pan-wizard 3.35.0 → 3.37.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +3645 -0
- package/README.md +46 -52
- package/agents/pan-conductor.md +2 -3
- package/agents/pan-counterfactual.md +1 -2
- package/agents/pan-debugger.md +14 -3
- package/agents/pan-design-checker.md +1 -2
- package/agents/pan-designer.md +2 -3
- package/agents/pan-document_code.md +1 -2
- package/agents/pan-executor.md +25 -0
- package/agents/pan-hardener.md +4 -3
- package/agents/pan-integration-checker.md +1 -2
- package/agents/pan-knowledge.md +1 -2
- package/agents/pan-meta-reviewer.md +3 -3
- package/agents/pan-optimizer.md +4 -4
- package/agents/pan-phase-researcher.md +80 -475
- package/agents/pan-plan-checker.md +40 -3
- package/agents/pan-planner.md +21 -7
- package/agents/pan-previewer.md +3 -2
- package/agents/pan-project-researcher.md +72 -493
- package/agents/pan-release.md +1 -1
- package/agents/pan-reviewer.md +14 -3
- package/agents/pan-roadmapper.md +12 -7
- package/agents/pan-verifier.md +9 -4
- package/bin/install-lib.cjs +134 -14
- package/bin/install.js +59 -12
- package/commands/pan/army.md +1 -1
- package/commands/pan/autodev.md +76 -0
- package/commands/pan/cost.md +21 -1
- package/commands/pan/exec-phase.md +3 -4
- package/commands/pan/focus-auto.md +14 -7
- package/commands/pan/focus-design.md +4 -5
- package/commands/pan/focus-docs.md +24 -0
- package/commands/pan/learn.md +13 -6
- package/commands/pan/milestone-audit.md +8 -1
- package/commands/pan/milestone-done.md +10 -5
- package/commands/pan/milestone-new.md +2 -2
- package/commands/pan/new-project.md +15 -12
- package/commands/pan/plan-phase.md +15 -9
- package/commands/pan/preview.md +1 -1
- package/commands/pan/quick.md +1 -1
- package/commands/pan/review-deep.md +11 -5
- package/commands/pan/settings.md +6 -0
- package/commands/pan/verify-phase.md +6 -2
- package/docs/AGENTS.md +917 -0
- package/docs/ARCHITECTURE.md +1054 -0
- package/docs/CLI-REFERENCE.md +4582 -0
- package/docs/COMPARISON.md +116 -0
- package/docs/DEVELOPMENT.md +490 -0
- package/docs/ECOSYSTEM-REVIEW-2026-06.md +107 -0
- package/docs/ECOSYSTEM-REVIEW-2026-08.md +349 -0
- package/docs/EXAMPLES.md +584 -0
- package/docs/FAQ.md +254 -0
- package/docs/FIELD-HARVEST-2026-07.md +88 -0
- package/docs/FIELD-REPORT-army-2026-06.md +121 -0
- package/docs/HOOKS.md +638 -0
- package/docs/IMPROVEMENT-TODO.md +246 -0
- package/docs/INTERNALS.md +485 -0
- package/docs/MIGRATION-v2-to-v3.md +258 -0
- package/docs/SECURITY-HARDENING.md +230 -0
- package/docs/SKILLS-ENHANCEMENT-PLAN-V2.md +122 -0
- package/docs/SKILLS-FULL-TEXT.md +9685 -0
- package/docs/SKILLS-REFERENCE.md +1037 -0
- package/docs/TROUBLESHOOTING.md +1084 -0
- package/docs/USER-GUIDE.md +1437 -0
- package/docs/archive/SKILLS-ENHANCEMENT-PLAN.md +230 -0
- package/docs/audits/AUDIT-2026-08-e2e-ROADMAP.md +117 -0
- package/docs/audits/AUDIT-2026-08-e2e-VERIFICATION.md +693 -0
- package/docs/audits/AUDIT-2026-08-e2e.md +1546 -0
- package/docs/audits/OUTSTANDING-2026-08.md +275 -0
- package/docs/audits/STABILITY-TEST-2026-08.md +640 -0
- package/docs/branding/README-brand-draft.md +119 -0
- package/docs/branding/handoff/IMPLEMENTATION.md +205 -0
- package/docs/branding/handoff/README-brand-draft.md +119 -0
- package/docs/branding/handoff/image-prompts.md +46 -0
- package/docs/branding/image-prompts.md +46 -0
- package/docs/branding/nano-banana-prompts.txt +60 -0
- package/docs/context-monitor.md +13 -0
- package/docs/decisions/ADR-0001-strategic-feature-prioritization.md +83 -0
- package/docs/decisions/ADR-0002-copilot-cli-runtime.md +71 -0
- package/docs/decisions/ADR-0003-smart-execution.md +84 -0
- package/docs/decisions/ADR-0004-copilot-cli-interaction.md +89 -0
- package/docs/decisions/ADR-0005-command-naming.md +80 -0
- package/docs/decisions/ADR-0006-focus-commands.md +101 -0
- package/docs/decisions/ADR-0007-ship-v1.md +73 -0
- package/docs/decisions/ADR-0008-map-codebase-mermaid-togaf.md +80 -0
- package/docs/decisions/ADR-0009-production-deployment-checklist.md +68 -0
- package/docs/decisions/ADR-0010-standards-integration.md +79 -0
- package/docs/decisions/ADR-0011-help-commands-rewrite.md +64 -0
- package/docs/decisions/ADR-0012-contributors-attribution-index.md +42 -0
- package/docs/decisions/ADR-0013-strategic-feature-additions.md +53 -0
- package/docs/decisions/ADR-0014-internal-cleanup-code-quality.md +53 -0
- package/docs/decisions/ADR-0015-focus-auto-runner.md +296 -0
- package/docs/decisions/ADR-0016-enhanced-doc-sync.md +54 -0
- package/docs/decisions/ADR-0017-enhanced-focus-design.md +69 -0
- package/docs/decisions/ADR-0018-user-reality-testing.md +84 -0
- package/docs/decisions/ADR-0019-end-to-end-user-acceptance-testing.md +113 -0
- package/docs/decisions/ADR-0020-ai-drift-prevention.md +55 -0
- package/docs/decisions/ADR-0021-codebase-mapper-v2.md +76 -0
- package/docs/decisions/ADR-0022-lifecycle-completeness.md +63 -0
- package/docs/decisions/ADR-0023-opus-4-7-adoption.md +89 -0
- package/docs/decisions/ADR-0024-spec-b-v2-completion.md +137 -0
- package/docs/decisions/ADR-0025-behavioral-guardrails-layer.md +129 -0
- package/docs/decisions/ADR-0026-self-improvement-loop.md +171 -0
- package/docs/decisions/ADR-0027-doc-code-link-graph.md +164 -0
- package/docs/decisions/ADR-0028-unified-agents-skills-target.md +46 -0
- package/docs/decisions/ADR-0029-experiment-evaluator.md +76 -0
- package/docs/decisions/ADR-0030-planning-concurrency.md +37 -0
- package/docs/decisions/ADR-0031-focus-auto-backlog-source.md +47 -0
- package/docs/decisions/ADR-0032-squad-model.md +64 -0
- package/docs/decisions/ADR-0033-army-campaign.md +40 -0
- package/docs/decisions/ADR-0034-scheduled-campaigns.md +57 -0
- package/docs/decisions/ADR-0035-army-hud-dashboard.md +40 -0
- package/docs/decisions/ADR-0036-agentic-memory-architecture.md +106 -0
- package/docs/decisions/ADR-0037-pin-security-agents-to-opus.md +51 -0
- package/docs/decisions/ADR-0038-skill-aligned-decomposition.md +41 -0
- package/docs/decisions/ADR-0039-worktree-as-project-variant.md +37 -0
- package/docs/decisions/ADR-0040-memory-injection-defense.md +55 -0
- package/docs/decisions/ADR-0041-mcp-dual-era-bridge.md +47 -0
- package/docs/decisions/ADR-0042-design-parity-both-flows.md +45 -0
- package/docs/decisions/ADR-0043-addressable-planning-roots.md +56 -0
- package/docs/decisions/ADR-0044-bound-the-cached-context.md +104 -0
- package/docs/decisions/ADR-0045-agent-plugins-bundle.md +76 -0
- package/docs/decisions/ADR-0046-subagent-frontmatter-decisions.md +39 -0
- package/docs/decisions/ADR-0047-pan-harness.md +76 -0
- package/docs/decisions/ADR-0048-one-core-per-project.md +43 -0
- package/docs/decisions/ADR-0049-machine-verdicts-and-findings-ledger.md +116 -0
- package/docs/specs/ai_drift_prevention_featureai.md +286 -0
- package/docs/specs/codebase_mapper_v2_featureai.md +930 -0
- package/docs/specs/command_naming_optimization_featureai.md +930 -0
- package/docs/specs/comp_analysis_markdown_v1.md +149 -0
- package/docs/specs/continuous_autonomous_execution_featureai.md +304 -0
- package/docs/specs/contributors_attribution_index_featureai.md +476 -0
- package/docs/specs/copilot_cli_interaction_featureai.md +868 -0
- package/docs/specs/copilot_cli_runtime_featureai.md +647 -0
- package/docs/specs/custom-skills-assessment.md +461 -0
- package/docs/specs/deployment_audit_findings_featureai.md +243 -0
- package/docs/specs/doc_code_link_graph_featureai.md +253 -0
- package/docs/specs/documentation_alignment_cleanup_featureai.md +69 -0
- package/docs/specs/e2e_visual_testing_featureai.md +615 -0
- package/docs/specs/end_to_end_user_acceptance_testing_featureai.md +810 -0
- package/docs/specs/evidence-loop-plan.md +76 -0
- package/docs/specs/evidence_loop_featureai.md +497 -0
- package/docs/specs/focus-auto-optimize-category.md +449 -0
- package/docs/specs/focus-auto-optimize-superplan.md +188 -0
- package/docs/specs/focus_auto_runner_featureai.md +1102 -0
- package/docs/specs/focus_design_v2_featureai.md +197 -0
- package/docs/specs/focus_sync_v2_featureai.md +202 -0
- package/docs/specs/git_integration_overhaul_featureai.md +326 -0
- package/docs/specs/glm_zai_provider_featureai.md +200 -0
- package/docs/specs/googlecli_adoption_featureai.md +661 -0
- package/docs/specs/help_commands_rewrite_featureai.md +860 -0
- package/docs/specs/industry_analysis_strategic_additions_featureai.md +231 -0
- package/docs/specs/industry_comparison_and_feature_gaps_featureai.md +674 -0
- package/docs/specs/internal_cleanup_code_quality_featureai.md +486 -0
- package/docs/specs/investigations/README.md +31 -0
- package/docs/specs/investigations/digests/2026-09-28.md +47 -0
- package/docs/specs/investigations/digests/2026-10-08.md +62 -0
- package/docs/specs/investigations/digests/2026-10-10.md +61 -0
- package/docs/specs/investigations/features/debugger-red-loop-first.md +55 -0
- package/docs/specs/investigations/features/harness-cross-host-behaviour.md +93 -0
- package/docs/specs/investigations/features/optimize-apply-regression-gate.md +66 -0
- package/docs/specs/investigations/features/reviewer-smell-baseline.md +57 -0
- package/docs/specs/investigations/features/trace-failure-capture.md +69 -0
- package/docs/specs/lifecycle_completeness_featureai.md +754 -0
- package/docs/specs/local-first-ci.md +42 -0
- package/docs/specs/map_codebase_mermaid_togaf_featureai.md +805 -0
- package/docs/specs/market-delta-2026-09-superplan.md +310 -0
- package/docs/specs/market-ideas-2026-09.md +131 -0
- package/docs/specs/market-ideas-2026-10.md +428 -0
- package/docs/specs/market-ideas-ledger.md +177 -0
- package/docs/specs/memory-optimization-2026-10.md +285 -0
- package/docs/specs/multi-track-planning-roots.md +143 -0
- package/docs/specs/multi_model_routing_featureai.md +687 -0
- package/docs/specs/multi_model_routing_superplan.md +197 -0
- package/docs/specs/opus_47_existing_enhancements_featureai.md +247 -0
- package/docs/specs/opus_47_extended_features_featureai.md +363 -0
- package/docs/specs/opus_47_extended_features_v2_featureai.md +397 -0
- package/docs/specs/pan_focus_commands_featureai.md +2336 -0
- package/docs/specs/pan_zcode_mcp_bridge_featureai.md +120 -0
- package/docs/specs/product-excellence-superplan.md +315 -0
- package/docs/specs/product_excellence_program_2026_q4_featureai.md +365 -0
- package/docs/specs/production_deployment_readiness_featureai.md +433 -0
- package/docs/specs/production_readiness_ship_v1_featureai.md +698 -0
- package/docs/specs/real_test_enforcement_featureai.md +213 -0
- package/docs/specs/reality-check-2026-09-superplan.md +362 -0
- package/docs/specs/right-sized-ceremony.md +74 -0
- package/docs/specs/roadmap_and_competition_2026_q4_featureai.md +390 -0
- package/docs/specs/self_improvement_loop_featureai.md +875 -0
- package/docs/specs/skill-aligned-decomposition.md +155 -0
- package/docs/specs/standards_integration_featureai.md +833 -0
- package/docs/specs/testing-system-redesign-2026-09.md +267 -0
- package/docs/specs/user_reality_testing_system_featureai.md +788 -0
- package/docs/specs/watch-triggers.json +22 -0
- package/docs/specs/workflow-integration-redesign.md +742 -0
- package/docs/specs/workflow_state_consistency_featureai.md +183 -0
- package/hooks/dist/pan-context-monitor.js +9 -2
- package/hooks/dist/pan-cost-logger.js +5 -0
- package/hooks/dist/pan-guard.js +28 -5
- package/hooks/dist/pan-stop-guard.js +9 -9
- package/hooks/dist/pan-trace-logger.js +5 -0
- package/package.json +10 -2
- package/pan-wizard-core/bin/lib/agents-md.cjs +57 -8
- package/pan-wizard-core/bin/lib/autodev-report.cjs +180 -0
- package/pan-wizard-core/bin/lib/autodev.cjs +642 -0
- package/pan-wizard-core/bin/lib/ci.cjs +143 -0
- package/pan-wizard-core/bin/lib/commands.cjs +1 -1
- package/pan-wizard-core/bin/lib/config.cjs +41 -5
- package/pan-wizard-core/bin/lib/constants.cjs +27 -1
- package/pan-wizard-core/bin/lib/core.cjs +31 -14
- package/pan-wizard-core/bin/lib/cost.cjs +160 -4
- package/pan-wizard-core/bin/lib/cross-family.cjs +164 -0
- package/pan-wizard-core/bin/lib/debug-cases.cjs +104 -0
- package/pan-wizard-core/bin/lib/doc-lint.cjs +5 -3
- package/pan-wizard-core/bin/lib/failure-class.cjs +81 -0
- package/pan-wizard-core/bin/lib/findings.cjs +148 -9
- package/pan-wizard-core/bin/lib/focus.cjs +1 -1
- package/pan-wizard-core/bin/lib/git.cjs +1 -1
- package/pan-wizard-core/bin/lib/hosts.cjs +88 -0
- package/pan-wizard-core/bin/lib/init.cjs +7 -2
- package/pan-wizard-core/bin/lib/knowledge.cjs +2 -0
- package/pan-wizard-core/bin/lib/merge.cjs +50 -0
- package/pan-wizard-core/bin/lib/mutate.cjs +268 -0
- package/pan-wizard-core/bin/lib/next-action.cjs +112 -0
- package/pan-wizard-core/bin/lib/optimize.cjs +2 -2
- package/pan-wizard-core/bin/lib/phase-gate.cjs +15 -9
- package/pan-wizard-core/bin/lib/phase-remove.cjs +91 -11
- package/pan-wizard-core/bin/lib/phase.cjs +85 -13
- package/pan-wizard-core/bin/lib/plan-lint.cjs +310 -0
- package/pan-wizard-core/bin/lib/preview.cjs +26 -4
- package/pan-wizard-core/bin/lib/requirements-lint.cjs +130 -0
- package/pan-wizard-core/bin/lib/research-lint.cjs +268 -0
- package/pan-wizard-core/bin/lib/review-deep.cjs +14 -22
- package/pan-wizard-core/bin/lib/review-vote.cjs +105 -0
- package/pan-wizard-core/bin/lib/roadmap.cjs +1 -0
- package/pan-wizard-core/bin/lib/runner.cjs +2 -16
- package/pan-wizard-core/bin/lib/sizing.cjs +356 -0
- package/pan-wizard-core/bin/lib/squads.cjs +1 -1
- package/pan-wizard-core/bin/lib/state-write.cjs +247 -0
- package/pan-wizard-core/bin/lib/test-gate.cjs +433 -0
- package/pan-wizard-core/bin/lib/uat.cjs +149 -0
- package/pan-wizard-core/bin/lib/usage.cjs +104 -0
- package/pan-wizard-core/bin/lib/verdict.cjs +26 -3
- package/pan-wizard-core/bin/lib/verify-plan-set.cjs +30 -1
- package/pan-wizard-core/bin/lib/verify-preflight.cjs +2 -2
- package/pan-wizard-core/bin/lib/verify-red.cjs +174 -0
- package/pan-wizard-core/bin/lib/verify-retro.cjs +3 -2
- package/pan-wizard-core/bin/lib/verify-scope.cjs +143 -2
- package/pan-wizard-core/bin/lib/verify.cjs +86 -31
- package/pan-wizard-core/bin/lib/worktree.cjs +70 -0
- package/pan-wizard-core/bin/pan-tools.cjs +257 -28
- package/pan-wizard-core/learnings/universal/autonomous-loop.md +2 -2
- package/pan-wizard-core/mcp/orchestrator.cjs +4 -1
- package/pan-wizard-core/references/continuation-format.md +3 -3
- package/pan-wizard-core/references/design-methodology.md +1 -1
- package/pan-wizard-core/references/guardrails.md +1 -1
- package/pan-wizard-core/references/handoff-decisions.md +1 -1
- package/pan-wizard-core/references/model-profiles.md +2 -3
- package/pan-wizard-core/references/questioning.md +24 -0
- package/pan-wizard-core/references/ui-brand.md +4 -6
- package/pan-wizard-core/references/verdict-contract.md +2 -1
- package/pan-wizard-core/templates/config.json +10 -2
- package/pan-wizard-core/templates/debug.md +1 -0
- package/pan-wizard-core/templates/design.md +5 -4
- package/pan-wizard-core/templates/discovery.md +2 -2
- package/pan-wizard-core/templates/phase-prompt.md +7 -0
- package/pan-wizard-core/templates/requirements.md +9 -0
- package/pan-wizard-core/templates/research.md +10 -0
- package/pan-wizard-core/templates/summary-complex.md +5 -1
- package/pan-wizard-core/templates/summary-minimal.md +4 -0
- package/pan-wizard-core/templates/summary-standard.md +5 -1
- package/pan-wizard-core/templates/summary.md +8 -1
- package/pan-wizard-core/workflows/add-phase.md +4 -6
- package/pan-wizard-core/workflows/assumptions.md +19 -1
- package/pan-wizard-core/workflows/diagnose-issues.md +2 -0
- package/pan-wizard-core/workflows/discuss-phase.md +4 -2
- package/pan-wizard-core/workflows/exec-lean.md +69 -0
- package/pan-wizard-core/workflows/exec-phase.md +79 -15
- package/pan-wizard-core/workflows/execute-plan.md +3 -1
- package/{commands/pan → pan-wizard-core/workflows}/focus-doc-audit.md +13 -29
- package/{commands/pan → pan-wizard-core/workflows}/focus-drift-walking.md +3 -18
- package/{commands/pan → pan-wizard-core/workflows}/focus-exec.md +8 -23
- package/{commands/pan → pan-wizard-core/workflows}/focus-plan.md +6 -20
- package/{commands/pan → pan-wizard-core/workflows}/focus-scan.md +3 -16
- package/{commands/pan → pan-wizard-core/workflows}/focus-sync.md +7 -21
- package/pan-wizard-core/workflows/health.md +6 -0
- package/pan-wizard-core/workflows/help.md +114 -77
- package/pan-wizard-core/workflows/insert-phase.md +4 -6
- package/pan-wizard-core/workflows/learn.md +3 -3
- package/pan-wizard-core/workflows/milestone-audit.md +2 -2
- package/pan-wizard-core/workflows/milestone-new.md +38 -95
- package/pan-wizard-core/workflows/new-project.md +96 -258
- package/pan-wizard-core/workflows/optimize.md +107 -5
- package/pan-wizard-core/workflows/phase-tests.md +10 -0
- package/pan-wizard-core/workflows/plan-phase.md +106 -45
- package/pan-wizard-core/workflows/profile.md +50 -0
- package/pan-wizard-core/workflows/progress.md +10 -0
- package/pan-wizard-core/workflows/quick.md +34 -43
- package/pan-wizard-core/workflows/remove-phase.md +11 -1
- package/pan-wizard-core/workflows/research-phase.md +118 -40
- package/pan-wizard-core/workflows/resume-project.md +18 -2
- package/pan-wizard-core/workflows/settings.md +33 -19
- package/pan-wizard-core/workflows/todo-check.md +6 -2
- package/pan-wizard-core/workflows/transition.md +22 -4
- package/pan-wizard-core/workflows/uat-walkthrough.md +90 -0
- package/pan-wizard-core/workflows/update.md +1 -1
- package/pan-wizard-core/workflows/verify-phase.md +39 -46
- package/scripts/build-agent-plugin.js +35 -15
- package/scripts/build-help.cjs +106 -0
- package/scripts/build-plugin.js +10 -1
- package/scripts/deprecate-old-versions.js +70 -3
- package/scripts/generate-skills-docs.py +1 -1
- package/scripts/git-hooks/pre-push +27 -0
- package/scripts/install-git-hooks.js +7 -5
- package/scripts/install-pre-push.cjs +38 -0
- package/scripts/land.cjs +151 -0
- package/scripts/local-ci.cjs +501 -0
- package/scripts/local-schedule.cjs +277 -0
- package/scripts/mutation-probe.cjs +7 -80
- package/scripts/prompt-lint.cjs +32 -0
- package/scripts/prompt-rules.cjs +156 -0
- package/scripts/run-tests.cjs +7 -1
- package/scripts/test-quality-lint.cjs +3 -3
- package/scripts/test-surface.cjs +175 -6
- package/agents/pan-research-synthesizer.md +0 -246
- package/commands/pan/audit-deployment.md +0 -386
- package/commands/pan/dashboard.md +0 -22
- package/commands/pan/discord.md +0 -19
- package/commands/pan/mcp-bridge.md +0 -143
- package/commands/pan/milestone-cleanup.md +0 -19
- package/commands/pan/milestone-gaps.md +0 -35
- package/commands/pan/optimize.md +0 -115
- package/commands/pan/profile.md +0 -76
- package/commands/pan/research-phase.md +0 -189
- package/pan-wizard-core/bin/lib/bridge.cjs +0 -269
- package/pan-wizard-core/templates/research-project/architecture.md +0 -204
- package/pan-wizard-core/templates/research-project/features.md +0 -147
- package/pan-wizard-core/templates/research-project/pitfalls.md +0 -200
- package/pan-wizard-core/templates/research-project/stack.md +0 -120
- package/pan-wizard-core/templates/research-project/summary.md +0 -170
|
@@ -0,0 +1,687 @@
|
|
|
1
|
+
# /featureAI — Multi-Model Routing Within Agent Workflows
|
|
2
|
+
|
|
3
|
+
**Source:** `docs/specs/comp_analysis_markdown_v1.md` — Enhancement #2
|
|
4
|
+
|
|
5
|
+
> "The research shows 30–70% cost reduction is achievable through intelligent model routing. PAN's 12 agents should support configurable model assignment — reasoning models (Claude Opus, GPT-5) for architecture and planning agents, mid-tier models (Sonnet, GPT-4.5) for coding agents, lightweight models (Haiku, GPT-4.5-mini) for verification and formatting agents. Stanford's FrugalGPT research demonstrated up to 98% cost reduction with cascade routing patterns."
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## Phase 0: Problem Framing
|
|
10
|
+
|
|
11
|
+
### 0.1 Problem Statement
|
|
12
|
+
|
|
13
|
+
PAN Wizard currently has a 3-tier model profile system (quality/balanced/budget) with per-agent overrides, but it is Anthropic-only (opus/sonnet/haiku) and static — the same model is used for every invocation of a given agent regardless of task complexity. As PAN expands to support multiple LLM providers (OpenAI, Google, local models) and as model pricing varies dramatically (Opus is ~75× more expensive than Haiku per token), users need intelligent routing that assigns the right model to the right agent for the right task. The competitive analysis shows 30–70% cost reduction is achievable through intelligent model routing, and Stanford's FrugalGPT demonstrated up to 98% reduction with cascade patterns. Without this, PAN users either overspend on quality profiles or sacrifice output quality on budget profiles — there is no middle ground that adapts per-invocation.
|
|
14
|
+
|
|
15
|
+
### 0.2 Scope
|
|
16
|
+
|
|
17
|
+
| In Scope | Out of Scope |
|
|
18
|
+
|----------|--------------|
|
|
19
|
+
| Multi-provider model identifiers in MODEL_PROFILES | Direct API calls to any LLM provider (PAN delegates to the host runtime) |
|
|
20
|
+
| Complexity-aware routing hints (task metadata → model tier) | Automatic model quality benchmarking |
|
|
21
|
+
| Cascade routing: try cheaper model first, escalate on failure | Token counting or cost tracking dashboards |
|
|
22
|
+
| User-configurable routing rules in config.json | Training or fine-tuning models |
|
|
23
|
+
| Provider-agnostic model aliases (reasoning/mid-tier/fast) | Rate limiting or quota management (host runtime's job) |
|
|
24
|
+
| Cost estimation display before execution | A2A or MCP protocol integration (separate feature) |
|
|
25
|
+
| Per-phase model override (e.g., critical phase → quality) | Real-time model switching mid-agent-execution |
|
|
26
|
+
| Runtime-specific model name resolution | Support for non-text modalities (vision, audio) |
|
|
27
|
+
|
|
28
|
+
### 0.3 Success Criteria (Measurable)
|
|
29
|
+
|
|
30
|
+
```
|
|
31
|
+
SC-1: MODEL_PROFILES supports provider-agnostic tier aliases (reasoning, mid, fast) alongside specific model names
|
|
32
|
+
SC-2: resolveModelInternal() resolves aliases to runtime-specific model names based on detected or configured provider
|
|
33
|
+
SC-3: config.json supports routing_strategy: "static" (current behavior), "complexity", or "cascade"
|
|
34
|
+
SC-4: Complexity routing uses task metadata (file count, wave count, requirement count) to select model tier
|
|
35
|
+
SC-5: Cascade routing attempts fast-tier first and escalates to mid/reasoning on structured failure detection
|
|
36
|
+
SC-6: /pan:profile command shows estimated cost multiplier per profile
|
|
37
|
+
SC-7: Per-phase model override via roadmap.md frontmatter (e.g., `model_tier: reasoning`)
|
|
38
|
+
SC-8: All 5 runtimes work correctly with the new routing (Claude Code, Codex, Gemini CLI, OpenCode, Copilot CLI)
|
|
39
|
+
SC-9: Existing 3-profile system (quality/balanced/budget) remains fully backward-compatible
|
|
40
|
+
SC-10: Unit tests cover all routing strategies, alias resolution, cascade logic, and per-phase overrides
|
|
41
|
+
SC-11: No regression in existing 1658 tests
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
### 0.4 User Stories
|
|
45
|
+
|
|
46
|
+
```
|
|
47
|
+
US-1: As a solo developer on a tight budget, I want PAN to automatically use
|
|
48
|
+
cheaper models for simple verification and formatting agents,
|
|
49
|
+
so that I can run full workflows without burning through my quota,
|
|
50
|
+
instead of manually toggling between quality and budget profiles.
|
|
51
|
+
|
|
52
|
+
US-2: As a team lead on a complex architecture phase, I want to mark specific
|
|
53
|
+
phases as "critical" so PAN uses reasoning-tier models for those phases,
|
|
54
|
+
so that architecture decisions get maximum quality,
|
|
55
|
+
instead of running everything at the same tier.
|
|
56
|
+
|
|
57
|
+
US-3: As a developer using Codex CLI (OpenAI), I want PAN's model profiles
|
|
58
|
+
to map to OpenAI model names automatically,
|
|
59
|
+
so that agent model selection works without manual configuration,
|
|
60
|
+
instead of the current Anthropic-only opus/sonnet/haiku hardcoding.
|
|
61
|
+
|
|
62
|
+
US-4: As a cost-conscious enterprise user, I want PAN to try a cheaper model
|
|
63
|
+
first and only escalate to expensive models when the cheap model fails,
|
|
64
|
+
so that 80%+ of routine tasks run at minimum cost,
|
|
65
|
+
instead of paying reasoning-tier prices for formatting and boilerplate.
|
|
66
|
+
|
|
67
|
+
US-5: As a PAN user about to run exec-phase on 3 plans, I want to see an
|
|
68
|
+
estimated cost comparison between profiles before committing,
|
|
69
|
+
so that I can make informed decisions about quality vs. spend,
|
|
70
|
+
instead of guessing.
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
---
|
|
74
|
+
|
|
75
|
+
## Phase 1: Internal Reconnaissance
|
|
76
|
+
|
|
77
|
+
### 1.1 Existing Capabilities
|
|
78
|
+
|
|
79
|
+
PAN already has a solid foundation for model routing:
|
|
80
|
+
|
|
81
|
+
| Component | Location | Current State |
|
|
82
|
+
|-----------|----------|--------------|
|
|
83
|
+
| `MODEL_PROFILES` constant | `pan-wizard-core/bin/lib/core.cjs:30-42` | 12 agents × 3 profiles (quality/balanced/budget) → opus/sonnet/haiku |
|
|
84
|
+
| `resolveModelInternal()` | `pan-wizard-core/bin/lib/core.cjs:497-511` | Checks `model_overrides` → falls back to profile lookup → maps opus→"inherit" |
|
|
85
|
+
| `cmdResolveModel()` | `pan-wizard-core/bin/lib/commands.cjs:274-295` | CLI wrapper: `pan-tools.cjs resolve-model <agent-type>` |
|
|
86
|
+
| `buildConfigDefaults()` | `pan-wizard-core/bin/lib/config.cjs:46-78` | Hardcoded defaults including `model_profile: "balanced"` |
|
|
87
|
+
| `/pan:profile` command | `commands/pan/profile.md` | Switches profile (quality/balanced/budget) with downgrade confirmation |
|
|
88
|
+
| `/pan:settings` command | `commands/pan/settings.md` | Interactive 7-question config including model profile |
|
|
89
|
+
| `model-profiles.md` reference | `pan-wizard-core/references/model-profiles.md` | Documentation consumed by agents explaining the profile system |
|
|
90
|
+
| Per-agent overrides | `config.json → model_overrides` | `{"pan-executor": "opus"}` — already supports per-agent override |
|
|
91
|
+
| Profile display | `pan-wizard-core/workflows/profile.md` | Shows model table after profile switch |
|
|
92
|
+
|
|
93
|
+
### 1.2 Key Design Constraints Discovered
|
|
94
|
+
|
|
95
|
+
1. **opus → "inherit" mapping**: When an agent resolves to "opus", it returns `"inherit"` so the host runtime uses its native top-tier model. This is already provider-agnostic for the top tier.
|
|
96
|
+
2. **Three valid model returns**: `"inherit"`, `"sonnet"`, `"haiku"` — these are passed directly to the Task tool's `model` parameter. The host runtime interprets them.
|
|
97
|
+
3. **Config merge pattern**: `buildConfigDefaults()` deep-merges user defaults with hardcoded defaults. New config keys must follow this pattern.
|
|
98
|
+
4. **Zero dependencies**: All routing logic must use Node.js builtins only.
|
|
99
|
+
5. **CommonJS required**: All `.cjs` modules — no ES module syntax.
|
|
100
|
+
|
|
101
|
+
### 1.3 Runtime Compatibility Analysis
|
|
102
|
+
|
|
103
|
+
| Runtime | Model Parameter Support | Native Model Names | Notes |
|
|
104
|
+
|---------|------------------------|-------------------|-------|
|
|
105
|
+
| Claude Code | `model` param in Task tool | claude-opus-4-0520, claude-sonnet-4-0514, claude-haiku-3-5-20241022 | Maps "inherit"/"sonnet"/"haiku" internally |
|
|
106
|
+
| Codex CLI | Model selection in config | gpt-5, gpt-4.5, gpt-4.5-mini | Would need alias mapping |
|
|
107
|
+
| Gemini CLI | Model selection in config | gemini-2.5-pro, gemini-2.5-flash, gemini-2.0-flash-lite | Would need alias mapping |
|
|
108
|
+
| OpenCode | Model selection in config | Supports multiple providers | Provider-dependent naming |
|
|
109
|
+
| Copilot CLI | Model selection in config | gpt-4o, claude-sonnet, etc. | Multi-provider via GitHub |
|
|
110
|
+
|
|
111
|
+
**Key insight**: Only Claude Code currently uses PAN's `resolve-model` output to set the Task tool's `model` parameter. The other 4 runtimes use their own model selection. This means the multi-provider routing primarily affects documentation and future API integration — the immediate value is in complexity-aware and cascade routing within the Anthropic tier.
|
|
112
|
+
|
|
113
|
+
---
|
|
114
|
+
|
|
115
|
+
## Phase 2: Competitive Analysis
|
|
116
|
+
|
|
117
|
+
### How Competitors Handle Model Routing
|
|
118
|
+
|
|
119
|
+
| Tool | Approach | Cost Savings Claimed | Limitations |
|
|
120
|
+
|------|----------|---------------------|-------------|
|
|
121
|
+
| **Cursor** | Per-request model dropdown + auto-routing in background | Not disclosed | Manual selection, no task-based routing |
|
|
122
|
+
| **Cline** | BYOK — user picks model per session | N/A (user manages) | No intelligent routing |
|
|
123
|
+
| **Aider** | `--model` flag, architect mode uses separate model | Community-reported savings | Two-model max (architect + editor) |
|
|
124
|
+
| **Continue.dev** | Model selection per action type (chat/edit/autocomplete) | Not disclosed | Action-type routing only, not complexity-based |
|
|
125
|
+
| **Roo Code** | Mode-specific model assignment (Architect/Code/Debug) | Not disclosed | 5 modes, static assignment |
|
|
126
|
+
| **Factory AI** | Proprietary multi-model orchestration | Claims significant savings | Closed source, enterprise only |
|
|
127
|
+
| **FrugalGPT** (Stanford) | Cascade: cheap → expensive on failure | Up to 98% cost reduction | Research paper, not productized |
|
|
128
|
+
| **Martian** | Model router API with quality prediction | 30-70% reduction | External dependency, not open-source |
|
|
129
|
+
| **OpenRouter** | Smart routing based on prompt characteristics | Varies by config | Third-party API dependency |
|
|
130
|
+
|
|
131
|
+
### Differentiation Opportunity
|
|
132
|
+
|
|
133
|
+
No open-source AI coding tool currently offers:
|
|
134
|
+
1. **Per-agent complexity-aware routing** — routing based on the specific task's complexity, not just the agent type
|
|
135
|
+
2. **Cascade execution** — try cheaper model, escalate on structured failure
|
|
136
|
+
3. **Provider-agnostic tier aliases** — abstract model names that resolve per-runtime
|
|
137
|
+
4. **Cost estimation before execution** — show comparative cost before committing to a profile
|
|
138
|
+
|
|
139
|
+
PAN can be first-to-market in the open-source space with all four.
|
|
140
|
+
|
|
141
|
+
---
|
|
142
|
+
|
|
143
|
+
## Phase 3: Design
|
|
144
|
+
|
|
145
|
+
### Architecture Decision
|
|
146
|
+
|
|
147
|
+
The feature is implemented as an **extension of the existing model resolution pipeline**, not a replacement. The current 3-profile system becomes a special case of the more general routing system.
|
|
148
|
+
|
|
149
|
+
### 3.1 Model Tier Aliases
|
|
150
|
+
|
|
151
|
+
Introduce provider-agnostic tier aliases that replace hardcoded model names:
|
|
152
|
+
|
|
153
|
+
```
|
|
154
|
+
reasoning → The provider's most capable model (opus, gpt-5, gemini-2.5-pro)
|
|
155
|
+
mid → The provider's balanced model (sonnet, gpt-4.5, gemini-2.5-flash)
|
|
156
|
+
fast → The provider's cheapest model (haiku, gpt-4.5-mini, gemini-2.0-flash-lite)
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
**Backward compatibility**: `opus`, `sonnet`, `haiku` continue to work as direct aliases for `reasoning`, `mid`, `fast` in the Anthropic provider.
|
|
160
|
+
|
|
161
|
+
### 3.2 Provider Resolution
|
|
162
|
+
|
|
163
|
+
```javascript
|
|
164
|
+
const PROVIDER_MODELS = {
|
|
165
|
+
anthropic: { reasoning: 'inherit', mid: 'sonnet', fast: 'haiku' },
|
|
166
|
+
openai: { reasoning: 'inherit', mid: 'mid', fast: 'fast' },
|
|
167
|
+
google: { reasoning: 'inherit', mid: 'mid', fast: 'fast' },
|
|
168
|
+
default: { reasoning: 'inherit', mid: 'sonnet', fast: 'haiku' },
|
|
169
|
+
};
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
Provider is detected from: (1) `config.json → provider` field, (2) runtime directory detection (`.claude/` → anthropic), (3) environment variable `PAN_PROVIDER`, (4) default to `anthropic`.
|
|
173
|
+
|
|
174
|
+
### 3.3 Updated MODEL_PROFILES
|
|
175
|
+
|
|
176
|
+
```javascript
|
|
177
|
+
const MODEL_PROFILES = {
|
|
178
|
+
'pan-planner': { quality: 'reasoning', balanced: 'reasoning', budget: 'mid' },
|
|
179
|
+
'pan-roadmapper': { quality: 'reasoning', balanced: 'mid', budget: 'mid' },
|
|
180
|
+
'pan-executor': { quality: 'reasoning', balanced: 'mid', budget: 'mid' },
|
|
181
|
+
'pan-phase-researcher': { quality: 'reasoning', balanced: 'mid', budget: 'fast' },
|
|
182
|
+
'pan-project-researcher': { quality: 'reasoning', balanced: 'mid', budget: 'fast' },
|
|
183
|
+
'pan-research-synthesizer': { quality: 'reasoning', balanced: 'mid', budget: 'fast' },
|
|
184
|
+
'pan-debugger': { quality: 'reasoning', balanced: 'mid', budget: 'mid' },
|
|
185
|
+
'pan-document_code': { quality: 'reasoning', balanced: 'fast', budget: 'fast' },
|
|
186
|
+
'pan-verifier': { quality: 'reasoning', balanced: 'mid', budget: 'fast' },
|
|
187
|
+
'pan-plan-checker': { quality: 'reasoning', balanced: 'mid', budget: 'fast' },
|
|
188
|
+
'pan-integration-checker': { quality: 'reasoning', balanced: 'mid', budget: 'fast' },
|
|
189
|
+
'pan-reviewer': { quality: 'reasoning', balanced: 'fast', budget: 'fast' },
|
|
190
|
+
};
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
### 3.4 Routing Strategies
|
|
194
|
+
|
|
195
|
+
#### Strategy 1: `static` (Default — Current Behavior)
|
|
196
|
+
|
|
197
|
+
Profile + per-agent override. No change from today.
|
|
198
|
+
|
|
199
|
+
#### Strategy 2: `complexity`
|
|
200
|
+
|
|
201
|
+
Task metadata determines whether to upgrade or downgrade from the profile's default tier:
|
|
202
|
+
|
|
203
|
+
```javascript
|
|
204
|
+
function resolveComplexityTier(baseTier, taskMetadata) {
|
|
205
|
+
const { fileCount, waveCount, requirementCount, isArchitectural } = taskMetadata;
|
|
206
|
+
|
|
207
|
+
const complexityScore =
|
|
208
|
+
(fileCount > 15 ? 2 : fileCount > 5 ? 1 : 0) +
|
|
209
|
+
(waveCount > 3 ? 2 : waveCount > 1 ? 1 : 0) +
|
|
210
|
+
(requirementCount > 5 ? 2 : requirementCount > 2 ? 1 : 0) +
|
|
211
|
+
(isArchitectural ? 3 : 0);
|
|
212
|
+
|
|
213
|
+
// Score 0-2: downgrade eligible (fast if base is mid, mid if base is reasoning)
|
|
214
|
+
// Score 3-5: use base tier
|
|
215
|
+
// Score 6+: upgrade eligible (mid if base is fast, reasoning if base is mid)
|
|
216
|
+
|
|
217
|
+
if (complexityScore <= 2 && baseTier !== 'fast') return downgrade(baseTier);
|
|
218
|
+
if (complexityScore >= 6 && baseTier !== 'reasoning') return upgrade(baseTier);
|
|
219
|
+
return baseTier;
|
|
220
|
+
}
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
Task metadata is extracted from the plan frontmatter (already available — `requirements`, `wave`, task list count).
|
|
224
|
+
|
|
225
|
+
#### Strategy 3: `cascade`
|
|
226
|
+
|
|
227
|
+
Try cheaper model first; escalate on structured failure:
|
|
228
|
+
|
|
229
|
+
```
|
|
230
|
+
1. Resolve base tier from profile
|
|
231
|
+
2. Downgrade one tier (reasoning→mid, mid→fast)
|
|
232
|
+
3. Execute agent with downgraded model
|
|
233
|
+
4. If agent output passes quality gate → done (saved cost)
|
|
234
|
+
5. If agent output fails quality gate → re-execute with base tier
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
**Quality gates** (simple heuristics, no external deps):
|
|
238
|
+
- Plan output: has required frontmatter keys, ≥N tasks, has success criteria
|
|
239
|
+
- Research output: has Sources section, has Confidence ratings, ≥N lines
|
|
240
|
+
- Verification output: has pass/fail verdict, checked all must-haves
|
|
241
|
+
- Executor: no error in commit, tests pass (already verified by PAN)
|
|
242
|
+
|
|
243
|
+
**Cascade is opt-in** and adds latency (potential double-execution). Recommended only for high-volume workflows.
|
|
244
|
+
|
|
245
|
+
### 3.5 Per-Phase Model Override
|
|
246
|
+
|
|
247
|
+
Allow roadmap.md phase sections to specify a model tier:
|
|
248
|
+
|
|
249
|
+
```markdown
|
|
250
|
+
## Phase 3: Authentication Architecture
|
|
251
|
+
<!-- model_tier: reasoning -->
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
When present, overrides the profile for ALL agents in that phase. Useful for flagging critical architecture phases.
|
|
255
|
+
|
|
256
|
+
### 3.6 Cost Estimation
|
|
257
|
+
|
|
258
|
+
Add cost multiplier display to `/pan:profile` and pre-execution summary:
|
|
259
|
+
|
|
260
|
+
```
|
|
261
|
+
Estimated cost multiplier vs budget:
|
|
262
|
+
quality: ~15× budget (Opus for all)
|
|
263
|
+
balanced: ~5× budget (Opus for planning, Sonnet elsewhere)
|
|
264
|
+
budget: 1× baseline (Sonnet + Haiku)
|
|
265
|
+
|
|
266
|
+
This phase (3 plans, 2 waves): ~4,200 tokens estimated
|
|
267
|
+
quality: ~$0.42
|
|
268
|
+
balanced: ~$0.18
|
|
269
|
+
budget: ~$0.04
|
|
270
|
+
```
|
|
271
|
+
|
|
272
|
+
Token estimates are rough (based on average agent consumption from PAN's own execution data, not real-time counting).
|
|
273
|
+
|
|
274
|
+
### 3.7 Config Schema Extension
|
|
275
|
+
|
|
276
|
+
```json
|
|
277
|
+
{
|
|
278
|
+
"model_profile": "balanced",
|
|
279
|
+
"model_overrides": {
|
|
280
|
+
"pan-executor": "reasoning"
|
|
281
|
+
},
|
|
282
|
+
"routing": {
|
|
283
|
+
"strategy": "static",
|
|
284
|
+
"provider": "auto",
|
|
285
|
+
"cascade_quality_gate": true,
|
|
286
|
+
"complexity_thresholds": {
|
|
287
|
+
"downgrade_max": 2,
|
|
288
|
+
"upgrade_min": 6
|
|
289
|
+
}
|
|
290
|
+
}
|
|
291
|
+
}
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
### Implementation Plan
|
|
295
|
+
|
|
296
|
+
```
|
|
297
|
+
1. pan-wizard-core/bin/lib/core.cjs
|
|
298
|
+
- Add PROVIDER_MODELS constant
|
|
299
|
+
- Update MODEL_PROFILES to use tier aliases (reasoning/mid/fast)
|
|
300
|
+
- Add detectProvider() function
|
|
301
|
+
- Add resolveTierToModel() function (alias → provider-specific model name)
|
|
302
|
+
- Add resolveComplexityTier() function
|
|
303
|
+
- Update resolveModelInternal() to support routing strategies
|
|
304
|
+
- Add estimateCostMultiplier() function
|
|
305
|
+
- Maintain backward compat: opus/sonnet/haiku still accepted in model_overrides
|
|
306
|
+
|
|
307
|
+
2. pan-wizard-core/bin/lib/config.cjs
|
|
308
|
+
- Update buildConfigDefaults() to include routing section
|
|
309
|
+
- Add routing config validation
|
|
310
|
+
|
|
311
|
+
3. pan-wizard-core/bin/lib/commands.cjs
|
|
312
|
+
- Update cmdResolveModel() to accept optional task metadata JSON
|
|
313
|
+
- Add cmdEstimateCost() for pre-execution cost display
|
|
314
|
+
|
|
315
|
+
4. pan-wizard-core/bin/pan-tools.cjs
|
|
316
|
+
- Wire resolve-model to accept --metadata flag
|
|
317
|
+
- Wire new estimate-cost command
|
|
318
|
+
|
|
319
|
+
5. pan-wizard-core/references/model-profiles.md
|
|
320
|
+
- Update with tier aliases, provider mapping, routing strategies
|
|
321
|
+
|
|
322
|
+
6. commands/pan/profile.md
|
|
323
|
+
- Add cost estimation display
|
|
324
|
+
|
|
325
|
+
7. pan-wizard-core/workflows/profile.md
|
|
326
|
+
- Add cost multiplier output step
|
|
327
|
+
|
|
328
|
+
8. pan-wizard-core/workflows/settings.md
|
|
329
|
+
- Add routing strategy question to interactive config
|
|
330
|
+
|
|
331
|
+
9. docs/USER-GUIDE.md
|
|
332
|
+
- Update Model Profiles section with new capabilities
|
|
333
|
+
|
|
334
|
+
10. docs/AGENTS.md
|
|
335
|
+
- Update Model Profiles section
|
|
336
|
+
|
|
337
|
+
11. docs/COMPARISON.md
|
|
338
|
+
- Update "Model Flexibility" row and "Where PAN Wizard Leads" section
|
|
339
|
+
|
|
340
|
+
12. docs/INTERNALS.md
|
|
341
|
+
- Document routing pipeline internals
|
|
342
|
+
|
|
343
|
+
13. tests/core.test.cjs
|
|
344
|
+
- New tests for tier alias resolution, provider detection, complexity routing,
|
|
345
|
+
cascade logic, per-phase override, backward compatibility
|
|
346
|
+
|
|
347
|
+
14. tests/config.test.cjs
|
|
348
|
+
- New tests for routing config defaults, validation
|
|
349
|
+
|
|
350
|
+
15. tests/commands.test.cjs
|
|
351
|
+
- New tests for resolve-model with metadata, estimate-cost command
|
|
352
|
+
```
|
|
353
|
+
|
|
354
|
+
### Test Plan
|
|
355
|
+
|
|
356
|
+
**Unit tests in `tests/core.test.cjs`:**
|
|
357
|
+
|
|
358
|
+
| # | Test | Type |
|
|
359
|
+
|---|------|------|
|
|
360
|
+
| 1 | PROVIDER_MODELS has entries for anthropic, openai, google, default | Happy path |
|
|
361
|
+
| 2 | MODEL_PROFILES uses tier aliases (reasoning/mid/fast) not model names | Happy path |
|
|
362
|
+
| 3 | resolveTierToModel('reasoning', 'anthropic') returns 'inherit' | Happy path |
|
|
363
|
+
| 4 | resolveTierToModel('mid', 'anthropic') returns 'sonnet' | Happy path |
|
|
364
|
+
| 5 | resolveTierToModel('fast', 'anthropic') returns 'haiku' | Happy path |
|
|
365
|
+
| 6 | resolveTierToModel('mid', 'openai') returns 'mid' | Happy path |
|
|
366
|
+
| 7 | Legacy 'opus' in model_overrides still resolves to 'inherit' | Backward compat |
|
|
367
|
+
| 8 | Legacy 'sonnet' in model_overrides still resolves to 'sonnet' | Backward compat |
|
|
368
|
+
| 9 | Legacy 'haiku' in model_overrides still resolves to 'haiku' | Backward compat |
|
|
369
|
+
| 10 | detectProvider() returns 'anthropic' when .claude/ exists | Happy path |
|
|
370
|
+
| 11 | detectProvider() returns configured provider when set in config | Happy path |
|
|
371
|
+
| 12 | detectProvider() returns 'default' when no runtime detected | Edge case |
|
|
372
|
+
| 13 | resolveComplexityTier downgrades on low complexity (score ≤2) | Happy path |
|
|
373
|
+
| 14 | resolveComplexityTier maintains tier on medium complexity (3-5) | Happy path |
|
|
374
|
+
| 15 | resolveComplexityTier upgrades on high complexity (score ≥6) | Happy path |
|
|
375
|
+
| 16 | resolveComplexityTier never downgrades 'fast' (already lowest) | Edge case |
|
|
376
|
+
| 17 | resolveComplexityTier never upgrades 'reasoning' (already highest) | Edge case |
|
|
377
|
+
| 18 | resolveComplexityTier with empty metadata returns base tier | Edge case |
|
|
378
|
+
| 19 | resolveModelInternal with strategy='static' behaves identically to current | Backward compat |
|
|
379
|
+
| 20 | resolveModelInternal with strategy='complexity' uses task metadata | Happy path |
|
|
380
|
+
| 21 | Per-agent override takes precedence over routing strategy | Priority |
|
|
381
|
+
| 22 | Per-phase override takes precedence over profile default | Priority |
|
|
382
|
+
| 23 | estimateCostMultiplier returns reasonable numbers for all 3 profiles | Happy path |
|
|
383
|
+
| 24 | Invalid routing strategy falls back to 'static' | Error case |
|
|
384
|
+
| 25 | Unknown provider falls back to 'default' | Error case |
|
|
385
|
+
| 26 | Config without routing section works (backward compat) | Backward compat |
|
|
386
|
+
|
|
387
|
+
**Unit tests in `tests/config.test.cjs`:**
|
|
388
|
+
|
|
389
|
+
| # | Test | Type |
|
|
390
|
+
|---|------|------|
|
|
391
|
+
| 27 | buildConfigDefaults includes routing section with strategy='static' | Happy path |
|
|
392
|
+
| 28 | User defaults can override routing.strategy | Happy path |
|
|
393
|
+
| 29 | Invalid routing strategy in user defaults is accepted (validated at resolve time) | Edge case |
|
|
394
|
+
|
|
395
|
+
**Unit tests in `tests/commands.test.cjs`:**
|
|
396
|
+
|
|
397
|
+
| # | Test | Type |
|
|
398
|
+
|---|------|------|
|
|
399
|
+
| 30 | resolve-model with --metadata flag uses complexity routing | Happy path |
|
|
400
|
+
| 31 | resolve-model without --metadata flag uses static routing | Backward compat |
|
|
401
|
+
| 32 | estimate-cost returns JSON with cost multipliers | Happy path |
|
|
402
|
+
| 33 | estimate-cost with --raw returns formatted text | Happy path |
|
|
403
|
+
|
|
404
|
+
**Runtime-specific tests:**
|
|
405
|
+
|
|
406
|
+
| # | Test | Type |
|
|
407
|
+
|---|------|------|
|
|
408
|
+
| 34 | Installer deploys updated model-profiles.md to all 5 runtimes | Runtime-specific |
|
|
409
|
+
| 35 | resolve-model output is valid for Claude Code Task model parameter | Runtime-specific |
|
|
410
|
+
|
|
411
|
+
---
|
|
412
|
+
|
|
413
|
+
## Phase 4: Specification Output
|
|
414
|
+
|
|
415
|
+
### Feature: Multi-Model Routing Within Agent Workflows
|
|
416
|
+
|
|
417
|
+
### Files to Create/Modify
|
|
418
|
+
|
|
419
|
+
| File | Action | Changes |
|
|
420
|
+
|------|--------|---------|
|
|
421
|
+
| `pan-wizard-core/bin/lib/core.cjs` | Modify | Add PROVIDER_MODELS, update MODEL_PROFILES to tier aliases, add detectProvider(), resolveTierToModel(), resolveComplexityTier(), estimateCostMultiplier(), update resolveModelInternal() |
|
|
422
|
+
| `pan-wizard-core/bin/lib/config.cjs` | Modify | Add routing defaults to buildConfigDefaults() |
|
|
423
|
+
| `pan-wizard-core/bin/lib/commands.cjs` | Modify | Update cmdResolveModel() for metadata, add cmdEstimateCost() |
|
|
424
|
+
| `pan-wizard-core/bin/pan-tools.cjs` | Modify | Wire estimate-cost command, update resolve-model case |
|
|
425
|
+
| `pan-wizard-core/references/model-profiles.md` | Modify | Full rewrite with tier aliases, provider mapping, routing docs |
|
|
426
|
+
| `commands/pan/profile.md` | Modify | Add cost estimation display |
|
|
427
|
+
| `pan-wizard-core/workflows/profile.md` | Modify | Add cost multiplier step |
|
|
428
|
+
| `pan-wizard-core/workflows/settings.md` | Modify | Add routing strategy question |
|
|
429
|
+
| `docs/USER-GUIDE.md` | Modify | Update Model Profiles section |
|
|
430
|
+
| `docs/AGENTS.md` | Modify | Update Model Profiles section |
|
|
431
|
+
| `docs/COMPARISON.md` | Modify | Update Model Flexibility row |
|
|
432
|
+
| `docs/INTERNALS.md` | Modify | Document routing pipeline |
|
|
433
|
+
| `tests/core.test.cjs` | Modify | Add ~26 tests for routing logic |
|
|
434
|
+
| `tests/config.test.cjs` | Modify | Add ~3 tests for routing config |
|
|
435
|
+
| `tests/commands.test.cjs` | Modify | Add ~4 tests for new/updated commands |
|
|
436
|
+
|
|
437
|
+
### Implementation Steps
|
|
438
|
+
|
|
439
|
+
#### Step 1: Core Routing Engine (`core.cjs`)
|
|
440
|
+
|
|
441
|
+
```javascript
|
|
442
|
+
// ─── Provider model mapping ──────────────────────────────────────────────────
|
|
443
|
+
|
|
444
|
+
const PROVIDER_MODELS = {
|
|
445
|
+
anthropic: { reasoning: 'inherit', mid: 'sonnet', fast: 'haiku' },
|
|
446
|
+
openai: { reasoning: 'inherit', mid: 'mid', fast: 'fast' },
|
|
447
|
+
google: { reasoning: 'inherit', mid: 'mid', fast: 'fast' },
|
|
448
|
+
default: { reasoning: 'inherit', mid: 'sonnet', fast: 'haiku' },
|
|
449
|
+
};
|
|
450
|
+
|
|
451
|
+
// Legacy alias mapping (backward compat)
|
|
452
|
+
const LEGACY_ALIASES = { opus: 'reasoning', sonnet: 'mid', haiku: 'fast' };
|
|
453
|
+
|
|
454
|
+
// Updated MODEL_PROFILES with tier aliases
|
|
455
|
+
const MODEL_PROFILES = {
|
|
456
|
+
'pan-planner': { quality: 'reasoning', balanced: 'reasoning', budget: 'mid' },
|
|
457
|
+
'pan-roadmapper': { quality: 'reasoning', balanced: 'mid', budget: 'mid' },
|
|
458
|
+
// ... (all 12 agents as in Phase 3.3)
|
|
459
|
+
};
|
|
460
|
+
```
|
|
461
|
+
|
|
462
|
+
```javascript
|
|
463
|
+
function detectProvider(cwd, config) {
|
|
464
|
+
// 1. Explicit config
|
|
465
|
+
if (config.routing?.provider && config.routing.provider !== 'auto') {
|
|
466
|
+
return config.routing.provider;
|
|
467
|
+
}
|
|
468
|
+
// 2. Environment variable
|
|
469
|
+
if (process.env.PAN_PROVIDER) return process.env.PAN_PROVIDER;
|
|
470
|
+
// 3. Runtime directory detection
|
|
471
|
+
const checks = [
|
|
472
|
+
['.claude', 'anthropic'], ['.codex', 'openai'],
|
|
473
|
+
['.gemini', 'google'], ['.opencode', 'openai'], ['.github', 'default'],
|
|
474
|
+
];
|
|
475
|
+
for (const [dir, provider] of checks) {
|
|
476
|
+
try { if (fs.statSync(path.join(cwd, dir)).isDirectory()) return provider; }
|
|
477
|
+
catch { /* continue */ }
|
|
478
|
+
}
|
|
479
|
+
return 'default';
|
|
480
|
+
}
|
|
481
|
+
|
|
482
|
+
function resolveTierToModel(tier, provider) {
|
|
483
|
+
// Handle legacy names
|
|
484
|
+
const normalizedTier = LEGACY_ALIASES[tier] || tier;
|
|
485
|
+
const providerMap = PROVIDER_MODELS[provider] || PROVIDER_MODELS['default'];
|
|
486
|
+
return providerMap[normalizedTier] || providerMap['mid'];
|
|
487
|
+
}
|
|
488
|
+
```
|
|
489
|
+
|
|
490
|
+
```javascript
|
|
491
|
+
function resolveComplexityTier(baseTier, taskMetadata) {
|
|
492
|
+
if (!taskMetadata) return baseTier;
|
|
493
|
+
const { fileCount = 0, waveCount = 0, requirementCount = 0, isArchitectural = false } = taskMetadata;
|
|
494
|
+
|
|
495
|
+
const score =
|
|
496
|
+
(fileCount > 15 ? 2 : fileCount > 5 ? 1 : 0) +
|
|
497
|
+
(waveCount > 3 ? 2 : waveCount > 1 ? 1 : 0) +
|
|
498
|
+
(requirementCount > 5 ? 2 : requirementCount > 2 ? 1 : 0) +
|
|
499
|
+
(isArchitectural ? 3 : 0);
|
|
500
|
+
|
|
501
|
+
const thresholds = taskMetadata.thresholds || { downgrade_max: 2, upgrade_min: 6 };
|
|
502
|
+
const tiers = ['fast', 'mid', 'reasoning'];
|
|
503
|
+
const idx = tiers.indexOf(baseTier);
|
|
504
|
+
if (idx === -1) return baseTier;
|
|
505
|
+
|
|
506
|
+
if (score <= thresholds.downgrade_max && idx > 0) return tiers[idx - 1];
|
|
507
|
+
if (score >= thresholds.upgrade_min && idx < 2) return tiers[idx + 1];
|
|
508
|
+
return baseTier;
|
|
509
|
+
}
|
|
510
|
+
```
|
|
511
|
+
|
|
512
|
+
```javascript
|
|
513
|
+
// Updated resolveModelInternal
|
|
514
|
+
function resolveModelInternal(cwd, agentType, taskMetadata) {
|
|
515
|
+
const config = loadConfig(cwd);
|
|
516
|
+
const provider = detectProvider(cwd, config);
|
|
517
|
+
|
|
518
|
+
// 1. Per-agent override (highest priority)
|
|
519
|
+
const override = config.model_overrides?.[agentType];
|
|
520
|
+
if (override) {
|
|
521
|
+
return resolveTierToModel(override, provider);
|
|
522
|
+
}
|
|
523
|
+
|
|
524
|
+
// 2. Profile lookup
|
|
525
|
+
const profile = config.model_profile || 'balanced';
|
|
526
|
+
const agentModels = MODEL_PROFILES[agentType];
|
|
527
|
+
if (!agentModels) return resolveTierToModel('mid', provider);
|
|
528
|
+
|
|
529
|
+
let tier = agentModels[profile] || agentModels['balanced'] || 'mid';
|
|
530
|
+
|
|
531
|
+
// 3. Routing strategy
|
|
532
|
+
const strategy = config.routing?.strategy || 'static';
|
|
533
|
+
if (strategy === 'complexity' && taskMetadata) {
|
|
534
|
+
const thresholds = config.routing?.complexity_thresholds;
|
|
535
|
+
tier = resolveComplexityTier(tier, { ...taskMetadata, thresholds });
|
|
536
|
+
}
|
|
537
|
+
// Note: cascade is handled by the orchestrator workflow, not here
|
|
538
|
+
|
|
539
|
+
return resolveTierToModel(tier, provider);
|
|
540
|
+
}
|
|
541
|
+
```
|
|
542
|
+
|
|
543
|
+
```javascript
|
|
544
|
+
const COST_MULTIPLIERS = { reasoning: 15, mid: 3, fast: 1 };
|
|
545
|
+
|
|
546
|
+
function estimateCostMultiplier(profile) {
|
|
547
|
+
let total = 0;
|
|
548
|
+
const agents = Object.keys(MODEL_PROFILES);
|
|
549
|
+
for (const agent of agents) {
|
|
550
|
+
const tier = MODEL_PROFILES[agent][profile] || 'mid';
|
|
551
|
+
total += COST_MULTIPLIERS[tier] || 3;
|
|
552
|
+
}
|
|
553
|
+
return { profile, total, average: +(total / agents.length).toFixed(1), agentCount: agents.length };
|
|
554
|
+
}
|
|
555
|
+
```
|
|
556
|
+
|
|
557
|
+
#### Step 2: Config Defaults (`config.cjs`)
|
|
558
|
+
|
|
559
|
+
Add to `buildConfigDefaults()`:
|
|
560
|
+
|
|
561
|
+
```javascript
|
|
562
|
+
routing: {
|
|
563
|
+
strategy: 'static',
|
|
564
|
+
provider: 'auto',
|
|
565
|
+
cascade_quality_gate: true,
|
|
566
|
+
complexity_thresholds: {
|
|
567
|
+
downgrade_max: 2,
|
|
568
|
+
upgrade_min: 6,
|
|
569
|
+
},
|
|
570
|
+
},
|
|
571
|
+
```
|
|
572
|
+
|
|
573
|
+
And include in the deep-merge:
|
|
574
|
+
|
|
575
|
+
```javascript
|
|
576
|
+
routing: { ...hardcoded.routing, ...(userDefaults.routing || {}) },
|
|
577
|
+
```
|
|
578
|
+
|
|
579
|
+
#### Step 3: CLI Commands (`commands.cjs`, `pan-tools.cjs`)
|
|
580
|
+
|
|
581
|
+
Update `cmdResolveModel` to accept optional metadata:
|
|
582
|
+
|
|
583
|
+
```javascript
|
|
584
|
+
function cmdResolveModel(cwd, agentType, raw, metadataJson) {
|
|
585
|
+
if (!agentType) error('agent-type required');
|
|
586
|
+
|
|
587
|
+
let taskMetadata = null;
|
|
588
|
+
if (metadataJson) {
|
|
589
|
+
try { taskMetadata = JSON.parse(metadataJson); }
|
|
590
|
+
catch { /* ignore invalid metadata, use static routing */ }
|
|
591
|
+
}
|
|
592
|
+
|
|
593
|
+
const config = loadConfig(cwd);
|
|
594
|
+
const profile = config.model_profile || 'balanced';
|
|
595
|
+
const strategy = config.routing?.strategy || 'static';
|
|
596
|
+
const model = resolveModelInternal(cwd, agentType, taskMetadata);
|
|
597
|
+
|
|
598
|
+
const result = { model, profile, strategy };
|
|
599
|
+
output(result, raw, model);
|
|
600
|
+
}
|
|
601
|
+
```
|
|
602
|
+
|
|
603
|
+
Add `estimate-cost` command:
|
|
604
|
+
|
|
605
|
+
```javascript
|
|
606
|
+
function cmdEstimateCost(cwd, raw) {
|
|
607
|
+
const estimates = ['quality', 'balanced', 'budget'].map(estimateCostMultiplier);
|
|
608
|
+
output({ estimates }, raw, estimates.map(e =>
|
|
609
|
+
`${e.profile}: ~${e.average}× baseline (${e.agentCount} agents)`
|
|
610
|
+
).join('\n'));
|
|
611
|
+
}
|
|
612
|
+
```
|
|
613
|
+
|
|
614
|
+
Wire in `pan-tools.cjs`:
|
|
615
|
+
|
|
616
|
+
```javascript
|
|
617
|
+
case 'estimate-cost': {
|
|
618
|
+
commands.cmdEstimateCost(cwd, raw);
|
|
619
|
+
break;
|
|
620
|
+
}
|
|
621
|
+
```
|
|
622
|
+
|
|
623
|
+
#### Step 4: Documentation Updates
|
|
624
|
+
|
|
625
|
+
- **model-profiles.md**: Full rewrite with tier aliases, provider mapping, all 3 routing strategies, cascade quality gates, per-phase override syntax
|
|
626
|
+
- **USER-GUIDE.md**: Update Model Profiles section with new routing options and cost estimation
|
|
627
|
+
- **AGENTS.md**: Update Model Profiles section with tier aliases
|
|
628
|
+
- **COMPARISON.md**: Update "Model Flexibility" to highlight multi-provider routing
|
|
629
|
+
- **INTERNALS.md**: Document the full resolution pipeline
|
|
630
|
+
|
|
631
|
+
#### Step 5: Command/Workflow Updates
|
|
632
|
+
|
|
633
|
+
- **profile.md**: Add cost estimation display after model table
|
|
634
|
+
- **profile.md workflow**: Add step to call `estimate-cost` and display results
|
|
635
|
+
- **settings.md workflow**: Add "Routing Strategy" question (Static/Complexity/Cascade)
|
|
636
|
+
|
|
637
|
+
### Tests Required
|
|
638
|
+
|
|
639
|
+
35 total tests across 3 test files (see Phase 3 Test Plan for full breakdown).
|
|
640
|
+
|
|
641
|
+
### Documentation Updates
|
|
642
|
+
|
|
643
|
+
| Document | Section | Change |
|
|
644
|
+
|----------|---------|--------|
|
|
645
|
+
| `docs/USER-GUIDE.md` | Model Profiles | Add tier aliases, routing strategies, cost estimation |
|
|
646
|
+
| `docs/AGENTS.md` | Model Profiles | Update table to show tier aliases |
|
|
647
|
+
| `docs/COMPARISON.md` | Model Flexibility row | Highlight multi-provider + intelligent routing |
|
|
648
|
+
| `docs/INTERNALS.md` | Per-Agent Overrides | Document full resolution pipeline |
|
|
649
|
+
| `pan-wizard-core/references/model-profiles.md` | Entire file | Rewrite with tier aliases and routing strategies |
|
|
650
|
+
| `CHANGELOG.md` | Next release | Document feature addition |
|
|
651
|
+
|
|
652
|
+
### Runtime Matrix
|
|
653
|
+
|
|
654
|
+
| Runtime | Supported | Notes |
|
|
655
|
+
|---------|-----------|-------|
|
|
656
|
+
| Claude Code | ✅ | Full support — resolve-model output drives Task tool model parameter |
|
|
657
|
+
| Codex | ✅ | Provider detection maps to openai, tier aliases resolve correctly |
|
|
658
|
+
| Gemini | ✅ | Provider detection maps to google, tier aliases resolve correctly |
|
|
659
|
+
| OpenCode | ✅ | Provider detection maps to openai (default), configurable |
|
|
660
|
+
| GitHub Copilot | ✅ | Provider detection maps to default, full alias support |
|
|
661
|
+
|
|
662
|
+
---
|
|
663
|
+
|
|
664
|
+
## Phase 5: Risk Assessment
|
|
665
|
+
|
|
666
|
+
| Risk | Impact | Likelihood | Mitigation |
|
|
667
|
+
|------|--------|------------|------------|
|
|
668
|
+
| Breaking existing model_overrides with legacy names (opus/sonnet/haiku) | **High** — users with custom config.json lose override behavior | Medium | LEGACY_ALIASES map ensures opus→reasoning, sonnet→mid, haiku→fast. Existing string values continue to work. Test explicitly. |
|
|
669
|
+
| Cascade routing doubles execution time on failure | **Medium** — slower for agents that consistently need higher tier | Medium | Cascade is opt-in only. Default remains `static`. Document clearly that cascade adds latency. |
|
|
670
|
+
| Complexity scoring produces unexpected tier assignments | **Medium** — wrong model for task complexity | Medium | Conservative thresholds (downgrade ≤2, upgrade ≥6). Configurable via `complexity_thresholds`. Users can tune or disable. |
|
|
671
|
+
| Provider detection wrong (e.g., multiple runtime dirs exist) | **Low** — resolves to wrong provider's model names | Low | Priority order: config explicit > env var > directory scan. First match wins. Users can always set explicit provider. |
|
|
672
|
+
| Cost estimates are inaccurate | **Low** — misleading but not functionally harmful | High | Label all estimates as "approximate". Use relative multipliers, not dollar amounts. |
|
|
673
|
+
| Non-Anthropic runtimes don't use resolve-model output | **Low** — feature is invisible to 4 of 5 runtimes today | N/A | This is expected. The tier alias system prepares for future runtime integration. Document this honestly. |
|
|
674
|
+
| resolveModelInternal signature change (new `taskMetadata` param) | **Medium** — internal callers break | Low | Parameter is optional with default `undefined`. Existing calls with 2 args continue to work identically (strategy defaults to 'static', no metadata → no complexity routing). |
|
|
675
|
+
| Config migration — existing config.json missing routing section | **Low** — new section not present | High | `buildConfigDefaults()` provides defaults. `resolveModelInternal()` uses `config.routing?.strategy \|\| 'static'` — missing section = current behavior. Zero migration needed. |
|
|
676
|
+
|
|
677
|
+
### Priority Recommendation
|
|
678
|
+
|
|
679
|
+
**Implement in this order:**
|
|
680
|
+
|
|
681
|
+
1. **Tier aliases + legacy compat** (lowest risk, highest long-term value) — enables multi-provider future
|
|
682
|
+
2. **Provider detection** (low risk, foundational for multi-runtime) — auto-detection with explicit override
|
|
683
|
+
3. **Cost estimation** (zero risk, immediate user value) — read-only display, no behavior change
|
|
684
|
+
4. **Complexity routing** (medium risk, significant cost savings) — opt-in strategy with conservative defaults
|
|
685
|
+
5. **Cascade routing** (highest risk, highest savings potential) — implement last, opt-in only, requires quality gate definitions
|
|
686
|
+
|
|
687
|
+
Steps 1-3 can ship together as a single release. Steps 4-5 should be separate releases with their own test cycles.
|