pan-wizard 3.35.0 → 3.37.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +3645 -0
- package/README.md +46 -52
- package/agents/pan-conductor.md +2 -3
- package/agents/pan-counterfactual.md +1 -2
- package/agents/pan-debugger.md +14 -3
- package/agents/pan-design-checker.md +1 -2
- package/agents/pan-designer.md +2 -3
- package/agents/pan-document_code.md +1 -2
- package/agents/pan-executor.md +25 -0
- package/agents/pan-hardener.md +4 -3
- package/agents/pan-integration-checker.md +1 -2
- package/agents/pan-knowledge.md +1 -2
- package/agents/pan-meta-reviewer.md +3 -3
- package/agents/pan-optimizer.md +4 -4
- package/agents/pan-phase-researcher.md +80 -475
- package/agents/pan-plan-checker.md +40 -3
- package/agents/pan-planner.md +21 -7
- package/agents/pan-previewer.md +3 -2
- package/agents/pan-project-researcher.md +72 -493
- package/agents/pan-release.md +1 -1
- package/agents/pan-reviewer.md +14 -3
- package/agents/pan-roadmapper.md +12 -7
- package/agents/pan-verifier.md +9 -4
- package/bin/install-lib.cjs +134 -14
- package/bin/install.js +59 -12
- package/commands/pan/army.md +1 -1
- package/commands/pan/autodev.md +76 -0
- package/commands/pan/cost.md +21 -1
- package/commands/pan/exec-phase.md +3 -4
- package/commands/pan/focus-auto.md +14 -7
- package/commands/pan/focus-design.md +4 -5
- package/commands/pan/focus-docs.md +24 -0
- package/commands/pan/learn.md +13 -6
- package/commands/pan/milestone-audit.md +8 -1
- package/commands/pan/milestone-done.md +10 -5
- package/commands/pan/milestone-new.md +2 -2
- package/commands/pan/new-project.md +15 -12
- package/commands/pan/plan-phase.md +15 -9
- package/commands/pan/preview.md +1 -1
- package/commands/pan/quick.md +1 -1
- package/commands/pan/review-deep.md +11 -5
- package/commands/pan/settings.md +6 -0
- package/commands/pan/verify-phase.md +6 -2
- package/docs/AGENTS.md +917 -0
- package/docs/ARCHITECTURE.md +1054 -0
- package/docs/CLI-REFERENCE.md +4582 -0
- package/docs/COMPARISON.md +116 -0
- package/docs/DEVELOPMENT.md +490 -0
- package/docs/ECOSYSTEM-REVIEW-2026-06.md +107 -0
- package/docs/ECOSYSTEM-REVIEW-2026-08.md +349 -0
- package/docs/EXAMPLES.md +584 -0
- package/docs/FAQ.md +254 -0
- package/docs/FIELD-HARVEST-2026-07.md +88 -0
- package/docs/FIELD-REPORT-army-2026-06.md +121 -0
- package/docs/HOOKS.md +638 -0
- package/docs/IMPROVEMENT-TODO.md +246 -0
- package/docs/INTERNALS.md +485 -0
- package/docs/MIGRATION-v2-to-v3.md +258 -0
- package/docs/SECURITY-HARDENING.md +230 -0
- package/docs/SKILLS-ENHANCEMENT-PLAN-V2.md +122 -0
- package/docs/SKILLS-FULL-TEXT.md +9685 -0
- package/docs/SKILLS-REFERENCE.md +1037 -0
- package/docs/TROUBLESHOOTING.md +1084 -0
- package/docs/USER-GUIDE.md +1437 -0
- package/docs/archive/SKILLS-ENHANCEMENT-PLAN.md +230 -0
- package/docs/audits/AUDIT-2026-08-e2e-ROADMAP.md +117 -0
- package/docs/audits/AUDIT-2026-08-e2e-VERIFICATION.md +693 -0
- package/docs/audits/AUDIT-2026-08-e2e.md +1546 -0
- package/docs/audits/OUTSTANDING-2026-08.md +275 -0
- package/docs/audits/STABILITY-TEST-2026-08.md +640 -0
- package/docs/branding/README-brand-draft.md +119 -0
- package/docs/branding/handoff/IMPLEMENTATION.md +205 -0
- package/docs/branding/handoff/README-brand-draft.md +119 -0
- package/docs/branding/handoff/image-prompts.md +46 -0
- package/docs/branding/image-prompts.md +46 -0
- package/docs/branding/nano-banana-prompts.txt +60 -0
- package/docs/context-monitor.md +13 -0
- package/docs/decisions/ADR-0001-strategic-feature-prioritization.md +83 -0
- package/docs/decisions/ADR-0002-copilot-cli-runtime.md +71 -0
- package/docs/decisions/ADR-0003-smart-execution.md +84 -0
- package/docs/decisions/ADR-0004-copilot-cli-interaction.md +89 -0
- package/docs/decisions/ADR-0005-command-naming.md +80 -0
- package/docs/decisions/ADR-0006-focus-commands.md +101 -0
- package/docs/decisions/ADR-0007-ship-v1.md +73 -0
- package/docs/decisions/ADR-0008-map-codebase-mermaid-togaf.md +80 -0
- package/docs/decisions/ADR-0009-production-deployment-checklist.md +68 -0
- package/docs/decisions/ADR-0010-standards-integration.md +79 -0
- package/docs/decisions/ADR-0011-help-commands-rewrite.md +64 -0
- package/docs/decisions/ADR-0012-contributors-attribution-index.md +42 -0
- package/docs/decisions/ADR-0013-strategic-feature-additions.md +53 -0
- package/docs/decisions/ADR-0014-internal-cleanup-code-quality.md +53 -0
- package/docs/decisions/ADR-0015-focus-auto-runner.md +296 -0
- package/docs/decisions/ADR-0016-enhanced-doc-sync.md +54 -0
- package/docs/decisions/ADR-0017-enhanced-focus-design.md +69 -0
- package/docs/decisions/ADR-0018-user-reality-testing.md +84 -0
- package/docs/decisions/ADR-0019-end-to-end-user-acceptance-testing.md +113 -0
- package/docs/decisions/ADR-0020-ai-drift-prevention.md +55 -0
- package/docs/decisions/ADR-0021-codebase-mapper-v2.md +76 -0
- package/docs/decisions/ADR-0022-lifecycle-completeness.md +63 -0
- package/docs/decisions/ADR-0023-opus-4-7-adoption.md +89 -0
- package/docs/decisions/ADR-0024-spec-b-v2-completion.md +137 -0
- package/docs/decisions/ADR-0025-behavioral-guardrails-layer.md +129 -0
- package/docs/decisions/ADR-0026-self-improvement-loop.md +171 -0
- package/docs/decisions/ADR-0027-doc-code-link-graph.md +164 -0
- package/docs/decisions/ADR-0028-unified-agents-skills-target.md +46 -0
- package/docs/decisions/ADR-0029-experiment-evaluator.md +76 -0
- package/docs/decisions/ADR-0030-planning-concurrency.md +37 -0
- package/docs/decisions/ADR-0031-focus-auto-backlog-source.md +47 -0
- package/docs/decisions/ADR-0032-squad-model.md +64 -0
- package/docs/decisions/ADR-0033-army-campaign.md +40 -0
- package/docs/decisions/ADR-0034-scheduled-campaigns.md +57 -0
- package/docs/decisions/ADR-0035-army-hud-dashboard.md +40 -0
- package/docs/decisions/ADR-0036-agentic-memory-architecture.md +106 -0
- package/docs/decisions/ADR-0037-pin-security-agents-to-opus.md +51 -0
- package/docs/decisions/ADR-0038-skill-aligned-decomposition.md +41 -0
- package/docs/decisions/ADR-0039-worktree-as-project-variant.md +37 -0
- package/docs/decisions/ADR-0040-memory-injection-defense.md +55 -0
- package/docs/decisions/ADR-0041-mcp-dual-era-bridge.md +47 -0
- package/docs/decisions/ADR-0042-design-parity-both-flows.md +45 -0
- package/docs/decisions/ADR-0043-addressable-planning-roots.md +56 -0
- package/docs/decisions/ADR-0044-bound-the-cached-context.md +104 -0
- package/docs/decisions/ADR-0045-agent-plugins-bundle.md +76 -0
- package/docs/decisions/ADR-0046-subagent-frontmatter-decisions.md +39 -0
- package/docs/decisions/ADR-0047-pan-harness.md +76 -0
- package/docs/decisions/ADR-0048-one-core-per-project.md +43 -0
- package/docs/decisions/ADR-0049-machine-verdicts-and-findings-ledger.md +116 -0
- package/docs/specs/ai_drift_prevention_featureai.md +286 -0
- package/docs/specs/codebase_mapper_v2_featureai.md +930 -0
- package/docs/specs/command_naming_optimization_featureai.md +930 -0
- package/docs/specs/comp_analysis_markdown_v1.md +149 -0
- package/docs/specs/continuous_autonomous_execution_featureai.md +304 -0
- package/docs/specs/contributors_attribution_index_featureai.md +476 -0
- package/docs/specs/copilot_cli_interaction_featureai.md +868 -0
- package/docs/specs/copilot_cli_runtime_featureai.md +647 -0
- package/docs/specs/custom-skills-assessment.md +461 -0
- package/docs/specs/deployment_audit_findings_featureai.md +243 -0
- package/docs/specs/doc_code_link_graph_featureai.md +253 -0
- package/docs/specs/documentation_alignment_cleanup_featureai.md +69 -0
- package/docs/specs/e2e_visual_testing_featureai.md +615 -0
- package/docs/specs/end_to_end_user_acceptance_testing_featureai.md +810 -0
- package/docs/specs/evidence-loop-plan.md +76 -0
- package/docs/specs/evidence_loop_featureai.md +497 -0
- package/docs/specs/focus-auto-optimize-category.md +449 -0
- package/docs/specs/focus-auto-optimize-superplan.md +188 -0
- package/docs/specs/focus_auto_runner_featureai.md +1102 -0
- package/docs/specs/focus_design_v2_featureai.md +197 -0
- package/docs/specs/focus_sync_v2_featureai.md +202 -0
- package/docs/specs/git_integration_overhaul_featureai.md +326 -0
- package/docs/specs/glm_zai_provider_featureai.md +200 -0
- package/docs/specs/googlecli_adoption_featureai.md +661 -0
- package/docs/specs/help_commands_rewrite_featureai.md +860 -0
- package/docs/specs/industry_analysis_strategic_additions_featureai.md +231 -0
- package/docs/specs/industry_comparison_and_feature_gaps_featureai.md +674 -0
- package/docs/specs/internal_cleanup_code_quality_featureai.md +486 -0
- package/docs/specs/investigations/README.md +31 -0
- package/docs/specs/investigations/digests/2026-09-28.md +47 -0
- package/docs/specs/investigations/digests/2026-10-08.md +62 -0
- package/docs/specs/investigations/digests/2026-10-10.md +61 -0
- package/docs/specs/investigations/features/debugger-red-loop-first.md +55 -0
- package/docs/specs/investigations/features/harness-cross-host-behaviour.md +93 -0
- package/docs/specs/investigations/features/optimize-apply-regression-gate.md +66 -0
- package/docs/specs/investigations/features/reviewer-smell-baseline.md +57 -0
- package/docs/specs/investigations/features/trace-failure-capture.md +69 -0
- package/docs/specs/lifecycle_completeness_featureai.md +754 -0
- package/docs/specs/local-first-ci.md +42 -0
- package/docs/specs/map_codebase_mermaid_togaf_featureai.md +805 -0
- package/docs/specs/market-delta-2026-09-superplan.md +310 -0
- package/docs/specs/market-ideas-2026-09.md +131 -0
- package/docs/specs/market-ideas-2026-10.md +428 -0
- package/docs/specs/market-ideas-ledger.md +177 -0
- package/docs/specs/memory-optimization-2026-10.md +285 -0
- package/docs/specs/multi-track-planning-roots.md +143 -0
- package/docs/specs/multi_model_routing_featureai.md +687 -0
- package/docs/specs/multi_model_routing_superplan.md +197 -0
- package/docs/specs/opus_47_existing_enhancements_featureai.md +247 -0
- package/docs/specs/opus_47_extended_features_featureai.md +363 -0
- package/docs/specs/opus_47_extended_features_v2_featureai.md +397 -0
- package/docs/specs/pan_focus_commands_featureai.md +2336 -0
- package/docs/specs/pan_zcode_mcp_bridge_featureai.md +120 -0
- package/docs/specs/product-excellence-superplan.md +315 -0
- package/docs/specs/product_excellence_program_2026_q4_featureai.md +365 -0
- package/docs/specs/production_deployment_readiness_featureai.md +433 -0
- package/docs/specs/production_readiness_ship_v1_featureai.md +698 -0
- package/docs/specs/real_test_enforcement_featureai.md +213 -0
- package/docs/specs/reality-check-2026-09-superplan.md +362 -0
- package/docs/specs/right-sized-ceremony.md +74 -0
- package/docs/specs/roadmap_and_competition_2026_q4_featureai.md +390 -0
- package/docs/specs/self_improvement_loop_featureai.md +875 -0
- package/docs/specs/skill-aligned-decomposition.md +155 -0
- package/docs/specs/standards_integration_featureai.md +833 -0
- package/docs/specs/testing-system-redesign-2026-09.md +267 -0
- package/docs/specs/user_reality_testing_system_featureai.md +788 -0
- package/docs/specs/watch-triggers.json +22 -0
- package/docs/specs/workflow-integration-redesign.md +742 -0
- package/docs/specs/workflow_state_consistency_featureai.md +183 -0
- package/hooks/dist/pan-context-monitor.js +9 -2
- package/hooks/dist/pan-cost-logger.js +5 -0
- package/hooks/dist/pan-guard.js +28 -5
- package/hooks/dist/pan-stop-guard.js +9 -9
- package/hooks/dist/pan-trace-logger.js +5 -0
- package/package.json +10 -2
- package/pan-wizard-core/bin/lib/agents-md.cjs +57 -8
- package/pan-wizard-core/bin/lib/autodev-report.cjs +180 -0
- package/pan-wizard-core/bin/lib/autodev.cjs +642 -0
- package/pan-wizard-core/bin/lib/ci.cjs +143 -0
- package/pan-wizard-core/bin/lib/commands.cjs +1 -1
- package/pan-wizard-core/bin/lib/config.cjs +41 -5
- package/pan-wizard-core/bin/lib/constants.cjs +27 -1
- package/pan-wizard-core/bin/lib/core.cjs +31 -14
- package/pan-wizard-core/bin/lib/cost.cjs +160 -4
- package/pan-wizard-core/bin/lib/cross-family.cjs +164 -0
- package/pan-wizard-core/bin/lib/debug-cases.cjs +104 -0
- package/pan-wizard-core/bin/lib/doc-lint.cjs +5 -3
- package/pan-wizard-core/bin/lib/failure-class.cjs +81 -0
- package/pan-wizard-core/bin/lib/findings.cjs +148 -9
- package/pan-wizard-core/bin/lib/focus.cjs +1 -1
- package/pan-wizard-core/bin/lib/git.cjs +1 -1
- package/pan-wizard-core/bin/lib/hosts.cjs +88 -0
- package/pan-wizard-core/bin/lib/init.cjs +7 -2
- package/pan-wizard-core/bin/lib/knowledge.cjs +2 -0
- package/pan-wizard-core/bin/lib/merge.cjs +50 -0
- package/pan-wizard-core/bin/lib/mutate.cjs +268 -0
- package/pan-wizard-core/bin/lib/next-action.cjs +112 -0
- package/pan-wizard-core/bin/lib/optimize.cjs +2 -2
- package/pan-wizard-core/bin/lib/phase-gate.cjs +15 -9
- package/pan-wizard-core/bin/lib/phase-remove.cjs +91 -11
- package/pan-wizard-core/bin/lib/phase.cjs +85 -13
- package/pan-wizard-core/bin/lib/plan-lint.cjs +310 -0
- package/pan-wizard-core/bin/lib/preview.cjs +26 -4
- package/pan-wizard-core/bin/lib/requirements-lint.cjs +130 -0
- package/pan-wizard-core/bin/lib/research-lint.cjs +268 -0
- package/pan-wizard-core/bin/lib/review-deep.cjs +14 -22
- package/pan-wizard-core/bin/lib/review-vote.cjs +105 -0
- package/pan-wizard-core/bin/lib/roadmap.cjs +1 -0
- package/pan-wizard-core/bin/lib/runner.cjs +2 -16
- package/pan-wizard-core/bin/lib/sizing.cjs +356 -0
- package/pan-wizard-core/bin/lib/squads.cjs +1 -1
- package/pan-wizard-core/bin/lib/state-write.cjs +247 -0
- package/pan-wizard-core/bin/lib/test-gate.cjs +433 -0
- package/pan-wizard-core/bin/lib/uat.cjs +149 -0
- package/pan-wizard-core/bin/lib/usage.cjs +104 -0
- package/pan-wizard-core/bin/lib/verdict.cjs +26 -3
- package/pan-wizard-core/bin/lib/verify-plan-set.cjs +30 -1
- package/pan-wizard-core/bin/lib/verify-preflight.cjs +2 -2
- package/pan-wizard-core/bin/lib/verify-red.cjs +174 -0
- package/pan-wizard-core/bin/lib/verify-retro.cjs +3 -2
- package/pan-wizard-core/bin/lib/verify-scope.cjs +143 -2
- package/pan-wizard-core/bin/lib/verify.cjs +86 -31
- package/pan-wizard-core/bin/lib/worktree.cjs +70 -0
- package/pan-wizard-core/bin/pan-tools.cjs +257 -28
- package/pan-wizard-core/learnings/universal/autonomous-loop.md +2 -2
- package/pan-wizard-core/mcp/orchestrator.cjs +4 -1
- package/pan-wizard-core/references/continuation-format.md +3 -3
- package/pan-wizard-core/references/design-methodology.md +1 -1
- package/pan-wizard-core/references/guardrails.md +1 -1
- package/pan-wizard-core/references/handoff-decisions.md +1 -1
- package/pan-wizard-core/references/model-profiles.md +2 -3
- package/pan-wizard-core/references/questioning.md +24 -0
- package/pan-wizard-core/references/ui-brand.md +4 -6
- package/pan-wizard-core/references/verdict-contract.md +2 -1
- package/pan-wizard-core/templates/config.json +10 -2
- package/pan-wizard-core/templates/debug.md +1 -0
- package/pan-wizard-core/templates/design.md +5 -4
- package/pan-wizard-core/templates/discovery.md +2 -2
- package/pan-wizard-core/templates/phase-prompt.md +7 -0
- package/pan-wizard-core/templates/requirements.md +9 -0
- package/pan-wizard-core/templates/research.md +10 -0
- package/pan-wizard-core/templates/summary-complex.md +5 -1
- package/pan-wizard-core/templates/summary-minimal.md +4 -0
- package/pan-wizard-core/templates/summary-standard.md +5 -1
- package/pan-wizard-core/templates/summary.md +8 -1
- package/pan-wizard-core/workflows/add-phase.md +4 -6
- package/pan-wizard-core/workflows/assumptions.md +19 -1
- package/pan-wizard-core/workflows/diagnose-issues.md +2 -0
- package/pan-wizard-core/workflows/discuss-phase.md +4 -2
- package/pan-wizard-core/workflows/exec-lean.md +69 -0
- package/pan-wizard-core/workflows/exec-phase.md +79 -15
- package/pan-wizard-core/workflows/execute-plan.md +3 -1
- package/{commands/pan → pan-wizard-core/workflows}/focus-doc-audit.md +13 -29
- package/{commands/pan → pan-wizard-core/workflows}/focus-drift-walking.md +3 -18
- package/{commands/pan → pan-wizard-core/workflows}/focus-exec.md +8 -23
- package/{commands/pan → pan-wizard-core/workflows}/focus-plan.md +6 -20
- package/{commands/pan → pan-wizard-core/workflows}/focus-scan.md +3 -16
- package/{commands/pan → pan-wizard-core/workflows}/focus-sync.md +7 -21
- package/pan-wizard-core/workflows/health.md +6 -0
- package/pan-wizard-core/workflows/help.md +114 -77
- package/pan-wizard-core/workflows/insert-phase.md +4 -6
- package/pan-wizard-core/workflows/learn.md +3 -3
- package/pan-wizard-core/workflows/milestone-audit.md +2 -2
- package/pan-wizard-core/workflows/milestone-new.md +38 -95
- package/pan-wizard-core/workflows/new-project.md +96 -258
- package/pan-wizard-core/workflows/optimize.md +107 -5
- package/pan-wizard-core/workflows/phase-tests.md +10 -0
- package/pan-wizard-core/workflows/plan-phase.md +106 -45
- package/pan-wizard-core/workflows/profile.md +50 -0
- package/pan-wizard-core/workflows/progress.md +10 -0
- package/pan-wizard-core/workflows/quick.md +34 -43
- package/pan-wizard-core/workflows/remove-phase.md +11 -1
- package/pan-wizard-core/workflows/research-phase.md +118 -40
- package/pan-wizard-core/workflows/resume-project.md +18 -2
- package/pan-wizard-core/workflows/settings.md +33 -19
- package/pan-wizard-core/workflows/todo-check.md +6 -2
- package/pan-wizard-core/workflows/transition.md +22 -4
- package/pan-wizard-core/workflows/uat-walkthrough.md +90 -0
- package/pan-wizard-core/workflows/update.md +1 -1
- package/pan-wizard-core/workflows/verify-phase.md +39 -46
- package/scripts/build-agent-plugin.js +35 -15
- package/scripts/build-help.cjs +106 -0
- package/scripts/build-plugin.js +10 -1
- package/scripts/deprecate-old-versions.js +70 -3
- package/scripts/generate-skills-docs.py +1 -1
- package/scripts/git-hooks/pre-push +27 -0
- package/scripts/install-git-hooks.js +7 -5
- package/scripts/install-pre-push.cjs +38 -0
- package/scripts/land.cjs +151 -0
- package/scripts/local-ci.cjs +501 -0
- package/scripts/local-schedule.cjs +277 -0
- package/scripts/mutation-probe.cjs +7 -80
- package/scripts/prompt-lint.cjs +32 -0
- package/scripts/prompt-rules.cjs +156 -0
- package/scripts/run-tests.cjs +7 -1
- package/scripts/test-quality-lint.cjs +3 -3
- package/scripts/test-surface.cjs +175 -6
- package/agents/pan-research-synthesizer.md +0 -246
- package/commands/pan/audit-deployment.md +0 -386
- package/commands/pan/dashboard.md +0 -22
- package/commands/pan/discord.md +0 -19
- package/commands/pan/mcp-bridge.md +0 -143
- package/commands/pan/milestone-cleanup.md +0 -19
- package/commands/pan/milestone-gaps.md +0 -35
- package/commands/pan/optimize.md +0 -115
- package/commands/pan/profile.md +0 -76
- package/commands/pan/research-phase.md +0 -189
- package/pan-wizard-core/bin/lib/bridge.cjs +0 -269
- package/pan-wizard-core/templates/research-project/architecture.md +0 -204
- package/pan-wizard-core/templates/research-project/features.md +0 -147
- package/pan-wizard-core/templates/research-project/pitfalls.md +0 -200
- package/pan-wizard-core/templates/research-project/stack.md +0 -120
- package/pan-wizard-core/templates/research-project/summary.md +0 -170
|
@@ -0,0 +1,875 @@
|
|
|
1
|
+
# featureAI: Cross-Project Self-Improvement Loop — Idea → External Build → Learn → Promote → Ship
|
|
2
|
+
|
|
3
|
+
> **Generated**: 2026-04-27
|
|
4
|
+
> **Mode**: `--full` (all phases)
|
|
5
|
+
> **Feature**: `/pan:experiment` family — autonomous external builds drive PAN's own evolution
|
|
6
|
+
> **Target version**: v3.7.0 (XL feature, ships across 3-4 waves)
|
|
7
|
+
> **Predecessors**: focus_auto_runner_featureai.md (autonomous loop), googlecli_adoption_featureai.md (behavioral guardrails layer), Session 33 Playwright VSCode integration patterns
|
|
8
|
+
> **Status**: design — awaiting acceptance
|
|
9
|
+
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
## Phase 0: Problem Framing & Demand
|
|
13
|
+
|
|
14
|
+
### 0.1 Problem Statement
|
|
15
|
+
|
|
16
|
+
PAN Wizard ships v3.5+ with a **self-learning system inside a project** — `pan-trace-logger` captures every agent spawn, `/pan:learn` analyzes the session via `pan-optimizer`, `/pan:optimize apply` writes back to that project's `.planning/memory/`. The loop is closed *for one project at a time*.
|
|
17
|
+
|
|
18
|
+
The PAN tool itself does not learn from this. Each PAN release ships frozen behavioral rules (`references/guardrails.md`, workflow text, agent instructions). Real-world session data — the gold mine of "what actually goes wrong when AI agents drive PAN" — stays trapped in user projects' `.planning/optimization/` directories. We dogfood PAN on PAN itself (`d:\PanWizard\.planning/`), but those learnings are **per-session memory**, not part of the shipped product. The next release of PAN does not know what the previous release's sessions discovered.
|
|
19
|
+
|
|
20
|
+
This matters NOW because:
|
|
21
|
+
|
|
22
|
+
- v3.5 just shipped the trace + learn + optimize trio. The pieces exist. What's missing is the **promote step** that lifts findings from one session's memory into shipped artifacts.
|
|
23
|
+
- v3.6 (just shipped) added `references/guardrails.md` — a human-authored behavioral file consumed by every workflow. This proves the *delivery channel* for shipped behavioral content. We need only build the *production channel* (AI-derived patterns that flow into shipped files).
|
|
24
|
+
- The MEMORY note "Cross-Session Learning (PATTERNS.md auto-capture)" was deferred for exactly this reason: needs a design spec. This is that spec.
|
|
25
|
+
- Real-world AI coding tooling (Cursor Background Agents, Aider's MCP integrations, Claude Code's Skills) is converging on session telemetry as a product feedback loop. PAN's existing trace infrastructure is unique among workflow tools; not closing the meta-loop wastes the lead.
|
|
26
|
+
|
|
27
|
+
**The cost of not building this:** PAN's quality plateau is gated by what its maintainers can hand-author into `references/`, `workflows/`, and agent instructions. Every session's lessons stay local. PAN's improvement velocity is bounded by maintainer attention, not by accumulated session evidence.
|
|
28
|
+
|
|
29
|
+
### 0.2 Demand Evidence
|
|
30
|
+
|
|
31
|
+
| Evidence | Source | Finding |
|
|
32
|
+
|----------|--------|---------|
|
|
33
|
+
| **User-stated explicit request** | This conversation | "design into pan a system whereby we build and then call learn and then we bring the file back to optimise PAN" |
|
|
34
|
+
| **MEMORY note** | `MEMORY.md` "Known Deferred Items" | "Cross-Session Learning (PATTERNS.md auto-capture): L effort, needs design spec" |
|
|
35
|
+
| **Existing infrastructure** | `optimize.cjs`, `pan-optimizer.md`, `pan-trace-logger.js`, `/pan:learn`, `/pan:optimize` | Trace + analyze + apply already work intra-project. Promote + ship are the missing ~30% |
|
|
36
|
+
| **Existing delivery channel** | `references/guardrails.md` shipped in v3.6.0 to all 5 runtimes | Proves AI-readable shipped reference docs work; same path serves AI-derived patterns |
|
|
37
|
+
| **Existing process automation** | `tests/scenarios/*.test.cjs` already spawn fresh installs in temp dirs via `createScenarioRunner` | Cross-process orchestration is a known-cost operation in this codebase |
|
|
38
|
+
| **Existing VSCode automation** | `tests/e2e/playwright.config.mjs`, `tests/e2e/vscode-helpers.mjs` (Session 33) | Precedent for driving VSCode programmatically — though the simpler `claude` CLI subprocess path covers the autonomous use case |
|
|
39
|
+
| **Industry convergence** | Cursor Background Agents, Aider --watch-files, Copilot Coding Agent | Autonomous external project execution is a 2026 standard; PAN's gap is closing the *learning* side, not just the execution side |
|
|
40
|
+
|
|
41
|
+
**Demand status:** STRONG — direct user request + deferred-with-design-spec marker + the prerequisite infrastructure is all in place.
|
|
42
|
+
|
|
43
|
+
### 0.3 Scope Definition
|
|
44
|
+
|
|
45
|
+
| In Scope | Out of Scope (and why) |
|
|
46
|
+
|----------|------------------------|
|
|
47
|
+
| Idea-input format (`idea.md` schema) PAN can consume as a Phase-0 spec | Multi-stakeholder idea collection workflow — single user, single idea, single experiment |
|
|
48
|
+
| Experiment scaffolding: create isolated project folder outside source repo with `.planning/` and idea doc | Experiments inside the source repo — installer guard already prohibits self-install; same logic applies |
|
|
49
|
+
| External AI agent invocation via subprocess (`claude -p`, `codex`, `gemini`, `opencode`) | VSCode UI automation as the *primary* path — adds Playwright dependency to the runtime; reserve as advanced opt-in |
|
|
50
|
+
| Stream observation: tail external project's `state.md`, `current-agent-id.txt`, summary files; surface progress in PAN session | Bidirectional control — external agent runs autonomously; PAN observes but doesn't intervene mid-build (cleaner failure semantics) |
|
|
51
|
+
| Telemetry harvest: copy `.planning/optimization/`, `agent-history.json`, traces back to PAN source as `experiments/<id>/` | Cross-experiment aggregation — one experiment, one harvest; aggregation is a separate v3.8+ feature |
|
|
52
|
+
| `/pan:learn --experiment <id>` runs `pan-optimizer` against harvested data | Re-running pan-optimizer continuously during the experiment — single post-build analysis is sufficient |
|
|
53
|
+
| `pan-tools learn promote --pattern <id> --scope <universal\|internal>` writes findings to `pan-wizard-core/learnings/` | Auto-promote (no human review) — risky; manual review step is the v3.7 design; auto-promote is a v3.8 ramp |
|
|
54
|
+
| Two-tier output: `learnings/universal/` (shipped to all installs) vs `learnings/internal/` (PAN-source-only) | Three or more tiers — universal/internal binary is sufficient |
|
|
55
|
+
| Workflows reference `learnings/universal/*.md` like they reference `references/*.md` | New runtime adapter logic — `learnings/` ships via the same glob that ships `references/` |
|
|
56
|
+
| Budget cap + timeout + circuit breaker on external runs (inherits focus-auto pattern) | Cost-tracking integration — `cost.cjs` already records spend; cross-experiment cost analysis is v3.8 |
|
|
57
|
+
| Tests: scenario coverage for the experiment lifecycle (init → run → harvest → promote) | Integration test that drives a real Claude Code subprocess — too slow for CI; mocked subprocess is sufficient |
|
|
58
|
+
|
|
59
|
+
### 0.4 Success Criteria (Measurable)
|
|
60
|
+
|
|
61
|
+
```
|
|
62
|
+
SC-1: /pan:experiment new <slug> --idea <file> creates an isolated experiment folder OUTSIDE the source repo with .planning/ and idea.md
|
|
63
|
+
SC-2: /pan:experiment run <id> spawns an external AI agent (Claude/Codex/Gemini/OpenCode) pointed at the experiment folder
|
|
64
|
+
SC-3: Experiment runner streams progress back to PAN session (state.md changes, summary.md appearances)
|
|
65
|
+
SC-4: Loop stops on ANY of: build success (final summary written), budget exhausted, timeout, circuit breaker (3 consecutive task failures)
|
|
66
|
+
SC-5: /pan:experiment harvest <id> copies .planning/optimization/ + agent-history.json + state.md from experiment folder back to d:\PanWizard\experiments\<id>\
|
|
67
|
+
SC-6: /pan:learn --experiment <id> runs pan-optimizer over harvested data, writes report to experiments/<id>/learnings/report-*.md
|
|
68
|
+
SC-7: pan-tools learn promote --pattern <id> --scope universal extracts a finding to pan-wizard-core/learnings/universal/<topic>.md
|
|
69
|
+
SC-8: pan-tools learn promote --pattern <id> --scope internal extracts a finding to pan-wizard-core/learnings/internal/<topic>.md (NOT shipped to user installs)
|
|
70
|
+
SC-9: Workflows reference learnings/universal/*.md the same way they reference references/*.md (already proven in v3.6.0)
|
|
71
|
+
SC-10: Installer ships pan-wizard-core/learnings/universal/ to all 5 runtimes; learnings/internal/ stays in source only
|
|
72
|
+
SC-11: Tests: 30+ new tests across experiment.test.cjs, learn-promote.test.cjs, scenarios/learnings-installed.test.cjs
|
|
73
|
+
SC-12: Zero regressions in existing 2588 tests
|
|
74
|
+
SC-13: A complete loop (idea.md → experiment run → harvest → learn → promote → re-install verifies pattern shipped) executes end-to-end in < 10 minutes for a trivial idea
|
|
75
|
+
SC-14: Spec works regardless of which runtime drives the external project (Claude/Codex/Gemini/OpenCode); GitHub Copilot CLI is opt-in (no headless mode)
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
### 0.5 User Stories
|
|
79
|
+
|
|
80
|
+
```
|
|
81
|
+
As a PAN maintainer with an idea for a small CLI tool,
|
|
82
|
+
I want to /pan:experiment new my-idea --idea my-idea.md, run it externally,
|
|
83
|
+
have it produce a real working project AND telemetry on every PAN-driven decision,
|
|
84
|
+
and then /pan:learn promote the most-cited findings into PAN's shipped behavioral docs,
|
|
85
|
+
so that the next PAN release literally got smarter from this build,
|
|
86
|
+
instead of having to manually transcribe lessons from .planning/optimization/ into references/guardrails.md.
|
|
87
|
+
|
|
88
|
+
As a PAN maintainer reviewing a fresh experiment harvest,
|
|
89
|
+
I want a clear universal-vs-internal scope decision per finding,
|
|
90
|
+
so that PAN-specific quirks (e.g., "always commit individually" because of the source repo's hooks)
|
|
91
|
+
don't pollute generalizable patterns shipped to user installs,
|
|
92
|
+
instead of relying on judgment under deadline.
|
|
93
|
+
|
|
94
|
+
As a PAN maintainer running 5 experiments overnight on different ideas,
|
|
95
|
+
I want each experiment isolated in its own folder, with its own budget cap,
|
|
96
|
+
and a single command (/pan:experiment harvest --all) to bring telemetry back the next morning,
|
|
97
|
+
so that overnight automation is trustworthy,
|
|
98
|
+
instead of waking up to a runaway external process or merged-state corruption.
|
|
99
|
+
|
|
100
|
+
As a PAN user (downstream consumer),
|
|
101
|
+
I want PAN's shipped behavioral docs to evolve based on actual usage signals,
|
|
102
|
+
so that recurring AI failure modes get codified as guardrails before they bite me,
|
|
103
|
+
instead of running into the same shortcut shortcut my teammates already encountered last quarter.
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
---
|
|
107
|
+
|
|
108
|
+
## Phase 1: Internal Reconnaissance
|
|
109
|
+
|
|
110
|
+
### 1.1 Existing Pieces (already shipped)
|
|
111
|
+
|
|
112
|
+
The 70% that exists:
|
|
113
|
+
|
|
114
|
+
| Capability | Module / file | Notes |
|
|
115
|
+
|------------|---------------|-------|
|
|
116
|
+
| Per-session trace logging | `hooks/pan-trace-logger.js` (5th hook, SubagentStop) | Auto-creates day-scoped sessions, logs `decision:agent_completion` events |
|
|
117
|
+
| Trace storage | `pan-wizard-core/bin/lib/optimize.cjs` `getOptimizeDir()` | `.planning/optimization/traces/sess_<id>/` |
|
|
118
|
+
| Pattern analysis | `pan-wizard-core/bin/lib/optimize.cjs` `analyzeEvents()` | Frequency maps, agent stats, error rates, redundancy detection |
|
|
119
|
+
| Report generation | `pan-wizard-core/bin/lib/optimize.cjs` `generateLocalReport()` + `pan-optimizer` agent | Structured findings with `## Auto-Apply Actions` JSON block |
|
|
120
|
+
| Apply reports | `pan-wizard-core/bin/lib/optimize.cjs` `applyReportRecommendations()` | Writes to `.planning/memory/`, project config |
|
|
121
|
+
| `/pan:learn` command | `commands/pan/learn.md` + `pan-wizard-core/workflows/learn.md` | Invokes pan-optimizer, shows score |
|
|
122
|
+
| `/pan:optimize` command | `commands/pan/optimize.md` + `optimize.cjs` cmd functions | trace init/end/status/list, learn, apply, list, stats |
|
|
123
|
+
| Behavioral content delivery | `pan-wizard-core/references/*.md` (14 files), workflow cross-references | Proven channel for AI-readable shipped content (v3.6.0) |
|
|
124
|
+
| External-process orchestration | `tests/helpers.cjs` `createScenarioRunner()` | Spawns fresh installer in temp dir, returns runner with `run()` + `cleanup()` |
|
|
125
|
+
| VSCode-specific automation | `tests/e2e/vscode-helpers.mjs` (Session 33) | Playwright + @vscode/test-electron — exists but unnecessary for autonomous CLI runs |
|
|
126
|
+
| Budget + circuit-breaker pattern | `pan-wizard-core/bin/lib/focus.cjs` (focus-auto) | Triple safety: per-cycle budget + cumulative cap + max cycles + zero-progress halt |
|
|
127
|
+
|
|
128
|
+
### 1.2 Missing Pieces (the 30% to build)
|
|
129
|
+
|
|
130
|
+
| Gap | Module / artifact | Effort |
|
|
131
|
+
|-----|------------------|--------|
|
|
132
|
+
| Experiment lifecycle data | `pan-wizard-core/bin/lib/experiment.cjs` (28th core module) | M (4 pts) |
|
|
133
|
+
| Experiment commands | `commands/pan/experiment.md` (52nd command) | S (2 pts) |
|
|
134
|
+
| External agent runner | `pan-wizard-core/bin/lib/runner.cjs` (29th core module) — subprocess invocation, output streaming | M (4 pts) |
|
|
135
|
+
| Idea-doc schema | `pan-wizard-core/templates/idea.md` (27th template) | XS (1 pt) |
|
|
136
|
+
| Harvest logic | `experiment.cjs` `harvestExperiment()` | S (2 pts) |
|
|
137
|
+
| `pan-experiment-runner` agent | `agents/pan-experiment-runner.md` (21st agent) — drives the external instance | M (4 pts) |
|
|
138
|
+
| Promote logic | `pan-wizard-core/bin/lib/optimize.cjs` `promotePattern()` | M (4 pts) |
|
|
139
|
+
| Two-tier learnings dir | `pan-wizard-core/learnings/universal/` and `pan-wizard-core/learnings/internal/` | XS (1 pt) |
|
|
140
|
+
| Installer ships `learnings/universal/` only | `bin/install.js` (verify glob; likely already covers any new top-level dir under pan-wizard-core/) | XS (1 pt) |
|
|
141
|
+
| Workflow re-references | exec-phase, plan-phase, focus-exec workflows reference `learnings/universal/` | S (2 pts) |
|
|
142
|
+
| Tests | `experiment.test.cjs`, `runner.test.cjs`, `learn-promote.test.cjs`, `scenarios/learnings-installed.test.cjs` | M-L (10-15 pts) |
|
|
143
|
+
| Documentation | USER-GUIDE, ARCHITECTURE updates + ADR-0026 | S (2 pts) |
|
|
144
|
+
|
|
145
|
+
**Total estimated effort:** ~30-40 pts. XL feature; ships across 3-4 sessions/waves.
|
|
146
|
+
|
|
147
|
+
### 1.3 Architectural Constraints
|
|
148
|
+
|
|
149
|
+
| Constraint | Source | Impact |
|
|
150
|
+
|------------|--------|--------|
|
|
151
|
+
| Source repo never receives a PAN install | CLAUDE.md "Self-Protection Gate" | Experiments MUST live outside `d:\PanWizard\` |
|
|
152
|
+
| Manifest tracks every shipped file | Existing v3.5+ behavior | New `learnings/universal/*.md` files appear in manifest checksum |
|
|
153
|
+
| 5-runtime parity is mandatory | CLAUDE.md "5 Target Runtimes" | Shipped learnings install identically to all 5 runtime dirs |
|
|
154
|
+
| Zero runtime deps | Existing project rule | Subprocess invocation uses `child_process.spawn` (Node built-in) — no new packages |
|
|
155
|
+
| Behavioral content is markdown + frontmatter | `references/`, `workflows/`, `agents/` patterns | `learnings/universal/*.md` follows the same shape |
|
|
156
|
+
|
|
157
|
+
### 1.4 Cross-Runtime Compatibility
|
|
158
|
+
|
|
159
|
+
| Runtime | External invocation | Headless support | Notes |
|
|
160
|
+
|---------|---------------------|-------------------|-------|
|
|
161
|
+
| Claude Code | `claude -p "<prompt>"` or via Skills/MCP | ✅ headless via `-p` flag | Primary target |
|
|
162
|
+
| Codex CLI | `codex exec "<prompt>"` (or similar) | ✅ | Verify exact flag in Codex docs at implementation time |
|
|
163
|
+
| Gemini CLI | `gemini -p "<prompt>"` | ✅ | Verify flag |
|
|
164
|
+
| OpenCode | `opencode "<prompt>"` | ✅ | Verify flag |
|
|
165
|
+
| GitHub Copilot CLI | (no general-purpose subprocess prompt mode known) | ❌ | Opt-out for now; document limitation |
|
|
166
|
+
|
|
167
|
+
The runner module abstracts these via a per-runtime adapter pattern (similar to existing installer adapters). Configuration: `pan-tools experiment run <id> --runtime claude` (defaults to the runtime PAN was installed for).
|
|
168
|
+
|
|
169
|
+
---
|
|
170
|
+
|
|
171
|
+
## Phase 2: Source Pattern Analysis
|
|
172
|
+
|
|
173
|
+
### 2.1 Patterns Adopted
|
|
174
|
+
|
|
175
|
+
**From `focus.cjs` (focus-auto runner — Session 26):**
|
|
176
|
+
- Triple safety net: per-cycle budget + cumulative budget + max cycles + zero-progress circuit breaker
|
|
177
|
+
- Run state in `.planning/focus/auto-run.json` — cross-session resumability via `--continue`
|
|
178
|
+
- `determineStopReason()` enumerated stop reasons returned in JSON
|
|
179
|
+
|
|
180
|
+
The experiment runner inherits all of these. State file: `.planning/experiments/<id>/run-state.json`.
|
|
181
|
+
|
|
182
|
+
**From `optimize.cjs` (circular optimization loop — Session 31):**
|
|
183
|
+
- Trace session lifecycle (`initTraceSession`, `logTraceEvent`, `endTraceSession`)
|
|
184
|
+
- Local analysis (`analyzeEvents`) producing structured findings
|
|
185
|
+
- Auto-apply JSON block at the end of optimizer reports
|
|
186
|
+
- `applyReportRecommendations()` for surgical writes
|
|
187
|
+
|
|
188
|
+
The promote step extends the apply pattern: instead of writing to `.planning/memory/` (intra-project), it writes to `pan-wizard-core/learnings/{universal,internal}/<topic>.md` (source repo, shippable on next release).
|
|
189
|
+
|
|
190
|
+
**From `tests/helpers.cjs` (`createScenarioRunner` — Session 18):**
|
|
191
|
+
- `child_process.execSync` invocation pattern
|
|
192
|
+
- Temp dir lifecycle (`mkdtempSync` + `rmSync` cleanup)
|
|
193
|
+
- Per-runtime adapter via `RUNTIME_DIR` map
|
|
194
|
+
|
|
195
|
+
The runner module generalizes this from "install + run command" to "scaffold + drive autonomous build."
|
|
196
|
+
|
|
197
|
+
**From v3.6.0 (`references/guardrails.md` shipped to 5 runtimes):**
|
|
198
|
+
- Single-file behavioral content shipped via `references/` directory
|
|
199
|
+
- Workflows reference by relative path
|
|
200
|
+
- All 5 runtimes get the file through existing installer glob
|
|
201
|
+
|
|
202
|
+
The `learnings/universal/` directory mirrors this pattern exactly — same delivery channel, AI-derived rather than human-authored content.
|
|
203
|
+
|
|
204
|
+
### 2.2 Patterns Explicitly NOT Adopted
|
|
205
|
+
|
|
206
|
+
- **Auto-promote without human review.** Risky in v3.7. The promote step is gated on a human pressing the trigger (`pan-tools learn promote --pattern <id>`). Auto-promote ramps in v3.8+ once we have N=10+ successful manual promotes to learn the filter.
|
|
207
|
+
- **VSCode UI automation as primary path.** Playwright + @vscode/test-electron infrastructure exists (Session 33), but using it for autonomous external runs adds: (a) GUI dependency, (b) headless display setup, (c) cross-platform display drivers. CLI subprocess is sufficient for the autonomous case. VSCode UI automation stays for testing purposes.
|
|
208
|
+
- **Bidirectional control mid-run.** External agent runs autonomously; PAN observes via file-tail, doesn't inject prompts. Cleaner failure semantics — if the external agent gets stuck, the timeout/circuit-breaker fires; PAN doesn't try to "save" it. Rationale: every intervention path is a new bug surface.
|
|
209
|
+
- **Multi-experiment aggregation.** v3.7 ships single-experiment promote. Aggregation across N experiments is v3.8 — needs more design (which findings are universal vs idiosyncratic to N specific experiments?).
|
|
210
|
+
|
|
211
|
+
---
|
|
212
|
+
|
|
213
|
+
## Phase 3: Design
|
|
214
|
+
|
|
215
|
+
### 3.1 Architecture Overview
|
|
216
|
+
|
|
217
|
+
```
|
|
218
|
+
PAN source repo (d:\PanWizard\)
|
|
219
|
+
──────────────────────────────────
|
|
220
|
+
1. /pan:experiment new my-idea
|
|
221
|
+
↓
|
|
222
|
+
2. Creates D:\pan-experiments\my-idea\
|
|
223
|
+
├── .planning/
|
|
224
|
+
│ ├── state.md
|
|
225
|
+
│ ├── idea.md ← from --idea <file>
|
|
226
|
+
│ └── ...
|
|
227
|
+
└── (PAN install for chosen runtime)
|
|
228
|
+
|
|
229
|
+
3. /pan:experiment run my-idea --runtime claude
|
|
230
|
+
↓
|
|
231
|
+
runner.cjs spawns: claude -p "/pan:new-project --auto @idea.md"
|
|
232
|
+
↓
|
|
233
|
+
External Claude Code session runs autonomously inside the experiment folder.
|
|
234
|
+
Builds the idea using the same PAN workflows we just shipped (with guardrails!)
|
|
235
|
+
|
|
236
|
+
4. Trace logger inside experiment captures every agent spawn.
|
|
237
|
+
PAN session tails experiment's state.md / agent-history.json / summaries.
|
|
238
|
+
|
|
239
|
+
5. /pan:experiment harvest my-idea
|
|
240
|
+
↓
|
|
241
|
+
Copies experiment's .planning/optimization/, agent-history.json,
|
|
242
|
+
state.md, summary files BACK to source repo:
|
|
243
|
+
d:\PanWizard\experiments\my-idea\
|
|
244
|
+
├── traces/
|
|
245
|
+
├── reports/
|
|
246
|
+
└── snapshot/
|
|
247
|
+
|
|
248
|
+
6. /pan:learn --experiment my-idea
|
|
249
|
+
↓
|
|
250
|
+
pan-optimizer (existing agent) reads experiments/my-idea/, generates:
|
|
251
|
+
experiments/my-idea/learnings/report-<timestamp>.md
|
|
252
|
+
with ## Findings + ## Auto-Apply Actions JSON
|
|
253
|
+
|
|
254
|
+
7. pan-tools learn promote --pattern <id> --scope universal
|
|
255
|
+
↓
|
|
256
|
+
Extracts the finding into:
|
|
257
|
+
pan-wizard-core/learnings/universal/<topic>.md
|
|
258
|
+
|
|
259
|
+
8. /pan:check → npm test → git commit → v3.7.x release
|
|
260
|
+
↓
|
|
261
|
+
Next user install gets the finding shipped under
|
|
262
|
+
.claude/pan-wizard-core/learnings/universal/<topic>.md
|
|
263
|
+
Workflows already reference learnings/universal/ — auto-loaded.
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
### 3.2 Module Plan
|
|
267
|
+
|
|
268
|
+
#### `pan-wizard-core/bin/lib/experiment.cjs` (new — 28th module)
|
|
269
|
+
|
|
270
|
+
```js
|
|
271
|
+
// Pure-ish functions. Side effects only at write boundaries.
|
|
272
|
+
|
|
273
|
+
const PAN_EXPERIMENTS_ROOT_DEFAULT = path.join(os.homedir(), 'pan-experiments');
|
|
274
|
+
// Configurable via .planning/config.json `experiments_root`
|
|
275
|
+
|
|
276
|
+
function newExperiment(slug, opts) {
|
|
277
|
+
// opts: { ideaPath, runtime, root, budget }
|
|
278
|
+
// 1. Validate slug (lowercase, hyphens, max 40 chars)
|
|
279
|
+
// 2. Compute path: <root>/<slug>
|
|
280
|
+
// 3. Refuse if path exists (no clobber)
|
|
281
|
+
// 4. Refuse if path is inside source repo (PAN_SOURCE_ROOT guard)
|
|
282
|
+
// 5. mkdir, copy idea.md to <path>/.planning/idea.md (or write template)
|
|
283
|
+
// 6. Run installer: node <source>/bin/install.js --<runtime> --local at <path>
|
|
284
|
+
// 7. Write <path>/.planning/experiments/<id>/manifest.json with metadata
|
|
285
|
+
// 8. Return { experiment_id, path, runtime, idea_path, created_at }
|
|
286
|
+
}
|
|
287
|
+
|
|
288
|
+
function listExperiments(opts) {
|
|
289
|
+
// List all experiments under root, with status, runtime, created_at
|
|
290
|
+
}
|
|
291
|
+
|
|
292
|
+
function getExperimentManifest(slug, opts) {
|
|
293
|
+
// Read <root>/<slug>/.planning/experiments/<slug>/manifest.json
|
|
294
|
+
}
|
|
295
|
+
|
|
296
|
+
function harvestExperiment(slug, opts) {
|
|
297
|
+
// 1. Locate experiment path
|
|
298
|
+
// 2. Source: <experiment>/.planning/optimization/, agent-history.json, state.md, phases/
|
|
299
|
+
// 3. Destination: d:\PanWizard\experiments\<slug>\
|
|
300
|
+
// 4. Copy with directory structure preserved
|
|
301
|
+
// 5. Write harvest manifest with timestamp + size + checksum
|
|
302
|
+
// 6. Return { harvested_files, total_bytes, harvest_path }
|
|
303
|
+
}
|
|
304
|
+
|
|
305
|
+
function pruneExperiment(slug, opts) {
|
|
306
|
+
// After harvest + promote, optionally delete the experiment folder
|
|
307
|
+
// Soft delete: rename to <slug>-archived-<timestamp>
|
|
308
|
+
}
|
|
309
|
+
|
|
310
|
+
module.exports = {
|
|
311
|
+
newExperiment, listExperiments, getExperimentManifest,
|
|
312
|
+
harvestExperiment, pruneExperiment,
|
|
313
|
+
PAN_EXPERIMENTS_ROOT_DEFAULT,
|
|
314
|
+
};
|
|
315
|
+
```
|
|
316
|
+
|
|
317
|
+
#### `pan-wizard-core/bin/lib/runner.cjs` (new — 29th module)
|
|
318
|
+
|
|
319
|
+
```js
|
|
320
|
+
// External agent subprocess orchestration.
|
|
321
|
+
// Per-runtime adapter pattern.
|
|
322
|
+
|
|
323
|
+
const RUNTIME_RUNNERS = {
|
|
324
|
+
claude: { bin: 'claude', args: (prompt) => ['-p', prompt] },
|
|
325
|
+
codex: { bin: 'codex', args: (prompt) => ['exec', prompt] },
|
|
326
|
+
gemini: { bin: 'gemini', args: (prompt) => ['-p', prompt] },
|
|
327
|
+
opencode: { bin: 'opencode', args: (prompt) => [prompt] },
|
|
328
|
+
copilot: null, // unsupported (no headless prompt mode)
|
|
329
|
+
};
|
|
330
|
+
|
|
331
|
+
function runExperiment(slug, opts) {
|
|
332
|
+
// opts: { runtime, prompt, timeout, budget, onProgress }
|
|
333
|
+
// 1. Validate runtime support
|
|
334
|
+
// 2. Build command args
|
|
335
|
+
// 3. spawn child_process.spawn() in experiment dir
|
|
336
|
+
// 4. Stream stdout/stderr via opts.onProgress callback (line-buffered)
|
|
337
|
+
// 5. Watch experiment's .planning/state.md / agent-history.json for status
|
|
338
|
+
// 6. Enforce timeout
|
|
339
|
+
// 7. Enforce circuit breaker (3 consecutive task failures)
|
|
340
|
+
// 8. Persist run-state.json after every status change
|
|
341
|
+
// 9. Return { exit_code, status, elapsed_ms, summaries_count, stop_reason }
|
|
342
|
+
}
|
|
343
|
+
|
|
344
|
+
function tailExperimentState(slug, opts) {
|
|
345
|
+
// File-watcher loop on state.md, agent-history.json
|
|
346
|
+
// Emits `state_change`, `agent_spawned`, `agent_completed`, `summary_written` events
|
|
347
|
+
// Used by runExperiment internally; also exposed for monitoring without re-running
|
|
348
|
+
}
|
|
349
|
+
|
|
350
|
+
function stopExperiment(slug) {
|
|
351
|
+
// Read PID from run-state.json, send SIGTERM, fall back to SIGKILL
|
|
352
|
+
}
|
|
353
|
+
|
|
354
|
+
module.exports = { runExperiment, tailExperimentState, stopExperiment, RUNTIME_RUNNERS };
|
|
355
|
+
```
|
|
356
|
+
|
|
357
|
+
#### `pan-wizard-core/bin/lib/optimize.cjs` (extend existing — add `promotePattern`)
|
|
358
|
+
|
|
359
|
+
```js
|
|
360
|
+
// New export, alongside existing analyzeEvents, generateLocalReport, etc.
|
|
361
|
+
|
|
362
|
+
function promotePattern(reportPath, patternId, opts) {
|
|
363
|
+
// opts: { scope: 'universal'|'internal', topic: string, sourceRoot: string }
|
|
364
|
+
// 1. Read report file, extract the named pattern from ## Findings
|
|
365
|
+
// 2. Validate pattern has: id, summary, evidence (>= 1 trace event), proposed_rule
|
|
366
|
+
// 3. Compute output path:
|
|
367
|
+
// universal → <sourceRoot>/pan-wizard-core/learnings/universal/<topic>.md
|
|
368
|
+
// internal → <sourceRoot>/pan-wizard-core/learnings/internal/<topic>.md
|
|
369
|
+
// 4. Append (or create) the topic file with the new pattern
|
|
370
|
+
// 5. Front-matter: pattern_ids list, last_updated, source_experiments list
|
|
371
|
+
// 6. Return { promoted_to: path, pattern_id, scope, topic }
|
|
372
|
+
}
|
|
373
|
+
|
|
374
|
+
function listPromotedPatterns(sourceRoot) {
|
|
375
|
+
// Walk learnings/{universal,internal}/, return inventory
|
|
376
|
+
}
|
|
377
|
+
|
|
378
|
+
function unpromotePattern(topicFile, patternId) {
|
|
379
|
+
// Remove a pattern from a topic file (for rollback)
|
|
380
|
+
}
|
|
381
|
+
```
|
|
382
|
+
|
|
383
|
+
### 3.3 Command Surface
|
|
384
|
+
|
|
385
|
+
#### `commands/pan/experiment.md` (new — 52nd command)
|
|
386
|
+
|
|
387
|
+
```
|
|
388
|
+
/pan:experiment new <slug> --idea <path> [--runtime <r>] [--root <path>] [--budget <pts>]
|
|
389
|
+
/pan:experiment list [--status <pending|running|done|failed>]
|
|
390
|
+
/pan:experiment run <slug> [--runtime <r>] [--timeout <sec>] [--no-stream]
|
|
391
|
+
/pan:experiment status <slug>
|
|
392
|
+
/pan:experiment stop <slug>
|
|
393
|
+
/pan:experiment harvest <slug> [--prune]
|
|
394
|
+
/pan:experiment harvest --all
|
|
395
|
+
/pan:experiment archive <slug>
|
|
396
|
+
/pan:experiment delete <slug> --confirm
|
|
397
|
+
```
|
|
398
|
+
|
|
399
|
+
CLI subcommands (`pan-tools experiment <sub>`):
|
|
400
|
+
- `experiment new <slug> --idea <path>`
|
|
401
|
+
- `experiment list`
|
|
402
|
+
- `experiment manifest <slug>`
|
|
403
|
+
- `experiment run <slug>` (or via `runner.cjs` directly)
|
|
404
|
+
- `experiment harvest <slug>`
|
|
405
|
+
- `experiment prune <slug>`
|
|
406
|
+
|
|
407
|
+
#### Extended `/pan:learn` and `/pan:optimize`
|
|
408
|
+
|
|
409
|
+
```
|
|
410
|
+
/pan:learn --experiment <slug> # NEW: run pan-optimizer over harvested experiment
|
|
411
|
+
pan-tools learn promote --pattern <id> --scope universal --topic <name>
|
|
412
|
+
pan-tools learn unpromote --pattern <id> --topic <name>
|
|
413
|
+
pan-tools learn list-promoted
|
|
414
|
+
```
|
|
415
|
+
|
|
416
|
+
### 3.4 New Agent: `pan-experiment-runner`
|
|
417
|
+
|
|
418
|
+
```yaml
|
|
419
|
+
---
|
|
420
|
+
name: pan-experiment-runner
|
|
421
|
+
description: Drives an external AI coding session against an experiment folder. Tails state, observes summaries, decides when the build is "done enough." Read-only relative to PAN source; writes only to <experiment-folder>/.planning/.
|
|
422
|
+
tools: Read, Bash, Glob, Grep
|
|
423
|
+
color: orange
|
|
424
|
+
thinking: enabled
|
|
425
|
+
thinking_budget: 6000
|
|
426
|
+
---
|
|
427
|
+
|
|
428
|
+
<role>
|
|
429
|
+
You are the experiment runner. You drive an AUTONOMOUS external Claude Code (or other CLI) instance against an isolated experiment folder. Your job: observe, not intervene.
|
|
430
|
+
|
|
431
|
+
You are NOT the agent doing the build. You are the watchdog. The external instance does the actual work.
|
|
432
|
+
|
|
433
|
+
**You may NOT:**
|
|
434
|
+
- Edit files in the experiment folder
|
|
435
|
+
- Inject prompts into the external instance
|
|
436
|
+
- Modify the source repo
|
|
437
|
+
|
|
438
|
+
**You MAY:**
|
|
439
|
+
- Read the experiment's .planning/ state
|
|
440
|
+
- Tail the experiment's agent-history.json
|
|
441
|
+
- Decide when to declare success / failure / timeout based on stop conditions
|
|
442
|
+
- Update PAN's .planning/experiments/<slug>/run-state.json
|
|
443
|
+
- Surface progress to the orchestrating user
|
|
444
|
+
</role>
|
|
445
|
+
|
|
446
|
+
<stop_conditions>
|
|
447
|
+
1. **success**: experiment's state.md status = "done" AND the milestone summary file exists
|
|
448
|
+
2. **budget_exhausted**: external instance's cumulative trace event count > opts.budget * SPEND_RATIO
|
|
449
|
+
3. **timeout**: elapsed wall-clock > opts.timeout
|
|
450
|
+
4. **circuit_breaker**: 3 consecutive agent failures in agent-history.json
|
|
451
|
+
5. **manual_stop**: PAN session writes <experiment>/.planning/STOP signal file
|
|
452
|
+
</stop_conditions>
|
|
453
|
+
|
|
454
|
+
<observations_to_log>
|
|
455
|
+
Every state transition writes a trace event in PAN's experiment record:
|
|
456
|
+
- agent_spawned (with task description)
|
|
457
|
+
- agent_completed (with status, elapsed)
|
|
458
|
+
- summary_written (with phase, plan)
|
|
459
|
+
- state_change (with old → new status)
|
|
460
|
+
- stop_condition_hit (with reason)
|
|
461
|
+
</observations_to_log>
|
|
462
|
+
```
|
|
463
|
+
|
|
464
|
+
### 3.5 Idea Document Schema
|
|
465
|
+
|
|
466
|
+
`pan-wizard-core/templates/idea.md`:
|
|
467
|
+
|
|
468
|
+
```markdown
|
|
469
|
+
---
|
|
470
|
+
title: <one-line idea title>
|
|
471
|
+
created: <ISO-8601>
|
|
472
|
+
created_by: <user>
|
|
473
|
+
runtime_preference: claude # claude|codex|gemini|opencode
|
|
474
|
+
budget: 80 # XS-XL points
|
|
475
|
+
priority: medium # low|medium|high
|
|
476
|
+
---
|
|
477
|
+
|
|
478
|
+
# Idea: <title>
|
|
479
|
+
|
|
480
|
+
## Problem
|
|
481
|
+
What user pain does this address?
|
|
482
|
+
|
|
483
|
+
## Success Criteria
|
|
484
|
+
What does "this works" look like? Be concrete.
|
|
485
|
+
|
|
486
|
+
## Scope
|
|
487
|
+
| In Scope | Out of Scope |
|
|
488
|
+
|----------|--------------|
|
|
489
|
+
| ... | ... |
|
|
490
|
+
|
|
491
|
+
## Constraints
|
|
492
|
+
- Tech stack
|
|
493
|
+
- Dependencies
|
|
494
|
+
- Deadlines
|
|
495
|
+
- Runtime support requirements
|
|
496
|
+
|
|
497
|
+
## Reference Material (optional)
|
|
498
|
+
- @path/to/related-doc.md
|
|
499
|
+
- https://link/to/inspiration
|
|
500
|
+
|
|
501
|
+
## Notes (optional)
|
|
502
|
+
Free-form context the external agent should know.
|
|
503
|
+
```
|
|
504
|
+
|
|
505
|
+
### 3.6 Two-Tier Learnings Layout
|
|
506
|
+
|
|
507
|
+
```
|
|
508
|
+
pan-wizard-core/
|
|
509
|
+
├── references/ # human-authored, shipped to all installs (existing — guardrails, tdd, etc.)
|
|
510
|
+
├── learnings/ # NEW
|
|
511
|
+
│ ├── universal/ # AI-derived, shipped to all installs (consumed by user-project workflows)
|
|
512
|
+
│ │ ├── exec-patterns.md
|
|
513
|
+
│ │ ├── plan-patterns.md
|
|
514
|
+
│ │ ├── verification-patterns.md
|
|
515
|
+
│ │ └── debugging-patterns.md
|
|
516
|
+
│ └── internal/ # AI-derived, source-only (consumed when working on PAN itself)
|
|
517
|
+
│ ├── pan-dev-patterns.md
|
|
518
|
+
│ └── installer-patterns.md
|
|
519
|
+
└── ...
|
|
520
|
+
```
|
|
521
|
+
|
|
522
|
+
Each topic file structure:
|
|
523
|
+
|
|
524
|
+
```markdown
|
|
525
|
+
---
|
|
526
|
+
topic: exec-patterns
|
|
527
|
+
last_updated: 2026-04-30
|
|
528
|
+
patterns:
|
|
529
|
+
- id: P-001
|
|
530
|
+
summary: Always run npm test:all before phase complete
|
|
531
|
+
promoted_at: 2026-04-30
|
|
532
|
+
source_experiments: [my-idea-1, my-idea-2]
|
|
533
|
+
- id: P-002
|
|
534
|
+
summary: When verification fails on missing artifact, scaffold first then fix
|
|
535
|
+
promoted_at: 2026-05-02
|
|
536
|
+
source_experiments: [my-idea-3]
|
|
537
|
+
---
|
|
538
|
+
|
|
539
|
+
# Exec Patterns (AI-derived)
|
|
540
|
+
|
|
541
|
+
This file is auto-maintained by `pan-tools learn promote`. Each pattern was
|
|
542
|
+
extracted from one or more experiment runs (see source_experiments). Patterns
|
|
543
|
+
are advisory — the orchestrator should weight them against current context.
|
|
544
|
+
|
|
545
|
+
## P-001 — Always run `npm test:all` before phase complete
|
|
546
|
+
|
|
547
|
+
**Evidence:** 14 trace events across experiments my-idea-1, my-idea-2 where
|
|
548
|
+
`/pan:verify-phase` passed but a downstream scenario test caught a regression.
|
|
549
|
+
|
|
550
|
+
**Rule:** Before marking a phase complete, run the full test suite, not just
|
|
551
|
+
the phase's own tests. If `test:all` script exists, prefer it over `test`.
|
|
552
|
+
|
|
553
|
+
**Applies in:** exec-phase, focus-exec, milestone-done
|
|
554
|
+
|
|
555
|
+
## P-002 — ...
|
|
556
|
+
```
|
|
557
|
+
|
|
558
|
+
### 3.7 Installer Changes
|
|
559
|
+
|
|
560
|
+
`bin/install.js` glob already covers any new top-level dir under `pan-wizard-core/`. Verify:
|
|
561
|
+
|
|
562
|
+
```js
|
|
563
|
+
// Around bin/install.js line ~XXX (verify exact location at impl)
|
|
564
|
+
const corePaths = [
|
|
565
|
+
'pan-wizard-core/bin',
|
|
566
|
+
'pan-wizard-core/workflows',
|
|
567
|
+
'pan-wizard-core/templates',
|
|
568
|
+
'pan-wizard-core/references',
|
|
569
|
+
'pan-wizard-core/learnings/universal', // NEW — explicitly include
|
|
570
|
+
// 'pan-wizard-core/learnings/internal' INTENTIONALLY OMITTED
|
|
571
|
+
];
|
|
572
|
+
```
|
|
573
|
+
|
|
574
|
+
Adding `learnings/universal` explicitly (instead of relying on glob) makes the universal-vs-internal split discoverable in install.js. Tests assert `learnings/internal/` is NOT in any install dir.
|
|
575
|
+
|
|
576
|
+
### 3.8 Workflow Cross-References
|
|
577
|
+
|
|
578
|
+
Following the v3.6.0 pattern, 4 workflows gain a reference to `learnings/universal/`:
|
|
579
|
+
|
|
580
|
+
```markdown
|
|
581
|
+
<required_reading>
|
|
582
|
+
@~/.claude/pan-wizard-core/references/guardrails.md
|
|
583
|
+
@~/.claude/pan-wizard-core/learnings/universal/ <!-- NEW: directory glob loads all topic files -->
|
|
584
|
+
</required_reading>
|
|
585
|
+
```
|
|
586
|
+
|
|
587
|
+
Workflows updated:
|
|
588
|
+
- `exec-phase.md`
|
|
589
|
+
- `plan-phase.md`
|
|
590
|
+
- `verify-phase.md`
|
|
591
|
+
- `execute-plan.md`
|
|
592
|
+
|
|
593
|
+
### 3.9 End-to-End Flow Walkthrough
|
|
594
|
+
|
|
595
|
+
User runs:
|
|
596
|
+
|
|
597
|
+
```bash
|
|
598
|
+
# 1. Capture an idea
|
|
599
|
+
echo "# Idea: Build a tiny markdown linter CLI" > my-idea.md
|
|
600
|
+
# (fill in problem, success, scope, constraints)
|
|
601
|
+
|
|
602
|
+
# 2. Scaffold experiment
|
|
603
|
+
pan-tools experiment new md-lint --idea my-idea.md --runtime claude --budget 60
|
|
604
|
+
# → creates D:/pan-experiments/md-lint/, installs PAN for claude, copies idea.md
|
|
605
|
+
|
|
606
|
+
# 3. Run the experiment (autonomous external Claude Code)
|
|
607
|
+
pan-tools experiment run md-lint
|
|
608
|
+
# → spawns: claude -p "/pan:new-project --auto @.planning/idea.md"
|
|
609
|
+
# inside D:/pan-experiments/md-lint/
|
|
610
|
+
# → tails its state.md; PAN session prints periodic progress
|
|
611
|
+
|
|
612
|
+
# 4. After ~10-30 min, experiment completes (or hits a stop condition)
|
|
613
|
+
pan-tools experiment harvest md-lint
|
|
614
|
+
# → copies optimization/, agent-history, state, summaries to
|
|
615
|
+
# d:/PanWizard/experiments/md-lint/
|
|
616
|
+
|
|
617
|
+
# 5. Run pan-optimizer over harvested data
|
|
618
|
+
/pan:learn --experiment md-lint
|
|
619
|
+
# → produces d:/PanWizard/experiments/md-lint/learnings/report-<ts>.md
|
|
620
|
+
|
|
621
|
+
# 6. Review report. Promote select findings:
|
|
622
|
+
pan-tools learn promote --pattern P-007 --scope universal --topic exec-patterns
|
|
623
|
+
# → appends pattern to pan-wizard-core/learnings/universal/exec-patterns.md
|
|
624
|
+
|
|
625
|
+
pan-tools learn promote --pattern P-013 --scope internal --topic pan-dev-patterns
|
|
626
|
+
# → appends to pan-wizard-core/learnings/internal/pan-dev-patterns.md
|
|
627
|
+
|
|
628
|
+
# 7. Verify shipping behavior
|
|
629
|
+
npm run test:all # asserts learnings/universal/ ships, internal/ does not
|
|
630
|
+
node bin/install.js --claude --local # in d:/pantesting, verify universal/ present, internal/ absent
|
|
631
|
+
|
|
632
|
+
# 8. Ship as v3.7.x
|
|
633
|
+
# CHANGELOG entry. ADR entry. Commit. Tag.
|
|
634
|
+
```
|
|
635
|
+
|
|
636
|
+
### 3.10 Implementation Plan (4 Waves Across v3.7.0 → v3.7.3)
|
|
637
|
+
|
|
638
|
+
| Wave | Version | Items | Pts | Days |
|
|
639
|
+
|------|---------|-------|-----|------|
|
|
640
|
+
| W1: Experiment scaffolding + idea schema | v3.7.0 | experiment.cjs (new + list + manifest), idea.md template, experiment.md command (subset), tests | 12 | 1 |
|
|
641
|
+
| W2: External runner + observation | v3.7.1 | runner.cjs, pan-experiment-runner agent, run/status/stop subcommands, file-tail, run-state.json | 10 | 1 |
|
|
642
|
+
| W3: Harvest + extended /pan:learn | v3.7.2 | experiment.harvestExperiment(), `/pan:learn --experiment`, harvest/prune subcommands, scenario tests | 8 | 1 |
|
|
643
|
+
| W4: Promote + ship | v3.7.3 | optimize.promotePattern(), learn promote/unpromote/list-promoted, learnings/{universal,internal}/ tier, installer + workflow refs, ADR-0026 | 10 | 1 |
|
|
644
|
+
|
|
645
|
+
**Total: ~40 pts, 4 sessions, ships across 4 patch versions OR consolidated as v3.7.0 if shipped as a single wave.**
|
|
646
|
+
|
|
647
|
+
Recommendation: ship as **v3.7.0 single wave** (40 pts is doable in one focused session with the spec already written). Each wave above can also be a separate session if the user prefers staged rollout.
|
|
648
|
+
|
|
649
|
+
### 3.11 Test Plan
|
|
650
|
+
|
|
651
|
+
Target: 30+ new tests.
|
|
652
|
+
|
|
653
|
+
#### Unit tests
|
|
654
|
+
|
|
655
|
+
- `tests/experiment.test.cjs` (8-10 tests): newExperiment validation, root path, slug rules, manifest, list, prune
|
|
656
|
+
- `tests/runner.test.cjs` (6-8 tests): runtime adapter selection, mocked subprocess spawn, timeout enforcement, circuit breaker, run-state persistence
|
|
657
|
+
- `tests/learn-promote.test.cjs` (6-8 tests): promotePattern shape, scope routing, topic file append, unpromote round-trip, list-promoted
|
|
658
|
+
|
|
659
|
+
#### Scenario tests
|
|
660
|
+
|
|
661
|
+
- `tests/scenarios/learnings-installed.test.cjs` (10 tests): `learnings/universal/*.md` present in all 5 runtime installs, `learnings/internal/` ABSENT in all 5 runtime installs (negative assertion), idea.md template ships, experiment.md command ships
|
|
662
|
+
- `tests/scenarios/experiment-lifecycle.test.cjs` (5 tests): mocked end-to-end — new → manifest → harvest → promote — without spawning a real external process
|
|
663
|
+
|
|
664
|
+
#### Integration tests (optional, opt-in via env flag)
|
|
665
|
+
|
|
666
|
+
- `tests/integration/real-experiment.test.cjs` — gated by `PAN_REAL_EXPERIMENT=1`, runs an actual `claude -p` against a trivial idea. Skipped in CI by default; available for manual smoke before each release.
|
|
667
|
+
|
|
668
|
+
### 3.12 Documentation
|
|
669
|
+
|
|
670
|
+
- `docs/USER-GUIDE.md` — new "Self-Improvement Loop" section under Advanced Features, walking through the end-to-end flow
|
|
671
|
+
- `docs/ARCHITECTURE.md` — new "v3.7.0 additions — Self-Improvement Loop" section listing the 2 new modules + new agent + 2 new top-level dirs
|
|
672
|
+
- `docs/decisions/ADR-0026-self-improvement-loop.md` — records: (a) external-process model (subprocess vs VSCode), (b) two-tier learnings split (universal vs internal), (c) manual promote (defer auto-promote to v3.8), (d) experiment root outside source repo
|
|
673
|
+
- `CHANGELOG.md` — v3.7.0 entry
|
|
674
|
+
|
|
675
|
+
---
|
|
676
|
+
|
|
677
|
+
## Phase 4: Specification — Ready-to-Implement
|
|
678
|
+
|
|
679
|
+
### Files to Create
|
|
680
|
+
|
|
681
|
+
1. `pan-wizard-core/bin/lib/experiment.cjs` (new — 28th core module, ~400 LOC)
|
|
682
|
+
2. `pan-wizard-core/bin/lib/runner.cjs` (new — 29th core module, ~300 LOC)
|
|
683
|
+
3. `pan-wizard-core/templates/idea.md` (new — 27th template)
|
|
684
|
+
4. `pan-wizard-core/learnings/universal/.gitkeep` (placeholder — first promote creates topic files)
|
|
685
|
+
5. `pan-wizard-core/learnings/internal/.gitkeep` (placeholder)
|
|
686
|
+
6. `pan-wizard-core/learnings/README.md` (explains universal vs internal split)
|
|
687
|
+
7. `commands/pan/experiment.md` (new — 52nd command, full subcommand surface)
|
|
688
|
+
8. `agents/pan-experiment-runner.md` (new — 21st agent)
|
|
689
|
+
9. `tests/experiment.test.cjs`
|
|
690
|
+
10. `tests/runner.test.cjs`
|
|
691
|
+
11. `tests/learn-promote.test.cjs`
|
|
692
|
+
12. `tests/scenarios/learnings-installed.test.cjs`
|
|
693
|
+
13. `tests/scenarios/experiment-lifecycle.test.cjs`
|
|
694
|
+
14. `docs/decisions/ADR-0026-self-improvement-loop.md`
|
|
695
|
+
|
|
696
|
+
### Files to Modify
|
|
697
|
+
|
|
698
|
+
1. `pan-wizard-core/bin/lib/optimize.cjs` — add `promotePattern`, `listPromotedPatterns`, `unpromotePattern` exports
|
|
699
|
+
2. `pan-wizard-core/bin/pan-tools.cjs` — register `experiment` and `learn promote/unpromote/list-promoted` subcommand routes
|
|
700
|
+
3. `commands/pan/learn.md` — extend `--experiment <slug>` flag + promote subcommand surface
|
|
701
|
+
4. `pan-wizard-core/workflows/learn.md` — add experiment-aware path
|
|
702
|
+
5. `pan-wizard-core/workflows/exec-phase.md` — reference `learnings/universal/`
|
|
703
|
+
6. `pan-wizard-core/workflows/plan-phase.md` — reference `learnings/universal/`
|
|
704
|
+
7. `pan-wizard-core/workflows/verify-phase.md` — reference `learnings/universal/`
|
|
705
|
+
8. `pan-wizard-core/workflows/execute-plan.md` — reference `learnings/universal/`
|
|
706
|
+
9. `bin/install.js` — explicit `learnings/universal` in shipped paths; tests assert `learnings/internal` is NOT shipped
|
|
707
|
+
10. `bin/install-lib.cjs` — verify glob handles new dir; add helper if needed
|
|
708
|
+
11. `package.json` — bump to 3.7.0
|
|
709
|
+
12. `CHANGELOG.md` — v3.7.0 entry
|
|
710
|
+
13. `docs/USER-GUIDE.md` — new section
|
|
711
|
+
14. `docs/ARCHITECTURE.md` — new section
|
|
712
|
+
15. `MEMORY.md` — refresh counts (29 modules, 21 agents, 52 commands, ~2620 tests)
|
|
713
|
+
|
|
714
|
+
### Step Order (recommended for single-session ship)
|
|
715
|
+
|
|
716
|
+
1. Write `experiment.cjs` + `runner.cjs` skeletons with all exports stubbed
|
|
717
|
+
2. Write all 5 test files (TDD-first; tests fail)
|
|
718
|
+
3. Implement experiment.cjs functions (newExperiment → list → manifest → harvest → prune)
|
|
719
|
+
4. Implement runner.cjs (RUNTIME_RUNNERS map, runExperiment with mocked subprocess for tests)
|
|
720
|
+
5. Implement promotePattern in optimize.cjs
|
|
721
|
+
6. Wire pan-tools.cjs dispatcher
|
|
722
|
+
7. Run unit tests — green
|
|
723
|
+
8. Create commands/pan/experiment.md, agents/pan-experiment-runner.md
|
|
724
|
+
9. Add workflow cross-references
|
|
725
|
+
10. Add `learnings/universal/`, `learnings/internal/` dirs with .gitkeep + README
|
|
726
|
+
11. Update bin/install.js for two-tier delivery
|
|
727
|
+
12. Run scenario tests — green (verifies shipping behavior)
|
|
728
|
+
13. Manual install verification in d:\pantesting (5 runtimes)
|
|
729
|
+
14. Manual smoke test: spawn a real `claude -p` against a trivial idea (opt-in integration)
|
|
730
|
+
15. Update CHANGELOG, USER-GUIDE, ARCHITECTURE, MEMORY
|
|
731
|
+
16. Write ADR-0026
|
|
732
|
+
17. Bump version to 3.7.0
|
|
733
|
+
18. Commit + tag
|
|
734
|
+
|
|
735
|
+
### Test Coverage Summary
|
|
736
|
+
|
|
737
|
+
| File | Test count | Type |
|
|
738
|
+
|------|------------|------|
|
|
739
|
+
| experiment.test.cjs | 10 | Unit |
|
|
740
|
+
| runner.test.cjs | 8 | Unit |
|
|
741
|
+
| learn-promote.test.cjs | 8 | Unit |
|
|
742
|
+
| scenarios/learnings-installed.test.cjs | 10 | Scenario |
|
|
743
|
+
| scenarios/experiment-lifecycle.test.cjs | 5 | Scenario |
|
|
744
|
+
| **Total new** | **~41** | — |
|
|
745
|
+
|
|
746
|
+
### Runtime Matrix
|
|
747
|
+
|
|
748
|
+
| Runtime | Experiment runner support | Receives `learnings/universal/` | Receives `learnings/internal/` |
|
|
749
|
+
|---------|---------------------------|--------------------------------|--------------------------------|
|
|
750
|
+
| Claude | ✅ via `claude -p` | ✅ | ❌ (intentional — source-only) |
|
|
751
|
+
| Codex | ✅ via `codex exec` | ✅ | ❌ |
|
|
752
|
+
| Gemini | ✅ via `gemini -p` | ✅ | ❌ |
|
|
753
|
+
| OpenCode | ✅ via `opencode` | ✅ | ❌ |
|
|
754
|
+
| GitHub Copilot | ❌ runner unsupported (no headless mode) | ✅ | ❌ |
|
|
755
|
+
|
|
756
|
+
GitHub Copilot users can still consume promoted learnings; they just can't drive an experiment runner. Documented limitation.
|
|
757
|
+
|
|
758
|
+
---
|
|
759
|
+
|
|
760
|
+
## Phase 5: Risk Assessment
|
|
761
|
+
|
|
762
|
+
| Risk | Impact | Likelihood | Mitigation |
|
|
763
|
+
|------|--------|------------|------------|
|
|
764
|
+
| Spawned external agent runs unbounded, consumes cost | High | Medium | Triple safety: timeout (default 30 min) + budget cap (default 80 pts) + circuit breaker (3 consecutive failures). Test coverage for each. |
|
|
765
|
+
| Experiment folder pollutes user disk | Low | Medium | Default root `~/pan-experiments/`; configurable. Document `pan-tools experiment delete` for cleanup. |
|
|
766
|
+
| Promote step writes garbage into shipped learnings | High | Medium | **Manual review gate** — promote requires explicit `--pattern <id>`; no auto-promote in v3.7. Tests verify file shape; ADR-0026 records the manual-only design. |
|
|
767
|
+
| Universal vs internal split is wrong (PAN-internal pattern shipped as universal) | Medium | High | Filter heuristic in promote: detect references to `pan-wizard-core/`, `bin/install.js`, `commands/pan/*` paths → suggest `--scope internal`. Human can override. |
|
|
768
|
+
| External agent doesn't return useful telemetry (e.g., gemini -p doesn't trigger PAN trace logger) | Medium | Medium | The trace logger fires on Claude Code SubagentStop; non-Claude runtimes log differently. Start with Claude as the primary; document fallback as "no auto-trace, manual harvest" for v3.7. |
|
|
769
|
+
| `claude -p` flag may not exist or differ across versions | Medium | Medium | Implement adapter map. Test against current Claude Code version at impl time. Document required Claude Code version in `references/`. |
|
|
770
|
+
| Cross-platform subprocess differences (Windows vs Unix) | Medium | Low | Use Node's `child_process.spawn` with `shell: false`. Existing scenario tests already cover Windows; new tests use same pattern. |
|
|
771
|
+
| Harvest copies sensitive data | Medium | Low | Document: idea.md is the only "sensitive" file; harvest excludes it by default; `--include-idea` opt-in flag for full archive. |
|
|
772
|
+
| `learnings/internal/` accidentally ships | High | Low | Negative test in `scenarios/learnings-installed.test.cjs`: assert NOT in any install dir. Installer test runs in CI on every change. |
|
|
773
|
+
| Cumulative learnings grow unbounded | Low | Medium | Each topic file has a soft size cap (10KB recommendation); patterns can be retired via `unpromote`. v3.8+ add an automatic age-based prune. |
|
|
774
|
+
| pan-experiment-runner agent intervenes in external instance | High | Low | Agent definition explicitly forbids editing experiment files. Agent tools list excludes Edit/Write. Tests assert no writes outside `<experiment>/.planning/`. |
|
|
775
|
+
| Test count grows fast — CI time increases | Low | High | New tests are mostly unit (fast). Integration test gated by env flag (excluded from CI). Mocked subprocess in scenario tests stays under 10s. |
|
|
776
|
+
| Spec/impl mismatch on runtime headless flags | Medium | Medium | Implementation phase verifies each runtime's actual headless invocation. Adapter map updates accordingly. ADR-0026 records what was verified vs assumed. |
|
|
777
|
+
|
|
778
|
+
### Backward Compatibility
|
|
779
|
+
|
|
780
|
+
- All proposals are additive. No breaking changes to existing commands, workflows, agents, tests.
|
|
781
|
+
- `learnings/` is a new top-level directory under `pan-wizard-core/`. Existing v3.6.x installs upgrading to v3.7.0 receive the new dir + universal contents via standard `pan update` flow.
|
|
782
|
+
- Workflows reference `learnings/universal/` AFTER existing `references/` references. If the dir is missing (manual install corruption), workflows degrade gracefully — the `@~/.claude/.../learnings/universal/` glob just resolves to no files.
|
|
783
|
+
- `optimize.cjs` extensions (`promotePattern`, etc.) are new exports. Existing `analyzeEvents`, `generateLocalReport`, etc. unchanged.
|
|
784
|
+
|
|
785
|
+
### What's Explicitly NOT in v3.7.0
|
|
786
|
+
|
|
787
|
+
- **Auto-promote** without human review — defer to v3.8+ once 10+ manual promotes inform the filter
|
|
788
|
+
- **Cross-experiment aggregation** — defer to v3.8+ (different design problem)
|
|
789
|
+
- **VSCode UI automation as runner path** — keep CLI subprocess; VSCode automation stays for tests
|
|
790
|
+
- **Bidirectional control mid-run** — observation-only by design
|
|
791
|
+
- **GitHub Copilot CLI runner** — no headless prompt mode known; document limitation
|
|
792
|
+
- **Experiment-driven evaluation** (LLM-as-judge over the experiment's outputs) — separate eval feature, see googlecli_adoption_featureai.md notes
|
|
793
|
+
- **Sandboxed/containerized experiments** — process isolation via separate folder is sufficient for v3.7; container isolation if needed later
|
|
794
|
+
|
|
795
|
+
---
|
|
796
|
+
|
|
797
|
+
## Appendix A: Open Questions (resolve before W4)
|
|
798
|
+
|
|
799
|
+
1. **Should `learnings/universal/` topic files be human-editable, or AI-only?** Recommendation: AI-only via `promote/unpromote`. Human edits create drift between topic frontmatter `pattern_ids` and body content. For human-authored content, use `references/`.
|
|
800
|
+
|
|
801
|
+
2. **Should the experiment runner use a single per-runtime adapter, or call the user's installed `claude` PATH?** Recommendation: PATH lookup (`which claude`). Document as a prerequisite in USER-GUIDE.
|
|
802
|
+
|
|
803
|
+
3. **Should `harvest --all` interleave reports or keep per-experiment dirs?** Recommendation: per-experiment dirs (cleaner traceability). `learn promote --aggregate` is a v3.8 feature for cross-experiment patterns.
|
|
804
|
+
|
|
805
|
+
4. **Should `learnings/universal/exec-patterns.md` REPLACE or AUGMENT `references/guardrails.md`?** Recommendation: augment. Guardrails are human-authored canonical rules; learnings are AI-derived advisory patterns. Workflows reference both.
|
|
806
|
+
|
|
807
|
+
5. **Should runtime adapters for codex/gemini/opencode ship in v3.7.0 or claude-only?** Recommendation: claude-only in W2 (v3.7.1); add other adapters in v3.7.4 follow-up. Faster initial ship; same surface.
|
|
808
|
+
|
|
809
|
+
6. **Should we add a `pan-tools experiment replay <slug>` to re-run from a specific phase?** Recommendation: defer to v3.8. Replay needs idempotency design that v3.7 doesn't need.
|
|
810
|
+
|
|
811
|
+
---
|
|
812
|
+
|
|
813
|
+
## Appendix B: Mapping to Existing PAN Surface
|
|
814
|
+
|
|
815
|
+
| New element | Closest existing analog | Drift risk |
|
|
816
|
+
|-------------|-------------------------|------------|
|
|
817
|
+
| `experiment.cjs` | `focus.cjs` (auto-runner) | Low — same lifecycle pattern |
|
|
818
|
+
| `runner.cjs` | `tests/helpers.cjs createScenarioRunner()` | Low — generalizes existing pattern |
|
|
819
|
+
| `pan-experiment-runner` agent | `pan-optimizer` agent | Low — same observation-only stance |
|
|
820
|
+
| `learnings/universal/` | `references/` | None — same delivery channel |
|
|
821
|
+
| `learnings/internal/` | `pan-wizard-core/.dev-only/` (doesn't exist) | New concept; needs negative test |
|
|
822
|
+
| `idea.md` template | `requirements.md`, `project_spec_*.md` (existing patterns) | Low — same shape |
|
|
823
|
+
| `pan-tools experiment new/run/harvest` | `pan-tools focus auto/scan/plan/exec` | Low — same subcommand pattern |
|
|
824
|
+
| `pan-tools learn promote` | `pan-tools optimize apply` | Medium — new write-target (source repo, not project) |
|
|
825
|
+
|
|
826
|
+
---
|
|
827
|
+
|
|
828
|
+
## Appendix C: Timeline & Dependencies
|
|
829
|
+
|
|
830
|
+
```
|
|
831
|
+
v3.6.0 ✓ shipped 2026-04-27 (behavioral guardrails layer)
|
|
832
|
+
│
|
|
833
|
+
├─ unblocks: shipping behavioral content via references/ pattern (proven)
|
|
834
|
+
│
|
|
835
|
+
v3.7.0 (this spec) — Self-Improvement Loop
|
|
836
|
+
│
|
|
837
|
+
├─ W1: experiment scaffolding (12 pts)
|
|
838
|
+
├─ W2: runner + observation (10 pts)
|
|
839
|
+
├─ W3: harvest + extended /pan:learn (8 pts)
|
|
840
|
+
├─ W4: promote + ship (10 pts)
|
|
841
|
+
│
|
|
842
|
+
└─ unblocks:
|
|
843
|
+
├─ v3.8.0: auto-promote (rules-based filter, AI-confidence threshold)
|
|
844
|
+
├─ v3.8.0: cross-experiment aggregation
|
|
845
|
+
├─ v3.9.0: container-isolated experiments (if needed for security)
|
|
846
|
+
└─ v3.9.0: experiment-replay-from-phase (idempotency design)
|
|
847
|
+
```
|
|
848
|
+
|
|
849
|
+
---
|
|
850
|
+
|
|
851
|
+
## Verification Checklist (pre-merge for v3.7.0)
|
|
852
|
+
|
|
853
|
+
- [ ] `pan-wizard-core/bin/lib/experiment.cjs` exists, exports newExperiment / listExperiments / harvestExperiment / pruneExperiment
|
|
854
|
+
- [ ] `pan-wizard-core/bin/lib/runner.cjs` exists, exports runExperiment / tailExperimentState / stopExperiment / RUNTIME_RUNNERS
|
|
855
|
+
- [ ] `pan-wizard-core/bin/lib/optimize.cjs` exports promotePattern / listPromotedPatterns / unpromotePattern (added)
|
|
856
|
+
- [ ] `commands/pan/experiment.md` registered, runs all 8 subcommands
|
|
857
|
+
- [ ] `agents/pan-experiment-runner.md` exists, has correct tool list (Read/Bash/Glob/Grep — NO Edit/Write)
|
|
858
|
+
- [ ] `pan-wizard-core/templates/idea.md` exists with required frontmatter fields
|
|
859
|
+
- [ ] `pan-wizard-core/learnings/universal/` exists; ships to all 5 runtime installs
|
|
860
|
+
- [ ] `pan-wizard-core/learnings/internal/` exists; does NOT ship to any install (negative test passes)
|
|
861
|
+
- [ ] 4 long workflows reference `learnings/universal/`
|
|
862
|
+
- [ ] `bin/install.js` explicit two-tier handling verified by tests
|
|
863
|
+
- [ ] `npm test` passes, 41 new tests added
|
|
864
|
+
- [ ] `npm run test:scenarios` passes, including learnings shipping tests
|
|
865
|
+
- [ ] Manual install in d:\pantesting for 2+ runtimes confirms learnings/universal/ shipped, learnings/internal/ absent
|
|
866
|
+
- [ ] Manual smoke: real `claude -p` experiment runs end-to-end on a trivial idea, harvest + promote complete the loop
|
|
867
|
+
- [ ] CHANGELOG.md, USER-GUIDE.md, ARCHITECTURE.md updated
|
|
868
|
+
- [ ] ADR-0026 committed
|
|
869
|
+
- [ ] MEMORY.md refreshed (29 modules, 21 agents, 52 commands, ~2620 tests)
|
|
870
|
+
- [ ] `package.json` version is 3.7.0
|
|
871
|
+
- [ ] All v3.6.0 behavioral guardrails still in effect (regression check)
|
|
872
|
+
|
|
873
|
+
---
|
|
874
|
+
|
|
875
|
+
**End of spec. Awaiting acceptance to begin v3.7.0 W1.**
|