pan-wizard 3.36.0 → 3.37.0

package/docs/IMPROVEMENT-TODO.md

@@ -0,0 +1,246 @@
+# PAN Wizard Improvement TODO
+
+This document turns the April 2026 project review into an actionable improvement backlog. It focuses on release safety, documentation trust, installer reliability, and long-term maintainability.
+
+> **June 2026 update:** a follow-up ecosystem review with time-critical items (Gemini→Antigravity transition, model-table staleness, runtime format migrations, skills-standard convergence) lives in [ECOSYSTEM-REVIEW-2026-06.md](ECOSYSTEM-REVIEW-2026-06.md). Items there are proposals until promoted into this backlog or an ADR.
+
+> **September 2026 update:** an interim market-delta scan (checked on `2026-09-10`, four weeks after the August ecosystem review) found one standard the August review missed — Agent Plugins, a vendor-neutral plugin bundle that Copilot CLI, Codex and Antigravity now load natively — plus a stale rate row for the current default Claude model and a plugin build that omits the native workflows. The prioritised queue lives in [specs/market-delta-2026-09-superplan.md](specs/market-delta-2026-09-superplan.md); items there are proposals until executed.
+
+> **September 2026 reality check (`2026-09-10`):** the first run of the `/reality-check` dev skill re-verified the same-day scan against primary sources and a packed install into all five runtimes. It found the scan's "no release since May" claim wrong for BMAD and for GSD (which continues as gsd-core and is PAN's closest peer), a wrong rationale for the flagship bump (the newest Fable-tier model is what the `fable` and `best` aliases resolve to, not any plan's default), a `validate health` verdict that exits 0 when broken, and a comparison matrix whose "no competitor" claims no longer hold. Findings and the sized queue live in the addendum of [specs/market-delta-2026-09-superplan.md](specs/market-delta-2026-09-superplan.md); the narrative is section 9 of [ECOSYSTEM-REVIEW-2026-08.md](ECOSYSTEM-REVIEW-2026-08.md), and the detailed execution plan is [specs/reality-check-2026-09-superplan.md](specs/reality-check-2026-09-superplan.md).
+
+> **September 2026 market idea scan (`2026-09-21`):** the first run of the `/market-ideas` dev skill swept the five host tools, the direct peers, new entrants and the standards for ideas PAN should take, and triaged them against the code and a new ideas ledger so no idea is proposed twice and no rejected idea returns without a new fact. The ledger is [specs/market-ideas-ledger.md](specs/market-ideas-ledger.md); the sized queue for this run is [specs/market-ideas-2026-09.md](specs/market-ideas-2026-09.md). Items there are proposals until executed.
+
+> **September 2026 reality check (`2026-09-22`, executed `2026-09-23`):** the second run of `/reality-check` found the default Claude model priced at its predecessor's rate, Gemini and OpenAI rate rows that had never matched their pricing pages, routing tiers that named no real model, Gemini CLI hooks registered under event names Gemini does not have, and a unified-skills install that left Claude Code with no commands. The queue that fixed them, and the follow-ups it found, are the second addendum of [specs/market-delta-2026-09-superplan.md](specs/market-delta-2026-09-superplan.md); the narrative is section 10 of [ECOSYSTEM-REVIEW-2026-08.md](ECOSYSTEM-REVIEW-2026-08.md).
+
+> **October 2026 market idea scan (`2026-10-03`):** the second run of `/market-ideas` covered the window since the first. Its highest-leverage finding is that Codex's source cuts any skill delivered through an Agent Plugin to a fixed byte budget, and many of PAN's bundled skill bodies were longer; the queue confirmed it live (M25) and the bundle now splits long skills (S8). Peers also shipped fixes for gaps PAN had when the scan ran, all closed the same day (M21, M22, M23, M27, M28): decision checkpoints that auto-pick the first option, a skipped test gate that still reads as a pass, free text that can break a planning table, no stale-verification check, and dependent plans that run after a failed one. The ledger is [specs/market-ideas-ledger.md](specs/market-ideas-ledger.md); the sized queue is [specs/market-ideas-2026-10.md](specs/market-ideas-2026-10.md), and path changes found on the way are handed to `/reality-check` there. Items are proposals until executed.
+
+> **October 2026 (`2026-10-11`):** a market idea scan with a feature-level read of Superpowers, a full doc audit (fixes plus code fixes found by the prompt sweeps), and a reality check ran the same day. The reality check found OpenCode 2 applies PAN's config as written but does not load its plugin, and that the verifier's test gate is enforced by prompt only. The queue is the Addendum of [market-ideas-2026-10.md](specs/market-ideas-2026-10.md); the delta is §11 of [ECOSYSTEM-REVIEW-2026-08.md](ECOSYSTEM-REVIEW-2026-08.md).
+
+> **October 2026 (`2026-10-12`), realigned:** the owner set feature excellence first and exposure later. The program is [product_excellence_program_2026_q4_featureai.md](specs/product_excellence_program_2026_q4_featureai.md): code-enforced gates for every guarantee, a measured review and verification stack, a deterministic autonomy loop, harness v2 (model doubles, record/replay, statistics, seeded-defect scoreboards), product hygiene, and PAN developing PAN. The session order, and what has landed, is in [product-excellence-superplan.md](specs/product-excellence-superplan.md).
+
+> **October 2026 market idea scan (`2026-10-19`):** a delta run of `/market-ideas`. Its highest-leverage finding is a live defect: Claude Code refuses a subagent's write to a file named like a report (`summary.md` and a few others), and PAN's research synthesizer writes exactly such a file, so research synthesis fails at its write step there. A live probe on this machine confirmed it. Also queued: recommended-model names that Copilot is deprecating, a rate row for a new Gemini flagship, truncated-plan detection, baselining a plan's own gates, and a "pre-existing" class for review findings. The queue is the `2026-10-19` Addendum of [market-ideas-2026-10.md](specs/market-ideas-2026-10.md), with path changes handed to `/reality-check` there; the ledger is [specs/market-ideas-ledger.md](specs/market-ideas-ledger.md). Items are proposals until executed.
+
+> **October 2026 reality check (`2026-10-19`):** a delta run the same day. The suite, the release gates, CI and harness tier 0 pass, and a packed five-runtime install writes every claimed surface. Several recorded statuses lag the tree (work described as open shipped in the last two releases). The changelog's OpenCode 2 claim rests on a simulated loader while Homebrew now installs OpenCode 2. The native exec-waves script lacks gates its markdown twin runs. The Copilot double-hook concern is refuted (PAN's Copilot hook copies defer to the Claude registration). The delta is §12 of [ECOSYSTEM-REVIEW-2026-08.md](ECOSYSTEM-REVIEW-2026-08.md); the queue is the reality-check Addendum of [market-ideas-2026-10.md](specs/market-ideas-2026-10.md).
+
+Do not duplicate filesystem-derived counts here. `CLAUDE.md` remains the single source of truth for command, agent, module, workflow, test, hook, spec, and ADR counts.
+
+## Goals
+
+- Make releases harder to publish in a broken or partially validated state.
+- Keep documentation accurate without manually chasing drift.
+- Make installer failures visible and diagnosable.
+- Reduce maintenance risk in the largest CLI and core modules.
+- Preserve the current strengths: zero runtime dependencies, broad tests, and multi-runtime support.
+
+## Priority Work
+
+### P0: Release Gate Hardening — DONE
+
+> **Status:** shipped. `scripts/release-check.js` runs as `npm run release:check` and as `prepublishOnly`, and the tag-triggered release workflow reruns it before publishing. The gate has since grown past this list: `doc-lint counts` over `docs/`, the doc↔code link graph, a zero-runtime-dependency assertion inside the pack gate, both distribution bundles with a stale-`dist/` digest check (`2026-09-10`), and a coverage gate that fails when a dispatcher arm never ran, overall line or function coverage drops below its floor, or a module group drops below its line floor (skipped on Node < 22). The flow is documented under "Release Process" in [DEVELOPMENT.md](DEVELOPMENT.md). Original task list retained.
+
+Current issue: `prepublishOnly` only runs the hook build step. That is too light for a CLI installer that ships commands, agents, workflows, hooks, runtime adapters, and generated install layouts.
+
+Tasks:
+
+- Add a release validation script, for example `npm run release:check`.
+- Include these checks in the release gate:
+  - `npm run build:hooks`
+  - `npm run test:all`
+  - `npm audit --omit=dev`
+  - `npm pack --dry-run --json`
+  - A packed-tarball install smoke test in a temp directory.
+- Update `prepublishOnly` to call the release validation script.
+- Document the release flow in `docs/DEVELOPMENT.md` or a dedicated `docs/RELEASE.md`.
+
+Acceptance criteria:
+
+- A publish attempt fails before upload if tests, hook build, audit, package dry-run, or smoke install fails.
+- The release process can be run locally with one command.
+- The release document names the required commands but does not duplicate test counts.
+
+### P0: Installer Manifest Verification — partially DONE
+
+> **Status:** per-file write failures in the installer's copy paths are collected as warnings instead of swallowed, `verifyInstalled()` checks each required install directory after the copy, the installer then runs `verifyInstall()` over its manifest and exits 1 on a missing or empty tracked file, `pan-tools validate deployment` verifies an install against its manifest after the fact, and `verifyInstall()` has unit tests for missing and 0-byte tracked files, a missing or empty dispatcher, and a missing or file-less manifest. Still open: a regression test that makes a copy or write fail mid-install and checks the installer reports it. Original task list retained.
+
+Current issue: parts of `bin/install.js` swallow copy/write errors. Some later checks prove that directories exist, but not that every expected file landed correctly.
+
+Tasks:
+
+- Replace empty `catch {}` blocks in installer copy paths with warning collection or explicit failure where the file is required.
+- Add a manifest verification pass after install.
+- Verify all required commands, agents, hooks, core files, references, templates, and workflows expected for the selected runtime are present.
+- Report optional failures as warnings and required failures as fatal errors.
+- Add regression tests for partial copy failure behavior.
+
+Acceptance criteria:
+
+- A missing required installed file causes a clear installer failure.
+- Optional cleanup failures do not hide required install failures.
+- Tests cover at least one simulated copy/write failure.
+
+### P1: Documentation Drift Guard — DONE
+
+> **Status:** shipped. `pan-tools doc-lint counts <dir>` flags filesystem-derived counts outside `CLAUDE.md`, runs over `docs/` as release Gate 4 and in `tests/doc-lint.test.cjs` (`ATTRIBUTION.md`, which the task list names, and `README.md` sit at the repo root, outside both scans), and `tests/claude-md-counts.test.cjs` pins the counts table's own contract. Historical changelog entries keep their snapshots by design.
+
+Current issue: the project rule says counts live only in `CLAUDE.md`, but other docs and specs still contain count-like values. This makes docs look authoritative after they have gone stale.
+
+Tasks:
+
+- Add a doc-lint rule that flags filesystem-derived counts outside `CLAUDE.md`.
+- Allow historical changelog entries only if the team wants changelog snapshots to remain as release history.
+- Clean current docs and specs by replacing numeric counts with qualitative language or links to `CLAUDE.md`.
+- Include attribution and architecture docs in the drift scan.
+
+Acceptance criteria:
+
+- CI or `npm run test:all` catches new duplicated count drift.
+- Current user-facing docs no longer embed mutable project counts.
+- `CLAUDE.md` remains the only active count table.
+
+### P1: Test Result Artifact Cleanup — DONE
+
+> **Status:** resolved as generated artifacts: `test-results/` is gitignored and nothing under it is tracked.
+
+Current issue: `test-results/latest.txt` and `test-results/scenarios.txt` can become stale and misleading.
+
+Tasks:
+
+- Decide whether test result files are generated artifacts or release evidence.
+- If generated, remove tracked stale files and keep `test-results/` ignored.
+- If release evidence, add `npm run refresh:test-results` and call it during release prep.
+- Ensure output encoding is readable on Windows and Unix terminals.
+
+Acceptance criteria:
+
+- The repository no longer contains stale test output from old package versions.
+- The policy for test-result files is documented.
+
+### P1: Runtime Artifact Ignore Rules — DONE
+
+> **Status:** shipped. `.gitignore` carries `.claude/*.lock` and `.claude/scheduled_tasks.lock` alongside the self-install artifact rules; the dev `.claude/commands`, `.claude/agents` and `.claude/workflows` stay trackable.
+
+Current issue: local runtime files can still appear in the source worktree, such as `.claude/*.lock` files.
+
+Tasks:
+
+- Add ignore rules for local Claude runtime lock files.
+- Audit `.gitignore` for other runtime-local files that should never be committed.
+- Keep development-only `.claude/commands`, `.claude/agents`, and `.claude/workflows` available as intended.
+
+Acceptance criteria:
+
+- Local scheduler/lock artifacts no longer show up as untracked source changes.
+- Existing dev command and agent files remain trackable.
+
+### P2: Large Module Decomposition — module half DONE (2026-06-11)
+
+The lib-module half shipped behind a compatibility harness (`tests/module-surface.test.cjs` pins every module's export surface against a committed fixture; behavior stays pinned by the rest of the suite):
+
+- `verify.cjs` decomposed by validation domain: drift detection, retrospective, deployment validation, and pre-execution gates each extracted to a focused `verify-*.cjs` submodule, re-exported through the facade.
+- `commands.cjs`: error-pattern/session-history/learnings lifecycle extracted to `commands-learnings.cjs` (with the shared phase-summary collector, keeping dependencies one-directional).
+- `phase.cjs`: removal + renumbering cascade extracted to `phase-remove.cjs`.
+- Public CLI behavior unchanged throughout — every consumer goes through the original facades, and the harness fails any extraction that drops or mutates an export.
+
+Remaining (open):
+
+- Split the central `pan-tools.cjs` switch into command-family routers or a table-driven dispatcher. Lower urgency now that the implementation modules are focused; do it behind the same harness plus dispatch-level contract tests.
+
+### P2: Hook Build Naming and Dependency Cleanup — DONE
+
+> **Status:** copy-only chosen. `build:hooks` copies the hooks listed in `HOOKS_TO_COPY` (`scripts/build-hooks.js`) to `hooks/dist/`; ARCHITECTURE, DEVELOPMENT, HOOKS and CONTRIBUTING all say so; no bundler remains in `devDependencies` (only `@playwright/test` and `@vscode/test-electron`).
+
+Current issue: docs and dependencies imply hooks are compiled with esbuild, but the build script currently copies pure Node.js files.
+
+Tasks:
+
+- Choose one direction:
+  - Keep copy-only hooks and rename/document the command accordingly, or
+  - Implement actual esbuild bundling if bundling is still desired.
+- Align `docs/ARCHITECTURE.md`, `docs/DEVELOPMENT.md`, `docs/HOOKS.md`, `ATTRIBUTION.md`, and `package.json` with the chosen direction.
+- Remove unused development dependencies if copy-only remains the design.
+
+Acceptance criteria:
+
+- Build docs, script behavior, and dependencies agree.
+- Contributors do not need to infer whether hook files are copied or bundled.
+
+### P2: Package Documentation Policy — DONE
+
+> **Status:** decided yes (`2026-10-22`): the `files` allowlist in `package.json` now ships `docs/` (minus the brand images under `docs/branding/assets/`) and `CHANGELOG.md` beside `bin`, `commands`, `agents`, `pan-wizard-core` (minus internal learnings), `hooks/dist`, `scripts`, `assets` and `pan-zcode`, so the README's relative `docs/` links resolve in an installed package too. `tests/package-contract.test.cjs` asserts that `files` is an explicit allowlist that keeps `dist`, `harness`, `marketplace` and `tests` out, and that `docs/` and `CHANGELOG.md` ship without the brand images.
+
+Current issue: the npm package includes `README.md` automatically, but not the full `docs/` directory. That may be intentional, but the policy should be explicit.
+
+Tasks:
+
+- Decide whether user-facing docs should ship in the npm package.
+- If yes, add selected docs or the full docs directory to the package files list.
+- If no, document that npm installs rely on GitHub for full documentation.
+- Keep package size checked with `npm pack --dry-run --json`.
+
+Acceptance criteria:
+
+- The package contents match the intended user experience.
+- Package-size changes are visible in release review.
+
+### P2: Optional LLM Evaluator for Self-Improvement Runner — ADR WRITTEN, awaiting review
+
+> **Status (2026-06-11):** the design spike shipped as [ADR-0029](decisions/ADR-0029-experiment-evaluator.md) — pluggable evaluator, host-CLI backend (no SDK dependency), off by default, runner-scoped with a hard "one stop authority" fence. Implementation stays blocked until the maintainer accepts the ADR. Original task list retained below for review context.
+
+Current issue: the v3.7.0 self-improvement loop (`runner.cjs`, `experiment.cjs`, ADR-0026) stops experiments on deterministic signals: the session's exit, a timeout, and milestone-completion detection in the experiment's `state.md`. For experiments whose success criterion is genuinely qualitative ("did this prompt variant produce a better outcome?"), a deterministic stop is a blunt condition. Claude Code's built-in `/goal` command demonstrates the worker/evaluator-split pattern (separate small model judges completion after every turn), but it is Claude-only and outside PAN's surface.
+
+The case for adopting this pattern in PAN is narrow and specific to the experiment runner. The rest of PAN (phase, focus, verify, retro) has been deliberately moving toward deterministic gates; reintroducing LLM-as-judge there would be regression. Inside `runner.cjs`, where the work being evaluated is itself probabilistic, an evaluator model is the right tool.
+
+Tasks:
+
+- Write an ADR before any code, capturing why the experiment runner is the only place this exception applies and what would have to change for the scope to grow.
+- Specify the evaluator contract: input is experiment transcript plus a goal condition string, output is a yes/no decision plus a short reason recorded to the experiment log.
+- Decide model access without breaking the zero-runtime-dependency posture. Options to evaluate: opt-in Anthropic SDK behind a config flag, shelling to the host runtime's CLI, or a pluggable evaluator interface that ships with no default backend.
+- Identify which experiment shapes benefit (prompt-variant comparison, output-quality judgment) and which must remain deterministic (anything wired to phase or verify state).
+- Specify failure modes: missing credentials, evaluator timeout, ambiguous response. Decide whether each fails the experiment, falls back to deterministic stop, or warns and continues.
+- Confirm cross-runtime behavior: the runner is the only consumer, so non-Claude runtimes should degrade gracefully when the feature is off.
+
+Acceptance criteria:
+
+- ADR shipped and reviewed before any implementation work.
+- Feature lives behind an explicit config flag, off by default.
+- No change to phase, focus, verify, or retro code paths.
+- Behavior documented in `commands/pan/experiment.md` and `pan-wizard-core/workflows/` only after the ADR lands.
+
+Reference: Claude Code's `/goal` ([docs](https://code.claude.com/docs/en/goal)) introduced the worker/evaluator-split pattern. This item adopts the pattern in one subsystem where probabilistic judgment is appropriate, without adopting the Claude-Code-specific implementation.
+
+## Suggested Execution Order
+
+> Shipped except: the dispatcher split (P2 decomposition), the copy-failure regression test (P0 manifest verification) and the optional LLM evaluator (P2, ADR awaiting review) — see the statuses above. Kept for the record.
+
+1. Add runtime artifact ignore rules and clean stale local artifacts.
+2. Fix or remove stale test-result files.
+3. Add `release:check` and wire it into `prepublishOnly`.
+4. Add package dry-run and packed-install smoke validation.
+5. Add doc drift linting and clean current duplicated counts.
+6. Harden installer manifest verification.
+7. Align hook build naming, docs, and dependencies.
+8. Begin large-module decomposition behind compatibility tests.
+
+Deferred (not on the critical path): the LLM evaluator design spike for the self-improvement runner. ADR first, scoped strictly to `runner.cjs`.
+
+## Verification Commands
+
+Use these during implementation and before release:
+
+```bash
+npm run build:hooks
+npm test
+npm run test:scenarios
+npm run test:all
+npm audit --omit=dev
+npm pack --dry-run --json
+```
+
+For manual installation testing, use a separate test directory. Do not install PAN into the source repository.
+
+## Notes
+
+- Preserve the zero-runtime-dependency posture unless there is a strong reason to change it.
+- Keep installer changes conservative; installation failures should become clearer, not noisier.
+- Prefer automated drift checks over periodic manual cleanup.
+- Treat docs as product surface. For this project, stale docs are a real reliability bug because agents and users both consume them.