pan-wizard 3.36.0 → 3.37.0

package/docs/decisions/ADR-0049-machine-verdicts-and-findings-ledger.md

@@ -0,0 +1,116 @@
+# ADR-0049: Machine verdicts and the findings ledger
+
+## Status
+
+**Accepted — 2026-10-12** by the maintainer (recorded in `docs/specs/product_excellence_program_2026_q4_featureai.md` §3.4). Evidence-bound finding closure and stable finding ids (item V6 of that program) landed as the amendment to D5 and D6 below (`2026-10-12`).
+
+Proposed — 2026-09-28; shipped in v3.32.0 (2026-09-29). Implemented on `feat/evidence-loop` (spec `docs/specs/evidence_loop_featureai.md`, plan `docs/specs/evidence-loop-plan.md`); the maintainer accepts or amends it at release review. The decisions below were made in `/featureAI` on the maintainer's instruction to design, plan and build the evidence loop from the 2026-09-28 investigation and the September market-ideas queue.
+
+## Context
+
+PAN's judges already decide whether work passed. The plan checker gates plans, the reviewer gates code, the design checker gates designs, and the verifier gates phases. What PAN did not have is a record of those decisions. Measured on 2026-09-28 (aggregate counts only, spec §1.1):
+
+- **Verdicts were prose that each workflow parsed its own way.**
+  - exec-phase grepped `status:` from the verification frontmatter.
+  - plan-phase matched headings.
+  - The review step read an error count from a shell variable nothing set.
+- **The reviewer's report was never saved.** `/pan:review-deep` read a `review.md` that no step wrote, from a path no phase directory has. Its parser also wanted bullet findings, while the reviewer writes tables. The reviewer half of every deep review therefore merged as zero findings.
+- **Nothing recorded a decision about a finding.** No record said a finding had been deferred, dismissed or fixed, or why. Milestone audits reconstructed tech debt from prose.
+- **The optimisation loop saw no failures in the field.**
+  - Across 14 field projects: 3,968 trace events, zero `error`, `gap` or `correction` events.
+  - Across 1,969 subagent transcripts on the same machine: 869 failed tool calls, 23.5% of spawns affected.
+  - The failure signal depended on agents choosing to call `optimize trace log`. They did on success branches and never on error branches.
+- **`optimize apply` could not be undone.** Running it twice on one report appended everything twice.
+
+Two peers converged on the verification side (market ideas MI-030 and MI-031):
+- Spec Kit's convergence classes: missing, partial, contradicts, unrequested.
+- BMAD's review triage: every finding ends as patch, defer, dismiss-with-reason or decision, and never silently.
+
+The Growing Harness paper (arXiv:2609.26760) supports runtime-recorded, localised failures and revertible changes.
+
+## Decision
+
+**D1: Verdict capture is a verb the workflow needs, not a hook.** `pan-tools findings record` reads a judge's report and prints the verdict the workflow branches on. That replaces the greps and heading matches, and the verdict gets recorded because the workflow needs the answer. It works on all five runtimes; per-agent transcripts exist on one. There is one writer per verdict.
+
+**D2: One machine block for the judges that return text; the verifier keeps its frontmatter.**
+- The plan checker, reviewer and design checker end their report with a fenced `pan-verdict` JSON block: `contract` `"1.0"`, agent, the agent's own verdict word, outcome `pass`/`fail`/`needs_human`, and findings, each with a class, severity, location and summary.
+- The last block in a report wins.
+- The verifier's verification.md frontmatter already is a machine contract, the one `/pan:plan-phase --gaps` consumes. A frontmatter adapter reads it, rather than asking the verifier for a second block that could disagree with it. The frontmatter gains `unrequested:` (M11).
+
+**D3: Reports are saved in the phase directory by the orchestrator.** The judges stay read-only. The orchestrator writes the returned text verbatim to `NN-review.md`, `NN-plan-check.md` or `NN-design-check.md`.
+
+**D4: An append-only project ledger, folded on read.** `.planning/findings.jsonl` holds verdict, finding and disposition rows.
+- Status is derived, never stored: the latest disposition wins, and `fixed` gives way to `open` when a newer verdict reports the finding again.
+- One file per project, because phase numbers continue across milestones; each row also records its milestone.
+- It is the same append-only pattern as the cost and trace ledgers.
+
+**D5: Deterministic identities.**
+- A recorded artifact is identified by its line-ending-normalised content hash, so recording it twice is a no-op.
+- A finding's id hashes its phase, agent, class, location and normalised summary.
+- `attempt` counts one agent's verdicts on one phase.
+
+**D6: Automatic status changes only where the evidence supports them.**
+- ~~A new verdict closes the same agent's earlier open findings on that phase that it no longer reports.~~ **Amended `2026-10-12` (program V6):** silence is not evidence. An earlier open finding closes only when the judge answers it `resolved` with evidence (a `resolutions` entry in its block, or a truth in the verifier's `re_verification.gaps_closed`), or, for a `code-fact:<fact>` finding (program V9), when that fact passes. A finding the new verdict neither reports nor answers stays open and is listed as `unaddressed`; `still_present` counts as a re-report. The `human`/`unrequested` exemption is retired with the rule it exempted from.
+- Deliberate dispositions are never overridden by a later report.
+
+**D5 amendment (`2026-10-12`, program V6):** a reported finding whose exact id is new but which matches an earlier finding of the same agent and phase (same class family, where `missing` and `partial` are one family; same file, ignoring line numbers; same truth or a summary with token overlap of at least one half) keeps the earlier id. `findings list --phase N --agent A` reports `churn`, the share of ids that appear or disappear between consecutive reports.
+
+**D7: Dispositions need reasons, and workflows dispose what they continue past.** The vocabulary is `fixed`, `deferred`, `dismissed`, `decision`. The last three require a reason, and the verb refuses without one. Workflows record the reason whenever a phase moves on with findings open:
+- warnings accepted at review;
+- "continue anyway" at `NEEDS_FIXES`;
+- "force proceed" after the plan revision limit;
+- design caveats after the design revision limit.
+
+Milestone audits read `findings debt`.
+
+**D8: The trace hook captures the failures only it can see.** On the agent-transcript path only, the hook records failed tool calls as `error/tool_error` events:
+- grouped per spawn, at most 10 events;
+- redacted before they are written, because some projects commit their traces;
+- `execution.error_pattern_learning: false` turns capture off; the key was reserved until now.
+
+Verdicts do not go through the hook (D1).
+
+**D9: Every optimizer apply is recorded, and a revert refuses what it cannot undo safely.**
+- An `apply_id` with per-action records makes `optimize revert` exact.
+- A revert refuses hand-edited files, and reverts are last-in, first-out per file.
+- Applying the same report twice writes nothing the second time.
+- The advisory before/after `optimize verdict` is deferred until the loop is used: one apply run existed in 14 field projects. Trigger: apply runs in three or more projects.
+
+**D10: `contract` versions the machine-readable outputs.** `"1.0"` appears on the verdict block, on the `findings` outputs, and on `state`, `state json`, `progress` and their MCP resources (M17). The rule is additive within a major version; readers ignore unknown fields.
+
+## Rejected alternatives
+
+- **The hook parses verdicts from transcripts.** It works on one runtime of five, and it would make a second writer beside the workflow, with ambiguous deduplication.
+- **Keep the prose `optimize trace log` calls for verdicts.** Measured: in the field they fired on success branches and never on error branches.
+- **A `pan-verdict` block in verification.md too.** Two sources of truth in one file. The frontmatter is already the contract `plan-phase --gaps` reads.
+- **Parse each judge's prose format.** The reviewer-table versus deep-review-bullet mismatch is the failure mode. One contract, one parser.
+- **Dispositions in the verification template.** The status would live in files the model rewrites. The verb enforces the reason at the write, and the ledger keeps the history.
+- **A per-phase findings file.** It would need phase-directory resolution in every writer, and milestone audits read across phases anyway.
+- **Automatic revert on regression.** A PAN project has no fixed held-out task set, so any regression signal is advisory. Revert stays a user decision.
+
+## Consequences
+
+- **Positive.**
+  - Workflows branch on one parsed verdict per judge, with the old reads kept as fallbacks.
+  - Milestone audits get tech debt with the reasons recorded at the time.
+  - Gap-closure rounds show what they resolved.
+  - `/pan:review-deep` finally merges the reviewer's findings.
+  - The optimiser sees failures and ranks them by recurrence.
+  - An apply can be undone.
+- **Negative, accepted.**
+  - ~~A finding reworded between attempts appears as fixed and re-opened under a new id.~~ Fixed by the D5 amendment: a rewording keeps its id.
+  - Since the D6 amendment, a judge that does not answer its open findings leaves them open, so a phase can carry open findings past a passing verdict until someone answers or disposes them.
+  - The judges' reports gain a block of JSON at the end.
+  - The phase directory gains up to three report files.
+- **Negative, mitigated.**
+  - Captured error text could hold secrets. It is redacted and capped, with an off switch.
+  - An unparseable block leaves the verdict unrecorded, so every workflow keeps its previous parse as the `||` fallback.
+- **Runtime reach.** Verdicts, findings, dispositions, revert and `contract` work on all five runtimes. Tool-failure capture is Claude-only: it needs per-agent transcripts.
+  - Copilot's documented camelCase payload is now read by both loggers, so its spawns carry their agent name. There are still no failures, because Copilot has no per-subagent transcript.
+  - Gemini has no subagent-completion event, and OpenCode registers no hooks.
+
+## Verification
+
+- Unit suites for the contract, the ledger, the scope check, the hook capture (fixture shaped from real transcript records), revert, and the workflow drift. Each fix was revert-proven against the committed code.
+- Tier-0 harness scenario `evidence-loop`: the whole loop runs green through a deployed install.
+- Tier-1 `tool-error-capture`, first run 2026-09-28 ($0.50): a real subagent's failed `npm test` was captured from its own transcript by the installed hook.

package/docs/specs/ai_drift_prevention_featureai.md

@@ -0,0 +1,286 @@
+# AI Drift Prevention System — Feature Specification
+
+**Generated:** 2026-03-07
+**Version:** 1.0
+**Status:** Proposed
+**ADR:** ADR-0020
+
+---
+
+## Problem Statement
+
+AI drift occurs when an AI coding assistant gradually deviates from a project's plan, conventions, coding standards, and architectural decisions during extended sessions or across session boundaries. This manifests as: using wrong patterns (e.g., `console.log` instead of `output()`), forgetting file naming conventions, straying from planned task scope, inventing unplanned features, or losing track of project state. For PAN Wizard's target users — developers using AI assistants for complex multi-session projects — drift is the #1 cause of wasted context tokens, rework, and subtle bugs.
+
+## Demand Evidence
+
+| Evidence Type | Source | Finding |
+|---|---|---|
+| Personal pain (user-stated) | This conversation | User explicitly asked for drift prevention analysis |
+| Community pattern | Reddit, HN, Dev.to | Universal complaint: "AI forgets conventions mid-session" |
+| PAN Wizard's own CLAUDE.md | Project conventions | Extensive convention docs = evidence drift is actively managed |
+| Competitor parity | Cursor (.cursorrules), Aider (CONVENTIONS.md) | Every major tool has SOME drift prevention |
+| Industry trend | 2026 guardrails movement | Shift from speed to quality — drift prevention is table-stakes |
+
+## Success Criteria
+
+```
+SC-1: AI sessions produce code matching project conventions >=95% of the time (measurable via drift score)
+SC-2: Cross-session context loss reduced — AI can resume with conventions in <2 commands
+SC-3: Drift detection runs in <200ms on typical projects
+SC-4: Works across Claude Code, OpenCode, Gemini CLI, Codex, Copilot CLI (all 5 runtimes)
+SC-5: No regression in existing 1510+ tests
+SC-6: Zero runtime dependencies maintained
+```
+
+## Competitive Landscape
+
+### Current State (March 2026)
+
+| Tool | Convention Docs | Active Enforcement | Drift Scoring | Cross-Session |
+|---|---|---|---|---|
+| **PAN Wizard** | Standards + CONVENTIONS.md | None (passive) | **None** | State + Resume + Memory |
+| **Aider** | CONVENTIONS.md | None | None | Git log |
+| **Cursor** | .cursor/rules/*.mdc | None | None | None |
+| **Cline** | .clinerules | Human review | None | None |
+| **Windsurf** | Rules files | None | None | Memory |
+| **Copilot WS** | Spec files | None | None | None |
+| **CodeScene** | External | CI quality gates | Quality score | External |
+| **Codacy** | External | Real-time | Quality score | External |
+
+**Key Finding:** No tool has built-in, zero-dependency drift scoring. PAN is already the leader in drift prevention infrastructure. The gap is measurement and active enforcement.
+
+### Novel Approaches from Industry Research
+
+1. **Structured protocols** — Output structured summaries with rationale
+2. **Guardrail + Validation pairs** — Rules + automated checks working together
+3. **Circuit breakers** — Warning threshold + hard threshold
+4. **Multi-agent layered validation** — Writer -> Critic -> Tester -> Compliance
+5. **Git-based isolation** — Every action reversible via worktrees
+6. **Spec-driven development** — Specs as primary artifacts
+
+## Strategic Analysis
+
+### Blue Ocean Framework
+
+| Action | Decision |
+|---|---|
+| **ELIMINATE** | External tool dependencies, IDE-specific monitoring |
+| **REDUCE** | Manual convention checking, human-in-the-loop friction |
+| **RAISE** | Convention enforcement: passive -> active. Drift detection: post-hoc -> during-execution |
+| **CREATE** | **Drift Score** (quantitative, nobody has this). Convention Checker (zero-dep). Drift Report. Pre-commit gate. |
+
+### Moat Score: 28/30
+
+| Moat Type | Score |
+|---|---|
+| Context Engineering | 5/5 |
+| Cross-Platform | 4/5 |
+| Developer Experience | 5/5 |
+| Zero Dependencies | 5/5 |
+| State Persistence | 4/5 |
+| Verification Quality | 5/5 |
+
+### Recommendation
+
+**Build** — PAN Wizard already has the strongest drift prevention infrastructure. Add quantitative Drift Score and automated convention checking to become the only tool answering "how much did the AI drift?" No other tool can answer this today. This is a Blue Ocean opportunity at Genesis stage in the Wardley map.
+
+## Design
+
+### Architecture
+
+```
+pan-tools.cjs → 'drift-check' case
+  → verify.cjs: cmdDriftCheck(cwd, raw, args)
+    → parseConventionRules(content)     -- from CONVENTIONS.md
+    → getChangedFiles(cwd, sinceRef)    -- git diff --name-only
+    → checkFileConventions(path, content, rules)  -- per file
+    → calculateDriftScore(violations, filesChecked)
+    → output(result, raw, summary)
+```
+
+### Convention Rule Format
+
+Rules extracted from CONVENTIONS.md markdown code blocks:
+
+```markdown
+## Error Handling
+- Use `safeReadFile()` instead of `existsSync()` + `readFileSync()`
+- Wrap all `writeFileSync` calls in try-catch
+
+## Output
+- Use `output(data, raw, label)` — never `console.log` for JSON output
+- Use `error(message)` — never `console.error` or `throw` to user
+```
+
+Parsed into:
+```json
+[
+  {"id": "safe-read", "antiPattern": "existsSync", "message": "Use safeReadFile() instead of existsSync+readFileSync", "severity": "warning"},
+  {"id": "output-fn", "antiPattern": "console\\.log", "message": "Use output() instead of console.log", "severity": "error"},
+  {"id": "error-fn", "antiPattern": "console\\.error", "message": "Use error() instead of console.error", "severity": "error"}
+]
+```
+
+### Built-in PAN Convention Rules
+
+```javascript
+const BUILTIN_CONVENTION_RULES = [
+  { id: 'no-console-log', antiPattern: /\bconsole\.log\b/, message: 'Use output() instead of console.log', severity: 'error', fileGlob: '*.cjs' },
+  { id: 'no-console-error', antiPattern: /\bconsole\.error\b/, message: 'Use error() instead of console.error', severity: 'error', fileGlob: '*.cjs' },
+  { id: 'no-existsSync', antiPattern: /\bexistsSync\b/, message: 'Use safeReadFile() or fileAccessible() instead of existsSync', severity: 'warning', fileGlob: '*.cjs' },
+  { id: 'no-throw-to-user', antiPattern: /\bthrow new Error\b/, message: 'Use error() function instead of throw', severity: 'warning', fileGlob: '*.cjs' },
+  { id: 'cmd-naming', antiPattern: /^function\s+(?!cmd)[a-z]+\w*\(cwd,\s*raw/, message: 'Public command functions should be named cmd*', severity: 'info', fileGlob: 'lib/*.cjs' },
+  { id: 'toPosix-paths', antiPattern: /output\([^)]*path\.join/, message: 'Wrap path.join() in toPosix() for output', severity: 'warning', fileGlob: '*.cjs' },
+  { id: 'write-try-catch', antiPattern: /writeFileSync\b(?![^]*catch)/, message: 'Wrap writeFileSync in try-catch', severity: 'warning', fileGlob: '*.cjs' },
+];
+```
+
+### Drift Score Formula
+
+```
+drift_score = min(1.0, weighted_violations / (files_checked * max_rules_per_file))
+
+where:
+  weighted_violations = sum(3 for errors, 1 for warnings, 0.5 for info)
+  max_rules_per_file = total_rules * 0.3 (30% expected violation ceiling)
+```
+
+Score interpretation:
+- 0.0 - 0.2: **Clean** — minimal or no drift
+- 0.2 - 0.5: **Low** — some convention deviations, review recommended
+- 0.5 - 0.8: **Medium** — significant drift, action recommended
+- 0.8 - 1.0: **High** — severe drift, conventions being ignored
+
+### Command Interface
+
+```
+pan-tools drift-check [--since <ref>] [--threshold <0.0-1.0>] [--files <path,...>]
+
+Arguments:
+  --since <ref>      Git ref to diff against (default: HEAD)
+  --threshold <n>    Pass/fail threshold (default: 0.5)
+  --files <paths>    Comma-separated specific files to check
+
+Output (JSON):
+{
+  "drift_score": 0.35,
+  "verdict": "low",
+  "violations": [
+    {"file": "lib/verify.cjs", "line": 45, "rule": "no-console-log", "message": "Use output() instead of console.log", "severity": "error"}
+  ],
+  "violation_count": 3,
+  "files_checked": 8,
+  "conventions_loaded": 12,
+  "summary": "drift: 0.35 (low) — 3 violations in 8 files"
+}
+
+Exit codes: 0 = pass, 1 = fail (above threshold)
+```
+
+### Error Handling
+
+| Condition | Output | Behavior |
+|---|---|---|
+| No CONVENTIONS.md | `{"drift_score": 0, "violations": [], "summary": "no conventions loaded"}` | Graceful pass |
+| Not a git repo | `{"error": "git repository not found"}` | Clean error |
+| Invalid --threshold | `{"error": "threshold must be 0.0-1.0"}` | Arg validation |
+| No changed files | `{"drift_score": 0, "violations": [], "summary": "no files changed"}` | Clean pass |
+| Invalid regex in convention | Skip rule, add warning | Degraded but functional |
+
+## Feature Ladder
+
+| Version | Scope | Value | Effort |
+|---|---|---|---|
+| **v0 (MVP)** | `drift-check` command + rule parser + git diff + built-in rules + score | Quantitative drift measurement | S-M (18 pts) |
+| **v1** | Pre-commit hook + drift-report.md + health --drift + verify integration | Automated prevention in workflow | M (16 pts) |
+| **v2** | Cross-session trends + auto-discovery + custom rule DSL + per-runtime profiles | Drift analytics and learning | L (30 pts) |
+
+## Implementation Tasks
+
+| # | ID | Title | Files | Effort | Pts | Priority |
+|---|---|---|---|---|---|---|
+| 1 | D.1 | Convention rule parser | verify.cjs | S | 2 | P2 |
+| 2 | D.2 | File convention checker | verify.cjs | S | 2 | P2 |
+| 3 | D.3 | Drift score calculator | verify.cjs | XS | 1 | P2 |
+| 4 | D.4 | cmdDriftCheck orchestrator | verify.cjs | M | 4 | P2 |
+| 5 | D.5 | CLI dispatcher routing | pan-tools.cjs | XS | 1 | P2 |
+| 6 | D.6 | Health check --drift | verify.cjs | S | 2 | P3 |
+| 7 | D.7 | Built-in convention rules | constants.cjs | S | 2 | P3 |
+| 8 | D.8 | Test suite (20+ tests) | tests/drift-check.test.cjs | M | 4 | P2 |
+| 9 | D.9 | Documentation | README, USER-GUIDE, CHANGELOG | S | 2 | P4 |
+
+**Total: 20 points (v0 MVP)**
+
+### Dependency Graph
+
+```
+D.7 (Constants) ──> D.1 (Parser) ──> D.2 (Checker) ──> D.3 (Score) ──> D.4 (Command)
+                                                                           ├──> D.5 (Dispatch)
+                                                                           ├──> D.6 (Health)
+                                                                           └──> D.8 (Tests) ──> D.9 (Docs)
+```
+
+## Test Plan
+
+### Test Pyramid (enforced)
+
+| Level | Count | What It Catches |
+|---|---|---|
+| Unit | 8+ | Rule parsing, file checking, score calculation |
+| Integration | 6+ | Git diff, arg parsing, JSON output, thresholds |
+| E2E | 2+ | Full workflow with real conventions |
+
+### Key Test Cases
+
+1. parseConventionRules: empty content -> []
+2. parseConventionRules: markdown with code blocks -> extracted rules
+3. parseConventionRules: invalid regex -> skipped, no crash
+4. checkFileConventions: clean file -> 0 violations
+5. checkFileConventions: console.log in .cjs -> violation
+6. checkFileConventions: existsSync usage -> warning
+7. calculateDriftScore: 0 violations -> 0.0
+8. calculateDriftScore: errors weighted 3x
+9. calculateDriftScore: capped at 1.0
+10. cmdDriftCheck: no conventions -> score 0
+11. cmdDriftCheck: with violations -> correct JSON
+12. cmdDriftCheck: --threshold exceeded -> exit 1
+13. cmdDriftCheck: --files specific path
+14. cmdDriftCheck: non-git dir -> error
+15. cmdDriftCheck: --since HEAD~3
+16. cmdDriftCheck: --raw output format
+17. Built-in rules match PAN patterns
+18. CRLF line endings handled
+19. Binary files skipped
+20. Large file (>100KB) skipped
+
+## Security
+
+- **Attack surface:** Read-only analysis. No code modification. No external calls.
+- **Path safety:** Applied via execFileSync (no shell injection) + path.resolve + project root check.
+- **Output sanitization:** toPosix paths, no absolute paths, no env vars, no stack traces.
+- **Content validation:** Regex try-catch for convention rules. No eval/Function.
+
+## Risk Register
+
+| Risk | Probability | Impact | Mitigation |
+|---|---|---|---|
+| False positive violations | Medium | Medium | Severity levels + configurable threshold |
+| Convention format variations | Medium | Low | Built-in rules as fallback |
+| Cross-platform paths | Medium | Medium | toPosix(), CRLF-safe patterns |
+| Performance on large repos | Low | Low | File count limit (100), size limit (100KB) |
+
+## Sources
+
+- [Guardrails for Agentic Coding](https://jvaneyck.wordpress.com/2026/02/22/guardrails-for-agentic-coding-how-to-move-up-the-ladder-without-lowering-your-bar/)
+- [AI Code Quality in 2026](https://tfir.io/ai-code-quality-2026-guardrails/)
+- [8 Tactics to Reduce Context Drift](https://lumenalta.com/insights/8-tactics-to-reduce-context-drift-with-parallel-ai-agents)
+- [AI Coding Agents: Coherence Through Orchestration](https://mikemason.ca/writing/ai-coding-agents-jan-2026/)
+- [Codified Context (arxiv)](https://arxiv.org/html/2602.20478v1)
+- [Cursor Rules Guide 2026](https://www.agentrulegen.com/guides/cursor-rules-guide)
+- [Aider Conventions](https://aider.chat/docs/usage/conventions.html)
+- [How We Prevent AI Agent Drift](https://dev.to/singhdevhub/how-we-prevent-ai-agents-drift-code-slop-generation-2eb7)
+- [CodeScene AI Guardrails](https://codescene.com/resources/use-cases/prevent-ai-generated-technical-debt)
+
+---
+
+*Generated by /featureAI — 2026-03-07*