@aperant/framework 0.6.2

package/workflows/verify-proof.md

@@ -0,0 +1,481 @@
+<purpose>
+Visual proof-of-work verification with structured evidence capture. Creates verification.json
+that tracks every test with action → evidence → verdict. Produces HTML proof report.
+
+This workflow is adversarial — it assumes things are broken until proven working.
+Every claim requires evidence. Every screenshot follows an interaction.
+
+OUTPUT FORMAT: verification.json (NOT VERIFICATION.md). JSON is the source of truth.
+</purpose>
+
+<context_profile>
+@packages/framework/context/installed/verification.md
+</context_profile>
+
+<template>
+@packages/framework/templates/proof-verification.md
+</template>
+
+<process>
+
+<step name="load_config" priority="first">
+## 1. Load Config and Context
+
+Read `.aperant/config.json` for verification preferences:
+```bash
+cat .aperant/config.json 2>/dev/null || echo "{}"
+```
+
+Read project constitution:
+```bash
+cat AGENTS.md 2>/dev/null || cat CLAUDE.md 2>/dev/null || echo "No constitution"
+```
+
+Read spec and plan:
+```bash
+cat spec.md 2>/dev/null
+cat implementation_plan.json 2>/dev/null
+cat testable_features.json 2>/dev/null
+```
+
+Generate run ID and create proof directory:
+```bash
+RUN_ID="$(date +%H%M%S)"
+APP_NAME=$(node -e "try{console.log(JSON.parse(require('fs').readFileSync('package.json','utf8')).name)}catch{console.log('app')}" 2>/dev/null)
+PROOF_DIR=".aperant/proof/${APP_NAME}/$(date +%Y%m%d)/${RUN_ID}"
+mkdir -p "${PROOF_DIR}/screenshots"
+```
+</step>
+
+<step name="pre_check">
+## 2. Pre-Check
+
+If `implementation_plan.json` exists, verify all subtasks are completed.
+If any are pending: report and exit.
+
+Check which MCP tools are available:
+- Try `mcp__electron__get_electron_window_info` — if responds, Electron is available
+- Try `mcp__puppeteer__puppeteer_navigate` — if responds, browser is available
+- Check `apt-proof-video --version` — if available, CDP video capture is enabled
+
+Record available tools in verification.json `meta.tools` array.
+</step>
+
+<step name="automated_checks">
+## 3. Run Automated Checks (with output capture)
+
+Run each check and capture the FULL terminal output for the proof report.
+
+**IMPORTANT:** Capture output, don't just check exit codes.
+
+```bash
+# Tests
+pnpm test 2>&1 | head -200
+```
+
+```bash
+# Type check
+pnpm typecheck 2>&1 | head -100
+```
+
+```bash
+# Lint
+pnpm lint 2>&1 | head -100
+```
+
+Record results in verification.json `automated_checks` object with full output.
+</step>
+
+<step name="discover_features">
+## 4. Deep Feature Discovery + Coverage Gate
+
+**This step determines verification quality. Do not rush it.**
+
+### 4a. Check the Feature Registry FIRST
+
+```bash
+ls .aperant/features/ 2>/dev/null
+cat .aperant/features/{area}.json 2>/dev/null
+```
+
+If `.aperant/features/{area}.json` exists for the area under test:
+- **Use it as your primary source** — it lists every sub-feature, wiring status, and test hints
+- Still verify against current code (the registry may be slightly stale)
+- If you discover features NOT in the registry, add them and flag: "Unregistered feature found"
+
+If no feature registry exists:
+- Suggest: "No feature registry found. Consider running `/apt:scan {area}` first for exhaustive coverage."
+- Proceed with manual discovery below, but warn that coverage may be incomplete
+
+### 4b. Read the source code
+
+You MUST read the actual source code for the features under test. Do not guess.
+For example, if testing a terminal feature:
+- Read the component files to find every prop, every button, every interaction
+- Read settings/config components to find every slider, dropdown, toggle
+- Read toolbar components to find every button and its handler
+- Read any test files for hints about expected behavior
+
+### 4c. Enumerate ALL sub-features
+
+For each feature area, list every individual testable element:
+- Every button → what does it do when clicked?
+- Every input/slider → what range does it accept? What does it change?
+- Every dropdown → what are the options? What does each option do?
+- Every toggle → what does it enable/disable?
+- Every keyboard shortcut → what does it trigger?
+
+### 4d. Compare against reference (if available)
+
+If the feature was ported from another codebase, or there is a design reference:
+- What should it look like when working correctly?
+- What visual quality markers indicate production-readiness?
+- A clean shell prompt with path and git branch is NOT the same as a blank terminal with cursor artifacts
+
+### 4e. Assess what CAN be tested via MCP
+
+For each sub-feature, decide:
+- **testable**: can interact via MCP click/type/navigate + verify via screenshot
+- **observable**: can see in screenshot but cannot interact (e.g., xterm.js internal rendering)
+- **blocked**: cannot test or observe via available tools
+
+Mark blocked items honestly. Do NOT mark them as pass.
+
+### 4f. Coverage Gate (MANDATORY — blocks test execution)
+
+After generating your test list, calculate coverage against the feature registry:
+
+```
+registered_count = total sub-features in .aperant/features/{area}.json
+tested_count = number of tests written (each linking to a registry_id)
+coverage = tested_count / registered_count
+```
+
+**Coverage rules:**
+- If `coverage < 50%`: **BLOCK** — "Only {tested_count}/{registered_count} features have tests. Add more tests before proceeding."
+  - Do NOT proceed to step 5. Go back and write more tests.
+- If `coverage < 80%`: **WARN** — list the top untested features by importance. Proceed but note the gap.
+- If `coverage >= 80%`: Proceed normally.
+
+**Every untested registered feature MUST go in the `skipped` array** with a reason.
+Valid skip reasons:
+- `time_constraint` — would take too long relative to value
+- `blocked_by_platform` — MCP tools cannot interact with this element
+- `depends_on_other` — requires a feature not yet implemented
+- `not_applicable` — feature is deprecated or not relevant to this verification
+
+Silence about untested features is NOT acceptable. Every registered sub-feature must appear
+either as a test (with `registry_id`) or in the `skipped` array.
+</step>
+
+<step name="generate_tests">
+## 5. Generate Test Cases
+
+For EVERY sub-feature discovered in step 4, generate an **action-based test**.
+
+Each test MUST have:
+- **id**: sequential test ID (e.g., "TEST-01")
+- **registry_id**: the sub-feature ID from `.aperant/features/{area}.json` (e.g., "terminals-grid-auto-tile")
+- **name**: descriptive test name
+- **precondition**: what must be true before testing (e.g., "app is running, terminal tab is active")
+- **action**: a specific thing to DO (e.g., "Type 'echo hello' and press Enter")
+- **expected**: a specific observable result (e.g., "Terminal displays 'hello' on the next line")
+
+**Rules for writing tests:**
+1. Every test involves an interaction (type, click, submit, navigate) OR a visual quality check
+2. "It renders" is NOT a valid test — "it renders X with properties Y when you do Z" IS
+3. If a feature has a text input, you must type into it
+4. If a feature has a button, you must click it
+5. If a feature has a slider, you must change it and verify the change took effect
+6. If a feature creates something, verify the creation AND its quality
+7. **Visual quality tests are valid** — "terminal shows clean prompt with path and git branch" IS a valid test
+8. **Artifact detection is valid** — "no rendering artifacts, stray characters, or broken formatting" IS a valid test
+9. Test sub-features exhaustively — if settings has 5 sliders, write 5 tests, not 1
+10. **Every test must reference a `registry_id`** from the feature registry. This links tests to sub-features and makes coverage tracking automatic.
+
+Write ALL tests into verification.json `tests` array before starting execution.
+
+**Gate:** Do not proceed to step 6 until ALL tests are written AND the coverage gate from step 4f passes.
+</step>
+
+<step name="start_video">
+## 6. Start Video Capture
+
+If CDP capture is available, start recording BEFORE executing tests:
+
+```bash
+apt-proof-video capture --port 9222 --duration {estimated_total_seconds} --fps 2 --output ${PROOF_DIR}
+```
+
+Run this in the background. The video will capture your ACTUAL test session —
+the same interactions that produce screenshots. This ensures video and screenshots
+are from the SAME session.
+
+**Important:** Estimate duration generously. Better to have extra video than to miss tests.
+- ~10 seconds per test is a reasonable estimate
+- Add 30 seconds for navigation overhead
+- If 20 tests: --duration 230
+
+**Note:** Captured frames are auto-stitched into an MP4 file by the tool. No manual assembly needed.
+
+If CDP is NOT available, note it and proceed without video.
+</step>
+
+<step name="execute_tests">
+## 7. Execute Tests (one at a time)
+
+For each test, in order:
+
+### 7a. Perform the Action
+Actually do what the test says:
+- If it says "type X" → use `mcp__electron__send_command_to_electron` with `fill_input` or `send_keyboard_shortcut`
+- If it says "click X" → use the appropriate click tool
+- If it says "navigate to X" → navigate there
+- If it says "run command X" → run it
+
+### 7b. Wait for Result
+Wait for the action to complete. If it involves loading, wait for the load.
+Add a short delay (1-2 seconds) so the CDP capture also captures this state.
+
+### 7c. Capture Evidence
+Take a screenshot AFTER the action completes:
+```
+Tool: mcp__electron__take_screenshot
+Args: {"outputPath": "${PROOF_DIR}/screenshots/{feature-id}-test-{n}.png"}
+```
+
+### 7d. Analyze Evidence
+**MANDATORY and CRITICAL.** Every screenshot must be analyzed in detail.
+
+You MUST describe what you see — not what you expect to see. Be adversarial:
+- Are there rendering artifacts? Stray characters? Broken formatting?
+- Does the UI look production-quality or does it look broken?
+- Compare against what a real user would expect to see
+- If testing a terminal: is the prompt clean? Is the path correct? Are there garbage characters?
+- If testing settings: did the value actually change? Is the control responsive?
+
+Write the analysis into the test's `analysis` field. Be specific and critical.
+"The terminal shows a cursor" is NOT analysis.
+"The terminal shows a blinking cursor at position 0,0 with no shell prompt, path, or git branch — this indicates the PTY session is not connected" IS analysis.
+
+If Gemini API is available, also run:
+```bash
+apt-proof-video analyze "${PROOF_DIR}/screenshots/{feature-id}-test-{n}.png" \
+  --model default \
+  --spec "{expected result from this test}"
+```
+
+### 7e. Record Result to verification.json
+After each test, update the test entry in verification.json `tests` array:
+
+```json
+{
+  "id": "TEST-01",
+  "registry_id": "terminals-grid-auto-tile",
+  "name": "Three Terminal Grid Auto-Tile",
+  "precondition": "2 terminals active",
+  "action": "Press Cmd+T",
+  "expected": "Grid reflows to 2+1 layout",
+  "actual": "Grid correctly tiled",
+  "evidence": "screenshots/03-grid.png",
+  "analysis": "Detailed adversarial analysis of what is actually visible in the screenshot...",
+  "result": "pass",
+  "severity": null
+}
+```
+
+Compare `expected` vs `actual`:
+- If they match → `result: "pass"`
+- If they partially match but have quality issues → `result: "fail"`, `severity: "minor"` or `"cosmetic"`
+- If they don't match → `result: "fail"`, set `severity` (`"blocker"`, `"major"`, `"minor"`, `"cosmetic"`), add to `gaps` array
+- If you can't test → `result: "blocked"`, explain why in `analysis`
+
+**Do NOT give a pass to something that has visual artifacts, broken rendering, or
+looks different from the expected production quality.**
+
+### 7f. Write verification.json after each test
+Write the updated verification.json to disk after each test completes.
+This ensures progress is saved even if the session is interrupted.
+
+**Gate:** Complete ALL tests before proceeding to step 8.
+</step>
+
+<step name="score_dimensions">
+## 8. Score QA Dimensions
+
+Based on the evidence collected (not assumed), score each dimension.
+Write scores to verification.json `dimensions` object:
+
+```json
+{
+  "dimensions": {
+    "completeness": {
+      "score": 7,
+      "rationale": "18/22 spec requirements have passing tests. 4 blocked by platform."
+    },
+    "correctness": {
+      "score": 8,
+      "rationale": "All tested features behave as expected. No blockers or majors."
+    },
+    "code_quality": {
+      "score": 9,
+      "rationale": "Zero lint errors, zero type errors."
+    },
+    "test_coverage": {
+      "score": 6,
+      "rationale": "Unit tests cover core logic but no integration tests for new grid layout."
+    }
+  }
+}
+```
+
+**Scoring rules:**
+
+**Completeness (1-10):**
+- Count how many spec requirements have passing tests
+- A requirement without a test scores 0
+- A requirement with a failing test scores 0
+- Sub-features not tested (e.g., individual settings) lower this score
+
+**Correctness (1-10):**
+- Based on automated check results AND manual test results
+- Any blocker = max score 3
+- Any major = max score 5
+- Visual artifacts or rendering issues count as failures
+
+**Code Quality (1-10):**
+- Based on lint results and code review (if done)
+
+**Test Coverage (1-10):**
+- Based on test suite results and whether new code has tests
+
+**Derive verdict:**
+- ALL dimensions >= 7: `"approved"`
+- ALL dimensions >= 5: `"approved_with_notes"`
+- ANY dimension < 5: `"rejected"`
+
+Write verdict to verification.json `verdict` field.
+</step>
+
+<step name="multi_model" optional="true">
+## 9. Multi-Model Verification (if enabled in config)
+
+Only run if `.aperant/config.json` has `multi_model.enabled: true`.
+
+For details on how to invoke each model, read `references/multi-model-verification.md`.
+For setup instructions, read `references/multi-model-setup.md`.
+
+Record each model's verdict in verification.json `model_verdicts` array.
+</step>
+
+<step name="generate_report">
+## 10. Generate HTML Proof Report
+
+Use the `apt-proof-video report` command to generate the HTML report from verification.json:
+
+```bash
+apt-proof-video report --proof-dir "${PROOF_DIR}" --title "{descriptive title}"
+```
+
+The report command:
+1. Reads verification.json as the primary data source
+2. Recursively scans for screenshots in subdirectories
+3. Generates feature checklists with pass/fail badges per test
+4. Embeds screenshots inline per test
+5. Shows automated checks with expandable output
+6. Shows QA dimension progress bars
+7. Shows gaps with severity badges
+8. Embeds video player (frames are auto-stitched to MP4)
+
+Also create convenience symlink inside .aperant/ (NOT at project root):
+```bash
+ln -sf "${PROOF_DIR}/report.html" .aperant/proof-report.html
+```
+</step>
+
+<step name="open_report">
+## 11. Open Report
+
+If `config.verification.open_report` is true (default):
+```bash
+open "${PROOF_DIR}/report.html"
+```
+</step>
+
+<step name="write_artifacts">
+## 12. Write Artifacts
+
+Write the following files:
+- `${PROOF_DIR}/verification.json` — the structured verification record (source of truth)
+- `${PROOF_DIR}/report.html` — self-contained HTML report (generated by apt-proof-video)
+- `${PROOF_DIR}/report.md` — markdown summary for PRs (generated by apt-proof-video)
+- `${PROOF_DIR}/qa_signoff.json` — QA verdict (inside proof dir, not project root)
+
+**Do NOT write VERIFICATION.md.** The JSON file is the only source of truth.
+
+Commit:
+```bash
+apt-tools commit "qa: proof verification {verdict}" --files ${PROOF_DIR}/verification.json ${PROOF_DIR}/report.html ${PROOF_DIR}/report.md ${PROOF_DIR}/qa_signoff.json
+```
+</step>
+
+<step name="report_output">
+## 13. Final Output
+
+```
+Proof Verification: {VERDICT}
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+Outcome: {one-line summary}
+
+Checks:
+  Tests:     {pass/fail} ({count} tests)
+  Types:     {pass/fail}
+  Lint:      {pass/fail}
+
+Scores:
+  Completeness:  {n}/10
+  Correctness:   {n}/10
+  Code Quality:  {n}/10
+  Test Coverage: {n}/10
+
+Coverage:
+  Registry:  {tested}/{total} features ({percent}%)
+  Skipped:   {n} features (see skipped array)
+
+Evidence:
+  Features:    {n} tested ({passed} pass, {failed} fail, {blocked} blocked)
+  Screenshots: {n} captured, {n} analyzed
+  Video:       {captured/not available}
+  Gaps:        {n} issues ({blockers} blockers, {majors} major)
+
+Proof: {PROOF_DIR}/
+  verification.json  (structured test record)
+  report.html        (opened in browser)
+  report.md          (for PR embedding)
+
+{If rejected:} Fix gaps and re-run /apt:verify-proof
+{If approved:} Next: /apt:review (optional code review)
+```
+</step>
+
+</process>
+
+<gates>
+## Workflow Gates
+
+These gates MUST be respected — do not skip them:
+
+1. **Pre-check gate** (step 2): All subtasks must be completed before verification starts
+2. **Feature discovery gate** (step 4): Source code MUST be read before writing tests
+3. **Coverage gate** (step 4f): Coverage < 50% blocks test execution. Every untested feature must be in the skipped array.
+4. **Test generation gate** (step 5): ALL tests must be written before execution starts
+5. **Registry link gate** (step 5): Every test must have a `registry_id` linking to the feature registry
+6. **Video gate** (step 6): Video capture starts BEFORE test execution, not after
+7. **Test execution gate** (step 7): ALL tests must be executed before scoring
+8. **Evidence gate** (step 7c-7d): Every test must have evidence AND critical analysis
+9. **Quality gate** (step 7d): Visual artifacts, broken rendering, and quality issues MUST be reported as failures
+10. **Scoring gate** (step 8): Scores must be derived from evidence, not assumed
+11. **Output format gate**: Write verification.json, NEVER VERIFICATION.md
+</gates>
+</process>
+</output>