@mmerterden/multi-agent-pipeline 18.0.0 → 19.1.0

package/pipeline/multi-agent-refs/phases/phase-5-test.md

@@ -1,182 +0,0 @@
-### Phase 5: Test
-
-> **TLDR**  -  Optional test gate. Offers to boot the simulator/emulator (UI Bug Hunter) or hand off to the user for manual QA. Needs an interactive prompt AND a worktree checkout, so it is in the phase set of `/multi-agent` alone and dropped by every `autopilot` or `--local` entry. Depth does not affect it: a Short run still reaches Phase 5. If issues found, loops back to Phase 3.
-
-<!-- progress-contract: applied -->
-Progress emission per `$HOME/.claude/multi-agent-refs/progress-contract.md`  -  lines for local-test prompt render, user-answer capture, repo checkout (if selected).
-
-#### Step 0  -  Test Gap Report (advisory)
-
-`state.testPolicy: none` → skip the gap scan (the gap IS the recorded policy) and run only pre-existing test targets; none → recorded no-op. Otherwise, before the local-checkout prompt, run the static test-gap detector. Heuristic, deterministic, no LLM, sub-second. The report ends up in `agent-log.md` under "Test Scenarios" and surfaces public symbols added in this branch that have no paired test.
-
-```bash
-STACK=$(jq -r '.analysis.stack.primary // "unknown"' "$STATE_FILE")
-case "$STACK" in
-  ios|swift)             SCAN_STACK=ios ;;
-  android|kotlin)        SCAN_STACK=android ;;
-  python)                SCAN_STACK=python ;;
-  node|typescript|js)    SCAN_STACK=node ;;
-  *)                     SCAN_STACK="" ;;
-esac
-if [ -n "$SCAN_STACK" ] && [ "${prefs_testGap_enabled:-true}" = "true" ]; then
-  GAP_FLAGS=""
-  [ "${prefs_testGap_scanTree:-false}" = "true" ] && GAP_FLAGS="$GAP_FLAGS --scan-tree"
-  [ "${prefs_testGap_promoteSeverity:-false}" = "true" ] && GAP_FLAGS="$GAP_FLAGS --severity-promote"
-  GAP_JSON=$(node $HOME/.claude/scripts/test-gap-scan.mjs \
-    --base "$BASE_BRANCH" --head HEAD --stack "$SCAN_STACK" $GAP_FLAGS 2>/dev/null)
-  echo "$GAP_JSON" | node $HOME/.claude/scripts/validate-test-gap.mjs - >/dev/null 2>&1 || GAP_JSON=""
-fi
-```
-
-**What the report contains** (per `$HOME/.claude/schemas/test-gap.schema.json`):
-
-| Field | Meaning |
-|---|---|
-| `gaps[].sourcePath` | source file with the unprotected symbol |
-| `gaps[].symbol` | symbol name |
-| `gaps[].kind` | rule id (e.g. `public_func`, `composable_fun`, `named_export`) |
-| `gaps[].severity` | `blocking` / `important` / `suggestion` (see severity table below) |
-| `gaps[].expectedTestPaths` | likely test paths the user should land at, in priority order |
-| `gaps[].hint` | stack-specific testing reminder (e.g. swiftui-qa.md 3-layer) |
-
-**Severity defaults**:
-
-| Symbol kind | Severity |
-|---|---|
-| `composable_fun`, `view_struct`, `config_struct`, `interface`, `objc_export`, `public_proto` | important |
-| Other public API additions (`public_func`, `open_fun`, `named_export`, `default_export`, ...) | suggestion |
-
-**Gating** (opt-in): if `prefs.testGap.blockingThreshold` is set and `gapBySeverity.important + gapBySeverity.blocking` exceeds it, Phase 5 surfaces the report as a Phase 4 rework finding and loops back. **Default off**  -  gaps render as advisory under "Test Gap Report" only.
-
-**Telemetry**:
-
-```bash
-LOG_METRIC_FORWARD_TO_TRACKER=0 $HOME/.claude/scripts/log-metric.sh "$TASK_ID" 5 test_gap.scanned \
-  stack=$SCAN_STACK \
-  sources=$(jq '.totals.sourcesScanned' <<< "$GAP_JSON") \
-  gaps=$(jq '.totals.gapCount' <<< "$GAP_JSON")
-```
-
-(No tracker forwarding  -  the scanner has no token cost.)
-
-**Figma reference panel (when `state.evidence.figma[]` is non-empty).** Before the local-checkout prompt, print a single block listing each captured frame so the user has a side-by-side reference during manual test:
-
-```
-Figma evidence (tier=<n>):
-  <fileKey>:<nodeId>  <canonicalComponentName>
-    screenshot: <screenshotUrl or local path>
-  <fileKey>:<nodeId>  <canonicalComponentName>
-    screenshot: <screenshotUrl or local path>
-```
-
-Tier 1 / Tier 2 records print `screenshotUrl` from the captured evidence (Tier 2 URLs expire after 30 days, re-fetch on the spot if needed). Tier 3 records print the local path to the user-attached screenshot. The block is informational; it never blocks the prompt.
-
-1. Ask with a native `AskUserQuestion` picker (never a typed y/N prompt). The options MUST make the local-checkout side effect explicit  -  testing removes the worktree and checks the branch out into the main repo:
-   - `question`: "Check out locally to test now?" (rendered in `outputLanguage`)
-   - `header`: "Test" (English, <=12 chars)
-   - `options`:
-     - `{ label: "Test now", description: "Removes the worktree and checks the branch out into the main repo for Xcode / manual test" }`
-     - `{ label: "Skip", description: "Stay in the worktree and go to Phase 6" }`
-   - **Skip** → set `state.phases["5"].status = "skipped"` (so Phase 6 can offer the local-checkout prompt) → Phase 6
-   - **Test now** → set `state.phases["5"].status = "in_progress"` → continue:
-2. **Commit changes in worktree BEFORE removing** (WIP commit to preserve work):
-   ```
-   git -C {worktree-path} add -A
-   git -C {worktree-path} commit -m "WIP: {jiraId}  -  changes for user test"
-   ```
-3. Remove worktree, checkout branch in main repo:
-   ```
-   git worktree remove .worktrees/{jiraId} --force
-   git checkout {branch-name}
-   ```
-   Branch now has the WIP commit  -  all changes are preserved.
-4. Show test instructions:
-   ```
-   Switched to branch: {branch-name}
-   To test: Xcode -> build -> run -> manual test
-   "ok" -> proceeds to Phase 6 (WIP commit will be replaced via git reset HEAD~1 + proper commit)
-   "fix: ..." -> worktree is recreated, returns to Phase 3
-   ```
-5. Mark the tracker as waiting, then wait for the user response:
-   ```bash
-   bash $HOME/.claude/scripts/phase-tracker.sh now 5 "awaiting local test (user)"
-   bash $HOME/.claude/scripts/phase-tracker.sh render
-   ```
-   The waiting state persists in `tracker-state.json` across the handoff; `/multi-agent:resume-local` and `/multi-agent:manual-test` CONTINUE this state file and never re-init it (`$HOME/.claude/multi-agent-refs/tracker-contract.md` "Continuation runs").
-
-   **"ok" is a structured result, not a word.** Before "ok" is accepted, the run writes `$WORKTREE/.pipeline/manual-test.json`: one entry per acceptance criterion, the criteria taken from the analysis doc test plan (Section 15 / 20), the plan tasks, and the user's own words in the reply. Every criterion records what was seen; a criterion that was not tried says so with a reason.
-   ```json
-   {"criteria":[{"spec":"<quote>","source":"analysis 15.2 | plan task 3 | user","observed":"<what was seen>","verdict":"pass|fail|not-tested","reason":"<required when not-tested>","screenshot":"<path or null>"}],"verdict":"passed|failed"}
-   ```
-   Then gate it:
-   ```bash
-   node $HOME/.claude/scripts/evidence-gate.mjs --claim manual --status passed --evidence "$WORKTREE/.pipeline/manual-test.json"
-   ```
-   Add `--require-screenshot` when `state.visualEvidence.required` is true: a passing criterion then has to name a file that exists, because a `screenshot` key pointing nowhere is not evidence.
-
-   Exit 1 means the "ok" is not accepted: tell the user which criterion is missing evidence (a `fail` verdict, or `not-tested` without a reason) and wait for the next reply. Exit 0 marks Phase 5 completed with `Result "local test passed (user)"`. The "fix: ..." path below is unchanged.
-6. If fix needed:
-   - Branch already has WIP commit (from step 2)  -  changes are safe
-   - **Heal stale admin state first** (same contract as Phase 0  -  step 3's
-     `worktree remove` or an interrupted run can leave a stale entry, so a bare
-     re-add fails with `already exists`/`already registered`):
-     ```bash
-     git -C "$PROJECT_ROOT" worktree prune 2>/dev/null || true
-     if git -C "$PROJECT_ROOT" worktree list --porcelain | grep -qF "{worktree-path}"; then
-       git -C "$PROJECT_ROOT" worktree unlock "{worktree-path}" 2>/dev/null || true
-     fi
-     ```
-     Phase 0's repo residue guard is already in `.git/info/exclude`  -  no re-add.
-   - Recreate worktree from branch: `git -C $PROJECT_ROOT worktree add {worktree-path} {branch}`
-   - Re-set git identity: `git -C {worktree-path} config user.name/email` (from state)
-   - Go back to Phase 3
-7. Log: "Phase 5: Test  -  {result}"
-
-**CRITICAL**: Never remove a worktree with uncommitted changes. Always WIP commit first.
-
-#### Automated Device Checks (on-demand)
-
-Before or during user testing, run device-level audits via Bash if user requests. See `audit-guide.md` for commands.
-
-| Check               | When              | Command                                   |
-| ------------------- | ----------------- | ----------------------------------------- |
-| UI flow video       | `state.visualEvidence.required` AND Phase 3 recorded none | `capture-evidence.sh video start` -> drive the flow -> `video stop` -> `fit`. See below |
-| Accessibility audit | UI changes        | `mcp__multi-agent-toolkit__{ios,android}_accessibility_audit` |
-| Biometric test      | Auth flow changes | ios: `mcp__multi-agent-toolkit__ios_biometric` (android: manual) |
-| Launch time         | Perf-sensitive changes | ios: app-launch instrument · android: `mcp__multi-agent-toolkit__android_launch_time` |
-| Visual test         | Any UI changes    | `/multi-agent test` (sim-test, both platforms) |
-| Snapshot regression | Component / pixel-stable UI changes | ios: `mcp__multi-agent-toolkit__ios_visual_diff` · android: `mcp__multi-agent-toolkit__android_screenshot` + compare |
-| Store screenshots   | `taskType === screenshot` | ios: `ios_status_bar({preset: "clean"})` · android: `android_screenshot` |
-
-##### UI flow video, when Phase 3 produced none
-
-Phase 3 Step 3.55 is the primary host and runs in every mode. Phase 5 is the richer one where it exists: the device is up and a person is watching, so the flow is one somebody confirmed. It adds, never replaces.
-
-Run only when `visualEvidence.required` and `visualEvidence.video.file` is empty: re-check the device with `$HOME/.claude/scripts/probe-evidence-capability.sh --only device`, then `$HOME/.claude/scripts/capture-evidence.sh video start` -> drive the flow (`run-ui-tests.sh run`, or the Phase 5 scenarios by hand) -> `video stop` -> `fit`. The cap is `visualEvidence.maxVideoSeconds`, read through `capture-evidence.sh limits` so one value serves both. Update `videoTier` and `videoTierReason` with what ran.
-
-When the intake answer was `unit` there is no recording here either: overriding it in a phase the user may not be watching makes the question decorative.
-
-Results included in Phase 7 report. MCP tools preferred when available  -  concise structured output, lower token cost.
-
-**Snapshot regression flow (optional):** when the task changes a stable component, capture a screenshot before the change (baseline) and after (current), then call `ios_visual_diff({baseline, current, max_diff_pct: 1.0})`. Threshold can be relaxed for animated / non-deterministic regions  -  keep `max_diff_pct ≤ 1.0` for static layouts.
-
-#### Security Audit (store-readiness)
-
-When the task touches authentication, keychain, network, or is scheduled for an imminent release, launch the `security-auditor` subagent to run an OWASP Mobile Top 10 pass plus App Store / Play Store compliance checks:
-
-```
-Agent(subagent_type: "security-auditor", prompt: "<diff + context>")
-```
-
-Returns severity-tagged findings (Critical / High / Medium). Critical items block Phase 6 just like Phase 4 blockers; High items are logged and surfaced in Phase 7 report. Skipped by default  -  opt-in for release branches or on explicit `/multi-agent "<task>" --audit` flag.
-
-#### Telemetry  -  token forwarding
-
-When the security-auditor or any other Phase 5 sub-agent runs, forward its token totals so Phase 7's Cost Breakdown captures Phase 5:
-
-```bash
-LOG_METRIC_FORWARD_TO_TRACKER=1 $HOME/.claude/scripts/log-metric.sh "$TASK_ID" 5 audit.completed \
-  model=opus tokens_in=$IN tokens_out=$OUT duration_ms=$DUR
-```
-
-If Phase 5 is purely user-driven (no sub-agent ran), no token forwarding is required and the cost block stays empty for this phase. Best-effort. See `$HOME/.claude/multi-agent-refs/progress-contract.md#token-telemetry-forwarding`.