forgedock 1.0.22 → 1.1.2

package/commands/pipeline-health/phase-3-analyze.md

@@ -0,0 +1,204 @@
+---
+install: extras
+---
+<!-- SPDX-FileCopyrightText: Copyright (c) RapierCraft Studios -->
+<!-- SPDX-License-Identifier: AGPL-3.0-or-later -->
+
+# /pipeline-health — Phase 3: Analyze & Correlate
+
+## Phase 3: Analyze & Correlate
+
+### 3A: Defect category breakdown
+
+**3A-I: Pre-merge defects** — Group review findings (from Phase 2B) by agent prefix.
+
+First, count findings per category from the Phase 2B output:
+
+```bash
+# Count review findings per agent prefix (SEC, AUTH, BILL, DB, FE, INFRA, API, CONC, SCRP)
+# These are the current-period counts to compare against PRIOR_* vars from Phase 1C
+COUNT_SEC=$(echo "$FINDINGS_RAW" | grep -c '"SEC' || echo 0)
+COUNT_AUTH=$(echo "$FINDINGS_RAW" | grep -c '"AUTH' || echo 0)
+COUNT_BILL=$(echo "$FINDINGS_RAW" | grep -c '"BILL' || echo 0)
+COUNT_DB=$(echo "$FINDINGS_RAW" | grep -c '"DB' || echo 0)
+COUNT_FE=$(echo "$FINDINGS_RAW" | grep -c '"FE' || echo 0)
+COUNT_INFRA=$(echo "$FINDINGS_RAW" | grep -c '"INFRA' || echo 0)
+COUNT_API=$(echo "$FINDINGS_RAW" | grep -c '"API' || echo 0)
+COUNT_CONC=$(echo "$FINDINGS_RAW" | grep -c '"CONC' || echo 0)
+COUNT_SCRP=$(echo "$FINDINGS_RAW" | grep -c '"SCRP' || echo 0)
+# ISSUE_SPEC from Phase 2L audit findings
+COUNT_ISSUE_SPEC=$ISSUE_SPEC_COUNT
+
+TOTAL_FINDINGS=$((COUNT_SEC + COUNT_AUTH + COUNT_BILL + COUNT_DB + COUNT_FE + COUNT_INFRA + COUNT_API + COUNT_CONC + COUNT_SCRP))
+
+# Compute trend for each category using _trend helper from Phase 1C
+TREND_SEC=$(       _trend "$COUNT_SEC"        "$PRIOR_SEC")
+TREND_AUTH=$(      _trend "$COUNT_AUTH"       "$PRIOR_AUTH")
+TREND_BILL=$(      _trend "$COUNT_BILL"       "$PRIOR_BILL")
+TREND_DB=$(        _trend "$COUNT_DB"         "$PRIOR_DB")
+TREND_FE=$(        _trend "$COUNT_FE"         "$PRIOR_FE")
+TREND_INFRA=$(     _trend "$COUNT_INFRA"      "$PRIOR_INFRA")
+TREND_API=$(       _trend "$COUNT_API"        "$PRIOR_API")
+TREND_CONC=$(      _trend "$COUNT_CONC"       "$PRIOR_CONC")
+TREND_SCRP=$(      _trend "$COUNT_SCRP"       "$PRIOR_SCRP")
+TREND_ISSUE_SPEC=$(_trend "$COUNT_ISSUE_SPEC" "$PRIOR_ISSUE_SPEC")
+
+_pct() { [ "$TOTAL_FINDINGS" -gt 0 ] && echo "scale=0; $1 * 100 / $TOTAL_FINDINGS" | bc || echo "0"; }
+```
+
+Then render the table using the computed values:
+
+| Category | Count | % of Total | Trend vs Prior Period |
+|----------|-------|------------|---------------------|
+| SEC | $COUNT_SEC | $(_pct $COUNT_SEC)% | $TREND_SEC |
+| AUTH | $COUNT_AUTH | $(_pct $COUNT_AUTH)% | $TREND_AUTH |
+| BILL | $COUNT_BILL | $(_pct $COUNT_BILL)% | $TREND_BILL |
+| DB | $COUNT_DB | $(_pct $COUNT_DB)% | $TREND_DB |
+| FE | $COUNT_FE | $(_pct $COUNT_FE)% | $TREND_FE |
+| INFRA | $COUNT_INFRA | $(_pct $COUNT_INFRA)% | $TREND_INFRA |
+| API | $COUNT_API | $(_pct $COUNT_API)% | $TREND_API |
+| CONC | $COUNT_CONC | $(_pct $COUNT_CONC)% | $TREND_CONC |
+| SCRP | $COUNT_SCRP | $(_pct $COUNT_SCRP)% | $TREND_SCRP |
+| ISSUE_SPEC | $COUNT_ISSUE_SPEC | $(_pct $COUNT_ISSUE_SPEC)% | $TREND_ISSUE_SPEC |
+
+**If PRIOR_* vars are all empty (baseline run)**: all trend cells show "—" (no prior period to compare against).
+
+**Note**: ISSUE_SPEC findings come from `/audit` (`audit-finding` label + body contains "ISSUE_SPEC"), not from review agents. They represent upstream failures — bad issue specs that the pipeline followed correctly but that produced wrong or wasted work.
+
+**3A-II: Post-deploy defects** — Group audit findings (from Phase 2L) by failure point:
+
+| Failure Point | Count | % of Total | Meaning |
+|---------------|-------|------------|---------|
+| INVESTIGATION | ? | ?% | Wrong verdict or missed scope — root cause misdiagnosed |
+| IMPLEMENTATION | ? | ?% | Code written incorrectly or incompletely |
+| REVIEW | ? | ?% | Review agents missed the issue or wrong agents triggered |
+| REVIEW_FALSE_NEG | ? | ?% | Review agent saw area but approved a defect (escape signal) |
+| QUALITY_GATE | ? | ?% | Static check should have caught this pattern but didn't |
+| DEPLOY_GATE | ? | ?% | Deploy-info should have flagged risk but didn't |
+| CONTEXT | ? | ?% | Missed relevant prior work or related code paths |
+| ARCHITECT | ? | ?% | Missed code paths or wrong implementation plan |
+| ORCHESTRATION | ? | ?% | Wave ordering wrong or dependency missed |
+| ISSUE_SPEC | ? | ?% | Bad issue spec — pipeline followed it correctly, spec was wrong |
+| OTHER | ? | ?% | Uncategorized or compound failure |
+
+**Cross-signal analysis**: Compare 3A-I (pre-merge review findings) with 3A-II (post-deploy audit findings) to identify where the pipeline gates are misaligned. High REVIEW_FALSE_NEG in 3A-II with low REVIEW findings per PR in 3A-I suggests review agents are approving too liberally rather than finding real defects.
+
+**Identify the top 3 defect categories across both tables.** These are where the builder/review is weakest and where pipeline improvements would have the highest ROI.
+
+### 3B: Prompt change impact
+
+For each Forge commit in the window:
+1. What command was changed?
+2. What was the intent? (read the commit message)
+3. Did the target metric improve after this change?
+   - If the change was to `review-pr-agents.md` → did false positive rate change?
+   - If the change was to `work-on.md` Step 3G → did build failure rate change?
+   - If the change was to `orchestrate.md` → did milestone merge success rate change?
+
+**Caution**: Correlation ≠ causation. Note confounding factors (e.g., milestone size, new developers).
+
+### 3C: Compute health score
+
+```
+Health Score = weighted average of:
+  Pre-merge quality:
+  - Build pass rate (staging→main):         weight 12%    target 95%+
+  - Review findings per PR:                 weight 9%     target < 0.5
+  - Manual fix-up rate:                     weight 8%     target < 15%
+  - False positive rate:                    weight 6%     target < 10%
+  - Pipeline self-correction rate:          weight 7%     target < 15%
+
+  Post-deploy quality:
+  - Post-deploy failure rate (Phase 2L):    weight 9%     target < 5%
+  - Review escape rate (Phase 2L):          weight 9%     target < 2%
+
+  Pipeline health signals:
+  - Issue quality score (Phase 2J):         weight 8%     target > 80%
+  - Investigation accuracy (Phase 2I):      weight 6%     target > 85%
+  - Milestone merge success rate:           weight 6%     target 80%+
+  - Orchestration efficiency (Phase 2K):    weight 5%     target score > 70
+  - Original work ratio:                   weight 4%     target > 70%
+  - Issue close velocity (days):            weight 1%     target < 2 days
+
+  Transcript analytics (supplementary — Phase 2M–2Q):
+  - Transcript health score (Phase 2N/2Q):  weight 10%    target > 70
+```
+
+**Total weight**: 12+9+8+6+7+9+9+8+6+6+5+4+1+10 = 100%.
+
+**Post-deploy failure rate** (Phase 2L): scored as `max(0, 100 - (rate / 5%) * 100)` — i.e., 0% failure rate = 100 score, 5% failure rate = 0 score. If no audit findings exist in the window, score = 100.
+
+**Review escape rate** (Phase 2L): scored as `max(0, 100 - (rate / 2%) * 100)` — i.e., 0% escape rate = 100 score, 2% escape rate = 0 score. If no REVIEW_FALSE_NEG findings exist, score = 100. This metric directly measures review agent effectiveness: a high escape rate with a low findings-per-PR rate indicates review agents are approving too liberally.
+
+**Issue quality score** is computed from Phase 2J sub-metrics: invalidation rate (40%), ISSUE_SPEC findings (30%), re-investigation rate (20%), structure compliance (10%). A perfect score (100) requires: 0% invalidation, 0 ISSUE_SPEC audit findings, 0 re-investigations, 100% structure compliance.
+
+**Investigation accuracy** comes from Phase 2I trajectory data: (CONFIRMED + PARTIAL) / total issues with trajectory comments. If no trajectory data is available in the window, omit this metric from the weighted average and redistribute its 7% proportionally across the other metrics.
+
+**Orchestration efficiency score** is computed from Phase 2K metrics: avg idle% (40% weight — lower is better, 0% idle = 100 score), avg resumes per agent (35% weight — 0 resumes = 100, ≥2 resumes = 0), clean agent rate (25% weight — equals clean_rate%). If no persisted `FORGE:AUDIT-AGENTS` data is available in the window, omit this metric and redistribute its 5% proportionally across the remaining metrics.
+
+**Transcript health score** (Phase 2N/2Q — supplementary signal, weight 10%): Computed from three Phase 2N/2Q output variables. If `TRANSCRIPT_DATA_AVAILABLE=false`, skip this metric and redistribute its 10% proportionally across the remaining metrics.
+
+```bash
+if [ "${TRANSCRIPT_DATA_AVAILABLE:-false}" = "true" ]; then
+  # stall_rate_score: 0% stall rate = 100, 10%+ stall rate = 0
+  # STALL_TURN_RATE is a percentage integer from Phase 2N (e.g., 5 for 5%)
+  STALL_TURN_RATE=${STALL_TURN_RATE:-0}
+  STALL_RATE_SCORE=$(echo "scale=0; r=$STALL_TURN_RATE; if(r>10) 0 else (10-r)*10" | bc 2>/dev/null || echo 0)
+
+  # compaction_score: 0 compactions = 100, 5+ compactions = 0
+  # COMPACTION_TOTAL is the total compaction events across all sessions from Phase 2N
+  COMPACTION_TOTAL=${COMPACTION_TOTAL:-0}
+  COMPACTION_SCORE=$(echo "scale=0; c=$COMPACTION_TOTAL; if(c>5) 0 else (5-c)*20" | bc 2>/dev/null || echo 0)
+
+  # model_split_score: 0% opus = 100, 100% opus = 0
+  # OPUS_PCT is derived from Phase 2Q's model_distribution output (integer 0-100)
+  # If not set (Phase 2Q skipped), use 0 as conservative fallback
+  OPUS_PCT=${OPUS_PCT:-0}
+  MODEL_SPLIT_SCORE=$(echo "scale=0; 100 - $OPUS_PCT" | bc 2>/dev/null || echo 100)
+  [ "$MODEL_SPLIT_SCORE" -lt 0 ] 2>/dev/null && MODEL_SPLIT_SCORE=0
+
+  # Weighted composite: stall 50% + compaction 30% + model split 20%
+  TRANSCRIPT_HEALTH_SCORE=$(echo "scale=0; ($STALL_RATE_SCORE * 50 + $COMPACTION_SCORE * 30 + $MODEL_SPLIT_SCORE * 20) / 100" | bc 2>/dev/null || echo 0)
+  echo "Transcript health score: $TRANSCRIPT_HEALTH_SCORE/100 (stall_rate_score=$STALL_RATE_SCORE compaction_score=$COMPACTION_SCORE model_split_score=$MODEL_SPLIT_SCORE)"
+  TRANSCRIPT_SCORE_AVAILABLE=true
+else
+  TRANSCRIPT_HEALTH_SCORE=""
+  TRANSCRIPT_SCORE_AVAILABLE=false
+  echo "Transcript health score: N/A (TRANSCRIPT_DATA_AVAILABLE=false — 10% weight redistributed proportionally)"
+fi
+```
+
+**Transcript flow qualitative flags** (supplementary, appear in report alongside the numeric score):
+- Stall turn rate > 10%: flag as ⚠️ `High stall rate — agents spending >10% of turns in long waits`
+- Compaction events > 5 in window: flag as ⚠️ `Frequent context compaction — consider reducing prompt or tool output size`
+- Stall turn rate ≤ 10% and compaction ≤ 5: flag as ✅ `Session flow healthy`
+- Data not available: flag as ℹ️ `Transcript flow metrics unavailable (requires Phase 2M session data)`
+
+**Aggregated session metrics** (Phase 2Q): If `$TRANSCRIPT_AGGREGATION_AVAILABLE` is true, embed `$AGGREGATED_TRANSCRIPT_METRICS` verbatim into the \"Aggregated Session Metrics\" section of the Phase 5A report. The guardrail footnotes (5 harmful-takeaway warnings from #347) MUST be included in that section even when session count is low — they protect against misreading the metrics table. If `TRANSCRIPT_AGGREGATION_AVAILABLE` is false, emit the unavailability note from Phase 2Q.
+
+**Anomaly signal**: If Phase 2I anomaly rate exceeds 10% of issues, flag it as a warning in the health report even though it does not feed the score directly — high anomaly rates indicate pipeline instability not captured by outcome metrics.
+
+Score 0-100. Categorize:
+- 90-100: Excellent — pipeline is shipping clean code
+- 70-89: Good — some defect categories need attention
+- 50-69: Needs work — specific pipeline links are underperforming
+- < 50: Critical — systemic issues in the pipeline
+
+**Health score trend**: Compare the computed score against `$PRIOR_HEALTH_SCORE` from Phase 1C:
+
+```bash
+# Compute score trend string for report header
+if [ -z "$PRIOR_HEALTH_SCORE" ]; then
+    SCORE_TREND="(baseline — no prior period)"
+elif [ "$HEALTH_SCORE" -gt "$PRIOR_HEALTH_SCORE" ] 2>/dev/null; then
+    SCORE_TREND="(↑ from $PRIOR_HEALTH_SCORE — +$(( HEALTH_SCORE - PRIOR_HEALTH_SCORE )) pts)"
+elif [ "$HEALTH_SCORE" -lt "$PRIOR_HEALTH_SCORE" ] 2>/dev/null; then
+    SCORE_TREND="(↓ from $PRIOR_HEALTH_SCORE — -$(( PRIOR_HEALTH_SCORE - HEALTH_SCORE )) pts)"
+else
+    SCORE_TREND="(= unchanged from $PRIOR_HEALTH_SCORE)"
+fi
+echo "Health Score: $HEALTH_SCORE/100 $SCORE_TREND"
+```
+
+---
+

package/commands/pipeline-health/phase-4-proposals.md

@@ -0,0 +1,117 @@
+---
+install: extras
+---
+<!-- SPDX-FileCopyrightText: Copyright (c) RapierCraft Studios -->
+<!-- SPDX-License-Identifier: AGPL-3.0-or-later -->
+
+# /pipeline-health — Phase 4: Generate Improvement Proposals
+
+## Phase 4: Generate Improvement Proposals
+
+### 4A: Pattern recurrence analysis — check-promotion threshold <!-- Added: forge#1331 -->
+
+Before the general improvement proposals, scan for `FORGE:PATTERN` tags in review-finding issues closed in the analysis window. When the same pattern slug appears on 3+ distinct issues, emit a check-promotion issue:
+
+```bash
+# Collect all review-finding issues closed in the window with FORGE:PATTERN annotations
+# {GH_FLAG} and {GH_REPO} are resolved from forge.yaml (see Config Resolution at top of file)
+PATTERN_ISSUES=$(gh issue list {GH_FLAG} \
+  --label "review-finding" \
+  --state closed \
+  --search "FORGE:PATTERN" \
+  --limit 100 \
+  --json number,title,body,closedAt \
+  --jq ".[] | select(.closedAt >= \"$SINCE\") | {number: .number, title: .title, body: .body}" 2>/dev/null || echo "[]")
+
+# Extract pattern slugs from FORGE:PATTERN HTML comments in issue bodies
+# Format in issue body: <!-- FORGE:PATTERN: {slug} -->
+PATTERN_LIST=$(echo "$PATTERN_ISSUES" | jq -r '
+  .body |
+  scan("<!-- FORGE:PATTERN: ([^\\s>]+) -->") |
+  .[0]
+' 2>/dev/null | sort | uniq -c | sort -rn)
+
+# For each pattern that appears 3+ times, emit a check-promotion issue
+while IFS= read -r line; do
+  COUNT=$(echo "$line" | awk '{print $1}')
+  SLUG=$(echo "$line" | awk '{print $2}')
+  [ -z "$SLUG" ] && continue
+  [ "$COUNT" -lt 3 ] && continue
+
+  # Check if a promotion issue already exists for this slug (avoid duplicates)
+  EXISTING_PROMOTION=$(gh issue list {GH_FLAG} \
+    --label "check-promotion" \
+    --state open \
+    --search "FORGE:PATTERN: $SLUG" \
+    --limit 5 \
+    --json number --jq '.[0].number' 2>/dev/null || echo "")
+
+  if [ -n "$EXISTING_PROMOTION" ]; then
+    echo "Check-promotion already open for pattern '$SLUG' as #${EXISTING_PROMOTION} — skipping."
+    continue
+  fi
+
+  # Collect the source finding issue numbers for this pattern
+  SOURCE_FINDINGS=$(echo "$PATTERN_ISSUES" | jq -r --arg slug "$SLUG" '
+    select(.body | test("<!-- FORGE:PATTERN: " + $slug + " -->")) |
+    "#\(.number)"
+  ' 2>/dev/null | head -10 | tr '\n' ' ')
+
+  gh issue create {GH_FLAG} \
+    --title "feat(quality-gate): promote pattern '$SLUG' to deterministic registry check (recurred ${COUNT}x)" \
+    --label "check-promotion,priority:P2" \
+    --body "$(cat <<CHECK_EOF
+## Problem
+
+The review-finding pattern \`$SLUG\` has recurred ${COUNT} times in the analysis window ($(date -u +%Y-%m-%d) window). Each recurrence costs a full investigate→build→review→merge cycle. A static check catches this in milliseconds.
+
+## Source Findings
+
+${SOURCE_FINDINGS}
+
+## Task
+
+1. Read the source findings above to understand the defect class mechanically.
+2. Write \`scripts/check-registry/${SLUG}.sh\` following the check contract in \`scripts/check-registry/README.md\`.
+3. Add an entry to \`scripts/check-registry/manifest.json\` referencing the new script.
+4. Verify the check: test it by reintroducing the defect pattern and confirm the gate blocks.
+5. PR, review, merge — the check runs on all subsequent quality-gate invocations.
+
+## Acceptance Criteria
+
+- [ ] \`scripts/check-registry/${SLUG}.sh\` exists, exits 1 on match with correct output format
+- [ ] Manifest entry added: \`{ "slug": "${SLUG}", "script": "...", "source_findings": [...], ... }\`
+- [ ] Seeded repo test: reintroduce the \`$SLUG\` pattern → quality gate blocks with REGISTRY-${SLUG} finding
+- [ ] Gate runtime increase ≤5s for the registered check batch
+- [ ] No regression: a clean changeset (no pattern) → gate passes
+
+## Pattern Tag
+
+<!-- FORGE:PATTERN: ${SLUG} -->
+
+<!-- AUTO-CREATED: pipeline-health Phase 4A — pattern recurrence threshold exceeded -->
+CHECK_EOF
+  )"
+  echo "Created check-promotion issue for pattern '$SLUG' (recurred ${COUNT}x from: $SOURCE_FINDINGS)"
+done <<< "$PATTERN_LIST"
+```
+
+**Recurrence threshold**: 3 occurrences in the analysis window. The threshold is intentionally low — the cost of a false positive is one extra check that rarely fires; the cost of a false negative is another full pipeline cycle for a defect that a grep already knows about.
+
+---
+
+### 4B: General improvement proposals
+
+For each of the top 3 defect categories:
+
+1. **Diagnose**: Why is the builder producing these defects? Read the relevant section of `work-on.md` — is there a check for this category? Is it too vague? Missing entirely?
+
+2. **Prescribe**: What specific change to which command file would reduce these defects?
+   - If the builder lacks a check → propose adding one to `work-on.md` or `quality-gate.md`
+   - If the review agent misses things → propose tightening the agent template in `review-pr-agents.md`
+   - If the orchestrator doesn't catch integration failures → propose a new inter-wave check
+
+3. **Scope**: Estimate the change size (lines added/modified) and risk level.
+
+---
+

package/commands/pipeline-health/phase-5-report.md

@@ -0,0 +1,284 @@
+---
+install: extras
+---
+<!-- SPDX-FileCopyrightText: Copyright (c) RapierCraft Studios -->
+<!-- SPDX-License-Identifier: AGPL-3.0-or-later -->
+
+# /pipeline-health — Phase 5: Report & Track
+
+## Phase 5: Report & Track
+
+### 5A: Post health report
+
+Create a detailed health report as a GitHub issue in the Forge repo:
+
+```bash
+gh issue create -R $FORGE_REPO \
+  --title "Pipeline Health: $REPO — $(date +%Y-%m-%d) — Score: [SCORE]/100" \
+  --label "health-report" \
+  --body "$(cat <<'EOF'
+## Pipeline Health Report
+
+**Project**: $REPO
+**Period**: $SINCE → $(date +%Y-%m-%dT%H:%M:%SZ)
+**Health Score**: [SCORE]/100 ([CATEGORY]) $SCORE_TREND
+
+## Metrics
+
+### Pre-Merge Quality
+
+| Metric | Value | Target | Status |
+|--------|-------|--------|--------|
+| Build pass rate (staging→main) | ?% | 95%+ | ✅/⚠️/❌ |
+| Review findings per PR | ? | < 0.5 | ✅/⚠️/❌ |
+| Manual fix-up rate | ?% | < 15% | ✅/⚠️/❌ |
+| False positive rate | ?% | < 10% | ✅/⚠️/❌ |
+| Pipeline self-correction rate | ?% | < 15% | ✅/⚠️/❌ |
+
+### Post-Deploy Quality
+
+| Metric | Value | Target | Status |
+|--------|-------|--------|--------|
+| Post-deploy failure rate | ?% | < 5% | ✅/⚠️/❌ |
+| Review escape rate (REVIEW_FALSE_NEG) | ?% | < 2% | ✅/⚠️/❌ |
+| Mean time to detect | ? days | < 7 days | ✅/⚠️/❌ |
+| Audit findings (total in window) | ? | — | ℹ️ |
+| — by failure point | see breakdown | — | ℹ️ |
+
+### Pipeline Health Signals
+
+| Metric | Value | Target | Status |
+|--------|-------|--------|--------|
+| Issue quality score | ?/100 | > 80 | ✅/⚠️/❌ |
+| — Investigation invalidation rate | ?% | < 15% | ✅/⚠️/❌ |
+| — ISSUE_SPEC audit findings | ? | 0 | ✅/⚠️/❌ |
+| — Re-investigation rate | ? | 0 | ✅/⚠️/❌ |
+| — Issue structure compliance | ?% | > 80% | ✅/⚠️/❌ |
+| Investigation accuracy (trajectory) | ?% | > 85% | ✅/⚠️/❌ |
+| Orchestration efficiency score | ?/100 | > 70 | ✅/⚠️/❌ |
+| — Avg agent idle% | ?% | < 30% | ✅/⚠️/❌ |
+| — Avg resume cycles per agent | ? | < 1 | ✅/⚠️/❌ |
+| — Clean agent rate | ?% | > 60% | ✅/⚠️/❌ |
+| — Batch stall rate | ?% (?/? issues) | < 10% | ✅/⚠️/❌ |
+| — Escalated to needs-human | ? issues | 0 | ✅/⚠️/❌ |
+| Original work ratio | ?% | > 70% | ✅/⚠️/❌ |
+| Milestone merge success | ?% | 80%+ | ✅/⚠️/❌ |
+| Issue close velocity | ? days | < 2 days | ✅/⚠️/❌ |
+| Transcript health score (Phase 2N/2Q) | ?/100 $(_trend $TRANSCRIPT_HEALTH_SCORE $PRIOR_TRANSCRIPT_SCORE) | > 70 | ✅/⚠️/❌/ℹ️ |
+| — Session flow flag | [✅ Healthy / ⚠️ flag / ℹ️ unavailable] | — | ℹ️ |
+| — Aggregated session metrics (Phase 2Q) | [✅ Available / ℹ️ unavailable] | — | ℹ️ |
+| Avg cost per issue (FORGE:DECISION_RECORD) | $? / ℹ️ unavailable | — | ℹ️ |
+
+## Trajectory Analytics
+
+_From Phase 2I — FORGE:TRAJECTORY comments on [N] closed issues in window_
+
+| Metric | Value |
+|--------|-------|
+| Issues with trajectory data | ? / ? closed |
+| Investigation accuracy | ?% (CONFIRMED: ?, PARTIAL: ?, INVALID: ?) |
+| Task type distribution | Bug Fix: ?, Feature: ?, Refactor: ?, Maintenance: ? |
+| Anomalies flagged | ? issues |
+| Notable anomalies | [list top 3 or "None"] |
+
+_If no trajectory data available in window: "No FORGE:TRAJECTORY comments found — pipeline may be running pre-trajectory version or window is too narrow."_
+
+## Orchestration Efficiency
+
+_From Phase 2K — persisted FORGE:AUDIT-AGENTS summaries ([N] sessions, [M] agents total) + FORGE:STALL_DETECTED annotations_
+
+| Metric | Value | Target | Status |
+|--------|-------|--------|--------|
+| Avg agent idle% | ?% | < 30% | ✅/⚠️/❌ |
+| Avg resume cycles per agent | ? | < 1 | ✅/⚠️/❌ |
+| Clean agent rate | ?% | > 60% | ✅/⚠️/❌ |
+| Batch stall rate | ?% (?/? issues) | < 10% | ✅/⚠️/❌ |
+| Escalated to needs-human | ? issues | 0 | ✅/⚠️/❌ |
+
+**Top stall boundaries** (phase transitions causing the most stalls):
+| Boundary | Occurrences |
+|----------|-------------|
+| [boundary_1] | [N] |
+| [boundary_2] | [N] |
+| [boundary_3] | [N] |
+
+_If no FORGE:AUDIT-AGENTS data available: "No persisted audit-agents summaries found in window. Run \`audit-agents --persist\` after orchestration runs to enable this metric. Proxy: [N] issues had resume/stall anomalies in trajectory data."_
+
+_Batch stall rate: sourced from \`FORGE:STALL_DETECTED\` comments (requires \`/orchestrate\` Step 4B.5 to be active). N/A if stall detector is not deployed._
+
+## Transcript Flow Metrics
+
+_From Phase 2N — JSONL transcript analytics ([N] pipeline sessions, [M] subagent sessions in window)_
+
+| Metric | Value | Signal |
+|--------|-------|--------|
+| Pipeline sessions parsed | ? parent, ? subagent | ℹ️ |
+| Total turns with timing | ? | ℹ️ |
+| Avg turn duration | ?s | ℹ️ |
+| Max turn duration | ?s | ℹ️ |
+| Stall turns (>120s) | ? (?%) | ✅/⚠️ |
+| Rate-limit indicator turns (60–120s) | ? | ℹ️ |
+| Context compaction events | ? | ✅/⚠️ |
+| Session flow flag | [✅ Healthy / ⚠️ High stall rate / ⚠️ Frequent compaction] | — |
+
+**Top tools by call frequency**:
+| Tool | Calls |
+|------|-------|
+| [tool_1] | [N] |
+| [tool_2] | [N] |
+| [tool_3] | [N] |
+
+_If transcript data unavailable: "Transcript flow metrics unavailable — Phase 2M found no pipeline sessions in window (SESSION_TOTAL=0). Run `/pipeline-health` on the same machine where Claude Code sessions are stored."_
+
+## Aggregated Session Metrics
+
+_From Phase 2Q — privacy-safe aggregation of [N] pipeline sessions (session IDs excluded; structural data only)_
+
+| Metric | Value (median) | Range [P25–P75] | Context |
+|--------|---------------|-----------------|---------|
+| Active compute time | ?s | [?s–?s] | Agent tool-call time only — excludes human idle. Target: — (baseline) |
+| Context compaction events | ? | [?–?] | Per session. >2 median = prompts/diffs may be oversized |
+| Sonnet / Opus split | ?% / ?% | — | Opus% = rate-limit fallback pressure |
+
+**Tool call distribution** (top 5 tools, all sessions combined):
+| Tool | Total Calls |
+|------|-------------|
+| [tool_1] | [N] |
+| [tool_2] | [N] |
+| [tool_3] | [N] |
+| [tool_4] | [N] |
+| [tool_5] | [N] |
+
+**Active time by command type** (median):
+| Command | Median duration | Range [P25–P75] | Sessions |
+|---------|-----------------|-----------------|----------|
+| [command_1] | ?s | [?s–?s] | ? |
+| [command_2] | ?s | [?s–?s] | ? |
+
+**Interpretation guardrails** (sourced from investigation #347):
+
+> ⚠️ **Guard 1**: Duration shown is *agent compute time*, not wall-clock. Wall-clock varies 2–10x due to human idle. "Sessions are long" ≠ prompts are bad. Compare against the command-type median, not a global average.
+
+> ⚠️ **Guard 2**: Tool call count is normalized per task type. A 15-file feature legitimately requires 200+ calls; a 1-file fix needs 30–50. Flag only if count per files-changed is ≥3× the command-type median.
+
+> ⚠️ **Guard 3**: Skill invocation count reflects which pipeline commands ran — not whether they ran correctly. High skill count without checking workflow:* label progression is a false signal.
+
+> ⚠️ **Guard 4**: Opus% is rate-limit pressure proxy — high opus% means fallback occurred, not that better models were chosen. Remedy: spread sessions across time or reduce prompt token usage.
+
+> ⚠️ **Guard 5**: Compaction events indicate large context windows, not inefficiency. Flag only if median compaction_count > 2 AND correlated with review-finding quality degradation.
+
+_If aggregated metrics unavailable: "Phase 2Q did not produce data — TRANSCRIPT_DATA_AVAILABLE=false or PER_SESSION_METRICS_RAW was empty. Run `/pipeline-health` on the machine where Claude Code sessions are stored."_
+
+## Post-Deploy Failure Analysis
+
+_From Phase 2L — \`audit-finding\` issues in the Forge repo ([N] findings in window)_
+
+| Metric | Value | Target |
+|--------|-------|--------|
+| Post-deploy failure rate | ?% | < 5% |
+| Review escape rate | ?% | < 2% |
+| Mean time to detect | ? days | < 7 days |
+
+**Failure point distribution** ([N] total audit findings):
+| Failure Point | Count | % | Meaning |
+|---------------|-------|---|---------|
+| REVIEW_FALSE_NEG | ? | ?% | Review agent saw area but approved defect |
+| IMPLEMENTATION | ? | ?% | Code written incorrectly |
+| INVESTIGATION | ? | ?% | Wrong verdict or missed scope |
+| DEPLOY_GATE | ? | ?% | Deploy-info missed risk signal |
+| [other] | ? | ?% | — |
+
+_If no audit findings in window: "No \`audit-finding\` issues in window — post-deploy failure rate = 0%."_
+
+## Defect Category Breakdown
+
+_Pre-merge review findings by agent prefix (from Phase 3A-I). Trends computed vs prior report #$PRIOR_NUMBER (or "—" if baseline run)._
+
+| Category | Count | % of Total | Trend vs Prior Period |
+|----------|-------|------------|---------------------|
+| SEC | $COUNT_SEC | $(_pct $COUNT_SEC)% | $TREND_SEC |
+| AUTH | $COUNT_AUTH | $(_pct $COUNT_AUTH)% | $TREND_AUTH |
+| BILL | $COUNT_BILL | $(_pct $COUNT_BILL)% | $TREND_BILL |
+| DB | $COUNT_DB | $(_pct $COUNT_DB)% | $TREND_DB |
+| FE | $COUNT_FE | $(_pct $COUNT_FE)% | $TREND_FE |
+| INFRA | $COUNT_INFRA | $(_pct $COUNT_INFRA)% | $TREND_INFRA |
+| API | $COUNT_API | $(_pct $COUNT_API)% | $TREND_API |
+| CONC | $COUNT_CONC | $(_pct $COUNT_CONC)% | $TREND_CONC |
+| SCRP | $COUNT_SCRP | $(_pct $COUNT_SCRP)% | $TREND_SCRP |
+| ISSUE_SPEC | $COUNT_ISSUE_SPEC | $(_pct $COUNT_ISSUE_SPEC)% | $TREND_ISSUE_SPEC |
+
+_Post-deploy audit findings by failure point — see Phase 3A-II table above._
+
+## Prompt Change Impact
+
+[Analysis from Phase 3B]
+
+## Top 3 Improvement Proposals
+
+### 1. [Category] — [Proposal title]
+**File**: `commands/[file].md`
+**Change**: [Description]
+**Expected impact**: [Metric] should improve by ~[X]%
+**Risk**: [Low/Medium/High]
+
+### 2. ...
+
+### 3. ...
+
+## Forge Commits in Window
+
+[Git log from Phase 2A]
+EOF
+)"
+```
+
+### 5B: Create improvement issues in Forge
+
+For each improvement proposal, create a trackable issue using the `/issue` structure.
+
+**Before creating**: Read the target file (`commands/[file].md`) and verify that the section reference in the issue body is accurate — confirm the section heading and line range exist.
+
+**Title format**: Use conventional commit prefix based on proposal type:
+- Proposed fix for a broken behaviour → `fix([command]): [description]`
+- New capability or enhancement → `feat([command]): [description]`
+- Restructuring without behaviour change → `refactor([command]): [description]`
+
+**Priority label**: Assign based on how far the metric deviates from its target:
+- Metric ≥ 20% below target → `P1`
+- Metric 10–20% below target → `P2`
+- Metric < 10% below target → `P3`
+
+```bash
+gh issue create -R $FORGE_REPO \
+  --title "fix([command]): [description]" \
+  --label "[bug|enhancement|feature],[P1|P2|P3]" \
+  --body "## Problem
+
+[1–3 sentences describing what the pipeline health metric reveals. Include the current value and target value.]
+
+**Source**: Health report #[REPORT_NUMBER] — [metric]: [current value] (target: [target value])
+
+## Root Cause (if known)
+
+\`commands/[file].md\` [section] — [what the section currently does that causes the metric to suffer]
+
+## Affected Files
+
+Files that need changes:
+
+1. \`commands/[file].md\` — [what needs to change in this file]
+
+## Expected Behavior
+
+After the fix, [metric] should reach [target value]. [Describe the concrete behaviour change in the command.]
+
+## Acceptance Criteria
+
+- [ ] [Specific, testable criterion tied to the metric]
+- [ ] [Specific, testable criterion tied to the command behaviour]
+- [ ] No regression in other pipeline-health metrics
+"
+```
+
+---
+

package/commands/pipeline-health/phase-6-summary.md

@@ -0,0 +1,52 @@
+---
+install: extras
+---
+<!-- SPDX-FileCopyrightText: Copyright (c) RapierCraft Studios -->
+<!-- SPDX-License-Identifier: AGPL-3.0-or-later -->
+
+# /pipeline-health — Phase 6: Summary
+
+## Phase 6: Summary
+
+Print to the user:
+
+```
+Forge Pipeline Health — [REPO]
+Score: [SCORE]/100 ([CATEGORY]) $SCORE_TREND
+
+Pre-merge quality:
+  Build pass rate:          [X]% (target 95%)
+  Findings per PR:          [X]  (target < 0.5)
+  False positive rate:      [X]% (target < 10%)
+  Manual fix-up rate:       [X]% (target < 15%)
+  Self-correction rate:     [X]% (target < 15%)
+
+Post-deploy quality:
+  Post-deploy failure rate: [X]% (target < 5%) — [N] audit findings
+  Review escape rate:       [X]% (target < 2%) — [N] REVIEW_FALSE_NEG
+  Mean time to detect:      [X] days (target < 7 days)
+  Top failure points:       [FP1] ([N]), [FP2] ([N]), [FP3] ([N])
+
+Pipeline health signals:
+  Issue quality score:      [X]/100 (target > 80)
+    Invalidation rate:      [X]% (target < 15%)
+    ISSUE_SPEC findings:    [X]  (target 0)
+    Re-investigations:      [X]  (target 0)
+    Structure compliance:   [X]% (target > 80%)
+  Investigation accuracy:   [X]% (target > 85%) — [N] issues with trajectory data
+    CONFIRMED: [N] | PARTIAL: [N] | INVALID: [N]
+  Anomalies flagged:        [N] issues
+  Task type distribution:   Bug Fix: [N], Feature: [N], Refactor: [N], Maintenance: [N]
+  Orchestration efficiency: [X]/100 (target > 70) — [N] sessions, [M] agents
+    Avg idle%:              [X]% (target < 30%)
+    Avg resume cycles:      [X] (target < 1)
+    Clean agent rate:       [X]% (target > 60%)
+    Top stall boundary:     [boundary] ([N] occurrences)
+  Original work ratio:      [X]% (target > 70%)
+  Milestone merge rate:     [X]% (target 80%)
+
+Top defect categories: [CAT1] ([N]), [CAT2] ([N]), [CAT3] ([N])
+
+Proposed improvements: [N] issues created in $FORGE_REPO
+Health report: [ISSUE_URL]
+```