forgedock 1.1.1 → 1.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (171) hide show
  1. package/README.md +53 -8
  2. package/bin/batch-runner.mjs +339 -0
  3. package/bin/demo.mjs +92 -7
  4. package/bin/engine/phases.mjs +20 -0
  5. package/bin/engine/state.mjs +8 -3
  6. package/bin/engine-cli.mjs +177 -5
  7. package/bin/forgedock.mjs +409 -26
  8. package/bin/hooks/interactive-engine.mjs +509 -0
  9. package/bin/hooks/pre-tool-use.mjs +498 -0
  10. package/bin/hooks/subagent-stop-enforce.mjs +197 -0
  11. package/bin/journey.mjs +72 -14
  12. package/bin/registry.mjs +139 -0
  13. package/bin/report.mjs +447 -0
  14. package/bin/runner.mjs +68 -19
  15. package/bin/settings-hook.mjs +235 -1
  16. package/commands/adopt.md +4 -1
  17. package/commands/analytics.md +4 -1
  18. package/commands/audit-agents.md +110 -27
  19. package/commands/audit.md +20 -1
  20. package/commands/autopilot.md +80 -8
  21. package/commands/changelog.md +2 -1
  22. package/commands/ci-audit.md +2 -1
  23. package/commands/cleanup.md +161 -1
  24. package/commands/compat-audit.md +2 -1
  25. package/commands/deploy-info.md +58 -1
  26. package/commands/design-bench.md +5 -1
  27. package/commands/design-render-critique-loop.md +1 -1
  28. package/commands/diagnose.md +2 -1
  29. package/commands/explain.md +2 -1
  30. package/commands/forge-stats.md +121 -2
  31. package/commands/forgedock-init.md +1 -1
  32. package/commands/geo-audit.md +3 -0
  33. package/commands/incident-response.md +2 -1
  34. package/commands/issue.md +23 -2
  35. package/commands/milestone.md +77 -12
  36. package/commands/optimize.md +2 -1
  37. package/commands/orchestrate/config.md +67 -0
  38. package/commands/orchestrate/phase-1-resolve.md +159 -0
  39. package/commands/orchestrate/phase-2-triage.md +126 -0
  40. package/commands/orchestrate/phase-2.5-synthesis.md +156 -0
  41. package/commands/orchestrate/phase-3-dependency.md +591 -0
  42. package/commands/orchestrate/phase-4-execution.md +962 -0
  43. package/commands/orchestrate/phase-5-cleanup.md +47 -0
  44. package/commands/orchestrate/phase-6-report.md +198 -0
  45. package/commands/orchestrate/safety.md +62 -0
  46. package/commands/orchestrate.md +25 -2045
  47. package/commands/pipeline-health/config.md +34 -0
  48. package/commands/pipeline-health/phase-1-context.md +180 -0
  49. package/commands/pipeline-health/phase-2-metrics.md +1729 -0
  50. package/commands/pipeline-health/phase-3-analyze.md +204 -0
  51. package/commands/pipeline-health/phase-4-proposals.md +117 -0
  52. package/commands/pipeline-health/phase-5-report.md +284 -0
  53. package/commands/pipeline-health/phase-6-summary.md +52 -0
  54. package/commands/pipeline-health.md +25 -2401
  55. package/commands/pipeline-resume.md +24 -1
  56. package/commands/pipeline-status.md +65 -0
  57. package/commands/qa-sweep.md +4 -1
  58. package/commands/quality-gate.md +248 -10
  59. package/commands/replay.md +2 -1
  60. package/commands/review-pr-agents/api.md +135 -0
  61. package/commands/review-pr-agents/auth.md +137 -0
  62. package/commands/review-pr-agents/billing.md +101 -0
  63. package/commands/review-pr-agents/concurrency.md +145 -0
  64. package/commands/review-pr-agents/database.md +103 -0
  65. package/commands/review-pr-agents/frontend.md +89 -0
  66. package/commands/review-pr-agents/infra.md +385 -0
  67. package/commands/review-pr-agents/protocols.md +185 -0
  68. package/commands/review-pr-agents/scraper.md +185 -0
  69. package/commands/review-pr-agents/security.md +406 -0
  70. package/commands/review-pr-agents.md +45 -1722
  71. package/commands/review-pr-staging.md +196 -40
  72. package/commands/review-pr.md +591 -52
  73. package/commands/rollback.md +2 -1
  74. package/commands/scope.md +2 -1
  75. package/commands/security-audit.md +2 -1
  76. package/commands/signal-planner.md +623 -0
  77. package/commands/sync-ecosystem.md +1 -0
  78. package/commands/test-gate.md +17 -1
  79. package/commands/validate.md +2 -1
  80. package/commands/work-on/build/architect.md +28 -0
  81. package/commands/work-on/build/context.md +2 -0
  82. package/commands/work-on/build/implement.md +3 -1
  83. package/commands/work-on/build/validate.md +26 -7
  84. package/commands/work-on/build.md +163 -24
  85. package/commands/work-on/close.md +130 -3
  86. package/commands/work-on/decompose.md +19 -1
  87. package/commands/work-on/investigate.md +117 -2
  88. package/commands/work-on/review.md +24 -4
  89. package/commands/work-on-monolithic.md +7 -2
  90. package/commands/work-on.md +202 -4
  91. package/docs/BRAND.md +14 -7
  92. package/docs/CI.md +2 -2
  93. package/docs/CODEX.md +36 -1
  94. package/docs/CONFIG.md +5 -8
  95. package/docs/FORGE-PROTOCOL.md +15 -1064
  96. package/docs/anthropic-devrel-outreach.md +2 -2
  97. package/docs/articles/command-decomposition-token-savings.md +214 -0
  98. package/docs/articles/model-release-playbook.md +131 -0
  99. package/docs/articles/per-repo-adaptive-scripts-token-savings.md +27 -25
  100. package/docs/claude-breakpoints.json +75 -5
  101. package/docs/design/abc-benchmark.md +1 -1
  102. package/docs/design/design-spec-schema.md +1 -1
  103. package/docs/design/divergent-generation.md +1 -1
  104. package/docs/design/eval-gate-runbook.md +178 -0
  105. package/docs/design/show-hn-draft.md +26 -1
  106. package/docs/design/token-cost-baseline-1298.md +59 -0
  107. package/docs/eval/README.md +117 -0
  108. package/docs/eval/corpus/seed-v1.json +6 -0
  109. package/docs/eval/published/claude-sonnet-4-6-2026-07-04.json +33 -0
  110. package/docs/site/annotations-explained.md +24 -0
  111. package/docs/site/command-learning-path.md +51 -14
  112. package/docs/site/command-reference.md +339 -113
  113. package/docs/site/for-companies.md +95 -0
  114. package/docs/site/forge-annotation-protocol.md +12 -256
  115. package/docs/site/getting-started.md +43 -4
  116. package/docs/site/github-is-the-memory.md +113 -0
  117. package/docs/site/how-it-works.md +3 -13
  118. package/docs/site/index.md +9 -1
  119. package/docs/spec/eval-run-result.md +147 -0
  120. package/docs/spec/forge-protocol-v1.md +441 -18
  121. package/docs/spec/label-state-machine.md +55 -0
  122. package/docs/spec/review-protocol.md +158 -0
  123. package/docs/spec-graph-schema.md +28 -0
  124. package/docs/superpowers/specs/2026-07-04-economic-self-governance-design.md +181 -0
  125. package/docs/superpowers/specs/2026-07-04-provenance-trail-design.md +189 -0
  126. package/examples/forgedock-demo/README.md +41 -22
  127. package/examples/forgedock-demo/issues/06-bug-tags-type-validation.md +43 -0
  128. package/examples/forgedock-demo/issues/07-bug-negative-offset-pagination.md +42 -0
  129. package/examples/forgedock-demo/issues/08-bug-patch-mass-assignment.md +49 -0
  130. package/examples/forgedock-demo/issues/09-bug-count-injection.md +47 -0
  131. package/examples/forgedock-demo/issues/10-feature-sort-param.md +39 -0
  132. package/examples/forgedock-demo/issues/11-feature-patch-tags.md +37 -0
  133. package/examples/forgedock-demo/issues/12-feature-bulk-create.md +42 -0
  134. package/examples/forgedock-demo/issues/13-feature-archive-notes.md +46 -0
  135. package/examples/forgedock-demo/issues/14-refactor-extract-title-validation.md +41 -0
  136. package/examples/forgedock-demo/issues/15-refactor-protected-route-flag.md +43 -0
  137. package/examples/forgedock-demo/issues/16-refactor-extract-store.md +45 -0
  138. package/examples/forgedock-demo/issues/17-perf-tags-dedup-set.md +35 -0
  139. package/examples/forgedock-demo/issues/18-perf-remove-redundant-sort.md +37 -0
  140. package/examples/forgedock-demo/issues/19-docs-api-reference-full.md +53 -0
  141. package/examples/forgedock-demo/issues/20-docs-intentional-flaw-convention.md +46 -0
  142. package/examples/forgedock-demo/scripts/smoke.js +18 -1
  143. package/examples/forgedock-demo/src/db.js +36 -11
  144. package/examples/forgedock-demo/src/routes/notes.js +106 -9
  145. package/examples/forgedock-demo/src/server.js +4 -1
  146. package/package.json +3 -1
  147. package/packages/protocol/LICENSE +21 -0
  148. package/packages/protocol/src/cli.js +135 -0
  149. package/packages/protocol/src/emit.js +146 -0
  150. package/packages/protocol/src/index.js +43 -0
  151. package/packages/protocol/src/parse.js +229 -0
  152. package/packages/protocol/src/types.js +171 -0
  153. package/packages/protocol/src/validate.js +122 -0
  154. package/scripts/bench-topology-cost.mjs +372 -0
  155. package/scripts/check-command-docs-drift.sh +198 -0
  156. package/scripts/check-protocol-restatements.sh +176 -0
  157. package/scripts/check-registry/README.md +72 -0
  158. package/scripts/check-registry/manifest.json +6 -0
  159. package/scripts/code-index.sh +454 -0
  160. package/scripts/conformance-check.mjs +168 -0
  161. package/scripts/doctor-pipeline-state.sh +484 -0
  162. package/scripts/eval-gate-baseline.json +17 -0
  163. package/scripts/eval-gate-scorecard.mjs +242 -0
  164. package/scripts/eval-scorecard-aggregator.mjs +252 -0
  165. package/scripts/flaky-quarantine.sh +241 -0
  166. package/scripts/forge-annotation.sh +567 -0
  167. package/scripts/issue-dedup.sh +236 -0
  168. package/scripts/validate-annotation-node.mjs +233 -0
  169. package/scripts/worktree-lifecycle.sh +3 -3
  170. package/templates/devdocs/agent/using-forgedock.md +14 -0
  171. package/docs/site/.vitepress/config.mjs +0 -85
@@ -1,6 +1,7 @@
1
1
  ---
2
2
  description: Self-analysis — measures pipeline performance, correlates with prompt changes, proposes improvements
3
3
  argument-hint: [project repo slug or "all"]
4
+ install: extras
4
5
  ---
5
6
  <!-- SPDX-FileCopyrightText: Copyright (c) RapierCraft Studios -->
6
7
  <!-- SPDX-License-Identifier: AGPL-3.0-or-later -->
@@ -9,2413 +10,36 @@ argument-hint: [project repo slug or "all"]
9
10
 
10
11
  **Input**: $ARGUMENTS
11
12
 
12
- You are the Forge pipeline's self-awareness layer. Your job is to measure how well the pipeline is performing, correlate performance with recent prompt changes, identify weak spots, and propose concrete improvements.
13
+ You are the Forge pipeline's self-awareness layer. Your job is to measure how well the pipeline is performing,
14
+ correlate performance with recent prompt changes, identify weak spots, and propose concrete improvements.
13
15
 
14
- ## Config
16
+ This file is the slim dispatcher. Detailed phase content lives in `commands/pipeline-health/`.
15
17
 
16
- Read project identity from `forge.yaml` before running any phase:
18
+ ## Execution Order
17
19
 
18
- ```bash
19
- CONFIG_FILE="${FORGE_CONFIG:-forge.yaml}"
20
- GH_REPO=$(yq e '.project.owner + "/" + .project.repo' "$CONFIG_FILE")
21
- GH_FLAG="-R $GH_REPO"
22
- # FORGE_REPO: the self-pipeline repo where pipeline-health issues are filed.
23
- # Set project.forge_repo in forge.yaml if your pipeline repo differs from GH_REPO.
24
- # Example: project:
25
- # forge_repo: "my-org/my-forge"
26
- FORGE_REPO=$(yq '.project.forge_repo // ""' "$CONFIG_FILE")
27
- [ -z "$FORGE_REPO" ] && FORGE_REPO="$GH_REPO"
28
- FORGE_HOME=$(yq e '.paths.root' "$CONFIG_FILE")
29
- echo "Forge repo: $FORGE_REPO"
30
- echo "Forge home: $FORGE_HOME"
31
- ```
32
-
33
- **FORGE_HOME**: `$FORGE_HOME` (set from `forge.yaml` → `paths.root`)
34
- **This command is READ-ONLY on the target project.** It creates issues in the Forge repo only.
35
-
36
- ---
37
-
38
- ## Phase 1: Identify Context
39
-
40
- ### 1A: Resolve the target project
41
-
42
- ```bash
43
- # Default to current repo if no argument
44
- if [ -z "$ARGUMENTS" ] || [ "$ARGUMENTS" = "all" ]; then
45
- REPO=$(gh repo view --json nameWithOwner --jq '.nameWithOwner' 2>/dev/null)
46
- else
47
- REPO="$ARGUMENTS"
48
- fi
49
- echo "Target project: $REPO"
50
- ```
51
-
52
- ### 1B: Get the analysis window
53
-
54
- ```bash
55
- # Last pipeline-health run (if any) — from Forge git log
56
- LAST_RUN=$(cd "$FORGE_HOME" && git log --all --oneline --grep="pipeline-health: $REPO" --format="%aI" -1 2>/dev/null || echo "")
57
- if [ -z "$LAST_RUN" ]; then
58
- # First run — analyze last 30 days
59
- SINCE=$(date -d "30 days ago" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -v-30d +%Y-%m-%dT%H:%M:%SZ)
60
- echo "First health check — analyzing last 30 days"
61
- else
62
- SINCE="$LAST_RUN"
63
- echo "Last health check: $LAST_RUN — analyzing since then"
64
- fi
65
- ```
66
-
67
- ### 1C: Load Prior Health Report
68
-
69
- Fetch the most recent `health-report` issue for the same repo and parse its metrics for trend computation. This must run before Phase 2 so that PRIOR_* variables are available when Phase 3A builds the trend column.
70
-
71
- ```bash
72
- # Find the most recent health-report issue for this repo
73
- PRIOR_REPORT=$(gh issue list -R $FORGE_REPO \
74
- --state all --label "health-report" --limit 10 \
75
- --json number,title,body \
76
- --jq "[.[] | select(.title | contains(\"$REPO\"))] | sort_by(.number) | last")
77
-
78
- if [ -z "$PRIOR_REPORT" ] || [ "$PRIOR_REPORT" = "null" ]; then
79
- echo "No prior health report found for $REPO — this is a baseline run. All trends will show '—'."
80
- PRIOR_HEALTH_SCORE=""
81
- PRIOR_BUILD_PASS_RATE=""
82
- PRIOR_FINDINGS_PER_PR=""
83
- PRIOR_FALSE_POSITIVE_RATE=""
84
- PRIOR_FIXUP_RATE=""
85
- PRIOR_SELF_CORRECTION_RATE=""
86
- PRIOR_ORIGINAL_WORK_RATIO=""
87
- PRIOR_MILESTONE_MERGE_RATE=""
88
- PRIOR_POST_DEPLOY_FAILURE_RATE=""
89
- PRIOR_REVIEW_ESCAPE_RATE=""
90
- PRIOR_SEC=0; PRIOR_AUTH=0; PRIOR_BILL=0; PRIOR_DB=0; PRIOR_FE=0
91
- PRIOR_INFRA=0; PRIOR_API=0; PRIOR_CONC=0; PRIOR_SCRP=0; PRIOR_ISSUE_SPEC=0
92
- PRIOR_TRANSCRIPT_SCORE=""
93
- PRIOR_SESSION_COUNT=""
94
- PRIOR_COMPACTION_MEDIAN=""
95
- else
96
- PRIOR_BODY=$(echo "$PRIOR_REPORT" | jq -r '.body')
97
- PRIOR_NUMBER=$(echo "$PRIOR_REPORT" | jq -r '.number')
98
- echo "Prior health report: #$PRIOR_NUMBER — parsing metrics for trend computation"
99
-
100
- # Parse health score: matches "**Health Score**: 78/100" or "Score: 78/100"
101
- PRIOR_HEALTH_SCORE=$(echo "$PRIOR_BODY" | grep -oP '(?<=\*\*Health Score\*\*: )\d+(?=/100)' | head -1)
102
- [ -z "$PRIOR_HEALTH_SCORE" ] && PRIOR_HEALTH_SCORE=$(echo "$PRIOR_BODY" | grep -oP '(?<=Score: )\d+(?=/100)' | head -1)
103
-
104
- # Parse metric table rows — format: "| Metric name | VALUE | target | status |"
105
- # Build pass rate
106
- PRIOR_BUILD_PASS_RATE=$(echo "$PRIOR_BODY" | grep -i "Build pass rate" | grep -oP '\d+(?=%)' | head -1)
107
- # Review findings per PR
108
- PRIOR_FINDINGS_PER_PR=$(echo "$PRIOR_BODY" | grep -i "Review findings per PR" | grep -oP '[0-9]+\.?[0-9]*' | sed -n '2p')
109
- # False positive rate
110
- PRIOR_FALSE_POSITIVE_RATE=$(echo "$PRIOR_BODY" | grep -i "False positive rate" | grep -oP '\d+(?=%)' | head -1)
111
- # Manual fix-up rate
112
- PRIOR_FIXUP_RATE=$(echo "$PRIOR_BODY" | grep -i "Manual fix-up rate" | grep -oP '\d+(?=%)' | head -1)
113
- # Pipeline self-correction rate
114
- PRIOR_SELF_CORRECTION_RATE=$(echo "$PRIOR_BODY" | grep -i "self-correction rate" | grep -oP '\d+(?=%)' | head -1)
115
- # Original work ratio
116
- PRIOR_ORIGINAL_WORK_RATIO=$(echo "$PRIOR_BODY" | grep -i "Original work ratio" | grep -oP '\d+(?=%)' | head -1)
117
- # Milestone merge success
118
- PRIOR_MILESTONE_MERGE_RATE=$(echo "$PRIOR_BODY" | grep -i "Milestone merge success" | grep -oP '\d+(?=%)' | head -1)
119
- # Post-deploy failure rate
120
- PRIOR_POST_DEPLOY_FAILURE_RATE=$(echo "$PRIOR_BODY" | grep -i "Post-deploy failure rate" | grep -oP '\d+(?=%)' | head -1)
121
- # Review escape rate
122
- PRIOR_REVIEW_ESCAPE_RATE=$(echo "$PRIOR_BODY" | grep -i "Review escape rate" | grep -oP '\d+(?=%)' | head -1)
123
- # Transcript health score (from Phase 3C — supplementary signal)
124
- PRIOR_TRANSCRIPT_SCORE=$(echo "$PRIOR_BODY" | grep -i "Transcript health score" | grep -oP '\d+(?=/100)' | head -1)
125
- # Pipeline sessions in window (from Transcript Flow Metrics section)
126
- PRIOR_SESSION_COUNT=$(echo "$PRIOR_BODY" | grep -i "Pipeline sessions parsed" | grep -oP '^\d+' | head -1)
127
- # Compaction events median (from Aggregated Session Metrics table)
128
- PRIOR_COMPACTION_MEDIAN=$(echo "$PRIOR_BODY" | grep -i "Context compaction events" | grep -oP '[0-9]+\.?[0-9]*' | head -1)
129
-
130
- # Parse defect category counts from the "Defect Category Breakdown" section
131
- # Matches table rows like "| SEC | 4 | 22% | ↑ +1 |" or "| SEC | 4 | 22% | — |"
132
- _parse_cat() {
133
- echo "$PRIOR_BODY" | grep -P "^\| $1 \|" | grep -oP '(?<=\| )\d+(?= \|)' | head -1
134
- }
135
- PRIOR_SEC=$( _parse_cat "SEC"); [ -z "$PRIOR_SEC" ] && PRIOR_SEC=0
136
- PRIOR_AUTH=$( _parse_cat "AUTH"); [ -z "$PRIOR_AUTH" ] && PRIOR_AUTH=0
137
- PRIOR_BILL=$( _parse_cat "BILL"); [ -z "$PRIOR_BILL" ] && PRIOR_BILL=0
138
- PRIOR_DB=$( _parse_cat "DB"); [ -z "$PRIOR_DB" ] && PRIOR_DB=0
139
- PRIOR_FE=$( _parse_cat "FE"); [ -z "$PRIOR_FE" ] && PRIOR_FE=0
140
- PRIOR_INFRA=$( _parse_cat "INFRA"); [ -z "$PRIOR_INFRA" ] && PRIOR_INFRA=0
141
- PRIOR_API=$( _parse_cat "API"); [ -z "$PRIOR_API" ] && PRIOR_API=0
142
- PRIOR_CONC=$( _parse_cat "CONC"); [ -z "$PRIOR_CONC" ] && PRIOR_CONC=0
143
- PRIOR_SCRP=$( _parse_cat "SCRP"); [ -z "$PRIOR_SCRP" ] && PRIOR_SCRP=0
144
- PRIOR_ISSUE_SPEC=$(_parse_cat "ISSUE_SPEC"); [ -z "$PRIOR_ISSUE_SPEC" ] && PRIOR_ISSUE_SPEC=0
145
-
146
- echo "Prior health score: ${PRIOR_HEALTH_SCORE:-N/A}"
147
- echo "Prior defect counts: SEC=$PRIOR_SEC AUTH=$PRIOR_AUTH BILL=$PRIOR_BILL DB=$PRIOR_DB FE=$PRIOR_FE INFRA=$PRIOR_INFRA API=$PRIOR_API CONC=$PRIOR_CONC SCRP=$PRIOR_SCRP ISSUE_SPEC=$PRIOR_ISSUE_SPEC"
148
- echo "Prior transcript: score=${PRIOR_TRANSCRIPT_SCORE:-N/A} sessions=${PRIOR_SESSION_COUNT:-N/A} compaction_median=${PRIOR_COMPACTION_MEDIAN:-N/A}"
149
- fi
150
-
151
- # Helper function: compute trend string for a numeric category count
152
- # Usage: _trend CURRENT PRIOR → outputs "↑ +N", "↓ -N", "=", or "—"
153
- _trend() {
154
- local current="$1" prior="$2"
155
- if [ -z "$prior" ] || [ "$prior" = "0" ] && [ "$current" = "0" ]; then
156
- echo "—"
157
- elif [ -z "$prior" ]; then
158
- echo "—"
159
- elif [ "$current" -gt "$prior" ] 2>/dev/null; then
160
- echo "↑ +$(( current - prior ))"
161
- elif [ "$current" -lt "$prior" ] 2>/dev/null; then
162
- echo "↓ -$(( prior - current ))"
163
- else
164
- echo "="
165
- fi
166
- }
167
- ```
168
-
169
- ---
170
-
171
- ## Phase 2: Collect Pipeline Metrics
172
-
173
- Run ALL of these in parallel (note: Phase 2I fetches trajectory comments per issue — run it concurrently with the others but expect it to take longer for large windows; Phase 2K queries persisted audit-agents summaries from Forge's tracking issue and is independent of the target project; Phase 2L queries `audit-finding` labeled issues in the Forge repo and is independent of the target project; Phase 2M scans `~/.claude/history.jsonl` and maps session IDs to JSONL files on disk — it is independent of the target project and completes in <5s for 25K entries; Phase 2N reads local JSONL transcript files using the JSONL Parser Utility and aggregates pipeline flow metrics — it requires Phase 2M to have run first and depends on `SESSION_JSONL_FILES`; Phase 2O extracts per-session structured metrics (phase detection, model distribution, tool call counts, agent spawns) from the same JSONL files — it requires Phase 2N to have run first and reads `TRANSCRIPT_DATA_AVAILABLE` set by Phase 2N; Phase 2P analyzes subagent JSONL files per session, maps each agent to its pipeline role, and computes per-role efficiency metrics — it requires Phase 2O to have run and depends on `TRANSCRIPT_DATA_AVAILABLE` and `SESSION_JSONL_FILES`; Phase 2Q aggregates per-session metrics from Phase 2O into a privacy-safe, report-ready summary — it strips session IDs and content fields, computes distributions, and emits guardrail footnotes — it requires Phase 2O to have run and reads `PER_SESSION_METRICS_RAW`):
174
-
175
- ### 2A: Forge changelog (what changed in the pipeline itself)
176
-
177
- ```bash
178
- cd "$FORGE_HOME"
179
- git log --oneline --since="$SINCE" -- commands/ docs/
180
- ```
181
-
182
- Record each commit — these are the independent variables. Each prompt change is a hypothesis: "this change should improve X."
183
-
184
- ### 2B: Review finding metrics (the primary quality signal)
185
-
186
- ```bash
187
- # Total review findings created in window
188
- gh issue list -R $REPO --state all --label "review-finding" --json number,createdAt,labels,title,state \
189
- --jq "[.[] | select(.createdAt > \"$SINCE\")] | length"
190
-
191
- # Breakdown by severity (from title/labels)
192
- gh issue list -R $REPO --state all --label "review-finding" --limit 500 --json number,title,labels,state,createdAt \
193
- --jq "[.[] | select(.createdAt > \"$SINCE\")]"
194
- ```
195
-
196
- **Categorize each finding** by its agent prefix (SEC, AUTH, BILL, CONC, SCRP, FE, API, DB, INFRA) extracted from the title or finding tags.
197
-
198
- ### 2C: Invalidation rate (false positive signal)
199
-
200
- ```bash
201
- # Review findings that were closed as invalid
202
- gh issue list -R $REPO --state closed --label "review-finding,workflow:invalid" --json number,createdAt \
203
- --jq "[.[] | select(.createdAt > \"$SINCE\")] | length"
204
-
205
- # Total review findings closed in window
206
- gh issue list -R $REPO --state closed --label "review-finding" --json number,createdAt \
207
- --jq "[.[] | select(.createdAt > \"$SINCE\")] | length"
208
- ```
209
-
210
- **False positive rate** = invalid / total closed. Target: < 10%.
211
-
212
- ### 2D: Build failure rate
213
-
214
- ```bash
215
- # staging→main PRs and their outcomes
216
- gh pr list -R $REPO --state all --base main --json number,state,mergedAt,createdAt \
217
- --jq "[.[] | select(.createdAt > \"$SINCE\")] | {total: length, merged: [.[] | select(.mergedAt != null)] | length, failed: [.[] | select(.state == \"CLOSED\" and .mergedAt == null)] | length}"
218
-
219
- # milestone→staging PRs and their outcomes
220
- gh pr list -R $REPO --state all --limit 200 --json number,state,mergedAt,createdAt,baseRefName,headRefName \
221
- --jq "[.[] | select(.createdAt > \"$SINCE\" and .baseRefName == \"staging\" and (.headRefName | startswith(\"milestone/\")))] | {total: length, merged: [.[] | select(.mergedAt != null)] | length, failed: [.[] | select(.state == \"CLOSED\" and .mergedAt == null)] | length}"
222
- ```
223
-
224
- ### 2E: Review findings per PR (density metric)
225
-
226
- ```bash
227
- # PRs merged in window
228
- MERGED_COUNT=$(gh pr list -R $REPO --state closed --json mergedAt,createdAt \
229
- --jq "[.[] | select(.mergedAt != null and .createdAt > \"$SINCE\")] | length")
230
-
231
- # Review findings created in window
232
- FINDING_COUNT=$(gh issue list -R $REPO --state all --label "review-finding" --json createdAt \
233
- --jq "[.[] | select(.createdAt > \"$SINCE\")] | length")
234
-
235
- echo "Findings per PR: $(echo "scale=2; $FINDING_COUNT / $MERGED_COUNT" | bc 2>/dev/null || echo 'N/A')"
236
- ```
237
-
238
- Target: < 0.5 findings per PR (meaning most PRs pass clean).
239
-
240
- ### 2F: Issue lifecycle velocity
241
-
242
- ```bash
243
- # Average time from issue creation to close (in days) for non-review-findings
244
- gh issue list -R $REPO --state closed --json createdAt,closedAt,labels \
245
- --jq "[.[] | select(.createdAt > \"$SINCE\" and (.labels | map(.name) | index(\"review-finding\") | not))] | .[].createdAt" | head -20
246
- ```
247
-
248
- ### 2G: Pipeline command usage (which commands are exercised most)
249
-
250
- ```bash
251
- # Count PR titles and issue comments that reference pipeline actions
252
- gh pr list -R $REPO --state all --limit 200 --json title,createdAt \
253
- --jq "[.[] | select(.createdAt > \"$SINCE\")] | [.[].title] | group_by(. | split(\":\" )[0]) | map({prefix: .[0] | split(\":\")[0], count: length}) | sort_by(-.count)"
254
- ```
255
-
256
- ### 2H: Manual fix-up commits (builder completeness signal)
257
-
258
- The pipeline builds a PR, but sometimes a human must intervene on the same branch to patch gaps the builder missed. These fix-up commits are a direct measure of builder completeness.
259
-
260
- **Detection strategy**:
261
-
262
- ```bash
263
- # For each PR merged in the window, get the full commit list with authors
264
- MERGED_PRS=$(gh pr list -R $REPO --state closed --limit 100 --json number,headRefName,mergedAt,createdAt \
265
- --jq "[.[] | select(.mergedAt != null and .createdAt > \"$SINCE\")] | .[].number")
266
-
267
- for PR in $MERGED_PRS; do
268
- # Get all commits on this PR
269
- gh api /repos/$REPO/pulls/$PR/commits \
270
- --jq '.[] | {sha: .sha[0:7], author: .author.login, message: .commit.message | split("\n")[0], date: .commit.author.date}'
271
- done
272
- ```
273
-
274
- **Classification rules** — for each PR's commit list:
275
-
276
- 1. **Pipeline commit**: author is the project bot account, or commit message matches Forge conventional-commit pattern (`Fix:`, `Feat:`, `Refactor:` with issue reference).
277
- 2. **Manual fix-up commit**: author is a human AND the commit appears AFTER the initial pipeline commits on the same PR AND the message indicates a correction (e.g., contains "fix", "patch", "missing", "forgot", "actually", "also need", "oops", or references the same component the pipeline just touched).
278
- 3. **Intentional human commit**: human commit that is clearly additive scope (new feature, unrelated change) — exclude from fix-up count.
279
-
280
- When ambiguous, read the commit message and diff summary to determine if the human was correcting a pipeline gap or adding new scope.
281
-
282
- **Also classify PRs by origin** — for each merged PR, determine if it is:
283
- - **Original work**: primary feature/bug PR from an issue
284
- - **Review-finding fix**: PR that fixes a review-finding issue (references a `review-finding` labeled issue)
285
- - **Squash regression fix**: PR whose title/body indicates restoring code lost during a milestone squash merge
286
-
287
- **Metrics to compute**:
288
- - **Fix-up rate**: PRs with ≥1 manual fix-up commit / total pipeline-built PRs. Target: < 15%.
289
- - **Fix-up density**: total manual fix-up commits / total pipeline-built PRs. Target: < 0.3.
290
- - **Pipeline self-correction rate**: review-finding fix PRs / original work PRs. Target: < 15%.
291
- - **Squash regression rate**: squash regression fix PRs / total PRs. Target: 0%.
292
- - **Original work ratio**: original work PRs / total PRs. Target: > 70%.
293
- - **Fix-up categories**: group by what was missing (config sync, edge case, test gap, incomplete refactor, wrong API usage, UI polish).
294
-
295
- **Why this matters**: Review findings catch defects post-build. Fix-up commits reveal defects the builder produced that were caught informally (by the developer, by testing, or by deploy failure) — these are "silent" quality gaps that review findings don't capture. The PR composition breakdown shows how much pipeline capacity goes to forward progress vs self-correction.
296
-
297
- ### 2I: Trajectory analytics (pipeline completeness signal)
298
-
299
- `FORGE:TRAJECTORY` comments are posted on every completed issue by `/work-on`. They contain the per-issue phase table, investigation verdict, task type, key decisions, and anomalies. Aggregating these reveals investigation accuracy, task type distribution, and anomaly patterns — signals invisible to the PR/issue label metrics above.
300
-
301
- **Note**: This section makes ~1 API call per closed issue in the window (to fetch comments). For windows with 100+ issues this can take 2–3 minutes.
302
-
303
- ```bash
304
- # Collect all issues closed in the window
305
- CLOSED_ISSUES=$(gh issue list -R $REPO --state closed --limit 100 --json number,closedAt \
306
- --jq "[.[] | select(.closedAt > \"$SINCE\")] | .[].number")
307
-
308
- # Accumulators
309
- TRAJ_TOTAL=0
310
- TRAJ_CONFIRMED=0
311
- TRAJ_PARTIAL=0
312
- TRAJ_INVALID=0
313
- declare -A TASK_TYPE_COUNTS
314
- ANOMALY_STRINGS=()
315
-
316
- for NUM in $CLOSED_ISSUES; do
317
- TRAJ=$(gh api repos/$REPO/issues/$NUM/comments \
318
- --jq '.[] | select(.body | contains("FORGE:TRAJECTORY")) | .body' 2>/dev/null)
319
-
320
- # Skip issues with no trajectory comment (pre-pipeline or manually closed)
321
- [ -z "$TRAJ" ] && continue
322
-
323
- TRAJ_TOTAL=$((TRAJ_TOTAL + 1))
324
-
325
- # Parse investigation verdict from the phase table row
326
- # Matches: "| Phase 1: Investigation | ✅ CONFIRMED (HIGH) |" or "✅ PARTIAL (MEDIUM) |" etc.
327
- VERDICT=$(echo "$TRAJ" | grep -oP '(?<=Investigation \| ✅ )(CONFIRMED|PARTIAL|INVALID)' | head -1)
328
- case "$VERDICT" in
329
- CONFIRMED) TRAJ_CONFIRMED=$((TRAJ_CONFIRMED + 1)) ;;
330
- PARTIAL) TRAJ_PARTIAL=$((TRAJ_PARTIAL + 1)) ;;
331
- INVALID) TRAJ_INVALID=$((TRAJ_INVALID + 1)) ;;
332
- esac
333
-
334
- # Parse task type from "Task type: Feature" or "Task type: Bug Fix"
335
- TASK_TYPE=$(echo "$TRAJ" | grep -oP '(?<=Task type: )[^\|]+' | tr -d ' ' | head -1)
336
- if [ -n "$TASK_TYPE" ]; then
337
- TASK_TYPE_COUNTS[$TASK_TYPE]=$(( ${TASK_TYPE_COUNTS[$TASK_TYPE]:-0} + 1 ))
338
- fi
339
-
340
- # Collect non-None anomaly strings
341
- ANOMALY=$(echo "$TRAJ" | grep -oP '(?<=\*\*Anomalies\*\*: )(?!None)[^\n]+' | head -1)
342
- [ -n "$ANOMALY" ] && ANOMALY_STRINGS+=("$NUM: $ANOMALY")
343
- done
344
-
345
- # Compute investigation accuracy rate (CONFIRMED + PARTIAL) / total
346
- TRAJ_ACCURATE=$((TRAJ_CONFIRMED + TRAJ_PARTIAL))
347
- if [ "$TRAJ_TOTAL" -gt 0 ]; then
348
- echo "Trajectory coverage: $TRAJ_TOTAL / $(echo "$CLOSED_ISSUES" | wc -w) closed issues"
349
- echo "Investigation accuracy: $TRAJ_ACCURATE / $TRAJ_TOTAL ($(echo "scale=0; $TRAJ_ACCURATE * 100 / $TRAJ_TOTAL" | bc)%)"
350
- echo " CONFIRMED: $TRAJ_CONFIRMED | PARTIAL: $TRAJ_PARTIAL | INVALID: $TRAJ_INVALID"
351
- echo ""
352
- echo "Task type distribution:"
353
- for TT in "${!TASK_TYPE_COUNTS[@]}"; do
354
- echo " $TT: ${TASK_TYPE_COUNTS[$TT]}"
355
- done
356
- echo ""
357
- echo "Anomalies flagged: ${#ANOMALY_STRINGS[@]}"
358
- for A in "${ANOMALY_STRINGS[@]}"; do echo " - Issue $A"; done
359
- else
360
- echo "No FORGE:TRAJECTORY comments found in window — no trajectory data available"
361
- fi
362
- ```
363
-
364
- **Metrics to compute**:
365
- - **Investigation accuracy rate**: (CONFIRMED + PARTIAL) / total issues with trajectory. Target: > 85%.
366
- - **INVALID rate**: INVALID / total. Target: < 15% (same as Phase 2J invalidation rate — these should converge).
367
- - **Task type distribution**: counts of Bug Fix / Feature / Refactor / Maintenance. Reveals whether the pipeline is doing forward-progress work or mostly self-correction.
368
- - **Anomaly count**: total issues with non-None anomaly strings. Target: < 10% of window. High anomaly rates indicate pipeline instability.
369
-
370
- **Why this matters**: Labels tell you what issues are in what state. Trajectory comments tell you *why* — what the investigator concluded, what anomalies were hit, and what decisions were made. Investigation accuracy is the deepest signal for whether the investigation phase is operating correctly. Task type distribution shows whether the pipeline is delivering forward-progress work (Feature/Bug Fix) vs. churn (Refactor/Maintenance cycles).
371
-
372
- ### 2J: Issue quality metrics (pipeline entry-point signal)
373
-
374
- Bad issues — vague descriptions, missing files, wrong domains — cause cascading failures: wasted investigation, incomplete fixes, and audit findings classified as `ISSUE_SPEC`. This section measures how healthy the pipeline's entry point is.
375
-
376
- ```bash
377
- # 1. Investigation invalidation rate
378
- # Issues closed as workflow:invalid (excluding review-findings — those are tracked in 2C)
379
- INVALID_ISSUES=$(gh issue list -R $REPO --state closed --label "workflow:invalid" --limit 500 --json number,createdAt,labels \
380
- --jq "[.[] | select(.createdAt > \"$SINCE\" and (.labels | map(.name) | index(\"review-finding\") | not))] | length")
381
-
382
- TOTAL_INVESTIGATED=$(gh issue list -R $REPO --state all --limit 500 --json number,createdAt,labels \
383
- --jq "[.[] | select(.createdAt > \"$SINCE\" and (.labels | map(.name) | any(. == \"workflow:investigating\" or . == \"workflow:building\" or . == \"workflow:in-review\" or . == \"workflow:merged\" or . == \"workflow:invalid\"))) ] | length")
384
-
385
- echo "Investigation invalidation rate: $INVALID_ISSUES / $TOTAL_INVESTIGATED"
386
- # Target: < 15%
387
-
388
- # 2. ISSUE_SPEC audit findings
389
- # Count of /audit findings classified as ISSUE_SPEC in the window
390
- ISSUE_SPEC_COUNT=$(gh issue list -R $FORGE_REPO --state all --label "audit-finding" --limit 200 \
391
- --json number,title,body,createdAt \
392
- --jq "[.[] | select(.createdAt > \"$SINCE\" and (.body | contains(\"ISSUE_SPEC\")))] | length")
393
-
394
- echo "ISSUE_SPEC audit findings: $ISSUE_SPEC_COUNT"
395
- # Target: 0. Any ISSUE_SPEC finding means /issue adoption is insufficient or issue quality is degraded.
396
-
397
- # 3. Re-investigation rate
398
- # Issues where FORGE:INVESTIGATOR comment was deleted and recreated (interrupted investigation → restart)
399
- # Proxy: issues that were labeled workflow:investigating more than once (check via timeline events)
400
- REINVESTIGATED=$(gh issue list -R $REPO --state all --limit 500 --json number,createdAt \
401
- --jq "[.[] | select(.createdAt > \"$SINCE\")] | .[].number" | while read NUM; do
402
- # Check if issue has timeline events showing workflow:investigating label added 2+ times
403
- INVEST_EVENTS=$(gh api repos/$REPO/issues/$NUM/timeline \
404
- --jq '[.[] | select(.event == "labeled" and .label.name == "workflow:investigating")] | length' 2>/dev/null || echo 0)
405
- if [ "$INVEST_EVENTS" -gt 1 ] 2>/dev/null; then echo "$NUM"; fi
406
- done | wc -l)
407
-
408
- echo "Re-investigation count: $REINVESTIGATED"
409
- # Target: 0. Each re-investigation signals the original issue spec was unclear or wrong.
410
-
411
- # 4. Issue structure compliance (/issue adoption proxy)
412
- # Issues created in window that have an "Affected Files" section in their body
413
- TOTAL_ISSUES=$(gh issue list -R $REPO --state all --limit 500 --json number,createdAt,labels \
414
- --jq "[.[] | select(.createdAt > \"$SINCE\" and (.labels | map(.name) | index(\"review-finding\") | not))] | length")
415
-
416
- STRUCTURED_ISSUES=$(gh issue list -R $REPO --state all --limit 500 --json number,createdAt,body,labels \
417
- --jq "[.[] | select(.createdAt > \"$SINCE\" and (.labels | map(.name) | index(\"review-finding\") | not) and (.body | contains(\"Affected Files\")))] | length")
418
-
419
- echo "Issue structure compliance: $STRUCTURED_ISSUES / $TOTAL_ISSUES"
420
- # Target: > 80%. Issues without Affected Files are harder to investigate and more likely to produce ISSUE_SPEC failures.
421
- ```
422
-
423
- **Issue quality score** = weighted average of sub-metrics:
424
- - Invalidation rate: 40% weight (direct signal — these issues failed)
425
- - ISSUE_SPEC audit findings: 30% weight (downstream cost signal)
426
- - Re-investigation rate: 20% weight (process cost signal)
427
- - Structure compliance: 10% weight (adoption/leading indicator)
428
-
429
- **Why this matters**: The investigation phase can only be as good as the issue it receives. A 20% invalidation rate means 1 in 5 investigations was a complete waste. ISSUE_SPEC audit findings are the downstream cost of bad issues reaching production. Tracking these signals closes the loop between issue creation quality and pipeline output quality.
430
-
431
- ### 2K: Orchestration efficiency (agent throughput signal)
432
-
433
- `/audit-agents` measures per-run agent efficiency (idle%, resume cycles, stall boundaries) but produces ephemeral output. This section collects persisted `<!-- FORGE:AUDIT-AGENTS -->` summary comments (posted by `audit-agents --persist`) from the Forge tracking issue and aggregates them into pipeline-wide efficiency metrics. If no persisted data exists, it falls back to mining resume-cycle anomalies from FORGE:TRAJECTORY comments collected in Phase 2I.
434
-
435
- **Step 1 — Query persisted audit-agents summaries**:
436
-
437
- ```bash
438
- # Persisted audit-agents summaries are posted to a designated Forge tracking issue
439
- # tagged with label "orchestration-metrics". Find it:
440
- TRACKING_ISSUE=$(gh issue list -R $FORGE_REPO \
441
- --state open --label "orchestration-metrics" --limit 1 \
442
- --json number --jq '.[0].number' 2>/dev/null)
443
-
444
- if [ -n "$TRACKING_ISSUE" ]; then
445
- # Fetch all FORGE:AUDIT-AGENTS comments posted since $SINCE
446
- AUDIT_COMMENTS=$(gh api repos/$FORGE_REPO/issues/$TRACKING_ISSUE/comments \
447
- --jq "[.[] | select(.body | contains(\"FORGE:AUDIT-AGENTS\")) | select(.created_at > \"$SINCE\")]")
448
- echo "Found $(echo "$AUDIT_COMMENTS" | jq 'length') persisted audit-agents summaries in window"
449
- else
450
- echo "No orchestration-metrics tracking issue found — skipping persisted data query"
451
- AUDIT_COMMENTS="[]"
452
- fi
453
- ```
454
-
455
- **Step 2 — Extract metrics from persisted summaries**:
456
-
457
- Each `<!-- FORGE:AUDIT-AGENTS -->` comment contains a structured block (posted by `audit-agents --persist`) with these fields:
458
-
459
- ```
460
- Session: {SESSION_ID}
461
- Date: {ISO_TIMESTAMP}
462
- Agents: {TOTAL_AGENT_COUNT}
463
- Avg idle%: {VALUE}
464
- Avg resumes: {VALUE}
465
- Clean agents: {N}/{TOTAL}
466
- Stall boundaries: {boundary_1}({count}), {boundary_2}({count}), ...
467
- ```
468
-
469
- ```bash
470
- # Parse each comment and accumulate metrics
471
- ORCH_SESSIONS=0
472
- ORCH_TOTAL_IDLE=0
473
- ORCH_TOTAL_RESUMES=0
474
- ORCH_CLEAN_AGENTS=0
475
- ORCH_TOTAL_AGENTS=0
476
- declare -A STALL_BOUNDARY_COUNTS
477
-
478
- echo "$AUDIT_COMMENTS" | jq -r '.[] | .body + "\u0000"' | while IFS= read -r -d $'\0' COMMENT; do
479
- ORCH_SESSIONS=$((ORCH_SESSIONS + 1))
480
-
481
- AVG_IDLE=$(echo "$COMMENT" | grep -oP '(?<=Avg idle%: )[0-9.]+' | head -1)
482
- AVG_RESUMES=$(echo "$COMMENT" | grep -oP '(?<=Avg resumes: )[0-9.]+' | head -1)
483
- CLEAN=$(echo "$COMMENT" | grep -oP '(?<=Clean agents: )\d+' | head -1)
484
- TOTAL=$(echo "$COMMENT" | grep -oP '(?<=Clean agents: \d{1,3}\/)\d+' | head -1)
485
-
486
- [ -n "$AVG_IDLE" ] && ORCH_TOTAL_IDLE=$(echo "$ORCH_TOTAL_IDLE + $AVG_IDLE" | bc)
487
- [ -n "$AVG_RESUMES" ] && ORCH_TOTAL_RESUMES=$(echo "$ORCH_TOTAL_RESUMES + $AVG_RESUMES" | bc)
488
- [ -n "$CLEAN" ] && ORCH_CLEAN_AGENTS=$((ORCH_CLEAN_AGENTS + CLEAN))
489
- [ -n "$TOTAL" ] && ORCH_TOTAL_AGENTS=$((ORCH_TOTAL_AGENTS + TOTAL))
490
-
491
- # Accumulate stall boundary counts
492
- BOUNDARIES=$(echo "$COMMENT" | grep -oP '(?<=Stall boundaries: )[^\n]+' | head -1)
493
- # Format: "investigate→build(4), context→architect(3)"
494
- echo "$BOUNDARIES" | grep -oP '[a-z→]+\(\d+\)' | while IFS= read -r ENTRY; do
495
- BOUNDARY=$(echo "$ENTRY" | grep -oP '^[a-z→]+')
496
- COUNT=$(echo "$ENTRY" | grep -oP '\d+')
497
- STALL_BOUNDARY_COUNTS[$BOUNDARY]=$(( ${STALL_BOUNDARY_COUNTS[$BOUNDARY]:-0} + COUNT ))
498
- done
499
- done
500
-
501
- # Compute aggregate metrics
502
- if [ "$ORCH_SESSIONS" -gt 0 ]; then
503
- AVG_IDLE_OVERALL=$(echo "scale=1; $ORCH_TOTAL_IDLE / $ORCH_SESSIONS" | bc)
504
- AVG_RESUMES_OVERALL=$(echo "scale=2; $ORCH_TOTAL_RESUMES / $ORCH_SESSIONS" | bc)
505
- CLEAN_RATE=$(echo "scale=0; $ORCH_CLEAN_AGENTS * 100 / $ORCH_TOTAL_AGENTS" | bc 2>/dev/null || echo "N/A")
506
- echo "Orchestration efficiency ($ORCH_SESSIONS sessions, $ORCH_TOTAL_AGENTS agents):"
507
- echo " Avg idle%: $AVG_IDLE_OVERALL% (target: < 30%)"
508
- echo " Avg resumes per agent: $AVG_RESUMES_OVERALL (target: < 1)"
509
- echo " Clean agent rate: $CLEAN_RATE% (target: > 60%)"
510
- echo " Top stall boundaries:"
511
- for B in "${!STALL_BOUNDARY_COUNTS[@]}"; do
512
- echo " $B: ${STALL_BOUNDARY_COUNTS[$B]} occurrences"
513
- done | sort -t: -k2 -rn | head -5
514
- ORCH_DATA_AVAILABLE=true
515
- else
516
- echo "No persisted audit-agents data in window"
517
- ORCH_DATA_AVAILABLE=false
518
- fi
519
- ```
520
-
521
- **Step 3 — Fallback: mine FORGE:TRAJECTORY for resume-cycle signals**:
522
-
523
- If `$ORCH_DATA_AVAILABLE` is false (no persisted summaries), use the trajectory data already collected in Phase 2I as a proxy. FORGE:TRAJECTORY `**Anomalies**` strings from `/work-on` often include resume cycle counts and stall descriptions.
524
-
525
- ```bash
526
- if [ "$ORCH_DATA_AVAILABLE" = "false" ]; then
527
- echo "Falling back to FORGE:TRAJECTORY anomaly mining for orchestration proxy signals"
528
- # Count how many trajectory issues had anomalies mentioning "resume" or "stall"
529
- RESUME_ANOMALIES=0
530
- for A in "${ANOMALY_STRINGS[@]}"; do
531
- echo "$A" | grep -qi 'resume\|stall\|end.turn\|routing' && RESUME_ANOMALIES=$((RESUME_ANOMALIES + 1))
532
- done
533
- echo "Issues with resume/stall anomalies in trajectory: $RESUME_ANOMALIES / $TRAJ_TOTAL"
534
- echo "(Proxy only — run 'audit-agents --persist' after orchestration runs for full efficiency data)"
535
- fi
536
- ```
537
-
538
- **Metrics to compute**:
539
- - **Avg agent idle%**: average idle time percentage across all agents in all sessions. Target: < 30%.
540
- - **Avg resume cycles per agent**: how often orchestrator must resume stalled agents. Target: < 1.
541
- - **Clean agent rate**: % of agents that complete without any stalls. Target: > 60%.
542
- - **Top stall boundaries**: which phase transitions cause the most stalls (e.g., investigate→build, context→architect). No target — use for diagnosis.
543
-
544
- **Why this matters**: Outcome metrics (findings per PR, fix-up rate) measure what the pipeline produces. Orchestration efficiency measures how much wasted motion occurs getting there. An agent that takes 40 minutes to do 8 minutes of actual work has 80% idle time — all of which is pipeline overhead. Reducing stall time directly improves throughput without changing code quality. This is the only metric that reveals whether the routing loop in `work-on.md` is functioning or causing agents to stop and wait.
545
-
546
- **Step 4 — Batch stall rate (from FORGE:STALL_DETECTED annotations)**:
547
-
548
- Query issues that received a `<!-- FORGE:STALL_DETECTED -->` comment within the analysis window. This measures how often the orchestrator's time-based stall detector fired — distinct from the resume-cycle count (which comes from completion-event stalls).
549
-
550
- ```bash
551
- # Denominator: issues that entered the pipeline in the window (have any workflow:* label)
552
- # Using pipeline-active issues avoids inflating the denominator with all 500 issues in window
553
- BATCH_TOTAL=$(gh issue list -R $GH_REPO \
554
- --state all --limit 500 \
555
- --json number,createdAt,labels \
556
- --jq "[.[] | select(.createdAt > \"$SINCE\") | select(.labels | map(.name) | any(startswith(\"workflow:\")))] | length" \
557
- 2>/dev/null || echo 0)
558
-
559
- # Enumerate pipeline-active issue numbers for per-issue comment queries
560
- PIPELINE_ISSUE_NUMS=$(gh issue list -R $GH_REPO \
561
- --state all --limit 500 \
562
- --json number,createdAt,labels \
563
- --jq "[.[] | select(.createdAt > \"$SINCE\") | select(.labels | map(.name) | any(startswith(\"workflow:\"))) | .number]" \
564
- 2>/dev/null || echo "[]")
565
-
566
- # Count issues with at least one FORGE:STALL_DETECTED comment
567
- # Use wc -l pattern (not variable accumulation) to avoid bash subshell scope loss
568
- STALL_COUNT=$(echo "$PIPELINE_ISSUE_NUMS" | jq -r '.[]' | while read -r NUM; do
569
- STALL_COMMENTS=$(gh api repos/$GH_REPO/issues/${NUM}/comments \
570
- --jq '[.[] | select(.body | contains("FORGE:STALL_DETECTED"))] | length' 2>/dev/null || echo 0)
571
- [ "$STALL_COMMENTS" -gt 0 ] && echo "$NUM"
572
- done | wc -l)
573
-
574
- # Count issues that escalated (2+ stall comments = auto-resume exhausted)
575
- ESCALATED_COUNT=$(echo "$PIPELINE_ISSUE_NUMS" | jq -r '.[]' | while read -r NUM; do
576
- STALL_COMMENTS=$(gh api repos/$GH_REPO/issues/${NUM}/comments \
577
- --jq '[.[] | select(.body | contains("FORGE:STALL_DETECTED"))] | length' 2>/dev/null || echo 0)
578
- [ "$STALL_COMMENTS" -ge 2 ] && echo "$NUM"
579
- done | wc -l)
580
-
581
- # Compute batch stall rate
582
- if [ "$BATCH_TOTAL" -gt 0 ]; then
583
- BATCH_STALL_RATE=$(echo "scale=1; $STALL_COUNT * 100 / $BATCH_TOTAL" | bc 2>/dev/null || echo "N/A")
584
- echo "Batch stall rate: ${BATCH_STALL_RATE}% (${STALL_COUNT}/${BATCH_TOTAL} pipeline issues stalled) — target: < 10%"
585
- echo "Escalated to needs-human: ${ESCALATED_COUNT} issues"
586
- else
587
- BATCH_STALL_RATE="N/A"
588
- echo "Batch stall rate: N/A (no pipeline issues in window or FORGE:STALL_DETECTED not yet deployed)"
589
- fi
590
- export BATCH_STALL_RATE STALL_COUNT BATCH_TOTAL ESCALATED_COUNT
591
- ```
592
-
593
- **Target**: `BATCH_STALL_RATE` < 10% (stalled issues / total issues in window). A rate above 10% indicates systemic pipeline unreliability — agents stopping without reaching terminal state frequently enough to require orchestrator intervention.
594
-
595
- **Note**: `BATCH_STALL_RATE` requires `FORGE:STALL_DETECTED` comments, which are written by the stall detector in `/orchestrate` Step 4B.5. If the stall detector is not deployed, this metric returns N/A and falls back to the resume-cycle proxy from Step 3.
596
-
597
- ### 2L: Post-deploy failure metrics (pipeline escape signal)
598
-
599
- `/audit` creates `audit-finding` labeled issues in the Forge repo whenever a defect escapes the pipeline and reaches staging or production. Each finding is tagged with a `**Failure point**:` field indicating which pipeline phase failed (INVESTIGATION, IMPLEMENTATION, REVIEW, REVIEW_FALSE_NEG, DEPLOY_GATE, etc.). Aggregating these reveals the post-deploy defect rate and review escape rate — signals invisible to the pre-merge metrics above.
600
-
601
- **Why this matters**: Build pass rate measures whether PRs merge successfully. Post-deploy failure rate measures whether those merged PRs actually work correctly in production. The health score can read 90+ while production failures accumulate if post-deploy data is excluded. Review escape rate directly measures review agent effectiveness: a REVIEW_FALSE_NEG finding means the review agent saw the change but approved a defect — the most actionable failure mode.
602
-
603
- ```bash
604
- # All audit-finding issues created in the window
605
- AUDIT_FINDINGS=$(gh issue list -R $FORGE_REPO --state all --label "audit-finding" \
606
- --limit 200 --json number,title,body,createdAt \
607
- --jq "[.[] | select(.createdAt > \"$SINCE\")]")
608
-
609
- AUDIT_TOTAL=$(echo "$AUDIT_FINDINGS" | jq 'length')
610
- echo "Audit findings in window: $AUDIT_TOTAL"
611
-
612
- # Post-deploy failure rate = audit findings / PRs merged in window
613
- # (MERGED_COUNT computed in Phase 2E)
614
- if [ "$MERGED_COUNT" -gt 0 ] && [ "$AUDIT_TOTAL" -gt 0 ]; then
615
- echo "Post-deploy failure rate: $(echo "scale=1; $AUDIT_TOTAL * 100 / $MERGED_COUNT" | bc)% (target: < 5%)"
616
- else
617
- echo "Post-deploy failure rate: 0% (no audit findings in window)"
618
- fi
619
-
620
- # Review escape rate = REVIEW_FALSE_NEG findings / PRs merged in window
621
- REVIEW_ESCAPE_COUNT=$(echo "$AUDIT_FINDINGS" | \
622
- jq '[.[] | select(.body | test("\\*\\*Failure point\\*\\*:.*REVIEW_FALSE_NEG"))] | length')
623
- echo "Review escape count: $REVIEW_ESCAPE_COUNT"
624
- if [ "$MERGED_COUNT" -gt 0 ]; then
625
- echo "Review escape rate: $(echo "scale=1; $REVIEW_ESCAPE_COUNT * 100 / $MERGED_COUNT" | bc)% (target: < 2%)"
626
- fi
627
-
628
- # Failure category distribution (by Failure point field)
629
- echo ""
630
- echo "Failure point distribution:"
631
- echo "$AUDIT_FINDINGS" | jq -r \
632
- '.[] | .body | capture("\\*\\*Failure point\\*\\*: *(?<fp>[^\\n]+)").fp // "UNKNOWN"' \
633
- | sort | uniq -c | sort -rn
634
-
635
- # Mean time to detect (days between issue createdAt and linked PR's mergedAt)
636
- # Simplified: use audit finding createdAt as proxy for detection time relative to window start
637
- # Full MTTD requires joining audit-finding body for the source PR number and querying its mergedAt
638
- echo ""
639
- echo "Mean time to detect (proxy — days from window open to finding creation):"
640
- echo "$AUDIT_FINDINGS" | jq -r \
641
- --arg since "$SINCE" \
642
- '[.[] | {days: ((.createdAt | fromdateiso8601) - ($since | fromdateiso8601)) / 86400}] | if length > 0 then (map(.days) | add / length) else 0 end' \
643
- | awk '{printf "%.1f days\n", $1}'
644
-
645
- # Severity breakdown (P1 vs P2 vs P3 from labels)
646
- echo ""
647
- echo "Severity breakdown:"
648
- gh issue list -R $FORGE_REPO --state all --label "audit-finding" \
649
- --limit 200 --json number,createdAt,labels \
650
- --jq "[.[] | select(.createdAt > \"$SINCE\")] | group_by(.labels | map(.name) | map(select(startswith(\"P\"))) | .[0]) | map({severity: .[0].labels | map(.name) | map(select(startswith(\"P\"))) | .[0], count: length})"
651
- ```
652
-
653
- **Metrics to compute**:
654
- - **Post-deploy failure rate**: audit findings in window / PRs merged in window. Target: < 5%. Each finding represents a defect that passed the full pipeline and required post-merge remediation.
655
- - **Review escape rate**: REVIEW_FALSE_NEG findings / PRs merged in window. Target: < 2%. This is the most direct measure of review agent effectiveness — these are defects the review agent saw but approved.
656
- - **Failure category distribution**: breakdown by failure point (INVESTIGATION, IMPLEMENTATION, REVIEW, REVIEW_FALSE_NEG, DEPLOY_GATE, etc.). No target — use for diagnosis. High IMPLEMENTATION counts indicate builder gaps; high REVIEW_FALSE_NEG counts indicate review agent gaps.
657
- - **Mean time to detect (MTD)**: average days from PR merge to audit-finding creation. Target: < 7 days. Long MTD means defects linger undetected in production.
658
-
659
- ### 2M: Session discovery and date-gated filtering (conversation transcript analytics)
660
-
661
- This phase locates conversation JSONL files for pipeline command sessions (work-on, orchestrate, review-pr, quality-gate, review-pr-staging) that ran within the analysis window. It produces `SESSION_JSONL_FILES` — consumed by downstream Conversation Transcript Analytics phases to extract flow metrics (phase durations, tool patterns, subagent efficiency, stalls, rate-limits).
662
-
663
- **Why `history.jsonl` and not mtime**: `history.jsonl` timestamps record session START time (accurate to the minute). File mtime records the last WRITE — it trails session start by an average of 156 minutes. Date-gating on mtime would misclassify sessions that started before `$SINCE` but were still writing at the cutoff.
664
-
665
- **Why XML `<command-name>` tag and not `display` field**: The `display` field records what the user TYPED, not what actually ran. A session where the user types `/work-on 350` then immediately `/clear` appears as a pipeline session in `display` but the JSONL contains only a `/clear` run. In testing, the `display`-only filter produced a 40% false positive rate. The `<command-name>` XML tag in the first few lines of the JSONL file records the actual executed command and is authoritative.
666
-
667
- **Path construction algorithm** (empirically verified from investigation #348):
668
-
669
- ```bash
670
- # Convert a Claude Code project path to its ~/.claude/projects/ directory name
671
- # Rule: both '/' and '.' are replaced with '-'
672
- # Examples:
673
- # /home/user/projects/myproject -> -home-user-projects-myproject
674
- # /home/user/projects/myproject/.claude/worktrees/fix-branch
675
- # -> -home-user-projects-myproject--claude-worktrees-fix-branch
676
- # (note: .claude -> --claude because '.' becomes '-' and surrounding '/' also become '-')
677
- _session_project_dir() {
678
- echo "$1" | sed 's|[/.]|-|g'
679
- }
680
- ```
681
-
682
- **Step 1 — Parse history.jsonl and filter to pipeline sessions in window**:
683
-
684
- ```bash
685
- HISTORY_FILE="$HOME/.claude/history.jsonl"
686
- PIPELINE_COMMANDS='work-on|orchestrate|review-pr|quality-gate|review-pr-staging'
687
-
688
- if [ ! -f "$HISTORY_FILE" ]; then
689
- echo "Phase 2M: $HISTORY_FILE not found — skipping session discovery"
690
- SESSION_JSONL_FILES=()
691
- SESSION_TOTAL=0
692
- else
693
- echo "Phase 2M: parsing $HISTORY_FILE for pipeline sessions since $SINCE"
694
-
695
- # Extract pipeline sessions: entries where display starts with a pipeline command
696
- # and timestamp is after $SINCE. One JSON object per line.
697
- # Output: tab-separated sessionId, project, timestamp, display
698
- SESSION_CANDIDATES=$(jq -r --arg since "$SINCE" --arg cmds "$PIPELINE_COMMANDS" '
699
- select(
700
- .timestamp != null and
701
- .timestamp > $since and
702
- .sessionId != null and
703
- .project != null and
704
- .display != null and
705
- (.display | test("^/(" + $cmds + ")( |$)"))
706
- ) |
707
- [.sessionId, .project, .timestamp, (.display | split(" ") | .[0])] | @tsv
708
- ' "$HISTORY_FILE" 2>/dev/null)
709
-
710
- # Deduplicate by sessionId — keep only the earliest entry per session
711
- # (earliest timestamp = session start time, most accurate for date filtering)
712
- DEDUPED_SESSIONS=$(echo "$SESSION_CANDIDATES" | sort -t$'\t' -k3,3 | sort -t$'\t' -k1,1 -u)
713
-
714
- SESSION_COUNT=$(echo "$DEDUPED_SESSIONS" | grep -c . 2>/dev/null || echo 0)
715
- echo "Phase 2M: found $SESSION_COUNT unique pipeline sessions in window (pre-disk verification)"
716
- fi
717
- ```
718
-
719
- **Step 2 — Map session IDs to JSONL files on disk**:
720
-
721
- ```bash
722
- SESSION_JSONL_FILES=()
723
- SESSION_MISSING_COUNT=0
724
- SESSION_VERIFIED_COUNT=0
725
- SESSION_FP_COUNT=0
726
-
727
- CLAUDE_PROJECTS_DIR="$HOME/.claude/projects"
728
-
729
- while IFS=$'\t' read -r SESSION_ID PROJECT TIMESTAMP DISPLAY_CMD; do
730
- [ -z "$SESSION_ID" ] && continue
731
-
732
- # Convert project path to directory name
733
- PROJECT_DIR=$(_session_project_dir "$PROJECT")
734
- JSONL_PATH="$CLAUDE_PROJECTS_DIR/$PROJECT_DIR/$SESSION_ID.jsonl"
735
-
736
- # Skip if file does not exist on disk (older/pruned sessions, different machine)
737
- if [ ! -f "$JSONL_PATH" ]; then
738
- SESSION_MISSING_COUNT=$((SESSION_MISSING_COUNT + 1))
739
- continue
740
- fi
741
-
742
- # Verify actual command via first <command-name> XML tag in JSONL
743
- # Pipeline commands are stored with double-slash prefix: //work-on, //orchestrate, etc.
744
- # Read only the first 5 lines (enough to find the command tag — it appears in the first message)
745
- ACTUAL_CMD=$(head -5 "$JSONL_PATH" 2>/dev/null | \
746
- grep -oP '(?<=<command-name>)//[^<]+(?=</command-name>)' | head -1)
747
-
748
- # Strip the double-slash prefix for comparison
749
- ACTUAL_CMD_CLEAN="${ACTUAL_CMD#//}"
750
-
751
- # Verify the actual command matches a pipeline command
752
- if ! echo "$ACTUAL_CMD_CLEAN" | grep -qP "^($PIPELINE_COMMANDS)$"; then
753
- SESSION_FP_COUNT=$((SESSION_FP_COUNT + 1))
754
- continue # False positive — display said pipeline command but JSONL ran something else
755
- fi
756
-
757
- # Determine subagent directory (JSONL files for agent sub-calls within this session)
758
- # Sub-agent JSONL files share the same project dir but have different session IDs
759
- # The parent session's own JSONL is the primary; subagents appear as sibling files
760
- SUBAGENT_DIR="$CLAUDE_PROJECTS_DIR/$PROJECT_DIR"
761
-
762
- SESSION_JSONL_FILES+=("$SESSION_ID $ACTUAL_CMD_CLEAN $TIMESTAMP $JSONL_PATH $SUBAGENT_DIR")
763
- SESSION_VERIFIED_COUNT=$((SESSION_VERIFIED_COUNT + 1))
764
-
765
- done <<< "$DEDUPED_SESSIONS"
766
-
767
- SESSION_TOTAL=${#SESSION_JSONL_FILES[@]}
768
-
769
- echo "Phase 2M: session discovery complete"
770
- echo " Verified pipeline sessions: $SESSION_TOTAL"
771
- echo " Missing on disk (pruned): $SESSION_MISSING_COUNT"
772
- echo " False positives filtered: $SESSION_FP_COUNT"
773
- echo ""
774
- echo "SESSION_JSONL_FILES contains $SESSION_TOTAL entries — available for transcript analytics phases"
775
- ```
776
-
777
- **Output format** (each element in `SESSION_JSONL_FILES` is a tab-separated record):
778
-
779
- | Field | Content | Example |
780
- |-------|---------|---------|
781
- | `sessionId` | Claude Code session UUID | `abc123-def456-...` |
782
- | `command` | Actual pipeline command run | `work-on` |
783
- | `timestamp` | Session start (from history.jsonl) | `2026-05-28T14:23:01Z` |
784
- | `jsonlPath` | Absolute path to JSONL file | `~/.claude/projects/-home-.../abc123.jsonl` |
785
- | `subagentDir` | Directory containing sibling JSONL files | `~/.claude/projects/-home-.../` |
786
-
787
- **Performance characteristics** (from investigation #348 empirical measurement):
788
- - `history.jsonl` parse + filter (25K entries): **<100ms**
789
- - mtime scan of 900 disk files: **<2ms**
790
- - Head-5 JSONL read for XML tag: **negligible** (<1ms per file)
791
- - Total Phase 2M runtime: **<5s** for 25K history entries with 300 pipeline sessions
792
-
793
- **Skip conditions**: If `~/.claude/history.jsonl` does not exist (different machine, fresh install), set `SESSION_TOTAL=0` and emit a warning. Downstream transcript analytics phases must gracefully skip when `SESSION_TOTAL == 0`.
794
-
795
- ---
796
-
797
- ### JSONL Parser Utility
798
-
799
- <!-- FORGE:JSONL_PARSER_UTIL -->
800
-
801
- This utility defines a reusable Python snippet for parsing Claude Code's local conversation JSONL files. It is **not a numbered phase** — it is a helper block that transcript analytics phases (added by the Conversation Transcript Analytics milestone) invoke at runtime.
802
-
803
- **Schema source**: #346 (investigate: map Claude Code JSONL transcript schema) — 8 message types mapped across 979 records.
804
-
805
- **Privacy contract**: Raw conversation content is NEVER emitted. Only structural metadata is extracted: tool names, skill identifiers, agent descriptions, timing data, and session metadata.
806
-
807
- **Usage** (phases that call this parser pass a JSONL file path):
808
- ```bash
809
- # Parse a single JSONL file and emit one structured event per line (NDJSON)
810
- JSONL_FILE="~/.claude/projects/{project}/{sessionId}.jsonl"
811
- python3 - "$JSONL_FILE" << 'PYEOF'
812
- import json, sys, os
813
-
814
- jsonl_file = sys.argv[1] if len(sys.argv) > 1 else None
815
- if not jsonl_file:
816
- print(json.dumps({"error": "No JSONL file path provided"}), flush=True)
817
- sys.exit(1)
818
-
819
- def parse_jsonl_file(path):
820
- """Parse a Claude Code JSONL session file. Emits one JSON event per line."""
821
- try:
822
- with open(path, "r", encoding="utf-8") as f:
823
- lines = f.readlines()
824
- except (OSError, IOError) as e:
825
- print(json.dumps({"error": f"Cannot open {path}: {e}"}), flush=True)
826
- return
827
-
828
- for lineno, raw in enumerate(lines, 1):
829
- raw = raw.strip()
830
- if not raw:
831
- continue
832
- try:
833
- msg = json.loads(raw)
834
- except json.JSONDecodeError:
835
- # Malformed line — skip silently (schema resilience rule)
836
- continue
837
-
838
- msg_type = msg.get("type", "")
839
- timestamp = msg.get("timestamp", "")
840
- session_id = msg.get("sessionId", "")
841
- is_subagent = bool(msg.get("isSidechain", False))
842
- agent_id = msg.get("agentId")
843
-
844
- # Base event skeleton — only structural fields, no content
845
- base = {
846
- "timestamp": timestamp,
847
- "session_id": session_id,
848
- "is_subagent": is_subagent,
849
- "lineno": lineno,
850
- }
851
- if agent_id:
852
- base["agent_id"] = agent_id
853
-
854
- if msg_type == "assistant":
855
- # Extract tool_use blocks from assistant message content
856
- content = msg.get("message", {}).get("content", [])
857
- if not isinstance(content, list):
858
- content = []
859
- for block in content:
860
- if not isinstance(block, dict):
861
- continue
862
- block_type = block.get("type", "")
863
- if block_type == "tool_use":
864
- tool_name = block.get("name", "")
865
- tool_input = block.get("input", {}) or {}
866
-
867
- if tool_name == "Skill":
868
- # Skill invocation — extract skill name and args (structural only)
869
- event = {**base,
870
- "event_type": "skill",
871
- "tool_name": "Skill",
872
- "skill_name": tool_input.get("skill", ""),
873
- "skill_args": tool_input.get("args", ""),
874
- }
875
- elif tool_name == "Agent":
876
- # Agent spawn — extract description and model (no prompt content)
877
- event = {**base,
878
- "event_type": "agent_spawn",
879
- "tool_name": "Agent",
880
- "agent_desc": tool_input.get("description", ""),
881
- "agent_model": tool_input.get("model", ""),
882
- }
883
- else:
884
- # All other tool calls — name only, no input content
885
- event = {**base,
886
- "event_type": "tool_use",
887
- "tool_name": tool_name,
888
- }
889
- print(json.dumps(event), flush=True)
890
- # text and thinking blocks: skip (raw content — privacy rule)
891
-
892
- elif msg_type == "system":
893
- subtype = msg.get("subtype", "")
894
- if subtype == "turn_duration":
895
- # Authoritative phase timing signal — use durationMs, NOT timestamp arithmetic
896
- event = {**base,
897
- "event_type": "turn_duration",
898
- "duration_ms": msg.get("durationMs", 0),
899
- "parent_uuid": msg.get("parentUuid", ""),
900
- }
901
- print(json.dumps(event), flush=True)
902
- elif subtype == "compact_boundary":
903
- # Context compaction marker
904
- compact_meta = msg.get("compactMetadata", {}) or {}
905
- event = {**base,
906
- "event_type": "compact",
907
- "pre_tokens": compact_meta.get("preTokens", 0),
908
- "trigger": compact_meta.get("trigger", ""),
909
- }
910
- print(json.dumps(event), flush=True)
911
- # local_command and other system subtypes: skip (content-bearing)
20
+ Read and execute phases in sequence. Each phase file is self-contained.
912
21
 
913
- elif msg_type == "user":
914
- # user messages: only emit session-start marker for first message (parentUuid=null)
915
- if msg.get("parentUuid") is None and not is_subagent:
916
- event = {**base,
917
- "event_type": "session_start",
918
- "is_meta": bool(msg.get("isMeta", False)),
919
- }
920
- print(json.dumps(event), flush=True)
22
+ | Step | File | Description |
23
+ |------|------|-------------|
24
+ | 0 | `pipeline-health/config.md` | Configuration — READ FIRST |
25
+ | 1 | `pipeline-health/phase-1-context.md` | Identify context (target project, analysis window, prior report) |
26
+ | 2 | `pipeline-health/phase-2-metrics.md` | Collect pipeline metrics (review findings, build rates, transcript analytics) |
27
+ | 3 | `pipeline-health/phase-3-analyze.md` | Analyze & correlate (defect breakdown, prompt change impact, health score) |
28
+ | 4 | `pipeline-health/phase-4-proposals.md` | Generate improvement proposals |
29
+ | 5 | `pipeline-health/phase-5-report.md` | Report & track (post health report, create improvement issues) |
30
+ | 6 | `pipeline-health/phase-6-summary.md` | Summary |
921
31
 
922
- elif msg_type == "progress":
923
- # Agent progress — only structural signal (agentId, parentToolUseID)
924
- data = msg.get("data", {}) or {}
925
- if data.get("type") == "agent_progress":
926
- event = {**base,
927
- "event_type": "agent_progress",
928
- "progress_agent_id": data.get("agentId", ""),
929
- "parent_tool_use_id": msg.get("parentToolUseID", ""),
930
- }
931
- print(json.dumps(event), flush=True)
932
- # hook_progress: skip
32
+ ## Quick Reference
933
33
 
934
- elif msg_type in ("queue-operation", "pr-link", "last-prompt", "file-history-snapshot"):
935
- # Known non-analytics types — skip silently
936
- pass
937
-
938
- else:
939
- # Unknown type — emit as unknown_type for forward-compatibility diagnostics
940
- event = {**base,
941
- "event_type": "unknown_type",
942
- "type": msg_type,
943
- }
944
- print(json.dumps(event), flush=True)
945
-
946
- parse_jsonl_file(jsonl_file)
947
- PYEOF
948
- ```
949
-
950
- **Calling convention**: Pass the JSONL file path as the first argument:
951
- ```bash
952
- python3 -c "..." "$JSONL_FILE" # inline version (for short scripts)
953
- python3 - "$JSONL_FILE" << 'PYEOF' # heredoc version — '-' reads from stdin, arg follows
954
- ```
955
-
956
- **Output format** (NDJSON — one JSON object per line):
957
-
958
- | `event_type` | Fields emitted | Source |
959
- |---|---|---|
960
- | `tool_use` | `tool_name`, `timestamp`, `session_id`, `is_subagent` | `assistant` content `tool_use` blocks |
961
- | `skill` | `tool_name="Skill"`, `skill_name`, `skill_args`, `timestamp`, `is_subagent` | `assistant` content where `name=="Skill"` |
962
- | `agent_spawn` | `tool_name="Agent"`, `agent_desc`, `agent_model`, `timestamp`, `is_subagent` | `assistant` content where `name=="Agent"` |
963
- | `turn_duration` | `duration_ms`, `parent_uuid`, `timestamp` | `system {subtype: "turn_duration"}` |
964
- | `compact` | `pre_tokens`, `trigger`, `timestamp` | `system {subtype: "compact_boundary"}` |
965
- | `session_start` | `is_meta`, `timestamp` | first `user` message (`parentUuid=null`) |
966
- | `agent_progress` | `progress_agent_id`, `parent_tool_use_id`, `timestamp` | `progress {data.type: "agent_progress"}` |
967
- | `unknown_type` | `type` | any unrecognized `type` value |
968
-
969
- **Schema resilience rules**:
970
- 1. `msg.get('field')` throughout — absent fields return `None`, never crash
971
- 2. Malformed JSON lines are silently skipped (try/except per line)
972
- 3. Unknown `type` values emit `unknown_type` event — not silently dropped
973
- 4. Both `usage` variants handled (parser doesn't touch `usage` — structural extraction only)
974
- 5. `is_subagent` populated from `isSidechain` field, not filename heuristic
975
-
976
- **Subagent detection**:
977
- - `isSidechain: true` → `is_subagent: true` in all events from that file
978
- - `agentId` field present → `agent_id` field in output
979
- - Subagent files located at: `~/.claude/projects/{project}/{sessionId}/subagents/agent-{agentId}.jsonl`
980
-
981
- **Worktree session detection** (for session discovery phases):
982
- ```bash
983
- # Project directory name regex to detect worktree sessions
984
- echo "$PROJECT_DIR" | grep -qP '^-.*--claude-worktrees-.*$' && echo "worktree" || echo "main"
985
- ```
986
-
987
- ---
988
-
989
- ### 2N: Transcript flow metrics (session efficiency signal)
990
-
991
- Parses the JSONL files discovered in Phase 2M to extract pipeline session flow metrics: per-turn durations, tool call frequency, context compaction events, stall turns (turns taking >2 min), and rate-limit indicator turns. This is the only metric that sees _inside_ a running session — all other Phase 2 metrics observe outcomes (PRs merged, findings created) not internal dynamics.
992
-
993
- **Prerequisite**: Phase 2M must have run. If `SESSION_TOTAL == 0`, skip this phase and set `TRANSCRIPT_DATA_AVAILABLE=false`.
994
-
995
- **This phase reads local disk only — no GitHub API calls.**
996
-
997
- **Step 1 — Check prerequisite and select files to parse**:
998
-
999
- ```bash
1000
- if [ "${SESSION_TOTAL:-0}" -eq 0 ]; then
1001
- echo "Phase 2N: SESSION_TOTAL=0 (Phase 2M found no sessions or history.jsonl missing) — skipping"
1002
- TRANSCRIPT_DATA_AVAILABLE=false
1003
- else
1004
- TRANSCRIPT_DATA_AVAILABLE=true
1005
-
1006
- # Performance guard: cap at 50 sessions to bound parse time.
1007
- # If SESSION_TOTAL > 50: select 30 most recent + 10 largest (by JSONL file size) + 10 random from remainder.
1008
- # Deduplication is implicit — most-recent and largest sets are drawn from the same array without replacement.
1009
- if [ "${SESSION_TOTAL}" -gt 50 ]; then
1010
- echo "Phase 2N: $SESSION_TOTAL sessions found — sampling 50 (30 most recent + 10 largest + 10 random) for performance"
1011
-
1012
- # SESSION_JSONL_FILES is a bash array; each element: sessionId\tcommand\ttimestamp\tjsonlPath\tsubagentDir
1013
- # Step A: sort by timestamp (field 3) descending → pick first 30
1014
- SORTED_BY_TIME=$(printf '%s\n' "${SESSION_JSONL_FILES[@]}" | sort -t$'\t' -k3,3 -r)
1015
- RECENT_30=$(echo "$SORTED_BY_TIME" | head -30)
1016
- RECENT_IDS=$(echo "$RECENT_30" | cut -f1)
1017
-
1018
- # Step B: from remaining entries (not in RECENT_IDS), sort by JSONL file size → pick 10 largest
1019
- REMAINDER=$(printf '%s\n' "${SESSION_JSONL_FILES[@]}" | grep -vFf <(echo "$RECENT_IDS") 2>/dev/null || true)
1020
- LARGEST_10=""
1021
- if [ -n "$REMAINDER" ]; then
1022
- LARGEST_10=$(echo "$REMAINDER" | while IFS=$'\t' read -r SID CMD TS JP SD; do
1023
- SZ=$(stat -c%s "$JP" 2>/dev/null || echo 0)
1024
- printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$SZ" "$SID" "$CMD" "$TS" "$JP" "$SD"
1025
- done | sort -rn | head -10 | cut -f2-)
1026
- LARGEST_IDS=$(echo "$LARGEST_10" | cut -f1)
1027
- fi
1028
-
1029
- # Step C: from entries not yet selected, pick 10 at random
1030
- EXCLUDED=$(printf '%s\n%s\n' "$RECENT_IDS" "${LARGEST_IDS:-}")
1031
- POOL=$(printf '%s\n' "${SESSION_JSONL_FILES[@]}" | grep -vFf <(echo "$EXCLUDED") 2>/dev/null || true)
1032
- RANDOM_10=""
1033
- if [ -n "$POOL" ]; then
1034
- RANDOM_10=$(echo "$POOL" | shuf | head -10)
1035
- fi
1036
-
1037
- # Combine and rebuild SESSION_JSONL_FILES (deduplicated)
1038
- COMBINED=$(printf '%s\n%s\n%s\n' "$RECENT_30" "${LARGEST_10:-}" "${RANDOM_10:-}" | grep -v '^$' | sort -u)
1039
- SESSION_TOTAL_ORIGINAL=$SESSION_TOTAL
1040
- SESSION_JSONL_FILES=()
1041
- while IFS= read -r ENTRY; do
1042
- [ -z "$ENTRY" ] && continue
1043
- SESSION_JSONL_FILES+=("$ENTRY")
1044
- done <<< "$COMBINED"
1045
- SESSION_TOTAL=${#SESSION_JSONL_FILES[@]}
1046
- echo "Phase 2N: sampled $SESSION_TOTAL of $SESSION_TOTAL_ORIGINAL sessions (30 most recent + 10 largest + 10 random)"
1047
- else
1048
- echo "Phase 2N: parsing $SESSION_TOTAL session JSONL files for flow metrics"
1049
- SESSION_TOTAL_ORIGINAL=$SESSION_TOTAL
1050
- fi
1051
- fi
1052
- ```
1053
-
1054
- **Step 2 — Parse each session and aggregate metrics (inline Python)**:
1055
-
1056
- **IMPORTANT**: Run as a single Python script. JSONL files can be large (500+ lines, 100KB+). Budget: 2 minutes total.
1057
-
1058
- ```python
1059
- import json, sys, os
1060
- from collections import defaultdict
1061
-
1062
- # Read SESSION_JSONL_FILES from Phase 2M environment
1063
- # Each entry is a tab-separated record: sessionId \t command \t timestamp \t jsonlPath \t subagentDir
1064
- session_records_raw = os.environ.get('SESSION_JSONL_FILES_RAW', '')
1065
-
1066
- # Aggregate accumulators
1067
- all_durations_ms = [] # from turn_duration events
1068
- total_tool_counts = defaultdict(int) # tool_name → call count
1069
- total_compactions = 0 # compact_boundary events
1070
- total_stall_turns = 0 # turns > 120000 ms
1071
- total_rate_limit_turns = 0 # turns 60000–120000 ms (possible rate-limit)
1072
- parent_session_count = 0
1073
- subagent_session_count = 0
1074
- parse_errors = 0
1075
-
1076
- def process_jsonl_file(jsonl_path, is_subagent=False):
1077
- """Parse one JSONL file using the JSONL Parser Utility event model."""
1078
- global total_compactions, total_stall_turns, total_rate_limit_turns, parse_errors
1079
-
1080
- try:
1081
- with open(jsonl_path, encoding='utf-8', errors='replace') as f:
1082
- for line in f:
1083
- line = line.strip()
1084
- if not line:
1085
- continue
1086
- try:
1087
- msg = json.loads(line)
1088
- except json.JSONDecodeError:
1089
- parse_errors += 1
1090
- continue
1091
-
1092
- msg_type = msg.get('type', '')
1093
- file_is_subagent = bool(msg.get('isSidechain', False)) or is_subagent
1094
-
1095
- if msg_type == 'assistant':
1096
- # Tool call frequency
1097
- content = msg.get('message', {}).get('content', [])
1098
- if isinstance(content, list):
1099
- for block in content:
1100
- if isinstance(block, dict) and block.get('type') == 'tool_use':
1101
- total_tool_counts[block.get('name', 'unknown')] += 1
1102
-
1103
- elif msg_type == 'system':
1104
- subtype = msg.get('subtype', '')
1105
- if subtype == 'turn_duration':
1106
- dur = msg.get('durationMs')
1107
- if isinstance(dur, (int, float)) and dur > 0:
1108
- all_durations_ms.append(dur)
1109
- if dur > 120000:
1110
- total_stall_turns += 1
1111
- elif dur > 60000:
1112
- total_rate_limit_turns += 1
1113
- elif subtype == 'compact_boundary':
1114
- total_compactions += 1
1115
-
1116
- # Unknown types: silently skip (schema resilience)
1117
-
1118
- except (IOError, OSError):
1119
- parse_errors += 1
1120
-
1121
- # Process each session from SESSION_JSONL_FILES_RAW
1122
- for record in session_records_raw.strip().split('\n'):
1123
- if not record.strip():
1124
- continue
1125
- parts = record.split('\t')
1126
- if len(parts) < 4:
1127
- continue
1128
- session_id, command, timestamp, jsonl_path = parts[0], parts[1], parts[2], parts[3]
1129
- subagent_dir = parts[4] if len(parts) > 4 else ''
1130
-
1131
- # Expand ~ in path
1132
- jsonl_path = os.path.expanduser(jsonl_path)
1133
- if not os.path.isfile(jsonl_path):
1134
- continue
1135
-
1136
- parent_session_count += 1
1137
- process_jsonl_file(jsonl_path, is_subagent=False)
1138
-
1139
- # Also parse subagent files in the same session directory
1140
- if subagent_dir:
1141
- sub_dir = os.path.expanduser(os.path.join(os.path.dirname(jsonl_path), session_id, 'subagents'))
1142
- if os.path.isdir(sub_dir):
1143
- for fname in os.listdir(sub_dir):
1144
- if fname.startswith('agent-') and fname.endswith('.jsonl'):
1145
- subagent_session_count += 1
1146
- process_jsonl_file(os.path.join(sub_dir, fname), is_subagent=True)
1147
-
1148
- # Output aggregated metrics
1149
- total_turns = len(all_durations_ms)
1150
- avg_duration_ms = sum(all_durations_ms) / len(all_durations_ms) if all_durations_ms else 0
1151
- max_duration_ms = max(all_durations_ms) if all_durations_ms else 0
1152
- stall_rate = (total_stall_turns / total_turns * 100) if total_turns > 0 else 0
1153
-
1154
- print(f"\n=== Phase 2N: Transcript Flow Metrics ===")
1155
- print(f"Pipeline sessions parsed: {parent_session_count} parent, {subagent_session_count} subagent")
1156
- print(f"Total turns with timing: {total_turns}")
1157
- print(f"Avg turn duration: {avg_duration_ms/1000:.1f}s")
1158
- print(f"Max turn duration: {max_duration_ms/1000:.1f}s")
1159
- print(f"Stall turns (>120s): {total_stall_turns} ({stall_rate:.1f}% of turns) — target: < 10%")
1160
- print(f"Rate-limit indicator turns (60–120s): {total_rate_limit_turns}")
1161
- print(f"Context compaction events: {total_compactions}")
1162
- print()
1163
- print("Tool call frequency (top 10):")
1164
- for tool, count in sorted(total_tool_counts.items(), key=lambda x: -x[1])[:10]:
1165
- print(f" {tool}: {count}")
1166
- ```
1167
-
1168
- To run this step:
1169
-
1170
- ```bash
1171
- if [ "$TRANSCRIPT_DATA_AVAILABLE" = "true" ]; then
1172
- # Serialize SESSION_JSONL_FILES array to newline-separated string for Python
1173
- export SESSION_JSONL_FILES_RAW
1174
- SESSION_JSONL_FILES_RAW=$(printf '%s\n' "${SESSION_JSONL_FILES[@]}")
1175
- PHASE_2N_OUTPUT=$(python3 - << 'PYEOF'
1176
- # [paste the Python script above here at runtime]
1177
- PYEOF
1178
- )
1179
- echo "$PHASE_2N_OUTPUT"
1180
-
1181
- # Extract bash variables from Phase 2N output for use in Phase 3C health score
1182
- # "Stall turns (>120s): N (X.Y% of turns) — target: < 10%"
1183
- STALL_TURN_RATE=$(echo "$PHASE_2N_OUTPUT" | grep -oP 'Stall turns \(>120s\): \d+ \(\K[\d.]+(?=% of turns)')
1184
- STALL_TURN_RATE=${STALL_TURN_RATE:-0}
1185
- # "Context compaction events: N"
1186
- COMPACTION_TOTAL=$(echo "$PHASE_2N_OUTPUT" | grep -oP 'Context compaction events: \K\d+')
1187
- COMPACTION_TOTAL=${COMPACTION_TOTAL:-0}
1188
- export STALL_TURN_RATE COMPACTION_TOTAL
1189
- echo "Phase 2N exports: STALL_TURN_RATE=$STALL_TURN_RATE COMPACTION_TOTAL=$COMPACTION_TOTAL"
1190
- fi
1191
34
  ```
1192
-
1193
- **Metrics produced**:
1194
- - **Pipeline sessions in window**: sessions confirmed in Phase 2M that actually ran a pipeline command (`work-on`, `orchestrate`, `review-pr`, etc.)
1195
- - **Avg turn duration (ms)**: mean of all `system.turn_duration.durationMs` values. Uses `durationMs` directly — NOT timestamp arithmetic (timestamps are non-monotonic).
1196
- - **Stall turns**: turns exceeding 120s. Each stall is a turn where the agent spent >2 min — often a rate-limit backoff or very large context. Target: < 10% of turns.
1197
- - **Rate-limit indicator turns**: turns 60–120s — possible but not certain rate-limit waits. Informational.
1198
- - **Context compaction events**: count of `system.compact_boundary` records. High compaction rates suggest prompts or tool outputs are too large, causing agents to lose intermediate state.
1199
- - **Tool call frequency**: which tools dominate. Reveals whether agents are read-heavy (Bash/Read/Grep) vs coordination-heavy (Skill/Agent).
1200
-
1201
- **Skip condition**: If `TRANSCRIPT_DATA_AVAILABLE` is false, emit:
1202
- > `Phase 2N: Transcript flow metrics unavailable — no pipeline sessions found by Phase 2M (SESSION_TOTAL=0). Run /pipeline-health on the same machine where Claude Code sessions are stored.`
1203
-
1204
- ---
1205
-
1206
- ### 2O: Per-session metric extraction (structured session metrics)
1207
-
1208
- Reads the same JSONL files discovered by Phase 2M to produce a **structured metrics dict per session**. Where Phase 2N aggregates across all sessions, Phase 2O isolates each session independently — enabling downstream phases (subagent flow #352, privacy aggregation #353, health report #354) to bucket, normalize, and compare sessions by command type, task complexity, and model distribution.
1209
-
1210
- **Prerequisite**: Phase 2N must have run. If `TRANSCRIPT_DATA_AVAILABLE != true`, skip this phase.
1211
-
1212
- **This phase reads local disk only — no GitHub API calls.**
1213
-
1214
- **Metrics per session (HIGH and MEDIUM actionability only per #347)**:
1215
- - `total_duration_ms` — sum of all `system.turn_duration.durationMs` values (active computation time; NOT wall-clock)
1216
- - `tool_call_counts` — dict of tool_name → count for all tool_use blocks
1217
- - `skill_invocations` — list of `{phase_name, skill_name, skill_args}` in call order
1218
- - `agent_spawns` — list of `{agent_desc, model}` for each Agent tool call
1219
- - `model_distribution` — dict of model_name → count (from `message.model` on assistant messages; proxy for sonnet/opus fallback pressure)
1220
- - `compaction_count` — count of `system {subtype: compact_boundary}` events
1221
-
1222
- **NOT included** (SKIP tier from #347): wall-clock duration, rate-limit event detection.
1223
-
1224
- **Step 1 — Check prerequisite**:
1225
-
1226
- ```bash
1227
- if [ "${TRANSCRIPT_DATA_AVAILABLE:-false}" != "true" ]; then
1228
- echo "Phase 2O: TRANSCRIPT_DATA_AVAILABLE=false (Phase 2N skipped or no sessions) — skipping"
1229
- PER_SESSION_METRICS_RAW=""
1230
- else
1231
- echo "Phase 2O: extracting per-session metrics from $SESSION_TOTAL sessions"
1232
- fi
35
+ Read: $FORGE_HOME/commands/pipeline-health/config.md # ALWAYS READ FIRST
36
+ Read: $FORGE_HOME/commands/pipeline-health/phase-1-context.md
37
+ Read: $FORGE_HOME/commands/pipeline-health/phase-2-metrics.md
38
+ Read: $FORGE_HOME/commands/pipeline-health/phase-3-analyze.md
39
+ Read: $FORGE_HOME/commands/pipeline-health/phase-4-proposals.md
40
+ Read: $FORGE_HOME/commands/pipeline-health/phase-5-report.md
41
+ Read: $FORGE_HOME/commands/pipeline-health/phase-6-summary.md
1233
42
  ```
1234
43
 
1235
- **Step 2 — Parse each session and emit structured NDJSON (inline Python)**:
1236
-
1237
- ```python
1238
- import json, sys, os
1239
- from collections import defaultdict
1240
-
1241
- # Skill name → pipeline phase name mapping
1242
- SKILL_PHASE_MAP = {
1243
- 'work-on': 'work-on',
1244
- 'quality-gate': 'build:validate',
1245
- 'review-pr': 'review',
1246
- 'review-pr-staging':'review',
1247
- 'cleanup': 'close',
1248
- 'orchestrate': 'orchestrate',
1249
- 'frontend-design': 'build:implement',
1250
- 'validate': 'validate',
1251
- 'audit': 'audit',
1252
- 'pipeline-health': 'pipeline-health',
1253
- }
1254
-
1255
- def parse_session_jsonl(jsonl_path):
1256
- """Parse one JSONL file and return per-session metric dict."""
1257
- metrics = {
1258
- 'total_duration_ms': 0,
1259
- 'tool_call_counts': defaultdict(int),
1260
- 'skill_invocations': [], # [{phase_name, skill_name, skill_args}]
1261
- 'agent_spawns': [], # [{agent_desc, model}]
1262
- 'model_distribution': defaultdict(int), # model_name → count
1263
- 'compaction_count': 0,
1264
- 'parse_errors': 0,
1265
- }
1266
-
1267
- try:
1268
- with open(jsonl_path, encoding='utf-8', errors='replace') as f:
1269
- for line in f:
1270
- line = line.strip()
1271
- if not line:
1272
- continue
1273
- try:
1274
- msg = json.loads(line)
1275
- except json.JSONDecodeError:
1276
- metrics['parse_errors'] += 1
1277
- continue
1278
-
1279
- msg_type = msg.get('type', '')
1280
-
1281
- if msg_type == 'assistant':
1282
- # Track model distribution (SAFE field per #347)
1283
- model = msg.get('message', {}).get('model', '')
1284
- if model:
1285
- metrics['model_distribution'][model] += 1
1286
-
1287
- # Extract tool_use blocks
1288
- content = msg.get('message', {}).get('content', [])
1289
- if not isinstance(content, list):
1290
- content = []
1291
- for block in content:
1292
- if not isinstance(block, dict):
1293
- continue
1294
- if block.get('type') != 'tool_use':
1295
- continue
1296
- tool_name = block.get('name', 'unknown')
1297
- tool_input = block.get('input', {}) or {}
1298
-
1299
- # Count all tool calls by name
1300
- metrics['tool_call_counts'][tool_name] += 1
1301
-
1302
- if tool_name == 'Skill':
1303
- # Map skill name to pipeline phase
1304
- skill_name = tool_input.get('skill', '')
1305
- skill_args = tool_input.get('args', '')
1306
- phase_name = SKILL_PHASE_MAP.get(skill_name, 'other')
1307
- metrics['skill_invocations'].append({
1308
- 'phase_name': phase_name,
1309
- 'skill_name': skill_name,
1310
- 'skill_args': str(skill_args)[:80], # truncate — no raw content
1311
- })
1312
- elif tool_name == 'Agent':
1313
- # Extract description and model only — NOT prompt content (UNSAFE per #347)
1314
- metrics['agent_spawns'].append({
1315
- 'agent_desc': str(tool_input.get('description', ''))[:120],
1316
- 'model': tool_input.get('model', ''),
1317
- })
1318
-
1319
- elif msg_type == 'system':
1320
- subtype = msg.get('subtype', '')
1321
- if subtype == 'turn_duration':
1322
- dur = msg.get('durationMs')
1323
- if isinstance(dur, (int, float)) and dur > 0:
1324
- metrics['total_duration_ms'] += dur
1325
- elif subtype == 'compact_boundary':
1326
- metrics['compaction_count'] += 1
1327
-
1328
- # All other message types: skip (content-bearing or irrelevant)
1329
-
1330
- except (IOError, OSError):
1331
- metrics['parse_errors'] += 1
1332
-
1333
- return metrics
1334
-
1335
- # Read SESSION_JSONL_FILES from Phase 2M
1336
- session_records_raw = os.environ.get('SESSION_JSONL_FILES_RAW', '')
1337
-
1338
- session_results = []
1339
- total_sessions = 0
1340
- print(f"\n=== Phase 2O: Per-Session Metric Extraction ===")
1341
-
1342
- for record in session_records_raw.strip().split('\n'):
1343
- if not record.strip():
1344
- continue
1345
- parts = record.split('\t')
1346
- if len(parts) < 4:
1347
- continue
1348
- session_id = parts[0]
1349
- command = parts[1]
1350
- timestamp = parts[2]
1351
- jsonl_path = os.path.expanduser(parts[3])
1352
-
1353
- if not os.path.isfile(jsonl_path):
1354
- continue
1355
-
1356
- total_sessions += 1
1357
- m = parse_session_jsonl(jsonl_path)
1358
-
1359
- # Serialize to JSON-safe dict (convert defaultdicts)
1360
- session_record = {
1361
- 'session_id': session_id,
1362
- 'command': command,
1363
- 'timestamp': timestamp,
1364
- 'total_duration_ms': m['total_duration_ms'],
1365
- 'tool_call_counts': dict(m['tool_call_counts']),
1366
- 'skill_invocations': m['skill_invocations'],
1367
- 'agent_spawns': m['agent_spawns'],
1368
- 'model_distribution': dict(m['model_distribution']),
1369
- 'compaction_count': m['compaction_count'],
1370
- 'parse_errors': m['parse_errors'],
1371
- }
1372
- session_results.append(session_record)
1373
- # Emit as NDJSON line (consumed by downstream phases via PER_SESSION_METRICS_RAW)
1374
- print(json.dumps(session_record), flush=True)
1375
-
1376
- # Print summary table
1377
- print(f"\nParsed {total_sessions} sessions.")
1378
- print(f"{'Session':<20} {'Command':<16} {'DurationS':>9} {'Tools':>6} {'Skills':>7} {'Agents':>7} {'Compact':>8} {'Sonnet/Opus':>12}")
1379
- print("-" * 92)
1380
- for s in session_results:
1381
- dur_s = s['total_duration_ms'] / 1000
1382
- tools = sum(s['tool_call_counts'].values())
1383
- skills = len(s['skill_invocations'])
1384
- agents = len(s['agent_spawns'])
1385
- compact = s['compaction_count']
1386
- sonnet = sum(v for k, v in s['model_distribution'].items() if 'sonnet' in k.lower())
1387
- opus = sum(v for k, v in s['model_distribution'].items() if 'opus' in k.lower())
1388
- sid_short = s['session_id'][:18]
1389
- cmd_short = s['command'][:14]
1390
- print(f"{sid_short:<20} {cmd_short:<16} {dur_s:>9.1f} {tools:>6} {skills:>7} {agents:>7} {compact:>8} {sonnet:>5}/{opus:<5}")
1391
- ```
1392
-
1393
- To run this step:
1394
-
1395
- ```bash
1396
- if [ "${TRANSCRIPT_DATA_AVAILABLE:-false}" = "true" ]; then
1397
- export SESSION_JSONL_FILES_RAW
1398
- SESSION_JSONL_FILES_RAW=$(printf '%s\n' "${SESSION_JSONL_FILES[@]}")
1399
- PER_SESSION_METRICS_RAW=$(python3 - << 'PYEOF'
1400
- # [paste the Python script above here at runtime]
1401
- PYEOF
1402
- )
1403
- echo "$PER_SESSION_METRICS_RAW"
1404
- export PER_SESSION_METRICS_RAW
1405
- fi
1406
- ```
1407
-
1408
- **Metrics produced per session**:
1409
- - **`total_duration_ms`**: sum of `system.turn_duration.durationMs` — active agent compute time, excludes human idle. Use this; NOT wall-clock timestamps.
1410
- - **`tool_call_counts`**: dict of every tool name called and how many times. Bash:Edit ratio reveals investigator vs builder behaviour.
1411
- - **`skill_invocations`**: ordered list of Skill calls with mapped phase names. Reveals which pipeline phases ran in this session and how many times each was invoked.
1412
- - **`agent_spawns`**: list of Agent tool calls with model used. Reveals subagent spawning patterns and model choices.
1413
- - **`model_distribution`**: sonnet vs opus request count. Proxy for API fallback pressure — a session with many opus requests may have experienced rate-limiting.
1414
- - **`compaction_count`**: per-session context compaction events. >1 = session spanned multiple context windows.
1415
-
1416
- **Output variable**: `PER_SESSION_METRICS_RAW` — NDJSON string (one JSON record per line), one record per session. Consumed by Phase 2O's print output and by downstream phases (#352 subagent flow, #353 privacy aggregation, #354 health report integration). Export it with `export PER_SESSION_METRICS_RAW` so downstream phases can read it.
1417
-
1418
- **Skip condition**: If `TRANSCRIPT_DATA_AVAILABLE` is false, emit:
1419
- > `Phase 2O: Per-session metrics unavailable — Phase 2N did not produce session data. SESSION_TOTAL must be > 0 for this phase to run.`
1420
-
1421
- ---
1422
-
1423
- ### 2P: Subagent flow analysis (role-mapped efficiency signal)
1424
-
1425
- Parses subagent JSONL files for each pipeline session to extract per-role efficiency metrics: which pipeline role each subagent performed, how long each agent ran, how many tools it called, and whether the same role was retried multiple times within a session. This phase complements Phase 2N's aggregate-level flow metrics with role-level breakdowns — revealing which pipeline phases (investigate, build, review) are slowest, most retry-prone, or most model-diverse.
1426
-
1427
- **Prerequisite**: Phase 2N must have run. If `TRANSCRIPT_DATA_AVAILABLE` is false, skip this phase.
1428
-
1429
- **This phase reads local disk only — no GitHub API calls.**
1430
-
1431
- **Role mapping rules** (applied to `agent_desc` from `agent_spawn` events):
1432
-
1433
- | Pattern | Mapped Role |
1434
- |---------|-------------|
1435
- | `^(Investigate|investigate)\s+` | `investigate` |
1436
- | `^(Work on|work on|Resume #\d+ build|Continue #\d+ build)\s+` | `build` |
1437
- | `^(Review|review|Resume.*review|Continue.*review)\s*` | `review` |
1438
- | `^Review\+Close\s+` | `review` |
1439
- | `^(Resume #\d+ close|Continue #\d+ close)\s*` | `close` |
1440
- | `^Quality gate\s*` | `quality-gate` |
1441
- | `^(Orchestrate|orchestrate)\s+` | `orchestrate` |
1442
- | domain agent names (e.g., `API bug hunter`, `Billing.*review`, `Security.*review`, `Frontend.*quality`, `General security`, `Auth.*review`, `Infra.*review`) | `review` |
1443
- | No pattern matches | `other` |
1444
-
1445
- **Retry detection**: Within a single session, if the same role is assigned to more than one subagent, subsequent agents are classified as retried (positional detection: second+ agent in same session+role = retry).
1446
-
1447
- **Step 1 — Check prerequisite**:
1448
-
1449
- ```bash
1450
- if [ "${TRANSCRIPT_DATA_AVAILABLE:-false}" != "true" ]; then
1451
- echo "Phase 2P: TRANSCRIPT_DATA_AVAILABLE=false — skipping subagent flow analysis"
1452
- SUBAGENT_DATA_AVAILABLE=false
1453
- else
1454
- echo "Phase 2P: analyzing subagent flow for $SESSION_TOTAL sessions"
1455
- SUBAGENT_DATA_AVAILABLE=true
1456
- fi
1457
- ```
1458
-
1459
- **Step 2 — Parse agent_spawn events and subagent files (inline Python)**:
1460
-
1461
- **IMPORTANT**: Run as a single Python script. Budget: 2 minutes total.
1462
-
1463
- ```python
1464
- import json, sys, os, re
1465
- from collections import defaultdict
1466
-
1467
- # Role mapping: ordered list of (regex_pattern, role_name) tuples
1468
- # Evaluated in order; first match wins. Domain review agents match last (broad patterns).
1469
- ROLE_PATTERNS = [
1470
- (re.compile(r'^(Investigate|investigate)\s+', re.I), 'investigate'),
1471
- (re.compile(r'^(Work on|work on)\s+', re.I), 'build'),
1472
- (re.compile(r'^Resume #\d+ build', re.I), 'build'),
1473
- (re.compile(r'^Continue #\d+ build', re.I), 'build'),
1474
- (re.compile(r'^forge#\d+\b', re.I), 'build'),
1475
- (re.compile(r'^Review\+Close\s+', re.I), 'review'),
1476
- (re.compile(r'^Resume.*close', re.I), 'close'),
1477
- (re.compile(r'^Continue.*close', re.I), 'close'),
1478
- (re.compile(r'^Quality gate', re.I), 'quality-gate'),
1479
- (re.compile(r'^(Orchestrate|orchestrate)\s+', re.I), 'orchestrate'),
1480
- (re.compile(r'^(Review|review)\s+', re.I), 'review'),
1481
- (re.compile(r'^Resume.*review', re.I), 'review'),
1482
- (re.compile(r'^Continue.*review', re.I), 'review'),
1483
- # Domain review agent patterns (broad — must come after specific patterns)
1484
- (re.compile(r'(bug hunter|security|quality audit|quality review|billing.*review|auth.*review|infra.*review|frontend.*review|API.*review|API.*audit|DB.*review|concurr.*review)', re.I), 'review'),
1485
- (re.compile(r'General security', re.I), 'review'),
1486
- ]
1487
-
1488
- def map_role(agent_desc):
1489
- """Map agent_desc string to a pipeline role name."""
1490
- for pattern, role in ROLE_PATTERNS:
1491
- if pattern.search(agent_desc):
1492
- return role
1493
- return 'other'
1494
-
1495
- # Read SESSION_JSONL_FILES_RAW from Phase 2M/2N environment
1496
- session_records_raw = os.environ.get('SESSION_JSONL_FILES_RAW', '')
1497
-
1498
- # Per-role accumulators
1499
- # role → list of agent records: {session_id, desc, model, duration_ms, tool_count, compact_count, is_retry}
1500
- role_agents = defaultdict(list)
1501
-
1502
- def compute_subagent_metrics(subagent_path):
1503
- """Return (duration_ms, tool_count, compact_count) for a subagent JSONL file."""
1504
- duration_ms = 0
1505
- tool_count = 0
1506
- compact_count = 0
1507
- try:
1508
- with open(subagent_path, encoding='utf-8', errors='replace') as f:
1509
- for line in f:
1510
- line = line.strip()
1511
- if not line:
1512
- continue
1513
- try:
1514
- msg = json.loads(line)
1515
- except json.JSONDecodeError:
1516
- continue
1517
- msg_type = msg.get('type', '')
1518
- if msg_type == 'assistant':
1519
- content = msg.get('message', {}).get('content', [])
1520
- if isinstance(content, list):
1521
- for block in content:
1522
- if isinstance(block, dict) and block.get('type') == 'tool_use':
1523
- tool_count += 1
1524
- elif msg_type == 'system':
1525
- subtype = msg.get('subtype', '')
1526
- if subtype == 'turn_duration':
1527
- dur = msg.get('durationMs')
1528
- if isinstance(dur, (int, float)) and dur > 0:
1529
- duration_ms += dur
1530
- elif subtype == 'compact_boundary':
1531
- compact_count += 1
1532
- except (IOError, OSError):
1533
- pass
1534
- return duration_ms, tool_count, compact_count
1535
-
1536
- # Process each parent session
1537
- for record in session_records_raw.strip().split('\n'):
1538
- if not record.strip():
1539
- continue
1540
- parts = record.split('\t')
1541
- if len(parts) < 4:
1542
- continue
1543
- session_id, command, timestamp, jsonl_path = parts[0], parts[1], parts[2], parts[3]
1544
- jsonl_path = os.path.expanduser(jsonl_path)
1545
- if not os.path.isfile(jsonl_path):
1546
- continue
1547
-
1548
- # Collect agent_spawn events from the parent JSONL (structural only)
1549
- spawned_agents = [] # list of {desc, model}
1550
- try:
1551
- with open(jsonl_path, encoding='utf-8', errors='replace') as f:
1552
- for line in f:
1553
- line = line.strip()
1554
- if not line:
1555
- continue
1556
- try:
1557
- msg = json.loads(line)
1558
- except json.JSONDecodeError:
1559
- continue
1560
- if msg.get('type') != 'assistant':
1561
- continue
1562
- content = msg.get('message', {}).get('content', [])
1563
- if not isinstance(content, list):
1564
- continue
1565
- for block in content:
1566
- if not isinstance(block, dict):
1567
- continue
1568
- if block.get('type') == 'tool_use' and block.get('name') == 'Agent':
1569
- inp = block.get('input', {}) or {}
1570
- agent_desc = inp.get('description', '')
1571
- agent_model = inp.get('model', 'unknown')
1572
- spawned_agents.append({'desc': agent_desc, 'model': agent_model})
1573
- except (IOError, OSError):
1574
- continue
1575
-
1576
- if not spawned_agents:
1577
- continue
1578
-
1579
- # Subagent JSONL directory: dirname(jsonl_path)/{session_id}/subagents/
1580
- sub_dir = os.path.join(os.path.dirname(jsonl_path), session_id, 'subagents')
1581
-
1582
- # Track roles seen in this session for retry detection (positional)
1583
- session_role_counts = defaultdict(int) # role → spawn count in this session
1584
- spawn_counter = 0 # overall counter across all roles (positional file index)
1585
-
1586
- for spawn in spawned_agents:
1587
- role = map_role(spawn['desc'])
1588
- session_role_counts[role] += 1
1589
- # Positional retry: second+ spawn of same role in same session = retry
1590
- is_retry = (session_role_counts[role] > 1)
1591
-
1592
- # Best-effort: associate this spawn with a subagent file by position (sorted order)
1593
- # Cannot map tool_use id to agent file hash — take file at overall spawn position
1594
- duration_ms = 0
1595
- tool_count = 0
1596
- compact_count = 0
1597
-
1598
- if os.path.isdir(sub_dir):
1599
- agent_files = sorted([
1600
- f for f in os.listdir(sub_dir)
1601
- if f.startswith('agent-') and f.endswith('.jsonl')
1602
- ])
1603
- # Use overall spawn position (across all roles) as index into sorted file list
1604
- if spawn_counter < len(agent_files):
1605
- fpath = os.path.join(sub_dir, agent_files[spawn_counter])
1606
- duration_ms, tool_count, compact_count = compute_subagent_metrics(fpath)
1607
-
1608
- spawn_counter += 1
1609
- role_agents[role].append({
1610
- 'session_id': session_id,
1611
- 'desc': spawn['desc'],
1612
- 'model': spawn['model'],
1613
- 'duration_ms': duration_ms,
1614
- 'tool_count': tool_count,
1615
- 'compact_count': compact_count,
1616
- 'is_retry': is_retry,
1617
- })
1618
-
1619
- # Aggregate metrics by role
1620
- print("\n=== Phase 2P: Subagent Flow Analysis ===")
1621
- print()
1622
-
1623
- all_roles = sorted(role_agents.keys())
1624
- model_distribution = defaultdict(int)
1625
-
1626
- for role in all_roles:
1627
- agents = role_agents[role]
1628
- total = len(agents)
1629
- retried = sum(1 for a in agents if a.get('is_retry', False))
1630
- retry_rate = (retried / total * 100) if total > 0 else 0
1631
-
1632
- durations = [a['duration_ms'] for a in agents if a['duration_ms'] > 0]
1633
- avg_dur_s = (sum(durations) / len(durations) / 1000) if durations else 0
1634
- max_dur_s = (max(durations) / 1000) if durations else 0
1635
-
1636
- tool_counts = [a['tool_count'] for a in agents]
1637
- avg_tools = (sum(tool_counts) / len(tool_counts)) if tool_counts else 0
1638
-
1639
- compacts = sum(a['compact_count'] for a in agents)
1640
-
1641
- for a in agents:
1642
- model_distribution[a['model']] += 1
1643
-
1644
- print(f"Role: {role} ({total} agents, {retried} retried — retry rate: {retry_rate:.0f}%)")
1645
- print(f" Avg duration: {avg_dur_s:.1f}s Max: {max_dur_s:.1f}s")
1646
- print(f" Avg tool calls: {avg_tools:.1f} Total compactions: {compacts}")
1647
- print()
1648
-
1649
- print("Model distribution (across all roles):")
1650
- for model, count in sorted(model_distribution.items(), key=lambda x: -x[1]):
1651
- print(f" {model}: {count} agents")
1652
-
1653
- total_agents = sum(len(v) for v in role_agents.values())
1654
- total_retried = sum(sum(1 for a in v if a.get('is_retry', False)) for v in role_agents.values())
1655
- print()
1656
- print(f"Total subagents analyzed: {total_agents}")
1657
- print(f"Total retried agents: {total_retried} ({(total_retried/total_agents*100) if total_agents else 0:.1f}%)")
1658
- ```
1659
-
1660
- To run this step:
1661
-
1662
- ```bash
1663
- if [ "$SUBAGENT_DATA_AVAILABLE" = "true" ]; then
1664
- export SESSION_JSONL_FILES_RAW
1665
- python3 - << 'PYEOF'
1666
- # [paste the Python script above here at runtime]
1667
- PYEOF
1668
- fi
1669
- ```
1670
-
1671
- **Metrics produced**:
1672
- - **Per-role agent count**: how many subagents handled each pipeline role (investigate, build, review, quality-gate, close, orchestrate, other) in the window.
1673
- - **Retry rate by role**: % of agents for that role that were retried (same role spawned more than once per session). High retry rates indicate stall or failure patterns for that phase.
1674
- - **Avg duration by role (seconds)**: mean of all `turn_duration.durationMs` values summed per subagent. Uses `durationMs` — NOT timestamp arithmetic.
1675
- - **Avg tool calls by role**: mean tool call count per subagent in that role. High counts may indicate agents doing excessive re-reads or exploration.
1676
- - **Model distribution**: breakdown of `sonnet` vs `opus` (and other models) across all subagents. Opus appearances indicate rate-limit fallback or explicit override — they cost more and run slower.
1677
- - **Total compactions**: count of `compact_boundary` events across all subagents in the role. High compaction indicates agents hitting context limits — prompt size or tool output volume issue.
1678
-
1679
- **Interpretation guide**:
1680
- - **Retry rate > 20% for a role** → that pipeline phase is fragile: agents stall, produce errors, or get externally resumed more than expected.
1681
- - **Avg duration for build > 600s** → builder agents are slow — likely doing excessive re-reads, hitting rate limits, or working on high-complexity issues that may need decomposition.
1682
- - **Opus count > 10% of total** → significant rate-limit fallback occurring — consider spreading pipeline sessions or reducing prompt sizes.
1683
- - **High compactions in review role** → review agents are loading too much context (large diffs or too many files) and compacting mid-review, which degrades finding quality.
1684
-
1685
- **Skip condition**: If `SUBAGENT_DATA_AVAILABLE` is false, emit:
1686
- > `Phase 2P: Subagent flow analysis unavailable — TRANSCRIPT_DATA_AVAILABLE=false from Phase 2N. Requires Phase 2M session discovery to have found pipeline sessions on this machine.`
1687
-
1688
- ---
1689
-
1690
- ### 2Q: Privacy-safe aggregation and harmful-takeaway guardrails
1691
-
1692
- Aggregates per-session metrics from Phase 2O's `PER_SESSION_METRICS_RAW` into a sanitized, report-ready summary. **No content fields pass through this phase** — only structural data (tool names, counts, durations, models). Session IDs are stripped. Output is context-framed with distribution ranges (median + P25/P75) to prevent misleading averages. Harmful-takeaway guardrail footnotes are emitted as a block for Phase 5A to embed directly in the health report.
1693
-
1694
- **Prerequisite**: Phase 2O must have run. If `TRANSCRIPT_DATA_AVAILABLE` is false or `PER_SESSION_METRICS_RAW` is empty, skip and set `AGGREGATED_TRANSCRIPT_METRICS=""`.
1695
-
1696
- **This phase reads only `PER_SESSION_METRICS_RAW` (in-memory) — no disk reads, no GitHub API calls.**
1697
-
1698
- **Field whitelist** (deny by default — only these fields are read from each session record):
1699
- - `command` — pipeline command name (safe: structural)
1700
- - `total_duration_ms` — active compute time (safe: timing only)
1701
- - `tool_call_counts` — dict of tool_name → count (safe: tool names are structural)
1702
- - `skill_invocations[].skill_name` — skill name only (safe: structural); `skill_args` is NOT read
1703
- - `model_distribution` — model_name → count dict (safe: model names only); authoritative source for sonnet/opus split
1704
- - `compaction_count` — integer (safe: count only)
1705
-
1706
- **Fields explicitly excluded** (even though present in Phase 2O output):
1707
- - `session_id` — opaque UUID; not useful for pipeline improvement; excluded from all output
1708
- - `skill_invocations[].skill_args` — may contain issue numbers (safe) but also file paths or branch names (contextual); excluded to keep the whitelist strict
1709
- - `agent_spawns[].model` — excluded from model_freq to prevent double-counting with `model_distribution`; spawn intent ≠ response actuality; Phase 2P uses this field separately for spawn pattern analysis <!-- Added: forge#369 -->
1710
- - `agent_spawns[].agent_desc` — may contain file paths or code context; excluded per #347 privacy classification
1711
-
1712
- **Step 1 — Prerequisite guard**:
1713
-
1714
- ```bash
1715
- if [ "${TRANSCRIPT_DATA_AVAILABLE:-false}" != "true" ] || [ -z "${PER_SESSION_METRICS_RAW:-}" ]; then
1716
- echo "Phase 2Q: skipping — TRANSCRIPT_DATA_AVAILABLE=false or PER_SESSION_METRICS_RAW empty"
1717
- AGGREGATED_TRANSCRIPT_METRICS=""
1718
- TRANSCRIPT_AGGREGATION_AVAILABLE=false
1719
- else
1720
- echo "Phase 2Q: aggregating privacy-safe metrics from PER_SESSION_METRICS_RAW"
1721
- TRANSCRIPT_AGGREGATION_AVAILABLE=true
1722
- fi
1723
- ```
1724
-
1725
- **Step 2 — Aggregate and sanitize (inline Python)**:
1726
-
1727
- ```python
1728
- import json, sys, os, statistics
1729
- from collections import defaultdict
1730
-
1731
- raw = os.environ.get('PER_SESSION_METRICS_RAW', '')
1732
- if not raw.strip():
1733
- print('Phase 2Q: PER_SESSION_METRICS_RAW is empty — nothing to aggregate')
1734
- sys.exit(0)
1735
-
1736
- # Accumulators (whitelisted fields only — no session_id, no skill_args, no agent_desc)
1737
- durations_ms = [] # total_duration_ms per session
1738
- tool_freq = defaultdict(int) # tool_name → total call count across all sessions
1739
- skill_freq = defaultdict(int) # skill_name → invocation count across all sessions
1740
- model_freq = defaultdict(int) # model_name → message count across all sessions
1741
- compaction_counts = [] # compaction_count per session
1742
- command_buckets = defaultdict(list) # command → list of duration_ms (for task-type normalization)
1743
- sessions_parsed = 0
1744
- sessions_skipped = 0
1745
-
1746
- for line in raw.strip().split('\n'):
1747
- line = line.strip()
1748
- if not line:
1749
- continue
1750
- try:
1751
- rec = json.loads(line)
1752
- except json.JSONDecodeError:
1753
- sessions_skipped += 1
1754
- continue
1755
-
1756
- sessions_parsed += 1
1757
-
1758
- # session_id: intentionally NOT read — stripped by omission
1759
- command = str(rec.get('command', 'unknown'))[:32] # safe: command name only
1760
-
1761
- dur = rec.get('total_duration_ms', 0)
1762
- if isinstance(dur, (int, float)) and dur >= 0:
1763
- durations_ms.append(dur)
1764
- command_buckets[command].append(dur)
1765
-
1766
- # Tool call counts: tool names only (structural)
1767
- for tool_name, count in (rec.get('tool_call_counts') or {}).items():
1768
- if isinstance(count, int):
1769
- tool_freq[str(tool_name)[:64]] += count
1770
-
1771
- # Skill invocations: skill_name only — skill_args intentionally NOT read
1772
- for inv in (rec.get('skill_invocations') or []):
1773
- if isinstance(inv, dict):
1774
- sk = inv.get('skill_name', '')
1775
- if sk:
1776
- skill_freq[str(sk)[:32]] += 1
1777
-
1778
- # Model distribution: model name → count (authoritative source for sonnet/opus split)
1779
- # agent_spawns[].model is intentionally NOT included here — it counts spawn intent, not
1780
- # actual response messages, and overlaps with model_distribution in all realistic sessions.
1781
- for m, cnt in (rec.get('model_distribution') or {}).items():
1782
- if isinstance(cnt, int):
1783
- model_freq[str(m)[:64]] += cnt
1784
-
1785
- cc = rec.get('compaction_count', 0)
1786
- if isinstance(cc, int):
1787
- compaction_counts.append(cc)
1788
-
1789
- # Distribution helpers (no numpy — pure stdlib)
1790
- def _median(lst):
1791
- if not lst: return 0
1792
- s = sorted(lst)
1793
- n = len(s)
1794
- return s[n // 2] if n % 2 == 1 else (s[n//2 - 1] + s[n//2]) / 2
1795
-
1796
- def _p25(lst):
1797
- if not lst: return 0
1798
- s = sorted(lst)
1799
- return s[max(0, len(s)//4 - 1)]
1800
-
1801
- def _p75(lst):
1802
- if not lst: return 0
1803
- s = sorted(lst)
1804
- return s[min(len(s)-1, (3*len(s))//4)]
1805
-
1806
- n = sessions_parsed
1807
- dur_median_s = _median(durations_ms) / 1000 if durations_ms else 0
1808
- dur_p25_s = _p25(durations_ms) / 1000 if durations_ms else 0
1809
- dur_p75_s = _p75(durations_ms) / 1000 if durations_ms else 0
1810
- compact_median = _median(compaction_counts) if compaction_counts else 0
1811
- compact_p25 = _p25(compaction_counts) if compaction_counts else 0
1812
- compact_p75 = _p75(compaction_counts) if compaction_counts else 0
1813
-
1814
- # Top tools (top 8 by frequency)
1815
- top_tools = sorted(tool_freq.items(), key=lambda x: -x[1])[:8]
1816
- # Top skills
1817
- top_skills = sorted(skill_freq.items(), key=lambda x: -x[1])[:6]
1818
- # Model split
1819
- total_model_msgs = sum(model_freq.values()) or 1
1820
- sonnet_msgs = sum(v for k, v in model_freq.items() if 'sonnet' in k.lower())
1821
- opus_msgs = sum(v for k, v in model_freq.items() if 'opus' in k.lower())
1822
-
1823
- print(f"\n=== Phase 2Q: Privacy-Safe Aggregated Transcript Metrics ===")
1824
- print(f"Sessions aggregated: {n} (skipped/malformed: {sessions_skipped})")
1825
- print(f"Session IDs: NOT INCLUDED (stripped by whitelist)")
1826
- print()
1827
- print(f"Active compute duration (agent time, excludes human idle):")
1828
- print(f" Median: {dur_median_s:.0f}s [P25: {dur_p25_s:.0f}s — P75: {dur_p75_s:.0f}s] (across {n} pipeline runs)")
1829
- print()
1830
- print(f"Context compaction events per session:")
1831
- print(f" Median: {compact_median:.1f} [P25: {compact_p25:.1f} — P75: {compact_p75:.1f}]")
1832
- print()
1833
- print("Tool call distribution (top 8, all sessions combined):")
1834
- for tool, cnt in top_tools:
1835
- print(f" {tool}: {cnt}")
1836
- print()
1837
- print("Skill invocations (top 6):")
1838
- for skill, cnt in top_skills:
1839
- print(f" {skill}: {cnt}")
1840
- print()
1841
- print(f"Model distribution: sonnet={sonnet_msgs} ({100*sonnet_msgs//total_model_msgs}%) opus={opus_msgs} ({100*opus_msgs//total_model_msgs}%)")
1842
- print()
1843
- print("Command-type duration buckets (median active time per command):")
1844
- for cmd, dlist in sorted(command_buckets.items()):
1845
- med = _median(dlist)/1000
1846
- p25 = _p25(dlist)/1000
1847
- p75 = _p75(dlist)/1000
1848
- print(f" {cmd}: {med:.0f}s median [P25: {p25:.0f}s — P75: {p75:.0f}s] (n={len(dlist)})")
1849
- ```
1850
-
1851
- To run this step:
1852
-
1853
- ```bash
1854
- if [ "${TRANSCRIPT_AGGREGATION_AVAILABLE:-false}" = "true" ]; then
1855
- AGGREGATED_TRANSCRIPT_METRICS=$(python3 - << 'PYEOF'
1856
- # [paste the Python script above here at runtime]
1857
- PYEOF
1858
- )
1859
- echo "$AGGREGATED_TRANSCRIPT_METRICS"
1860
- export AGGREGATED_TRANSCRIPT_METRICS
1861
-
1862
- # Extract OPUS_PCT from Phase 2Q output for use in Phase 3C health score
1863
- # "Model distribution: sonnet=N (X%) opus=M (Y%)"
1864
- OPUS_PCT=$(echo "$AGGREGATED_TRANSCRIPT_METRICS" | grep -oP 'opus=\d+ \(\K\d+(?=%\))')
1865
- OPUS_PCT=${OPUS_PCT:-0}
1866
- export OPUS_PCT
1867
- echo "Phase 2Q exports: OPUS_PCT=$OPUS_PCT"
1868
- fi
1869
- ```
1870
-
1871
- **Output variable**: `AGGREGATED_TRANSCRIPT_METRICS` — plain text block (human-readable). Consumed by Phase 5A to populate the \"Aggregated Session Metrics\" section of the health report. Export with `export AGGREGATED_TRANSCRIPT_METRICS`.
1872
-
1873
- **Guardrail footnotes** — embed these verbatim in the Phase 5A report section (select ≥3; all 5 apply when transcript data is available):
1874
-
1875
- > ⚠️ **Harmful-takeaway guard [1/5]**: Session duration shown here is *agent compute time* (active tool-call time), not wall-clock. Wall-clock varies 2–10x due to human idle. \"Sessions are long\" does not mean prompts are bad — it means the agent was computing for a long time. Compare against the task-type median, not a global average.
1876
-
1877
- > ⚠️ **Harmful-takeaway guard [2/5]**: Tool call count is normalized per task type. A 15-file feature legitimately requires 200+ tool calls; a 1-file bug fix should need 30–50. \"High tool count\" without task-type context is a false signal. Flag only if count per files-changed is ≥3× the command-type median.
1878
-
1879
- > ⚠️ **Harmful-takeaway guard [3/5]**: Skill invocation count reflects *which pipeline commands ran*, not whether they ran correctly. A session that calls \`work-on\` 6 times may be a routing loop (bad) or a 6-phase investigation (expected). Do not treat high skill counts as pipeline bloat without checking the workflow:* label progression.
1880
-
1881
- > ⚠️ **Harmful-takeaway guard [4/5]**: Model distribution (sonnet/opus split) is a rate-limit pressure proxy, not a quality signal. High opus% means rate limits forced fallback — it does not mean the session used better models by choice. Remedy: spread pipeline sessions across time or reduce prompt token usage.
1882
-
1883
- > ⚠️ **Harmful-takeaway guard [5/5]**: Compaction events signal large context windows, not inefficiency. A review session scanning a 20-file diff legitimately compacts once. Flag only if compaction_count > 2 per session (median) and correlate with review-finding quality degradation — not with compaction count alone.
1884
-
1885
- **Threshold/target values**: Use '—' on first run (no prior baseline), matching the existing Phase 1C pattern. On subsequent runs, compare median duration and compaction rate against prior report's values.
1886
-
1887
- **Skip condition**: If `TRANSCRIPT_DATA_AVAILABLE` is false or `PER_SESSION_METRICS_RAW` is empty, emit:
1888
- > `Phase 2Q: Aggregated transcript metrics unavailable — Phase 2O did not produce session data. Run /pipeline-health on the machine where Claude Code sessions are stored.`
1889
-
1890
- ---
1891
-
1892
- ## Phase 3: Analyze & Correlate
1893
-
1894
- ### 3A: Defect category breakdown
1895
-
1896
- **3A-I: Pre-merge defects** — Group review findings (from Phase 2B) by agent prefix.
1897
-
1898
- First, count findings per category from the Phase 2B output:
1899
-
1900
- ```bash
1901
- # Count review findings per agent prefix (SEC, AUTH, BILL, DB, FE, INFRA, API, CONC, SCRP)
1902
- # These are the current-period counts to compare against PRIOR_* vars from Phase 1C
1903
- COUNT_SEC=$(echo "$FINDINGS_RAW" | grep -c '"SEC' || echo 0)
1904
- COUNT_AUTH=$(echo "$FINDINGS_RAW" | grep -c '"AUTH' || echo 0)
1905
- COUNT_BILL=$(echo "$FINDINGS_RAW" | grep -c '"BILL' || echo 0)
1906
- COUNT_DB=$(echo "$FINDINGS_RAW" | grep -c '"DB' || echo 0)
1907
- COUNT_FE=$(echo "$FINDINGS_RAW" | grep -c '"FE' || echo 0)
1908
- COUNT_INFRA=$(echo "$FINDINGS_RAW" | grep -c '"INFRA' || echo 0)
1909
- COUNT_API=$(echo "$FINDINGS_RAW" | grep -c '"API' || echo 0)
1910
- COUNT_CONC=$(echo "$FINDINGS_RAW" | grep -c '"CONC' || echo 0)
1911
- COUNT_SCRP=$(echo "$FINDINGS_RAW" | grep -c '"SCRP' || echo 0)
1912
- # ISSUE_SPEC from Phase 2L audit findings
1913
- COUNT_ISSUE_SPEC=$ISSUE_SPEC_COUNT
1914
-
1915
- TOTAL_FINDINGS=$((COUNT_SEC + COUNT_AUTH + COUNT_BILL + COUNT_DB + COUNT_FE + COUNT_INFRA + COUNT_API + COUNT_CONC + COUNT_SCRP))
1916
-
1917
- # Compute trend for each category using _trend helper from Phase 1C
1918
- TREND_SEC=$( _trend "$COUNT_SEC" "$PRIOR_SEC")
1919
- TREND_AUTH=$( _trend "$COUNT_AUTH" "$PRIOR_AUTH")
1920
- TREND_BILL=$( _trend "$COUNT_BILL" "$PRIOR_BILL")
1921
- TREND_DB=$( _trend "$COUNT_DB" "$PRIOR_DB")
1922
- TREND_FE=$( _trend "$COUNT_FE" "$PRIOR_FE")
1923
- TREND_INFRA=$( _trend "$COUNT_INFRA" "$PRIOR_INFRA")
1924
- TREND_API=$( _trend "$COUNT_API" "$PRIOR_API")
1925
- TREND_CONC=$( _trend "$COUNT_CONC" "$PRIOR_CONC")
1926
- TREND_SCRP=$( _trend "$COUNT_SCRP" "$PRIOR_SCRP")
1927
- TREND_ISSUE_SPEC=$(_trend "$COUNT_ISSUE_SPEC" "$PRIOR_ISSUE_SPEC")
1928
-
1929
- _pct() { [ "$TOTAL_FINDINGS" -gt 0 ] && echo "scale=0; $1 * 100 / $TOTAL_FINDINGS" | bc || echo "0"; }
1930
- ```
1931
-
1932
- Then render the table using the computed values:
1933
-
1934
- | Category | Count | % of Total | Trend vs Prior Period |
1935
- |----------|-------|------------|---------------------|
1936
- | SEC | $COUNT_SEC | $(_pct $COUNT_SEC)% | $TREND_SEC |
1937
- | AUTH | $COUNT_AUTH | $(_pct $COUNT_AUTH)% | $TREND_AUTH |
1938
- | BILL | $COUNT_BILL | $(_pct $COUNT_BILL)% | $TREND_BILL |
1939
- | DB | $COUNT_DB | $(_pct $COUNT_DB)% | $TREND_DB |
1940
- | FE | $COUNT_FE | $(_pct $COUNT_FE)% | $TREND_FE |
1941
- | INFRA | $COUNT_INFRA | $(_pct $COUNT_INFRA)% | $TREND_INFRA |
1942
- | API | $COUNT_API | $(_pct $COUNT_API)% | $TREND_API |
1943
- | CONC | $COUNT_CONC | $(_pct $COUNT_CONC)% | $TREND_CONC |
1944
- | SCRP | $COUNT_SCRP | $(_pct $COUNT_SCRP)% | $TREND_SCRP |
1945
- | ISSUE_SPEC | $COUNT_ISSUE_SPEC | $(_pct $COUNT_ISSUE_SPEC)% | $TREND_ISSUE_SPEC |
1946
-
1947
- **If PRIOR_* vars are all empty (baseline run)**: all trend cells show "—" (no prior period to compare against).
1948
-
1949
- **Note**: ISSUE_SPEC findings come from `/audit` (`audit-finding` label + body contains "ISSUE_SPEC"), not from review agents. They represent upstream failures — bad issue specs that the pipeline followed correctly but that produced wrong or wasted work.
1950
-
1951
- **3A-II: Post-deploy defects** — Group audit findings (from Phase 2L) by failure point:
1952
-
1953
- | Failure Point | Count | % of Total | Meaning |
1954
- |---------------|-------|------------|---------|
1955
- | INVESTIGATION | ? | ?% | Wrong verdict or missed scope — root cause misdiagnosed |
1956
- | IMPLEMENTATION | ? | ?% | Code written incorrectly or incompletely |
1957
- | REVIEW | ? | ?% | Review agents missed the issue or wrong agents triggered |
1958
- | REVIEW_FALSE_NEG | ? | ?% | Review agent saw area but approved a defect (escape signal) |
1959
- | QUALITY_GATE | ? | ?% | Static check should have caught this pattern but didn't |
1960
- | DEPLOY_GATE | ? | ?% | Deploy-info should have flagged risk but didn't |
1961
- | CONTEXT | ? | ?% | Missed relevant prior work or related code paths |
1962
- | ARCHITECT | ? | ?% | Missed code paths or wrong implementation plan |
1963
- | ORCHESTRATION | ? | ?% | Wave ordering wrong or dependency missed |
1964
- | ISSUE_SPEC | ? | ?% | Bad issue spec — pipeline followed it correctly, spec was wrong |
1965
- | OTHER | ? | ?% | Uncategorized or compound failure |
1966
-
1967
- **Cross-signal analysis**: Compare 3A-I (pre-merge review findings) with 3A-II (post-deploy audit findings) to identify where the pipeline gates are misaligned. High REVIEW_FALSE_NEG in 3A-II with low REVIEW findings per PR in 3A-I suggests review agents are approving too liberally rather than finding real defects.
1968
-
1969
- **Identify the top 3 defect categories across both tables.** These are where the builder/review is weakest and where pipeline improvements would have the highest ROI.
1970
-
1971
- ### 3B: Prompt change impact
1972
-
1973
- For each Forge commit in the window:
1974
- 1. What command was changed?
1975
- 2. What was the intent? (read the commit message)
1976
- 3. Did the target metric improve after this change?
1977
- - If the change was to `review-pr-agents.md` → did false positive rate change?
1978
- - If the change was to `work-on.md` Step 3G → did build failure rate change?
1979
- - If the change was to `orchestrate.md` → did milestone merge success rate change?
1980
-
1981
- **Caution**: Correlation ≠ causation. Note confounding factors (e.g., milestone size, new developers).
1982
-
1983
- ### 3C: Compute health score
1984
-
1985
- ```
1986
- Health Score = weighted average of:
1987
- Pre-merge quality:
1988
- - Build pass rate (staging→main): weight 12% target 95%+
1989
- - Review findings per PR: weight 9% target < 0.5
1990
- - Manual fix-up rate: weight 8% target < 15%
1991
- - False positive rate: weight 6% target < 10%
1992
- - Pipeline self-correction rate: weight 7% target < 15%
1993
-
1994
- Post-deploy quality:
1995
- - Post-deploy failure rate (Phase 2L): weight 9% target < 5%
1996
- - Review escape rate (Phase 2L): weight 9% target < 2%
1997
-
1998
- Pipeline health signals:
1999
- - Issue quality score (Phase 2J): weight 8% target > 80%
2000
- - Investigation accuracy (Phase 2I): weight 6% target > 85%
2001
- - Milestone merge success rate: weight 6% target 80%+
2002
- - Orchestration efficiency (Phase 2K): weight 5% target score > 70
2003
- - Original work ratio: weight 4% target > 70%
2004
- - Issue close velocity (days): weight 1% target < 2 days
2005
-
2006
- Transcript analytics (supplementary — Phase 2M–2Q):
2007
- - Transcript health score (Phase 2N/2Q): weight 10% target > 70
2008
- ```
2009
-
2010
- **Total weight**: 12+9+8+6+7+9+9+8+6+6+5+4+1+10 = 100%.
2011
-
2012
- **Post-deploy failure rate** (Phase 2L): scored as `max(0, 100 - (rate / 5%) * 100)` — i.e., 0% failure rate = 100 score, 5% failure rate = 0 score. If no audit findings exist in the window, score = 100.
2013
-
2014
- **Review escape rate** (Phase 2L): scored as `max(0, 100 - (rate / 2%) * 100)` — i.e., 0% escape rate = 100 score, 2% escape rate = 0 score. If no REVIEW_FALSE_NEG findings exist, score = 100. This metric directly measures review agent effectiveness: a high escape rate with a low findings-per-PR rate indicates review agents are approving too liberally.
2015
-
2016
- **Issue quality score** is computed from Phase 2J sub-metrics: invalidation rate (40%), ISSUE_SPEC findings (30%), re-investigation rate (20%), structure compliance (10%). A perfect score (100) requires: 0% invalidation, 0 ISSUE_SPEC audit findings, 0 re-investigations, 100% structure compliance.
2017
-
2018
- **Investigation accuracy** comes from Phase 2I trajectory data: (CONFIRMED + PARTIAL) / total issues with trajectory comments. If no trajectory data is available in the window, omit this metric from the weighted average and redistribute its 7% proportionally across the other metrics.
2019
-
2020
- **Orchestration efficiency score** is computed from Phase 2K metrics: avg idle% (40% weight — lower is better, 0% idle = 100 score), avg resumes per agent (35% weight — 0 resumes = 100, ≥2 resumes = 0), clean agent rate (25% weight — equals clean_rate%). If no persisted `FORGE:AUDIT-AGENTS` data is available in the window, omit this metric and redistribute its 5% proportionally across the remaining metrics.
2021
-
2022
- **Transcript health score** (Phase 2N/2Q — supplementary signal, weight 10%): Computed from three Phase 2N/2Q output variables. If `TRANSCRIPT_DATA_AVAILABLE=false`, skip this metric and redistribute its 10% proportionally across the remaining metrics.
2023
-
2024
- ```bash
2025
- if [ "${TRANSCRIPT_DATA_AVAILABLE:-false}" = "true" ]; then
2026
- # stall_rate_score: 0% stall rate = 100, 10%+ stall rate = 0
2027
- # STALL_TURN_RATE is a percentage integer from Phase 2N (e.g., 5 for 5%)
2028
- STALL_TURN_RATE=${STALL_TURN_RATE:-0}
2029
- STALL_RATE_SCORE=$(echo "scale=0; r=$STALL_TURN_RATE; if(r>10) 0 else (10-r)*10" | bc 2>/dev/null || echo 0)
2030
-
2031
- # compaction_score: 0 compactions = 100, 5+ compactions = 0
2032
- # COMPACTION_TOTAL is the total compaction events across all sessions from Phase 2N
2033
- COMPACTION_TOTAL=${COMPACTION_TOTAL:-0}
2034
- COMPACTION_SCORE=$(echo "scale=0; c=$COMPACTION_TOTAL; if(c>5) 0 else (5-c)*20" | bc 2>/dev/null || echo 0)
2035
-
2036
- # model_split_score: 0% opus = 100, 100% opus = 0
2037
- # OPUS_PCT is derived from Phase 2Q's model_distribution output (integer 0-100)
2038
- # If not set (Phase 2Q skipped), use 0 as conservative fallback
2039
- OPUS_PCT=${OPUS_PCT:-0}
2040
- MODEL_SPLIT_SCORE=$(echo "scale=0; 100 - $OPUS_PCT" | bc 2>/dev/null || echo 100)
2041
- [ "$MODEL_SPLIT_SCORE" -lt 0 ] 2>/dev/null && MODEL_SPLIT_SCORE=0
2042
-
2043
- # Weighted composite: stall 50% + compaction 30% + model split 20%
2044
- TRANSCRIPT_HEALTH_SCORE=$(echo "scale=0; ($STALL_RATE_SCORE * 50 + $COMPACTION_SCORE * 30 + $MODEL_SPLIT_SCORE * 20) / 100" | bc 2>/dev/null || echo 0)
2045
- echo "Transcript health score: $TRANSCRIPT_HEALTH_SCORE/100 (stall_rate_score=$STALL_RATE_SCORE compaction_score=$COMPACTION_SCORE model_split_score=$MODEL_SPLIT_SCORE)"
2046
- TRANSCRIPT_SCORE_AVAILABLE=true
2047
- else
2048
- TRANSCRIPT_HEALTH_SCORE=""
2049
- TRANSCRIPT_SCORE_AVAILABLE=false
2050
- echo "Transcript health score: N/A (TRANSCRIPT_DATA_AVAILABLE=false — 10% weight redistributed proportionally)"
2051
- fi
2052
- ```
2053
-
2054
- **Transcript flow qualitative flags** (supplementary, appear in report alongside the numeric score):
2055
- - Stall turn rate > 10%: flag as ⚠️ `High stall rate — agents spending >10% of turns in long waits`
2056
- - Compaction events > 5 in window: flag as ⚠️ `Frequent context compaction — consider reducing prompt or tool output size`
2057
- - Stall turn rate ≤ 10% and compaction ≤ 5: flag as ✅ `Session flow healthy`
2058
- - Data not available: flag as ℹ️ `Transcript flow metrics unavailable (requires Phase 2M session data)`
2059
-
2060
- **Aggregated session metrics** (Phase 2Q): If `$TRANSCRIPT_AGGREGATION_AVAILABLE` is true, embed `$AGGREGATED_TRANSCRIPT_METRICS` verbatim into the \"Aggregated Session Metrics\" section of the Phase 5A report. The guardrail footnotes (5 harmful-takeaway warnings from #347) MUST be included in that section even when session count is low — they protect against misreading the metrics table. If `TRANSCRIPT_AGGREGATION_AVAILABLE` is false, emit the unavailability note from Phase 2Q.
2061
-
2062
- **Anomaly signal**: If Phase 2I anomaly rate exceeds 10% of issues, flag it as a warning in the health report even though it does not feed the score directly — high anomaly rates indicate pipeline instability not captured by outcome metrics.
2063
-
2064
- Score 0-100. Categorize:
2065
- - 90-100: Excellent — pipeline is shipping clean code
2066
- - 70-89: Good — some defect categories need attention
2067
- - 50-69: Needs work — specific pipeline links are underperforming
2068
- - < 50: Critical — systemic issues in the pipeline
2069
-
2070
- **Health score trend**: Compare the computed score against `$PRIOR_HEALTH_SCORE` from Phase 1C:
2071
-
2072
- ```bash
2073
- # Compute score trend string for report header
2074
- if [ -z "$PRIOR_HEALTH_SCORE" ]; then
2075
- SCORE_TREND="(baseline — no prior period)"
2076
- elif [ "$HEALTH_SCORE" -gt "$PRIOR_HEALTH_SCORE" ] 2>/dev/null; then
2077
- SCORE_TREND="(↑ from $PRIOR_HEALTH_SCORE — +$(( HEALTH_SCORE - PRIOR_HEALTH_SCORE )) pts)"
2078
- elif [ "$HEALTH_SCORE" -lt "$PRIOR_HEALTH_SCORE" ] 2>/dev/null; then
2079
- SCORE_TREND="(↓ from $PRIOR_HEALTH_SCORE — -$(( PRIOR_HEALTH_SCORE - HEALTH_SCORE )) pts)"
2080
- else
2081
- SCORE_TREND="(= unchanged from $PRIOR_HEALTH_SCORE)"
2082
- fi
2083
- echo "Health Score: $HEALTH_SCORE/100 $SCORE_TREND"
2084
- ```
2085
-
2086
- ---
2087
-
2088
- ## Phase 4: Generate Improvement Proposals
2089
-
2090
- For each of the top 3 defect categories:
2091
-
2092
- 1. **Diagnose**: Why is the builder producing these defects? Read the relevant section of `work-on.md` — is there a check for this category? Is it too vague? Missing entirely?
2093
-
2094
- 2. **Prescribe**: What specific change to which command file would reduce these defects?
2095
- - If the builder lacks a check → propose adding one to `work-on.md` or `quality-gate.md`
2096
- - If the review agent misses things → propose tightening the agent template in `review-pr-agents.md`
2097
- - If the orchestrator doesn't catch integration failures → propose a new inter-wave check
2098
-
2099
- 3. **Scope**: Estimate the change size (lines added/modified) and risk level.
2100
-
2101
- ---
2102
-
2103
- ## Phase 5: Report & Track
2104
-
2105
- ### 5A: Post health report
2106
-
2107
- Create a detailed health report as a GitHub issue in the Forge repo:
2108
-
2109
- ```bash
2110
- gh issue create -R $FORGE_REPO \
2111
- --title "Pipeline Health: $REPO — $(date +%Y-%m-%d) — Score: [SCORE]/100" \
2112
- --label "health-report" \
2113
- --body "$(cat <<'EOF'
2114
- ## Pipeline Health Report
2115
-
2116
- **Project**: $REPO
2117
- **Period**: $SINCE → $(date +%Y-%m-%dT%H:%M:%SZ)
2118
- **Health Score**: [SCORE]/100 ([CATEGORY]) $SCORE_TREND
2119
-
2120
- ## Metrics
2121
-
2122
- ### Pre-Merge Quality
2123
-
2124
- | Metric | Value | Target | Status |
2125
- |--------|-------|--------|--------|
2126
- | Build pass rate (staging→main) | ?% | 95%+ | ✅/⚠️/❌ |
2127
- | Review findings per PR | ? | < 0.5 | ✅/⚠️/❌ |
2128
- | Manual fix-up rate | ?% | < 15% | ✅/⚠️/❌ |
2129
- | False positive rate | ?% | < 10% | ✅/⚠️/❌ |
2130
- | Pipeline self-correction rate | ?% | < 15% | ✅/⚠️/❌ |
2131
-
2132
- ### Post-Deploy Quality
2133
-
2134
- | Metric | Value | Target | Status |
2135
- |--------|-------|--------|--------|
2136
- | Post-deploy failure rate | ?% | < 5% | ✅/⚠️/❌ |
2137
- | Review escape rate (REVIEW_FALSE_NEG) | ?% | < 2% | ✅/⚠️/❌ |
2138
- | Mean time to detect | ? days | < 7 days | ✅/⚠️/❌ |
2139
- | Audit findings (total in window) | ? | — | ℹ️ |
2140
- | — by failure point | see breakdown | — | ℹ️ |
2141
-
2142
- ### Pipeline Health Signals
2143
-
2144
- | Metric | Value | Target | Status |
2145
- |--------|-------|--------|--------|
2146
- | Issue quality score | ?/100 | > 80 | ✅/⚠️/❌ |
2147
- | — Investigation invalidation rate | ?% | < 15% | ✅/⚠️/❌ |
2148
- | — ISSUE_SPEC audit findings | ? | 0 | ✅/⚠️/❌ |
2149
- | — Re-investigation rate | ? | 0 | ✅/⚠️/❌ |
2150
- | — Issue structure compliance | ?% | > 80% | ✅/⚠️/❌ |
2151
- | Investigation accuracy (trajectory) | ?% | > 85% | ✅/⚠️/❌ |
2152
- | Orchestration efficiency score | ?/100 | > 70 | ✅/⚠️/❌ |
2153
- | — Avg agent idle% | ?% | < 30% | ✅/⚠️/❌ |
2154
- | — Avg resume cycles per agent | ? | < 1 | ✅/⚠️/❌ |
2155
- | — Clean agent rate | ?% | > 60% | ✅/⚠️/❌ |
2156
- | — Batch stall rate | ?% (?/? issues) | < 10% | ✅/⚠️/❌ |
2157
- | — Escalated to needs-human | ? issues | 0 | ✅/⚠️/❌ |
2158
- | Original work ratio | ?% | > 70% | ✅/⚠️/❌ |
2159
- | Milestone merge success | ?% | 80%+ | ✅/⚠️/❌ |
2160
- | Issue close velocity | ? days | < 2 days | ✅/⚠️/❌ |
2161
- | Transcript health score (Phase 2N/2Q) | ?/100 $(_trend $TRANSCRIPT_HEALTH_SCORE $PRIOR_TRANSCRIPT_SCORE) | > 70 | ✅/⚠️/❌/ℹ️ |
2162
- | — Session flow flag | [✅ Healthy / ⚠️ flag / ℹ️ unavailable] | — | ℹ️ |
2163
- | — Aggregated session metrics (Phase 2Q) | [✅ Available / ℹ️ unavailable] | — | ℹ️ |
2164
-
2165
- ## Trajectory Analytics
2166
-
2167
- _From Phase 2I — FORGE:TRAJECTORY comments on [N] closed issues in window_
2168
-
2169
- | Metric | Value |
2170
- |--------|-------|
2171
- | Issues with trajectory data | ? / ? closed |
2172
- | Investigation accuracy | ?% (CONFIRMED: ?, PARTIAL: ?, INVALID: ?) |
2173
- | Task type distribution | Bug Fix: ?, Feature: ?, Refactor: ?, Maintenance: ? |
2174
- | Anomalies flagged | ? issues |
2175
- | Notable anomalies | [list top 3 or "None"] |
2176
-
2177
- _If no trajectory data available in window: "No FORGE:TRAJECTORY comments found — pipeline may be running pre-trajectory version or window is too narrow."_
2178
-
2179
- ## Orchestration Efficiency
2180
-
2181
- _From Phase 2K — persisted FORGE:AUDIT-AGENTS summaries ([N] sessions, [M] agents total) + FORGE:STALL_DETECTED annotations_
2182
-
2183
- | Metric | Value | Target | Status |
2184
- |--------|-------|--------|--------|
2185
- | Avg agent idle% | ?% | < 30% | ✅/⚠️/❌ |
2186
- | Avg resume cycles per agent | ? | < 1 | ✅/⚠️/❌ |
2187
- | Clean agent rate | ?% | > 60% | ✅/⚠️/❌ |
2188
- | Batch stall rate | ?% (?/? issues) | < 10% | ✅/⚠️/❌ |
2189
- | Escalated to needs-human | ? issues | 0 | ✅/⚠️/❌ |
2190
-
2191
- **Top stall boundaries** (phase transitions causing the most stalls):
2192
- | Boundary | Occurrences |
2193
- |----------|-------------|
2194
- | [boundary_1] | [N] |
2195
- | [boundary_2] | [N] |
2196
- | [boundary_3] | [N] |
2197
-
2198
- _If no FORGE:AUDIT-AGENTS data available: "No persisted audit-agents summaries found in window. Run \`audit-agents --persist\` after orchestration runs to enable this metric. Proxy: [N] issues had resume/stall anomalies in trajectory data."_
2199
-
2200
- _Batch stall rate: sourced from \`FORGE:STALL_DETECTED\` comments (requires \`/orchestrate\` Step 4B.5 to be active). N/A if stall detector is not deployed._
2201
-
2202
- ## Transcript Flow Metrics
2203
-
2204
- _From Phase 2N — JSONL transcript analytics ([N] pipeline sessions, [M] subagent sessions in window)_
2205
-
2206
- | Metric | Value | Signal |
2207
- |--------|-------|--------|
2208
- | Pipeline sessions parsed | ? parent, ? subagent | ℹ️ |
2209
- | Total turns with timing | ? | ℹ️ |
2210
- | Avg turn duration | ?s | ℹ️ |
2211
- | Max turn duration | ?s | ℹ️ |
2212
- | Stall turns (>120s) | ? (?%) | ✅/⚠️ |
2213
- | Rate-limit indicator turns (60–120s) | ? | ℹ️ |
2214
- | Context compaction events | ? | ✅/⚠️ |
2215
- | Session flow flag | [✅ Healthy / ⚠️ High stall rate / ⚠️ Frequent compaction] | — |
2216
-
2217
- **Top tools by call frequency**:
2218
- | Tool | Calls |
2219
- |------|-------|
2220
- | [tool_1] | [N] |
2221
- | [tool_2] | [N] |
2222
- | [tool_3] | [N] |
2223
-
2224
- _If transcript data unavailable: "Transcript flow metrics unavailable — Phase 2M found no pipeline sessions in window (SESSION_TOTAL=0). Run `/pipeline-health` on the same machine where Claude Code sessions are stored."_
2225
-
2226
- ## Aggregated Session Metrics
2227
-
2228
- _From Phase 2Q — privacy-safe aggregation of [N] pipeline sessions (session IDs excluded; structural data only)_
2229
-
2230
- | Metric | Value (median) | Range [P25–P75] | Context |
2231
- |--------|---------------|-----------------|---------|
2232
- | Active compute time | ?s | [?s–?s] | Agent tool-call time only — excludes human idle. Target: — (baseline) |
2233
- | Context compaction events | ? | [?–?] | Per session. >2 median = prompts/diffs may be oversized |
2234
- | Sonnet / Opus split | ?% / ?% | — | Opus% = rate-limit fallback pressure |
2235
-
2236
- **Tool call distribution** (top 5 tools, all sessions combined):
2237
- | Tool | Total Calls |
2238
- |------|-------------|
2239
- | [tool_1] | [N] |
2240
- | [tool_2] | [N] |
2241
- | [tool_3] | [N] |
2242
- | [tool_4] | [N] |
2243
- | [tool_5] | [N] |
2244
-
2245
- **Active time by command type** (median):
2246
- | Command | Median duration | Range [P25–P75] | Sessions |
2247
- |---------|-----------------|-----------------|----------|
2248
- | [command_1] | ?s | [?s–?s] | ? |
2249
- | [command_2] | ?s | [?s–?s] | ? |
2250
-
2251
- **Interpretation guardrails** (sourced from investigation #347):
2252
-
2253
- > ⚠️ **Guard 1**: Duration shown is *agent compute time*, not wall-clock. Wall-clock varies 2–10x due to human idle. "Sessions are long" ≠ prompts are bad. Compare against the command-type median, not a global average.
2254
-
2255
- > ⚠️ **Guard 2**: Tool call count is normalized per task type. A 15-file feature legitimately requires 200+ calls; a 1-file fix needs 30–50. Flag only if count per files-changed is ≥3× the command-type median.
2256
-
2257
- > ⚠️ **Guard 3**: Skill invocation count reflects which pipeline commands ran — not whether they ran correctly. High skill count without checking workflow:* label progression is a false signal.
2258
-
2259
- > ⚠️ **Guard 4**: Opus% is rate-limit pressure proxy — high opus% means fallback occurred, not that better models were chosen. Remedy: spread sessions across time or reduce prompt token usage.
2260
-
2261
- > ⚠️ **Guard 5**: Compaction events indicate large context windows, not inefficiency. Flag only if median compaction_count > 2 AND correlated with review-finding quality degradation.
2262
-
2263
- _If aggregated metrics unavailable: "Phase 2Q did not produce data — TRANSCRIPT_DATA_AVAILABLE=false or PER_SESSION_METRICS_RAW was empty. Run `/pipeline-health` on the machine where Claude Code sessions are stored."_
2264
-
2265
- ## Post-Deploy Failure Analysis
2266
-
2267
- _From Phase 2L — \`audit-finding\` issues in the Forge repo ([N] findings in window)_
2268
-
2269
- | Metric | Value | Target |
2270
- |--------|-------|--------|
2271
- | Post-deploy failure rate | ?% | < 5% |
2272
- | Review escape rate | ?% | < 2% |
2273
- | Mean time to detect | ? days | < 7 days |
2274
-
2275
- **Failure point distribution** ([N] total audit findings):
2276
- | Failure Point | Count | % | Meaning |
2277
- |---------------|-------|---|---------|
2278
- | REVIEW_FALSE_NEG | ? | ?% | Review agent saw area but approved defect |
2279
- | IMPLEMENTATION | ? | ?% | Code written incorrectly |
2280
- | INVESTIGATION | ? | ?% | Wrong verdict or missed scope |
2281
- | DEPLOY_GATE | ? | ?% | Deploy-info missed risk signal |
2282
- | [other] | ? | ?% | — |
2283
-
2284
- _If no audit findings in window: "No \`audit-finding\` issues in window — post-deploy failure rate = 0%."_
2285
-
2286
- ## Defect Category Breakdown
2287
-
2288
- _Pre-merge review findings by agent prefix (from Phase 3A-I). Trends computed vs prior report #$PRIOR_NUMBER (or "—" if baseline run)._
2289
-
2290
- | Category | Count | % of Total | Trend vs Prior Period |
2291
- |----------|-------|------------|---------------------|
2292
- | SEC | $COUNT_SEC | $(_pct $COUNT_SEC)% | $TREND_SEC |
2293
- | AUTH | $COUNT_AUTH | $(_pct $COUNT_AUTH)% | $TREND_AUTH |
2294
- | BILL | $COUNT_BILL | $(_pct $COUNT_BILL)% | $TREND_BILL |
2295
- | DB | $COUNT_DB | $(_pct $COUNT_DB)% | $TREND_DB |
2296
- | FE | $COUNT_FE | $(_pct $COUNT_FE)% | $TREND_FE |
2297
- | INFRA | $COUNT_INFRA | $(_pct $COUNT_INFRA)% | $TREND_INFRA |
2298
- | API | $COUNT_API | $(_pct $COUNT_API)% | $TREND_API |
2299
- | CONC | $COUNT_CONC | $(_pct $COUNT_CONC)% | $TREND_CONC |
2300
- | SCRP | $COUNT_SCRP | $(_pct $COUNT_SCRP)% | $TREND_SCRP |
2301
- | ISSUE_SPEC | $COUNT_ISSUE_SPEC | $(_pct $COUNT_ISSUE_SPEC)% | $TREND_ISSUE_SPEC |
2302
-
2303
- _Post-deploy audit findings by failure point — see Phase 3A-II table above._
2304
-
2305
- ## Prompt Change Impact
2306
-
2307
- [Analysis from Phase 3B]
2308
-
2309
- ## Top 3 Improvement Proposals
2310
-
2311
- ### 1. [Category] — [Proposal title]
2312
- **File**: `commands/[file].md`
2313
- **Change**: [Description]
2314
- **Expected impact**: [Metric] should improve by ~[X]%
2315
- **Risk**: [Low/Medium/High]
2316
-
2317
- ### 2. ...
2318
-
2319
- ### 3. ...
2320
-
2321
- ## Forge Commits in Window
2322
-
2323
- [Git log from Phase 2A]
2324
- EOF
2325
- )"
2326
- ```
2327
-
2328
- ### 5B: Create improvement issues in Forge
2329
-
2330
- For each improvement proposal, create a trackable issue using the `/issue` structure.
2331
-
2332
- **Before creating**: Read the target file (`commands/[file].md`) and verify that the section reference in the issue body is accurate — confirm the section heading and line range exist.
2333
-
2334
- **Title format**: Use conventional commit prefix based on proposal type:
2335
- - Proposed fix for a broken behaviour → `fix([command]): [description]`
2336
- - New capability or enhancement → `feat([command]): [description]`
2337
- - Restructuring without behaviour change → `refactor([command]): [description]`
2338
-
2339
- **Priority label**: Assign based on how far the metric deviates from its target:
2340
- - Metric ≥ 20% below target → `P1`
2341
- - Metric 10–20% below target → `P2`
2342
- - Metric < 10% below target → `P3`
2343
-
2344
- ```bash
2345
- gh issue create -R $FORGE_REPO \
2346
- --title "fix([command]): [description]" \
2347
- --label "[bug|enhancement|feature],[P1|P2|P3]" \
2348
- --body "## Problem
2349
-
2350
- [1–3 sentences describing what the pipeline health metric reveals. Include the current value and target value.]
2351
-
2352
- **Source**: Health report #[REPORT_NUMBER] — [metric]: [current value] (target: [target value])
2353
-
2354
- ## Root Cause (if known)
2355
-
2356
- \`commands/[file].md\` [section] — [what the section currently does that causes the metric to suffer]
2357
-
2358
- ## Affected Files
2359
-
2360
- Files that need changes:
2361
-
2362
- 1. \`commands/[file].md\` — [what needs to change in this file]
2363
-
2364
- ## Expected Behavior
2365
-
2366
- After the fix, [metric] should reach [target value]. [Describe the concrete behaviour change in the command.]
2367
-
2368
- ## Acceptance Criteria
2369
-
2370
- - [ ] [Specific, testable criterion tied to the metric]
2371
- - [ ] [Specific, testable criterion tied to the command behaviour]
2372
- - [ ] No regression in other pipeline-health metrics
2373
- "
2374
- ```
2375
-
2376
- ---
2377
-
2378
- ## Phase 6: Summary
2379
-
2380
- Print to the user:
2381
-
2382
- ```
2383
- Forge Pipeline Health — [REPO]
2384
- Score: [SCORE]/100 ([CATEGORY]) $SCORE_TREND
2385
-
2386
- Pre-merge quality:
2387
- Build pass rate: [X]% (target 95%)
2388
- Findings per PR: [X] (target < 0.5)
2389
- False positive rate: [X]% (target < 10%)
2390
- Manual fix-up rate: [X]% (target < 15%)
2391
- Self-correction rate: [X]% (target < 15%)
2392
-
2393
- Post-deploy quality:
2394
- Post-deploy failure rate: [X]% (target < 5%) — [N] audit findings
2395
- Review escape rate: [X]% (target < 2%) — [N] REVIEW_FALSE_NEG
2396
- Mean time to detect: [X] days (target < 7 days)
2397
- Top failure points: [FP1] ([N]), [FP2] ([N]), [FP3] ([N])
2398
-
2399
- Pipeline health signals:
2400
- Issue quality score: [X]/100 (target > 80)
2401
- Invalidation rate: [X]% (target < 15%)
2402
- ISSUE_SPEC findings: [X] (target 0)
2403
- Re-investigations: [X] (target 0)
2404
- Structure compliance: [X]% (target > 80%)
2405
- Investigation accuracy: [X]% (target > 85%) — [N] issues with trajectory data
2406
- CONFIRMED: [N] | PARTIAL: [N] | INVALID: [N]
2407
- Anomalies flagged: [N] issues
2408
- Task type distribution: Bug Fix: [N], Feature: [N], Refactor: [N], Maintenance: [N]
2409
- Orchestration efficiency: [X]/100 (target > 70) — [N] sessions, [M] agents
2410
- Avg idle%: [X]% (target < 30%)
2411
- Avg resume cycles: [X] (target < 1)
2412
- Clean agent rate: [X]% (target > 60%)
2413
- Top stall boundary: [boundary] ([N] occurrences)
2414
- Original work ratio: [X]% (target > 70%)
2415
- Milestone merge rate: [X]% (target 80%)
2416
-
2417
- Top defect categories: [CAT1] ([N]), [CAT2] ([N]), [CAT3] ([N])
2418
-
2419
- Proposed improvements: [N] issues created in $FORGE_REPO
2420
- Health report: [ISSUE_URL]
2421
- ```
44
+ The command reads only the phase file(s) relevant to the current step rather than loading
45
+ the full 2500-line monolith upfront.