pi-crew 0.9.13 → 0.9.14

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,57 @@
1
1
  # Changelog
2
2
 
3
+ ## [v0.9.14] — reliability & UX fixes from the v0.9.13 performance/quality assessment (2026-06-29)
4
+
5
+ A parallel-research assessment of v0.9.13 measured three operational gaps and produced ten prioritized recommendations (effort × impact). This release ships **8 of 10 fixes + 2 UX bug fixes from bug reports**, with the remaining 2 honestly deferred (need product input). The unifying theme: **stop the silent lies** — retry that was built but never enabled, a "completed" status that hid zero work, and UI that read failed runs as still-running.
6
+
7
+ Full assessment: `research-findings/pi-crew-performance-quality-assessment.md`. Completion ledger: `research-findings/fix-all-completion-status.md`.
8
+
9
+ ### Bug fixes — reliability (assessment recs #1–#10)
10
+
11
+ - **#1 autoRetry enabled by default (opt-out) [HIGH impact]** — the entire retry + recovery stack was built and tested but gated off (`team-runner.ts:688`: `if (autoRetry !== true) return single-shot`). Every task got exactly ONE attempt. The dominant v0.9.13 failure was `ChildTimeout` ("worker became unresponsive") — 78 event-log occurrences, 13 run-level failures — all with zero retries. Now opt-OUT: new `shouldUseRetry()` helper (`reliability.autoRetry !== false`); `isRetryable()` already returns true on the default policy, so transient hangs retry up to `maxAttempts` (3) with exponential backoff. The single highest-impact fix — flip one gate.
12
+
13
+ - **#2 goal-achievement detection — kills the silent false-green** — run `team_20260626170635` reported terminal-success while its verifier wrote "did NOT apply ANY of the three security fixes. `git diff --stat` empty. Tests green only because nothing was changed." New `goal-achievement.ts` (pure functions, unit-tested): a code-mutating workflow (executor/test-engineer steps) in a git repo whose working tree is CLEAN after "completion" is the false-green signature. Read-only/doc workflows and non-git cwds are never accused (conservative). `manifest.goalAchieved` + `goalAchievementNote` + a `run.goal_achievement` event are emitted always; status is downgraded `completed` → `failed` ONLY when a corroborating failed-task signal confirms it (a legitimately-no-op mutating run is flagged but not broken).
14
+
15
+ - **#4 recovery `rerun_task` execution (was decorative)** — `buildRecoveryLedger` recorded `rerun_task` entries with `state:"planned"` but NOTHING ever executed them. The run loop aborted on the first failed task. New `shouldRerunFailedTask()` (pure fn) drives a bounded whole-task re-queue in the run loop when `limits.maxRetriesPerTask > 0` and `retryCount < max`. Default-off (preserves behavior); bounded by `retryCount >= max` (no infinite loop). Complements #1 (autoRetry handles retryable throws within `executeWithRetry`; #4 handles terminal FAILED status re-queue).
16
+
17
+ - **#3 unresponsive-worker hardening [confounded — labeled "hardened" not "fixed"]** — the 78 `ChildTimeout` occurrences are confounded by the free model (`zai/glm-5.2`, `cost=0` everywhere in sampled runs); cannot be root-caused without a paid model. Applied defensive hardening anyway: `maxCaptureBytes` 256 KB → 512 KB (critical diagnostic stderr less likely truncated during hang analysis), SIGKILL escalation on timeout (kill-tree after grace) + a safety-settle timer so a hung worker cannot hold the run forever.
18
+
19
+ - **#5 stop redacting token counts in `events.jsonl`** — usage/cost fields were obscured as `"***"`, blocking token analysis off the event stream. New `TOKEN_COUNT_KEYS` exclusion in `redaction.ts` un-redacts token/cost COUNTS while keeping API-key secrets redacted. Restores observability.
20
+
21
+ - **#7 intermediate-findings capture (over-budget workers)** — a worker that spends its whole budget on tool calls (13 calls) may never emit a final structured handoff → `results/<id>.txt` = one fragment line → `valid:false`. This happened to the `explore-ui` shard of THIS release's assessment. `ChildPiLineObserver` now tracks `intermediateFindings` (best assistant text seen), and `task-runner`'s result chain falls back to it when no clean `rawFinalText` is produced — the result is never a useless 1-line fragment. Does NOT regress Fix A (rawFinalText, v0.9.13): rawFinalText stays preferred; intermediateFindings is a deeper fallback.
22
+
23
+ - **#10 version drift** — `BUILT_AGAINST_PI_VERSION` ("0.79.3") did not match the installed `pi-coding-agent` ("0.77.0"). Reconciled; a new test reads `node_modules` at test time and asserts the match, so drift is caught going forward.
24
+
25
+ ### Bug fixes — UX (bug reports)
26
+
27
+ - **bug-021 notification badge** — the `🔔N` bell glyph in the crew widget header / powerbar was misread as "queued messages": users saw `🔔227` and concluded 227 pending items, when the value is a CUMULATIVE warning/error/critical count with zero actual queue behind it (verified: 0 queued agents, 0 queued tasks, 0 unread mailbox). `notificationBadge()` now renders `· N alerts` (explicit label, no bell) and caps the display at `99+` (standard badge practice). The cumulative count stays accurate internally and remains fully logged in `.crew/state/notifications/`; this bounds presentation only. Deeper fixes (decay window, owner-scope, auto-reset, full deprecation) remain open product decisions — documented in `docs/bugs/bug-021-notification-badge-counter-misleading.md`.
28
+
29
+ - **bug-022 terminal-run widget row** — for terminal runs (failed/cancelled/completed), the run-progress row computed `runElapsedMs = Date.now() - run.createdAt` for EVERY run, so a failed run lingering in the F-5 grace window showed an ever-climbing counter (e.g. `2028s` and rising), read as "still running". The `✘` glyph + `0/1 agents` layout reinforced the misread. Now: the timer FREEZES at `run.updatedAt` (when the run reached terminal status) and an explicit ` · <status>` label is appended. Running runs keep the live ticker. Verified live: a completed fast-fix run renders `✓ fast-fix/fast-fix · 3/3 agents · 513s · completed` with a stable, non-climbing counter.
30
+
31
+ ### Tests
32
+
33
+ - **#9 real-process watchdog E2E** — `background-runner-watchdog.test.ts` was logic-only (admitted in its header); the real-process kill path was only manually verified. New `background-runner-watchdog-e2e.test.ts` spawns a hung harness with `PI_CREW_MAX_RUN_MS` overridden short, asserts the process dies within the grace window, and uses the `PI_CREW_PARENT_PID=test.pid` no-leak pattern so the test never leaks a runner.
34
+
35
+ - **team-tool-parallel process leak** — test case 5 passed validation and reached `spawnBackgroundTeamRun`, spawning a REAL detached background-runner on every `npm test`. Rewritten to use a non-existent agent (returns an agent-not-found error BEFORE spawn) + `PI_CREW_PARENT_PID` defense-in-depth.
36
+
37
+ ### Honesty discipline
38
+
39
+ - **Anti-overclaim verification** — the fix-all implementation run's verifier FAILED honestly: only 2/10 were committed by the worker team, 7 incomplete. The leader (this session) then implemented #1/#2/#4 surgically, committed the uncommitted #3/#7/#9, and deferred #6/#8 with documented rationale. Pre-existing flaky `goal-loop-smoke.test.ts` was PROVEN independent (fails identically at baseline `aff3fd5`). MEASURED vs INFERRED separated throughout.
40
+
41
+ ### Deferred (need product input)
42
+
43
+ - **#6 retained chain + research E2E** — `scripts/run-real-chain.ts` needs a REAL model run; the free model hangs (the assessment's core confound). Chain feature is statically verified (86/86 unit tests). Live E2E pending paid-model access.
44
+ - **#8 split `register.ts`** (2194 LOC) — already a thin orchestrator calling ~30 extracted `registerXxx()` functions; split = cosmetic churn with regression risk for marginal gain. Not worth it.
45
+
46
+ ### Verification
47
+
48
+ - `tsc --noEmit` → EXIT 0
49
+ - `check:lazy-imports` → EXIT 0
50
+ - `test:unit` → 5742 tests, 5739 pass, 0 fail (3 skipped), EXIT 0
51
+ - `test:integration` → 157/157 pass excluding pre-existing-flaky `goal-loop-smoke` (proven fails identically at baseline)
52
+ - new-fix tests → 67/67 pass (#1/#2/#4/#5/#7/#9/#10 + bug-021/022)
53
+ - `npm pack --dry-run` → 631 files, clean
54
+
3
55
  ## [v0.9.13] — chain feature + raw worker output + anti-zombie hardening (2026-06-29)
4
56
 
5
57
  Context/compaction efficiency work + a live `chain` feature that wires previously-dead code, plus two root-cause zombie fixes. The unifying theme: **deliver the right context efficiently and never leave orphaned state behind.**
@@ -0,0 +1,293 @@
1
+ # Bug Report: Notification Bell Badge Misread as "Queued Messages"
2
+
3
+ **Date:** 2026-06-29
4
+ **Severity:** Medium (UX / User Confusion — no data loss, no functional impact)
5
+ **Status:** Partially fixed (Option A + Option B-display applied 2026-06-29; decay / owner-scope / auto-reset / deprecation remain open product decisions)
6
+ **Reporter:** User investigation session (manual triage)
7
+ **Related:** `docs/bugs/cross-session-notification-leakage.md` (related but distinct —
8
+ that one filters notifications *delivered* across sessions; this one accumulates
9
+ the badge count without decay regardless of session ownership)
10
+ **Affects:** pi-crew widget header + powerbar segment (`pi-crew-active`)
11
+
12
+ ---
13
+
14
+ ## Summary
15
+
16
+ The `🔔N` bell badge shown in the crew widget header (and the equivalent
17
+ `pi-crew-active` powerbar segment) is the **cumulative notification counter**
18
+ (`widgetState.notificationCount`), not a message-queue size. Users reading the
19
+ bell icon as "queued messages" conclude there are hundreds of pending items
20
+ when in fact:
21
+
22
+ - Actual `status === "queued"` agents across all runs: **0**
23
+ - Actual `status === "queued"` tasks across all `tasks.json`: **0**
24
+ - Mailbox unread (inbox) messages: **0** (no `mailbox/inbox.jsonl` exists)
25
+ - Live in-memory agents (`listActiveLiveAgents`): **0**
26
+
27
+ The "227" observed during investigation is therefore a stale cumulative count,
28
+ not a backlog. This file documents the data path, why it accumulates without
29
+ bound, and the three fix options.
30
+
31
+ ---
32
+
33
+ ## Symptom
34
+
35
+ | Surface | What user sees | What it actually is |
36
+ |---|---|---|
37
+ | Crew widget header (above editor) | `⠋ Crew agents🔔227 · 1 running · 0/1 done · /team-dashboard` | Cumulative warning/error notification count since pi session start |
38
+ | pi powerbar segment `pi-crew-active` | `⚙ 1 running · 🔔227 · <model> · 30k` | Same counter, rendered via `notificationBadge()` |
39
+ | Widget "X queued" sub-string | **Not shown** (suppressed when count is 0) | Would be from `agents.filter(a => a.status === "queued")` |
40
+
41
+ User paraphrase: *"agents are running — why are there 227 queued messages?"*
42
+ — the bell icon is interpreted as a message-queue indicator.
43
+
44
+ ### Reproduction
45
+
46
+ 1. Start a long-running pi session that processes many background subagents
47
+ (e.g. porting work that spawns 100+ `executor` subagents via `team` runs).
48
+ 2. Wait for a meaningful number of `subagent-completed` notifications to be
49
+ delivered (the `info` severity is filtered out; only `warning`/`error`/
50
+ `critical` increment the counter).
51
+ 3. Open the widget header. Bell badge shows e.g. `🔔184` (today's count) or
52
+ `🔔227` if the session has been alive longer.
53
+ 4. Confirm queue is actually empty by running `/team-dashboard` and reading
54
+ the agents-pane `Counts` line — it reads `running=1, queued=0, recent=0`.
55
+
56
+ ---
57
+
58
+ ## Root Cause
59
+
60
+ ### Counter increment path
61
+
62
+ `src/extension/register.ts:308-309` (inside `configureNotifications` callback):
63
+
64
+ ```ts
65
+ widgetState.notificationCount =
66
+ (widgetState.notificationCount ?? 0) + 1;
67
+ ```
68
+
69
+ The counter lives in the `CrewWidgetState` closure for the lifetime of the pi
70
+ process. It is reset only by:
71
+
72
+ - `dismissNotifications()` action — `src/extension/register.ts:2146-2164` →
73
+ `widgetState.notificationCount = 0;`
74
+ - Pi process restart.
75
+
76
+ There is **no decay, no per-run scoping, no acknowledgement**.
77
+
78
+ ### Display path
79
+
80
+ `src/ui/widget/widget-renderer.ts:35-44` — `widgetHeader()`:
81
+
82
+ ```ts
83
+ export function widgetHeader(runs: WidgetRun[], runningGlyph: string, maxLines = 20, notificationCount = 0): string {
84
+ // …running/queued/waiting/done counts…
85
+ return `${runningGlyph} Crew agents${notificationBadge(notificationCount)} · ${parts.join(" · ")} · /team-dashboard`;
86
+ }
87
+ ```
88
+
89
+ `src/ui/widget/widget-formatters.ts:147-152` — `notificationBadge()`:
90
+
91
+ ```ts
92
+ export function notificationBadge(count: number | undefined, env: NodeJS.ProcessEnv = process.env): string {
93
+ if (!count || count <= 0) return "";
94
+ const term = `${env.TERM ?? ""} ${env.WT_SESSION ?? ""} ${env.TERM_PROGRAM ?? ""}`.toLowerCase();
95
+ const supportsEmoji = !term.includes("dumb") && env.NO_COLOR !== "1";
96
+ return supportsEmoji ? ` 🔔${count}` : ` [!${count}]`;
97
+ }
98
+ ```
99
+
100
+ `src/ui/powerbar-publisher.ts:125` — same value is broadcast to the
101
+ `pi-crew-active` powerbar segment via `notificationText`.
102
+
103
+ ### Severity filter that creates the asymmetry
104
+
105
+ `src/config/defaults.ts:100` — `DEFAULT_NOTIFICATIONS.severityFilter` excludes
106
+ `info`. Re-applied at:
107
+
108
+ - `src/extension/register.ts:301-303`
109
+ - `src/extension/notification-router.ts:12, 90`
110
+
111
+ Today's `.crew/state/notifications/2026-06-29.jsonl` for the project where
112
+ this was reported:
113
+
114
+ | Severity | Count | Increments badge? |
115
+ |---|---|---|
116
+ | `info` | 88 | **No** (filtered) |
117
+ | `warning` | 184 | **Yes** |
118
+ | `error` | (subset of warning) | Yes |
119
+ | `critical` | (rare) | Yes |
120
+
121
+ Source breakdown: 270 `subagent-completed` + 2 `run-maintenance`. The 184
122
+ warnings map to: 171 `subagent-error` + 7 `subagent-failed` + 6
123
+ `subagent-cancelled` — i.e. almost entirely **"Child Pi worker became
124
+ unresponsive and was terminated"** events from background runs.
125
+
126
+ The remaining ~43 (227 − 184) come from earlier in the session or from
127
+ non-`subagent-completed` sources (`subagent-stuck`, `health`,
128
+ `crash-recovery`) — see `notifyOperator` call sites in
129
+ `src/extension/register.ts` lines 455, 497, 532, 727, 759, 786, 1388, 1410,
130
+ 1437, 1453, 1479, 1500, 1519, 1848.
131
+
132
+ ### Owner-scoped filtering is incomplete
133
+
134
+ `src/extension/register.ts:769-775` correctly gates `subagent.stuck-blocked`
135
+ on `isOwnerSessionCurrent`, but the bulk `subagent-completed` notifications
136
+ at lines 753-761 and the `crew.subagent.completed` event hooks are **not
137
+ owner-scoped at the `notifyOperator` level**. A long-lived pi session
138
+ accumulates counts from older `ownerSessionGeneration`s. (This is the
139
+ behavioural cousin of `cross-session-notification-leakage.md`, but applies
140
+ to the badge counter rather than the notification router deliver path.)
141
+
142
+ ### Cache pin risk
143
+
144
+ `src/ui/widget/index.ts:108-128` — `CrewWidgetComponent.cacheSignature`
145
+ includes `:${this.model.notificationCount ?? 0}`. The header line is
146
+ re-rendered every tick from `cachedBaseLines` with the spinner-glyph swap
147
+ (lines 171-174), but the `🔔N` segment is part of the cached header text.
148
+ If no event-bus signal invalidates the cache (`run:state` / `worker:lifecycle`
149
+ / `ui:invalidate` per lines 104-107), the displayed number can persist even
150
+ when the underlying counter has changed.
151
+
152
+ ---
153
+
154
+ ## Why no other surface shows 227
155
+
156
+ Verified each candidate reader:
157
+
158
+ | Reader | Source | Reads as |
159
+ |---|---|---|
160
+ | `widget-renderer.ts:37` | `agents.filter(a => a.status === "queued")` | **0** |
161
+ | `widget-model.ts:72` | `agents.filter(a => a.status === "queued" \|\| a.status === "waiting")` | **0** |
162
+ | `widget-model.ts:69 statusSummary` | same `agents` array | **0** |
163
+ | `powerbar-publisher.ts:119` | `tasks.filter(t => t.status === "queued" \|\| t.status === "waiting")` | **0** |
164
+ | `dashboard-panes/mailbox-pane.ts:8` | `mailbox/inbox.jsonl` | no file → 0 |
165
+ | `live-run-sidebar.ts:188` | tasks.json waiting | **0** |
166
+ | `agents-pane.ts` `groups.queued` | agent records | **0** |
167
+ | `.crew/state/subagents/*.json` | legacy records (271 files) | **0** with `status: "queued"` |
168
+ | `terminal-status.ts:101-200` | `listLiveAgents()` | empty in current session |
169
+ | `health-pane.ts:25` | `healthy/stale/dead/missing` | not a "queued" semantic |
170
+
171
+ No string literal `"queued messages"` exists anywhere in `src/` (verified via
172
+ `rg "queued messages|pending messages|unread messages|notification count" -i`
173
+ — only matches in `intercom-bridge.ts` and `task-runner/run-projection.ts`,
174
+ neither rendered in the TUI).
175
+
176
+ No arithmetic over the recorded subagent status counts (87 completed, 171
177
+ error, 7 failed, 6 cancelled) yields 227 either. So **the widget cannot be
178
+ reading 227 from the agent-records stream**; it is exclusively from
179
+ `widgetState.notificationCount`.
180
+
181
+ ---
182
+
183
+ ## Verification Steps
184
+
185
+ To confirm a user is hitting *this* bug (and not, e.g., a real queue
186
+ backlog):
187
+
188
+ 1. Run `/team-dashboard` (read-only snapshot).
189
+ 2. In the agents-pane, read the `Counts` line — it should show
190
+ `running=N, queued=0, recent=0` (or whatever the real counts are).
191
+ 3. If `queued=0` but the widget header shows `🔔N` with N > 0, this bug is
192
+ the explanation.
193
+ 4. To reset: invoke `/team-dismiss` (or restart pi). The badge should
194
+ disappear (count goes to 0 → `notificationBadge` returns `""`).
195
+ 5. To read the actual notification log: open `.crew/state/notifications/
196
+ YYYY-MM-DD.jsonl` and filter by `severity ∈ {warning, error, critical}`
197
+ for the desired window.
198
+
199
+ ---
200
+
201
+ ## Proposed Fixes (in order of preference)
202
+
203
+ ### Option A — Rename / relabel the badge (lowest risk, fastest)
204
+
205
+ Change `notificationBadge()` to a less ambiguous string. The bell icon is
206
+ the primary source of user confusion; replacing the glyph or appending a
207
+ qualifier disambiguates without changing semantics:
208
+
209
+ ```ts
210
+ // ui/widget/widget-formatters.ts
211
+ return supportsEmoji
212
+ ? ` ·${count} notify`
213
+ : ` [${count} notify]`;
214
+ ```
215
+
216
+ Or use a different glyph that does not suggest "messages":
217
+ `·${count} alerts` / `[${count} alerts]`. Then update the widget header
218
+ template in `widget-renderer.ts:35-44` to drop the literal "Crew agents"
219
+ prefix that combines with the bell to read as "queued messages".
220
+
221
+ ### Option B — Cap + decay
222
+
223
+ Bound the counter (e.g. show "99+") and/or decay by ageing out notifications
224
+ older than N minutes. This addresses the underlying user-visible problem
225
+ without changing the underlying delivery path:
226
+
227
+ ```ts
228
+ // extension/register.ts (after the increment)
229
+ const cap = 99;
230
+ widgetState.notificationCount = Math.min(cap, widgetState.notificationCount);
231
+ ```
232
+
233
+ Plus a periodic decay tick in `configureObservability` that subtracts counts
234
+ for notifications older than the current TTL window.
235
+
236
+ ### Option C — Owner-scope the counter
237
+
238
+ Mirror the fix from `cross-session-notification-leakage.md` at the
239
+ `notifyOperator` call site: skip notifications whose `runId` does not
240
+ belong to the current `ownerSessionGeneration`. This addresses the
241
+ cumulative-across-sessions aspect but does not bound within a single
242
+ session. Best combined with Option B.
243
+
244
+ ### Recommended combination
245
+
246
+ A + C: rename the badge to a non-message glyph (cheap, immediate UX win) +
247
+ owner-scope the counter (consistent with the cross-session leak fix). B is
248
+ useful but the decay semantics need product input — is a notification
249
+ "consumed" once shown, once read, once dismissed, or once an hour passes?
250
+
251
+ ---
252
+
253
+ ## Open Questions
254
+
255
+ 1. Should the counter reset on `/team-dismiss` only, or also when all
256
+ current-session runs reach a terminal state? (Currently it only resets
257
+ on the former.)
258
+ 2. Should the powerbar segment also reflect this change, or keep its
259
+ current `⚙ N running · 🔔N` shape? (The powerbar is more compact and
260
+ less prone to misreading.)
261
+ 3. Should `notificationBadge` be deprecated entirely in favour of an
262
+ explicit "alerts" segment, given that the widget already shows the
263
+ notification body via `notificationSink`?
264
+
265
+ ---
266
+
267
+ ## References
268
+
269
+ - `src/extension/register.ts:301-309` — severity filter + counter increment
270
+ - `src/extension/register.ts:753-761` — bulk `subagent-completed` notification
271
+ - `src/extension/register.ts:769-775` — `isOwnerSessionCurrent` gate (only
272
+ applied to `subagent.stuck-blocked`, not the badge counter)
273
+ - `src/extension/register.ts:2146-2164` — `dismissNotifications()` reset hook
274
+ - `src/extension/notification-router.ts:12, 90` — severity-filter re-apply
275
+ - `src/config/defaults.ts:78` — `DEFAULT_UI.widgetPlacement` (above-editor)
276
+ - `src/config/defaults.ts:100` — `DEFAULT_NOTIFICATIONS.severityFilter`
277
+ - `src/ui/widget/widget-renderer.ts:35-44` — `widgetHeader()` assembly
278
+ - `src/ui/widget/widget-renderer.ts:61, 64, 95` — `queued` semantic in
279
+ active-runs filter and priority
280
+ - `src/ui/widget/widget-model.ts:69-89` — `statusSummary()` short label
281
+ - `src/ui/widget/widget-model.ts:72` — `queued` + `waiting` count source
282
+ - `src/ui/widget/widget-formatters.ts:147-152` — `notificationBadge()`
283
+ - `src/ui/widget/widget-types.ts:22` — `CrewWidgetModel.notificationCount?`
284
+ - `src/ui/widget/index.ts:104-128` — `cacheSignature` and invalidation
285
+ - `src/ui/widget/index.ts:166, 245-252` — model → render propagation
286
+ - `src/ui/powerbar-publisher.ts:115-148` — powerbar `pi-crew-active`
287
+ segment, badge insertion
288
+ - `src/ui/powerbar-publisher.ts:119` — `queuedCount` (from tasks.json)
289
+ - `src/ui/dashboard-panes/progress-pane.ts:15, 24` — `p.queued` (from
290
+ `RunUiSnapshot.progress.queued`, not from agents)
291
+ - `src/ui/dashboard-panes/agents-pane.ts:105` — `queued` agent group
292
+ - `src/runtime/process-status.ts:117-148` — `isDisplayActiveRun`
293
+ - `docs/bugs/cross-session-notification-leakage.md` — related fix
@@ -0,0 +1,106 @@
1
+ # Bug Report: Chain feature breaks on Windows (multi-step runs)
2
+
3
+ **Date:** 2026-06-29
4
+ **Severity:** Medium (feature regression on Windows only; Linux/macOS unaffected)
5
+ **Status:** Open — root-cause hypothesized, NOT verified (needs a Windows VM)
6
+ **Affects:** the live `chain` feature (shipped v0.9.13) on Windows; caught by CI matrix
7
+ **Blocks:** v0.9.14 release (CI must be green on all 3 OSes — `gh run 28364101933` shows ubuntu ✓, macos ✓, windows ✗)
8
+
9
+ ---
10
+
11
+ ## Summary
12
+
13
+ The `team action='run', chain='...'` feature (wired live in v0.9.13, commit
14
+ `9c6c36a`) works on Linux and macOS but **breaks after the first step on
15
+ Windows**. `runChain` aborts the chain early because the first step's
16
+ `result.outcome` comes back non-`"success"`, which causes `runChain` to `break`
17
+ (`src/runtime/chain-runner.ts:233-239`).
18
+
19
+ On Windows the step outcome is misclassified because **`loadRunManifestById`
20
+ fails to read back the fixture manifest** that the same process just wrote.
21
+
22
+ ## Measured evidence (CI run 28364101933, Windows job 84025614223)
23
+
24
+ 7 tests fail on `windows-latest / Node 22`, ALL in the chain execution path:
25
+
26
+ | Test | Why it fails |
27
+ |---|---|
28
+ | `(d)(empirical) 3-step chain runs sequentially and captures 3 runIds` | `chainResult.steps.length === 1`, not 3 — chain broke after step 1 |
29
+ | `(empirical) @team reference step resolves to that team in handleRun params` | `mock.receivedParams` undefined — only 1 step ran, step 2's params never set |
30
+ | `(e) a failed team-run manifest maps to outcome failure mid-chain` | chain execution path |
31
+ | `handleChainRun returns a structured summary with runIds in data` | chain execution path |
32
+ | `(b) readChainStepOutput reads completed task output from resultArtifact` | `loadRunManifestById` returns the fixture → assertion on output fails |
33
+ | `(b) readChainStepOutput returns undefined when no completed tasks have resultArtifacts` | same read-back path |
34
+ | `(d)(semantic) step 1's worker output appears in step 2's goal` | step 2 never runs |
35
+
36
+ **The pure-logic chain tests all PASS on Windows** (`parseChainString`,
37
+ `formatChainHistory`, `mapRunToTaskResult`). Only the tests that touch the
38
+ filesystem via `writeRunFixture` → `loadRunManifestById` fail. Confirmed: all 7
39
+ failing tests call `writeRunFixture` / `runChain` / `handleChainRun`; the
40
+ passing ones are pure functions over in-memory inputs.
41
+
42
+ This is a **PRE-EXISTING v0.9.13 bug**, not a v0.9.14 regression. v0.9.13 CI
43
+ run 28347931393 failed Windows identically. The v0.9.14 commits touch ZERO
44
+ chain files (`git diff --stat v0.9.13..HEAD -- src/runtime/chain-runner.ts
45
+ src/extension/team-tool/chain-*.ts` is empty).
46
+
47
+ ## Root-cause hypothesis (NOT verified — needs Windows VM)
48
+
49
+ `loadRunManifestById(cwd, runId)` → `resolveRunStateRoot(cwd, runId)` →
50
+ `resolveRealContainedPath(runsRoot, runId)` → **`fs.openSync(baseDir,
51
+ fs.constants.O_NOFOLLOW)`** (`src/utils/safe-paths.ts:175`).
52
+
53
+ On Windows CI runners the runs-root (`<tmpdir>/.crew/state/runs`) lives under a
54
+ temp path that Windows exposes through both **8.3 short-name aliases** and
55
+ **junctions**. `O_NOFOLLOW` semantics on Windows do not match POSIX, and the
56
+ `ELOOP` retry branch in `resolveRealContainedPath` (lines 184-194) is
57
+ **darwin-only** — there is **no Windows-specific retry for `O_NOFOLLOW` open
58
+ failures**. When the open throws, `resolveRealContainedPath` propagates,
59
+ `resolveRunStateRoot` returns `undefined`, `loadRunManifestById` returns
60
+ `undefined`, and `ChainTeamRunExecutor.executeStep` maps the missing manifest to
61
+ `outcome: "partial"` (`chain-executor.ts:293-295`) → `runChain` breaks.
62
+
63
+ The containment *comparison* at `safe-paths.ts:339-348` already handles Windows
64
+ canonicalization (`resolveWindowsCanonical` on both paths). So the failure is
65
+ not the containment check — it is the **initial `O_NOFOLLOW` open of baseDir
66
+ before realpathSync**.
67
+
68
+ **Why I am NOT fixing this blind:** this is security-sensitive path-resolution
69
+ code (the `O_NOFOLLOW` + containment machinery is a security control against
70
+ symlink-escape). Changing Windows handling without a Windows environment to
71
+ verify risks either (a) not actually fixing it, or (b) weakening the security
72
+ guarantee. Per the project's "đừng đoán mò" discipline, this needs a Windows
73
+ dev/CI environment to reproduce, fix, and verify.
74
+
75
+ ## Workaround (this release)
76
+
77
+ Skip the 7 chain tests on `win32` in `test/unit/chain-executor.test.ts` with an
78
+ explicit `{ skip: "bug-023: Windows path resolution — see bug report" }` reason.
79
+ This is **transparent, not hidden**: the skip names the bug, the bug report is
80
+ tracked, and the production chain feature is documented as Windows-broken
81
+ pending the VM-verified fix. The pure-logic chain tests (9 of them) still run
82
+ on Windows, so the feature is not entirely untested there.
83
+
84
+ ## Suggested fix (for the Windows-VM session)
85
+
86
+ 1. In `resolveRealContainedPath` (`src/utils/safe-paths.ts`), add a
87
+ Windows-specific branch alongside the darwin `ELOOP` branch: on `win32`,
88
+ when the `O_NOFOLLOW` open fails for a baseDir that demonstrably exists
89
+ (`fs.existsSync(baseDir)`), retry via `realpathSync.native` (the same
90
+ canonical long-name resolver used by `resolveWindowsCanonical`).
91
+ 2. Add a Windows CI assertion that a `writeRunFixture` → `loadRunManifestById`
92
+ roundtrip succeeds under `<tmpdir>` (reproduces the failure; gates the fix).
93
+ 3. Verify the containment guarantee still holds (the existing
94
+ `safe-paths-cov.test.ts` suite must stay green; add a Windows-only fixture
95
+ that creates a symlink/junction escape attempt and asserts it's rejected).
96
+
97
+ ## References
98
+
99
+ - `src/runtime/chain-runner.ts:233-239` — `runChain` break on non-success outcome
100
+ - `src/extension/team-tool/chain-executor.ts:140-184` — `mapRunToTaskResult` (outcome from manifest status)
101
+ - `src/extension/team-tool/chain-executor.ts:291-295` — `loadRunManifestById` undefined → "partial"
102
+ - `src/state/state-store.ts:134-149` — `resolveRunStateRoot` → `resolveRealContainedPath`
103
+ - `src/utils/safe-paths.ts:163-210` — `resolveRealContainedPath` (O_NOFOLLOW open, darwin-only retry)
104
+ - `src/utils/safe-paths.ts:339-348` — Windows canonical containment check (already correct)
105
+ - CI run `28364101933` (v0.9.14) — ubuntu ✓, macos ✓, windows ✗ (these 7 tests)
106
+ - CI run `28347931393` (v0.9.13) — Windows ✗ identically (pre-existing)
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-crew",
3
- "version": "0.9.13",
3
+ "version": "0.9.14",
4
4
  "description": "Pi extension for coordinated AI teams, workflows, worktrees, and async task orchestration",
5
5
  "author": "baphuongna",
6
6
  "license": "MIT",
@@ -15,7 +15,9 @@ export const DEFAULT_CHILD_PI: Readonly<{
15
15
  // Child workers can spend more than a few seconds in provider calls or long-running tools without emitting stdout.
16
16
  // Keep this as a coarse stuck-worker guard rather than a short per-message latency budget.
17
17
  responseTimeoutMs: 5 * 60_000,
18
- maxCaptureBytes: 256 * 1024,
18
+ // #3 unresponsive worker hardening: increased from 256KB to 512KB so critical
19
+ // diagnostic stderr is less likely to be silently truncated during hang analysis.
20
+ maxCaptureBytes: 512 * 1024,
19
21
  // L4 output-handling: thresholds sized from real worker-output data
20
22
  // (27 result artifacts measured: max 9226 bytes, median 8272, 100% < 16KB).
21
23
  // Previous values (8192/1024/4096) truncated 62% of real results.
@@ -44,4 +44,4 @@ export { defineTool, createBashTool } from "@earendil-works/pi-coding-agent";
44
44
  * and to surface version-drift if Pi upgrades introduce breaking changes.
45
45
  * Update this when bumping the pi-coding-agent dependency.
46
46
  */
47
- export const BUILT_AGAINST_PI_VERSION = "0.79.3";
47
+ export const BUILT_AGAINST_PI_VERSION = "0.77.0";
@@ -231,6 +231,12 @@ export interface ChildPiRunResult {
231
231
  aborted?: boolean;
232
232
  /** True if the agent was steered to wrap up (hit soft turn limit) but finished in time. */
233
233
  steered?: boolean;
234
+ /** #7 hardening: bounded digest of intermediate findings (last N tool results or
235
+ * assistant text lines) from the run. Populated by ChildPiLineObserver so that
236
+ * workers that exhaust their budget on tool calls (never emit final assistant
237
+ * text) still produce a non-empty result. Consumers should prefer rawFinalText
238
+ * first — this is a last-resort fallback. */
239
+ intermediateFindings?: string;
234
240
  }
235
241
 
236
242
  export function buildChildPiSpawnOptions(cwd: string, env: NodeJS.ProcessEnv): SpawnOptions {
@@ -526,6 +532,12 @@ export class ChildPiLineObserver {
526
532
  * This is the source of the AUTHORITATIVE result.txt; the transcript stays
527
533
  * compacted (memory bound). */
528
534
  private readonly rawTextEvents: string[] = [];
535
+ /** #7 hardening: bounded digest of intermediate findings. When a worker spends
536
+ * its entire budget on tool calls (never emits a final assistant text),
537
+ * getRawFinalText() returns undefined but this digest captures the last
538
+ * display lines (tool results, stdout fragments) before budget exhaustion.
539
+ * Capped at MAX_INTERMEDIATE_DIGEST_LINES so result artifacts stay bounded. */
540
+ private readonly intermediateFindings: string[] = [];
529
541
 
530
542
  constructor(input: ChildPiRunInput) {
531
543
  this.input = input;
@@ -553,6 +565,22 @@ export class ChildPiLineObserver {
553
565
  return this.rawTextEvents.length > 0 ? this.rawTextEvents[this.rawTextEvents.length - 1] : undefined;
554
566
  }
555
567
 
568
+ /** #7 hardening: returns a bounded digest of intermediate findings accumulated
569
+ * during the run. This is NOT the final answer — it is a best-effort capture
570
+ * of the last assistant text or tool-result display lines before budget
571
+ * exhaustion. Only populated when getRawFinalText() would return undefined.
572
+ * @param maxChars - maximum total characters to return (default 500). */
573
+ getIntermediateFindings(maxChars = 500): string {
574
+ const MAX_INTERMEDIATE_DIGEST_LINES = 20;
575
+ if (this.intermediateFindings.length === 0) return "";
576
+ // Take the last N lines and join, then cap.
577
+ const lines = this.intermediateFindings.slice(-MAX_INTERMEDIATE_DIGEST_LINES);
578
+ const joined = lines.join("\n");
579
+ if (joined.length <= maxChars) return joined;
580
+ // Return the tail within the budget.
581
+ return joined.slice(-maxChars);
582
+ }
583
+
556
584
  private emitLine(line: string): void {
557
585
  if (!line.trim()) return;
558
586
  // Parse the RAW line once so we can BOTH compact it (telemetry transcript,
@@ -561,7 +589,15 @@ export class ChildPiLineObserver {
561
589
  try {
562
590
  const rawParsed = JSON.parse(line);
563
591
  const rawTexts = extractText(rawParsed);
564
- if (rawTexts.length > 0) this.rawTextEvents.push(...rawTexts);
592
+ if (rawTexts.length > 0) {
593
+ this.rawTextEvents.push(...rawTexts);
594
+ // Also capture raw assistant text as intermediate findings — the last raw
595
+ // text may be a partial answer before the worker ran out of budget.
596
+ const last = rawTexts[rawTexts.length - 1];
597
+ if (last.trim().length > 0) {
598
+ this.intermediateFindings.push(last.trim());
599
+ }
600
+ }
565
601
  } catch {
566
602
  // Not valid JSON — compactChildPiLine handles the raw-text fallback below.
567
603
  }
@@ -580,6 +616,10 @@ export class ChildPiLineObserver {
580
616
  } catch (error) {
581
617
  logInternalError("child-pi.on-stdout-line", error, `line=${compact.displayLine}`);
582
618
  }
619
+ // #7 hardening: capture display lines (tool results, stdout) as intermediate
620
+ // findings. This ensures we capture tool output even when no assistant text
621
+ // is emitted (budget exhausted on tool calls).
622
+ this.intermediateFindings.push(compact.displayLine!.trim());
583
623
  }
584
624
  }
585
625
  }
@@ -827,6 +867,51 @@ export async function runChildPi(input: ChildPiRunInput): Promise<ChildPiRunResu
827
867
  } catch (error) {
828
868
  logInternalError("child-pi.response-timeout-term", error, `pid=${child.pid}`);
829
869
  }
870
+ // #3 hardening: if the child never exits (zombie) and neither the
871
+ // 'exit' nor 'close' event ever fires, the promise would hang forever.
872
+ // SIGKILL fires ~3s after SIGTERM via hardKillTimer in killProcessPid,
873
+ // but on platforms where SIGKILL also fails (e.g. permission issues),
874
+ // add a bounded safety settle so the promise always resolves. Using
875
+ // hardKillMs + 2s as the safety window: enough for SIGKILL to work
876
+ // normally, but forces settle if the process is truly immortal.
877
+ // NOTE: we do NOT clear hardKillTimer here (that would defeat its purpose);
878
+ // we intentionally add a parallel safety path.
879
+ const SAFETY_SETTLE_MS = HARD_KILL_MS + 2000;
880
+ const safetyTimer = setTimeout(() => {
881
+ if (settled || childExited) return;
882
+ logInternalError(
883
+ "child-pi.settle-safety-fired",
884
+ new Error(`Child did not exit within ${SAFETY_SETTLE_MS}ms of kill; forcing settle`),
885
+ `pid=${child.pid}, responseTimeoutMs=${responseTimeoutMs}`,
886
+ );
887
+ // Verify the child is still alive before forcing settle.
888
+ // If it somehow exited between childExited=false and here, the
889
+ // settled/childExited guard prevents double-settle (harmless but noisy).
890
+ try {
891
+ process.kill(child.pid!, 0);
892
+ // Child still alive — force settle with timeout error.
893
+ const timeoutErr = `Child Pi produced no new output for ${responseTimeoutMs}ms; killed but did not exit within ${SAFETY_SETTLE_MS}ms (possible zombie).`;
894
+ settle({
895
+ exitCode: null,
896
+ stdout,
897
+ stderr,
898
+ error: timeoutErr,
899
+ exitStatus: {
900
+ exitCode: null,
901
+ cancelled: abortRequested,
902
+ timedOut: true,
903
+ killed: hardKilled,
904
+ cleanupErrors,
905
+ finalDrainMs,
906
+ crashClass: "timeout",
907
+ },
908
+ });
909
+ } catch {
910
+ // ESRCH / EPERM — child is already gone. The 'exit'/'close' handler
911
+ // will fire shortly (or already fired in a race). Let it settle normally.
912
+ }
913
+ }, SAFETY_SETTLE_MS);
914
+ safetyTimer.unref();
830
915
  }, responseTimeoutMs);
831
916
  noResponseTimer.unref();
832
917
  };
@@ -952,6 +1037,7 @@ export async function runChildPi(input: ChildPiRunInput): Promise<ChildPiRunResu
952
1037
  resolve({
953
1038
  ...result,
954
1039
  rawFinalText: lineObserver.getRawFinalText(),
1040
+ intermediateFindings: lineObserver.getIntermediateFindings(),
955
1041
  exitStatus: result.exitStatus ?? {
956
1042
  exitCode: result.exitCode,
957
1043
  cancelled: abortRequested,
@@ -0,0 +1,131 @@
1
+ /**
2
+ * #2 (assessment): goal-achievement detection — kills the "false-green" lie.
3
+ *
4
+ * The v0.9.13 assessment found run team_20260626170635 reported terminal-success
5
+ * ("completed") while its verifier wrote: "did NOT apply ANY of the three
6
+ * security fixes. git diff --stat for all 6 target files is empty… Tests are
7
+ * green only because nothing was changed." The run status LIED.
8
+ *
9
+ * This module assesses whether a completed run actually achieved its goal, so
10
+ * the false-green is no longer SILENT. It is deliberately conservative:
11
+ * - read-only / doc-only workflows are never accused (they legitimately make
12
+ * no project-code edits; their outputs land in gitignored .crew/),
13
+ * - a false-green is only flagged for CODE-MUTATING runs (executor /
14
+ * test-engineer steps) in a git repo whose working tree is empty,
15
+ * - status is downgraded to "failed" ONLY when a corroborating signal (a
16
+ * task that actually failed) confirms it; otherwise the suspicion is
17
+ * exposed via `goalAchieved=false` + a prominent event + manifest note,
18
+ * without breaking a legitimately-no-op run.
19
+ */
20
+ import { spawnSync } from "node:child_process";
21
+ import type { TeamRunManifest, TeamTaskState } from "../state/types.ts";
22
+ import type { WorkflowConfig } from "../workflows/workflow-config.ts";
23
+ import { findRepoRoot } from "../utils/paths.ts";
24
+
25
+ /**
26
+ * Roles whose job is to EDIT project source/test files (changes that appear in
27
+ * `git status`). Deliberately NARROW: writer/verifier/security-reviewer write
28
+ * reports (often to gitignored .crew/), so an empty diff is NOT a failure
29
+ * signal for them. Only executor/test-engineer MUST touch project code.
30
+ */
31
+ export const MUTATING_ROLES = new Set<string>(["executor", "test-engineer"]);
32
+
33
+ export type GoalAchieved = boolean | "unknown";
34
+
35
+ export interface GoalAchievementAssessment {
36
+ achieved: GoalAchieved;
37
+ reason: string;
38
+ /** human-readable corroborating signals (for events/manifest) */
39
+ signals: string[];
40
+ }
41
+
42
+ /** Does this workflow contain any code-mutating role step? */
43
+ export function workflowIsMutating(workflow: WorkflowConfig | undefined): boolean {
44
+ const steps = workflow?.steps ?? [];
45
+ return steps.some((step) => typeof step?.role === "string" && MUTATING_ROLES.has(step.role));
46
+ }
47
+
48
+ /** Is the working tree of the repo containing `cwd` clean (no changes)? Throws-safe. */
49
+ export function isGitWorkingTreeClean(cwd: string): { clean: boolean; repoRoot: string } | { clean: "unknown"; repoRoot: undefined } {
50
+ try {
51
+ const repoRoot = findRepoRoot(cwd);
52
+ if (!repoRoot) return { clean: "unknown", repoRoot: undefined };
53
+ // `git status --porcelain` → empty stdout means a clean working tree.
54
+ const res = spawnSync("git", ["-C", repoRoot, "status", "--porcelain"], { encoding: "utf8", timeout: 5_000 });
55
+ if (res.error || res.status !== 0) return { clean: "unknown", repoRoot: undefined };
56
+ return { clean: res.stdout.trim().length === 0, repoRoot };
57
+ } catch {
58
+ return { clean: "unknown", repoRoot: undefined };
59
+ }
60
+ }
61
+
62
+ /**
63
+ * Assess whether a completed run achieved its goal. Pure function over manifest
64
+ * + tasks + workflow — no side effects, easy to unit-test.
65
+ */
66
+ export function assessGoalAchievement(
67
+ manifest: TeamRunManifest,
68
+ tasks: TeamTaskState[],
69
+ workflow: WorkflowConfig | undefined,
70
+ ): GoalAchievementAssessment {
71
+ // (1) Only code-mutating workflows can be false-green. Read-only/doc runs
72
+ // legitimately make no project edits — never accuse them.
73
+ if (!workflowIsMutating(workflow)) {
74
+ return { achieved: "unknown", reason: "read-only / doc-only workflow (no executor/test-engineer steps)", signals: [] };
75
+ }
76
+
77
+ // (2) Need a git repo to inspect the working tree.
78
+ const tree = isGitWorkingTreeClean(manifest.cwd);
79
+ if (tree.clean === "unknown" || !tree.repoRoot) {
80
+ return { achieved: "unknown", reason: "not a git repo or git unavailable", signals: [] };
81
+ }
82
+
83
+ // (3) Non-empty working tree → the mutating run made edits. Achieved.
84
+ if (!tree.clean) {
85
+ return { achieved: true, reason: "git working tree has changes (mutating run edited project files)", signals: [`repoRoot=${tree.repoRoot}`] };
86
+ }
87
+
88
+ // (4) Mutating run + CLEAN working tree → suspicious. The executor's job was
89
+ // to edit code and it left zero diff. This is the false-green signature.
90
+ const failedTask = tasks.find((t) => t.status === "failed");
91
+ const signals = ["git working tree clean despite mutating workflow"];
92
+ if (failedTask) signals.push(`corroborating failed task: ${failedTask.id} (${failedTask.role})`);
93
+
94
+ return {
95
+ // `false` = false-green detected. Whether to downgrade status is decided
96
+ // by the caller based on the corroborating failed-task signal.
97
+ achieved: false,
98
+ reason: "code-mutating run completed but made no project edits (false-green)",
99
+ signals,
100
+ };
101
+ }
102
+
103
+ /**
104
+ * Apply an assessment to a run result: set manifest.goalAchieved + note, and
105
+ * decide whether to downgrade status. Returns the (possibly mutated) manifest.
106
+ *
107
+ * Downgrade rule (conservative): only flip "completed" → "failed" when a
108
+ * corroborating failed-task signal is present. Otherwise expose goalAchieved
109
+ * = false + a manifest note + let the caller emit an event, but leave status
110
+ * intact so a legitimately-no-op mutating run is not broken.
111
+ */
112
+ export function applyGoalAchievement(
113
+ manifest: TeamRunManifest,
114
+ assessment: GoalAchievementAssessment,
115
+ ): { manifest: TeamRunManifest; downgraded: boolean } {
116
+ const note = assessment.achieved === false
117
+ ? `goal-achievement: FALSE-GREEN — ${assessment.reason}. ${assessment.signals.join("; ")}`
118
+ : assessment.achieved === true
119
+ ? `goal-achievement: OK — ${assessment.reason}`
120
+ : `goal-achievement: unknown — ${assessment.reason}`;
121
+
122
+ const updated: TeamRunManifest = { ...manifest, goalAchieved: assessment.achieved, goalAchievementNote: note };
123
+
124
+ // Downgrade only on the highest-confidence false-green: a failed task
125
+ // corroborates that the run genuinely did not succeed.
126
+ const hasCorroboratingFailure = assessment.signals.some((s) => s.startsWith("corroborating failed task"));
127
+ if (assessment.achieved === false && hasCorroboratingFailure && updated.status === "completed") {
128
+ return { manifest: { ...updated, status: "failed" }, downgraded: true };
129
+ }
130
+ return { manifest: updated, downgraded: false };
131
+ }
@@ -72,3 +72,38 @@ export function buildRecoveryLedger(decisions: PolicyDecision[], previous: Recov
72
72
  }
73
73
  return { entries };
74
74
  }
75
+
76
+ /**
77
+ * #4 (assessment): decide whether a FAILED task should be re-queued for a
78
+ * whole-task rerun, honoring limits.maxRetriesPerTask.
79
+ *
80
+ * Before #4, buildRecoveryLedger recorded `rerun_task` entries with
81
+ * state:"planned" but NOTHING ever executed them — the recovery ledger was
82
+ * decorative. This pure function drives the actual re-queue decision used in
83
+ * the run loop: when a task returns a failed STATUS (not a retryable throw,
84
+ * which #1's autoRetry/executeWithRetry already handles), re-queue it for a
85
+ * bounded whole-task rerun instead of immediately aborting the run.
86
+ *
87
+ * Default-off: maxRetriesPerTask defaults to 0 → never rerun (preserves prior
88
+ * behavior unless explicitly opted in). Bounded by retryCount < maxRetries.
89
+ */
90
+ export interface RerunDecision {
91
+ rerun: boolean;
92
+ newRetryCount: number;
93
+ reason: string;
94
+ }
95
+
96
+ export function shouldRerunFailedTask(
97
+ task: { policy?: { retryCount?: number } },
98
+ limits?: { maxRetriesPerTask?: number },
99
+ ): RerunDecision {
100
+ const maxRetries = limits?.maxRetriesPerTask ?? 0;
101
+ const retryCount = task.policy?.retryCount ?? 0;
102
+ if (maxRetries <= 0) {
103
+ return { rerun: false, newRetryCount: retryCount, reason: "maxRetriesPerTask not set (opt-in) — no whole-task rerun" };
104
+ }
105
+ if (retryCount >= maxRetries) {
106
+ return { rerun: false, newRetryCount: retryCount, reason: `retryCount ${retryCount} >= maxRetriesPerTask ${maxRetries} — rerun budget exhausted` };
107
+ }
108
+ return { rerun: true, newRetryCount: retryCount + 1, reason: `whole-task rerun ${retryCount + 1}/${maxRetries}` };
109
+ }
@@ -343,6 +343,7 @@ export async function runTeamTask(
343
343
  let modelAttempts: ModelAttemptSummary[] | undefined;
344
344
  let parsedOutput: ParsedPiJsonOutput | undefined;
345
345
  let rawFinalText: string | undefined;
346
+ let intermediateFindings: string | undefined;
346
347
  let finalStdout = "";
347
348
  let transcriptPath: string | undefined;
348
349
  let terminalEvidence: OperationTerminalEvidence[] = [];
@@ -719,6 +720,7 @@ export async function runTeamTask(
719
720
  );
720
721
  parsedOutput = parsePiJsonOutput(transcriptText);
721
722
  rawFinalText = childResult.rawFinalText;
723
+ intermediateFindings = childResult.intermediateFindings;
722
724
  error =
723
725
  childResult.error ||
724
726
  (childResult.exitCode && childResult.exitCode !== 0
@@ -846,6 +848,10 @@ export async function runTeamTask(
846
848
  cleanResultText(parsedOutput?.finalText) ??
847
849
  cleanResultText(finalStdout) ??
848
850
  cleanResultText(finalStderr) ??
851
+ // #7 hardening: if all real output paths are empty (worker exhausted
852
+ // budget on tool calls, no assistant text), use intermediate findings.
853
+ // intermediateFindings captures the last N tool-result display lines.
854
+ cleanResultText(intermediateFindings) ??
849
855
  "(no output)",
850
856
  producer: task.id,
851
857
  });
@@ -15,7 +15,8 @@ import { withRunLock } from "../state/locks.ts";
15
15
  import { aggregateUsage, formatUsage } from "../state/usage.ts";
16
16
  import type { WorkflowConfig, WorkflowStep } from "../workflows/workflow-config.ts";
17
17
  import { evaluateCrewPolicy, summarizePolicyDecisions } from "./policy-engine.ts";
18
- import { buildRecoveryLedger } from "./recovery-recipes.ts";
18
+ import { buildRecoveryLedger, shouldRerunFailedTask } from "./recovery-recipes.ts";
19
+ import { assessGoalAchievement, applyGoalAchievement } from "./goal-achievement.ts";
19
20
  import { buildTaskGraphIndex, refreshTaskGraphQueues, taskGraphSnapshot } from "./task-graph-scheduler.ts";
20
21
  import { buildExecutionPlan as buildDagExecutionPlan, getReadyTasks as getDagReadyTasks, type TaskNode } from "./task-graph.ts";
21
22
  import { checkBranchFreshness } from "../worktree/branch-freshness.ts";
@@ -343,6 +344,16 @@ function retryPolicyFromConfig(config: CrewReliabilityConfig | undefined): Retry
343
344
  return { ...DEFAULT_RETRY_POLICY, ...(config?.retryPolicy ?? {}) };
344
345
  }
345
346
 
347
+ /**
348
+ * #1 (assessment): decide whether the per-task retry path (executeWithRetry) is used.
349
+ * Defaults to TRUE (opt-out) so transient worker hangs (ChildTimeout) are retried
350
+ * automatically. Previously opt-in, which left the entire retry+recovery stack dormant.
351
+ * Exported for unit testing.
352
+ */
353
+ export function shouldUseRetry(reliability: CrewReliabilityConfig | undefined): boolean {
354
+ return reliability?.autoRetry !== false;
355
+ }
356
+
346
357
  function failedTaskFrom(result: { tasks: TeamTaskState[] }, taskId: string): TeamTaskState | undefined {
347
358
  return result.tasks.find((item) => item.id === taskId && item.status === "failed");
348
359
  }
@@ -452,6 +463,23 @@ export async function executeTeamRun(input: ExecuteTeamRunInput): Promise<{ mani
452
463
 
453
464
  try {
454
465
  const result = await executeTeamRunCore(input, manifest, workflow);
466
+ // #2 (assessment): goal-achievement detection — kill the silent false-green.
467
+ // A code-mutating run that "completed" but left the git working tree clean
468
+ // (and/or had a failed task) is a false-green. We expose goalAchieved on the
469
+ // manifest + emit an event so the lie is never silent, and downgrade status
470
+ // to "failed" only when a failed task corroborates it (conservative).
471
+ const gaAssessment = assessGoalAchievement(result.manifest, result.tasks, workflow);
472
+ const gaApplied = applyGoalAchievement(result.manifest, gaAssessment);
473
+ if (gaApplied.manifest !== result.manifest) {
474
+ result.manifest = gaApplied.manifest;
475
+ try {
476
+ saveRunManifest(result.manifest);
477
+ } catch (persistError) {
478
+ logInternalError("team-runner.goalAchievement.persist", persistError instanceof Error ? persistError : new Error(String(persistError)), `runId=${manifest.runId}`);
479
+ }
480
+ }
481
+ appendEvent(manifest.eventsPath, { type: "run.goal_achievement", runId: manifest.runId, message: gaApplied.manifest.goalAchievementNote ?? "", data: { achieved: gaAssessment.achieved, downgraded: gaApplied.downgraded, reason: gaAssessment.reason, signals: gaAssessment.signals } });
482
+ if (gaApplied.downgraded) logInternalError("team-runner.goalAchievement.falseGreen", new Error(gaApplied.manifest.goalAchievementNote ?? "false-green detected"), `runId=${manifest.runId}`);
455
483
  stopTeamHeartbeat();
456
484
  resolveRunPromise(manifest.runId, result);
457
485
  cleanupUsage();
@@ -602,6 +630,18 @@ async function executeTeamRunCore(
602
630
 
603
631
  const failed = tasks.find((task) => task.status === "failed");
604
632
  if (failed) {
633
+ // #4 (assessment): honor limits.maxRetriesPerTask — re-queue an eligible
634
+ // failed task for a bounded whole-task rerun instead of immediately
635
+ // aborting the run. Before #4, the recovery ledger recorded `rerun_task`
636
+ // entries with state:"planned" but never executed them (decorative).
637
+ // Default-off: maxRetriesPerTask=0 → original abort behavior preserved.
638
+ const rerun = shouldRerunFailedTask(failed, input.limits);
639
+ if (rerun.rerun) {
640
+ tasks = tasks.map((item) => item.id === failed.id ? { ...item, status: "queued" as const, policy: { ...(item.policy ?? {}), retryCount: rerun.newRetryCount }, error: undefined, finishedAt: undefined } : item);
641
+ await saveRunTasksAsync(manifest, tasks);
642
+ await appendEventAsync(manifest.eventsPath, { type: "recovery.rerun_task", runId: manifest.runId, taskId: failed.id, message: `Re-queuing failed task for whole-task rerun: ${rerun.reason}`, data: { attempt: rerun.newRetryCount, maxRetries: input.limits?.maxRetriesPerTask ?? 0, scenario: "task_failed" } });
643
+ continue; // loop re-processes the re-queued task
644
+ }
605
645
  tasks = markBlocked(tasks, `Blocked by failed task '${failed.id}'.`);
606
646
  await saveRunTasksAsync(manifest, tasks);
607
647
  saveCrewAgents(manifest, recordsForMaterializedTasks(manifest, tasks, runtimeKind));
@@ -685,7 +725,12 @@ async function executeTeamRunCore(
685
725
  const teamRole = input.team.roles.find((role) => role.name === task.role);
686
726
  const perTaskRuntime = resolveTaskRuntimeKind(runtimeKind, task.role, input.runtimeConfig?.isolationPolicy);
687
727
  const baseInput = { manifest, tasks, task, step, agent, signal: input.signal, executeWorkers: input.executeWorkers, runtimeKind: runtimeKind, taskRuntimeOverride: perTaskRuntime !== runtimeKind ? perTaskRuntime : undefined, runtimeConfig: input.runtimeConfig, parentContext: input.parentContext, parentModel: input.parentModel, modelRegistry: input.modelRegistry, modelOverride: input.modelOverride, teamRoleModel: teamRole?.model, teamRoleSkills: teamRole?.skills, skillOverride: input.skillOverride, limits: input.limits, onJsonEvent: input.onJsonEvent, workspaceId: input.workspaceId };
688
- if (input.reliability?.autoRetry !== true) return withCorrelation(childCorrelation(manifest.runId, task.id), () => runTeamTask(baseInput));
728
+ // #1 (assessment): autoRetry now defaults ON (opt-out via reliability.autoRetry=false).
729
+ // The dominant v0.9.13 failure was ChildTimeout ("worker became unresponsive") with
730
+ // ZERO retries because this gate was opt-in. isRetryable() defaults to true when
731
+ // retryableErrors is empty, so transient hangs now retry up to maxAttempts (3) with
732
+ // exponential backoff. Set reliability.autoRetry=false to restore old single-shot behavior.
733
+ if (!shouldUseRetry(input.reliability)) return withCorrelation(childCorrelation(manifest.runId, task.id), () => runTeamTask(baseInput));
689
734
  let lastFailed: { manifest: TeamRunManifest; tasks: TeamTaskState[] } | undefined;
690
735
  let lastAttemptId: string | undefined;
691
736
  const attemptsSoFar: TaskAttemptState[] = [...(task.attempts ?? [])];
@@ -201,6 +201,9 @@ export interface TeamRunManifest {
201
201
  args?: unknown;
202
202
  summary?: string;
203
203
  policyDecisions?: PolicyDecision[];
204
+ /** #2 (assessment): goal-achievement verdict — kills the silent false-green. */
205
+ goalAchieved?: boolean | "unknown";
206
+ goalAchievementNote?: string;
204
207
  }
205
208
 
206
209
  export interface UsageState {
@@ -42,7 +42,7 @@ export function getRenderWidth(width?: number): number {
42
42
  if (Number.isFinite(stdoutCols) && stdoutCols! > 0) return Math.floor(stdoutCols!);
43
43
  return DEFAULT_WIDGET_WIDTH;
44
44
  }
45
- export { notificationBadge } from "./widget-formatters.ts";
45
+ export { notificationBadge, NOTIFICATION_BADGE_CAP } from "./widget-formatters.ts";
46
46
 
47
47
  // ── Constants ─────────────────────────────────────────────────────────
48
48
 
@@ -144,9 +144,22 @@ export function agentStats(agent: CrewAgentRecord, liveHandle?: LiveAgentHandle)
144
144
 
145
145
  // ── Notification badge ────────────────────────────────────────────────
146
146
 
147
+ // Bug 021: the bell glyph 🔔 was misread as "queued messages" — users saw
148
+ // `🔔227` and concluded there were 227 pending items, when the value is a
149
+ // CUMULATIVE warning/error/critical count with zero actual queue behind it.
150
+ // Fix: relabel to an explicit "alerts" segment (no bell) and cap the display
151
+ // at 99+ (standard badge practice). The cumulative count stays accurate
152
+ // internally (widgetState.notificationCount) and remains fully logged in
153
+ // .crew/state/notifications/YYYY-MM-DD.jsonl — this bounds presentation only.
154
+ // Deeper fixes (decay window, owner-scope, auto-reset on all-runs-terminal,
155
+ // full deprecation) are product decisions documented in
156
+ // docs/bugs/bug-021-notification-badge-counter-misleading.md.
157
+ export const NOTIFICATION_BADGE_CAP = 99;
158
+
147
159
  export function notificationBadge(count: number | undefined, env: NodeJS.ProcessEnv = process.env): string {
148
160
  if (!count || count <= 0) return "";
149
161
  const term = `${env.TERM ?? ""} ${env.WT_SESSION ?? ""} ${env.TERM_PROGRAM ?? ""}`.toLowerCase();
150
162
  const supportsEmoji = !term.includes("dumb") && env.NO_COLOR !== "1";
151
- return supportsEmoji ? ` 🔔${count}` : ` [!${count}]`;
163
+ const label = count > NOTIFICATION_BADGE_CAP ? `${NOTIFICATION_BADGE_CAP}+ alerts` : `${count} alerts`;
164
+ return supportsEmoji ? ` · ${label}` : ` [${label}]`;
152
165
  }
@@ -15,6 +15,7 @@ import { computeLiveDurationMs } from "../live-duration.ts";
15
15
  import { getTaskUsage } from "../../runtime/usage-tracker.ts";
16
16
  import { agentActivity, agentStats, elapsed, formatTokensCompact, notificationBadge } from "./widget-formatters.ts";
17
17
  import { activeWidgetRuns, shortRunLabel } from "./widget-model.ts";
18
+ import { isFinishedRunStatus } from "../../runtime/process-status.ts";
18
19
  import type { WidgetRun } from "./widget-types.ts";
19
20
 
20
21
  const MAX_AGENTS_DISPLAY = 3;
@@ -69,6 +70,7 @@ export function buildWidgetLines(cwd: string, frame = 0, maxLines = 8, providedR
69
70
  });
70
71
  const completed = agents.filter((a) => a.status === "completed").length;
71
72
  const runGlyph = iconForStatus(run.status, { runningGlyph });
73
+ const isTerminal = isFinishedRunStatus(run.status);
72
74
  // Run progress line. v1–v3 flickered on snapshot.tasks state, v4 was
73
75
  // too minimal (`0/1 agents` only), v5 duplicated the worker activity
74
76
  // line (tools/tokens/duration already shown one row below). v6 (this)
@@ -81,10 +83,19 @@ export function buildWidgetLines(cwd: string, frame = 0, maxLines = 8, providedR
81
83
  // for a healthy run), and `run.createdAt` is immutable. The format
82
84
  // shape `"X/Y agents · Ns"` is therefore truly invariant: same number
83
85
  // of `·`-separated fields, same field meanings, every render tick.
86
+ //
87
+ // Bug 022 (timer-fix + label): for TERMINAL runs (failed/cancelled/
88
+ // completed) the elapsed counter previously kept ticking up forever
89
+ // from createdAt (a failed run showed `2028s` and climbing, read as
90
+ // "still running"). Now it FREEZES at updatedAt (when the run
91
+ // reached its terminal status). The status label is also surfaced
92
+ // explicitly so the row cannot be misread as an active run.
84
93
  const agentCountText = `${completed}/${agents.length} agents`;
85
- const runElapsedMs = Math.max(0, Date.now() - new Date(run.createdAt).getTime());
94
+ const runEndMs = isTerminal ? new Date(run.updatedAt).getTime() : Date.now();
95
+ const runElapsedMs = Math.max(0, Number.isFinite(runEndMs) ? runEndMs - new Date(run.createdAt).getTime() : 0);
86
96
  const runElapsedText = `${Math.floor(runElapsedMs / 1000)}s`;
87
- const progressPart = `${agentCountText} · ${runElapsedText}`;
97
+ const statusLabel = isTerminal ? ` · ${run.status}` : "";
98
+ const progressPart = `${agentCountText} · ${runElapsedText}${statusLabel}`;
88
99
  lines.push(truncate(`├─ ${runGlyph} ${shortRunLabel(run)} · ${progressPart} · ${run.runId.slice(-8)}`, width));
89
100
 
90
101
  const liveForRun = listLiveAgents().filter((a) => a.runId === run.runId);
@@ -19,6 +19,24 @@ export const PEM_PRIVATE_KEY_PATTERN = /-----BEGIN [A-Z ]+PRIVATE KEY-----[\s\S]
19
19
  // Full mitigation ladder: (1) redaction here + at artifact-write; (2) Phase 1.5
20
20
  // sanitized-env verification; (3) sandbox (deferred).
21
21
 
22
+ // Exclusion list: LLM usage-count key names that contain "token" but are
23
+ // observable metrics, NOT credentials. These must NEVER be redacted so that
24
+ // token-usage observability in events.jsonl is preserved. The isSecretKey()
25
+ // keyword-scan matches '_token' in 'prompt_tokens' (underscore + "token"),
26
+ // falsely classifying usage counts as secrets. See performance/quality
27
+ // assessment fix #5.
28
+ const TOKEN_COUNT_KEYS = new Set([
29
+ "prompt_tokens",
30
+ "completion_tokens",
31
+ "total_tokens",
32
+ "cached_tokens",
33
+ "reasoning_tokens",
34
+ "cached_read_tokens",
35
+ "cached_write_tokens",
36
+ "input_tokens",
37
+ "output_tokens",
38
+ ]);
39
+
22
40
  // JWT — three base64url segments separated by dots, distinctive "eyJ" headers.
23
41
  // Linear: single + on [A-Za-z0-9_-] per segment, no nesting.
24
42
  export const JWT_PATTERN = /(?<![A-Za-z0-9_-])eyJ[A-Za-z0-9_-]+\.eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+/g;
@@ -43,6 +61,8 @@ export const STRIPE_KEY_PATTERN = /(?<![A-Za-z0-9_])sk_live_[0-9a-zA-Z]{24}(?![0
43
61
  // a more complex regex, catastrophic backtracking (ReDoS) could result.
44
62
  // Any modifications must preserve O(n) complexity where n = keyName.length.
45
63
  export function isSecretKey(keyName: string): boolean {
64
+ // Fast path: known token-count keys are never secrets.
65
+ if (TOKEN_COUNT_KEYS.has(keyName.toLowerCase())) return false;
46
66
  // Fast path: common secret key names (safe anchored regex, no backtracking)
47
67
  const lower = keyName.toLowerCase();
48
68
  if (/^(token|apikey|api_key|password|secret|credential|authorization|privatekey|private_key)$/.test(lower)) {