pi-crew 0.9.13 → 0.9.14
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +52 -0
- package/docs/bugs/bug-021-notification-badge-counter-misleading.md +293 -0
- package/docs/bugs/bug-023-chain-windows-path-resolution.md +106 -0
- package/package.json +1 -1
- package/src/config/defaults.ts +3 -1
- package/src/extension/pi-api.ts +1 -1
- package/src/runtime/child-pi.ts +87 -1
- package/src/runtime/goal-achievement.ts +131 -0
- package/src/runtime/recovery-recipes.ts +35 -0
- package/src/runtime/task-runner.ts +6 -0
- package/src/runtime/team-runner.ts +47 -2
- package/src/state/types.ts +3 -0
- package/src/ui/widget/index.ts +1 -1
- package/src/ui/widget/widget-formatters.ts +14 -1
- package/src/ui/widget/widget-renderer.ts +13 -2
- package/src/utils/redaction.ts +20 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,57 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [v0.9.14] — reliability & UX fixes from the v0.9.13 performance/quality assessment (2026-06-29)
|
|
4
|
+
|
|
5
|
+
A parallel-research assessment of v0.9.13 measured three operational gaps and produced ten prioritized recommendations (effort × impact). This release ships **8 of 10 fixes + 2 UX bug fixes from bug reports**, with the remaining 2 honestly deferred (need product input). The unifying theme: **stop the silent lies** — retry that was built but never enabled, a "completed" status that hid zero work, and UI that read failed runs as still-running.
|
|
6
|
+
|
|
7
|
+
Full assessment: `research-findings/pi-crew-performance-quality-assessment.md`. Completion ledger: `research-findings/fix-all-completion-status.md`.
|
|
8
|
+
|
|
9
|
+
### Bug fixes — reliability (assessment recs #1–#10)
|
|
10
|
+
|
|
11
|
+
- **#1 autoRetry enabled by default (opt-out) [HIGH impact]** — the entire retry + recovery stack was built and tested but gated off (`team-runner.ts:688`: `if (autoRetry !== true) return single-shot`). Every task got exactly ONE attempt. The dominant v0.9.13 failure was `ChildTimeout` ("worker became unresponsive") — 78 event-log occurrences, 13 run-level failures — all with zero retries. Now opt-OUT: new `shouldUseRetry()` helper (`reliability.autoRetry !== false`); `isRetryable()` already returns true on the default policy, so transient hangs retry up to `maxAttempts` (3) with exponential backoff. The single highest-impact fix — flip one gate.
|
|
12
|
+
|
|
13
|
+
- **#2 goal-achievement detection — kills the silent false-green** — run `team_20260626170635` reported terminal-success while its verifier wrote "did NOT apply ANY of the three security fixes. `git diff --stat` empty. Tests green only because nothing was changed." New `goal-achievement.ts` (pure functions, unit-tested): a code-mutating workflow (executor/test-engineer steps) in a git repo whose working tree is CLEAN after "completion" is the false-green signature. Read-only/doc workflows and non-git cwds are never accused (conservative). `manifest.goalAchieved` + `goalAchievementNote` + a `run.goal_achievement` event are emitted always; status is downgraded `completed` → `failed` ONLY when a corroborating failed-task signal confirms it (a legitimately-no-op mutating run is flagged but not broken).
|
|
14
|
+
|
|
15
|
+
- **#4 recovery `rerun_task` execution (was decorative)** — `buildRecoveryLedger` recorded `rerun_task` entries with `state:"planned"` but NOTHING ever executed them. The run loop aborted on the first failed task. New `shouldRerunFailedTask()` (pure fn) drives a bounded whole-task re-queue in the run loop when `limits.maxRetriesPerTask > 0` and `retryCount < max`. Default-off (preserves behavior); bounded by `retryCount >= max` (no infinite loop). Complements #1 (autoRetry handles retryable throws within `executeWithRetry`; #4 handles terminal FAILED status re-queue).
|
|
16
|
+
|
|
17
|
+
- **#3 unresponsive-worker hardening [confounded — labeled "hardened" not "fixed"]** — the 78 `ChildTimeout` occurrences are confounded by the free model (`zai/glm-5.2`, `cost=0` everywhere in sampled runs); cannot be root-caused without a paid model. Applied defensive hardening anyway: `maxCaptureBytes` 256 KB → 512 KB (critical diagnostic stderr less likely truncated during hang analysis), SIGKILL escalation on timeout (kill-tree after grace) + a safety-settle timer so a hung worker cannot hold the run forever.
|
|
18
|
+
|
|
19
|
+
- **#5 stop redacting token counts in `events.jsonl`** — usage/cost fields were obscured as `"***"`, blocking token analysis off the event stream. New `TOKEN_COUNT_KEYS` exclusion in `redaction.ts` un-redacts token/cost COUNTS while keeping API-key secrets redacted. Restores observability.
|
|
20
|
+
|
|
21
|
+
- **#7 intermediate-findings capture (over-budget workers)** — a worker that spends its whole budget on tool calls (13 calls) may never emit a final structured handoff → `results/<id>.txt` = one fragment line → `valid:false`. This happened to the `explore-ui` shard of THIS release's assessment. `ChildPiLineObserver` now tracks `intermediateFindings` (best assistant text seen), and `task-runner`'s result chain falls back to it when no clean `rawFinalText` is produced — the result is never a useless 1-line fragment. Does NOT regress Fix A (rawFinalText, v0.9.13): rawFinalText stays preferred; intermediateFindings is a deeper fallback.
|
|
22
|
+
|
|
23
|
+
- **#10 version drift** — `BUILT_AGAINST_PI_VERSION` ("0.79.3") did not match the installed `pi-coding-agent` ("0.77.0"). Reconciled; a new test reads `node_modules` at test time and asserts the match, so drift is caught going forward.
|
|
24
|
+
|
|
25
|
+
### Bug fixes — UX (bug reports)
|
|
26
|
+
|
|
27
|
+
- **bug-021 notification badge** — the `🔔N` bell glyph in the crew widget header / powerbar was misread as "queued messages": users saw `🔔227` and concluded 227 pending items, when the value is a CUMULATIVE warning/error/critical count with zero actual queue behind it (verified: 0 queued agents, 0 queued tasks, 0 unread mailbox). `notificationBadge()` now renders `· N alerts` (explicit label, no bell) and caps the display at `99+` (standard badge practice). The cumulative count stays accurate internally and remains fully logged in `.crew/state/notifications/`; this bounds presentation only. Deeper fixes (decay window, owner-scope, auto-reset, full deprecation) remain open product decisions — documented in `docs/bugs/bug-021-notification-badge-counter-misleading.md`.
|
|
28
|
+
|
|
29
|
+
- **bug-022 terminal-run widget row** — for terminal runs (failed/cancelled/completed), the run-progress row computed `runElapsedMs = Date.now() - run.createdAt` for EVERY run, so a failed run lingering in the F-5 grace window showed an ever-climbing counter (e.g. `2028s` and rising), read as "still running". The `✘` glyph + `0/1 agents` layout reinforced the misread. Now: the timer FREEZES at `run.updatedAt` (when the run reached terminal status) and an explicit ` · <status>` label is appended. Running runs keep the live ticker. Verified live: a completed fast-fix run renders `✓ fast-fix/fast-fix · 3/3 agents · 513s · completed` with a stable, non-climbing counter.
|
|
30
|
+
|
|
31
|
+
### Tests
|
|
32
|
+
|
|
33
|
+
- **#9 real-process watchdog E2E** — `background-runner-watchdog.test.ts` was logic-only (admitted in its header); the real-process kill path was only manually verified. New `background-runner-watchdog-e2e.test.ts` spawns a hung harness with `PI_CREW_MAX_RUN_MS` overridden short, asserts the process dies within the grace window, and uses the `PI_CREW_PARENT_PID=test.pid` no-leak pattern so the test never leaks a runner.
|
|
34
|
+
|
|
35
|
+
- **team-tool-parallel process leak** — test case 5 passed validation and reached `spawnBackgroundTeamRun`, spawning a REAL detached background-runner on every `npm test`. Rewritten to use a non-existent agent (returns an agent-not-found error BEFORE spawn) + `PI_CREW_PARENT_PID` defense-in-depth.
|
|
36
|
+
|
|
37
|
+
### Honesty discipline
|
|
38
|
+
|
|
39
|
+
- **Anti-overclaim verification** — the fix-all implementation run's verifier FAILED honestly: only 2/10 were committed by the worker team, 7 incomplete. The leader (this session) then implemented #1/#2/#4 surgically, committed the uncommitted #3/#7/#9, and deferred #6/#8 with documented rationale. Pre-existing flaky `goal-loop-smoke.test.ts` was PROVEN independent (fails identically at baseline `aff3fd5`). MEASURED vs INFERRED separated throughout.
|
|
40
|
+
|
|
41
|
+
### Deferred (need product input)
|
|
42
|
+
|
|
43
|
+
- **#6 retained chain + research E2E** — `scripts/run-real-chain.ts` needs a REAL model run; the free model hangs (the assessment's core confound). Chain feature is statically verified (86/86 unit tests). Live E2E pending paid-model access.
|
|
44
|
+
- **#8 split `register.ts`** (2194 LOC) — already a thin orchestrator calling ~30 extracted `registerXxx()` functions; split = cosmetic churn with regression risk for marginal gain. Not worth it.
|
|
45
|
+
|
|
46
|
+
### Verification
|
|
47
|
+
|
|
48
|
+
- `tsc --noEmit` → EXIT 0
|
|
49
|
+
- `check:lazy-imports` → EXIT 0
|
|
50
|
+
- `test:unit` → 5742 tests, 5739 pass, 0 fail (3 skipped), EXIT 0
|
|
51
|
+
- `test:integration` → 157/157 pass excluding pre-existing-flaky `goal-loop-smoke` (proven fails identically at baseline)
|
|
52
|
+
- new-fix tests → 67/67 pass (#1/#2/#4/#5/#7/#9/#10 + bug-021/022)
|
|
53
|
+
- `npm pack --dry-run` → 631 files, clean
|
|
54
|
+
|
|
3
55
|
## [v0.9.13] — chain feature + raw worker output + anti-zombie hardening (2026-06-29)
|
|
4
56
|
|
|
5
57
|
Context/compaction efficiency work + a live `chain` feature that wires previously-dead code, plus two root-cause zombie fixes. The unifying theme: **deliver the right context efficiently and never leave orphaned state behind.**
|
|
@@ -0,0 +1,293 @@
|
|
|
1
|
+
# Bug Report: Notification Bell Badge Misread as "Queued Messages"
|
|
2
|
+
|
|
3
|
+
**Date:** 2026-06-29
|
|
4
|
+
**Severity:** Medium (UX / User Confusion — no data loss, no functional impact)
|
|
5
|
+
**Status:** Partially fixed (Option A + Option B-display applied 2026-06-29; decay / owner-scope / auto-reset / deprecation remain open product decisions)
|
|
6
|
+
**Reporter:** User investigation session (manual triage)
|
|
7
|
+
**Related:** `docs/bugs/cross-session-notification-leakage.md` (related but distinct —
|
|
8
|
+
that one filters notifications *delivered* across sessions; this one accumulates
|
|
9
|
+
the badge count without decay regardless of session ownership)
|
|
10
|
+
**Affects:** pi-crew widget header + powerbar segment (`pi-crew-active`)
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## Summary
|
|
15
|
+
|
|
16
|
+
The `🔔N` bell badge shown in the crew widget header (and the equivalent
|
|
17
|
+
`pi-crew-active` powerbar segment) is the **cumulative notification counter**
|
|
18
|
+
(`widgetState.notificationCount`), not a message-queue size. Users reading the
|
|
19
|
+
bell icon as "queued messages" conclude there are hundreds of pending items
|
|
20
|
+
when in fact:
|
|
21
|
+
|
|
22
|
+
- Actual `status === "queued"` agents across all runs: **0**
|
|
23
|
+
- Actual `status === "queued"` tasks across all `tasks.json`: **0**
|
|
24
|
+
- Mailbox unread (inbox) messages: **0** (no `mailbox/inbox.jsonl` exists)
|
|
25
|
+
- Live in-memory agents (`listActiveLiveAgents`): **0**
|
|
26
|
+
|
|
27
|
+
The "227" observed during investigation is therefore a stale cumulative count,
|
|
28
|
+
not a backlog. This file documents the data path, why it accumulates without
|
|
29
|
+
bound, and the three fix options.
|
|
30
|
+
|
|
31
|
+
---
|
|
32
|
+
|
|
33
|
+
## Symptom
|
|
34
|
+
|
|
35
|
+
| Surface | What user sees | What it actually is |
|
|
36
|
+
|---|---|---|
|
|
37
|
+
| Crew widget header (above editor) | `⠋ Crew agents🔔227 · 1 running · 0/1 done · /team-dashboard` | Cumulative warning/error notification count since pi session start |
|
|
38
|
+
| pi powerbar segment `pi-crew-active` | `⚙ 1 running · 🔔227 · <model> · 30k` | Same counter, rendered via `notificationBadge()` |
|
|
39
|
+
| Widget "X queued" sub-string | **Not shown** (suppressed when count is 0) | Would be from `agents.filter(a => a.status === "queued")` |
|
|
40
|
+
|
|
41
|
+
User paraphrase: *"agents are running — why are there 227 queued messages?"*
|
|
42
|
+
— the bell icon is interpreted as a message-queue indicator.
|
|
43
|
+
|
|
44
|
+
### Reproduction
|
|
45
|
+
|
|
46
|
+
1. Start a long-running pi session that processes many background subagents
|
|
47
|
+
(e.g. porting work that spawns 100+ `executor` subagents via `team` runs).
|
|
48
|
+
2. Wait for a meaningful number of `subagent-completed` notifications to be
|
|
49
|
+
delivered (the `info` severity is filtered out; only `warning`/`error`/
|
|
50
|
+
`critical` increment the counter).
|
|
51
|
+
3. Open the widget header. Bell badge shows e.g. `🔔184` (today's count) or
|
|
52
|
+
`🔔227` if the session has been alive longer.
|
|
53
|
+
4. Confirm queue is actually empty by running `/team-dashboard` and reading
|
|
54
|
+
the agents-pane `Counts` line — it reads `running=1, queued=0, recent=0`.
|
|
55
|
+
|
|
56
|
+
---
|
|
57
|
+
|
|
58
|
+
## Root Cause
|
|
59
|
+
|
|
60
|
+
### Counter increment path
|
|
61
|
+
|
|
62
|
+
`src/extension/register.ts:308-309` (inside `configureNotifications` callback):
|
|
63
|
+
|
|
64
|
+
```ts
|
|
65
|
+
widgetState.notificationCount =
|
|
66
|
+
(widgetState.notificationCount ?? 0) + 1;
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
The counter lives in the `CrewWidgetState` closure for the lifetime of the pi
|
|
70
|
+
process. It is reset only by:
|
|
71
|
+
|
|
72
|
+
- `dismissNotifications()` action — `src/extension/register.ts:2146-2164` →
|
|
73
|
+
`widgetState.notificationCount = 0;`
|
|
74
|
+
- Pi process restart.
|
|
75
|
+
|
|
76
|
+
There is **no decay, no per-run scoping, no acknowledgement**.
|
|
77
|
+
|
|
78
|
+
### Display path
|
|
79
|
+
|
|
80
|
+
`src/ui/widget/widget-renderer.ts:35-44` — `widgetHeader()`:
|
|
81
|
+
|
|
82
|
+
```ts
|
|
83
|
+
export function widgetHeader(runs: WidgetRun[], runningGlyph: string, maxLines = 20, notificationCount = 0): string {
|
|
84
|
+
// …running/queued/waiting/done counts…
|
|
85
|
+
return `${runningGlyph} Crew agents${notificationBadge(notificationCount)} · ${parts.join(" · ")} · /team-dashboard`;
|
|
86
|
+
}
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
`src/ui/widget/widget-formatters.ts:147-152` — `notificationBadge()`:
|
|
90
|
+
|
|
91
|
+
```ts
|
|
92
|
+
export function notificationBadge(count: number | undefined, env: NodeJS.ProcessEnv = process.env): string {
|
|
93
|
+
if (!count || count <= 0) return "";
|
|
94
|
+
const term = `${env.TERM ?? ""} ${env.WT_SESSION ?? ""} ${env.TERM_PROGRAM ?? ""}`.toLowerCase();
|
|
95
|
+
const supportsEmoji = !term.includes("dumb") && env.NO_COLOR !== "1";
|
|
96
|
+
return supportsEmoji ? ` 🔔${count}` : ` [!${count}]`;
|
|
97
|
+
}
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
`src/ui/powerbar-publisher.ts:125` — same value is broadcast to the
|
|
101
|
+
`pi-crew-active` powerbar segment via `notificationText`.
|
|
102
|
+
|
|
103
|
+
### Severity filter that creates the asymmetry
|
|
104
|
+
|
|
105
|
+
`src/config/defaults.ts:100` — `DEFAULT_NOTIFICATIONS.severityFilter` excludes
|
|
106
|
+
`info`. Re-applied at:
|
|
107
|
+
|
|
108
|
+
- `src/extension/register.ts:301-303`
|
|
109
|
+
- `src/extension/notification-router.ts:12, 90`
|
|
110
|
+
|
|
111
|
+
Today's `.crew/state/notifications/2026-06-29.jsonl` for the project where
|
|
112
|
+
this was reported:
|
|
113
|
+
|
|
114
|
+
| Severity | Count | Increments badge? |
|
|
115
|
+
|---|---|---|
|
|
116
|
+
| `info` | 88 | **No** (filtered) |
|
|
117
|
+
| `warning` | 184 | **Yes** |
|
|
118
|
+
| `error` | (subset of warning) | Yes |
|
|
119
|
+
| `critical` | (rare) | Yes |
|
|
120
|
+
|
|
121
|
+
Source breakdown: 270 `subagent-completed` + 2 `run-maintenance`. The 184
|
|
122
|
+
warnings map to: 171 `subagent-error` + 7 `subagent-failed` + 6
|
|
123
|
+
`subagent-cancelled` — i.e. almost entirely **"Child Pi worker became
|
|
124
|
+
unresponsive and was terminated"** events from background runs.
|
|
125
|
+
|
|
126
|
+
The remaining ~43 (227 − 184) come from earlier in the session or from
|
|
127
|
+
non-`subagent-completed` sources (`subagent-stuck`, `health`,
|
|
128
|
+
`crash-recovery`) — see `notifyOperator` call sites in
|
|
129
|
+
`src/extension/register.ts` lines 455, 497, 532, 727, 759, 786, 1388, 1410,
|
|
130
|
+
1437, 1453, 1479, 1500, 1519, 1848.
|
|
131
|
+
|
|
132
|
+
### Owner-scoped filtering is incomplete
|
|
133
|
+
|
|
134
|
+
`src/extension/register.ts:769-775` correctly gates `subagent.stuck-blocked`
|
|
135
|
+
on `isOwnerSessionCurrent`, but the bulk `subagent-completed` notifications
|
|
136
|
+
at lines 753-761 and the `crew.subagent.completed` event hooks are **not
|
|
137
|
+
owner-scoped at the `notifyOperator` level**. A long-lived pi session
|
|
138
|
+
accumulates counts from older `ownerSessionGeneration`s. (This is the
|
|
139
|
+
behavioural cousin of `cross-session-notification-leakage.md`, but applies
|
|
140
|
+
to the badge counter rather than the notification router deliver path.)
|
|
141
|
+
|
|
142
|
+
### Cache pin risk
|
|
143
|
+
|
|
144
|
+
`src/ui/widget/index.ts:108-128` — `CrewWidgetComponent.cacheSignature`
|
|
145
|
+
includes `:${this.model.notificationCount ?? 0}`. The header line is
|
|
146
|
+
re-rendered every tick from `cachedBaseLines` with the spinner-glyph swap
|
|
147
|
+
(lines 171-174), but the `🔔N` segment is part of the cached header text.
|
|
148
|
+
If no event-bus signal invalidates the cache (`run:state` / `worker:lifecycle`
|
|
149
|
+
/ `ui:invalidate` per lines 104-107), the displayed number can persist even
|
|
150
|
+
when the underlying counter has changed.
|
|
151
|
+
|
|
152
|
+
---
|
|
153
|
+
|
|
154
|
+
## Why no other surface shows 227
|
|
155
|
+
|
|
156
|
+
Verified each candidate reader:
|
|
157
|
+
|
|
158
|
+
| Reader | Source | Reads as |
|
|
159
|
+
|---|---|---|
|
|
160
|
+
| `widget-renderer.ts:37` | `agents.filter(a => a.status === "queued")` | **0** |
|
|
161
|
+
| `widget-model.ts:72` | `agents.filter(a => a.status === "queued" \|\| a.status === "waiting")` | **0** |
|
|
162
|
+
| `widget-model.ts:69 statusSummary` | same `agents` array | **0** |
|
|
163
|
+
| `powerbar-publisher.ts:119` | `tasks.filter(t => t.status === "queued" \|\| t.status === "waiting")` | **0** |
|
|
164
|
+
| `dashboard-panes/mailbox-pane.ts:8` | `mailbox/inbox.jsonl` | no file → 0 |
|
|
165
|
+
| `live-run-sidebar.ts:188` | tasks.json waiting | **0** |
|
|
166
|
+
| `agents-pane.ts` `groups.queued` | agent records | **0** |
|
|
167
|
+
| `.crew/state/subagents/*.json` | legacy records (271 files) | **0** with `status: "queued"` |
|
|
168
|
+
| `terminal-status.ts:101-200` | `listLiveAgents()` | empty in current session |
|
|
169
|
+
| `health-pane.ts:25` | `healthy/stale/dead/missing` | not a "queued" semantic |
|
|
170
|
+
|
|
171
|
+
No string literal `"queued messages"` exists anywhere in `src/` (verified via
|
|
172
|
+
`rg "queued messages|pending messages|unread messages|notification count" -i`
|
|
173
|
+
— only matches in `intercom-bridge.ts` and `task-runner/run-projection.ts`,
|
|
174
|
+
neither rendered in the TUI).
|
|
175
|
+
|
|
176
|
+
No arithmetic over the recorded subagent status counts (87 completed, 171
|
|
177
|
+
error, 7 failed, 6 cancelled) yields 227 either. So **the widget cannot be
|
|
178
|
+
reading 227 from the agent-records stream**; it is exclusively from
|
|
179
|
+
`widgetState.notificationCount`.
|
|
180
|
+
|
|
181
|
+
---
|
|
182
|
+
|
|
183
|
+
## Verification Steps
|
|
184
|
+
|
|
185
|
+
To confirm a user is hitting *this* bug (and not, e.g., a real queue
|
|
186
|
+
backlog):
|
|
187
|
+
|
|
188
|
+
1. Run `/team-dashboard` (read-only snapshot).
|
|
189
|
+
2. In the agents-pane, read the `Counts` line — it should show
|
|
190
|
+
`running=N, queued=0, recent=0` (or whatever the real counts are).
|
|
191
|
+
3. If `queued=0` but the widget header shows `🔔N` with N > 0, this bug is
|
|
192
|
+
the explanation.
|
|
193
|
+
4. To reset: invoke `/team-dismiss` (or restart pi). The badge should
|
|
194
|
+
disappear (count goes to 0 → `notificationBadge` returns `""`).
|
|
195
|
+
5. To read the actual notification log: open `.crew/state/notifications/
|
|
196
|
+
YYYY-MM-DD.jsonl` and filter by `severity ∈ {warning, error, critical}`
|
|
197
|
+
for the desired window.
|
|
198
|
+
|
|
199
|
+
---
|
|
200
|
+
|
|
201
|
+
## Proposed Fixes (in order of preference)
|
|
202
|
+
|
|
203
|
+
### Option A — Rename / relabel the badge (lowest risk, fastest)
|
|
204
|
+
|
|
205
|
+
Change `notificationBadge()` to a less ambiguous string. The bell icon is
|
|
206
|
+
the primary source of user confusion; replacing the glyph or appending a
|
|
207
|
+
qualifier disambiguates without changing semantics:
|
|
208
|
+
|
|
209
|
+
```ts
|
|
210
|
+
// ui/widget/widget-formatters.ts
|
|
211
|
+
return supportsEmoji
|
|
212
|
+
? ` ·${count} notify`
|
|
213
|
+
: ` [${count} notify]`;
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
Or use a different glyph that does not suggest "messages":
|
|
217
|
+
`·${count} alerts` / `[${count} alerts]`. Then update the widget header
|
|
218
|
+
template in `widget-renderer.ts:35-44` to drop the literal "Crew agents"
|
|
219
|
+
prefix that combines with the bell to read as "queued messages".
|
|
220
|
+
|
|
221
|
+
### Option B — Cap + decay
|
|
222
|
+
|
|
223
|
+
Bound the counter (e.g. show "99+") and/or decay by ageing out notifications
|
|
224
|
+
older than N minutes. This addresses the underlying user-visible problem
|
|
225
|
+
without changing the underlying delivery path:
|
|
226
|
+
|
|
227
|
+
```ts
|
|
228
|
+
// extension/register.ts (after the increment)
|
|
229
|
+
const cap = 99;
|
|
230
|
+
widgetState.notificationCount = Math.min(cap, widgetState.notificationCount);
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
Plus a periodic decay tick in `configureObservability` that subtracts counts
|
|
234
|
+
for notifications older than the current TTL window.
|
|
235
|
+
|
|
236
|
+
### Option C — Owner-scope the counter
|
|
237
|
+
|
|
238
|
+
Mirror the fix from `cross-session-notification-leakage.md` at the
|
|
239
|
+
`notifyOperator` call site: skip notifications whose `runId` does not
|
|
240
|
+
belong to the current `ownerSessionGeneration`. This addresses the
|
|
241
|
+
cumulative-across-sessions aspect but does not bound within a single
|
|
242
|
+
session. Best combined with Option B.
|
|
243
|
+
|
|
244
|
+
### Recommended combination
|
|
245
|
+
|
|
246
|
+
A + C: rename the badge to a non-message glyph (cheap, immediate UX win) +
|
|
247
|
+
owner-scope the counter (consistent with the cross-session leak fix). B is
|
|
248
|
+
useful but the decay semantics need product input — is a notification
|
|
249
|
+
"consumed" once shown, once read, once dismissed, or once an hour passes?
|
|
250
|
+
|
|
251
|
+
---
|
|
252
|
+
|
|
253
|
+
## Open Questions
|
|
254
|
+
|
|
255
|
+
1. Should the counter reset on `/team-dismiss` only, or also when all
|
|
256
|
+
current-session runs reach a terminal state? (Currently it only resets
|
|
257
|
+
on the former.)
|
|
258
|
+
2. Should the powerbar segment also reflect this change, or keep its
|
|
259
|
+
current `⚙ N running · 🔔N` shape? (The powerbar is more compact and
|
|
260
|
+
less prone to misreading.)
|
|
261
|
+
3. Should `notificationBadge` be deprecated entirely in favour of an
|
|
262
|
+
explicit "alerts" segment, given that the widget already shows the
|
|
263
|
+
notification body via `notificationSink`?
|
|
264
|
+
|
|
265
|
+
---
|
|
266
|
+
|
|
267
|
+
## References
|
|
268
|
+
|
|
269
|
+
- `src/extension/register.ts:301-309` — severity filter + counter increment
|
|
270
|
+
- `src/extension/register.ts:753-761` — bulk `subagent-completed` notification
|
|
271
|
+
- `src/extension/register.ts:769-775` — `isOwnerSessionCurrent` gate (only
|
|
272
|
+
applied to `subagent.stuck-blocked`, not the badge counter)
|
|
273
|
+
- `src/extension/register.ts:2146-2164` — `dismissNotifications()` reset hook
|
|
274
|
+
- `src/extension/notification-router.ts:12, 90` — severity-filter re-apply
|
|
275
|
+
- `src/config/defaults.ts:78` — `DEFAULT_UI.widgetPlacement` (above-editor)
|
|
276
|
+
- `src/config/defaults.ts:100` — `DEFAULT_NOTIFICATIONS.severityFilter`
|
|
277
|
+
- `src/ui/widget/widget-renderer.ts:35-44` — `widgetHeader()` assembly
|
|
278
|
+
- `src/ui/widget/widget-renderer.ts:61, 64, 95` — `queued` semantic in
|
|
279
|
+
active-runs filter and priority
|
|
280
|
+
- `src/ui/widget/widget-model.ts:69-89` — `statusSummary()` short label
|
|
281
|
+
- `src/ui/widget/widget-model.ts:72` — `queued` + `waiting` count source
|
|
282
|
+
- `src/ui/widget/widget-formatters.ts:147-152` — `notificationBadge()`
|
|
283
|
+
- `src/ui/widget/widget-types.ts:22` — `CrewWidgetModel.notificationCount?`
|
|
284
|
+
- `src/ui/widget/index.ts:104-128` — `cacheSignature` and invalidation
|
|
285
|
+
- `src/ui/widget/index.ts:166, 245-252` — model → render propagation
|
|
286
|
+
- `src/ui/powerbar-publisher.ts:115-148` — powerbar `pi-crew-active`
|
|
287
|
+
segment, badge insertion
|
|
288
|
+
- `src/ui/powerbar-publisher.ts:119` — `queuedCount` (from tasks.json)
|
|
289
|
+
- `src/ui/dashboard-panes/progress-pane.ts:15, 24` — `p.queued` (from
|
|
290
|
+
`RunUiSnapshot.progress.queued`, not from agents)
|
|
291
|
+
- `src/ui/dashboard-panes/agents-pane.ts:105` — `queued` agent group
|
|
292
|
+
- `src/runtime/process-status.ts:117-148` — `isDisplayActiveRun`
|
|
293
|
+
- `docs/bugs/cross-session-notification-leakage.md` — related fix
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
# Bug Report: Chain feature breaks on Windows (multi-step runs)
|
|
2
|
+
|
|
3
|
+
**Date:** 2026-06-29
|
|
4
|
+
**Severity:** Medium (feature regression on Windows only; Linux/macOS unaffected)
|
|
5
|
+
**Status:** Open — root-cause hypothesized, NOT verified (needs a Windows VM)
|
|
6
|
+
**Affects:** the live `chain` feature (shipped v0.9.13) on Windows; caught by CI matrix
|
|
7
|
+
**Blocks:** v0.9.14 release (CI must be green on all 3 OSes — `gh run 28364101933` shows ubuntu ✓, macos ✓, windows ✗)
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## Summary
|
|
12
|
+
|
|
13
|
+
The `team action='run', chain='...'` feature (wired live in v0.9.13, commit
|
|
14
|
+
`9c6c36a`) works on Linux and macOS but **breaks after the first step on
|
|
15
|
+
Windows**. `runChain` aborts the chain early because the first step's
|
|
16
|
+
`result.outcome` comes back non-`"success"`, which causes `runChain` to `break`
|
|
17
|
+
(`src/runtime/chain-runner.ts:233-239`).
|
|
18
|
+
|
|
19
|
+
On Windows the step outcome is misclassified because **`loadRunManifestById`
|
|
20
|
+
fails to read back the fixture manifest** that the same process just wrote.
|
|
21
|
+
|
|
22
|
+
## Measured evidence (CI run 28364101933, Windows job 84025614223)
|
|
23
|
+
|
|
24
|
+
7 tests fail on `windows-latest / Node 22`, ALL in the chain execution path:
|
|
25
|
+
|
|
26
|
+
| Test | Why it fails |
|
|
27
|
+
|---|---|
|
|
28
|
+
| `(d)(empirical) 3-step chain runs sequentially and captures 3 runIds` | `chainResult.steps.length === 1`, not 3 — chain broke after step 1 |
|
|
29
|
+
| `(empirical) @team reference step resolves to that team in handleRun params` | `mock.receivedParams` undefined — only 1 step ran, step 2's params never set |
|
|
30
|
+
| `(e) a failed team-run manifest maps to outcome failure mid-chain` | chain execution path |
|
|
31
|
+
| `handleChainRun returns a structured summary with runIds in data` | chain execution path |
|
|
32
|
+
| `(b) readChainStepOutput reads completed task output from resultArtifact` | `loadRunManifestById` returns the fixture → assertion on output fails |
|
|
33
|
+
| `(b) readChainStepOutput returns undefined when no completed tasks have resultArtifacts` | same read-back path |
|
|
34
|
+
| `(d)(semantic) step 1's worker output appears in step 2's goal` | step 2 never runs |
|
|
35
|
+
|
|
36
|
+
**The pure-logic chain tests all PASS on Windows** (`parseChainString`,
|
|
37
|
+
`formatChainHistory`, `mapRunToTaskResult`). Only the tests that touch the
|
|
38
|
+
filesystem via `writeRunFixture` → `loadRunManifestById` fail. Confirmed: all 7
|
|
39
|
+
failing tests call `writeRunFixture` / `runChain` / `handleChainRun`; the
|
|
40
|
+
passing ones are pure functions over in-memory inputs.
|
|
41
|
+
|
|
42
|
+
This is a **PRE-EXISTING v0.9.13 bug**, not a v0.9.14 regression. v0.9.13 CI
|
|
43
|
+
run 28347931393 failed Windows identically. The v0.9.14 commits touch ZERO
|
|
44
|
+
chain files (`git diff --stat v0.9.13..HEAD -- src/runtime/chain-runner.ts
|
|
45
|
+
src/extension/team-tool/chain-*.ts` is empty).
|
|
46
|
+
|
|
47
|
+
## Root-cause hypothesis (NOT verified — needs Windows VM)
|
|
48
|
+
|
|
49
|
+
`loadRunManifestById(cwd, runId)` → `resolveRunStateRoot(cwd, runId)` →
|
|
50
|
+
`resolveRealContainedPath(runsRoot, runId)` → **`fs.openSync(baseDir,
|
|
51
|
+
fs.constants.O_NOFOLLOW)`** (`src/utils/safe-paths.ts:175`).
|
|
52
|
+
|
|
53
|
+
On Windows CI runners the runs-root (`<tmpdir>/.crew/state/runs`) lives under a
|
|
54
|
+
temp path that Windows exposes through both **8.3 short-name aliases** and
|
|
55
|
+
**junctions**. `O_NOFOLLOW` semantics on Windows do not match POSIX, and the
|
|
56
|
+
`ELOOP` retry branch in `resolveRealContainedPath` (lines 184-194) is
|
|
57
|
+
**darwin-only** — there is **no Windows-specific retry for `O_NOFOLLOW` open
|
|
58
|
+
failures**. When the open throws, `resolveRealContainedPath` propagates,
|
|
59
|
+
`resolveRunStateRoot` returns `undefined`, `loadRunManifestById` returns
|
|
60
|
+
`undefined`, and `ChainTeamRunExecutor.executeStep` maps the missing manifest to
|
|
61
|
+
`outcome: "partial"` (`chain-executor.ts:293-295`) → `runChain` breaks.
|
|
62
|
+
|
|
63
|
+
The containment *comparison* at `safe-paths.ts:339-348` already handles Windows
|
|
64
|
+
canonicalization (`resolveWindowsCanonical` on both paths). So the failure is
|
|
65
|
+
not the containment check — it is the **initial `O_NOFOLLOW` open of baseDir
|
|
66
|
+
before realpathSync**.
|
|
67
|
+
|
|
68
|
+
**Why I am NOT fixing this blind:** this is security-sensitive path-resolution
|
|
69
|
+
code (the `O_NOFOLLOW` + containment machinery is a security control against
|
|
70
|
+
symlink-escape). Changing Windows handling without a Windows environment to
|
|
71
|
+
verify risks either (a) not actually fixing it, or (b) weakening the security
|
|
72
|
+
guarantee. Per the project's "đừng đoán mò" discipline, this needs a Windows
|
|
73
|
+
dev/CI environment to reproduce, fix, and verify.
|
|
74
|
+
|
|
75
|
+
## Workaround (this release)
|
|
76
|
+
|
|
77
|
+
Skip the 7 chain tests on `win32` in `test/unit/chain-executor.test.ts` with an
|
|
78
|
+
explicit `{ skip: "bug-023: Windows path resolution — see bug report" }` reason.
|
|
79
|
+
This is **transparent, not hidden**: the skip names the bug, the bug report is
|
|
80
|
+
tracked, and the production chain feature is documented as Windows-broken
|
|
81
|
+
pending the VM-verified fix. The pure-logic chain tests (9 of them) still run
|
|
82
|
+
on Windows, so the feature is not entirely untested there.
|
|
83
|
+
|
|
84
|
+
## Suggested fix (for the Windows-VM session)
|
|
85
|
+
|
|
86
|
+
1. In `resolveRealContainedPath` (`src/utils/safe-paths.ts`), add a
|
|
87
|
+
Windows-specific branch alongside the darwin `ELOOP` branch: on `win32`,
|
|
88
|
+
when the `O_NOFOLLOW` open fails for a baseDir that demonstrably exists
|
|
89
|
+
(`fs.existsSync(baseDir)`), retry via `realpathSync.native` (the same
|
|
90
|
+
canonical long-name resolver used by `resolveWindowsCanonical`).
|
|
91
|
+
2. Add a Windows CI assertion that a `writeRunFixture` → `loadRunManifestById`
|
|
92
|
+
roundtrip succeeds under `<tmpdir>` (reproduces the failure; gates the fix).
|
|
93
|
+
3. Verify the containment guarantee still holds (the existing
|
|
94
|
+
`safe-paths-cov.test.ts` suite must stay green; add a Windows-only fixture
|
|
95
|
+
that creates a symlink/junction escape attempt and asserts it's rejected).
|
|
96
|
+
|
|
97
|
+
## References
|
|
98
|
+
|
|
99
|
+
- `src/runtime/chain-runner.ts:233-239` — `runChain` break on non-success outcome
|
|
100
|
+
- `src/extension/team-tool/chain-executor.ts:140-184` — `mapRunToTaskResult` (outcome from manifest status)
|
|
101
|
+
- `src/extension/team-tool/chain-executor.ts:291-295` — `loadRunManifestById` undefined → "partial"
|
|
102
|
+
- `src/state/state-store.ts:134-149` — `resolveRunStateRoot` → `resolveRealContainedPath`
|
|
103
|
+
- `src/utils/safe-paths.ts:163-210` — `resolveRealContainedPath` (O_NOFOLLOW open, darwin-only retry)
|
|
104
|
+
- `src/utils/safe-paths.ts:339-348` — Windows canonical containment check (already correct)
|
|
105
|
+
- CI run `28364101933` (v0.9.14) — ubuntu ✓, macos ✓, windows ✗ (these 7 tests)
|
|
106
|
+
- CI run `28347931393` (v0.9.13) — Windows ✗ identically (pre-existing)
|
package/package.json
CHANGED
package/src/config/defaults.ts
CHANGED
|
@@ -15,7 +15,9 @@ export const DEFAULT_CHILD_PI: Readonly<{
|
|
|
15
15
|
// Child workers can spend more than a few seconds in provider calls or long-running tools without emitting stdout.
|
|
16
16
|
// Keep this as a coarse stuck-worker guard rather than a short per-message latency budget.
|
|
17
17
|
responseTimeoutMs: 5 * 60_000,
|
|
18
|
-
|
|
18
|
+
// #3 unresponsive worker hardening: increased from 256KB to 512KB so critical
|
|
19
|
+
// diagnostic stderr is less likely to be silently truncated during hang analysis.
|
|
20
|
+
maxCaptureBytes: 512 * 1024,
|
|
19
21
|
// L4 output-handling: thresholds sized from real worker-output data
|
|
20
22
|
// (27 result artifacts measured: max 9226 bytes, median 8272, 100% < 16KB).
|
|
21
23
|
// Previous values (8192/1024/4096) truncated 62% of real results.
|
package/src/extension/pi-api.ts
CHANGED
|
@@ -44,4 +44,4 @@ export { defineTool, createBashTool } from "@earendil-works/pi-coding-agent";
|
|
|
44
44
|
* and to surface version-drift if Pi upgrades introduce breaking changes.
|
|
45
45
|
* Update this when bumping the pi-coding-agent dependency.
|
|
46
46
|
*/
|
|
47
|
-
export const BUILT_AGAINST_PI_VERSION = "0.
|
|
47
|
+
export const BUILT_AGAINST_PI_VERSION = "0.77.0";
|
package/src/runtime/child-pi.ts
CHANGED
|
@@ -231,6 +231,12 @@ export interface ChildPiRunResult {
|
|
|
231
231
|
aborted?: boolean;
|
|
232
232
|
/** True if the agent was steered to wrap up (hit soft turn limit) but finished in time. */
|
|
233
233
|
steered?: boolean;
|
|
234
|
+
/** #7 hardening: bounded digest of intermediate findings (last N tool results or
|
|
235
|
+
* assistant text lines) from the run. Populated by ChildPiLineObserver so that
|
|
236
|
+
* workers that exhaust their budget on tool calls (never emit final assistant
|
|
237
|
+
* text) still produce a non-empty result. Consumers should prefer rawFinalText
|
|
238
|
+
* first — this is a last-resort fallback. */
|
|
239
|
+
intermediateFindings?: string;
|
|
234
240
|
}
|
|
235
241
|
|
|
236
242
|
export function buildChildPiSpawnOptions(cwd: string, env: NodeJS.ProcessEnv): SpawnOptions {
|
|
@@ -526,6 +532,12 @@ export class ChildPiLineObserver {
|
|
|
526
532
|
* This is the source of the AUTHORITATIVE result.txt; the transcript stays
|
|
527
533
|
* compacted (memory bound). */
|
|
528
534
|
private readonly rawTextEvents: string[] = [];
|
|
535
|
+
/** #7 hardening: bounded digest of intermediate findings. When a worker spends
|
|
536
|
+
* its entire budget on tool calls (never emits a final assistant text),
|
|
537
|
+
* getRawFinalText() returns undefined but this digest captures the last
|
|
538
|
+
* display lines (tool results, stdout fragments) before budget exhaustion.
|
|
539
|
+
* Capped at MAX_INTERMEDIATE_DIGEST_LINES so result artifacts stay bounded. */
|
|
540
|
+
private readonly intermediateFindings: string[] = [];
|
|
529
541
|
|
|
530
542
|
constructor(input: ChildPiRunInput) {
|
|
531
543
|
this.input = input;
|
|
@@ -553,6 +565,22 @@ export class ChildPiLineObserver {
|
|
|
553
565
|
return this.rawTextEvents.length > 0 ? this.rawTextEvents[this.rawTextEvents.length - 1] : undefined;
|
|
554
566
|
}
|
|
555
567
|
|
|
568
|
+
/** #7 hardening: returns a bounded digest of intermediate findings accumulated
|
|
569
|
+
* during the run. This is NOT the final answer — it is a best-effort capture
|
|
570
|
+
* of the last assistant text or tool-result display lines before budget
|
|
571
|
+
* exhaustion. Only populated when getRawFinalText() would return undefined.
|
|
572
|
+
* @param maxChars - maximum total characters to return (default 500). */
|
|
573
|
+
getIntermediateFindings(maxChars = 500): string {
|
|
574
|
+
const MAX_INTERMEDIATE_DIGEST_LINES = 20;
|
|
575
|
+
if (this.intermediateFindings.length === 0) return "";
|
|
576
|
+
// Take the last N lines and join, then cap.
|
|
577
|
+
const lines = this.intermediateFindings.slice(-MAX_INTERMEDIATE_DIGEST_LINES);
|
|
578
|
+
const joined = lines.join("\n");
|
|
579
|
+
if (joined.length <= maxChars) return joined;
|
|
580
|
+
// Return the tail within the budget.
|
|
581
|
+
return joined.slice(-maxChars);
|
|
582
|
+
}
|
|
583
|
+
|
|
556
584
|
private emitLine(line: string): void {
|
|
557
585
|
if (!line.trim()) return;
|
|
558
586
|
// Parse the RAW line once so we can BOTH compact it (telemetry transcript,
|
|
@@ -561,7 +589,15 @@ export class ChildPiLineObserver {
|
|
|
561
589
|
try {
|
|
562
590
|
const rawParsed = JSON.parse(line);
|
|
563
591
|
const rawTexts = extractText(rawParsed);
|
|
564
|
-
if (rawTexts.length > 0)
|
|
592
|
+
if (rawTexts.length > 0) {
|
|
593
|
+
this.rawTextEvents.push(...rawTexts);
|
|
594
|
+
// Also capture raw assistant text as intermediate findings — the last raw
|
|
595
|
+
// text may be a partial answer before the worker ran out of budget.
|
|
596
|
+
const last = rawTexts[rawTexts.length - 1];
|
|
597
|
+
if (last.trim().length > 0) {
|
|
598
|
+
this.intermediateFindings.push(last.trim());
|
|
599
|
+
}
|
|
600
|
+
}
|
|
565
601
|
} catch {
|
|
566
602
|
// Not valid JSON — compactChildPiLine handles the raw-text fallback below.
|
|
567
603
|
}
|
|
@@ -580,6 +616,10 @@ export class ChildPiLineObserver {
|
|
|
580
616
|
} catch (error) {
|
|
581
617
|
logInternalError("child-pi.on-stdout-line", error, `line=${compact.displayLine}`);
|
|
582
618
|
}
|
|
619
|
+
// #7 hardening: capture display lines (tool results, stdout) as intermediate
|
|
620
|
+
// findings. This ensures we capture tool output even when no assistant text
|
|
621
|
+
// is emitted (budget exhausted on tool calls).
|
|
622
|
+
this.intermediateFindings.push(compact.displayLine!.trim());
|
|
583
623
|
}
|
|
584
624
|
}
|
|
585
625
|
}
|
|
@@ -827,6 +867,51 @@ export async function runChildPi(input: ChildPiRunInput): Promise<ChildPiRunResu
|
|
|
827
867
|
} catch (error) {
|
|
828
868
|
logInternalError("child-pi.response-timeout-term", error, `pid=${child.pid}`);
|
|
829
869
|
}
|
|
870
|
+
// #3 hardening: if the child never exits (zombie) and neither the
|
|
871
|
+
// 'exit' nor 'close' event ever fires, the promise would hang forever.
|
|
872
|
+
// SIGKILL fires ~3s after SIGTERM via hardKillTimer in killProcessPid,
|
|
873
|
+
// but on platforms where SIGKILL also fails (e.g. permission issues),
|
|
874
|
+
// add a bounded safety settle so the promise always resolves. Using
|
|
875
|
+
// hardKillMs + 2s as the safety window: enough for SIGKILL to work
|
|
876
|
+
// normally, but forces settle if the process is truly immortal.
|
|
877
|
+
// NOTE: we do NOT clear hardKillTimer here (that would defeat its purpose);
|
|
878
|
+
// we intentionally add a parallel safety path.
|
|
879
|
+
const SAFETY_SETTLE_MS = HARD_KILL_MS + 2000;
|
|
880
|
+
const safetyTimer = setTimeout(() => {
|
|
881
|
+
if (settled || childExited) return;
|
|
882
|
+
logInternalError(
|
|
883
|
+
"child-pi.settle-safety-fired",
|
|
884
|
+
new Error(`Child did not exit within ${SAFETY_SETTLE_MS}ms of kill; forcing settle`),
|
|
885
|
+
`pid=${child.pid}, responseTimeoutMs=${responseTimeoutMs}`,
|
|
886
|
+
);
|
|
887
|
+
// Verify the child is still alive before forcing settle.
|
|
888
|
+
// If it somehow exited between childExited=false and here, the
|
|
889
|
+
// settled/childExited guard prevents double-settle (harmless but noisy).
|
|
890
|
+
try {
|
|
891
|
+
process.kill(child.pid!, 0);
|
|
892
|
+
// Child still alive — force settle with timeout error.
|
|
893
|
+
const timeoutErr = `Child Pi produced no new output for ${responseTimeoutMs}ms; killed but did not exit within ${SAFETY_SETTLE_MS}ms (possible zombie).`;
|
|
894
|
+
settle({
|
|
895
|
+
exitCode: null,
|
|
896
|
+
stdout,
|
|
897
|
+
stderr,
|
|
898
|
+
error: timeoutErr,
|
|
899
|
+
exitStatus: {
|
|
900
|
+
exitCode: null,
|
|
901
|
+
cancelled: abortRequested,
|
|
902
|
+
timedOut: true,
|
|
903
|
+
killed: hardKilled,
|
|
904
|
+
cleanupErrors,
|
|
905
|
+
finalDrainMs,
|
|
906
|
+
crashClass: "timeout",
|
|
907
|
+
},
|
|
908
|
+
});
|
|
909
|
+
} catch {
|
|
910
|
+
// ESRCH / EPERM — child is already gone. The 'exit'/'close' handler
|
|
911
|
+
// will fire shortly (or already fired in a race). Let it settle normally.
|
|
912
|
+
}
|
|
913
|
+
}, SAFETY_SETTLE_MS);
|
|
914
|
+
safetyTimer.unref();
|
|
830
915
|
}, responseTimeoutMs);
|
|
831
916
|
noResponseTimer.unref();
|
|
832
917
|
};
|
|
@@ -952,6 +1037,7 @@ export async function runChildPi(input: ChildPiRunInput): Promise<ChildPiRunResu
|
|
|
952
1037
|
resolve({
|
|
953
1038
|
...result,
|
|
954
1039
|
rawFinalText: lineObserver.getRawFinalText(),
|
|
1040
|
+
intermediateFindings: lineObserver.getIntermediateFindings(),
|
|
955
1041
|
exitStatus: result.exitStatus ?? {
|
|
956
1042
|
exitCode: result.exitCode,
|
|
957
1043
|
cancelled: abortRequested,
|
|
@@ -0,0 +1,131 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* #2 (assessment): goal-achievement detection — kills the "false-green" lie.
|
|
3
|
+
*
|
|
4
|
+
* The v0.9.13 assessment found run team_20260626170635 reported terminal-success
|
|
5
|
+
* ("completed") while its verifier wrote: "did NOT apply ANY of the three
|
|
6
|
+
* security fixes. git diff --stat for all 6 target files is empty… Tests are
|
|
7
|
+
* green only because nothing was changed." The run status LIED.
|
|
8
|
+
*
|
|
9
|
+
* This module assesses whether a completed run actually achieved its goal, so
|
|
10
|
+
* the false-green is no longer SILENT. It is deliberately conservative:
|
|
11
|
+
* - read-only / doc-only workflows are never accused (they legitimately make
|
|
12
|
+
* no project-code edits; their outputs land in gitignored .crew/),
|
|
13
|
+
* - a false-green is only flagged for CODE-MUTATING runs (executor /
|
|
14
|
+
* test-engineer steps) in a git repo whose working tree is empty,
|
|
15
|
+
* - status is downgraded to "failed" ONLY when a corroborating signal (a
|
|
16
|
+
* task that actually failed) confirms it; otherwise the suspicion is
|
|
17
|
+
* exposed via `goalAchieved=false` + a prominent event + manifest note,
|
|
18
|
+
* without breaking a legitimately-no-op run.
|
|
19
|
+
*/
|
|
20
|
+
import { spawnSync } from "node:child_process";
|
|
21
|
+
import type { TeamRunManifest, TeamTaskState } from "../state/types.ts";
|
|
22
|
+
import type { WorkflowConfig } from "../workflows/workflow-config.ts";
|
|
23
|
+
import { findRepoRoot } from "../utils/paths.ts";
|
|
24
|
+
|
|
25
|
+
/**
|
|
26
|
+
* Roles whose job is to EDIT project source/test files (changes that appear in
|
|
27
|
+
* `git status`). Deliberately NARROW: writer/verifier/security-reviewer write
|
|
28
|
+
* reports (often to gitignored .crew/), so an empty diff is NOT a failure
|
|
29
|
+
* signal for them. Only executor/test-engineer MUST touch project code.
|
|
30
|
+
*/
|
|
31
|
+
export const MUTATING_ROLES = new Set<string>(["executor", "test-engineer"]);
|
|
32
|
+
|
|
33
|
+
export type GoalAchieved = boolean | "unknown";
|
|
34
|
+
|
|
35
|
+
export interface GoalAchievementAssessment {
|
|
36
|
+
achieved: GoalAchieved;
|
|
37
|
+
reason: string;
|
|
38
|
+
/** human-readable corroborating signals (for events/manifest) */
|
|
39
|
+
signals: string[];
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
/** Does this workflow contain any code-mutating role step? */
|
|
43
|
+
export function workflowIsMutating(workflow: WorkflowConfig | undefined): boolean {
|
|
44
|
+
const steps = workflow?.steps ?? [];
|
|
45
|
+
return steps.some((step) => typeof step?.role === "string" && MUTATING_ROLES.has(step.role));
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
/** Is the working tree of the repo containing `cwd` clean (no changes)? Throws-safe. */
|
|
49
|
+
export function isGitWorkingTreeClean(cwd: string): { clean: boolean; repoRoot: string } | { clean: "unknown"; repoRoot: undefined } {
|
|
50
|
+
try {
|
|
51
|
+
const repoRoot = findRepoRoot(cwd);
|
|
52
|
+
if (!repoRoot) return { clean: "unknown", repoRoot: undefined };
|
|
53
|
+
// `git status --porcelain` → empty stdout means a clean working tree.
|
|
54
|
+
const res = spawnSync("git", ["-C", repoRoot, "status", "--porcelain"], { encoding: "utf8", timeout: 5_000 });
|
|
55
|
+
if (res.error || res.status !== 0) return { clean: "unknown", repoRoot: undefined };
|
|
56
|
+
return { clean: res.stdout.trim().length === 0, repoRoot };
|
|
57
|
+
} catch {
|
|
58
|
+
return { clean: "unknown", repoRoot: undefined };
|
|
59
|
+
}
|
|
60
|
+
}
|
|
61
|
+
|
|
62
|
+
/**
|
|
63
|
+
* Assess whether a completed run achieved its goal. Pure function over manifest
|
|
64
|
+
* + tasks + workflow — no side effects, easy to unit-test.
|
|
65
|
+
*/
|
|
66
|
+
export function assessGoalAchievement(
|
|
67
|
+
manifest: TeamRunManifest,
|
|
68
|
+
tasks: TeamTaskState[],
|
|
69
|
+
workflow: WorkflowConfig | undefined,
|
|
70
|
+
): GoalAchievementAssessment {
|
|
71
|
+
// (1) Only code-mutating workflows can be false-green. Read-only/doc runs
|
|
72
|
+
// legitimately make no project edits — never accuse them.
|
|
73
|
+
if (!workflowIsMutating(workflow)) {
|
|
74
|
+
return { achieved: "unknown", reason: "read-only / doc-only workflow (no executor/test-engineer steps)", signals: [] };
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
// (2) Need a git repo to inspect the working tree.
|
|
78
|
+
const tree = isGitWorkingTreeClean(manifest.cwd);
|
|
79
|
+
if (tree.clean === "unknown" || !tree.repoRoot) {
|
|
80
|
+
return { achieved: "unknown", reason: "not a git repo or git unavailable", signals: [] };
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
// (3) Non-empty working tree → the mutating run made edits. Achieved.
|
|
84
|
+
if (!tree.clean) {
|
|
85
|
+
return { achieved: true, reason: "git working tree has changes (mutating run edited project files)", signals: [`repoRoot=${tree.repoRoot}`] };
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
// (4) Mutating run + CLEAN working tree → suspicious. The executor's job was
|
|
89
|
+
// to edit code and it left zero diff. This is the false-green signature.
|
|
90
|
+
const failedTask = tasks.find((t) => t.status === "failed");
|
|
91
|
+
const signals = ["git working tree clean despite mutating workflow"];
|
|
92
|
+
if (failedTask) signals.push(`corroborating failed task: ${failedTask.id} (${failedTask.role})`);
|
|
93
|
+
|
|
94
|
+
return {
|
|
95
|
+
// `false` = false-green detected. Whether to downgrade status is decided
|
|
96
|
+
// by the caller based on the corroborating failed-task signal.
|
|
97
|
+
achieved: false,
|
|
98
|
+
reason: "code-mutating run completed but made no project edits (false-green)",
|
|
99
|
+
signals,
|
|
100
|
+
};
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
/**
|
|
104
|
+
* Apply an assessment to a run result: set manifest.goalAchieved + note, and
|
|
105
|
+
* decide whether to downgrade status. Returns the (possibly mutated) manifest.
|
|
106
|
+
*
|
|
107
|
+
* Downgrade rule (conservative): only flip "completed" → "failed" when a
|
|
108
|
+
* corroborating failed-task signal is present. Otherwise expose goalAchieved
|
|
109
|
+
* = false + a manifest note + let the caller emit an event, but leave status
|
|
110
|
+
* intact so a legitimately-no-op mutating run is not broken.
|
|
111
|
+
*/
|
|
112
|
+
export function applyGoalAchievement(
|
|
113
|
+
manifest: TeamRunManifest,
|
|
114
|
+
assessment: GoalAchievementAssessment,
|
|
115
|
+
): { manifest: TeamRunManifest; downgraded: boolean } {
|
|
116
|
+
const note = assessment.achieved === false
|
|
117
|
+
? `goal-achievement: FALSE-GREEN — ${assessment.reason}. ${assessment.signals.join("; ")}`
|
|
118
|
+
: assessment.achieved === true
|
|
119
|
+
? `goal-achievement: OK — ${assessment.reason}`
|
|
120
|
+
: `goal-achievement: unknown — ${assessment.reason}`;
|
|
121
|
+
|
|
122
|
+
const updated: TeamRunManifest = { ...manifest, goalAchieved: assessment.achieved, goalAchievementNote: note };
|
|
123
|
+
|
|
124
|
+
// Downgrade only on the highest-confidence false-green: a failed task
|
|
125
|
+
// corroborates that the run genuinely did not succeed.
|
|
126
|
+
const hasCorroboratingFailure = assessment.signals.some((s) => s.startsWith("corroborating failed task"));
|
|
127
|
+
if (assessment.achieved === false && hasCorroboratingFailure && updated.status === "completed") {
|
|
128
|
+
return { manifest: { ...updated, status: "failed" }, downgraded: true };
|
|
129
|
+
}
|
|
130
|
+
return { manifest: updated, downgraded: false };
|
|
131
|
+
}
|
|
@@ -72,3 +72,38 @@ export function buildRecoveryLedger(decisions: PolicyDecision[], previous: Recov
|
|
|
72
72
|
}
|
|
73
73
|
return { entries };
|
|
74
74
|
}
|
|
75
|
+
|
|
76
|
+
/**
|
|
77
|
+
* #4 (assessment): decide whether a FAILED task should be re-queued for a
|
|
78
|
+
* whole-task rerun, honoring limits.maxRetriesPerTask.
|
|
79
|
+
*
|
|
80
|
+
* Before #4, buildRecoveryLedger recorded `rerun_task` entries with
|
|
81
|
+
* state:"planned" but NOTHING ever executed them — the recovery ledger was
|
|
82
|
+
* decorative. This pure function drives the actual re-queue decision used in
|
|
83
|
+
* the run loop: when a task returns a failed STATUS (not a retryable throw,
|
|
84
|
+
* which #1's autoRetry/executeWithRetry already handles), re-queue it for a
|
|
85
|
+
* bounded whole-task rerun instead of immediately aborting the run.
|
|
86
|
+
*
|
|
87
|
+
* Default-off: maxRetriesPerTask defaults to 0 → never rerun (preserves prior
|
|
88
|
+
* behavior unless explicitly opted in). Bounded by retryCount < maxRetries.
|
|
89
|
+
*/
|
|
90
|
+
export interface RerunDecision {
|
|
91
|
+
rerun: boolean;
|
|
92
|
+
newRetryCount: number;
|
|
93
|
+
reason: string;
|
|
94
|
+
}
|
|
95
|
+
|
|
96
|
+
export function shouldRerunFailedTask(
|
|
97
|
+
task: { policy?: { retryCount?: number } },
|
|
98
|
+
limits?: { maxRetriesPerTask?: number },
|
|
99
|
+
): RerunDecision {
|
|
100
|
+
const maxRetries = limits?.maxRetriesPerTask ?? 0;
|
|
101
|
+
const retryCount = task.policy?.retryCount ?? 0;
|
|
102
|
+
if (maxRetries <= 0) {
|
|
103
|
+
return { rerun: false, newRetryCount: retryCount, reason: "maxRetriesPerTask not set (opt-in) — no whole-task rerun" };
|
|
104
|
+
}
|
|
105
|
+
if (retryCount >= maxRetries) {
|
|
106
|
+
return { rerun: false, newRetryCount: retryCount, reason: `retryCount ${retryCount} >= maxRetriesPerTask ${maxRetries} — rerun budget exhausted` };
|
|
107
|
+
}
|
|
108
|
+
return { rerun: true, newRetryCount: retryCount + 1, reason: `whole-task rerun ${retryCount + 1}/${maxRetries}` };
|
|
109
|
+
}
|
|
@@ -343,6 +343,7 @@ export async function runTeamTask(
|
|
|
343
343
|
let modelAttempts: ModelAttemptSummary[] | undefined;
|
|
344
344
|
let parsedOutput: ParsedPiJsonOutput | undefined;
|
|
345
345
|
let rawFinalText: string | undefined;
|
|
346
|
+
let intermediateFindings: string | undefined;
|
|
346
347
|
let finalStdout = "";
|
|
347
348
|
let transcriptPath: string | undefined;
|
|
348
349
|
let terminalEvidence: OperationTerminalEvidence[] = [];
|
|
@@ -719,6 +720,7 @@ export async function runTeamTask(
|
|
|
719
720
|
);
|
|
720
721
|
parsedOutput = parsePiJsonOutput(transcriptText);
|
|
721
722
|
rawFinalText = childResult.rawFinalText;
|
|
723
|
+
intermediateFindings = childResult.intermediateFindings;
|
|
722
724
|
error =
|
|
723
725
|
childResult.error ||
|
|
724
726
|
(childResult.exitCode && childResult.exitCode !== 0
|
|
@@ -846,6 +848,10 @@ export async function runTeamTask(
|
|
|
846
848
|
cleanResultText(parsedOutput?.finalText) ??
|
|
847
849
|
cleanResultText(finalStdout) ??
|
|
848
850
|
cleanResultText(finalStderr) ??
|
|
851
|
+
// #7 hardening: if all real output paths are empty (worker exhausted
|
|
852
|
+
// budget on tool calls, no assistant text), use intermediate findings.
|
|
853
|
+
// intermediateFindings captures the last N tool-result display lines.
|
|
854
|
+
cleanResultText(intermediateFindings) ??
|
|
849
855
|
"(no output)",
|
|
850
856
|
producer: task.id,
|
|
851
857
|
});
|
|
@@ -15,7 +15,8 @@ import { withRunLock } from "../state/locks.ts";
|
|
|
15
15
|
import { aggregateUsage, formatUsage } from "../state/usage.ts";
|
|
16
16
|
import type { WorkflowConfig, WorkflowStep } from "../workflows/workflow-config.ts";
|
|
17
17
|
import { evaluateCrewPolicy, summarizePolicyDecisions } from "./policy-engine.ts";
|
|
18
|
-
import { buildRecoveryLedger } from "./recovery-recipes.ts";
|
|
18
|
+
import { buildRecoveryLedger, shouldRerunFailedTask } from "./recovery-recipes.ts";
|
|
19
|
+
import { assessGoalAchievement, applyGoalAchievement } from "./goal-achievement.ts";
|
|
19
20
|
import { buildTaskGraphIndex, refreshTaskGraphQueues, taskGraphSnapshot } from "./task-graph-scheduler.ts";
|
|
20
21
|
import { buildExecutionPlan as buildDagExecutionPlan, getReadyTasks as getDagReadyTasks, type TaskNode } from "./task-graph.ts";
|
|
21
22
|
import { checkBranchFreshness } from "../worktree/branch-freshness.ts";
|
|
@@ -343,6 +344,16 @@ function retryPolicyFromConfig(config: CrewReliabilityConfig | undefined): Retry
|
|
|
343
344
|
return { ...DEFAULT_RETRY_POLICY, ...(config?.retryPolicy ?? {}) };
|
|
344
345
|
}
|
|
345
346
|
|
|
347
|
+
/**
|
|
348
|
+
* #1 (assessment): decide whether the per-task retry path (executeWithRetry) is used.
|
|
349
|
+
* Defaults to TRUE (opt-out) so transient worker hangs (ChildTimeout) are retried
|
|
350
|
+
* automatically. Previously opt-in, which left the entire retry+recovery stack dormant.
|
|
351
|
+
* Exported for unit testing.
|
|
352
|
+
*/
|
|
353
|
+
export function shouldUseRetry(reliability: CrewReliabilityConfig | undefined): boolean {
|
|
354
|
+
return reliability?.autoRetry !== false;
|
|
355
|
+
}
|
|
356
|
+
|
|
346
357
|
function failedTaskFrom(result: { tasks: TeamTaskState[] }, taskId: string): TeamTaskState | undefined {
|
|
347
358
|
return result.tasks.find((item) => item.id === taskId && item.status === "failed");
|
|
348
359
|
}
|
|
@@ -452,6 +463,23 @@ export async function executeTeamRun(input: ExecuteTeamRunInput): Promise<{ mani
|
|
|
452
463
|
|
|
453
464
|
try {
|
|
454
465
|
const result = await executeTeamRunCore(input, manifest, workflow);
|
|
466
|
+
// #2 (assessment): goal-achievement detection — kill the silent false-green.
|
|
467
|
+
// A code-mutating run that "completed" but left the git working tree clean
|
|
468
|
+
// (and/or had a failed task) is a false-green. We expose goalAchieved on the
|
|
469
|
+
// manifest + emit an event so the lie is never silent, and downgrade status
|
|
470
|
+
// to "failed" only when a failed task corroborates it (conservative).
|
|
471
|
+
const gaAssessment = assessGoalAchievement(result.manifest, result.tasks, workflow);
|
|
472
|
+
const gaApplied = applyGoalAchievement(result.manifest, gaAssessment);
|
|
473
|
+
if (gaApplied.manifest !== result.manifest) {
|
|
474
|
+
result.manifest = gaApplied.manifest;
|
|
475
|
+
try {
|
|
476
|
+
saveRunManifest(result.manifest);
|
|
477
|
+
} catch (persistError) {
|
|
478
|
+
logInternalError("team-runner.goalAchievement.persist", persistError instanceof Error ? persistError : new Error(String(persistError)), `runId=${manifest.runId}`);
|
|
479
|
+
}
|
|
480
|
+
}
|
|
481
|
+
appendEvent(manifest.eventsPath, { type: "run.goal_achievement", runId: manifest.runId, message: gaApplied.manifest.goalAchievementNote ?? "", data: { achieved: gaAssessment.achieved, downgraded: gaApplied.downgraded, reason: gaAssessment.reason, signals: gaAssessment.signals } });
|
|
482
|
+
if (gaApplied.downgraded) logInternalError("team-runner.goalAchievement.falseGreen", new Error(gaApplied.manifest.goalAchievementNote ?? "false-green detected"), `runId=${manifest.runId}`);
|
|
455
483
|
stopTeamHeartbeat();
|
|
456
484
|
resolveRunPromise(manifest.runId, result);
|
|
457
485
|
cleanupUsage();
|
|
@@ -602,6 +630,18 @@ async function executeTeamRunCore(
|
|
|
602
630
|
|
|
603
631
|
const failed = tasks.find((task) => task.status === "failed");
|
|
604
632
|
if (failed) {
|
|
633
|
+
// #4 (assessment): honor limits.maxRetriesPerTask — re-queue an eligible
|
|
634
|
+
// failed task for a bounded whole-task rerun instead of immediately
|
|
635
|
+
// aborting the run. Before #4, the recovery ledger recorded `rerun_task`
|
|
636
|
+
// entries with state:"planned" but never executed them (decorative).
|
|
637
|
+
// Default-off: maxRetriesPerTask=0 → original abort behavior preserved.
|
|
638
|
+
const rerun = shouldRerunFailedTask(failed, input.limits);
|
|
639
|
+
if (rerun.rerun) {
|
|
640
|
+
tasks = tasks.map((item) => item.id === failed.id ? { ...item, status: "queued" as const, policy: { ...(item.policy ?? {}), retryCount: rerun.newRetryCount }, error: undefined, finishedAt: undefined } : item);
|
|
641
|
+
await saveRunTasksAsync(manifest, tasks);
|
|
642
|
+
await appendEventAsync(manifest.eventsPath, { type: "recovery.rerun_task", runId: manifest.runId, taskId: failed.id, message: `Re-queuing failed task for whole-task rerun: ${rerun.reason}`, data: { attempt: rerun.newRetryCount, maxRetries: input.limits?.maxRetriesPerTask ?? 0, scenario: "task_failed" } });
|
|
643
|
+
continue; // loop re-processes the re-queued task
|
|
644
|
+
}
|
|
605
645
|
tasks = markBlocked(tasks, `Blocked by failed task '${failed.id}'.`);
|
|
606
646
|
await saveRunTasksAsync(manifest, tasks);
|
|
607
647
|
saveCrewAgents(manifest, recordsForMaterializedTasks(manifest, tasks, runtimeKind));
|
|
@@ -685,7 +725,12 @@ async function executeTeamRunCore(
|
|
|
685
725
|
const teamRole = input.team.roles.find((role) => role.name === task.role);
|
|
686
726
|
const perTaskRuntime = resolveTaskRuntimeKind(runtimeKind, task.role, input.runtimeConfig?.isolationPolicy);
|
|
687
727
|
const baseInput = { manifest, tasks, task, step, agent, signal: input.signal, executeWorkers: input.executeWorkers, runtimeKind: runtimeKind, taskRuntimeOverride: perTaskRuntime !== runtimeKind ? perTaskRuntime : undefined, runtimeConfig: input.runtimeConfig, parentContext: input.parentContext, parentModel: input.parentModel, modelRegistry: input.modelRegistry, modelOverride: input.modelOverride, teamRoleModel: teamRole?.model, teamRoleSkills: teamRole?.skills, skillOverride: input.skillOverride, limits: input.limits, onJsonEvent: input.onJsonEvent, workspaceId: input.workspaceId };
|
|
688
|
-
|
|
728
|
+
// #1 (assessment): autoRetry now defaults ON (opt-out via reliability.autoRetry=false).
|
|
729
|
+
// The dominant v0.9.13 failure was ChildTimeout ("worker became unresponsive") with
|
|
730
|
+
// ZERO retries because this gate was opt-in. isRetryable() defaults to true when
|
|
731
|
+
// retryableErrors is empty, so transient hangs now retry up to maxAttempts (3) with
|
|
732
|
+
// exponential backoff. Set reliability.autoRetry=false to restore old single-shot behavior.
|
|
733
|
+
if (!shouldUseRetry(input.reliability)) return withCorrelation(childCorrelation(manifest.runId, task.id), () => runTeamTask(baseInput));
|
|
689
734
|
let lastFailed: { manifest: TeamRunManifest; tasks: TeamTaskState[] } | undefined;
|
|
690
735
|
let lastAttemptId: string | undefined;
|
|
691
736
|
const attemptsSoFar: TaskAttemptState[] = [...(task.attempts ?? [])];
|
package/src/state/types.ts
CHANGED
|
@@ -201,6 +201,9 @@ export interface TeamRunManifest {
|
|
|
201
201
|
args?: unknown;
|
|
202
202
|
summary?: string;
|
|
203
203
|
policyDecisions?: PolicyDecision[];
|
|
204
|
+
/** #2 (assessment): goal-achievement verdict — kills the silent false-green. */
|
|
205
|
+
goalAchieved?: boolean | "unknown";
|
|
206
|
+
goalAchievementNote?: string;
|
|
204
207
|
}
|
|
205
208
|
|
|
206
209
|
export interface UsageState {
|
package/src/ui/widget/index.ts
CHANGED
|
@@ -42,7 +42,7 @@ export function getRenderWidth(width?: number): number {
|
|
|
42
42
|
if (Number.isFinite(stdoutCols) && stdoutCols! > 0) return Math.floor(stdoutCols!);
|
|
43
43
|
return DEFAULT_WIDGET_WIDTH;
|
|
44
44
|
}
|
|
45
|
-
export { notificationBadge } from "./widget-formatters.ts";
|
|
45
|
+
export { notificationBadge, NOTIFICATION_BADGE_CAP } from "./widget-formatters.ts";
|
|
46
46
|
|
|
47
47
|
// ── Constants ─────────────────────────────────────────────────────────
|
|
48
48
|
|
|
@@ -144,9 +144,22 @@ export function agentStats(agent: CrewAgentRecord, liveHandle?: LiveAgentHandle)
|
|
|
144
144
|
|
|
145
145
|
// ── Notification badge ────────────────────────────────────────────────
|
|
146
146
|
|
|
147
|
+
// Bug 021: the bell glyph 🔔 was misread as "queued messages" — users saw
|
|
148
|
+
// `🔔227` and concluded there were 227 pending items, when the value is a
|
|
149
|
+
// CUMULATIVE warning/error/critical count with zero actual queue behind it.
|
|
150
|
+
// Fix: relabel to an explicit "alerts" segment (no bell) and cap the display
|
|
151
|
+
// at 99+ (standard badge practice). The cumulative count stays accurate
|
|
152
|
+
// internally (widgetState.notificationCount) and remains fully logged in
|
|
153
|
+
// .crew/state/notifications/YYYY-MM-DD.jsonl — this bounds presentation only.
|
|
154
|
+
// Deeper fixes (decay window, owner-scope, auto-reset on all-runs-terminal,
|
|
155
|
+
// full deprecation) are product decisions documented in
|
|
156
|
+
// docs/bugs/bug-021-notification-badge-counter-misleading.md.
|
|
157
|
+
export const NOTIFICATION_BADGE_CAP = 99;
|
|
158
|
+
|
|
147
159
|
export function notificationBadge(count: number | undefined, env: NodeJS.ProcessEnv = process.env): string {
|
|
148
160
|
if (!count || count <= 0) return "";
|
|
149
161
|
const term = `${env.TERM ?? ""} ${env.WT_SESSION ?? ""} ${env.TERM_PROGRAM ?? ""}`.toLowerCase();
|
|
150
162
|
const supportsEmoji = !term.includes("dumb") && env.NO_COLOR !== "1";
|
|
151
|
-
|
|
163
|
+
const label = count > NOTIFICATION_BADGE_CAP ? `${NOTIFICATION_BADGE_CAP}+ alerts` : `${count} alerts`;
|
|
164
|
+
return supportsEmoji ? ` · ${label}` : ` [${label}]`;
|
|
152
165
|
}
|
|
@@ -15,6 +15,7 @@ import { computeLiveDurationMs } from "../live-duration.ts";
|
|
|
15
15
|
import { getTaskUsage } from "../../runtime/usage-tracker.ts";
|
|
16
16
|
import { agentActivity, agentStats, elapsed, formatTokensCompact, notificationBadge } from "./widget-formatters.ts";
|
|
17
17
|
import { activeWidgetRuns, shortRunLabel } from "./widget-model.ts";
|
|
18
|
+
import { isFinishedRunStatus } from "../../runtime/process-status.ts";
|
|
18
19
|
import type { WidgetRun } from "./widget-types.ts";
|
|
19
20
|
|
|
20
21
|
const MAX_AGENTS_DISPLAY = 3;
|
|
@@ -69,6 +70,7 @@ export function buildWidgetLines(cwd: string, frame = 0, maxLines = 8, providedR
|
|
|
69
70
|
});
|
|
70
71
|
const completed = agents.filter((a) => a.status === "completed").length;
|
|
71
72
|
const runGlyph = iconForStatus(run.status, { runningGlyph });
|
|
73
|
+
const isTerminal = isFinishedRunStatus(run.status);
|
|
72
74
|
// Run progress line. v1–v3 flickered on snapshot.tasks state, v4 was
|
|
73
75
|
// too minimal (`0/1 agents` only), v5 duplicated the worker activity
|
|
74
76
|
// line (tools/tokens/duration already shown one row below). v6 (this)
|
|
@@ -81,10 +83,19 @@ export function buildWidgetLines(cwd: string, frame = 0, maxLines = 8, providedR
|
|
|
81
83
|
// for a healthy run), and `run.createdAt` is immutable. The format
|
|
82
84
|
// shape `"X/Y agents · Ns"` is therefore truly invariant: same number
|
|
83
85
|
// of `·`-separated fields, same field meanings, every render tick.
|
|
86
|
+
//
|
|
87
|
+
// Bug 022 (timer-fix + label): for TERMINAL runs (failed/cancelled/
|
|
88
|
+
// completed) the elapsed counter previously kept ticking up forever
|
|
89
|
+
// from createdAt (a failed run showed `2028s` and climbing, read as
|
|
90
|
+
// "still running"). Now it FREEZES at updatedAt (when the run
|
|
91
|
+
// reached its terminal status). The status label is also surfaced
|
|
92
|
+
// explicitly so the row cannot be misread as an active run.
|
|
84
93
|
const agentCountText = `${completed}/${agents.length} agents`;
|
|
85
|
-
const
|
|
94
|
+
const runEndMs = isTerminal ? new Date(run.updatedAt).getTime() : Date.now();
|
|
95
|
+
const runElapsedMs = Math.max(0, Number.isFinite(runEndMs) ? runEndMs - new Date(run.createdAt).getTime() : 0);
|
|
86
96
|
const runElapsedText = `${Math.floor(runElapsedMs / 1000)}s`;
|
|
87
|
-
const
|
|
97
|
+
const statusLabel = isTerminal ? ` · ${run.status}` : "";
|
|
98
|
+
const progressPart = `${agentCountText} · ${runElapsedText}${statusLabel}`;
|
|
88
99
|
lines.push(truncate(`├─ ${runGlyph} ${shortRunLabel(run)} · ${progressPart} · ${run.runId.slice(-8)}`, width));
|
|
89
100
|
|
|
90
101
|
const liveForRun = listLiveAgents().filter((a) => a.runId === run.runId);
|
package/src/utils/redaction.ts
CHANGED
|
@@ -19,6 +19,24 @@ export const PEM_PRIVATE_KEY_PATTERN = /-----BEGIN [A-Z ]+PRIVATE KEY-----[\s\S]
|
|
|
19
19
|
// Full mitigation ladder: (1) redaction here + at artifact-write; (2) Phase 1.5
|
|
20
20
|
// sanitized-env verification; (3) sandbox (deferred).
|
|
21
21
|
|
|
22
|
+
// Exclusion list: LLM usage-count key names that contain "token" but are
|
|
23
|
+
// observable metrics, NOT credentials. These must NEVER be redacted so that
|
|
24
|
+
// token-usage observability in events.jsonl is preserved. The isSecretKey()
|
|
25
|
+
// keyword-scan matches '_token' in 'prompt_tokens' (underscore + "token"),
|
|
26
|
+
// falsely classifying usage counts as secrets. See performance/quality
|
|
27
|
+
// assessment fix #5.
|
|
28
|
+
const TOKEN_COUNT_KEYS = new Set([
|
|
29
|
+
"prompt_tokens",
|
|
30
|
+
"completion_tokens",
|
|
31
|
+
"total_tokens",
|
|
32
|
+
"cached_tokens",
|
|
33
|
+
"reasoning_tokens",
|
|
34
|
+
"cached_read_tokens",
|
|
35
|
+
"cached_write_tokens",
|
|
36
|
+
"input_tokens",
|
|
37
|
+
"output_tokens",
|
|
38
|
+
]);
|
|
39
|
+
|
|
22
40
|
// JWT — three base64url segments separated by dots, distinctive "eyJ" headers.
|
|
23
41
|
// Linear: single + on [A-Za-z0-9_-] per segment, no nesting.
|
|
24
42
|
export const JWT_PATTERN = /(?<![A-Za-z0-9_-])eyJ[A-Za-z0-9_-]+\.eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+/g;
|
|
@@ -43,6 +61,8 @@ export const STRIPE_KEY_PATTERN = /(?<![A-Za-z0-9_])sk_live_[0-9a-zA-Z]{24}(?![0
|
|
|
43
61
|
// a more complex regex, catastrophic backtracking (ReDoS) could result.
|
|
44
62
|
// Any modifications must preserve O(n) complexity where n = keyName.length.
|
|
45
63
|
export function isSecretKey(keyName: string): boolean {
|
|
64
|
+
// Fast path: known token-count keys are never secrets.
|
|
65
|
+
if (TOKEN_COUNT_KEYS.has(keyName.toLowerCase())) return false;
|
|
46
66
|
// Fast path: common secret key names (safe anchored regex, no backtracking)
|
|
47
67
|
const lower = keyName.toLowerCase();
|
|
48
68
|
if (/^(token|apikey|api_key|password|secret|credential|authorization|privatekey|private_key)$/.test(lower)) {
|