pi-crew 0.11.0 → 0.11.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +155 -9
- package/README.md +161 -1037
- package/agents/verifier.md +18 -7
- package/dist/index.mjs +744 -90644
- package/docs/README.md +57 -46
- package/docs/architecture.md +87 -33
- package/docs/commands-reference.md +9 -5
- package/docs/troubleshooting.md +3 -2
- package/package.json +1 -3
- package/schema.json +39 -0
- package/skills/real-test-pi-crew/REPORT-TEMPLATE.md +2 -0
- package/skills/real-test-pi-crew/SKILL.md +371 -34
- package/src/agents/agent-config.ts +1 -1
- package/src/agents/discover-agents.ts +1 -1
- package/src/config/config-validation.ts +15 -2
- package/src/config/config.ts +47 -13
- package/src/config/defaults.ts +0 -1
- package/src/config/env-vars.ts +35 -0
- package/src/config/types.ts +19 -5
- package/src/errors.ts +2 -2
- package/src/extension/async-notifier.ts +23 -0
- package/src/extension/crew-vibes/config.ts +0 -21
- package/src/extension/crew-vibes/index.ts +0 -2
- package/src/extension/crew-vibes/render.ts +1 -50
- package/src/extension/help.ts +21 -12
- package/src/extension/knowledge-injection.ts +2 -1
- package/src/extension/management.ts +8 -3
- package/src/extension/notification-sink.ts +17 -0
- package/src/extension/register.ts +7 -2
- package/src/extension/registration/command-utils.ts +28 -2
- package/src/extension/registration/commands/dashboard.ts +11 -1
- package/src/extension/registration/commands/manage.ts +36 -19
- package/src/extension/registration/commands/run.ts +24 -2
- package/src/extension/registration/commands/shared.ts +23 -1
- package/src/extension/registration/commands/status.ts +25 -2
- package/src/extension/registration/context-builder.ts +8 -2
- package/src/extension/registration/health-notify-policy.ts +100 -0
- package/src/extension/registration/lazy-configurers.ts +35 -0
- package/src/extension/registration/lifecycle-handlers.ts +91 -30
- package/src/extension/registration/lifecycle.ts +75 -10
- package/src/extension/registration/observability.ts +98 -35
- package/src/extension/registration/registration-types.ts +7 -5
- package/src/extension/registration/runtime-cleanup.ts +9 -3
- package/src/extension/registration/subagent-helpers.ts +38 -0
- package/src/extension/registration/subagent-tools.ts +16 -6
- package/src/extension/registration/team-tool.ts +10 -3
- package/src/extension/registration/terminal-status-wiring.ts +172 -0
- package/src/extension/registration/viewers.ts +6 -0
- package/src/extension/registration/wire-cross-extension.ts +28 -0
- package/src/extension/run-compare.ts +220 -0
- package/src/extension/run-export.ts +37 -5
- package/src/extension/run-maintenance.ts +155 -5
- package/src/extension/team-tool/dispatch/index.ts +3 -2
- package/src/extension/team-tool/dispatch/manage.ts +5 -2
- package/src/extension/team-tool/goal.ts +4 -1
- package/src/extension/team-tool/handle-settings.ts +33 -4
- package/src/extension/team-tool/health-monitor.ts +21 -7
- package/src/extension/team-tool/lifecycle-actions.ts +49 -1
- package/src/extension/team-tool/plan.ts +10 -0
- package/src/extension/team-tool/routing-hint.ts +63 -0
- package/src/extension/team-tool/status.ts +4 -0
- package/src/extension/team-tool.ts +52 -6
- package/src/extension/webhook-notify.ts +382 -0
- package/src/observability/metric-sink.ts +12 -2
- package/src/prompt/prompt-runtime.ts +82 -31
- package/src/prompt/worker-events-channel.ts +12 -0
- package/src/runtime/README.md +1 -1
- package/src/runtime/async-runner.ts +87 -1
- package/src/runtime/background-runner.ts +313 -234
- package/src/runtime/broker/crew-broker.ts +17 -11
- package/src/runtime/broker/delegate/shadow-lifecycle.ts +92 -0
- package/src/runtime/broker/wait-status-cache.ts +1 -1
- package/src/runtime/child-pi/child-pi-timers.ts +1 -1
- package/src/runtime/child-pi/mock-fixtures.ts +48 -0
- package/src/runtime/crew-agent-records.ts +337 -45
- package/src/runtime/deadletter.ts +43 -1
- package/src/runtime/delegate-spawn.ts +5 -1
- package/src/runtime/dispatch-batch.ts +72 -5
- package/src/runtime/goal-workflow/goal-loop-runner.ts +73 -4
- package/src/runtime/heartbeat/heartbeat-watcher.ts +7 -0
- package/src/runtime/model/model-fallback.ts +21 -1
- package/src/runtime/model/pi-args.ts +8 -10
- package/src/runtime/recovery/crash-recovery.ts +25 -1
- package/src/runtime/run-worker.ts +12 -1
- package/src/runtime/scheduling/global-worker-cap.ts +13 -6
- package/src/runtime/scheduling/run-coalesced-task-group.ts +27 -1
- package/src/runtime/scheduling/scheduler.ts +49 -13
- package/src/runtime/scheduling/semaphore.ts +148 -20
- package/src/runtime/scratchpad/README.md +1 -1
- package/src/runtime/scratchpad/protocol.ts +1 -1
- package/src/runtime/settings-store.ts +1 -1
- package/src/runtime/skill-instructions.ts +22 -0
- package/src/runtime/stale-reconciler.ts +85 -13
- package/src/runtime/task-display.ts +1 -1
- package/src/runtime/task-runner/pre-execution.ts +26 -2
- package/src/runtime/task-runner/prompt-builder.ts +142 -45
- package/src/runtime/task-runner.ts +21 -1
- package/src/runtime/team-runner.ts +38 -1
- package/src/runtime/workspace-lock.ts +4 -1
- package/src/schema/config-schema.ts +18 -0
- package/src/schema/team-tool-schema.ts +17 -0
- package/src/state/atomic-write.ts +53 -0
- package/src/state/contracts.ts +109 -0
- package/src/state/coordination/locks.ts +191 -33
- package/src/state/coordination/mailbox.ts +140 -15
- package/src/state/crew-init.ts +87 -12
- package/src/state/event-log/cursor.ts +37 -1
- package/src/state/event-log/event-log-rotation.ts +72 -7
- package/src/state/stores/active-run-registry.ts +13 -1
- package/src/state/stores/state-store.ts +112 -22
- package/src/state/types.ts +4 -0
- package/src/ui/adaptive-card.ts +65 -0
- package/src/ui/agents-jobs-browser.ts +70 -64
- package/src/ui/card-colors.ts +36 -7
- package/src/ui/dashboard-panes/agents-pane.ts +55 -14
- package/src/ui/dashboard-panes/cancellation-pane.ts +0 -42
- package/src/ui/dashboard-panes/health-pane.ts +7 -5
- package/src/ui/dashboard-panes/mailbox-pane.ts +22 -6
- package/src/ui/dashboard-panes/metrics-pane.ts +15 -7
- package/src/ui/dashboard-panes/pane-theme.ts +21 -0
- package/src/ui/dashboard-panes/plan-pane.ts +63 -30
- package/src/ui/dashboard-panes/progress-pane.ts +3 -2
- package/src/ui/dashboard-panes/schedules-pane.ts +44 -21
- package/src/ui/dashboard-panes/transcript-pane.ts +11 -5
- package/src/ui/dwf-phase-display.ts +3 -20
- package/src/ui/format-helpers.ts +22 -0
- package/src/ui/heartbeat-aggregator.ts +34 -0
- package/src/ui/inline-panel/crew-editor.ts +13 -3
- package/src/ui/inline-panel/index.ts +60 -4
- package/src/ui/keybinding-map.ts +251 -35
- package/src/ui/live-conversation-overlay.ts +180 -47
- package/src/ui/live-run-sidebar.ts +134 -55
- package/src/ui/mascot.ts +32 -16
- package/src/ui/overlays/agent-picker-overlay.ts +81 -26
- package/src/ui/overlays/confirm-overlay.ts +55 -29
- package/src/ui/overlays/help-overlay.ts +108 -53
- package/src/ui/overlays/mailbox-compose-overlay.ts +89 -50
- package/src/ui/overlays/mailbox-detail-overlay.ts +137 -57
- package/src/ui/powerbar-publisher.ts +0 -1
- package/src/ui/rail.ts +333 -0
- package/src/ui/run-dashboard.ts +193 -79
- package/src/ui/run-snapshot-cache.ts +18 -1
- package/src/ui/settings-overlay.ts +81 -39
- package/src/ui/spinner.ts +26 -2
- package/src/ui/terminal-status.ts +7 -1
- package/src/ui/theme-adapter.ts +0 -45
- package/src/ui/theme-discovery.ts +12 -6
- package/src/ui/tool-progress-formatter.ts +128 -9
- package/src/ui/tool-renderers/brief-mode.ts +10 -67
- package/src/ui/tool-renderers/index.ts +374 -523
- package/src/ui/transcript-viewer.ts +30 -12
- package/src/ui/widget/index.ts +32 -52
- package/src/ui/widget/task-list.ts +64 -32
- package/src/ui/widget/widget-formatters.ts +3 -402
- package/src/ui/widget/widget-model.ts +28 -7
- package/src/ui/widget/widget-renderer.ts +201 -128
- package/src/ui/widget/widget-types.ts +0 -2
- package/src/utils/incremental-reader.ts +11 -3
- package/src/utils/paths.ts +94 -12
- package/src/utils/project-markers.ts +40 -0
- package/src/utils/visual.ts +0 -4
- package/src/worktree/worktree-manager.ts +206 -26
- package/workflows/distill.workflow.md +3 -3
- package/workflows/fast-fix.workflow.md +1 -1
- package/workflows/plan-execute.workflow.md +1 -1
- package/workflows/review.workflow.md +1 -1
- package/workflows/strict-fast-fix.workflow.md +1 -1
- package/docs/migration-v0.4-v0.5.md +0 -208
- package/docs/runtime-flow.md +0 -148
- package/src/extension/crew-vibes/figures.ts +0 -22
- package/src/extension/crew-vibes/font-detect.ts +0 -71
- package/src/ui/dynamic-border.ts +0 -35
- package/src/ui/loaders.ts +0 -6
- package/src/ui/overlay-stack.ts +0 -148
|
@@ -1,8 +1,10 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: real-test-pi-crew
|
|
3
3
|
description: >
|
|
4
|
-
|
|
5
|
-
|
|
4
|
+
LIVE end-to-end battery for pi-crew changes — unit-green ≠ works in a real Pi session; this skill proves the latter.
|
|
5
|
+
USE WHEN: you changed src/runtime/broker|child-pi|surface|prompt|state, src/ui, src/config, src/schema/team-tool-schema.ts, agents/*.md, workflows, CI, or dist/ — before committing; a worker hung/died/"no output for 600000ms"; a run went green but panes/UI/ask/steer look wrong; someone says "verify it really works".
|
|
6
|
+
TIERS AT A GLANCE: T1 critical (21s) · T2 kill-switch · T3 bundle+md5 · T4/T8 live-session sync · T5/T6 TUI probes · T7 smoke run · T9 team-tool feature battery · T10 surface panes (tmux + herdr) · T11 regression battery (a–j) · T12 agent-frontmatter contracts · T13 real-run UI render. A decision table at the TOP of the file maps changed-path → required tiers.
|
|
7
|
+
When NOT to use: unit tests for isolated modules (use the test runner directly); pure test execution.
|
|
6
8
|
|
|
7
9
|
origin: pi-crew
|
|
8
10
|
triggers:
|
|
@@ -54,18 +56,90 @@ triggers:
|
|
|
54
56
|
- "silent bash"
|
|
55
57
|
- "worker killed mid command"
|
|
56
58
|
- "tier 12"
|
|
59
|
+
- "tier 13"
|
|
60
|
+
- "check the UI"
|
|
61
|
+
- "run the UI"
|
|
62
|
+
- "render every surface"
|
|
63
|
+
- "full UI"
|
|
64
|
+
- "does the card look right"
|
|
65
|
+
- "undefined in the widget"
|
|
66
|
+
- "spinner keeps spinning"
|
|
67
|
+
- "hint cut off"
|
|
68
|
+
- "tofu glyph"
|
|
69
|
+
- "catalog png"
|
|
70
|
+
- "response timeout"
|
|
71
|
+
- "no output for 600000"
|
|
72
|
+
- "dead worker"
|
|
73
|
+
- "tail pipe"
|
|
74
|
+
- "run-state layout"
|
|
75
|
+
- "symlink defense"
|
|
76
|
+
- "mutation check"
|
|
57
77
|
---
|
|
58
78
|
|
|
59
79
|
# real-test-pi-crew
|
|
60
80
|
|
|
61
81
|
End-to-end verification discipline for pi-crew changes. Distilled from the broker Phase-4 rollout (commits `1cb2dca` → `d599578` → `612e18b` → `4186284`, July 2026). The pain this skill prevents: shipping code that compiles + unit-tests-green but breaks in the user's live Pi session, or hangs the verifier worker.
|
|
62
82
|
|
|
63
|
-
|
|
83
|
+
## Nhận dạng nhanh — Tier map + decision table
|
|
84
|
+
|
|
85
|
+
**Bản chất**: unit xanh ≠ chạy thật. Mỗi tier chứng minh MỘT lớp đảm bảo trong một pi session thật. Bảng dưới là chỉ mục — chi tiết ở section tương ứng; ngưỡng thời gian ở "Performance budget" cuối file.
|
|
86
|
+
|
|
87
|
+
| Tier | Chứng minh gì | Chi phí | Section |
|
|
88
|
+
|---|---|---|---|
|
|
89
|
+
| 1 | critical suites xanh (broker/UI) | 21s | Tier 1 |
|
|
90
|
+
| 2 | kill-switch tắt đúng qua 3 đường | 75s | Tier 2 |
|
|
91
|
+
| 3 | typecheck + rebuild bundle + md5 sync | 25s | Tier 3 |
|
|
92
|
+
| 4 | session sống nhận được bundle mới | <1s + restart | Tier 4 |
|
|
93
|
+
| 5 | TUI sống (tmux send-keys) | 5s | Tier 5 |
|
|
94
|
+
| 6 | TUI sống (pty bulk keys) | 5s | Tier 6 |
|
|
95
|
+
| 7 | smoke run — verifier không treo | 60–120s | Tier 7 |
|
|
96
|
+
| 8 | md5 session == md5 đĩa | <1s | Tier 8 |
|
|
97
|
+
| 9 | team-tool feature battery (9a–9g) | 30s + ~120s/spawn | Tier 9 |
|
|
98
|
+
| 10 | worker chạy trong pane thật (10a E2E · 10b tmux live · 10c herdr live) | 90–120s/lần | Tier 10 |
|
|
99
|
+
| 11 | regression battery ghim các fix cũ (11a–11j) | theo suite | Tier 11 |
|
|
100
|
+
| 12 | resource contract (agent .md frontmatter) | — | Tier 12 |
|
|
101
|
+
| 13 | render MỌI surface từ run thật trên đĩa | 150s | Tier 13 |
|
|
102
|
+
|
|
103
|
+
**Decision table — "đổi file X thì phải chạy tier gì"** (bản chi tiết đường dẫn ở phần When to use dưới):
|
|
104
|
+
|
|
105
|
+
| Đổi… | Tier bắt buộc (thứ tự) |
|
|
106
|
+
|---|---|
|
|
107
|
+
| `src/runtime/broker/` | T1 → T7 → T9b-W |
|
|
108
|
+
| `src/ui/` | T1 → (T5|T6) → T13 |
|
|
109
|
+
| `src/config/` (kể cả migration-validator) | T1 → T11c |
|
|
110
|
+
| `src/runtime/child-pi/` (spawn/kill/steer) | T7 → T9g |
|
|
111
|
+
| `src/runtime/async-runner.ts` / `src/runtime/background-runner.ts` (spawn, allowlist, entry-guard, `executeBackgroundRun` core) | T1 → T9b async (event `async.spawned` KHÔNG được có `data.inline:true` — battery phải chứng minh runner DETACHED thật) → T7; đụng `BACKGROUND_RUNNER_ENV_ALLOWLIST` → thêm T10b/T10c (quirk F4-2026-08-30: mux env bị strip ⇒ async headless âm thầm) |
|
|
112
|
+
| `src/runtime/surface/` | T10a → T10b → T10c (dò herdr bằng `HERDR_ENV=1`, KHÔNG phải `$TMUX`) |
|
|
113
|
+
| `src/prompt/` (ask/message/delegate/surface-worker) | T9b-W → T9g |
|
|
114
|
+
| `team-runner.ts` / `task-runner/` / `goal-workflow/plan-templates.ts` | T7 |
|
|
115
|
+
| `src/state/` | T7 → T9a → T11a |
|
|
116
|
+
| `paths.ts` / `project-markers.ts` / `stale-reconciler.ts` / `health-monitor.ts` | T11 pinned suites (layout/symlink) |
|
|
117
|
+
| `team-tool-schema.ts` / `registration/team-tool.ts` | T9 FULL — schema validate TRƯỚC handler; schema sai làm chết mọi action âm thầm |
|
|
118
|
+
| `agents/*.md` / `skills/*/SKILL.md` / discovery | T12 |
|
|
119
|
+
| `workflows/*.workflow.md` | T7 |
|
|
120
|
+
| `.github/workflows/ci.yml` | T11e |
|
|
121
|
+
| `scripts/wc-gate.mjs` | T11b |
|
|
122
|
+
| `dist/` (bundle) | T3 → T4 → T8 → T11j |
|
|
123
|
+
| bất kỳ đường dẫn trên, trước khi commit | gates đầy đủ: critical + unit + tsc + biome + md5 |
|
|
124
|
+
|
|
125
|
+
**When to use**: after any change to `src/runtime/broker/*.ts` (broker + tokens + issuer), `src/ui/`, `src/config/` (incl. `src/config/migration-validator.ts`), `src/extension/registration/lifecycle-handlers.ts`, `src/runtime/child-pi/*.ts` (worker spawn/kill/steering), `src/runtime/async-runner.ts` + `src/runtime/background-runner.ts` (detached spawn + allowlist + entry-guarded `executeBackgroundRun` core — async dispatch, xem quy tắc tier ở decision table), `src/runtime/heartbeat/*` + `src/ui/heartbeat-aggregator.ts` (ambient heartbeat — guest `gc-*` skip), `src/runtime/surface/*.ts` (MuxSurface providers, degrade, launch script), `src/prompt/*.ts` (worker-side tools: ask / message / delegate / surface-worker recorder), `src/runtime/goal-workflow/plan-templates.ts`, `src/runtime/team-runner.ts` or `src/runtime/task-runner/**` (scheduler / execution — Tier 7 smoke), `src/state/**` (durable state — Tier 7 + 9a events/status + **Tier 11a read-your-writes**), `src/utils/paths.ts` + `src/utils/project-markers.ts` + `src/runtime/stale-reconciler.ts` + `src/extension/team-tool/health-monitor.ts` (run-state layout + symlink defense — pinned suites listed in the 2026-09-20 update), `src/runtime/live-session/**` + `src/runtime/custom-tools/*` (live-session mode + worker custom tools), `src/schema/team-tool-schema.ts` (or any `Type.Unsafe({...})` schema definition), `src/extension/registration/team-tool.ts`, `workflows/*.workflow.md`, `.github/workflows/*.yml` (CI env — Tier 11e), `scripts/wc-gate.mjs` (Tier 11b), or before any commit touching these paths. Schema changes additionally require Tier 9 (feature battery) because the team tool's TypeBox schema is validated by pi-ai BEFORE the handler runs — a too-strict or malformed schema breaks every action silently. Surface changes additionally require Tier 10 (surface-mode battery) because surface is fail-closed: every failure degrades to headless and the run still goes green — only pane-level evidence proves the panes engaged. Resource `.md` changes (agent bodies/frontmatter, skill metadata, discovery, frontmatter parsing) additionally require **Tier 12** (resource-contract battery) because the agent/team/workflow frontmatter parser is line-based, not YAML — a folded scalar parses as `">"` for every consumer while all other tiers stay green.
|
|
64
126
|
|
|
65
127
|
> **Path map (2026-08-26 reorg + A1)**: `src/runtime/crew-broker*.ts` → `src/runtime/broker/`; `src/runtime/child-pi*.ts` → `src/runtime/child-pi/`; `src/runtime/plan-templates.ts` (flat) → `src/runtime/goal-workflow/plan-templates.ts`; NEW dirs `src/runtime/surface/` and `src/prompt/`. Test files moved with them (`test/unit/crew-broker-*.test.ts` → `test/unit/runtime/broker/`, `test/unit/keybinding-map.parity.test.ts` → `test/unit/ui/`, ...).
|
|
66
128
|
|
|
67
129
|
> **2026-09-11 update (Batch-1..10, branch `fix/bundle-skill-resolution-and-skill-meta`, tip `aa899a1e`)**: builtin agents 17 → **18** (librarian, oracle, designer, 3 councillors, orchestrator); every agent carries flat routing metadata; NEW **Tier 12** (resource-contract battery) for `agents/*.md` / `skills/*/SKILL.md` / discovery / frontmatter changes; staleness gate gained a path-leak scan (ARCH-7); `scripts/release-smoke.mjs` gained a tarball import + peer-install gate (ARCH-6); two new operational quirks documented (broker SIGTERM on long silent bash; `wait-request-broker.test.ts` 180s-per-file load flake). Orientation doc: `CONTEXT.md`.
|
|
68
130
|
|
|
131
|
+
> **2026-09-17 update (review-remediation wave F01–F20, 20 findings + baseline, `docs/archive/2026-09-17-pi-crew-review-verification.md`)**: three repo gates became **build-blocking in `ci.yml`** — `check:decision-drift`, `check:env-vars`, `check:event-types -- --enforce` (event registry 89 → **168** registered types, 0 drift); test-runner spawn deadline 900s → **1500s** + `PI_CREW_TEST_RUNNER_TIMEOUT_MS` override, and `resolveExitCode()` now fails CLOSED (ETIMEDOUT/spawn-error/signal-kill → non-zero; `NODE_TEST_CONTEXT` scrubbed from child env) — the silent exit-0 false-green class is closed; `PI_CREW_DEBUG_STALE` registered + routed via `getCrewEnv`. New state-machine contracts pinned by tests (all RED→GREEN): mailbox ack-sweep is FAIL-CLOSED on unreadable history (`mailbox-sweep-fail-closed.test.ts` — abort drops nothing, throttle window not consumed), run-lock staleMs steal of a LIVE in-process holder now warns `locks.steal-live-holder` (`run-lock-steal-live-holder-warn.test.ts`), async↔async run-lock mutual exclusion token-guarded. Suite anchors @ 2026-09-17: ~8012 unit tests (869 files), 130 pass + 4 skipped integration (35 files), `test:critical` 116, bundle 3357.3 KB md5 `25ffc88e9a13611f61871b4c4a5b17ab`.
|
|
132
|
+
|
|
133
|
+
> **2026-09-20 update (RR-010..020 wave, commits `864353b3` RR-010..014 + `fa140d24` RR-015..019 + `afd8782d` RR-020)**: RR-020 hardened the run-state layer — symlink defense via `existsSymlinkFreePath()` walking EVERY path component (`RUN_STATE_LAYOUT_SEGMENTS`), new jiti-safe `utils/project-markers.ts`, `crew-init` realpath-start + home/tmp boundary stop, ALL stale-reconciler run loops (incl. the two pre-deletion gates) on `readdirSync({withFileTypes:true})` + `Dirent.isDirectory()` (E1: quarantine-rename could previously write OUTSIDE the scanned tree through a symlinked `runs/<id>`), health-monitor Dirent-guarded + raw `JSON.parse` manifest reads (no quarantine path) + `readRunTasks` only on trusted `listRuns` roots, no-op writer gating (metric-sink on crewRoot existence; appendPruneAudit bails when root missing), `.pi/teams` keybinding override, `stableIOCacheKey` goal `?? ""`, event-log cursor omits `nextByteOffset` on limit-truncated delta, shadow discriminator `stepId === undefined`. 5 verification rounds incl. cold-verifier subagents + mutation checks (~17/18 caught). New pinned suites: `run-state-layout-parity` (14), `no-op-writers-no-crew-root` (8), `project-markers-parity` (7), `keybinding-map-override` (8), `shadow-task-dag-readiness` (7), `prompt-builder-cache-goal` (3), `event-log-cursor-limit-continuation` (3). **Worker-watchdog RCA (proven end-to-end, run `team_20260919150900_6af9ee2bcd57a7f3`)**: the 600s no-response watchdog (`child-pi-timers.ts` noResponseTimer) resets ONLY on child-Pi pipe data (stdout `child-pi.ts:1008` AND stderr `:1029`); pi's bash tool emits a `tool_execution_update` per output chunk (100ms throttle — `tools/bash.js` → `pi-agent-core/agent-loop.js` → print-mode `writeRawStdout`) so a STREAMING command keeps the worker alive indefinitely, while `| tail` / `| head` / `> file` buffering converts a chatty 12-min suite into guaranteed silence → `worker.response_timeout` "No output for 600000ms" at exactly cmd_start+600s → SIGTERM → exit 143 (pi's print-mode SIGTERM handler exits 143 itself). Genuinely-silent >600s commands (sleep, quiet build) die unpiped too. Suite anchors @ 2026-09-20: unit **8050 pass / 0 fail / 3 skipped** (874 files, ~11-12 min), integration **134 tests / 130 pass / 4 skipped** (15 suites, ~150s), `test:critical` **116** (~21s), bundle **3360.1 KB** md5 `2f45df91ed0580ee5bb5b9b57a8a8c3f` (`--committed-hash` OK).
|
|
134
|
+
|
|
135
|
+
> **2026-09-26 update (ambient-noise triage + temp-workspace hygiene + inline-async seam, commits `73790eaa`→`0006ddd5`, reports `real-test-2026-09-26-full-battery-manual.md` + addenda 1-5, ADRs `2026-09-26-inline-async-test-seam.md` / `2026-09-26-temp-workspace-hygiene.md`)**: (1) **Guest `gc-*` skip** — delegate-owned guest tasks have NO heartbeat channel; watcher + aggregator skip them at two independent layers (`agent==="delegate"` marker), guard-tests prove a genuinely-missing REAL worker is still flagged. (2) **Reconciler hygiene** — sweep batch rotates statelessly (`floor(now/60s) % batches`, starvation từ cluster kẹt đầu alphabet đã chết), `.cleanup-in-progress` sentinels cũ >10 phút được reclaim, test-runner dọn `pi-crew-*` pre-existing (mtime < suiteStart−30') sau mỗi suite — battery probe dirs trong /tmp phải KHÔNG đặt tên `pi-crew-*` hoặc phải tạo <30' trước T11. (3) **Inline-async test seam** — `PI_CREW_TEST_ASYNC_INLINE=1` + `PI_CREW_ALLOW_MOCK=1` chạy async run IN-PROCESS (`executeBackgroundRun` core shared với runner detached; `main()` giờ entry-guarded bằng `PI_CREW_BACKGROUND_RUNNER_ENTRY` hoặc argv check). Seam là công cụ UNIT TEST cho heavy async files (subagent-tools-integration: 5+min stall-prone → 14/14 trong 37s) — **battery T7/T9b/T10 TUYỆT ĐỐI không set** (xem row troubleshooting "seam leak"). New pinned suites: `stale-reconciler-rotation` (mutation-checked), `stale-reconciler-sentinel-reclaim`, `sweep-test-tmp` ×2, `async-runner-inline-seam` (mutation-checked), `heartbeat-watcher-guest-skip` ×2, `heartbeat-overlay` guest tests ×2. Suite anchors @ 2026-09-26: `npm test` **EXIT=0** — 8379 tests / 8372 pass / 0 fail / 7 skipped (~14 min), `test:critical` 116 (~25s), bundle **1677.8 KB** md5 `3860c0264940e7df91511da493fa375e`; CI flake Windows `subagent-tools-integration` spawn-stall **RESOLVED** bởi seam (đã từng tốn rerun ×4 trong một ngày).
|
|
136
|
+
|
|
137
|
+
> **2026-09-21 update (F-L2 — steer boot-window loss, found by the live feature battery)**: a steer written to `<artifactsRoot>/steering/<taskId>.jsonl` while the worker extension is still LOADING was silently lost forever — `registerPiTeamsPromptRuntime` starts the file poll at load time, pi's loader gates action methods until bind (`sendMessage: notInitialized` THROWS), and the old poll advanced `lastOffset` to `stat.size` BEFORE the line loop while its per-line catch swallowed the throw as "malformed line" (run `team_20260921035840`, root-caused by watcher-driven bisection probes R7-R9: +0.65s post-boot = lost, +2.5s+ = delivered). Fix (`src/prompt/prompt-runtime.ts`): walk the buffer by byte offset, advance only past terminal verdicts (delivered/non-steer/rejected/malformed), rewind to the failing line on a DELIVERY failure, `createSeenSteerIdSet.unmark()` releases the dedup id so the retry is not skipped. Regression suite `test/unit/prompt/prompt-runtime-steer-boot-window.test.ts` (4 tests) — mutation-checked both layers (offset-revert → 4 fail; unmark-removal → exactly the id-bearing test fails). Note: `dist/index.mjs` is UNCHANGED by this fix — child workers load prompt-runtime from the SOURCE path (`pi-args.ts PROMPT_RUNTIME_EXTENSION_PATH`), so the fix is live without a bundle bump.
|
|
138
|
+
>
|
|
139
|
+
> **F-L2 live re-verification (post-restart, 2026-09-21 afternoon)** — two independent proofs: (a) **lab**: pre-write the steer, then `PI_CREW_STEERING_FILE=<file> pi --extension <repo>/src/prompt/prompt-runtime.ts -p '<long task>'` — stderr shows `[pi-crew:prompt-runtime.steer-delivery-failed] Extension runtime not initialized…` ×2 (pre-bind ticks now LOGGED instead of swallowed) then the post-bind tick delivers and the model complies; (b) **real team run**: watcher writes at **+0.51s** after the steering dir appears (the previously-lost window) → worker session log gains the `custom_message` and the probe token lands in `results/<taskId>.txt`. Suite anchors @ 2026-09-21 (post-fix): unit **8006 tests / 8003 pass / 0 fail / 3 skipped** (875 files; 1088s under load), integration **134 / 130 pass / 4 skipped**, `test:critical` **116**, bundle **3360.1 KB** md5 `2f45df91ed0580ee5bb5b9b57a8a8c3f` (UNCHANGED — see the source-path note above).
|
|
140
|
+
>
|
|
141
|
+
> **Steering channel filename is RUN-KIND-DEPENDENT — the #1 way to fake a failure** (cost this session a full false-negative investigation): **team workflow runs** poll `<artifactsRoot>/steering/<taskId>.jsonl` (`child-executor.ts:562` → env `PI_CREW_STEERING_FILE` → `prompt-runtime.ts:855`), e.g. `01_assess.jsonl`, `adaptive-01-executor.jsonl`; **crew_agent / direct subagent runs** use the agent-slot name `<slot>-agent.jsonl` (e.g. `01_01-agent.jsonl`). A watcher writing `01_01-agent.jsonl` into a team run's steering dir writes a file **no component reads** (0-byte `<taskId>.jsonl` sits untouched, the phantom file accumulates bytes) — it looks exactly like the boot-window loss but is a probe bug. Always derive the name from the run kind, or write every `*.jsonl` present in the dir (see Tier 9g).
|
|
142
|
+
|
|
69
143
|
## Core principle: disk ≠ live Pi
|
|
70
144
|
|
|
71
145
|
Two locations hold pi-crew state:
|
|
@@ -122,8 +196,12 @@ The skill maps to existing CI gates as follows:
|
|
|
122
196
|
| Bundle-staleness check (incl. **ARCH-7 path-leak scan** since `7d18508b` — line-scans `dist/index.mjs` + structural sourcemap check for tracked-source leaks) | Tier 3 last step | `scripts/check-bundle-staleness.mjs`; `--committed-hash` mode = Tier 11j release gate |
|
|
123
197
|
| `npm run test:bundle` (bundle import smoke, 2 tests) | Tier 3 post-build sanity | `test/unit/bundle-load.test.ts` |
|
|
124
198
|
| `node scripts/release-smoke.mjs` (manual, release cut) | Tier 3/11j companion | ARCH-6: installs pi-* peer deps, `import()`s the tarball-installed bundle (`:77`), shape-checks exports |
|
|
125
|
-
| Full `npm test` (= unit
|
|
199
|
+
| Full `npm test` (= unit 869 files + integration 35 @ 2026-09-17) | n/a — too slow for in-loop | CI only; slow tier (3 files) is a SEPARATE glob `test:integration:slow` — only `npm run test:full` includes it |
|
|
126
200
|
| `PI_CREW_SMOKE=1` env | Tier 11e | set ONLY in `weekly-smoke.yml` (auth-gated); nightly.yml deliberately does NOT (comment at `:24`) |
|
|
201
|
+
| `npm run check:decision-drift` | Tier 11 (doc↔code) | **build-blocking in `.github/workflows/ci.yml` since 2026-09-17** (review-remediation baseline): every `PI_CREW_*` token cited in `docs/decisions/*.md` must exist in `src/` |
|
|
202
|
+
| `npm run check:env-vars` | Tier 11 | **build-blocking in `ci.yml` since 2026-09-17**: every `PI_CREW_*` read must be routed via `getCrewEnv` + registered |
|
|
203
|
+
| `npm run check:event-types -- --enforce` | Tier 9a/11 | **build-blocking in `ci.yml` since 2026-09-17**: event registry (`src/state/contracts.ts`) must cover every emitted type — 168 registered / 0 drift @ 2026-09-17 (was 89, drift silent) |
|
|
204
|
+
| `PI_CREW_TEST_RUNNER_TIMEOUT_MS` env | Tier 1 companion | overrides the test-runner spawn deadline (`scripts/test-runner.mjs:206`); default **1_500_000 ms** since 2026-09-17 (was 900_000 — sized for a 5800-test suite; the full suite outgrew it and the ETIMEDOUT became a false green pre-F05) |
|
|
127
205
|
|
|
128
206
|
To add Tier 1 to a pre-commit hook:
|
|
129
207
|
|
|
@@ -149,11 +227,11 @@ To add Tier 1 to CI as a fast-feedback gate (under 30s):
|
|
|
149
227
|
|
|
150
228
|
---
|
|
151
229
|
|
|
152
|
-
## Tier 1 — Critical unit tests (~21s,
|
|
230
|
+
## Tier 1 — Critical unit tests (~21s, 116 tests, the only suite you need for broker/UI changes)
|
|
153
231
|
|
|
154
232
|
**What**: run the curated 14-file fast subset.
|
|
155
233
|
|
|
156
|
-
**Why this exists**: full `npm run test:unit` runs
|
|
234
|
+
**Why this exists**: full `npm run test:unit` runs 875 files (was 642 at skill-writing time — it keeps growing), several minutes. Verifier worker response timeout would kill the worker mid-run → run = "hang". The fix (introduced in commit `1cb2dca`) splits out a `test:critical` subset covering exactly what changed in the broker/UI work.
|
|
157
235
|
|
|
158
236
|
**How**:
|
|
159
237
|
|
|
@@ -161,7 +239,7 @@ To add Tier 1 to CI as a fast-feedback gate (under 30s):
|
|
|
161
239
|
time npm run test:critical
|
|
162
240
|
```
|
|
163
241
|
|
|
164
|
-
Expected output: `# tests
|
|
242
|
+
Expected output: `# tests 116 # pass 116 # fail 0 # duration_ms ~21000`. (Count was 97 at v0.9.46, 101 at v0.9.66, 102 at the waitMethodsEnabled flip, **116 @ 2026-09-20** — verify with the actual run; the skill's hard-coded numbers drift between releases.)
|
|
165
243
|
|
|
166
244
|
**References**:
|
|
167
245
|
|
|
@@ -242,7 +320,7 @@ Compare the printed md5 against what the user's Pi session loaded. If they diffe
|
|
|
242
320
|
| Bundle builder | `scripts/build-bundle.mjs` (esbuild-based, bundles `index.bundle.ts` → `dist/index.mjs`) |
|
|
243
321
|
| Bundle resolution rule | `index.ts:1-25` (entrypoint docstring); also `scripts/build-bundle.mjs:14-20` (entrypoint preference); **symlink is live for source files but the bundled `dist/index.mjs` is loaded** |
|
|
244
322
|
| Postinstall hook | `scripts/postinstall.mjs:43` — best-effort bundle rebuild; falls back to strip-types if esbuild missing |
|
|
245
|
-
| Bundle md5 anchors | `1cc4d55e18add7b9a036c569143320b6` (Phase-4 flip, ~2.78 MB) → `16e29d053bd370e24f40df147dadcb79` (v0.9.66, 2026-08-11) → `9b557ac106b82e1ee33d39dd0d6c7dd7` (post-MuxSurface-A1 main, 2026-08-27) → `945720b1ad25673d86e263cdd834532f` (post-Batch-10 branch tip `aa899a1e`, 2026-09-11, ~3.30 MB). **Always check current**: `md5sum dist/index.mjs` |
|
|
323
|
+
| Bundle md5 anchors | `1cc4d55e18add7b9a036c569143320b6` (Phase-4 flip, ~2.78 MB) → `16e29d053bd370e24f40df147dadcb79` (v0.9.66, 2026-08-11) → `9b557ac106b82e1ee33d39dd0d6c7dd7` (post-MuxSurface-A1 main, 2026-08-27) → `945720b1ad25673d86e263cdd834532f` (post-Batch-10 branch tip `aa899a1e`, 2026-09-11, ~3.30 MB) → `25ffc88e9a13611f61871b4c4a5b17ab` (review-remediation F01–F20 wave, 2026-09-17, 3357.3 KB / 3,437,867 B). **Always check current**: `md5sum dist/index.mjs` |
|
|
246
324
|
|
|
247
325
|
---
|
|
248
326
|
|
|
@@ -388,7 +466,7 @@ else:
|
|
|
388
466
|
|
|
389
467
|
**What**: prove the verifier worker completes within `RESPONSE_TIMEOUT_MS` (**600s since the stuck-worker hardening — was 300s when this skill was distilled; `DEFAULT_CHILD_PI.responseTimeoutMs = 10 * 60_000`**).
|
|
390
468
|
|
|
391
|
-
**Why this is its own tier**: `test:critical` covers unit-level invariants, but the verifier LLM is a separate failure mode — it reads the verifier prompt from `src/runtime/goal-workflow/plan-templates.ts:144, 147` (taskTemplate strings) or from `workflows/*.workflow.md` (workflow verifier sections), then decides which bash command to run. If the prompt says "Run tests" without specifying which, the LLM runs `npm test` (
|
|
469
|
+
**Why this is its own tier**: `test:critical` covers unit-level invariants, but the verifier LLM is a separate failure mode — it reads the verifier prompt from `src/runtime/goal-workflow/plan-templates.ts:144, 147` (taskTemplate strings) or from `workflows/*.workflow.md` (workflow verifier sections), then decides which bash command to run. If the prompt says "Run tests" without specifying which, the LLM runs `npm test` (875 files) and the worker gets killed by the response timeout with exit 143.
|
|
392
470
|
|
|
393
471
|
**How** (from parent Pi session — `team` is a tool, not a shell command):
|
|
394
472
|
|
|
@@ -403,7 +481,7 @@ team:
|
|
|
403
481
|
async: false # synchronous: wait for completion before returning
|
|
404
482
|
```
|
|
405
483
|
|
|
406
|
-
The `team` tool is described in the agent's system prompt. Use `team action='status' <runId>` to inspect mid-run, `team action='events' <runId
|
|
484
|
+
The `team` tool is described in the agent's system prompt. Use `team action='status' <runId>` to inspect mid-run, `team action='events' <runId>` for the event log (**no `limit` param** — it shows the last 500 of the cursor tail; passing `limit` is silently ignored), `team action='cancel' <runId>` to abort.
|
|
407
485
|
|
|
408
486
|
**Real measured outcomes from this session** (July 2026, under the old 300s timeout — wall-clock shape still representative):
|
|
409
487
|
|
|
@@ -430,7 +508,7 @@ The `team` tool is described in the agent's system prompt. Use `team action='sta
|
|
|
430
508
|
|
|
431
509
|
1. **Verifier LLM runs `npm test`** (full unit + integration suite, >4 min) instead of `npm run test:critical`. Symptom: worker killed with exit 143 at the response timeout (300s historically — the measured runs below predate the bump to 600s). Fix: rewrite the verifier prompt to specify the exact fast command AND include "Do NOT run `npm test` or `npm run test:unit`".
|
|
432
510
|
2. **Verifier LLM improvises** with a clean-cache `npm test` run anyway. The cache directive ("cache to `.crew/cache/`", "do NOT re-run") catches this — the second worker that observes a cached log should not re-run.
|
|
433
|
-
3. **
|
|
511
|
+
3. **Silent-command watchdog kill (mechanism proven end-to-end 2026-09-20, run `team_20260919150900_6af9ee2bcd57a7f3`; the Batch-1 postmortem `postmortem-batch-1-sigterm.md` was the same class)**: the 600s no-response timer resets ONLY on child-Pi stdout/stderr data; pi's bash tool emits a `tool_execution_update` per output chunk (100ms throttle), so a STREAMING command survives indefinitely — but `npm run test:unit 2>&1 | tail -25` buffers everything until EOF → zero events for ~700s → `worker.response_timeout` "No output for 600000ms" at exactly cmd_start+600s → SIGTERM (exit 143, from pi's own print-mode SIGTERM handler) WHILE the suite is still running. The worker is ALIVE (fresh heartbeat) — silence, not death, is what kills. Fix: NEVER wrap a long command in `| tail`/`| head`/`> file` inside a worker (stream it, or split <5 min chunks); if you MUST buffer stdout, do NOT merge `2>&1` (subprocess stderr chunks still produce updates → keep the timer alive); genuinely-silent >600s workloads need `PI_TEAMS_CHILD_RESPONSE_TIMEOUT_MS`. Diagnosis: `worker.response_timeout` + fresh `heartbeat.json` + last `tool_execution_update` in `.crew/state/runs/<runId>/background.log` sitting at the silent command's start = this class, not a crash. The killed worker's findings are recoverable from that background.log.
|
|
434
512
|
|
|
435
513
|
---
|
|
436
514
|
|
|
@@ -481,13 +559,13 @@ If the two md5s match → session is on the latest code. If not → user must `/
|
|
|
481
559
|
- `team action='health'` — run-state scan
|
|
482
560
|
- `team action='doctor' focus='zombies'` — orphan subagent + orphan surface-pane scan (read-only)
|
|
483
561
|
- `team action='status' runId='<recent>' details=false` — compact
|
|
484
|
-
- `team action='events' runId='<recent>'` — full event lifecycle
|
|
562
|
+
- `team action='events' runId='<recent>'` — full event lifecycle (no `limit` param — bounded to the last 500 events by the cursor reader)
|
|
485
563
|
- `team action='summary' runId='<recent>'` — cost/by-role report
|
|
486
564
|
- `team action='get' resource='workflow' team='implementation'` — resource inspect
|
|
487
565
|
- `team action='explain' runId='<recent>'` — markdown render
|
|
488
566
|
- `team action='worktrees' runId='<recent>'` — workspace listing
|
|
489
567
|
- `team action='graph' runId='<recent>'` — task-graph render (newer action)
|
|
490
|
-
- `team action='search'
|
|
568
|
+
- `team action='search' goal='<text>'` — search runs/goals (the handler accepts `goal` or `task`; **there is NO `query` field** — passing one returns "Unrecognized team tool parameter field". A no-match search answers "No results found." — that is a PASS, not an error)
|
|
491
569
|
- `team-settings` (slash) or `team action='settings' config={args:'get runtime.surface.mode'}` — config surface incl. the surface/nesting keys
|
|
492
570
|
2. **9b. Spawn paths** (cost tokens — one probe each is enough):
|
|
493
571
|
- `team action='run'` sync (fast-fix, trivial goal) — proves sync run + child-pi spawn + provider-extension loading
|
|
@@ -499,6 +577,13 @@ If the two md5s match → session is on the latest code. If not → user must `/
|
|
|
499
577
|
- `steer_subagent` while a background subagent runs — proves live steering (timing-sensitive; was listed under 9c, but it is the canonical name now — `crew_agent_steer` is the alias)
|
|
500
578
|
3. **9b-W. Worker-tool paths** (cost tokens — proven via goal text that instructs the worker to call the tool; one probe each):
|
|
501
579
|
- **ask round-trip**: goal says "use the `ask` tool to ask the parent <question>, wait for the reply". Proves `wait.request` → park → `team action='respond'` → pickup. **The gate `broker.waitMethodsEnabled` defaulted to `false` until 2026-08-26 (`ceb9a68d` flipped it) — ask slept silently for weeks while every wait.request was rejected `policy-disabled`.** If a worker "answers its own question" instead of asking, the gate or the prompt guidance regressed. Rejections are never silent: a `policy.action` event lands in `events.jsonl`.
|
|
580
|
+
- **Leader turn latency > ask deadline → false timeout.** Default `ASK_TIMEOUT_SEC_DEFAULT = 480s`, and the effective value is clamped to `ASK_TIMEOUT_SEC_CEILING = 480s` (`prompt-runtime.ts:305-313,718`) — a model-supplied `timeoutSec: 900` is silently clamped to 480 (the typebox schema advertises `max 3600`, but that max is never used for the clamp; only the broker accepts up to 3600). If the orchestrator session's own turn latency (ambient notifications, long tool calls) exceeds the deadline, the worker times out before the leader's `respond` runs. **Fix: answer from a DETACHED responder**, not from the agent turn — a small node watcher that polls `events.jsonl` for `ask.requested`, then appends a `kind:"response"` line (with the exact `questionId`) to `<runDir>/mailbox/inbox.jsonl` (the file `findAskResponse` reads). It answers in ~250ms regardless of turn latency. A working reference is `/tmp/ask-responder.mjs` (mkdir -p the mailbox dir first — a run that has never had a mailbox has NO `mailbox/` dir yet; the PRODUCT writer `appendMailboxMessage`→`ensureRunMailbox` creates it, but a raw probe append does not).
|
|
581
|
+
- **THE PROBE-APPEND ENVELOPE IS STRICT — a bare `{kind:"response",questionId,text}` is SILENTLY DROPPED** (measured 2026-09-23, cost a full false-negative "ask round-trip is broken" investigation). `parseMailboxMessage` (`src/state/coordination/mailbox.ts:278-292`) requires ALL of `id`, `runId`, `direction` (`"inbox"|"outbox"`), `from`, `to`, `body` (NOT `text`), `createdAt`, `status` (`queued|delivered|acknowledged`) — any missing/wrong field returns `undefined`, the line is skipped, and the parked worker times out with zero error. Correct probe line:
|
|
582
|
+
```json
|
|
583
|
+
{"id":"probe-<ts>","runId":"<runId>","direction":"inbox","from":"leader","to":"<taskId>","body":"<answer>","createdAt":"<iso>","status":"delivered","kind":"response","priority":"normal","deliveryMode":"next_turn","taskId":"<taskId>","questionId":"<qid>","data":{"action":"respond","kind":"response"}}
|
|
584
|
+
```
|
|
585
|
+
Verify the write worked BEFORE blaming the product: `validateMailbox(manifest)` reports every unparseable line as `{level:"error"}`, and a correct answer flips the parked task within ~0.5s (`ask.answered` + `task.resumed` in events.jsonl). If you see `ask.timedout` with no `ask.answered`, suspect your envelope first.
|
|
586
|
+
- **`ask.answered` alone does NOT prove the leader replied.** The worker's own `resolvePark` emits `ask.answered` ("…answered; task resumed.") on EVERY terminal path incl. timeout. The discriminator: a LEADER answer (`respond.ts:241`) carries `data.delivery` (`"mailbox"` or `"requeue"`); the self-resolve from `crew-broker.ts:1973` carries only `{questionId}`. Grep the payload before claiming a round-trip. Proof of a real round-trip = the worker's result contains the answer wrapped in `<dependency-context>` (`renderAskAnswer`).
|
|
502
587
|
- **message notify**: goal says "use the `message` tool to notify the parent when done". Proves `msg.send` (non-blocking) + the broker `from`-override (anti-spoof) + the wake pattern on the orchestrator session. Rate-limit 10 msg/60s per worker — a burst probe should hit the limit, not hang.
|
|
503
588
|
- **message DM/group**: goal says "DM task `<sibling taskId>` / send to group `x`" — proves `to:` routing + inbox pickup (delivered as fenced `<inbox-message>` DATA, not instructions).
|
|
504
589
|
- **delegate nesting**: goal says "use the `delegate` tool to spawn a child agent". Proves the role gate is open for every role (D8, default-on), the depth cap (`nesting.maxDepth: 4` — a depth-5 attempt must reject with the structured policy message + `delegate.rejected` event, never silently), and the nested-slot budget. Kill switch: `nesting.enabled: false` in **user** config only (sensitive — project config cannot flip it).
|
|
@@ -511,37 +596,83 @@ If the two md5s match → session is on the latest code. If not → user must `/
|
|
|
511
596
|
|
|
512
597
|
**9c. Lifecycle / recovery** (needs a *running* run — start an async run, then exercise these against its runId):
|
|
513
598
|
- `team action='wait' runId='...'` — block until completion
|
|
514
|
-
- `team action='steer' runId='...' message='...'` — inject a steering note mid-run
|
|
599
|
+
- `team action='steer' runId='...' taskId='...' message='...'` — inject a steering note mid-run (**all three params required**; omitting taskId → "steer requires runId, taskId, and message")
|
|
515
600
|
- `team action='status' runId='...' details=true` — full dump mid-run
|
|
516
601
|
- `team action='cache' subAction='...' runId='...'` — snapshot cache ops
|
|
517
|
-
- `team action='checkpoint' runId='...'` — state checkpoint
|
|
602
|
+
- `team action='checkpoint' runId='...' taskId='...'` — state checkpoint (**taskId required** → "Checkpoint requires runId and taskId.")
|
|
518
603
|
- `team action='cancel' runId='...'` — ⚠️ destructive (kills the run); use a throwaway run
|
|
519
604
|
- `team action='invalidate' runId='...'` — cache invalidation
|
|
520
605
|
- `team action='resume' runId='...'` / `retry` — resume a completed/failed run
|
|
521
606
|
- `team action='respond' taskId='...' message='...'` — mailbox reply (needs a waiting task)
|
|
522
607
|
- subagent steering (full procedure): `crew_agent run_in_background=true` a long task (e.g. `sleep 60`), then `steer_subagent` (alias `crew_agent_steer`) while it runs, then `get_subagent_result` — proves the steer arrived (timing-sensitive; assert the agent's output reflects the steer; the quick one-shot version lives in 9b)
|
|
523
608
|
|
|
524
|
-
**9d. Destructive** (⚠️ **requires explicit user confirmation** per the delegation policy — never run unprompted):
|
|
525
|
-
- `
|
|
526
|
-
- `team action='
|
|
527
|
-
- `team action='forget' runId='...'` —
|
|
609
|
+
**9d. Destructive** (⚠️ **requires explicit user confirmation** per the delegation policy — never run unprompted). Run everything against a SACRIFICIAL scratch cwd (with a project marker) — never the real root:
|
|
610
|
+
- Cheap finished runs for prune tests: dispatch `workflow='t9d-empty'` (a workflow-create'd no-steps workflow). It fails in ~2s at the trust gate (`PI_CREW_TRUST_PROJECT_DWF`) **but still persists a `failed` manifest** — a zero-token finished run. `finished` = completed|failed|cancelled (run-maintenance.ts:85). Exports/imports land in `.crew/imports/`, NOT `runs/` — they do NOT count for prune.
|
|
611
|
+
- `team action='prune' keep=<N>` — dryRun first (no confirm needed): verifies newest-first ordering. Real prune (`confirm:true`) writes `.crew/audit/prune.jsonl` (schema: `{action, keep, kept, removed, auditedAt}` — not event-schema).
|
|
612
|
+
- `team action='forget' runId='...' confirm:true` — removes state + artifacts; ownership-checked (foreign needs `force:true`). Batch cleanup of many runs: loop `handleForget({runId, confirm:true, force:true}, ctx)` from a strip-types script — the result is `{content:[{type:'text',text}]}`, NOT a plain string (stringifying it yields "[object Object]" — parse `.content[0].text` for "Forgot run").
|
|
613
|
+
- `team action='cleanup'` — three modes: `runId` (per-run worktree), no-runId project (guidance block; `force:true` also removes `.crew/`), `scope:'user'` (touches `~/.pi/agent` — dryRun ONLY unless the user says otherwise). `dryRun:true` needs no confirm.
|
|
528
614
|
- `team action='doctor' focus='zombies'` is READ-ONLY (safe) but the follow-up `kill <PID>` it suggests is destructive — confirm with the user before killing
|
|
615
|
+
- ⚠️ **Auto-prune keep=10 runs at EVERY pi session start** (`register.sessionStart.autoPruneProject`, lifecycle-handlers.ts) — it deleted all of one morning's battery-evidence runs the moment the next session started (audit: `.crew/audit/prune.jsonl`, removed=14 at 14:37:28). "Prune sees only ~11 finished" is NOT an F-L1 symptom — it's this auto-prune. **Export evidence runs immediately** (`action:'export'`) or they vanish at the next session start. It also entangled the spawn storm: each storm worker's session start pruned corpses (removed=49, removed=22 mid-storm).
|
|
529
616
|
|
|
530
617
|
**9e. Admin / mutation** (mutates config or workflow files — use a scratch project cwd or back up first):
|
|
531
|
-
-
|
|
618
|
+
- **The scratch cwd MUST carry a project marker** (`.git` / `.crew` / `.pi` / `package.json`). `cwd` is validated to stay under the session workspace root, and `findRepoRoot` WALKS UP looking for a marker — a marker-less scratch dir (e.g. an empty `/tmp/x`) silently resolves to the nearest ancestor project and your `create` lands in the REAL project's `.crew/teams/`. Measured 2026-09-21: an empty scratch dir wrote `t9e-probe-team.team.md` into `my_pi/.crew/teams/`. `mkdir -p <scratch>/.crew` first, then verify where the file landed.
|
|
619
|
+
- `team action='create' resource='team' ...` / `update` / `delete` — manage teams/agents/workflows. Params: `create` takes the name in `config.name`; `update`/`delete` take it in `team`/`agent`/`workflow` (NOT `name` — `name` is an unrecognized field). `delete` needs `confirm:true` (destructive-gated). **`update`/`delete` without an explicit `scope` used to be unable to see a PROJECT resource** (`findResource`'s default pool was `[...builtin, ...user]`, and `sourceMatches` drops builtin → the default degenerated to user-only) while the error message promised "mutable user/project scopes". Fixed 2026-09-21; if you hit "not found in mutable user/project scopes" on a resource you just created, pass `scope:'project'` and file it.
|
|
532
620
|
- `team action='init'` / `config` / `validate` / `autonomy` / `settings` — project setup
|
|
533
|
-
- `team action='workflow-create'` / `workflow-save` / `workflow-delete` / `workflow-get` / `workflow-list` — workflow CRUD
|
|
534
|
-
- `team action='import'` / `imports` / `export` — run data portability
|
|
621
|
+
- `team action='workflow-create'` / `workflow-save` / `workflow-delete` / `workflow-get` / `workflow-list` — workflow CRUD. `workflow-create` needs `confirm:true` + `config.name` + `config.script` (arbitrary-code-execution surface); `workflow-delete` needs `confirm:true`.
|
|
622
|
+
- `team action='import'` / `imports` / `export` — run data portability. `export` needs a `runId` that lives under the SAME root you are operating in (a scratch cwd cannot export a run from another project's `.crew/state/runs/`).
|
|
535
623
|
- `team action='parallel' tasks=[...]` — parallel dispatch (spawn path, costs tokens per task)
|
|
536
624
|
|
|
537
625
|
**9f. Background / scheduled** (expensive or niche):
|
|
538
|
-
- `team action='
|
|
539
|
-
- `
|
|
540
|
-
- `team action='
|
|
541
|
-
- `team action='
|
|
626
|
+
- **goal-loop starts via `team action='goal' config.subAction='start'`** — NOT `run runKind='goal-loop'` (runKind only takes effect on DYNAMIC workflows; on builtin teams it is ignored with a warning). Required: `config.evaluatorModel` (no default), budget — **`budgetTotal` is TOP-LEVEL, `config.budgetUnlimited` is IN config** (asymmetric; goal.ts handleStart reads `params.budgetTotal` but `params.config.budgetUnlimited`). maxTurns via `config.maxTurns`. Smoke 2026-09-21: dispatch + state machine + budget accounting proven (loop_start→turn_start→turn_terminal_status→loop_end, turnsUsed 1/2, budgetUsed 0); the turn run failed <1s with NO diagnostic trail — **root-caused 2026-09-22 (GL-1b)**: the turn dir WAS created synchronously (`createRunManifest` → state-store writes manifest+events at once) and lived under the PROJECT root `.crew/state/runs/` (probe used the WRONG subpath `.crew/runs/`); the failure reason was persisted ONLY inside that turn dir (manifest.summary + `run.failed` event) and then DELETED by the session-start auto-prune (keep=10). Fixed: GL-1 terminal path + loop catch now persist `lastTurnError` + `currentRunId:undefined` into GoalLoopState (goal JSON is never pruned) and emit `reason` on `goal.turn_terminal_status` / a `goal.loop_error` event. Lesson: (1) probe the REAL layout `<crewRoot>/state/runs/`, (2) any failure reason stored only in a run dir will vanish at the next session start — persist at goal level.
|
|
627
|
+
- `budgetTotal` used to be unsatisfiable from model calls: pi-ai stringifies numbers when a Union has a `Literal("")` branch, and budgetTotal lacked the stringified-number branch its five sibling params have — died at schema validation. Fixed 2026-09-21 (schema branch + normalizeLooseNumericFields coercion).
|
|
628
|
+
- `team action='schedule' cron='...' ...` — cron works; **`interval` was IMPOSSIBLE until 2026-09-21** (handle-schedule builds `${interval}ms` but both interval parsers lacked an `ms` unit → "Invalid schedule" on every numeric interval). Fixed in parseIntervalMs + detectSchedule. `remove`/`disable`/`enable` subActions live on **action='schedule'** (action='scheduled' is list-only).
|
|
629
|
+
- `team action='auto-summarize'` / `anchor` / `auto_boomerang` — **`auto_boomerang` with no subAction TOGGLES auto-summarize** (deliberate aliasing, misleading name) — it will silently flip real user state; restore with `auto-summarize config.subAction='off'`. Status reads are safe no-throw.
|
|
630
|
+
- `team action='api' runId='...'` — full manifest+artifacts dump (runId required)
|
|
631
|
+
- ⚠️ **Scheduled-job spawn storm (post-mortem 2026-09-21, ROOT-CAUSED 2026-09-22)**: `arm()`'s interval branch used `setInterval(fire, intervalMs)` — Node timers are 32-bit, so a LEGAL 90-day interval (7,776,000,000 ms > 2^31-1) overflowed to a **1ms hot fire loop** (`TimeoutOverflowWarning: Timeout duration was set to 1` — reproducible in one line), and `fire()` has no in-flight guard → one run spawned per tick: ~104 garbage runs, 50+ node processes. The `once` branch had the same overflow via `+30d` (2.59e9 ms). **Fixed 2026-09-22**: both branches now use armCron's clamped chained-hop pattern (MAX_TIMER_DELAY_MS). Containment recipe (still valid): remove job FIRST, then `pkill -f '[b]ackground-runner.ts --cwd <root>'` (bracket — self-match trap); cleanup via handleForget loop; manifest-less corpses are product-invisible (rm directly). Bursts every ~15min were the auto-prune (session-start keep=10) interleaving with the 1ms loop.
|
|
542
632
|
|
|
543
633
|
**Acceptance for 9c–9f**: the action returns a structured result (not `Unknown type` / not an empty error), and for spawn/lifecycle paths the run reaches the expected terminal status. For 9d/9e, the mutation is reversible or confined to scratch state.
|
|
544
634
|
|
|
635
|
+
### 9g. Steer round-trip probe (timing-sensitive — the F-L2 recipe)
|
|
636
|
+
|
|
637
|
+
**What**: prove a steering note written to the worker's steering JSONL actually reaches the model, INCLUDING when it is written in the boot window (before the worker extension binds).
|
|
638
|
+
|
|
639
|
+
**Why a dedicated recipe**: my turn boundaries are minutes long, so a mid-run steer cannot be issued from the parent session directly — use a **detached watcher** that polls for the run's steering dir and writes at a controlled delay. And the steering filename differs by run kind (see the F-L2 block): a wrong name produces a **false failure that looks exactly like the real bug**.
|
|
640
|
+
|
|
641
|
+
```bash
|
|
642
|
+
# 1. Arm the watcher FIRST (detached), THEN start the run in the same turn
|
|
643
|
+
# (two tool calls in one block run concurrently — a sync run blocks the turn).
|
|
644
|
+
# The watcher writes EVERY *.jsonl present, which is run-kind agnostic:
|
|
645
|
+
cat > /tmp/steer-watch.sh <<'EOF'
|
|
646
|
+
#!/bin/bash
|
|
647
|
+
DELAY=${1:-0.5}; LOG=/tmp/steer-watch.log
|
|
648
|
+
ART=<workspace>/.crew/artifacts # project crew root; user root = ~/.pi/agent/extensions/pi-crew
|
|
649
|
+
for i in $(seq 1 240); do
|
|
650
|
+
D=$(find $ART -type d -path "*/steering" -newermt "1 minute ago" 2>/dev/null | head -1)
|
|
651
|
+
[ -n "$D" ] && break; sleep 0.5
|
|
652
|
+
done
|
|
653
|
+
[ -z "$D" ] && { echo "NO dir" >> $LOG; exit 1; }
|
|
654
|
+
sleep "$DELAY"
|
|
655
|
+
for F in "$D"/*.jsonl; do
|
|
656
|
+
printf '%s\n' '{"type":"steer","message":"PROBE_TOKEN: end your final report with the exact token PROBE_TOKEN_ACK"}' >> "$F"
|
|
657
|
+
done
|
|
658
|
+
EOF
|
|
659
|
+
chmod +x /tmp/steer-watch.sh
|
|
660
|
+
(setsid nohup /tmp/steer-watch.sh 0.5 > /dev/null 2>&1 &)
|
|
661
|
+
# 2. In the SAME turn: team action='run' team='default' goal='<multi-turn task>'
|
|
662
|
+
# 3. Verify (three independent signals):
|
|
663
|
+
ls -la <artifactsRoot>/steering/ # which files exist + byte sizes
|
|
664
|
+
cat <artifactsRoot>/results/<taskId>.txt # probe token present = model complied
|
|
665
|
+
grep -c PROBE_TOKEN <run>/agents/<taskId>/events.jsonl # custom_message delivered
|
|
666
|
+
```
|
|
667
|
+
|
|
668
|
+
**Timing windows** (measured): write at **+0.5s** after the steering dir appears = boot window (pre-bind) — the F-L2 case; **+2.5s or later** = post-bind, delivered by the old code too. For a NEGATIVE control on an unfixed build, revert the offset logic and the same probe must lose the steer (mutation-check parity with the unit suite).
|
|
669
|
+
|
|
670
|
+
**Pitfalls that produce fake results**:
|
|
671
|
+
- Wrong filename (see above) → write every `*.jsonl` or derive from run kind.
|
|
672
|
+
- A steer that asks the worker to run a tool the role does not have (e.g. `bash` for the `planner`) → no compliance even though delivery worked. Phrase the probe as text compliance ("end your report with token X").
|
|
673
|
+
- Delivery is consumed at a turn boundary: the probe goal should be explicitly multi-turn (plan → paced writes → summary) so a boundary definitely exists after the retry lands. Measured: a planner's single short turn still received and acknowledged the steer (probe B), so this is a robustness preference, not a hard requirement.
|
|
674
|
+
- Truncate-at-spawn (`child-executor.ts:586-615`) wipes the file for each new incarnation, SKIPPING the wipe when `mtime >= taskDispatchStartedAtMs - 1` (the `racedSteerArrived` guard) — a steer written after dispatch survives; verify with `ls -la` (non-zero bytes = survived).
|
|
675
|
+
|
|
545
676
|
---
|
|
546
677
|
|
|
547
678
|
## Tier 10 — Surface-mode battery (MuxSurface A1, workers in real panes)
|
|
@@ -585,6 +716,12 @@ Acceptance: 3/3 cho mỗi suite khi điều kiện backend thỏa; skip vì thi
|
|
|
585
716
|
|
|
586
717
|
### 10b. Live surface run (từ parent Pi session)
|
|
587
718
|
|
|
719
|
+
**Chạy được từ một pi headless trong tmux** (không cần TUI tương tác): mở `tmux new-session -d`, bật `runtime.surface.visibleAgents=['*']` trong `~/.pi/agent/pi-crew.json`, rồi từ TRONG pane chạy `pi -p "<goal dùng team tool>"`. Evidence (đã đo 2026-09-21, run `team_20260921100246_f2ee9ba752e1b2ad`):
|
|
720
|
+
- `manifest.surface.provider == "tmux"` + `workerPids` non-empty (chỉ nhánh surface ghi `workerPids`)
|
|
721
|
+
- `events.jsonl`: 3× `worker.surface_spawned` (kèm `paneId` `%2`/`%3`/`%4`) + 3× `worker.surface_closed`, **0** `surface.degraded`, **0** `worker.surface_gate_blocked`
|
|
722
|
+
- `tmux list-panes -a` trong lúc run cho thấy pane mang taskId; sau run **không còn pane nào** (auto-exit) — chỉ còn pane shell của bạn
|
|
723
|
+
- **Nhớ khôi phục** `visibleAgents` về `[]` sau probe (nếu không, mọi run sau đều mở pane thật)
|
|
724
|
+
|
|
588
725
|
```text
|
|
589
726
|
1. team-settings set runtime.surface.visibleAgents '["*"]' # hoặc agent cụ thể, vd '["executor"]'
|
|
590
727
|
2. team action='run' team='fast-fix' goal='<trivial>' async=false
|
|
@@ -602,6 +739,8 @@ Evidence cần thu: pane id + title từ `list-panes` TRONG lúc run, và pane b
|
|
|
602
739
|
|
|
603
740
|
herdr chỉ được detect khi **chính pi session đang chạy trong một herdr pane** (design decision — không đoán mò qua socket nếu pi không thuộc herd). Socket API newline-JSON qua `~/.config/herdr/herdr.sock` (herdr 0.8.2+): 1 request = 1 connection, envelope `{"event":...}` (underscore), `pane.read` cần source `"visible"`. Nếu không có herdr: 10c skip với lý do "not in herdr pane" — chấp nhận được, miễn ghi rõ trong report.
|
|
604
741
|
|
|
742
|
+
**Cách dò ĐÚNG (bài học 2026-09-22)**: dò `HERDR_ENV=1` / `HERDR_PANE_ID` / `HERDR_TAB_ID` / `HERDR_WORKSPACE_ID` / `HERDR_SOCKET_PATH` — KHÔNG dò `$TMUX`/`$HERDR`/`$TERM_PROGRAM` (biến `$HERDR` không tồn tại; `$TMUX` rỗng là BÌNH THƯỜNG trong herdr). Một lần dò sai tên biến đã khiến 10c bị ghi nhầm "not runnable" trong khi pi vẫn đang chạy trong herdr. Cross-check bằng process tree: walk ppid lên đến khi gặp `herdr server`/`herdr`. Lưu ý priority: mode `auto` chọn **TMUX trước herdr** khi cả hai khả dụng — muốn ép herdr thì set `runtime.surface.mode: "herdr"`. Probe nhanh không tốn token: `resolveSurfaceDetailed(process.env, {runtime:{surface:{mode:'auto',visibleAgents:['*']}}}, 'executor', 0)` qua strip-types — kỳ vọng `{provider:{kind:"herdr"}}`. Live evidence (run `team_20260922013428_5bf4d1e315ef5e45`): `manifest.surface.provider == "herdr"`, 3 workerPids, 3× surface_spawned với paneId thật (`w2:pBA/pBB/pBC` trong workspace người dùng), 3× surface_closed, 0 degraded/gate_blocked; workers tự xác nhận env markers trong pane.
|
|
743
|
+
|
|
605
744
|
### Surface failure modes → symptom map
|
|
606
745
|
|
|
607
746
|
| Symptom | Likely cause | Recovery |
|
|
@@ -611,7 +750,7 @@ herdr chỉ được detect khi **chính pi session đang chạy trong một her
|
|
|
611
750
|
| Worker thứ 7 trở đi headless | Pane cap `MAX_SURFACE_WORKERS = 6` (`src/runtime/surface/resolve-surface.ts`) — hardcoded A1 | By design; config cap là A2 defer |
|
|
612
751
|
| Surface worker chết liên tục → quay lại headless | Degrade lockout: cause-group lockout + spawn-fail streak 3 | Đọc `events.jsonl` (degrade.classify events); fix gốc nhân (thường là launch script env) |
|
|
613
752
|
| Pane ở lại sau crash host | Orphan pane — doctor chưa quét | `team action='doctor' focus='zombies'` liệt kê + đóng; sweepLaunchScripts dọn script TTL |
|
|
614
|
-
| herdr không được detect | pi không chạy trong herdr pane | By design; chạy pi trong herdr pane rồi thử lại |
|
|
753
|
+
| herdr không được detect | pi không chạy trong herdr pane | By design; chạy pi trong herdr pane rồi thử lại. ⚠️ Nhưng trước khi kết luận vậy: dò `HERDR_ENV=1`/`HERDR_PANE_ID` (không phải `$TMUX`/`$HERDR`) + walk process tree tìm `herdr server` — probe sai tên biến từng cho kết luận sai "not runnable" (2026-09-22) |
|
|
615
754
|
| herdr worker xong việc nhưng host treo tới deadline 600s | Provider thiếu subscribe `pane.exited` (herdr 0.8.2 không push `pane.closed` cho exit tự nhiên) — đã fix `01af9a78` | Chạy 10a herdr suite; đọc subscription wiring trong `herdr-provider.ts` |
|
|
616
755
|
| herdr subscription im lặng / mux-dead ngay lập tức | Frame `\n\n` (tự nối newline trên wrapper đã nối sẵn) — server coi empty line là malformed | Xem unit "wire framing" trong `herdr-provider.test.ts`; đừng thêm `\n` ở tầng provider |
|
|
617
756
|
| Async run tưởng "luôn headless" | **KHÔNG còn** — nhưng 2 lớp phải cùng mở: (1) hard-gate async bỏ 2026-08-27; (2) `BACKGROUND_RUNNER_ENV_ALLOWLIST` từng strip `TMUX`/`HERDR_*` khỏi detached runner → async vẫn gate `no-mux` dù host trong mux (battery 2026-08-30 Finding 2, fix `f0a41a16` thêm đủ mux env vào allow-list) | Test với live mux + `visibleAgents` set: async run PHẢI có `worker.surface_spawned` (verified live `team_20260830144901`: 3/3 panes, tab riêng, tab đóng khi run end); nếu chỉ thấy `no-mux` → kiểm allow-list trước khi nghi gate |
|
|
@@ -641,7 +780,7 @@ grep -n "Log the event first" src/runtime/recovery/crash-recovery.ts # design
|
|
|
641
780
|
# 3. buffered-site census — snapshot & audit:
|
|
642
781
|
grep -rln "appendEventBuffered" src/ | wc -l # 16 files / ~70 raw matches (incl. imports+definition) at v0.10.5; audited live conversions = 43; EVERY new site needs the reader-audit
|
|
643
782
|
# 4. the full gate — test:critical has NO stores/dwf/recovery coverage:
|
|
644
|
-
npm run test:unit #
|
|
783
|
+
npm run test:unit # 875 files, ~8006 tests, ~11-12 min (670-730s measured 2026-09-20; 1088s under load 2026-09-21) — MANDATORY after any delayed-write conversion program
|
|
645
784
|
```
|
|
646
785
|
|
|
647
786
|
### 11b. wc-gate enforcement (M4 done-gate)
|
|
@@ -808,6 +947,129 @@ node --experimental-strip-types --no-warnings --test --test-force-exit \
|
|
|
808
947
|
|
|
809
948
|
---
|
|
810
949
|
|
|
950
|
+
## Tier 13 — Real-run UI render battery (every surface, real state, no fixtures)
|
|
951
|
+
|
|
952
|
+
**What**: run a REAL team run, then render **every UI surface from that run's on-disk state** (`manifest.json` / `agents.json` / `tasks.json` / the snapshot cache / the real widget model) at several widths, and assert the cross-surface invariants that per-module unit tests structurally cannot see.
|
|
953
|
+
|
|
954
|
+
**Why this is its own tier**: unit tests assert whatever strings their author chose with fixtures their author built; executor summaries claim behaviour. Neither can catch a surface painting something that maps to **no data field** or **contradicts the state**. Measured 2026-09-16: a 4-executor parallel UI migration + an independent verifier + **7857 green tests** left **7 real defects** in the UI — one render pass over a real run found all of them in ~2 minutes. The battery is cheap; the class of bug it kills is invisible to everything else.
|
|
955
|
+
|
|
956
|
+
| Defect class | Real example (2026-09-16) | Why unit tests missed it |
|
|
957
|
+
|---|---|---|
|
|
958
|
+
| **Invented string** (hardcoded, maps to no field) | sidebar painted `122ab0dd · completed · right default` | no test asserts the ABSENCE of unexplained words |
|
|
959
|
+
| **Cross-surface inconsistency** | `fast-fix/fast-fix` in sidebar AND dashboard (each site joined `team/workflow` itself) | per-file fixtures used differing team names, so neither site looked wrong |
|
|
960
|
+
| **Wrong state glyph** | dock spun `⠹` on a finished run (`0 running · 3/3 done`) | the header always received a spinner frame; no test crossed "0 running" with the glyph |
|
|
961
|
+
| **Truncation eats the important token** | dashboard run list painted `› ✓ 122ab0dd complete…` — the status cut in half | fixture goals fit the width, so the narrow-width fallback never ran |
|
|
962
|
+
| **Wire format leaking into TUI** | sidebar `input=2780, output=3715, cacheRead=57216, cost=0.000000, turns=0` | the shared `formatUsage` helper IS correct for CLI output; only the TUI usage is wrong |
|
|
963
|
+
| **Unit-less / wrong-scale numbers** | `314.7s` instead of `5m44s` in the agents pane | tests asserted the numbers, never the format |
|
|
964
|
+
| **Pluralisation** | `1 runs`, `3 agents`, `1 tools` | fixtures used plural counts |
|
|
965
|
+
| **Optional segment unguarded** | live `undefined — ↓·enter` after a run finished — the pure builder guarded it, the **component path did not** | the builder test passed; the component path had no zero-runs test |
|
|
966
|
+
|
|
967
|
+
**When required**: ANY change under `src/ui/**`, or any change that alters what a surface prints (a formatter, a status-slot map, a label helper). Also required after parallel/subagent UI work — delegated claims are hypotheses until rendered.
|
|
968
|
+
|
|
969
|
+
**How**:
|
|
970
|
+
|
|
971
|
+
```bash
|
|
972
|
+
# 0. Produce real state — a real run (read-only goal keeps it safe + fast):
|
|
973
|
+
# from the parent Pi session: team action='run' team='fast-fix' goal='<read-only 1-question task>'
|
|
974
|
+
# → note the runId; state lands in <workspace>/.crew/state/runs/<runId>/
|
|
975
|
+
# 1. Write the harness to /tmp (NEVER into the repo) and render every surface.
|
|
976
|
+
# Template (adjust imports to the surfaces you touched):
|
|
977
|
+
cat > /tmp/full-ui.ts <<'EOF'
|
|
978
|
+
import * as fs from "node:fs";
|
|
979
|
+
import { asCrewTheme } from "/ABS/PATH/pi-crew/src/ui/theme-adapter.ts";
|
|
980
|
+
import { formatCompactToolProgress } from "/ABS/PATH/pi-crew/src/ui/tool-progress-formatter.ts";
|
|
981
|
+
import { teamToolRenderer } from "/ABS/PATH/pi-crew/src/ui/tool-renderers/index.ts";
|
|
982
|
+
import { buildWidgetLines } from "/ABS/PATH/pi-crew/src/ui/widget/widget-renderer.ts";
|
|
983
|
+
import { buildTaskListLines } from "/ABS/PATH/pi-crew/src/ui/widget/task-list.ts";
|
|
984
|
+
import { LiveRunSidebar } from "/ABS/PATH/pi-crew/src/ui/live-run-sidebar.ts";
|
|
985
|
+
|
|
986
|
+
const CWD = "/ABS/PATH/WORKSPACE"; // where .crew/state lives
|
|
987
|
+
const RUN = `${CWD}/.crew/state/runs/<runId>`;
|
|
988
|
+
const manifest = JSON.parse(fs.readFileSync(`${RUN}/manifest.json`, "utf8"));
|
|
989
|
+
const agents = JSON.parse(fs.readFileSync(`${RUN}/agents.json`, "utf8"));
|
|
990
|
+
const tasks = JSON.parse(fs.readFileSync(`${RUN}/tasks.json`, "utf8"));
|
|
991
|
+
const theme = asCrewTheme({});
|
|
992
|
+
const R = (c: any, w = 118) => c.render(w).map((l: string) => l.replace(/\s+$/, "")).join("\n");
|
|
993
|
+
const L = (ls: string[]) => ls.map((l) => l.replace(/\s+$/, "")).join("\n");
|
|
994
|
+
const hdr = (t: string) => `\n═══ ${t} ═══`;
|
|
995
|
+
const details = { action: "run", status: manifest.status, runId: manifest.runId, team: manifest.team, agentRecords: agents };
|
|
996
|
+
|
|
997
|
+
console.log(hdr("CALL"), R(teamToolRenderer.renderCall({ action: "run", goal: manifest.goal, team: manifest.team }, theme, { argsComplete: true })));
|
|
998
|
+
// STREAMING must go through the PRODUCER, not a hand-built string:
|
|
999
|
+
const stream = formatCompactToolProgress({ agentId: manifest.runId, status: "running", runId: manifest.runId,
|
|
1000
|
+
startedAt: new Date(manifest.createdAt).getTime(), manifest, tasks, agents });
|
|
1001
|
+
console.log(hdr("STREAMING"), R(teamToolRenderer.renderResult({ details: { action: "run" }, content: [{ type: "text", text: stream }] }, { isPartial: true }, theme, {})));
|
|
1002
|
+
console.log(hdr("COLLAPSED"), R(teamToolRenderer.renderResult({ details }, { action: "run" }, theme, {})));
|
|
1003
|
+
console.log(hdr("EXPANDED"), R(teamToolRenderer.renderResult({ details }, { action: "run" }, theme, { expanded: true })));
|
|
1004
|
+
console.log(hdr("EXPANDED@80"), R(teamToolRenderer.renderResult({ details }, { action: "run" }, theme, { expanded: true }), 80));
|
|
1005
|
+
|
|
1006
|
+
// WidgetRun NEEDS `snapshot` (the plan card reads snapshot.tasks) — a bare {run, agents} renders EMPTY and looks like a bug:
|
|
1007
|
+
const { createRunSnapshotCache } = await import("/ABS/PATH/pi-crew/src/ui/run-snapshot-cache.ts");
|
|
1008
|
+
const snap = createRunSnapshotCache(CWD).refreshIfStale(manifest.runId);
|
|
1009
|
+
const done = [{ run: manifest, agents, snapshot: snap }];
|
|
1010
|
+
const live = [{ run: { ...manifest, status: "running" }, agents: agents.map((a: any, i: number) => ({ ...a, status: i === 0 ? "running" : a.status })), snapshot: snap }];
|
|
1011
|
+
const dead = [{ run: { ...manifest, status: "failed" }, agents, snapshot: snap }];
|
|
1012
|
+
for (const [name, runs, w] of [["done", done, 118], ["done@50", done, 50], ["running", live, 118], ["failed", dead, 118]] as const) {
|
|
1013
|
+
console.log(hdr(`DOCK ${name}`), L(buildWidgetLines(CWD, 0, 8, runs as never, 0, w, {})));
|
|
1014
|
+
}
|
|
1015
|
+
console.log(hdr("PLAN CARD"), L(buildTaskListLines(done as never, 118, theme)) || "(empty)");
|
|
1016
|
+
|
|
1017
|
+
const sidebar = new LiveRunSidebar({ cwd: CWD, runId: manifest.runId, done: () => undefined, theme: {}, config: {} as never });
|
|
1018
|
+
console.log(hdr("SIDEBAR"), R(sidebar, 118));
|
|
1019
|
+
// Dashboard + browser: see docs/ui-samples/capture.ts sections 13/14 for the exact constructors.
|
|
1020
|
+
EOF
|
|
1021
|
+
node --experimental-strip-types --no-warnings /tmp/full-ui.ts | tee /tmp/full-ui.txt
|
|
1022
|
+
|
|
1023
|
+
# 2. Invariant sweep over the rendered text (all of these must be ZERO hits):
|
|
1024
|
+
grep -nE "undefined|[╭╮╰╯├┤]|->" /tmp/full-ui.txt # unguarded segment / retired frame / legacy separator
|
|
1025
|
+
grep -nE "\b1 (runs|tools|agents|tasks|edits)\b" /tmp/full-ui.txt # pluralisation (note: `11 tools` legitimately contains `1 tools` — anchor the match)
|
|
1026
|
+
grep -nE "input=|output=|cacheRead=|cost=[0-9]" /tmp/full-ui.txt # wire format leaked into a TUI surface
|
|
1027
|
+
python3 - <<'PY'
|
|
1028
|
+
import re
|
|
1029
|
+
bad = []
|
|
1030
|
+
for ln in open("/tmp/full-ui.txt", encoding="utf-8"):
|
|
1031
|
+
if "═══" in ln: continue
|
|
1032
|
+
# spinner present while nothing is running?
|
|
1033
|
+
if re.search(r"[⠁-⣿]", ln) and re.search(r"0 running", ln): bad.append(("spinner+0-running", ln.rstrip()))
|
|
1034
|
+
plain = re.sub(r"\x1b\[[0-9;]*m", "", ln)
|
|
1035
|
+
if len(plain.rstrip("\n")) > 120: bad.append(("over-width", plain.rstrip()))
|
|
1036
|
+
if "··" in plain and "↓·enter" in plain and plain.rstrip().endswith("…"): bad.append(("hint clipped", plain.rstrip()))
|
|
1037
|
+
print("BAD:", bad if bad else "none")
|
|
1038
|
+
PY
|
|
1039
|
+
|
|
1040
|
+
# 3. Narrow-width survival: the actionable hint must be the LAST thing to go.
|
|
1041
|
+
# Render the dock at 40/50/60 and assert `↓·enter` (or its tail) is still there.
|
|
1042
|
+
|
|
1043
|
+
# 4. Cross-surface consistency (same run, every surface):
|
|
1044
|
+
# - team label identical everywhere (`fast-fix`, never `fast-fix/fast-fix`)
|
|
1045
|
+
# - run id rendered at the same width everywhere (shortId = last 8)
|
|
1046
|
+
# - durations in ONE format (`5m44s`, never `344.3s`)
|
|
1047
|
+
# - usage in ONE format (`↑2.8k ↓3.7k`, never `input=…`)
|
|
1048
|
+
# - status word never truncated (`completed`, never `complete…`)
|
|
1049
|
+
```
|
|
1050
|
+
|
|
1051
|
+
**Mandatory rules for this tier**
|
|
1052
|
+
|
|
1053
|
+
- **Render REAL state, never hand-built fixtures.** Fixtures are how the seven defects above survived: the author picks values that fit and read well. `team action='run'` costs ~120s and gives you goals, ids, counts and usage that are the wrong length, the wrong shape and the wrong scale — which is the point.
|
|
1054
|
+
- **Feed STREAMING through the producer** (`formatCompactToolProgress`), not a string you typed. Half the streaming bugs live in the producer→parser contract (`test/unit/runtime/core/tool-progress-formatter.test.ts`).
|
|
1055
|
+
- **Render EVERY state you support**: running / done / failed / focused / idle / narrow. A glyph or fallback that is correct in one state is routinely wrong in another (spinner on a finished run; a hint clipped at 50 columns).
|
|
1056
|
+
- **`undefined` sweep must cover the COMPONENT path**, not just the pure builder. `widget-renderer.buildWidgetLines` had `if (!zero) return []` for three weeks while `src/ui/widget/index.ts` composed the same row with an unguarded template literal — the live `undefined — ↓·enter`.
|
|
1057
|
+
- **Any word you cannot trace to a data field is a bug.** For each literal in the rendered output ask "which field/derivation produced this?" — `right default` had no answer.
|
|
1058
|
+
- Keep the harness in `/tmp`. It is a probe, not a deliverable; if it is worth keeping, it belongs in the repo's catalog script (see below), not in a random test file.
|
|
1059
|
+
|
|
1060
|
+
**Catalog (if the repo ships one, `docs/ui-samples/`)**
|
|
1061
|
+
|
|
1062
|
+
- `capture.ts` renders real components into `captures/*.txt`; `render_png.py` turns them into terminal-style PNGs. Re-run BOTH after any UI change, then look at one image — text captures hide font-level breakage.
|
|
1063
|
+
- `render_png.py` must **fail loudly when the font has no glyph** for a character (the coverage self-check compares each glyph against the `.notdef` box). DejaVu Sans Mono — the best-covered mono font on a stock Linux box — lacks the **braille spinner range** (U+2800–U+28FF) and `⟳ ⏰ ⎿`, so a naive render ships `□` for every running row. Mappings in use: braille → `◐`, `⟳` → `↻`, `⎿` → `└`, `⏰`/`⏱` → `o`.
|
|
1064
|
+
- Set **line-height == font-size**, otherwise box-drawing rails paint as a dashed line.
|
|
1065
|
+
- Heavy box glyphs (`┏ ┃ ┗`) render with light strokes in DejaVu/Noto — a font trait, not a capture bug. Document it instead of hunting fonts.
|
|
1066
|
+
|
|
1067
|
+
**Acceptance**: every surface renders; the invariant sweep returns zero hits; each state (running/done/failed/focused/idle/narrow) renders the right glyph and keeps the actionable hint; no `undefined`, no retired frame glyph, no `->`, no wire format, no invented word; durations/usage/plurals consistent across surfaces; catalog (if present) regenerated and one PNG visually inspected.
|
|
1068
|
+
|
|
1069
|
+
**Reference implementations**: `docs/ui-samples/capture.ts` (sections 13–18 render dashboard/browser/inline panel/transcript/live-conversation/settings from a REAL temp run written through the state-store APIs), `docs/ui-samples/render_png.py` (glyph coverage self-check + substitutions), `docs/UI-DESIGN-SYSTEM.md` (the grammar each surface must follow), `src/ui/rail.ts` (the single source of glyphs/helpers).
|
|
1070
|
+
|
|
1071
|
+
---
|
|
1072
|
+
|
|
811
1073
|
## Anti-patterns (the cost is real, observed in this session)
|
|
812
1074
|
|
|
813
1075
|
| Anti-pattern | Cost | Where fixed | Reference |
|
|
@@ -843,7 +1105,16 @@ node --experimental-strip-types --no-warnings --test --test-force-exit \
|
|
|
843
1105
|
| **Duplicated defaults map drift (G17-class)** (P0 remediation, `b6eba80f`): 2 bản EFFECTIVE_DEFAULTS (`settings-overlay.ts`, `handle-settings.ts`) hardcode `"aboveEditor"` trong khi nguồn chân lý (defaults.ts/install.mjs) nói `"bottom"` — suite không có test so 2 bản với nhau, drift sống sót qua 7500 tests. | `b6eba80f` | Defaults phải có MỘT nguồn chân lý, hoặc test so các bản sao. Live probe: `team-settings get <key>`. Xem Tier 11g. |
|
|
844
1106
|
| **Test vacuous — assert trên fixture chứ không trên wiring** (P1 remediation, `09dda842`): migration-validator test 2 từng assert key tự chế không có trong registry → luôn pass dù validator chưa được wire vào registerPiTeams. | `09dda842` | Test phải dùng key THẬT từ registry (`PI_CREW_BROKER_DIAG_UI` severity "removed"), và wiring test phải prove call-site (register.ts:68), không chỉ prove pure function. |
|
|
845
1107
|
| **Folded YAML scalar (`key: >`) in agent/team/workflow frontmatter** (Batch-9 regression, fixed `aa899a1e`): `utils/frontmatter.ts` is line-based — folded descriptions parsed as literal `">"` for ALL 17 agents while typecheck/lint/test:critical stayed green (skills unaffected: real `yaml` package). Symptom: guidance renders `name (builtin): >`, When-NOT text missing. | `aa899a1e` | Agent/teams/workflows frontmatter values stay single-line; quote values containing `": "` (parser strips symmetric quotes); run the Tier 12b dual-parse probe after EVERY resource `.md` frontmatter edit. Folded scalars remain fine in `skills/*/SKILL.md` only. |
|
|
846
|
-
|
|
|
1108
|
+
| | **Fix + green test ≠ fix pinned** (RR-020 round 4, 2026-09-19): a new test written after the fix can be VACUOUS — it kept passing even with the fix reverted (asserted a fixture path that never exercised the repair). 5 verification rounds with ~17/18 mutation checks caught it. | n/a (process) | Mutation-check every fix+test pair: revert the fix → the new test MUST fail → restore the fix. "Test passes" proves nothing about wiring; only "test fails without the fix" does. Scripted per-file backup/restore (`/tmp/mutate.py` pattern) makes this cheap |
|
|
1109
|
+
| **Worker killed mid long-silent-bash** (Batch-1 postmortem): one >5–10 min command emits no output events → watchdog SIGTERM mid-run — exit 143 WHILE the command runs. Work was intact; manual re-run green — the kill WAS the "hang". **2026-09-20 sharpening**: the trigger is EVENT SILENCE, not duration — `\| tail`/`\| head`/`> file` buffering manufactured silence from a chatty suite (proven: `worker.response_timeout` at exactly cmd_start+600s, heartbeat fresh, 29 earlier streaming commands all survived); genuinely-silent commands die unpiped too. | n/a (quirk — `CONTEXT.md` Flagged #1; P2 candidate) | Never `\| tail` a long suite in a worker — stream or split it; if buffering stdout, keep stderr out of the pipe (`2>&1` removes the last update source); raise `PI_TEAMS_CHILD_RESPONSE_TIMEOUT_MS` for legit silent workloads. On exit-143-mid-command: re-run manually BEFORE diagnosing a code bug; mine `.crew/state/runs/<runId>/background.log` for the killed worker's findings. Postmortem: `postmortem-batch-1-sigterm.md`; full RCA: run `team_20260919150900_6af9ee2bcd57a7f3` |
|
|
1110
|
+
| **`pkill -f "test:unit"` kills your own shell** (2026-09-16): the pattern matches the `bash -c` cmdline of the tool call that runs it, so the command dies before it starts the replacement suite — and the log file simply never appears. | n/a (self-inflicted) | Kill by pid (`pgrep -f test-runner` → `kill <pid>`), or match a pattern the sink does not contain | Tier 13 "How" step 0 — detached start: `(setsid nohup npm run test:unit > /tmp/suite.log 2>&1 < /dev/null &)` |
|
|
1111
|
+
| **`pgrep -f test-runner` read as liveness** (2026-09-16): the same self-match makes it report `RUNNING` forever, so a suite that died 10 minutes ago looks alive and the "result" you read is a truncated log. | n/a (self-inflicted) | Liveness = `ps -eo pid,etime,cmd \| grep -E "test-runner\|node --test"` with the grep itself excluded | Tier 13 harness step 1 — a run is DONE only when the `# tests/# pass/# fail` block exists at EOF |
|
|
1112
|
+
| **Reading a truncated suite log as a summary** (2026-09-16): the runner died with `Test runner error: spawnSync … ETIMEDOUT` at subtest 3946/7860 under load; the last numbers looked like a verdict. | n/a (runner) | The verdict is the `# tests/# pass/# fail` block at EOF — absent means the run did not finish; re-run in the foreground with nothing else heavy running. Since 2026-09-17 (F05) the runner itself fails CLOSED on this class — `resolveExitCode()` (`scripts/test-runner.mjs:44`) maps ETIMEDOUT/spawn-error/signal-kill to a non-zero exit, so wrappers/CI see red instead of a silent exit-0 false green; the spawn deadline also rose 900s→1500s (`PI_CREW_TEST_RUNNER_TIMEOUT_MS` override) | Failure symptoms row "Test runner error: spawnSync … ETIMEDOUT" |
|
|
1113
|
+
| **Running the full suite on a tree you are still editing** (2026-09-16): a 12-minute suite started before the last fixes reports failures that no longer exist — and passes fixes that were not in yet. | n/a (process) | Freeze the tree first; if you must edit, the run is VOID — say so instead of quoting its numbers | Done-criteria: "Full `test:unit` fresh-run" |
|
|
1114
|
+
| **Grepping the bundle without context or escape-awareness** (2026-09-16): esbuild escapes non-ASCII as `\u250F` (uppercase hex) so glyph greps miss, while `"->"` (the progress **wire format**, `roleSeparator`) and `Crew agents` (error message + task-graph markdown) are permanent false positives. | n/a (permanent) | Grep the codepoint escape (`u250F`, `u258F`) and always print ±90 chars of context before concluding | Tier 3 bundle check + Tier 13 step 2 (escape-aware proof snippet) |
|
|
1115
|
+
| **Trusting a delegated UI claim** (2026-09-16): four parallel executors + a verifier reported "dedupe applied" and "hints canonical"; the rendered surfaces still showed `fast-fix/fast-fix` and an invented `· right default`. Summaries describe intent. | n/a (process) | Render the surface with REAL state before believing any UI claim — including your own | Tier 13 (this entire tier exists for this class) |
|
|
1116
|
+
| **Declaring a red gate "pre-existing" (or "my regression") from reasoning alone** (2026-09-16): `check:env-vars` was red; the file was untouched, but that is an argument, not evidence. | n/a (process) | Prove it in a clean-HEAD worktree: `git worktree add -q /tmp/pc-head HEAD && (cd /tmp/pc-head && node scripts/check-env-vars.mjs); git worktree remove --force /tmp/pc-head` | Same recipe as the broker-flake proof (Tier 1 / Failure symptoms) |
|
|
1117
|
+
| **Shipping a catalog PNG with tofu boxes** (2026-09-16): the catalog rendered the braille spinner + `⟳` + `⎿` as `□` — invisible in the `.txt` captures, obvious the moment the image was opened. | `docs/ui-samples/render_png.py` glyph self-check | Make the renderer FAIL on any character it cannot paint (compare against the `.notdef` box), then map the offender | Tier 13 "Catalog" + `render_png.py` (`_has_glyph`, `BRAILLE_TO`) |
|
|
847
1118
|
| **Treating `wait-request-broker.test.ts` load-timeout as a product bug**: the test runner's per-file 180s timeout is below this file's full-suite runtime under parallel load — fails only with the whole suite, passes in isolation. Pre-existing flake, NOT a regression from your change. | n/a (test infra) | Re-run the single file before fixing anything: `node scripts/test-runner.mjs test/unit/runtime/broker/wait-request-broker.test.ts`. Green in isolation = infra flake; move on. |
|
|
848
1119
|
|
|
849
1120
|
---
|
|
@@ -874,9 +1145,36 @@ When a tier fails, the recovery is usually quick. Match the symptom to the cause
|
|
|
874
1145
|
| `delegate` rejects with a policy message | By design when depth cap hit (`maxDepth: 4`) or `nesting.enabled: false` in USER config (sensitive — project cannot flip) | Check depth in the rejection payload; `delegate.rejected` event in events.jsonl confirms the structured (non-silent) path |
|
|
875
1146
|
| herdr provider never engages | pi is not itself running inside a herdr pane (design: no socket guessing) | Run pi inside herdr, then `runtime.surface.mode` auto/`herdr`; verify `~/.config/herdr/herdr.sock` responds |
|
|
876
1147
|
| Surface run >5 phút bị stale-reconcile giết oan (worker khỏe, pane sống) | F1 (đã fix f12f4f5d + af2f8eb4): recorder chỉ flush ở turn boundary → lastSeen đóng băng giữa turn; reconciler cũ time-based không pid-gate. **Bẫy đa host**: MỘT pi session chạy bundle cũ cũng đủ giết run của session khác (sweep quét mọi runs) — tát cả host phải cùng version | Kiểm tra mọi pi process cùng bundle (`ps` lstart vs dist mtime); `PI_CREW_DEBUG_STALE=1` sidecar /tmp/pi-crew-f1-debug.log ghi mọi verdict STALE để bắt hung thủ; kỳ vọng sidecar rỗng khi mọi host đã fix |
|
|
877
|
-
| Worker exits 143 WHILE a long bash command is still running (no LLM-activity window before the kill) |
|
|
1148
|
+
| Worker exits 143 WHILE a long bash command is still running (no LLM-activity window before the kill) | Event-silence watchdog (`CONTEXT.md` Flagged #1). **2026-09-20**: the observed variant IS `worker.response_timeout` ("No output for 600000ms") — check the command for `\| tail`/`\| head`/`> file` buffering FIRST | Split the command; emit progress between steps; re-run the suite manually — work is usually intact. See `postmortem-batch-1-sigterm.md` |
|
|
1149
|
+
| `worker.response_timeout` "No output for 600000ms", exit 143 at exactly cmd_start+600s, worker heartbeat FRESH | Event-silence kill (not a crash): the 600s timer resets only on child-Pi pipe data; the running command emitted no output events for 600s — usually `\| tail`/`\| head`/`> file` buffering, or a genuinely silent command | Unbuffer (stream) or split the command; if buffering stdout, keep `2>&1` OUT of the pipe (stderr chunks still reset the timer); `PI_TEAMS_CHILD_RESPONSE_TIMEOUT_MS` for legit silent workloads. Mine `background.log` — the killed worker's findings survive |
|
|
1150
|
+
| Ambient "N dead worker(s)" notification for a run that already reached a terminal status | Stale health marker re-firing on a terminal run — no live process behind it | Verify before acting: `ps -p <pid>` (dead), `manifest.status` (failed/done), `exit-code.txt`. Nothing to recover; marker clears via `/team-dashboard → health → K` (kill stale). Never kill a pid on a stale heartbeat alone — check pid + log first |
|
|
1151
|
+
| Ambient "heartbeat dead"/"missing heartbeat" cho task guest `gc-*` (depth-2 delegate) trên run ĐANG CHẠY | Guest tasks không có kênh heartbeat (lifecycle do `delegate.requested/admitted/completed` sở hữu) — fixed 2026-09-26: watcher `73790eaa` + aggregator `b7ab5ef2` skip `agent==="delegate"` ở CẢ HAI lớp | Nếu tái xuất hiện trên build ≥ `b7ab5ef2`: check skip marker còn nguyên ở cả 2 nơi (watcher task-loop + `summarizeHeartbeats`). KHÔNG bao giờ kết luận worker thật chết chỉ từ notification của guest task — verify pid/manifest như row trên |
|
|
1152
|
+
| Battery async run "xanh" nhưng event `async.spawned` có `data.inline:true`, hoặc không có process runner detached nào trong `ps` | **Seam leak**: shell có `PI_CREW_TEST_ASYNC_INLINE=1` (thường + `PI_CREW_ALLOW_MOCK=1`) export từ phiên test trước — async dispatch chạy in-process, battery đang chứng minh đường SAI (seam là unit-test tool) | Preflight battery: `env | grep PI_CREW_TEST_ASYNC_INLINE` phải RỖNG; sau mỗi async run trong T7/T9b: assert `async.spawned` KHÔNG có `data.inline`. Nếu dính: `unset` cả 2 biến rồi chạy lại tier |
|
|
1153
|
+
| Steer written during worker BOOT never reaches the model (no error, no log; later steers work) | F-L2 (fixed 2026-09-21): poll advanced its offset past an entry whose `pi.sendMessage` threw pre-bind ("not initialized") and the per-line catch ate the throw — one-shot silent loss | Fixed in `prompt-runtime.ts` (offset rewind + dedup unmark); pinned by `prompt-runtime-steer-boot-window.test.ts`. On older builds: re-send the steer after ~5s — a steer that lands post-bind delivers fine |
|
|
1154
|
+
| Steer "lost" but the file you wrote is FULL while a 0-byte `<taskId>.jsonl` sits beside it | **Probe bug, not a product bug** — the steering filename depends on run kind: team runs poll `<taskId>.jsonl`, crew_agent/subagent runs use `<slot>-agent.jsonl`; you wrote the wrong channel (no component reads it) | Re-run writing every `*.jsonl` in the steering dir (Tier 9g), or derive the name from the run kind. Cross-check `results/<taskId>.txt` + the worker session's `custom_message` before calling it a loss |
|
|
1155
|
+
| Fix verified in unit tests but a live probe still fails | Live path ≠ unit path: wrong filename (above), wrong role/tool for the probe instruction, or the session did not cold-start the changed module | Confirm the loaded artifact (`--extension <path>` resolution for workers is the SOURCE file, so a restart is enough; parent-side needs bundle+restart), then re-run Tier 9g with text-only compliance |
|
|
878
1156
|
| Full `test:unit` fails ONLY on `wait-request-broker.test.ts` under parallel load | Per-file 180s runner timeout vs the file's real runtime (passes isolated) | `node scripts/test-runner.mjs test/unit/runtime/broker/wait-request-broker.test.ts` — green in isolation = infra flake, not a regression |
|
|
1157
|
+
| `undefined` (or `undefined — …`) painted in a live surface | an optional segment interpolated into a template literal without a guard. **Check BOTH paths**: the pure builder and the component that composes the same row (2026-09-16: `widget-renderer.buildWidgetLines` guarded it, `src/ui/widget/index.ts` did not). | Guard before composing (`if (!x) return [] / return undefined`), add a regression lock that renders the zero-state through the COMPONENT, and sweep every `${optional}` on the surface. |
|
|
1158
|
+
| Spinner keeps spinning after the run finished (`⠹ … 0 running`) | the surface hardcodes a spinner frame instead of deriving the glyph from state | Derive it: spinner only while an agent/run is actually running, otherwise the outcome glyph (`✓`/`✗`). Assert in the battery that "0 running" never coexists with a braille glyph. |
|
|
1159
|
+
| Actionable hint clipped at a narrow width (`···· ↓…`) | the leader/hint is composed at a pinned budget and the WHOLE line is truncated afterwards, so the tail dies first | Compose the tail with the shared leader helper at `budget = min(pinned, width - 2)` so the LEFT segment is trimmed (with `…`) and the hint survives. Probe at 40/50/60 columns. |
|
|
1160
|
+
| A status/word truncated mid-token (`complete…`, `fast-f…`) | a narrow-width fallback truncating `head · meta` as ONE string | Give the fallback an explicit priority order (status > id > goal > meta) and clip the lowest-priority segment; never `truncate()` a concatenation that mixes a must-survive token with a droppable one. |
|
|
1161
|
+
| A surface shows a word nobody can trace to data (`· right default`) | a hardcoded literal left in a template | Delete it or replace it with the real field (`run.workspaceMode`). Add "every literal traces to a field" to the Tier 13 sweep. |
|
|
1162
|
+
| `input=2780, output=3715, cost=0.000000` inside a TUI panel | a `key=value` formatter (CLI/status output) reused where the compact TUI form belongs | Keep the wire formatter for CLI/log output; add a compact form for the rail (`↑2.8k ↓3.7k`, cost only when > 0). Same for durations: `formatDuration` (`5m44s`), never `(ms/1000).toFixed(1)}s`. |
|
|
1163
|
+
| Catalog PNG shows `□` / dashed rails | the render font lacks the glyph (braille spinner, `⟳ ⏰ ⎿`) or `line-height > font-size` | Re-run with the coverage self-check in `render_png.py`; keep the substitution map up to date; set `LH = FS`. |
|
|
1164
|
+
| `Test runner error: spawnSync … ETIMEDOUT` mid-suite | the runner's own spawn deadline hit under load (a real test spawns node children); deadline is **1500s since 2026-09-17** (900s before) | Re-run in the foreground with nothing else heavy running; a truncated log is not a verdict. If it repeats on one file, run that file alone. On a loaded box you may raise `PI_CREW_TEST_RUNNER_TIMEOUT_MS` — but note the run now exits non-zero (fail-closed, F05), so a wrapper green is trustworthy |
|
|
879
1165
|
| Guidance / `team action='list'` shows an agent description as `>` or missing When-NOT text | Folded-scalar frontmatter (`description: >`) — the line-based parser reads `>` literally (CONTEXT.md Flagged #4) | Restore the single-line value (double-quote it if it contains `": "`); re-run the Tier 12b dual-parse probe |
|
|
1166
|
+
| Worker's `ask` always times out though the orchestrator *did* answer | Orchestrator turn latency (ambient notifications, a long tool call) exceeded the ask deadline — the reply lands after the park expired. Deadline is `min(max(1, timeoutSec), 480)`s and `timeoutSec: 900` is **silently clamped to 480** (`prompt-runtime.ts:718`; schema advertises max 3600, never used) | Answer from a DETACHED watcher that appends `kind:"response"` + the exact `questionId` to `<runDir>/mailbox/inbox.jsonl` (mkdir -p first — an untouched run has no `mailbox/` yet), not from the agent turn. See Tier 9b-W ask recipe |
|
|
1167
|
+
| `ask.answered` in `events.jsonl` but the worker result still says timed out | The worker's own `resolvePark` emits `ask.answered` on every terminal path (incl. timeout) — it is not a leader reply. Discriminator: leader answers carry `data.delivery` (`respond.ts:241`); the self-resolve (`crew-broker.ts:1973`) carries only `{questionId}` | Grep the `ask.answered` payload for `data.delivery`; a real round-trip shows the answer inside `<dependency-context>` in `results/<taskId>.txt` |
|
|
1168
|
+
| Parked task times out even though your responder appended a response line | Probe envelope incomplete — `parseMailboxMessage` requires `id/runId/direction/from/to/body/createdAt/status` and drops the line silently otherwise (a bare `{kind:"response",questionId,text}` is the classic mistake; note `body`, not `text`) | Re-append with the full envelope (see Tier 9b-W), or run `validateMailbox` to see the parse errors. Verify delivery: `ask.answered` with `data.delivery:"mailbox"` + `task.resumed` within ~0.5s |
|
|
1169
|
+
| Parked task + `manifest.waitState` set but no `mailbox/` dir in the run | Expected on a run whose workers never used mailbox/ask before — the dir is created lazily by the product writer (`appendMailboxMessage`→`ensureRunMailbox`), not at run creation | Not a bug. A raw probe append must `mkdir -p` itself |
|
|
1170
|
+
| Same agent shows a DIFFERENT duration on two surfaces of the same finished run (dashboard `29m39s` vs tool card `8m22s`) | One surface measured from `completedAt`, the other from `now`. The dashboard's non-live branch computed `nowMs - startedAt` and ignored `completedAt`, so every finished agent of an old run inflated forever (fixed 2026-09-21: prefer a sane `completedAt`, reject NaN/future/before-startedAt). **Any surface that shows a duration for a completed agent must end at `completedAt`.** | Cross-surface duration check is a Tier 13 invariant — grep the same taskId's duration on every surface and require one value |
|
|
1171
|
+
| `update`/`delete` says "not found in mutable user/project scopes" for a resource you just created in the project | `findResource`'s default pool omitted `discovery.project` (fixed 2026-09-21). Pass `scope:'project'` as a workaround; file it if it recurs | Tier 9e: create → update/delete WITHOUT `scope` is the probe that catches it (the old unit test always passed `scope` and masked it) |
|
|
1172
|
+
| Admin probe wrote into the REAL project's `.crew/` instead of your scratch dir | The scratch cwd had no project marker, so `findRepoRoot` walked up and latched onto the real project | `mkdir -p <scratch>/.crew` before the probe; verify the reported `filePath` points at the scratch |
|
|
1173
|
+
| Catalog `captures/*.txt` diff is huge after a UI change and never settles | `capture.ts` uses `new Date()` + live spinner frames + time-derived run ids, so 7/18 files change on every run. **Do not commit the drift** — restore the captures and commit only the real change; re-run `render_png.py` for the glyph self-check (it fails loud on a font-less glyph, which is the part worth running) | Tier 13 catalog step: treat the `.txt` diff as noise unless it shows a real formatting change |
|
|
1174
|
+
| Export→import of a tmux-surface run fails: "events[i].time must be a string" | `worker-events-channel` (the surface worker's self-report path) used a raw O_APPEND writer that never stamped `time` — every worker.started/completed of a surface run was time-less and the bundle validator rejects the whole bundle. Fixed 2026-09-21 (stamp at the write() choke point). OLD surface-run data stays un-importable | Export a surface run and grep its events.jsonl for `"type":"worker.` entries lacking `time` before assuming transport corruption |
|
|
1175
|
+
| `team action='goal' budgetTotal=100000` rejected at validation ("must be equal to constant / must be number") | pi-ai stringifies numeric args on `Literal("")` unions; budgetTotal lacked the stringified-number branch its sibling budget params have. Fixed 2026-09-21 | Workaround pre-fix: `config.budgetUnlimited:true` (note: in config, NOT top-level) |
|
|
1176
|
+
| `team action='schedule' interval=<ms>` always says `Invalid schedule "<N>ms". Use "5m", …` | handle-schedule builds `${interval}ms`; both interval parsers lacked an `ms` unit — every numeric interval died. Fixed 2026-09-21 | The suggested "5m" formats only work via `cron`/`once` params, not `interval` (interval is schema-typed as a plain ms number) |
|
|
1177
|
+
| Runs keep appearing under one scheduled job; children of YOUR pi process; machine thrashing | **Interval ≥ 2^31-1 ms (≥24.8 days) overflowed the Node timer to a 1ms hot loop** (fixed 2026-09-22 — clamped hops). Check `.crew/audit/prune.jsonl` for interleaved auto-prunes. Contain: remove job, then `pkill -f '[b]ackground-runner.ts --cwd <root>'` | Any new timer scheduling must clamp below MAX_TIMER_DELAY_MS — grep for `setInterval(`/`setTimeout(` with computed delays |
|
|
880
1178
|
|
|
881
1179
|
## Performance budget (per-tier soft limits)
|
|
882
1180
|
|
|
@@ -894,6 +1192,7 @@ When a tier fails, the recovery is usually quick. Match the symptom to the cause
|
|
|
894
1192
|
| 10a (surface E2E suite) | 90s | 180s | tmux server issue or a real spawn/degrade regression — investigate, don't bump |
|
|
895
1193
|
| 10b (live surface run) | ~120s (one fast-fix run) | 600s (worker hard limit) | Pane never engaged (check `visibleAgents`) or auto-exit failed leaving panes open |
|
|
896
1194
|
| 10c (herdr path) | ~120s | 600s | herdr socket protocol drift — check `herdr api schema --json` against `src/runtime/surface/herdr-provider.ts` |
|
|
1195
|
+
| 13 (real-run UI render: 1 real fast-fix run + harness + sweep) | 150s | 300s | A surface is reading disk on the paint path, or the run itself hung — the render is <10ms, so a slow Tier 13 means the harness is doing I/O it shouldn't |
|
|
897
1196
|
|
|
898
1197
|
If a tier runs over the hard limit, **stop and investigate** — don't bump the budget silently. The budget exists precisely so regressions in test runtime (which usually means a regression in test setup/teardown) are caught early.
|
|
899
1198
|
|
|
@@ -995,6 +1294,8 @@ Use this to answer "đủ full tính năng chưa?" without re-deriving. Every us
|
|
|
995
1294
|
| Surface panes tmux/herdr (A1) | `src/runtime/surface/` | T10 (10a E2E + 10b live + 10c herdr) |
|
|
996
1295
|
| Broker (mailbox, steer, tokens) | `src/runtime/broker/` | T1/T2 + 9c steer/respond + T10a test #2 |
|
|
997
1296
|
| Dashboard + keybindings + overlays | `src/ui/`, commands `src/extension/registration/commands/` | T5/T6 probe + parity golden test |
|
|
1297
|
+
| **UI surfaces render from real run state** (tool card, dock widget, plan card, sidebar, dashboard, browser, overlays) | `src/ui/**`, grammar in `docs/UI-DESIGN-SYSTEM.md`, primitives in `src/ui/rail.ts` | T13 (real-run render battery: state glyphs, invented strings, pluralisation, truncation priority, usage/duration formats, narrow-width hint survival) |
|
|
1298
|
+
| UI catalog artifacts (`captures/*.txt` + `png/*.png`, regenerated from real components) | `docs/ui-samples/capture.ts`, `docs/ui-samples/render_png.py` | T13 catalog step (regenerate both + open one PNG; renderer fails on a missing glyph) |
|
|
998
1299
|
| Slash commands (8: run/status/doctor/help/dashboard/settings/init/config) | `commands/{run,status,manage,dashboard}.ts` | T5 send-keys một lệnh `/team-*` |
|
|
999
1300
|
| team-settings / config | `src/extension/team-tool/handle-settings.ts` | 9a settings get + 10b set visibleAgents |
|
|
1000
1301
|
| Worktree isolation | `src/worktree/` | 9a worktrees + 9b run `workspaceMode='worktree'` |
|
|
@@ -1024,6 +1325,10 @@ Use this to answer "đủ full tính năng chưa?" without re-deriving. Every us
|
|
|
1024
1325
|
| Byte-stable worker prefix (ARCH-3) | `src/runtime/task-runner/prompt-builder.ts` stablePrefix/dynamicSuffix split | byte-identity unit test (strictEqual) |
|
|
1025
1326
|
| Release tarball import gate (ARCH-6) | `scripts/release-smoke.mjs` (installs pi-* peers, `import()`s installed bundle `:77`, shape-checks exports) | release cut: `node scripts/release-smoke.mjs` |
|
|
1026
1327
|
| Bundle path-leak scan (ARCH-7) | `scripts/check-bundle-staleness.mjs` (line-scan dist + structural sourcemap check) | T3 staleness run + T11j |
|
|
1328
|
+
| Run-state layout + symlink defense (`existsSymlinkFreePath` component walk, Dirent-guarded scans, E1 quarantine-rename fix) | `src/utils/paths.ts`, `src/utils/project-markers.ts`, `src/state/crew-init.ts`, `src/runtime/stale-reconciler.ts`, `src/extension/team-tool/health-monitor.ts` | `test/unit/run-state-layout-parity.test.ts` (14) + `test/unit/utils/project-markers-parity.test.ts` (7) + live `team action='health'` |
|
|
1329
|
+
| No-op writer gating (metric-sink crewRoot, prune-audit bail) | `src/observability/metric-sink.ts`, `src/extension/run-maintenance.ts` | `test/unit/extension/no-op-writers-no-crew-root.test.ts` (8) |
|
|
1330
|
+
| Worker-dashboard keybinding override (`.pi/teams`) | `src/ui/keybinding-map.ts` | `test/unit/ui/keybinding-map-override.test.ts` (8) |
|
|
1331
|
+
| Steer delivery incl. boot window (pre-bind retry) | `src/prompt/prompt-runtime.ts` (poll + `createSeenSteerIdSet.unmark`), `src/runtime/child-pi/child-pi-spawn.ts:283` (env), `src/runtime/task-runner/child-executor.ts:562` (filename) | `prompt-runtime-steer-boot-window.test.ts` (4) + **Tier 9g** live (watcher at +0.5s; lab variant with `PI_CREW_STEERING_FILE` + `pi --extension <repo>/src/prompt/prompt-runtime.ts`) |
|
|
1027
1332
|
|
|
1028
1333
|
---
|
|
1029
1334
|
|
|
@@ -1047,6 +1352,8 @@ The skill mentions specific commits, line numbers, and version pins. As the code
|
|
|
1047
1352
|
| Verify frontmatter stays single-line/quoted | Each `agents/`, `teams/`, `workflows/` `.md` edit | Tier 12b dual-parse probe — BOTH discovery and strict `yaml` must pass |
|
|
1048
1353
|
| Verify staleness leak-scan still runs | Each `check-bundle-staleness.mjs` edit | `node scripts/check-bundle-staleness.mjs` after `build:bundle` — exit 0 (staleness + path-leak) |
|
|
1049
1354
|
| Verify release-smoke peer pins + import gate | Each `release-smoke.mjs` edit / release cut | `node scripts/release-smoke.mjs` — peer install + import + shape checks green |
|
|
1355
|
+
| Verify suite anchors + bundle md5 chain | Each release / big wave | Update the anchors line (unit/integration/critical counts + bundle size/md5) — see the 2026-09-17 → 2026-09-20 update blocks; `md5sum dist/index.mjs` + `node scripts/check-bundle-staleness.mjs --committed-hash` |
|
|
1356
|
+
| Verify steering channel names + spawn-truncate guard | Each `src/runtime/task-runner/child-executor.ts` / `child-pi-spawn.ts` / `prompt-runtime.ts` edit | `grep -n 'steering.*taskId}.jsonl' src/runtime/task-runner/child-executor.ts` (task-id channel) + `grep -n 'PI_CREW_STEERING_FILE' src/runtime/child-pi/child-pi-spawn.ts` + `grep -n racedSteerArrived src/runtime/task-runner/child-executor.ts` — update Tier 9g if the naming/guard changes |
|
|
1050
1357
|
|
|
1051
1358
|
The skill does NOT need to be updated for every commit — only when the cited lines/files move. Consider it a "living reference" not a "live spec".
|
|
1052
1359
|
|
|
@@ -1055,7 +1362,7 @@ The skill does NOT need to be updated for every commit — only when the cited l
|
|
|
1055
1362
|
## Quick reference — exact commands
|
|
1056
1363
|
|
|
1057
1364
|
```bash
|
|
1058
|
-
# Tier 1 (critical unit, ~21s,
|
|
1365
|
+
# Tier 1 (critical unit, ~21s, 116 tests)
|
|
1059
1366
|
npm run test:critical
|
|
1060
1367
|
# Tier 2 (3-path proof, broker changes only)
|
|
1061
1368
|
PI_CREW_BROKER=0 npm run test:critical
|
|
@@ -1081,10 +1388,13 @@ md5sum dist/index.mjs
|
|
|
1081
1388
|
md5sum "$(npm root -g)"/pi-crew/dist/index.mjs 2>/dev/null \
|
|
1082
1389
|
|| md5sum ../node_modules/pi-crew/dist/index.mjs
|
|
1083
1390
|
# Tier 9 (feature battery — from parent Pi session, tool calls not shell)
|
|
1084
|
-
# read-only: team action=list / recommend / health / doctor / status / events / summary / get / explain / worktrees / settings
|
|
1391
|
+
# read-only: team action=list / recommend / health / doctor / status / events / summary / get / explain / worktrees / graph / search goal='...' / settings
|
|
1085
1392
|
# spawn: team action=run (sync) ; team action=run async=true ; team action=run chain='"A" -> "B"'
|
|
1086
1393
|
# Agent (direct) ; crew_agent run_in_background=true + get_subagent_result ; steer_subagent
|
|
1087
1394
|
# worker tools (goal-text probes): ask round-trip ; message notify/DM/group ; delegate nesting (depth-cap reject)
|
|
1395
|
+
# steer round-trip (incl. boot window) → Tier 9g: arm a detached watcher writing EVERY *.jsonl at +0.5s, then start the run
|
|
1396
|
+
# verify: results/<taskId>.txt carries the probe token + custom_message in the worker session log
|
|
1397
|
+
# NEVER write a single hardcoded filename — team runs poll <taskId>.jsonl, crew_agent runs <slot>-agent.jsonl
|
|
1088
1398
|
# reproduce the two silent schema failures:
|
|
1089
1399
|
# node --input-type=module -e "import {Value} from '@sinclair/typebox/value'; import {TeamToolParams} from './src/schema/team-tool-schema.ts'; Value.Check(TeamToolParams, {action:'list', skill:'', config:{}})" # throws 'Unknown type' = Type.Unsafe-without-Kind bug
|
|
1090
1400
|
# Tier 10 (surface battery)
|
|
@@ -1111,6 +1421,22 @@ node -e 'const yaml=require("yaml"),fs=require("fs");let ok=0;for(const f of fs.
|
|
|
1111
1421
|
node --experimental-strip-types --no-warnings --test --test-force-exit test/unit/bundle-skill-resolution.test.ts test/unit/extension/registration/tool-loop-guard.test.ts test/unit/runtime/core/skill-instructions.test.ts # 12d
|
|
1112
1422
|
node scripts/check-bundle-staleness.mjs # staleness + ARCH-7 path-leak scan (also after every build:bundle)
|
|
1113
1423
|
node scripts/release-smoke.mjs # release cut: peer install + tarball import + shape check (ARCH-6)
|
|
1424
|
+
# Tier 13 (real-run UI render battery — any src/ui/** change, or after ANY delegated UI work)
|
|
1425
|
+
# from the parent Pi session: team action='run' team='fast-fix' goal='<read-only 1-question task>' # ~120s, gives REAL state
|
|
1426
|
+
# then render every surface from that run's on-disk state (harness template in Tier 13; keep it in /tmp):
|
|
1427
|
+
node --experimental-strip-types --no-warnings /tmp/full-ui.ts | tee /tmp/full-ui.txt
|
|
1428
|
+
grep -nE "undefined|[╭╮╰╯├┤]|->" /tmp/full-ui.txt # 0 hits (unguarded segment / retired frame / legacy separator)
|
|
1429
|
+
grep -nE "(^|[^0-9])1 (runs|tools|agents|tasks)\b" /tmp/full-ui.txt # 0 hits (anchor the plural check — `11 tools` contains `1 tools`)
|
|
1430
|
+
grep -nE "input=|output=|cacheRead=|cost=[0-9]" /tmp/full-ui.txt # 0 hits (wire format leaked into a TUI panel)
|
|
1431
|
+
# state glyph: a braille spinner must never coexist with `0 running`; hint (`↓·enter`) must survive at 40/50/60 cols
|
|
1432
|
+
# catalog: regenerate BOTH artifacts, then LOOK at one image
|
|
1433
|
+
node --experimental-strip-types --no-warnings docs/ui-samples/capture.ts # captures/*.txt
|
|
1434
|
+
python3 docs/ui-samples/render_png.py # png/*.png (fails loudly on a glyph the font lacks)
|
|
1435
|
+
# bundle-side proof (escape-aware — esbuild writes \u250F uppercase):
|
|
1436
|
+
python3 - <<'PY'
|
|
1437
|
+
s=open("dist/index.mjs",encoding="utf-8").read()
|
|
1438
|
+
for k,v in {"rail open u250F":"u250F" in s,"section u2523":"u2523" in s,"retired u256D":"u256D" in s,"raw undefined-hint":"undefined \u2014 " in s}.items(): print(f"{k}: {v}")
|
|
1439
|
+
PY
|
|
1114
1440
|
# 11a full gate (after ANY delayed-write conversion program): npm run test:unit # ~7500 tests, 15-18 min
|
|
1115
1441
|
```
|
|
1116
1442
|
|
|
@@ -1120,7 +1446,7 @@ node scripts/release-smoke.mjs # release c
|
|
|
1120
1446
|
|
|
1121
1447
|
Before claiming "tested":
|
|
1122
1448
|
|
|
1123
|
-
- [ ] Tier 1: `test:critical` fresh-run, all pass (<25s). Count varies by release — was 97 at v0.9.46, 101 at v0.9.66,
|
|
1449
|
+
- [ ] Tier 1: `test:critical` fresh-run, all pass (<25s). Count varies by release — was 97 at v0.9.46, 101 at v0.9.66, 102 at the waitMethodsEnabled flip, **116 @ 2026-09-20**; record the actual count in the report.
|
|
1124
1450
|
- [ ] Tier 2: 3-path proof all pass — **required if you touched `src/config/defaults.ts` or `src/extension/registration/lifecycle-handlers.ts`**
|
|
1125
1451
|
- [ ] Tier 3: `npm run typecheck` exit 0, `npm run build:bundle` exit 0
|
|
1126
1452
|
- [ ] Tier 4: bundle md5 matches what the session loaded (or user has `/quit`-ed + reopened)
|
|
@@ -1128,6 +1454,7 @@ Before claiming "tested":
|
|
|
1128
1454
|
- [ ] Tier 7: smoke team run for any `src/runtime/goal-workflow/plan-templates.ts` or `workflows/*.workflow.md` change — completed, no hang, verifier output under 60s
|
|
1129
1455
|
- [ ] Tier 8: final md5 sync check passed
|
|
1130
1456
|
- [ ] Tier 9: feature battery — **required if you touched `src/schema/team-tool-schema.ts`, `src/extension/registration/team-tool.ts`, any `Type.Unsafe({...})` schema, or any armed-role tool list (`agents/*.md` / `src/config/role-tools.ts`)**. 9a read-only batch all return clean; one probe per 9b spawn path (sync / async / chain / `Agent` / `crew_agent`+`get_subagent_result`) completes with `consistency=1`. Run 9c–9f only when the change touches their code path; **at least one full 9c/9e/9f sweep per release is recommended so the battery stays proven** (see `real-test-2026-08-11-scratchpad-I-batch.md`); 9d (destructive) requires explicit user confirmation. **After every run: `git status` to catch unauthorized agent edits.**
|
|
1457
|
+
- [ ] Tier 13: real-run render battery — **required for ANY `src/ui/**` change** (and after any parallel/delegated UI work). One real `team action='run'` producing on-disk state; every surface rendered from that state at 118 + a narrow width; invariant sweep clean (no `undefined` / retired frame glyph / `->` / wire format / invented word / bad plural); the correct glyph per state (running vs done vs failed — never a spinner with `0 running`); the actionable hint survives the narrowest width; durations, usage and run ids formatted identically on every surface; catalog regenerated (capture + PNG) and one image visually inspected. **Fixtures do not count** — the defects this tier exists to catch (invented strings, wrong-state glyphs, truncation that eats a must-survive token) are invisible to author-chosen fixture values.
|
|
1131
1458
|
- [ ] **Output report**: save `docs/real-test/reports/real-test-<YYYY-MM-DD>-<slug>.md` from `skills/real-test-pi-crew/REPORT-TEMPLATE.md`, filled DURING the run with per-tier evidence (counts/md5/runId) — not reconstructed from memory afterward. This is what makes past runs verifiable instead of trust-the-summary.
|
|
1132
1459
|
- [ ] Tier 10: surface battery — **required if you touched `src/runtime/surface/**`, `src/prompt/surface-worker.ts`, the surface branch of `src/runtime/child-pi/child-pi.ts`, or the surface config keys**. 10a E2E 3/3 per backend available (tmux trong tmux; herdr ngoài tmux + socket sống — skip vì thiếu mux là correct-by-design nhưng KHÔNG tính pass cho backend đó); 10b live run với session ĐÃ reload bundle mới (xem Anti-patterns "file-md5 only") + `visibleAgents` set + pane-level evidence (pane id/title during run, `worker.surface_spawned`/`worker.surface_closed` events, pane auto-closed after — KHÔNG dùng `manifest.surface.panes` làm evidence engage, xem Anti-patterns "panes == {}"); 10c herdr live chỉ khi pi chạy trong herdr pane (skip kèm lý do nếu không).
|
|
1133
1460
|
- [ ] Tier 11: remediation regression battery — **required if you touched `src/state/**` write paths, `migration-validator.ts`/its wiring, `scripts/wc-gate.mjs` or `ci` scripts, `.github/workflows/*` env, EFFECTIVE_DEFAULTS maps, or you are cutting a release**. Sub-checks a–j per Tier 11; 11a item 4 (full `test:unit`) mandatory after any delayed-write conversion program, skippable for doc-only changes. Record: buffered-site census count, wc-gate max, staleness `--committed-hash` result.
|
|
@@ -1147,7 +1474,7 @@ Decision docs:
|
|
|
1147
1474
|
Source files (critical paths):
|
|
1148
1475
|
- `src/config/defaults.ts:191` — `DEFAULT_BROKER` (`:205` `waitMethodsEnabled: true`), `:221` `DEFAULT_NESTING`, `:252` `resolveBrokerEnvOverride`
|
|
1149
1476
|
- `src/extension/registration/lifecycle-handlers.ts:1026-1039` — `effectiveEnabled()` (precedence)
|
|
1150
|
-
- `src/runtime/child-pi/child-pi-constants.ts:23` — `RESPONSE_TIMEOUT_MS = 300_000
|
|
1477
|
+
- `src/runtime/child-pi/child-pi-constants.ts:23` — `RESPONSE_TIMEOUT_MS = DEFAULT_CHILD_PI.responseTimeoutMs` = **600_000** (`src/config/defaults.ts:26`; was 300_000 before the stuck-worker hardening — see Tier 7)
|
|
1151
1478
|
- `src/runtime/goal-workflow/plan-templates.ts:144, 147, 151` — verifier `taskTemplate` + `verificationCommand`
|
|
1152
1479
|
- `src/runtime/broker/crew-broker.ts` — broker server (per-connection gate, NDJSON framing)
|
|
1153
1480
|
- `src/runtime/broker/crew-broker-client.ts` — client (`isEventFrame()` distinguishes event vs response frames)
|
|
@@ -1199,6 +1526,16 @@ Workflow files:
|
|
|
1199
1526
|
- `workflows/plan-execute.workflow.md:30` — verifier prompt
|
|
1200
1527
|
- `workflows/review.workflow.md:31` — verifier prompt
|
|
1201
1528
|
|
|
1529
|
+
UI design system + catalog (Tier 13):
|
|
1530
|
+
- `docs/UI-DESIGN-SYSTEM.md` — the RAIL grammar every surface must follow (glyphs `┏ ┣ ┃ ┗`, canopy `NAME ▸ SUBJECT`, dot leaders, eighth-block gauge, cursor `›`, overflow `▲/▼`, hint format) + the 7 surface classes + the width contract
|
|
1531
|
+
- `src/ui/rail.ts` — SINGLE SOURCE of the glyphs/helpers (`RAIL`, `canopyLine`, `sectionLine`, `railLine`, `railLeaders`, `gaugeBar`, `statusSlot/Badge/Icon`, `overflowHint`, `formatHint`/`keyToken`, `CURSOR`/`ACTIVE`, `dedupeAgentLabel`, `padVisual`/`truncVisual`). Surfaces must import from here, never re-declare `┏`/`┃`/`▕` locally.
|
|
1532
|
+
- `src/ui/adaptive-card.ts` — width-deferred wrapper (`render(width)` is the real width; a frame baked for 116 columns tears at 100)
|
|
1533
|
+
- `src/ui/format-helpers.ts` — `formatCount` (pluralisation: `1 tool`), `formatDuration` (`5m44s`, never `314.7s`), `teamWorkflowLabel` (collapses `team/team`), `truncLine`
|
|
1534
|
+
- `src/ui/widget/widget-renderer.ts` — dock row (`buildWidgetLines`, `idleWidgetLine`, `widgetActivityGlyph`, `widgetRailSlot`, `dockTail` hint budget) — the zero-runs branch is the live `undefined — ↓·enter` regression site
|
|
1535
|
+
- `src/ui/live-run-sidebar.ts` / `src/ui/run-dashboard.ts` / `src/ui/agents-jobs-browser.ts` / `src/ui/settings-overlay.ts` / `src/ui/dashboard-panes/*` — surfaces migrated to RAIL 2026-09-16
|
|
1536
|
+
- `docs/ui-samples/capture.ts` (real renders incl. sections 13–18 from a run written through the state-store APIs) + `docs/ui-samples/render_png.py` (glyph coverage self-check + substitution map) + `docs/ui-samples/README.md`
|
|
1537
|
+
- Tests: `test/unit/ui/rail.test.ts`, `dock-rail`, `dashboard-rail`, `panes-rail`, `overlays-rail` (grammar locks), `test/unit/ui/tool-renderers-redesign.test.ts` (card), `test/unit/ui/tool-renderers-frame-width.test.ts` (width invariant)
|
|
1538
|
+
|
|
1202
1539
|
Resource-contract files (Tier 12):
|
|
1203
1540
|
- `src/utils/frontmatter.ts` — LINE-BASED parser (`parseLines`): single-line values, symmetric-quote strip (`aa899a1e`); folded scalars unsupported for agents/teams/workflows (skills use the real `yaml` package — folded OK there)
|
|
1204
1541
|
- `src/agents/discover-agents.ts:388-391, 476` — flat routing keys (`useWhen`/`avoidWhen`/`cost`/`category` as top-level CSV); discovery cache TTL ~30s (`invalidateAgentDiscoveryCache()`)
|