opencode-thrum-pro 0.1.13 → 0.1.14

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (24) hide show
  1. package/assets/skills/coordinator-assessing-agent-completion/SKILL.md +17 -20
  2. package/assets/skills/coordinator-deploying-a-node/SKILL.md +2 -2
  3. package/assets/skills/coordinator-hotpath-merge-gate/SKILL.md +9 -1
  4. package/assets/skills/coordinator-merging-code/SKILL.md +79 -5
  5. package/assets/skills/coordinator-philosophy-merge-gate/SKILL.md +1 -1
  6. package/assets/skills/coordinator-rolling-out-to-nodes/SKILL.md +2 -2
  7. package/assets/skills/diagnosing-fleet-and-daemon/SKILL.md +1 -1
  8. package/assets/skills/implementer-receiving-dispatch/SKILL.md +1 -1
  9. package/assets/skills/orchestrate/SKILL.md +1 -1
  10. package/assets/skills/persistent-watcher-archetype/SKILL.md +1 -1
  11. package/assets/skills/persistent-watcher-archetype/resources/roster-watch.sh +3 -3
  12. package/assets/skills/project-setup/SKILL.md +7 -12
  13. package/assets/skills/project-setup/resources/implementation-agent.md +2 -2
  14. package/assets/skills/researcher-maintaining-memory/SKILL.md +1 -1
  15. package/assets/skills/thrum/resources/CLI_REFERENCE.md +1 -1
  16. package/assets/skills/thrum/resources/LISTENER_PATTERN.md +2 -2
  17. package/assets/skills/using-monitors/SKILL.md +8 -3
  18. package/assets/skills/using-tmux-ssh/SKILL.md +32 -21
  19. package/assets/skills/using-tmux-ssh/resources/tmux-ssh +1158 -0
  20. package/assets/skills/verify-against-plan/SKILL.md +1 -1
  21. package/dist/installer.js +3 -5
  22. package/package.json +1 -1
  23. package/assets/skills/coordinator-context-monitoring/SKILL.md +0 -685
  24. package/assets/skills/coordinator-post-restart-sweep/SKILL.md +0 -135
@@ -1,685 +0,0 @@
1
- ---
2
- name: coordinator-context-monitoring
3
- description: "Use when managing live implementer/brainstormer agents during a long coordination session, at epic merge gates, after a busy dispatch hour, or whenever you suspect an agent is approaching context limits. Prevents 97%-context silent blow-ups by running a sweep + pre-emptive restart before the agent degrades. Safe to wire into a recurring cron that INVOKES this skill — the skill applies tier-ladder judgment, surfacing (and, only when explicitly opted in via config, autonomously restarting) at the >85% tier. What's forbidden is a cron/script that fires restarts unconditionally without going through this skill's tier ladder."
4
- ---
5
-
6
- # Coordinator: Context Monitoring and Pre-emptive Restart
7
-
8
- ## When to invoke
9
-
10
- Trigger this pattern at each of:
11
-
12
- - Receipt of an `ALERT: flagged=…` message from the `context-monitoring` thrum
13
- monitor (the canonical scheduled sweep, outer tick `@every 30m`, self-gated
14
- so effective rate is slower; see "How the scheduled sweep works")
15
- - Epic merge gates (after merging a sub-epic — E6.4, E6.5, etc.)
16
- - After dispatching 3+ tasks in quick succession
17
- - After any agent has been running for 60+ minutes without a restart
18
- - On observing slow or degraded responses from an implementer
19
- - Manually whenever the session feels "intense" (lots of cycles in a short
20
- window)
21
-
22
- The skill applies tier-ladder judgment: the >85% tier surfaces a
23
- `recommend-restart-extended` reason (or, when `restart_actuation=true` in
24
- config, runs a snapshot-gated restart) only on actual ctx %, not on every sweep.
25
- What's forbidden is a script that bypasses this skill and fires
26
- `thrum tmux restart --force` unconditionally (that violates
27
- `feedback_restart_discipline` — burn the runway, don't restart on schedule).
28
-
29
- ## Coordinator self-restart — ALWAYS the EXTENDED form, compact-extended FIRST
30
-
31
- This skill's tier ladder (Steps 3–5) governs how the coordinator restarts
32
- **other** agents. When the **coordinator restarts ITSELF**, a different rule
33
- applies and it is absolute — but the current ruling
34
- REVERSES which command that is:
35
-
36
- > **The coordinator ALWAYS uses the EXTENDED snapshot grade for itself — never
37
- > the standard/compact form. Which command carries that grade is
38
- > runtime-directed: `/thrum:compact-extended` on any runtime with a verified
39
- > compact-equivalent skill (the default), `/thrum:restart-extended` ONLY
40
- > as the fallback on a runtime without one (today: muse, copilot).**
41
-
42
- This is a REVERSAL of the prior absolute rule ("ALWAYS restart-extended, NEVER
43
- compact") — that rule is retired, not merely relaxed. Do not restore it.
44
-
45
- Rationale: the coordinator is the nerve center. Its handoff must carry the full
46
- 16-section snapshot (wire contracts, capability matrix, cycle history, open
47
- L-questions, anticipated Q&A, design rationale) so the next session resumes
48
- steady-state coordination without re-deriving fleet state — that requirement
49
- is UNCHANGED and is what "EXTENDED" governs; only the compact-vs-restart choice
50
- changed. Compacting (rather than restarting) additionally keeps the SAME
51
- process/session alive — sub-agents, background loops, and cron schedules the
52
- coordinator owns survive the compaction, where a restart would drop them.
53
- There is no ctx-band at which the coordinator downgrades to the standard/
54
- compact-grade snapshot — the tier ladder below governs OTHER agents, not this.
55
-
56
- Practical trigger points for coordinator self-restart/compact:
57
-
58
- - A warn-tier context nudge fires on the coordinator's own pane (e.g. 70%+).
59
- - The coordinator hits a clean checkpoint and elects a fresh session.
60
- - Rate-limit / stuck-state recovery on the coordinator itself.
61
-
62
- In every one of these: run `/thrum:compact-extended` if the coordinator's
63
- runtime has a compact equivalent (the default); `/thrum:restart-extended` only
64
- if it does not. (The role-rule `coord-always-restart-extended` — `Coordinator
65
- ALWAYS uses /thrum:restart-extended` — is RETIRED; see
66
- `thrum memory` for its retirement record. This skill is the always-loaded home
67
- so the CURRENT rule survives even when memory isn't consulted.)
68
-
69
- Note the distinction from Steps 3–5 below: those steps drive `/thrum:restart`
70
- (Step 4) and `/thrum:restart-extended` (Step 5) into
71
- **implementer/brainstormer** panes per their ctx tier — that tiering is correct
72
- for worker agents. The carve-out here is ONLY about the coordinator's own
73
- restart/compact, which is unconditionally the extended grade, compact-first.
74
-
75
- ## How the scheduled sweep works (v0.10.6+ — thrum monitor)
76
-
77
- The sweep runs as a daemon-managed `thrum monitor` job named
78
- `context-monitoring`, registered with an `@every 30m` schedule. The 30-min tick
79
- is the outer envelope — the sweep is self-gated: daytime ticks only fire
80
- when the fleet is busy (`agent_status=working`), and overnight ticks fire only
81
- every other tick (effective 60-min rate), so most ticks may be silent no-ops.
82
- There is a SINGLE monitor for ALL lenses in v1 — not one per lens. The job
83
- invokes
84
- `scripts/error-and-context-agent-sweep.sh --no-nudge --out /tmp/agent-sweep.txt`.
85
- The script emits a single consolidated `ALERT:` line to stdout when ANY agent
86
- crosses a threshold (ctx >= 50% OR api-error OR capture-fail); when the fleet is
87
- clean, the script is silent so no message fires. The monitor's
88
- `--match '^ALERT:'` filter routes the ALERT line as a message to the
89
- `coordinator` role (fanned out to every LIVE local coordinator — if no
90
- coordinator is currently live, the sweep is a silent no-op rather than queuing
91
- for an offline one), which triggers this skill.
92
-
93
- Format of the ALERT line:
94
-
95
- ```text
96
- ALERT: flagged=N stuck=S stuck_working=W tier3=T tier2=U — agent_a(92%,api-err,STUCK,stuck-working); agent_b(88%); …
97
- ```
98
-
99
- or, on a benign-only tick once the periodic heartbeat interval has elapsed:
100
-
101
- ```text
102
- ALERT: all-clear — N agents healthy, 0 actionable (sweep alive)
103
- ```
104
-
105
- - `flagged` — total agents needing attention
106
- - `stuck` — api-errored on TWO consecutive sweeps (state file tracks this across
107
- runs)
108
- - `stuck_working` — agent_status=working AND tmux quiet > threshold AND no
109
- recent JSONL tool calls; threshold tunable via `--silence-threshold-min`,
110
- default 10 min
111
- - `tier3` — count with ctx >= 85% (snapshot-gated restart candidates; see
112
- Step 5)
113
- - `tier2` — count with 70-84% ctx (tmux-send nudge candidates)
114
- - Per-agent segment: `name(ctx%,reason-if-any,classifier)` joined by `;`
115
-
116
- ### RUN vs DELIVER
117
-
118
- The sweep now ALWAYS computes flags every scheduled tick (daytime; overnight
119
- keeps its even-tick-only cadence) — RUN is no longer gated on a coarse
120
- fleet-busy signal, which used to silently suppress genuinely actionable signals
121
- before flags were even computed.
122
-
123
- DELIVERY of the `ALERT:` line is a separate decision, made AFTER flags are
124
- computed:
125
-
126
- - **ACTIONABLE** (delivers the real ALERT) — tier2/tier3 ctx, workstall, stuck /
127
- stuck_working, waiting-on-coord, blocked-on-human, capture-fail,
128
- ctx-unknown-idle, api-err, idle-mid-task (abandoned), finished-impl (recommend
129
- teardown), awaiting-restart, pending-human.
130
- - **BENIGN-ONLY** (never delivers on its own) — idle-no-task and/or bare ctx
131
- tier1 (50-69%) alone. `flagged=20 tier3=0` where every one of those 20 is
132
- benign-only does **not** fire an ALERT — `flagged` counts everything written
133
- to the per-agent report; it does not imply delivery.
134
- - **Periodic all-clear heartbeat** — configurable via `.thrum/config.json`
135
- `heartbeat.allclear_interval_min` (default 60). If no actionable ALERT has
136
- fired for that long, the sweep emits the minimal `all-clear` line above so a
137
- benign-quiet fleet is never mistaken for a dead monitor. Any delivered ALERT
138
- (actionable or all-clear) resets this clock.
139
-
140
- The full per-agent report stays at `/tmp/agent-sweep.txt` (overwritten each
141
- sweep) for on-demand drill-down — read it AFTER receiving an ALERT to see which
142
- specific panes are at risk.
143
-
144
- The previous keepalive-cron pattern (CronCreate `<id>`) is deprecated in
145
- favor of this scheduled monitor. The bookkeeping responsibility moves out of the
146
- coordinator's per-session re-add chore and into the daemon's durable monitors
147
- table (survives daemon restart, no per-session re-init needed for this monitor —
148
- though OTHER CronCreate jobs may still require it per
149
- `feedback_cron_reinit_each_session`).
150
-
151
- ## Lenses
152
-
153
- Every lens SURFACES or RECOMMENDS by default — none take autonomous action on
154
- their own. The ONLY exception is `context_tiers`' snapshot-gated restart, and
155
- even that requires an explicit `restart_actuation: true` opt-in in config. There
156
- is a single monitor for all lenses in v1, not one per lens.
157
-
158
- ### Default-ON lenses (implemented, E1–E6)
159
-
160
- | Lens | Default | Disposition |
161
- | --------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
162
- | `context_tiers` | ON | The ctx%/stuck-working ladder (L1). Sole autonomous actuator when `restart_actuation=true`; otherwise RECOMMEND via `recommend-restart-extended` reason in ALERT. |
163
- | `idle_mid_task` | ON | L2 — 30-min idle-with-open-task detection (bead cross-ref). RECOMMEND `/thrum:sleep-extended`; a no-task variant REPORTs `idle-no-task` at lower priority. |
164
- | `snapshot_awaiting_restart` | ON | L3 — pane text + fresh-snapshot two-signal heuristic. RECOMMEND `thrum tmux restart <agent>` (never autonomous). |
165
- | `blocked_on_human_modal` | ON | L4 — detects spend-limit/permission/consent modal prompts. DETECTION ONLY, never auto-answers; routes into the L5 ledger. |
166
- | `pending_human_ledger` | ON | L5 — flat JSONL ledger of items awaiting a human. EXEMPT from progressive backoff — surfaces every tick. `hb_ledger_resolve` (the resolution path) is defined but has zero call sites — nothing marks an entry resolved, so the ledger is append-only-forever, not "until resolved." |
167
- | `waiting_on_coord` | ON | L9 — 21-rule pattern match (folded in from the standalone waiting-on-coord sweep) + warm-hold exemption. RECOMMEND coordinator answer. |
168
-
169
- ### Default-OFF lenses (E8, flag-gated — NOT YET IMPLEMENTED as of E7)
170
-
171
- These lenses are reserved for E8. They exist as config flags for future work but
172
- produce no output until enabled.
173
-
174
- | Lens | Default | Disposition |
175
- | ----------------------- | ------- | -------------------------------- |
176
- | `merge_ready_queue` | OFF | L6 — REPORT |
177
- | `merge_reconciliation` | OFF | L7 — REPORT |
178
- | `base_deploy_staleness` | OFF | L8 — REPORT |
179
- | `ready_vs_capacity` | OFF | L10 — RECOMMEND |
180
- | `stale_agent_gc` | OFF | L11 — propose-only GC, RECOMMEND |
181
- | `daemon_wedge_trend` | OFF | L12 — REPORT |
182
-
183
- ## Configuration
184
-
185
- The heartbeat system is configured under the `heartbeat` key in
186
- `.thrum/config.json` (Go type: `HeartbeatConfig`). Example:
187
-
188
- ```json
189
- {
190
- "heartbeat": {
191
- "schema_version": 1,
192
- "enabled": true,
193
- "cadence_minutes_day": 30,
194
- "cadence_minutes_overnight": 60,
195
- "busy_only_daytime": true,
196
- "busy_signal": "working_or_activity_30m",
197
- "overnight_window": {
198
- "from": "22:00",
199
- "to": "07:00",
200
- "tz": "America/New_York"
201
- },
202
- "backoff": { "start_after": 2, "multiplier": 2, "floor_every": 4 },
203
- "lenses": {
204
- "context_tiers": {
205
- "enabled": true,
206
- "params": {
207
- "restart_actuation": false,
208
- "degrading_flag_after_ticks": 3
209
- }
210
- },
211
- "idle_mid_task": { "enabled": true },
212
- "...": "..."
213
- }
214
- }
215
- }
216
- ```
217
-
218
- **Field reference:**
219
-
220
- | Field | Purpose |
221
- | -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
222
- | `cadence_minutes_day` | Daytime cadence target (default 30 min). The underlying monitor runs `@every 30m` and self-gates so only busy-fleet daytime ticks fire. |
223
- | `cadence_minutes_overnight` | Overnight cadence target (default 60 min). The monitor fires every other `@every 30m` tick overnight (effective 60-min rate). |
224
- | `busy_only_daytime` | When `true`, daytime ticks are skipped unless any agent has `agent_status=working` (busy-signal gate). |
225
- | `busy_signal` | Algorithm for "is the fleet busy?" Default `working_or_activity_30m`: any agent working OR activity in the last 30 min. |
226
- | `overnight_window.tz` | **MUST be an explicit IANA timezone** (e.g. `"America/New_York"`). If unset, the sweep warns and falls back to always-daytime-rate rather than silently guessing. |
227
- | `backoff.start_after` | Number of consecutive unchanged detections of the same problem before progressive backoff begins. |
228
- | `backoff.multiplier` | Each backoff step multiplies the surface interval by this factor (default 2×). |
229
- | `backoff.floor_every` | The surface interval is capped at this many ticks — the problem is never fully silenced (always surfaces at least once per `floor_every` ticks). |
230
- | `lenses.<name>.enabled` | Enable/disable a lens individually. |
231
- | `lenses.<name>.params` | Per-lens parameters. The only actuation opt-in in the whole system is `context_tiers.params.restart_actuation` (default `false`). |
232
- | `lenses.context_tiers.params.degrading_flag_after_ticks` | After this many consecutive tier3 surfaced ticks without a successful restart, a `degrading-no-restart` reason is added to the ALERT (default 3). |
233
-
234
- ### Editing the config (there is NO `thrum config set`)
235
-
236
- `thrum config` only has a `show` subcommand — you **hand-edit
237
- `.thrum/config.json`** to configure the heartbeat. Two gotchas make this
238
- non-obvious; both bite silently.
239
-
240
- **1. All-or-nothing merge (the sharp footgun).** The config
241
- loader (`LoadThrumConfig`) applies
242
- `DefaultHeartbeatConfig()` **only when the stanza is effectively absent**
243
- (`schema_version == 0` AND zero lenses):
244
-
245
- ```go
246
- if cfg.Heartbeat.SchemaVersion == 0 && len(cfg.Heartbeat.Lenses) == 0 {
247
- cfg.Heartbeat = DefaultHeartbeatConfig()
248
- }
249
- ```
250
-
251
- There is **no field-level deep merge.** The moment you write a stanza with
252
- `"schema_version": 1`, every field you _omit_ takes its Go zero-value —
253
- `cadence_minutes_day: 0`, `enabled: false`, and **all 12 lenses disabled**. So
254
- "I'll just set the timezone" with a three-line stanza silently turns the whole
255
- sweep off. **Always write the COMPLETE stanza** — copy
256
- `DefaultHeartbeatConfig()` in full, then change
257
- only what you need.
258
-
259
- **2. `overnight_window.tz` MUST be an explicit IANA zone.** Default is `""` (the
260
- source comment reads "TZ set by operator"). Left unset, the sweep prints
261
- `warning: heartbeat.overnight_window.tz not configured — using local time for gating`
262
- and falls back to the box's local clock (warns rather than silently
263
- guessing). Set it to the operator's zone, e.g. `"America/Los_Angeles"` (PST/PDT)
264
- or `"America/New_York"` (EST/EDT). Never `"local"`.
265
-
266
- **Procedure** — back up, write the full stanza, validate, done (no restart):
267
-
268
- ```bash
269
- cp .thrum/config.json .thrum/config.json.bak
270
- # Build a COMPLETE heartbeat block mirroring DefaultHeartbeatConfig with your tz,
271
- # then splice it in (jq keeps the rest of config.json intact):
272
- jq --slurpfile hb /tmp/hb-block.json '.heartbeat = $hb[0]' .thrum/config.json > /tmp/c.json \
273
- && mv /tmp/c.json .thrum/config.json
274
- jq -e '.heartbeat.overnight_window.tz != "" and (.heartbeat.lenses|length)==12' .thrum/config.json
275
- ```
276
-
277
- **No daemon restart is required.** The scheduled monitor invokes the sweep
278
- _script_, which reads `.thrum/config.json` directly on every tick — the new tz
279
- takes effect on the next sweep. (`thrum config show` reflects the daemon's
280
- in-memory config, cached at boot, so it will lag until the next daemon start;
281
- that lag does not affect the script-driven sweep.) Avoid a
282
- restart-just-for-config — it needlessly risks a cold-boot WAL-replay
283
- brick.
284
-
285
- ## Subagent model selection
286
-
287
- > **Model tiers:** pass an explicit `model:` on every dispatch; choose model and effort per the `choosing-subagent-models` skill (read `runtime.role_models` at the moment of use). Never let a sub-agent inherit your own model.
288
-
289
- ## Step 1 — Run the sweep
290
-
291
- ```bash
292
- bash scripts/error-and-context-agent-sweep.sh --out /tmp/agent-sweep.txt
293
- grep -E 'ctx_used:|^===== ' /tmp/agent-sweep.txt
294
- ```
295
-
296
- The script emits one `ctx_used: X.X%` line per live agent. It captures the
297
- Claude Code status bar footer, normalizing UTF-8 non-breaking spaces
298
- (`\xc2\xa0`) before matching `Ctx Used: X.X%`. Runtimes without that footer
299
- (Codex, Cursor) fall back to `(n/a)`.
300
-
301
- ## Step 2 — Threshold logic
302
-
303
- ### 🔴 ORCHESTRATORS ARE NOT ON THE CONTEXT LADDER — they restart on a PLAN BOUNDARY
304
-
305
- **The context thresholds below are a COORDINATOR rule. Orchestrators inherited it
306
- rather than being chosen for it, and it is not cost-optimal for them.** Do not
307
- hold an orchestrator open to reach 70–75%, and do not restart one mid-plan to
308
- chase the cost curve.
309
-
310
- **The orchestrator rule:**
311
-
312
- > **Restart an orchestrator at PLAN COMPLETION — where completion means the plan
313
- > is MERGED *and* the follow-up beads filed during that plan have been FIXED.**
314
- > Not at the merge report. Not at the merge.
315
-
316
- **Why the follow-ups clause is load-bearing, not a detail.** The standing
317
- fix-while-warm rule says a bug found mid-work gets FIXED while warm, because
318
- file-and-move-on costs ~10x — the research has to be redone before the fix can
319
- start. Bugs found during a plan are therefore filed and swept up at the END of
320
- the plan, while the orchestrator *and its implementers* still hold the context.
321
- **A restart at the merge boundary destroys exactly that warmth, and does so
322
- INVISIBLY — the follow-ups still get done, just cold and at the 10x price.** A
323
- restart rule keyed to "the merge" does not merely mistime a restart; it silently
324
- repeals fix-while-warm.
325
-
326
- **Why the split is real, not stylistic**:
327
-
328
- - Per-turn cost scales LINEARLY with context size (the whole prefix is re-sent
329
- and billed at the cache-read rate every turn), so cumulative session cost is
330
- roughly QUADRATIC in how far context is allowed to grow. Break-even for a
331
- restart is **under one turn** once context is meaningfully above the re-prime
332
- floor.
333
- - **Orchestrators re-prime far cheaper than coordinators, and it is structural:**
334
- the prime output filter gives ONLY `role=="coordinator"` the full
335
- project-state passthrough. Measured orchestrator re-prime ≈ 30k tokens.
336
- - Coordinators keep the ladder below for a stated reason — larger re-entry cost,
337
- plus decision-dense context a snapshot cannot faithfully reconstruct.
338
-
339
- **What the plan boundary buys:** it is the one point where the cost argument and
340
- the preserve-in-flight-judgment argument AGREE. The work has just been handed off
341
- in a merge report and the follow-ups are closed, so in-flight judgment is at its
342
- minimum and state is externalised into beads.
343
-
344
- **Applying it in this sweep:** an orchestrator flagged in the 50–85% bands is
345
- NOT a restart candidate on context alone. Check whether its plan is complete
346
- (merged + follow-ups closed). If it is, restart regardless of how low its
347
- context is. If it is not, leave it alone regardless of how high.
348
-
349
-
350
-
351
- | ctx_tier | stuck_working | Action |
352
- | -------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
353
- | < 50% | N | No action — agent has runway |
354
- | < 50% | Y | Tmux-send "are you stuck? `/continue` if waiting" nudge; if not recovered in next sweep, surface to operator |
355
- | 50-70% | N | **NOT a warning** — OPTIONAL restart-if-IDLE only: directed inbox restart request when the agent is idle. A BUSY agent (or a coordinator, or a warm-hold/on-call/parked consultant) at 50-70% gets no action — see role/state carve-out below. |
356
- | 50-70% | Y | Tmux-send nudge; defer the tier-1 directed restart request until the next sweep confirms the pane is active again AND ctx is still in this band |
357
- | 70-85% | N | Tmux-send `/thrum:restart` (bypasses inbox; more forceful) — for coordinators/warm-hold consultants ACTIVELY WORKING, this band doesn't apply until the role/state carve-out's higher floor (75%, see below) |
358
- | 70-85% | Y | Surface to operator immediately (degraded + stuck → human-eyes-needed) |
359
- | > 85% | any | Snapshot-gated restart (opt-in): surface `recommend-restart-extended` by default; autonomous restart only when `restart_actuation=true` in config (see Step 5) |
360
- | `(n/a)` | any | Pane capture failed OR runtime has no Ctx footer — check tmux session manually |
361
-
362
- **Role/state-aware carve-out:** coordinators, and any agent explicitly standing by as a
363
- warm-hold:/on-call:/parked: consultant in an ongoing impl, are treated exactly
364
- like a coordinator — while ACTIVELY WORKING (not idle) they only flag at
365
- **ctx >= 75%**, not the normal 50% `CTX_THRESHOLD`. An IDLE coordinator (or idle
366
- consultant) still flags at the normal threshold — idle restarts are cheap, no
367
- exemption needed there. Implemented as `hb_effective_ctx_threshold` in
368
- `scripts/heartbeat-lib.sh` — the 75% floor is a named var
369
- (`HB_COORD_CTX_THRESHOLD_WORKING`), a one-line swap if it ever changes. Locked
370
- test scenarios: coord@72%+working → NOT flagged; coord@76%+working → flagged;
371
- coord@60%+idle → flagged; non-coordinator roles are unaffected (still 50% while
372
- working or idle).
373
-
374
- **Warm-hold/on-call/parked exemption:** if the agent's `intent` field starts with `^warm-hold:`,
375
- `^on-call:`, or `^parked:`, skip nudge + restart for all tiers below >85% (and
376
- apply the coordinator-equivalent 75% floor above while working). The >85%
377
- bracket still fires regardless. These are EXPLICIT DECLARED states, not a
378
- heuristic guess at coordination intent — a consultant/on-call/done-planning
379
- researcher must set one of these prefixes ITSELF via
380
- `thrum session set-intent "warm-hold: <reason>"` (or `on-call:`/`parked:`).
381
-
382
- **A coordinator CANNOT set another agent's intent.** `thrum session set-intent`
383
- takes TEXT only — it sets the intent for the CURRENT SESSION and accepts no
384
- agent argument and no `--to`. To get an intent onto another agent, message that
385
- agent and have it set its own; then verify in the focused fleet view with
386
- `thrum team @AGENT --offline --include-inactive --json`
387
- rather than by counting acks.
388
-
389
- **Background-child-aware busy detection:** an agent with an OPEN in-progress bead, a recent JSONL tool_use
390
- (`state == "working"`), or a pane showing it's driving background sub-agents
391
- (`Waiting for N background agent...`) is BUSY, not idle — even if pane-silent.
392
- This is checked BEFORE the idle/abandoned classification (`hb_idle_classify`'s
393
- `is_busy` param) so an orchestrator quietly waiting on its own dispatched
394
- sub-agents is never misread as abandoned just because it hasn't committed
395
- recently.
396
-
397
- **Finished-impl teardown:** a closed bead + idle + no
398
- new dispatch classifies as `finished`, which surfaces a `recommend-teardown`
399
- ALERT to the coordinator — it is a RECOMMENDATION only, never autonomous;
400
- teardown itself stays coordinator-actuated (kill-tmux → worktree-teardown per
401
- lifecycle discipline in `implementer-status-and-handoff`).
402
-
403
- ## Step 2.5 — Stuck-working axis
404
-
405
- `stuck_working` is **orthogonal** to `ctx_tier`. They can compound. Treat the
406
- table above as a composite lookup: the action depends on the _cell_, not on
407
- either axis alone.
408
-
409
- `stuck_working` measures "is this agent currently unable to make progress?"
410
- while `ctx_tier` measures "how soon will this agent degrade if left alone?" An
411
- agent can be:
412
-
413
- - ctx-fine but stuck-working (waiting on a hung tool call) → nudge to unstick
414
- - ctx-degraded but not stuck-working (working normally but burning context) →
415
- restart cleanly
416
- - both → composite action per the table; high-degradation cases skip the nudge
417
- step since the agent likely can't recover from the nudge alone
418
-
419
- The sweep's STUCK-WORKING classification (sweep script's per-agent
420
- `stuck_working` line + ALERT-line `stuck_working=N` axis) fires when:
421
-
422
- 1. `agent_status = "working"` (the agent claims to be mid-work; set by the
423
- Pattern D self-write in role skills), AND
424
- 2. tmux silence > `SILENCE_THRESHOLD_MIN` minutes (default 10; tunable via
425
- `--silence-threshold-min N`), AND
426
- 3. JSONL transcript's last assistant message has `stop_reason = tool_use` AND
427
- was emitted more than `SILENCE_THRESHOLD_MIN` minutes ago (Claude runtime
428
- only; non-Claude runtimes use tmux silence alone)
429
- 4. `intent` does NOT start with `^warm-hold:` (warm-hold exemption)
430
-
431
- The classification is sweep-side only — it does NOT write to identity files. The
432
- identity-file `agent_status="stuck"` semantic stays reserved for
433
- `permission.markAgentStuck` writes (permission-cadence give-up).
434
-
435
- **Reliability ladder rationale:** the inbox → tmux-send → snapshot-gated restart
436
- progression goes from polite to forceful. Inbox messages can fail (delivery
437
- bugs, agent too degraded to check inbox at high ctx, or the documented self-echo
438
- regression). Tmux-send types literally into the agent's input field — bypasses
439
- inbox entirely. The snapshot-gated restart (Step 5) bypasses the agent's
440
- willingness but not the worktree safety: it first solicits a snapshot, then
441
- restarts gracefully only after one lands.
442
-
443
- Agents degrade _silently_ before they blow; 50% is the "still coherent enough to
444
- write a good snapshot" window.
445
-
446
- ## Step 3 — Directed inbox restart request for 50% – 70% agents
447
-
448
- ```bash
449
- thrum send --to @<agent_name> --stdin <<'EOF'
450
- Your context is at ~X%. Please run /thrum:restart now — write your snapshot and I will re-dispatch after.
451
- EOF
452
- ```
453
-
454
- The agent writes their restart snapshot to
455
- `.thrum/agents/<agent>/sessions/<timestamp>-restart.md` and goes idle.
456
- Coordinator waits for the "snapshot written" acknowledgement before
457
- re-dispatching the next task.
458
-
459
- If the agent acks without writing the snapshot first (anti-pattern), gently
460
- remind them to write the snapshot before going idle so the next session restores
461
- cleanly.
462
-
463
- If the agent doesn't respond within ~5 minutes OR you see them keep working,
464
- escalate to Step 4 (tmux-send nudge).
465
-
466
- ## Step 4 — Tmux-send nudge for 70% – 85% agents
467
-
468
- The inbox path may not be reaching them at this context level. Bypass the inbox
469
- by typing the restart command directly into their input field:
470
-
471
- ```bash
472
- thrum tmux send <tmux_session_name> '/thrum:restart'
473
- ```
474
-
475
- If it exits 3 (Enter WITHHELD, text already typed) or 4 (verdict unknown), do NOT retype it: `thrum tmux capture` the pane and submit with `thrum tmux send <tmux_session_name> --keys Enter` only if no dialog is showing.
476
-
477
- (Find the tmux session name in sweep output — it's typically the worktree
478
- basename, e.g. `myepic-impl`, NOT the agent_id.)
479
-
480
- This causes the runtime to immediately execute `/thrum:restart` as if the user
481
- typed it. The agent writes their snapshot + restarts.
482
-
483
- If the tmux-send doesn't trigger a restart within ~5 minutes (agent may be too
484
- degraded to process input), escalate to Step 5 (snapshot-gated restart).
485
-
486
- ## Step 5 — Snapshot-gated restart for >85% agents
487
-
488
- The >85% tier is the ONLY autonomous actuator in the whole heartbeat system —
489
- and even it is **opt-in, off by default**. The behavior is controlled by
490
- `heartbeat.lenses.context_tiers.params.restart_actuation` in config (default:
491
- `false`).
492
-
493
- ### When `restart_actuation` is `false` (the default)
494
-
495
- The sweep does NOT restart anything. It emits a `recommend-restart-extended`
496
- reason in the ALERT line, surfacing the situation to the coordinator role. When
497
- this reason appears, decide whether to run `/thrum:restart-extended` on the
498
- agent — a deliberate, human-in-the-loop action. No automatic restart fires.
499
-
500
- ### When `restart_actuation` is `true` (explicitly enabled by the operator)
501
-
502
- The sweep runs a 4-step snapshot-gated procedure automatically:
503
-
504
- 1. **Capture pre-send timestamp** — record the current wall-clock time.
505
- 2. **Send `/thrum:restart-extended` via raw tmux send-keys** — types the restart
506
- command directly into the agent's pane (bypasses inbox).
507
- 3. **Sleep a settle window** — default 180 s, tunable via `HB_SNAPSHOT_SETTLE`
508
- env var.
509
- 4. **Check for a FRESH snapshot file** — looks for a `*-restart.md` file in
510
- `.thrum/agents/<agent>/sessions/` written within the freshness window
511
- (default 200 s, tunable via `HB_SNAPSHOT_WINDOW` — a distinct env var from
512
- Step 3's settle window, `HB_SNAPSHOT_SETTLE`):
513
- - **Fresh snapshot found:** runs `thrum tmux restart <agent_id>` (graceful).
514
- **NEVER `--force`.** The `thrum tmux restart` call in this path takes no
515
- `--force` flag, by design — if the agent wrote a snapshot, there is no
516
- reason to force-interrupt mid-call.
517
- - **No fresh snapshot:** does NOT force anything. The agent is tracked in a
518
- `no_snapshot` backstop instead (see A2 note below).
519
-
520
- **NEVER `--force`.** This path does not use `--force` under any circumstances.
521
- Forceful interruption of a mid-tool-call agent is not worth the risk when the
522
- graceful path is available and the agent has runway before catastrophic
523
- degradation.
524
-
525
- ### Safety checks that still apply
526
-
527
- The following guards from "Pre-restart safety checks" carry forward and are
528
- checked BEFORE the snapshot-gated procedure fires:
529
-
530
- - **Warm-hold exemption** — if `intent` starts with `^warm-hold:`, skip ALL
531
- restart actions for this agent (the >85% bracket is NOT exempt from warm-hold
532
- in the snapshot-gated model; an operator who sets warm-hold is taking explicit
533
- responsibility).
534
- - **Cooldown** — do not restart the same agent twice within 30 minutes. If an
535
- agent crosses threshold again that fast, surface to operator rather than
536
- restart-loop.
537
- - **No-restart-mid-merge** — never restart an agent whose pane shows a Git merge
538
- conflict or active rebase. Surface to operator instead.
539
-
540
- ### "degrading-no-restart" backstop
541
-
542
- If an agent stays at tier3 without a successful restart landing — whether
543
- because `restart_actuation` is `false`, or because a snapshot never appeared
544
- within the settle window — for N consecutive surfaced ticks (default: 3,
545
- config-tunable via
546
- `heartbeat.lenses.context_tiers.params.degrading_flag_after_ticks`), the sweep
547
- adds a distinct `degrading-no-restart` reason to the ALERT. This ensures the
548
- coordinator still sees that the agent needs eyes even when the system alone
549
- cannot fix it.
550
-
551
- ### After a successful restart
552
-
553
- See "Post-restart procedure" below.
554
-
555
- ## Step 6 — API-error recovery (who nudges depends on how the sweep runs)
556
-
557
- When you run the sweep **manually** (the command in Step 1, no flags), it still
558
- auto-nudges every agent whose pane shows an `API Error` line — it types
559
- `continue` into the affected pane via `tmux send-keys` (bypassing the
560
- `thrum tmux send` wrapper queue, which stalls on fully-silent panes). The report header lists who was auto-nudged:
561
-
562
- ```text
563
- # auto-nudged 3 agent(s) on api_errors with 'continue':
564
- # - impl_foo @ foo-impl:0.0
565
- # ...
566
- ```
567
-
568
- Pass `--no-nudge` (alias `--report-only`) to get a pure detection run with **no
569
- pane writes** — useful to inspect before acting.
570
-
571
- **The daemon does NOT auto-nudge from the sweep.** The daemon-hosted built-in
572
- sweep always runs the script with `--no-nudge`: a daemon must not silently type
573
- into agent panes as a side-effect of a _detection_ sweep. It reports flagged
574
- agents to the coordinator role only.
575
-
576
- **Deliberate daemon auto-remediation is a separate, opt-in feature**
577
- (`daemon.auto_remediation.enabled`, **default OFF**). When an operator turns it
578
- on, the daemon — not this skill — applies the recovery tier ladder:
579
-
580
- - 1st detection of an agent's API error → one `continue` nudge.
581
- - Still erroring on the next tick (nudge didn't clear it) → it escalates
582
- SUSPECTED-STUCK to the operator chain instead of re-nudging (no nudge-loops).
583
- - It skips panes mid git merge/rebase, audits every nudge/escalation, and resets
584
- an agent's state on recovery.
585
-
586
- So if `daemon.auto_remediation.enabled` is on, you don't manage API-error
587
- recovery by hand — the daemon does, with the same no-repeat + escalate
588
- safeguards described here.
589
-
590
- Anthropic 529s and rate limits are transient (typically resolve in
591
- seconds-to-minutes); the agent's previous tool call is queued in-session, so a
592
- single `continue` reactivates them without losing in-flight state.
593
-
594
- **When to escalate (manual runs / daemon off):** if the same agent shows an API
595
- error on TWO consecutive sweeps despite a nudge, the issue isn't transient —
596
- surface to operator as SUSPECTED-STUCK and investigate manually
597
- (status.claude.com, network, account limits). (When daemon auto-remediation is
598
- on, it raises this escalation for you.)
599
-
600
- **When NOT to nudge:** before shipping a release, to fold one more fix into the
601
- current cycle while an agent is stuck, run with `--no-nudge` (or hold the
602
- manual sweep). Once `continue` fires, the agent
603
- resumes its previous tool call immediately — there's no recovery window.
604
-
605
- ## Pre-restart safety checks
606
-
607
- Whether triggered by the scheduled `context-monitoring` thrum monitor or by the
608
- coordinator manually invoking the skill, run these guards BEFORE firing a
609
- restart:
610
-
611
- 1. **Verify the daemon is reachable**:
612
- Run `thrum daemon status`; on error, skip the sweep and surface to operator.
613
- Inspect the local online roster with `thrum team --json` (`.team.members`);
614
- use `--offline --include-inactive` for offline, stale and retired rows.
615
- 2. **Confirm the monitor is alive**: `thrum monitor list` should show
616
- `context-monitoring` in `running` status with a non-empty `SCHEDULE` column.
617
- If absent or dead, the scheduled ALERTs aren't firing and the skill must be
618
- invoked manually from a recurring cron until the monitor is restored. (See
619
- "Re-register the monitor" below.)
620
- 3. **Check if any agent is mid-commit** (active tool call): look for
621
- `Running bash` or active spinner in the sweep pane lines — if so, defer
622
- restart for that agent until the tool completes.
623
- 4. **Never restart (graceful or otherwise) an agent whose pane shows a Git merge
624
- conflict or active rebase** — that corrupts the worktree. Surface to operator
625
- instead.
626
- 5. **Cooldown**: do not restart the same agent twice within 30 minutes. If an
627
- agent crosses threshold again that fast, something's wrong with their
628
- workload — surface to operator rather than restart-loop.
629
-
630
- ### Re-register the monitor
631
-
632
- If `thrum monitor list` doesn't show `context-monitoring`, register it from the
633
- main repo. The script path must be absolute (the daemon runs the command from a
634
- clean env, not the coordinator's shell), so resolve it from the repo root first:
635
-
636
- ```bash
637
- SCRIPT="$(git -C /path/to/thrum/main-repo rev-parse --show-toplevel)/scripts/error-and-context-agent-sweep.sh"
638
- thrum monitor add \
639
- --name context-monitoring \
640
- --schedule "@every 30m" \
641
- --match '^ALERT:' \
642
- --to coordinator \
643
- -- bash "$SCRIPT" --no-nudge --out /tmp/agent-sweep.txt
644
- ```
645
-
646
- (Or substitute the literal absolute path if no shell is handy in the
647
- daemon's environment.) The monitor fires one-shot per scheduled tick; in between
648
- ticks the child does not run, so there's no continuous CPU cost from the sweep.
649
- Note that `@every 30m` is the outer envelope — the sweep script self-gates
650
- so most ticks produce no ALERT output and thus no message fires.
651
-
652
- ## Post-restart procedure
653
-
654
- When a restart fires (Step 4 or 5 — tmux-send nudge or snapshot-gated restart):
655
-
656
- 1. Surface a brief status note to the coordinator: "Restarted @\<agent_name\>
657
- at \<ctx\>% — snapshot at
658
- `.thrum/agents/<agent>/sessions/<timestamp>-restart.md`".
659
- 2. Wait for the agent to come back online (`thrum team @AGENT --offline --include-inactive` shows them online again,
660
- or their pane shows the runtime prompt).
661
- 3. Re-send their current dispatch with the full scope + plan refs + AC targets —
662
- treat them as a fresh implementer who needs the full briefing again.
663
- 4. Note any WIP files they may have left in their worktree from the prior
664
- attempt (they'll audit salvage-vs-discard before substantive code).
665
- 5. If they had a partial DONE, the coordinator's git log should still have it;
666
- the agent may need a pointer to commits they shipped before the blow.
667
-
668
- ## Reference
669
-
670
- - **Sweep script**: `scripts/error-and-context-agent-sweep.sh` (captures
671
- `ctx_used: X.X%` from Claude JSONL transcript; falls back to pane scan for
672
- non-Claude runtimes). Now part of a sweep-script family (sibling:
673
- `waiting-on-coord-agent-sweep.sh`).
674
- - **Memory key**: project-local `coordinator-rule-context-check-broadcast` may
675
- capture project-specific tweaks; load via
676
- `thrum memory list --kind agent_rule --scope role`.
677
- - **Related discipline**:
678
- - `feedback_restart_discipline` — burn the runway; don't preempt-restart at
679
- clean checkpoints
680
- - `feedback_byte_equality_pane_detection` — pane-snapshot byte diffs are
681
- unreliable; use structural anchors + settle windows
682
- - **Why thresholds matter**: agents at 97% context silently produce degraded
683
- output (missed instructions, partial tool calls, slow responses) before they
684
- blow. The 70%/85% thresholds give the system 15-30% runway to extract a
685
- snapshot or trigger a graceful restart cleanly.