opencode-thrum-pro 0.1.12 → 0.1.14
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/assets/commands/thrum-compact-extended.md +5 -0
- package/assets/commands/thrum-compact.md +5 -0
- package/assets/skills/coordinator-assessing-agent-completion/SKILL.md +17 -20
- package/assets/skills/coordinator-deploying-a-node/SKILL.md +2 -2
- package/assets/skills/coordinator-hotpath-merge-gate/SKILL.md +11 -2
- package/assets/skills/coordinator-merging-code/SKILL.md +79 -5
- package/assets/skills/coordinator-philosophy-merge-gate/SKILL.md +2 -2
- package/assets/skills/coordinator-rolling-out-to-nodes/SKILL.md +2 -2
- package/assets/skills/diagnosing-fleet-and-daemon/SKILL.md +1 -1
- package/assets/skills/implementer-receiving-dispatch/SKILL.md +1 -1
- package/assets/skills/orchestrate/SKILL.md +1 -1
- package/assets/skills/persistent-watcher-archetype/SKILL.md +24 -8
- package/assets/skills/persistent-watcher-archetype/resources/roster-watch.sh +3 -3
- package/assets/skills/project-setup/SKILL.md +7 -12
- package/assets/skills/project-setup/resources/implementation-agent.md +2 -2
- package/assets/skills/researcher-maintaining-memory/SKILL.md +1 -1
- package/assets/skills/thrum/resources/CLI_REFERENCE.md +1 -1
- package/assets/skills/thrum/resources/LISTENER_PATTERN.md +2 -2
- package/assets/skills/using-monitors/SKILL.md +156 -18
- package/assets/skills/using-tmux-ssh/SKILL.md +121 -0
- package/assets/skills/using-tmux-ssh/resources/tmux-ssh +1158 -0
- package/assets/skills/verify-against-plan/SKILL.md +1 -1
- package/dist/installer.js +3 -5
- package/package.json +1 -1
- package/assets/skills/coordinator-context-monitoring/SKILL.md +0 -685
- package/assets/skills/coordinator-post-restart-sweep/SKILL.md +0 -135
|
@@ -1,685 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: coordinator-context-monitoring
|
|
3
|
-
description: "Use when managing live implementer/brainstormer agents during a long coordination session, at epic merge gates, after a busy dispatch hour, or whenever you suspect an agent is approaching context limits. Prevents 97%-context silent blow-ups by running a sweep + pre-emptive restart before the agent degrades. Safe to wire into a recurring cron that INVOKES this skill — the skill applies tier-ladder judgment, surfacing (and, only when explicitly opted in via config, autonomously restarting) at the >85% tier. What's forbidden is a cron/script that fires restarts unconditionally without going through this skill's tier ladder."
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
# Coordinator: Context Monitoring and Pre-emptive Restart
|
|
7
|
-
|
|
8
|
-
## When to invoke
|
|
9
|
-
|
|
10
|
-
Trigger this pattern at each of:
|
|
11
|
-
|
|
12
|
-
- Receipt of an `ALERT: flagged=…` message from the `context-monitoring` thrum
|
|
13
|
-
monitor (the canonical scheduled sweep, outer tick `@every 30m`, self-gated
|
|
14
|
-
so effective rate is slower; see "How the scheduled sweep works")
|
|
15
|
-
- Epic merge gates (after merging a sub-epic — E6.4, E6.5, etc.)
|
|
16
|
-
- After dispatching 3+ tasks in quick succession
|
|
17
|
-
- After any agent has been running for 60+ minutes without a restart
|
|
18
|
-
- On observing slow or degraded responses from an implementer
|
|
19
|
-
- Manually whenever the session feels "intense" (lots of cycles in a short
|
|
20
|
-
window)
|
|
21
|
-
|
|
22
|
-
The skill applies tier-ladder judgment: the >85% tier surfaces a
|
|
23
|
-
`recommend-restart-extended` reason (or, when `restart_actuation=true` in
|
|
24
|
-
config, runs a snapshot-gated restart) only on actual ctx %, not on every sweep.
|
|
25
|
-
What's forbidden is a script that bypasses this skill and fires
|
|
26
|
-
`thrum tmux restart --force` unconditionally (that violates
|
|
27
|
-
`feedback_restart_discipline` — burn the runway, don't restart on schedule).
|
|
28
|
-
|
|
29
|
-
## Coordinator self-restart — ALWAYS the EXTENDED form, compact-extended FIRST
|
|
30
|
-
|
|
31
|
-
This skill's tier ladder (Steps 3–5) governs how the coordinator restarts
|
|
32
|
-
**other** agents. When the **coordinator restarts ITSELF**, a different rule
|
|
33
|
-
applies and it is absolute — but the current ruling
|
|
34
|
-
REVERSES which command that is:
|
|
35
|
-
|
|
36
|
-
> **The coordinator ALWAYS uses the EXTENDED snapshot grade for itself — never
|
|
37
|
-
> the standard/compact form. Which command carries that grade is
|
|
38
|
-
> runtime-directed: `/thrum:compact-extended` on any runtime with a verified
|
|
39
|
-
> compact-equivalent skill (the default), `/thrum:restart-extended` ONLY
|
|
40
|
-
> as the fallback on a runtime without one (today: muse, copilot).**
|
|
41
|
-
|
|
42
|
-
This is a REVERSAL of the prior absolute rule ("ALWAYS restart-extended, NEVER
|
|
43
|
-
compact") — that rule is retired, not merely relaxed. Do not restore it.
|
|
44
|
-
|
|
45
|
-
Rationale: the coordinator is the nerve center. Its handoff must carry the full
|
|
46
|
-
16-section snapshot (wire contracts, capability matrix, cycle history, open
|
|
47
|
-
L-questions, anticipated Q&A, design rationale) so the next session resumes
|
|
48
|
-
steady-state coordination without re-deriving fleet state — that requirement
|
|
49
|
-
is UNCHANGED and is what "EXTENDED" governs; only the compact-vs-restart choice
|
|
50
|
-
changed. Compacting (rather than restarting) additionally keeps the SAME
|
|
51
|
-
process/session alive — sub-agents, background loops, and cron schedules the
|
|
52
|
-
coordinator owns survive the compaction, where a restart would drop them.
|
|
53
|
-
There is no ctx-band at which the coordinator downgrades to the standard/
|
|
54
|
-
compact-grade snapshot — the tier ladder below governs OTHER agents, not this.
|
|
55
|
-
|
|
56
|
-
Practical trigger points for coordinator self-restart/compact:
|
|
57
|
-
|
|
58
|
-
- A warn-tier context nudge fires on the coordinator's own pane (e.g. 70%+).
|
|
59
|
-
- The coordinator hits a clean checkpoint and elects a fresh session.
|
|
60
|
-
- Rate-limit / stuck-state recovery on the coordinator itself.
|
|
61
|
-
|
|
62
|
-
In every one of these: run `/thrum:compact-extended` if the coordinator's
|
|
63
|
-
runtime has a compact equivalent (the default); `/thrum:restart-extended` only
|
|
64
|
-
if it does not. (The role-rule `coord-always-restart-extended` — `Coordinator
|
|
65
|
-
ALWAYS uses /thrum:restart-extended` — is RETIRED; see
|
|
66
|
-
`thrum memory` for its retirement record. This skill is the always-loaded home
|
|
67
|
-
so the CURRENT rule survives even when memory isn't consulted.)
|
|
68
|
-
|
|
69
|
-
Note the distinction from Steps 3–5 below: those steps drive `/thrum:restart`
|
|
70
|
-
(Step 4) and `/thrum:restart-extended` (Step 5) into
|
|
71
|
-
**implementer/brainstormer** panes per their ctx tier — that tiering is correct
|
|
72
|
-
for worker agents. The carve-out here is ONLY about the coordinator's own
|
|
73
|
-
restart/compact, which is unconditionally the extended grade, compact-first.
|
|
74
|
-
|
|
75
|
-
## How the scheduled sweep works (v0.10.6+ — thrum monitor)
|
|
76
|
-
|
|
77
|
-
The sweep runs as a daemon-managed `thrum monitor` job named
|
|
78
|
-
`context-monitoring`, registered with an `@every 30m` schedule. The 30-min tick
|
|
79
|
-
is the outer envelope — the sweep is self-gated: daytime ticks only fire
|
|
80
|
-
when the fleet is busy (`agent_status=working`), and overnight ticks fire only
|
|
81
|
-
every other tick (effective 60-min rate), so most ticks may be silent no-ops.
|
|
82
|
-
There is a SINGLE monitor for ALL lenses in v1 — not one per lens. The job
|
|
83
|
-
invokes
|
|
84
|
-
`scripts/error-and-context-agent-sweep.sh --no-nudge --out /tmp/agent-sweep.txt`.
|
|
85
|
-
The script emits a single consolidated `ALERT:` line to stdout when ANY agent
|
|
86
|
-
crosses a threshold (ctx >= 50% OR api-error OR capture-fail); when the fleet is
|
|
87
|
-
clean, the script is silent so no message fires. The monitor's
|
|
88
|
-
`--match '^ALERT:'` filter routes the ALERT line as a message to the
|
|
89
|
-
`coordinator` role (fanned out to every LIVE local coordinator — if no
|
|
90
|
-
coordinator is currently live, the sweep is a silent no-op rather than queuing
|
|
91
|
-
for an offline one), which triggers this skill.
|
|
92
|
-
|
|
93
|
-
Format of the ALERT line:
|
|
94
|
-
|
|
95
|
-
```text
|
|
96
|
-
ALERT: flagged=N stuck=S stuck_working=W tier3=T tier2=U — agent_a(92%,api-err,STUCK,stuck-working); agent_b(88%); …
|
|
97
|
-
```
|
|
98
|
-
|
|
99
|
-
or, on a benign-only tick once the periodic heartbeat interval has elapsed:
|
|
100
|
-
|
|
101
|
-
```text
|
|
102
|
-
ALERT: all-clear — N agents healthy, 0 actionable (sweep alive)
|
|
103
|
-
```
|
|
104
|
-
|
|
105
|
-
- `flagged` — total agents needing attention
|
|
106
|
-
- `stuck` — api-errored on TWO consecutive sweeps (state file tracks this across
|
|
107
|
-
runs)
|
|
108
|
-
- `stuck_working` — agent_status=working AND tmux quiet > threshold AND no
|
|
109
|
-
recent JSONL tool calls; threshold tunable via `--silence-threshold-min`,
|
|
110
|
-
default 10 min
|
|
111
|
-
- `tier3` — count with ctx >= 85% (snapshot-gated restart candidates; see
|
|
112
|
-
Step 5)
|
|
113
|
-
- `tier2` — count with 70-84% ctx (tmux-send nudge candidates)
|
|
114
|
-
- Per-agent segment: `name(ctx%,reason-if-any,classifier)` joined by `;`
|
|
115
|
-
|
|
116
|
-
### RUN vs DELIVER
|
|
117
|
-
|
|
118
|
-
The sweep now ALWAYS computes flags every scheduled tick (daytime; overnight
|
|
119
|
-
keeps its even-tick-only cadence) — RUN is no longer gated on a coarse
|
|
120
|
-
fleet-busy signal, which used to silently suppress genuinely actionable signals
|
|
121
|
-
before flags were even computed.
|
|
122
|
-
|
|
123
|
-
DELIVERY of the `ALERT:` line is a separate decision, made AFTER flags are
|
|
124
|
-
computed:
|
|
125
|
-
|
|
126
|
-
- **ACTIONABLE** (delivers the real ALERT) — tier2/tier3 ctx, workstall, stuck /
|
|
127
|
-
stuck_working, waiting-on-coord, blocked-on-human, capture-fail,
|
|
128
|
-
ctx-unknown-idle, api-err, idle-mid-task (abandoned), finished-impl (recommend
|
|
129
|
-
teardown), awaiting-restart, pending-human.
|
|
130
|
-
- **BENIGN-ONLY** (never delivers on its own) — idle-no-task and/or bare ctx
|
|
131
|
-
tier1 (50-69%) alone. `flagged=20 tier3=0` where every one of those 20 is
|
|
132
|
-
benign-only does **not** fire an ALERT — `flagged` counts everything written
|
|
133
|
-
to the per-agent report; it does not imply delivery.
|
|
134
|
-
- **Periodic all-clear heartbeat** — configurable via `.thrum/config.json`
|
|
135
|
-
`heartbeat.allclear_interval_min` (default 60). If no actionable ALERT has
|
|
136
|
-
fired for that long, the sweep emits the minimal `all-clear` line above so a
|
|
137
|
-
benign-quiet fleet is never mistaken for a dead monitor. Any delivered ALERT
|
|
138
|
-
(actionable or all-clear) resets this clock.
|
|
139
|
-
|
|
140
|
-
The full per-agent report stays at `/tmp/agent-sweep.txt` (overwritten each
|
|
141
|
-
sweep) for on-demand drill-down — read it AFTER receiving an ALERT to see which
|
|
142
|
-
specific panes are at risk.
|
|
143
|
-
|
|
144
|
-
The previous keepalive-cron pattern (CronCreate `<id>`) is deprecated in
|
|
145
|
-
favor of this scheduled monitor. The bookkeeping responsibility moves out of the
|
|
146
|
-
coordinator's per-session re-add chore and into the daemon's durable monitors
|
|
147
|
-
table (survives daemon restart, no per-session re-init needed for this monitor —
|
|
148
|
-
though OTHER CronCreate jobs may still require it per
|
|
149
|
-
`feedback_cron_reinit_each_session`).
|
|
150
|
-
|
|
151
|
-
## Lenses
|
|
152
|
-
|
|
153
|
-
Every lens SURFACES or RECOMMENDS by default — none take autonomous action on
|
|
154
|
-
their own. The ONLY exception is `context_tiers`' snapshot-gated restart, and
|
|
155
|
-
even that requires an explicit `restart_actuation: true` opt-in in config. There
|
|
156
|
-
is a single monitor for all lenses in v1, not one per lens.
|
|
157
|
-
|
|
158
|
-
### Default-ON lenses (implemented, E1–E6)
|
|
159
|
-
|
|
160
|
-
| Lens | Default | Disposition |
|
|
161
|
-
| --------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
162
|
-
| `context_tiers` | ON | The ctx%/stuck-working ladder (L1). Sole autonomous actuator when `restart_actuation=true`; otherwise RECOMMEND via `recommend-restart-extended` reason in ALERT. |
|
|
163
|
-
| `idle_mid_task` | ON | L2 — 30-min idle-with-open-task detection (bead cross-ref). RECOMMEND `/thrum:sleep-extended`; a no-task variant REPORTs `idle-no-task` at lower priority. |
|
|
164
|
-
| `snapshot_awaiting_restart` | ON | L3 — pane text + fresh-snapshot two-signal heuristic. RECOMMEND `thrum tmux restart <agent>` (never autonomous). |
|
|
165
|
-
| `blocked_on_human_modal` | ON | L4 — detects spend-limit/permission/consent modal prompts. DETECTION ONLY, never auto-answers; routes into the L5 ledger. |
|
|
166
|
-
| `pending_human_ledger` | ON | L5 — flat JSONL ledger of items awaiting a human. EXEMPT from progressive backoff — surfaces every tick. `hb_ledger_resolve` (the resolution path) is defined but has zero call sites — nothing marks an entry resolved, so the ledger is append-only-forever, not "until resolved." |
|
|
167
|
-
| `waiting_on_coord` | ON | L9 — 21-rule pattern match (folded in from the standalone waiting-on-coord sweep) + warm-hold exemption. RECOMMEND coordinator answer. |
|
|
168
|
-
|
|
169
|
-
### Default-OFF lenses (E8, flag-gated — NOT YET IMPLEMENTED as of E7)
|
|
170
|
-
|
|
171
|
-
These lenses are reserved for E8. They exist as config flags for future work but
|
|
172
|
-
produce no output until enabled.
|
|
173
|
-
|
|
174
|
-
| Lens | Default | Disposition |
|
|
175
|
-
| ----------------------- | ------- | -------------------------------- |
|
|
176
|
-
| `merge_ready_queue` | OFF | L6 — REPORT |
|
|
177
|
-
| `merge_reconciliation` | OFF | L7 — REPORT |
|
|
178
|
-
| `base_deploy_staleness` | OFF | L8 — REPORT |
|
|
179
|
-
| `ready_vs_capacity` | OFF | L10 — RECOMMEND |
|
|
180
|
-
| `stale_agent_gc` | OFF | L11 — propose-only GC, RECOMMEND |
|
|
181
|
-
| `daemon_wedge_trend` | OFF | L12 — REPORT |
|
|
182
|
-
|
|
183
|
-
## Configuration
|
|
184
|
-
|
|
185
|
-
The heartbeat system is configured under the `heartbeat` key in
|
|
186
|
-
`.thrum/config.json` (Go type: `HeartbeatConfig`). Example:
|
|
187
|
-
|
|
188
|
-
```json
|
|
189
|
-
{
|
|
190
|
-
"heartbeat": {
|
|
191
|
-
"schema_version": 1,
|
|
192
|
-
"enabled": true,
|
|
193
|
-
"cadence_minutes_day": 30,
|
|
194
|
-
"cadence_minutes_overnight": 60,
|
|
195
|
-
"busy_only_daytime": true,
|
|
196
|
-
"busy_signal": "working_or_activity_30m",
|
|
197
|
-
"overnight_window": {
|
|
198
|
-
"from": "22:00",
|
|
199
|
-
"to": "07:00",
|
|
200
|
-
"tz": "America/New_York"
|
|
201
|
-
},
|
|
202
|
-
"backoff": { "start_after": 2, "multiplier": 2, "floor_every": 4 },
|
|
203
|
-
"lenses": {
|
|
204
|
-
"context_tiers": {
|
|
205
|
-
"enabled": true,
|
|
206
|
-
"params": {
|
|
207
|
-
"restart_actuation": false,
|
|
208
|
-
"degrading_flag_after_ticks": 3
|
|
209
|
-
}
|
|
210
|
-
},
|
|
211
|
-
"idle_mid_task": { "enabled": true },
|
|
212
|
-
"...": "..."
|
|
213
|
-
}
|
|
214
|
-
}
|
|
215
|
-
}
|
|
216
|
-
```
|
|
217
|
-
|
|
218
|
-
**Field reference:**
|
|
219
|
-
|
|
220
|
-
| Field | Purpose |
|
|
221
|
-
| -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
222
|
-
| `cadence_minutes_day` | Daytime cadence target (default 30 min). The underlying monitor runs `@every 30m` and self-gates so only busy-fleet daytime ticks fire. |
|
|
223
|
-
| `cadence_minutes_overnight` | Overnight cadence target (default 60 min). The monitor fires every other `@every 30m` tick overnight (effective 60-min rate). |
|
|
224
|
-
| `busy_only_daytime` | When `true`, daytime ticks are skipped unless any agent has `agent_status=working` (busy-signal gate). |
|
|
225
|
-
| `busy_signal` | Algorithm for "is the fleet busy?" Default `working_or_activity_30m`: any agent working OR activity in the last 30 min. |
|
|
226
|
-
| `overnight_window.tz` | **MUST be an explicit IANA timezone** (e.g. `"America/New_York"`). If unset, the sweep warns and falls back to always-daytime-rate rather than silently guessing. |
|
|
227
|
-
| `backoff.start_after` | Number of consecutive unchanged detections of the same problem before progressive backoff begins. |
|
|
228
|
-
| `backoff.multiplier` | Each backoff step multiplies the surface interval by this factor (default 2×). |
|
|
229
|
-
| `backoff.floor_every` | The surface interval is capped at this many ticks — the problem is never fully silenced (always surfaces at least once per `floor_every` ticks). |
|
|
230
|
-
| `lenses.<name>.enabled` | Enable/disable a lens individually. |
|
|
231
|
-
| `lenses.<name>.params` | Per-lens parameters. The only actuation opt-in in the whole system is `context_tiers.params.restart_actuation` (default `false`). |
|
|
232
|
-
| `lenses.context_tiers.params.degrading_flag_after_ticks` | After this many consecutive tier3 surfaced ticks without a successful restart, a `degrading-no-restart` reason is added to the ALERT (default 3). |
|
|
233
|
-
|
|
234
|
-
### Editing the config (there is NO `thrum config set`)
|
|
235
|
-
|
|
236
|
-
`thrum config` only has a `show` subcommand — you **hand-edit
|
|
237
|
-
`.thrum/config.json`** to configure the heartbeat. Two gotchas make this
|
|
238
|
-
non-obvious; both bite silently.
|
|
239
|
-
|
|
240
|
-
**1. All-or-nothing merge (the sharp footgun).** The config
|
|
241
|
-
loader (`LoadThrumConfig`) applies
|
|
242
|
-
`DefaultHeartbeatConfig()` **only when the stanza is effectively absent**
|
|
243
|
-
(`schema_version == 0` AND zero lenses):
|
|
244
|
-
|
|
245
|
-
```go
|
|
246
|
-
if cfg.Heartbeat.SchemaVersion == 0 && len(cfg.Heartbeat.Lenses) == 0 {
|
|
247
|
-
cfg.Heartbeat = DefaultHeartbeatConfig()
|
|
248
|
-
}
|
|
249
|
-
```
|
|
250
|
-
|
|
251
|
-
There is **no field-level deep merge.** The moment you write a stanza with
|
|
252
|
-
`"schema_version": 1`, every field you _omit_ takes its Go zero-value —
|
|
253
|
-
`cadence_minutes_day: 0`, `enabled: false`, and **all 12 lenses disabled**. So
|
|
254
|
-
"I'll just set the timezone" with a three-line stanza silently turns the whole
|
|
255
|
-
sweep off. **Always write the COMPLETE stanza** — copy
|
|
256
|
-
`DefaultHeartbeatConfig()` in full, then change
|
|
257
|
-
only what you need.
|
|
258
|
-
|
|
259
|
-
**2. `overnight_window.tz` MUST be an explicit IANA zone.** Default is `""` (the
|
|
260
|
-
source comment reads "TZ set by operator"). Left unset, the sweep prints
|
|
261
|
-
`warning: heartbeat.overnight_window.tz not configured — using local time for gating`
|
|
262
|
-
and falls back to the box's local clock (warns rather than silently
|
|
263
|
-
guessing). Set it to the operator's zone, e.g. `"America/Los_Angeles"` (PST/PDT)
|
|
264
|
-
or `"America/New_York"` (EST/EDT). Never `"local"`.
|
|
265
|
-
|
|
266
|
-
**Procedure** — back up, write the full stanza, validate, done (no restart):
|
|
267
|
-
|
|
268
|
-
```bash
|
|
269
|
-
cp .thrum/config.json .thrum/config.json.bak
|
|
270
|
-
# Build a COMPLETE heartbeat block mirroring DefaultHeartbeatConfig with your tz,
|
|
271
|
-
# then splice it in (jq keeps the rest of config.json intact):
|
|
272
|
-
jq --slurpfile hb /tmp/hb-block.json '.heartbeat = $hb[0]' .thrum/config.json > /tmp/c.json \
|
|
273
|
-
&& mv /tmp/c.json .thrum/config.json
|
|
274
|
-
jq -e '.heartbeat.overnight_window.tz != "" and (.heartbeat.lenses|length)==12' .thrum/config.json
|
|
275
|
-
```
|
|
276
|
-
|
|
277
|
-
**No daemon restart is required.** The scheduled monitor invokes the sweep
|
|
278
|
-
_script_, which reads `.thrum/config.json` directly on every tick — the new tz
|
|
279
|
-
takes effect on the next sweep. (`thrum config show` reflects the daemon's
|
|
280
|
-
in-memory config, cached at boot, so it will lag until the next daemon start;
|
|
281
|
-
that lag does not affect the script-driven sweep.) Avoid a
|
|
282
|
-
restart-just-for-config — it needlessly risks a cold-boot WAL-replay
|
|
283
|
-
brick.
|
|
284
|
-
|
|
285
|
-
## Subagent model selection
|
|
286
|
-
|
|
287
|
-
> **Model tiers:** pass an explicit `model:` on every dispatch; choose model and effort per the `choosing-subagent-models` skill (read `runtime.role_models` at the moment of use). Never let a sub-agent inherit your own model.
|
|
288
|
-
|
|
289
|
-
## Step 1 — Run the sweep
|
|
290
|
-
|
|
291
|
-
```bash
|
|
292
|
-
bash scripts/error-and-context-agent-sweep.sh --out /tmp/agent-sweep.txt
|
|
293
|
-
grep -E 'ctx_used:|^===== ' /tmp/agent-sweep.txt
|
|
294
|
-
```
|
|
295
|
-
|
|
296
|
-
The script emits one `ctx_used: X.X%` line per live agent. It captures the
|
|
297
|
-
Claude Code status bar footer, normalizing UTF-8 non-breaking spaces
|
|
298
|
-
(`\xc2\xa0`) before matching `Ctx Used: X.X%`. Runtimes without that footer
|
|
299
|
-
(Codex, Cursor) fall back to `(n/a)`.
|
|
300
|
-
|
|
301
|
-
## Step 2 — Threshold logic
|
|
302
|
-
|
|
303
|
-
### 🔴 ORCHESTRATORS ARE NOT ON THE CONTEXT LADDER — they restart on a PLAN BOUNDARY
|
|
304
|
-
|
|
305
|
-
**The context thresholds below are a COORDINATOR rule. Orchestrators inherited it
|
|
306
|
-
rather than being chosen for it, and it is not cost-optimal for them.** Do not
|
|
307
|
-
hold an orchestrator open to reach 70–75%, and do not restart one mid-plan to
|
|
308
|
-
chase the cost curve.
|
|
309
|
-
|
|
310
|
-
**The orchestrator rule:**
|
|
311
|
-
|
|
312
|
-
> **Restart an orchestrator at PLAN COMPLETION — where completion means the plan
|
|
313
|
-
> is MERGED *and* the follow-up beads filed during that plan have been FIXED.**
|
|
314
|
-
> Not at the merge report. Not at the merge.
|
|
315
|
-
|
|
316
|
-
**Why the follow-ups clause is load-bearing, not a detail.** The standing
|
|
317
|
-
fix-while-warm rule says a bug found mid-work gets FIXED while warm, because
|
|
318
|
-
file-and-move-on costs ~10x — the research has to be redone before the fix can
|
|
319
|
-
start. Bugs found during a plan are therefore filed and swept up at the END of
|
|
320
|
-
the plan, while the orchestrator *and its implementers* still hold the context.
|
|
321
|
-
**A restart at the merge boundary destroys exactly that warmth, and does so
|
|
322
|
-
INVISIBLY — the follow-ups still get done, just cold and at the 10x price.** A
|
|
323
|
-
restart rule keyed to "the merge" does not merely mistime a restart; it silently
|
|
324
|
-
repeals fix-while-warm.
|
|
325
|
-
|
|
326
|
-
**Why the split is real, not stylistic**:
|
|
327
|
-
|
|
328
|
-
- Per-turn cost scales LINEARLY with context size (the whole prefix is re-sent
|
|
329
|
-
and billed at the cache-read rate every turn), so cumulative session cost is
|
|
330
|
-
roughly QUADRATIC in how far context is allowed to grow. Break-even for a
|
|
331
|
-
restart is **under one turn** once context is meaningfully above the re-prime
|
|
332
|
-
floor.
|
|
333
|
-
- **Orchestrators re-prime far cheaper than coordinators, and it is structural:**
|
|
334
|
-
the prime output filter gives ONLY `role=="coordinator"` the full
|
|
335
|
-
project-state passthrough. Measured orchestrator re-prime ≈ 30k tokens.
|
|
336
|
-
- Coordinators keep the ladder below for a stated reason — larger re-entry cost,
|
|
337
|
-
plus decision-dense context a snapshot cannot faithfully reconstruct.
|
|
338
|
-
|
|
339
|
-
**What the plan boundary buys:** it is the one point where the cost argument and
|
|
340
|
-
the preserve-in-flight-judgment argument AGREE. The work has just been handed off
|
|
341
|
-
in a merge report and the follow-ups are closed, so in-flight judgment is at its
|
|
342
|
-
minimum and state is externalised into beads.
|
|
343
|
-
|
|
344
|
-
**Applying it in this sweep:** an orchestrator flagged in the 50–85% bands is
|
|
345
|
-
NOT a restart candidate on context alone. Check whether its plan is complete
|
|
346
|
-
(merged + follow-ups closed). If it is, restart regardless of how low its
|
|
347
|
-
context is. If it is not, leave it alone regardless of how high.
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
| ctx_tier | stuck_working | Action |
|
|
352
|
-
| -------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
353
|
-
| < 50% | N | No action — agent has runway |
|
|
354
|
-
| < 50% | Y | Tmux-send "are you stuck? `/continue` if waiting" nudge; if not recovered in next sweep, surface to operator |
|
|
355
|
-
| 50-70% | N | **NOT a warning** — OPTIONAL restart-if-IDLE only: directed inbox restart request when the agent is idle. A BUSY agent (or a coordinator, or a warm-hold/on-call/parked consultant) at 50-70% gets no action — see role/state carve-out below. |
|
|
356
|
-
| 50-70% | Y | Tmux-send nudge; defer the tier-1 directed restart request until the next sweep confirms the pane is active again AND ctx is still in this band |
|
|
357
|
-
| 70-85% | N | Tmux-send `/thrum:restart` (bypasses inbox; more forceful) — for coordinators/warm-hold consultants ACTIVELY WORKING, this band doesn't apply until the role/state carve-out's higher floor (75%, see below) |
|
|
358
|
-
| 70-85% | Y | Surface to operator immediately (degraded + stuck → human-eyes-needed) |
|
|
359
|
-
| > 85% | any | Snapshot-gated restart (opt-in): surface `recommend-restart-extended` by default; autonomous restart only when `restart_actuation=true` in config (see Step 5) |
|
|
360
|
-
| `(n/a)` | any | Pane capture failed OR runtime has no Ctx footer — check tmux session manually |
|
|
361
|
-
|
|
362
|
-
**Role/state-aware carve-out:** coordinators, and any agent explicitly standing by as a
|
|
363
|
-
warm-hold:/on-call:/parked: consultant in an ongoing impl, are treated exactly
|
|
364
|
-
like a coordinator — while ACTIVELY WORKING (not idle) they only flag at
|
|
365
|
-
**ctx >= 75%**, not the normal 50% `CTX_THRESHOLD`. An IDLE coordinator (or idle
|
|
366
|
-
consultant) still flags at the normal threshold — idle restarts are cheap, no
|
|
367
|
-
exemption needed there. Implemented as `hb_effective_ctx_threshold` in
|
|
368
|
-
`scripts/heartbeat-lib.sh` — the 75% floor is a named var
|
|
369
|
-
(`HB_COORD_CTX_THRESHOLD_WORKING`), a one-line swap if it ever changes. Locked
|
|
370
|
-
test scenarios: coord@72%+working → NOT flagged; coord@76%+working → flagged;
|
|
371
|
-
coord@60%+idle → flagged; non-coordinator roles are unaffected (still 50% while
|
|
372
|
-
working or idle).
|
|
373
|
-
|
|
374
|
-
**Warm-hold/on-call/parked exemption:** if the agent's `intent` field starts with `^warm-hold:`,
|
|
375
|
-
`^on-call:`, or `^parked:`, skip nudge + restart for all tiers below >85% (and
|
|
376
|
-
apply the coordinator-equivalent 75% floor above while working). The >85%
|
|
377
|
-
bracket still fires regardless. These are EXPLICIT DECLARED states, not a
|
|
378
|
-
heuristic guess at coordination intent — a consultant/on-call/done-planning
|
|
379
|
-
researcher must set one of these prefixes ITSELF via
|
|
380
|
-
`thrum session set-intent "warm-hold: <reason>"` (or `on-call:`/`parked:`).
|
|
381
|
-
|
|
382
|
-
**A coordinator CANNOT set another agent's intent.** `thrum session set-intent`
|
|
383
|
-
takes TEXT only — it sets the intent for the CURRENT SESSION and accepts no
|
|
384
|
-
agent argument and no `--to`. To get an intent onto another agent, message that
|
|
385
|
-
agent and have it set its own; then verify in the focused fleet view with
|
|
386
|
-
`thrum team @AGENT --offline --include-inactive --json`
|
|
387
|
-
rather than by counting acks.
|
|
388
|
-
|
|
389
|
-
**Background-child-aware busy detection:** an agent with an OPEN in-progress bead, a recent JSONL tool_use
|
|
390
|
-
(`state == "working"`), or a pane showing it's driving background sub-agents
|
|
391
|
-
(`Waiting for N background agent...`) is BUSY, not idle — even if pane-silent.
|
|
392
|
-
This is checked BEFORE the idle/abandoned classification (`hb_idle_classify`'s
|
|
393
|
-
`is_busy` param) so an orchestrator quietly waiting on its own dispatched
|
|
394
|
-
sub-agents is never misread as abandoned just because it hasn't committed
|
|
395
|
-
recently.
|
|
396
|
-
|
|
397
|
-
**Finished-impl teardown:** a closed bead + idle + no
|
|
398
|
-
new dispatch classifies as `finished`, which surfaces a `recommend-teardown`
|
|
399
|
-
ALERT to the coordinator — it is a RECOMMENDATION only, never autonomous;
|
|
400
|
-
teardown itself stays coordinator-actuated (kill-tmux → worktree-teardown per
|
|
401
|
-
lifecycle discipline in `implementer-status-and-handoff`).
|
|
402
|
-
|
|
403
|
-
## Step 2.5 — Stuck-working axis
|
|
404
|
-
|
|
405
|
-
`stuck_working` is **orthogonal** to `ctx_tier`. They can compound. Treat the
|
|
406
|
-
table above as a composite lookup: the action depends on the _cell_, not on
|
|
407
|
-
either axis alone.
|
|
408
|
-
|
|
409
|
-
`stuck_working` measures "is this agent currently unable to make progress?"
|
|
410
|
-
while `ctx_tier` measures "how soon will this agent degrade if left alone?" An
|
|
411
|
-
agent can be:
|
|
412
|
-
|
|
413
|
-
- ctx-fine but stuck-working (waiting on a hung tool call) → nudge to unstick
|
|
414
|
-
- ctx-degraded but not stuck-working (working normally but burning context) →
|
|
415
|
-
restart cleanly
|
|
416
|
-
- both → composite action per the table; high-degradation cases skip the nudge
|
|
417
|
-
step since the agent likely can't recover from the nudge alone
|
|
418
|
-
|
|
419
|
-
The sweep's STUCK-WORKING classification (sweep script's per-agent
|
|
420
|
-
`stuck_working` line + ALERT-line `stuck_working=N` axis) fires when:
|
|
421
|
-
|
|
422
|
-
1. `agent_status = "working"` (the agent claims to be mid-work; set by the
|
|
423
|
-
Pattern D self-write in role skills), AND
|
|
424
|
-
2. tmux silence > `SILENCE_THRESHOLD_MIN` minutes (default 10; tunable via
|
|
425
|
-
`--silence-threshold-min N`), AND
|
|
426
|
-
3. JSONL transcript's last assistant message has `stop_reason = tool_use` AND
|
|
427
|
-
was emitted more than `SILENCE_THRESHOLD_MIN` minutes ago (Claude runtime
|
|
428
|
-
only; non-Claude runtimes use tmux silence alone)
|
|
429
|
-
4. `intent` does NOT start with `^warm-hold:` (warm-hold exemption)
|
|
430
|
-
|
|
431
|
-
The classification is sweep-side only — it does NOT write to identity files. The
|
|
432
|
-
identity-file `agent_status="stuck"` semantic stays reserved for
|
|
433
|
-
`permission.markAgentStuck` writes (permission-cadence give-up).
|
|
434
|
-
|
|
435
|
-
**Reliability ladder rationale:** the inbox → tmux-send → snapshot-gated restart
|
|
436
|
-
progression goes from polite to forceful. Inbox messages can fail (delivery
|
|
437
|
-
bugs, agent too degraded to check inbox at high ctx, or the documented self-echo
|
|
438
|
-
regression). Tmux-send types literally into the agent's input field — bypasses
|
|
439
|
-
inbox entirely. The snapshot-gated restart (Step 5) bypasses the agent's
|
|
440
|
-
willingness but not the worktree safety: it first solicits a snapshot, then
|
|
441
|
-
restarts gracefully only after one lands.
|
|
442
|
-
|
|
443
|
-
Agents degrade _silently_ before they blow; 50% is the "still coherent enough to
|
|
444
|
-
write a good snapshot" window.
|
|
445
|
-
|
|
446
|
-
## Step 3 — Directed inbox restart request for 50% – 70% agents
|
|
447
|
-
|
|
448
|
-
```bash
|
|
449
|
-
thrum send --to @<agent_name> --stdin <<'EOF'
|
|
450
|
-
Your context is at ~X%. Please run /thrum:restart now — write your snapshot and I will re-dispatch after.
|
|
451
|
-
EOF
|
|
452
|
-
```
|
|
453
|
-
|
|
454
|
-
The agent writes their restart snapshot to
|
|
455
|
-
`.thrum/agents/<agent>/sessions/<timestamp>-restart.md` and goes idle.
|
|
456
|
-
Coordinator waits for the "snapshot written" acknowledgement before
|
|
457
|
-
re-dispatching the next task.
|
|
458
|
-
|
|
459
|
-
If the agent acks without writing the snapshot first (anti-pattern), gently
|
|
460
|
-
remind them to write the snapshot before going idle so the next session restores
|
|
461
|
-
cleanly.
|
|
462
|
-
|
|
463
|
-
If the agent doesn't respond within ~5 minutes OR you see them keep working,
|
|
464
|
-
escalate to Step 4 (tmux-send nudge).
|
|
465
|
-
|
|
466
|
-
## Step 4 — Tmux-send nudge for 70% – 85% agents
|
|
467
|
-
|
|
468
|
-
The inbox path may not be reaching them at this context level. Bypass the inbox
|
|
469
|
-
by typing the restart command directly into their input field:
|
|
470
|
-
|
|
471
|
-
```bash
|
|
472
|
-
thrum tmux send <tmux_session_name> '/thrum:restart'
|
|
473
|
-
```
|
|
474
|
-
|
|
475
|
-
If it exits 3 (Enter WITHHELD, text already typed) or 4 (verdict unknown), do NOT retype it: `thrum tmux capture` the pane and submit with `thrum tmux send <tmux_session_name> --keys Enter` only if no dialog is showing.
|
|
476
|
-
|
|
477
|
-
(Find the tmux session name in sweep output — it's typically the worktree
|
|
478
|
-
basename, e.g. `myepic-impl`, NOT the agent_id.)
|
|
479
|
-
|
|
480
|
-
This causes the runtime to immediately execute `/thrum:restart` as if the user
|
|
481
|
-
typed it. The agent writes their snapshot + restarts.
|
|
482
|
-
|
|
483
|
-
If the tmux-send doesn't trigger a restart within ~5 minutes (agent may be too
|
|
484
|
-
degraded to process input), escalate to Step 5 (snapshot-gated restart).
|
|
485
|
-
|
|
486
|
-
## Step 5 — Snapshot-gated restart for >85% agents
|
|
487
|
-
|
|
488
|
-
The >85% tier is the ONLY autonomous actuator in the whole heartbeat system —
|
|
489
|
-
and even it is **opt-in, off by default**. The behavior is controlled by
|
|
490
|
-
`heartbeat.lenses.context_tiers.params.restart_actuation` in config (default:
|
|
491
|
-
`false`).
|
|
492
|
-
|
|
493
|
-
### When `restart_actuation` is `false` (the default)
|
|
494
|
-
|
|
495
|
-
The sweep does NOT restart anything. It emits a `recommend-restart-extended`
|
|
496
|
-
reason in the ALERT line, surfacing the situation to the coordinator role. When
|
|
497
|
-
this reason appears, decide whether to run `/thrum:restart-extended` on the
|
|
498
|
-
agent — a deliberate, human-in-the-loop action. No automatic restart fires.
|
|
499
|
-
|
|
500
|
-
### When `restart_actuation` is `true` (explicitly enabled by the operator)
|
|
501
|
-
|
|
502
|
-
The sweep runs a 4-step snapshot-gated procedure automatically:
|
|
503
|
-
|
|
504
|
-
1. **Capture pre-send timestamp** — record the current wall-clock time.
|
|
505
|
-
2. **Send `/thrum:restart-extended` via raw tmux send-keys** — types the restart
|
|
506
|
-
command directly into the agent's pane (bypasses inbox).
|
|
507
|
-
3. **Sleep a settle window** — default 180 s, tunable via `HB_SNAPSHOT_SETTLE`
|
|
508
|
-
env var.
|
|
509
|
-
4. **Check for a FRESH snapshot file** — looks for a `*-restart.md` file in
|
|
510
|
-
`.thrum/agents/<agent>/sessions/` written within the freshness window
|
|
511
|
-
(default 200 s, tunable via `HB_SNAPSHOT_WINDOW` — a distinct env var from
|
|
512
|
-
Step 3's settle window, `HB_SNAPSHOT_SETTLE`):
|
|
513
|
-
- **Fresh snapshot found:** runs `thrum tmux restart <agent_id>` (graceful).
|
|
514
|
-
**NEVER `--force`.** The `thrum tmux restart` call in this path takes no
|
|
515
|
-
`--force` flag, by design — if the agent wrote a snapshot, there is no
|
|
516
|
-
reason to force-interrupt mid-call.
|
|
517
|
-
- **No fresh snapshot:** does NOT force anything. The agent is tracked in a
|
|
518
|
-
`no_snapshot` backstop instead (see A2 note below).
|
|
519
|
-
|
|
520
|
-
**NEVER `--force`.** This path does not use `--force` under any circumstances.
|
|
521
|
-
Forceful interruption of a mid-tool-call agent is not worth the risk when the
|
|
522
|
-
graceful path is available and the agent has runway before catastrophic
|
|
523
|
-
degradation.
|
|
524
|
-
|
|
525
|
-
### Safety checks that still apply
|
|
526
|
-
|
|
527
|
-
The following guards from "Pre-restart safety checks" carry forward and are
|
|
528
|
-
checked BEFORE the snapshot-gated procedure fires:
|
|
529
|
-
|
|
530
|
-
- **Warm-hold exemption** — if `intent` starts with `^warm-hold:`, skip ALL
|
|
531
|
-
restart actions for this agent (the >85% bracket is NOT exempt from warm-hold
|
|
532
|
-
in the snapshot-gated model; an operator who sets warm-hold is taking explicit
|
|
533
|
-
responsibility).
|
|
534
|
-
- **Cooldown** — do not restart the same agent twice within 30 minutes. If an
|
|
535
|
-
agent crosses threshold again that fast, surface to operator rather than
|
|
536
|
-
restart-loop.
|
|
537
|
-
- **No-restart-mid-merge** — never restart an agent whose pane shows a Git merge
|
|
538
|
-
conflict or active rebase. Surface to operator instead.
|
|
539
|
-
|
|
540
|
-
### "degrading-no-restart" backstop
|
|
541
|
-
|
|
542
|
-
If an agent stays at tier3 without a successful restart landing — whether
|
|
543
|
-
because `restart_actuation` is `false`, or because a snapshot never appeared
|
|
544
|
-
within the settle window — for N consecutive surfaced ticks (default: 3,
|
|
545
|
-
config-tunable via
|
|
546
|
-
`heartbeat.lenses.context_tiers.params.degrading_flag_after_ticks`), the sweep
|
|
547
|
-
adds a distinct `degrading-no-restart` reason to the ALERT. This ensures the
|
|
548
|
-
coordinator still sees that the agent needs eyes even when the system alone
|
|
549
|
-
cannot fix it.
|
|
550
|
-
|
|
551
|
-
### After a successful restart
|
|
552
|
-
|
|
553
|
-
See "Post-restart procedure" below.
|
|
554
|
-
|
|
555
|
-
## Step 6 — API-error recovery (who nudges depends on how the sweep runs)
|
|
556
|
-
|
|
557
|
-
When you run the sweep **manually** (the command in Step 1, no flags), it still
|
|
558
|
-
auto-nudges every agent whose pane shows an `API Error` line — it types
|
|
559
|
-
`continue` into the affected pane via `tmux send-keys` (bypassing the
|
|
560
|
-
`thrum tmux send` wrapper queue, which stalls on fully-silent panes). The report header lists who was auto-nudged:
|
|
561
|
-
|
|
562
|
-
```text
|
|
563
|
-
# auto-nudged 3 agent(s) on api_errors with 'continue':
|
|
564
|
-
# - impl_foo @ foo-impl:0.0
|
|
565
|
-
# ...
|
|
566
|
-
```
|
|
567
|
-
|
|
568
|
-
Pass `--no-nudge` (alias `--report-only`) to get a pure detection run with **no
|
|
569
|
-
pane writes** — useful to inspect before acting.
|
|
570
|
-
|
|
571
|
-
**The daemon does NOT auto-nudge from the sweep.** The daemon-hosted built-in
|
|
572
|
-
sweep always runs the script with `--no-nudge`: a daemon must not silently type
|
|
573
|
-
into agent panes as a side-effect of a _detection_ sweep. It reports flagged
|
|
574
|
-
agents to the coordinator role only.
|
|
575
|
-
|
|
576
|
-
**Deliberate daemon auto-remediation is a separate, opt-in feature**
|
|
577
|
-
(`daemon.auto_remediation.enabled`, **default OFF**). When an operator turns it
|
|
578
|
-
on, the daemon — not this skill — applies the recovery tier ladder:
|
|
579
|
-
|
|
580
|
-
- 1st detection of an agent's API error → one `continue` nudge.
|
|
581
|
-
- Still erroring on the next tick (nudge didn't clear it) → it escalates
|
|
582
|
-
SUSPECTED-STUCK to the operator chain instead of re-nudging (no nudge-loops).
|
|
583
|
-
- It skips panes mid git merge/rebase, audits every nudge/escalation, and resets
|
|
584
|
-
an agent's state on recovery.
|
|
585
|
-
|
|
586
|
-
So if `daemon.auto_remediation.enabled` is on, you don't manage API-error
|
|
587
|
-
recovery by hand — the daemon does, with the same no-repeat + escalate
|
|
588
|
-
safeguards described here.
|
|
589
|
-
|
|
590
|
-
Anthropic 529s and rate limits are transient (typically resolve in
|
|
591
|
-
seconds-to-minutes); the agent's previous tool call is queued in-session, so a
|
|
592
|
-
single `continue` reactivates them without losing in-flight state.
|
|
593
|
-
|
|
594
|
-
**When to escalate (manual runs / daemon off):** if the same agent shows an API
|
|
595
|
-
error on TWO consecutive sweeps despite a nudge, the issue isn't transient —
|
|
596
|
-
surface to operator as SUSPECTED-STUCK and investigate manually
|
|
597
|
-
(status.claude.com, network, account limits). (When daemon auto-remediation is
|
|
598
|
-
on, it raises this escalation for you.)
|
|
599
|
-
|
|
600
|
-
**When NOT to nudge:** before shipping a release, to fold one more fix into the
|
|
601
|
-
current cycle while an agent is stuck, run with `--no-nudge` (or hold the
|
|
602
|
-
manual sweep). Once `continue` fires, the agent
|
|
603
|
-
resumes its previous tool call immediately — there's no recovery window.
|
|
604
|
-
|
|
605
|
-
## Pre-restart safety checks
|
|
606
|
-
|
|
607
|
-
Whether triggered by the scheduled `context-monitoring` thrum monitor or by the
|
|
608
|
-
coordinator manually invoking the skill, run these guards BEFORE firing a
|
|
609
|
-
restart:
|
|
610
|
-
|
|
611
|
-
1. **Verify the daemon is reachable**:
|
|
612
|
-
Run `thrum daemon status`; on error, skip the sweep and surface to operator.
|
|
613
|
-
Inspect the local online roster with `thrum team --json` (`.team.members`);
|
|
614
|
-
use `--offline --include-inactive` for offline, stale and retired rows.
|
|
615
|
-
2. **Confirm the monitor is alive**: `thrum monitor list` should show
|
|
616
|
-
`context-monitoring` in `running` status with a non-empty `SCHEDULE` column.
|
|
617
|
-
If absent or dead, the scheduled ALERTs aren't firing and the skill must be
|
|
618
|
-
invoked manually from a recurring cron until the monitor is restored. (See
|
|
619
|
-
"Re-register the monitor" below.)
|
|
620
|
-
3. **Check if any agent is mid-commit** (active tool call): look for
|
|
621
|
-
`Running bash` or active spinner in the sweep pane lines — if so, defer
|
|
622
|
-
restart for that agent until the tool completes.
|
|
623
|
-
4. **Never restart (graceful or otherwise) an agent whose pane shows a Git merge
|
|
624
|
-
conflict or active rebase** — that corrupts the worktree. Surface to operator
|
|
625
|
-
instead.
|
|
626
|
-
5. **Cooldown**: do not restart the same agent twice within 30 minutes. If an
|
|
627
|
-
agent crosses threshold again that fast, something's wrong with their
|
|
628
|
-
workload — surface to operator rather than restart-loop.
|
|
629
|
-
|
|
630
|
-
### Re-register the monitor
|
|
631
|
-
|
|
632
|
-
If `thrum monitor list` doesn't show `context-monitoring`, register it from the
|
|
633
|
-
main repo. The script path must be absolute (the daemon runs the command from a
|
|
634
|
-
clean env, not the coordinator's shell), so resolve it from the repo root first:
|
|
635
|
-
|
|
636
|
-
```bash
|
|
637
|
-
SCRIPT="$(git -C /path/to/thrum/main-repo rev-parse --show-toplevel)/scripts/error-and-context-agent-sweep.sh"
|
|
638
|
-
thrum monitor add \
|
|
639
|
-
--name context-monitoring \
|
|
640
|
-
--schedule "@every 30m" \
|
|
641
|
-
--match '^ALERT:' \
|
|
642
|
-
--to coordinator \
|
|
643
|
-
-- bash "$SCRIPT" --no-nudge --out /tmp/agent-sweep.txt
|
|
644
|
-
```
|
|
645
|
-
|
|
646
|
-
(Or substitute the literal absolute path if no shell is handy in the
|
|
647
|
-
daemon's environment.) The monitor fires one-shot per scheduled tick; in between
|
|
648
|
-
ticks the child does not run, so there's no continuous CPU cost from the sweep.
|
|
649
|
-
Note that `@every 30m` is the outer envelope — the sweep script self-gates
|
|
650
|
-
so most ticks produce no ALERT output and thus no message fires.
|
|
651
|
-
|
|
652
|
-
## Post-restart procedure
|
|
653
|
-
|
|
654
|
-
When a restart fires (Step 4 or 5 — tmux-send nudge or snapshot-gated restart):
|
|
655
|
-
|
|
656
|
-
1. Surface a brief status note to the coordinator: "Restarted @\<agent_name\>
|
|
657
|
-
at \<ctx\>% — snapshot at
|
|
658
|
-
`.thrum/agents/<agent>/sessions/<timestamp>-restart.md`".
|
|
659
|
-
2. Wait for the agent to come back online (`thrum team @AGENT --offline --include-inactive` shows them online again,
|
|
660
|
-
or their pane shows the runtime prompt).
|
|
661
|
-
3. Re-send their current dispatch with the full scope + plan refs + AC targets —
|
|
662
|
-
treat them as a fresh implementer who needs the full briefing again.
|
|
663
|
-
4. Note any WIP files they may have left in their worktree from the prior
|
|
664
|
-
attempt (they'll audit salvage-vs-discard before substantive code).
|
|
665
|
-
5. If they had a partial DONE, the coordinator's git log should still have it;
|
|
666
|
-
the agent may need a pointer to commits they shipped before the blow.
|
|
667
|
-
|
|
668
|
-
## Reference
|
|
669
|
-
|
|
670
|
-
- **Sweep script**: `scripts/error-and-context-agent-sweep.sh` (captures
|
|
671
|
-
`ctx_used: X.X%` from Claude JSONL transcript; falls back to pane scan for
|
|
672
|
-
non-Claude runtimes). Now part of a sweep-script family (sibling:
|
|
673
|
-
`waiting-on-coord-agent-sweep.sh`).
|
|
674
|
-
- **Memory key**: project-local `coordinator-rule-context-check-broadcast` may
|
|
675
|
-
capture project-specific tweaks; load via
|
|
676
|
-
`thrum memory list --kind agent_rule --scope role`.
|
|
677
|
-
- **Related discipline**:
|
|
678
|
-
- `feedback_restart_discipline` — burn the runway; don't preempt-restart at
|
|
679
|
-
clean checkpoints
|
|
680
|
-
- `feedback_byte_equality_pane_detection` — pane-snapshot byte diffs are
|
|
681
|
-
unreliable; use structural anchors + settle windows
|
|
682
|
-
- **Why thresholds matter**: agents at 97% context silently produce degraded
|
|
683
|
-
output (missed instructions, partial tool calls, slow responses) before they
|
|
684
|
-
blow. The 70%/85% thresholds give the system 15-30% runway to extract a
|
|
685
|
-
snapshot or trigger a graceful restart cleanly.
|