opencode-thrum-pro 0.1.0 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/assets/commands/thrum-_stamp-protocol.md +65 -0
- package/assets/skills/brainstorm-steward/SKILL.md +15 -0
- package/assets/skills/choosing-subagent-models/SKILL.md +19 -11
- package/assets/skills/conversation-only-context/SKILL.md +186 -0
- package/assets/skills/coordinator-dispatching-work/SKILL.md +28 -8
- package/assets/skills/coordinator-hotpath-merge-gate/SKILL.md +68 -0
- package/assets/skills/coordinator-merging-code/SKILL.md +24 -0
- package/assets/skills/coordinator-running-brainstorm-cycles/SKILL.md +11 -7
- package/assets/skills/efficient-multi-agent-research/SKILL.md +7 -6
- package/assets/skills/persistent-watcher-archetype/SKILL.md +68 -9
- package/assets/skills/persistent-watcher-archetype/resources/thrum-capture-fallback.sh +123 -0
- package/assets/skills/persistent-watcher-archetype/resources/thrum-watch-pane-capture.sh +379 -0
- package/assets/skills/prime-agent/SKILL.md +2 -4
- package/assets/skills/project-setup/SKILL.md +118 -21
- package/assets/skills/update-queue/SKILL.md +3 -1
- package/assets/thrum_allowlist.json +34 -0
- package/dist/allowlist-sync.test.d.ts +1 -0
- package/dist/allowlist-sync.test.js +44 -0
- package/dist/index.js +24 -1
- package/dist/installer.d.ts +29 -0
- package/dist/installer.js +356 -1
- package/dist/installer.permissions.test.d.ts +1 -0
- package/dist/installer.permissions.test.js +602 -0
- package/dist/permission-fixture.integration.test.d.ts +1 -0
- package/dist/permission-fixture.integration.test.js +506 -0
- package/package.json +3 -2
|
@@ -42,6 +42,13 @@ every agent in your `roster`:
|
|
|
42
42
|
pattern check, not a judgment call.
|
|
43
43
|
- Never approve blind: read the actual command text first.
|
|
44
44
|
- Verify by re-capture after acting — never trust exit status alone.
|
|
45
|
+
- To unblock a REMOTE roster member's modal, use `thrum tmux send <agent> " "`
|
|
46
|
+
(proxies by agent name to the owning peer daemon and appends Enter, so a
|
|
47
|
+
space+Enter confirms the default-focused option) — never `thrum tmux key`,
|
|
48
|
+
which is local-socket-only and errors on a remote agent. This grants only
|
|
49
|
+
the leftmost/default option; the bright line above still applies. From the
|
|
50
|
+
CLI/peer-router path this send is QUEUED behind the conservative
|
|
51
|
+
monitor-silence wait, not instant — don't re-send while waiting it out.
|
|
45
52
|
|
|
46
53
|
## Your own restart (auto-restart-at-ctx-threshold)
|
|
47
54
|
|
|
@@ -171,25 +178,77 @@ failure this archetype exists to prevent, inflicted on itself by the wrong
|
|
|
171
178
|
choice of plumbing.
|
|
172
179
|
|
|
173
180
|
Register the loop as a `thrum monitor` instead — a daemon-scheduled job,
|
|
174
|
-
independent of your session, that keeps running across your restarts
|
|
181
|
+
independent of your session, that keeps running across your restarts.
|
|
182
|
+
|
|
183
|
+
**A ready-to-use script ships with this skill** — `resources/thrum-watch-
|
|
184
|
+
pane-capture.sh` (paired with `resources/thrum-capture-fallback.sh`,
|
|
185
|
+
thrum-3mhrt). It is generic and roster-driven: it reads your own
|
|
186
|
+
`watch_params.json`, captures each roster member's pane (local `thrum tmux
|
|
187
|
+
capture` first, always — that already reaches remote agents by name via the
|
|
188
|
+
normal rpcrouter proxy; see "SSH fallback" below for what the second script
|
|
189
|
+
is for), runs `thrum detect --category permission`, applies the two-capture
|
|
190
|
+
stability check, writes one report file, and emits one matchable summary
|
|
191
|
+
line. It is mechanical only — no auto-escalate, no auto-key — exactly the
|
|
192
|
+
scope this section already describes; judgment stays in your own turn.
|
|
193
|
+
|
|
194
|
+
**Setup (one-time per watcher instance):** `CLAUDE_PLUGIN_ROOT` is a
|
|
195
|
+
Claude-Code-hook-only env var — it is NOT present in `thrum monitor`'s
|
|
196
|
+
scheduled-process environment, so the script cannot locate itself inside
|
|
197
|
+
the plugin tree at run time. Copy both scripts out of the skill's
|
|
198
|
+
`resources/` into your own agent directory first:
|
|
175
199
|
|
|
176
200
|
```bash
|
|
177
|
-
|
|
178
|
-
|
|
201
|
+
cp "${CLAUDE_PLUGIN_ROOT}/skills/persistent-watcher-archetype/resources/thrum-watch-pane-capture.sh" \
|
|
202
|
+
"${CLAUDE_PLUGIN_ROOT}/skills/persistent-watcher-archetype/resources/thrum-capture-fallback.sh" \
|
|
203
|
+
.thrum/agents/<you>/
|
|
204
|
+
chmod +x .thrum/agents/<you>/thrum-watch-pane-capture.sh .thrum/agents/<you>/thrum-capture-fallback.sh
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
Then register the monitor against that absolute copy path —
|
|
208
|
+
**`--notify-on-success` is MANDATORY, not optional decoration**: a
|
|
209
|
+
`--schedule`d job delivers NOTHING on `--match` alone (thrum-ruz1z §5c —
|
|
210
|
+
this fails completely silently, every status field reads healthy, and it
|
|
211
|
+
is exactly the trap a literal reading of the syntax below used to walk
|
|
212
|
+
readers into):
|
|
213
|
+
|
|
214
|
+
```bash
|
|
215
|
+
thrum monitor start --name <you>-pane-watch \
|
|
216
|
+
--match "^watch-tick: (cycle done|RESOLUTION-FAIL)" \
|
|
179
217
|
--to @<you> \
|
|
180
|
-
--
|
|
218
|
+
--notify-on-success \
|
|
219
|
+
--schedule '*/15 * * * *' \
|
|
220
|
+
-- /absolute/path/to/.thrum/agents/<you>/thrum-watch-pane-capture.sh
|
|
181
221
|
```
|
|
182
222
|
|
|
183
|
-
The
|
|
184
|
-
|
|
185
|
-
act — `thrum monitor` matches that line and delivers it to you as a thrum
|
|
186
|
-
message, which is your wake signal. Judgment (duties 1-4 above) happens in
|
|
187
|
-
your own turn when you read that message, not in the script.
|
|
223
|
+
(The `RESOLUTION-FAIL` alternative in `--match` is the script's own
|
|
224
|
+
fork-default signal — see "SSH fallback" below.)
|
|
188
225
|
|
|
189
226
|
At the start of every cycle, confirm the monitor is still running
|
|
190
227
|
(`thrum monitor list`) — if it's missing or dead, re-`thrum monitor start`
|
|
191
228
|
it rather than assuming someone else will notice.
|
|
192
229
|
|
|
230
|
+
### SSH fallback (resilience net, not the primary path)
|
|
231
|
+
|
|
232
|
+
`thrum tmux capture <agent>` already reaches remote agents by name via the
|
|
233
|
+
rpcrouter proxy — that is how fleet-wide watching works today for a healthy
|
|
234
|
+
peer. `thrum-capture-fallback.sh` exists for when that proxy path is
|
|
235
|
+
broken on the caller's side (known bug classes: thrum-7vwgy same-host
|
|
236
|
+
dual-daemon caller-identity collision; thrum-zkqut phantom-routing
|
|
237
|
+
regression on a daemon build predating `EnsureProxies`). It is a strict
|
|
238
|
+
FALLBACK, never primary and never SSH-first: `thrum-watch-pane-capture.sh`
|
|
239
|
+
always tries the local proxy path first, and only retries via an SSH hop —
|
|
240
|
+
using a per-agent SSH target resolved from `thrum state`
|
|
241
|
+
(`agent_pool:<agent>` → box hostname → matched against `topology` →
|
|
242
|
+
`ssh.target`/`ssh.user`/`repo_path`) — on the specific known proxy-failure
|
|
243
|
+
error signature. Resolution fails CLOSED on any gap (missing box, no
|
|
244
|
+
matching topology row, a placeholder `repo_path`) — that agent simply gets
|
|
245
|
+
no SSH fallback for the run, it is not skipped from capture entirely, and
|
|
246
|
+
local-only capture failures still show up in the report. If EVERY roster
|
|
247
|
+
agent fails resolution in one run (likely a fleet-wide topology/agent_pool
|
|
248
|
+
data gap, not routine per-agent degradation), the script flips its summary
|
|
249
|
+
line to the `RESOLUTION-FAIL` marker so `--match` treats it as a real
|
|
250
|
+
problem rather than a clean tick.
|
|
251
|
+
|
|
193
252
|
## Relationship to the deterministic context-monitoring sweep
|
|
194
253
|
|
|
195
254
|
You are NOT a replacement for `scripts/error-and-context-agent-sweep.sh`
|
|
@@ -0,0 +1,123 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# thrum-capture-fallback.sh — capture/key an agent, falling back to an SSH
|
|
3
|
+
# hop when the local `thrum tmux capture|key` proxy path fails with a known
|
|
4
|
+
# cross-daemon proxy-routing error signature (thrum-7vwgy / thrum-zkqut
|
|
5
|
+
# class bugs: same-host dual-daemon caller-identity collision, and a
|
|
6
|
+
# phantom-routing regression on daemon builds predating EnsureProxies).
|
|
7
|
+
#
|
|
8
|
+
# `thrum tmux capture <agent>` ALREADY reaches remote agents by name via the
|
|
9
|
+
# rpcrouter proxy — this script is a resilience FALLBACK for when that path
|
|
10
|
+
# is broken on the CALLER's side, not a replacement for it and not a
|
|
11
|
+
# fleet-wide-watching enabler in its own right. Local-first, always.
|
|
12
|
+
#
|
|
13
|
+
# This papers over those bugs rather than fixing them — it does not touch
|
|
14
|
+
# thrum's daemon/peer/proxy config. It retries the identical command via
|
|
15
|
+
# SSH, cd'd into the TARGET AGENT's own repo checkout on its box, using a
|
|
16
|
+
# caller-resolved ssh target/user/repo. There is no hardcoded host or repo
|
|
17
|
+
# in this script — resolution is the CALLER's job (see the companion
|
|
18
|
+
# thrum-watch-pane-capture.sh's resolve_agent_ssh, which reads thrum state
|
|
19
|
+
# agent_pool/topology to find the right box + repo per agent).
|
|
20
|
+
#
|
|
21
|
+
# Usage:
|
|
22
|
+
# thrum-capture-fallback.sh capture <agent_name> <ssh_target> <ssh_user> <repo_path>
|
|
23
|
+
# thrum-capture-fallback.sh key <agent_name> <ssh_target> <ssh_user> <repo_path> <key1> [key2 ...]
|
|
24
|
+
#
|
|
25
|
+
# <ssh_target> may be the literal string "NONE" to mean "no SSH fallback is
|
|
26
|
+
# available for this agent" (e.g. the caller's target resolution failed
|
|
27
|
+
# closed on a missing/placeholder topology.repo_path) — in that case a
|
|
28
|
+
# local failure is returned as-is, with no SSH attempt at all.
|
|
29
|
+
#
|
|
30
|
+
# <ssh_user> may be empty ("") to let ssh/its own config resolve the user
|
|
31
|
+
# (e.g. when <ssh_target> is itself a configured Host alias) — in that case
|
|
32
|
+
# the ssh destination is just <ssh_target>, not user@target.
|
|
33
|
+
#
|
|
34
|
+
# Exit status: propagates the underlying thrum command's exit status from
|
|
35
|
+
# whichever path (local or SSH) actually produced the returned output.
|
|
36
|
+
set -uo pipefail
|
|
37
|
+
|
|
38
|
+
MODE="${1:?usage: $0 <capture|key> <agent_name> <ssh_target> <ssh_user> <repo_path> [keys...]}"
|
|
39
|
+
AGENT="${2:?usage: $0 <capture|key> <agent_name> <ssh_target> <ssh_user> <repo_path> [keys...]}"
|
|
40
|
+
SSH_TARGET="${3:?usage: $0 <capture|key> <agent_name> <ssh_target> <ssh_user> <repo_path> [keys...]}"
|
|
41
|
+
SSH_USER="${4-}"
|
|
42
|
+
REPO_PATH="${5-}"
|
|
43
|
+
shift 5 || true
|
|
44
|
+
EXTRA_ARGS=("$@")
|
|
45
|
+
|
|
46
|
+
# Known local-proxy failure signatures that mean "try the SSH fallback."
|
|
47
|
+
# MUST STAY IN SYNC BY HAND with thrum-watch-pane-capture.sh's identical
|
|
48
|
+
# is_proxy_failure() -- no shared-sourcing across the two files. This
|
|
49
|
+
# script is also directly callable standalone (e.g. `key` to approve/deny
|
|
50
|
+
# a modal on a cross-daemon agent), so it re-derives the signature
|
|
51
|
+
# independently rather than trusting a caller's prior classification.
|
|
52
|
+
is_proxy_failure() {
|
|
53
|
+
grep -qE 'rpcrouter: caller-peer lacks required capability|peer unreachable|circuit open, dial skipped' <<<"$1"
|
|
54
|
+
}
|
|
55
|
+
|
|
56
|
+
ssh_destination() {
|
|
57
|
+
if [ -n "${SSH_USER}" ]; then
|
|
58
|
+
printf '%s@%s' "${SSH_USER}" "${SSH_TARGET}"
|
|
59
|
+
else
|
|
60
|
+
printf '%s' "${SSH_TARGET}"
|
|
61
|
+
fi
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
run_local() {
|
|
65
|
+
case "$MODE" in
|
|
66
|
+
# "${EXTRA_ARGS[@]:-}" not "${EXTRA_ARGS[@]}": a DECLARED-EMPTY array
|
|
67
|
+
# subscripted with [@] is a fatal unbound-variable under `set -u` on
|
|
68
|
+
# bash <4.4 (a real bug, not a style choice -- `key` mode with zero
|
|
69
|
+
# key args crashes on this fleet's /bin/bash 3.2.57 without the ":-").
|
|
70
|
+
capture) thrum tmux capture "$AGENT" 2>&1 ;;
|
|
71
|
+
key) thrum tmux key "$AGENT" "${EXTRA_ARGS[@]:-}" 2>&1 ;;
|
|
72
|
+
esac
|
|
73
|
+
}
|
|
74
|
+
|
|
75
|
+
run_ssh() {
|
|
76
|
+
local dest cmd
|
|
77
|
+
dest="$(ssh_destination)"
|
|
78
|
+
case "$MODE" in
|
|
79
|
+
capture) cmd="thrum tmux capture '$AGENT'" ;;
|
|
80
|
+
key) cmd="thrum tmux key '$AGENT' ${EXTRA_ARGS[*]:-}" ;;
|
|
81
|
+
esac
|
|
82
|
+
# Client-side expansion of REPO_PATH/cmd into the remote command string is
|
|
83
|
+
# intentional -- that's how the resolved repo path and thrum invocation
|
|
84
|
+
# reach the remote shell at all. REPO_PATH and AGENT are quoted here so a
|
|
85
|
+
# space in either (e.g. a topology repo_path with a space) doesn't break
|
|
86
|
+
# the remote `cd` -- consistent with fail-closed-on-bad-topology-data
|
|
87
|
+
# everywhere else in this pair of scripts.
|
|
88
|
+
# shellcheck disable=SC2029
|
|
89
|
+
ssh "${dest}" "cd '${REPO_PATH}' && ${cmd}" 2>&1
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
# ssh-add retry: if the SSH hop itself fails at the connection layer (not a
|
|
93
|
+
# thrum-level error), the caller's identities may need re-adding to the
|
|
94
|
+
# keychain-backed ssh-agent. No default keys ship here -- this is a
|
|
95
|
+
# generic plugin script, and baking in one operator's specific key
|
|
96
|
+
# filenames would be wrong for every other fleet/operator that copies it.
|
|
97
|
+
# Set THRUM_CAPTURE_FALLBACK_SSH_KEYS (space-separated paths) per-operator
|
|
98
|
+
# if this retry needs to add specific identities; with it unset, `ssh-add
|
|
99
|
+
# --apple-use-keychain` still runs with no extra args, which adds ssh's
|
|
100
|
+
# own default identities.
|
|
101
|
+
retry_with_ssh_add() {
|
|
102
|
+
local keys="${THRUM_CAPTURE_FALLBACK_SSH_KEYS:-}"
|
|
103
|
+
# Intentional word-splitting of a space-separated key-path list below.
|
|
104
|
+
# shellcheck disable=SC2086
|
|
105
|
+
ssh-add --apple-use-keychain ${keys} >/dev/null 2>&1
|
|
106
|
+
run_ssh
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
OUT="$(run_local)"
|
|
110
|
+
STATUS=$?
|
|
111
|
+
|
|
112
|
+
if [ "${STATUS}" -ne 0 ] && is_proxy_failure "${OUT}" && [ "${SSH_TARGET}" != "NONE" ]; then
|
|
113
|
+
OUT="$(run_ssh)"
|
|
114
|
+
STATUS=$?
|
|
115
|
+
# SSH connection-layer failure (not a thrum proxy error) -> try ssh-add once.
|
|
116
|
+
if [ "${STATUS}" -ne 0 ] && grep -qE 'Permission denied|Could not resolve hostname|Connection refused|Connection timed out|Too many authentication failures' <<<"${OUT}"; then
|
|
117
|
+
OUT="$(retry_with_ssh_add)"
|
|
118
|
+
STATUS=$?
|
|
119
|
+
fi
|
|
120
|
+
fi
|
|
121
|
+
|
|
122
|
+
echo "${OUT}"
|
|
123
|
+
exit "${STATUS}"
|
|
@@ -0,0 +1,379 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
#
|
|
3
|
+
# thrum-watch-pane-capture.sh
|
|
4
|
+
#
|
|
5
|
+
# The generic, roster-driven pane-watch capture-loop script that ships with
|
|
6
|
+
# the persistent-watcher-archetype plugin skill (thrum-3mhrt). Works for
|
|
7
|
+
# BOTH a persistent-watcher-archetype watcher and the brainstorm-steward
|
|
8
|
+
# (one parameterized script serves both — the only differences, roster
|
|
9
|
+
# contents and cadence, are already `watch_params.json` fields, not script
|
|
10
|
+
# forks).
|
|
11
|
+
#
|
|
12
|
+
# WHAT THIS SCRIPT DOES (single, non-resident pass, driven by `thrum
|
|
13
|
+
# monitor`'s own schedule — never a loop; the next wake is the monitor's
|
|
14
|
+
# job, not this script's):
|
|
15
|
+
# Read roster from watch_params.json -> resolve each roster member's
|
|
16
|
+
# cross-daemon SSH-fallback target ONCE per run (thrum state
|
|
17
|
+
# agent_pool/topology; fail CLOSED on any gap, never guess a path) ->
|
|
18
|
+
# capture each member's pane (local `thrum tmux capture` first, ALWAYS —
|
|
19
|
+
# `thrum tmux capture <agent>` already reaches remote agents by name via
|
|
20
|
+
# the normal rpcrouter proxy; the SSH hop below is a resilience FALLBACK
|
|
21
|
+
# for when that path is broken on the caller's side, e.g. thrum-7vwgy/
|
|
22
|
+
# thrum-zkqut-class bugs, never a fleet-wide-watching enabler in its own
|
|
23
|
+
# right, and never SSH-first) -> run each capture through `thrum detect
|
|
24
|
+
# --category permission` -> for a STABLE (unchanged across two captures,
|
|
25
|
+
# two seconds apart) detected permission prompt, flag it in the report ->
|
|
26
|
+
# write ONE report file with a section per agent (pane tail + status) ->
|
|
27
|
+
# emit ONE matchable summary line on stdout for `thrum monitor` to catch
|
|
28
|
+
# and deliver as a wake message.
|
|
29
|
+
#
|
|
30
|
+
# DELIBERATE SCOPE LIMIT: this script is MECHANICAL ONLY — capture, detect,
|
|
31
|
+
# emit. It never auto-escalates and never auto-approves/denies a modal
|
|
32
|
+
# (never calls `thrum tmux key`/`thrum send`). The persistent-watcher-
|
|
33
|
+
# archetype skill is explicit that judgment (approve/cancel/escalate/nudge)
|
|
34
|
+
# happens in the watcher's OWN turn when it reads the wake message this
|
|
35
|
+
# script's summary line triggers, not in a script. A bash script cannot
|
|
36
|
+
# safely reproduce that judgment.
|
|
37
|
+
#
|
|
38
|
+
# SETUP (per watcher/steward instance — see the skill's SKILL.md for the
|
|
39
|
+
# full recipe): copy BOTH this script and thrum-capture-fallback.sh from
|
|
40
|
+
# the skill's resources/ into your own `.thrum/agents/<you>/`, then
|
|
41
|
+
# register `thrum monitor start` against that absolute copy path.
|
|
42
|
+
# `CLAUDE_PLUGIN_ROOT` is a Claude-Code-hook-only env var, NOT present in
|
|
43
|
+
# `thrum monitor`'s scheduled-process environment, so the script cannot
|
|
44
|
+
# locate itself inside the plugin tree at run time — it must be copied
|
|
45
|
+
# next to the watch_params.json it reads.
|
|
46
|
+
#
|
|
47
|
+
# This script relies on that copy location to find its own paths: it lives
|
|
48
|
+
# at `<repo>/.thrum/agents/<name>/thrum-watch-pane-capture.sh`, so its own
|
|
49
|
+
# directory IS `.thrum/agents/<name>` and `<name>` (its own dirname's
|
|
50
|
+
# basename) IS the watcher/steward's agent name — used to namespace log/
|
|
51
|
+
# report/lock files so multiple watchers in one repo never collide.
|
|
52
|
+
#
|
|
53
|
+
# USAGE:
|
|
54
|
+
# thrum-watch-pane-capture.sh
|
|
55
|
+
#
|
|
56
|
+
# THRUM MONITOR EXAMPLE (see SKILL.md for the full, current recipe —
|
|
57
|
+
# --notify-on-success is MANDATORY: a --schedule'd job delivers NOTHING on
|
|
58
|
+
# --match alone, thrum-ruz1z/S5c):
|
|
59
|
+
# thrum monitor start --name <you>-pane-watch \
|
|
60
|
+
# --match "^watch-tick: (cycle done|RESOLUTION-FAIL)" \
|
|
61
|
+
# --to @<you> \
|
|
62
|
+
# --notify-on-success \
|
|
63
|
+
# --schedule '*/15 * * * *' \
|
|
64
|
+
# -- /path/to/.thrum/agents/<you>/thrum-watch-pane-capture.sh
|
|
65
|
+
#
|
|
66
|
+
set -uo pipefail
|
|
67
|
+
|
|
68
|
+
# --- locate paths ------------------------------------------------------
|
|
69
|
+
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
|
|
70
|
+
AGENT_NAME="$(basename -- "${SCRIPT_DIR}")"
|
|
71
|
+
# SCRIPT_DIR is <repo>/.thrum/agents/<AGENT_NAME> per the setup recipe above
|
|
72
|
+
# -- never hardcode REPO_ROOT (the prior box-specific adaptation this
|
|
73
|
+
# script generalizes did, and that was exactly the thing to fix).
|
|
74
|
+
REPO_ROOT="$(cd -- "${SCRIPT_DIR}/../../.." >/dev/null 2>&1 && pwd)"
|
|
75
|
+
PARAMS_FILE="${SCRIPT_DIR}/watch_params.json"
|
|
76
|
+
FALLBACK_SCRIPT="${SCRIPT_DIR}/thrum-capture-fallback.sh"
|
|
77
|
+
|
|
78
|
+
LOG_DIR="${REPO_ROOT}/.thrum/var/log"
|
|
79
|
+
LOCK_DIR="${REPO_ROOT}/.thrum/var/${AGENT_NAME}-pane-watch.lockdir"
|
|
80
|
+
LOG_FILE="${LOG_DIR}/${AGENT_NAME}-pane-watch.log"
|
|
81
|
+
REPORT_FILE="${LOG_DIR}/${AGENT_NAME}-pane-watch-report.txt"
|
|
82
|
+
|
|
83
|
+
mkdir -p "${LOG_DIR}"
|
|
84
|
+
|
|
85
|
+
# --- logging -------------------------------------------------------------
|
|
86
|
+
# ALWAYS write a line, even on a no-op run: a silent run is
|
|
87
|
+
# indistinguishable from a run that never happened.
|
|
88
|
+
log() {
|
|
89
|
+
printf '%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$1" >>"${LOG_FILE}"
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
# --- single-instance lock -------------------------------------------------
|
|
93
|
+
# macOS ships no `flock`(1) by default, so use `mkdir` as the atomic lock
|
|
94
|
+
# primitive instead (mkdir on an existing dir fails atomically on every
|
|
95
|
+
# POSIX filesystem -- same guarantee flock gives, no dependency, and
|
|
96
|
+
# `thrum monitor` resolves `#!/usr/bin/env bash` to macOS system /bin/bash
|
|
97
|
+
# 3.2 which also lacks flock's usual companion, `exec N>file`+flock -n N
|
|
98
|
+
# patterns some other watchers use).
|
|
99
|
+
# Stale-lock guard: if the lock dir is older than 10 min, a prior run
|
|
100
|
+
# almost certainly crashed without cleaning up -- reclaim it rather than
|
|
101
|
+
# wedging every future cycle forever.
|
|
102
|
+
if ! mkdir "${LOCK_DIR}" 2>/dev/null; then
|
|
103
|
+
if [ -d "${LOCK_DIR}" ]; then
|
|
104
|
+
LOCK_AGE=$(( $(date +%s) - $(stat -f %m "${LOCK_DIR}" 2>/dev/null || echo 0) ))
|
|
105
|
+
if [ "${LOCK_AGE}" -gt 600 ]; then
|
|
106
|
+
log "WARN: stale lock dir (${LOCK_AGE}s old) -- reclaiming"
|
|
107
|
+
rm -r -- "${LOCK_DIR}"
|
|
108
|
+
mkdir "${LOCK_DIR}" || { log "SKIP: could not reclaim stale lock, exiting"; exit 0; }
|
|
109
|
+
else
|
|
110
|
+
log "SKIP: another instance holds the lock (${LOCK_AGE}s old), exiting"
|
|
111
|
+
exit 0
|
|
112
|
+
fi
|
|
113
|
+
else
|
|
114
|
+
log "SKIP: mkdir lock failed for an unexpected reason, exiting"
|
|
115
|
+
exit 0
|
|
116
|
+
fi
|
|
117
|
+
fi
|
|
118
|
+
|
|
119
|
+
log "START"
|
|
120
|
+
|
|
121
|
+
# --- scratch dir for pane captures ----------------------------------------
|
|
122
|
+
WORKDIR="$(mktemp -d "${TMPDIR:-/tmp}/${AGENT_NAME}-pane-watch.XXXXXX")"
|
|
123
|
+
# Single combined trap: both the lock dir AND the scratch dir must be
|
|
124
|
+
# cleaned up on exit, regardless of which path (success/error/SKIP) got
|
|
125
|
+
# here -- a second `trap ... EXIT` would silently REPLACE this one, not
|
|
126
|
+
# add to it, so both cleanups live in the same trap.
|
|
127
|
+
trap 'rm -r -- "${LOCK_DIR}" "${WORKDIR}"' EXIT
|
|
128
|
+
|
|
129
|
+
# --- load roster from watch_params.json -----------------------------------
|
|
130
|
+
if [ ! -f "${PARAMS_FILE}" ]; then
|
|
131
|
+
log "ERROR: watch_params.json not found at ${PARAMS_FILE}, exiting"
|
|
132
|
+
exit 1
|
|
133
|
+
fi
|
|
134
|
+
|
|
135
|
+
# `mapfile`/`readarray` are bash-4+ only; `thrum monitor`'s scheduled-run
|
|
136
|
+
# process resolves `bash` to macOS's system /bin/bash (3.2), not whatever
|
|
137
|
+
# Homebrew bash an interactive shell's PATH might pick up -- so array
|
|
138
|
+
# population must stick to bash-3.2-portable constructs throughout this
|
|
139
|
+
# script (no mapfile/readarray, no `local -n`, no bash-4 associative
|
|
140
|
+
# arrays -- parallel indexed arrays instead). Test with `/bin/bash
|
|
141
|
+
# thrum-watch-pane-capture.sh`, not just `bash ...` or `./...`, or this
|
|
142
|
+
# class of bug only surfaces on the live scheduled run (thrum-dnatz).
|
|
143
|
+
ROSTER=()
|
|
144
|
+
while IFS= read -r line; do
|
|
145
|
+
[ -n "${line}" ] && ROSTER+=("${line}")
|
|
146
|
+
done < <(jq -r '.roster[]?' "${PARAMS_FILE}")
|
|
147
|
+
|
|
148
|
+
if [ "${#ROSTER[@]}" -eq 0 ]; then
|
|
149
|
+
log "NOOP: roster is empty in watch_params.json"
|
|
150
|
+
exit 0
|
|
151
|
+
fi
|
|
152
|
+
log "roster: ${ROSTER[*]}"
|
|
153
|
+
|
|
154
|
+
# NOTE: this script does NOT define its own is_proxy_failure() -- the
|
|
155
|
+
# local-vs-SSH decision happens ENTIRELY inside thrum-capture-fallback.sh
|
|
156
|
+
# (see capture_agent below, which always routes through that wrapper), so
|
|
157
|
+
# duplicating the signature here would be genuinely dead code, not a
|
|
158
|
+
# sync-source. If that division of responsibility ever changes, add the
|
|
159
|
+
# check back here too and see the "MUST STAY IN SYNC" note in
|
|
160
|
+
# thrum-capture-fallback.sh's own is_proxy_failure().
|
|
161
|
+
|
|
162
|
+
# --- per-agent SSH-fallback target resolution -----------------------------
|
|
163
|
+
# Resolved ONCE per roster agent, for the whole run (never guess a path;
|
|
164
|
+
# fail CLOSED on any gap -- an agent with an unresolved target simply gets
|
|
165
|
+
# no SSH fallback, it is NOT skipped from capture entirely: local capture
|
|
166
|
+
# is always attempted regardless of resolution outcome).
|
|
167
|
+
#
|
|
168
|
+
# thrum state get agent_pool:<agent> -> .entry.value.box (a hostname)
|
|
169
|
+
# thrum state list --kind topology -> enumerate topology SCOPES only
|
|
170
|
+
# thrum state show topology:<scope> -> per-scope .entry.value, matched
|
|
171
|
+
# by .value.hostname == the box
|
|
172
|
+
# hostname above
|
|
173
|
+
#
|
|
174
|
+
# NOTE ON THE TWO-STEP TOPOLOGY LOOKUP: `thrum state list --kind topology
|
|
175
|
+
# --json`'s CLI output does NOT include each entry's `.value` (verified
|
|
176
|
+
# against cmd/thrum/state.go's stateListEntry struct, which has no Value
|
|
177
|
+
# field, and confirmed live -- `state list --json` returns only
|
|
178
|
+
# kind/scope/method/as_of/established_by). Only `state get`/`state show`
|
|
179
|
+
# emit the full entry including `.value`. So topology resolution is
|
|
180
|
+
# necessarily two calls: `list` to enumerate scopes, `show` per scope for
|
|
181
|
+
# the value -- a single-call `.value.hostname` scan is not achievable
|
|
182
|
+
# against the current CLI, regardless of how the field/scope names read.
|
|
183
|
+
is_placeholder() {
|
|
184
|
+
case "$1" in
|
|
185
|
+
""|PENDING-LEON|PENDING|TODO|TBD) return 0 ;;
|
|
186
|
+
*) return 1 ;;
|
|
187
|
+
esac
|
|
188
|
+
}
|
|
189
|
+
|
|
190
|
+
TOPO_HOSTNAME=()
|
|
191
|
+
TOPO_SSH_TARGET=()
|
|
192
|
+
TOPO_SSH_USER=()
|
|
193
|
+
TOPO_REPO_PATH=()
|
|
194
|
+
|
|
195
|
+
build_topology_table() {
|
|
196
|
+
local scopes_json scope show_json hostname ssh_target ssh_user repo_path
|
|
197
|
+
if ! scopes_json="$(thrum state list --kind topology --json 2>>"${LOG_FILE}")"; then
|
|
198
|
+
log "WARN: thrum state list --kind topology failed -- ALL per-agent ssh-fallback resolution will fail closed this run"
|
|
199
|
+
return 0
|
|
200
|
+
fi
|
|
201
|
+
while IFS= read -r scope; do
|
|
202
|
+
[ -n "${scope}" ] || continue
|
|
203
|
+
if ! show_json="$(thrum state show "topology:${scope}" --json 2>>"${LOG_FILE}")"; then
|
|
204
|
+
log "WARN: thrum state show topology:${scope} failed, skipping this topology row"
|
|
205
|
+
continue
|
|
206
|
+
fi
|
|
207
|
+
hostname="$(printf '%s' "${show_json}" | jq -r '.entry.value.hostname // empty')"
|
|
208
|
+
[ -n "${hostname}" ] || continue
|
|
209
|
+
ssh_target="$(printf '%s' "${show_json}" | jq -r '.entry.value.ssh.target // empty')"
|
|
210
|
+
ssh_user="$(printf '%s' "${show_json}" | jq -r '.entry.value.ssh.user // empty')"
|
|
211
|
+
repo_path="$(printf '%s' "${show_json}" | jq -r '.entry.value.repo_path // empty')"
|
|
212
|
+
TOPO_HOSTNAME+=("${hostname}")
|
|
213
|
+
TOPO_SSH_TARGET+=("${ssh_target}")
|
|
214
|
+
TOPO_SSH_USER+=("${ssh_user}")
|
|
215
|
+
TOPO_REPO_PATH+=("${repo_path}")
|
|
216
|
+
done < <(printf '%s' "${scopes_json}" | jq -r '.entries[]?.scope')
|
|
217
|
+
}
|
|
218
|
+
|
|
219
|
+
# Parallel arrays, indexed 1:1 with ROSTER -- bash 3.2 has no associative
|
|
220
|
+
# arrays, so "agent -> resolved target" is a lookup-by-matching-ROSTER-
|
|
221
|
+
# index, not a map.
|
|
222
|
+
RESOLVED_SSH_TARGET=()
|
|
223
|
+
RESOLVED_SSH_USER=()
|
|
224
|
+
RESOLVED_SSH_REPO=()
|
|
225
|
+
RESOLVED_COUNT=0
|
|
226
|
+
|
|
227
|
+
resolve_roster_targets() {
|
|
228
|
+
local agent pool_json box i matched row_found
|
|
229
|
+
for agent in "${ROSTER[@]}"; do
|
|
230
|
+
matched=0
|
|
231
|
+
row_found=0
|
|
232
|
+
if pool_json="$(thrum state get "agent_pool:${agent}" --json 2>>"${LOG_FILE}")"; then
|
|
233
|
+
box="$(printf '%s' "${pool_json}" | jq -r '.entry.value.box // empty')"
|
|
234
|
+
if [ -n "${box}" ]; then
|
|
235
|
+
for i in "${!TOPO_HOSTNAME[@]}"; do
|
|
236
|
+
if [ "${TOPO_HOSTNAME[$i]}" = "${box}" ]; then
|
|
237
|
+
row_found=1
|
|
238
|
+
if is_placeholder "${TOPO_REPO_PATH[$i]}" || is_placeholder "${TOPO_SSH_TARGET[$i]}"; then
|
|
239
|
+
log "agent=${agent} WARN topology row for host=${box} has a placeholder/missing ssh.target or repo_path -- fail-closed, no ssh fallback for this agent"
|
|
240
|
+
else
|
|
241
|
+
RESOLVED_SSH_TARGET+=("${TOPO_SSH_TARGET[$i]}")
|
|
242
|
+
RESOLVED_SSH_USER+=("${TOPO_SSH_USER[$i]}")
|
|
243
|
+
RESOLVED_SSH_REPO+=("${TOPO_REPO_PATH[$i]}")
|
|
244
|
+
RESOLVED_COUNT=$((RESOLVED_COUNT + 1))
|
|
245
|
+
matched=1
|
|
246
|
+
fi
|
|
247
|
+
break
|
|
248
|
+
fi
|
|
249
|
+
done
|
|
250
|
+
if [ "${row_found}" -eq 0 ]; then
|
|
251
|
+
log "agent=${agent} WARN no topology row matches host=${box} -- fail-closed, no ssh fallback for this agent"
|
|
252
|
+
fi
|
|
253
|
+
else
|
|
254
|
+
log "agent=${agent} WARN agent_pool entry has no .value.box -- fail-closed, no ssh fallback for this agent"
|
|
255
|
+
fi
|
|
256
|
+
else
|
|
257
|
+
log "agent=${agent} WARN thrum state get agent_pool:${agent} failed -- fail-closed, no ssh fallback for this agent"
|
|
258
|
+
fi
|
|
259
|
+
if [ "${matched}" -eq 0 ]; then
|
|
260
|
+
RESOLVED_SSH_TARGET+=("NONE")
|
|
261
|
+
RESOLVED_SSH_USER+=("")
|
|
262
|
+
RESOLVED_SSH_REPO+=("")
|
|
263
|
+
fi
|
|
264
|
+
done
|
|
265
|
+
}
|
|
266
|
+
|
|
267
|
+
build_topology_table
|
|
268
|
+
resolve_roster_targets
|
|
269
|
+
|
|
270
|
+
# Fork default (coordinator-resolved, thrum-3mhrt plan): if EVERY roster
|
|
271
|
+
# agent failed resolution, that is itself worth surfacing as a real
|
|
272
|
+
# problem (likely a fleet-wide topology/agent_pool data gap), not a clean
|
|
273
|
+
# tick -- flip the summary line's marker so `--match` treats it as a
|
|
274
|
+
# monitor-level FAIL rather than routine per-agent degradation. This is
|
|
275
|
+
# ORTHOGONAL to whether local capture actually needs the fallback this
|
|
276
|
+
# run -- it fires on a total resolution outage regardless.
|
|
277
|
+
RESOLUTION_ALL_FAILED=0
|
|
278
|
+
if [ "${RESOLVED_COUNT}" -eq 0 ]; then
|
|
279
|
+
RESOLUTION_ALL_FAILED=1
|
|
280
|
+
log "WARN: ALL ${#ROSTER[@]} roster agent(s) failed ssh-fallback target resolution this run"
|
|
281
|
+
fi
|
|
282
|
+
|
|
283
|
+
# --- capture helper: local first, ALWAYS; SSH fallback only inside the ---
|
|
284
|
+
# --- wrapper, only on the known proxy-failure signature -------------------
|
|
285
|
+
# Always routes through thrum-capture-fallback.sh (never calls `thrum tmux
|
|
286
|
+
# capture` directly itself) so the local/SSH/ssh-add retry logic lives in
|
|
287
|
+
# exactly one place. "Local or cross-daemon" is decided EMPIRICALLY by the
|
|
288
|
+
# wrapper's own local attempt succeeding/failing -- never by an up-front
|
|
289
|
+
# hostname-equality heuristic here.
|
|
290
|
+
capture_agent() {
|
|
291
|
+
local agent="$1" out_file="$2" idx="$3"
|
|
292
|
+
bash "${FALLBACK_SCRIPT}" capture "${agent}" \
|
|
293
|
+
"${RESOLVED_SSH_TARGET[$idx]}" "${RESOLVED_SSH_USER[$idx]}" "${RESOLVED_SSH_REPO[$idx]}" \
|
|
294
|
+
>"${out_file}" 2>&1
|
|
295
|
+
}
|
|
296
|
+
|
|
297
|
+
FLAGGED=0
|
|
298
|
+
{
|
|
299
|
+
echo "# ${AGENT_NAME} pane-watch report"
|
|
300
|
+
echo "# generated: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
|
301
|
+
echo "# roster: ${ROSTER[*]}"
|
|
302
|
+
echo "# ssh-fallback resolved: ${RESOLVED_COUNT}/${#ROSTER[@]} roster agent(s)"
|
|
303
|
+
echo
|
|
304
|
+
} >"${REPORT_FILE}"
|
|
305
|
+
|
|
306
|
+
ROSTER_IDX=0
|
|
307
|
+
for agent in "${ROSTER[@]}"; do
|
|
308
|
+
CAP1="${WORKDIR}/${agent}.cap1"
|
|
309
|
+
CAP2="${WORKDIR}/${agent}.cap2"
|
|
310
|
+
|
|
311
|
+
if ! capture_agent "${agent}" "${CAP1}" "${ROSTER_IDX}"; then
|
|
312
|
+
log "agent=${agent} WARN capture failed (exit code), recording report entry anyway"
|
|
313
|
+
fi
|
|
314
|
+
|
|
315
|
+
if [ ! -s "${CAP1}" ]; then
|
|
316
|
+
log "agent=${agent} WARN empty capture, skipping detect"
|
|
317
|
+
{
|
|
318
|
+
echo "===== @${agent} ====="
|
|
319
|
+
echo "status: CAPTURE FAILED (empty output -- check local + ssh-fallback paths; ssh-fallback target: ${RESOLVED_SSH_TARGET[$ROSTER_IDX]})"
|
|
320
|
+
echo
|
|
321
|
+
} >>"${REPORT_FILE}"
|
|
322
|
+
ROSTER_IDX=$((ROSTER_IDX + 1))
|
|
323
|
+
continue
|
|
324
|
+
fi
|
|
325
|
+
|
|
326
|
+
STATUS="clean"
|
|
327
|
+
MATCH_NAME=""
|
|
328
|
+
|
|
329
|
+
# thrum detect exit-code contract (cmd/thrum/detect.go):
|
|
330
|
+
# exit 0 + matched entry name on stdout -> match
|
|
331
|
+
# exit 1, silent -> no match
|
|
332
|
+
# `if VAR=$(...); then` is exempt from `set -e` semantics for this
|
|
333
|
+
# purpose (we run under set -u only, not -e), so a "no match" does not
|
|
334
|
+
# abort the loop -- this IS the intended branch.
|
|
335
|
+
if MATCH_NAME=$(thrum detect --category permission --pane-file "${CAP1}" --runtime claude 2>>"${LOG_FILE}"); then
|
|
336
|
+
# Stability check: a pane holding unsubmitted text or a live human/
|
|
337
|
+
# agent mid-composition is not idle. Require the pane to be unchanged
|
|
338
|
+
# across two captures, two seconds apart, before flagging.
|
|
339
|
+
sleep 2
|
|
340
|
+
if capture_agent "${agent}" "${CAP2}" "${ROSTER_IDX}" && [ -s "${CAP2}" ]; then
|
|
341
|
+
if diff -q "${CAP1}" "${CAP2}" >/dev/null; then
|
|
342
|
+
FLAGGED=$((FLAGGED + 1))
|
|
343
|
+
STATUS="FLAGGED: stable permission prompt (pattern=${MATCH_NAME})"
|
|
344
|
+
log "agent=${agent} pattern=${MATCH_NAME} STABLE permission prompt detected"
|
|
345
|
+
else
|
|
346
|
+
STATUS="pattern=${MATCH_NAME} detected but UNSTABLE across two captures -- pane is live, not flagging"
|
|
347
|
+
log "agent=${agent} pattern=${MATCH_NAME} UNSTABLE across two captures -- leaving alone"
|
|
348
|
+
fi
|
|
349
|
+
else
|
|
350
|
+
STATUS="pattern=${MATCH_NAME} detected, re-capture failed -- not flagging without stability confirmation"
|
|
351
|
+
log "agent=${agent} WARN re-capture failed during stability check, not flagging"
|
|
352
|
+
fi
|
|
353
|
+
else
|
|
354
|
+
log "agent=${agent} no permission prompt detected"
|
|
355
|
+
fi
|
|
356
|
+
|
|
357
|
+
{
|
|
358
|
+
echo "===== @${agent} ====="
|
|
359
|
+
echo "status: ${STATUS}"
|
|
360
|
+
echo "--- pane tail (last 40 lines) ---"
|
|
361
|
+
tail -n 40 "${CAP1}"
|
|
362
|
+
echo "--- end pane ---"
|
|
363
|
+
echo
|
|
364
|
+
} >>"${REPORT_FILE}"
|
|
365
|
+
|
|
366
|
+
ROSTER_IDX=$((ROSTER_IDX + 1))
|
|
367
|
+
done
|
|
368
|
+
|
|
369
|
+
log "DONE flagged=${FLAGGED} resolved=${RESOLVED_COUNT}/${#ROSTER[@]} report=${REPORT_FILE}"
|
|
370
|
+
|
|
371
|
+
# One matchable line for `thrum monitor --match` to catch and deliver.
|
|
372
|
+
# Two shapes: the routine tick, and the resolution-outage marker from the
|
|
373
|
+
# fork default above -- SKILL.md's registration example's --match regex
|
|
374
|
+
# must cover BOTH ("^watch-tick: (cycle done|RESOLUTION-FAIL)").
|
|
375
|
+
if [ "${RESOLUTION_ALL_FAILED}" -eq 1 ]; then
|
|
376
|
+
echo "watch-tick: RESOLUTION-FAIL $(date -u +%Y-%m-%dT%H:%M:%SZ) -- ${#ROSTER[@]} agent(s), 0 ssh-fallback targets resolved -- report: ${REPORT_FILE}"
|
|
377
|
+
else
|
|
378
|
+
echo "watch-tick: cycle done $(date -u +%Y-%m-%dT%H:%M:%SZ) -- ${#ROSTER[@]} agent(s), ${FLAGGED} flagged, ${RESOLVED_COUNT} ssh-fallback-ready -- report: ${REPORT_FILE}"
|
|
379
|
+
fi
|
|
@@ -85,8 +85,6 @@ needed, defer to `thrum prime`.
|
|
|
85
85
|
|
|
86
86
|
Once both steps complete, proceed with the work the wake-primer asked for.
|
|
87
87
|
|
|
88
|
-
The
|
|
89
|
-
|
|
90
|
-
`.thrum/agents/<agent_id>/state.md` via `thrum agent state update` /
|
|
91
|
-
`recover`) were retired — superseded by `thrum state` (personal_state kind). The
|
|
88
|
+
The former agent-state update and recovery skills were retired. Personal
|
|
89
|
+
continuity now uses `thrum state` (personal_state kind). The
|
|
92
90
|
`last_seen_skills.txt` baseline write in Step 2 above is unaffected.
|