@mu-crew/mu 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +611 -0
- package/README.md +328 -0
- package/dist/cli.js +22146 -0
- package/dist/cli.js.map +1 -0
- package/dist/index.d.ts +4853 -0
- package/dist/index.js +9740 -0
- package/dist/index.js.map +1 -0
- package/docs/.m-reflow-5f7exa_b.md +509 -0
- package/docs/.m-reflow-5v_034rw.md +545 -0
- package/docs/.m-reflow-7yaw8ubc.md +513 -0
- package/docs/ARCHITECTURE.md +997 -0
- package/docs/HANDOVER.md +486 -0
- package/docs/ROADMAP.md +277 -0
- package/docs/USAGE_GUIDE.md +2615 -0
- package/docs/VISION.md +487 -0
- package/docs/VOCABULARY.md +481 -0
- package/docs/img/tui-dashboard.png +0 -0
- package/package.json +87 -0
- package/skills/mu/REMOTE_WORKERS.md +604 -0
- package/skills/mu/SKILL.md +369 -0
|
@@ -0,0 +1,604 @@
|
|
|
1
|
+
# Remote workers
|
|
2
|
+
|
|
3
|
+
Running mu agents on another machine. A recipe, not a feature: mu has
|
|
4
|
+
no remote backend, no ssh code, and no host registry, and it needs
|
|
5
|
+
none.
|
|
6
|
+
|
|
7
|
+
Read this before spawning your first remote agent. The recipe is short;
|
|
8
|
+
the traps below it are the part that costs time.
|
|
9
|
+
|
|
10
|
+
**If you read one thing:** on a session-capped host, never leave an
|
|
11
|
+
attach pane open — it holds the only ssh channel and silently breaks
|
|
12
|
+
`git fetch`, `murmur collect` and every other ssh. Attach to look, then
|
|
13
|
+
`mu agent close`. See § Never leave an attach pane open.
|
|
14
|
+
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
## The model
|
|
18
|
+
|
|
19
|
+
**The pane is local, the process is remote.** `mu agent spawn
|
|
20
|
+
--command 'ssh <host> -t "..."'` starts an ordinary tmux pane whose
|
|
21
|
+
foreground process happens to be ssh. Everything mu does is
|
|
22
|
+
pane-shaped, so `mu agent send`, `mu agent read`, status detection and
|
|
23
|
+
the reaper all work unchanged, across the network, with no mu changes.
|
|
24
|
+
|
|
25
|
+
**One orchestrator DB; panes may be remote.** All state stays on your
|
|
26
|
+
machine. Do not run a second mu on the host to "coordinate": ownership
|
|
27
|
+
is machine-local by construction — `tasks.owner_id` is an FK into the
|
|
28
|
+
`agents` table, which never syncs — so a remote mu could not claim
|
|
29
|
+
tasks in your DAG. Two half-views, no benefit.
|
|
30
|
+
|
|
31
|
+
**murmur sees the REMOTE pane** — the opposite of the line above. Its
|
|
32
|
+
extension runs inside the agent's process, so it claims the pane on the
|
|
33
|
+
HOST; mu's local pane holds an ssh client and reports nothing. Two
|
|
34
|
+
addresses, one worker.
|
|
35
|
+
|
|
36
|
+
murmur ties them back together by looking for the AGENT NAME in your
|
|
37
|
+
local pane's command line. When it matches, the remote row shows
|
|
38
|
+
`attached here %N` and enter focuses your existing pane instead of
|
|
39
|
+
opening a second connection — which matters on a capped host, where the
|
|
40
|
+
second one fails.
|
|
41
|
+
|
|
42
|
+
Both recipes below work, for the same reason: the agent name is in the
|
|
43
|
+
argv either way. A direct spawn carries it in the remote command; the
|
|
44
|
+
detached shape carries it in the session name, because you named the
|
|
45
|
+
session after the agent. **Keep doing that** — `-s mu-<agent>` is what
|
|
46
|
+
makes the attachment findable, and a session named anything else costs
|
|
47
|
+
you the hint.
|
|
48
|
+
|
|
49
|
+
Do not bother putting `MU_AGENT_NAME=` in front of the attach command
|
|
50
|
+
to help it along. An environment prefix is consumed by your shell and
|
|
51
|
+
never reaches the process arguments, which is all `ps` reports on macOS.
|
|
52
|
+
|
|
53
|
+
What mu does NOT know about a remote agent: the workspace. There is no
|
|
54
|
+
`vcs_workspaces` row, so no `mu workspace list / refresh / commits /
|
|
55
|
+
free`, no `behind` column, no staleness warning on claim, and no
|
|
56
|
+
auto-free on close. You own that bookkeeping.
|
|
57
|
+
|
|
58
|
+
---
|
|
59
|
+
|
|
60
|
+
## Never leave an attach pane open
|
|
61
|
+
|
|
62
|
+
On a host that caps sessions per connection, **your attach pane holds
|
|
63
|
+
the only ssh channel**. While it is open, `git fetch`, `git push`,
|
|
64
|
+
`murmur collect`, `rsync`, and plain ssh can fail with the misleading
|
|
65
|
+
`Permission denied (keyboard-interactive)` error.
|
|
66
|
+
|
|
67
|
+
**Attach to look, then close immediately.** `mu agent close <name>`
|
|
68
|
+
detaches without stopping a detached-tmux agent. Poll with `murmur
|
|
69
|
+
collect` + `murmur status`; use mule for orchestrator commands because
|
|
70
|
+
its separate ControlPath cannot contend with the attach pane. See
|
|
71
|
+
§ When the host limits concurrent sessions for diagnosis and the
|
|
72
|
+
remote-agent setup.
|
|
73
|
+
|
|
74
|
+
When a collect fails, current murmur names an attachment that holds the
|
|
75
|
+
channel. Otherwise inspect it directly:
|
|
76
|
+
|
|
77
|
+
```bash
|
|
78
|
+
ps -o pid=,command= -ax | grep "[s]sh <host>"
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
### Better: attach with the peer's jump command
|
|
82
|
+
|
|
83
|
+
If murmur knows the host, ask it how to reach the host interactively
|
|
84
|
+
rather than hardcoding ssh:
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
JUMP=$(murmur peer list --json | jq -r '.[]|select(.name=="dev").jump_command')
|
|
88
|
+
mu agent spawn worker-1 -w big --command "${JUMP//\{pane\}/mu-worker-1}"
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
That command may name a transport taking **no** ssh session at all,
|
|
92
|
+
which removes the contention rather than managing it.
|
|
93
|
+
|
|
94
|
+
**A spawn that lands on an auth prompt looks exactly like a healthy
|
|
95
|
+
one.** Measured: with the slot busy, `ssh dev -t "tmux attach"` fell
|
|
96
|
+
back to a fresh connection and stopped at `Enter a passcode:`. The spawn
|
|
97
|
+
succeeded, the row appeared, `mu agent list` showed `needs_input` — and
|
|
98
|
+
`mu agent send` pasted the whole prompt into the passcode field and
|
|
99
|
+
reported success. The agent received nothing.
|
|
100
|
+
|
|
101
|
+
So confirm a remote agent actually got the work before trusting it
|
|
102
|
+
(`mu agent read <name> -n 20`, or its context percentage), and never
|
|
103
|
+
send anything sensitive to an unconfirmed pane.
|
|
104
|
+
|
|
105
|
+
---
|
|
106
|
+
|
|
107
|
+
## The recipe
|
|
108
|
+
|
|
109
|
+
```bash
|
|
110
|
+
# 1. WORKSPACE — you create it; --workspace does NOT work remotely
|
|
111
|
+
ssh dev 'git -C ~/repo worktree add ~/ws/worker-1'
|
|
112
|
+
|
|
113
|
+
# 2. RECORD — persist location and per-worker baseline together
|
|
114
|
+
mu task note t1 -w big "REMOTE: dev:~/ws/worker-1
|
|
115
|
+
REMOTE_BASE: worker-1:$(ssh dev 'cd ~/ws/worker-1 && git rev-parse HEAD')"
|
|
116
|
+
|
|
117
|
+
# 3. SPAWN — env vars go INSIDE the command (tmux -e stops at the hop)
|
|
118
|
+
mu agent spawn worker-1 -w big --command \
|
|
119
|
+
'ssh dev -t "cd ~/ws/worker-1 && MU_MANAGED_AGENT=1 MU_AGENT_NAME=worker-1 MU_WORKSTREAM=big pi --approve"'
|
|
120
|
+
|
|
121
|
+
# 4. CLAIM + SEND — identical to a local agent
|
|
122
|
+
mu task claim t1 -w big --for worker-1 --evidence 'remote on dev'
|
|
123
|
+
mu agent send worker-1 -w big '...'
|
|
124
|
+
|
|
125
|
+
# 5. WAIT — run the claim's one-shot Next: command once per turn
|
|
126
|
+
# Set a deadline; at expiry read the pane, then release or re-dispatch.
|
|
127
|
+
job=$(mule run --host dev --max-secs 30 \
|
|
128
|
+
'cd ~/ws/worker-1 && git rev-parse HEAD 2>/dev/null || echo unreadable')
|
|
129
|
+
mule wait "$job" >/dev/null && sha=$(mule tail "$job")
|
|
130
|
+
case "$sha" in (*[!0-9a-fA-F]*|'') :;; (*) [ "${#sha}" -eq 40 ] && \
|
|
131
|
+
{ [ "$sha" = 24481fe24481fe24481fe24481fe24481fe2448 ] || \
|
|
132
|
+
mu task close t1 -w big --evidence "worker-1 committed $sha"; };; esac
|
|
133
|
+
|
|
134
|
+
# 6. COLLECT — fetch straight from the remote worktree
|
|
135
|
+
git fetch "ssh://dev/~/ws/worker-1" HEAD && git cherry-pick FETCH_HEAD
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
Local and remote agents mix freely in one workstream. The DAG, tracks,
|
|
139
|
+
`mu state` and `mu task wait` do not care where a pane's process runs
|
|
140
|
+
— task status is a row in YOUR database, written by you, so a wait on
|
|
141
|
+
it is exact.
|
|
142
|
+
|
|
143
|
+
**Agent STATUS is different, and this is the one place "nothing
|
|
144
|
+
changes" is false.** `busy` / `needs_input` / `idle` come from reading
|
|
145
|
+
the local pane's scrollback, which for a remote worker is a nested tmux
|
|
146
|
+
rendered over ssh. What you get is redraw lag and quiet periods that
|
|
147
|
+
look like idleness, so anything derived from status is unreliable here:
|
|
148
|
+
|
|
149
|
+
| waiting on | remote? |
|
|
150
|
+
| --- | --- |
|
|
151
|
+
| `mu task wait` (task status) | exact — a DB poll, once something closes the task |
|
|
152
|
+
| the reaper | fires, but see below |
|
|
153
|
+
| stall detection | unreliable |
|
|
154
|
+
| `mu agent wait --first` | same, it is status-based |
|
|
155
|
+
|
|
156
|
+
Measured both directions in one session: a stall fired at 300s against
|
|
157
|
+
a worker that was visibly mid-turn, and `mu agent list` showed
|
|
158
|
+
`needs_input` for another that was working. Give remote waits a generous
|
|
159
|
+
`--timeout`; see `mu task wait --help` for stall controls.
|
|
160
|
+
|
|
161
|
+
The reaper is right for a DIRECT spawn — the connection dying really
|
|
162
|
+
does kill that agent — and wrong for a detached-tmux one, where the
|
|
163
|
+
agent outlives the ssh but mu reaps the task anyway. See § A dropped
|
|
164
|
+
connection reaps the task but NOT the commit.
|
|
165
|
+
|
|
166
|
+
**If you need trustworthy remote status, ask murmur, not mu.** Its
|
|
167
|
+
extension runs inside pi ON THE HOST and pushes state, so nothing is
|
|
168
|
+
scraped and no ssh hop distorts it:
|
|
169
|
+
|
|
170
|
+
```bash
|
|
171
|
+
murmur collect && murmur status --json # activity is host-reported
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
That is the division worth remembering: mu owns the work, murmur owns
|
|
175
|
+
what the agent is doing.
|
|
176
|
+
|
|
177
|
+
### On step 2 — the note is load-bearing
|
|
178
|
+
|
|
179
|
+
The task note survives connection loss and context compaction. Keep the
|
|
180
|
+
location as `REMOTE: <host>:<path>` and each baseline as
|
|
181
|
+
`REMOTE_BASE: <agent>:<sha>`. `mu task claim --for <agent>` uses both to
|
|
182
|
+
print a complete one-shot poll-and-close command. For a crew, record one
|
|
183
|
+
baseline per worker and use one bounded mule job for every path:
|
|
184
|
+
|
|
185
|
+
```bash
|
|
186
|
+
mule run --max-secs 30 'for d in ~/ws/*/; do printf "%s %s\\n" \
|
|
187
|
+
"$(basename $d)" "$(cd $d && git rev-parse HEAD 2>/dev/null || echo unreadable)"; done'
|
|
188
|
+
mule wait <job>; mule tail <job>
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
Each line names the worker and sha. Accept only 40 hex characters;
|
|
192
|
+
`unreadable` or an empty field means retry next turn, never progress.
|
|
193
|
+
|
|
194
|
+
### On step 5 — wait on commits
|
|
195
|
+
|
|
196
|
+
**Poll once per turn, never sleep in a tool call.** A `while true` loop
|
|
197
|
+
with `sleep` holds the call; aborting it can leave remote work and a
|
|
198
|
+
capped ssh channel running. Set a wall-clock deadline before dispatch.
|
|
199
|
+
At expiry, read the pane, then release or re-dispatch instead of extending
|
|
200
|
+
the wait silently.
|
|
201
|
+
|
|
202
|
+
On a session-capped host, run the claim's `Next:` command through mule
|
|
203
|
+
between other work. Bare ssh polls can be refused with an empty sha,
|
|
204
|
+
which looks like progress. The command captures the new sha and closes
|
|
205
|
+
the task with that sha as evidence. The commit tells you;
|
|
206
|
+
the DAG stays IN_PROGRESS until the orchestrator closes it. This feeds
|
|
207
|
+
`mu task wait` so blocked tasks advance; it does not replace it.
|
|
208
|
+
|
|
209
|
+
### Recovering the location
|
|
210
|
+
|
|
211
|
+
`mu state` lists exact `REMOTE:` lines as its remote-worker inventory,
|
|
212
|
+
and the recovery command is mechanical:
|
|
213
|
+
|
|
214
|
+
```bash
|
|
215
|
+
git fetch "ssh://<host>/<path>" HEAD && git cherry-pick FETCH_HEAD
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
### On step 6 — fetching from a worktree
|
|
219
|
+
|
|
220
|
+
`git fetch "ssh://<host>/<path>" HEAD` reads a remote worktree
|
|
221
|
+
**directly**. No shared remote, no push, no bare repo in between. This
|
|
222
|
+
is the part people expect to be hard and it is not.
|
|
223
|
+
|
|
224
|
+
- It exits **128** on failure, so `&&` chaining is safe.
|
|
225
|
+
- **Quote the URL.** `~` is legal in an `ssh://` URL (git's own docs
|
|
226
|
+
list `ssh://host/~user/path`), but unquoted it is expanded by your
|
|
227
|
+
LOCAL shell into your laptop's home.
|
|
228
|
+
- Two workers editing one file still conflict on cherry-pick, exactly
|
|
229
|
+
as locally. Bucket work by file cluster, not by machine.
|
|
230
|
+
|
|
231
|
+
### Run the merged suite where the workers are
|
|
232
|
+
|
|
233
|
+
The reason to send work to a big machine is to stop paying for it on a
|
|
234
|
+
small one. Cherry-picking remote work and then running the whole suite
|
|
235
|
+
locally gives that back — and it is the default thing to do, so say the
|
|
236
|
+
other thing explicitly.
|
|
237
|
+
|
|
238
|
+
**Do not re-run what the worker ran.** Its green on its own tree is the
|
|
239
|
+
evidence you asked for. What is unverified is the MERGE: the worker
|
|
240
|
+
forked from a base that has since moved, so the only new information is
|
|
241
|
+
in the combination.
|
|
242
|
+
|
|
243
|
+
So cherry-pick locally (it is a few seconds of IO), then push the merged
|
|
244
|
+
head to the same host and run the gate there:
|
|
245
|
+
|
|
246
|
+
```bash
|
|
247
|
+
git cherry-pick <sha>
|
|
248
|
+
git push -q "ssh://dev/~/hacking/<repo>.git" HEAD:refs/heads/main
|
|
249
|
+
ssh dev 'cd ~/hacking/<checkout> && git fetch -q origin \
|
|
250
|
+
&& git reset -q --hard origin/main && npm run check'
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
**On a session-capped host, run that gate through mule instead.** The
|
|
254
|
+
last line holds the channel for the whole suite — minutes — which is
|
|
255
|
+
precisely when your other tooling starts failing with a credentials
|
|
256
|
+
error that has nothing to do with credentials:
|
|
257
|
+
|
|
258
|
+
```bash
|
|
259
|
+
mule run --cwd ~/hacking/<checkout> --wait \
|
|
260
|
+
'git fetch -q origin && git reset -q --hard origin/main && npm run check'
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
Same work, dispatched detached on mule's own connection, and `--wait`
|
|
264
|
+
exits with the suite's own code. See § When the host limits concurrent
|
|
265
|
+
sessions.
|
|
266
|
+
|
|
267
|
+
That host already has a checkout and warm dependencies — the same ones
|
|
268
|
+
the worker used — so the marginal cost is near zero, while the same run
|
|
269
|
+
on a laptop is minutes of CPU per integration.
|
|
270
|
+
|
|
271
|
+
**Keep two things local.** A **platform-sensitive** subset, because a
|
|
272
|
+
remote green does not prove a local green when the bug is
|
|
273
|
+
platform-shaped: macOS `ps` returns argv where Linux `ps -e` appends the
|
|
274
|
+
environment, and a real bug lived in exactly that gap. And the **final**
|
|
275
|
+
gate before the push that matters, since that one is about your tree
|
|
276
|
+
rather than the worker's.
|
|
277
|
+
|
|
278
|
+
---
|
|
279
|
+
|
|
280
|
+
## Traps
|
|
281
|
+
|
|
282
|
+
### Use the host's real CLI command
|
|
283
|
+
|
|
284
|
+
The wrapper you run locally is not what a bare `pi` gives you on the
|
|
285
|
+
host. If your local `$MU_PI_COMMAND` is `pi-meta --pi-meta-no-solo
|
|
286
|
+
--approve`, spawning plain `pi` remotely yields a live pane, a healthy
|
|
287
|
+
status, and an agent with **no models configured** — it looks fine
|
|
288
|
+
until you send it work. Check first:
|
|
289
|
+
|
|
290
|
+
```bash
|
|
291
|
+
ssh dev 'command -v pi-meta'
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
### Tell the worker that mu is absent
|
|
295
|
+
|
|
296
|
+
`mu` is usually not installed on the host, and the DB is on your
|
|
297
|
+
laptop regardless. The in-pane worker loop (`mu task claim` / `note` /
|
|
298
|
+
`close`) therefore cannot run. Say so in the prompt and have the
|
|
299
|
+
worker print its sha instead; YOU claim, note and close from the
|
|
300
|
+
orchestrator. Omit this and the worker burns a turn on `command not
|
|
301
|
+
found`, or worse, silently fails to close and your `mu task wait`
|
|
302
|
+
hangs.
|
|
303
|
+
|
|
304
|
+
### Everything is orchestrator-PULL
|
|
305
|
+
|
|
306
|
+
The host frequently cannot resolve your laptop at all — a corporate
|
|
307
|
+
devserver typically has no route back to a NAT'd machine. So: you push
|
|
308
|
+
setup out, you fetch commits back. Never write a recipe in which the
|
|
309
|
+
host reaches you, and never assume a peer can `git fetch` from you.
|
|
310
|
+
|
|
311
|
+
### A dropped connection reaps the task but NOT the commit
|
|
312
|
+
|
|
313
|
+
If the ssh client dies — network drop, laptop sleep, VPN blip — the
|
|
314
|
+
pane dies with it, and mu behaves exactly as for a dead local agent:
|
|
315
|
+
the agent row goes, and the reaper reverts the task `IN_PROGRESS →
|
|
316
|
+
OPEN` with a `[reaper]` note. That is correct and desirable.
|
|
317
|
+
|
|
318
|
+
But the worker's **commit is still on the host**, and mu has no record
|
|
319
|
+
of where. Before re-dispatching, fetch from the path in the task note
|
|
320
|
+
and look: re-running the task blind duplicates work that already
|
|
321
|
+
exists. This is the strongest argument for step 2.
|
|
322
|
+
|
|
323
|
+
### A crashed remote worker disappears rather than reporting `crashed`
|
|
324
|
+
|
|
325
|
+
murmur records `crashed` only when a **pane outlives the process** in
|
|
326
|
+
it: that is how an unreported death leaves a trace. Locally it holds —
|
|
327
|
+
a pi exits inside a shell pane and the pane stays. It does not hold
|
|
328
|
+
here, because the agent is the remote session's only process, so tmux
|
|
329
|
+
reaps the session with it and the row is simply gone at the next
|
|
330
|
+
collect.
|
|
331
|
+
|
|
332
|
+
Measured: SIGKILL a remote agent and murmur reports **zero rows and
|
|
333
|
+
zero crashed**, not a crash. Nothing distinguishes "it died" from "it
|
|
334
|
+
finished and I closed it". A second consequence of the same rule: the
|
|
335
|
+
crash path only fires for an agent killed **mid-turn**, since an idle
|
|
336
|
+
agent is already `stopped` and a stopped owner dying reads as a normal
|
|
337
|
+
finish.
|
|
338
|
+
|
|
339
|
+
So do not wait for a crash signal on a remote worker. The durable
|
|
340
|
+
traces are the ones mu already gives you:
|
|
341
|
+
|
|
342
|
+
- the reaper flipping the task back to `OPEN` with a `[reaper]` note
|
|
343
|
+
- the `REMOTE:` task note, which is where the commit is
|
|
344
|
+
|
|
345
|
+
`ssh <host> 'tmux ls'` confirms whether the session is really gone.
|
|
346
|
+
|
|
347
|
+
### Cleaning up
|
|
348
|
+
|
|
349
|
+
mu will not remove a remote worktree, because it does not know it
|
|
350
|
+
exists:
|
|
351
|
+
|
|
352
|
+
```bash
|
|
353
|
+
ssh dev 'git -C ~/repo worktree remove ~/ws/worker-1'
|
|
354
|
+
```
|
|
355
|
+
|
|
356
|
+
`mu agent kick` signals the local pane's foreground process group —
|
|
357
|
+
that is the ssh client, not the remote agent. Use `mu agent close` and
|
|
358
|
+
respawn.
|
|
359
|
+
|
|
360
|
+
---
|
|
361
|
+
|
|
362
|
+
## mu and murmur, and what you lose without it
|
|
363
|
+
|
|
364
|
+
[murmur](https://github.com/mu-crew/murmur) is optional and
|
|
365
|
+
**strictly additive**. Nothing in mu needs it: the DAG, claim/close/
|
|
366
|
+
wait, workspaces, spawn/send/read, the reaper, `mu state`, this whole
|
|
367
|
+
remote recipe and `git fetch` collection all work with murmur absent.
|
|
368
|
+
What you lose is the *view* — cross-machine agent state, the tmux
|
|
369
|
+
status pills, `prefix+a`, the attachment hint, and `murmur peer list`
|
|
370
|
+
for host reachability. Guard on it (`if it is installed`) rather than
|
|
371
|
+
assuming it.
|
|
372
|
+
|
|
373
|
+
The division: **mu owns the work, murmur owns what an agent is doing.**
|
|
374
|
+
murmur never places work.
|
|
375
|
+
|
|
376
|
+
| question | ask |
|
|
377
|
+
| --- | --- |
|
|
378
|
+
| what should happen next | mu — the DAG |
|
|
379
|
+
| who owns this task | mu — `claim` / `close` |
|
|
380
|
+
| is the task done | mu — `task wait`, a DB poll |
|
|
381
|
+
| what is this agent doing right now | murmur — pushed from inside pi |
|
|
382
|
+
| is anything blocked on me, anywhere | murmur |
|
|
383
|
+
| which host can I reach | murmur — `peer list` |
|
|
384
|
+
|
|
385
|
+
**The seam is three env vars, and it is load-bearing.** `mu agent
|
|
386
|
+
spawn` injects `MU_MANAGED_AGENT=1`, `MU_AGENT_NAME` and
|
|
387
|
+
`MU_WORKSTREAM`; pi inherits them and murmur's extension reads them.
|
|
388
|
+
That one mechanism gives you:
|
|
389
|
+
|
|
390
|
+
1. `driver=orchestrated`, so crew stays out of the human's status bar
|
|
391
|
+
unless it is blocked or crashed — true for local and remote alike
|
|
392
|
+
2. the workstream and agent name on the row, for grouping
|
|
393
|
+
3. the attachment back-reference for a remote worker
|
|
394
|
+
|
|
395
|
+
For a remote worker they go **inside** the ssh command (tmux `-e` stops
|
|
396
|
+
at the hop). Miss them and the agent reports `driver=human`: it appears
|
|
397
|
+
in your own counts as if it were yours.
|
|
398
|
+
|
|
399
|
+
## Picking a host
|
|
400
|
+
|
|
401
|
+
mu does not track hosts and should not; that is murmur's job. If it is
|
|
402
|
+
installed:
|
|
403
|
+
|
|
404
|
+
```bash
|
|
405
|
+
murmur peer list --json | jq -r '.[] | select(.error) | .name'
|
|
406
|
+
```
|
|
407
|
+
|
|
408
|
+
Read it correctly, because it is **best-effort by design**:
|
|
409
|
+
|
|
410
|
+
- `ssh` is **last-known** reachability; `error` is the **current**
|
|
411
|
+
attempt. A host can read `warm` and be failing right now.
|
|
412
|
+
- The command exits **0** either way — a fleet with sleeping machines
|
|
413
|
+
is the normal state, not a fault.
|
|
414
|
+
- So branch on `.error`, never on presence in the list.
|
|
415
|
+
|
|
416
|
+
`murmur status` and `pick` are polling paths and stay silent whatever
|
|
417
|
+
the fleet is doing. `murmur collect` is the deliberate dial: it prints
|
|
418
|
+
one line per host it could not reach, so it is the one to run for "can I
|
|
419
|
+
reach this host *right now*".
|
|
420
|
+
|
|
421
|
+
### Reaching a host is two different questions
|
|
422
|
+
|
|
423
|
+
A peer carries `target` (for a COMMAND — always ssh, used by the
|
|
424
|
+
collector) and a jump command (for a HUMAN — need not be ssh):
|
|
425
|
+
|
|
426
|
+
```bash
|
|
427
|
+
murmur peer set <name> --jump-command '<command with {pane}>'
|
|
428
|
+
```
|
|
429
|
+
|
|
430
|
+
Opaque template: murmur substitutes `{pane}` and runs the rest
|
|
431
|
+
unparsed. The default reproduces `ssh -t <target> tmux attach`, so an
|
|
432
|
+
unconfigured peer behaves as before.
|
|
433
|
+
|
|
434
|
+
On a session-capped host this is the difference between holding the one
|
|
435
|
+
slot for your whole visit and holding nothing (see the ET section).
|
|
436
|
+
Don't hardcode `et` — a site wrapper may add VPN selection and its own
|
|
437
|
+
binary resolution, which is why the value is opaque.
|
|
438
|
+
|
|
439
|
+
---
|
|
440
|
+
|
|
441
|
+
## When the host limits concurrent sessions
|
|
442
|
+
|
|
443
|
+
Rare, but it presents as a credentials bug, so learn to recognise it.
|
|
444
|
+
|
|
445
|
+
Most sshd allow 10 sessions per connection (`MaxSessions`, default 10)
|
|
446
|
+
and the recipe above is all you need. A hardened host may set
|
|
447
|
+
**`MaxSessions 1`**: then the long-lived ssh holding your AGENT consumes
|
|
448
|
+
the only channel, and every other ssh — including `git fetch` — is
|
|
449
|
+
refused with the misleading 2FA error described in § Never leave an
|
|
450
|
+
attach pane open.
|
|
451
|
+
|
|
452
|
+
**Diagnostic:** if `mu agent read` works fine while a plain `ssh
|
|
453
|
+
<host> true` fails, it is session exhaustion, not credentials.
|
|
454
|
+
|
|
455
|
+
**But do not stop there — that test is necessary, not sufficient.** The
|
|
456
|
+
same `Permission denied (keyboard-interactive)` is also what a **dead
|
|
457
|
+
ssh agent** produces: `SSH_AUTH_SOCK` points at a socket that no longer
|
|
458
|
+
exists, key auth cannot be attempted, and sshd falls back to 2FA and
|
|
459
|
+
sits on a passcode prompt you cannot see. One session cost an hour to
|
|
460
|
+
this, diagnosed as session exhaustion throughout.
|
|
461
|
+
|
|
462
|
+
Check both, in this order — the second is local and instant:
|
|
463
|
+
|
|
464
|
+
```bash
|
|
465
|
+
ssh-add -l # "Error connecting to agent" => dead agent, NOT the cap
|
|
466
|
+
ssh <host> true # fails while `mu agent read` works => the cap
|
|
467
|
+
```
|
|
468
|
+
|
|
469
|
+
And note the prompt is **invisible unless the ssh runs inside a pane you
|
|
470
|
+
can capture**. That is what finally exposed it: the refusal looks like a
|
|
471
|
+
silent failure because the question is being asked somewhere with no
|
|
472
|
+
terminal attached.
|
|
473
|
+
|
|
474
|
+
`ControlMaster no` looks like it should help and does not. It governs
|
|
475
|
+
master *creation* only; a refused channel falls back regardless.
|
|
476
|
+
Measured with `no` set: three of four concurrent calls still produced
|
|
477
|
+
the misleading 2FA error. There is no client-side fix — `MaxSessions`
|
|
478
|
+
exists only in `sshd_config`, and the client discovers the cap only by
|
|
479
|
+
being refused.
|
|
480
|
+
|
|
481
|
+
### Fix: run commands through mule
|
|
482
|
+
|
|
483
|
+
[mule](https://github.com/mu-crew/mule) opens a separate ssh
|
|
484
|
+
ControlPath and dispatches detached jobs. On a `MaxSessions 1` host,
|
|
485
|
+
five concurrent bare calls produced **1 of 5** successes; mule produced
|
|
486
|
+
**5 of 5**. A multi-minute mule job also left a concurrent plain ssh
|
|
487
|
+
working.
|
|
488
|
+
|
|
489
|
+
Route commands by failure mode, not duration. Use mule for long work
|
|
490
|
+
and whenever refusal could look like success. Eight concurrent bare
|
|
491
|
+
`rev-parse` polls returned one sha and seven empty results; mule
|
|
492
|
+
dispatched all eight. Keep worktree setup and `murmur collect` bare
|
|
493
|
+
because they fail loudly. See `mule --help` and the relevant subcommand
|
|
494
|
+
help for flags, job control, warnings, and exit codes.
|
|
495
|
+
|
|
496
|
+
**A mule job must be entirely remote.** The host has no route back to
|
|
497
|
+
your laptop, so a local-endpoint `git fetch`, `git push`, or rsync
|
|
498
|
+
cannot run inside the job. Collect from the orchestrator as described
|
|
499
|
+
in § Everything is orchestrator-PULL.
|
|
500
|
+
|
|
501
|
+
**Use `mule run --tui` for an interactive/full-screen agent when you want
|
|
502
|
+
mule's timeout, runtime, transcript, and cleanup.** Plain `mule run` pipes
|
|
503
|
+
output into the artifact, so attaching finds the pane but no live TUI.
|
|
504
|
+
mule prints the exact murmur jump and `mu agent spawn --command` lines after
|
|
505
|
+
TUI dispatch. mu still needs that local attach pane for read/send/wait; closing
|
|
506
|
+
it detaches without stopping the mule job. Use plain detached remote tmux when
|
|
507
|
+
you need none of mule's artifacts or lifecycle controls.
|
|
508
|
+
|
|
509
|
+
#### mule exit 3 is a handback
|
|
510
|
+
|
|
511
|
+
A missing ssh master can require a human to touch a hardware key. Ask
|
|
512
|
+
the operator to run the command mule prints. Do not retry, run
|
|
513
|
+
`ssh -MNf` yourself, or fall back to `ssh <host> <command>`; each avoids
|
|
514
|
+
the required handback or recreates the capped-channel failure.
|
|
515
|
+
|
|
516
|
+
#### Exit 4 is "not yet"; exit 5 is "never"
|
|
517
|
+
|
|
518
|
+
Exit 4 and 6 mean poll again because the job may still complete. Exit 5
|
|
519
|
+
means no result will arrive. None means the work itself failed. Read the
|
|
520
|
+
full table in `mule --help`.
|
|
521
|
+
|
|
522
|
+
### Fix: detached remote tmux (for the agent itself)
|
|
523
|
+
|
|
524
|
+
Run the agent in its own tmux session on the host, and attach to
|
|
525
|
+
*that*. The agent's lifetime is then decoupled from the connection, so
|
|
526
|
+
the session can be released.
|
|
527
|
+
|
|
528
|
+
```bash
|
|
529
|
+
# Agent runs DETACHED on the host; the ssh returns immediately
|
|
530
|
+
ssh dev 'tmux new-session -d -s mu-worker-1 -c ~/ws/worker-1 \
|
|
531
|
+
"MU_MANAGED_AGENT=1 MU_AGENT_NAME=worker-1 MU_WORKSTREAM=big pi --approve"'
|
|
532
|
+
|
|
533
|
+
# Attach a local pane to it; claim/send are then normal
|
|
534
|
+
mu agent spawn worker-1 -w big --command 'ssh dev -t "tmux attach -t mu-worker-1"'
|
|
535
|
+
|
|
536
|
+
# COLLECT: detach FIRST to free the session, then fetch
|
|
537
|
+
mu agent close worker-1 -w big
|
|
538
|
+
git fetch "ssh://dev/~/ws/worker-1" HEAD && git cherry-pick FETCH_HEAD
|
|
539
|
+
|
|
540
|
+
# Reattach — same session, LLM context intact
|
|
541
|
+
mu agent spawn worker-1 -w big --command 'ssh dev -t "tmux attach -t mu-worker-1"'
|
|
542
|
+
```
|
|
543
|
+
|
|
544
|
+
Two consequences, both counterintuitive:
|
|
545
|
+
|
|
546
|
+
- **An attached pane blocks concurrent ssh just as much as a direct
|
|
547
|
+
one.** Nesting does not make the host concurrent; it makes detaching
|
|
548
|
+
cheap and non-destructive. You must close the pane BEFORE fetching —
|
|
549
|
+
or use mule, which is on its own channel and does not care.
|
|
550
|
+
- **`mu agent close` detaches, it does not stop the agent** — the
|
|
551
|
+
inverse of local semantics, and the whole point. To actually stop
|
|
552
|
+
one: `ssh dev 'tmux kill-session -t mu-worker-1'`. Skip that and you
|
|
553
|
+
accumulate orphaned remote sessions mu cannot see; `ssh dev 'tmux
|
|
554
|
+
ls'` is the only inventory.
|
|
555
|
+
|
|
556
|
+
This adds a THIRD address: local pane → ssh → remote tmux session →
|
|
557
|
+
agent pane. `kick` reaches only the first, `ssh dev 'tmux ls'` is the
|
|
558
|
+
only view of the third. Keep the session name equal to the agent name
|
|
559
|
+
— mu records neither, so it is the only handle tying them together.
|
|
560
|
+
|
|
561
|
+
The session name is load-bearing beyond readability: murmur finds your
|
|
562
|
+
attachment by spotting the agent name in the local pane's argv, and
|
|
563
|
+
with this shape the session name is the only place it appears. `-s
|
|
564
|
+
mu-worker-1` for agent `worker-1` gives you `attached here %N` on the
|
|
565
|
+
remote row; `-s scratch` silently does not.
|
|
566
|
+
|
|
567
|
+
The upside beyond unblocking `git fetch`: a dropped connection no
|
|
568
|
+
longer reaps the task, since the agent outlives the ssh session, and a
|
|
569
|
+
reattach preserves full LLM context.
|
|
570
|
+
|
|
571
|
+
Use this shape only where you need it. On an ordinary host it is
|
|
572
|
+
pointless indirection.
|
|
573
|
+
|
|
574
|
+
### For your own interactive work, use ET instead
|
|
575
|
+
|
|
576
|
+
The recipe above is for mu agents, which need a pane whose process mu
|
|
577
|
+
controls. Your own shell on the host has an easier answer: **Eternal
|
|
578
|
+
Terminal holds no ssh session at all.** It bootstraps over ssh and then
|
|
579
|
+
hands off to `etserver` on its own transport, so `MaxSessions` never
|
|
580
|
+
counts it.
|
|
581
|
+
|
|
582
|
+
```bash
|
|
583
|
+
et dev # or your site's wrapper, e.g. `x2ssh -et dev`
|
|
584
|
+
```
|
|
585
|
+
|
|
586
|
+
**An aborted ET or `x2ssh` leaves local processes behind.** Both survive
|
|
587
|
+
the abort of the tool call that started them and go on holding state, so
|
|
588
|
+
a retry stacks a second one on top. `pkill -f x2ssh` before retrying.
|
|
589
|
+
|
|
590
|
+
Verified on a `MaxSessions 1` host, twice: an interactive `ssh dev`
|
|
591
|
+
starved every other ssh for as long as it stayed open, while an ET
|
|
592
|
+
session left `ssh dev true` succeeding throughout — and the same held
|
|
593
|
+
with a nested `tmux attach` live inside it, with a concurrent `murmur
|
|
594
|
+
collect` reaching the host. So ET costs the capped slot nothing even
|
|
595
|
+
while you are sitting in a remote agent, which is what makes it usable
|
|
596
|
+
as a murmur jump command.
|
|
597
|
+
|
|
598
|
+
So the clean split on a capped host is ET for you, one `ssh -MNf <host>`
|
|
599
|
+
master for tooling, and detached tmux for mu agents. The three do not
|
|
600
|
+
compete.
|
|
601
|
+
|
|
602
|
+
Note ET cannot serve murmur or `git fetch` — it exposes no multiplexing
|
|
603
|
+
socket to attach to. That is exactly why it pairs well: it takes none of
|
|
604
|
+
the capped slots those tools need.
|