@mu-crew/mu 1.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,604 @@
1
+ # Remote workers
2
+
3
+ Running mu agents on another machine. A recipe, not a feature: mu has
4
+ no remote backend, no ssh code, and no host registry, and it needs
5
+ none.
6
+
7
+ Read this before spawning your first remote agent. The recipe is short;
8
+ the traps below it are the part that costs time.
9
+
10
+ **If you read one thing:** on a session-capped host, never leave an
11
+ attach pane open — it holds the only ssh channel and silently breaks
12
+ `git fetch`, `murmur collect` and every other ssh. Attach to look, then
13
+ `mu agent close`. See § Never leave an attach pane open.
14
+
15
+ ---
16
+
17
+ ## The model
18
+
19
+ **The pane is local, the process is remote.** `mu agent spawn
20
+ --command 'ssh <host> -t "..."'` starts an ordinary tmux pane whose
21
+ foreground process happens to be ssh. Everything mu does is
22
+ pane-shaped, so `mu agent send`, `mu agent read`, status detection and
23
+ the reaper all work unchanged, across the network, with no mu changes.
24
+
25
+ **One orchestrator DB; panes may be remote.** All state stays on your
26
+ machine. Do not run a second mu on the host to "coordinate": ownership
27
+ is machine-local by construction — `tasks.owner_id` is an FK into the
28
+ `agents` table, which never syncs — so a remote mu could not claim
29
+ tasks in your DAG. Two half-views, no benefit.
30
+
31
+ **murmur sees the REMOTE pane** — the opposite of the line above. Its
32
+ extension runs inside the agent's process, so it claims the pane on the
33
+ HOST; mu's local pane holds an ssh client and reports nothing. Two
34
+ addresses, one worker.
35
+
36
+ murmur ties them back together by looking for the AGENT NAME in your
37
+ local pane's command line. When it matches, the remote row shows
38
+ `attached here %N` and enter focuses your existing pane instead of
39
+ opening a second connection — which matters on a capped host, where the
40
+ second one fails.
41
+
42
+ Both recipes below work, for the same reason: the agent name is in the
43
+ argv either way. A direct spawn carries it in the remote command; the
44
+ detached shape carries it in the session name, because you named the
45
+ session after the agent. **Keep doing that** — `-s mu-<agent>` is what
46
+ makes the attachment findable, and a session named anything else costs
47
+ you the hint.
48
+
49
+ Do not bother putting `MU_AGENT_NAME=` in front of the attach command
50
+ to help it along. An environment prefix is consumed by your shell and
51
+ never reaches the process arguments, which is all `ps` reports on macOS.
52
+
53
+ What mu does NOT know about a remote agent: the workspace. There is no
54
+ `vcs_workspaces` row, so no `mu workspace list / refresh / commits /
55
+ free`, no `behind` column, no staleness warning on claim, and no
56
+ auto-free on close. You own that bookkeeping.
57
+
58
+ ---
59
+
60
+ ## Never leave an attach pane open
61
+
62
+ On a host that caps sessions per connection, **your attach pane holds
63
+ the only ssh channel**. While it is open, `git fetch`, `git push`,
64
+ `murmur collect`, `rsync`, and plain ssh can fail with the misleading
65
+ `Permission denied (keyboard-interactive)` error.
66
+
67
+ **Attach to look, then close immediately.** `mu agent close <name>`
68
+ detaches without stopping a detached-tmux agent. Poll with `murmur
69
+ collect` + `murmur status`; use mule for orchestrator commands because
70
+ its separate ControlPath cannot contend with the attach pane. See
71
+ § When the host limits concurrent sessions for diagnosis and the
72
+ remote-agent setup.
73
+
74
+ When a collect fails, current murmur names an attachment that holds the
75
+ channel. Otherwise inspect it directly:
76
+
77
+ ```bash
78
+ ps -o pid=,command= -ax | grep "[s]sh <host>"
79
+ ```
80
+
81
+ ### Better: attach with the peer's jump command
82
+
83
+ If murmur knows the host, ask it how to reach the host interactively
84
+ rather than hardcoding ssh:
85
+
86
+ ```bash
87
+ JUMP=$(murmur peer list --json | jq -r '.[]|select(.name=="dev").jump_command')
88
+ mu agent spawn worker-1 -w big --command "${JUMP//\{pane\}/mu-worker-1}"
89
+ ```
90
+
91
+ That command may name a transport taking **no** ssh session at all,
92
+ which removes the contention rather than managing it.
93
+
94
+ **A spawn that lands on an auth prompt looks exactly like a healthy
95
+ one.** Measured: with the slot busy, `ssh dev -t "tmux attach"` fell
96
+ back to a fresh connection and stopped at `Enter a passcode:`. The spawn
97
+ succeeded, the row appeared, `mu agent list` showed `needs_input` — and
98
+ `mu agent send` pasted the whole prompt into the passcode field and
99
+ reported success. The agent received nothing.
100
+
101
+ So confirm a remote agent actually got the work before trusting it
102
+ (`mu agent read <name> -n 20`, or its context percentage), and never
103
+ send anything sensitive to an unconfirmed pane.
104
+
105
+ ---
106
+
107
+ ## The recipe
108
+
109
+ ```bash
110
+ # 1. WORKSPACE — you create it; --workspace does NOT work remotely
111
+ ssh dev 'git -C ~/repo worktree add ~/ws/worker-1'
112
+
113
+ # 2. RECORD — persist location and per-worker baseline together
114
+ mu task note t1 -w big "REMOTE: dev:~/ws/worker-1
115
+ REMOTE_BASE: worker-1:$(ssh dev 'cd ~/ws/worker-1 && git rev-parse HEAD')"
116
+
117
+ # 3. SPAWN — env vars go INSIDE the command (tmux -e stops at the hop)
118
+ mu agent spawn worker-1 -w big --command \
119
+ 'ssh dev -t "cd ~/ws/worker-1 && MU_MANAGED_AGENT=1 MU_AGENT_NAME=worker-1 MU_WORKSTREAM=big pi --approve"'
120
+
121
+ # 4. CLAIM + SEND — identical to a local agent
122
+ mu task claim t1 -w big --for worker-1 --evidence 'remote on dev'
123
+ mu agent send worker-1 -w big '...'
124
+
125
+ # 5. WAIT — run the claim's one-shot Next: command once per turn
126
+ # Set a deadline; at expiry read the pane, then release or re-dispatch.
127
+ job=$(mule run --host dev --max-secs 30 \
128
+ 'cd ~/ws/worker-1 && git rev-parse HEAD 2>/dev/null || echo unreadable')
129
+ mule wait "$job" >/dev/null && sha=$(mule tail "$job")
130
+ case "$sha" in (*[!0-9a-fA-F]*|'') :;; (*) [ "${#sha}" -eq 40 ] && \
131
+ { [ "$sha" = 24481fe24481fe24481fe24481fe24481fe2448 ] || \
132
+ mu task close t1 -w big --evidence "worker-1 committed $sha"; };; esac
133
+
134
+ # 6. COLLECT — fetch straight from the remote worktree
135
+ git fetch "ssh://dev/~/ws/worker-1" HEAD && git cherry-pick FETCH_HEAD
136
+ ```
137
+
138
+ Local and remote agents mix freely in one workstream. The DAG, tracks,
139
+ `mu state` and `mu task wait` do not care where a pane's process runs
140
+ — task status is a row in YOUR database, written by you, so a wait on
141
+ it is exact.
142
+
143
+ **Agent STATUS is different, and this is the one place "nothing
144
+ changes" is false.** `busy` / `needs_input` / `idle` come from reading
145
+ the local pane's scrollback, which for a remote worker is a nested tmux
146
+ rendered over ssh. What you get is redraw lag and quiet periods that
147
+ look like idleness, so anything derived from status is unreliable here:
148
+
149
+ | waiting on | remote? |
150
+ | --- | --- |
151
+ | `mu task wait` (task status) | exact — a DB poll, once something closes the task |
152
+ | the reaper | fires, but see below |
153
+ | stall detection | unreliable |
154
+ | `mu agent wait --first` | same, it is status-based |
155
+
156
+ Measured both directions in one session: a stall fired at 300s against
157
+ a worker that was visibly mid-turn, and `mu agent list` showed
158
+ `needs_input` for another that was working. Give remote waits a generous
159
+ `--timeout`; see `mu task wait --help` for stall controls.
160
+
161
+ The reaper is right for a DIRECT spawn — the connection dying really
162
+ does kill that agent — and wrong for a detached-tmux one, where the
163
+ agent outlives the ssh but mu reaps the task anyway. See § A dropped
164
+ connection reaps the task but NOT the commit.
165
+
166
+ **If you need trustworthy remote status, ask murmur, not mu.** Its
167
+ extension runs inside pi ON THE HOST and pushes state, so nothing is
168
+ scraped and no ssh hop distorts it:
169
+
170
+ ```bash
171
+ murmur collect && murmur status --json # activity is host-reported
172
+ ```
173
+
174
+ That is the division worth remembering: mu owns the work, murmur owns
175
+ what the agent is doing.
176
+
177
+ ### On step 2 — the note is load-bearing
178
+
179
+ The task note survives connection loss and context compaction. Keep the
180
+ location as `REMOTE: <host>:<path>` and each baseline as
181
+ `REMOTE_BASE: <agent>:<sha>`. `mu task claim --for <agent>` uses both to
182
+ print a complete one-shot poll-and-close command. For a crew, record one
183
+ baseline per worker and use one bounded mule job for every path:
184
+
185
+ ```bash
186
+ mule run --max-secs 30 'for d in ~/ws/*/; do printf "%s %s\\n" \
187
+ "$(basename $d)" "$(cd $d && git rev-parse HEAD 2>/dev/null || echo unreadable)"; done'
188
+ mule wait <job>; mule tail <job>
189
+ ```
190
+
191
+ Each line names the worker and sha. Accept only 40 hex characters;
192
+ `unreadable` or an empty field means retry next turn, never progress.
193
+
194
+ ### On step 5 — wait on commits
195
+
196
+ **Poll once per turn, never sleep in a tool call.** A `while true` loop
197
+ with `sleep` holds the call; aborting it can leave remote work and a
198
+ capped ssh channel running. Set a wall-clock deadline before dispatch.
199
+ At expiry, read the pane, then release or re-dispatch instead of extending
200
+ the wait silently.
201
+
202
+ On a session-capped host, run the claim's `Next:` command through mule
203
+ between other work. Bare ssh polls can be refused with an empty sha,
204
+ which looks like progress. The command captures the new sha and closes
205
+ the task with that sha as evidence. The commit tells you;
206
+ the DAG stays IN_PROGRESS until the orchestrator closes it. This feeds
207
+ `mu task wait` so blocked tasks advance; it does not replace it.
208
+
209
+ ### Recovering the location
210
+
211
+ `mu state` lists exact `REMOTE:` lines as its remote-worker inventory,
212
+ and the recovery command is mechanical:
213
+
214
+ ```bash
215
+ git fetch "ssh://<host>/<path>" HEAD && git cherry-pick FETCH_HEAD
216
+ ```
217
+
218
+ ### On step 6 — fetching from a worktree
219
+
220
+ `git fetch "ssh://<host>/<path>" HEAD` reads a remote worktree
221
+ **directly**. No shared remote, no push, no bare repo in between. This
222
+ is the part people expect to be hard and it is not.
223
+
224
+ - It exits **128** on failure, so `&&` chaining is safe.
225
+ - **Quote the URL.** `~` is legal in an `ssh://` URL (git's own docs
226
+ list `ssh://host/~user/path`), but unquoted it is expanded by your
227
+ LOCAL shell into your laptop's home.
228
+ - Two workers editing one file still conflict on cherry-pick, exactly
229
+ as locally. Bucket work by file cluster, not by machine.
230
+
231
+ ### Run the merged suite where the workers are
232
+
233
+ The reason to send work to a big machine is to stop paying for it on a
234
+ small one. Cherry-picking remote work and then running the whole suite
235
+ locally gives that back — and it is the default thing to do, so say the
236
+ other thing explicitly.
237
+
238
+ **Do not re-run what the worker ran.** Its green on its own tree is the
239
+ evidence you asked for. What is unverified is the MERGE: the worker
240
+ forked from a base that has since moved, so the only new information is
241
+ in the combination.
242
+
243
+ So cherry-pick locally (it is a few seconds of IO), then push the merged
244
+ head to the same host and run the gate there:
245
+
246
+ ```bash
247
+ git cherry-pick <sha>
248
+ git push -q "ssh://dev/~/hacking/<repo>.git" HEAD:refs/heads/main
249
+ ssh dev 'cd ~/hacking/<checkout> && git fetch -q origin \
250
+ && git reset -q --hard origin/main && npm run check'
251
+ ```
252
+
253
+ **On a session-capped host, run that gate through mule instead.** The
254
+ last line holds the channel for the whole suite — minutes — which is
255
+ precisely when your other tooling starts failing with a credentials
256
+ error that has nothing to do with credentials:
257
+
258
+ ```bash
259
+ mule run --cwd ~/hacking/<checkout> --wait \
260
+ 'git fetch -q origin && git reset -q --hard origin/main && npm run check'
261
+ ```
262
+
263
+ Same work, dispatched detached on mule's own connection, and `--wait`
264
+ exits with the suite's own code. See § When the host limits concurrent
265
+ sessions.
266
+
267
+ That host already has a checkout and warm dependencies — the same ones
268
+ the worker used — so the marginal cost is near zero, while the same run
269
+ on a laptop is minutes of CPU per integration.
270
+
271
+ **Keep two things local.** A **platform-sensitive** subset, because a
272
+ remote green does not prove a local green when the bug is
273
+ platform-shaped: macOS `ps` returns argv where Linux `ps -e` appends the
274
+ environment, and a real bug lived in exactly that gap. And the **final**
275
+ gate before the push that matters, since that one is about your tree
276
+ rather than the worker's.
277
+
278
+ ---
279
+
280
+ ## Traps
281
+
282
+ ### Use the host's real CLI command
283
+
284
+ The wrapper you run locally is not what a bare `pi` gives you on the
285
+ host. If your local `$MU_PI_COMMAND` is `pi-meta --pi-meta-no-solo
286
+ --approve`, spawning plain `pi` remotely yields a live pane, a healthy
287
+ status, and an agent with **no models configured** — it looks fine
288
+ until you send it work. Check first:
289
+
290
+ ```bash
291
+ ssh dev 'command -v pi-meta'
292
+ ```
293
+
294
+ ### Tell the worker that mu is absent
295
+
296
+ `mu` is usually not installed on the host, and the DB is on your
297
+ laptop regardless. The in-pane worker loop (`mu task claim` / `note` /
298
+ `close`) therefore cannot run. Say so in the prompt and have the
299
+ worker print its sha instead; YOU claim, note and close from the
300
+ orchestrator. Omit this and the worker burns a turn on `command not
301
+ found`, or worse, silently fails to close and your `mu task wait`
302
+ hangs.
303
+
304
+ ### Everything is orchestrator-PULL
305
+
306
+ The host frequently cannot resolve your laptop at all — a corporate
307
+ devserver typically has no route back to a NAT'd machine. So: you push
308
+ setup out, you fetch commits back. Never write a recipe in which the
309
+ host reaches you, and never assume a peer can `git fetch` from you.
310
+
311
+ ### A dropped connection reaps the task but NOT the commit
312
+
313
+ If the ssh client dies — network drop, laptop sleep, VPN blip — the
314
+ pane dies with it, and mu behaves exactly as for a dead local agent:
315
+ the agent row goes, and the reaper reverts the task `IN_PROGRESS →
316
+ OPEN` with a `[reaper]` note. That is correct and desirable.
317
+
318
+ But the worker's **commit is still on the host**, and mu has no record
319
+ of where. Before re-dispatching, fetch from the path in the task note
320
+ and look: re-running the task blind duplicates work that already
321
+ exists. This is the strongest argument for step 2.
322
+
323
+ ### A crashed remote worker disappears rather than reporting `crashed`
324
+
325
+ murmur records `crashed` only when a **pane outlives the process** in
326
+ it: that is how an unreported death leaves a trace. Locally it holds —
327
+ a pi exits inside a shell pane and the pane stays. It does not hold
328
+ here, because the agent is the remote session's only process, so tmux
329
+ reaps the session with it and the row is simply gone at the next
330
+ collect.
331
+
332
+ Measured: SIGKILL a remote agent and murmur reports **zero rows and
333
+ zero crashed**, not a crash. Nothing distinguishes "it died" from "it
334
+ finished and I closed it". A second consequence of the same rule: the
335
+ crash path only fires for an agent killed **mid-turn**, since an idle
336
+ agent is already `stopped` and a stopped owner dying reads as a normal
337
+ finish.
338
+
339
+ So do not wait for a crash signal on a remote worker. The durable
340
+ traces are the ones mu already gives you:
341
+
342
+ - the reaper flipping the task back to `OPEN` with a `[reaper]` note
343
+ - the `REMOTE:` task note, which is where the commit is
344
+
345
+ `ssh <host> 'tmux ls'` confirms whether the session is really gone.
346
+
347
+ ### Cleaning up
348
+
349
+ mu will not remove a remote worktree, because it does not know it
350
+ exists:
351
+
352
+ ```bash
353
+ ssh dev 'git -C ~/repo worktree remove ~/ws/worker-1'
354
+ ```
355
+
356
+ `mu agent kick` signals the local pane's foreground process group —
357
+ that is the ssh client, not the remote agent. Use `mu agent close` and
358
+ respawn.
359
+
360
+ ---
361
+
362
+ ## mu and murmur, and what you lose without it
363
+
364
+ [murmur](https://github.com/mu-crew/murmur) is optional and
365
+ **strictly additive**. Nothing in mu needs it: the DAG, claim/close/
366
+ wait, workspaces, spawn/send/read, the reaper, `mu state`, this whole
367
+ remote recipe and `git fetch` collection all work with murmur absent.
368
+ What you lose is the *view* — cross-machine agent state, the tmux
369
+ status pills, `prefix+a`, the attachment hint, and `murmur peer list`
370
+ for host reachability. Guard on it (`if it is installed`) rather than
371
+ assuming it.
372
+
373
+ The division: **mu owns the work, murmur owns what an agent is doing.**
374
+ murmur never places work.
375
+
376
+ | question | ask |
377
+ | --- | --- |
378
+ | what should happen next | mu — the DAG |
379
+ | who owns this task | mu — `claim` / `close` |
380
+ | is the task done | mu — `task wait`, a DB poll |
381
+ | what is this agent doing right now | murmur — pushed from inside pi |
382
+ | is anything blocked on me, anywhere | murmur |
383
+ | which host can I reach | murmur — `peer list` |
384
+
385
+ **The seam is three env vars, and it is load-bearing.** `mu agent
386
+ spawn` injects `MU_MANAGED_AGENT=1`, `MU_AGENT_NAME` and
387
+ `MU_WORKSTREAM`; pi inherits them and murmur's extension reads them.
388
+ That one mechanism gives you:
389
+
390
+ 1. `driver=orchestrated`, so crew stays out of the human's status bar
391
+ unless it is blocked or crashed — true for local and remote alike
392
+ 2. the workstream and agent name on the row, for grouping
393
+ 3. the attachment back-reference for a remote worker
394
+
395
+ For a remote worker they go **inside** the ssh command (tmux `-e` stops
396
+ at the hop). Miss them and the agent reports `driver=human`: it appears
397
+ in your own counts as if it were yours.
398
+
399
+ ## Picking a host
400
+
401
+ mu does not track hosts and should not; that is murmur's job. If it is
402
+ installed:
403
+
404
+ ```bash
405
+ murmur peer list --json | jq -r '.[] | select(.error) | .name'
406
+ ```
407
+
408
+ Read it correctly, because it is **best-effort by design**:
409
+
410
+ - `ssh` is **last-known** reachability; `error` is the **current**
411
+ attempt. A host can read `warm` and be failing right now.
412
+ - The command exits **0** either way — a fleet with sleeping machines
413
+ is the normal state, not a fault.
414
+ - So branch on `.error`, never on presence in the list.
415
+
416
+ `murmur status` and `pick` are polling paths and stay silent whatever
417
+ the fleet is doing. `murmur collect` is the deliberate dial: it prints
418
+ one line per host it could not reach, so it is the one to run for "can I
419
+ reach this host *right now*".
420
+
421
+ ### Reaching a host is two different questions
422
+
423
+ A peer carries `target` (for a COMMAND — always ssh, used by the
424
+ collector) and a jump command (for a HUMAN — need not be ssh):
425
+
426
+ ```bash
427
+ murmur peer set <name> --jump-command '<command with {pane}>'
428
+ ```
429
+
430
+ Opaque template: murmur substitutes `{pane}` and runs the rest
431
+ unparsed. The default reproduces `ssh -t <target> tmux attach`, so an
432
+ unconfigured peer behaves as before.
433
+
434
+ On a session-capped host this is the difference between holding the one
435
+ slot for your whole visit and holding nothing (see the ET section).
436
+ Don't hardcode `et` — a site wrapper may add VPN selection and its own
437
+ binary resolution, which is why the value is opaque.
438
+
439
+ ---
440
+
441
+ ## When the host limits concurrent sessions
442
+
443
+ Rare, but it presents as a credentials bug, so learn to recognise it.
444
+
445
+ Most sshd allow 10 sessions per connection (`MaxSessions`, default 10)
446
+ and the recipe above is all you need. A hardened host may set
447
+ **`MaxSessions 1`**: then the long-lived ssh holding your AGENT consumes
448
+ the only channel, and every other ssh — including `git fetch` — is
449
+ refused with the misleading 2FA error described in § Never leave an
450
+ attach pane open.
451
+
452
+ **Diagnostic:** if `mu agent read` works fine while a plain `ssh
453
+ <host> true` fails, it is session exhaustion, not credentials.
454
+
455
+ **But do not stop there — that test is necessary, not sufficient.** The
456
+ same `Permission denied (keyboard-interactive)` is also what a **dead
457
+ ssh agent** produces: `SSH_AUTH_SOCK` points at a socket that no longer
458
+ exists, key auth cannot be attempted, and sshd falls back to 2FA and
459
+ sits on a passcode prompt you cannot see. One session cost an hour to
460
+ this, diagnosed as session exhaustion throughout.
461
+
462
+ Check both, in this order — the second is local and instant:
463
+
464
+ ```bash
465
+ ssh-add -l # "Error connecting to agent" => dead agent, NOT the cap
466
+ ssh <host> true # fails while `mu agent read` works => the cap
467
+ ```
468
+
469
+ And note the prompt is **invisible unless the ssh runs inside a pane you
470
+ can capture**. That is what finally exposed it: the refusal looks like a
471
+ silent failure because the question is being asked somewhere with no
472
+ terminal attached.
473
+
474
+ `ControlMaster no` looks like it should help and does not. It governs
475
+ master *creation* only; a refused channel falls back regardless.
476
+ Measured with `no` set: three of four concurrent calls still produced
477
+ the misleading 2FA error. There is no client-side fix — `MaxSessions`
478
+ exists only in `sshd_config`, and the client discovers the cap only by
479
+ being refused.
480
+
481
+ ### Fix: run commands through mule
482
+
483
+ [mule](https://github.com/mu-crew/mule) opens a separate ssh
484
+ ControlPath and dispatches detached jobs. On a `MaxSessions 1` host,
485
+ five concurrent bare calls produced **1 of 5** successes; mule produced
486
+ **5 of 5**. A multi-minute mule job also left a concurrent plain ssh
487
+ working.
488
+
489
+ Route commands by failure mode, not duration. Use mule for long work
490
+ and whenever refusal could look like success. Eight concurrent bare
491
+ `rev-parse` polls returned one sha and seven empty results; mule
492
+ dispatched all eight. Keep worktree setup and `murmur collect` bare
493
+ because they fail loudly. See `mule --help` and the relevant subcommand
494
+ help for flags, job control, warnings, and exit codes.
495
+
496
+ **A mule job must be entirely remote.** The host has no route back to
497
+ your laptop, so a local-endpoint `git fetch`, `git push`, or rsync
498
+ cannot run inside the job. Collect from the orchestrator as described
499
+ in § Everything is orchestrator-PULL.
500
+
501
+ **Use `mule run --tui` for an interactive/full-screen agent when you want
502
+ mule's timeout, runtime, transcript, and cleanup.** Plain `mule run` pipes
503
+ output into the artifact, so attaching finds the pane but no live TUI.
504
+ mule prints the exact murmur jump and `mu agent spawn --command` lines after
505
+ TUI dispatch. mu still needs that local attach pane for read/send/wait; closing
506
+ it detaches without stopping the mule job. Use plain detached remote tmux when
507
+ you need none of mule's artifacts or lifecycle controls.
508
+
509
+ #### mule exit 3 is a handback
510
+
511
+ A missing ssh master can require a human to touch a hardware key. Ask
512
+ the operator to run the command mule prints. Do not retry, run
513
+ `ssh -MNf` yourself, or fall back to `ssh <host> <command>`; each avoids
514
+ the required handback or recreates the capped-channel failure.
515
+
516
+ #### Exit 4 is "not yet"; exit 5 is "never"
517
+
518
+ Exit 4 and 6 mean poll again because the job may still complete. Exit 5
519
+ means no result will arrive. None means the work itself failed. Read the
520
+ full table in `mule --help`.
521
+
522
+ ### Fix: detached remote tmux (for the agent itself)
523
+
524
+ Run the agent in its own tmux session on the host, and attach to
525
+ *that*. The agent's lifetime is then decoupled from the connection, so
526
+ the session can be released.
527
+
528
+ ```bash
529
+ # Agent runs DETACHED on the host; the ssh returns immediately
530
+ ssh dev 'tmux new-session -d -s mu-worker-1 -c ~/ws/worker-1 \
531
+ "MU_MANAGED_AGENT=1 MU_AGENT_NAME=worker-1 MU_WORKSTREAM=big pi --approve"'
532
+
533
+ # Attach a local pane to it; claim/send are then normal
534
+ mu agent spawn worker-1 -w big --command 'ssh dev -t "tmux attach -t mu-worker-1"'
535
+
536
+ # COLLECT: detach FIRST to free the session, then fetch
537
+ mu agent close worker-1 -w big
538
+ git fetch "ssh://dev/~/ws/worker-1" HEAD && git cherry-pick FETCH_HEAD
539
+
540
+ # Reattach — same session, LLM context intact
541
+ mu agent spawn worker-1 -w big --command 'ssh dev -t "tmux attach -t mu-worker-1"'
542
+ ```
543
+
544
+ Two consequences, both counterintuitive:
545
+
546
+ - **An attached pane blocks concurrent ssh just as much as a direct
547
+ one.** Nesting does not make the host concurrent; it makes detaching
548
+ cheap and non-destructive. You must close the pane BEFORE fetching —
549
+ or use mule, which is on its own channel and does not care.
550
+ - **`mu agent close` detaches, it does not stop the agent** — the
551
+ inverse of local semantics, and the whole point. To actually stop
552
+ one: `ssh dev 'tmux kill-session -t mu-worker-1'`. Skip that and you
553
+ accumulate orphaned remote sessions mu cannot see; `ssh dev 'tmux
554
+ ls'` is the only inventory.
555
+
556
+ This adds a THIRD address: local pane → ssh → remote tmux session →
557
+ agent pane. `kick` reaches only the first, `ssh dev 'tmux ls'` is the
558
+ only view of the third. Keep the session name equal to the agent name
559
+ — mu records neither, so it is the only handle tying them together.
560
+
561
+ The session name is load-bearing beyond readability: murmur finds your
562
+ attachment by spotting the agent name in the local pane's argv, and
563
+ with this shape the session name is the only place it appears. `-s
564
+ mu-worker-1` for agent `worker-1` gives you `attached here %N` on the
565
+ remote row; `-s scratch` silently does not.
566
+
567
+ The upside beyond unblocking `git fetch`: a dropped connection no
568
+ longer reaps the task, since the agent outlives the ssh session, and a
569
+ reattach preserves full LLM context.
570
+
571
+ Use this shape only where you need it. On an ordinary host it is
572
+ pointless indirection.
573
+
574
+ ### For your own interactive work, use ET instead
575
+
576
+ The recipe above is for mu agents, which need a pane whose process mu
577
+ controls. Your own shell on the host has an easier answer: **Eternal
578
+ Terminal holds no ssh session at all.** It bootstraps over ssh and then
579
+ hands off to `etserver` on its own transport, so `MaxSessions` never
580
+ counts it.
581
+
582
+ ```bash
583
+ et dev # or your site's wrapper, e.g. `x2ssh -et dev`
584
+ ```
585
+
586
+ **An aborted ET or `x2ssh` leaves local processes behind.** Both survive
587
+ the abort of the tool call that started them and go on holding state, so
588
+ a retry stacks a second one on top. `pkill -f x2ssh` before retrying.
589
+
590
+ Verified on a `MaxSessions 1` host, twice: an interactive `ssh dev`
591
+ starved every other ssh for as long as it stayed open, while an ET
592
+ session left `ssh dev true` succeeding throughout — and the same held
593
+ with a nested `tmux attach` live inside it, with a concurrent `murmur
594
+ collect` reaching the host. So ET costs the capped slot nothing even
595
+ while you are sitting in a remote agent, which is what makes it usable
596
+ as a murmur jump command.
597
+
598
+ So the clean split on a capped host is ET for you, one `ssh -MNf <host>`
599
+ master for tooling, and detached tmux for mu agents. The three do not
600
+ compete.
601
+
602
+ Note ET cannot serve murmur or `git fetch` — it exposes no multiplexing
603
+ socket to attach to. That is exactly why it pairs well: it takes none of
604
+ the capped slots those tools need.