@celilo/cli 0.18.0 → 0.20.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (34) hide show
  1. package/CELILO_SUBSYSTEMS.md +4 -2
  2. package/package.json +4 -4
  3. package/src/api/remote-client.test.ts +86 -2
  4. package/src/api/serve.ts +242 -38
  5. package/src/api/sessions.test.ts +196 -0
  6. package/src/api/sessions.ts +278 -0
  7. package/src/cli/commands/apt-upgrade.test.ts +20 -1
  8. package/src/cli/commands/apt-upgrade.ts +12 -2
  9. package/src/cli/commands/backup-sweep.ts +25 -9
  10. package/src/cli/commands/events.ts +150 -4
  11. package/src/cli/commands/module-update.test.ts +72 -1
  12. package/src/cli/commands/module-update.ts +68 -22
  13. package/src/cli/commands/system-migrate.test.ts +56 -0
  14. package/src/cli/commands/system-migrate.ts +52 -4
  15. package/src/cli/completion.ts +2 -0
  16. package/src/cli/index.ts +27 -3
  17. package/src/db/migration-status.test.ts +114 -0
  18. package/src/db/migration-status.ts +78 -0
  19. package/src/db/schema-introspection.ts +8 -1
  20. package/src/services/backup-metadata.ts +17 -0
  21. package/src/services/backup-staging.test.ts +98 -0
  22. package/src/services/backup-staging.ts +73 -1
  23. package/src/services/backup-sweep.test.ts +15 -0
  24. package/src/services/backup-sweep.ts +17 -1
  25. package/src/services/bus-interview-park.test.ts +179 -0
  26. package/src/services/bus-interview.ts +17 -6
  27. package/src/services/events-daemon.test.ts +244 -0
  28. package/src/services/events-daemon.ts +295 -8
  29. package/src/services/fleet-checks.test.ts +75 -4
  30. package/src/services/fleet-checks.ts +82 -12
  31. package/src/services/interview-errors.ts +37 -0
  32. package/src/services/remote-responder.test.ts +83 -0
  33. package/src/services/remote-responder.ts +31 -10
  34. package/src/services/responder-probe.ts +3 -1
@@ -232,7 +232,9 @@ Creation, scheduling and freshness. A module declares an `on_backup` hook and a
232
232
 
233
233
  ## Events
234
234
 
235
- - **Event bus** — `packages/event-bus/src/index.ts` — `Bus`, `openBus`, `defineEvents`, `defineHandler`, `runDispatcher`, pattern matching + timer ticks (`emitDueTimerTicks`, `retentionSweep`). **Exactly one dispatcher per bus**: `runDispatcher` refuses to start while another dispatcher's process is alive (`assertSoleDispatcher` in `dispatcher.ts`, liveness via `bus.liveDispatchers()` — `kill(pid,0)`, not heartbeat age, since a tick blocks for as long as its slowest handler). The exclusion is here rather than in the systemd unit because a stranded dispatcher can sit outside the unit's cgroup where `KillMode` cannot reach it (#580). `bus.health()` reports `dispatcherCount`/`dispatchers` and a `duplicate_dispatcher` status; `checkDispatcher` (`services/fleet-checks.ts`) fails on more than one.
235
+ - **Migration interrogation** — `apps/celilo/src/db/migration-status.ts` — `getMigrationStatus(sqlite, migrationsFolder)` reports applied count, latest applied migration BY TAG, and pending ones by name (joining `__drizzle_migrations.created_at` to drizzle journal `when`, which is exact). Surface: `celilo system migrate --status`, which opens the DB **read-only** on purpose — `getDb()` auto-migrates on open, so a status routed through it would repair what it claims to report and could never say "pending". Paired with `findSchemaDrift` (`db/schema-introspection.ts`), which is column-aware: a table COUNT cannot distinguish "the column migration applied" from "nothing happened", which is why a rollout asserting `backups.pid` had to reach for `sqlite3` over SSH. `checkSchemaDrift` (`services/fleet-checks.ts`) fails on a missing table, a missing column, OR an unapplied journal migration, and its summary names tables AND columns so the operator can see what was checked.
236
+ - **Event bus** — `packages/event-bus/src/index.ts` — `Bus`, `openBus`, `defineEvents`, `defineHandler`, `runDispatcher`, pattern matching + timer ticks (`emitDueTimerTicks`, `retentionSweep`). **Exactly one dispatcher per bus**: `runDispatcher` refuses to start while another dispatcher's process is alive (`assertSoleDispatcher` in `dispatcher.ts`, liveness via `bus.liveDispatchers()` — `kill(pid,0)`, not heartbeat age, since a tick blocks for as long as its slowest handler). The exclusion is here rather than in the systemd unit because a stranded dispatcher can sit outside the unit's cgroup where `KillMode` cannot reach it (#580). `bus.health()` reports `dispatcherCount`/`dispatchers` and a `duplicate_dispatcher` status; `checkDispatcher` (`services/fleet-checks.ts`) fails on more than one. **Supervision** (`services/events-daemon.ts`): the unit is named `celilo-events.service` in BOTH the user and system scope, so the two are indistinguishable in every operator-facing string — `install-daemon` (which defaults to **user** scope) therefore refuses when the other scope's unit exists, and `checkDispatcher` compares the live pid against each installed unit's `MainPID` (`unitMainPid`) rather than merely testing that a unit file exists, since a file nobody is running is not supervision (#610).
237
+ - **Dispatcher supervision (install / restart)** — `apps/celilo/src/services/events-daemon.ts` — `installDaemon`/`planDaemonInstall`/`uninstallDaemon`/`readInstalledUnit` write the systemd unit or launchd plist without touching supervisor state, and `restartDaemon` (+ pure `orphanDispatcherPids`, `resolveRestartScope`, `supervisorCommands`) is the one verb that DOES cycle it. CLI: `celilo events install-daemon|uninstall-daemon|show-daemon|restart-daemon [--system]`. `restart-daemon` exists because the dispatcher runs the code it LOADED: celilo-mgr sat 9 days on event-bus v0.1.8 after apt installed v0.2.0, running the very bug the release fixed (celilo#604). Two things make it non-trivial and both are load-bearing: (1) it stops any live dispatcher the supervisor does not own first — an ORPHAN (PPID 1) is invisible to `systemctl restart`, and `assertSoleDispatcher` then crash-loops the unit while the old code keeps serving; (2) it verifies on the BUS that a NEW pid is live reporting `BUS_VERSION`, never off systemctl's exit code, which returns 0 into exactly that crash loop. System scope shells `sudo systemctl` (the unit is root-owned; celilo runs unprivileged), covered by the scoped `/etc/sudoers.d/celilo-events-restart` conffile that `celilo-bootstrap` ships — a unit test asserts the argv and the grant cannot drift apart.
236
238
 
237
239
  ## Remote API (drive the CLI over the wire)
238
240
 
@@ -246,7 +248,7 @@ Run any celilo command on celilo-mgr over SSH instead of screen-scraping `ssh <h
246
248
  - **Access control** — `apps/celilo/src/services/api-access.ts` — `grantPrincipal`, `isAuthorized` (deny-by-default, `command:subcommand` grants), `renderAuthorizedKeys`. Table: `api_principals` (`apps/celilo/src/db/schema.ts`). CLI: `apps/celilo/src/cli/commands/api.ts` (`api grant|list|revoke|authorized-keys|key new`).
247
249
  - **Mid-run interview bridge (`kind:daemon` responder)** — `apps/celilo/src/services/remote-responder.ts` — `startRemoteResponder` bridges bus `interview.required.*` ↔ wire.
248
250
  - **Server provisioning** — the `celilo-bootstrap` deb (`packaging/celilo-bootstrap/scripts/postinst`) creates the non-root `celilo-api` landing account + sshd; membership in the `celilo` group + `/etc/sudoers.d/celilo` (`!use_pty`) gives api-serve DB access via the wrapper's sudo-drop.
249
- - **Self-upgrade (apt)** — `celilo apt-upgrade` (`apps/celilo/src/cli/commands/apt-upgrade.ts`) upgrades the deb-installed `celilo`/`celilo-bootstrap` packages (`apt-get update` → `--only-upgrade install`) then spawns a fresh `celilo system migrate` (ISS-0100). It's the RW target behind the MCP's registry-derived `celilo_apt_upgrade` tool; the celilo user's two apt invocations are scoped-sudo'd by `/etc/sudoers.d/celilo-apt-upgrade`, shipped by `celilo-bootstrap`. **This upgrades celilo ITSELF — not the modules it manages. For those, see Module auto-upgrade below; the two are routinely confused.**
251
+ - **Self-upgrade (apt)** — `celilo apt-upgrade` (`apps/celilo/src/cli/commands/apt-upgrade.ts`) upgrades the deb-installed `celilo`/`celilo-bootstrap` packages (`apt-get update` → `--only-upgrade install`) then spawns a fresh `celilo system migrate` (ISS-0100), then `celilo events restart-daemon` so the dispatcher actually runs the code just installed — a failure there fails the whole command and names which steps DID complete, because "upgraded" while the dispatcher serves stale code is the silent state celilo#604 documents. It's the RW target behind the MCP's registry-derived `celilo_apt_upgrade` tool; the celilo user's two apt invocations are scoped-sudo'd by `/etc/sudoers.d/celilo-apt-upgrade`, shipped by `celilo-bootstrap`. **This upgrades celilo ITSELF — not the modules it manages. For those, see Module auto-upgrade below; the two are routinely confused.**
250
252
  - **Module auto-upgrade (registry-poll CD)** — the *pull* half of continuous deployment: celilo-mgr polls the registry and upgrades opted-in modules unattended. Spec: `openspec/specs/module-auto-upgrade/spec.md`. Entry points: `apps/celilo/src/cli/commands/module-upgrade.ts` — `runRegistryPoll` (the `--poll` path), `selectPollTargets` (pure: `autoUpgrade && latest && change ∉ {up-to-date, ahead}`), `upgradeOneModule` (update → backup → deploy → verify), `needsPreUpgradeBackup`, `pickAutoUpgrade`/`pickUpgradePolicy` (both fail closed/safe); `classifyVersionChange` in `module-update.ts` (treats a registry `+N` revision as a patch); `resolveDeployPosture` in `apps/celilo/src/services/deploy-posture.ts`. Trigger: celilo-mgmt's `registry-poll` subscription (`modules/celilo-mgmt/manifest.yml`) on `timer.tick.15m` with handler **`celilo module upgrade --poll`** — the flag is REQUIRED, since the dispatcher appends the event id positionally and a bare handler would consume it as the optional module name (silent: 3108 deliveries, 0 successes). Operator controls are framework config keys settable on ANY module (`FRAMEWORK_CONFIG_KEYS` in `module-config.ts`): `auto_upgrade` (opt-in, default false) and `upgrade_policy` (`by-semver`|`always-safe`|`always-fast`), validated at set time because both readers fail open. ⚠️ `always-safe` guarantees safe *posture*, NOT a backup — `needsPreUpgradeBackup` also requires the TARGET manifest to declare an `on_backup` hook, else it warns and proceeds. Confirm a data-bearing module declares `on_backup` before enabling `auto_upgrade` on it. The *build* half (app CI publishing a `.netapp` on merge) is not yet shipped — `openspec/changes/build-bus-poll-cd`.
251
253
  - **MCP service (`@celilo/mcp`)** — `packages/mcp/src/` — an operator-facing stdio MCP server (official `@modelcontextprotocol/sdk`, bin `celilo-mcp`) that drives a remote celilo server over the Remote API for an AI client. Two-item config (`config.ts`: `server` + `defaultUser`, env or `~/.config/celilo-mcp/config.json`). Dual-principal auth (`auth.ts`: `celilo-mcp auth setup` enrolls read-only `celilo-mcp-ro` + full `celilo-mcp-rw` ed25519 keypairs, prints the exact `celilo api grant` lines the operator runs server-side). Transport (`transport.ts`): reuses `@celilo/core` `runRemoteClient`, selecting the principal by `ssh -i <key>` and capturing structured output. Tool surface is generated LIVE from the server's command registry — `registry-fetch.ts` fetches `celilo commands --json` (+ `service list --json` for configured providers) over the RO principal on connect; `tools-from-registry.ts` (pure) projects that into one tool per runnable leaf, grouped by top-level command (`celilo_module_*`, `celilo_proxmox_*`, …), each with a Zod input schema from the leaf's args/flags and a read/write tag → RO/RW routing, plus a generic `celilo_run` escape hatch. Auto-detect hides provider-gated groups (e.g. `celilo_proxmox_*` until a Proxmox service is configured) and re-detects on a timer, emitting `notifications/tools/list_changed` when the surface changes. Coverage gate (`tests/coverage.test.ts`) asserts every registry leaf maps to a tool. Composite RO troubleshooting tools (`troubleshoot.ts` pure correlation + `troubleshoot-tools.ts` thin adapters): `celilo_assess_module <id>` and `celilo_fleet_status` correlate `celilo audit --json` (the drift backbone) with the `module list --json` roster into a per-module / fleet-wide verdict. Design: `openspec/changes/celilo-mcp-service/proposal.md`. (Distinct from the dev/ops `@celilo/mcp-server` below.)
252
254
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@celilo/cli",
3
- "version": "0.18.0",
3
+ "version": "0.20.0",
4
4
  "description": "Celilo — home lab orchestration CLI",
5
5
  "type": "module",
6
6
  "bin": {
@@ -57,10 +57,10 @@
57
57
  },
58
58
  "dependencies": {
59
59
  "@aws-sdk/client-s3": "^3.1024.0",
60
- "@celilo/capabilities": "^0.10.0",
60
+ "@celilo/capabilities": "^0.11.0",
61
61
  "@celilo/cli-display": "^0.1.10",
62
- "@celilo/core": "^0.3.2",
63
- "@celilo/event-bus": "^0.2.0",
62
+ "@celilo/core": "^0.5.0",
63
+ "@celilo/event-bus": "^0.3.0",
64
64
  "@clack/prompts": "^1.1.0",
65
65
  "ajv": "^8.18.0",
66
66
  "drizzle-orm": "^0.36.4",
@@ -72,7 +72,7 @@ test('renders a forwarded interview and sends the answer back', async () => {
72
72
  push({ type: 'interview', id: 'q1', kind: 'text', message: 'Hostname?' });
73
73
 
74
74
  const seen: Array<{ id: string }> = [];
75
- const code = await runRemoteClient('ignored', ['module', 'deploy', 'site'], {
75
+ const outcome = await runRemoteClient('ignored', ['module', 'deploy', 'site'], {
76
76
  openTransport: () => transport,
77
77
  out: { write() {} },
78
78
  renderInterview: async (iv) => {
@@ -81,7 +81,7 @@ test('renders a forwarded interview and sends the answer back', async () => {
81
81
  },
82
82
  });
83
83
 
84
- expect(code).toBe(0);
84
+ expect(outcome).toEqual({ status: 'result', exitCode: 0 });
85
85
  expect(seen).toHaveLength(1);
86
86
  expect(seen[0].id).toBe('q1');
87
87
  expect(writes.some((w) => w.includes('"command"'))).toBe(true);
@@ -89,3 +89,87 @@ test('renders a forwarded interview and sends the answer back', async () => {
89
89
  expect(answer).toBeDefined();
90
90
  expect(JSON.parse(answer as string)).toEqual({ type: 'answer', id: 'q1', value: 'myhost' });
91
91
  });
92
+
93
+ /**
94
+ * Regression for the fabricated "operator declined".
95
+ *
96
+ * The default renderer prompts with clack, which reads keypresses off stdin
97
+ * whether or not stdin is a terminal. Driven over the MCP (stdin = the JSON-RPC
98
+ * stream) the next newline submitted the prompt at its `initialValue` — the
99
+ * question's `defaultValue` — so a breaking update nobody saw came back as a
100
+ * considered "no". With no terminal the client must say it cannot answer.
101
+ *
102
+ * And it must say so as `unanswerable`, NOT as an `answer` of any shape: an
103
+ * `answer` is what consumes the query and destroys a question nobody decided
104
+ * (celilo#609). The server parks and replies `blocked`.
105
+ */
106
+ test('no TTY and no renderer → sends unanswerable and returns blocked, never an answer', async () => {
107
+ const writes: string[] = [];
108
+ const encoder = new TextEncoder();
109
+ let controller!: ReadableStreamDefaultController<Uint8Array>;
110
+ const stdout = new ReadableStream<Uint8Array>({
111
+ start(c) {
112
+ controller = c;
113
+ },
114
+ });
115
+ const push = (obj: unknown) => controller.enqueue(encoder.encode(`${JSON.stringify(obj)}\n`));
116
+
117
+ const transport: RemoteTransport = {
118
+ stdin: {
119
+ write(chunk: string) {
120
+ writes.push(chunk);
121
+ // What a #609 server does with "I can't decide": park, don't resolve.
122
+ if (chunk.includes('"unanswerable"')) {
123
+ push({
124
+ type: 'blocked',
125
+ sessionId: 'sess-1',
126
+ eventId: '42',
127
+ question: 'Apply breaking update for iptables (1.0.2+9 → 2.0.0+1)?',
128
+ key: 'module-upgrade:iptables.apply_breaking',
129
+ });
130
+ controller.close();
131
+ }
132
+ },
133
+ },
134
+ stdout,
135
+ kill() {},
136
+ exited: Promise.resolve(1),
137
+ };
138
+
139
+ push({ type: 'ready', protocolVersion: 1 });
140
+ push({
141
+ type: 'interview',
142
+ id: 'q1',
143
+ scope: 'module-upgrade:iptables',
144
+ key: 'apply_breaking',
145
+ kind: 'confirm',
146
+ message: 'Apply breaking update for iptables (1.0.2+9 → 2.0.0+1)?',
147
+ defaultValue: 'false',
148
+ });
149
+
150
+ // No `renderInterview` — exactly what runRemoteCapture used to do. bun test
151
+ // runs with a piped stdin, i.e. the MCP server's situation.
152
+ expect(process.stdin.isTTY).toBeFalsy();
153
+ const outcome = await runRemoteClient('ignored', ['module', 'update'], {
154
+ openTransport: () => transport,
155
+ out: { write() {} },
156
+ });
157
+
158
+ // Never an answer — that would consume the query.
159
+ expect(writes.find((w) => w.includes('"answer"'))).toBeUndefined();
160
+
161
+ const unanswerable = writes.find((w) => w.includes('"unanswerable"'));
162
+ expect(unanswerable).toBeDefined();
163
+ const parsed = JSON.parse(unanswerable as string) as { id: string; reason: string };
164
+ expect(parsed.id).toBe('q1');
165
+ expect(parsed.reason).toContain('module-upgrade:iptables.apply_breaking');
166
+
167
+ // And the caller is told where it stands rather than being handed a decline.
168
+ expect(outcome).toEqual({
169
+ status: 'blocked',
170
+ sessionId: 'sess-1',
171
+ eventId: '42',
172
+ question: 'Apply breaking update for iptables (1.0.2+9 → 2.0.0+1)?',
173
+ key: 'module-upgrade:iptables.apply_breaking',
174
+ });
175
+ });
package/src/api/serve.ts CHANGED
@@ -2,13 +2,19 @@
2
2
  * Remote API server — `celilo api-serve --principal=<id>` (Slices 1–3).
3
3
  *
4
4
  * Invoked as the sshd forced command for an enrolled key, so it starts already
5
- * authenticated as `principal`. Reads `command`/`answer` messages as NDJSON on
6
- * stdin/stdout, authorizes each command against the principal's grants
7
- * (deny-by-default), and — if allowed — runs it as a child `celilo` process,
8
- * translating the child's protocol-mode output into `progress`/`log` and a
9
- * terminal `result`. While a command runs, a remote responder bridges the event
10
- * bus to the wire so mid-run `interview`s are answered by the client. Every
11
- * attempt is audited to stderr.
5
+ * authenticated as `principal`. Reads `command`/`answer`/`unanswerable`/
6
+ * `attach`/`cancel` messages as NDJSON on stdin/stdout, authorizes each command
7
+ * against the principal's grants (deny-by-default), and — if allowed — runs it
8
+ * as a child `celilo` process, translating the child's protocol-mode output into
9
+ * `progress`/`log` and a terminal `result`. While a command runs, a remote
10
+ * responder bridges the event bus to the wire so mid-run `interview`s are
11
+ * answered by the client. Every attempt is audited to stderr.
12
+ *
13
+ * When the client cannot decide a question, the command **parks**: the query is
14
+ * left unanswered on the bus, the child stays alive past the end of the ssh
15
+ * session, its output is buffered in the session registry, and the client is
16
+ * told `blocked` with the session and query ids (celilo#609). Another responder
17
+ * answers by event id; a later `attach` collects the outcome.
12
18
  */
13
19
 
14
20
  import { createInterface } from 'node:readline';
@@ -16,16 +22,30 @@ import {
16
22
  API_PROTOCOL_VERSION,
17
23
  ClientMessageSchema,
18
24
  type ServerMessage,
25
+ ServerMessageSchema,
19
26
  translateOutputLine,
20
27
  } from '@celilo/core';
21
28
  import { parseArguments } from '../cli/parser';
22
29
  import { getEventBusPath } from '../config/paths';
23
30
  import { isAuthorized } from '../services/api-access';
31
+ import { EVENT_TYPES } from '../services/bus-interview';
24
32
  import { type WireInterview, startRemoteResponder } from '../services/remote-responder';
33
+ import {
34
+ SessionWriter,
35
+ abandonSession,
36
+ expiryReason,
37
+ readSession,
38
+ reapExpiredSessions,
39
+ replayOutput,
40
+ stillParked,
41
+ } from './sessions';
25
42
 
26
43
  /** Exit code returned to the client when authz denies a command. */
27
44
  const EXIT_PERMISSION_DENIED = 126;
28
45
 
46
+ /** How often an attached client polls a session's buffer for new output. */
47
+ const ATTACH_POLL_MS = 250;
48
+
29
49
  function send(msg: ServerMessage): void {
30
50
  process.stdout.write(`${JSON.stringify(msg)}\n`);
31
51
  }
@@ -44,8 +64,20 @@ function opOf(argv: string[]): { command: string; subcommand?: string; label: st
44
64
  return { command: parsed.command, subcommand: parsed.subcommand, label };
45
65
  }
46
66
 
67
+ /**
68
+ * Emit a message to the attached client (if any) AND to the session buffer, so
69
+ * output produced while nobody is attached is not lost.
70
+ */
71
+ function emit(session: SessionWriter, msg: ServerMessage, attached: () => boolean): void {
72
+ session.append(msg);
73
+ if (attached()) send(msg);
74
+ }
75
+
47
76
  /** Drain a child stream line-by-line, forwarding each line as a message. */
48
- async function pumpLines(stream: ReadableStream<Uint8Array>): Promise<void> {
77
+ async function pumpLines(
78
+ stream: ReadableStream<Uint8Array>,
79
+ forward: (msg: ServerMessage) => void,
80
+ ): Promise<void> {
49
81
  const decoder = new TextDecoder();
50
82
  let buffer = '';
51
83
  for await (const chunk of stream) {
@@ -54,17 +86,17 @@ async function pumpLines(stream: ReadableStream<Uint8Array>): Promise<void> {
54
86
  while (nl >= 0) {
55
87
  const line = buffer.slice(0, nl);
56
88
  buffer = buffer.slice(nl + 1);
57
- send(translateOutputLine(line));
89
+ forward(translateOutputLine(line));
58
90
  nl = buffer.indexOf('\n');
59
91
  }
60
92
  }
61
93
  if (buffer.length > 0) {
62
- send(translateOutputLine(buffer));
94
+ forward(translateOutputLine(buffer));
63
95
  }
64
96
  }
65
97
 
66
98
  /** Run an authorized command as a child; returns its exit code. */
67
- async function runCommand(argv: string[]): Promise<number> {
99
+ async function runCommand(argv: string[], forward: (msg: ServerMessage) => void): Promise<number> {
68
100
  // Re-invoke this same CLI as a child. The child is non-TTY (piped stdout) so
69
101
  // ProgressDisplay resolves to protocol mode and emits `[progress:*]` markers.
70
102
  const child = Bun.spawn([process.execPath, Bun.main, ...argv], {
@@ -73,49 +105,137 @@ async function runCommand(argv: string[]): Promise<number> {
73
105
  stderr: 'pipe',
74
106
  });
75
107
 
76
- await Promise.all([pumpLines(child.stdout), pumpLines(child.stderr)]);
108
+ await Promise.all([pumpLines(child.stdout, forward), pumpLines(child.stderr, forward)]);
77
109
  const exitCode = await child.exited;
78
- send({ type: 'result', success: exitCode === 0, exitCode });
110
+ forward({ type: 'result', success: exitCode === 0, exitCode });
79
111
  return exitCode;
80
112
  }
81
113
 
82
- async function handleCommand(
83
- principal: string,
84
- argv: string[],
85
- busDbPath: string,
86
- ask: (interview: WireInterview) => Promise<unknown>,
87
- ): Promise<void> {
88
- const { command, subcommand, label } = opOf(argv);
89
-
90
- if (!(await isAuthorized(principal, command, subcommand))) {
91
- send({ type: 'error', error: `permission denied: "${principal}" is not granted "${label}"` });
114
+ /**
115
+ * Replay a parked session's buffered output to the attaching client, then tail
116
+ * it until the session goes terminal. Deny-by-default on the owning principal.
117
+ */
118
+ async function handleAttach(principal: string, sessionId: string): Promise<void> {
119
+ const record = readSession(sessionId);
120
+ if (!record) {
121
+ send({ type: 'error', error: `no such session: ${sessionId}` });
122
+ send({ type: 'result', success: false, exitCode: 1 });
123
+ return;
124
+ }
125
+ if (record.principal !== principal) {
126
+ send({
127
+ type: 'error',
128
+ error: `permission denied: session ${sessionId} belongs to another principal`,
129
+ });
92
130
  send({ type: 'result', success: false, exitCode: EXIT_PERMISSION_DENIED });
93
- audit(principal, label, 'deny');
131
+ audit(principal, `attach:${sessionId}`, 'deny');
94
132
  return;
95
133
  }
134
+ audit(principal, `attach:${sessionId}`, 'allow');
96
135
 
97
- // Bridge the bus to the wire so mid-run interviews reach the client.
98
- const responder = startRemoteResponder({ busDbPath, ask, emittedBy: `api:${principal}` });
99
- try {
100
- const exitCode = await runCommand(argv);
101
- audit(principal, label, 'allow', exitCode);
102
- } finally {
103
- responder.close();
136
+ let cursor = 0;
137
+ for (;;) {
138
+ const { messages, next } = replayOutput(sessionId, cursor);
139
+ cursor = next;
140
+ for (const line of messages) {
141
+ const parsed = ServerMessageSchema.safeParse(JSON.parse(line));
142
+ if (!parsed.success) continue;
143
+ send(parsed.data);
144
+ if (parsed.data.type === 'result') return;
145
+ }
146
+ const current = readSession(sessionId);
147
+ if (!current) return;
148
+ // Still waiting on a decision → say so rather than tailing forever. Asked of
149
+ // the bus, not of the record: the answer arrives from a third party the
150
+ // owning process never hears from, so its `state` lags.
151
+ if (current.parkedEventId && stillParked(current, getEventBusPath())) {
152
+ send({
153
+ type: 'blocked',
154
+ sessionId,
155
+ eventId: current.parkedEventId,
156
+ question: current.question ?? '',
157
+ key: current.questionKey ?? undefined,
158
+ });
159
+ return;
160
+ }
161
+ await new Promise((r) => setTimeout(r, ATTACH_POLL_MS));
104
162
  }
105
163
  }
106
164
 
165
+ /** Abandon a parked session on the operator's say-so, before its TTL. */
166
+ function handleCancel(principal: string, sessionId: string): void {
167
+ const record = readSession(sessionId);
168
+ if (!record) {
169
+ send({ type: 'error', error: `no such session: ${sessionId}` });
170
+ send({ type: 'result', success: false, exitCode: 1 });
171
+ return;
172
+ }
173
+ if (record.principal !== principal) {
174
+ send({
175
+ type: 'error',
176
+ error: `permission denied: session ${sessionId} belongs to another principal`,
177
+ });
178
+ send({ type: 'result', success: false, exitCode: EXIT_PERMISSION_DENIED });
179
+ audit(principal, `cancel:${sessionId}`, 'deny');
180
+ return;
181
+ }
182
+ abandonSession(record, {
183
+ busDbPath: getEventBusPath(),
184
+ reason: `Session ${sessionId} was cancelled by ${principal} with "${record.question ?? 'the question'}" still unanswered.`,
185
+ emittedBy: `api:${principal}`,
186
+ });
187
+ audit(principal, `cancel:${sessionId}`, 'allow');
188
+ send({ type: 'result', success: true, exitCode: 0 });
189
+ }
190
+
107
191
  export async function apiServeMode(principal: string): Promise<void> {
108
192
  send({ type: 'ready', protocolVersion: API_PROTOCOL_VERSION });
109
193
 
110
194
  const busDbPath = getEventBusPath();
111
- const pendingAnswers = new Map<string, (value: unknown) => void>();
195
+ // A session whose owning process died (reboot, OOM) would otherwise stay
196
+ // parked forever, holding whatever its command holds.
197
+ reapExpiredSessions({ busDbPath });
198
+
199
+ // When the ssh client goes away, sshd hangs up its forced command. Ignoring
200
+ // SIGHUP is what lets a parked command outlive the session that started it —
201
+ // the TTL reaper, not the transport, is what bounds it now.
202
+ process.on('SIGHUP', () => {});
203
+
204
+ const pendingAnswers = new Map<
205
+ string,
206
+ { resolve: (value: unknown) => void; reject: (error: Error) => void; interview: WireInterview }
207
+ >();
208
+
209
+ /** Set while a command is running; the parked child outlives the transport. */
210
+ const live: { session: SessionWriter | null; clientAttached: boolean } = {
211
+ session: null,
212
+ clientAttached: true,
213
+ };
214
+ let commandRunning: Promise<void> | null = null;
112
215
 
113
216
  const ask = (interview: WireInterview): Promise<unknown> =>
114
- new Promise((resolve) => {
115
- pendingAnswers.set(interview.id, resolve);
217
+ new Promise((resolve, reject) => {
218
+ pendingAnswers.set(interview.id, { resolve, reject, interview });
219
+ if (!live.clientAttached) {
220
+ // Nobody to ask. Re-park on THIS question: a detached command that was
221
+ // answered and moved on is now waiting on a *different* event, and a
222
+ // record still naming the previous one is actively wrong — it hides the
223
+ // live question from `events list-unanswered`, and it aims the TTL
224
+ // reaper at a question that was already decided, so the reaper retires
225
+ // the session while the child is still alive and blocked.
226
+ live.session?.park({
227
+ eventId: interview.id,
228
+ eventType: EVENT_TYPES.interviewRequired(interview.scope, interview.key),
229
+ question: interview.message,
230
+ questionKey: `${interview.scope}.${interview.key}`,
231
+ });
232
+ return;
233
+ }
116
234
  send({
117
235
  type: 'interview',
118
236
  id: interview.id,
237
+ scope: interview.scope,
238
+ key: interview.key,
119
239
  kind: interview.kind,
120
240
  message: interview.message,
121
241
  description: interview.description,
@@ -126,6 +246,46 @@ export async function apiServeMode(principal: string): Promise<void> {
126
246
  });
127
247
  });
128
248
 
249
+ const handleCommand = async (argv: string[]): Promise<void> => {
250
+ const { command, subcommand, label } = opOf(argv);
251
+
252
+ if (!(await isAuthorized(principal, command, subcommand))) {
253
+ send({ type: 'error', error: `permission denied: "${principal}" is not granted "${label}"` });
254
+ send({ type: 'result', success: false, exitCode: EXIT_PERMISSION_DENIED });
255
+ audit(principal, label, 'deny');
256
+ return;
257
+ }
258
+
259
+ const writer = SessionWriter.create({ principal, argv });
260
+ live.session = writer;
261
+ const reaper = setTimeout(
262
+ () => {
263
+ if (writer.current.state !== 'parked') return;
264
+ abandonSession(writer.current, {
265
+ busDbPath,
266
+ reason: expiryReason(writer.current),
267
+ emittedBy: `api:${principal}`,
268
+ });
269
+ writer.finish('abandoned');
270
+ },
271
+ Math.max(0, writer.current.expiresAt - Date.now()),
272
+ );
273
+
274
+ // Bridge the bus to the wire so mid-run interviews reach the client.
275
+ const responder = startRemoteResponder({ busDbPath, ask, emittedBy: `api:${principal}` });
276
+ try {
277
+ const exitCode = await runCommand(argv, (msg) =>
278
+ emit(writer, msg, () => live.clientAttached),
279
+ );
280
+ writer.finish('finished');
281
+ audit(principal, label, 'allow', exitCode);
282
+ } finally {
283
+ clearTimeout(reaper);
284
+ responder.close();
285
+ live.session = null;
286
+ }
287
+ };
288
+
129
289
  const rl = createInterface({ input: process.stdin, terminal: false });
130
290
 
131
291
  for await (const line of rl) {
@@ -140,20 +300,64 @@ export async function apiServeMode(principal: string): Promise<void> {
140
300
  }
141
301
 
142
302
  if (msg.type === 'answer') {
143
- const resolve = pendingAnswers.get(msg.id);
144
- if (resolve) {
303
+ const pending = pendingAnswers.get(msg.id);
304
+ if (pending) {
145
305
  pendingAnswers.delete(msg.id);
146
- resolve(msg.value);
306
+ live.session?.unpark();
307
+ pending.resolve(msg.value);
308
+ }
309
+ continue;
310
+ }
311
+
312
+ if (msg.type === 'unanswerable') {
313
+ // The client cannot decide. Do NOT resolve the question — park on it and
314
+ // tell the client where it stands. The responder, seeing `ask` reject,
315
+ // emits nothing on the bus, so the query is still there to be answered.
316
+ const pending = pendingAnswers.get(msg.id);
317
+ if (!pending) continue;
318
+ pendingAnswers.delete(msg.id);
319
+ const { interview } = pending;
320
+ pending.reject(new Error(msg.reason));
321
+ if (live.session) {
322
+ live.session.park({
323
+ eventId: msg.id,
324
+ eventType: EVENT_TYPES.interviewRequired(interview.scope, interview.key),
325
+ question: interview.message,
326
+ questionKey: `${interview.scope}.${interview.key}`,
327
+ });
328
+ send({
329
+ type: 'blocked',
330
+ sessionId: live.session.id,
331
+ eventId: msg.id,
332
+ question: interview.message,
333
+ key: `${interview.scope}.${interview.key}`,
334
+ });
147
335
  }
148
336
  continue;
149
337
  }
150
338
 
339
+ if (msg.type === 'attach') {
340
+ await handleAttach(principal, msg.sessionId);
341
+ continue;
342
+ }
343
+
344
+ if (msg.type === 'cancel') {
345
+ handleCancel(principal, msg.sessionId);
346
+ continue;
347
+ }
348
+
151
349
  if (msg.type === 'command') {
152
350
  // Fire-and-forget so the read loop keeps consuming `answer` messages while
153
351
  // the command runs — mid-run interviews are answered in-flight.
154
- void handleCommand(principal, msg.argv, busDbPath, ask);
352
+ commandRunning = handleCommand(msg.argv);
353
+ void commandRunning;
155
354
  }
156
355
  }
157
356
 
357
+ // stdin closed: the client is gone. A parked command must NOT die with it —
358
+ // that is the bug. Stay alive (output goes to the session buffer) until the
359
+ // command finishes or the reaper abandons it.
360
+ live.clientAttached = false;
361
+ if (commandRunning) await commandRunning;
158
362
  process.exit(0);
159
363
  }