@celilo/cli 0.18.0 → 0.20.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CELILO_SUBSYSTEMS.md +4 -2
- package/package.json +4 -4
- package/src/api/remote-client.test.ts +86 -2
- package/src/api/serve.ts +242 -38
- package/src/api/sessions.test.ts +196 -0
- package/src/api/sessions.ts +278 -0
- package/src/cli/commands/apt-upgrade.test.ts +20 -1
- package/src/cli/commands/apt-upgrade.ts +12 -2
- package/src/cli/commands/backup-sweep.ts +25 -9
- package/src/cli/commands/events.ts +150 -4
- package/src/cli/commands/module-update.test.ts +72 -1
- package/src/cli/commands/module-update.ts +68 -22
- package/src/cli/commands/system-migrate.test.ts +56 -0
- package/src/cli/commands/system-migrate.ts +52 -4
- package/src/cli/completion.ts +2 -0
- package/src/cli/index.ts +27 -3
- package/src/db/migration-status.test.ts +114 -0
- package/src/db/migration-status.ts +78 -0
- package/src/db/schema-introspection.ts +8 -1
- package/src/services/backup-metadata.ts +17 -0
- package/src/services/backup-staging.test.ts +98 -0
- package/src/services/backup-staging.ts +73 -1
- package/src/services/backup-sweep.test.ts +15 -0
- package/src/services/backup-sweep.ts +17 -1
- package/src/services/bus-interview-park.test.ts +179 -0
- package/src/services/bus-interview.ts +17 -6
- package/src/services/events-daemon.test.ts +244 -0
- package/src/services/events-daemon.ts +295 -8
- package/src/services/fleet-checks.test.ts +75 -4
- package/src/services/fleet-checks.ts +82 -12
- package/src/services/interview-errors.ts +37 -0
- package/src/services/remote-responder.test.ts +83 -0
- package/src/services/remote-responder.ts +31 -10
- package/src/services/responder-probe.ts +3 -1
package/CELILO_SUBSYSTEMS.md
CHANGED
|
@@ -232,7 +232,9 @@ Creation, scheduling and freshness. A module declares an `on_backup` hook and a
|
|
|
232
232
|
|
|
233
233
|
## Events
|
|
234
234
|
|
|
235
|
-
- **
|
|
235
|
+
- **Migration interrogation** — `apps/celilo/src/db/migration-status.ts` — `getMigrationStatus(sqlite, migrationsFolder)` reports applied count, latest applied migration BY TAG, and pending ones by name (joining `__drizzle_migrations.created_at` to drizzle journal `when`, which is exact). Surface: `celilo system migrate --status`, which opens the DB **read-only** on purpose — `getDb()` auto-migrates on open, so a status routed through it would repair what it claims to report and could never say "pending". Paired with `findSchemaDrift` (`db/schema-introspection.ts`), which is column-aware: a table COUNT cannot distinguish "the column migration applied" from "nothing happened", which is why a rollout asserting `backups.pid` had to reach for `sqlite3` over SSH. `checkSchemaDrift` (`services/fleet-checks.ts`) fails on a missing table, a missing column, OR an unapplied journal migration, and its summary names tables AND columns so the operator can see what was checked.
|
|
236
|
+
- **Event bus** — `packages/event-bus/src/index.ts` — `Bus`, `openBus`, `defineEvents`, `defineHandler`, `runDispatcher`, pattern matching + timer ticks (`emitDueTimerTicks`, `retentionSweep`). **Exactly one dispatcher per bus**: `runDispatcher` refuses to start while another dispatcher's process is alive (`assertSoleDispatcher` in `dispatcher.ts`, liveness via `bus.liveDispatchers()` — `kill(pid,0)`, not heartbeat age, since a tick blocks for as long as its slowest handler). The exclusion is here rather than in the systemd unit because a stranded dispatcher can sit outside the unit's cgroup where `KillMode` cannot reach it (#580). `bus.health()` reports `dispatcherCount`/`dispatchers` and a `duplicate_dispatcher` status; `checkDispatcher` (`services/fleet-checks.ts`) fails on more than one. **Supervision** (`services/events-daemon.ts`): the unit is named `celilo-events.service` in BOTH the user and system scope, so the two are indistinguishable in every operator-facing string — `install-daemon` (which defaults to **user** scope) therefore refuses when the other scope's unit exists, and `checkDispatcher` compares the live pid against each installed unit's `MainPID` (`unitMainPid`) rather than merely testing that a unit file exists, since a file nobody is running is not supervision (#610).
|
|
237
|
+
- **Dispatcher supervision (install / restart)** — `apps/celilo/src/services/events-daemon.ts` — `installDaemon`/`planDaemonInstall`/`uninstallDaemon`/`readInstalledUnit` write the systemd unit or launchd plist without touching supervisor state, and `restartDaemon` (+ pure `orphanDispatcherPids`, `resolveRestartScope`, `supervisorCommands`) is the one verb that DOES cycle it. CLI: `celilo events install-daemon|uninstall-daemon|show-daemon|restart-daemon [--system]`. `restart-daemon` exists because the dispatcher runs the code it LOADED: celilo-mgr sat 9 days on event-bus v0.1.8 after apt installed v0.2.0, running the very bug the release fixed (celilo#604). Two things make it non-trivial and both are load-bearing: (1) it stops any live dispatcher the supervisor does not own first — an ORPHAN (PPID 1) is invisible to `systemctl restart`, and `assertSoleDispatcher` then crash-loops the unit while the old code keeps serving; (2) it verifies on the BUS that a NEW pid is live reporting `BUS_VERSION`, never off systemctl's exit code, which returns 0 into exactly that crash loop. System scope shells `sudo systemctl` (the unit is root-owned; celilo runs unprivileged), covered by the scoped `/etc/sudoers.d/celilo-events-restart` conffile that `celilo-bootstrap` ships — a unit test asserts the argv and the grant cannot drift apart.
|
|
236
238
|
|
|
237
239
|
## Remote API (drive the CLI over the wire)
|
|
238
240
|
|
|
@@ -246,7 +248,7 @@ Run any celilo command on celilo-mgr over SSH instead of screen-scraping `ssh <h
|
|
|
246
248
|
- **Access control** — `apps/celilo/src/services/api-access.ts` — `grantPrincipal`, `isAuthorized` (deny-by-default, `command:subcommand` grants), `renderAuthorizedKeys`. Table: `api_principals` (`apps/celilo/src/db/schema.ts`). CLI: `apps/celilo/src/cli/commands/api.ts` (`api grant|list|revoke|authorized-keys|key new`).
|
|
247
249
|
- **Mid-run interview bridge (`kind:daemon` responder)** — `apps/celilo/src/services/remote-responder.ts` — `startRemoteResponder` bridges bus `interview.required.*` ↔ wire.
|
|
248
250
|
- **Server provisioning** — the `celilo-bootstrap` deb (`packaging/celilo-bootstrap/scripts/postinst`) creates the non-root `celilo-api` landing account + sshd; membership in the `celilo` group + `/etc/sudoers.d/celilo` (`!use_pty`) gives api-serve DB access via the wrapper's sudo-drop.
|
|
249
|
-
- **Self-upgrade (apt)** — `celilo apt-upgrade` (`apps/celilo/src/cli/commands/apt-upgrade.ts`) upgrades the deb-installed `celilo`/`celilo-bootstrap` packages (`apt-get update` → `--only-upgrade install`) then spawns a fresh `celilo system migrate` (ISS-0100). It's the RW target behind the MCP's registry-derived `celilo_apt_upgrade` tool; the celilo user's two apt invocations are scoped-sudo'd by `/etc/sudoers.d/celilo-apt-upgrade`, shipped by `celilo-bootstrap`. **This upgrades celilo ITSELF — not the modules it manages. For those, see Module auto-upgrade below; the two are routinely confused.**
|
|
251
|
+
- **Self-upgrade (apt)** — `celilo apt-upgrade` (`apps/celilo/src/cli/commands/apt-upgrade.ts`) upgrades the deb-installed `celilo`/`celilo-bootstrap` packages (`apt-get update` → `--only-upgrade install`) then spawns a fresh `celilo system migrate` (ISS-0100), then `celilo events restart-daemon` so the dispatcher actually runs the code just installed — a failure there fails the whole command and names which steps DID complete, because "upgraded" while the dispatcher serves stale code is the silent state celilo#604 documents. It's the RW target behind the MCP's registry-derived `celilo_apt_upgrade` tool; the celilo user's two apt invocations are scoped-sudo'd by `/etc/sudoers.d/celilo-apt-upgrade`, shipped by `celilo-bootstrap`. **This upgrades celilo ITSELF — not the modules it manages. For those, see Module auto-upgrade below; the two are routinely confused.**
|
|
250
252
|
- **Module auto-upgrade (registry-poll CD)** — the *pull* half of continuous deployment: celilo-mgr polls the registry and upgrades opted-in modules unattended. Spec: `openspec/specs/module-auto-upgrade/spec.md`. Entry points: `apps/celilo/src/cli/commands/module-upgrade.ts` — `runRegistryPoll` (the `--poll` path), `selectPollTargets` (pure: `autoUpgrade && latest && change ∉ {up-to-date, ahead}`), `upgradeOneModule` (update → backup → deploy → verify), `needsPreUpgradeBackup`, `pickAutoUpgrade`/`pickUpgradePolicy` (both fail closed/safe); `classifyVersionChange` in `module-update.ts` (treats a registry `+N` revision as a patch); `resolveDeployPosture` in `apps/celilo/src/services/deploy-posture.ts`. Trigger: celilo-mgmt's `registry-poll` subscription (`modules/celilo-mgmt/manifest.yml`) on `timer.tick.15m` with handler **`celilo module upgrade --poll`** — the flag is REQUIRED, since the dispatcher appends the event id positionally and a bare handler would consume it as the optional module name (silent: 3108 deliveries, 0 successes). Operator controls are framework config keys settable on ANY module (`FRAMEWORK_CONFIG_KEYS` in `module-config.ts`): `auto_upgrade` (opt-in, default false) and `upgrade_policy` (`by-semver`|`always-safe`|`always-fast`), validated at set time because both readers fail open. ⚠️ `always-safe` guarantees safe *posture*, NOT a backup — `needsPreUpgradeBackup` also requires the TARGET manifest to declare an `on_backup` hook, else it warns and proceeds. Confirm a data-bearing module declares `on_backup` before enabling `auto_upgrade` on it. The *build* half (app CI publishing a `.netapp` on merge) is not yet shipped — `openspec/changes/build-bus-poll-cd`.
|
|
251
253
|
- **MCP service (`@celilo/mcp`)** — `packages/mcp/src/` — an operator-facing stdio MCP server (official `@modelcontextprotocol/sdk`, bin `celilo-mcp`) that drives a remote celilo server over the Remote API for an AI client. Two-item config (`config.ts`: `server` + `defaultUser`, env or `~/.config/celilo-mcp/config.json`). Dual-principal auth (`auth.ts`: `celilo-mcp auth setup` enrolls read-only `celilo-mcp-ro` + full `celilo-mcp-rw` ed25519 keypairs, prints the exact `celilo api grant` lines the operator runs server-side). Transport (`transport.ts`): reuses `@celilo/core` `runRemoteClient`, selecting the principal by `ssh -i <key>` and capturing structured output. Tool surface is generated LIVE from the server's command registry — `registry-fetch.ts` fetches `celilo commands --json` (+ `service list --json` for configured providers) over the RO principal on connect; `tools-from-registry.ts` (pure) projects that into one tool per runnable leaf, grouped by top-level command (`celilo_module_*`, `celilo_proxmox_*`, …), each with a Zod input schema from the leaf's args/flags and a read/write tag → RO/RW routing, plus a generic `celilo_run` escape hatch. Auto-detect hides provider-gated groups (e.g. `celilo_proxmox_*` until a Proxmox service is configured) and re-detects on a timer, emitting `notifications/tools/list_changed` when the surface changes. Coverage gate (`tests/coverage.test.ts`) asserts every registry leaf maps to a tool. Composite RO troubleshooting tools (`troubleshoot.ts` pure correlation + `troubleshoot-tools.ts` thin adapters): `celilo_assess_module <id>` and `celilo_fleet_status` correlate `celilo audit --json` (the drift backbone) with the `module list --json` roster into a per-module / fleet-wide verdict. Design: `openspec/changes/celilo-mcp-service/proposal.md`. (Distinct from the dev/ops `@celilo/mcp-server` below.)
|
|
252
254
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@celilo/cli",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.20.0",
|
|
4
4
|
"description": "Celilo — home lab orchestration CLI",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -57,10 +57,10 @@
|
|
|
57
57
|
},
|
|
58
58
|
"dependencies": {
|
|
59
59
|
"@aws-sdk/client-s3": "^3.1024.0",
|
|
60
|
-
"@celilo/capabilities": "^0.
|
|
60
|
+
"@celilo/capabilities": "^0.11.0",
|
|
61
61
|
"@celilo/cli-display": "^0.1.10",
|
|
62
|
-
"@celilo/core": "^0.
|
|
63
|
-
"@celilo/event-bus": "^0.
|
|
62
|
+
"@celilo/core": "^0.5.0",
|
|
63
|
+
"@celilo/event-bus": "^0.3.0",
|
|
64
64
|
"@clack/prompts": "^1.1.0",
|
|
65
65
|
"ajv": "^8.18.0",
|
|
66
66
|
"drizzle-orm": "^0.36.4",
|
|
@@ -72,7 +72,7 @@ test('renders a forwarded interview and sends the answer back', async () => {
|
|
|
72
72
|
push({ type: 'interview', id: 'q1', kind: 'text', message: 'Hostname?' });
|
|
73
73
|
|
|
74
74
|
const seen: Array<{ id: string }> = [];
|
|
75
|
-
const
|
|
75
|
+
const outcome = await runRemoteClient('ignored', ['module', 'deploy', 'site'], {
|
|
76
76
|
openTransport: () => transport,
|
|
77
77
|
out: { write() {} },
|
|
78
78
|
renderInterview: async (iv) => {
|
|
@@ -81,7 +81,7 @@ test('renders a forwarded interview and sends the answer back', async () => {
|
|
|
81
81
|
},
|
|
82
82
|
});
|
|
83
83
|
|
|
84
|
-
expect(
|
|
84
|
+
expect(outcome).toEqual({ status: 'result', exitCode: 0 });
|
|
85
85
|
expect(seen).toHaveLength(1);
|
|
86
86
|
expect(seen[0].id).toBe('q1');
|
|
87
87
|
expect(writes.some((w) => w.includes('"command"'))).toBe(true);
|
|
@@ -89,3 +89,87 @@ test('renders a forwarded interview and sends the answer back', async () => {
|
|
|
89
89
|
expect(answer).toBeDefined();
|
|
90
90
|
expect(JSON.parse(answer as string)).toEqual({ type: 'answer', id: 'q1', value: 'myhost' });
|
|
91
91
|
});
|
|
92
|
+
|
|
93
|
+
/**
|
|
94
|
+
* Regression for the fabricated "operator declined".
|
|
95
|
+
*
|
|
96
|
+
* The default renderer prompts with clack, which reads keypresses off stdin
|
|
97
|
+
* whether or not stdin is a terminal. Driven over the MCP (stdin = the JSON-RPC
|
|
98
|
+
* stream) the next newline submitted the prompt at its `initialValue` — the
|
|
99
|
+
* question's `defaultValue` — so a breaking update nobody saw came back as a
|
|
100
|
+
* considered "no". With no terminal the client must say it cannot answer.
|
|
101
|
+
*
|
|
102
|
+
* And it must say so as `unanswerable`, NOT as an `answer` of any shape: an
|
|
103
|
+
* `answer` is what consumes the query and destroys a question nobody decided
|
|
104
|
+
* (celilo#609). The server parks and replies `blocked`.
|
|
105
|
+
*/
|
|
106
|
+
test('no TTY and no renderer → sends unanswerable and returns blocked, never an answer', async () => {
|
|
107
|
+
const writes: string[] = [];
|
|
108
|
+
const encoder = new TextEncoder();
|
|
109
|
+
let controller!: ReadableStreamDefaultController<Uint8Array>;
|
|
110
|
+
const stdout = new ReadableStream<Uint8Array>({
|
|
111
|
+
start(c) {
|
|
112
|
+
controller = c;
|
|
113
|
+
},
|
|
114
|
+
});
|
|
115
|
+
const push = (obj: unknown) => controller.enqueue(encoder.encode(`${JSON.stringify(obj)}\n`));
|
|
116
|
+
|
|
117
|
+
const transport: RemoteTransport = {
|
|
118
|
+
stdin: {
|
|
119
|
+
write(chunk: string) {
|
|
120
|
+
writes.push(chunk);
|
|
121
|
+
// What a #609 server does with "I can't decide": park, don't resolve.
|
|
122
|
+
if (chunk.includes('"unanswerable"')) {
|
|
123
|
+
push({
|
|
124
|
+
type: 'blocked',
|
|
125
|
+
sessionId: 'sess-1',
|
|
126
|
+
eventId: '42',
|
|
127
|
+
question: 'Apply breaking update for iptables (1.0.2+9 → 2.0.0+1)?',
|
|
128
|
+
key: 'module-upgrade:iptables.apply_breaking',
|
|
129
|
+
});
|
|
130
|
+
controller.close();
|
|
131
|
+
}
|
|
132
|
+
},
|
|
133
|
+
},
|
|
134
|
+
stdout,
|
|
135
|
+
kill() {},
|
|
136
|
+
exited: Promise.resolve(1),
|
|
137
|
+
};
|
|
138
|
+
|
|
139
|
+
push({ type: 'ready', protocolVersion: 1 });
|
|
140
|
+
push({
|
|
141
|
+
type: 'interview',
|
|
142
|
+
id: 'q1',
|
|
143
|
+
scope: 'module-upgrade:iptables',
|
|
144
|
+
key: 'apply_breaking',
|
|
145
|
+
kind: 'confirm',
|
|
146
|
+
message: 'Apply breaking update for iptables (1.0.2+9 → 2.0.0+1)?',
|
|
147
|
+
defaultValue: 'false',
|
|
148
|
+
});
|
|
149
|
+
|
|
150
|
+
// No `renderInterview` — exactly what runRemoteCapture used to do. bun test
|
|
151
|
+
// runs with a piped stdin, i.e. the MCP server's situation.
|
|
152
|
+
expect(process.stdin.isTTY).toBeFalsy();
|
|
153
|
+
const outcome = await runRemoteClient('ignored', ['module', 'update'], {
|
|
154
|
+
openTransport: () => transport,
|
|
155
|
+
out: { write() {} },
|
|
156
|
+
});
|
|
157
|
+
|
|
158
|
+
// Never an answer — that would consume the query.
|
|
159
|
+
expect(writes.find((w) => w.includes('"answer"'))).toBeUndefined();
|
|
160
|
+
|
|
161
|
+
const unanswerable = writes.find((w) => w.includes('"unanswerable"'));
|
|
162
|
+
expect(unanswerable).toBeDefined();
|
|
163
|
+
const parsed = JSON.parse(unanswerable as string) as { id: string; reason: string };
|
|
164
|
+
expect(parsed.id).toBe('q1');
|
|
165
|
+
expect(parsed.reason).toContain('module-upgrade:iptables.apply_breaking');
|
|
166
|
+
|
|
167
|
+
// And the caller is told where it stands rather than being handed a decline.
|
|
168
|
+
expect(outcome).toEqual({
|
|
169
|
+
status: 'blocked',
|
|
170
|
+
sessionId: 'sess-1',
|
|
171
|
+
eventId: '42',
|
|
172
|
+
question: 'Apply breaking update for iptables (1.0.2+9 → 2.0.0+1)?',
|
|
173
|
+
key: 'module-upgrade:iptables.apply_breaking',
|
|
174
|
+
});
|
|
175
|
+
});
|
package/src/api/serve.ts
CHANGED
|
@@ -2,13 +2,19 @@
|
|
|
2
2
|
* Remote API server — `celilo api-serve --principal=<id>` (Slices 1–3).
|
|
3
3
|
*
|
|
4
4
|
* Invoked as the sshd forced command for an enrolled key, so it starts already
|
|
5
|
-
* authenticated as `principal`. Reads `command`/`answer
|
|
6
|
-
* stdin/stdout, authorizes each command
|
|
7
|
-
* (deny-by-default), and — if allowed — runs it
|
|
8
|
-
* translating the child's protocol-mode output into
|
|
9
|
-
* terminal `result`. While a command runs, a remote
|
|
10
|
-
* bus to the wire so mid-run `interview`s are
|
|
11
|
-
* attempt is audited to stderr.
|
|
5
|
+
* authenticated as `principal`. Reads `command`/`answer`/`unanswerable`/
|
|
6
|
+
* `attach`/`cancel` messages as NDJSON on stdin/stdout, authorizes each command
|
|
7
|
+
* against the principal's grants (deny-by-default), and — if allowed — runs it
|
|
8
|
+
* as a child `celilo` process, translating the child's protocol-mode output into
|
|
9
|
+
* `progress`/`log` and a terminal `result`. While a command runs, a remote
|
|
10
|
+
* responder bridges the event bus to the wire so mid-run `interview`s are
|
|
11
|
+
* answered by the client. Every attempt is audited to stderr.
|
|
12
|
+
*
|
|
13
|
+
* When the client cannot decide a question, the command **parks**: the query is
|
|
14
|
+
* left unanswered on the bus, the child stays alive past the end of the ssh
|
|
15
|
+
* session, its output is buffered in the session registry, and the client is
|
|
16
|
+
* told `blocked` with the session and query ids (celilo#609). Another responder
|
|
17
|
+
* answers by event id; a later `attach` collects the outcome.
|
|
12
18
|
*/
|
|
13
19
|
|
|
14
20
|
import { createInterface } from 'node:readline';
|
|
@@ -16,16 +22,30 @@ import {
|
|
|
16
22
|
API_PROTOCOL_VERSION,
|
|
17
23
|
ClientMessageSchema,
|
|
18
24
|
type ServerMessage,
|
|
25
|
+
ServerMessageSchema,
|
|
19
26
|
translateOutputLine,
|
|
20
27
|
} from '@celilo/core';
|
|
21
28
|
import { parseArguments } from '../cli/parser';
|
|
22
29
|
import { getEventBusPath } from '../config/paths';
|
|
23
30
|
import { isAuthorized } from '../services/api-access';
|
|
31
|
+
import { EVENT_TYPES } from '../services/bus-interview';
|
|
24
32
|
import { type WireInterview, startRemoteResponder } from '../services/remote-responder';
|
|
33
|
+
import {
|
|
34
|
+
SessionWriter,
|
|
35
|
+
abandonSession,
|
|
36
|
+
expiryReason,
|
|
37
|
+
readSession,
|
|
38
|
+
reapExpiredSessions,
|
|
39
|
+
replayOutput,
|
|
40
|
+
stillParked,
|
|
41
|
+
} from './sessions';
|
|
25
42
|
|
|
26
43
|
/** Exit code returned to the client when authz denies a command. */
|
|
27
44
|
const EXIT_PERMISSION_DENIED = 126;
|
|
28
45
|
|
|
46
|
+
/** How often an attached client polls a session's buffer for new output. */
|
|
47
|
+
const ATTACH_POLL_MS = 250;
|
|
48
|
+
|
|
29
49
|
function send(msg: ServerMessage): void {
|
|
30
50
|
process.stdout.write(`${JSON.stringify(msg)}\n`);
|
|
31
51
|
}
|
|
@@ -44,8 +64,20 @@ function opOf(argv: string[]): { command: string; subcommand?: string; label: st
|
|
|
44
64
|
return { command: parsed.command, subcommand: parsed.subcommand, label };
|
|
45
65
|
}
|
|
46
66
|
|
|
67
|
+
/**
|
|
68
|
+
* Emit a message to the attached client (if any) AND to the session buffer, so
|
|
69
|
+
* output produced while nobody is attached is not lost.
|
|
70
|
+
*/
|
|
71
|
+
function emit(session: SessionWriter, msg: ServerMessage, attached: () => boolean): void {
|
|
72
|
+
session.append(msg);
|
|
73
|
+
if (attached()) send(msg);
|
|
74
|
+
}
|
|
75
|
+
|
|
47
76
|
/** Drain a child stream line-by-line, forwarding each line as a message. */
|
|
48
|
-
async function pumpLines(
|
|
77
|
+
async function pumpLines(
|
|
78
|
+
stream: ReadableStream<Uint8Array>,
|
|
79
|
+
forward: (msg: ServerMessage) => void,
|
|
80
|
+
): Promise<void> {
|
|
49
81
|
const decoder = new TextDecoder();
|
|
50
82
|
let buffer = '';
|
|
51
83
|
for await (const chunk of stream) {
|
|
@@ -54,17 +86,17 @@ async function pumpLines(stream: ReadableStream<Uint8Array>): Promise<void> {
|
|
|
54
86
|
while (nl >= 0) {
|
|
55
87
|
const line = buffer.slice(0, nl);
|
|
56
88
|
buffer = buffer.slice(nl + 1);
|
|
57
|
-
|
|
89
|
+
forward(translateOutputLine(line));
|
|
58
90
|
nl = buffer.indexOf('\n');
|
|
59
91
|
}
|
|
60
92
|
}
|
|
61
93
|
if (buffer.length > 0) {
|
|
62
|
-
|
|
94
|
+
forward(translateOutputLine(buffer));
|
|
63
95
|
}
|
|
64
96
|
}
|
|
65
97
|
|
|
66
98
|
/** Run an authorized command as a child; returns its exit code. */
|
|
67
|
-
async function runCommand(argv: string[]): Promise<number> {
|
|
99
|
+
async function runCommand(argv: string[], forward: (msg: ServerMessage) => void): Promise<number> {
|
|
68
100
|
// Re-invoke this same CLI as a child. The child is non-TTY (piped stdout) so
|
|
69
101
|
// ProgressDisplay resolves to protocol mode and emits `[progress:*]` markers.
|
|
70
102
|
const child = Bun.spawn([process.execPath, Bun.main, ...argv], {
|
|
@@ -73,49 +105,137 @@ async function runCommand(argv: string[]): Promise<number> {
|
|
|
73
105
|
stderr: 'pipe',
|
|
74
106
|
});
|
|
75
107
|
|
|
76
|
-
await Promise.all([pumpLines(child.stdout), pumpLines(child.stderr)]);
|
|
108
|
+
await Promise.all([pumpLines(child.stdout, forward), pumpLines(child.stderr, forward)]);
|
|
77
109
|
const exitCode = await child.exited;
|
|
78
|
-
|
|
110
|
+
forward({ type: 'result', success: exitCode === 0, exitCode });
|
|
79
111
|
return exitCode;
|
|
80
112
|
}
|
|
81
113
|
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
114
|
+
/**
|
|
115
|
+
* Replay a parked session's buffered output to the attaching client, then tail
|
|
116
|
+
* it until the session goes terminal. Deny-by-default on the owning principal.
|
|
117
|
+
*/
|
|
118
|
+
async function handleAttach(principal: string, sessionId: string): Promise<void> {
|
|
119
|
+
const record = readSession(sessionId);
|
|
120
|
+
if (!record) {
|
|
121
|
+
send({ type: 'error', error: `no such session: ${sessionId}` });
|
|
122
|
+
send({ type: 'result', success: false, exitCode: 1 });
|
|
123
|
+
return;
|
|
124
|
+
}
|
|
125
|
+
if (record.principal !== principal) {
|
|
126
|
+
send({
|
|
127
|
+
type: 'error',
|
|
128
|
+
error: `permission denied: session ${sessionId} belongs to another principal`,
|
|
129
|
+
});
|
|
92
130
|
send({ type: 'result', success: false, exitCode: EXIT_PERMISSION_DENIED });
|
|
93
|
-
audit(principal,
|
|
131
|
+
audit(principal, `attach:${sessionId}`, 'deny');
|
|
94
132
|
return;
|
|
95
133
|
}
|
|
134
|
+
audit(principal, `attach:${sessionId}`, 'allow');
|
|
96
135
|
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
136
|
+
let cursor = 0;
|
|
137
|
+
for (;;) {
|
|
138
|
+
const { messages, next } = replayOutput(sessionId, cursor);
|
|
139
|
+
cursor = next;
|
|
140
|
+
for (const line of messages) {
|
|
141
|
+
const parsed = ServerMessageSchema.safeParse(JSON.parse(line));
|
|
142
|
+
if (!parsed.success) continue;
|
|
143
|
+
send(parsed.data);
|
|
144
|
+
if (parsed.data.type === 'result') return;
|
|
145
|
+
}
|
|
146
|
+
const current = readSession(sessionId);
|
|
147
|
+
if (!current) return;
|
|
148
|
+
// Still waiting on a decision → say so rather than tailing forever. Asked of
|
|
149
|
+
// the bus, not of the record: the answer arrives from a third party the
|
|
150
|
+
// owning process never hears from, so its `state` lags.
|
|
151
|
+
if (current.parkedEventId && stillParked(current, getEventBusPath())) {
|
|
152
|
+
send({
|
|
153
|
+
type: 'blocked',
|
|
154
|
+
sessionId,
|
|
155
|
+
eventId: current.parkedEventId,
|
|
156
|
+
question: current.question ?? '',
|
|
157
|
+
key: current.questionKey ?? undefined,
|
|
158
|
+
});
|
|
159
|
+
return;
|
|
160
|
+
}
|
|
161
|
+
await new Promise((r) => setTimeout(r, ATTACH_POLL_MS));
|
|
104
162
|
}
|
|
105
163
|
}
|
|
106
164
|
|
|
165
|
+
/** Abandon a parked session on the operator's say-so, before its TTL. */
|
|
166
|
+
function handleCancel(principal: string, sessionId: string): void {
|
|
167
|
+
const record = readSession(sessionId);
|
|
168
|
+
if (!record) {
|
|
169
|
+
send({ type: 'error', error: `no such session: ${sessionId}` });
|
|
170
|
+
send({ type: 'result', success: false, exitCode: 1 });
|
|
171
|
+
return;
|
|
172
|
+
}
|
|
173
|
+
if (record.principal !== principal) {
|
|
174
|
+
send({
|
|
175
|
+
type: 'error',
|
|
176
|
+
error: `permission denied: session ${sessionId} belongs to another principal`,
|
|
177
|
+
});
|
|
178
|
+
send({ type: 'result', success: false, exitCode: EXIT_PERMISSION_DENIED });
|
|
179
|
+
audit(principal, `cancel:${sessionId}`, 'deny');
|
|
180
|
+
return;
|
|
181
|
+
}
|
|
182
|
+
abandonSession(record, {
|
|
183
|
+
busDbPath: getEventBusPath(),
|
|
184
|
+
reason: `Session ${sessionId} was cancelled by ${principal} with "${record.question ?? 'the question'}" still unanswered.`,
|
|
185
|
+
emittedBy: `api:${principal}`,
|
|
186
|
+
});
|
|
187
|
+
audit(principal, `cancel:${sessionId}`, 'allow');
|
|
188
|
+
send({ type: 'result', success: true, exitCode: 0 });
|
|
189
|
+
}
|
|
190
|
+
|
|
107
191
|
export async function apiServeMode(principal: string): Promise<void> {
|
|
108
192
|
send({ type: 'ready', protocolVersion: API_PROTOCOL_VERSION });
|
|
109
193
|
|
|
110
194
|
const busDbPath = getEventBusPath();
|
|
111
|
-
|
|
195
|
+
// A session whose owning process died (reboot, OOM) would otherwise stay
|
|
196
|
+
// parked forever, holding whatever its command holds.
|
|
197
|
+
reapExpiredSessions({ busDbPath });
|
|
198
|
+
|
|
199
|
+
// When the ssh client goes away, sshd hangs up its forced command. Ignoring
|
|
200
|
+
// SIGHUP is what lets a parked command outlive the session that started it —
|
|
201
|
+
// the TTL reaper, not the transport, is what bounds it now.
|
|
202
|
+
process.on('SIGHUP', () => {});
|
|
203
|
+
|
|
204
|
+
const pendingAnswers = new Map<
|
|
205
|
+
string,
|
|
206
|
+
{ resolve: (value: unknown) => void; reject: (error: Error) => void; interview: WireInterview }
|
|
207
|
+
>();
|
|
208
|
+
|
|
209
|
+
/** Set while a command is running; the parked child outlives the transport. */
|
|
210
|
+
const live: { session: SessionWriter | null; clientAttached: boolean } = {
|
|
211
|
+
session: null,
|
|
212
|
+
clientAttached: true,
|
|
213
|
+
};
|
|
214
|
+
let commandRunning: Promise<void> | null = null;
|
|
112
215
|
|
|
113
216
|
const ask = (interview: WireInterview): Promise<unknown> =>
|
|
114
|
-
new Promise((resolve) => {
|
|
115
|
-
pendingAnswers.set(interview.id, resolve);
|
|
217
|
+
new Promise((resolve, reject) => {
|
|
218
|
+
pendingAnswers.set(interview.id, { resolve, reject, interview });
|
|
219
|
+
if (!live.clientAttached) {
|
|
220
|
+
// Nobody to ask. Re-park on THIS question: a detached command that was
|
|
221
|
+
// answered and moved on is now waiting on a *different* event, and a
|
|
222
|
+
// record still naming the previous one is actively wrong — it hides the
|
|
223
|
+
// live question from `events list-unanswered`, and it aims the TTL
|
|
224
|
+
// reaper at a question that was already decided, so the reaper retires
|
|
225
|
+
// the session while the child is still alive and blocked.
|
|
226
|
+
live.session?.park({
|
|
227
|
+
eventId: interview.id,
|
|
228
|
+
eventType: EVENT_TYPES.interviewRequired(interview.scope, interview.key),
|
|
229
|
+
question: interview.message,
|
|
230
|
+
questionKey: `${interview.scope}.${interview.key}`,
|
|
231
|
+
});
|
|
232
|
+
return;
|
|
233
|
+
}
|
|
116
234
|
send({
|
|
117
235
|
type: 'interview',
|
|
118
236
|
id: interview.id,
|
|
237
|
+
scope: interview.scope,
|
|
238
|
+
key: interview.key,
|
|
119
239
|
kind: interview.kind,
|
|
120
240
|
message: interview.message,
|
|
121
241
|
description: interview.description,
|
|
@@ -126,6 +246,46 @@ export async function apiServeMode(principal: string): Promise<void> {
|
|
|
126
246
|
});
|
|
127
247
|
});
|
|
128
248
|
|
|
249
|
+
const handleCommand = async (argv: string[]): Promise<void> => {
|
|
250
|
+
const { command, subcommand, label } = opOf(argv);
|
|
251
|
+
|
|
252
|
+
if (!(await isAuthorized(principal, command, subcommand))) {
|
|
253
|
+
send({ type: 'error', error: `permission denied: "${principal}" is not granted "${label}"` });
|
|
254
|
+
send({ type: 'result', success: false, exitCode: EXIT_PERMISSION_DENIED });
|
|
255
|
+
audit(principal, label, 'deny');
|
|
256
|
+
return;
|
|
257
|
+
}
|
|
258
|
+
|
|
259
|
+
const writer = SessionWriter.create({ principal, argv });
|
|
260
|
+
live.session = writer;
|
|
261
|
+
const reaper = setTimeout(
|
|
262
|
+
() => {
|
|
263
|
+
if (writer.current.state !== 'parked') return;
|
|
264
|
+
abandonSession(writer.current, {
|
|
265
|
+
busDbPath,
|
|
266
|
+
reason: expiryReason(writer.current),
|
|
267
|
+
emittedBy: `api:${principal}`,
|
|
268
|
+
});
|
|
269
|
+
writer.finish('abandoned');
|
|
270
|
+
},
|
|
271
|
+
Math.max(0, writer.current.expiresAt - Date.now()),
|
|
272
|
+
);
|
|
273
|
+
|
|
274
|
+
// Bridge the bus to the wire so mid-run interviews reach the client.
|
|
275
|
+
const responder = startRemoteResponder({ busDbPath, ask, emittedBy: `api:${principal}` });
|
|
276
|
+
try {
|
|
277
|
+
const exitCode = await runCommand(argv, (msg) =>
|
|
278
|
+
emit(writer, msg, () => live.clientAttached),
|
|
279
|
+
);
|
|
280
|
+
writer.finish('finished');
|
|
281
|
+
audit(principal, label, 'allow', exitCode);
|
|
282
|
+
} finally {
|
|
283
|
+
clearTimeout(reaper);
|
|
284
|
+
responder.close();
|
|
285
|
+
live.session = null;
|
|
286
|
+
}
|
|
287
|
+
};
|
|
288
|
+
|
|
129
289
|
const rl = createInterface({ input: process.stdin, terminal: false });
|
|
130
290
|
|
|
131
291
|
for await (const line of rl) {
|
|
@@ -140,20 +300,64 @@ export async function apiServeMode(principal: string): Promise<void> {
|
|
|
140
300
|
}
|
|
141
301
|
|
|
142
302
|
if (msg.type === 'answer') {
|
|
143
|
-
const
|
|
144
|
-
if (
|
|
303
|
+
const pending = pendingAnswers.get(msg.id);
|
|
304
|
+
if (pending) {
|
|
145
305
|
pendingAnswers.delete(msg.id);
|
|
146
|
-
|
|
306
|
+
live.session?.unpark();
|
|
307
|
+
pending.resolve(msg.value);
|
|
308
|
+
}
|
|
309
|
+
continue;
|
|
310
|
+
}
|
|
311
|
+
|
|
312
|
+
if (msg.type === 'unanswerable') {
|
|
313
|
+
// The client cannot decide. Do NOT resolve the question — park on it and
|
|
314
|
+
// tell the client where it stands. The responder, seeing `ask` reject,
|
|
315
|
+
// emits nothing on the bus, so the query is still there to be answered.
|
|
316
|
+
const pending = pendingAnswers.get(msg.id);
|
|
317
|
+
if (!pending) continue;
|
|
318
|
+
pendingAnswers.delete(msg.id);
|
|
319
|
+
const { interview } = pending;
|
|
320
|
+
pending.reject(new Error(msg.reason));
|
|
321
|
+
if (live.session) {
|
|
322
|
+
live.session.park({
|
|
323
|
+
eventId: msg.id,
|
|
324
|
+
eventType: EVENT_TYPES.interviewRequired(interview.scope, interview.key),
|
|
325
|
+
question: interview.message,
|
|
326
|
+
questionKey: `${interview.scope}.${interview.key}`,
|
|
327
|
+
});
|
|
328
|
+
send({
|
|
329
|
+
type: 'blocked',
|
|
330
|
+
sessionId: live.session.id,
|
|
331
|
+
eventId: msg.id,
|
|
332
|
+
question: interview.message,
|
|
333
|
+
key: `${interview.scope}.${interview.key}`,
|
|
334
|
+
});
|
|
147
335
|
}
|
|
148
336
|
continue;
|
|
149
337
|
}
|
|
150
338
|
|
|
339
|
+
if (msg.type === 'attach') {
|
|
340
|
+
await handleAttach(principal, msg.sessionId);
|
|
341
|
+
continue;
|
|
342
|
+
}
|
|
343
|
+
|
|
344
|
+
if (msg.type === 'cancel') {
|
|
345
|
+
handleCancel(principal, msg.sessionId);
|
|
346
|
+
continue;
|
|
347
|
+
}
|
|
348
|
+
|
|
151
349
|
if (msg.type === 'command') {
|
|
152
350
|
// Fire-and-forget so the read loop keeps consuming `answer` messages while
|
|
153
351
|
// the command runs — mid-run interviews are answered in-flight.
|
|
154
|
-
|
|
352
|
+
commandRunning = handleCommand(msg.argv);
|
|
353
|
+
void commandRunning;
|
|
155
354
|
}
|
|
156
355
|
}
|
|
157
356
|
|
|
357
|
+
// stdin closed: the client is gone. A parked command must NOT die with it —
|
|
358
|
+
// that is the bug. Stay alive (output goes to the session buffer) until the
|
|
359
|
+
// command finishes or the reaper abandons it.
|
|
360
|
+
live.clientAttached = false;
|
|
361
|
+
if (commandRunning) await commandRunning;
|
|
158
362
|
process.exit(0);
|
|
159
363
|
}
|