@celilo/cli 0.9.1 → 0.11.0-alpha.0

package/CELILO_CORE_MODULES.md

@@ -44,7 +44,7 @@ Each entry: `module id` — what it is — **provides** / **requires** capabilit
 - **forgejo** — self-hosted Forgejo git forge (git-over-SSH, OIDC, public ingress). **provides:** `source_forge`. **requires:** `public_web`, `idp`, `firewall`, `dns_registrar`. See `v2/CI_PIPELINE.md`.
 - **forgejo-runner** — host-mode (LXC) Forgejo Actions runner for unit/lint jobs; outbound-only, lives in dmz. **requires:** `source_forge`. See `v2/FORGEJO_RUNNER.md`.
 - **forgejo-builder** — VM-based, Docker-capable Forgejo Actions runner (`requires.system.type: vm`) for hermetic release builds + the cele2e suite. **requires:** `source_forge`.
-- **npm-cache-node** — self-hosted npm registry as a pass-through disk cache (proxies upstream). **requires:** `public_web`, `dns_registrar`. See `v2/NPM_CACHE_NODE.md`.
+- **npm-cache-node** — self-hosted npm registry: a pass-through disk cache (proxies upstream) plus a `PUT /<pkg>` publish endpoint that stores locally published `@celilo/*` tarballs as authoritative local-origin (never overwritten by upstream). PUT is gated on a static operator-set publish token (`publish_tokens` secret, SHA-256 model; fail-closed — absent/bad token → 401); reads stay open (it's a mirror). **requires:** `public_web`, `dns_registrar`. See `v2/NPM_CACHE_NODE.md`.
 
 ## Applications
 

package/CELILO_SUBSYSTEMS.md

@@ -77,7 +77,23 @@ see `design/README.md`. Companion doc: [CELILO_CORE_MODULES.md](./CELILO_CORE_MO
 
 - **Event bus** — `packages/event-bus/src/index.ts` — `Bus`, `openBus`, `defineEvents`, `defineHandler`, `runDispatcher`, pattern matching + timer ticks (`emitDueTimerTicks`, `retentionSweep`).
 
+## Remote API (drive the CLI over the wire)
+
+Run any celilo command on celilo-mgr over SSH instead of screen-scraping `ssh <host> celilo …`. Typed, streamed, per-operation authz, mid-run interviews. Design: `v2/API_COMMUNICATION.md`.
+
+- **Lightweight core (`@celilo/core`)** — `packages/core/src/` — the transport primitives lifted out of `@celilo/cli` so a consumer (e.g. the MCP server) can reach the wire without dragging Ink/React/drizzle/aws-sdk: `command-registry.ts` (`COMMANDS` + `CommandDef`/`ArgDef`/`FlagDef`), `protocol.ts`, `remote-client.ts`. Public surface: `packages/core/src/index.ts`.
+- **Wire protocol** — `packages/core/src/protocol.ts` (`@celilo/core`) — versioned NDJSON tagged union (`command`/`progress`/`log`/`result`/`error`/`interview`/`answer`) + `translateOutputLine`.
+- **Registry serialization** — `apps/celilo/src/cli/commands/commands-json.ts` — `celilo commands --json` prints the full `COMMANDS` tree as JSON; the live source of truth the MCP fetches to generate its tool surface (so it mirrors whatever celilo version the server runs). `service list --json` similarly exposes configured providers for MCP auto-detect. `module list --json` prints the module roster (id/version/state) as stable JSON — the backbone the MCP composite troubleshooting tools correlate `audit --json` findings against.
+- **Server** — `apps/celilo/src/api/serve.ts` (`apiServeMode`); the `celilo api-serve --principal=<id>` sshd forced-command entry point (dispatched in `apps/celilo/src/cli/index.ts`). Authorizes per principal, runs the command as a protocol-mode child, streams output, audits to stderr.
+- **Client** — `packages/core/src/remote-client.ts` (`@celilo/core`) — `resolveRemote` (`--remote <dest>` / `CELILO_REMOTE`), `runRemoteClient` (`ssh -T`, renders progress via the local ProgressDisplay, answers interviews via clack).
+- **Access control** — `apps/celilo/src/services/api-access.ts` — `grantPrincipal`, `isAuthorized` (deny-by-default, `command:subcommand` grants), `renderAuthorizedKeys`. Table: `api_principals` (`apps/celilo/src/db/schema.ts`). CLI: `apps/celilo/src/cli/commands/api.ts` (`api grant|list|revoke|authorized-keys|key new`).
+- **Mid-run interview bridge (`kind:daemon` responder)** — `apps/celilo/src/services/remote-responder.ts` — `startRemoteResponder` bridges bus `interview.required.*` ↔ wire.
+- **Server provisioning** — the `celilo-bootstrap` deb (`packaging/celilo-bootstrap/scripts/postinst`) creates the non-root `celilo-api` landing account + sshd; membership in the `celilo` group + `/etc/sudoers.d/celilo` (`!use_pty`) gives api-serve DB access via the wrapper's sudo-drop.
+- **Self-upgrade (apt)** — `celilo apt-upgrade` (`apps/celilo/src/cli/commands/apt-upgrade.ts`) upgrades the deb-installed `celilo`/`celilo-bootstrap` packages (`apt-get update` → `--only-upgrade install`) then spawns a fresh `celilo system migrate` (ISS-0100). It's the RW target behind the MCP's registry-derived `celilo_apt_upgrade` tool; the celilo user's two apt invocations are scoped-sudo'd by `/etc/sudoers.d/celilo-apt-upgrade`, shipped by `celilo-bootstrap`.
+- **MCP service (`@celilo/mcp`)** — `packages/mcp/src/` — an operator-facing stdio MCP server (official `@modelcontextprotocol/sdk`, bin `celilo-mcp`) that drives a remote celilo server over the Remote API for an AI client. Two-item config (`config.ts`: `server` + `defaultUser`, env or `~/.config/celilo-mcp/config.json`). Dual-principal auth (`auth.ts`: `celilo-mcp auth setup` enrolls read-only `celilo-mcp-ro` + full `celilo-mcp-rw` ed25519 keypairs, prints the exact `celilo api grant` lines the operator runs server-side). Transport (`transport.ts`): reuses `@celilo/core` `runRemoteClient`, selecting the principal by `ssh -i <key>` and capturing structured output. Tool surface is generated LIVE from the server's command registry — `registry-fetch.ts` fetches `celilo commands --json` (+ `service list --json` for configured providers) over the RO principal on connect; `tools-from-registry.ts` (pure) projects that into one tool per runnable leaf, grouped by top-level command (`celilo_module_*`, `celilo_proxmox_*`, …), each with a Zod input schema from the leaf's args/flags and a read/write tag → RO/RW routing, plus a generic `celilo_run` escape hatch. Auto-detect hides provider-gated groups (e.g. `celilo_proxmox_*` until a Proxmox service is configured) and re-detects on a timer, emitting `notifications/tools/list_changed` when the surface changes. Coverage gate (`tests/coverage.test.ts`) asserts every registry leaf maps to a tool. Composite RO troubleshooting tools (`troubleshoot.ts` pure correlation + `troubleshoot-tools.ts` thin adapters): `celilo_assess_module <id>` and `celilo_fleet_status` correlate `celilo audit --json` (the drift backbone) with the `module list --json` roster into a per-module / fleet-wide verdict. Design: `v2/CELILO_MCP_SERVICE.md`. (Distinct from the dev/ops `@celilo/mcp-server` below.)
+
 ## E2E simulation
 
 - **cele2e harness** — `packages/e2e/src/` — `runner.ts`, `container-manager.ts` (`startNetwork`, `reconnectNetwork`), `network-builder.ts` (`NetworkBuilder`).
 - **Run wrapper** — `infra/scripts/cele2e-run.sh` lives in the separate `infra/` clone, **not** in this repo. See the cele2e section of the repo-root `CLAUDE.md` for the operator workflow.
+- **MCP server (agent-driven e2e)** — `packages/mcp-server/src/index.ts` — stdio MCP server exposing `start_run`/`run_status`/`run_result`/`stop_run` (detached cele2e runs read off the event bus, no ANSI scraping) + `env_check` (docker VM, run-lock, shared-infra, mgmt image CLI version, netapps). Dev/ops tool, `private`, not shipped to consumers.

package/drizzle/0014_api_principals.sql

@@ -0,0 +1,10 @@
+CREATE TABLE `api_principals` (
+	`id` text PRIMARY KEY NOT NULL,
+	`name` text NOT NULL,
+	`public_key` text NOT NULL,
+	`grants` text DEFAULT '[]' NOT NULL,
+	`created_at` integer DEFAULT (unixepoch()) NOT NULL,
+	`updated_at` integer DEFAULT (unixepoch()) NOT NULL
+);
+--> statement-breakpoint
+CREATE UNIQUE INDEX `api_principals_name_unique` ON `api_principals` (`name`);

package/drizzle/meta/_journal.json

@@ -99,6 +99,13 @@
       "when": 1782456724000,
       "tag": "0013_dns_view_overrides",
       "breakpoints": true
+    },
+    {
+      "idx": 14,
+      "version": "6",
+      "when": 1783000000000,
+      "tag": "0014_api_principals",
+      "breakpoints": true
     }
   ]
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@celilo/cli",
-  "version": "0.9.1",
+  "version": "0.11.0-alpha.0",
   "description": "Celilo — home lab orchestration CLI",
   "type": "module",
   "bin": {
@@ -16,7 +16,13 @@
     "CELILO_SUBSYSTEMS.md",
     "CELILO_CORE_MODULES.md"
   ],
-  "keywords": ["celilo", "homelab", "orchestration", "ansible", "terraform"],
+  "keywords": [
+    "celilo",
+    "homelab",
+    "orchestration",
+    "ansible",
+    "terraform"
+  ],
   "license": "MIT",
   "repository": {
     "type": "git",
@@ -49,9 +55,10 @@
   },
   "dependencies": {
     "@aws-sdk/client-s3": "^3.1024.0",
-    "@celilo/capabilities": "^0.6.0",
-    "@celilo/cli-display": "^0.1.9",
-    "@celilo/event-bus": "^0.1.7",
+    "@celilo/capabilities": "0.6.0-alpha.0",
+    "@celilo/cli-display": "0.1.9-alpha.1",
+    "@celilo/core": "^0.1.0",
+    "@celilo/event-bus": "0.1.8-alpha.0",
     "@clack/prompts": "^1.1.0",
     "ajv": "^8.18.0",
     "drizzle-orm": "^0.36.4",
@@ -72,5 +79,6 @@
     "ink-testing-library": "^4.0.0",
     "typescript": "^5.9.3",
     "zod-to-json-schema": "^3.25.2"
-  }
+  },
+  "gitHead": "8fe2dff30ef6a59a0aed48e5501403e43be8429e"
 }

package/src/api/protocol.test.ts

@@ -0,0 +1,76 @@
+import { describe, expect, test } from 'bun:test';
+import { translateOutputLine } from '@celilo/core';
+
+describe('translateOutputLine', () => {
+  test('start marker → progress start', () => {
+    expect(translateOutputLine('[progress:start] Deploying caddy | Deployed caddy')).toEqual({
+      type: 'progress',
+      kind: 'start',
+      doing: 'Deploying caddy',
+      done: 'Deployed caddy',
+    });
+  });
+
+  test('push marker → progress push', () => {
+    expect(translateOutputLine('[progress:push] Sub step | Sub done')).toEqual({
+      type: 'progress',
+      kind: 'push',
+      doing: 'Sub step',
+      done: 'Sub done',
+    });
+  });
+
+  test('done marker with override message', () => {
+    expect(translateOutputLine('[progress:done] all good')).toEqual({
+      type: 'progress',
+      kind: 'done',
+      message: 'all good',
+    });
+  });
+
+  test('done marker without message', () => {
+    expect(translateOutputLine('[progress:done]')).toEqual({
+      type: 'progress',
+      kind: 'done',
+      message: undefined,
+    });
+  });
+
+  test('fail marker → progress fail', () => {
+    expect(translateOutputLine('[progress:fail] it broke')).toEqual({
+      type: 'progress',
+      kind: 'fail',
+      message: 'it broke',
+    });
+  });
+
+  test('sub marker → progress sub', () => {
+    expect(translateOutputLine('[progress:sub] running terraform')).toEqual({
+      type: 'progress',
+      kind: 'sub',
+      message: 'running terraform',
+    });
+  });
+
+  test('legacy instant marker → progress message', () => {
+    expect(translateOutputLine('[progress] heads up')).toEqual({
+      type: 'progress',
+      kind: 'message',
+      message: 'heads up',
+    });
+  });
+
+  test('plain line → log', () => {
+    expect(translateOutputLine('just some output')).toEqual({
+      type: 'log',
+      message: 'just some output',
+    });
+  });
+
+  test('a line that only resembles a marker → log', () => {
+    expect(translateOutputLine('see [progress:start] in the docs')).toEqual({
+      type: 'log',
+      message: 'see [progress:start] in the docs',
+    });
+  });
+});

package/src/api/remote-client.test.ts

@@ -0,0 +1,91 @@
+import { describe, expect, test } from 'bun:test';
+import { type RemoteTransport, resolveRemote, runRemoteClient } from '@celilo/core';
+
+const argv = (rest: string[]) => ['bun', 'celilo', ...rest];
+
+describe('resolveRemote', () => {
+  test('leading --remote <dest>', () => {
+    expect(resolveRemote(argv(['--remote', 'host', 'module', 'list']), undefined)).toEqual({
+      dest: 'host',
+      commandArgv: ['module', 'list'],
+    });
+  });
+
+  test('leading --remote=<dest>', () => {
+    expect(resolveRemote(argv(['--remote=user@host', 'status']), undefined)).toEqual({
+      dest: 'user@host',
+      commandArgv: ['status'],
+    });
+  });
+
+  test('CELILO_REMOTE env', () => {
+    expect(resolveRemote(argv(['module', 'list']), 'envhost')).toEqual({
+      dest: 'envhost',
+      commandArgv: ['module', 'list'],
+    });
+  });
+
+  test('leading flag beats env', () => {
+    expect(resolveRemote(argv(['--remote', 'flaghost', 'status']), 'envhost')?.dest).toBe(
+      'flaghost',
+    );
+  });
+
+  test('local invocation → null', () => {
+    expect(resolveRemote(argv(['module', 'list']), undefined)).toBeNull();
+  });
+
+  test('--remote with no dest → null', () => {
+    expect(resolveRemote(argv(['--remote']), undefined)).toBeNull();
+  });
+});
+
+test('renders a forwarded interview and sends the answer back', async () => {
+  const writes: string[] = [];
+  const encoder = new TextEncoder();
+  let controller!: ReadableStreamDefaultController<Uint8Array>;
+  const stdout = new ReadableStream<Uint8Array>({
+    start(c) {
+      controller = c;
+    },
+  });
+  const push = (obj: unknown) => controller.enqueue(encoder.encode(`${JSON.stringify(obj)}\n`));
+
+  const transport: RemoteTransport = {
+    stdin: {
+      write(chunk: string) {
+        writes.push(chunk);
+        // The command's answer arrived — finish the command.
+        if (chunk.includes('"answer"')) {
+          push({ type: 'result', success: true, exitCode: 0 });
+          controller.close();
+        }
+      },
+    },
+    stdout,
+    kill() {},
+    exited: Promise.resolve(0),
+  };
+
+  // Server script: greet, then ask one interview.
+  push({ type: 'ready', protocolVersion: 1 });
+  push({ type: 'interview', id: 'q1', kind: 'text', message: 'Hostname?' });
+
+  const seen: Array<{ id: string }> = [];
+  const code = await runRemoteClient('ignored', ['module', 'deploy', 'site'], {
+    openTransport: () => transport,
+    out: { write() {} },
+    renderInterview: async (iv) => {
+      seen.push(iv);
+      return 'myhost';
+    },
+  });
+
+  expect(code).toBe(0);
+  expect(seen).toHaveLength(1);
+  expect(seen[0].id).toBe('q1');
+  expect(writes.some((w) => w.includes('"command"'))).toBe(true);
+  const answer = writes.find((w) => w.includes('"answer"'));
+  expect(answer).toBeDefined();
+  expect(JSON.parse(answer as string)).toEqual({ type: 'answer', id: 'q1', value: 'myhost' });
+});

package/src/api/serve.ts

@@ -0,0 +1,159 @@
+/**
+ * Remote API server — `celilo api-serve --principal=<id>` (Slices 1–3).
+ *
+ * Invoked as the sshd forced command for an enrolled key, so it starts already
+ * authenticated as `principal`. Reads `command`/`answer` messages as NDJSON on
+ * stdin/stdout, authorizes each command against the principal's grants
+ * (deny-by-default), and — if allowed — runs it as a child `celilo` process,
+ * translating the child's protocol-mode output into `progress`/`log` and a
+ * terminal `result`. While a command runs, a remote responder bridges the event
+ * bus to the wire so mid-run `interview`s are answered by the client. Every
+ * attempt is audited to stderr.
+ */
+
+import { createInterface } from 'node:readline';
+import {
+  API_PROTOCOL_VERSION,
+  ClientMessageSchema,
+  type ServerMessage,
+  translateOutputLine,
+} from '@celilo/core';
+import { parseArguments } from '../cli/parser';
+import { getEventBusPath } from '../config/paths';
+import { isAuthorized } from '../services/api-access';
+import { type WireInterview, startRemoteResponder } from '../services/remote-responder';
+
+/** Exit code returned to the client when authz denies a command. */
+const EXIT_PERMISSION_DENIED = 126;
+
+function send(msg: ServerMessage): void {
+  process.stdout.write(`${JSON.stringify(msg)}\n`);
+}
+
+function audit(principal: string, op: string, decision: string, exitCode?: number): void {
+  const suffix = exitCode === undefined ? '' : ` exit=${exitCode}`;
+  process.stderr.write(
+    `[api-audit] principal=${principal} op=${op} decision=${decision}${suffix}\n`,
+  );
+}
+
+/** Derive the `command`/`subcommand` an argv would run, via the real parser. */
+function opOf(argv: string[]): { command: string; subcommand?: string; label: string } {
+  const parsed = parseArguments(['bun', 'celilo', ...argv]);
+  const label = parsed.subcommand ? `${parsed.command}:${parsed.subcommand}` : parsed.command;
+  return { command: parsed.command, subcommand: parsed.subcommand, label };
+}
+
+/** Drain a child stream line-by-line, forwarding each line as a message. */
+async function pumpLines(stream: ReadableStream<Uint8Array>): Promise<void> {
+  const decoder = new TextDecoder();
+  let buffer = '';
+  for await (const chunk of stream) {
+    buffer += decoder.decode(chunk, { stream: true });
+    let nl = buffer.indexOf('\n');
+    while (nl >= 0) {
+      const line = buffer.slice(0, nl);
+      buffer = buffer.slice(nl + 1);
+      send(translateOutputLine(line));
+      nl = buffer.indexOf('\n');
+    }
+  }
+  if (buffer.length > 0) {
+    send(translateOutputLine(buffer));
+  }
+}
+
+/** Run an authorized command as a child; returns its exit code. */
+async function runCommand(argv: string[]): Promise<number> {
+  // Re-invoke this same CLI as a child. The child is non-TTY (piped stdout) so
+  // ProgressDisplay resolves to protocol mode and emits `[progress:*]` markers.
+  const child = Bun.spawn([process.execPath, Bun.main, ...argv], {
+    stdin: 'ignore',
+    stdout: 'pipe',
+    stderr: 'pipe',
+  });
+
+  await Promise.all([pumpLines(child.stdout), pumpLines(child.stderr)]);
+  const exitCode = await child.exited;
+  send({ type: 'result', success: exitCode === 0, exitCode });
+  return exitCode;
+}
+
+async function handleCommand(
+  principal: string,
+  argv: string[],
+  busDbPath: string,
+  ask: (interview: WireInterview) => Promise<unknown>,
+): Promise<void> {
+  const { command, subcommand, label } = opOf(argv);
+
+  if (!(await isAuthorized(principal, command, subcommand))) {
+    send({ type: 'error', error: `permission denied: "${principal}" is not granted "${label}"` });
+    send({ type: 'result', success: false, exitCode: EXIT_PERMISSION_DENIED });
+    audit(principal, label, 'deny');
+    return;
+  }
+
+  // Bridge the bus to the wire so mid-run interviews reach the client.
+  const responder = startRemoteResponder({ busDbPath, ask, emittedBy: `api:${principal}` });
+  try {
+    const exitCode = await runCommand(argv);
+    audit(principal, label, 'allow', exitCode);
+  } finally {
+    responder.close();
+  }
+}
+
+export async function apiServeMode(principal: string): Promise<void> {
+  send({ type: 'ready', protocolVersion: API_PROTOCOL_VERSION });
+
+  const busDbPath = getEventBusPath();
+  const pendingAnswers = new Map<string, (value: unknown) => void>();
+
+  const ask = (interview: WireInterview): Promise<unknown> =>
+    new Promise((resolve) => {
+      pendingAnswers.set(interview.id, resolve);
+      send({
+        type: 'interview',
+        id: interview.id,
+        kind: interview.kind,
+        message: interview.message,
+        description: interview.description,
+        defaultValue: interview.defaultValue,
+        placeholder: interview.placeholder,
+        options: interview.options,
+        required: interview.required,
+      });
+    });
+
+  const rl = createInterface({ input: process.stdin, terminal: false });
+
+  for await (const line of rl) {
+    if (!line.trim()) continue;
+
+    let msg: ReturnType<typeof ClientMessageSchema.parse>;
+    try {
+      msg = ClientMessageSchema.parse(JSON.parse(line));
+    } catch (error) {
+      send({ type: 'error', error: error instanceof Error ? error.message : String(error) });
+      continue;
+    }
+
+    if (msg.type === 'answer') {
+      const resolve = pendingAnswers.get(msg.id);
+      if (resolve) {
+        pendingAnswers.delete(msg.id);
+        resolve(msg.value);
+      }
+      continue;
+    }
+
+    if (msg.type === 'command') {
+      // Fire-and-forget so the read loop keeps consuming `answer` messages while
+      // the command runs — mid-run interviews are answered in-flight.
+      void handleCommand(principal, msg.argv, busDbPath, ask);
+    }
+  }
+
+  process.exit(0);
+}

package/src/cli/command-tree-parser.ts

@@ -177,7 +177,9 @@ export class CommandTreeParser {
       'usage',
       'options',
       'examples',
-      'commands',
+      // NB: 'commands' is intentionally NOT excluded — it's a real command
+      // (`celilo commands`). The "Commands:" section header is filtered earlier
+      // by the section-header regex, so it never reaches this check.
       'description',
       'example',
       'for',

package/src/cli/commands/api.ts

@@ -0,0 +1,194 @@
+/**
+ * `celilo api ...` — manage remote-API principals (Slice 2a).
+ *
+ * grant / list / revoke operate on the api_principals table; authorized-keys
+ * renders the forced-command file for the API account. See v2/API_COMMUNICATION.md.
+ */
+
+import { existsSync, mkdirSync, readFileSync } from 'node:fs';
+import { homedir } from 'node:os';
+import { dirname, join } from 'node:path';
+import {
+  grantPrincipal,
+  listPrincipals,
+  renderAuthorizedKeys,
+  revokePrincipal,
+  validatePrincipalName,
+} from '../../services/api-access';
+import { celiloIntro } from '../prompts';
+import type { CommandResult } from '../types';
+
+function errMsg(error: unknown): string {
+  return error instanceof Error ? error.message : String(error);
+}
+
+/** `celilo api grant <principal> --key <pubkey|path> --can <grant[,grant...]>` */
+export async function handleApiGrant(
+  args: string[],
+  flags: Record<string, boolean | string> = {},
+): Promise<CommandResult> {
+  try {
+    const name = args[0];
+    if (!name) {
+      return {
+        success: false,
+        error: 'Usage: celilo api grant <principal> --key <pubkey|path> --can <grant[,grant...]>',
+      };
+    }
+
+    const keyArg = typeof flags.key === 'string' ? flags.key : '';
+    if (!keyArg) {
+      return {
+        success: false,
+        error: '--key <public-key-or-path> is required (e.g. ~/.ssh/id_ed25519.pub)',
+      };
+    }
+
+    const canArg = typeof flags.can === 'string' ? flags.can : '';
+    if (!canArg) {
+      return {
+        success: false,
+        error: '--can <grant[,grant...]> is required (e.g. --can module:deploy,service:*)',
+      };
+    }
+
+    const publicKey = existsSync(keyArg) ? readFileSync(keyArg, 'utf8').trim() : keyArg.trim();
+    const grants = canArg
+      .split(',')
+      .map((g) => g.trim())
+      .filter(Boolean);
+
+    const { principal, created } = await grantPrincipal({ name, publicKey, grants });
+
+    return {
+      success: true,
+      message: `${created ? 'Granted' : 'Updated'} API access for "${principal.name}" → ${grants.join(', ')}\n\nInstall the forced-command line on celilo-mgr with:\n  celilo api authorized-keys >> ~celilo-api/.ssh/authorized_keys`,
+    };
+  } catch (error) {
+    return { success: false, error: `Failed to grant API access: ${errMsg(error)}` };
+  }
+}
+
+/** `celilo api list` */
+export async function handleApiList(): Promise<CommandResult> {
+  try {
+    celiloIntro('API Principals');
+    const principals = await listPrincipals();
+
+    if (principals.length === 0) {
+      console.log('No API principals.\n');
+      console.log('Grant access:');
+      console.log('  celilo api grant <name> --key ~/.ssh/id_ed25519.pub --can module:deploy');
+      return { success: true, message: 'No API principals found' };
+    }
+
+    console.log('');
+    for (const p of principals) {
+      const parts = p.publicKey.split(/\s+/);
+      const keyType = parts[0] ?? '';
+      const comment = parts.length > 2 ? parts.slice(2).join(' ') : '';
+      console.log(`${p.name}`);
+      console.log(`  Grants: ${p.grants.join(', ') || '(none)'}`);
+      console.log(`  Key:    ${keyType}${comment ? ` (${comment})` : ''}`);
+      console.log('');
+    }
+
+    console.log(`Total: ${principals.length} principal${principals.length === 1 ? '' : 's'}\n`);
+    return { success: true, message: `Found ${principals.length} principal(s)` };
+  } catch (error) {
+    return { success: false, error: `Failed to list API principals: ${errMsg(error)}` };
+  }
+}
+
+/** `celilo api revoke <principal>` */
+export async function handleApiRevoke(args: string[]): Promise<CommandResult> {
+  try {
+    const name = args[0];
+    if (!name) {
+      return { success: false, error: 'Usage: celilo api revoke <principal>' };
+    }
+
+    const removed = await revokePrincipal(name);
+    if (!removed) {
+      return { success: false, error: `No API principal named "${name}".` };
+    }
+
+    return {
+      success: true,
+      message: `Revoked API access for "${name}".\n\nRe-render celilo-mgr's authorized_keys to drop the line:\n  celilo api authorized-keys > ~celilo-api/.ssh/authorized_keys`,
+    };
+  } catch (error) {
+    return { success: false, error: `Failed to revoke API access: ${errMsg(error)}` };
+  }
+}
+
+/** `celilo api authorized-keys` — print the forced-command file for the API account. */
+export async function handleApiAuthorizedKeys(): Promise<CommandResult> {
+  try {
+    const content = await renderAuthorizedKeys();
+    return { success: true, message: content, rawOutput: true };
+  } catch (error) {
+    return { success: false, error: `Failed to render authorized_keys: ${errMsg(error)}` };
+  }
+}
+
+/** `celilo api key new <name>` — generate a client-side keypair for API access. */
+export async function handleApiKeyNew(args: string[]): Promise<CommandResult> {
+  try {
+    const name = args[0];
+    if (!name) {
+      return { success: false, error: 'Usage: celilo api key new <name>' };
+    }
+    validatePrincipalName(name);
+
+    // Prefer $HOME (operator shell always sets it, and it's test-controllable);
+    // homedir() is the fallback for the rare unset case.
+    const keyPath = join(process.env.HOME || homedir(), '.ssh', `celilo-api-${name}`);
+    if (existsSync(keyPath) || existsSync(`${keyPath}.pub`)) {
+      return {
+        success: false,
+        error: `A key already exists at ${keyPath}. Remove it or choose another name.`,
+      };
+    }
+
+    mkdirSync(dirname(keyPath), { recursive: true, mode: 0o700 });
+
+    const gen = Bun.spawnSync([
+      'ssh-keygen',
+      '-t',
+      'ed25519',
+      '-N',
+      '',
+      '-C',
+      `celilo-api-${name}`,
+      '-f',
+      keyPath,
+    ]);
+    if (gen.exitCode !== 0) {
+      return {
+        success: false,
+        error: `ssh-keygen failed: ${gen.stderr.toString().trim() || `exit ${gen.exitCode}`}`,
+      };
+    }
+
+    const pubkey = readFileSync(`${keyPath}.pub`, 'utf8').trim();
+    return {
+      success: true,
+      message: [
+        `Generated API keypair for "${name}":`,
+        `  private: ${keyPath}  (keep secret — never share)`,
+        `  public:  ${keyPath}.pub`,
+        '',
+        `Public key: ${pubkey}`,
+        '',
+        'Enroll the public key on celilo-mgr:',
+        `  celilo api grant ${name} --key ${keyPath}.pub --can module:deploy`,
+        '',
+        'Then run commands remotely (configure a Host alias in ~/.ssh/config as needed):',
+        '  celilo --remote celilo-api@celilo-mgr <command>',
+      ].join('\n'),
+    };
+  } catch (error) {
+    return { success: false, error: `Failed to generate API key: ${errMsg(error)}` };
+  }
+}

package/src/cli/commands/apt-upgrade.test.ts

@@ -0,0 +1,33 @@
+import { describe, expect, test } from 'bun:test';
+import { handleAptUpgrade } from './apt-upgrade';
+
+describe('handleAptUpgrade', () => {
+  test('runs the three steps in order when each succeeds', async () => {
+    const seen: string[][] = [];
+    const result = await handleAptUpgrade([], {}, (argv) => {
+      seen.push(argv);
+      return { status: 0 };
+    });
+
+    expect(result.success).toBe(true);
+    expect(seen).toEqual([
+      ['sudo', 'apt-get', 'update'],
+      ['sudo', 'apt-get', '-y', '--only-upgrade', 'install', 'celilo', 'celilo-bootstrap'],
+      ['/usr/local/bin/celilo', 'system', 'migrate'],
+    ]);
+  });
+
+  test('stops at the first failing step and does not run later ones', async () => {
+    const seen: string[][] = [];
+    const result = await handleAptUpgrade([], {}, (argv) => {
+      seen.push(argv);
+      // Fail the apt upgrade step (index 1).
+      return { status: argv.includes('install') ? 100 : 0 };
+    });
+
+    expect(result.success).toBe(false);
+    if (!result.success) expect(result.error).toContain('apt-get upgrade');
+    // update + install ran; migrate did NOT.
+    expect(seen).toHaveLength(2);
+  });
+});