spinuptui 0.20.1 → 0.21.0

package/CHANGELOG.md

@@ -11,6 +11,69 @@ versions; such changes are called out here.
 
 ## [Unreleased]
 
+## [0.21.0] - 2026-07-09
+
+### Added
+- **`spinuptui ssh <domain>`** — non-interactive CLI subcommand for external
+  tooling (e.g. an incident-diagnostics agent that's handed only a domain).
+  Resolves the domain to its site/server, builds the SSH target, and runs a
+  live connectivity probe, returning JSON with a classified failure reason
+  (site not found, ambiguous domain match, server has no IP, key not
+  granted, token rejected, etc.) when it can't connect.
+- **`spinuptui incidents <domain>` / `--all`** — surfaces Uptime Kuma's
+  down/up incident history as JSON, scoped to sites SpinupTUI manages
+  monitoring for. `--all` sweeps every such site over a single Kuma
+  connection; `--hours` sets the lookback window (default 24). Pairs with
+  `spinuptui ssh` for an external agent's discover → access → report loop.
+- **Server-wide PHP-fatal sentinel monitor.** Fixes a real monitoring blind
+  spot: SpinupWP's page cache keeps serving `/` even while PHP is fataling on
+  anything the cache doesn't already have, so the existing health/fingerprint
+  monitors can sleep through a real outage (confirmed against a 2026-07-07
+  incident that went undetected for 54 of its 68 minutes). A root-level cron
+  (one per server, added via the monitoring overlay's repair flow once sudo
+  is connected — same place the Redis sentinel is set up) tails every site's
+  error/debug logs each minute for new `PHP Fatal error` lines and reports
+  up/down to a single new Kuma push monitor, with the actually-affected
+  site domain(s) carried in the down message. Surfaces automatically in
+  `spinuptui incidents --all`.
+- **Cache-bypass monitor, opt-in per site.**
+  Closes the other half of the page-cache blind spot: a wedged PHP-FPM pool
+  that never logs a fatal is invisible to both the plain health monitor and
+  the new fatal sentinel above, since nginx serves the cache without ever
+  touching PHP-FPM. This registers an ordinary Kuma HTTP monitor with a
+  `Cookie: wordpress_no_cache=1` header — the same page-cache bypass
+  `siteDoctor.ts` already uses — so it actually exercises PHP on every check
+  (verified live: `fastcgi-cache-bypass-reason: COOKIE`). Deliberately opt-in
+  (not automatic) and defaults to a 1h check window (30m/1h/2h/4h/6h
+  choices), since forcing a real render costs the site something, unlike the
+  cache-served health/fingerprint checks — meant for sites with an actual
+  history of this failure mode, not the whole fleet.
+
+### Changed
+- **Site/server monitoring overlay rebuilt as a full-screen two-pane browser**
+  (`m` in the sites pane), replacing the small centered popover. A narrow
+  left list of the site's monitor kinds (with a live status dot) sits next
+  to a wide detail pane showing whichever one is selected — its one-line
+  definition, a longer explanation of the actual mechanism, live status, and
+  how to act on it. Grew out of real confusion testing the popover version:
+  six monitor kinds across two site contexts didn't fit legibly in a 68-col
+  box, and a separate glossary overlay left the explanation disconnected
+  from live status. Also **collapses the per-kind action keys into one `a`**
+  that acts on whichever monitor is currently highlighted (front page
+  selected → calibration picker; cache bypass selected → its picker;
+  anything else → register/repair) — fewer keys to remember now that the
+  detail pane shows which monitor you're acting on. The overlay also gained:
+  a passive "Alerts: …" header line (fetched once per open, not polled) so
+  notification wiring is visible without pressing `n`; `o` to deep-link
+  straight to the selected monitor's own page in Uptime Kuma; and `x` to
+  remove the front-page or cache-bypass monitor (confirm-gated) — scoped to
+  just those two kinds, since the other four re-derive themselves
+  automatically on the next repair and would silently reappear.
+- **The header's fleet-wide "N monitors down" badge now names the domain(s)**
+  (falling back to a bare count once there are too many to list) and points
+  at the Search tab to find them, instead of an unactionable number with no
+  way to tell which site it means.
+
 ## [0.20.1] - 2026-07-08
 
 ### Changed
@@ -996,7 +1059,8 @@ Initial tagged release.
 ### Notes
 - Read-only release: works with a SpinupWP **Read Only** API token.
 
-[Unreleased]: https://github.com/mwender/spinupwp-tui/compare/v0.20.1...HEAD
+[Unreleased]: https://github.com/mwender/spinupwp-tui/compare/v0.21.0...HEAD
+[0.21.0]: https://github.com/mwender/spinupwp-tui/compare/v0.20.1...v0.21.0
 [0.20.1]: https://github.com/mwender/spinupwp-tui/compare/v0.20.0...v0.20.1
 [0.20.0]: https://github.com/mwender/spinupwp-tui/compare/v0.19.1...v0.20.0
 [0.19.1]: https://github.com/mwender/spinupwp-tui/compare/v0.19.0...v0.19.1

package/README.md

@@ -65,12 +65,19 @@
   [how it works](docs/clone-wizard-explained.md)
 - **Privileged writes over SSH** (`S` / `K`) — grant or revoke SSH keys directly,
   since the API can't. → [docs/privileged-ssh-writes.md](docs/privileged-ssh-writes.md)
-- **Uptime Kuma monitoring** (`m`) — wire a site into your own
-  [Uptime Kuma](https://github.com/louislam/uptime-kuma) for uptime, load, and
-  health checks. → [docs/uptime-kuma.md](docs/uptime-kuma.md)
-- **Site monitoring: fingerprints, a doctor, and a Redis sentinel** (`M`) — a
-  front-page identity check, a cache/outage doctor, and alert wiring.
-  → [docs/site-monitoring.md](docs/site-monitoring.md)
+- **Site & server monitoring** (`m`) — a two-pane browser of a site's Uptime
+  Kuma monitors (health, front-page fingerprint, an opt-in cache-bypass
+  check, and, for vanity/server sites, load/Redis/PHP-fatal sentinels), each
+  with a live status dot, an in-context explanation, and one action key to
+  register/recalibrate/remove it — plus a cache/outage doctor and alert
+  wiring. → [docs/site-monitoring.md](docs/site-monitoring.md) ·
+  [docs/uptime-kuma.md](docs/uptime-kuma.md)
+- **Non-interactive CLI subcommands for external tooling** — `spinuptui ssh <domain>`
+  resolves a domain to a live SSH target with a live connectivity probe;
+  `spinuptui incidents <domain>` / `--all` surfaces Uptime Kuma's down/up
+  history as JSON. Built so another agent/script can go from just a domain to
+  "what's wrong and how do I get in" with no manual lookup.
+  → [CLI subcommands](#cli-subcommands)
 - **Completion toasts** — a non-focus-stealing toast when a background write
   (PHP upgrade, reboot, DNS resolve, …) finishes.
 - **Release notes** — an in-app "what's new" after every update, sourced
@@ -156,6 +163,15 @@ to the version in the header (and in the `?` About panel).
 spinuptui            Launch the dashboard
 spinuptui login      Set or update your saved API token
 spinuptui where      Show the config path and which source the token came from
+spinuptui ssh <domain>  Print SSH access info for a site as JSON (resolves the
+                     site's server, builds its SSH target, and runs a live
+                     connectivity probe — for external tooling, e.g. an
+                     incident-diagnostics agent handed only a domain)
+spinuptui incidents <domain> | --all [--hours N]  Print Uptime Kuma down/up
+                     incidents as JSON, scoped to sites SpinupTUI manages
+                     monitoring for (config.json's kumaMonitors) — --all
+                     sweeps every such site in one Kuma connection, --hours
+                     sets the lookback window (default 24)
 spinuptui --version  Print the version
 spinuptui --help     Show help
 ```
@@ -232,8 +248,7 @@ These can be set in `config.json` or via an environment variable:
 | `u` | Upgrade a site's PHP version (Servers / Stacks / Search; needs a Read/Write token) |
 | `H` | Enable / disable HTTPS on a site (Servers / Stacks / Search; needs a Read/Write token) |
 | `P` | Purge a site's page cache + object cache (Servers / Stacks / Search; needs a Read/Write token) |
-| `m` | Site monitoring via Uptime Kuma (Servers tab, sites pane — same overlay as `M`) |
-| `M` | Site monitoring overlay — add monitors, `f` front-page check, `d` site doctor, `n` alert wiring, vanity refresh/rotation (Servers / Stacks / Search) |
+| `m` / `M` | Site/server monitoring — a two-pane browser of this site's Uptime Kuma monitors (`M` is an alias, since lowercase `m` is taken in Search/Stacks). Inside: `↑`/`↓` selects a monitor, `a` registers/recalibrates/repairs it (Front page and Cache bypass open a check-window picker), `x` removes Front page or Cache bypass, `o` opens the selected monitor in Kuma, `d` runs the site doctor, `n` shows/edits alert wiring, and vanity sites add `R`/`r` for page refresh/secret rotation (Servers / Stacks / Search) |
 | `a` | Server actions: reboot / restart a service (Servers / Search; needs a Read/Write token) |
 | `c` | Create a new server (Servers tab; needs a Read/Write token) |
 | `V` | Add a vanity site at the server's own hostname — DNS + site + HTTPS + SSH-key handoff (Servers tab; offered when no hostname site exists; needs a Read/Write token) |
@@ -286,9 +301,12 @@ Needs a Read/Write token. Full details: [docs/server-actions.md](docs/server-act
 
 ## Site monitoring
 
-`M` on a site opens the monitoring overlay — uptime/Redis monitors, a
-template-identity front-page check, a cache/outage doctor, and Kuma alert
-wiring, all against your own Uptime Kuma instance. Full details:
+`m` on a site opens a full-screen two-pane monitor browser — health,
+front-page fingerprint, an opt-in cache-bypass check, and (vanity/server
+sites) load/Redis/PHP-fatal sentinels, each with live status, an in-context
+explanation, and a single action key to register/recalibrate/remove it —
+plus a cache/outage doctor and Kuma alert wiring, all against your own
+Uptime Kuma instance. Full details:
 [docs/site-monitoring.md](docs/site-monitoring.md) ·
 [docs/uptime-kuma.md](docs/uptime-kuma.md).
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "spinuptui",
-  "version": "0.20.1",
+  "version": "0.21.0",
   "description": "SpinupTUI — a terminal dashboard for browsing, monitoring, and managing your SpinupWP servers and sites.",
   "type": "module",
   "bin": {

package/src/config.ts

@@ -116,6 +116,9 @@ export interface KumaMonitorRef {
   pushToken?: string
   redisId?: number // "{server} redis" push monitor fed by the cron's redis-cli ping (vanity/server domains)
   redisToken?: string
+  fatalId?: number // "{server} php-fatal" server-wide sentinel fed by the root cron's error/debug.log tail (vanity/server domains)
+  fatalToken?: string
+  bypassId?: number // "{domain} cache-bypass" opt-in http monitor with a page-cache-bypass Cookie header (per-site, no token — Kuma polls it directly)
   fingerprintId?: number // keyword monitor asserting the front page serves its own template
   // What fingerprint calibration derived (lib/siteFingerprint.ts) — shown in the
   // overlay and kept so a recalibration can explain what it's replacing.

package/src/index.tsx

@@ -12,9 +12,13 @@ import { StoreProvider } from "./ui/store.tsx"
 import { App } from "./ui/App.tsx"
 import { Onboarding } from "./ui/Onboarding.tsx"
 import { isDevMode } from "./dev/devMode.ts"
+import { SpinupWPClient } from "./api/client.ts"
+import { resolveSshAccess } from "./lib/cliSsh.ts"
+import { resolveIncidents } from "./lib/cliIncidents.ts"
 
 const args = process.argv.slice(2)
-const command = args.find((a) => !a.startsWith("-"))
+const positionals = args.filter((a) => !a.startsWith("-"))
+const command = positionals[0]
 
 if (args.includes("-h") || args.includes("--help") || command === "help") {
   const cfg = loadConfig()
@@ -24,6 +28,9 @@ Usage:
   spinuptui            Launch the dashboard
   spinuptui login      Set or update your saved API token
   spinuptui where      Print the config file path and token source
+  spinuptui ssh <domain>  Print SSH access info for a site (JSON)
+  spinuptui incidents <domain> | --all [--hours N]  Print Uptime Kuma
+                       down/up incidents for a site or the whole fleet (JSON)
   spinuptui --version  Print the version
   spinuptui --help     Show this help
 
@@ -48,6 +55,42 @@ if (command === "where") {
   process.exit(0)
 }
 
+if (command === "ssh") {
+  const domain = positionals[1]
+  if (!domain) {
+    console.error(JSON.stringify({ ok: false, reason: "usage", message: "Usage: spinuptui ssh <domain>" }))
+    process.exit(1)
+  }
+  const cfg = loadConfig()
+  const client = new SpinupWPClient(cfg)
+  const result = await resolveSshAccess(domain, client, cfg)
+  console.log(JSON.stringify(result))
+  process.exit(result.ok ? 0 : 1)
+}
+
+if (command === "incidents") {
+  const rest = args.slice(1)
+  const hoursIdx = rest.indexOf("--hours")
+  const hours = hoursIdx !== -1 ? Number(rest[hoursIdx + 1]) : 24
+  const allFlag = rest.includes("--all")
+  const domainArg = rest.find((a, i) => !a.startsWith("-") && !(hoursIdx !== -1 && i === hoursIdx + 1))
+
+  if ((!domainArg && !allFlag) || (domainArg && allFlag) || !Number.isFinite(hours) || hours <= 0) {
+    console.error(
+      JSON.stringify({
+        ok: false,
+        reason: "usage",
+        message: "Usage: spinuptui incidents <domain> | spinuptui incidents --all [--hours N]",
+      }),
+    )
+    process.exit(1)
+  }
+  const cfg = loadConfig()
+  const result = await resolveIncidents(cfg, { domain: domainArg, hours })
+  console.log(JSON.stringify(result))
+  process.exit(result.ok ? 0 : 1)
+}
+
 // `spinuptui login` forces the onboarding wizard even when a token already exists,
 // so the token can be (re)saved to the config file for global use.
 const forceLogin = command === "login"

package/src/lib/cliIncidents.ts

@@ -0,0 +1,126 @@
+// `spinuptui incidents <domain>` / `spinuptui incidents --all` — surfaces
+// Uptime Kuma's down/up incident history for the fleet SpinupTUI itself
+// manages monitoring for, for external tooling (e.g. an incident-diagnostics
+// agent polling on its own schedule). Scoped to config.kumaMonitors, not
+// Kuma's full live monitor list — a monitor added by hand in Kuma with no
+// corresponding SpinupWP site would have nowhere actionable for a caller to
+// go, so it's deliberately excluded.
+
+import type { AppConfig, KumaMonitorRef } from "../config.ts"
+import { withKuma, KumaError, type KumaBeat } from "./uptimeKuma.ts"
+
+export type IncidentKind = "health" | "push" | "redis" | "fingerprint" | "fatal" | "bypass"
+export type IncidentStatus = "down" | "up" | "pending" | "maintenance"
+
+export interface IncidentEvent {
+  domain: string
+  kind: IncidentKind
+  status: IncidentStatus
+  time: string
+  message: string
+}
+
+export type IncidentsResult =
+  | { ok: true; windowHours: number; events: IncidentEvent[] }
+  | { ok: false; reason: "kuma_not_configured" | "no_monitors_registered" | "kuma_error"; message: string }
+
+const STATUS_LABEL: Record<number, IncidentStatus> = { 0: "down", 1: "up", 2: "pending", 3: "maintenance" }
+
+const KIND_FIELDS: Array<[keyof KumaMonitorRef, IncidentKind]> = [
+  ["healthId", "health"],
+  ["pushId", "push"],
+  ["redisId", "redis"],
+  ["fingerprintId", "fingerprint"],
+  ["fatalId", "fatal"],
+  ["bypassId", "bypass"],
+]
+
+interface Target {
+  domain: string
+  kind: IncidentKind
+  id: number
+}
+
+export async function resolveIncidents(
+  cfg: AppConfig,
+  opts: { domain?: string; hours: number },
+): Promise<IncidentsResult> {
+  if (!cfg.uptimeKuma) {
+    return {
+      ok: false,
+      reason: "kuma_not_configured",
+      message: "No Uptime Kuma connection configured in SpinupTUI.",
+    }
+  }
+
+  const domains = opts.domain ? [opts.domain] : Object.keys(cfg.kumaMonitors)
+  const targets: Target[] = []
+  for (const domain of domains) {
+    const ref = cfg.kumaMonitors[domain]
+    if (!ref) continue
+    for (const [field, kind] of KIND_FIELDS) {
+      const id = ref[field]
+      if (typeof id === "number") targets.push({ domain, kind, id })
+    }
+  }
+
+  if (targets.length === 0) {
+    return {
+      ok: false,
+      reason: "no_monitors_registered",
+      message: opts.domain
+        ? `No Kuma monitors registered for "${opts.domain}" in SpinupTUI.`
+        : "No Kuma monitors registered for any site in SpinupTUI.",
+    }
+  }
+
+  try {
+    const { result } = await withKuma(cfg.uptimeKuma, async (kuma) => {
+      const perTarget = await Promise.all(
+        targets.map(async (t) => {
+          try {
+            return { t, beats: await kuma.getMonitorBeats(t.id, opts.hours) }
+          } catch {
+            return { t, beats: [] as KumaBeat[] }
+          }
+        }),
+      )
+      const events: IncidentEvent[] = []
+      for (const { t, beats } of perTarget) {
+        for (const b of beats) {
+          if (!b.important) continue
+          const status = STATUS_LABEL[b.status] ?? "pending"
+          const message = b.msg || ""
+          // The fatal sentinel is registered under the server's vanity domain,
+          // not the site that's actually broken — a down beat pushed BY OUR
+          // OWN cron carries the real, comma-separated affected domain(s) as
+          // msg instead. But Kuma can ALSO generate its own synthetic down
+          // beats on this same push monitor (e.g. "No heartbeat in the time
+          // window" if the cron itself stops beating) — those are an
+          // infrastructure signal about the server, not a parseable domain
+          // list, and must NOT be mistaken for one (confirmed live: this
+          // exact message has no commas and isn't a hostname). Only messages
+          // that parse cleanly as a comma-separated list of hostname-shaped
+          // tokens get split into per-domain events; anything else — Kuma's
+          // own wording included — stays tagged to the vanity domain like
+          // every other kind.
+          const domainList = message.split(",").map((d) => d.trim())
+          const looksLikeDomainList = domainList.every((d) => /^[a-zA-Z0-9](?:[a-zA-Z0-9-]*[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]*[a-zA-Z0-9])?)+$/.test(d))
+          if (t.kind === "fatal" && status === "down" && looksLikeDomainList) {
+            for (const domain of domainList) {
+              events.push({ domain, kind: t.kind, status, time: b.time, message })
+            }
+            continue
+          }
+          events.push({ domain: t.domain, kind: t.kind, status, time: b.time, message })
+        }
+      }
+      events.sort((a, b) => a.time.localeCompare(b.time))
+      return events
+    })
+    return { ok: true, windowHours: opts.hours, events: result }
+  } catch (err) {
+    const message = err instanceof KumaError ? err.message : `Unexpected error: ${(err as Error).message}`
+    return { ok: false, reason: "kuma_error", message }
+  }
+}

package/src/lib/cliSsh.ts

@@ -0,0 +1,161 @@
+// `spinuptui ssh <domain>` — non-interactive SSH access lookup for external
+// tooling (e.g. an incident-diagnostics agent that's handed only a domain).
+// Resolves domain -> site -> server -> SSH target, then runs a live
+// connectivity probe (BatchMode, same as probeSite) so the result is a
+// trustworthy go/no-go rather than a guess.
+
+import type { AppConfig } from "../config.ts"
+import type { SpinupWPClientLike } from "../api/client.ts"
+import { ApiError } from "../api/client.ts"
+import { resolveSiteSshTarget, SSH_OPTS } from "./probe.ts"
+
+export type SshAccessReason =
+  | "no_token"
+  | "token_rejected"
+  | "site_not_found"
+  | "multiple_matches"
+  | "server_has_no_ip"
+  | "permission_denied"
+  | "connection_failed"
+  | "ssh_error"
+  | "api_error"
+
+export interface SshAccessCandidate {
+  siteId: number
+  serverId: number
+  server: string
+}
+
+export type SshAccessResult =
+  | { ok: true; domain: string; primaryDomain: string; sshTarget: string; port: number | null; server: string }
+  | {
+      ok: false
+      domain: string
+      reason: SshAccessReason
+      message: string
+      remedy?: string
+      candidates?: SshAccessCandidate[]
+    }
+
+const GRANT_KEY_REMEDY =
+  "Run `spinuptui`, select this site in the Browser view, and press K to grant this device's key (sudo must be connected on the server first — press S)."
+
+export async function resolveSshAccess(
+  domain: string,
+  client: SpinupWPClientLike,
+  cfg: AppConfig,
+): Promise<SshAccessResult> {
+  if (!cfg.token) {
+    return { ok: false, domain, reason: "no_token", message: "No API token configured. Run `spinuptui login`." }
+  }
+
+  let site
+  let server
+  try {
+    const sites = await client.listSites()
+    const matches = sites.filter(
+      (s) => s.domain === domain || s.additional_domains?.some((a) => a.domain === domain),
+    )
+    if (matches.length === 0) {
+      return {
+        ok: false,
+        domain,
+        reason: "site_not_found",
+        message: `No site matching "${domain}" found in this SpinupWP account.`,
+      }
+    }
+    if (matches.length > 1) {
+      const candidates: SshAccessCandidate[] = await Promise.all(
+        matches.map(async (s) => {
+          const srv = await client.getServer(s.server_id)
+          return { siteId: s.id, serverId: s.server_id, server: srv.name }
+        }),
+      )
+      return {
+        ok: false,
+        domain,
+        reason: "multiple_matches",
+        message: `"${domain}" matches ${matches.length} sites in this account — cannot pick one automatically.`,
+        candidates,
+      }
+    }
+    site = matches[0]
+    server = await client.getServer(site.server_id)
+  } catch (err) {
+    if (err instanceof ApiError && err.status === 401) {
+      return {
+        ok: false,
+        domain,
+        reason: "token_rejected",
+        message: "Saved API token was rejected (401) — it may be expired or revoked.",
+        remedy: "Run `spinuptui login` to save a fresh token.",
+      }
+    }
+    const message = err instanceof ApiError ? err.message : `Unexpected error: ${(err as Error).message}`
+    return { ok: false, domain, reason: "api_error", message }
+  }
+
+  if (!server.ip_address) {
+    return {
+      ok: false,
+      domain,
+      reason: "server_has_no_ip",
+      message: `Server "${server.name}" has no IP address on file.`,
+    }
+  }
+
+  const target = resolveSiteSshTarget(site, server, cfg.sshUser)
+  const port = server.ssh_port && server.ssh_port !== 22 ? server.ssh_port : null
+  const portOpt = port ? ["-p", String(port)] : []
+
+  let proc: ReturnType<typeof Bun.spawn>
+  try {
+    proc = Bun.spawn(["ssh", ...SSH_OPTS, ...portOpt, target, "true"], {
+      stdout: "pipe",
+      stderr: "pipe",
+      stdin: "ignore",
+    })
+  } catch (err) {
+    return { ok: false, domain, reason: "ssh_error", message: `Failed to launch ssh: ${(err as Error).message}` }
+  }
+
+  const timeout = setTimeout(() => {
+    try {
+      proc.kill()
+    } catch {
+      /* already gone */
+    }
+  }, 15000)
+
+  const exitCode = await proc.exited
+  clearTimeout(timeout)
+  const stderr = await new Response(proc.stderr as ReadableStream<Uint8Array>).text()
+
+  if (exitCode === 0) {
+    return { ok: true, domain, primaryDomain: site.domain, sshTarget: target, port, server: server.name }
+  }
+
+  if (/permission denied/i.test(stderr)) {
+    return {
+      ok: false,
+      domain,
+      reason: "permission_denied",
+      message: stderr.trim().split("\n").slice(-2).join(" ") || "Permission denied.",
+      remedy: GRANT_KEY_REMEDY,
+    }
+  }
+  if (/(connection timed out|operation timed out|connection refused|could not resolve)/i.test(stderr)) {
+    return {
+      ok: false,
+      domain,
+      reason: "connection_failed",
+      message: stderr.trim().split("\n").slice(-2).join(" ") || `ssh exited with code ${exitCode}`,
+    }
+  }
+  return {
+    ok: false,
+    domain,
+    reason: "ssh_error",
+    message: stderr.trim().split("\n").slice(-2).join(" ") || `ssh exited with code ${exitCode}`,
+  }
+}

package/src/lib/probe.ts

@@ -60,7 +60,7 @@ export function resolveSiteSshTarget(site: Site, server: Server | undefined, ssh
   return `${user}@${ip}`
 }
 
-const SSH_OPTS = [
+export const SSH_OPTS = [
   "-o", "BatchMode=yes",
   "-o", "ConnectTimeout=7",
   "-o", "StrictHostKeyChecking=accept-new",

package/src/lib/ssh.ts

@@ -444,6 +444,52 @@ async function runSudoSiteOp(
   return { ok: true, target: `${siteUser}@${ip}` }
 }
 
+export type SudoServerResult = { ok: true } | { ok: false; error: string }
+
+// Same probe → maybe-password → `sudo -S -p '' bash -s` idiom as runSudoSiteOp,
+// but scoped to the SERVER as root — no Site/site_user involved. Used for ops
+// that need to read/write across every site's directory (e.g. the fatal-log
+// monitor's cron, which globs /sites/* — a regular site user can't even list
+// that directory, confirmed live).
+export async function runSudoServerOp(
+  server: Server,
+  opts: { sudoUser: string; sudoPassword: string },
+  build: () => string,
+  marker: string,
+): Promise<SudoServerResult> {
+  const ip = server.ip_address
+  if (!ip) return { ok: false, error: "Server has no IP address." }
+  const target = `${opts.sudoUser}@${ip}`
+  const port = server.ssh_port ?? null
+
+  const probe = await runSshStdin(target, port, "sudo -n true 2>&1", null, 15000)
+  let needPassword = false
+  if (probe.code === 0) {
+    needPassword = false
+  } else if (probe.code === 255 || probe.code === -1) {
+    return { ok: false, error: lastErrorLine(probe.stdout + "\n" + probe.stderr, "Couldn't connect over SSH — is your key on the server for this user?") }
+  } else {
+    const out = (probe.stdout + " " + probe.stderr).toLowerCase()
+    if (/not in the sudoers|not allowed to run|may not run|unknown user|is not allowed/.test(out)) {
+      return { ok: false, error: `${opts.sudoUser} can't run sudo on this server.` }
+    }
+    needPassword = true
+    if (!opts.sudoPassword) return { ok: false, error: "A sudo password is required for this server." }
+  }
+
+  const script = build()
+  const payload = needPassword ? `${opts.sudoPassword}\n${script}\n` : `${script}\n`
+  const res = await runSshStdin(target, port, "sudo -S -p '' bash -s", payload, 30000)
+  if (res.code !== 0 || !res.stdout.includes(marker)) {
+    const out = (res.stderr + " " + res.stdout).toLowerCase()
+    if (/incorrect password|sorry, try again|authentication failure/.test(out)) {
+      return { ok: false, error: "Sudo password was rejected." }
+    }
+    return { ok: false, error: lastErrorLine(res.stderr || res.stdout, `The remote command failed (ssh exit ${res.code}).`) }
+  }
+  return { ok: true }
+}
+
 // Append the given public keys to a site user's authorized_keys, via the server's
 // sudo user. Idempotent (re-running never duplicates a line). The caller resolves
 // which keys to deploy (machine key via ensureSpinupKey, personal via listPersonalKeys).