@rackbops/ac-agent 2.0.0-alpha.47 → 2.0.0-alpha.48

package/README.md

@@ -106,10 +106,58 @@ against the new bundle, `install` stops the old process and starts it again thro
 `stop`/`start` machinery the `ac-agent stop`/`ac-agent start` commands use, then confirms the new
 process is actually answering (a health check, retried for a few seconds — the OS command that
 launches a process can report success before that process has actually started listening) before
-printing `restarted: now serving <version>`. If the restart fails outright, or the health check
-never comes back healthy, the install is still reported as a success — the printed line is
-`restart failed: <reason>; run ac-agent stop && ac-agent start`, and that manual sequence is the
-fallback. A fresh install, with nothing running yet, prints neither line.
+printing `restarted: now serving <version>`. Only a health answer reporting the **new** version
+counts (#659), so the old process still answering on the port is not mistaken for the new one.
+
+Before copying the new bundle in over a running agent, `install` keeps the one it replaces as
+`bin/ac-agent.prev.mjs` in the state dir (#659; one backup, overwritten by the next install). If the
+new bundle never answers health at its own version, `install` rolls back:
+1. It stops the task/unit again. On Windows this ends `launch.js`'s retry loop, so it can't rerun the
+   broken bundle in the middle of the restore.
+2. It restores `ac-agent.prev.mjs` and starts it.
+3. It confirms the previous version answers, then prints
+   `rolled back: now serving <previous version> (<reason>)`. The previous version is read from the
+   running agent's own health answer just before it is stopped. If that read got no version (no
+   `cert.pem` to read it with, no answer, an error answer, or one without a version), any healthy
+   answer is accepted, and the line says
+   `now serving the previous bundle`.
+
+Whenever a rollback is possible, the restart confirmation waits up to a minute instead of a few
+seconds, because a false failure would undo a good release (the Windows cold start has never been
+timed). This applies to a plain install too, such as Tooling's `Update-AcAgent.ps1`. A new bundle that
+never comes up therefore takes about two minutes to report (a minute waiting for it, up to another
+confirming the rollback; a hung connection adds to each attempt). If the install fails before the
+restart (registering the task/unit, the certificate) and a backup was made, the running agent's bundle
+is copied back onto the disk; the error is reported as before.
+
+The install then **fails** (exit 1), because it did not take. If the rollback fails too, it prints
+`rollback failed: ...`, leaves both bundles in place, and also exits 1; `ac-agent start` is the
+fallback.
+
+When there was no bundle to go back to, a restart that fails outright still reports the install as a
+success, with `restart failed: <reason>; run ac-agent stop && ac-agent start`, and that manual
+sequence is the fallback. A fresh install, with nothing running yet, prints neither line.
+
+`install --update` is what the agent's own update route launches; it isn't meant to be run by hand.
+It differs from a plain install in two ways:
+- Its restart confirmation (when an agent is running to restart) is always the minute-long one,
+  even with no backup to roll back to.
+- It records each step in `update.json` in the state dir:
+  `{status: "installing" | "ok" | "rolled-back" | "failed", from, target, at, message?}`, where
+  `from` is null when the previous version could not be read.
+  - Once `installing` is written, the outcome is recorded as `ok`, `rolled-back` or `failed`. That
+    covers a failure before the restart, and no agent running afterwards.
+  - A failure before `installing` is written (reading the config, the backup copy) leaves the
+    previous record as it was.
+  - It writes a temp file and renames it over, so a reader never sees a partial record.
+  - A write that fails (for example a locked file) prints a `warning: could not record update
+    status ...` line and never changes the install's outcome. The record then still shows the last
+    write that landed, which may be `installing`, so a reader should treat an old `installing` as
+    unknown.
+
+A plain install never writes `update.json`. `ac-agent version --json` prints
+`{"version", "apiVersion"}`, which is how a staged bundle's agent API version is read without starting
+it.
 
 ### The firewall step
 
@@ -223,7 +271,8 @@ when this session's own AppData writes to the resolved config/state dirs are vir
 packaged app (see **Run install from a normal shell** above), which also prints a `warning: this
 session's AppData is virtualized (<packageFamily>); the logon task cannot see <stateDir> -- see
 README, Install` line to stderr. `serve` logs the same warning line once at startup, in the same
-circumstance, without refusing to start.
+circumstance, without refusing to start. `update` (#659) is the last `install --update`'s record from
+`update.json`, or `null` when there is none or it is unreadable.
 
 ## Pair
 
@@ -275,7 +324,7 @@ or `systemctl --user stop/start`) without touching the registration itself.
 | | Windows | Linux |
 |---|---|---|
 | Config (`agent.toml`) | `%APPDATA%\ac-agent\` | `~/.config/ac-agent/` |
-| State (cert, key, token hash, grants, log, pid, the installed bundle copy) | `%LOCALAPPDATA%\ac-agent\` | `~/.local/state/ac-agent/` |
+| State (cert, key, token hash, grants, log, pid, the installed bundle copy and its one backup, `update.json`) | `%LOCALAPPDATA%\ac-agent\` | `~/.local/state/ac-agent/` |
 
 Both are overridable with `AC_AGENT_CONFIG_DIR` / `AC_AGENT_STATE_DIR` — **absolute paths only**; a
 relative override throws, naming the value. `install` resolves these once, in its own process, and

package/dist/ac-agent.mjs

@@ -47,7 +47,7 @@ function _mergeNamespaces(n, m) {
 }
 
 var name = "@rackbops/ac-agent";
-var version = "2.0.0-alpha.47";
+var version = "2.0.0-alpha.48";
 var type = "module";
 var description = "The per-machine host agent (installed with npx @rackbops/ac-agent): an HTTPS health server plus install/status/uninstall for a Windows logon task or a systemd user unit.";
 var bin = {
@@ -21945,36 +21945,67 @@ function detectVirtualization(deps) {
 function bundleTargetPath(stateDir) {
     return path.join(stateDir, "bin", "ac-agent.mjs");
 }
-// #277 review round 1 (correctness lens): `deps.platform.start()`/`.stop()` are backed by
-// `runPowerShell`/`runCommand` (cli.ts), which resolve unconditionally on the child process's
-// `close` event and never inspect its exit code -- a `Start-ScheduledTask`/`systemctl start` that
-// runs but fails (bad permissions, a corrupted registration, ...) would otherwise be
-// indistinguishable here from one that actually worked, so `installService` could print `restarted:
-// now serving <version>` while nothing is actually listening. `confirmHealthy` below closes that gap
-// by requiring a real, successful `/v1/health` response -- not just "the launch command exited" --
-// before the restart is reported as a success. The new process is several process-creation hops away
-// on Windows (Start-ScheduledTask -> wscript.exe -> JScript -> node), so this retries briefly rather
-// than judging on a single immediate check.
-//
-// #277 review round 2: this budget (worst case: 7 delays x 400ms = 2.8s, plus the fetch attempts
-// themselves) is a deliberately generous, but UNVERIFIED-ON-WINDOWS, guess -- live verification for
-// this fix only covered Linux (nitro, over ssh); no live Windows box was installed against (Melody's
-// install refuses under MSIX AppData virtualization regardless, #263; MILE/SHIORI are roshne's own
-// boxes, out of scope here). A real Start-ScheduledTask -> wscript.exe -> JScript -> node cold start
-// that took longer than this window would report a false "restart failed" over an agent that's
-// actually fine a moment later -- a safe failure direction (never a false "restarted"), but still
-// worth a live Windows timing check before trusting this number long-term.
-const HEALTH_CHECK_ATTEMPTS = 8;
-const HEALTH_CHECK_DELAY_MS = 400;
-async function confirmHealthy(deps, port, certPem, expectedFingerprint) {
-    for (let attempt = 0; attempt < HEALTH_CHECK_ATTEMPTS; attempt++) {
+/** The one previous bundle `install` keeps while it restarts onto a new one (#659): restored when
+ *  the new bundle never answers health at its own version. Overwritten by the next install. */
+function bundleBackupPath(stateDir) {
+    return path.join(stateDir, "bin", "ac-agent.prev.mjs");
+}
+function updateRecordPath(stateDir) {
+    return path.join(stateDir, "update.json");
+}
+/** Best-effort: the record reports an install's outcome, it never decides it. A write that fails
+ *  (on Windows, `renameSync` over a file a reader holds open can throw EPERM/EBUSY; a full disk) is
+ *  printed and swallowed -- left to throw inside the restart sequence, it would roll back a release
+ *  that came up healthy (#659 review round 1). */
+function writeUpdateRecord(deps, record) {
+    const target = updateRecordPath(deps.stateDir);
+    const tmp = `${target}.tmp`;
+    try {
+        deps.writeFile(tmp, `${JSON.stringify(record, null, 2)}\n`);
+        deps.renameFile(tmp, target);
+    }
+    catch (err) {
+        deps.print(`warning: could not record update status ${record.status} in ${target}: ${err instanceof Error ? err.message : String(err)}`);
+    }
+}
+/** `update.json` parsed, or null when absent or unreadable -- `status` reports it as-is. */
+function readUpdateRecord(deps) {
+    const text = deps.readFile(updateRecordPath(deps.stateDir));
+    if (text === null)
+        return null;
+    try {
+        const value = JSON.parse(text);
+        return typeof value === "object" && value !== null && typeof value.status === "string"
+            ? value
+            : null;
+    }
+    catch {
+        return null;
+    }
+}
+const HEALTH_BUDGET = { attempts: 8, delayMs: 400 };
+/** `install --update` (#659): a false failure here rolls back a good release, so the update path
+ *  waits up to a minute -- the Windows cold start (Start-ScheduledTask -> wscript -> node) has never
+ *  been timed live, see above. */
+const UPDATE_HEALTH_BUDGET = { attempts: 60, delayMs: 1000 };
+/** Waits for a health answer that is `ok` and, when `expectedVersion` is given, reports exactly that
+ *  version (#659): the old process still answering on the port is not the new bundle coming up. */
+async function confirmHealthy(deps, port, certPem, expectedFingerprint, expectedVersion, budget) {
+    let lastVersion;
+    for (let attempt = 0; attempt < budget.attempts; attempt++) {
         const health = await deps.fetchHealth(port, certPem, expectedFingerprint);
-        if (health?.ok)
-            return;
-        if (attempt < HEALTH_CHECK_ATTEMPTS - 1) {
-            await new Promise((resolve) => setTimeout(resolve, HEALTH_CHECK_DELAY_MS));
+        if (health?.ok) {
+            if (expectedVersion === null || health.version === expectedVersion)
+                return;
+            lastVersion = health.version;
+        }
+        if (attempt < budget.attempts - 1) {
+            await new Promise((resolve) => setTimeout(resolve, budget.delayMs));
         }
     }
+    if (lastVersion !== undefined) {
+        throw new Error(`port ${port} still answers version ${lastVersion}, not ${expectedVersion}`);
+    }
     throw new Error(`new process did not answer a health check on port ${port}`);
 }
 async function installService(deps) {
@@ -22028,28 +22059,72 @@ async function installService(deps) {
         deps.print("warning: AC_AGENT_REPOS_ROOT overrides repos_root; the saved value will take effect when it is unset");
     }
     const bundlePath = bundleTargetPath(deps.stateDir);
-    deps.copyFile(deps.bundleSourcePath, bundlePath);
-    const { certPem } = await ensureCert({
-        stateDir: deps.stateDir,
-        hostname: deps.hostname(),
-        ips: deps.ips,
-        readFile: deps.readFile,
-        writeFile: (p, content, mode) => deps.writeFile(p, content, mode),
-    });
-    const fp = fingerprint(certPem);
-    // #263: refuse before registering a task/unit nothing can ever see, rather than leave a broken
-    // registration behind for `status`/`start` to fail against later. Must run after the state-dir
-    // writes above (so a genuinely virtualized session's own writes are what the probe finds) and
-    // before `deps.platform.install` (which is what actually registers the task/unit on Windows).
-    const virtualization = detectVirtualization(deps);
-    if (virtualization.virtualized) {
-        throw new Error(installRefusalMessage(virtualization.packageFamily, virtualization.vfsPath));
+    const backupPath = bundleBackupPath(deps.stateDir);
+    // #659: an install over a running agent keeps the bundle it is replacing, and the version that
+    // bundle is serving, so a new bundle that never comes up can be undone. Read before the copy: once
+    // the copy lands, the file on disk is no longer what the running process loaded.
+    const wasRunning = isAgentRunning(deps);
+    let backedUp = false;
+    let previousVersion = null;
+    if (wasRunning && deps.exists(bundlePath)) {
+        previousVersion = await runningVersion(deps, port);
+        deps.copyFile(bundlePath, backupPath);
+        backedUp = true;
+    }
+    const record = (status, message) => {
+        if (!deps.updateMode)
+            return;
+        writeUpdateRecord(deps, {
+            status,
+            from: previousVersion,
+            target: deps.version,
+            at: nowIso(deps),
+            ...(message !== undefined ? { message } : {}),
+        });
+    };
+    record("installing");
+    let certPem;
+    let fp;
+    let result;
+    try {
+        deps.copyFile(deps.bundleSourcePath, bundlePath);
+        certPem = (await ensureCert({
+            stateDir: deps.stateDir,
+            hostname: deps.hostname(),
+            ips: deps.ips,
+            readFile: deps.readFile,
+            writeFile: (p, content, mode) => deps.writeFile(p, content, mode),
+        })).certPem;
+        fp = fingerprint(certPem);
+        // #263: refuse before registering a task/unit nothing can ever see, rather than leave a broken
+        // registration behind for `status`/`start` to fail against later. Must run after the state-dir
+        // writes above (so a genuinely virtualized session's own writes are what the probe finds) and
+        // before `deps.platform.install` (which is what actually registers the task/unit on Windows).
+        const virtualization = detectVirtualization(deps);
+        if (virtualization.virtualized) {
+            throw new Error(installRefusalMessage(virtualization.packageFamily, virtualization.vfsPath));
+        }
+        result = await deps.platform.install({
+            nodePath: deps.nodePath,
+            bundlePath,
+            user: deps.user,
+        });
+    }
+    catch (err) {
+        // #659 review round 1: a failure before the restart leaves the old process running, so put its
+        // bundle back on disk (the next crash restart must load what was serving, not the half-installed
+        // new one) and record the outcome -- `installing` must never be the last word.
+        if (backedUp) {
+            try {
+                deps.copyFile(backupPath, bundlePath);
+            }
+            catch {
+                // the backup is still in place for a manual restore; the original error is what matters
+            }
+        }
+        record("failed", err instanceof Error ? err.message : String(err));
+        throw err;
     }
-    const result = await deps.platform.install({
-        nodePath: deps.nodePath,
-        bundlePath,
-        user: deps.user,
-    });
     deps.print(`config: ${path.join(deps.configDir, "agent.toml")} (${loaded.source})`);
     deps.print(`state: ${deps.stateDir}`);
     deps.print(`fingerprint: ${fp}`);
@@ -22063,19 +22138,101 @@ async function installService(deps) {
     // path -- so `stopService`'s #238 ordering (task/unit stop before the orphaned-pid kill) is
     // exercised here unchanged, just followed by the same `platform.start()` the `start` verb calls.
     if (isAgentRunning(deps)) {
+        // The long budget whenever a rollback is possible, not only under --update: a false failure there
+        // stops a healthy new process and restores the old bundle. Tooling's Update-AcAgent.ps1 runs a
+        // plain `install`, and the Windows cold start has never been timed (#659 review round 1). With
+        // nothing to roll back to, a failure only prints, so the short budget stays.
+        const budget = deps.updateMode || backedUp
+            ? (deps.rollbackHealthBudget ?? UPDATE_HEALTH_BUDGET)
+            : HEALTH_BUDGET;
+        let restartError = null;
         try {
             await stopService(deps);
             await deps.platform.start();
-            await confirmHealthy(deps, port, certPem, fp);
-            deps.print(`restarted: now serving ${deps.version}`);
+            await confirmHealthy(deps, port, certPem, fp, deps.version, budget);
         }
         catch (err) {
-            // The install itself already succeeded (the task/unit is registered against the new bundle);
-            // a restart failure must not be reported as an install failure -- the operator's fallback is
-            // the same manual stop/start #238 already documents.
-            deps.print(`restart failed: ${err instanceof Error ? err.message : String(err)}; run ac-agent stop && ac-agent start`);
+            restartError = err;
         }
+        if (restartError === null) {
+            // Recorded only after the restart sequence: a failed record write must never look like a
+            // failed restart (#659 review round 1).
+            deps.print(`restarted: now serving ${deps.version}`);
+            record("ok");
+            return;
+        }
+        const reason = restartError instanceof Error ? restartError.message : String(restartError);
+        if (!backedUp) {
+            // Nothing to go back to (no bundle was in place): the install itself already succeeded (the
+            // task/unit is registered against the new bundle), so a restart failure must not be reported
+            // as an install failure -- the operator's fallback is the manual stop/start #238 documents.
+            deps.print(`restart failed: ${reason}; run ac-agent stop && ac-agent start`);
+            record("failed", reason);
+            return;
+        }
+        await rollBack(deps, {
+            port,
+            certPem,
+            fp,
+            bundlePath,
+            backupPath,
+            previousVersion,
+            budget,
+            reason,
+        });
+        return;
     }
+    // Nothing was running to restart (the agent died between the pre-check and now, or a fresh install).
+    // Under --update the agent that launched this install was running, so this is a failure the record
+    // must say, not leave at `installing` (#659 review round 2).
+    record("failed", "no agent was running after the install; run ac-agent start");
+}
+function nowIso(deps) {
+    return (deps.now ?? (() => new Date()))().toISOString();
+}
+/** The version the running agent reports before it is stopped (#659), read with the cert already in
+ *  the state dir -- null when there is no cert yet or the agent does not answer. */
+async function runningVersion(deps, port) {
+    const certPem = deps.readFile(path.join(deps.stateDir, "cert.pem"));
+    if (certPem === null)
+        return null;
+    const health = await deps.fetchHealth(port, certPem, fingerprint(certPem));
+    return health?.ok && typeof health.version === "string" ? health.version : null;
+}
+/** #659: the new bundle never answered health at its own version, so restore the one it replaced.
+ *  Stops first, in #238's order (the task/unit, then the pid) -- on Windows that ends `launch.js`'s
+ *  retry loop, which would otherwise rerun the broken bundle a minute later, mid-restore. Throws
+ *  either way, so `install` exits 1: the install did not take. */
+async function rollBack(deps, ctx) {
+    const record = (status, message) => {
+        if (!deps.updateMode)
+            return;
+        writeUpdateRecord(deps, {
+            status,
+            from: ctx.previousVersion,
+            target: deps.version,
+            at: nowIso(deps),
+            message,
+        });
+    };
+    try {
+        await stopService(deps);
+        deps.copyFile(ctx.backupPath, ctx.bundlePath);
+        await deps.platform.start();
+        await confirmHealthy(deps, ctx.port, ctx.certPem, ctx.fp, ctx.previousVersion, ctx.budget);
+    }
+    catch (err) {
+        const why = err instanceof Error ? err.message : String(err);
+        deps.print(`rollback failed: ${why}; the new bundle failed first (${ctx.reason}). ` +
+            `${ctx.backupPath} holds the previous bundle: copy it over ${ctx.bundlePath} if it is not ` +
+            "already there, then run ac-agent start");
+        record("failed", `${ctx.reason}; rollback failed: ${why}`);
+        throw new Error(`install did not take and the rollback failed: ${why}`);
+    }
+    const serving = ctx.previousVersion ?? "the previous bundle";
+    deps.print(`rolled back: now serving ${serving} (${ctx.reason})`);
+    record("rolled-back", ctx.reason);
+    throw new Error(`install did not take: ${deps.version} never came up; rolled back to ${serving}`);
 }
 async function status(deps, now = () => new Date()) {
     const loaded = loadConfig(deps.configDir, {
@@ -22113,6 +22270,7 @@ async function status(deps, now = () => new Date()) {
         grants,
         virtualized: virtualization.virtualized,
         lanAddress: deps.ips[0] ?? null,
+        update: readUpdateRecord(deps),
     };
 }
 /** Generates a one-time pairing code (#48): refuses when already paired unless `force` (which is how
@@ -22609,6 +22767,8 @@ async function dispatch(argv, io) {
                         port: { type: "string" },
                         name: { type: "string" },
                         "repos-root": { type: "string" },
+                        // #659: set only by the agent's own update route -- see ServiceDeps.updateMode.
+                        update: { type: "boolean", default: false },
                     },
                     allowPositionals: false,
                 });
@@ -22619,6 +22779,7 @@ async function dispatch(argv, io) {
                     port: values.port !== undefined ? Number.parseInt(values.port, 10) : undefined,
                     name: values.name,
                     reposRoot,
+                    update: values.update ?? false,
                 });
                 return 0;
             }
@@ -22661,9 +22822,19 @@ async function dispatch(argv, io) {
             case "publish-adopt":
                 await io.publishAdopt();
                 return 0;
-            case "version":
-                io.print(packageJson.version);
+            case "version": {
+                const { values } = parseArgs({
+                    args: rest,
+                    options: { json: { type: "boolean", default: false } },
+                    allowPositionals: false,
+                });
+                // #659: `--json` is how the update route reads a staged bundle's API version without
+                // starting it.
+                io.print(values.json
+                    ? JSON.stringify({ version: packageJson.version, apiVersion: AGENT_API_VERSION })
+                    : packageJson.version);
                 return 0;
+            }
             default:
                 io.printError(USAGE);
                 return 1;
@@ -22854,6 +23025,13 @@ async function buildPlatformOps(port, unitPath, launchScriptPath, configDir, sta
 function systemdUnitPath(homedir) {
     return path.posix.join(homedir, ".config", "systemd", "user", "ac-agent.service");
 }
+/** A `/v1/health` answer as `install`/`status` read it: the HTTP status, plus the reported version
+ *  when the body carries one (#659) -- that version is what tells a restarted new bundle from the old
+ *  process still answering on the port. */
+function healthResultFrom(ok, body) {
+    const version = typeof body === "object" && body !== null ? body.version : undefined;
+    return typeof version === "string" ? { ok, version } : { ok };
+}
 async function buildServiceDeps(overrides = {}) {
     const os = await import('node:os');
     const fs = await import('node:fs');
@@ -22886,6 +23064,7 @@ async function buildServiceDeps(overrides = {}) {
         },
         readFile,
         writeFile: (p, content, mode) => fs.writeFileSync(p, content, mode !== undefined ? { mode } : undefined),
+        renameFile: (from, to) => fs.renameSync(from, to),
         removeFile: (p) => {
             try {
                 fs.unlinkSync(p);
@@ -22912,6 +23091,7 @@ async function buildServiceDeps(overrides = {}) {
         machineNameOverride: overrides.name,
         portOverride: overrides.port,
         reposRootOverride: overrides.reposRoot,
+        updateMode: overrides.update ?? false,
         env: process.env,
         osPlatform: process.platform,
         readdir: (p) => fs.readdirSync(p),
@@ -22939,7 +23119,14 @@ async function buildServiceDeps(overrides = {}) {
                 const res = await undiciFetch(`https://127.0.0.1:${healthPort}/v1/health`, {
                     dispatcher: agent,
                 });
-                return { ok: res.ok };
+                let body;
+                try {
+                    body = await res.json();
+                }
+                catch {
+                    // an unparseable body still counts by its status alone, as before
+                }
+                return healthResultFrom(res.ok, body);
             }
             catch {
                 return null;
@@ -74959,4 +75146,4 @@ var tunnel = /*#__PURE__*/Object.freeze({
   registerTunnelVerb: registerTunnelVerb
 });
 
-export { AGENT_API_VERSION, buildRepoFs, buildWhichFs, dispatch, isEntryPoint, main, realExists };
+export { AGENT_API_VERSION, buildRepoFs, buildWhichFs, dispatch, healthResultFrom, isEntryPoint, main, realExists };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@rackbops/ac-agent",
-  "version": "2.0.0-alpha.47",
+  "version": "2.0.0-alpha.48",
   "private": false,
   "type": "module",
   "description": "The per-machine host agent (installed with npx @rackbops/ac-agent): an HTTPS health server plus install/status/uninstall for a Windows logon task or a systemd user unit.",