@catalyst-cloud/cli 0.16.41 → 0.16.43

package/CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## 0.16.43
+
+`catalyst ask accept` now waits for an answer that has not reached Catalyst yet, instead of failing. If you accept an ask a few seconds after answering it, Catalyst may not have your answer comment yet. The command used to stop with a refusal, and the same command worked when you ran it again. Now it waits as long as Catalyst asks and tries again, for up to 30 seconds. If the answer still has not arrived, it says so and tells you when to run the same command again. A refusal that will not change on retry is reported at once, with its reason.
+
+## 0.16.42
+
+A busy machine now takes a waiting release after one day. Before, a busy machine could keep putting off a release that cannot take over from running work in place. Now the next check asks Catalyst to send the machine no new work, waits for the running work to finish, then moves to the release and takes work again. `catalyst runner update --check` says when a machine is waiting like this, and `catalyst runner auto-update off` ends the wait. If the update cannot restore the machine's private thoughts storage, the machine keeps taking no new work, and `catalyst runner update` tells you what to do. This release also makes updates on Catalyst's own machines more reliable.
+
 ## 0.16.41
 
 A runner's automatic update schedule now applies a newly approved runner release at its next check, instead of waiting for one of four set times of day. An owner who wants set times can still choose them with `catalyst update policy windows <HH:MM,...>`, and `catalyst update policy windows default` goes back to applying at the next check. `catalyst runner update` still applies a release at once, and `catalyst update policy` shows which setting is in effect.

package/README.md

@@ -346,7 +346,7 @@ The skill files themselves are written by whichever install command you ran, in
 
 The new CLI is installed beside the old one, and your `catalyst` command runs it at the same path. A running local replica restarts on it and keeps its synced data; a stopped one stays stopped. CLI, SDK and image updates keep their existing policy against moving to an older version. Ordinary CLI startup never replaces skills with its bundled copy.
 
-On a machine checking for releases every 15 minutes (`catalyst runner auto-update on`), that check applies the same component policies. Otherwise components move when you run `catalyst update`. That check moves the runner images to a newer release as soon as it finds one, with no set times of day. An update does not stop work that is already running: that work finishes on the image it started with. If you want updates only at set times, `catalyst update policy windows 06:00,18:00 --time-zone America/Chicago` sets your own times, `catalyst update policy windows default` goes back to none, and `catalyst runner update` or `catalyst update` applies a waiting release at once. Each component is on by default; turn one off with `catalyst update policy <component> off` (for example `cli`), and `catalyst update policy` lists every setting. If you turned the installer's daily update off, the CLI and both skills packs start held, and `catalyst update policy cli on` turns the CLI back on.
+On a machine checking for releases every 15 minutes (`catalyst runner auto-update on`), that check applies the same component policies. Otherwise components move when you run `catalyst update`. That check moves the runner images to a newer release as soon as it finds one, with no set times of day. An update does not stop work that is already running: that work finishes on the image it started with. When work is running and the new release cannot take over from it in place, the check leaves the release waiting. Once a release has waited a day, the next check asks Catalyst to send the machine no new work. It waits for the running work to finish, however long that takes, then moves to the release and takes work again. `catalyst runner update --check` says when a machine is waiting like this, and `catalyst runner auto-update off` ends the wait. If you want updates only at set times, `catalyst update policy windows 06:00,18:00 --time-zone America/Chicago` sets your own times, `catalyst update policy windows default` goes back to none, and `catalyst runner update` or `catalyst update` applies a waiting release at once. Each component is on by default; turn one off with `catalyst update policy <component> off` (for example `cli`), and `catalyst update policy` lists every setting. If you turned the installer's daily update off, the CLI and both skills packs start held, and `catalyst update policy cli on` turns the CLI back on.
 
 The installer's daily job runs the same release check, so it moves the CLI and the Cloud skills only to the approved release and keeps your holds. Until the development skills join the release, it refreshes them from their repository.
 

package/dist/ask.js

@@ -9,7 +9,7 @@ import { CliError, UsageError } from "./errors.js";
 import { apiClient } from "./transport.js";
 import { fetchAllPages } from "./pagination.js";
 import { fetchWorkflowStates, postAgent, resolveIssue } from "./write.js";
-export async function cmdAsk(args, ctx) {
+export async function cmdAsk(args, ctx, deps = {}) {
     const [sub, ...rest] = positionals(args);
     if (!sub)
         throw new UsageError("ask needs a subcommand: raise | accept | answer | proxy | disown | list");
@@ -33,7 +33,7 @@ export async function cmdAsk(args, ctx) {
         case "raise":
             return raise(args, ctx, doc, api);
         case "accept":
-            return accept(args, ctx, doc, api, rest);
+            return accept(args, ctx, doc, api, rest, deps.sleep ?? ((ms) => new Promise((r) => setTimeout(r, ms))));
         case "answer":
             return answerByProxy(args, ctx, doc, api, proxyBody);
         case "list":
@@ -128,20 +128,36 @@ function askGates(args, options, doc) {
     }
     return { repo, pr: pr.length === 1 ? pr[0] : pr, labels: labels.length ? labels : ["hold"], releasedBy: released };
 }
-async function accept(args, ctx, doc, api, rest) {
+/** The most `ask accept` waits, in total, for an answer comment the cloud has not copied yet. */
+export const ASK_ACCEPT_RETRY_BUDGET_MS = 30_000;
+async function accept(args, ctx, doc, api, rest, sleep) {
     const askTicket = rest[0];
     const answer = flagString(args, "answer");
     const role = flagString(args, "role");
     if (!askTicket || !answer || !role)
         throw new UsageError("ask accept needs <askTicket> --answer <commentId> --role <role>");
     const ask = await resolveIssue(api, askTicket);
-    const result = await postAgent(api, doc, "ask-accept", {
-        askIssueId: ask.id,
-        answerCommentId: answer,
-        acceptedByRole: role,
-    });
-    ctx.stdout(args.json ? JSON.stringify(result) : `ask ${ask.identifier}: answer ${answer} recorded by ${role}`);
-    return 0;
+    const body = { askIssueId: ask.id, answerCommentId: answer, acceptedByRole: role };
+    // An answer posted seconds ago may not have reached the cloud yet. The cloud refuses that one case
+    // as retryable and says how long to wait; every other refusal is final and is not retried.
+    let waited = 0;
+    for (;;) {
+        const result = await postAgent(api, doc, "ask-accept", body, { accept: [409] });
+        if (result.outcome !== "refused") {
+            ctx.stdout(args.json ? JSON.stringify(result) : `ask ${ask.identifier}: answer ${answer} recorded by ${role}`);
+            return 0;
+        }
+        const waitMs = Math.max(1, result.retryAfterSeconds ?? 10) * 1000;
+        if (result.retryable === true && waited + waitMs <= ASK_ACCEPT_RETRY_BUDGET_MS) {
+            await sleep(waitMs);
+            waited += waitMs;
+            continue;
+        }
+        const reason = result.reason ?? "refused";
+        throw new CliError(result.retryable === true
+            ? `ask ${ask.identifier}: answer ${answer} has not reached the cloud yet (${reason}) after ${waited / 1000}s of retries; run the same command again in ${result.retryAfterSeconds ?? 10}s`
+            : `ask ${ask.identifier}: accept refused (${reason})`, result.retryable === true ? "ask-accept-retry" : "ask-accept-refused", 2, 409);
+    }
 }
 /** The cloud's bounds on an answer by proxy, checked here first so a refusal never costs a write. */
 const QUOTE_MAX = 4000;

package/dist/cli.js

@@ -531,7 +531,7 @@ export async function main(argv, ctx = defaultCtx(), deps = {}) {
             case "write":
                 return await cmdWrite(args, ctx, deps.write);
             case "ask":
-                return await cmdAsk(args, ctx);
+                return await cmdAsk(args, ctx, { sleep: deps.sleep });
             case "ready":
                 return await cmdReady(args, ctx, {
                     skillNames: CUSTOMER_SKILLS,

package/dist/runner-command.js

@@ -1,11 +1,11 @@
 import { spawnSync } from "node:child_process";
-import { realpathSync } from "node:fs";
+import { existsSync, realpathSync } from "node:fs";
 import { isAbsolute, resolve } from "node:path";
 import { flagBool, flagString, positionals } from "./args.js";
 import { CliError, UsageError } from "./errors.js";
 import { autoUpdateStatus, custodyNode, disableAutoUpdate, enableAutoUpdate, scheduledPath, } from "./runner-auto-update.js";
-import { dockerRunnerUpdateEngine, readReleaseState, RELEASE_COMPONENTS, runnerDir, runRunnerUpdate, updateResultLines, lastProblem, lastProblemLines, } from "./runner-update.js";
-import { dockerFleetUpdateEngine, postRollLines, readPostRollResult, runFleetPostRollCheck, runFleetUpdate } from "./runner-update-fleet.js";
+import { dockerRunnerUpdateEngine, readReleaseState, RELEASE_COMPONENTS, runnerDir, runRunnerUpdate, updateResultLines, lastProblem, lastProblemLines, drainLines, custodyUnattested, CUSTODY_UNATTESTED_LINE, releaseUpdateDrain, } from "./runner-update.js";
+import { dockerFleetUpdateEngine, postRollLines, readPostRollResult, runFleetPostRollCheck, runFleetUpdate, } from "./runner-update-fleet.js";
 import { runUpdate, sharedReleaseRead, updateReportJson, updateReportLines } from "./update.js";
 import { effectivePolicy, readUpdatePolicy, setComponentPolicy } from "./update-policy.js";
 import { resolveClaude } from "./update-plugin.js";
@@ -69,6 +69,18 @@ function statusLine(status) {
         return "Automatic runner updates need your systemd user manager, which is not running for you. Run: sudo loginctl enable-linger $USER, then catalyst runner auto-update on";
     return `Automatic runner updates are not available on this machine (${status.detail}). Run catalyst runner update yourself.`;
 }
+function autoLine(result, at) {
+    return `${at.toISOString()} ${result.outcome} ${result.detail} generation=${result.generation ?? "none"}`;
+}
+function drainReleaseLine(result) {
+    if (result === "released")
+        return "Catalyst can send this machine new phases again: it had stopped sending them so the machine could update.";
+    if (result === "pending")
+        return "The update that was waiting for running phases ends within a few seconds, and Catalyst can then send this machine new phases again.";
+    if (result === "failed")
+        return "Catalyst may still be holding new phases back from this machine. Run catalyst runner auto-update off again to retry.";
+    return null;
+}
 /** Why this machine's images must not move now: an owner's hold, or settings that cannot be read
  *  (which never count as "everything on"). Null when every image may move. */
 function imageHold(ctx) {
@@ -145,7 +157,9 @@ export async function cmdRunner(args, ctx, deps = {}) {
             if (args.json)
                 ctx.stdout(JSON.stringify(extra ? { ...result, components: extra.rows } : result));
             else if (auto) {
-                ctx.stdout(`${ctx.now().toISOString()} ${result.outcome} ${result.detail} generation=${result.generation ?? "none"}`);
+                ctx.stdout(autoLine(result, ctx.now()));
+                if (custodyUnattested(result))
+                    ctx.stdout(CUSTODY_UNATTESTED_LINE);
                 if (extra)
                     ctx.stdout(extra.line);
             }
@@ -160,13 +174,27 @@ export async function cmdRunner(args, ctx, deps = {}) {
         // windows; then only in a window. An owner's own run applies at once (CTC-7465, update-windows.ts).
         if (hold !== null) {
             const retained = { outcome: "retained", detail: hold, generation: readReleaseState(dir()).generation, exitCode: hold === "owner_setting" ? 0 : 1 };
+            if (hold === "owner_setting" && !fleet) {
+                const released = await releaseUpdateDrain(engine, dir(), { cloud: false });
+                if (released === "failed" && !auto && !args.json)
+                    ctx.stdout(drainReleaseLine(released));
+            }
             // Unreadable settings move nothing; a hold leaves the other components to the release check.
             return print(retained, hold === "owner_setting" ? await components(undefined) : null);
         }
         const ownerWindows = auto && !check && !fleet ? readUpdatePolicy(ctx.home).windows : undefined;
+        // A fleet supervisor that misses its qualification deadline names each scope it had not qualified,
+        // and why, on stderr before the restore, so stdout keeps its one line or JSON object (CTC-8163).
         const result = fleet
-            ? await runFleetUpdate(deps.fleetEngine ?? dockerFleetUpdateEngine({ env: ctx.env }), { dir: dir(), check })
-            : await runRunnerUpdate(engine, { dir: dir(), check, ...(ownerWindows ? { window: windowBounds(ctx.now().getTime(), ownerWindows) } : {}) });
+            ? await runFleetUpdate(deps.fleetEngine ?? dockerFleetUpdateEngine({ env: ctx.env }), {
+                dir: dir(), check, ...deps.fleetClock,
+                onQualificationPending: (line) => ctx.stderr(`catalyst: ${line}`),
+            })
+            : await runRunnerUpdate(engine, {
+                dir: dir(), check,
+                ...(ownerWindows ? { window: windowBounds(ctx.now().getTime(), ownerWindows) } : {}),
+                ...(auto && !check ? { drain: true, held: () => imageHold(ctx), ...(args.json ? {} : { onPass: (pass) => ctx.stdout(autoLine(pass, ctx.now())) }) } : {}),
+            });
         // `runner update --check` also lists every installed component against the approved release, as
         // `catalyst update --check` does. The image fields stay where scripts already read them.
         if (check && !fleet && !auto) {
@@ -190,10 +218,10 @@ export async function cmdRunner(args, ctx, deps = {}) {
             const state = readReleaseState(dir());
             if (args.json) {
                 const { schemaVersion, release, components, projects } = updateReportJson(report);
-                ctx.stdout(JSON.stringify({ ...result, schemaVersion, release, components, projects, lastProblem: lastProblem(state) }));
+                ctx.stdout(JSON.stringify({ ...result, schemaVersion, release, components, projects, lastProblem: lastProblem(state), drain: state.drain === null ? null : { generation: state.drain.generation, acknowledgedAtMs: state.drain.acknowledgedAtMs }, deferral: state.deferral }));
             }
             else
-                for (const line of [...updateResultLines(result), ...lastProblemLines(state), ...updateReportLines(report)])
+                for (const line of [...updateResultLines(result), ...lastProblemLines(state), ...drainLines(state), ...updateReportLines(report)])
                     ctx.stdout(line);
             return result.exitCode;
         }
@@ -229,12 +257,25 @@ export async function cmdRunner(args, ctx, deps = {}) {
         if (mode === "off" || (mode === "on" && status.state === "on"))
             for (const id of RELEASE_COMPONENTS)
                 setComponentPolicy(ctx.home, id, mode === "on");
+        let updateDrain = "none";
+        if (mode === "off" && !fleet) {
+            let runner = null;
+            try {
+                runner = dir();
+            }
+            catch { /* No runner directory to release. */ }
+            if (runner !== null && existsSync(resolve(runner, ".env")) && existsSync(resolve(runner, "compose.yaml")))
+                updateDrain = await releaseUpdateDrain(deps.engine ?? dockerRunnerUpdateEngine({ env: ctx.env }), runner, { cloud: true });
+        }
         if (args.json) {
             const state = mode === "status" && dirFlag !== undefined ? readReleaseState(dirFlag) : undefined;
-            ctx.stdout(JSON.stringify({ ...status, ...(state ? { release: state } : {}) }));
+            ctx.stdout(JSON.stringify({ ...status, ...(state ? { release: state } : {}), ...(mode === "off" ? { updateDrain } : {}) }));
         }
         else {
             ctx.stdout(statusLine(status));
+            const line = mode === "off" ? drainReleaseLine(updateDrain) : null;
+            if (line !== null)
+                ctx.stdout(line);
             // A single runner's schedule prints what it always has; a fleet's says so, and for which directory.
             if (status.state === "on" && status.fleet)
                 ctx.stdout(`This is a supervisor fleet's schedule: it updates the compose.json in ${status.dir}.`);