toolroll 0.9.77 → 0.9.78

package/CHANGELOG.md

@@ -1,5 +1,86 @@
 # Changelog
 
+## 0.9.78 — 2026-10-29
+
+- **Toolroll says when it is restarting.** The worker and console now record
+  when they start, keep a heartbeat every 15 seconds, and record when they
+  begin stopping and when they have stopped. While one is stopping it takes no
+  new work, `/healthz` answers "draining" at once, pages say "Toolroll is
+  restarting", and a new start waits only until the old one has finished
+  stopping instead of a fixed window. A worker that stopped cleanly never
+  holds its projects. A second live `up` is still refused.
+- **The database is copied every few minutes.** Besides scheduled backups,
+  the worker copies the database online every 5 minutes (change it, or the
+  folder, in Settings → Backups), checks each copy and keeps the newest 24
+  and one a day for a week. It never holds up the console or the worker. A
+  deploy uses the latest copy when it is recent and matches the database
+  exactly, instead of making a new one. Restoring stays a manual
+  `toolroll restore`.
+- **See where the time went.** A task's page has a "Where the time went"
+  section under Details: when each part of its release check started and
+  ended, and each step of its deploys, including what a deploy waited for.
+  Console pages that take over a second keep a short record of where the
+  time went, which an instance operator reads on `/health`. Only page names,
+  step names and reasons are kept, never addresses or form contents.
+- **A crashed worker is noticed in a minute, not three.** Workers check in
+  every 15 seconds and are treated as gone after a minute of silence. A
+  project lease is never taken for longer than a minute.
+- **A release's version bump alone no longer owes the upgrade path.** Toolroll
+  now decides from both commits' verified files that package.json and
+  package-lock.json changed only in the project's own version (the check's
+  own rule); any other change to them still owes it.
+- **The macOS suite runs on the candidate before it merges.** Pull requests
+  and main pushes now run the unit suite on hosted macOS with Node 22 and 24,
+  in four shards each, joined into `node 22 · macos-latest` and
+  `node 24 · macos-latest` exactly like the Linux checks. `toolroll release`
+  waits for them; main's protection not requiring them yet is reported, not
+  refused. After the merge, publication waits only for the tag's publish job,
+  no longer for main's own CI run on the merge commit. The nightly and tag
+  macOS jobs are renamed so no other workflow shares a required name.
+- **A release is published without waiting for Homebrew.** `toolroll release`
+  ends with "Published; Homebrew pending" once npm and the GitHub release are
+  verified, instead of waiting up to 45 minutes for the tap's checks. Homebrew
+  is its own state in the release journal (pending, failed or current), and
+  running the same command again finishes it: it waits for the tap change's
+  checks, then merges exactly that change, with the checksum still taken from
+  npm's tarball. A tap already on a newer version is never moved back, checked
+  again right before the merge.
+- **A release whose checks run outside the sandbox now reruns its own check
+  after Toolroll's machine fails.** Such a release can't carry later commits,
+  so it used to file without release terms, and a check stopped by Toolroll's
+  own shutdown, a lost build or a command that never started waited for a
+  person. Its approval now seals a release check purpose instead: the check
+  is held to the release's rules (a complete report, no CI standing in for
+  its units), and only a failure Toolroll itself confirmed as its own
+  infrastructure reruns, on the same commit under the same approval, at most
+  twice. Another commit, a changed scope, release check or check access, or
+  an approval that no longer stands never reruns. It never carries a later
+  commit. (Schema v138 adds the column; saved scopes are unchanged.)
+- **A deploy no longer gets stuck behind an `up` that won't stop.** When the
+  old service's processes outlive their stop (a supervisor that exited while
+  its `up` ignored SIGTERM), the deployment stops them itself, but only
+  processes proved, by exact runtime path and arguments, to be that service's
+  supervisor, `up` or console and worker. Anything it can't prove refuses and
+  nothing is signalled.
+- **A killed `up` no longer blocks the next start for three minutes.** `up`
+  records each process it gives its worker credential to. A new start that
+  proves every one of them has exited signs the old worker off and takes over
+  at once; a second live `up` is still refused.
+- **`up` signs off however it is stopped.** SIGTERM, Ctrl-C, its supervisor
+  dying or its channel closing all end with the worker retired. A stop that
+  hangs past the worker's grace stops the console and worker outright
+  instead of waiting for launchd to kill it.
+- **Deployments wait long enough, and say why.** Restoring the previous
+  service, starting the new one and waiting for its project leases now allow
+  two minutes (past a stale worker heartbeat's one) and name what they are
+  waiting for. "Current work must finish first" now says what is running and
+  what to do.
+- **One command puts back a deployment that stopped half way.**
+  `toolroll release --recover-deploy` shows what recovery will do for the
+  unfinished deployment; `--yes` does it: stops only that service's proved
+  leftovers, starts the previous service again and lifts the update pause.
+  It is safe to run again.
+
 ## 0.9.77 — 2026-10-29
 
 - **CI finishes in about a third of the time.** The unit suite runs in four

package/dist/agent-fence.js

@@ -33,6 +33,7 @@ import { basename, dirname, isAbsolute, join, relative, resolve, sep } from "nod
 import { readMonitoring } from "./monitoring-settings.js";
 import { existingFolders } from "./names.js";
 import { ensureGitViewsRoot, gitViewsRoot } from "./git-views.js";
+import { ensureRunnerProcessFolder, RUNNER_PROCESS_DIR } from "./runner-process.js";
 import { DEPS_FOLDER, linkedKey, readyCopy, sharedCopies } from "./shared-deps.js";
 import { SEATBELT_PROBE, sandboxRuntimeProblem, srtSpawn } from "./sandbox-runtime.js";
 import { protectedStorageDenies, protectedStorageNames } from "./protected-storage.js";
@@ -167,6 +168,21 @@ export function agentFence(options) {
             catch { /* fenced below either way */ }
         }
         fence.add(runFolder);
+        const runnerFolder = join(state, RUNNER_PROCESS_DIR);
+        try {
+            ensureRunnerProcessFolder(runnerFolder);
+        }
+        catch { /* both the name and its target stay denied */ }
+        fence.add(runnerFolder);
+        fence.add(real(runnerFolder));
+        // A copy may first appear after launch. Fence its permanent namespace even when empty.
+        const backups = join(state, "backups");
+        try {
+            mkdirSync(backups, { mode: 0o700 });
+        }
+        catch { /* name and target remain denied */ }
+        fence.add(backups);
+        fence.add(real(backups));
         // Only a folder that is Toolroll's own (~/.config/toolroll or an older name,
         // the desktop app's "Toolroll") is fenced whole: a database kept
         // in a shared folder (a projects folder, a test's temp folder) must not
@@ -214,7 +230,7 @@ export function agentFence(options) {
     // Secret, state, database, explicit-extra and home-config denies are never exemptible.
     for (const path of options.boundary?.protectedShell ?? [])
         fence.add(real(path));
-    const protectedPaths = new Set([...fence].flatMap(path => path === runFolder ? [path, real(path)] : [real(path)]));
+    const protectedPaths = new Set([...fence].flatMap(path => path === runFolder || [RUNNER_PROCESS_DIR, "backups"].includes(basename(path)) ? [path, real(path)] : [real(path)]));
     // Protected storage (chat apps, browser profiles, password managers): every launch, every policy, never exemptible,
     // kept by the name it was resolved under as well as where it leads.
     const storage = options.protectedStorage ?? protectedStorageDenies(home);
@@ -226,7 +242,7 @@ export function agentFence(options) {
         fence.add(path);
     const binding = options.binding;
     const needed = binding === undefined || binding === null ? [] : [binding.gitdir, real(join(binding.main, ".git"))];
-    return [...new Set([...[...fence].flatMap(path => path === runFolder ? [path, real(path)] : [real(path)]), ...storage.flatMap(path => [path, real(path)])])].filter(path => {
+    return [...new Set([...[...fence].flatMap(path => path === runFolder || [RUNNER_PROCESS_DIR, "backups"].includes(basename(path)) ? [path, real(path)] : [real(path)]), ...storage.flatMap(path => [path, real(path)])])].filter(path => {
         if (protectedPaths.has(path) || !needed.some(target => target === path || inside(target, path)))
             return true;
         console.warn(`Dropped build fence entry covering this checkout's own Git data: ${path}`);

package/dist/backup-ui.d.ts

@@ -4,9 +4,11 @@
  * disclosures.
  */
 import { type Html } from "./html.js";
-import type { BackupRun, BackupSettings } from "./store.js";
+import type { BackupRun, BackupSettings, LiveBackupState } from "./store.js";
 /** The schedules on offer and the most copies kept. Here rather than in backup.ts so the page never loads `node:sqlite`. */
 export declare const BACKUP_EVERY_HOURS: readonly [1, 6, 12, 24];
+/** The continuous copy's choices, in minutes (live-backup.ts LIVE_EVERY_MINUTES; here so the page never loads it). */
+export declare const LIVE_EVERY: readonly [1, 5, 15, 30];
 export declare const MAX_KEEP = 100;
 export declare const BACKUP_CSS: string;
 export type BackupView = {
@@ -16,6 +18,12 @@ export type BackupView = {
     runs: BackupRun[];
     csrf: string;
     now: Date;
+    /** The continuous copy (live-backup.ts): its settings, latest verified copy and folder. */
+    live?: {
+        state: LiveBackupState;
+        folder: string;
+        defaultFolder: string;
+    };
 };
 export declare function backupHtml(view: BackupView, notice: {
     said?: string | null;

package/dist/backup-ui.js

@@ -8,6 +8,8 @@ import { bytesWords } from "./storage.js";
 import { whenUtc } from "./when-html.js";
 /** The schedules on offer and the most copies kept. Here rather than in backup.ts so the page never loads `node:sqlite`. */
 export const BACKUP_EVERY_HOURS = [1, 6, 12, 24];
+/** The continuous copy's choices, in minutes (live-backup.ts LIVE_EVERY_MINUTES; here so the page never loads it). */
+export const LIVE_EVERY = [1, 5, 15, 30];
 export const MAX_KEEP = 100;
 const when = (at) => whenUtc(at);
 const whenWords = (at) => `${at.slice(0, 16).replace("T", " ")} UTC`;
@@ -23,6 +25,19 @@ export const BACKUP_CSS = `.backups{max-width:720px;min-width:0;overflow-wrap:an
     `.backups li{display:flex;gap:8px 12px;flex-wrap:wrap;padding:8px 0;border-top:1px solid var(--line, rgba(0,0,0,.08));font-size:.875rem}` +
     `.backups li .failed{color:var(--danger)}.backups code{white-space:nowrap}.backups pre{overflow-x:auto;margin:8px 0}` +
     `@media (max-width:900px){.backups .backup-state form,.backups .backup-state button{width:100%}.backups .pair{grid-template-columns:1fr}}`;
+/** The continuous copy in one line: its latest verified copy, or why there is none. */
+function liveLine(view) {
+    const live = view.live;
+    if (live === undefined)
+        return "";
+    if (!live.state.settings.enabled)
+        return html `<p class="meta">Continuous copy is off.</p>`;
+    if (live.state.problem !== null)
+        return html `<p class="problem" role="alert">The continuous copy failed: ${live.state.problem}</p>`;
+    if (live.state.latest === null)
+        return html `<p class="meta">The first continuous copy is made within a few minutes of Toolroll starting.</p>`;
+    return html `<p class="meta" data-live-copy>Continuous copy checked ${when(live.state.latest.madeAt)}, every ${live.state.settings.everyMinutes === 1 ? "minute" : `${live.state.settings.everyMinutes} minutes`}.</p>`;
+}
 /** The state in one line, and the next backup (if any), for the top of the page. */
 function state(view) {
     const last = view.runs[0];
@@ -31,13 +46,13 @@ function state(view) {
         : good === undefined ? "The first scheduled backup runs within a minute or two of the console starting."
             : `Next one about ${whenWords(new Date(Date.parse(good.startedAt) + view.settings.everyHours * 3_600_000).toISOString())}.`;
     if (last === undefined)
-        return html `<h2>No backups yet</h2><p class="meta">${next}</p>`;
+        return html `<h2>No backups yet</h2><p class="meta">${next}</p>${liveLine(view)}`;
     if (last.ok === null)
         return html `<h2>Backing up</h2><p class="meta">Started ${when(last.startedAt)}.</p>`;
     if (last.ok === false) {
         return html `<h2 data-backup-state="failed">Last backup failed</h2><p class="problem" role="alert">${when(last.startedAt)}: ${last.error ?? "no reason was recorded"}</p><p class="meta">${good === undefined ? "No backup has succeeded yet." : html `Last good backup ${when(good.startedAt)}.`} ${next}</p>`;
     }
-    return html `<h2 data-backup-state="ok">Backed up ${when(last.startedAt)}</h2><p class="meta">${last.bytes === null ? "" : `${bytesWords(last.bytes)} · `}${next}</p>`;
+    return html `<h2 data-backup-state="ok">Backed up ${when(last.startedAt)}</h2><p class="meta">${last.bytes === null ? "" : `${bytesWords(last.bytes)} · `}${next}</p>${liveLine(view)}`;
 }
 export function backupHtml(view, notice) {
     const s = view.settings;
@@ -46,5 +61,15 @@ export function backupHtml(view, notice) {
         ...BACKUP_EVERY_HOURS.map(hours => html `<option value="${hours}"${s.enabled && s.everyHours === hours ? html ` selected` : ""}>${everyWords(hours)}</option>`)];
     const runs = view.runs.map(run => html `<li data-backup-run="${run.id}"><span>${when(run.startedAt)}</span>${run.ok === true ? html `<span>Succeeded${run.bytes === null ? "" : ` · ${bytesWords(run.bytes)}`}</span>${run.file === null ? "" : html `<span class="mono meta">${fileName(run.file)}</span>`}`
         : run.ok === false ? html `<span class="failed">Failed: ${run.error ?? "no reason recorded"}</span>` : html `<span>Running</span>`}<span class="meta">${run.trigger === "manual" ? "by hand" : "scheduled"}</span></li>`);
-    return html `<section class="backups">${note}<div class="card backup-state"><div>${state(view)}</div>${postForm("/settings/backups/now", html `<button type="submit">Back up now</button>`)}</div><h2>Schedule</h2>${postForm("/settings/backups", html `<fieldset><div class="pair"><label>Back up<select name="every">${choices}</select></label><label>Keep the newest<input type="number" name="keep" min="1" max="${MAX_KEEP}" step="1" value="${s.keep}" required></label></div><label>Folder<input type="text" name="folder" value="${s.folder ?? ""}" placeholder="${view.defaultFolder}" spellcheck="false"></label>${reauth(html `<label>Your Toolroll password<input type="password" name="password" autocomplete="current-password"></label>`)}</fieldset><button type="submit">Save</button>`)}<details><summary>Recent backups${view.runs.length === 0 ? "" : ` (${view.runs.length})`}</summary>${runs.length === 0 ? html `<p class="meta">None yet.</p>` : html `<ol>${runs}</ol>`}<p class="meta">In <span class="mono">${view.folder}</span>. Only the database is copied: provider keys and sign-in files stay out.</p></details><details><summary>Restore a backup</summary><p>Stop Toolroll, then run:</p><pre><code>toolroll restore &lt;file&gt; --dry-run</code></pre><p class="meta">The dry run checks the backup (its version and ledger chain) and changes nothing. Run it again without <code>--dry-run</code> to restore; the current database is kept as a copy first.</p></details></section>`;
+    return html `<section class="backups">${note}<div class="card backup-state"><div>${state(view)}</div>${postForm("/settings/backups/now", html `<button type="submit">Back up now</button>`)}</div><h2>Schedule</h2>${postForm("/settings/backups", html `<fieldset><div class="pair"><label>Back up<select name="every">${choices}</select></label><label>Keep the newest<input type="number" name="keep" min="1" max="${MAX_KEEP}" step="1" value="${s.keep}" required></label></div><label>Folder<input type="text" name="folder" value="${s.folder ?? ""}" placeholder="${view.defaultFolder}" spellcheck="false"></label>${liveFields(view)}${reauth(html `<label>Your Toolroll password<input type="password" name="password" autocomplete="current-password"></label>`)}</fieldset><button type="submit">Save</button>`)}<details><summary>Recent backups${view.runs.length === 0 ? "" : ` (${view.runs.length})`}</summary>${runs.length === 0 ? html `<p class="meta">None yet.</p>` : html `<ol>${runs}</ol>`}<p class="meta">In <span class="mono">${view.folder}</span>. Only the database is copied: provider keys and sign-in files stay out.</p></details><details><summary>Restore a backup</summary><p>Stop Toolroll, then run:</p><pre><code>toolroll restore &lt;file&gt; --dry-run</code></pre><p class="meta">The dry run checks the backup (its version and ledger chain) and changes nothing. Run it again without <code>--dry-run</code> to restore; the current database is kept as a copy first.</p>${view.live === undefined ? "" : html `<p class="meta">Continuous copies are in <span class="mono">${view.live.folder}</span>; restore one the same way.</p>`}</details></section>`;
+}
+/** The continuous copy's two choices, beside the schedule's: how often, and where (blank: beside the database). */
+function liveFields(view) {
+    const live = view.live;
+    if (live === undefined)
+        return "";
+    const on = live.state.settings;
+    const choices = [html `<option value="off"${on.enabled ? "" : html ` selected`}>Off</option>`,
+        ...LIVE_EVERY.map(minutes => html `<option value="${minutes}"${on.enabled && on.everyMinutes === minutes ? html ` selected` : ""}>${minutes === 1 ? "Every minute" : `Every ${minutes} minutes`}</option>`)];
+    return html `<div class="pair"><label>Copy continuously<select name="live_every">${choices}</select></label><label>Copy folder<input type="text" name="live_folder" value="${on.folder ?? ""}" placeholder="${live.defaultFolder}" spellcheck="false"></label></div>`;
 }

package/dist/backup.js

@@ -19,6 +19,7 @@ import { basename, dirname, isAbsolute, join, resolve } from "node:path";
 import { createRequire } from "node:module";
 import { BACKUP_EVERY_HOURS, MAX_KEEP } from "./backup-ui.js";
 import { verifyLedgerChain } from "./ledger-chain.js";
+import { DEFAULT_LIVENESS_MS } from "./runner.js";
 import { openStore, readSchemaVersion, SCHEMA_VERSION } from "./store.js";
 /** A scheduled backup's name: the database it came from (see `databaseTag`), then when it was made. Backups made
  * before the tag was added have none. Retention only ever removes files named like this. */
@@ -28,8 +29,8 @@ export const OLDEST_RESTORABLE = 103;
 export { BACKUP_EVERY_HOURS, MAX_KEEP };
 /** How long the console's lease lasts without a renewal: a console that stopped holds it no longer than this. */
 const LEASE_MS = 3 * 60_000;
-/** A worker that heartbeat this recently is taken to be running. */
-const RUNNER_FRESH_MS = 3 * 60_000;
+/** A worker that heartbeat this recently is taken to be running (runner.ts DEFAULT_LIVENESS_MS). */
+const RUNNER_FRESH_MS = DEFAULT_LIVENESS_MS;
 export const defaultBackupFolder = (databaseFile) => join(dirname(databaseFile), "backups");
 export const backupFolderOf = (settings, databaseFile) => settings.folder ?? defaultBackupFolder(databaseFile);
 /** 2026-09-28T05:34:39.123Z → 2026-09-28-053439 (UTC). */
@@ -287,7 +288,7 @@ export async function restoreDatabase(options) {
         try {
             const running = runningNow(store, now);
             if (running.length > 0)
-                refuse("running", `Toolroll is running (${running.join(" and ")}). Stop it first, then restore. If it has just stopped, wait a few minutes.`);
+                refuse("running", `Toolroll is running (${running.join(" and ")}). Stop it first, then restore. If it has just stopped, wait a minute.`);
             folder = backupFolderOf(store.backupSettings(), databaseFile);
         }
         finally {