alvin-bot 4.8.5 → 4.8.7

package/CHANGELOG.md

@@ -2,6 +2,113 @@
 
 All notable changes to Alvin Bot are documented here.
 
+## [4.8.7] — 2026-04-11
+
+### 🐛 `/update` now detects stale-runtime (rebuild without restart)
+
+Caught immediately after publishing 4.8.6 on the Mac mini: `/update` reported "Already up to date — no new commits" even though the running process was on **v4.8.5** while the disk was already built at **v4.8.6**. The user could see the version mismatch in `/status` (v4.8.5) but `/update` refused to acknowledge it.
+
+**Root cause**: The updater only compared **git commits** (or **npm registry version**) against the local install. It never checked whether the **running process's in-memory version** was older than the **on-disk built version**. This is the dev/CI loop scenario:
+
+1. You edit src/, bump package.json, commit + push
+2. `npm run build` regenerates dist/ at the new version
+3. The running process has the OLD code in memory
+4. You run `/update` in Telegram
+5. git: HEAD == origin/main (just pushed) → 0 commits behind → "up to date"
+6. Process never restarts → keeps running OLD code
+
+**Fix**: New `isRuntimeStale()` check at the very start of `runUpdate()`. Compares `BOT_VERSION` (in-memory at process start) against `package.json.version` from disk via the existing semver compare. If disk is newer, returns success with `requiresRestart=true` immediately — skip the git/npm fetch entirely, just signal a restart so the fresh code takes effect.
+
+After 4.8.7, running `/update` after a manual rebuild will correctly say *"Disk is already built at vX, running vY. Restarting to pick up the new code..."* and trigger the restart.
+
+### ✨ Internal watchdog with crash-loop brake (`src/services/watchdog.ts`)
+
+Ali asked for "derbe persistent" — already 95% there with `KeepAlive: true` from 4.8.6, but the missing piece was a brake to stop the bot from infinite-restart-looping if a deterministic crash happens (corrupt state file, missing dependency, broken upgrade).
+
+**New module**: `src/services/watchdog.ts`. Two responsibilities:
+
+**1. Liveness beacon**. Every 30 s the bot writes `~/.alvin-bot/state/watchdog.json` with `{lastBeat, pid, bootTime, crashCount, crashWindowStart, version}`. Fast disk write, no I/O blocking.
+
+**2. Crash-loop brake**. On every fresh boot, the watchdog reads the previous beacon:
+
+- If the previous beacon is **less than 90 s old** → the previous process exited very recently → that's a crash (or a deliberate restart, treated the same way for the brake's purpose). Increment `crashCount`.
+- If the previous beacon is **older than 90 s** → previous process had clean uptime → reset counter to 0.
+- The crash window is **10 minutes**. Crashes within this window accumulate; older ones don't count.
+- If `crashCount` reaches **10**, the brake engages:
+  - Writes `~/.alvin-bot/state/crash-loop.alert` with the timestamp, version, error log path, and recovery steps
+  - Tries to `launchctl unload -w` its own LaunchAgent so launchd stops retrying (otherwise `KeepAlive: true` would keep burning CPU forever)
+  - Exits with code 3
+
+**3. Recovery**. After **5 minutes of clean uptime**, the watchdog auto-resets the crash counter to 0. So a healthy bot that occasionally has a transient hiccup doesn't slowly accumulate toward the brake over days.
+
+**4. Brake check at startup**. `checkCrashLoopBrake()` runs in `index.ts` **before** any expensive init — if the alert file already exists, the bot exits cleanly with code 3 and tries to unload itself again. This prevents launchd from spinning the bot up just to write the same alert over and over.
+
+**Recovery from a tripped brake**:
+
+```bash
+# 1. Investigate the error log
+cat ~/.alvin-bot/logs/alvin-bot.err.log
+
+# 2. Fix whatever was wrong
+# 3. Remove the alert file
+rm ~/.alvin-bot/state/crash-loop.alert
+
+# 4. Reload the LaunchAgent
+alvin-bot launchd install
+```
+
+**What this catches**:
+
+- Process crashes (segfault, OOM kill) → exit non-zero → brake increments
+- `process.exit()` from unhandled rejection → similar
+- Tight crash loops → brake engages at 10 within 10 min
+- Corrupted state files that crash on read → brake engages eventually
+
+**What this does NOT catch (yet)**:
+
+- Event-loop deadlocks where the process is alive but completely frozen. The watchdog beacon needs the event loop to be alive, so it can't detect freeze. A future release will add an external sister LaunchAgent (`com.alvinbot.watchdog`) that runs every 2 minutes via `StartInterval` and kills the main bot if its beacon file is too stale. Tracked as a follow-up.
+
+**Telemetry surface**: `alvin-bot status` could read the beacon file in a future release to show "crash count: X in last Y minutes" — for now, the alert file is the main user-facing signal.
+
+### 🛡 LaunchAgent: ProcessType + LimitLoadToSessionType
+
+Two small plist hardening tweaks:
+
+- **`ProcessType: Background`** — explicit hint to launchd that this is a long-running background service. macOS treats Background processes with friendlier scheduling and is less likely to kill them under memory pressure (vs `Standard` which is the default for unlabeled jobs).
+- **`LimitLoadToSessionType: Aqua`** — only loads in user GUI sessions. Prevents the LaunchAgent from accidentally loading in non-GUI contexts (e.g. SSH login session) where it would not have Keychain access. Defensive: matches our existing assumption that the bot needs the GUI keychain unlocked for Claude SDK OAuth.
+
+These don't change behaviour for normal use, but they're explicit about our intent. macOS will treat the bot as a proper background service rather than a generic foreground job.
+
+### Tests
+
+87 still passing — no test changes (the stale-runtime check is a fast-path branch that doesn't disturb the existing git/npm logic).
+
+## [4.8.6] — 2026-04-11
+
+### 🐛 LaunchAgent: `/restart` left the bot down forever
+
+Caught on the Mac mini production bot: running `/restart` in Telegram killed the bot cleanly but the process never came back, leaving the bot dead until manual intervention.
+
+**Root cause**: The 4.6.0 LaunchAgent plist template hardcoded a conditional `KeepAlive`:
+
+```xml
+<key>KeepAlive</key>
+<dict>
+    <key>SuccessfulExit</key>
+    <false/>   <!-- don't restart on normal exit -->
+    <key>Crashed</key>
+    <true/>    <!-- only restart on crash -->
+</dict>
+```
+
+That meant launchd would only auto-restart on **crashes**, not on normal exits. But `/restart` (and `/update`) work by calling `process.exit(0)` — a deliberate clean exit — and relying on the process manager to bring the bot back up. With pm2 this always worked because pm2's default is "restart on any exit". With launchd's conditional KeepAlive, `process.exit(0)` was the ONE exit code that guaranteed the bot stayed down.
+
+**Fix**: Plist template now uses `<key>KeepAlive</key><true/>` — unconditional restart on any exit. Matches pm2's default behavior. `ThrottleInterval` dropped from 10s to 5s so recovery is quicker.
+
+**Migration for existing installs**: re-run `alvin-bot launchd install` to get the new plist. The install script unloads the old plist, writes the new one, and reloads it — existing data and running state are preserved.
+
+Also removed the stale "(PM2)" suffix from the `/restart` Telegram command description — it's just "Restart the bot" now, since the command works identically with both pm2 and launchd.
+
 ## [4.8.5] — 2026-04-11
 
 ### 🐛 `/update` now works for npm-global installs

package/bin/cli.js

@@ -1461,15 +1461,16 @@ function renderLaunchdPlist({ label, nodePath, entryPoint, cwd, home, logDir })
     <true/>
 
     <key>KeepAlive</key>
-    <dict>
-        <key>SuccessfulExit</key>
-        <false/>
-        <key>Crashed</key>
-        <true/>
-    </dict>
+    <true/>
 
     <key>ThrottleInterval</key>
-    <integer>10</integer>
+    <integer>5</integer>
+
+    <key>ProcessType</key>
+    <string>Background</string>
+
+    <key>LimitLoadToSessionType</key>
+    <string>Aqua</string>
 
     <key>StandardOutPath</key>
     <string>${logDir}/alvin-bot.out.log</string>

package/dist/handlers/commands.js

@@ -156,7 +156,7 @@ export function registerCommands(bot) {
         { command: "webui", description: "Open Web UI in browser" },
         { command: "setup", description: "Configure API keys & platforms" },
         { command: "cancel", description: "Cancel running request" },
-        { command: "restart", description: "Restart the bot (PM2)" },
+        { command: "restart", description: "Restart the bot" },
         { command: "update", description: "Pull latest, build, restart" },
         { command: "autoupdate", description: "Auto-update on|off|status" },
     ]).catch(err => console.error("Failed to set bot commands:", err));

package/dist/index.js

@@ -14,6 +14,11 @@ if (hasLegacyData()) {
 }
 // 3. Seed defaults for any files that don't exist yet (fresh install)
 seedDefaults();
+// 4. Crash-loop brake check — if we've crashed N times in a short window,
+//    refuse to start, write an alert file, and unload our LaunchAgent so
+//    launchd stops retrying. Runs BEFORE any expensive init so a broken
+//    state file doesn't tank the whole CPU.
+checkCrashLoopBrake();
 // ── Normal imports (safe now — DATA_DIR is ready) ──────────────────
 import { Bot, InlineKeyboard } from "grammy";
 import { config } from "./config.js";
@@ -76,6 +81,7 @@ import { loadSkills } from "./services/skills.js";
 import { loadHooks } from "./services/hooks.js";
 import { registerShutdownHandler } from "./services/restart.js";
 import { cancelAllSubAgents } from "./services/subagents.js";
+import { startWatchdog, stopWatchdog, checkCrashLoopBrake } from "./services/watchdog.js";
 import { getRegistry } from "./engine.js";
 import { scanAssets } from "./services/asset-index.js";
 // Scan asset directory and generate INDEX.json + INDEX.md
@@ -235,6 +241,7 @@ const shutdown = async () => {
     // agents can post a cancellation message to Telegram before the bot
     // stops. Capped at 5s internally so a hang can't block shutdown.
     await cancelAllSubAgents(true);
+    stopWatchdog();
     stopScheduler();
     stopSessionCleanup();
     if (queueInterval)
@@ -472,6 +479,8 @@ if (bot) {
             console.log(`   Users: ${config.allowedUsers.length} authorized`);
             // Start heartbeat monitor
             startHeartbeat();
+            // Start internal watchdog (crash-loop brake + liveness beacon)
+            startWatchdog();
             // Index memory vectors in background (non-blocking)
             initEmbeddings().catch(() => { });
         },
@@ -483,5 +492,6 @@ else {
     console.log(`   WebUI: http://localhost:${process.env.WEB_PORT || 3100}`);
     // Start heartbeat monitor even without Telegram
     startHeartbeat();
+    startWatchdog();
     initEmbeddings().catch(() => { });
 }

package/dist/services/updater.js

@@ -19,6 +19,7 @@ import { resolve, dirname } from "path";
 import { fileURLToPath } from "url";
 import fs from "fs";
 import os from "os";
+import { BOT_VERSION } from "../version.js";
 const execAsync = promisify(exec);
 const PROJECT_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "../..");
 const DATA_DIR = process.env.ALVIN_DATA_DIR || resolve(os.homedir(), ".alvin-bot");
@@ -84,12 +85,40 @@ function compareSemver(a, b) {
     }
     return 0;
 }
+/**
+ * Is the running bot's in-memory version older than what's already built
+ * on disk? This happens when the dev/CI rebuilt the bot mid-session and
+ * the process hasn't restarted yet. A manual /update without a git/npm
+ * fetch should still trigger a restart in this case so the fresh code
+ * takes effect.
+ */
+function isRuntimeStale() {
+    const onDisk = readLocalVersion();
+    if (!onDisk || !BOT_VERSION || BOT_VERSION === "unknown")
+        return false;
+    return compareSemver(BOT_VERSION, onDisk) < 0;
+}
 /** Pull latest changes, install deps, rebuild. Returns a structured result
  *  instead of throwing so the /update command can report cleanly to Telegram.
  *  Dispatches to the git path for source installs and the npm path for
- *  npm-global installs. */
+ *  npm-global installs.
+ *
+ *  Before doing any fetch, checks whether the disk is already newer than
+ *  the running process (i.e. someone rebuilt between the process start
+ *  and this call). If so, returns success with requiresRestart=true so
+ *  the command handler can trigger a graceful restart.
+ */
 export async function runUpdate() {
     try {
+        // Stale-runtime check: disk is already newer than the running code.
+        if (isRuntimeStale()) {
+            const onDisk = readLocalVersion();
+            return {
+                ok: true,
+                message: `Disk is already built at v${onDisk}, running v${BOT_VERSION}. Restarting to pick up the new code...`,
+                requiresRestart: true,
+            };
+        }
         if (isOwnGitRepo()) {
             return await runGitUpdate();
         }

package/dist/services/watchdog.js

@@ -0,0 +1,236 @@
+/**
+ * Internal Watchdog — Self-monitoring for crash-loop detection.
+ *
+ * Writes a liveness beacon file every 30 s with the current pid + boot
+ * time + crash counter. On startup, reads the beacon to detect whether
+ * the previous process exited cleanly or crashed. If too many crashes
+ * happen in a short window, refuses to keep restarting and writes an
+ * alert file so the user can investigate.
+ *
+ * Persistence layers this complements:
+ *   - launchd KeepAlive: true → restarts on any exit (good)
+ *   - ThrottleInterval: 5     → minimum 5 s between restarts (good)
+ *   - This watchdog            → caps the total restart count so we
+ *                                don't burn CPU on a truly broken state
+ *
+ * What this CAN catch:
+ *   - Process crash → exit non-zero → launchd restarts → next boot reads
+ *     beacon, sees a recent exit, increments crash counter
+ *   - Tight crash loop → counter accumulates → hits brake at 10
+ *
+ * What this CANNOT catch (yet):
+ *   - True event-loop deadlocks (process alive but frozen). That requires
+ *     an external watchdog process — tracked as a follow-up.
+ */
+import fs from "fs";
+import { resolve } from "path";
+import os from "os";
+import { execSync } from "child_process";
+import { BOT_VERSION } from "../version.js";
+const DATA_DIR = process.env.ALVIN_DATA_DIR || resolve(os.homedir(), ".alvin-bot");
+const STATE_DIR = resolve(DATA_DIR, "state");
+const BEACON_FILE = resolve(STATE_DIR, "watchdog.json");
+const ALERT_FILE = resolve(STATE_DIR, "crash-loop.alert");
+const BEACON_INTERVAL_MS = 30_000; // write a beacon every 30 s
+const CRASH_WINDOW_MS = 10 * 60 * 1000; // 10 min — crashes within this count toward the brake
+const CRASH_BRAKE_THRESHOLD = 10; // after this many crashes in the window, brake
+const STALE_BEACON_MS = 90_000; // a beacon older than this is considered "old enough that previous process really exited"
+const RECOVERY_UPTIME_MS = 5 * 60 * 1000; // 5 min of clean uptime resets the counter
+let beaconTimer = null;
+let resetTimer = null;
+let bootTime = 0;
+function ensureStateDir() {
+    try {
+        fs.mkdirSync(STATE_DIR, { recursive: true });
+    }
+    catch (err) {
+        console.error("[watchdog] failed to create state dir:", err);
+    }
+}
+function readBeacon() {
+    try {
+        const raw = fs.readFileSync(BEACON_FILE, "utf-8");
+        const parsed = JSON.parse(raw);
+        if (typeof parsed.lastBeat === "number" &&
+            typeof parsed.pid === "number" &&
+            typeof parsed.bootTime === "number" &&
+            typeof parsed.crashCount === "number" &&
+            typeof parsed.crashWindowStart === "number" &&
+            typeof parsed.version === "string") {
+            return parsed;
+        }
+        return null;
+    }
+    catch {
+        return null;
+    }
+}
+function writeBeacon(data) {
+    try {
+        fs.writeFileSync(BEACON_FILE, JSON.stringify(data, null, 0), "utf-8");
+    }
+    catch (err) {
+        console.error("[watchdog] failed to write beacon:", err);
+    }
+}
+function writeAlert(reason, crashCount) {
+    try {
+        const content = [
+            `Alvin Bot crash-loop brake hit at ${new Date().toISOString()}`,
+            `Version: ${BOT_VERSION}`,
+            `Crashes in the last ${CRASH_WINDOW_MS / 60_000} minutes: ${crashCount}`,
+            `Threshold: ${CRASH_BRAKE_THRESHOLD}`,
+            ``,
+            `Reason: ${reason}`,
+            ``,
+            `The bot will refuse to start until this file is removed AND the`,
+            `LaunchAgent is reloaded. Investigate the recent error log:`,
+            `  ${resolve(DATA_DIR, "logs", "alvin-bot.err.log")}`,
+            ``,
+            `Recovery steps once you've fixed the underlying issue:`,
+            `  rm "${ALERT_FILE}"`,
+            `  alvin-bot launchd install   # or just kickstart the service`,
+            ``,
+        ].join("\n");
+        fs.writeFileSync(ALERT_FILE, content, "utf-8");
+    }
+    catch (err) {
+        console.error("[watchdog] failed to write alert:", err);
+    }
+}
+/**
+ * Check whether the watchdog has hit the crash-loop brake. Called once
+ * at startup, BEFORE most of the bot initializes. If the brake is set
+ * (alert file exists), the bot exits cleanly with code 3 — and because
+ * launchd's KeepAlive will keep retrying, we also try to unload our
+ * own LaunchAgent so the retries stop.
+ */
+export function checkCrashLoopBrake() {
+    if (!fs.existsSync(ALERT_FILE))
+        return;
+    console.error("");
+    console.error("==================================================");
+    console.error("⛔ alvin-bot crash-loop brake is engaged");
+    console.error("==================================================");
+    try {
+        const content = fs.readFileSync(ALERT_FILE, "utf-8");
+        console.error(content);
+    }
+    catch { /* ignore */ }
+    // Attempt to unload our own LaunchAgent so launchd stops retrying.
+    // If we don't do this, launchd just KeepAlive's us forever and we
+    // burn CPU writing the same alert.
+    if (process.platform === "darwin") {
+        try {
+            const home = os.homedir();
+            const plistPath = resolve(home, "Library", "LaunchAgents", "com.alvinbot.app.plist");
+            if (fs.existsSync(plistPath)) {
+                execSync(`launchctl unload -w "${plistPath}"`, { stdio: "pipe" });
+                console.error("[watchdog] LaunchAgent unloaded — bot will not auto-restart.");
+            }
+        }
+        catch (err) {
+            console.error("[watchdog] failed to unload LaunchAgent:", err);
+        }
+    }
+    // Exit with a distinct code so logs make the cause obvious
+    process.exit(3);
+}
+/**
+ * Start the watchdog. Called from src/index.ts after all services are
+ * initialized. Reads the previous beacon, increments crash counter if
+ * the previous run exited recently, schedules the periodic beacon
+ * writer, and schedules a recovery-mark reset after RECOVERY_UPTIME_MS
+ * of clean uptime.
+ */
+export function startWatchdog() {
+    ensureStateDir();
+    bootTime = Date.now();
+    const previous = readBeacon();
+    let crashCount = 0;
+    let crashWindowStart = bootTime;
+    if (previous) {
+        const timeSinceLastBeat = bootTime - previous.lastBeat;
+        const inWindow = bootTime - previous.crashWindowStart < CRASH_WINDOW_MS;
+        if (timeSinceLastBeat < STALE_BEACON_MS) {
+            // Previous process exited very recently → that's a crash (or a
+            // graceful exit immediately followed by a restart, which we treat
+            // the same way for the brake — the goal is to detect rapid cycles).
+            if (inWindow) {
+                crashCount = previous.crashCount + 1;
+                crashWindowStart = previous.crashWindowStart;
+            }
+            else {
+                // Previous crash was outside the window → reset counter
+                crashCount = 1;
+            }
+            console.log(`[watchdog] detected restart after ${Math.round(timeSinceLastBeat / 1000)}s — crash ${crashCount}/${CRASH_BRAKE_THRESHOLD} in current ${CRASH_WINDOW_MS / 60_000}min window`);
+            if (crashCount >= CRASH_BRAKE_THRESHOLD) {
+                console.error(`[watchdog] crash-loop brake triggered (${crashCount} crashes in ${CRASH_WINDOW_MS / 60_000}min)`);
+                writeAlert(`Process restarted ${crashCount} times within ${CRASH_WINDOW_MS / 60_000} minutes. Last beacon was ${Math.round(timeSinceLastBeat / 1000)}s ago. Most likely a deterministic crash on startup.`, crashCount);
+                // Re-use the brake check to unload + exit cleanly
+                checkCrashLoopBrake();
+            }
+        }
+        else {
+            // Previous beacon was old → process had clean uptime before exit,
+            // OR system was rebooted between runs. Reset crash count.
+            crashCount = 0;
+            crashWindowStart = bootTime;
+        }
+    }
+    // Write the first beacon immediately so a fresh restart updates the file
+    writeBeacon({
+        lastBeat: bootTime,
+        pid: process.pid,
+        bootTime,
+        crashCount,
+        crashWindowStart,
+        version: BOT_VERSION,
+    });
+    // Periodic beacon writer
+    beaconTimer = setInterval(() => {
+        writeBeacon({
+            lastBeat: Date.now(),
+            pid: process.pid,
+            bootTime,
+            crashCount,
+            crashWindowStart,
+            version: BOT_VERSION,
+        });
+    }, BEACON_INTERVAL_MS);
+    // Schedule a recovery counter reset after RECOVERY_UPTIME_MS of clean
+    // uptime. If we make it that far without dying, the bot is healthy
+    // again and we shouldn't penalize a future single crash.
+    resetTimer = setTimeout(() => {
+        if (crashCount > 0) {
+            console.log(`[watchdog] ${RECOVERY_UPTIME_MS / 60_000}min clean uptime — resetting crash counter from ${crashCount} to 0`);
+            crashCount = 0;
+            crashWindowStart = Date.now();
+            writeBeacon({
+                lastBeat: Date.now(),
+                pid: process.pid,
+                bootTime,
+                crashCount,
+                crashWindowStart,
+                version: BOT_VERSION,
+            });
+        }
+    }, RECOVERY_UPTIME_MS);
+    console.log(`[watchdog] started — beacon every ${BEACON_INTERVAL_MS / 1000}s, brake at ${CRASH_BRAKE_THRESHOLD} crashes per ${CRASH_WINDOW_MS / 60_000}min, recovery after ${RECOVERY_UPTIME_MS / 60_000}min uptime`);
+}
+/**
+ * Stop the watchdog cleanly. Called from the shutdown handler in
+ * index.ts so beacon timers don't keep the process alive after the
+ * grammy bot has stopped.
+ */
+export function stopWatchdog() {
+    if (beaconTimer) {
+        clearInterval(beaconTimer);
+        beaconTimer = null;
+    }
+    if (resetTimer) {
+        clearTimeout(resetTimer);
+        resetTimer = null;
+    }
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "alvin-bot",
-  "version": "4.8.5",
+  "version": "4.8.7",
   "description": "Alvin Bot — Your personal AI agent on Telegram, WhatsApp, Discord, Signal, and Web.",
   "type": "module",
   "main": "dist/index.js",