@inetafrica/open-claudia 3.1.26 → 3.1.27

package/CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## v3.1.27 — The outage pause stops feeding the wedge-exit clock (the fleet-wide Telegram reboot loop)
+
+- **Symptom.** Every Telegram-channel bot was rebooting itself in a loop while no other bot was: rashmi 214 restarts in 7d15h, euvy 210 in 7d17h, ruk-ai 62 in 44h, rahil-coders 11 — 4 of 4 pods running the `telegram` channel. All 12 pods running only `kazee, spaces, web` sat at 0. All 16 share one node and one egress IP, so node, address and NetworkPolicy preset were all controlled for; the *unrestricted* bot (euvy, allow-all egress) restarted most, which retired the egress theory rather than confirming it. The channel was the only discriminator.
+- **Root cause.** `_wedgedSince` is stamped on the first polling hiccup and cleared in exactly one place — `_armHealthyCheck`, on a completed `getUpdates`. `_enterOutagePause()` cleared `_healthyTimer` (the only thing that can reach that reset) and never touched `_wedgedSince`. So every second spent inside the outage pause — a state whose entire documented purpose is *"a reboot can't fix a dead network... never exits"* — still counted toward the 10-minute reboot budget. A link that flapped a few times inside ten minutes rebooted the pod anyway, moments after this very method had twice decided a reboot could not help. Both bots exited almost exactly 10 minutes after their FIRST hiccup regardless of what happened in between: ruk-ai 07:41:02 → 07:51:11 = 10m09s across three outage pauses; euvy 07:33:37 → 07:43:42 = 10m05s.
+- **Fix.** `_enterOutagePause()` now retires the wedge clock (`_wedgedSince = 0`, `_healBackoff = 0`). A failed *fresh* probe is positive evidence that the streak so far was the network, not an in-loop poll wedge — and the exit backstop exists only for the in-loop case. Clearing is safe because `_onPollingHiccup` no-ops while paused: a real in-loop wedge that survives the resume starts an honest new streak.
+- **Proof, not assertion.** New regression test `testOutagePauseDoesNotFeedTheWedgeExitClock` in `test-telegram-poll-recovery.js` replays the production sequence line for line (hiccup → three genuine outage pauses with successful re-probes → ruk-ai's exact 10m09s mark on a reachable link). It **fails against unmodified code** with the reboot firing, and passes with the fix. `testWedgeExitFiresWhenIdle` still passes, so the genuine in-loop backstop is intact, not disabled.
+- **Still open, deliberately not fixed here.** Why the path to `api.telegram.org` flaps at all is unexplained — the probe genuinely fails, three confirmed ~30s outages inside ten minutes on ruk-ai. This change stops the pointless reboots; it does not restore connectivity during a flap. Separately worth asking whether the wedge-exit backstop earns its keep at all: `_heal()` already destroys the agent, nulls `bot._polling` and dials a fresh socket — everything a new process does for the poll path — and 210 restarts never helped.
+
 ## v3.1.26 — Claude Opus 5 in the model picker
 
 - **`claude-opus-5` added to the Claude provider model list** (`CLAUDE_MODELS`), slotted between `claude-fable-5` and `claude-opus-4-8`. Opus 5 launched 2026-07-24 (near-Fable-5 capability at roughly half the price; Anthropic's new default on Max). Verified live before listing: a real CLI round-trip with `--model claude-opus-5` succeeded. Free-text `/model claude-opus-5` already worked (the `/model` handler doesn't validate against the list); this puts it on the picker buttons for `/model` and `/login`. Default model unchanged (`claude-opus-4-8`).

package/channels/telegram/adapter.js

@@ -281,6 +281,17 @@ class TelegramAdapter {
     if (this._pausedForOutage) return;
     this._pausedForOutage = true;
     if (this._healthyTimer) { clearTimeout(this._healthyTimer); this._healthyTimer = null; }
+    // Retire the wedge clock. A failed fresh probe is positive evidence that the
+    // streak so far was the network, not an in-loop poll wedge — and the exit
+    // backstop exists ONLY for the in-loop case. Leaving it stamped counted every
+    // paused second toward the 10-min reboot budget, so a link that flapped a few
+    // times inside ten minutes rebooted the pod anyway, right after this very
+    // method decided a reboot couldn't help (the fleet-wide loop of 2026-08-01:
+    // rashmi 214 restarts, euvy 210, ruk-ai 62 — every Telegram pod, no kazee one).
+    // Clearing here is safe because _onPollingHiccup no-ops while paused: if a real
+    // in-loop wedge survives the resume, the next hiccup starts an honest new streak.
+    this._wedgedSince = 0;
+    this._healBackoff = 0;
     console.error(`${this._stamp()} Telegram unreachable — host-network outage; pausing polls, re-probing every 30s (no reboot, no reconnect hammering).`);
     // Stop the fast retry loop now (clean stop: no zombie loop, socket severed).
     Promise.resolve(this._stopPollingClean("host-network outage pause")).catch(() => {});

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@inetafrica/open-claudia",
-  "version": "3.1.26",
+  "version": "3.1.27",
   "description": "An always-on, provider-agnostic coding-agent harness for Claude Code and OpenAI Codex via chat",
   "main": "bot.js",
   "bin": {

package/test-provider-language.js

@@ -322,7 +322,16 @@ assert.match(pkg.description, /provider-agnostic coding-agent harness/i);
 // 3.1.26 approved by Sumeet 2026-07-25 ("Do it"): add claude-opus-5 (launched
 // 2026-07-24) to the Claude provider model picker; verified live via a CLI
 // round-trip with --model claude-opus-5 before listing.
-assert.strictEqual(pkg.version, "3.1.26", "release version must remain unchanged without explicit approval");
+// 3.1.27 approved by Sumeet 2026-08-02 ("Yes") after he asked for proof rather
+// than assertion ("so are you sure these are the issues and the fix or any
+// tests you can run to confirm?"): the outage pause stops feeding the
+// wedge-exit clock. _enterOutagePause cleared _healthyTimer — the only thing
+// that resets _wedgedSince — but never _wedgedSince itself, so every second
+// spent in a state whose whole purpose is "the network is down, a reboot
+// cannot help" still counted toward the 10-minute reboot budget. Reproduced by
+// a new regression test that replays the prod log line for line and fails
+// against unmodified code.
+assert.strictEqual(pkg.version, "3.1.27", "release version must remain unchanged without explicit approval");
 for (const keyword of ["claude", "codex", "coding-agent", "provider-agnostic"]) {
   assert.ok(pkg.keywords.includes(keyword), `package keyword missing: ${keyword}`);
 }

package/test-telegram-poll-recovery.js

@@ -511,6 +511,118 @@ async function testHealthySignalClearsWedgeOnceAPollCompletes() {
   assert.deepStrictEqual(hiccups, [], "a genuinely recovered restart must NOT re-arm a heal");
 }
 
+// THE FLAPPING-NETWORK REBOOT LOOP (live 2026-08-01: rashmi 214 restarts/7d15h, euvy
+// 210/7d17h, ruk-ai 62/44h — all four Telegram pods, none of the twelve kazee-only ones).
+// _wedgedSince is stamped on the FIRST hiccup and cleared in exactly ONE place:
+// _armHealthyCheck, on a COMPLETED getUpdates. _enterOutagePause clears _healthyTimer —
+// the only thing that can reset it — and never touches _wedgedSince. So every second
+// spent inside the outage pause, a state whose whole purpose is "the network is down, a
+// reboot cannot help, never exit", still counts toward the 10-minute reboot budget. A
+// link that flaps a few times inside ten minutes therefore reboots the pod anyway, even
+// though the code twice decided a reboot was the wrong answer. In prod both bots exited
+// almost exactly 10 min after their FIRST hiccup regardless of what happened in between
+// (ruk-ai 07:41:02→07:51:11 = 10m09s with three outage pauses; euvy 07:33:37→07:43:42 =
+// 10m05s). Fails before the fix: the exit fires on accumulated flap time.
+async function testOutagePauseDoesNotFeedTheWedgeExitClock() {
+  const realNow = Date.now;
+  let now = 1_700_000_000_000;
+  Date.now = () => now;
+
+  let exited = false;
+  let reachable = true;
+  let tickFn = null;
+
+  const adapter = Object.create(TelegramAdapter.prototype);
+  adapter._healTimer = null;
+  adapter._healthyTimer = null;
+  adapter._conflictSince = 0;
+  adapter._conflictTimer = null;
+  adapter._wedgedSince = 0;
+  adapter._healBackoff = 0;
+  adapter._deferredExitLogAt = 0;
+  adapter._lastHiccupLog = 0;
+  adapter._pausedForOutage = false;
+  adapter._outageProbeTimer = null;
+  adapter._pollStartedAt = 0;
+  adapter._anyTurnActive = () => false; // idle throughout
+  adapter._probeTelegramReachable = async () => reachable;
+  adapter._agent = { destroy() {} };
+  adapter.bot = {
+    _polling: { _abort: false, _lastUpdate: 0 },
+    async stopPolling() {},
+    // A loop that never manages to complete a getUpdates while the link is flapping.
+    async startPolling() { if (!this._polling) this._polling = { _abort: false, _lastUpdate: 0 }; },
+  };
+
+  const realExit = process.exit;
+  process.exit = () => { exited = true; throw new Error("__exit__"); };
+  const realSetTimeout = global.setTimeout;
+  const realClearTimeout = global.clearTimeout;
+  global.setTimeout = (fn, ms) => {
+    if (ms === 500 || ms === 5000) fn();            // stop-clean internals fire inline
+    else if (ms === 30000) tickFn = fn;             // capture the outage re-probe tick
+    return { ms };
+  };
+  global.clearTimeout = () => {};
+
+  const heal = async () => {
+    adapter._healTimer = null;
+    try { await adapter._heal(); }
+    catch (e) { if (e.message !== "__exit__") throw e; }
+  };
+
+  try {
+    // T+0: the link starts flapping. First hiccup stamps the wedge clock.
+    adapter._onPollingHiccup("EFATAL: Error: socket hang up");
+    const firstHiccup = now;
+    assert.strictEqual(adapter._wedgedSince, firstHiccup, "the first hiccup stamps the wedge clock");
+
+    // T+3m: a fresh connection fails too ⇒ genuine host outage ⇒ pause, never reboot.
+    now += 3 * 60 * 1000;
+    reachable = false;
+    await heal();
+    assert.strictEqual(adapter._pausedForOutage, true, "an unreachable host must enter the outage pause");
+    assert.strictEqual(exited, false, "the outage pause must not reboot");
+
+    // T+3m30s: link returns, polling resumes — but the loop still can't complete a poll.
+    now += 30 * 1000;
+    reachable = true;
+    assert.strictEqual(typeof tickFn, "function", "the pause must schedule a slow re-probe");
+    await tickFn();
+    assert.strictEqual(adapter._pausedForOutage, false, "a successful re-probe leaves the pause");
+
+    // T+6m / T+9m: it flaps twice more, each correctly handled as an outage.
+    for (const _ of [1, 2]) {
+      now += 2 * 60 * 1000 + 30 * 1000;
+      adapter._onPollingHiccup("EFATAL: Error: socket hang up");
+      reachable = false;
+      tickFn = null;
+      await heal();
+      assert.strictEqual(exited, false, "each outage must be paused through, never rebooted");
+      now += 30 * 1000;
+      reachable = true;
+      if (tickFn) await tickFn();
+    }
+
+    // T+10m+: the link is up right now, but the accumulated flap time has blown the
+    // 10-minute budget. The bot must NOT reboot: nearly all of that window was spent in
+    // a state it had already, correctly, decided a reboot could not fix.
+    now = firstHiccup + 10 * 60 * 1000 + 9 * 1000; // ruk-ai's exact 10m09s
+    reachable = true;
+    adapter._onPollingHiccup("EFATAL: Error: socket hang up");
+    await heal();
+
+    assert.strictEqual(exited, false,
+      "time spent in a known host-network outage must NOT count toward the wedge-exit clock — " +
+      "this is the fleet-wide reboot loop (rashmi 214, euvy 210, ruk-ai 62 restarts)");
+  } finally {
+    Date.now = realNow;
+    process.exit = realExit;
+    global.setTimeout = realSetTimeout;
+    global.clearTimeout = realClearTimeout;
+  }
+}
+
 (async () => {
   await testStartDoesNotWaitForFirstLongPollAndArmsStallWatchdog();
   await testHealStopsPlainlyDestroysAgentAndRestartsIdle();
@@ -523,6 +635,7 @@ async function testHealthySignalClearsWedgeOnceAPollCompletes() {
   await testHealNullsPoisonedPollingSoLibRebuildsFresh();
   await testHealthySignalReArmsWhenNoPollCompleted();
   await testHealthySignalClearsWedgeOnceAPollCompletes();
+  await testOutagePauseDoesNotFeedTheWedgeExitClock();
   console.log("telegram poll recovery OK");
 })().catch((err) => {
   console.error(err.stack || err);