@inetafrica/open-claudia 3.1.26 → 3.1.28

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,22 @@
1
1
  # Changelog
2
2
 
3
+ ## v3.1.28 — The Telegram reboot loop was self-inflicted: the heal kept tearing down its own recovery
4
+
5
+ - **Symptom.** v3.1.27 shipped to all four Telegram pods and the loop continued (euvy: 6 restarts in 3h). Its fix was real but addressed a different failure — the dominant one never enters `_enterOutagePause` at all (zero "Telegram unreachable" lines in the crash logs). The decisive observation was that the bot reproduced it identically on a laptop — 155 self-exits in a local `bot.log` — which retired the cluster-network and conntrack theories outright. The shape is bursts: healthy for 2.7–4.2h, then ~10 minutes of failed heals, then `process.exit(1)`, repeat.
6
+ - **How it was found.** Reasoning from prod logs had already produced one wrong fix, so this time the local install was instrumented directly (unthrottled lifecycle events to a side file) and a real burst was captured. Two facts invisible to the console log settled it: every polling error after the first carried `_polling._abort === true` — meaning *we* aborted it, not Telegram — and `_lastUpdate` read 0 for 200+ seconds straight across 9 successful restarts. It also retired a number the previous investigation had reported as fact: the "~62s failure cadence" was an artifact of the 1/min log throttle. The real rate was one every ~15s.
7
+ - **Root cause — three stacked deadlocks, each independently fatal to recovery.** (1) `_heal` → `_stopPollingClean` → `_agent.destroy()` kills the in-flight `getUpdates`, which surfaces in `_onPollingHiccup` ~1ms later as `EFATAL: socket hang up`. `_healTimer` had just been nulled at the top of `_heal`, so the "a heal is already scheduled" guard didn't hold and the teardown scheduled *another* heal — which tore the socket down again. Self-sustaining. (2) The backoff capped at 15s while `polling.params.timeout` is 30s, so every restarted poll was aborted at the halfway mark: `_lastUpdate`, the only evidence of recovery, could never advance. (3) `_armHealthyCheck` cleared its own pending timer on every re-arm, so a heal every 15s starved a check that needs 35s of quiet — **zero healthy checks fired in the entire captured burst** — and that callback owns the only reset of `_wedgedSince` and `_healBackoff`. With nothing able to clear the wedge clock, it ran monotonically to the 10-minute backstop. One ordinary `ECONNRESET` was enough to cost ten minutes and a reboot.
8
+ - **Fix.** (1) A time-boxed self-abort window opened in `_stopPollingClean` makes `_onPollingHiccup` ignore the error our own teardown raises; it is never explicitly cleared, so a path that forgets to close it cannot permanently deafen the handler — the worst case is a dropped error that `_armHealthyCheck` re-arms 35s later anyway. (2) The heal ladder (now `_scheduleHeal`) escalates to 45s, clearing the 40s request timeout that bounds one `getUpdates`, so the late rungs give a poll genuine room to land; early rungs stay short because they run while polling is down. (3) Re-arming the healthy check no longer pushes its deadline back — the first arm always fires — while the baseline tracks the newest restart, so a poll that completed *before* the latest restart can't be mistaken for proof the latest restart worked. A failed `startPolling` now schedules its own retry rather than waiting for a hiccup that the suppression window means will never come.
9
+ - **Proof, not assertion.** `testOneNetworkBlipDoesNotSpiralIntoTheWedgeExitLoop` models the poller instead of stubbing it: `destroy()` raises the teardown error for real, a healthy poll takes the full 30s hold to complete, the library re-dials ~300ms after a fault (which is why a heal always had something to destroy), and a virtual clock runs a single `ECONNRESET` well past the 10-minute budget. Against unmodified code it reproduces the production failure exactly — a storm of `socket hang up`, then "Polling wedged >10min and Telegram is reachable — exiting for a clean restart" — and against the fix the bot recovers with the wedge cleared and the ladder reset. Three narrower tests pin each deadlock individually. The previous release's test passed against a bug it did not fix because it drove `_enterOutagePause()` directly, a path production never takes; the lesson — model the interaction, don't stub past it — is why this harness raises the error from `destroy()`.
10
+ - **Correction to v3.1.27's closing note.** That entry wondered whether the wedge-exit backstop earned its keep, observing that "210 restarts never helped". They did: a fresh process is the only thing that escapes the loop, which is exactly why each reboot bought 2.7–4.2h of health. The backstop stays until this fix is proven in the fleet.
11
+
12
+ ## v3.1.27 — The outage pause stops feeding the wedge-exit clock (the fleet-wide Telegram reboot loop)
13
+
14
+ - **Symptom.** Every Telegram-channel bot was rebooting itself in a loop while no other bot was: rashmi 214 restarts in 7d15h, euvy 210 in 7d17h, ruk-ai 62 in 44h, rahil-coders 11 — 4 of 4 pods running the `telegram` channel. All 12 pods running only `kazee, spaces, web` sat at 0. All 16 share one node and one egress IP, so node, address and NetworkPolicy preset were all controlled for; the *unrestricted* bot (euvy, allow-all egress) restarted most, which retired the egress theory rather than confirming it. The channel was the only discriminator.
15
+ - **Root cause.** `_wedgedSince` is stamped on the first polling hiccup and cleared in exactly one place — `_armHealthyCheck`, on a completed `getUpdates`. `_enterOutagePause()` cleared `_healthyTimer` (the only thing that can reach that reset) and never touched `_wedgedSince`. So every second spent inside the outage pause — a state whose entire documented purpose is *"a reboot can't fix a dead network... never exits"* — still counted toward the 10-minute reboot budget. A link that flapped a few times inside ten minutes rebooted the pod anyway, moments after this very method had twice decided a reboot could not help. Both bots exited almost exactly 10 minutes after their FIRST hiccup regardless of what happened in between: ruk-ai 07:41:02 → 07:51:11 = 10m09s across three outage pauses; euvy 07:33:37 → 07:43:42 = 10m05s.
16
+ - **Fix.** `_enterOutagePause()` now retires the wedge clock (`_wedgedSince = 0`, `_healBackoff = 0`). A failed *fresh* probe is positive evidence that the streak so far was the network, not an in-loop poll wedge — and the exit backstop exists only for the in-loop case. Clearing is safe because `_onPollingHiccup` no-ops while paused: a real in-loop wedge that survives the resume starts an honest new streak.
17
+ - **Proof, not assertion.** New regression test `testOutagePauseDoesNotFeedTheWedgeExitClock` in `test-telegram-poll-recovery.js` replays the production sequence line for line (hiccup → three genuine outage pauses with successful re-probes → ruk-ai's exact 10m09s mark on a reachable link). It **fails against unmodified code** with the reboot firing, and passes with the fix. `testWedgeExitFiresWhenIdle` still passes, so the genuine in-loop backstop is intact, not disabled.
18
+ - **Still open, deliberately not fixed here.** Why the path to `api.telegram.org` flaps at all is unexplained — the probe genuinely fails, three confirmed ~30s outages inside ten minutes on ruk-ai. This change stops the pointless reboots; it does not restore connectivity during a flap. Separately worth asking whether the wedge-exit backstop earns its keep at all: `_heal()` already destroys the agent, nulls `bot._polling` and dials a fresh socket — everything a new process does for the poll path — and 210 restarts never helped.
19
+
3
20
  ## v3.1.26 — Claude Opus 5 in the model picker
4
21
 
5
22
  - **`claude-opus-5` added to the Claude provider model list** (`CLAUDE_MODELS`), slotted between `claude-fable-5` and `claude-opus-4-8`. Opus 5 launched 2026-07-24 (near-Fable-5 capability at roughly half the price; Anthropic's new default on Max). Verified live before listing: a real CLI round-trip with `--model claude-opus-5` succeeded. Free-text `/model claude-opus-5` already worked (the `/model` handler doesn't validate against the list); this puts it on the picker buttons for `/model` and `/login`. Default model unchanged (`claude-opus-4-8`).
@@ -44,6 +44,9 @@ class TelegramAdapter {
44
44
  this._deferredExitLogAt = 0; // throttle the "deferring restart, turn in progress" log (1/min)
45
45
  this._lastHiccupLog = 0; // throttle the transient-error log (1/min)
46
46
  this._healBackoff = 0; // grows per consecutive heal, capped
47
+ this._selfAbortUntil = 0; // teardown grace: errors WE caused aren't faults
48
+ this._healthyBaseline = 0; // newest restart the pending healthy-check judges
49
+ this._healthyReArmMsg = ""; // reason text if that check finds no completed poll
47
50
  this._pausedForOutage = false; // host-network outage: poll loop stopped, slow re-probe owns recovery
48
51
  this._outageProbeTimer = null; // the ~30s re-probe loop that runs while paused
49
52
  this._pollWatchdogTimer = null; // detects a silent first/active getUpdates stall
@@ -112,6 +115,15 @@ class TelegramAdapter {
112
115
  // old exit-for-relaunch *was* the restart the bot kept doing on every idle
113
116
  // timeout (a sleep/wake burst could exhaust all 6 attempts in seconds).
114
117
  _onPollingHiccup(msg) {
118
+ // OUR OWN teardown, not a fault. _stopPollingClean destroys the agent, which
119
+ // kills the in-flight getUpdates and surfaces here as "socket hang up" 1-5ms
120
+ // later. Counting it re-armed the heal that had just torn the socket down —
121
+ // a self-sustaining loop that ran until the 10-min backstop exited the
122
+ // process (root-caused 2026-08-03 from a live capture: after the ONE genuine
123
+ // ECONNRESET, every subsequent error carried _polling._abort === true, i.e.
124
+ // we aborted it). The heal re-arms its own follow-up via _armHealthyCheck,
125
+ // so dropping these loses no recovery.
126
+ if (Date.now() < (this._selfAbortUntil || 0)) return;
115
127
  if (this._pausedForOutage) return; // paused for a host outage; the slow re-probe owns recovery
116
128
  // During a 409 streak the conflict handler owns the poll lifecycle; a heal
117
129
  // here would startPolling straight back into the conflict and its errors
@@ -125,8 +137,21 @@ class TelegramAdapter {
125
137
  console.log(`${this._stamp()} Polling hiccup (${msg}) — retrying on a fresh connection.`);
126
138
  }
127
139
  if (this._healTimer) return; // a heal is already scheduled; let it run
128
- this._healBackoff = Math.min(this._healBackoff + 1, 5);
129
- const delay = Math.min(15000, 500 * 2 ** this._healBackoff); // 1s … 15s
140
+ this._scheduleHeal();
141
+ }
142
+
143
+ // The heal ladder MUST out-wait one getUpdates or it can never observe
144
+ // success: a heal tears the socket down, so an interval shorter than the poll
145
+ // hold aborts every attempt before it can complete and _lastUpdate (the only
146
+ // evidence of recovery) stays pinned at 0 forever. The old 15s cap sat at HALF
147
+ // the 30s long-poll hold, which made recovery structurally impossible — a
148
+ // burst could only ever end at the 10-min wedge-exit. The cap now clears the
149
+ // 40s request timeout, which bounds even a hung poll, so the last rungs give a
150
+ // poll genuine room to land. Early rungs stay short: they run while polling is
151
+ // still down, where retrying fast is free and often enough.
152
+ _scheduleHeal() {
153
+ this._healBackoff = Math.min(this._healBackoff + 1, 7);
154
+ const delay = Math.min(45000, 500 * 2 ** this._healBackoff); // 1s, 2s, 4s, 8s, 16s, 32s, 45s
130
155
  this._healTimer = setTimeout(() => this._heal(), delay);
131
156
  }
132
157
 
@@ -141,6 +166,13 @@ class TelegramAdapter {
141
166
  // destroying the agent forces a hung half-open getUpdates to settle NOW
142
167
  // instead of never. The 5s race caps the wait if teardown gets swallowed.
143
168
  async _stopPollingClean(reason) {
169
+ // Open the self-abort window BEFORE destroying the agent: the "socket hang
170
+ // up" it raises lands in _onPollingHiccup within milliseconds. Time-boxed
171
+ // (never an explicit clear) so a path that forgets to close it can't deafen
172
+ // the hiccup handler permanently — the worst case is that a genuine error
173
+ // inside the window is dropped, and _armHealthyCheck still re-arms recovery
174
+ // 35s later. Covers the 5s stop race + 500ms settle + startPolling.
175
+ this._selfAbortUntil = Date.now() + 10000;
144
176
  const stopped = Promise.resolve(this.bot.stopPolling({ reason })).catch(() => {});
145
177
  try { this._agent.destroy(); } catch (e) {}
146
178
  await Promise.race([stopped, new Promise((r) => setTimeout(r, 5000))]);
@@ -244,7 +276,12 @@ class TelegramAdapter {
244
276
  this._pollStartedAt = Date.now();
245
277
  await this.bot.startPolling({ restart: false });
246
278
  } catch (e) {
247
- return; // restart didn't take — the next hiccup reschedules a heal
279
+ // Restart didn't take. This used to fall through to "the next hiccup
280
+ // reschedules a heal", but the self-abort window means the teardown we
281
+ // just did raises no hiccup — so nothing would be scheduled and the
282
+ // adapter would sit dead until the wedge-exit. Schedule it ourselves.
283
+ if (!this._healTimer) this._scheduleHeal();
284
+ return;
248
285
  }
249
286
  this._armHealthyCheck(this._pollStartedAt, "restart completed but no getUpdates within 35s");
250
287
  }
@@ -254,18 +291,29 @@ class TelegramAdapter {
254
291
  // nothing, so a blind timer would declare it healthy and reset _wedgedSince,
255
292
  // hiding the wedge until the 10-min backstop rebooted. Gate the reset on a
256
293
  // completed poll; otherwise re-arm a heal.
294
+ //
295
+ // Re-arming used to clearTimeout the pending check, which starved it: heals
296
+ // ran every 15s, the check needed 35s of quiet, so it NEVER fired — and it
297
+ // owns the only reset of _wedgedSince/_healBackoff, so the wedge clock ran
298
+ // unbroken to the 10-min exit (zero healthcheck events in a 200s+ live
299
+ // capture, 2026-08-03). Now the DEADLINE is never pushed back — the first arm
300
+ // wins and always fires — while the baseline tracks the newest restart, so a
301
+ // poll that completed before the latest restart can't be mistaken for proof
302
+ // that the latest restart worked.
257
303
  _armHealthyCheck(restartedAt, reArmMsg) {
258
- if (this._healthyTimer) clearTimeout(this._healthyTimer);
304
+ this._healthyBaseline = restartedAt;
305
+ this._healthyReArmMsg = reArmMsg;
306
+ if (this._healthyTimer) return;
259
307
  this._healthyTimer = setTimeout(() => {
260
308
  this._healthyTimer = null;
261
309
  const lastCompletedPoll = this.bot?._polling?._lastUpdate || 0;
262
- if (lastCompletedPoll >= restartedAt) {
310
+ if (lastCompletedPoll >= this._healthyBaseline) {
263
311
  if (this._wedgedSince) console.log(`${this._stamp()} Polling healthy again.`);
264
312
  this._wedgedSince = 0;
265
313
  this._healBackoff = 0;
266
314
  this._deferredExitLogAt = 0;
267
315
  } else {
268
- this._onPollingHiccup(reArmMsg);
316
+ this._onPollingHiccup(this._healthyReArmMsg);
269
317
  }
270
318
  }, 35000);
271
319
  }
@@ -281,6 +329,17 @@ class TelegramAdapter {
281
329
  if (this._pausedForOutage) return;
282
330
  this._pausedForOutage = true;
283
331
  if (this._healthyTimer) { clearTimeout(this._healthyTimer); this._healthyTimer = null; }
332
+ // Retire the wedge clock. A failed fresh probe is positive evidence that the
333
+ // streak so far was the network, not an in-loop poll wedge — and the exit
334
+ // backstop exists ONLY for the in-loop case. Leaving it stamped counted every
335
+ // paused second toward the 10-min reboot budget, so a link that flapped a few
336
+ // times inside ten minutes rebooted the pod anyway, right after this very
337
+ // method decided a reboot couldn't help (the fleet-wide loop of 2026-08-01:
338
+ // rashmi 214 restarts, euvy 210, ruk-ai 62 — every Telegram pod, no kazee one).
339
+ // Clearing here is safe because _onPollingHiccup no-ops while paused: if a real
340
+ // in-loop wedge survives the resume, the next hiccup starts an honest new streak.
341
+ this._wedgedSince = 0;
342
+ this._healBackoff = 0;
284
343
  console.error(`${this._stamp()} Telegram unreachable — host-network outage; pausing polls, re-probing every 30s (no reboot, no reconnect hammering).`);
285
344
  // Stop the fast retry loop now (clean stop: no zombie loop, socket severed).
286
345
  Promise.resolve(this._stopPollingClean("host-network outage pause")).catch(() => {});
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@inetafrica/open-claudia",
3
- "version": "3.1.26",
3
+ "version": "3.1.28",
4
4
  "description": "An always-on, provider-agnostic coding-agent harness for Claude Code and OpenAI Codex via chat",
5
5
  "main": "bot.js",
6
6
  "bin": {
@@ -322,7 +322,26 @@ assert.match(pkg.description, /provider-agnostic coding-agent harness/i);
322
322
  // 3.1.26 approved by Sumeet 2026-07-25 ("Do it"): add claude-opus-5 (launched
323
323
  // 2026-07-24) to the Claude provider model picker; verified live via a CLI
324
324
  // round-trip with --model claude-opus-5 before listing.
325
- assert.strictEqual(pkg.version, "3.1.26", "release version must remain unchanged without explicit approval");
325
+ // 3.1.27 approved by Sumeet 2026-08-02 ("Yes") after he asked for proof rather
326
+ // than assertion ("so are you sure these are the issues and the fix or any
327
+ // tests you can run to confirm?"): the outage pause stops feeding the
328
+ // wedge-exit clock. _enterOutagePause cleared _healthyTimer — the only thing
329
+ // that resets _wedgedSince — but never _wedgedSince itself, so every second
330
+ // spent in a state whose whole purpose is "the network is down, a reboot
331
+ // cannot help" still counted toward the 10-minute reboot budget. Reproduced by
332
+ // a new regression test that replays the prod log line for line and fails
333
+ // against unmodified code.
334
+ // 3.1.28 approved by Sumeet 2026-08-03 ("Ok fix it and push our a new version
335
+ // of open Claudia then set up a watcher to wait for the docker to complete
336
+ // building the update across all pods starting with one then if it connects
337
+ // fine do the rest") after v3.1.27 shipped and did NOT stop the loop. Live
338
+ // instrumentation of his own Mac install captured a full burst and root-caused
339
+ // it: the heal's own _agent.destroy() teardown re-entered _onPollingHiccup and
340
+ // scheduled the next heal, the 15s backoff cap sat at HALF the 30s long-poll
341
+ // hold so no getUpdates could ever complete, and _armHealthyCheck cleared its
342
+ // own pending timer on every re-arm so the only code that resets _wedgedSince
343
+ // never ran. One ordinary ECONNRESET cost 10 minutes and a process.exit.
344
+ assert.strictEqual(pkg.version, "3.1.28", "release version must remain unchanged without explicit approval");
326
345
  for (const keyword of ["claude", "codex", "coding-agent", "provider-agnostic"]) {
327
346
  assert.ok(pkg.keywords.includes(keyword), `package keyword missing: ${keyword}`);
328
347
  }
@@ -511,6 +511,382 @@ async function testHealthySignalClearsWedgeOnceAPollCompletes() {
511
511
  assert.deepStrictEqual(hiccups, [], "a genuinely recovered restart must NOT re-arm a heal");
512
512
  }
513
513
 
514
+ // THE FLAPPING-NETWORK REBOOT LOOP (live 2026-08-01: rashmi 214 restarts/7d15h, euvy
515
+ // 210/7d17h, ruk-ai 62/44h — all four Telegram pods, none of the twelve kazee-only ones).
516
+ // _wedgedSince is stamped on the FIRST hiccup and cleared in exactly ONE place:
517
+ // _armHealthyCheck, on a COMPLETED getUpdates. _enterOutagePause clears _healthyTimer —
518
+ // the only thing that can reset it — and never touches _wedgedSince. So every second
519
+ // spent inside the outage pause, a state whose whole purpose is "the network is down, a
520
+ // reboot cannot help, never exit", still counts toward the 10-minute reboot budget. A
521
+ // link that flaps a few times inside ten minutes therefore reboots the pod anyway, even
522
+ // though the code twice decided a reboot was the wrong answer. In prod both bots exited
523
+ // almost exactly 10 min after their FIRST hiccup regardless of what happened in between
524
+ // (ruk-ai 07:41:02→07:51:11 = 10m09s with three outage pauses; euvy 07:33:37→07:43:42 =
525
+ // 10m05s). Fails before the fix: the exit fires on accumulated flap time.
526
+ async function testOutagePauseDoesNotFeedTheWedgeExitClock() {
527
+ const realNow = Date.now;
528
+ let now = 1_700_000_000_000;
529
+ Date.now = () => now;
530
+
531
+ let exited = false;
532
+ let reachable = true;
533
+ let tickFn = null;
534
+
535
+ const adapter = Object.create(TelegramAdapter.prototype);
536
+ adapter._healTimer = null;
537
+ adapter._healthyTimer = null;
538
+ adapter._conflictSince = 0;
539
+ adapter._conflictTimer = null;
540
+ adapter._wedgedSince = 0;
541
+ adapter._healBackoff = 0;
542
+ adapter._deferredExitLogAt = 0;
543
+ adapter._lastHiccupLog = 0;
544
+ adapter._pausedForOutage = false;
545
+ adapter._outageProbeTimer = null;
546
+ adapter._pollStartedAt = 0;
547
+ adapter._anyTurnActive = () => false; // idle throughout
548
+ adapter._probeTelegramReachable = async () => reachable;
549
+ adapter._agent = { destroy() {} };
550
+ adapter.bot = {
551
+ _polling: { _abort: false, _lastUpdate: 0 },
552
+ async stopPolling() {},
553
+ // A loop that never manages to complete a getUpdates while the link is flapping.
554
+ async startPolling() { if (!this._polling) this._polling = { _abort: false, _lastUpdate: 0 }; },
555
+ };
556
+
557
+ const realExit = process.exit;
558
+ process.exit = () => { exited = true; throw new Error("__exit__"); };
559
+ const realSetTimeout = global.setTimeout;
560
+ const realClearTimeout = global.clearTimeout;
561
+ global.setTimeout = (fn, ms) => {
562
+ if (ms === 500 || ms === 5000) fn(); // stop-clean internals fire inline
563
+ else if (ms === 30000) tickFn = fn; // capture the outage re-probe tick
564
+ return { ms };
565
+ };
566
+ global.clearTimeout = () => {};
567
+
568
+ const heal = async () => {
569
+ adapter._healTimer = null;
570
+ try { await adapter._heal(); }
571
+ catch (e) { if (e.message !== "__exit__") throw e; }
572
+ };
573
+
574
+ try {
575
+ // T+0: the link starts flapping. First hiccup stamps the wedge clock.
576
+ adapter._onPollingHiccup("EFATAL: Error: socket hang up");
577
+ const firstHiccup = now;
578
+ assert.strictEqual(adapter._wedgedSince, firstHiccup, "the first hiccup stamps the wedge clock");
579
+
580
+ // T+3m: a fresh connection fails too ⇒ genuine host outage ⇒ pause, never reboot.
581
+ now += 3 * 60 * 1000;
582
+ reachable = false;
583
+ await heal();
584
+ assert.strictEqual(adapter._pausedForOutage, true, "an unreachable host must enter the outage pause");
585
+ assert.strictEqual(exited, false, "the outage pause must not reboot");
586
+
587
+ // T+3m30s: link returns, polling resumes — but the loop still can't complete a poll.
588
+ now += 30 * 1000;
589
+ reachable = true;
590
+ assert.strictEqual(typeof tickFn, "function", "the pause must schedule a slow re-probe");
591
+ await tickFn();
592
+ assert.strictEqual(adapter._pausedForOutage, false, "a successful re-probe leaves the pause");
593
+
594
+ // T+6m / T+9m: it flaps twice more, each correctly handled as an outage.
595
+ for (const _ of [1, 2]) {
596
+ now += 2 * 60 * 1000 + 30 * 1000;
597
+ adapter._onPollingHiccup("EFATAL: Error: socket hang up");
598
+ reachable = false;
599
+ tickFn = null;
600
+ await heal();
601
+ assert.strictEqual(exited, false, "each outage must be paused through, never rebooted");
602
+ now += 30 * 1000;
603
+ reachable = true;
604
+ if (tickFn) await tickFn();
605
+ }
606
+
607
+ // T+10m+: the link is up right now, but the accumulated flap time has blown the
608
+ // 10-minute budget. The bot must NOT reboot: nearly all of that window was spent in
609
+ // a state it had already, correctly, decided a reboot could not fix.
610
+ now = firstHiccup + 10 * 60 * 1000 + 9 * 1000; // ruk-ai's exact 10m09s
611
+ reachable = true;
612
+ adapter._onPollingHiccup("EFATAL: Error: socket hang up");
613
+ await heal();
614
+
615
+ assert.strictEqual(exited, false,
616
+ "time spent in a known host-network outage must NOT count toward the wedge-exit clock — " +
617
+ "this is the fleet-wide reboot loop (rashmi 214, euvy 210, ruk-ai 62 restarts)");
618
+ } finally {
619
+ Date.now = realNow;
620
+ process.exit = realExit;
621
+ global.setTimeout = realSetTimeout;
622
+ global.clearTimeout = realClearTimeout;
623
+ }
624
+ }
625
+
626
+ // The fleet reboot loop, reproduced. Everything above stubs the poller; this
627
+ // harness MODELS it, because the bug lived in the interaction the stubs elided:
628
+ // _stopPollingClean destroys the agent, and in production that kills the
629
+ // in-flight getUpdates and re-enters _onPollingHiccup ~1ms later. A stub whose
630
+ // destroy() is a no-op can never show that, which is exactly why v3.1.27's unit
631
+ // test passed against a bug it did not fix (it drove _enterOutagePause, a path
632
+ // the dominant failure never takes). Here destroy() raises the error for real,
633
+ // a poll takes the full 30s hold to complete, and a virtual clock runs the
634
+ // burst past the 10-minute wedge-exit budget.
635
+ function makeVirtualPoller(clock) {
636
+ const state = { polling: null, pendingPoll: null, onSelfAbort: null, startCount: 0 };
637
+ const bot = {
638
+ _polling: null,
639
+ async stopPolling() {
640
+ // A plain stop only sets the abort flag — the HTTP request is still open
641
+ // on the wire. It is _agent.destroy() below that actually kills it, which
642
+ // is precisely why the resulting error looks like a network fault.
643
+ if (state.polling) state.polling._abort = true;
644
+ },
645
+ async startPolling() {
646
+ state.startCount += 1;
647
+ state.polling = { _abort: false, _lastUpdate: 0 }; // fresh object ⇒ _lastUpdate resets to 0
648
+ bot._polling = state.polling;
649
+ const mine = state.polling;
650
+ // A healthy getUpdates completes only after the full 30s long-poll hold.
651
+ state.pendingPoll = clock.at(clock.now() + 30000, () => {
652
+ state.pendingPoll = null;
653
+ if (!mine._abort) mine._lastUpdate = clock.now();
654
+ });
655
+ },
656
+ };
657
+ const agent = {
658
+ destroy() {
659
+ // The real behaviour: tearing the socket down surfaces as a polling error.
660
+ // No re-dial here — a plain stop set _abort, so the lib's loop does not
661
+ // reschedule; the heal's own startPolling is what brings polling back.
662
+ if (state.pendingPoll) {
663
+ clock.cancel(state.pendingPoll);
664
+ state.pendingPoll = null;
665
+ if (state.onSelfAbort) state.onSelfAbort("EFATAL: Error: socket hang up");
666
+ }
667
+ },
668
+ };
669
+ // A poll that dies on its own (a genuine network fault, _abort still false).
670
+ // node-telegram-bot-api re-dials within ~300ms, which is why a request is
671
+ // ALWAYS back in flight by the time a heal runs — and therefore why the heal
672
+ // always had something to destroy. The live capture shows exactly this:
673
+ // reqPending was true at every heal:start.
674
+ state.crash = (msg) => {
675
+ if (state.pendingPoll) { clock.cancel(state.pendingPoll); state.pendingPoll = null; }
676
+ if (state.onSelfAbort) state.onSelfAbort(msg);
677
+ const mine = state.polling;
678
+ state.pendingPoll = clock.at(clock.now() + 300, () => {
679
+ state.pendingPoll = clock.at(clock.now() + 30000, () => {
680
+ state.pendingPoll = null;
681
+ if (mine && !mine._abort) mine._lastUpdate = clock.now();
682
+ });
683
+ });
684
+ };
685
+ return { bot, agent, state };
686
+ }
687
+
688
+ function makeClock(start) {
689
+ let now = start;
690
+ let seq = 0;
691
+ const timers = new Map();
692
+ return {
693
+ now: () => now,
694
+ at(when, fn) { const id = ++seq; timers.set(id, { when, fn }); return id; },
695
+ cancel(id) { timers.delete(id); },
696
+ pending: () => timers.size,
697
+ async advanceTo(target) {
698
+ for (;;) {
699
+ let pick = null;
700
+ for (const [id, t] of timers) {
701
+ if (t.when > target) continue;
702
+ if (!pick || t.when < pick.t.when || (t.when === pick.t.when && id < pick.id)) pick = { id, t };
703
+ }
704
+ if (!pick) break;
705
+ timers.delete(pick.id);
706
+ now = Math.max(now, pick.t.when);
707
+ await pick.t.fn();
708
+ }
709
+ now = target;
710
+ },
711
+ };
712
+ }
713
+
714
+ async function testOneNetworkBlipDoesNotSpiralIntoTheWedgeExitLoop() {
715
+ const clock = makeClock(1_700_000_000_000);
716
+ const { bot, agent, state } = makeVirtualPoller(clock);
717
+
718
+ const adapter = Object.create(TelegramAdapter.prototype);
719
+ adapter._healTimer = null;
720
+ adapter._healthyTimer = null;
721
+ adapter._healthyBaseline = 0;
722
+ adapter._healthyReArmMsg = "";
723
+ adapter._selfAbortUntil = 0;
724
+ adapter._wedgedSince = 0;
725
+ adapter._healBackoff = 0;
726
+ adapter._deferredExitLogAt = 0;
727
+ adapter._lastHiccupLog = 0;
728
+ adapter._pausedForOutage = false;
729
+ adapter._outageProbeTimer = null;
730
+ adapter._conflictSince = 0;
731
+ adapter._conflictTimer = null;
732
+ adapter._pollStartedAt = 0;
733
+ adapter._anyTurnActive = () => false;
734
+ adapter._probeTelegramReachable = async () => true; // Telegram was reachable throughout the real burst
735
+ adapter._agent = agent;
736
+ adapter.bot = bot;
737
+ state.onSelfAbort = (msg) => adapter._onPollingHiccup(msg);
738
+
739
+ let exited = false;
740
+ const realExit = process.exit;
741
+ const realNow = Date.now;
742
+ const realSetTimeout = global.setTimeout;
743
+ const realClearTimeout = global.clearTimeout;
744
+ Date.now = () => clock.now();
745
+ process.exit = () => { exited = true; throw new Error("__exit__"); };
746
+ global.setTimeout = (fn, ms) => {
747
+ if (ms === 500 || ms === 5000) { fn(); return { inline: true }; } // stop-clean internals
748
+ return clock.at(clock.now() + ms, fn);
749
+ };
750
+ global.clearTimeout = (id) => { if (id && !id.inline) clock.cancel(id); };
751
+
752
+ try {
753
+ await bot.startPolling(); // healthy, mid long-poll hold
754
+ await clock.advanceTo(clock.now() + 1000);
755
+ // ONE genuine socket reset — the only external event in the captured burst.
756
+ state.crash("EFATAL: Error: read ECONNRESET");
757
+ assert.ok(adapter._wedgedSince > 0, "a real error must open the wedge window");
758
+
759
+ // Run well past the 10-minute wedge-exit budget.
760
+ for (let i = 0; i < 40 && !exited; i += 1) {
761
+ try { await clock.advanceTo(clock.now() + 30000); }
762
+ catch (e) { if (e.message !== "__exit__") throw e; }
763
+ }
764
+
765
+ assert.strictEqual(exited, false,
766
+ "a single ECONNRESET must not end in process.exit — the heal's own teardown was " +
767
+ "re-triggering the heal, so the wedge clock never cleared and every burst ran to " +
768
+ "the 10-min backstop (155 self-exits observed locally, 210+ per pod in the fleet)");
769
+ assert.strictEqual(adapter._wedgedSince, 0,
770
+ "once a getUpdates completes again the wedge must clear");
771
+ assert.strictEqual(adapter._healBackoff, 0, "recovery must reset the heal ladder");
772
+ assert.ok(bot._polling._lastUpdate > 0,
773
+ "the poll loop must actually complete a getUpdates — pre-fix _lastUpdate stayed 0 forever " +
774
+ "because each heal aborted the 30s long-poll after 15s");
775
+ assert.ok(state.startCount <= 4,
776
+ `recovery must take a couple of restarts, not a storm (took ${state.startCount})`);
777
+ } finally {
778
+ Date.now = realNow;
779
+ process.exit = realExit;
780
+ global.setTimeout = realSetTimeout;
781
+ global.clearTimeout = realClearTimeout;
782
+ }
783
+ }
784
+
785
+ // Fix 1 in isolation: the "socket hang up" our own _agent.destroy() raises must
786
+ // not schedule a heal. Pre-fix this was the engine of the loop.
787
+ async function testOurOwnTeardownIsNotCountedAsAFault() {
788
+ const clock = makeClock(1_700_000_000_000);
789
+ const { bot, agent, state } = makeVirtualPoller(clock);
790
+
791
+ const adapter = Object.create(TelegramAdapter.prototype);
792
+ Object.assign(adapter, {
793
+ _healTimer: null, _healthyTimer: null, _healthyBaseline: 0, _healthyReArmMsg: "",
794
+ _selfAbortUntil: 0, _wedgedSince: 0, _healBackoff: 0, _deferredExitLogAt: 0,
795
+ _lastHiccupLog: 0, _pausedForOutage: false, _outageProbeTimer: null,
796
+ _conflictSince: 0, _conflictTimer: null, _pollStartedAt: 0,
797
+ _anyTurnActive: () => false, _probeTelegramReachable: async () => true,
798
+ _agent: agent, bot,
799
+ });
800
+
801
+ const scheduled = [];
802
+ const realNow = Date.now;
803
+ const realSetTimeout = global.setTimeout;
804
+ const realClearTimeout = global.clearTimeout;
805
+ Date.now = () => clock.now();
806
+ global.setTimeout = (fn, ms) => {
807
+ if (ms === 500 || ms === 5000) { fn(); return { inline: true }; }
808
+ scheduled.push(ms);
809
+ return clock.at(clock.now() + ms, fn);
810
+ };
811
+ global.clearTimeout = (id) => { if (id && !id.inline) clock.cancel(id); };
812
+
813
+ let selfAbortSeen = 0;
814
+ state.onSelfAbort = (msg) => { selfAbortSeen += 1; adapter._onPollingHiccup(msg); };
815
+
816
+ try {
817
+ await bot.startPolling();
818
+ await clock.advanceTo(clock.now() + 1000);
819
+ adapter._wedgedSince = clock.now(); // already in a wedge, as when a heal runs
820
+ scheduled.length = 0;
821
+ await adapter._heal(); // tears the socket down → raises our own error
822
+
823
+ assert.strictEqual(selfAbortSeen, 1, "the harness must actually raise the teardown error");
824
+ assert.deepStrictEqual(scheduled, [35000],
825
+ "the only timer a heal may arm is its 35s healthy check — pre-fix the teardown error " +
826
+ "also queued another heal, which tore down the socket again, forever");
827
+ assert.ok(clock.now() < adapter._selfAbortUntil,
828
+ "the self-abort window must be open across the restart");
829
+ } finally {
830
+ Date.now = realNow;
831
+ global.setTimeout = realSetTimeout;
832
+ global.clearTimeout = realClearTimeout;
833
+ }
834
+ }
835
+
836
+ // Fix 2: the heal ladder must out-wait one getUpdates. A cap below the poll
837
+ // hold makes recovery unobservable — _lastUpdate can never advance.
838
+ function testHealLadderOutWaitsTheLongPoll() {
839
+ const adapter = Object.create(TelegramAdapter.prototype);
840
+ adapter._healBackoff = 0;
841
+ const delays = [];
842
+ const realSetTimeout = global.setTimeout;
843
+ global.setTimeout = (fn, ms) => { delays.push(ms); return { ms }; };
844
+ try {
845
+ for (let i = 0; i < 10; i += 1) { adapter._healTimer = null; adapter._scheduleHeal(); }
846
+ } finally {
847
+ global.setTimeout = realSetTimeout;
848
+ }
849
+ const settled = delays[delays.length - 1];
850
+ assert.ok(settled > 40000,
851
+ `the settled heal interval (${settled}ms) must exceed the 40s request timeout that bounds ` +
852
+ "one getUpdates; the old 15s cap was HALF the 30s long-poll hold, so every attempt was " +
853
+ "aborted before it could complete and recovery was structurally impossible");
854
+ assert.ok(delays[0] <= 1000, "early rungs must still retry fast while polling is down");
855
+ }
856
+
857
+ // Fix 3: re-arming must not push the deadline back, or a heal cadence shorter
858
+ // than the check window starves the only code that clears the wedge.
859
+ function testHealthyCheckCannotBeStarvedByRepeatedReArms() {
860
+ const adapter = Object.create(TelegramAdapter.prototype);
861
+ adapter._healthyTimer = null;
862
+ adapter._healthyBaseline = 0;
863
+ adapter._healthyReArmMsg = "";
864
+
865
+ let armCount = 0;
866
+ let cleared = 0;
867
+ const realSetTimeout = global.setTimeout;
868
+ const realClearTimeout = global.clearTimeout;
869
+ global.setTimeout = (fn, ms) => { armCount += 1; return { ms, fn }; };
870
+ global.clearTimeout = () => { cleared += 1; };
871
+ try {
872
+ adapter._armHealthyCheck(1000, "first");
873
+ adapter._armHealthyCheck(2000, "second");
874
+ adapter._armHealthyCheck(3000, "third");
875
+ } finally {
876
+ global.setTimeout = realSetTimeout;
877
+ global.clearTimeout = realClearTimeout;
878
+ }
879
+
880
+ assert.strictEqual(armCount, 1,
881
+ "the pending healthy check must keep its original deadline — re-arming it on every heal " +
882
+ "is why zero healthy checks fired in a 200s+ live capture");
883
+ assert.strictEqual(cleared, 0, "a pending healthy check must never be cancelled by a re-arm");
884
+ assert.strictEqual(adapter._healthyBaseline, 3000,
885
+ "the baseline must track the NEWEST restart, so a poll completed before it cannot be " +
886
+ "mistaken for proof that the latest restart worked");
887
+ assert.strictEqual(adapter._healthyReArmMsg, "third", "the re-arm reason must track the newest restart too");
888
+ }
889
+
514
890
  (async () => {
515
891
  await testStartDoesNotWaitForFirstLongPollAndArmsStallWatchdog();
516
892
  await testHealStopsPlainlyDestroysAgentAndRestartsIdle();
@@ -523,6 +899,11 @@ async function testHealthySignalClearsWedgeOnceAPollCompletes() {
523
899
  await testHealNullsPoisonedPollingSoLibRebuildsFresh();
524
900
  await testHealthySignalReArmsWhenNoPollCompleted();
525
901
  await testHealthySignalClearsWedgeOnceAPollCompletes();
902
+ await testOutagePauseDoesNotFeedTheWedgeExitClock();
903
+ await testOneNetworkBlipDoesNotSpiralIntoTheWedgeExitLoop();
904
+ await testOurOwnTeardownIsNotCountedAsAFault();
905
+ testHealLadderOutWaitsTheLongPoll();
906
+ testHealthyCheckCannotBeStarvedByRepeatedReArms();
526
907
  console.log("telegram poll recovery OK");
527
908
  })().catch((err) => {
528
909
  console.error(err.stack || err);