@inetafrica/open-claudia 3.0.42 → 3.0.43
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +6 -0
- package/channels/telegram/adapter.js +122 -31
- package/package.json +1 -1
- package/test-provider-language.js +13 -1
- package/test-telegram-poll-recovery.js +121 -2
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## v3.0.43 — Telegram stops rebooting into a dead network: outages now pause-and-wait instead of restart-storming
|
|
4
|
+
|
|
5
|
+
- **A host-wide network blip no longer triggers a reboot storm.** v3.0.42 fixed the *in-loop* poll wedge (a poisoned reused poller) so it heals in place — but it still kept the 10-minute `process.exit(1)` backstop, which fired even when the *entire host* had lost the network (a laptop sleep/wake, or pod egress starvation where Telegram poll+send, Kazee WS and Spaces WS all drop together). Exiting into a dead network just relaunched straight back into the same outage, so you'd see "Back online and ready! Running v3.0.42" every ~14 minutes, busting the prompt cache and spamming the chat. Now the self-exit is **reachability-gated**: before it ever restarts the process it runs a fresh, connection-fresh HEAD probe to `api.telegram.org`, and only exits if that probe genuinely **succeeds** — proving the API is reachable and it's a real in-process wedge worth a clean restart, not the host being offline.
|
|
6
|
+
- **When the host is actually offline, the bot now pauses and waits instead of hammering.** On a confirmed-unreachable probe the adapter enters an *outage pause*: it **stops polling entirely** (so `node-telegram-bot-api` no longer respins `getUpdates` every ~300ms into a dead socket — the "not waiting between reconnects" hammering) and re-probes on a calm **30-second** cadence. The instant connectivity returns it resumes polling cleanly from a fresh poller (the update offset is preserved, so nothing is refetched or skipped) — no process restart, no "Back online" spam. The pause also suspends the health watchdog and hiccup handler so they don't fight the slow re-probe. Poll-health transitions are now timestamped for easier post-mortem.
|
|
7
|
+
- **Agent-space / OpenClaw:** no new deps, env, or schema change; Docker image builds FROM the existing `:base`. Change is confined to `channels/telegram/adapter.js` (the `_heal` backstop plus the new `_enterOutagePause`/`_armHealthyCheck` paths); covered by new cases in `test-telegram-poll-recovery.js` (the gated exit defers on a live host, pauses instead of rebooting on a dead one, and resumes when the network returns). Existing pods pick it up on their next approved upgrade.
|
|
8
|
+
|
|
3
9
|
## v3.0.42 — Telegram stops rebooting itself: the long-poll wedge now heals in place
|
|
4
10
|
|
|
5
11
|
- **The bot no longer restarts the whole process to recover from a stuck Telegram long-poll.** When the network stalls a poll mid-flight (a half-open socket after a laptop sleep/wake, or a NAT/IPv6 connection eviction), `node-telegram-bot-api` was left holding one reused internal poller whose abort flag got stuck "on". Restarting polling *in place* revived that same poisoned object, which fired a single `getUpdates` and then silently refused to schedule the next one — a dead loop that threw no error, so the heal looked like it "worked" while nothing was actually polling. The only thing that ever truly recovered was the 10-minute backstop calling `process.exit(1)` and letting the supervisor relaunch — i.e. the reboot storm you'd see. The heal now **discards the poisoned poller so a fresh one is built** (the update offset is preserved, so no messages are refetched or skipped), and it only declares polling "healthy again" once a `getUpdates` has genuinely **completed since the restart** — otherwise it re-arms another heal instead of blindly declaring victory. Net effect: transient poll wedges recover in seconds without a process restart; the 10-min self-exit becomes the rare true-last-resort it was meant to be.
|
|
@@ -44,6 +44,8 @@ class TelegramAdapter {
|
|
|
44
44
|
this._deferredExitLogAt = 0; // throttle the "deferring restart, turn in progress" log (1/min)
|
|
45
45
|
this._lastHiccupLog = 0; // throttle the transient-error log (1/min)
|
|
46
46
|
this._healBackoff = 0; // grows per consecutive heal, capped
|
|
47
|
+
this._pausedForOutage = false; // host-network outage: poll loop stopped, slow re-probe owns recovery
|
|
48
|
+
this._outageProbeTimer = null; // the ~30s re-probe loop that runs while paused
|
|
47
49
|
this._pollWatchdogTimer = null; // detects a silent first/active getUpdates stall
|
|
48
50
|
this._pollStartedAt = 0;
|
|
49
51
|
this._conflictSince = 0; // start of the current 409 streak
|
|
@@ -83,6 +85,7 @@ class TelegramAdapter {
|
|
|
83
85
|
}
|
|
84
86
|
|
|
85
87
|
_checkPollingHealth() {
|
|
88
|
+
if (this._pausedForOutage) return; // outage pause runs its own slow re-probe
|
|
86
89
|
if (this._conflictTimer || this._conflictSince) return; // 409 backoff owns recovery
|
|
87
90
|
const lastCompletedPoll = this.bot?._polling?._lastUpdate || this._pollStartedAt;
|
|
88
91
|
if (!lastCompletedPoll || Date.now() - lastCompletedPoll <= 75000) return;
|
|
@@ -95,6 +98,8 @@ class TelegramAdapter {
|
|
|
95
98
|
if (this._healthyTimer) { clearTimeout(this._healthyTimer); this._healthyTimer = null; }
|
|
96
99
|
if (this._conflictTimer) { clearTimeout(this._conflictTimer); this._conflictTimer = null; }
|
|
97
100
|
if (this._conflictClearTimer) { clearTimeout(this._conflictClearTimer); this._conflictClearTimer = null; }
|
|
101
|
+
if (this._outageProbeTimer) { clearTimeout(this._outageProbeTimer); this._outageProbeTimer = null; }
|
|
102
|
+
this._pausedForOutage = false;
|
|
98
103
|
try { await this._stopPollingClean("adapter stop"); } catch (e) {}
|
|
99
104
|
}
|
|
100
105
|
|
|
@@ -107,6 +112,7 @@ class TelegramAdapter {
|
|
|
107
112
|
// old exit-for-relaunch *was* the restart the bot kept doing on every idle
|
|
108
113
|
// timeout (a sleep/wake burst could exhaust all 6 attempts in seconds).
|
|
109
114
|
_onPollingHiccup(msg) {
|
|
115
|
+
if (this._pausedForOutage) return; // paused for a host outage; the slow re-probe owns recovery
|
|
110
116
|
// During a 409 streak the conflict handler owns the poll lifecycle; a heal
|
|
111
117
|
// here would startPolling straight back into the conflict and its errors
|
|
112
118
|
// would keep _wedgedSince pinned until the 10-min backstop killed us (the
|
|
@@ -116,7 +122,7 @@ class TelegramAdapter {
|
|
|
116
122
|
if (!this._wedgedSince) this._wedgedSince = now;
|
|
117
123
|
if (now - this._lastHiccupLog > 60000) {
|
|
118
124
|
this._lastHiccupLog = now;
|
|
119
|
-
console.log(
|
|
125
|
+
console.log(`${this._stamp()} Polling hiccup (${msg}) — retrying on a fresh connection.`);
|
|
120
126
|
}
|
|
121
127
|
if (this._healTimer) return; // a heal is already scheduled; let it run
|
|
122
128
|
this._healBackoff = Math.min(this._healBackoff + 1, 5);
|
|
@@ -148,31 +154,74 @@ class TelegramAdapter {
|
|
|
148
154
|
catch (e) { return false; }
|
|
149
155
|
}
|
|
150
156
|
|
|
157
|
+
_stamp() {
|
|
158
|
+
return new Date().toISOString();
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
// Can this HOST reach Telegram AT ALL right now? A one-shot HEAD-ish probe on
|
|
162
|
+
// a brand-new connection (agent:false so it never rides our possibly-wedged
|
|
163
|
+
// poll agent), bounded so the probe itself can't hang. Any HTTP response —
|
|
164
|
+
// even a 404 — proves the TCP+TLS path is alive; a reset/timeout means it
|
|
165
|
+
// isn't. This is the fresh-process-equivalent check that lets the wedge
|
|
166
|
+
// backstop tell a genuine in-loop poll wedge (a restart fixes it) from a
|
|
167
|
+
// host-wide network outage (every outbound connection is failing together —
|
|
168
|
+
// Telegram polls + sends AND the Kazee/Spaces websockets — so a reboot just
|
|
169
|
+
// relaunches into the same dead network and spams "Back online"). Never rejects.
|
|
170
|
+
_probeTelegramReachable() {
|
|
171
|
+
return new Promise((resolve) => {
|
|
172
|
+
let settled = false;
|
|
173
|
+
const done = (v) => { if (!settled) { settled = true; resolve(v); } };
|
|
174
|
+
try {
|
|
175
|
+
const req = https.get(
|
|
176
|
+
{ hostname: "api.telegram.org", port: 443, path: "/", agent: false, timeout: 8000 },
|
|
177
|
+
(res) => { res.resume(); done(true); },
|
|
178
|
+
);
|
|
179
|
+
req.on("timeout", () => { try { req.destroy(); } catch (e) {} done(false); });
|
|
180
|
+
req.on("error", () => done(false));
|
|
181
|
+
} catch (e) { done(false); }
|
|
182
|
+
});
|
|
183
|
+
}
|
|
184
|
+
|
|
151
185
|
async _heal() {
|
|
152
186
|
this._healTimer = null;
|
|
153
|
-
//
|
|
154
|
-
//
|
|
155
|
-
//
|
|
156
|
-
// the
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
//
|
|
163
|
-
//
|
|
164
|
-
//
|
|
165
|
-
//
|
|
166
|
-
//
|
|
167
|
-
//
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
187
|
+
// A sustained wedge (>60s, and not a 409 streak): probe ONE fresh connection
|
|
188
|
+
// to learn WHY before we keep restarting. This both distinguishes a genuine
|
|
189
|
+
// in-loop poll wedge from a host-network outage AND lets us stop hammering
|
|
190
|
+
// when the network itself is down. On a healthy network wedges clear in
|
|
191
|
+
// seconds and never reach this branch. (Never during a 409 streak: that's a
|
|
192
|
+
// token fight, not a network wedge, and the conflict handler owns the loop.)
|
|
193
|
+
if (this._wedgedSince && !this._conflictSince && Date.now() - this._wedgedSince > 60000) {
|
|
194
|
+
if (!(await this._probeTelegramReachable())) {
|
|
195
|
+
// A fresh connection to Telegram fails too: a host-wide network outage,
|
|
196
|
+
// not a poll wedge (we've watched the Telegram poll, Telegram sends, and
|
|
197
|
+
// the Kazee + Spaces websockets all drop in lockstep). A reboot can't fix
|
|
198
|
+
// a dead network — the fresh process can't reach Telegram either and just
|
|
199
|
+
// spams "Back online". Restarting the poll loop is equally futile and only
|
|
200
|
+
// hammers getUpdates every ~300ms (which can trip a 429 the instant the
|
|
201
|
+
// link flickers back). Stop polling entirely and wait it out on a slow
|
|
202
|
+
// re-probe; resume the moment connectivity returns. Never exits.
|
|
203
|
+
this._enterOutagePause();
|
|
204
|
+
return;
|
|
205
|
+
}
|
|
206
|
+
// Telegram answers a fresh connection, yet our poll loop still hasn't
|
|
207
|
+
// completed a getUpdates. If that's dragged past 10 minutes a restart-in-
|
|
208
|
+
// place plainly isn't clearing it — exit for a clean launchd relaunch. On a
|
|
209
|
+
// healthy network this never fires (the null-restart below clears an in-loop
|
|
210
|
+
// wedge in seconds); it replaces the old hair-trigger ~90s restart storm.
|
|
211
|
+
if (Date.now() - this._wedgedSince > 600000) {
|
|
212
|
+
if (this._anyTurnActive()) {
|
|
213
|
+
// A turn is in flight — never reboot mid-turn: it kills in-progress work
|
|
214
|
+
// and breaks the seamless resume. Defer the exit, fall through to another
|
|
215
|
+
// in-place recovery attempt, and let a later idle heal exit if needed.
|
|
216
|
+
if (!this._deferredExitLogAt || Date.now() - this._deferredExitLogAt > 60000) {
|
|
217
|
+
this._deferredExitLogAt = Date.now();
|
|
218
|
+
console.error(`${this._stamp()} Polling wedged >10min but a turn is in progress — deferring restart until idle.`);
|
|
219
|
+
}
|
|
220
|
+
} else {
|
|
221
|
+
console.error(`${this._stamp()} Polling wedged >10min and Telegram is reachable — exiting for a clean restart.`);
|
|
222
|
+
try { require("../../core/restart-reason").recordExitReason("Telegram polling wedged >10min while API reachable — self-exit for a clean restart", { code: 1 }); } catch (e) {}
|
|
223
|
+
process.exit(1);
|
|
171
224
|
}
|
|
172
|
-
} else {
|
|
173
|
-
console.error("Polling wedged >10min — exiting for a clean restart.");
|
|
174
|
-
try { require("../../core/restart-reason").recordExitReason("Telegram polling wedged >10min (long-poll unrecoverable) — self-exit for a clean restart", { code: 1 }); } catch (e) {}
|
|
175
|
-
process.exit(1);
|
|
176
225
|
}
|
|
177
226
|
}
|
|
178
227
|
try {
|
|
@@ -197,27 +246,69 @@ class TelegramAdapter {
|
|
|
197
246
|
} catch (e) {
|
|
198
247
|
return; // restart didn't take — the next hiccup reschedules a heal
|
|
199
248
|
}
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
249
|
+
this._armHealthyCheck(this._pollStartedAt, "restart completed but no getUpdates within 35s");
|
|
250
|
+
}
|
|
251
|
+
|
|
252
|
+
// Recovery is real only if a getUpdates has actually COMPLETED since the
|
|
253
|
+
// (re)start — not merely "no throw for 35s". A dead one-shot loop throws
|
|
254
|
+
// nothing, so a blind timer would declare it healthy and reset _wedgedSince,
|
|
255
|
+
// hiding the wedge until the 10-min backstop rebooted. Gate the reset on a
|
|
256
|
+
// completed poll; otherwise re-arm a heal.
|
|
257
|
+
_armHealthyCheck(restartedAt, reArmMsg) {
|
|
206
258
|
if (this._healthyTimer) clearTimeout(this._healthyTimer);
|
|
207
259
|
this._healthyTimer = setTimeout(() => {
|
|
208
260
|
this._healthyTimer = null;
|
|
209
261
|
const lastCompletedPoll = this.bot?._polling?._lastUpdate || 0;
|
|
210
262
|
if (lastCompletedPoll >= restartedAt) {
|
|
211
|
-
if (this._wedgedSince) console.log(
|
|
263
|
+
if (this._wedgedSince) console.log(`${this._stamp()} Polling healthy again.`);
|
|
212
264
|
this._wedgedSince = 0;
|
|
213
265
|
this._healBackoff = 0;
|
|
214
266
|
this._deferredExitLogAt = 0;
|
|
215
267
|
} else {
|
|
216
|
-
this._onPollingHiccup(
|
|
268
|
+
this._onPollingHiccup(reArmMsg);
|
|
217
269
|
}
|
|
218
270
|
}, 35000);
|
|
219
271
|
}
|
|
220
272
|
|
|
273
|
+
// Host-network outage recovery: STOP the poll loop entirely (kill the lib's
|
|
274
|
+
// ~300ms getUpdates retry spin) and re-probe Telegram on a slow ~30s cadence,
|
|
275
|
+
// resuming the instant a fresh connection succeeds. Nothing outbound can get
|
|
276
|
+
// through while the host has no network, so spinning only burns CPU and risks a
|
|
277
|
+
// 429 when the link flickers back — this is the "wait between reconnects"
|
|
278
|
+
// backoff. Polling and the socket.io adapters all auto-recover together when
|
|
279
|
+
// connectivity returns. Never exits.
|
|
280
|
+
_enterOutagePause() {
|
|
281
|
+
if (this._pausedForOutage) return;
|
|
282
|
+
this._pausedForOutage = true;
|
|
283
|
+
if (this._healthyTimer) { clearTimeout(this._healthyTimer); this._healthyTimer = null; }
|
|
284
|
+
console.error(`${this._stamp()} Telegram unreachable — host-network outage; pausing polls, re-probing every 30s (no reboot, no reconnect hammering).`);
|
|
285
|
+
// Stop the fast retry loop now (clean stop: no zombie loop, socket severed).
|
|
286
|
+
Promise.resolve(this._stopPollingClean("host-network outage pause")).catch(() => {});
|
|
287
|
+
const tick = async () => {
|
|
288
|
+
this._outageProbeTimer = null;
|
|
289
|
+
if (!this._pausedForOutage) return;
|
|
290
|
+
if (!(await this._probeTelegramReachable())) {
|
|
291
|
+
if (this._pausedForOutage) this._outageProbeTimer = setTimeout(tick, 30000);
|
|
292
|
+
return;
|
|
293
|
+
}
|
|
294
|
+
// Connectivity is back. Resume on a FRESH poll loop object (the offset
|
|
295
|
+
// survives in bot.options.polling params, so no updates are missed or
|
|
296
|
+
// refetched), then confirm a getUpdates actually completes.
|
|
297
|
+
this._pausedForOutage = false;
|
|
298
|
+
console.log(`${this._stamp()} Telegram reachable again — resuming polling after outage.`);
|
|
299
|
+
try {
|
|
300
|
+
this.bot._polling = null;
|
|
301
|
+
this._pollStartedAt = Date.now();
|
|
302
|
+
await this.bot.startPolling({ restart: false });
|
|
303
|
+
} catch (e) {
|
|
304
|
+
this._onPollingHiccup("resume after outage failed to start");
|
|
305
|
+
return;
|
|
306
|
+
}
|
|
307
|
+
this._armHealthyCheck(this._pollStartedAt, "resume after outage completed but no getUpdates within 35s");
|
|
308
|
+
};
|
|
309
|
+
this._outageProbeTimer = setTimeout(tick, 30000);
|
|
310
|
+
}
|
|
311
|
+
|
|
221
312
|
// Telegram 409s a getUpdates call while ANOTHER one holds the token: a
|
|
222
313
|
// ghost long-poll left by an uncleanly-killed predecessor (clears within
|
|
223
314
|
// its ≤30s poll timeout), or a genuinely running second poller. We hold
|
package/package.json
CHANGED
|
@@ -74,7 +74,19 @@ assert.match(pkg.description, /provider-agnostic coding-agent harness/i);
|
|
|
74
74
|
// builds a fresh object (offset survives via the shared params), and the
|
|
75
75
|
// "healthy" reset requires a getUpdates that actually COMPLETED since the restart
|
|
76
76
|
// (else it re-arms a heal) instead of blindly declaring victory after 35s.
|
|
77
|
-
|
|
77
|
+
// 3.0.43 approved by Sumeet 2026-07-18 ("Yeh push"): reachability-gated self-exit
|
|
78
|
+
// + outage-pause backoff, so a host-wide network blip no longer triggers a
|
|
79
|
+
// reboot storm. v3.0.42 healed the in-loop wedge but STILL rebooted after 10min
|
|
80
|
+
// even when the whole host had lost the network (laptop sleep, pod egress
|
|
81
|
+
// starvation) — exiting into a dead network just relaunched into the same
|
|
82
|
+
// outage, spamming "Back online" every ~14min. The 10-min self-exit is now gated
|
|
83
|
+
// on a fresh HEAD probe to api.telegram.org actually SUCCEEDING (proves a genuine
|
|
84
|
+
// in-loop wedge, not a dead host); if the probe fails the adapter enters an
|
|
85
|
+
// outage pause — it STOPS polling (no 300ms lib reconnect spin) and re-probes
|
|
86
|
+
// every 30s, resuming cleanly the moment connectivity returns. Poll-health logs
|
|
87
|
+
// are timestamped. Confined to channels/telegram/adapter.js; new cases in
|
|
88
|
+
// test-telegram-poll-recovery.js cover the pause+resume and the gated exit.
|
|
89
|
+
assert.strictEqual(pkg.version, "3.0.43", "release version must remain unchanged without explicit approval");
|
|
78
90
|
for (const keyword of ["claude", "codex", "coding-agent", "provider-agnostic"]) {
|
|
79
91
|
assert.ok(pkg.keywords.includes(keyword), `package keyword missing: ${keyword}`);
|
|
80
92
|
}
|
|
@@ -190,6 +190,7 @@ async function testWedgeExitDefersWhileTurnInFlight() {
|
|
|
190
190
|
adapter._deferredExitLogAt = 0;
|
|
191
191
|
adapter._healBackoff = 0;
|
|
192
192
|
adapter._anyTurnActive = () => true; // a turn is running/queued
|
|
193
|
+
adapter._probeTelegramReachable = async () => true; // reachable ⇒ genuine wedge; defer path
|
|
193
194
|
adapter._agent = { destroy() { calls.push("destroy"); } };
|
|
194
195
|
adapter.bot = {
|
|
195
196
|
async stopPolling() { calls.push("stop"); },
|
|
@@ -217,8 +218,10 @@ async function testWedgeExitDefersWhileTurnInFlight() {
|
|
|
217
218
|
assert.ok(adapter._deferredExitLogAt > 0, "the deferral must be logged (throttle stamp set)");
|
|
218
219
|
}
|
|
219
220
|
|
|
220
|
-
// Once the process is idle (no turn running/queued)
|
|
221
|
-
//
|
|
221
|
+
// Once the process is idle (no turn running/queued) AND Telegram answers a fresh
|
|
222
|
+
// connection — proving this is a real in-loop poll wedge, not a host-network
|
|
223
|
+
// outage — the backstop exits for a clean restart exactly as before. The
|
|
224
|
+
// deferral only postpones, never cancels.
|
|
222
225
|
async function testWedgeExitFiresWhenIdle() {
|
|
223
226
|
let exitCode;
|
|
224
227
|
const adapter = Object.create(TelegramAdapter.prototype);
|
|
@@ -227,6 +230,7 @@ async function testWedgeExitFiresWhenIdle() {
|
|
|
227
230
|
adapter._wedgedSince = Date.now() - 11 * 60 * 1000; // wedged >10min
|
|
228
231
|
adapter._deferredExitLogAt = 0;
|
|
229
232
|
adapter._anyTurnActive = () => false; // idle
|
|
233
|
+
adapter._probeTelegramReachable = async () => true; // Telegram reachable ⇒ genuine poll wedge
|
|
230
234
|
adapter._agent = { destroy() {} };
|
|
231
235
|
adapter.bot = { async stopPolling() {}, async startPolling() {} };
|
|
232
236
|
|
|
@@ -244,6 +248,116 @@ async function testWedgeExitFiresWhenIdle() {
|
|
|
244
248
|
assert.strictEqual(exitCode, 1, "the idle wedge backstop must exit(1) for a clean restart");
|
|
245
249
|
}
|
|
246
250
|
|
|
251
|
+
// THE REBOOT-STORM regression, now resolved by an OUTAGE PAUSE. When the wedge is
|
|
252
|
+
// a host-wide network outage — every outbound connection failing together
|
|
253
|
+
// (Telegram polls + sends AND the Kazee/Spaces websockets) — a reboot can't help:
|
|
254
|
+
// the fresh process can't reach Telegram either and just relaunches into the same
|
|
255
|
+
// dead network, spamming "Back online". So even when idle, the backstop must NOT
|
|
256
|
+
// exit while a fresh connection to Telegram also fails. Instead it STOPS the poll
|
|
257
|
+
// loop entirely (no ~300ms getUpdates hammering that can trip a 429 when the link
|
|
258
|
+
// flickers back) and waits on a slow re-probe. Fails before the fix because the
|
|
259
|
+
// idle backstop exits unconditionally.
|
|
260
|
+
async function testWedgeExitPausesInsteadOfRebootingOnHostOutage() {
|
|
261
|
+
const calls = [];
|
|
262
|
+
let exited = false;
|
|
263
|
+
const adapter = Object.create(TelegramAdapter.prototype);
|
|
264
|
+
adapter._healTimer = null;
|
|
265
|
+
adapter._healthyTimer = null;
|
|
266
|
+
adapter._conflictSince = 0;
|
|
267
|
+
adapter._wedgedSince = Date.now() - 11 * 60 * 1000; // wedged >10min
|
|
268
|
+
adapter._deferredExitLogAt = 0;
|
|
269
|
+
adapter._healBackoff = 0;
|
|
270
|
+
adapter._pausedForOutage = false;
|
|
271
|
+
adapter._outageProbeTimer = null;
|
|
272
|
+
adapter._anyTurnActive = () => false; // idle
|
|
273
|
+
adapter._probeTelegramReachable = async () => false; // host can't reach Telegram at all
|
|
274
|
+
adapter._agent = { destroy() { calls.push("destroy"); } };
|
|
275
|
+
adapter.bot = {
|
|
276
|
+
async stopPolling() { calls.push("stop"); },
|
|
277
|
+
async startPolling() { calls.push("start"); },
|
|
278
|
+
};
|
|
279
|
+
|
|
280
|
+
const realExit = process.exit;
|
|
281
|
+
process.exit = () => { exited = true; throw new Error("process.exit must not be called during a host-network outage"); };
|
|
282
|
+
const realSetTimeout = global.setTimeout;
|
|
283
|
+
const realClearTimeout = global.clearTimeout;
|
|
284
|
+
global.setTimeout = (fn, ms) => { if (ms === 500) fn(); return { ms }; };
|
|
285
|
+
global.clearTimeout = () => {};
|
|
286
|
+
|
|
287
|
+
try {
|
|
288
|
+
await adapter._heal();
|
|
289
|
+
await new Promise((r) => realSetTimeout(r, 5)); // let the fire-and-forget stop settle
|
|
290
|
+
} finally {
|
|
291
|
+
process.exit = realExit;
|
|
292
|
+
global.setTimeout = realSetTimeout;
|
|
293
|
+
global.clearTimeout = realClearTimeout;
|
|
294
|
+
}
|
|
295
|
+
|
|
296
|
+
assert.strictEqual(exited, false, "a host-network outage must NOT trigger a reboot (the fresh process can't reach the network either)");
|
|
297
|
+
assert.strictEqual(adapter._pausedForOutage, true, "an unreachable host must pause polling instead of rebooting");
|
|
298
|
+
assert.ok(calls.includes("stop") && calls.includes("destroy"),
|
|
299
|
+
"entering the outage pause must stop the poll loop and sever the socket");
|
|
300
|
+
assert.ok(!calls.includes("start"),
|
|
301
|
+
"while the host is unreachable the bot must NOT keep reconnecting (no ~300ms getUpdates hammering)");
|
|
302
|
+
assert.ok(adapter._outageProbeTimer, "the pause must schedule a slow re-probe to resume when the network returns");
|
|
303
|
+
}
|
|
304
|
+
|
|
305
|
+
// Companion: once the network returns, the outage pause's slow re-probe must
|
|
306
|
+
// RESUME polling on a FRESH loop object (offset preserved via bot.options.polling
|
|
307
|
+
// params) and leave the paused state — the bot recovers on its own, no reboot.
|
|
308
|
+
async function testOutagePauseResumesPollingWhenNetworkReturns() {
|
|
309
|
+
const calls = [];
|
|
310
|
+
let reachable = false;
|
|
311
|
+
let tickFn;
|
|
312
|
+
const adapter = Object.create(TelegramAdapter.prototype);
|
|
313
|
+
adapter._healTimer = null;
|
|
314
|
+
adapter._healthyTimer = null;
|
|
315
|
+
adapter._wedgedSince = Date.now() - 2 * 60 * 1000;
|
|
316
|
+
adapter._healBackoff = 0;
|
|
317
|
+
adapter._deferredExitLogAt = 0;
|
|
318
|
+
adapter._pausedForOutage = false;
|
|
319
|
+
adapter._outageProbeTimer = null;
|
|
320
|
+
adapter._pollStartedAt = 0;
|
|
321
|
+
adapter._onPollingHiccup = () => {};
|
|
322
|
+
adapter._probeTelegramReachable = async () => reachable;
|
|
323
|
+
adapter._agent = { destroy() { calls.push("destroy"); } };
|
|
324
|
+
adapter.bot = {
|
|
325
|
+
_polling: { _abort: true, _lastUpdate: 0 },
|
|
326
|
+
async stopPolling() { calls.push("stop"); },
|
|
327
|
+
async startPolling() {
|
|
328
|
+
calls.push("start");
|
|
329
|
+
if (!this._polling) this._polling = { _abort: false, _lastUpdate: Date.now() };
|
|
330
|
+
},
|
|
331
|
+
};
|
|
332
|
+
|
|
333
|
+
const realSetTimeout = global.setTimeout;
|
|
334
|
+
const realClearTimeout = global.clearTimeout;
|
|
335
|
+
global.setTimeout = (fn, ms) => {
|
|
336
|
+
if (ms === 500 || ms === 5000) fn(); // stop-clean internals fire inline
|
|
337
|
+
else if (ms === 30000 && !tickFn) tickFn = fn; // capture the outage re-probe tick
|
|
338
|
+
return { ms };
|
|
339
|
+
};
|
|
340
|
+
global.clearTimeout = () => {};
|
|
341
|
+
|
|
342
|
+
try {
|
|
343
|
+
adapter._enterOutagePause();
|
|
344
|
+
await new Promise((r) => realSetTimeout(r, 5)); // let the fire-and-forget stop settle
|
|
345
|
+
assert.strictEqual(adapter._pausedForOutage, true, "an unreachable host pauses polling");
|
|
346
|
+
assert.ok(!calls.includes("start"), "no reconnecting while paused (network is down)");
|
|
347
|
+
assert.strictEqual(typeof tickFn, "function", "the pause must schedule a slow re-probe");
|
|
348
|
+
|
|
349
|
+
reachable = true;
|
|
350
|
+
adapter.bot._polling = { _abort: true, _lastUpdate: 0 }; // stale object present before resume
|
|
351
|
+
await tickFn();
|
|
352
|
+
assert.strictEqual(adapter._pausedForOutage, false, "a successful re-probe must leave the pause");
|
|
353
|
+
assert.ok(calls.includes("start"), "a successful re-probe must resume polling");
|
|
354
|
+
assert.strictEqual(adapter.bot._polling._abort, false, "resume must build a FRESH poll loop object (offset preserved)");
|
|
355
|
+
} finally {
|
|
356
|
+
global.setTimeout = realSetTimeout;
|
|
357
|
+
global.clearTimeout = realClearTimeout;
|
|
358
|
+
}
|
|
359
|
+
}
|
|
360
|
+
|
|
247
361
|
// THE RESTART STORM regression. A wedge (half-open getUpdates after sleep/wake
|
|
248
362
|
// or a NAT/IPv6 eviction) leaves the lib's REUSED _polling object with `_abort`
|
|
249
363
|
// stuck true: the hung request never settled within our stop cap, so stop()'s
|
|
@@ -266,6 +380,7 @@ async function testHealNullsPoisonedPollingSoLibRebuildsFresh() {
|
|
|
266
380
|
adapter._healBackoff = 0;
|
|
267
381
|
adapter._pollStartedAt = ancient;
|
|
268
382
|
adapter._onPollingHiccup = () => {};
|
|
383
|
+
adapter._probeTelegramReachable = async () => true; // reachable ⇒ normal in-place heal path
|
|
269
384
|
adapter._agent = { destroy() {} };
|
|
270
385
|
adapter.bot = {
|
|
271
386
|
_polling: poisoned,
|
|
@@ -310,6 +425,7 @@ async function testHealthySignalReArmsWhenNoPollCompleted() {
|
|
|
310
425
|
adapter._healBackoff = 2;
|
|
311
426
|
adapter._deferredExitLogAt = 0;
|
|
312
427
|
adapter._pollStartedAt = 0;
|
|
428
|
+
adapter._probeTelegramReachable = async () => true; // reachable ⇒ normal in-place heal path
|
|
313
429
|
adapter._agent = { destroy() {} };
|
|
314
430
|
adapter.bot = {
|
|
315
431
|
_polling: { _abort: true, _lastUpdate: 0 },
|
|
@@ -360,6 +476,7 @@ async function testHealthySignalClearsWedgeOnceAPollCompletes() {
|
|
|
360
476
|
adapter._healBackoff = 3;
|
|
361
477
|
adapter._deferredExitLogAt = 12345;
|
|
362
478
|
adapter._pollStartedAt = 0;
|
|
479
|
+
adapter._probeTelegramReachable = async () => true; // reachable ⇒ normal in-place heal path
|
|
363
480
|
adapter._agent = { destroy() {} };
|
|
364
481
|
adapter.bot = {
|
|
365
482
|
_polling: { _abort: true, _lastUpdate: 0 },
|
|
@@ -401,6 +518,8 @@ async function testHealthySignalClearsWedgeOnceAPollCompletes() {
|
|
|
401
518
|
await test409ConflictSeversGhostSocketAndRestartsIdle();
|
|
402
519
|
await testWedgeExitDefersWhileTurnInFlight();
|
|
403
520
|
await testWedgeExitFiresWhenIdle();
|
|
521
|
+
await testWedgeExitPausesInsteadOfRebootingOnHostOutage();
|
|
522
|
+
await testOutagePauseResumesPollingWhenNetworkReturns();
|
|
404
523
|
await testHealNullsPoisonedPollingSoLibRebuildsFresh();
|
|
405
524
|
await testHealthySignalReArmsWhenNoPollCompleted();
|
|
406
525
|
await testHealthySignalClearsWedgeOnceAPollCompletes();
|