@inetafrica/open-claudia 3.0.41 → 3.0.43
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +11 -0
- package/channels/telegram/adapter.js +145 -31
- package/package.json +1 -1
- package/test-provider-language.js +22 -1
- package/test-telegram-poll-recovery.js +274 -2
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,16 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## v3.0.43 — Telegram stops rebooting into a dead network: outages now pause-and-wait instead of restart-storming
|
|
4
|
+
|
|
5
|
+
- **A host-wide network blip no longer triggers a reboot storm.** v3.0.42 fixed the *in-loop* poll wedge (a poisoned reused poller) so it heals in place — but it still kept the 10-minute `process.exit(1)` backstop, which fired even when the *entire host* had lost the network (a laptop sleep/wake, or pod egress starvation where Telegram poll+send, Kazee WS and Spaces WS all drop together). Exiting into a dead network just relaunched straight back into the same outage, so you'd see "Back online and ready! Running v3.0.42" every ~14 minutes, busting the prompt cache and spamming the chat. Now the self-exit is **reachability-gated**: before it ever restarts the process it runs a fresh, connection-fresh HEAD probe to `api.telegram.org`, and only exits if that probe genuinely **succeeds** — proving the API is reachable and it's a real in-process wedge worth a clean restart, not the host being offline.
|
|
6
|
+
- **When the host is actually offline, the bot now pauses and waits instead of hammering.** On a confirmed-unreachable probe the adapter enters an *outage pause*: it **stops polling entirely** (so `node-telegram-bot-api` no longer respins `getUpdates` every ~300ms into a dead socket — the "not waiting between reconnects" hammering) and re-probes on a calm **30-second** cadence. The instant connectivity returns it resumes polling cleanly from a fresh poller (the update offset is preserved, so nothing is refetched or skipped) — no process restart, no "Back online" spam. The pause also suspends the health watchdog and hiccup handler so they don't fight the slow re-probe. Poll-health transitions are now timestamped for easier post-mortem.
|
|
7
|
+
- **Agent-space / OpenClaw:** no new deps, env, or schema change; Docker image builds FROM the existing `:base`. Change is confined to `channels/telegram/adapter.js` (the `_heal` backstop plus the new `_enterOutagePause`/`_armHealthyCheck` paths); covered by new cases in `test-telegram-poll-recovery.js` (the gated exit defers on a live host, pauses instead of rebooting on a dead one, and resumes when the network returns). Existing pods pick it up on their next approved upgrade.
|
|
8
|
+
|
|
9
|
+
## v3.0.42 — Telegram stops rebooting itself: the long-poll wedge now heals in place
|
|
10
|
+
|
|
11
|
+
- **The bot no longer restarts the whole process to recover from a stuck Telegram long-poll.** When the network stalls a poll mid-flight (a half-open socket after a laptop sleep/wake, or a NAT/IPv6 connection eviction), `node-telegram-bot-api` was left holding one reused internal poller whose abort flag got stuck "on". Restarting polling *in place* revived that same poisoned object, which fired a single `getUpdates` and then silently refused to schedule the next one — a dead loop that threw no error, so the heal looked like it "worked" while nothing was actually polling. The only thing that ever truly recovered was the 10-minute backstop calling `process.exit(1)` and letting the supervisor relaunch — i.e. the reboot storm you'd see. The heal now **discards the poisoned poller so a fresh one is built** (the update offset is preserved, so no messages are refetched or skipped), and it only declares polling "healthy again" once a `getUpdates` has genuinely **completed since the restart** — otherwise it re-arms another heal instead of blindly declaring victory. Net effect: transient poll wedges recover in seconds without a process restart; the 10-min self-exit becomes the rare true-last-resort it was meant to be.
|
|
12
|
+
- **Agent-space / OpenClaw:** no new deps, env, or schema change; Docker image builds FROM the existing `:base`. Change is confined to `channels/telegram/adapter.js` (the `_heal` path); covered by three new cases in `test-telegram-poll-recovery.js` (proven to fail before the fix and pass after — the poisoned-object revival and the false "healthy" signal). Existing pods pick it up on their next approved upgrade.
|
|
13
|
+
|
|
3
14
|
## v3.0.41 — Verbose off now means a clean chat: the "Let me…" narration no longer lingers
|
|
4
15
|
|
|
5
16
|
- **With `/verbose` off, the interim progress message is now removed once a turn settles, so the chat is left with only the final answer.** During a multi-step turn the bot posts a throttled "working…" preview that shows the short "Let me check…/Let me look at…" narration emitted before each tool call. That preview used to *stay* — settling into a permanent message — regardless of `/verbose`, which only ever gated the extended-thinking (🧠) stream. So even with verbose off you'd be left with the narration trail as a leftover message that reads like thinking-out-loud. `finalizeStreamPreview` now deletes that preview when verbose is off; `/verbose on` still keeps the narration trail. Live progress during long turns is unchanged either way — the cleanup only affects what remains after the turn ends.
|
|
@@ -44,6 +44,8 @@ class TelegramAdapter {
|
|
|
44
44
|
this._deferredExitLogAt = 0; // throttle the "deferring restart, turn in progress" log (1/min)
|
|
45
45
|
this._lastHiccupLog = 0; // throttle the transient-error log (1/min)
|
|
46
46
|
this._healBackoff = 0; // grows per consecutive heal, capped
|
|
47
|
+
this._pausedForOutage = false; // host-network outage: poll loop stopped, slow re-probe owns recovery
|
|
48
|
+
this._outageProbeTimer = null; // the ~30s re-probe loop that runs while paused
|
|
47
49
|
this._pollWatchdogTimer = null; // detects a silent first/active getUpdates stall
|
|
48
50
|
this._pollStartedAt = 0;
|
|
49
51
|
this._conflictSince = 0; // start of the current 409 streak
|
|
@@ -83,6 +85,7 @@ class TelegramAdapter {
|
|
|
83
85
|
}
|
|
84
86
|
|
|
85
87
|
_checkPollingHealth() {
|
|
88
|
+
if (this._pausedForOutage) return; // outage pause runs its own slow re-probe
|
|
86
89
|
if (this._conflictTimer || this._conflictSince) return; // 409 backoff owns recovery
|
|
87
90
|
const lastCompletedPoll = this.bot?._polling?._lastUpdate || this._pollStartedAt;
|
|
88
91
|
if (!lastCompletedPoll || Date.now() - lastCompletedPoll <= 75000) return;
|
|
@@ -95,6 +98,8 @@ class TelegramAdapter {
|
|
|
95
98
|
if (this._healthyTimer) { clearTimeout(this._healthyTimer); this._healthyTimer = null; }
|
|
96
99
|
if (this._conflictTimer) { clearTimeout(this._conflictTimer); this._conflictTimer = null; }
|
|
97
100
|
if (this._conflictClearTimer) { clearTimeout(this._conflictClearTimer); this._conflictClearTimer = null; }
|
|
101
|
+
if (this._outageProbeTimer) { clearTimeout(this._outageProbeTimer); this._outageProbeTimer = null; }
|
|
102
|
+
this._pausedForOutage = false;
|
|
98
103
|
try { await this._stopPollingClean("adapter stop"); } catch (e) {}
|
|
99
104
|
}
|
|
100
105
|
|
|
@@ -107,6 +112,7 @@ class TelegramAdapter {
|
|
|
107
112
|
// old exit-for-relaunch *was* the restart the bot kept doing on every idle
|
|
108
113
|
// timeout (a sleep/wake burst could exhaust all 6 attempts in seconds).
|
|
109
114
|
_onPollingHiccup(msg) {
|
|
115
|
+
if (this._pausedForOutage) return; // paused for a host outage; the slow re-probe owns recovery
|
|
110
116
|
// During a 409 streak the conflict handler owns the poll lifecycle; a heal
|
|
111
117
|
// here would startPolling straight back into the conflict and its errors
|
|
112
118
|
// would keep _wedgedSince pinned until the 10-min backstop killed us (the
|
|
@@ -116,7 +122,7 @@ class TelegramAdapter {
|
|
|
116
122
|
if (!this._wedgedSince) this._wedgedSince = now;
|
|
117
123
|
if (now - this._lastHiccupLog > 60000) {
|
|
118
124
|
this._lastHiccupLog = now;
|
|
119
|
-
console.log(
|
|
125
|
+
console.log(`${this._stamp()} Polling hiccup (${msg}) — retrying on a fresh connection.`);
|
|
120
126
|
}
|
|
121
127
|
if (this._healTimer) return; // a heal is already scheduled; let it run
|
|
122
128
|
this._healBackoff = Math.min(this._healBackoff + 1, 5);
|
|
@@ -148,51 +154,159 @@ class TelegramAdapter {
|
|
|
148
154
|
catch (e) { return false; }
|
|
149
155
|
}
|
|
150
156
|
|
|
157
|
+
_stamp() {
|
|
158
|
+
return new Date().toISOString();
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
// Can this HOST reach Telegram AT ALL right now? A one-shot HEAD-ish probe on
|
|
162
|
+
// a brand-new connection (agent:false so it never rides our possibly-wedged
|
|
163
|
+
// poll agent), bounded so the probe itself can't hang. Any HTTP response —
|
|
164
|
+
// even a 404 — proves the TCP+TLS path is alive; a reset/timeout means it
|
|
165
|
+
// isn't. This is the fresh-process-equivalent check that lets the wedge
|
|
166
|
+
// backstop tell a genuine in-loop poll wedge (a restart fixes it) from a
|
|
167
|
+
// host-wide network outage (every outbound connection is failing together —
|
|
168
|
+
// Telegram polls + sends AND the Kazee/Spaces websockets — so a reboot just
|
|
169
|
+
// relaunches into the same dead network and spams "Back online"). Never rejects.
|
|
170
|
+
_probeTelegramReachable() {
|
|
171
|
+
return new Promise((resolve) => {
|
|
172
|
+
let settled = false;
|
|
173
|
+
const done = (v) => { if (!settled) { settled = true; resolve(v); } };
|
|
174
|
+
try {
|
|
175
|
+
const req = https.get(
|
|
176
|
+
{ hostname: "api.telegram.org", port: 443, path: "/", agent: false, timeout: 8000 },
|
|
177
|
+
(res) => { res.resume(); done(true); },
|
|
178
|
+
);
|
|
179
|
+
req.on("timeout", () => { try { req.destroy(); } catch (e) {} done(false); });
|
|
180
|
+
req.on("error", () => done(false));
|
|
181
|
+
} catch (e) { done(false); }
|
|
182
|
+
});
|
|
183
|
+
}
|
|
184
|
+
|
|
151
185
|
async _heal() {
|
|
152
186
|
this._healTimer = null;
|
|
153
|
-
//
|
|
154
|
-
//
|
|
155
|
-
//
|
|
156
|
-
// the
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
//
|
|
163
|
-
//
|
|
164
|
-
//
|
|
165
|
-
//
|
|
166
|
-
//
|
|
167
|
-
//
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
187
|
+
// A sustained wedge (>60s, and not a 409 streak): probe ONE fresh connection
|
|
188
|
+
// to learn WHY before we keep restarting. This both distinguishes a genuine
|
|
189
|
+
// in-loop poll wedge from a host-network outage AND lets us stop hammering
|
|
190
|
+
// when the network itself is down. On a healthy network wedges clear in
|
|
191
|
+
// seconds and never reach this branch. (Never during a 409 streak: that's a
|
|
192
|
+
// token fight, not a network wedge, and the conflict handler owns the loop.)
|
|
193
|
+
if (this._wedgedSince && !this._conflictSince && Date.now() - this._wedgedSince > 60000) {
|
|
194
|
+
if (!(await this._probeTelegramReachable())) {
|
|
195
|
+
// A fresh connection to Telegram fails too: a host-wide network outage,
|
|
196
|
+
// not a poll wedge (we've watched the Telegram poll, Telegram sends, and
|
|
197
|
+
// the Kazee + Spaces websockets all drop in lockstep). A reboot can't fix
|
|
198
|
+
// a dead network — the fresh process can't reach Telegram either and just
|
|
199
|
+
// spams "Back online". Restarting the poll loop is equally futile and only
|
|
200
|
+
// hammers getUpdates every ~300ms (which can trip a 429 the instant the
|
|
201
|
+
// link flickers back). Stop polling entirely and wait it out on a slow
|
|
202
|
+
// re-probe; resume the moment connectivity returns. Never exits.
|
|
203
|
+
this._enterOutagePause();
|
|
204
|
+
return;
|
|
205
|
+
}
|
|
206
|
+
// Telegram answers a fresh connection, yet our poll loop still hasn't
|
|
207
|
+
// completed a getUpdates. If that's dragged past 10 minutes a restart-in-
|
|
208
|
+
// place plainly isn't clearing it — exit for a clean launchd relaunch. On a
|
|
209
|
+
// healthy network this never fires (the null-restart below clears an in-loop
|
|
210
|
+
// wedge in seconds); it replaces the old hair-trigger ~90s restart storm.
|
|
211
|
+
if (Date.now() - this._wedgedSince > 600000) {
|
|
212
|
+
if (this._anyTurnActive()) {
|
|
213
|
+
// A turn is in flight — never reboot mid-turn: it kills in-progress work
|
|
214
|
+
// and breaks the seamless resume. Defer the exit, fall through to another
|
|
215
|
+
// in-place recovery attempt, and let a later idle heal exit if needed.
|
|
216
|
+
if (!this._deferredExitLogAt || Date.now() - this._deferredExitLogAt > 60000) {
|
|
217
|
+
this._deferredExitLogAt = Date.now();
|
|
218
|
+
console.error(`${this._stamp()} Polling wedged >10min but a turn is in progress — deferring restart until idle.`);
|
|
219
|
+
}
|
|
220
|
+
} else {
|
|
221
|
+
console.error(`${this._stamp()} Polling wedged >10min and Telegram is reachable — exiting for a clean restart.`);
|
|
222
|
+
try { require("../../core/restart-reason").recordExitReason("Telegram polling wedged >10min while API reachable — self-exit for a clean restart", { code: 1 }); } catch (e) {}
|
|
223
|
+
process.exit(1);
|
|
171
224
|
}
|
|
172
|
-
} else {
|
|
173
|
-
console.error("Polling wedged >10min — exiting for a clean restart.");
|
|
174
|
-
try { require("../../core/restart-reason").recordExitReason("Telegram polling wedged >10min (long-poll unrecoverable) — self-exit for a clean restart", { code: 1 }); } catch (e) {}
|
|
175
|
-
process.exit(1);
|
|
176
225
|
}
|
|
177
226
|
}
|
|
178
227
|
try {
|
|
179
228
|
await this._stopPollingClean("stale Telegram long-poll recovery");
|
|
180
229
|
await new Promise((r) => setTimeout(r, 500));
|
|
181
|
-
//
|
|
182
|
-
//
|
|
230
|
+
// THE wedge fix: the lib reuses this.bot._polling across start/stop and
|
|
231
|
+
// only reschedules the poll loop when `_abort === false`. A wedge (half-
|
|
232
|
+
// open socket after sleep/wake or a NAT/IPv6 eviction) can leave the old
|
|
233
|
+
// getUpdates unsettled past our 5s stop cap, so `_abort` stays stuck true
|
|
234
|
+
// — the restarted loop then fires ONE getUpdates and dies in its .finally,
|
|
235
|
+
// a silent dead loop that only process.exit ever cleared (the restart
|
|
236
|
+
// storm). Nulling forces startPolling to build a FRESH object with
|
|
237
|
+
// `_abort=false`; the offset survives via the shared bot.options.polling
|
|
238
|
+
// params, so no updates are refetched or skipped.
|
|
239
|
+
this.bot._polling = null;
|
|
240
|
+
// The fresh object's _lastUpdate starts at 0; startPolling below blocks up
|
|
241
|
+
// to ~30s on the first long-poll, so stamp the watchdog grace baseline NOW
|
|
242
|
+
// — else _checkPollingHealth would fall back to a stale _pollStartedAt and
|
|
243
|
+
// false-fire mid-first-poll.
|
|
244
|
+
this._pollStartedAt = Date.now();
|
|
183
245
|
await this.bot.startPolling({ restart: false });
|
|
184
246
|
} catch (e) {
|
|
185
247
|
return; // restart didn't take — the next hiccup reschedules a heal
|
|
186
248
|
}
|
|
187
|
-
|
|
249
|
+
this._armHealthyCheck(this._pollStartedAt, "restart completed but no getUpdates within 35s");
|
|
250
|
+
}
|
|
251
|
+
|
|
252
|
+
// Recovery is real only if a getUpdates has actually COMPLETED since the
|
|
253
|
+
// (re)start — not merely "no throw for 35s". A dead one-shot loop throws
|
|
254
|
+
// nothing, so a blind timer would declare it healthy and reset _wedgedSince,
|
|
255
|
+
// hiding the wedge until the 10-min backstop rebooted. Gate the reset on a
|
|
256
|
+
// completed poll; otherwise re-arm a heal.
|
|
257
|
+
_armHealthyCheck(restartedAt, reArmMsg) {
|
|
188
258
|
if (this._healthyTimer) clearTimeout(this._healthyTimer);
|
|
189
259
|
this._healthyTimer = setTimeout(() => {
|
|
190
260
|
this._healthyTimer = null;
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
261
|
+
const lastCompletedPoll = this.bot?._polling?._lastUpdate || 0;
|
|
262
|
+
if (lastCompletedPoll >= restartedAt) {
|
|
263
|
+
if (this._wedgedSince) console.log(`${this._stamp()} Polling healthy again.`);
|
|
264
|
+
this._wedgedSince = 0;
|
|
265
|
+
this._healBackoff = 0;
|
|
266
|
+
this._deferredExitLogAt = 0;
|
|
267
|
+
} else {
|
|
268
|
+
this._onPollingHiccup(reArmMsg);
|
|
269
|
+
}
|
|
270
|
+
}, 35000);
|
|
271
|
+
}
|
|
272
|
+
|
|
273
|
+
// Host-network outage recovery: STOP the poll loop entirely (kill the lib's
|
|
274
|
+
// ~300ms getUpdates retry spin) and re-probe Telegram on a slow ~30s cadence,
|
|
275
|
+
// resuming the instant a fresh connection succeeds. Nothing outbound can get
|
|
276
|
+
// through while the host has no network, so spinning only burns CPU and risks a
|
|
277
|
+
// 429 when the link flickers back — this is the "wait between reconnects"
|
|
278
|
+
// backoff. Polling and the socket.io adapters all auto-recover together when
|
|
279
|
+
// connectivity returns. Never exits.
|
|
280
|
+
_enterOutagePause() {
|
|
281
|
+
if (this._pausedForOutage) return;
|
|
282
|
+
this._pausedForOutage = true;
|
|
283
|
+
if (this._healthyTimer) { clearTimeout(this._healthyTimer); this._healthyTimer = null; }
|
|
284
|
+
console.error(`${this._stamp()} Telegram unreachable — host-network outage; pausing polls, re-probing every 30s (no reboot, no reconnect hammering).`);
|
|
285
|
+
// Stop the fast retry loop now (clean stop: no zombie loop, socket severed).
|
|
286
|
+
Promise.resolve(this._stopPollingClean("host-network outage pause")).catch(() => {});
|
|
287
|
+
const tick = async () => {
|
|
288
|
+
this._outageProbeTimer = null;
|
|
289
|
+
if (!this._pausedForOutage) return;
|
|
290
|
+
if (!(await this._probeTelegramReachable())) {
|
|
291
|
+
if (this._pausedForOutage) this._outageProbeTimer = setTimeout(tick, 30000);
|
|
292
|
+
return;
|
|
293
|
+
}
|
|
294
|
+
// Connectivity is back. Resume on a FRESH poll loop object (the offset
|
|
295
|
+
// survives in bot.options.polling params, so no updates are missed or
|
|
296
|
+
// refetched), then confirm a getUpdates actually completes.
|
|
297
|
+
this._pausedForOutage = false;
|
|
298
|
+
console.log(`${this._stamp()} Telegram reachable again — resuming polling after outage.`);
|
|
299
|
+
try {
|
|
300
|
+
this.bot._polling = null;
|
|
301
|
+
this._pollStartedAt = Date.now();
|
|
302
|
+
await this.bot.startPolling({ restart: false });
|
|
303
|
+
} catch (e) {
|
|
304
|
+
this._onPollingHiccup("resume after outage failed to start");
|
|
305
|
+
return;
|
|
306
|
+
}
|
|
307
|
+
this._armHealthyCheck(this._pollStartedAt, "resume after outage completed but no getUpdates within 35s");
|
|
308
|
+
};
|
|
309
|
+
this._outageProbeTimer = setTimeout(tick, 30000);
|
|
196
310
|
}
|
|
197
311
|
|
|
198
312
|
// Telegram 409s a getUpdates call while ANOTHER one holds the token: a
|
package/package.json
CHANGED
|
@@ -65,7 +65,28 @@ assert.match(pkg.description, /provider-agnostic coding-agent harness/i);
|
|
|
65
65
|
// permanent message regardless of /verbose — only extended-thinking was gated.
|
|
66
66
|
// finalizeStreamPreview now deletes the settled preview when /verbose is off, so
|
|
67
67
|
// chat carries only the final answer; verbose ON still keeps the narration trail.
|
|
68
|
-
|
|
68
|
+
// 3.0.42 approved by Sumeet 2026-07-18 ("Yes please because it's restarting way
|
|
69
|
+
// too much"): Telegram poll-recovery fix that stops the restart storm. A wedge
|
|
70
|
+
// (half-open getUpdates after sleep/wake or NAT/IPv6 eviction) left the lib's
|
|
71
|
+
// REUSED _polling object with _abort stuck true, so the in-place restart revived
|
|
72
|
+
// a dead one-shot loop — the heal "succeeded" yet nothing polled, and only the
|
|
73
|
+
// 10-min process.exit backstop recovered. The heal now nulls _polling so the lib
|
|
74
|
+
// builds a fresh object (offset survives via the shared params), and the
|
|
75
|
+
// "healthy" reset requires a getUpdates that actually COMPLETED since the restart
|
|
76
|
+
// (else it re-arms a heal) instead of blindly declaring victory after 35s.
|
|
77
|
+
// 3.0.43 approved by Sumeet 2026-07-18 ("Yeh push"): reachability-gated self-exit
|
|
78
|
+
// + outage-pause backoff, so a host-wide network blip no longer triggers a
|
|
79
|
+
// reboot storm. v3.0.42 healed the in-loop wedge but STILL rebooted after 10min
|
|
80
|
+
// even when the whole host had lost the network (laptop sleep, pod egress
|
|
81
|
+
// starvation) — exiting into a dead network just relaunched into the same
|
|
82
|
+
// outage, spamming "Back online" every ~14min. The 10-min self-exit is now gated
|
|
83
|
+
// on a fresh HEAD probe to api.telegram.org actually SUCCEEDING (proves a genuine
|
|
84
|
+
// in-loop wedge, not a dead host); if the probe fails the adapter enters an
|
|
85
|
+
// outage pause — it STOPS polling (no 300ms lib reconnect spin) and re-probes
|
|
86
|
+
// every 30s, resuming cleanly the moment connectivity returns. Poll-health logs
|
|
87
|
+
// are timestamped. Confined to channels/telegram/adapter.js; new cases in
|
|
88
|
+
// test-telegram-poll-recovery.js cover the pause+resume and the gated exit.
|
|
89
|
+
assert.strictEqual(pkg.version, "3.0.43", "release version must remain unchanged without explicit approval");
|
|
69
90
|
for (const keyword of ["claude", "codex", "coding-agent", "provider-agnostic"]) {
|
|
70
91
|
assert.ok(pkg.keywords.includes(keyword), `package keyword missing: ${keyword}`);
|
|
71
92
|
}
|
|
@@ -190,6 +190,7 @@ async function testWedgeExitDefersWhileTurnInFlight() {
|
|
|
190
190
|
adapter._deferredExitLogAt = 0;
|
|
191
191
|
adapter._healBackoff = 0;
|
|
192
192
|
adapter._anyTurnActive = () => true; // a turn is running/queued
|
|
193
|
+
adapter._probeTelegramReachable = async () => true; // reachable ⇒ genuine wedge; defer path
|
|
193
194
|
adapter._agent = { destroy() { calls.push("destroy"); } };
|
|
194
195
|
adapter.bot = {
|
|
195
196
|
async stopPolling() { calls.push("stop"); },
|
|
@@ -217,8 +218,10 @@ async function testWedgeExitDefersWhileTurnInFlight() {
|
|
|
217
218
|
assert.ok(adapter._deferredExitLogAt > 0, "the deferral must be logged (throttle stamp set)");
|
|
218
219
|
}
|
|
219
220
|
|
|
220
|
-
// Once the process is idle (no turn running/queued)
|
|
221
|
-
//
|
|
221
|
+
// Once the process is idle (no turn running/queued) AND Telegram answers a fresh
|
|
222
|
+
// connection — proving this is a real in-loop poll wedge, not a host-network
|
|
223
|
+
// outage — the backstop exits for a clean restart exactly as before. The
|
|
224
|
+
// deferral only postpones, never cancels.
|
|
222
225
|
async function testWedgeExitFiresWhenIdle() {
|
|
223
226
|
let exitCode;
|
|
224
227
|
const adapter = Object.create(TelegramAdapter.prototype);
|
|
@@ -227,6 +230,7 @@ async function testWedgeExitFiresWhenIdle() {
|
|
|
227
230
|
adapter._wedgedSince = Date.now() - 11 * 60 * 1000; // wedged >10min
|
|
228
231
|
adapter._deferredExitLogAt = 0;
|
|
229
232
|
adapter._anyTurnActive = () => false; // idle
|
|
233
|
+
adapter._probeTelegramReachable = async () => true; // Telegram reachable ⇒ genuine poll wedge
|
|
230
234
|
adapter._agent = { destroy() {} };
|
|
231
235
|
adapter.bot = { async stopPolling() {}, async startPolling() {} };
|
|
232
236
|
|
|
@@ -244,6 +248,269 @@ async function testWedgeExitFiresWhenIdle() {
|
|
|
244
248
|
assert.strictEqual(exitCode, 1, "the idle wedge backstop must exit(1) for a clean restart");
|
|
245
249
|
}
|
|
246
250
|
|
|
251
|
+
// THE REBOOT-STORM regression, now resolved by an OUTAGE PAUSE. When the wedge is
|
|
252
|
+
// a host-wide network outage — every outbound connection failing together
|
|
253
|
+
// (Telegram polls + sends AND the Kazee/Spaces websockets) — a reboot can't help:
|
|
254
|
+
// the fresh process can't reach Telegram either and just relaunches into the same
|
|
255
|
+
// dead network, spamming "Back online". So even when idle, the backstop must NOT
|
|
256
|
+
// exit while a fresh connection to Telegram also fails. Instead it STOPS the poll
|
|
257
|
+
// loop entirely (no ~300ms getUpdates hammering that can trip a 429 when the link
|
|
258
|
+
// flickers back) and waits on a slow re-probe. Fails before the fix because the
|
|
259
|
+
// idle backstop exits unconditionally.
|
|
260
|
+
async function testWedgeExitPausesInsteadOfRebootingOnHostOutage() {
|
|
261
|
+
const calls = [];
|
|
262
|
+
let exited = false;
|
|
263
|
+
const adapter = Object.create(TelegramAdapter.prototype);
|
|
264
|
+
adapter._healTimer = null;
|
|
265
|
+
adapter._healthyTimer = null;
|
|
266
|
+
adapter._conflictSince = 0;
|
|
267
|
+
adapter._wedgedSince = Date.now() - 11 * 60 * 1000; // wedged >10min
|
|
268
|
+
adapter._deferredExitLogAt = 0;
|
|
269
|
+
adapter._healBackoff = 0;
|
|
270
|
+
adapter._pausedForOutage = false;
|
|
271
|
+
adapter._outageProbeTimer = null;
|
|
272
|
+
adapter._anyTurnActive = () => false; // idle
|
|
273
|
+
adapter._probeTelegramReachable = async () => false; // host can't reach Telegram at all
|
|
274
|
+
adapter._agent = { destroy() { calls.push("destroy"); } };
|
|
275
|
+
adapter.bot = {
|
|
276
|
+
async stopPolling() { calls.push("stop"); },
|
|
277
|
+
async startPolling() { calls.push("start"); },
|
|
278
|
+
};
|
|
279
|
+
|
|
280
|
+
const realExit = process.exit;
|
|
281
|
+
process.exit = () => { exited = true; throw new Error("process.exit must not be called during a host-network outage"); };
|
|
282
|
+
const realSetTimeout = global.setTimeout;
|
|
283
|
+
const realClearTimeout = global.clearTimeout;
|
|
284
|
+
global.setTimeout = (fn, ms) => { if (ms === 500) fn(); return { ms }; };
|
|
285
|
+
global.clearTimeout = () => {};
|
|
286
|
+
|
|
287
|
+
try {
|
|
288
|
+
await adapter._heal();
|
|
289
|
+
await new Promise((r) => realSetTimeout(r, 5)); // let the fire-and-forget stop settle
|
|
290
|
+
} finally {
|
|
291
|
+
process.exit = realExit;
|
|
292
|
+
global.setTimeout = realSetTimeout;
|
|
293
|
+
global.clearTimeout = realClearTimeout;
|
|
294
|
+
}
|
|
295
|
+
|
|
296
|
+
assert.strictEqual(exited, false, "a host-network outage must NOT trigger a reboot (the fresh process can't reach the network either)");
|
|
297
|
+
assert.strictEqual(adapter._pausedForOutage, true, "an unreachable host must pause polling instead of rebooting");
|
|
298
|
+
assert.ok(calls.includes("stop") && calls.includes("destroy"),
|
|
299
|
+
"entering the outage pause must stop the poll loop and sever the socket");
|
|
300
|
+
assert.ok(!calls.includes("start"),
|
|
301
|
+
"while the host is unreachable the bot must NOT keep reconnecting (no ~300ms getUpdates hammering)");
|
|
302
|
+
assert.ok(adapter._outageProbeTimer, "the pause must schedule a slow re-probe to resume when the network returns");
|
|
303
|
+
}
|
|
304
|
+
|
|
305
|
+
// Companion: once the network returns, the outage pause's slow re-probe must
|
|
306
|
+
// RESUME polling on a FRESH loop object (offset preserved via bot.options.polling
|
|
307
|
+
// params) and leave the paused state — the bot recovers on its own, no reboot.
|
|
308
|
+
async function testOutagePauseResumesPollingWhenNetworkReturns() {
|
|
309
|
+
const calls = [];
|
|
310
|
+
let reachable = false;
|
|
311
|
+
let tickFn;
|
|
312
|
+
const adapter = Object.create(TelegramAdapter.prototype);
|
|
313
|
+
adapter._healTimer = null;
|
|
314
|
+
adapter._healthyTimer = null;
|
|
315
|
+
adapter._wedgedSince = Date.now() - 2 * 60 * 1000;
|
|
316
|
+
adapter._healBackoff = 0;
|
|
317
|
+
adapter._deferredExitLogAt = 0;
|
|
318
|
+
adapter._pausedForOutage = false;
|
|
319
|
+
adapter._outageProbeTimer = null;
|
|
320
|
+
adapter._pollStartedAt = 0;
|
|
321
|
+
adapter._onPollingHiccup = () => {};
|
|
322
|
+
adapter._probeTelegramReachable = async () => reachable;
|
|
323
|
+
adapter._agent = { destroy() { calls.push("destroy"); } };
|
|
324
|
+
adapter.bot = {
|
|
325
|
+
_polling: { _abort: true, _lastUpdate: 0 },
|
|
326
|
+
async stopPolling() { calls.push("stop"); },
|
|
327
|
+
async startPolling() {
|
|
328
|
+
calls.push("start");
|
|
329
|
+
if (!this._polling) this._polling = { _abort: false, _lastUpdate: Date.now() };
|
|
330
|
+
},
|
|
331
|
+
};
|
|
332
|
+
|
|
333
|
+
const realSetTimeout = global.setTimeout;
|
|
334
|
+
const realClearTimeout = global.clearTimeout;
|
|
335
|
+
global.setTimeout = (fn, ms) => {
|
|
336
|
+
if (ms === 500 || ms === 5000) fn(); // stop-clean internals fire inline
|
|
337
|
+
else if (ms === 30000 && !tickFn) tickFn = fn; // capture the outage re-probe tick
|
|
338
|
+
return { ms };
|
|
339
|
+
};
|
|
340
|
+
global.clearTimeout = () => {};
|
|
341
|
+
|
|
342
|
+
try {
|
|
343
|
+
adapter._enterOutagePause();
|
|
344
|
+
await new Promise((r) => realSetTimeout(r, 5)); // let the fire-and-forget stop settle
|
|
345
|
+
assert.strictEqual(adapter._pausedForOutage, true, "an unreachable host pauses polling");
|
|
346
|
+
assert.ok(!calls.includes("start"), "no reconnecting while paused (network is down)");
|
|
347
|
+
assert.strictEqual(typeof tickFn, "function", "the pause must schedule a slow re-probe");
|
|
348
|
+
|
|
349
|
+
reachable = true;
|
|
350
|
+
adapter.bot._polling = { _abort: true, _lastUpdate: 0 }; // stale object present before resume
|
|
351
|
+
await tickFn();
|
|
352
|
+
assert.strictEqual(adapter._pausedForOutage, false, "a successful re-probe must leave the pause");
|
|
353
|
+
assert.ok(calls.includes("start"), "a successful re-probe must resume polling");
|
|
354
|
+
assert.strictEqual(adapter.bot._polling._abort, false, "resume must build a FRESH poll loop object (offset preserved)");
|
|
355
|
+
} finally {
|
|
356
|
+
global.setTimeout = realSetTimeout;
|
|
357
|
+
global.clearTimeout = realClearTimeout;
|
|
358
|
+
}
|
|
359
|
+
}
|
|
360
|
+
|
|
361
|
+
// THE RESTART STORM regression. A wedge (half-open getUpdates after sleep/wake
|
|
362
|
+
// or a NAT/IPv6 eviction) leaves the lib's REUSED _polling object with `_abort`
|
|
363
|
+
// stuck true: the hung request never settled within our stop cap, so stop()'s
|
|
364
|
+
// `.finally(() => _abort = false)` never ran. node-telegram-bot-api's
|
|
365
|
+
// startPolling reuses this.bot._polling when it's present, so restarting in
|
|
366
|
+
// place revives the SAME poisoned object — its loop fires one getUpdates and
|
|
367
|
+
// dies in .finally (sees _abort===true ⇒ no reschedule). A silent dead loop:
|
|
368
|
+
// the heal "succeeds" (no throw) yet nothing polls, so only the 10-min backstop's
|
|
369
|
+
// process.exit ever recovered (the reboot storm the user saw). The heal must
|
|
370
|
+
// null _polling so the lib constructs a FRESH object with _abort=false; the
|
|
371
|
+
// offset survives via the shared bot.options.polling params. Fails before the
|
|
372
|
+
// fix because the poisoned object is revived in place.
|
|
373
|
+
async function testHealNullsPoisonedPollingSoLibRebuildsFresh() {
|
|
374
|
+
const ancient = Date.now() - 5 * 60 * 1000;
|
|
375
|
+
const poisoned = { _abort: true, _lastUpdate: ancient };
|
|
376
|
+
const adapter = Object.create(TelegramAdapter.prototype);
|
|
377
|
+
adapter._healTimer = null;
|
|
378
|
+
adapter._healthyTimer = null;
|
|
379
|
+
adapter._wedgedSince = Date.now() - 60000;
|
|
380
|
+
adapter._healBackoff = 0;
|
|
381
|
+
adapter._pollStartedAt = ancient;
|
|
382
|
+
adapter._onPollingHiccup = () => {};
|
|
383
|
+
adapter._probeTelegramReachable = async () => true; // reachable ⇒ normal in-place heal path
|
|
384
|
+
adapter._agent = { destroy() {} };
|
|
385
|
+
adapter.bot = {
|
|
386
|
+
_polling: poisoned,
|
|
387
|
+
async stopPolling() { /* wedge: the hung request never settles, _abort stays true */ },
|
|
388
|
+
async startPolling() {
|
|
389
|
+
// Faithful to the lib: reuse _polling if present, else construct fresh.
|
|
390
|
+
if (!this._polling) this._polling = { _abort: false, _lastUpdate: Date.now() };
|
|
391
|
+
},
|
|
392
|
+
};
|
|
393
|
+
|
|
394
|
+
const realSetTimeout = global.setTimeout;
|
|
395
|
+
const realClearTimeout = global.clearTimeout;
|
|
396
|
+
global.setTimeout = (fn, ms) => { if (ms === 500 || ms === 5000) fn(); return { ms }; };
|
|
397
|
+
global.clearTimeout = () => {};
|
|
398
|
+
try {
|
|
399
|
+
await adapter._heal();
|
|
400
|
+
} finally {
|
|
401
|
+
global.setTimeout = realSetTimeout;
|
|
402
|
+
global.clearTimeout = realClearTimeout;
|
|
403
|
+
}
|
|
404
|
+
|
|
405
|
+
assert.notStrictEqual(adapter.bot._polling, poisoned,
|
|
406
|
+
"heal must discard the poisoned _polling object, not revive it in place");
|
|
407
|
+
assert.strictEqual(adapter.bot._polling._abort, false,
|
|
408
|
+
"the rebuilt poller must start with _abort=false so its loop reschedules");
|
|
409
|
+
assert.ok(adapter.bot._polling._lastUpdate >= adapter._pollStartedAt,
|
|
410
|
+
"the fresh poller must be able to complete a getUpdates (a live loop, not a dead one-shot)");
|
|
411
|
+
}
|
|
412
|
+
|
|
413
|
+
// HONEST HEALTH SIGNAL regression. "No throw for 35s" is NOT recovery — a dead
|
|
414
|
+
// one-shot loop throws nothing. The old blind healthy-timer reset _wedgedSince
|
|
415
|
+
// unconditionally, so a restart that never completed a single getUpdates was
|
|
416
|
+
// declared healthy, hiding the wedge until the 10-min backstop rebooted. The
|
|
417
|
+
// check must require a getUpdates COMPLETED since the restart; if none did, it
|
|
418
|
+
// must re-arm a heal instead of declaring victory. Fails before the fix because
|
|
419
|
+
// the blind timer clears the wedge with no completed poll.
|
|
420
|
+
async function testHealthySignalReArmsWhenNoPollCompleted() {
|
|
421
|
+
const adapter = Object.create(TelegramAdapter.prototype);
|
|
422
|
+
adapter._healTimer = null;
|
|
423
|
+
adapter._healthyTimer = null;
|
|
424
|
+
adapter._wedgedSince = Date.now() - 60000;
|
|
425
|
+
adapter._healBackoff = 2;
|
|
426
|
+
adapter._deferredExitLogAt = 0;
|
|
427
|
+
adapter._pollStartedAt = 0;
|
|
428
|
+
adapter._probeTelegramReachable = async () => true; // reachable ⇒ normal in-place heal path
|
|
429
|
+
adapter._agent = { destroy() {} };
|
|
430
|
+
adapter.bot = {
|
|
431
|
+
_polling: { _abort: true, _lastUpdate: 0 },
|
|
432
|
+
async stopPolling() {},
|
|
433
|
+
async startPolling() {
|
|
434
|
+
// Models a restart whose loop never completes a getUpdates: _lastUpdate
|
|
435
|
+
// stays 0 (pre-fix this is the revived poisoned object; post-fix a fresh
|
|
436
|
+
// object that simply hasn't succeeded yet).
|
|
437
|
+
if (!this._polling) this._polling = { _abort: false, _lastUpdate: 0 };
|
|
438
|
+
},
|
|
439
|
+
};
|
|
440
|
+
let healthyFn;
|
|
441
|
+
const hiccups = [];
|
|
442
|
+
adapter._onPollingHiccup = (m) => hiccups.push(m);
|
|
443
|
+
|
|
444
|
+
const realSetTimeout = global.setTimeout;
|
|
445
|
+
const realClearTimeout = global.clearTimeout;
|
|
446
|
+
global.setTimeout = (fn, ms) => {
|
|
447
|
+
if (ms === 500 || ms === 5000) fn();
|
|
448
|
+
else if (ms >= 30000) healthyFn = fn; // capture the healthy check (30s pre-fix, 35s post)
|
|
449
|
+
return { ms };
|
|
450
|
+
};
|
|
451
|
+
global.clearTimeout = () => {};
|
|
452
|
+
try {
|
|
453
|
+
await adapter._heal();
|
|
454
|
+
assert.strictEqual(typeof healthyFn, "function", "heal must arm a healthy check");
|
|
455
|
+
healthyFn();
|
|
456
|
+
} finally {
|
|
457
|
+
global.setTimeout = realSetTimeout;
|
|
458
|
+
global.clearTimeout = realClearTimeout;
|
|
459
|
+
}
|
|
460
|
+
|
|
461
|
+
assert.ok(adapter._wedgedSince > 0,
|
|
462
|
+
"a restart with no completed getUpdates must NOT clear the wedge");
|
|
463
|
+
assert.match(hiccups[0] || "", /no getUpdates/i,
|
|
464
|
+
"an unrecovered restart must re-arm a heal, not declare victory");
|
|
465
|
+
}
|
|
466
|
+
|
|
467
|
+
// Companion happy-path guard: once a getUpdates DOES complete after the restart,
|
|
468
|
+
// the same healthy check clears the wedge and resets backoff (and must NOT
|
|
469
|
+
// re-arm a heal). Passes before and after the fix — it guards the fix from
|
|
470
|
+
// over-correcting into a bot that never declares itself recovered.
|
|
471
|
+
async function testHealthySignalClearsWedgeOnceAPollCompletes() {
|
|
472
|
+
const adapter = Object.create(TelegramAdapter.prototype);
|
|
473
|
+
adapter._healTimer = null;
|
|
474
|
+
adapter._healthyTimer = null;
|
|
475
|
+
adapter._wedgedSince = Date.now() - 60000;
|
|
476
|
+
adapter._healBackoff = 3;
|
|
477
|
+
adapter._deferredExitLogAt = 12345;
|
|
478
|
+
adapter._pollStartedAt = 0;
|
|
479
|
+
adapter._probeTelegramReachable = async () => true; // reachable ⇒ normal in-place heal path
|
|
480
|
+
adapter._agent = { destroy() {} };
|
|
481
|
+
adapter.bot = {
|
|
482
|
+
_polling: { _abort: true, _lastUpdate: 0 },
|
|
483
|
+
async stopPolling() {},
|
|
484
|
+
async startPolling() {
|
|
485
|
+
// A fresh, healthy poller that completes a getUpdates just after restart.
|
|
486
|
+
if (!this._polling) this._polling = { _abort: false, _lastUpdate: Date.now() + 1 };
|
|
487
|
+
},
|
|
488
|
+
};
|
|
489
|
+
let healthyFn;
|
|
490
|
+
const hiccups = [];
|
|
491
|
+
adapter._onPollingHiccup = (m) => hiccups.push(m);
|
|
492
|
+
|
|
493
|
+
const realSetTimeout = global.setTimeout;
|
|
494
|
+
const realClearTimeout = global.clearTimeout;
|
|
495
|
+
global.setTimeout = (fn, ms) => {
|
|
496
|
+
if (ms === 500 || ms === 5000) fn();
|
|
497
|
+
else if (ms >= 30000) healthyFn = fn;
|
|
498
|
+
return { ms };
|
|
499
|
+
};
|
|
500
|
+
global.clearTimeout = () => {};
|
|
501
|
+
try {
|
|
502
|
+
await adapter._heal();
|
|
503
|
+
healthyFn();
|
|
504
|
+
} finally {
|
|
505
|
+
global.setTimeout = realSetTimeout;
|
|
506
|
+
global.clearTimeout = realClearTimeout;
|
|
507
|
+
}
|
|
508
|
+
|
|
509
|
+
assert.strictEqual(adapter._wedgedSince, 0, "a completed getUpdates since restart must clear the wedge");
|
|
510
|
+
assert.strictEqual(adapter._healBackoff, 0, "recovery resets heal backoff");
|
|
511
|
+
assert.deepStrictEqual(hiccups, [], "a genuinely recovered restart must NOT re-arm a heal");
|
|
512
|
+
}
|
|
513
|
+
|
|
247
514
|
(async () => {
|
|
248
515
|
await testStartDoesNotWaitForFirstLongPollAndArmsStallWatchdog();
|
|
249
516
|
await testHealStopsPlainlyDestroysAgentAndRestartsIdle();
|
|
@@ -251,6 +518,11 @@ async function testWedgeExitFiresWhenIdle() {
|
|
|
251
518
|
await test409ConflictSeversGhostSocketAndRestartsIdle();
|
|
252
519
|
await testWedgeExitDefersWhileTurnInFlight();
|
|
253
520
|
await testWedgeExitFiresWhenIdle();
|
|
521
|
+
await testWedgeExitPausesInsteadOfRebootingOnHostOutage();
|
|
522
|
+
await testOutagePauseResumesPollingWhenNetworkReturns();
|
|
523
|
+
await testHealNullsPoisonedPollingSoLibRebuildsFresh();
|
|
524
|
+
await testHealthySignalReArmsWhenNoPollCompleted();
|
|
525
|
+
await testHealthySignalClearsWedgeOnceAPollCompletes();
|
|
254
526
|
console.log("telegram poll recovery OK");
|
|
255
527
|
})().catch((err) => {
|
|
256
528
|
console.error(err.stack || err);
|