@torrent-tv/proxy 2.58.2 → 2.59.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +12 -0
- package/assets/diagnostics/sctpstate.gdb +29 -0
- package/bin/cli.js +19 -1
- package/package.json +1 -1
- package/services/data-channel-handler.js +36 -4
- package/services/delivery-probe.js +110 -4
- package/services/memory-report.js +166 -0
- package/services/piece-store/shared-piece-store.js +791 -717
- package/services/torrent-worker/worker.js +5 -0
- package/services/usrsctp-state.js +146 -0
- package/test/memory-budget.test.js +69 -0
- package/test/probe-wedge-certainty.test.js +60 -0
- package/test/usrsctp-state.test.js +115 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,15 @@
|
|
|
1
|
+
## 2.59.0
|
|
2
|
+
|
|
3
|
+
- **New**: a wedge (roadmap item 11) is now declared, and evidence gathered for it automatically, even when the small-message shape means nothing ever queues. The only trigger that existed (`wedgeIsCertain`) requires a nonzero channel queue — confirmed 2026-08-28 by reading the code, not assuming it from the note: `bufferedAmount()` reads 0 the instant our bytes reach usrsctp, whatever usrsctp does with them next, so a wedge carrying only probes and control messages never set it and the last real episode's ring was never saved automatically. The delivery probe already computes a queue-independent verdict (`association-stopped`); it is now wired to the same evidence-gathering, gated on its OWN certainty rule rather than the raw verdict — a connection can sit behind by a bounded, non-growing amount for minutes (measured the same day, a backgrounded tab: gap held at 6-7 probes for 95+ seconds while `seen` kept climbing right along with `sent`) without anything being wrong. What a true wedge shows instead, checked against a session already known to be one: `seen` frozen at one value for over a minute while `sent` climbs unbounded. `probeWedgeIsCertain` asks whether the counter has stopped moving for longer than this connection's own history says a healthy gap ever takes — the same shape `wedgeIsCertain` already uses, applied to the probe's own counter.
|
|
4
|
+
- **New**: `usrsctp`'s live association state (peer receive window, pending data, retransmission timeout, congestion window) can now be read on either wedge declaration, automatically — `node_datachannel.node` ships unstripped, so the read is a gdb attach against the running process, no rebuild. The walk and its healthy baseline are `research/session-2026-08-27-28-freeze-onset-and-sessions.md`, section 1; the script that performs it ships in the package (`assets/diagnostics/sctpstate.gdb`) instead of surviving only as long as someone remembers to copy it back onto a host after a container is recreated. Nine episodes of this item have lacked exactly this reading.
|
|
5
|
+
|
|
6
|
+
## 2.58.3
|
|
7
|
+
|
|
8
|
+
- **New**: the proxy says what it is holding, once a minute — resident memory, heap, external and array buffers, the torrent stores in BYTES, and what the machine has left. It was killed on 2026-08-28 by the kernel's own out-of-memory killer at 2.4 GB resident (`exit code 137`, no core dump, `Out of memory: Killed process ... anon-rss: 2422628kB`) and the log had never recorded a single figure about memory. There was one final reading, taken by the kernel, and no series leading to it.
|
|
9
|
+
- **Fix**: the torrent stores share ONE budget instead of each taking its own. It was per torrent, so two torrents meant two of it, and nothing anywhere asked what the process as a whole was holding. On the film the proxy died under, one store had taken the full 504 MB.
|
|
10
|
+
- **Fix**: that budget is a share of `MemAvailable` rather than of `os.freemem()`. On Linux the second counts only the pages free at that instant, while the kernel deliberately keeps that number low by filling the rest with reclaimable cache — so the share it produced had little to do with what an allocation could actually obtain. The kernel publishes the estimate; we read it.
|
|
11
|
+
- **New**: the piece-store line reports megabytes beside its piece count. The count alone says nothing without the piece size, and the piece size differs per torrent: on that film, "63" meant 504 MB.
|
|
12
|
+
|
|
1
13
|
## 2.58.2
|
|
2
14
|
|
|
3
15
|
- **Fix**: `utp-native` moves to 2.5.3-ttv.8, which removes the whole crash family rather than another instance of it. Nine deaths in a fortnight had one shape — libuv holding a pointer into memory that had gone — because the structs carrying `uv_udp_t`, `uv_timer_t` and `uv_udp_send_t` were allocated by JavaScript as `Buffer.alloc(sizeof(...))`, putting them under the garbage collector while libuv's rule is that they must live until the close or completion callback has run. Every earlier fix reconciled the two owners with a rule and the next release found another way through. The module now allocates and frees that memory itself, at the point libuv has provably finished; JavaScript holds only a token, and freeing points the token at nothing so a late call does nothing instead of faulting. On the environment's own teardown nothing is freed at all — a deliberate leak while the process ends beats touching napi as it goes. Checked on the target: 77 tests pass, and 60 create/serve/destroy cycles leave memory flat.
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
set pagination off
|
|
2
|
+
set confirm off
|
|
3
|
+
set $base = (unsigned long) &system_base_info
|
|
4
|
+
set $hash = *(unsigned long *)($base + 0)
|
|
5
|
+
set $mask = *(unsigned long *)($base + 8)
|
|
6
|
+
printf "asochash=%p mask=%lu\n", $hash, $mask
|
|
7
|
+
set $i = 0
|
|
8
|
+
set $stcb = 0
|
|
9
|
+
while $i <= $mask && $stcb == 0
|
|
10
|
+
set $head = *(unsigned long *)($hash + $i * 8)
|
|
11
|
+
if $head != 0
|
|
12
|
+
set $stcb = $head
|
|
13
|
+
printf "bucket %lu -> stcb=%p\n", $i, $stcb
|
|
14
|
+
end
|
|
15
|
+
set $i = $i + 1
|
|
16
|
+
end
|
|
17
|
+
if $stcb == 0
|
|
18
|
+
printf "no association found (no viewer connected?)\n"
|
|
19
|
+
else
|
|
20
|
+
set $sock = *(unsigned long *)($stcb + 0)
|
|
21
|
+
printf "socket=%p asoc=%p\n", $sock, $stcb + 88
|
|
22
|
+
set $buf = (unsigned long) malloc(512)
|
|
23
|
+
set $lenp = (unsigned long) malloc(8)
|
|
24
|
+
set *(int *)$lenp = 512
|
|
25
|
+
set $rc = (int) usrsctp_getsockopt($sock, 132, 256, $buf, $lenp)
|
|
26
|
+
printf "getsockopt rc=%d len=%d\n", $rc, *(int *)$lenp
|
|
27
|
+
printf "state=%d rwnd=%u unackdata=%u penddata=%u instrms=%u outstrms=%u fragpoint=%u\n", *(int *)($buf+4), *(unsigned int *)($buf+8), *(unsigned short *)($buf+12), *(unsigned short *)($buf+14), *(unsigned short *)($buf+16), *(unsigned short *)($buf+18), *(unsigned int *)($buf+20)
|
|
28
|
+
printf "primary: state=%d cwnd=%u srtt=%u rto=%u mtu=%u\n", *(int *)($buf+24+4+128), *(unsigned int *)($buf+24+4+128+4), *(unsigned int *)($buf+24+4+128+8), *(unsigned int *)($buf+24+4+128+12), *(unsigned int *)($buf+24+4+128+16)
|
|
29
|
+
end
|
package/bin/cli.js
CHANGED
|
@@ -25,6 +25,8 @@ import { createWebRtcManager } from "../services/webrtc-manager.js";
|
|
|
25
25
|
import { createDataChannelHandler } from "../services/data-channel-handler.js";
|
|
26
26
|
import { pruneCoreDumps } from "../services/core-dumps.js";
|
|
27
27
|
import { adoptOrphanRingFiles, createPacketWitness, pruneWitnessCaptures } from "../services/packet-witness.js";
|
|
28
|
+
import { createUsrsctpStateReader } from "../services/usrsctp-state.js";
|
|
29
|
+
import { startMemoryReport } from "../services/memory-report.js";
|
|
28
30
|
import { collectHealthMetrics } from "../services/health-collector.js";
|
|
29
31
|
import { createPortMapper } from "../services/port-mapper.js";
|
|
30
32
|
import { classifyNat } from "../services/nat-classifier.js";
|
|
@@ -187,6 +189,9 @@ let webRtcManager = null;
|
|
|
187
189
|
*/
|
|
188
190
|
let packetWitness = null;
|
|
189
191
|
|
|
192
|
+
/** @type {ReturnType<typeof createUsrsctpStateReader> | null} */
|
|
193
|
+
let usrsctpStateReader = null;
|
|
194
|
+
|
|
190
195
|
/** @type {ReturnType<typeof createDataChannelHandler> | null} */
|
|
191
196
|
let dataChannelHandler = null;
|
|
192
197
|
|
|
@@ -334,6 +339,18 @@ try {
|
|
|
334
339
|
// the pruner recognises BEFORE anything starts a new ring over them.
|
|
335
340
|
void adoptOrphanRingFiles(packetWitness.dir).then(() => pruneWitnessCaptures(packetWitness.dir));
|
|
336
341
|
|
|
342
|
+
// Reads usrsctp's own association state via gdb the moment a wedge is
|
|
343
|
+
// declared (roadmap item 11) — no source rebuild, the module ships
|
|
344
|
+
// unstripped. A host without gdb just never gets a reading, the same way a
|
|
345
|
+
// host without tcpdump never gets a packet capture.
|
|
346
|
+
usrsctpStateReader = createUsrsctpStateReader({ log: (message) => logger.info(message) });
|
|
347
|
+
|
|
348
|
+
// What this process holds, once a minute. The kernel killed the proxy on
|
|
349
|
+
// 2026-08-28 at 2.4 GB resident and the log had never said a word about
|
|
350
|
+
// memory, so the growth that ended in that kill has no shape in any record we
|
|
351
|
+
// keep. RSS is the figure the OOM killer reads, so RSS is the figure to say.
|
|
352
|
+
startMemoryReport({ log: (message) => logger.info(message) });
|
|
353
|
+
|
|
337
354
|
logger.info(`Starting @torrent-tv/proxy v${PROXY_VERSION}`);
|
|
338
355
|
logger.info(`Local stream endpoint: http://${bindHost}:${actualPort}/stream`);
|
|
339
356
|
logger.info(`Advertised direct URL: ${directBaseUrl}`);
|
|
@@ -457,7 +474,8 @@ try {
|
|
|
457
474
|
getTransportSnapshot: (sessionId) => webRtcManager?.getTransportSnapshot(sessionId) ?? null,
|
|
458
475
|
// Records the wire when a queue stays wedged — how the rare one-way
|
|
459
476
|
// transmit death (roadmap item 10, 2026-08-24) gets its evidence.
|
|
460
|
-
witness: packetWitness
|
|
477
|
+
witness: packetWitness,
|
|
478
|
+
usrsctpState: usrsctpStateReader
|
|
461
479
|
});
|
|
462
480
|
|
|
463
481
|
webRtcManager = createWebRtcManager({
|
package/package.json
CHANGED
|
@@ -159,7 +159,7 @@ export function wedgeIsCertain({ queuedBytes, bytesPerSecond, flatForMs, longest
|
|
|
159
159
|
* @param {DataChannel} channel
|
|
160
160
|
* @returns {() => void} Stops the watch.
|
|
161
161
|
*/
|
|
162
|
-
function makeSendQueueWatcher({ log, getTransportSnapshot, witness }) {
|
|
162
|
+
function makeSendQueueWatcher({ log, getTransportSnapshot, witness, usrsctpState }) {
|
|
163
163
|
// Every channel of one connection reads the SAME transport counters — the
|
|
164
164
|
// snapshot describes the peer connection, not the channel — so the heartbeat
|
|
165
165
|
// belongs to the connection and is printed once for it. Printed per channel
|
|
@@ -479,6 +479,14 @@ function makeSendQueueWatcher({ log, getTransportSnapshot, witness }) {
|
|
|
479
479
|
connection.captureStarted = false;
|
|
480
480
|
}
|
|
481
481
|
}
|
|
482
|
+
if (usrsctpState && verdict.certain && peerStillSending) {
|
|
483
|
+
// Its own single-flight and cooldown, independent of the witness's —
|
|
484
|
+
// one gdb attach per wedge is enough, and a refusal here (already
|
|
485
|
+
// read, cooling down) costs nothing to retry on the next tick.
|
|
486
|
+
usrsctpState.maybeRead(
|
|
487
|
+
`send queue stuck ${queued}B, accepted-byte counter unmoved ${Math.round(flatForMs / 1000)}s`
|
|
488
|
+
);
|
|
489
|
+
}
|
|
482
490
|
}, SEND_QUEUE_SAMPLE_MS);
|
|
483
491
|
|
|
484
492
|
if (typeof timer.unref === "function") {
|
|
@@ -526,7 +534,18 @@ export function encodeFrame(idBytes, bytes, done) {
|
|
|
526
534
|
return frame;
|
|
527
535
|
}
|
|
528
536
|
|
|
529
|
-
export function createDataChannelHandler({
|
|
537
|
+
export function createDataChannelHandler({
|
|
538
|
+
proxyPort,
|
|
539
|
+
onLog,
|
|
540
|
+
getTransportSnapshot,
|
|
541
|
+
sourceRegistry,
|
|
542
|
+
witness,
|
|
543
|
+
// Reads usrsctp's own association state (services/usrsctp-state.js) the
|
|
544
|
+
// moment a wedge is declared, from either detector below. Optional: a host
|
|
545
|
+
// without gdb simply never gets a reading, same as the witness without
|
|
546
|
+
// tcpdump.
|
|
547
|
+
usrsctpState
|
|
548
|
+
}) {
|
|
530
549
|
/**
|
|
531
550
|
* Channels currently interested in one file's subtitle cues, keyed by
|
|
532
551
|
* `sourceKey:fileIndex`. Populated the moment a browser asks for an
|
|
@@ -608,13 +627,26 @@ export function createDataChannelHandler({ proxyPort, onLog, getTransportSnapsho
|
|
|
608
627
|
const { watchSendQueue, readDelivery } = makeSendQueueWatcher({
|
|
609
628
|
log: (message) => log(message),
|
|
610
629
|
getTransportSnapshot,
|
|
611
|
-
witness
|
|
630
|
+
witness,
|
|
631
|
+
usrsctpState
|
|
612
632
|
});
|
|
613
633
|
// Numbered probes on every channel, and the browser's echo of what it saw.
|
|
614
634
|
// The proxy's own counters cannot say whether bytes it handed to usrsctp were
|
|
615
635
|
// ever put on the wire; the far end can, and it keeps answering throughout a
|
|
616
636
|
// freeze. See services/delivery-probe.js.
|
|
617
|
-
|
|
637
|
+
//
|
|
638
|
+
// Also the ONLY wedge signal that does not require a nonzero channel queue —
|
|
639
|
+
// `wedgeIsCertain` above needs `queuedBytes > 0`, which small, infrequent
|
|
640
|
+
// traffic never produces (confirmed 2026-08-28: the 2026-08-27 episode's
|
|
641
|
+
// ring was never saved automatically for exactly this reason). So this is
|
|
642
|
+
// wired to the same evidence-gathering as the queue watcher.
|
|
643
|
+
const deliveryProbe = createDeliveryProbe({
|
|
644
|
+
log: (message) => log(message),
|
|
645
|
+
readDelivery,
|
|
646
|
+
getTransportSnapshot,
|
|
647
|
+
witness,
|
|
648
|
+
usrsctpState
|
|
649
|
+
});
|
|
618
650
|
|
|
619
651
|
/**
|
|
620
652
|
* @param {string} message
|
|
@@ -106,6 +106,32 @@ const ECHO_STALE_FALLBACK_MS = 5_000;
|
|
|
106
106
|
/** How often the probe state is written to the log while nothing changes. */
|
|
107
107
|
const REPORT_INTERVAL_MS = 5_000;
|
|
108
108
|
|
|
109
|
+
/**
|
|
110
|
+
* Whether the `seen` counter has stopped advancing for longer than this
|
|
111
|
+
* connection's own history says a healthy gap between two advances ever
|
|
112
|
+
* takes.
|
|
113
|
+
*
|
|
114
|
+
* The `association-stopped` verdict alone is not enough to act on: a
|
|
115
|
+
* connection can sit BEHIND by a bounded, roughly constant amount for
|
|
116
|
+
* minutes (measured 2026-08-28, session on a backgrounded tab — gap held at
|
|
117
|
+
* 6-7 probes for 95+ seconds while `seen` kept climbing right along with
|
|
118
|
+
* `sent`) without anything being wrong. What a true wedge shows instead,
|
|
119
|
+
* measured the same day against a session already known to be one
|
|
120
|
+
* (`d85ae4f5`): `seen` FROZEN at one value for over a minute while `sent`
|
|
121
|
+
* climbs unbounded. So the question is not "is there a gap" but "has the
|
|
122
|
+
* highest-seen number stopped moving at all, for longer than it has ever
|
|
123
|
+
* legitimately taken this connection to report an advance" — the same shape
|
|
124
|
+
* as {@link wedgeIsCertain} in `data-channel-handler.js`, applied to the
|
|
125
|
+
* probe's own counter instead of the transport's byte counter.
|
|
126
|
+
*
|
|
127
|
+
* @param {{ stuckForMs: number, longestHealthySeenGapMs: number, intervalMs?: number }} state
|
|
128
|
+
* @returns {{ certain: boolean, needMs: number }}
|
|
129
|
+
*/
|
|
130
|
+
export function probeWedgeIsCertain({ stuckForMs, longestHealthySeenGapMs, intervalMs = PROBE_INTERVAL_MS }) {
|
|
131
|
+
const needMs = Math.max(longestHealthySeenGapMs, intervalMs);
|
|
132
|
+
return { certain: stuckForMs >= needMs, needMs };
|
|
133
|
+
}
|
|
134
|
+
|
|
109
135
|
/**
|
|
110
136
|
* One connection's probe state.
|
|
111
137
|
*
|
|
@@ -119,6 +145,9 @@ const REPORT_INTERVAL_MS = 5_000;
|
|
|
119
145
|
* @property {number} echoes - How many echoes have arrived.
|
|
120
146
|
* @property {string} verdict - Last verdict reported, so a change is logged at once.
|
|
121
147
|
* @property {number} reportedAt - When the state was last written to the log.
|
|
148
|
+
* @property {number} lastSeenAdvanceAt - When any label's `seen` value last increased (0 = never yet).
|
|
149
|
+
* @property {number} longestHealthySeenGapMs - The longest gap between two advances this connection has shown while not flagged as a wedge.
|
|
150
|
+
* @property {boolean} probeCaptureStarted - One evidence-gathering attempt per wedge; reset once `seen` advances again.
|
|
122
151
|
* @property {ReturnType<typeof setInterval> | null} timer
|
|
123
152
|
*/
|
|
124
153
|
|
|
@@ -204,6 +233,11 @@ export function readProbeState(state) {
|
|
|
204
233
|
* @param {Object} options
|
|
205
234
|
* @param {(message: string) => void} options.log
|
|
206
235
|
* @param {number} [options.intervalMs]
|
|
236
|
+
* @param {(sessionId: string) => object | null} [options.getTransportSnapshot]
|
|
237
|
+
* Needed only to hand the witness a remote endpoint when this probe is the
|
|
238
|
+
* one declaring a wedge.
|
|
239
|
+
* @param {{ maybeCapture: (trigger: object) => boolean }} [options.witness]
|
|
240
|
+
* @param {{ maybeRead: (reasonText: string) => boolean }} [options.usrsctpState]
|
|
207
241
|
* @returns {{
|
|
208
242
|
* attach: (sessionId: string, tag: string, label: string, channel: import('node-datachannel').DataChannel) => void,
|
|
209
243
|
* detach: (sessionId: string, channel: import('node-datachannel').DataChannel) => void,
|
|
@@ -211,7 +245,14 @@ export function readProbeState(state) {
|
|
|
211
245
|
* dispose: () => void
|
|
212
246
|
* }}
|
|
213
247
|
*/
|
|
214
|
-
export function createDeliveryProbe({
|
|
248
|
+
export function createDeliveryProbe({
|
|
249
|
+
log,
|
|
250
|
+
intervalMs = PROBE_INTERVAL_MS,
|
|
251
|
+
readDelivery,
|
|
252
|
+
getTransportSnapshot,
|
|
253
|
+
witness,
|
|
254
|
+
usrsctpState
|
|
255
|
+
}) {
|
|
215
256
|
/** @type {Map<string, ProbeConnection>} */
|
|
216
257
|
const connections = new Map();
|
|
217
258
|
|
|
@@ -284,6 +325,47 @@ export function createDeliveryProbe({ log, intervalMs = PROBE_INTERVAL_MS, readD
|
|
|
284
325
|
connection.reportedAt = now;
|
|
285
326
|
log(`[dc-probe] ${connection.tag} ${verdict} — ${detail} at=${new Date(now).toISOString()}`);
|
|
286
327
|
}
|
|
328
|
+
|
|
329
|
+
// `association-stopped` alone is not certainty — see probeWedgeIsCertain.
|
|
330
|
+
// A connection that is merely lagging by a bounded amount reaches this
|
|
331
|
+
// verdict too (a hidden tab's own echo cadence, measured 2026-08-28), and
|
|
332
|
+
// `seen` keeps advancing right along with it. Only a `seen` value that has
|
|
333
|
+
// stopped moving ENTIRELY, for longer than this connection has ever shown
|
|
334
|
+
// as a legitimate gap, is the wedge this exists to catch.
|
|
335
|
+
const stuckForMs = connection.lastSeenAdvanceAt === 0 ? 0 : now - connection.lastSeenAdvanceAt;
|
|
336
|
+
if (verdict === "association-stopped") {
|
|
337
|
+
const { certain, needMs } = probeWedgeIsCertain({
|
|
338
|
+
stuckForMs,
|
|
339
|
+
longestHealthySeenGapMs: connection.longestHealthySeenGapMs
|
|
340
|
+
});
|
|
341
|
+
if (certain && !connection.probeCaptureStarted) {
|
|
342
|
+
connection.probeCaptureStarted = true;
|
|
343
|
+
const reasonText =
|
|
344
|
+
`probe seen-counter unmoved ${Math.round(stuckForMs / 1000)}s against the ` +
|
|
345
|
+
`${(needMs / 1000).toFixed(1)}s this connection's own history says is legitimate`;
|
|
346
|
+
if (witness) {
|
|
347
|
+
const snapshot = getTransportSnapshot?.(connection.id) ?? null;
|
|
348
|
+
const started = witness.maybeCapture({
|
|
349
|
+
sessionId: connection.id,
|
|
350
|
+
tag: connection.tag,
|
|
351
|
+
label: "probe",
|
|
352
|
+
remote: snapshot?.remote ?? null,
|
|
353
|
+
queuedBytes: 0,
|
|
354
|
+
stuckForMs
|
|
355
|
+
});
|
|
356
|
+
if (!started) {
|
|
357
|
+
connection.probeCaptureStarted = false;
|
|
358
|
+
}
|
|
359
|
+
}
|
|
360
|
+
if (usrsctpState) {
|
|
361
|
+
usrsctpState.maybeRead(reasonText);
|
|
362
|
+
}
|
|
363
|
+
}
|
|
364
|
+
} else {
|
|
365
|
+
// Not association-stopped any more: whatever was flagged has cleared,
|
|
366
|
+
// and a later wedge on the same connection deserves its own attempt.
|
|
367
|
+
connection.probeCaptureStarted = false;
|
|
368
|
+
}
|
|
287
369
|
}
|
|
288
370
|
|
|
289
371
|
return {
|
|
@@ -305,6 +387,9 @@ export function createDeliveryProbe({ log, intervalMs = PROBE_INTERVAL_MS, readD
|
|
|
305
387
|
echoes: 0,
|
|
306
388
|
verdict: "",
|
|
307
389
|
reportedAt: 0,
|
|
390
|
+
lastSeenAdvanceAt: 0,
|
|
391
|
+
longestHealthySeenGapMs: 0,
|
|
392
|
+
probeCaptureStarted: false,
|
|
308
393
|
timer: null
|
|
309
394
|
};
|
|
310
395
|
connections.set(sessionId, connection);
|
|
@@ -342,15 +427,36 @@ export function createDeliveryProbe({ log, intervalMs = PROBE_INTERVAL_MS, readD
|
|
|
342
427
|
if (!connection || !echo || typeof echo !== "object") {
|
|
343
428
|
return;
|
|
344
429
|
}
|
|
430
|
+
const now = Date.now();
|
|
345
431
|
const seen = echo.seen;
|
|
346
432
|
if (seen && typeof seen === "object") {
|
|
433
|
+
let advanced = false;
|
|
347
434
|
for (const [label, value] of Object.entries(seen)) {
|
|
348
|
-
if (Number.isInteger(value)) {
|
|
349
|
-
|
|
435
|
+
if (!Number.isInteger(value)) {
|
|
436
|
+
continue;
|
|
350
437
|
}
|
|
438
|
+
const previous = connection.seen.get(label);
|
|
439
|
+
if (!Number.isInteger(previous) || value > previous) {
|
|
440
|
+
advanced = true;
|
|
441
|
+
}
|
|
442
|
+
connection.seen.set(label, value);
|
|
443
|
+
}
|
|
444
|
+
// What a wedge shows is this counter frozen, not merely behind — see
|
|
445
|
+
// probeWedgeIsCertain. The gap since the last time ANY label moved is
|
|
446
|
+
// this connection's own answer to "how long may a healthy report take
|
|
447
|
+
// to arrive", recorded only while nothing is currently flagged (the
|
|
448
|
+
// same guard `longestHealthyFlatMs` uses): a stretch already under
|
|
449
|
+
// suspicion must not teach the detector to tolerate it.
|
|
450
|
+
if (advanced) {
|
|
451
|
+
if (connection.lastSeenAdvanceAt !== 0 && !connection.probeCaptureStarted) {
|
|
452
|
+
const gap = now - connection.lastSeenAdvanceAt;
|
|
453
|
+
if (gap > connection.longestHealthySeenGapMs) {
|
|
454
|
+
connection.longestHealthySeenGapMs = gap;
|
|
455
|
+
}
|
|
456
|
+
}
|
|
457
|
+
connection.lastSeenAdvanceAt = now;
|
|
351
458
|
}
|
|
352
459
|
}
|
|
353
|
-
const now = Date.now();
|
|
354
460
|
if (connection.echoAt !== 0) {
|
|
355
461
|
const sinceLast = now - connection.echoAt;
|
|
356
462
|
if (sinceLast > connection.echoIntervalMs) {
|
|
@@ -0,0 +1,166 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* @file What this process is holding, said out loud on a regular cadence.
|
|
3
|
+
*
|
|
4
|
+
* Written 2026-08-28, after the kernel killed the proxy and the log could not
|
|
5
|
+
* say why. The supervisor recorded `exit code 137` — SIGKILL, so no core dump —
|
|
6
|
+
* and the kernel ring buffer held the whole of what was known:
|
|
7
|
+
*
|
|
8
|
+
* Out of memory: Killed process 3036113 (MainThread)
|
|
9
|
+
* anon-rss: 2422628kB total-vm: 20856720kB oom_score_adj: 200
|
|
10
|
+
*
|
|
11
|
+
* Two point four gigabytes, on a host with under two free, and the addon is the
|
|
12
|
+
* first thing the kernel picks because Home Assistant gives addons a positive
|
|
13
|
+
* `oom_score_adj`. What the proxy had been logging all along was its share of a
|
|
14
|
+
* CPU. Nothing anywhere said how much memory it held, so the growth that ended
|
|
15
|
+
* in that line has no shape: one final reading taken by the kernel, and no
|
|
16
|
+
* series leading to it.
|
|
17
|
+
*
|
|
18
|
+
* This is that series. It costs one line a minute and reads only counters the
|
|
19
|
+
* runtime already maintains.
|
|
20
|
+
*/
|
|
21
|
+
|
|
22
|
+
import { readFile } from "node:fs/promises";
|
|
23
|
+
import os from "node:os";
|
|
24
|
+
|
|
25
|
+
/** How often the reading is taken and written. */
|
|
26
|
+
export const MEMORY_REPORT_INTERVAL_MS = 60_000;
|
|
27
|
+
|
|
28
|
+
/**
|
|
29
|
+
* What the process is holding, from the runtime's own counters.
|
|
30
|
+
*
|
|
31
|
+
* `rss` is what the kernel counts against us and therefore what the OOM killer
|
|
32
|
+
* reads. The rest says where it went: the JavaScript heap, and everything held
|
|
33
|
+
* outside it — which for this proxy is where the interesting growth lives,
|
|
34
|
+
* since torrent pieces sit in a `SharedArrayBuffer` and segment bodies pass
|
|
35
|
+
* through buffers.
|
|
36
|
+
*
|
|
37
|
+
* @returns {{ rss: number, heapUsed: number, heapTotal: number, external: number, arrayBuffers: number }}
|
|
38
|
+
*/
|
|
39
|
+
export function readProcessMemory() {
|
|
40
|
+
const usage = process.memoryUsage();
|
|
41
|
+
return {
|
|
42
|
+
rss: usage.rss,
|
|
43
|
+
heapUsed: usage.heapUsed,
|
|
44
|
+
heapTotal: usage.heapTotal,
|
|
45
|
+
external: usage.external,
|
|
46
|
+
arrayBuffers: usage.arrayBuffers ?? 0
|
|
47
|
+
};
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
/**
|
|
51
|
+
* How much memory the machine could still give out, in bytes.
|
|
52
|
+
*
|
|
53
|
+
* `os.freemem()` is the wrong quantity on Linux and the difference is not
|
|
54
|
+
* academic: it counts only pages that are free RIGHT NOW, while the kernel
|
|
55
|
+
* deliberately keeps that number low by filling the rest with reclaimable page
|
|
56
|
+
* cache. `MemAvailable` is the kernel's own estimate of what a new allocation
|
|
57
|
+
* could actually obtain, cache included. Reading the estimate the kernel
|
|
58
|
+
* publishes beats recomputing a worse one.
|
|
59
|
+
*
|
|
60
|
+
* @returns {Promise<number | null>} Bytes, or null where /proc is not there.
|
|
61
|
+
*/
|
|
62
|
+
export async function readAvailableMemory() {
|
|
63
|
+
try {
|
|
64
|
+
const text = await readFile("/proc/meminfo", "utf8");
|
|
65
|
+
const match = /^MemAvailable:\s+(\d+)\s+kB$/m.exec(text);
|
|
66
|
+
if (match) {
|
|
67
|
+
return Number(match[1]) * 1024;
|
|
68
|
+
}
|
|
69
|
+
} catch {
|
|
70
|
+
// silent-ok: not Linux, or /proc is not mounted. The fallback below is a
|
|
71
|
+
// worse answer, and saying so is the point of returning it separately.
|
|
72
|
+
}
|
|
73
|
+
return null;
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
/**
|
|
77
|
+
* Available memory, falling back to what the runtime can offer.
|
|
78
|
+
*
|
|
79
|
+
* @returns {Promise<{ bytes: number, measured: boolean }>}
|
|
80
|
+
*/
|
|
81
|
+
export async function availableMemory() {
|
|
82
|
+
const fromKernel = await readAvailableMemory();
|
|
83
|
+
if (fromKernel !== null) {
|
|
84
|
+
return { bytes: fromKernel, measured: true };
|
|
85
|
+
}
|
|
86
|
+
return { bytes: os.freemem(), measured: false };
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
/**
|
|
90
|
+
* Render a size the way a person reads one.
|
|
91
|
+
*
|
|
92
|
+
* @param {number} bytes
|
|
93
|
+
* @returns {string}
|
|
94
|
+
*/
|
|
95
|
+
function megabytes(bytes) {
|
|
96
|
+
return `${Math.round(bytes / (1024 * 1024))}MB`;
|
|
97
|
+
}
|
|
98
|
+
|
|
99
|
+
/**
|
|
100
|
+
* One line saying what the process holds and what the machine has left.
|
|
101
|
+
*
|
|
102
|
+
* Pure, so the wording and the arithmetic can be pinned without a running
|
|
103
|
+
* process. Store figures are given in BYTES rather than in pieces: the piece
|
|
104
|
+
* count is meaningless without the piece size, and the piece size differs per
|
|
105
|
+
* torrent — on the film this proxy died under, 63 pieces meant 504 MB.
|
|
106
|
+
*
|
|
107
|
+
* @param {Object} reading
|
|
108
|
+
* @param {{ rss: number, heapUsed: number, heapTotal: number, external: number, arrayBuffers: number }} reading.process
|
|
109
|
+
* @param {number} reading.availableBytes
|
|
110
|
+
* @param {boolean} reading.availableMeasured
|
|
111
|
+
* @param {{ name: string, residentBytes: number, budgetBytes: number }[]} [reading.stores]
|
|
112
|
+
* @returns {string}
|
|
113
|
+
*/
|
|
114
|
+
export function describeMemory({ process: usage, availableBytes, availableMeasured, stores = [] }) {
|
|
115
|
+
const storeResident = stores.reduce((total, store) => total + (store.residentBytes || 0), 0);
|
|
116
|
+
const storeBudget = stores.reduce((total, store) => total + (store.budgetBytes || 0), 0);
|
|
117
|
+
const storesPart = stores.length === 0
|
|
118
|
+
? "no torrent stores"
|
|
119
|
+
: `${stores.length} torrent store(s) holding ${megabytes(storeResident)} ` +
|
|
120
|
+
`of ${megabytes(storeBudget)} allowed`;
|
|
121
|
+
return (
|
|
122
|
+
`memory: rss=${megabytes(usage.rss)} heap=${megabytes(usage.heapUsed)}/${megabytes(usage.heapTotal)} ` +
|
|
123
|
+
`external=${megabytes(usage.external)} arrayBuffers=${megabytes(usage.arrayBuffers)}; ` +
|
|
124
|
+
`${storesPart}; ` +
|
|
125
|
+
`machine has ${megabytes(availableBytes)} available` +
|
|
126
|
+
`${availableMeasured ? "" : " (estimated — /proc/meminfo could not be read)"}`
|
|
127
|
+
);
|
|
128
|
+
}
|
|
129
|
+
|
|
130
|
+
/**
|
|
131
|
+
* Report memory on a timer until stopped.
|
|
132
|
+
*
|
|
133
|
+
* @param {Object} options
|
|
134
|
+
* @param {(message: string) => void} options.log
|
|
135
|
+
* @param {() => { name: string, residentBytes: number, budgetBytes: number }[]} [options.readStores]
|
|
136
|
+
* @param {number} [options.intervalMs]
|
|
137
|
+
* @returns {{ stop: () => void }}
|
|
138
|
+
*/
|
|
139
|
+
export function startMemoryReport({ log, readStores, intervalMs = MEMORY_REPORT_INTERVAL_MS }) {
|
|
140
|
+
const tick = async () => {
|
|
141
|
+
try {
|
|
142
|
+
const { bytes, measured } = await availableMemory();
|
|
143
|
+
let stores = [];
|
|
144
|
+
try {
|
|
145
|
+
stores = typeof readStores === "function" ? readStores() ?? [] : [];
|
|
146
|
+
} catch {
|
|
147
|
+
// silent-ok: a store list that cannot be read must not stop the reading
|
|
148
|
+
// that matters, which is the process's own.
|
|
149
|
+
}
|
|
150
|
+
log(describeMemory({
|
|
151
|
+
process: readProcessMemory(),
|
|
152
|
+
availableBytes: bytes,
|
|
153
|
+
availableMeasured: measured,
|
|
154
|
+
stores
|
|
155
|
+
}));
|
|
156
|
+
} catch {
|
|
157
|
+
// silent-ok: a reading that fails is not worth ending the series over.
|
|
158
|
+
}
|
|
159
|
+
};
|
|
160
|
+
void tick();
|
|
161
|
+
const timer = setInterval(() => { void tick(); }, intervalMs);
|
|
162
|
+
if (typeof timer.unref === "function") {
|
|
163
|
+
timer.unref();
|
|
164
|
+
}
|
|
165
|
+
return { stop: () => clearInterval(timer) };
|
|
166
|
+
}
|