@torrent-tv/proxy 2.64.5 → 2.64.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +23 -0
- package/package.json +1 -1
- package/server.js +1 -1
- package/services/hls-session-manager.js +71 -2
- package/services/memory-report.js +178 -33
- package/services/piece-store/piece-lru.js +16 -0
- package/services/piece-store/shared-piece-store.js +262 -137
- package/services/torrent-worker/client.js +7 -3
- package/services/torrent-worker/pool-adapter.js +1 -1
- package/services/torrent-worker/worker.js +20 -4
- package/test/memory-budget.test.js +63 -1
- package/test/piece-lru.test.js +18 -0
- package/test/piece-store-reservations.test.js +263 -0
- package/test/quality-variants.test.js +39 -0
- package/test/viewer-position-on-open.test.js +40 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,26 @@
|
|
|
1
|
+
## 2.64.7
|
|
2
|
+
|
|
3
|
+
- **Fix**: A file opened at a position starts its SOUND at that position. The audio rendition's start is worked out from the picture's read head less the buffer the viewer reports holding — sound, because a read head is the furthest request of any viewer and the picture sits behind it by however deep that buffer is. At a cold open there is no report yet, and the fallback subtracted the WHOLE 120 s look-ahead from a buffer that does not exist: field 2026-08-31, a page opened at 588 s started its sound at 460 s, 131 seconds of film nobody would hear, and the segment the viewer needed took 38.8 s to appear against the picture's 8.4 s — the audio encoder healthy at 2.3-3.1x throughout, simply given a running start it did not need. The reading now says WHICH of its three sources answered (`viewerPositionSource`): a seek and a served segment are request edges and keep the subtraction, while the opening position is not an edge — nothing has been asked for since the session was made, and a browser that has just opened holds nothing by construction (`research/cold-open-audio-start-2026-08-31.md`).
|
|
4
|
+
- **Fix**: The first encode run is positioned from the position the viewer ASKED for, not from the figure rounded to ten seconds. That rounding exists to answer one question — whether two viewers share a session — and `Math.round` can move a position FORWARD: 588 s became 590 s, which falls in segment #85 while the viewer at 588 s is inside #84. The player asked for a segment behind the run, the run was restarted onto it, and the 4.5 s it had already produced were thrown away.
|
|
5
|
+
|
|
6
|
+
## 2.64.6
|
|
7
|
+
|
|
8
|
+
- **New**: The torrent worker reads its own memory once a SECOND, and writes a line only when something moved. A minute cannot see what kills it: three times — 2026-08-30 14:00 and 23:19, 2026-08-31 13:27 — the worker's own line read `heap=28-36MB`, and by the sample after next the thread had been terminated for reaching its heap ceiling, with the whole rise fitting inside a single sixty-second gap. The reading and the line are now separate cadences: taken every second, written when the heap has moved by 25 MB or when a quiet minute is up, so a healthy session costs the same one line a minute it costs today and a runaway is a curve rather than a step.
|
|
9
|
+
- **New**: A heap snapshot of the WORKER's own isolate, on every new high-water above 400 MB, into the state directory, three kept. Every snapshot ever written by this proxy has been of the MAIN isolate — `v8.writeHeapSnapshot` snapshots the thread that calls it, and the main thread's heap is 26 MB while the one that dies climbs to 2240 MB. So the isolate that has died three times has never once been looked at, and the question "what is holding this memory" has had no way to be answered. Deliberately NOT taken as the ceiling is approached: a snapshot is about the size of the heap it is of, so asking for one at 1.9 GB on a machine with 600 MB left is a way to cause the kill being studied — and at one reading a second no 400 MB step is ever missed. `stateDir` now travels to the worker for this, because a worker cannot change the process's working directory and had no way to choose where a file lands.
|
|
10
|
+
- **New**: The memory line says the heap ceiling beside the heap — `heap=1800MB/1904MB of 2240MB allowed`. That ceiling is what the runtime terminates a thread for reaching, it is inherited from the main isolate by a worker created without `resourceLimits`, and without it the log said "30MB" with no way to tell how far that was from the end.
|
|
11
|
+
|
|
12
|
+
- **Fix**: Room for a piece in the store is an owned reservation, released in a `finally`, instead of a number one function increments and another decrements. Every consequence of the old pairing was a defect, and all of them are gone with it: a failure between `#claimSlot` and `#registerPiece` — a disk read that throws — lost a slot for the life of the process, because there was no `try/finally` anywhere on that path; `put`'s error path tried to correct that by guessing, and its own comments said so ("Heuristic:", "We conservatively decrement"), so it could take back a reservation belonging to a different claim in flight and let the store admit past its allowance until the next minute's revision evicted the excess. `stats()` now reports `outstanding`, and the periodic `piece-store` line prints it when it is not zero: a reservation that never comes back is invisible until the store cannot admit anything, and by then the reason is long gone.
|
|
13
|
+
- **Fix**: A claim waits on PROGRESS, not on activity. The five-second "every resident piece is pinned" error was skipped entirely while a spill was in flight or any reservation was held, so ONE lost reservation from the defect above made that error permanently unreachable and a read retried every 50 ms for the rest of the process's life — never completing, never failing, and never saying anything. The clock now restarts when something actually moves (a piece admitted, a spill finished, a pin released) and the error is raised when nothing has moved for five seconds, whatever is nominally in flight.
|
|
14
|
+
- **Fix**: The wait between attempts allocates once instead of once per pending spill. It attached a fresh pair of handlers to EVERY spill in flight on EVERY attempt and left an uncancelled timer behind each time, so a claim that could not be satisfied allocated in proportion to attempts times spills. One idempotent handler now, its timer cleared when it is woken, and a settling spill wakes the store itself — which is where that belonged, and which `#claimSlotOnce`'s own spills were not doing at all.
|
|
15
|
+
- **Fix**: Closing or destroying the store wakes whoever is waiting for room, and a claim checks for a closed store on every attempt. Callers inside `#claimSlot` were simply left asleep for ever.
|
|
16
|
+
- **Fix**: A piece handed back to memory while its own spill is still being written no longer reappears on disk. `DiskTier.write` records the index when it COMPLETES, and `put` called `forget` before that, so the completing write put the stale copy back and a later read returned bytes from before the rewrite. `put` now waits for that piece's spill to finish before forgetting it.
|
|
17
|
+
- **Fix**: A spill that fails while the allowance is being lowered is counted (`spill-failures` in the store's line) instead of thrown into nothing. Nobody awaits those spills, so the rethrow was an unhandled rejection, and an unhandled rejection in the torrent worker ends the thread — a second way to lose the torrent client on top of the one being investigated.
|
|
18
|
+
- **Fix**: Two concurrent revivals of the same piece no longer each allocate and register a buffer for it, leaving whoever holds the first reading memory the store has stopped tracking. The second finds the piece already back and uses it.
|
|
19
|
+
- **Fix**: `PieceLru` follows the store's live allowance. It was built with the capacity the store was created with and never revised, so `isFull()` answered against a number that had stopped being the limit — dormant only because nothing calls it today.
|
|
20
|
+
- **Chore**: The store accepts an injected disk tier so a test can hold a write open or make one fail on purpose. Four of the defects above live in what happens when the disk does not answer at once, and none of them was reachable from outside before (`test/piece-store-reservations.test.js`).
|
|
21
|
+
|
|
22
|
+
All eight were found by reading the file after the torrent worker was killed by its own JS heap limit on 2026-08-31 (`research/worker-heap-oom-2026-08-31.md`, §5). None of them is proven to be that growth; they are what the reading found, and each is a defect in its own right.
|
|
23
|
+
|
|
1
24
|
## 2.64.5
|
|
2
25
|
|
|
3
26
|
- **Fix**: A rung measured at 0.007x is no longer kept just because it is on screen. The `playingHeight` exemption is now checked after the `measured < 1` withdrawal, so a 4K HEVC transcode at 0.007x on a CM4 (field 2026-08-31, 0.1x at 23:45 and 0.007x at 06:57, 0.04s buffered) is withdrawn and the offer can become empty instead of stalling the viewer with no way to downgrade. `ownHeight` is kept only for a copied source (`!transcodeVideo`), not for a re-encode already running at 0.18x.
|
package/package.json
CHANGED
package/server.js
CHANGED
|
@@ -116,7 +116,7 @@ export async function startProxyServer({
|
|
|
116
116
|
// idled. Serving a segment shared that thread, so reading an already-finished
|
|
117
117
|
// 10 MB file took 12-23 s against 125 ms to hand it to the channel. The
|
|
118
118
|
// adapter keeps TorrentPool's interface, so nothing downstream changed.
|
|
119
|
-
const torrentPool = new WorkerTorrentPool({ maxDiskBytes, memoryBytes, onSubtitleCues });
|
|
119
|
+
const torrentPool = new WorkerTorrentPool({ maxDiskBytes, memoryBytes, stateDir, onSubtitleCues });
|
|
120
120
|
const selectedPort = await getPort({
|
|
121
121
|
port: buildPortCandidates(port)
|
|
122
122
|
});
|
|
@@ -1257,6 +1257,34 @@ function headsOf(session) {
|
|
|
1257
1257
|
return session.consumerHeads;
|
|
1258
1258
|
}
|
|
1259
1259
|
|
|
1260
|
+
/**
|
|
1261
|
+
* WHICH of the three readings answered, which is a different question from what
|
|
1262
|
+
* the answer was.
|
|
1263
|
+
*
|
|
1264
|
+
* It matters for one thing: `openedAt` is not a request edge. The other two are
|
|
1265
|
+
* — a seek and a requested segment are both places a viewer has moved to while
|
|
1266
|
+
* holding a buffer, so the picture is behind them by however deep that buffer
|
|
1267
|
+
* is. `openedAt` is where a session was created and nothing has been asked for
|
|
1268
|
+
* since, so the picture is exactly there and there is nothing to subtract.
|
|
1269
|
+
* Reading the number without knowing which of the three it was is what started
|
|
1270
|
+
* a cold open's audio two minutes early on 2026-08-31.
|
|
1271
|
+
*
|
|
1272
|
+
* @param {{ seeked?: number, lastRequestedStart?: number | null, openedAt?: number }} readings
|
|
1273
|
+
* @returns {"seeked" | "requested" | "opened" | "none"}
|
|
1274
|
+
*/
|
|
1275
|
+
export function viewerPositionSource({ seeked, lastRequestedStart, openedAt }) {
|
|
1276
|
+
if (Number.isFinite(seeked) && seeked > 0) {
|
|
1277
|
+
return "seeked";
|
|
1278
|
+
}
|
|
1279
|
+
if (Number.isFinite(lastRequestedStart) && lastRequestedStart > 0) {
|
|
1280
|
+
return "requested";
|
|
1281
|
+
}
|
|
1282
|
+
if (Number.isFinite(openedAt) && openedAt > 0) {
|
|
1283
|
+
return "opened";
|
|
1284
|
+
}
|
|
1285
|
+
return "none";
|
|
1286
|
+
}
|
|
1287
|
+
|
|
1260
1288
|
export function resolveViewerPosition({ seeked, lastRequestedStart, openedAt }) {
|
|
1261
1289
|
if (Number.isFinite(seeked) && seeked > 0) {
|
|
1262
1290
|
return seeked;
|
|
@@ -2408,8 +2436,19 @@ export class HlsSessionManager {
|
|
|
2408
2436
|
// asked for #152, and 45 s later the browser gave up with "no data arrived
|
|
2409
2437
|
// from the proxy" while the transcode ran happily at 9.9x through the
|
|
2410
2438
|
// opening credits.
|
|
2411
|
-
|
|
2412
|
-
|
|
2439
|
+
// From what the viewer ASKED for, not from the rounded figure. The rounding
|
|
2440
|
+
// exists to answer one question — is this the same session as somebody
|
|
2441
|
+
// else's — and it is the wrong number for this one, because `Math.round`
|
|
2442
|
+
// can move the position FORWARD: 588s became 590s, which falls in segment
|
|
2443
|
+
// #85 while the viewer at 588s is inside #84. The player then asked for a
|
|
2444
|
+
// segment behind the run, the run was restarted onto it, and the 4.5s it
|
|
2445
|
+
// had produced were thrown away (field 2026-08-31,
|
|
2446
|
+
// `research/cold-open-audio-start-2026-08-31.md`).
|
|
2447
|
+
const requestedStart = Number.isFinite(startPositionSeconds) && startPositionSeconds > 0
|
|
2448
|
+
? startPositionSeconds
|
|
2449
|
+
: 0;
|
|
2450
|
+
const firstIndex = requestedStart > 0
|
|
2451
|
+
? this.#segmentIndexForTime(session, requestedStart)
|
|
2413
2452
|
: 0;
|
|
2414
2453
|
await this.#startEncodeRun(session, firstIndex);
|
|
2415
2454
|
|
|
@@ -7824,10 +7863,40 @@ export class HlsSessionManager {
|
|
|
7824
7863
|
// would start the run where no request can ever reach it.
|
|
7825
7864
|
return Math.max(0, Math.min(earliestStated, readHead) - this.segmentDurationSec);
|
|
7826
7865
|
}
|
|
7866
|
+
if (this.#viewerPositionSourceOf(watching) === "opened") {
|
|
7867
|
+
// The session has not started. Nobody has seeked, nobody has asked for a
|
|
7868
|
+
// segment, and nobody has reported anything — so the read head is not a
|
|
7869
|
+
// request edge at all, it is where the viewer opened, and a browser that
|
|
7870
|
+
// has just opened holds no buffer by construction. Subtracting one here
|
|
7871
|
+
// is not erring "early, the cheap direction": it is the whole of the
|
|
7872
|
+
// start-up cost. Field 2026-08-31: a page opened at 588s started its
|
|
7873
|
+
// sound at 460s, 131 seconds of film nobody would hear, and the segment
|
|
7874
|
+
// the viewer needed took 38.8s to appear against the picture's 8.4s
|
|
7875
|
+
// (`research/cold-open-audio-start-2026-08-31.md`).
|
|
7876
|
+
return Math.max(0, readHead - this.segmentDurationSec);
|
|
7877
|
+
}
|
|
7827
7878
|
const buffered = deepestBuffer === null ? LOOKAHEAD_PAUSE_SECONDS : deepestBuffer;
|
|
7828
7879
|
return Math.max(0, readHead - buffered - this.segmentDurationSec);
|
|
7829
7880
|
}
|
|
7830
7881
|
|
|
7882
|
+
/**
|
|
7883
|
+
* Which reading gave this session's viewer position — see
|
|
7884
|
+
* {@link viewerPositionSource}.
|
|
7885
|
+
*
|
|
7886
|
+
* @param {HlsSession} session
|
|
7887
|
+
* @returns {"seeked" | "requested" | "opened" | "none"}
|
|
7888
|
+
*/
|
|
7889
|
+
#viewerPositionSourceOf(session) {
|
|
7890
|
+
const lastRequestedStart = Number.isInteger(session.lastRequestedSegment) && session.lastRequestedSegment > 0
|
|
7891
|
+
? this.#segmentStartTime(session, session.lastRequestedSegment)
|
|
7892
|
+
: null;
|
|
7893
|
+
return viewerPositionSource({
|
|
7894
|
+
seeked: session.viewerPositionSeconds,
|
|
7895
|
+
lastRequestedStart,
|
|
7896
|
+
openedAt: session.progress?.startPositionSeconds
|
|
7897
|
+
});
|
|
7898
|
+
}
|
|
7899
|
+
|
|
7831
7900
|
#viewerPositionOf(session) {
|
|
7832
7901
|
const lastRequestedStart = Number.isInteger(session.lastRequestedSegment) && session.lastRequestedSegment > 0
|
|
7833
7902
|
? this.#segmentStartTime(session, session.lastRequestedSegment)
|
|
@@ -19,7 +19,7 @@
|
|
|
19
19
|
* runtime already maintains.
|
|
20
20
|
*/
|
|
21
21
|
|
|
22
|
-
import { readFile, statfs } from "node:fs/promises";
|
|
22
|
+
import { readdir, readFile, rm, statfs } from "node:fs/promises";
|
|
23
23
|
import os from "node:os";
|
|
24
24
|
import v8 from "node:v8";
|
|
25
25
|
import path from "node:path";
|
|
@@ -27,6 +27,18 @@ import path from "node:path";
|
|
|
27
27
|
/** How often the reading is taken and written. */
|
|
28
28
|
export const MEMORY_REPORT_INTERVAL_MS = 60_000;
|
|
29
29
|
|
|
30
|
+
/**
|
|
31
|
+
* How often the torrent worker takes its own reading.
|
|
32
|
+
*
|
|
33
|
+
* A minute cannot see what killed it. Three times — 2026-08-30 14:00 and 23:19,
|
|
34
|
+
* 2026-08-31 13:27 — the worker's heap read 28-36 MB in one sample and the
|
|
35
|
+
* thread was dead by the sample after next, with the whole rise from 30 MB to
|
|
36
|
+
* the 2240 MB heap limit fitting inside a single gap. A second is short enough
|
|
37
|
+
* that the rise is a curve rather than a step, and the line is only WRITTEN when
|
|
38
|
+
* something moved, so a quiet session costs what it costs today.
|
|
39
|
+
*/
|
|
40
|
+
export const WORKER_MEMORY_SAMPLE_MS = 1_000;
|
|
41
|
+
|
|
30
42
|
/**
|
|
31
43
|
* What the process is holding, from the runtime's own counters.
|
|
32
44
|
*
|
|
@@ -36,16 +48,30 @@ export const MEMORY_REPORT_INTERVAL_MS = 60_000;
|
|
|
36
48
|
* since torrent pieces sit in a `SharedArrayBuffer` and segment bodies pass
|
|
37
49
|
* through buffers.
|
|
38
50
|
*
|
|
39
|
-
*
|
|
51
|
+
* `heapLimit` is this isolate's own ceiling, and it belongs beside the heap
|
|
52
|
+
* figures because it is what the runtime kills the thread for reaching — a
|
|
53
|
+
* worker created without `resourceLimits` inherits the main isolate's, 2240 MB
|
|
54
|
+
* on the addon host. Without it the log said 30 MB and gave no idea how far
|
|
55
|
+
* that was from the end.
|
|
56
|
+
*
|
|
57
|
+
* @returns {{ rss: number, heapUsed: number, heapTotal: number, external: number, arrayBuffers: number, heapLimit: number }}
|
|
40
58
|
*/
|
|
41
59
|
export function readProcessMemory() {
|
|
42
60
|
const usage = process.memoryUsage();
|
|
61
|
+
let heapLimit = 0;
|
|
62
|
+
try {
|
|
63
|
+
heapLimit = v8.getHeapStatistics().heap_size_limit ?? 0;
|
|
64
|
+
} catch {
|
|
65
|
+
// silent-ok: a missing ceiling leaves the term out, it does not cost the
|
|
66
|
+
// reading beside it.
|
|
67
|
+
}
|
|
43
68
|
return {
|
|
44
69
|
rss: usage.rss,
|
|
45
70
|
heapUsed: usage.heapUsed,
|
|
46
71
|
heapTotal: usage.heapTotal,
|
|
47
72
|
external: usage.external,
|
|
48
|
-
arrayBuffers: usage.arrayBuffers ?? 0
|
|
73
|
+
arrayBuffers: usage.arrayBuffers ?? 0,
|
|
74
|
+
heapLimit
|
|
49
75
|
};
|
|
50
76
|
}
|
|
51
77
|
|
|
@@ -198,7 +224,8 @@ export function describeMemory({
|
|
|
198
224
|
`committed ${megabytes(storeCommitted)} of ${megabytes(storeBudget)} allowed, ` +
|
|
199
225
|
`${megabytes(storeSpilled)} spilled to disk`;
|
|
200
226
|
const isolate =
|
|
201
|
-
`heap=${megabytes(usage.heapUsed)}/${megabytes(usage.heapTotal)}
|
|
227
|
+
`heap=${megabytes(usage.heapUsed)}/${megabytes(usage.heapTotal)}` +
|
|
228
|
+
`${usage.heapLimit ? ` of ${megabytes(usage.heapLimit)} allowed` : ""} ` +
|
|
202
229
|
`external=${megabytes(usage.external)} arrayBuffers=${megabytes(usage.arrayBuffers)}`;
|
|
203
230
|
if (scope === "thread") {
|
|
204
231
|
return `memory (${label || "thread"}): ${isolate}; ${storesPart}`;
|
|
@@ -213,6 +240,39 @@ export function describeMemory({
|
|
|
213
240
|
);
|
|
214
241
|
}
|
|
215
242
|
|
|
243
|
+
/**
|
|
244
|
+
* Whether this reading is worth writing down, and why.
|
|
245
|
+
*
|
|
246
|
+
* A series taken every second and printed every second is unreadable, and one
|
|
247
|
+
* printed every minute cannot see a rise that takes forty seconds. So the
|
|
248
|
+
* cadence of the READING and the cadence of the LINE are separate: the figure
|
|
249
|
+
* is taken often, and written when it has moved or when the quiet interval has
|
|
250
|
+
* passed. Pure, so the rule can be pinned without a clock.
|
|
251
|
+
*
|
|
252
|
+
* @param {Object} state
|
|
253
|
+
* @param {number} state.watchedBytes - The figure this scope is watching.
|
|
254
|
+
* @param {number} state.lastWrittenBytes
|
|
255
|
+
* @param {number} state.sinceWrittenMs
|
|
256
|
+
* @param {number} state.changeBytes - Movement that earns a line of its own.
|
|
257
|
+
* @param {number} state.quietMs - How long silence may last regardless.
|
|
258
|
+
* @returns {boolean}
|
|
259
|
+
*/
|
|
260
|
+
export function readingIsWorthWriting({
|
|
261
|
+
watchedBytes,
|
|
262
|
+
lastWrittenBytes,
|
|
263
|
+
sinceWrittenMs,
|
|
264
|
+
changeBytes,
|
|
265
|
+
quietMs
|
|
266
|
+
}) {
|
|
267
|
+
if (quietMs <= 0 || sinceWrittenMs >= quietMs) {
|
|
268
|
+
return true;
|
|
269
|
+
}
|
|
270
|
+
if (changeBytes <= 0) {
|
|
271
|
+
return false;
|
|
272
|
+
}
|
|
273
|
+
return Math.abs(watchedBytes - lastWrittenBytes) >= changeBytes;
|
|
274
|
+
}
|
|
275
|
+
|
|
216
276
|
/**
|
|
217
277
|
* Report memory on a timer until stopped.
|
|
218
278
|
*
|
|
@@ -225,7 +285,17 @@ export function describeMemory({
|
|
|
225
285
|
* @param {string} [options.label] - Which thread the isolate figures are of.
|
|
226
286
|
* @param {string} [options.diskPath] - Where pieces spill, for the free-space
|
|
227
287
|
* reading. Omitted, the disk term is left out rather than guessed.
|
|
228
|
-
* @param {number} [options.intervalMs]
|
|
288
|
+
* @param {number} [options.intervalMs] - How often the figure is READ.
|
|
289
|
+
* @param {number} [options.quietMs] - How long the line may stay silent while
|
|
290
|
+
* nothing moves. Zero writes every reading, which is what the process scope
|
|
291
|
+
* has always done.
|
|
292
|
+
* @param {number} [options.changeBytes] - Movement that earns a line before the
|
|
293
|
+
* quiet interval is up.
|
|
294
|
+
* @param {string} [options.snapshotDir] - Where heap snapshots are written.
|
|
295
|
+
* Defaults to the temporary directory, as the process scope always did.
|
|
296
|
+
* @param {number} [options.snapshotFloorBytes]
|
|
297
|
+
* @param {number} [options.snapshotGrowthBytes]
|
|
298
|
+
* @param {number} [options.keepSnapshots] - Newest to keep; zero keeps all.
|
|
229
299
|
* @returns {{ stop: () => void }}
|
|
230
300
|
*/
|
|
231
301
|
export function startMemoryReport({
|
|
@@ -234,9 +304,62 @@ export function startMemoryReport({
|
|
|
234
304
|
scope = "process",
|
|
235
305
|
label = "",
|
|
236
306
|
diskPath = "",
|
|
237
|
-
intervalMs = MEMORY_REPORT_INTERVAL_MS
|
|
307
|
+
intervalMs = MEMORY_REPORT_INTERVAL_MS,
|
|
308
|
+
quietMs = 0,
|
|
309
|
+
changeBytes = 0,
|
|
310
|
+
snapshotDir = "",
|
|
311
|
+
snapshotFloorBytes = 500 * 1024 * 1024,
|
|
312
|
+
snapshotGrowthBytes = 100 * 1024 * 1024,
|
|
313
|
+
keepSnapshots = 0
|
|
238
314
|
}) {
|
|
239
|
-
let
|
|
315
|
+
let highWater = 0;
|
|
316
|
+
let lastWrittenBytes = 0;
|
|
317
|
+
let lastWrittenAt = 0;
|
|
318
|
+
// The process watches what the kernel kills it for; a thread watches what the
|
|
319
|
+
// runtime kills IT for, which is its own heap and not the process's resident
|
|
320
|
+
// memory — the main isolate sat at 26 MB while the worker's heap climbed to
|
|
321
|
+
// its 2240 MB ceiling.
|
|
322
|
+
const watchedOf = (memory) => (scope === "thread" ? memory.heapTotal : memory.rss);
|
|
323
|
+
const slug = (label || scope).replace(/[^a-z0-9]+/gi, "-").toLowerCase();
|
|
324
|
+
const directory = snapshotDir || os.tmpdir();
|
|
325
|
+
|
|
326
|
+
/**
|
|
327
|
+
* @param {number} watchedBytes
|
|
328
|
+
* @param {string} why
|
|
329
|
+
* @returns {Promise<void>}
|
|
330
|
+
*/
|
|
331
|
+
const takeSnapshot = async (watchedBytes, why) => {
|
|
332
|
+
let snapPath = "";
|
|
333
|
+
try {
|
|
334
|
+
snapPath = path.join(directory, `heap-${slug}-${Date.now()}-${watchedBytes}.heapsnapshot`);
|
|
335
|
+
// Synchronous and proportional to the heap, so it stops this thread for
|
|
336
|
+
// as long as it takes to write. That is the price of the only reading
|
|
337
|
+
// that names what is holding the memory, and it is why it is bounded by
|
|
338
|
+
// a floor, by a growth step and by how many are kept.
|
|
339
|
+
v8.writeHeapSnapshot(snapPath);
|
|
340
|
+
log(`memory: wrote heap snapshot of the ${label || scope} to ${snapPath} (${megabytes(watchedBytes)}, ${why})`);
|
|
341
|
+
} catch {
|
|
342
|
+
// silent-ok: no snapshot is worse than a snapshot, and much better than
|
|
343
|
+
// ending the series that leads to the next one.
|
|
344
|
+
return;
|
|
345
|
+
}
|
|
346
|
+
if (keepSnapshots <= 0) {
|
|
347
|
+
return;
|
|
348
|
+
}
|
|
349
|
+
try {
|
|
350
|
+
const prefix = `heap-${slug}-`;
|
|
351
|
+
const mine = (await readdir(directory))
|
|
352
|
+
.filter((name) => name.startsWith(prefix) && name.endsWith(".heapsnapshot"))
|
|
353
|
+
.sort();
|
|
354
|
+
for (const name of mine.slice(0, Math.max(0, mine.length - keepSnapshots))) {
|
|
355
|
+
await rm(path.join(directory, name), { force: true });
|
|
356
|
+
}
|
|
357
|
+
} catch {
|
|
358
|
+
// silent-ok: a snapshot that could not be pruned is a disk-space problem
|
|
359
|
+
// for later, not a reason to lose the one just written.
|
|
360
|
+
}
|
|
361
|
+
};
|
|
362
|
+
|
|
240
363
|
const tick = async () => {
|
|
241
364
|
try {
|
|
242
365
|
let stores = [];
|
|
@@ -247,35 +370,57 @@ export function startMemoryReport({
|
|
|
247
370
|
// that matters, which is the process's own.
|
|
248
371
|
}
|
|
249
372
|
const processMemory = readProcessMemory();
|
|
373
|
+
const watched = watchedOf(processMemory);
|
|
374
|
+
const now = Date.now();
|
|
375
|
+
const write = readingIsWorthWriting({
|
|
376
|
+
watchedBytes: watched,
|
|
377
|
+
lastWrittenBytes,
|
|
378
|
+
sinceWrittenMs: lastWrittenAt === 0 ? Number.POSITIVE_INFINITY : now - lastWrittenAt,
|
|
379
|
+
changeBytes,
|
|
380
|
+
quietMs
|
|
381
|
+
});
|
|
382
|
+
|
|
383
|
+
let anonymousBytes = null;
|
|
250
384
|
if (scope === "thread") {
|
|
251
|
-
|
|
252
|
-
|
|
385
|
+
if (write) {
|
|
386
|
+
log(describeMemory({ scope, label, process: processMemory, stores }));
|
|
387
|
+
}
|
|
388
|
+
} else {
|
|
389
|
+
const { bytes, measured } = await availableMemory();
|
|
390
|
+
anonymousBytes = await readAnonymousMemory();
|
|
391
|
+
const diskFreeBytes = diskPath ? await readDiskFree(diskPath) : null;
|
|
392
|
+
if (write) {
|
|
393
|
+
log(describeMemory({
|
|
394
|
+
scope,
|
|
395
|
+
label,
|
|
396
|
+
process: processMemory,
|
|
397
|
+
availableBytes: bytes,
|
|
398
|
+
availableMeasured: measured,
|
|
399
|
+
anonymousBytes,
|
|
400
|
+
diskFreeBytes,
|
|
401
|
+
stores
|
|
402
|
+
}));
|
|
403
|
+
}
|
|
404
|
+
}
|
|
405
|
+
if (write) {
|
|
406
|
+
lastWrittenBytes = watched;
|
|
407
|
+
lastWrittenAt = now;
|
|
253
408
|
}
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
}));
|
|
267
|
-
// High-water and heap snapshot on growth — gives a file to open in Chrome DevTools
|
|
268
|
-
// when the kernel is about to kill the process. One snapshot per high-water.
|
|
269
|
-
if (processMemory.rss > highWaterRss + 100 * 1024 * 1024 && processMemory.rss > 500 * 1024 * 1024) {
|
|
270
|
-
highWaterRss = processMemory.rss;
|
|
271
|
-
try {
|
|
272
|
-
const snapPath = path.join(os.tmpdir(), `heap-${Date.now()}-${processMemory.rss}.heapsnapshot`);
|
|
273
|
-
v8.writeHeapSnapshot(snapPath);
|
|
274
|
-
log(`memory: wrote heap snapshot to ${snapPath} rss=${Math.round(processMemory.rss / (1024 * 1024))}MB anon=${anonymousBytes ? Math.round(anonymousBytes / (1024 * 1024)) : "?"}MB`);
|
|
275
|
-
} catch {}
|
|
409
|
+
|
|
410
|
+
// A snapshot per high-water, so there is a file to open when the growth
|
|
411
|
+
// has to be named rather than described.
|
|
412
|
+
//
|
|
413
|
+
// Deliberately NOT one taken as the ceiling is approached: a snapshot is
|
|
414
|
+
// written by the isolate itself and is about the size of its heap, so
|
|
415
|
+
// asking for one at 1.9 GB on a machine with 600 MB left is a good way to
|
|
416
|
+
// cause the kill being studied. Whatever holds 1.6 GB is the same thing
|
|
417
|
+
// that holds 2.2 GB, and at one reading a second no step is ever missed.
|
|
418
|
+
if (watched > highWater + snapshotGrowthBytes && watched > snapshotFloorBytes) {
|
|
419
|
+
highWater = watched;
|
|
420
|
+
await takeSnapshot(watched, "a new high-water");
|
|
276
421
|
}
|
|
277
|
-
if (anonymousBytes !== null && processMemory.rss > 800 * 1024 * 1024) {
|
|
278
|
-
log(`memory: high rss=${
|
|
422
|
+
if (scope !== "thread" && anonymousBytes !== null && processMemory.rss > 800 * 1024 * 1024) {
|
|
423
|
+
log(`memory: high rss=${megabytes(processMemory.rss)} anon=${megabytes(anonymousBytes)} heap=${megabytes(processMemory.heapUsed)} — watch for OOM`);
|
|
279
424
|
}
|
|
280
425
|
} catch {
|
|
281
426
|
// silent-ok: a reading that fails is not worth ending the series over.
|
|
@@ -64,6 +64,22 @@ export class PieceLru {
|
|
|
64
64
|
return this.#capacity;
|
|
65
65
|
}
|
|
66
66
|
|
|
67
|
+
/**
|
|
68
|
+
* Follow the store's live allowance, which moves with the machine's free
|
|
69
|
+
* memory. Without this the capacity stayed at whatever the store was created
|
|
70
|
+
* with, and {@link PieceLru#isFull} answered against a number that had not
|
|
71
|
+
* been the limit for some time.
|
|
72
|
+
*
|
|
73
|
+
* @param {number} capacity
|
|
74
|
+
* @returns {void}
|
|
75
|
+
*/
|
|
76
|
+
setCapacity(capacity) {
|
|
77
|
+
if (!Number.isInteger(capacity) || capacity < 1) {
|
|
78
|
+
return;
|
|
79
|
+
}
|
|
80
|
+
this.#capacity = capacity;
|
|
81
|
+
}
|
|
82
|
+
|
|
67
83
|
/**
|
|
68
84
|
* How many pieces are currently held by a reader.
|
|
69
85
|
*
|