@torrent-tv/proxy 2.64.5 → 2.64.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +18 -0
- package/package.json +1 -1
- package/server.js +1 -1
- package/services/memory-report.js +178 -33
- package/services/piece-store/piece-lru.js +16 -0
- package/services/piece-store/shared-piece-store.js +262 -137
- package/services/torrent-worker/client.js +7 -3
- package/services/torrent-worker/pool-adapter.js +1 -1
- package/services/torrent-worker/worker.js +20 -4
- package/test/memory-budget.test.js +63 -1
- package/test/piece-lru.test.js +18 -0
- package/test/piece-store-reservations.test.js +263 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,21 @@
|
|
|
1
|
+
## 2.64.6
|
|
2
|
+
|
|
3
|
+
- **New**: The torrent worker reads its own memory once a SECOND, and writes a line only when something moved. A minute cannot see what kills it: three times — 2026-08-30 14:00 and 23:19, 2026-08-31 13:27 — the worker's own line read `heap=28-36MB`, and by the sample after next the thread had been terminated for reaching its heap ceiling, with the whole rise fitting inside a single sixty-second gap. The reading and the line are now separate cadences: taken every second, written when the heap has moved by 25 MB or when a quiet minute is up, so a healthy session costs the same one line a minute it costs today and a runaway is a curve rather than a step.
|
|
4
|
+
- **New**: A heap snapshot of the WORKER's own isolate, on every new high-water above 400 MB, into the state directory, three kept. Every snapshot ever written by this proxy has been of the MAIN isolate — `v8.writeHeapSnapshot` snapshots the thread that calls it, and the main thread's heap is 26 MB while the one that dies climbs to 2240 MB. So the isolate that has died three times has never once been looked at, and the question "what is holding this memory" has had no way to be answered. Deliberately NOT taken as the ceiling is approached: a snapshot is about the size of the heap it is of, so asking for one at 1.9 GB on a machine with 600 MB left is a way to cause the kill being studied — and at one reading a second no 400 MB step is ever missed. `stateDir` now travels to the worker for this, because a worker cannot change the process's working directory and had no way to choose where a file lands.
|
|
5
|
+
- **New**: The memory line says the heap ceiling beside the heap — `heap=1800MB/1904MB of 2240MB allowed`. That ceiling is what the runtime terminates a thread for reaching, it is inherited from the main isolate by a worker created without `resourceLimits`, and without it the log said "30MB" with no way to tell how far that was from the end.
|
|
6
|
+
|
|
7
|
+
- **Fix**: Room for a piece in the store is an owned reservation, released in a `finally`, instead of a number one function increments and another decrements. Every consequence of the old pairing was a defect, and all of them are gone with it: a failure between `#claimSlot` and `#registerPiece` — a disk read that throws — lost a slot for the life of the process, because there was no `try/finally` anywhere on that path; `put`'s error path tried to correct that by guessing, and its own comments said so ("Heuristic:", "We conservatively decrement"), so it could take back a reservation belonging to a different claim in flight and let the store admit past its allowance until the next minute's revision evicted the excess. `stats()` now reports `outstanding`, and the periodic `piece-store` line prints it when it is not zero: a reservation that never comes back is invisible until the store cannot admit anything, and by then the reason is long gone.
|
|
8
|
+
- **Fix**: A claim waits on PROGRESS, not on activity. The five-second "every resident piece is pinned" error was skipped entirely while a spill was in flight or any reservation was held, so ONE lost reservation from the defect above made that error permanently unreachable and a read retried every 50 ms for the rest of the process's life — never completing, never failing, and never saying anything. The clock now restarts when something actually moves (a piece admitted, a spill finished, a pin released) and the error is raised when nothing has moved for five seconds, whatever is nominally in flight.
|
|
9
|
+
- **Fix**: The wait between attempts allocates once instead of once per pending spill. It attached a fresh pair of handlers to EVERY spill in flight on EVERY attempt and left an uncancelled timer behind each time, so a claim that could not be satisfied allocated in proportion to attempts times spills. One idempotent handler now, its timer cleared when it is woken, and a settling spill wakes the store itself — which is where that belonged, and which `#claimSlotOnce`'s own spills were not doing at all.
|
|
10
|
+
- **Fix**: Closing or destroying the store wakes whoever is waiting for room, and a claim checks for a closed store on every attempt. Callers inside `#claimSlot` were simply left asleep for ever.
|
|
11
|
+
- **Fix**: A piece handed back to memory while its own spill is still being written no longer reappears on disk. `DiskTier.write` records the index when it COMPLETES, and `put` called `forget` before that, so the completing write put the stale copy back and a later read returned bytes from before the rewrite. `put` now waits for that piece's spill to finish before forgetting it.
|
|
12
|
+
- **Fix**: A spill that fails while the allowance is being lowered is counted (`spill-failures` in the store's line) instead of thrown into nothing. Nobody awaits those spills, so the rethrow was an unhandled rejection, and an unhandled rejection in the torrent worker ends the thread — a second way to lose the torrent client on top of the one being investigated.
|
|
13
|
+
- **Fix**: Two concurrent revivals of the same piece no longer each allocate and register a buffer for it, leaving whoever holds the first reading memory the store has stopped tracking. The second finds the piece already back and uses it.
|
|
14
|
+
- **Fix**: `PieceLru` follows the store's live allowance. It was built with the capacity the store was created with and never revised, so `isFull()` answered against a number that had stopped being the limit — dormant only because nothing calls it today.
|
|
15
|
+
- **Chore**: The store accepts an injected disk tier so a test can hold a write open or make one fail on purpose. Four of the defects above live in what happens when the disk does not answer at once, and none of them was reachable from outside before (`test/piece-store-reservations.test.js`).
|
|
16
|
+
|
|
17
|
+
All eight were found by reading the file after the torrent worker was killed by its own JS heap limit on 2026-08-31 (`research/worker-heap-oom-2026-08-31.md`, §5). None of them is proven to be that growth; they are what the reading found, and each is a defect in its own right.
|
|
18
|
+
|
|
1
19
|
## 2.64.5
|
|
2
20
|
|
|
3
21
|
- **Fix**: A rung measured at 0.007x is no longer kept just because it is on screen. The `playingHeight` exemption is now checked after the `measured < 1` withdrawal, so a 4K HEVC transcode at 0.007x on a CM4 (field 2026-08-31, 0.1x at 23:45 and 0.007x at 06:57, 0.04s buffered) is withdrawn and the offer can become empty instead of stalling the viewer with no way to downgrade. `ownHeight` is kept only for a copied source (`!transcodeVideo`), not for a re-encode already running at 0.18x.
|
package/package.json
CHANGED
package/server.js
CHANGED
|
@@ -116,7 +116,7 @@ export async function startProxyServer({
|
|
|
116
116
|
// idled. Serving a segment shared that thread, so reading an already-finished
|
|
117
117
|
// 10 MB file took 12-23 s against 125 ms to hand it to the channel. The
|
|
118
118
|
// adapter keeps TorrentPool's interface, so nothing downstream changed.
|
|
119
|
-
const torrentPool = new WorkerTorrentPool({ maxDiskBytes, memoryBytes, onSubtitleCues });
|
|
119
|
+
const torrentPool = new WorkerTorrentPool({ maxDiskBytes, memoryBytes, stateDir, onSubtitleCues });
|
|
120
120
|
const selectedPort = await getPort({
|
|
121
121
|
port: buildPortCandidates(port)
|
|
122
122
|
});
|
|
@@ -19,7 +19,7 @@
|
|
|
19
19
|
* runtime already maintains.
|
|
20
20
|
*/
|
|
21
21
|
|
|
22
|
-
import { readFile, statfs } from "node:fs/promises";
|
|
22
|
+
import { readdir, readFile, rm, statfs } from "node:fs/promises";
|
|
23
23
|
import os from "node:os";
|
|
24
24
|
import v8 from "node:v8";
|
|
25
25
|
import path from "node:path";
|
|
@@ -27,6 +27,18 @@ import path from "node:path";
|
|
|
27
27
|
/** How often the reading is taken and written. */
|
|
28
28
|
export const MEMORY_REPORT_INTERVAL_MS = 60_000;
|
|
29
29
|
|
|
30
|
+
/**
|
|
31
|
+
* How often the torrent worker takes its own reading.
|
|
32
|
+
*
|
|
33
|
+
* A minute cannot see what killed it. Three times — 2026-08-30 14:00 and 23:19,
|
|
34
|
+
* 2026-08-31 13:27 — the worker's heap read 28-36 MB in one sample and the
|
|
35
|
+
* thread was dead by the sample after next, with the whole rise from 30 MB to
|
|
36
|
+
* the 2240 MB heap limit fitting inside a single gap. A second is short enough
|
|
37
|
+
* that the rise is a curve rather than a step, and the line is only WRITTEN when
|
|
38
|
+
* something moved, so a quiet session costs what it costs today.
|
|
39
|
+
*/
|
|
40
|
+
export const WORKER_MEMORY_SAMPLE_MS = 1_000;
|
|
41
|
+
|
|
30
42
|
/**
|
|
31
43
|
* What the process is holding, from the runtime's own counters.
|
|
32
44
|
*
|
|
@@ -36,16 +48,30 @@ export const MEMORY_REPORT_INTERVAL_MS = 60_000;
|
|
|
36
48
|
* since torrent pieces sit in a `SharedArrayBuffer` and segment bodies pass
|
|
37
49
|
* through buffers.
|
|
38
50
|
*
|
|
39
|
-
*
|
|
51
|
+
* `heapLimit` is this isolate's own ceiling, and it belongs beside the heap
|
|
52
|
+
* figures because it is what the runtime kills the thread for reaching — a
|
|
53
|
+
* worker created without `resourceLimits` inherits the main isolate's, 2240 MB
|
|
54
|
+
* on the addon host. Without it the log said 30 MB and gave no idea how far
|
|
55
|
+
* that was from the end.
|
|
56
|
+
*
|
|
57
|
+
* @returns {{ rss: number, heapUsed: number, heapTotal: number, external: number, arrayBuffers: number, heapLimit: number }}
|
|
40
58
|
*/
|
|
41
59
|
export function readProcessMemory() {
|
|
42
60
|
const usage = process.memoryUsage();
|
|
61
|
+
let heapLimit = 0;
|
|
62
|
+
try {
|
|
63
|
+
heapLimit = v8.getHeapStatistics().heap_size_limit ?? 0;
|
|
64
|
+
} catch {
|
|
65
|
+
// silent-ok: a missing ceiling leaves the term out, it does not cost the
|
|
66
|
+
// reading beside it.
|
|
67
|
+
}
|
|
43
68
|
return {
|
|
44
69
|
rss: usage.rss,
|
|
45
70
|
heapUsed: usage.heapUsed,
|
|
46
71
|
heapTotal: usage.heapTotal,
|
|
47
72
|
external: usage.external,
|
|
48
|
-
arrayBuffers: usage.arrayBuffers ?? 0
|
|
73
|
+
arrayBuffers: usage.arrayBuffers ?? 0,
|
|
74
|
+
heapLimit
|
|
49
75
|
};
|
|
50
76
|
}
|
|
51
77
|
|
|
@@ -198,7 +224,8 @@ export function describeMemory({
|
|
|
198
224
|
`committed ${megabytes(storeCommitted)} of ${megabytes(storeBudget)} allowed, ` +
|
|
199
225
|
`${megabytes(storeSpilled)} spilled to disk`;
|
|
200
226
|
const isolate =
|
|
201
|
-
`heap=${megabytes(usage.heapUsed)}/${megabytes(usage.heapTotal)}
|
|
227
|
+
`heap=${megabytes(usage.heapUsed)}/${megabytes(usage.heapTotal)}` +
|
|
228
|
+
`${usage.heapLimit ? ` of ${megabytes(usage.heapLimit)} allowed` : ""} ` +
|
|
202
229
|
`external=${megabytes(usage.external)} arrayBuffers=${megabytes(usage.arrayBuffers)}`;
|
|
203
230
|
if (scope === "thread") {
|
|
204
231
|
return `memory (${label || "thread"}): ${isolate}; ${storesPart}`;
|
|
@@ -213,6 +240,39 @@ export function describeMemory({
|
|
|
213
240
|
);
|
|
214
241
|
}
|
|
215
242
|
|
|
243
|
+
/**
|
|
244
|
+
* Whether this reading is worth writing down, and why.
|
|
245
|
+
*
|
|
246
|
+
* A series taken every second and printed every second is unreadable, and one
|
|
247
|
+
* printed every minute cannot see a rise that takes forty seconds. So the
|
|
248
|
+
* cadence of the READING and the cadence of the LINE are separate: the figure
|
|
249
|
+
* is taken often, and written when it has moved or when the quiet interval has
|
|
250
|
+
* passed. Pure, so the rule can be pinned without a clock.
|
|
251
|
+
*
|
|
252
|
+
* @param {Object} state
|
|
253
|
+
* @param {number} state.watchedBytes - The figure this scope is watching.
|
|
254
|
+
* @param {number} state.lastWrittenBytes
|
|
255
|
+
* @param {number} state.sinceWrittenMs
|
|
256
|
+
* @param {number} state.changeBytes - Movement that earns a line of its own.
|
|
257
|
+
* @param {number} state.quietMs - How long silence may last regardless.
|
|
258
|
+
* @returns {boolean}
|
|
259
|
+
*/
|
|
260
|
+
export function readingIsWorthWriting({
|
|
261
|
+
watchedBytes,
|
|
262
|
+
lastWrittenBytes,
|
|
263
|
+
sinceWrittenMs,
|
|
264
|
+
changeBytes,
|
|
265
|
+
quietMs
|
|
266
|
+
}) {
|
|
267
|
+
if (quietMs <= 0 || sinceWrittenMs >= quietMs) {
|
|
268
|
+
return true;
|
|
269
|
+
}
|
|
270
|
+
if (changeBytes <= 0) {
|
|
271
|
+
return false;
|
|
272
|
+
}
|
|
273
|
+
return Math.abs(watchedBytes - lastWrittenBytes) >= changeBytes;
|
|
274
|
+
}
|
|
275
|
+
|
|
216
276
|
/**
|
|
217
277
|
* Report memory on a timer until stopped.
|
|
218
278
|
*
|
|
@@ -225,7 +285,17 @@ export function describeMemory({
|
|
|
225
285
|
* @param {string} [options.label] - Which thread the isolate figures are of.
|
|
226
286
|
* @param {string} [options.diskPath] - Where pieces spill, for the free-space
|
|
227
287
|
* reading. Omitted, the disk term is left out rather than guessed.
|
|
228
|
-
* @param {number} [options.intervalMs]
|
|
288
|
+
* @param {number} [options.intervalMs] - How often the figure is READ.
|
|
289
|
+
* @param {number} [options.quietMs] - How long the line may stay silent while
|
|
290
|
+
* nothing moves. Zero writes every reading, which is what the process scope
|
|
291
|
+
* has always done.
|
|
292
|
+
* @param {number} [options.changeBytes] - Movement that earns a line before the
|
|
293
|
+
* quiet interval is up.
|
|
294
|
+
* @param {string} [options.snapshotDir] - Where heap snapshots are written.
|
|
295
|
+
* Defaults to the temporary directory, as the process scope always did.
|
|
296
|
+
* @param {number} [options.snapshotFloorBytes]
|
|
297
|
+
* @param {number} [options.snapshotGrowthBytes]
|
|
298
|
+
* @param {number} [options.keepSnapshots] - Newest to keep; zero keeps all.
|
|
229
299
|
* @returns {{ stop: () => void }}
|
|
230
300
|
*/
|
|
231
301
|
export function startMemoryReport({
|
|
@@ -234,9 +304,62 @@ export function startMemoryReport({
|
|
|
234
304
|
scope = "process",
|
|
235
305
|
label = "",
|
|
236
306
|
diskPath = "",
|
|
237
|
-
intervalMs = MEMORY_REPORT_INTERVAL_MS
|
|
307
|
+
intervalMs = MEMORY_REPORT_INTERVAL_MS,
|
|
308
|
+
quietMs = 0,
|
|
309
|
+
changeBytes = 0,
|
|
310
|
+
snapshotDir = "",
|
|
311
|
+
snapshotFloorBytes = 500 * 1024 * 1024,
|
|
312
|
+
snapshotGrowthBytes = 100 * 1024 * 1024,
|
|
313
|
+
keepSnapshots = 0
|
|
238
314
|
}) {
|
|
239
|
-
let
|
|
315
|
+
let highWater = 0;
|
|
316
|
+
let lastWrittenBytes = 0;
|
|
317
|
+
let lastWrittenAt = 0;
|
|
318
|
+
// The process watches what the kernel kills it for; a thread watches what the
|
|
319
|
+
// runtime kills IT for, which is its own heap and not the process's resident
|
|
320
|
+
// memory — the main isolate sat at 26 MB while the worker's heap climbed to
|
|
321
|
+
// its 2240 MB ceiling.
|
|
322
|
+
const watchedOf = (memory) => (scope === "thread" ? memory.heapTotal : memory.rss);
|
|
323
|
+
const slug = (label || scope).replace(/[^a-z0-9]+/gi, "-").toLowerCase();
|
|
324
|
+
const directory = snapshotDir || os.tmpdir();
|
|
325
|
+
|
|
326
|
+
/**
|
|
327
|
+
* @param {number} watchedBytes
|
|
328
|
+
* @param {string} why
|
|
329
|
+
* @returns {Promise<void>}
|
|
330
|
+
*/
|
|
331
|
+
const takeSnapshot = async (watchedBytes, why) => {
|
|
332
|
+
let snapPath = "";
|
|
333
|
+
try {
|
|
334
|
+
snapPath = path.join(directory, `heap-${slug}-${Date.now()}-${watchedBytes}.heapsnapshot`);
|
|
335
|
+
// Synchronous and proportional to the heap, so it stops this thread for
|
|
336
|
+
// as long as it takes to write. That is the price of the only reading
|
|
337
|
+
// that names what is holding the memory, and it is why it is bounded by
|
|
338
|
+
// a floor, by a growth step and by how many are kept.
|
|
339
|
+
v8.writeHeapSnapshot(snapPath);
|
|
340
|
+
log(`memory: wrote heap snapshot of the ${label || scope} to ${snapPath} (${megabytes(watchedBytes)}, ${why})`);
|
|
341
|
+
} catch {
|
|
342
|
+
// silent-ok: no snapshot is worse than a snapshot, and much better than
|
|
343
|
+
// ending the series that leads to the next one.
|
|
344
|
+
return;
|
|
345
|
+
}
|
|
346
|
+
if (keepSnapshots <= 0) {
|
|
347
|
+
return;
|
|
348
|
+
}
|
|
349
|
+
try {
|
|
350
|
+
const prefix = `heap-${slug}-`;
|
|
351
|
+
const mine = (await readdir(directory))
|
|
352
|
+
.filter((name) => name.startsWith(prefix) && name.endsWith(".heapsnapshot"))
|
|
353
|
+
.sort();
|
|
354
|
+
for (const name of mine.slice(0, Math.max(0, mine.length - keepSnapshots))) {
|
|
355
|
+
await rm(path.join(directory, name), { force: true });
|
|
356
|
+
}
|
|
357
|
+
} catch {
|
|
358
|
+
// silent-ok: a snapshot that could not be pruned is a disk-space problem
|
|
359
|
+
// for later, not a reason to lose the one just written.
|
|
360
|
+
}
|
|
361
|
+
};
|
|
362
|
+
|
|
240
363
|
const tick = async () => {
|
|
241
364
|
try {
|
|
242
365
|
let stores = [];
|
|
@@ -247,35 +370,57 @@ export function startMemoryReport({
|
|
|
247
370
|
// that matters, which is the process's own.
|
|
248
371
|
}
|
|
249
372
|
const processMemory = readProcessMemory();
|
|
373
|
+
const watched = watchedOf(processMemory);
|
|
374
|
+
const now = Date.now();
|
|
375
|
+
const write = readingIsWorthWriting({
|
|
376
|
+
watchedBytes: watched,
|
|
377
|
+
lastWrittenBytes,
|
|
378
|
+
sinceWrittenMs: lastWrittenAt === 0 ? Number.POSITIVE_INFINITY : now - lastWrittenAt,
|
|
379
|
+
changeBytes,
|
|
380
|
+
quietMs
|
|
381
|
+
});
|
|
382
|
+
|
|
383
|
+
let anonymousBytes = null;
|
|
250
384
|
if (scope === "thread") {
|
|
251
|
-
|
|
252
|
-
|
|
385
|
+
if (write) {
|
|
386
|
+
log(describeMemory({ scope, label, process: processMemory, stores }));
|
|
387
|
+
}
|
|
388
|
+
} else {
|
|
389
|
+
const { bytes, measured } = await availableMemory();
|
|
390
|
+
anonymousBytes = await readAnonymousMemory();
|
|
391
|
+
const diskFreeBytes = diskPath ? await readDiskFree(diskPath) : null;
|
|
392
|
+
if (write) {
|
|
393
|
+
log(describeMemory({
|
|
394
|
+
scope,
|
|
395
|
+
label,
|
|
396
|
+
process: processMemory,
|
|
397
|
+
availableBytes: bytes,
|
|
398
|
+
availableMeasured: measured,
|
|
399
|
+
anonymousBytes,
|
|
400
|
+
diskFreeBytes,
|
|
401
|
+
stores
|
|
402
|
+
}));
|
|
403
|
+
}
|
|
404
|
+
}
|
|
405
|
+
if (write) {
|
|
406
|
+
lastWrittenBytes = watched;
|
|
407
|
+
lastWrittenAt = now;
|
|
253
408
|
}
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
}));
|
|
267
|
-
// High-water and heap snapshot on growth — gives a file to open in Chrome DevTools
|
|
268
|
-
// when the kernel is about to kill the process. One snapshot per high-water.
|
|
269
|
-
if (processMemory.rss > highWaterRss + 100 * 1024 * 1024 && processMemory.rss > 500 * 1024 * 1024) {
|
|
270
|
-
highWaterRss = processMemory.rss;
|
|
271
|
-
try {
|
|
272
|
-
const snapPath = path.join(os.tmpdir(), `heap-${Date.now()}-${processMemory.rss}.heapsnapshot`);
|
|
273
|
-
v8.writeHeapSnapshot(snapPath);
|
|
274
|
-
log(`memory: wrote heap snapshot to ${snapPath} rss=${Math.round(processMemory.rss / (1024 * 1024))}MB anon=${anonymousBytes ? Math.round(anonymousBytes / (1024 * 1024)) : "?"}MB`);
|
|
275
|
-
} catch {}
|
|
409
|
+
|
|
410
|
+
// A snapshot per high-water, so there is a file to open when the growth
|
|
411
|
+
// has to be named rather than described.
|
|
412
|
+
//
|
|
413
|
+
// Deliberately NOT one taken as the ceiling is approached: a snapshot is
|
|
414
|
+
// written by the isolate itself and is about the size of its heap, so
|
|
415
|
+
// asking for one at 1.9 GB on a machine with 600 MB left is a good way to
|
|
416
|
+
// cause the kill being studied. Whatever holds 1.6 GB is the same thing
|
|
417
|
+
// that holds 2.2 GB, and at one reading a second no step is ever missed.
|
|
418
|
+
if (watched > highWater + snapshotGrowthBytes && watched > snapshotFloorBytes) {
|
|
419
|
+
highWater = watched;
|
|
420
|
+
await takeSnapshot(watched, "a new high-water");
|
|
276
421
|
}
|
|
277
|
-
if (anonymousBytes !== null && processMemory.rss > 800 * 1024 * 1024) {
|
|
278
|
-
log(`memory: high rss=${
|
|
422
|
+
if (scope !== "thread" && anonymousBytes !== null && processMemory.rss > 800 * 1024 * 1024) {
|
|
423
|
+
log(`memory: high rss=${megabytes(processMemory.rss)} anon=${megabytes(anonymousBytes)} heap=${megabytes(processMemory.heapUsed)} — watch for OOM`);
|
|
279
424
|
}
|
|
280
425
|
} catch {
|
|
281
426
|
// silent-ok: a reading that fails is not worth ending the series over.
|
|
@@ -64,6 +64,22 @@ export class PieceLru {
|
|
|
64
64
|
return this.#capacity;
|
|
65
65
|
}
|
|
66
66
|
|
|
67
|
+
/**
|
|
68
|
+
* Follow the store's live allowance, which moves with the machine's free
|
|
69
|
+
* memory. Without this the capacity stayed at whatever the store was created
|
|
70
|
+
* with, and {@link PieceLru#isFull} answered against a number that had not
|
|
71
|
+
* been the limit for some time.
|
|
72
|
+
*
|
|
73
|
+
* @param {number} capacity
|
|
74
|
+
* @returns {void}
|
|
75
|
+
*/
|
|
76
|
+
setCapacity(capacity) {
|
|
77
|
+
if (!Number.isInteger(capacity) || capacity < 1) {
|
|
78
|
+
return;
|
|
79
|
+
}
|
|
80
|
+
this.#capacity = capacity;
|
|
81
|
+
}
|
|
82
|
+
|
|
67
83
|
/**
|
|
68
84
|
* How many pieces are currently held by a reader.
|
|
69
85
|
*
|
|
@@ -23,6 +23,17 @@
|
|
|
23
23
|
* Evicting a piece deletes its entry and the memory is reclaimable by GC.
|
|
24
24
|
* `committed` therefore equals `resident`, not a high-water mark.
|
|
25
25
|
*
|
|
26
|
+
* **Room for a piece is an owned reservation, not a shared number.**
|
|
27
|
+
* `#claimSlot` hands back a release the caller runs in a `finally`; nothing
|
|
28
|
+
* else touches `#outstandingPieces`. It was a counter incremented in one
|
|
29
|
+
* function and decremented in another, and every consequence of that was a
|
|
30
|
+
* defect: a failure between the two lost a slot for the life of the process,
|
|
31
|
+
* `put`'s error path guessed at the correction and could take back a
|
|
32
|
+
* reservation belonging to a different claim, and one lost reservation made
|
|
33
|
+
* the five-second "everything is pinned" error permanently unreachable, so a
|
|
34
|
+
* read retried every 50 ms for ever without completing or failing. Read out of
|
|
35
|
+
* the field failure of 2026-08-31 (`research/worker-heap-oom-2026-08-31.md`).
|
|
36
|
+
*
|
|
26
37
|
* What this deliberately does NOT do is manage the disk as a cache of its own.
|
|
27
38
|
* Pieces evicted from memory are written once and read back on demand; the file
|
|
28
39
|
* is discarded whole when the torrent goes away. libtorrent 2.0 and webtor's
|
|
@@ -98,6 +109,15 @@ function availableMemorySync() {
|
|
|
98
109
|
|
|
99
110
|
const MIN_BUDGET_BYTES = 64 * 1024 * 1024;
|
|
100
111
|
const MIN_RESIDENT_PIECES = 2;
|
|
112
|
+
/**
|
|
113
|
+
* How long a claim may go without ANYTHING moving before it gives up.
|
|
114
|
+
*
|
|
115
|
+
* Measured against progress, not against activity. The earlier rule skipped
|
|
116
|
+
* this timer entirely while a spill was in flight or a reservation was held —
|
|
117
|
+
* so one reservation that was never returned made the timer unreachable and a
|
|
118
|
+
* read retried every 50 ms for the life of the process, never completing and
|
|
119
|
+
* never failing (field 2026-08-31).
|
|
120
|
+
*/
|
|
101
121
|
const PINNED_WAIT_MS = 5_000;
|
|
102
122
|
const CLAIM_RETRY_MS = 50;
|
|
103
123
|
|
|
@@ -111,8 +131,21 @@ export class SharedPieceStore {
|
|
|
111
131
|
#buffers = new Map();
|
|
112
132
|
/** @type {Map<number, Promise<void>>} */
|
|
113
133
|
#evicting = new Map();
|
|
134
|
+
/**
|
|
135
|
+
* Slots claimed but not yet filled.
|
|
136
|
+
*
|
|
137
|
+
* Handed out by {@link SharedPieceStore##claimSlot} as a release function the
|
|
138
|
+
* caller must call in a `finally`, never as a number one function increments
|
|
139
|
+
* and another decrements. The counter used to be paired across
|
|
140
|
+
* `#claimSlot`/`#registerPiece`, so any failure between the two lost a slot
|
|
141
|
+
* for the life of the process, and `put`'s error path tried to correct that
|
|
142
|
+
* by guessing — which could take back a reservation belonging to a different
|
|
143
|
+
* claim and let the store admit past its allowance.
|
|
144
|
+
*/
|
|
114
145
|
#outstandingPieces = 0;
|
|
115
146
|
#pinnedWaitStartedAt = 0;
|
|
147
|
+
/** When something last actually moved: a piece admitted, spilled or unpinned. */
|
|
148
|
+
#lastProgressAt = 0;
|
|
116
149
|
#waiters = [];
|
|
117
150
|
#lru;
|
|
118
151
|
#disk;
|
|
@@ -125,7 +158,8 @@ export class SharedPieceStore {
|
|
|
125
158
|
revivals: 0,
|
|
126
159
|
blockedByPins: 0,
|
|
127
160
|
waitedForPins: 0,
|
|
128
|
-
evictedOnRevise: 0
|
|
161
|
+
evictedOnRevise: 0,
|
|
162
|
+
spillFailures: 0
|
|
129
163
|
};
|
|
130
164
|
|
|
131
165
|
constructor(chunkLength, options = {}) {
|
|
@@ -147,7 +181,11 @@ export class SharedPieceStore {
|
|
|
147
181
|
this.#growthCeiling = this.#capacity;
|
|
148
182
|
this.#lru = new PieceLru(this.#capacity);
|
|
149
183
|
this.#name = options.name ?? "pieces";
|
|
150
|
-
|
|
184
|
+
// `options.disk` exists so a test can hold a write open or make one fail on
|
|
185
|
+
// purpose. Four of the defects fixed here live in what happens when the
|
|
186
|
+
// disk tier does not answer immediately or at all, and none of them is
|
|
187
|
+
// reachable from outside without saying so.
|
|
188
|
+
this.#disk = options.disk ?? new DiskTier({
|
|
151
189
|
directory: options.path ?? ".",
|
|
152
190
|
name: `${this.#name}.pieces`,
|
|
153
191
|
chunkLength
|
|
@@ -168,6 +206,10 @@ export class SharedPieceStore {
|
|
|
168
206
|
committedBytes: residentBytes,
|
|
169
207
|
budgetBytes: this.#growthCeiling * this.#chunkLength,
|
|
170
208
|
pinned: this.#lru.pinnedCount,
|
|
209
|
+
// Slots claimed and not yet filled. Reported because a reservation that
|
|
210
|
+
// is never returned is invisible until the store cannot admit anything,
|
|
211
|
+
// and by then the reason is long gone. At rest this is zero.
|
|
212
|
+
outstanding: this.#outstandingPieces,
|
|
171
213
|
spilled: this.#disk.size,
|
|
172
214
|
spilledBytes: this.#disk.size * this.#chunkLength,
|
|
173
215
|
...this.#counters
|
|
@@ -180,6 +222,11 @@ export class SharedPieceStore {
|
|
|
180
222
|
this.#capacity,
|
|
181
223
|
Math.max(MIN_RESIDENT_PIECES, Number.isFinite(wanted) ? wanted : this.#capacity)
|
|
182
224
|
);
|
|
225
|
+
// The LRU is told too. It was constructed with the store's original
|
|
226
|
+
// capacity and never revised, so `isFull()` answered against a number that
|
|
227
|
+
// had not been the limit for some time — dormant only because nothing calls
|
|
228
|
+
// it, which is a trap for whoever calls it next.
|
|
229
|
+
this.#lru.setCapacity(this.#growthCeiling);
|
|
183
230
|
// With per-piece buffers memory CAN be given back immediately, unlike the
|
|
184
231
|
// old growable pool. Eagerly evict excess to honour the new ceiling.
|
|
185
232
|
let evicted = 0;
|
|
@@ -197,23 +244,13 @@ export class SharedPieceStore {
|
|
|
197
244
|
this.#lru.remove(victim);
|
|
198
245
|
evicted += 1;
|
|
199
246
|
this.#counters.evictedOnRevise += 1;
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
(error) => {
|
|
208
|
-
this.#evicting.delete(victim);
|
|
209
|
-
this.#wake();
|
|
210
|
-
throw error;
|
|
211
|
-
}
|
|
212
|
-
);
|
|
213
|
-
this.#evicting.set(victim, spill);
|
|
214
|
-
}
|
|
215
|
-
if (evicted > 0) {
|
|
216
|
-
// Logged by the caller (worker) via reviseStoreBudgets, but also countable here.
|
|
247
|
+
// Nobody awaits this spill, so its failure has to end here. Rethrowing
|
|
248
|
+
// made it an unhandled rejection, and an unhandled rejection in the
|
|
249
|
+
// torrent worker ends the thread — a second way to lose the torrent
|
|
250
|
+
// client, on top of the one that already loses it.
|
|
251
|
+
void this.#spill(victim, victimBuffer).catch(() => {
|
|
252
|
+
this.#counters.spillFailures += 1;
|
|
253
|
+
});
|
|
217
254
|
}
|
|
218
255
|
return {
|
|
219
256
|
name: this.#name,
|
|
@@ -272,30 +309,102 @@ export class SharedPieceStore {
|
|
|
272
309
|
|
|
273
310
|
unpin(index) {
|
|
274
311
|
this.#lru.unpin(index);
|
|
275
|
-
this.#
|
|
312
|
+
this.#noteProgress();
|
|
276
313
|
}
|
|
277
314
|
|
|
315
|
+
/**
|
|
316
|
+
* Reserve room for one piece.
|
|
317
|
+
*
|
|
318
|
+
* @returns {Promise<() => void>} The release, which the caller MUST call in a
|
|
319
|
+
* `finally`. Calling it twice is harmless.
|
|
320
|
+
*/
|
|
278
321
|
async #claimSlot() {
|
|
279
322
|
for (;;) {
|
|
323
|
+
if (this.#closed) {
|
|
324
|
+
throw new Error("Piece store is closed.");
|
|
325
|
+
}
|
|
280
326
|
const ok = await this.#claimSlotOnce();
|
|
281
327
|
if (ok) {
|
|
282
|
-
|
|
328
|
+
let released = false;
|
|
329
|
+
return () => {
|
|
330
|
+
if (released) {
|
|
331
|
+
return;
|
|
332
|
+
}
|
|
333
|
+
released = true;
|
|
334
|
+
this.#outstandingPieces -= 1;
|
|
335
|
+
this.#noteProgress();
|
|
336
|
+
};
|
|
283
337
|
}
|
|
284
|
-
await
|
|
285
|
-
this.#waiters.push(resolve);
|
|
286
|
-
for (const spill of this.#evicting.values()) {
|
|
287
|
-
void spill.then(() => this.#wake(), () => this.#wake());
|
|
288
|
-
}
|
|
289
|
-
const retry = setTimeout(() => this.#wake(), CLAIM_RETRY_MS);
|
|
290
|
-
retry.unref?.();
|
|
291
|
-
});
|
|
338
|
+
await this.#waitForSlot();
|
|
292
339
|
}
|
|
293
340
|
}
|
|
294
341
|
|
|
342
|
+
/**
|
|
343
|
+
* Sleep until something moves, or until the retry interval, whichever first.
|
|
344
|
+
*
|
|
345
|
+
* One handler, idempotent, and its timer is cleared when it is woken. The
|
|
346
|
+
* earlier version attached a fresh pair of handlers to EVERY pending spill on
|
|
347
|
+
* every attempt and left a timer running each time, so a claim that could not
|
|
348
|
+
* be satisfied allocated in proportion to attempts times pending spills. A
|
|
349
|
+
* settling spill now wakes the store itself, which is where that belongs.
|
|
350
|
+
*
|
|
351
|
+
* @returns {Promise<void>}
|
|
352
|
+
*/
|
|
353
|
+
#waitForSlot() {
|
|
354
|
+
return new Promise((resolve) => {
|
|
355
|
+
let settled = false;
|
|
356
|
+
/** @type {ReturnType<typeof setTimeout> | null} */
|
|
357
|
+
let retry = null;
|
|
358
|
+
const finish = () => {
|
|
359
|
+
if (settled) {
|
|
360
|
+
return;
|
|
361
|
+
}
|
|
362
|
+
settled = true;
|
|
363
|
+
if (retry !== null) {
|
|
364
|
+
clearTimeout(retry);
|
|
365
|
+
}
|
|
366
|
+
resolve();
|
|
367
|
+
};
|
|
368
|
+
this.#waiters.push(finish);
|
|
369
|
+
retry = setTimeout(finish, CLAIM_RETRY_MS);
|
|
370
|
+
retry.unref?.();
|
|
371
|
+
});
|
|
372
|
+
}
|
|
373
|
+
|
|
295
374
|
#registerPiece(index, buffer) {
|
|
296
375
|
this.#buffers.set(index, buffer);
|
|
297
376
|
this.#lru.touch(index);
|
|
298
|
-
this.#
|
|
377
|
+
this.#noteProgress();
|
|
378
|
+
}
|
|
379
|
+
|
|
380
|
+
/**
|
|
381
|
+
* Write a piece out and account for it, from the one place that does it.
|
|
382
|
+
*
|
|
383
|
+
* @param {number} index
|
|
384
|
+
* @param {SharedArrayBuffer} buffer
|
|
385
|
+
* @returns {Promise<void>}
|
|
386
|
+
*/
|
|
387
|
+
#spill(index, buffer) {
|
|
388
|
+
const bytes = Buffer.from(buffer, 0, this.#lengthOf(index));
|
|
389
|
+
const spill = this.#disk.write(index, bytes).then(
|
|
390
|
+
() => {
|
|
391
|
+
this.#counters.spills += 1;
|
|
392
|
+
this.#evicting.delete(index);
|
|
393
|
+
this.#noteProgress();
|
|
394
|
+
},
|
|
395
|
+
(error) => {
|
|
396
|
+
this.#evicting.delete(index);
|
|
397
|
+
this.#noteProgress();
|
|
398
|
+
throw error;
|
|
399
|
+
}
|
|
400
|
+
);
|
|
401
|
+
this.#evicting.set(index, spill);
|
|
402
|
+
return spill;
|
|
403
|
+
}
|
|
404
|
+
|
|
405
|
+
/** Something actually moved: wake whoever is waiting and restart the clock. */
|
|
406
|
+
#noteProgress() {
|
|
407
|
+
this.#lastProgressAt = Date.now();
|
|
299
408
|
this.#wake();
|
|
300
409
|
}
|
|
301
410
|
|
|
@@ -311,25 +420,30 @@ export class SharedPieceStore {
|
|
|
311
420
|
// Reserve before suspension so concurrent callers see the reservation.
|
|
312
421
|
if (this.#buffers.size + this.#outstandingPieces < this.#growthCeiling) {
|
|
313
422
|
this.#outstandingPieces += 1;
|
|
423
|
+
this.#pinnedWaitStartedAt = 0;
|
|
314
424
|
return true;
|
|
315
425
|
}
|
|
316
426
|
|
|
317
427
|
const victim = this.#lru.evictionCandidate();
|
|
318
428
|
if (victim === null) {
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
|
|
429
|
+
// Nothing may leave. Wait while the store is still MOVING — a spill
|
|
430
|
+
// completing, a piece admitted, a pin released — and give up when it has
|
|
431
|
+
// not moved for PINNED_WAIT_MS, whatever is nominally in flight. The old
|
|
432
|
+
// rule asked whether anything was in flight rather than whether anything
|
|
433
|
+
// had happened, which is why one lost reservation could hold a read here
|
|
434
|
+
// for ever.
|
|
322
435
|
if (this.#pinnedWaitStartedAt === 0) {
|
|
323
436
|
this.#pinnedWaitStartedAt = Date.now();
|
|
324
437
|
}
|
|
325
|
-
|
|
438
|
+
const stillFor = Date.now() - Math.max(this.#pinnedWaitStartedAt, this.#lastProgressAt);
|
|
439
|
+
if (stillFor < PINNED_WAIT_MS) {
|
|
326
440
|
this.#counters.waitedForPins += 1;
|
|
327
441
|
return false;
|
|
328
442
|
}
|
|
329
443
|
this.#pinnedWaitStartedAt = 0;
|
|
330
444
|
this.#counters.blockedByPins += 1;
|
|
331
445
|
throw new Error(
|
|
332
|
-
`Every resident piece is pinned and
|
|
446
|
+
`Every resident piece is pinned and nothing moved for ${PINNED_WAIT_MS}ms; no slot can be freed.`
|
|
333
447
|
);
|
|
334
448
|
}
|
|
335
449
|
this.#pinnedWaitStartedAt = 0;
|
|
@@ -346,24 +460,95 @@ export class SharedPieceStore {
|
|
|
346
460
|
this.#lru.remove(victim);
|
|
347
461
|
this.#outstandingPieces += 1;
|
|
348
462
|
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
(
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
}
|
|
359
|
-
);
|
|
360
|
-
this.#evicting.set(victim, spill);
|
|
361
|
-
await spill;
|
|
463
|
+
try {
|
|
464
|
+
await this.#spill(victim, victimBuffer);
|
|
465
|
+
} catch (error) {
|
|
466
|
+
// The caller never received a release for this reservation, so it is
|
|
467
|
+
// given back here rather than left outstanding for ever.
|
|
468
|
+
this.#outstandingPieces -= 1;
|
|
469
|
+
this.#noteProgress();
|
|
470
|
+
throw error;
|
|
471
|
+
}
|
|
362
472
|
// Outstanding stays +1 for the caller; the slot for the new piece is now free.
|
|
363
473
|
return true;
|
|
364
474
|
}
|
|
365
475
|
|
|
366
|
-
|
|
476
|
+
/**
|
|
477
|
+
* Bring a spilled piece back into memory, once, however many callers ask.
|
|
478
|
+
*
|
|
479
|
+
* @param {number} index
|
|
480
|
+
* @returns {Promise<SharedArrayBuffer | null>} `null` when the piece is on
|
|
481
|
+
* neither tier.
|
|
482
|
+
*/
|
|
483
|
+
async #revive(index) {
|
|
484
|
+
const spill = this.#evicting.get(index);
|
|
485
|
+
if (spill) {
|
|
486
|
+
await spill.catch(() => undefined);
|
|
487
|
+
}
|
|
488
|
+
|
|
489
|
+
if (!this.#disk.has(index)) {
|
|
490
|
+
return null;
|
|
491
|
+
}
|
|
492
|
+
|
|
493
|
+
const release = await this.#claimSlot();
|
|
494
|
+
try {
|
|
495
|
+
// Another caller may have brought it back while this one waited for a
|
|
496
|
+
// slot. Registering a second buffer for the same piece would leave
|
|
497
|
+
// whoever holds the first reading memory nothing evicts.
|
|
498
|
+
const already = this.#buffers.get(index);
|
|
499
|
+
if (already !== undefined) {
|
|
500
|
+
this.#lru.touch(index);
|
|
501
|
+
this.#counters.fromMemory += 1;
|
|
502
|
+
return already;
|
|
503
|
+
}
|
|
504
|
+
const target = new SharedArrayBuffer(this.#lengthOf(index));
|
|
505
|
+
await this.#disk.read(index, Buffer.from(target));
|
|
506
|
+
this.#registerPiece(index, target);
|
|
507
|
+
this.#counters.fromDisk += 1;
|
|
508
|
+
this.#counters.revivals += 1;
|
|
509
|
+
return target;
|
|
510
|
+
} finally {
|
|
511
|
+
release();
|
|
512
|
+
}
|
|
513
|
+
}
|
|
514
|
+
|
|
515
|
+
/**
|
|
516
|
+
* A fresh buffer holding this piece's bytes.
|
|
517
|
+
*
|
|
518
|
+
* @param {number} index
|
|
519
|
+
* @param {Uint8Array} bytes
|
|
520
|
+
* @returns {SharedArrayBuffer}
|
|
521
|
+
*/
|
|
522
|
+
#copyIntoNewBuffer(index, bytes) {
|
|
523
|
+
const length = this.#lengthOf(index);
|
|
524
|
+
const sab = new SharedArrayBuffer(length);
|
|
525
|
+
const view = Buffer.from(sab);
|
|
526
|
+
if (bytes.copy) {
|
|
527
|
+
bytes.copy(view, 0, 0, length);
|
|
528
|
+
} else {
|
|
529
|
+
view.set(bytes.subarray(0, length), 0);
|
|
530
|
+
}
|
|
531
|
+
return sab;
|
|
532
|
+
}
|
|
533
|
+
|
|
534
|
+
/**
|
|
535
|
+
* Drop the disk copy of a piece that memory now holds — after any spill of
|
|
536
|
+
* that same piece has finished.
|
|
537
|
+
*
|
|
538
|
+
* `DiskTier.write` records the index when it COMPLETES, so forgetting while a
|
|
539
|
+
* spill of that index is still running let the completing write put it back,
|
|
540
|
+
* and a later read then returned the stale bytes.
|
|
541
|
+
*
|
|
542
|
+
* @param {number} index
|
|
543
|
+
* @returns {Promise<void>}
|
|
544
|
+
*/
|
|
545
|
+
async #forgetOnDisk(index) {
|
|
546
|
+
const spill = this.#evicting.get(index);
|
|
547
|
+
if (spill) {
|
|
548
|
+
await spill.catch(() => undefined);
|
|
549
|
+
}
|
|
550
|
+
this.#disk.forget(index);
|
|
551
|
+
}
|
|
367
552
|
|
|
368
553
|
put(index, bytes, callback = () => undefined) {
|
|
369
554
|
if (this.#closed) {
|
|
@@ -371,65 +556,28 @@ export class SharedPieceStore {
|
|
|
371
556
|
return;
|
|
372
557
|
}
|
|
373
558
|
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
const view = Buffer.from(sab);
|
|
381
|
-
if (bytes.copy) {
|
|
382
|
-
bytes.copy(view, 0, 0, length);
|
|
383
|
-
} else {
|
|
384
|
-
view.set(bytes.subarray(0, length), 0);
|
|
385
|
-
}
|
|
386
|
-
this.#buffers.set(index, sab);
|
|
559
|
+
const write = async () => {
|
|
560
|
+
// Already resident: the buffer is replaced, not added, so no slot is
|
|
561
|
+
// needed and none is claimed. A fresh buffer rather than a write into the
|
|
562
|
+
// old one, so a reader holding the old reference cannot see a torn write.
|
|
563
|
+
if (this.#buffers.has(index)) {
|
|
564
|
+
this.#buffers.set(index, this.#copyIntoNewBuffer(index, bytes));
|
|
387
565
|
this.#lru.touch(index);
|
|
388
|
-
this.#
|
|
389
|
-
|
|
390
|
-
|
|
391
|
-
|
|
392
|
-
}
|
|
566
|
+
await this.#forgetOnDisk(index);
|
|
567
|
+
this.#noteProgress();
|
|
568
|
+
return;
|
|
569
|
+
}
|
|
393
570
|
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
|
|
398
|
-
|
|
399
|
-
|
|
400
|
-
bytes.copy(view, 0, 0, length);
|
|
401
|
-
} else {
|
|
402
|
-
view.set(bytes.subarray(0, length), 0);
|
|
571
|
+
const release = await this.#claimSlot();
|
|
572
|
+
try {
|
|
573
|
+
this.#registerPiece(index, this.#copyIntoNewBuffer(index, bytes));
|
|
574
|
+
await this.#forgetOnDisk(index);
|
|
575
|
+
} finally {
|
|
576
|
+
release();
|
|
403
577
|
}
|
|
404
|
-
this.#registerPiece(index, sab);
|
|
405
|
-
this.#disk.forget(index);
|
|
406
578
|
};
|
|
407
579
|
|
|
408
|
-
write().then(() => callback(null), (error) =>
|
|
409
|
-
// If claim failed, outstanding was already incremented; correct it.
|
|
410
|
-
// #registerPiece decrements on success; on failure we must decrement too.
|
|
411
|
-
// But #claimSlotOnce already handles increment; we need to decrement if write threw before register.
|
|
412
|
-
// Easiest: if error and outstanding still +1 and piece not registered, decrement.
|
|
413
|
-
if (error) {
|
|
414
|
-
// If we reserved but never registered, outstanding is still +1.
|
|
415
|
-
// Check if piece is not in map and we have outstanding.
|
|
416
|
-
if (!this.#buffers.has(index) && this.#outstandingPieces > 0) {
|
|
417
|
-
// Only decrement if the failure happened before register.
|
|
418
|
-
// Heuristic: if error message is pinned exhaustion, it came from claimSlotOnce which did not increment? Actually claimSlotOnce increments only on success/eviction.
|
|
419
|
-
// For pinned error, outstanding was not incremented? Let's handle: claimSlot throws before increment? No, it throws after check, without increment.
|
|
420
|
-
// So only failures after claim (disk write etc) need decrement — those have outstanding +1.
|
|
421
|
-
// We conservatively decrement if outstanding >0 and piece not registered.
|
|
422
|
-
// But to avoid double-decrement we check if this specific write's outstanding is still held.
|
|
423
|
-
// Simple: decrement if outstanding >0 and piece not in map, and the error is not the pinned throw's pre-increment case.
|
|
424
|
-
// The pinned throw does not increment, so outstanding is 0 there.
|
|
425
|
-
if (this.#outstandingPieces > 0) {
|
|
426
|
-
this.#outstandingPieces -= 1;
|
|
427
|
-
this.#wake();
|
|
428
|
-
}
|
|
429
|
-
}
|
|
430
|
-
}
|
|
431
|
-
callback(error);
|
|
432
|
-
});
|
|
580
|
+
write().then(() => callback(null), (error) => callback(error));
|
|
433
581
|
}
|
|
434
582
|
|
|
435
583
|
get(index, options, callback) {
|
|
@@ -451,28 +599,14 @@ export class SharedPieceStore {
|
|
|
451
599
|
if (buffer !== undefined) {
|
|
452
600
|
this.#lru.touch(index);
|
|
453
601
|
this.#counters.fromMemory += 1;
|
|
454
|
-
|
|
455
|
-
return Buffer.from(view);
|
|
456
|
-
}
|
|
457
|
-
|
|
458
|
-
const spill = this.#evicting.get(index);
|
|
459
|
-
if (spill) {
|
|
460
|
-
await spill.catch(() => undefined);
|
|
602
|
+
return Buffer.from(Buffer.from(buffer, offset, length));
|
|
461
603
|
}
|
|
462
604
|
|
|
463
|
-
|
|
605
|
+
const revived = await this.#revive(index);
|
|
606
|
+
if (revived === null) {
|
|
464
607
|
throw new Error(`Piece ${index} is not in the store.`);
|
|
465
608
|
}
|
|
466
|
-
|
|
467
|
-
await this.#claimSlot();
|
|
468
|
-
const targetSab = new SharedArrayBuffer(pieceLength);
|
|
469
|
-
const target = Buffer.from(targetSab);
|
|
470
|
-
await this.#disk.read(index, target);
|
|
471
|
-
this.#registerPiece(index, targetSab);
|
|
472
|
-
this.#counters.fromDisk += 1;
|
|
473
|
-
this.#counters.revivals += 1;
|
|
474
|
-
const view = Buffer.from(targetSab, offset, length);
|
|
475
|
-
return Buffer.from(view);
|
|
609
|
+
return Buffer.from(Buffer.from(revived, offset, length));
|
|
476
610
|
};
|
|
477
611
|
|
|
478
612
|
fetch().then((bytes) => done(null, bytes), (error) => done(error));
|
|
@@ -517,30 +651,20 @@ export class SharedPieceStore {
|
|
|
517
651
|
return this.locate(index);
|
|
518
652
|
}
|
|
519
653
|
|
|
520
|
-
const
|
|
521
|
-
if (
|
|
522
|
-
await spill.catch(() => undefined);
|
|
523
|
-
}
|
|
524
|
-
|
|
525
|
-
if (!this.#disk.has(index)) {
|
|
654
|
+
const revived = await this.#revive(index);
|
|
655
|
+
if (revived === null) {
|
|
526
656
|
return null;
|
|
527
657
|
}
|
|
528
|
-
|
|
529
|
-
const pieceLength = this.#lengthOf(index);
|
|
530
|
-
await this.#claimSlot();
|
|
531
|
-
const targetSab = new SharedArrayBuffer(pieceLength);
|
|
532
|
-
const target = Buffer.from(targetSab);
|
|
533
|
-
await this.#disk.read(index, target);
|
|
534
|
-
this.#registerPiece(index, targetSab);
|
|
535
|
-
this.#counters.fromDisk += 1;
|
|
536
|
-
this.#counters.revivals += 1;
|
|
537
|
-
return this.locate(index);
|
|
658
|
+
return { buffer: revived, offset: 0, length: this.#lengthOf(index) };
|
|
538
659
|
}
|
|
539
660
|
|
|
540
661
|
close(callback = () => undefined) {
|
|
541
662
|
this.#closed = true;
|
|
542
663
|
liveStores.delete(this);
|
|
543
664
|
this.#buffers.clear();
|
|
665
|
+
// Whoever is waiting for a slot is woken and finds the store closed, which
|
|
666
|
+
// is an error they can report. Left asleep they simply never returned.
|
|
667
|
+
this.#wake();
|
|
544
668
|
this.#disk.close().then(() => callback(null), (error) => callback(error));
|
|
545
669
|
}
|
|
546
670
|
|
|
@@ -548,6 +672,7 @@ export class SharedPieceStore {
|
|
|
548
672
|
this.#closed = true;
|
|
549
673
|
liveStores.delete(this);
|
|
550
674
|
this.#buffers.clear();
|
|
675
|
+
this.#wake();
|
|
551
676
|
this.#disk.destroy().then(() => callback(null), (error) => callback(error));
|
|
552
677
|
}
|
|
553
678
|
}
|
|
@@ -67,11 +67,15 @@ export class TorrentWorkerClient {
|
|
|
67
67
|
#onSubtitleCues;
|
|
68
68
|
|
|
69
69
|
/**
|
|
70
|
-
* @param {{ maxDiskBytes?: number, memoryBytes?: number, onSubtitleCues?: (event: object) => void }} [options]
|
|
70
|
+
* @param {{ maxDiskBytes?: number, memoryBytes?: number, stateDir?: string, onSubtitleCues?: (event: object) => void }} [options]
|
|
71
71
|
*/
|
|
72
|
-
constructor({ maxDiskBytes, memoryBytes, onSubtitleCues } = {}) {
|
|
72
|
+
constructor({ maxDiskBytes, memoryBytes, stateDir, onSubtitleCues } = {}) {
|
|
73
73
|
this.#worker = new Worker(fileURLToPath(WORKER_URL), {
|
|
74
|
-
|
|
74
|
+
// `stateDir` travels because the worker writes heap snapshots of its own
|
|
75
|
+
// isolate there. It cannot choose a directory any other way: a worker may
|
|
76
|
+
// not change the process's working directory, and the isolate that has
|
|
77
|
+
// died three times is the one no snapshot has ever been taken of.
|
|
78
|
+
workerData: { maxDiskBytes, memoryBytes, stateDir }
|
|
75
79
|
});
|
|
76
80
|
this.#caller = createCaller(this.#worker);
|
|
77
81
|
this.#onSubtitleCues = onSubtitleCues ?? (() => undefined);
|
|
@@ -37,7 +37,7 @@ export class WorkerTorrentPool {
|
|
|
37
37
|
#torrents = new Map();
|
|
38
38
|
|
|
39
39
|
/**
|
|
40
|
-
* @param {{ maxDiskBytes?: number, memoryBytes?: number }} [options]
|
|
40
|
+
* @param {{ maxDiskBytes?: number, memoryBytes?: number, stateDir?: string }} [options]
|
|
41
41
|
*/
|
|
42
42
|
constructor(options = {}) {
|
|
43
43
|
this.#client = new TorrentWorkerClient(options);
|
|
@@ -29,7 +29,7 @@ import { createFileClaims } from "./file-claims.js";
|
|
|
29
29
|
import { readFragments, supplyFiguresFor } from "./piece-reader.js";
|
|
30
30
|
import { cuesHeldFor, declaredSubtitleTracksOf, subtitleTracksOf, warmSubtitleCues } from "./subtitle-cues.js";
|
|
31
31
|
import { Command, Event } from "./protocol.js";
|
|
32
|
-
import { startMemoryReport } from "../memory-report.js";
|
|
32
|
+
import { startMemoryReport, WORKER_MEMORY_SAMPLE_MS } from "../memory-report.js";
|
|
33
33
|
|
|
34
34
|
// Imported dynamically, and that is load-bearing: static imports are RESOLVED
|
|
35
35
|
// during linking, before any module body runs, so a statically imported pool
|
|
@@ -485,11 +485,25 @@ parentPort.on("message", async (message) => {
|
|
|
485
485
|
// a `SharedArrayBuffer` allocated here, so the main thread's counters cannot
|
|
486
486
|
// see it however carefully they are read — which is half the reason 650 MB of a
|
|
487
487
|
// 893 MB process had no explanation on 2026-08-28 (roadmap item 2).
|
|
488
|
+
// A second between readings, a minute between lines unless the heap moved by
|
|
489
|
+
// 25 MB, and a heap snapshot of THIS isolate on every new high-water above
|
|
490
|
+
// 400 MB. Three deaths — 2026-08-30 14:00 and 23:19, and
|
|
491
|
+
// 2026-08-31 13:27 — went from a 30 MB heap to the 2240 MB ceiling inside one
|
|
492
|
+
// sixty-second gap, and the only snapshots ever written were of the main
|
|
493
|
+
// isolate, whose heap is 26 MB. So the isolate that dies has never once been
|
|
494
|
+
// looked at (roadmap item 2, `research/worker-heap-oom-2026-08-31.md`).
|
|
488
495
|
startMemoryReport({
|
|
489
496
|
log,
|
|
490
497
|
readStores: collectStoreStats,
|
|
491
498
|
scope: "thread",
|
|
492
|
-
label: "torrent worker"
|
|
499
|
+
label: "torrent worker",
|
|
500
|
+
intervalMs: WORKER_MEMORY_SAMPLE_MS,
|
|
501
|
+
quietMs: 60_000,
|
|
502
|
+
changeBytes: 25 * 1024 * 1024,
|
|
503
|
+
snapshotDir: workerData?.stateDir || undefined,
|
|
504
|
+
snapshotFloorBytes: 400 * 1024 * 1024,
|
|
505
|
+
snapshotGrowthBytes: 400 * 1024 * 1024,
|
|
506
|
+
keepSnapshots: 3
|
|
493
507
|
});
|
|
494
508
|
|
|
495
509
|
const STORE_REPORT_INTERVAL_MS = 60_000;
|
|
@@ -518,7 +532,7 @@ setInterval(() => {
|
|
|
518
532
|
}
|
|
519
533
|
}
|
|
520
534
|
for (const stats of collectStoreStats()) {
|
|
521
|
-
const signature = `${stats.fromMemory}/${stats.fromDisk}/${stats.spills}/${stats.revivals}/${stats.blockedByPins}/${stats.evictedOnRevise}`;
|
|
535
|
+
const signature = `${stats.fromMemory}/${stats.fromDisk}/${stats.spills}/${stats.revivals}/${stats.blockedByPins}/${stats.evictedOnRevise}/${stats.spillFailures}`;
|
|
522
536
|
if (lastReported.get(stats.name) === signature) {
|
|
523
537
|
continue;
|
|
524
538
|
}
|
|
@@ -535,7 +549,9 @@ setInterval(() => {
|
|
|
535
549
|
`pinned=${stats.pinned} spilled=${stats.spilled} reads=${reads} (${fromMemoryShare}% from memory) ` +
|
|
536
550
|
`spills=${stats.spills} revivals=${stats.revivals}` +
|
|
537
551
|
(stats.blockedByPins > 0 ? ` blocked-by-pins=${stats.blockedByPins}` : "") +
|
|
538
|
-
(stats.evictedOnRevise > 0 ? ` evictedOnRevise=${stats.evictedOnRevise}` : "")
|
|
552
|
+
(stats.evictedOnRevise > 0 ? ` evictedOnRevise=${stats.evictedOnRevise}` : "") +
|
|
553
|
+
(stats.spillFailures > 0 ? ` spill-failures=${stats.spillFailures}` : "") +
|
|
554
|
+
(stats.outstanding > 0 ? ` outstanding=${stats.outstanding}` : "")
|
|
539
555
|
);
|
|
540
556
|
}
|
|
541
557
|
}, STORE_REPORT_INTERVAL_MS).unref();
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import test from "node:test";
|
|
2
2
|
import assert from "node:assert/strict";
|
|
3
3
|
|
|
4
|
-
import { describeMemory } from "../services/memory-report.js";
|
|
4
|
+
import { describeMemory, readingIsWorthWriting } from "../services/memory-report.js";
|
|
5
5
|
import {
|
|
6
6
|
budgetForNewStore,
|
|
7
7
|
SharedPieceStore,
|
|
@@ -154,3 +154,65 @@ test("a store's allowance follows the machine, and never passes its reservation"
|
|
|
154
154
|
await rm(directory, { recursive: true, force: true });
|
|
155
155
|
}
|
|
156
156
|
});
|
|
157
|
+
|
|
158
|
+
test("the line says how far the heap is from the ceiling it is killed for reaching", () => {
|
|
159
|
+
// The worker died three times for reaching 2240 MB while its own line said
|
|
160
|
+
// "heap=30MB" and nothing said what 30 MB was 30 MB of.
|
|
161
|
+
const line = describeMemory({
|
|
162
|
+
scope: "thread",
|
|
163
|
+
label: "torrent worker",
|
|
164
|
+
process: {
|
|
165
|
+
rss: 2549 * MEGABYTE,
|
|
166
|
+
heapUsed: 1800 * MEGABYTE,
|
|
167
|
+
heapTotal: 1904 * MEGABYTE,
|
|
168
|
+
external: 40 * MEGABYTE,
|
|
169
|
+
arrayBuffers: 20 * MEGABYTE,
|
|
170
|
+
heapLimit: 2240 * MEGABYTE
|
|
171
|
+
},
|
|
172
|
+
stores: []
|
|
173
|
+
});
|
|
174
|
+
assert.match(line, /heap=1800MB\/1904MB of 2240MB allowed/);
|
|
175
|
+
|
|
176
|
+
const withoutLimit = describeMemory({
|
|
177
|
+
scope: "thread",
|
|
178
|
+
label: "torrent worker",
|
|
179
|
+
process: { rss: 0, heapUsed: 12 * MEGABYTE, heapTotal: 20 * MEGABYTE, external: 0, arrayBuffers: 0 },
|
|
180
|
+
stores: []
|
|
181
|
+
});
|
|
182
|
+
assert.match(withoutLimit, /heap=12MB\/20MB external=/, "an unknown ceiling is left out, not printed as zero");
|
|
183
|
+
});
|
|
184
|
+
|
|
185
|
+
test("a reading is written when it moved, or when the silence has gone on long enough", () => {
|
|
186
|
+
const changeBytes = 25 * MEGABYTE;
|
|
187
|
+
const quietMs = 60_000;
|
|
188
|
+
const still = {
|
|
189
|
+
watchedBytes: 100 * MEGABYTE,
|
|
190
|
+
lastWrittenBytes: 100 * MEGABYTE,
|
|
191
|
+
sinceWrittenMs: 1_000,
|
|
192
|
+
changeBytes,
|
|
193
|
+
quietMs
|
|
194
|
+
};
|
|
195
|
+
|
|
196
|
+
assert.equal(readingIsWorthWriting(still), false, "a second of nothing is not worth a line");
|
|
197
|
+
assert.equal(
|
|
198
|
+
readingIsWorthWriting({ ...still, sinceWrittenMs: 60_000 }),
|
|
199
|
+
true,
|
|
200
|
+
"a quiet minute is still written, so a healthy session reads as it always did"
|
|
201
|
+
);
|
|
202
|
+
// The rise that killed the worker: 818 MB to 2203 MB inside one old sample.
|
|
203
|
+
assert.equal(
|
|
204
|
+
readingIsWorthWriting({ ...still, watchedBytes: 130 * MEGABYTE }),
|
|
205
|
+
true,
|
|
206
|
+
"growth of a quarter of a gigabyte cannot wait for the minute to be up"
|
|
207
|
+
);
|
|
208
|
+
assert.equal(
|
|
209
|
+
readingIsWorthWriting({ ...still, watchedBytes: 70 * MEGABYTE }),
|
|
210
|
+
true,
|
|
211
|
+
"memory given back is as interesting as memory taken"
|
|
212
|
+
);
|
|
213
|
+
assert.equal(
|
|
214
|
+
readingIsWorthWriting({ ...still, quietMs: 0 }),
|
|
215
|
+
true,
|
|
216
|
+
"no quiet interval means every reading is written, which is the process scope"
|
|
217
|
+
);
|
|
218
|
+
});
|
package/test/piece-lru.test.js
CHANGED
|
@@ -171,3 +171,21 @@ test("a reader moving its window replaces it instead of accumulating", () => {
|
|
|
171
171
|
assert.equal(lru.evictionCandidate(), 30, "the pieces already read are free again");
|
|
172
172
|
assert.equal(lru.protectedCount, 1);
|
|
173
173
|
});
|
|
174
|
+
|
|
175
|
+
test("the capacity follows the store's live allowance", () => {
|
|
176
|
+
const lru = new PieceLru(4);
|
|
177
|
+
for (const index of [40, 41]) {
|
|
178
|
+
lru.touch(index);
|
|
179
|
+
}
|
|
180
|
+
assert.equal(lru.isFull(), false, "two of four is not full");
|
|
181
|
+
|
|
182
|
+
// The store's allowance moves with the machine's free memory, and the LRU is
|
|
183
|
+
// told. Before this it kept the capacity it was built with for ever, so
|
|
184
|
+
// `isFull` answered against a number that had stopped being the limit.
|
|
185
|
+
lru.setCapacity(2);
|
|
186
|
+
assert.equal(lru.capacity, 2);
|
|
187
|
+
assert.equal(lru.isFull(), true, "two of two is full");
|
|
188
|
+
|
|
189
|
+
lru.setCapacity(0);
|
|
190
|
+
assert.equal(lru.capacity, 2, "a capacity below one is refused, not obeyed");
|
|
191
|
+
});
|
|
@@ -0,0 +1,263 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* @file Room for a piece is an owned reservation, and it comes back every time.
|
|
3
|
+
*
|
|
4
|
+
* Each case here is a defect read out of the field failure of 2026-08-31
|
|
5
|
+
* (`research/worker-heap-oom-2026-08-31.md`, §5), where the torrent worker was
|
|
6
|
+
* found holding reservations nobody could return. They all need the disk tier
|
|
7
|
+
* to be slow, or to fail, at a moment the caller chooses — which is what
|
|
8
|
+
* `options.disk` is for.
|
|
9
|
+
*/
|
|
10
|
+
|
|
11
|
+
import test from "node:test";
|
|
12
|
+
import assert from "node:assert/strict";
|
|
13
|
+
import { SharedPieceStore } from "../services/piece-store/shared-piece-store.js";
|
|
14
|
+
|
|
15
|
+
const CHUNK = 1024;
|
|
16
|
+
|
|
17
|
+
/**
|
|
18
|
+
* A disk tier the test drives: writes can be held open and either side can be
|
|
19
|
+
* made to fail.
|
|
20
|
+
*
|
|
21
|
+
* @param {{ failWrite?: boolean, failRead?: boolean, holdWrites?: boolean }} [behaviour]
|
|
22
|
+
*/
|
|
23
|
+
function makeDisk({ failWrite = false, failRead = false, holdWrites = false } = {}) {
|
|
24
|
+
const stored = new Map();
|
|
25
|
+
/** @type {Array<() => void>} */
|
|
26
|
+
const held = [];
|
|
27
|
+
return {
|
|
28
|
+
stored,
|
|
29
|
+
/** Let every held write finish. */
|
|
30
|
+
releaseWrites() {
|
|
31
|
+
const waiting = held.splice(0, held.length);
|
|
32
|
+
for (const resume of waiting) {
|
|
33
|
+
resume();
|
|
34
|
+
}
|
|
35
|
+
},
|
|
36
|
+
get heldCount() {
|
|
37
|
+
return held.length;
|
|
38
|
+
},
|
|
39
|
+
get size() {
|
|
40
|
+
return stored.size;
|
|
41
|
+
},
|
|
42
|
+
has(index) {
|
|
43
|
+
return stored.has(index);
|
|
44
|
+
},
|
|
45
|
+
async write(index, bytes) {
|
|
46
|
+
if (holdWrites) {
|
|
47
|
+
await new Promise((resolve) => held.push(resolve));
|
|
48
|
+
}
|
|
49
|
+
if (failWrite) {
|
|
50
|
+
throw new Error("the disk refused the write");
|
|
51
|
+
}
|
|
52
|
+
stored.set(index, Buffer.from(bytes));
|
|
53
|
+
},
|
|
54
|
+
async read(index, target) {
|
|
55
|
+
if (failRead) {
|
|
56
|
+
throw new Error("the disk refused the read");
|
|
57
|
+
}
|
|
58
|
+
const bytes = stored.get(index);
|
|
59
|
+
if (!bytes) {
|
|
60
|
+
throw new Error(`Piece ${index} is not on disk.`);
|
|
61
|
+
}
|
|
62
|
+
bytes.copy(target);
|
|
63
|
+
return bytes.length;
|
|
64
|
+
},
|
|
65
|
+
forget(index) {
|
|
66
|
+
stored.delete(index);
|
|
67
|
+
},
|
|
68
|
+
async close() {},
|
|
69
|
+
async destroy() {
|
|
70
|
+
stored.clear();
|
|
71
|
+
}
|
|
72
|
+
};
|
|
73
|
+
}
|
|
74
|
+
|
|
75
|
+
/**
|
|
76
|
+
* @param {object} [options]
|
|
77
|
+
* @returns {{ store: SharedPieceStore, disk: ReturnType<typeof makeDisk> }}
|
|
78
|
+
*/
|
|
79
|
+
function makeStore({ pieces = 2, totalPieces = 16, disk = makeDisk() } = {}) {
|
|
80
|
+
const store = new SharedPieceStore(CHUNK, {
|
|
81
|
+
length: CHUNK * totalPieces,
|
|
82
|
+
memoryBytes: CHUNK * pieces,
|
|
83
|
+
disk,
|
|
84
|
+
name: "test"
|
|
85
|
+
});
|
|
86
|
+
return { store, disk };
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
/**
|
|
90
|
+
* @param {number} index
|
|
91
|
+
* @returns {Buffer}
|
|
92
|
+
*/
|
|
93
|
+
function piece(index) {
|
|
94
|
+
const bytes = Buffer.allocUnsafeSlow(CHUNK);
|
|
95
|
+
bytes.fill(index % 256);
|
|
96
|
+
bytes.writeUInt32BE(index, 0);
|
|
97
|
+
return bytes;
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
const put = (store, index) =>
|
|
101
|
+
new Promise((resolve, reject) => store.put(index, piece(index), (error) => (error ? reject(error) : resolve())));
|
|
102
|
+
const get = (store, index) =>
|
|
103
|
+
new Promise((resolve, reject) =>
|
|
104
|
+
store.get(index, undefined, (error, bytes) => (error ? reject(error) : resolve(bytes)))
|
|
105
|
+
);
|
|
106
|
+
|
|
107
|
+
/**
|
|
108
|
+
* Wait until a condition holds, rather than for a chosen interval — a test that
|
|
109
|
+
* sleeps samples, it does not check (roadmap item 54).
|
|
110
|
+
*
|
|
111
|
+
* @param {() => boolean} holds
|
|
112
|
+
* @param {string} what
|
|
113
|
+
* @returns {Promise<void>}
|
|
114
|
+
*/
|
|
115
|
+
async function until(holds, what) {
|
|
116
|
+
const deadline = Date.now() + 5_000;
|
|
117
|
+
while (!holds()) {
|
|
118
|
+
if (Date.now() > deadline) {
|
|
119
|
+
throw new Error(`timed out waiting until ${what}`);
|
|
120
|
+
}
|
|
121
|
+
await new Promise((resolve) => setImmediate(resolve));
|
|
122
|
+
}
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
test("a revival that fails on the disk gives its slot back", async () => {
|
|
126
|
+
const disk = makeDisk({ failRead: true });
|
|
127
|
+
const { store } = makeStore({ pieces: 2, disk });
|
|
128
|
+
try {
|
|
129
|
+
await put(store, 0);
|
|
130
|
+
await put(store, 1);
|
|
131
|
+
await put(store, 2); // piece 0 is the least recently used, so it spills
|
|
132
|
+
|
|
133
|
+
await assert.rejects(() => get(store, 0), /refused the read/);
|
|
134
|
+
|
|
135
|
+
assert.equal(
|
|
136
|
+
store.stats().outstanding,
|
|
137
|
+
0,
|
|
138
|
+
"the slot claimed for the revival was never returned"
|
|
139
|
+
);
|
|
140
|
+
} finally {
|
|
141
|
+
await new Promise((resolve) => store.destroy(resolve));
|
|
142
|
+
}
|
|
143
|
+
});
|
|
144
|
+
|
|
145
|
+
test("a spill that fails gives back the slot the eviction claimed", async () => {
|
|
146
|
+
const disk = makeDisk({ failWrite: true });
|
|
147
|
+
const { store } = makeStore({ pieces: 2, disk });
|
|
148
|
+
try {
|
|
149
|
+
await put(store, 0);
|
|
150
|
+
await put(store, 1);
|
|
151
|
+
await assert.rejects(() => put(store, 2), /refused the write/);
|
|
152
|
+
|
|
153
|
+
assert.equal(store.stats().outstanding, 0, "the eviction kept its reservation");
|
|
154
|
+
} finally {
|
|
155
|
+
await new Promise((resolve) => store.destroy(resolve));
|
|
156
|
+
}
|
|
157
|
+
});
|
|
158
|
+
|
|
159
|
+
test("a claim that cannot be met ends in an error, not in waiting for ever", async () => {
|
|
160
|
+
const disk = makeDisk({ holdWrites: true });
|
|
161
|
+
const { store } = makeStore({ pieces: 2, disk });
|
|
162
|
+
try {
|
|
163
|
+
await put(store, 0);
|
|
164
|
+
await put(store, 1);
|
|
165
|
+
|
|
166
|
+
// Evicts one piece; its write is held, so the store has a spill in flight
|
|
167
|
+
// for as long as this test wants.
|
|
168
|
+
const spilling = put(store, 2);
|
|
169
|
+
await until(() => disk.heldCount > 0, "a write is in flight");
|
|
170
|
+
|
|
171
|
+
// Nothing left that may be evicted: one piece is being written out, the
|
|
172
|
+
// other is pinned. The old rule waited while anything was nominally in
|
|
173
|
+
// flight, which here is for ever.
|
|
174
|
+
store.pin(1);
|
|
175
|
+
await assert.rejects(() => put(store, 3), /nothing moved/);
|
|
176
|
+
|
|
177
|
+
store.unpin(1);
|
|
178
|
+
disk.releaseWrites();
|
|
179
|
+
await spilling;
|
|
180
|
+
} finally {
|
|
181
|
+
disk.releaseWrites();
|
|
182
|
+
await new Promise((resolve) => store.destroy(resolve));
|
|
183
|
+
}
|
|
184
|
+
});
|
|
185
|
+
|
|
186
|
+
test("closing the store fails whoever is waiting for room", async () => {
|
|
187
|
+
const disk = makeDisk({ holdWrites: true });
|
|
188
|
+
const { store } = makeStore({ pieces: 2, disk });
|
|
189
|
+
try {
|
|
190
|
+
await put(store, 0);
|
|
191
|
+
await put(store, 1);
|
|
192
|
+
const spilling = put(store, 2);
|
|
193
|
+
await until(() => disk.heldCount > 0, "a write is in flight");
|
|
194
|
+
store.pin(1);
|
|
195
|
+
|
|
196
|
+
const waiting = put(store, 3);
|
|
197
|
+
await until(() => store.stats().waitedForPins > 0, "the claim is waiting");
|
|
198
|
+
|
|
199
|
+
await new Promise((resolve) => store.close(resolve));
|
|
200
|
+
await assert.rejects(() => waiting, /closed/);
|
|
201
|
+
|
|
202
|
+
store.unpin(1);
|
|
203
|
+
disk.releaseWrites();
|
|
204
|
+
await spilling.catch(() => undefined);
|
|
205
|
+
} finally {
|
|
206
|
+
disk.releaseWrites();
|
|
207
|
+
}
|
|
208
|
+
});
|
|
209
|
+
|
|
210
|
+
test("a piece written back to memory is not resurrected on disk by its own spill", async () => {
|
|
211
|
+
const disk = makeDisk({ holdWrites: true });
|
|
212
|
+
const { store } = makeStore({ pieces: 3, disk });
|
|
213
|
+
try {
|
|
214
|
+
await put(store, 0);
|
|
215
|
+
await put(store, 1);
|
|
216
|
+
await put(store, 2);
|
|
217
|
+
|
|
218
|
+
// Piece 0 leaves memory because the machine's allowance fell; its write is
|
|
219
|
+
// still in flight. The allowance then recovers, so there is room again
|
|
220
|
+
// without waiting for that write.
|
|
221
|
+
store.reviseGrowthCeiling(CHUNK * 2);
|
|
222
|
+
await until(() => disk.heldCount > 0, "the spill of piece 0 is in flight");
|
|
223
|
+
store.reviseGrowthCeiling(CHUNK * 3);
|
|
224
|
+
|
|
225
|
+
// The swarm hands piece 0 back while that write is still going. The store
|
|
226
|
+
// must drop the disk copy AFTER the write has recorded it, not before —
|
|
227
|
+
// `DiskTier.write` adds the index on completion, so an early forget is
|
|
228
|
+
// undone and the next read of piece 0 comes back from before the rewrite.
|
|
229
|
+
let settled = false;
|
|
230
|
+
const rewritten = put(store, 0);
|
|
231
|
+
void rewritten.then(() => {
|
|
232
|
+
settled = true;
|
|
233
|
+
});
|
|
234
|
+
await until(() => store.stats().resident === 3, "piece 0 is back in memory");
|
|
235
|
+
assert.equal(settled, false, "the write finished without waiting for the piece's own spill");
|
|
236
|
+
|
|
237
|
+
disk.releaseWrites();
|
|
238
|
+
await rewritten;
|
|
239
|
+
|
|
240
|
+
assert.equal(disk.has(0), false, "the completing spill put the stale copy back");
|
|
241
|
+
} finally {
|
|
242
|
+
disk.releaseWrites();
|
|
243
|
+
await new Promise((resolve) => store.destroy(resolve));
|
|
244
|
+
}
|
|
245
|
+
});
|
|
246
|
+
|
|
247
|
+
test("a spill that fails while the allowance is lowered is counted, not thrown", async () => {
|
|
248
|
+
const disk = makeDisk({ failWrite: true });
|
|
249
|
+
const { store } = makeStore({ pieces: 4, disk });
|
|
250
|
+
try {
|
|
251
|
+
await put(store, 0);
|
|
252
|
+
await put(store, 1);
|
|
253
|
+
await put(store, 2);
|
|
254
|
+
await put(store, 3);
|
|
255
|
+
|
|
256
|
+
store.reviseGrowthCeiling(CHUNK * 2);
|
|
257
|
+
await until(() => store.stats().spillFailures > 0, "the failed spills are counted");
|
|
258
|
+
|
|
259
|
+
assert.equal(store.stats().outstanding, 0, "lowering the allowance held a reservation");
|
|
260
|
+
} finally {
|
|
261
|
+
await new Promise((resolve) => store.destroy(resolve));
|
|
262
|
+
}
|
|
263
|
+
});
|