@torrent-tv/proxy 2.61.0 → 2.63.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,791 +1,866 @@
1
- /**
2
- * @file Torrent pieces in shared memory, with disk as the second tier.
3
- *
4
- * Replaces the chunk store WebTorrent would otherwise build for itself. Two
5
- * reasons, one forced and one chosen.
6
- *
7
- * **Forced.** WebTorrent's own piece cache hands out the buffer it keeps using
8
- * and slices again on the next read. The torrent client now runs on its own
9
- * thread, and moving a piece to the main thread by transferring ownership
10
- * detached the cache's memory: proxy 2.9.71-2.9.73 answered every read with an
11
- * empty body (`Stream ends prematurely at 0`) and the failure was invisible,
12
- * because the error never reached the reader. Owning the memory ourselves
13
- * removes the question of whose it was.
14
- *
15
- * **Chosen.** Memory this side of the thread boundary can be *shared* memory,
16
- * which the main thread reads by offset instead of receiving as bytes — see
17
- * {@link SharedPieceStore#locate}. Measured on the field host: a piece copy
18
- * costs 3.64 ms, a read from the page cache into a buffer we own 7.63 ms, and
19
- * re-downloading a piece from the swarm ~1430 ms. So memory first, disk under
20
- * it, and never the swarm twice.
21
- *
22
- * What this deliberately does NOT do is manage the disk as a cache of its own.
23
- * Pieces evicted from memory are written once and read back on demand; the file
24
- * is discarded whole when the torrent goes away. libtorrent 2.0 and webtor's
25
- * seeder both concluded that a hand-rolled disk cache earns less than it costs,
26
- * and nothing here disagrees.
27
- */
28
-
29
- import os from "node:os";
30
- import { readFileSync } from "node:fs";
31
- import { PieceLru } from "./piece-lru.js";
32
- import { DiskTier } from "./disk-tier.js";
33
-
34
- /**
35
- * Live stores, so the worker can report on them.
36
- *
37
- * WebTorrent constructs the store itself, deep inside its own wrappers, so
38
- * there is no handle to reach for from outside. Registering here is what makes
39
- * the store's behaviour visible in the field at all — without it the first
40
- * strange case has nothing to go on.
41
- *
42
- * @type {Set<SharedPieceStore>}
43
- */
44
- const liveStores = new Set();
45
-
46
- /**
47
- * A snapshot of every live store, for logging.
48
- *
49
- * @returns {{ name: string, resident: number, capacity: number, residentBytes: number, budgetBytes: number, spilled: number, fromMemory: number, fromDisk: number, spills: number, revivals: number, blockedByPins: number }[]}
50
- */
51
- export function collectStoreStats() {
52
- return [...liveStores].map((store) => store.stats());
53
- }
54
-
55
- /**
56
- * The shared store behind a torrent, or `null` if it is not one of ours.
57
- *
58
- * WebTorrent wraps whatever store it is given — today in `ImmediateChunkStore`,
59
- * and historically in a piece cache as well — and offers no way to ask for the
60
- * innermost one. Walking the `store` chain finds it regardless of how many
61
- * wrappers there are or what order they sit in, which is sturdier than reaching
62
- * for a fixed `torrent.store.store`.
63
- *
64
- * @param {{ store?: object } | null | undefined} torrent
65
- * @returns {SharedPieceStore | null}
66
- */
67
- export function findSharedStore(torrent) {
68
- let candidate = torrent?.store;
69
- // Bounded rather than `while (candidate)`: a store that referenced itself
70
- // would otherwise hang the thread instead of failing.
71
- for (let depth = 0; candidate && depth < 8; depth += 1) {
72
- if (candidate instanceof SharedPieceStore) {
73
- return candidate;
74
- }
75
- candidate = candidate.store;
76
- }
77
- return null;
78
- }
79
-
80
- /**
81
- * Ceiling for the automatic budget, and the share of available memory it takes.
82
- *
83
- * A flat default would be a guess dressed as a decision: the proxy runs on
84
- * whatever the owner has, from a Pi to a rented box. So it is a share of what
85
- * the machine can actually give, capped.
86
- *
87
- * Two things about this were wrong until 2026-08-28, and the kernel found both.
88
- * It killed the proxy at 2.4 GB resident (`exit code 137`, no dump, `Out of
89
- * memory: Killed process ... anon-rss: 2422628kB`), on a host with under two
90
- * gigabytes to spare and an `oom_score_adj` of 200 that makes the addon the
91
- * first thing chosen.
92
- *
93
- * The first: the budget was **per torrent**, so two torrents meant two of it and
94
- * nothing anywhere asked what the process as a whole was holding. It is shared
95
- * now — {@link budgetForNewStore} divides what is allowed between the stores
96
- * that exist.
97
- *
98
- * The second: it was a share of `os.freemem()`, which on Linux counts only the
99
- * pages free at that instant while the kernel deliberately keeps that number low
100
- * by filling the rest with reclaimable cache. The kernel publishes its own
101
- * estimate of what an allocation could obtain — `MemAvailable` — and that is the
102
- * quantity to divide.
103
- */
104
- const MEMORY_BUDGET_CEILING_BYTES = 512 * 1024 * 1024;
105
- const AVAILABLE_MEMORY_SHARE = 0.25;
106
-
107
- /**
108
- * What all torrent stores together may hold, in bytes.
109
- *
110
- * Sampled when a store is made rather than kept, because what the machine has
111
- * to spare is not ours to predict: another container starting is as much a
112
- * change as another viewer arriving.
113
- *
114
- * @param {number} availableBytes - What the machine can still give out.
115
- * @returns {number}
116
- */
117
- export function totalStoreBudgetBytes(availableBytes) {
118
- const share = Math.floor(Math.max(availableBytes, 0) * AVAILABLE_MEMORY_SHARE);
119
- return Math.max(MIN_BUDGET_BYTES, Math.min(MEMORY_BUDGET_CEILING_BYTES, share));
120
- }
121
-
122
- /**
123
- * One store's share of the whole, given how many stores there will be.
124
- *
125
- * Divided rather than handed out whole: the failure this replaces is several
126
- * stores each taking the maximum. The floor is what a store needs to work at
127
- * all — below it the store thrashes to disk and the viewer pays for it — so a
128
- * proxy serving many torrents at once on a small machine will exceed the total,
129
- * and that is deliberate: refusing to serve is worse, and the memory report now
130
- * says plainly what is being held.
131
- *
132
- * @param {number} availableBytes
133
- * @param {number} storeCount - Stores that will exist, this one included.
134
- * @returns {number}
135
- */
136
- export function budgetForNewStore(availableBytes, storeCount) {
137
- const total = totalStoreBudgetBytes(availableBytes);
138
- const shares = Math.max(1, Math.floor(storeCount));
139
- return Math.max(MIN_BUDGET_BYTES, Math.floor(total / shares));
140
- }
141
-
142
- /**
143
- * Budget for one torrent's resident pieces when the caller names none.
144
- *
145
- * @returns {number}
146
- */
147
- function defaultMemoryBytes() {
148
- return budgetForNewStore(availableMemorySync(), liveStores.size + 1);
149
- }
150
-
151
- /**
152
- * What the machine can still give out, without waiting on a file read.
153
- *
154
- * The store is constructed synchronously, so the asynchronous reading in
155
- * `services/memory-report.js` cannot be used here. `MemAvailable` is read from
156
- * `/proc` with a blocking read, which is a few microseconds on a pseudo-file,
157
- * and `os.freemem()` remains the answer where `/proc` is not there.
158
- *
159
- * @returns {number}
160
- */
161
- function availableMemorySync() {
162
- try {
163
- const text = readFileSync("/proc/meminfo", "utf8");
164
- const match = /^MemAvailable:\s+(\d+)\s+kB$/m.exec(text);
165
- if (match) {
166
- return Number(match[1]) * 1024;
167
- }
168
- } catch {
169
- // silent-ok: not Linux, or /proc is not mounted.
170
- }
171
- return os.freemem();
172
- }
173
-
174
- /** Floor for the automatic budget — below this the store thrashes to disk. */
175
- const MIN_BUDGET_BYTES = 64 * 1024 * 1024;
176
- /**
177
- * Never keep fewer than this many pieces resident, whatever the budget says.
178
- *
179
- * Two is the smallest workable number rather than a round one: a piece being
180
- * read holds its slot, so a second slot must exist for the next piece to land
181
- * in. With one, a single reader would deadlock the store against itself.
182
- */
183
- const MIN_RESIDENT_PIECES = 2;
184
- /**
185
- * How long a caller waits for a pinned piece to be released before the store
186
- * calls it a deadlock. A pin lasts one read of one piece — milliseconds — so
187
- * anything approaching this is a reader waiting for itself.
188
- */
189
- const PINNED_WAIT_MS = 5_000;
190
- /** How often a wait for a slot looks again when no event is due to wake it. */
191
- const CLAIM_RETRY_MS = 50;
192
-
193
- /**
194
- * A chunk store holding pieces in a `SharedArrayBuffer`, spilling to disk.
195
- *
196
- * Implements the `abstract-chunk-store` shape WebTorrent expects — `put`,
197
- * `get`, `close`, `destroy` — plus {@link locate}, {@link pin} and
198
- * {@link unpin}, which is how the main thread reads a piece without it being
199
- * copied or moved.
200
- */
201
- export class SharedPieceStore {
202
- #chunkLength;
203
- #lastChunkLength;
204
- #lastChunkIndex;
205
- #capacity;
206
- /** @type {SharedArrayBuffer} */
207
- #shared;
208
- /** @type {Buffer} A view over the whole pool, for slot arithmetic. */
209
- #pool;
210
- /** Piece index slot number. */
211
- #slotOf = new Map();
212
- /** Slot numbers not currently holding a piece. */
213
- #freeSlots = [];
214
- /**
215
- * Pieces being written out to disk right now: index that write.
216
- *
217
- * Such a piece is in neither place — its slot has already been given away,
218
- * and the disk copy is not finished. A reader arriving in that window must
219
- * wait for the write instead of concluding the piece is gone.
220
- *
221
- * @type {Map<number, Promise<void>>}
222
- */
223
- #evicting = new Map();
224
- /**
225
- * Slots handed out but not yet recorded against a piece.
226
- *
227
- * A slot is claimed before the piece is copied into it, so between those two
228
- * moments the slot belongs to nobody the books know about. Without counting
229
- * them, a burst of concurrent puts — which is the normal case, pieces arrive
230
- * from many peers at once — sees an empty eviction list and concludes the
231
- * store is exhausted, when in fact it is merely mid-flight.
232
- */
233
- #outstandingSlots = 0;
234
- /** When the wait for a pinned piece began; 0 when nothing is waiting. */
235
- #pinnedWaitStartedAt = 0;
236
- /** Resolvers waiting for a slot to become claimable. @type {(() => void)[]} */
237
- #waiters = [];
238
- #lru;
239
- #disk;
240
- /** Slots backed by memory right now; grows towards {@link capacity}. */
241
- #allocatedSlots = 0;
242
- #closed = false;
243
- #name;
244
- /**
245
- * What the store has actually been doing. Reported, not just kept: the
246
- * balance between memory and disk reads is the number that says whether the
247
- * budget is right, and it cannot be guessed from outside.
248
- */
249
- #counters = {
250
- fromMemory: 0,
251
- fromDisk: 0,
252
- spills: 0,
253
- revivals: 0,
254
- blockedByPins: 0,
255
- waitedForPins: 0
256
- };
257
-
258
- /**
259
- * @param {number} chunkLength - Piece length, and therefore the slot size.
260
- * @param {object} [options]
261
- * @param {number} [options.length] - Total torrent length, so the short last piece is sized correctly.
262
- * @param {number} [options.memoryBytes] - Budget for resident pieces.
263
- * @param {string} [options.path] - Directory for the spill file.
264
- * @param {string} [options.name] - Spill file name; must be unique per torrent.
265
- */
266
- constructor(chunkLength, options = {}) {
267
- if (!Number.isInteger(chunkLength) || chunkLength < 1) {
268
- throw new Error(`Chunk length must be a positive integer, got ${chunkLength}.`);
269
- }
270
- this.#chunkLength = chunkLength;
271
-
272
- const totalLength = Number.isFinite(options.length) ? options.length : 0;
273
- this.#lastChunkIndex = totalLength > 0 ? Math.ceil(totalLength / chunkLength) - 1 : -1;
274
- const remainder = totalLength % chunkLength;
275
- this.#lastChunkLength = remainder === 0 ? chunkLength : remainder;
276
-
277
- const memoryBytes = Number.isFinite(options.memoryBytes) && options.memoryBytes > 0
278
- ? options.memoryBytes
279
- : defaultMemoryBytes();
280
- this.#capacity = Math.max(MIN_RESIDENT_PIECES, Math.floor(memoryBytes / chunkLength));
281
-
282
- // Grows into the budget instead of taking it up front. The budget is per
283
- // torrent, so claiming all of it on `add` would charge a host for pieces
284
- // nobody has asked for — and a torrent that is merely open, or one being
285
- // probed for its codecs, needs a handful of slots, not the ceiling.
286
- this.#shared = new SharedArrayBuffer(MIN_RESIDENT_PIECES * chunkLength, {
287
- maxByteLength: this.#capacity * chunkLength
288
- });
289
- this.#pool = Buffer.from(this.#shared);
290
- for (let slot = 0; slot < MIN_RESIDENT_PIECES; slot += 1) {
291
- this.#freeSlots.push(slot);
292
- }
293
- this.#allocatedSlots = MIN_RESIDENT_PIECES;
294
- this.#lru = new PieceLru(this.#capacity);
295
- this.#name = options.name ?? "pieces";
296
- this.#disk = new DiskTier({
297
- directory: options.path ?? ".",
298
- name: `${this.#name}.pieces`,
299
- chunkLength
300
- });
301
- liveStores.add(this);
302
- }
303
-
304
- /**
305
- * What this store has been doing, for the periodic report.
306
- *
307
- * @returns {{ name: string, resident: number, capacity: number, residentBytes: number, budgetBytes: number, spilled: number, fromMemory: number, fromDisk: number, spills: number, revivals: number, blockedByPins: number }}
308
- */
309
- stats() {
310
- return {
311
- name: this.#name,
312
- resident: this.#slotOf.size,
313
- capacity: this.#capacity,
314
- residentBytes: this.#slotOf.size * this.#chunkLength,
315
- budgetBytes: this.#capacity * this.#chunkLength,
316
- pinned: this.#lru.pinnedCount,
317
- spilled: this.#disk.size,
318
- ...this.#counters
319
- };
320
- }
321
-
322
- /** `abstract-chunk-store` exposes the piece size under this name. */
323
- get chunkLength() {
324
- return this.#chunkLength;
325
- }
326
-
327
- /**
328
- * The pool itself, so another thread can map the same memory and read a piece
329
- * by the offset {@link locate} reports.
330
- *
331
- * @returns {SharedArrayBuffer}
332
- */
333
- get sharedBuffer() {
334
- return this.#shared;
335
- }
336
-
337
- /** How many pieces fit in memory at once. */
338
- get capacity() {
339
- return this.#capacity;
340
- }
341
-
342
- /** How many pieces are resident right now. */
343
- get residentCount() {
344
- return this.#slotOf.size;
345
- }
346
-
347
- /** How many pieces have been spilled to disk. */
348
- get spilledCount() {
349
- return this.#disk.size;
350
- }
351
-
352
- /**
353
- * Length of a given piece the last one is usually short.
354
- *
355
- * @param {number} index
356
- * @returns {number}
357
- */
358
- #lengthOf(index) {
359
- return index === this.#lastChunkIndex ? this.#lastChunkLength : this.#chunkLength;
360
- }
361
-
362
- /**
363
- * Where a resident piece sits in the shared pool, or `null` if it is not
364
- * resident.
365
- *
366
- * The main thread reads straight from those bytes, so callers MUST hold a pin
367
- * across the read — see {@link pin}.
368
- *
369
- * @param {number} index
370
- * @returns {{ offset: number, length: number } | null}
371
- */
372
- locate(index) {
373
- const slot = this.#slotOf.get(index);
374
- if (slot === undefined) {
375
- return null;
376
- }
377
- return { offset: slot * this.#chunkLength, length: this.#lengthOf(index) };
378
- }
379
-
380
- /**
381
- * Hold a piece in memory across a read. Nested; release with {@link unpin}.
382
- *
383
- * @param {number} index
384
- * @returns {void}
385
- */
386
- pin(index) {
387
- this.#lru.pin(index);
388
- }
389
-
390
- /**
391
- * @param {number} index
392
- * @returns {void}
393
- */
394
- unpin(index) {
395
- this.#lru.unpin(index);
396
- // A released pin can be exactly what a caller waiting for a slot needs.
397
- this.#wake();
398
- }
399
-
400
- /**
401
- * Make a slot available, spilling the least recently used piece if need be.
402
- *
403
- * @returns {Promise<number>} Slot number.
404
- */
405
- async #claimSlot() {
406
- for (;;) {
407
- const slot = await this.#claimSlotOnce();
408
- if (slot !== null) {
409
- return slot;
410
- }
411
- // Nothing claimable this instant, but work is in flight that will make a
412
- // slot claimable: a spill finishing, or a piece being written into a slot
413
- // already handed out. Wait for either and look again, rather than failing
414
- // while the store is in the middle of making room.
415
- await new Promise((resolve) => {
416
- this.#waiters.push(resolve);
417
- for (const spill of this.#evicting.values()) {
418
- void spill.then(() => this.#wake(), () => this.#wake());
419
- }
420
- // A wake is not guaranteed to come. Waiting for a spill is safe — one
421
- // is in flight and will finish — but waiting for a PIN to be released
422
- // is not: if every piece is held and nothing else is happening, there
423
- // is no event left to fire, and the deadline that gives up cannot be
424
- // reached because it is only tested inside an attempt. That is a hang,
425
- // and it hung this store's own test for the full ten minutes a run is
426
- // allowed. So the wait also re-checks on a timer.
427
- const retry = setTimeout(() => this.#wake(), CLAIM_RETRY_MS);
428
- retry.unref?.();
429
- });
430
- }
431
- }
432
-
433
- /**
434
- * Record a piece against the slot it now occupies, and let waiters retry.
435
- *
436
- * @param {number} index
437
- * @param {number} slot
438
- * @returns {void}
439
- */
440
- #registerSlot(index, slot) {
441
- this.#slotOf.set(index, slot);
442
- this.#lru.touch(index);
443
- this.#outstandingSlots -= 1;
444
- this.#wake();
445
- }
446
-
447
- /**
448
- * Release everyone waiting for a slot; each rechecks for itself.
449
- *
450
- * @returns {void}
451
- */
452
- #wake() {
453
- const waiting = this.#waiters;
454
- this.#waiters = [];
455
- for (const resolve of waiting) {
456
- resolve();
457
- }
458
- }
459
-
460
- /**
461
- * One attempt at a slot: a number, or `null` when the caller should wait for
462
- * an in-flight spill and try again.
463
- *
464
- * @returns {Promise<number | null>}
465
- */
466
- async #claimSlotOnce() {
467
- // Every slot handed out below is counted BEFORE this function can suspend.
468
- // Counting it after an `await` would leave concurrent callers — which is
469
- // how pieces actually arrive — seeing an idle store and declaring it
470
- // exhausted while its slots are already spoken for.
471
- const free = this.#freeSlots.pop();
472
- if (free !== undefined) {
473
- this.#outstandingSlots += 1;
474
- return free;
475
- }
476
-
477
- // Room left in the budget: take more memory rather than evicting. Growing
478
- // replaces the view over the pool, so every slot offset stays valid — the
479
- // bytes do not move.
480
- if (this.#allocatedSlots < this.#capacity) {
481
- this.#allocatedSlots += 1;
482
- this.#shared.grow(this.#allocatedSlots * this.#chunkLength);
483
- this.#pool = Buffer.from(this.#shared);
484
- this.#outstandingSlots += 1;
485
- return this.#allocatedSlots - 1;
486
- }
487
-
488
- const victim = this.#lru.evictionCandidate();
489
- if (victim === null) {
490
- if (this.#evicting.size > 0 || this.#outstandingSlots > 0) {
491
- return null;
492
- }
493
- // Every resident piece is being READ right now. That is not a permanent
494
- // condition: a pin lasts as long as one read of one piece, and the reader
495
- // releases it a moment later. So wait for that, exactly as the loop above
496
- // waits for a spillpins now wake the waiters.
497
- //
498
- // It became reachable when a viewer could have three readers on one file
499
- // (2026-08-15: picture, the audio track chosen and the one left behind);
500
- // failing here ended a read with zero bytes, which ffmpeg reads as the
501
- // end of the file, so every encoder died and the session answered 500 to
502
- // everything after that.
503
- //
504
- // The deadline is what keeps a genuine deadlock visible: a reader that
505
- // holds a pin while waiting for a slot would otherwise wait for itself
506
- // for ever.
507
- if (this.#pinnedWaitStartedAt === 0) {
508
- this.#pinnedWaitStartedAt = Date.now();
509
- }
510
- if (Date.now() - this.#pinnedWaitStartedAt < PINNED_WAIT_MS) {
511
- this.#counters.waitedForPins += 1;
512
- return null;
513
- }
514
- this.#pinnedWaitStartedAt = 0;
515
- this.#counters.blockedByPins += 1;
516
- throw new Error(
517
- `Every resident piece is pinned and none was released in ${PINNED_WAIT_MS}ms; no slot can be freed.`
518
- );
519
- }
520
- this.#pinnedWaitStartedAt = 0;
521
-
522
- const slot = this.#slotOf.get(victim);
523
-
524
- // Claim the victim NOW, before the write can suspend us. Picking it and
525
- // releasing it either side of an `await` lets a second claim, arriving in
526
- // that gap, pick the same victim and be handed the same slot — after which
527
- // two pieces write over each other, both fail their hash, and the torrent
528
- // downloads them again, forever. Removing it from the books first makes the
529
- // choice atomic; `#evicting` keeps readers correct in the meantime.
530
- this.#slotOf.delete(victim);
531
- this.#lru.remove(victim);
532
- this.#outstandingSlots += 1;
533
-
534
- const bytes = this.#pool.subarray(slot * this.#chunkLength, slot * this.#chunkLength + this.#lengthOf(victim));
535
- const spill = this.#disk.write(victim, bytes).then(
536
- () => {
537
- this.#counters.spills += 1;
538
- this.#evicting.delete(victim);
539
- },
540
- (error) => {
541
- this.#evicting.delete(victim);
542
- throw error;
543
- }
544
- );
545
- this.#evicting.set(victim, spill);
546
- await spill;
547
- return slot;
548
- }
549
-
550
- /**
551
- * Store a piece.
552
- *
553
- * @param {number} index
554
- * @param {Uint8Array} bytes
555
- * @param {(error?: Error | null) => void} [callback]
556
- * @returns {void}
557
- */
558
- put(index, bytes, callback = () => undefined) {
559
- if (this.#closed) {
560
- queueMicrotask(() => callback(new Error("Piece store is closed.")));
561
- return;
562
- }
563
-
564
- const existing = this.#slotOf.get(index);
565
- const write = async () => {
566
- const slot = existing ?? (await this.#claimSlot());
567
- bytes.copy
568
- ? bytes.copy(this.#pool, slot * this.#chunkLength)
569
- : this.#pool.set(bytes, slot * this.#chunkLength);
570
- if (existing === undefined) {
571
- this.#registerSlot(index, slot);
572
- } else {
573
- this.#lru.touch(index);
574
- }
575
- // A newer copy is in memory; whatever is on disk is stale.
576
- this.#disk.forget(index);
577
- };
578
-
579
- write().then(() => callback(null), (error) => callback(error));
580
- }
581
-
582
- /**
583
- * Fetch a piece, or a range within it.
584
- *
585
- * Returns a buffer of its own rather than a view into the pool: WebTorrent
586
- * keeps what it is given — to verify a hash, to serve a peer — and the slot
587
- * underneath may be reused meanwhile. The thread-crossing path avoids this
588
- * copy entirely by going through {@link locate}.
589
- *
590
- * @param {number} index
591
- * @param {{ offset?: number, length?: number } | ((error: Error | null, bytes?: Buffer) => void)} [options]
592
- * @param {(error: Error | null, bytes?: Buffer) => void} [callback]
593
- * @returns {void}
594
- */
595
- get(index, options, callback) {
596
- if (typeof options === "function") {
597
- return this.get(index, undefined, options);
598
- }
599
- const done = callback ?? (() => undefined);
600
- if (this.#closed) {
601
- queueMicrotask(() => done(new Error("Piece store is closed.")));
602
- return;
603
- }
604
-
605
- const pieceLength = this.#lengthOf(index);
606
- const offset = options?.offset ?? 0;
607
- const length = options?.length ?? pieceLength - offset;
608
-
609
- const fetch = async () => {
610
- const slot = this.#slotOf.get(index);
611
- if (slot !== undefined) {
612
- this.#lru.touch(index);
613
- this.#counters.fromMemory += 1;
614
- const start = slot * this.#chunkLength + offset;
615
- return Buffer.from(this.#pool.subarray(start, start + length));
616
- }
617
-
618
- // Mid-spill: neither in memory nor yet on disk. See `reside` — reporting
619
- // it missing here would tell WebTorrent to fetch a piece we already have.
620
- const spill = this.#evicting.get(index);
621
- if (spill) {
622
- await spill.catch(() => undefined);
623
- }
624
-
625
- if (!this.#disk.has(index)) {
626
- throw new Error(`Piece ${index} is not in the store.`);
627
- }
628
-
629
- // Bring it back into memory: it was just asked for, so it is likely to be
630
- // asked for again, and the caller may follow up with `locate`.
631
- const revived = await this.#claimSlot();
632
- const target = this.#pool.subarray(
633
- revived * this.#chunkLength,
634
- revived * this.#chunkLength + pieceLength
635
- );
636
- await this.#disk.read(index, target);
637
- this.#registerSlot(index, revived);
638
- this.#counters.fromDisk += 1;
639
- this.#counters.revivals += 1;
640
- const start = revived * this.#chunkLength + offset;
641
- return Buffer.from(this.#pool.subarray(start, start + length));
642
- };
643
-
644
- fetch().then((bytes) => done(null, bytes), (error) => done(error));
645
- }
646
-
647
- /**
648
- * Ensure a piece is in memory and say where it sits — without copying it.
649
- *
650
- * This is {@link get} minus its final copy, and it exists for exactly one
651
- * caller: the reader that hands pieces to the other thread. That thread maps
652
- * the same {@link sharedBuffer}, so an offset and a length are all it needs,
653
- * and the bytes never move. `get` cannot serve that purpose because
654
- * WebTorrent keeps what `get` returns while the slot underneath may be
655
- * reused.
656
- *
657
- * The caller MUST hold a pin across the whole read — the returned offset
658
- * stays valid only while the piece is pinned.
659
- *
660
- * @param {number} index
661
- * @returns {Promise<{ offset: number, length: number } | null>} `null` when
662
- * the store holds no such piece, in memory or on disk.
663
- */
664
- /**
665
- * Declare the pieces a reader is about to need, so eviction takes something
666
- * else while it can. Replaces that reader's previous declaration.
667
- *
668
- * @param {string|number} readerId
669
- * @param {number} from - First piece, inclusive.
670
- * @param {number} to - Last piece, inclusive.
671
- * @returns {void}
672
- */
673
- protectRange(readerId, from, to) {
674
- this.#lru.protect(readerId, from, to);
675
- }
676
-
677
- /**
678
- * Forget a reader's declaration. Call it when the reader ends.
679
- *
680
- * @param {string|number} readerId
681
- * @returns {void}
682
- */
683
- /**
684
- * The windows live readers have declared. See {@link PieceLru.protectedRanges}.
685
- *
686
- * @returns {Array<{ from: number, to: number }>}
687
- */
688
- protectedRanges() {
689
- return this.#lru.protectedRanges();
690
- }
691
-
692
- releaseProtection(readerId) {
693
- this.#lru.unprotect(readerId);
694
- }
695
-
696
- /**
697
- * Bring back into memory, in parallel and without waiting, the pieces of a
698
- * range that have been spilled to disk.
699
- *
700
- * A piece is otherwise revived only when the reader arrives at it, one at a
701
- * time and in step with decoding, so a seek backward into content already
702
- * downloaded pays a disk round trip per piece. The disk is local; the whole
703
- * window can be brought back at once while the reader is still on its first
704
- * piece.
705
- *
706
- * Bounded, because each revival needs a slot and unbounded revival of a
707
- * window larger than the store would simply thrash. Errors are swallowed: a
708
- * failed warm-up costs nothing, the reader will ask for the piece properly.
709
- *
710
- * @param {number} from - First piece, inclusive.
711
- * @param {number} to - Last piece, inclusive.
712
- * @param {number} [limit] - Most pieces to revive at once.
713
- * @returns {number} How many revivals were started.
714
- */
715
- warmRange(from, to, limit = Math.max(1, Math.floor(this.#capacity / 4))) {
716
- if (this.#closed || !Number.isInteger(from) || !Number.isInteger(to)) {
717
- return 0;
718
- }
719
- let started = 0;
720
- for (let index = from; index <= to && started < limit; index += 1) {
721
- if (this.#slotOf.has(index) || !this.#disk.has(index)) {
722
- continue;
723
- }
724
- started += 1;
725
- void this.reside(index).catch(() => undefined);
726
- }
727
- return started;
728
- }
729
-
730
- async reside(index) {
731
- if (this.#closed) {
732
- throw new Error("Piece store is closed.");
733
- }
734
-
735
- const slot = this.#slotOf.get(index);
736
- if (slot !== undefined) {
737
- this.#lru.touch(index);
738
- this.#counters.fromMemory += 1;
739
- return this.locate(index);
740
- }
741
-
742
- // Caught mid-spill: the slot is already gone, the disk copy is not there
743
- // yet. Waiting is the only correct answer — reporting it missing would make
744
- // the caller re-download a piece we are in the middle of keeping.
745
- const spill = this.#evicting.get(index);
746
- if (spill) {
747
- await spill.catch(() => undefined);
748
- }
749
-
750
- if (!this.#disk.has(index)) {
751
- return null;
752
- }
753
-
754
- const pieceLength = this.#lengthOf(index);
755
- const revived = await this.#claimSlot();
756
- const target = this.#pool.subarray(
757
- revived * this.#chunkLength,
758
- revived * this.#chunkLength + pieceLength
759
- );
760
- await this.#disk.read(index, target);
761
- this.#registerSlot(index, revived);
762
- this.#counters.fromDisk += 1;
763
- this.#counters.revivals += 1;
764
- return this.locate(index);
765
- }
766
-
767
- /**
768
- * Close the store, keeping the spill file.
769
- *
770
- * @param {(error?: Error | null) => void} [callback]
771
- * @returns {void}
772
- */
773
- close(callback = () => undefined) {
774
- this.#closed = true;
775
- liveStores.delete(this);
776
- this.#disk.close().then(() => callback(null), (error) => callback(error));
777
- }
778
-
779
- /**
780
- * Close the store and delete everything it wrote.
781
- *
782
- * @param {(error?: Error | null) => void} [callback]
783
- * @returns {void}
784
- */
785
- destroy(callback = () => undefined) {
786
- this.#closed = true;
787
- liveStores.delete(this);
788
- this.#slotOf.clear();
789
- this.#disk.destroy().then(() => callback(null), (error) => callback(error));
790
- }
791
- }
1
+ /**
2
+ * @file Torrent pieces in shared memory, with disk as the second tier.
3
+ *
4
+ * Replaces the chunk store WebTorrent would otherwise build for itself. Two
5
+ * reasons, one forced and one chosen.
6
+ *
7
+ * **Forced.** WebTorrent's own piece cache hands out the buffer it keeps using
8
+ * and slices again on the next read. The torrent client now runs on its own
9
+ * thread, and moving a piece to the main thread by transferring ownership
10
+ * detached the cache's memory: proxy 2.9.71-2.9.73 answered every read with an
11
+ * empty body (`Stream ends prematurely at 0`) and the failure was invisible,
12
+ * because the error never reached the reader. Owning the memory ourselves
13
+ * removes the question of whose it was.
14
+ *
15
+ * **Chosen.** Memory this side of the thread boundary can be *shared* memory,
16
+ * which the main thread reads by offset instead of receiving as bytes — see
17
+ * {@link SharedPieceStore#locate}. Measured on the field host: a piece copy
18
+ * costs 3.64 ms, a read from the page cache into a buffer we own 7.63 ms, and
19
+ * re-downloading a piece from the swarm ~1430 ms. So memory first, disk under
20
+ * it, and never the swarm twice.
21
+ *
22
+ * What this deliberately does NOT do is manage the disk as a cache of its own.
23
+ * Pieces evicted from memory are written once and read back on demand; the file
24
+ * is discarded whole when the torrent goes away. libtorrent 2.0 and webtor's
25
+ * seeder both concluded that a hand-rolled disk cache earns less than it costs,
26
+ * and nothing here disagrees.
27
+ */
28
+
29
+ import os from "node:os";
30
+ import { readFileSync } from "node:fs";
31
+ import { PieceLru } from "./piece-lru.js";
32
+ import { DiskTier } from "./disk-tier.js";
33
+
34
+ /**
35
+ * Live stores, so the worker can report on them.
36
+ *
37
+ * WebTorrent constructs the store itself, deep inside its own wrappers, so
38
+ * there is no handle to reach for from outside. Registering here is what makes
39
+ * the store's behaviour visible in the field at all — without it the first
40
+ * strange case has nothing to go on.
41
+ *
42
+ * @type {Set<SharedPieceStore>}
43
+ */
44
+ const liveStores = new Set();
45
+
46
+ /**
47
+ * A snapshot of every live store, for logging.
48
+ *
49
+ * @returns {{ name: string, resident: number, capacity: number, residentBytes: number, budgetBytes: number, spilled: number, fromMemory: number, fromDisk: number, spills: number, revivals: number, blockedByPins: number }[]}
50
+ */
51
+ export function collectStoreStats() {
52
+ return [...liveStores].map((store) => store.stats());
53
+ }
54
+
55
+ /**
56
+ * The shared store behind a torrent, or `null` if it is not one of ours.
57
+ *
58
+ * WebTorrent wraps whatever store it is given — today in `ImmediateChunkStore`,
59
+ * and historically in a piece cache as well — and offers no way to ask for the
60
+ * innermost one. Walking the `store` chain finds it regardless of how many
61
+ * wrappers there are or what order they sit in, which is sturdier than reaching
62
+ * for a fixed `torrent.store.store`.
63
+ *
64
+ * @param {{ store?: object } | null | undefined} torrent
65
+ * @returns {SharedPieceStore | null}
66
+ */
67
+ export function findSharedStore(torrent) {
68
+ let candidate = torrent?.store;
69
+ // Bounded rather than `while (candidate)`: a store that referenced itself
70
+ // would otherwise hang the thread instead of failing.
71
+ for (let depth = 0; candidate && depth < 8; depth += 1) {
72
+ if (candidate instanceof SharedPieceStore) {
73
+ return candidate;
74
+ }
75
+ candidate = candidate.store;
76
+ }
77
+ return null;
78
+ }
79
+
80
+ /**
81
+ * Ceiling for the automatic budget, and the share of available memory it takes.
82
+ *
83
+ * A flat default would be a guess dressed as a decision: the proxy runs on
84
+ * whatever the owner has, from a Pi to a rented box. So it is a share of what
85
+ * the machine can actually give, capped.
86
+ *
87
+ * Two things about this were wrong until 2026-08-28, and the kernel found both.
88
+ * It killed the proxy at 2.4 GB resident (`exit code 137`, no dump, `Out of
89
+ * memory: Killed process ... anon-rss: 2422628kB`), on a host with under two
90
+ * gigabytes to spare and an `oom_score_adj` of 200 that makes the addon the
91
+ * first thing chosen.
92
+ *
93
+ * The first: the budget was **per torrent**, so two torrents meant two of it and
94
+ * nothing anywhere asked what the process as a whole was holding. It is shared
95
+ * now — {@link budgetForNewStore} divides what is allowed between the stores
96
+ * that exist.
97
+ *
98
+ * The second: it was a share of `os.freemem()`, which on Linux counts only the
99
+ * pages free at that instant while the kernel deliberately keeps that number low
100
+ * by filling the rest with reclaimable cache. The kernel publishes its own
101
+ * estimate of what an allocation could obtain — `MemAvailable` — and that is the
102
+ * quantity to divide.
103
+ */
104
+ const MEMORY_BUDGET_CEILING_BYTES = 512 * 1024 * 1024;
105
+ const AVAILABLE_MEMORY_SHARE = 0.25;
106
+
107
+ /**
108
+ * What all torrent stores together may hold, in bytes.
109
+ *
110
+ * Sampled when a store is made rather than kept, because what the machine has
111
+ * to spare is not ours to predict: another container starting is as much a
112
+ * change as another viewer arriving.
113
+ *
114
+ * @param {number} availableBytes - What the machine can still give out.
115
+ * @returns {number}
116
+ */
117
+ export function totalStoreBudgetBytes(availableBytes) {
118
+ const share = Math.floor(Math.max(availableBytes, 0) * AVAILABLE_MEMORY_SHARE);
119
+ return Math.max(MIN_BUDGET_BYTES, Math.min(MEMORY_BUDGET_CEILING_BYTES, share));
120
+ }
121
+
122
+ /**
123
+ * One store's share of the whole, given how many stores there will be.
124
+ *
125
+ * Divided rather than handed out whole: the failure this replaces is several
126
+ * stores each taking the maximum. The floor is what a store needs to work at
127
+ * all — below it the store thrashes to disk and the viewer pays for it — so a
128
+ * proxy serving many torrents at once on a small machine will exceed the total,
129
+ * and that is deliberate: refusing to serve is worse, and the memory report now
130
+ * says plainly what is being held.
131
+ *
132
+ * @param {number} availableBytes
133
+ * @param {number} storeCount - Stores that will exist, this one included.
134
+ * @returns {number}
135
+ */
136
+ export function budgetForNewStore(availableBytes, storeCount) {
137
+ const total = totalStoreBudgetBytes(availableBytes);
138
+ const shares = Math.max(1, Math.floor(storeCount));
139
+ return Math.max(MIN_BUDGET_BYTES, Math.floor(total / shares));
140
+ }
141
+
142
+ /**
143
+ * Re-derive every live store's growth ceiling from what the machine has NOW.
144
+ *
145
+ * A budget settled at birth is not a budget: the machine it was taken from
146
+ * changes, and this proxy is one process among many on a host that hands its
147
+ * addons a positive `oom_score_adj`. Lowering a ceiling frees nothing that is
148
+ * already committed — the pool cannot shrink — but it stops the growth that
149
+ * would otherwise carry on into memory the machine no longer has, and sends
150
+ * those pieces to disk instead, which is what the disk tier is for.
151
+ *
152
+ * Never above the reservation each store was created with: `maxByteLength` was
153
+ * fixed from it and `grow()` cannot pass it.
154
+ *
155
+ * @returns {{ name: string, ceilingBytes: number, committedBytes: number }[]}
156
+ * What each store may now grow to, for the caller to report.
157
+ */
158
+ export function reviseStoreBudgets() {
159
+ const share = budgetForNewStore(availableMemorySync(), liveStores.size);
160
+ const revised = [];
161
+ for (const store of liveStores) {
162
+ revised.push(store.reviseGrowthCeiling(share));
163
+ }
164
+ return revised;
165
+ }
166
+
167
+ /**
168
+ * Budget for one torrent's resident pieces when the caller names none.
169
+ *
170
+ * @returns {number}
171
+ */
172
+ function defaultMemoryBytes() {
173
+ return budgetForNewStore(availableMemorySync(), liveStores.size + 1);
174
+ }
175
+
176
+ /**
177
+ * What the machine can still give out, without waiting on a file read.
178
+ *
179
+ * The store is constructed synchronously, so the asynchronous reading in
180
+ * `services/memory-report.js` cannot be used here. `MemAvailable` is read from
181
+ * `/proc` with a blocking read, which is a few microseconds on a pseudo-file,
182
+ * and `os.freemem()` remains the answer where `/proc` is not there.
183
+ *
184
+ * @returns {number}
185
+ */
186
+ function availableMemorySync() {
187
+ try {
188
+ const text = readFileSync("/proc/meminfo", "utf8");
189
+ const match = /^MemAvailable:\s+(\d+)\s+kB$/m.exec(text);
190
+ if (match) {
191
+ return Number(match[1]) * 1024;
192
+ }
193
+ } catch {
194
+ // silent-ok: not Linux, or /proc is not mounted.
195
+ }
196
+ return os.freemem();
197
+ }
198
+
199
+ /** Floor for the automatic budget — below this the store thrashes to disk. */
200
+ const MIN_BUDGET_BYTES = 64 * 1024 * 1024;
201
+ /**
202
+ * Never keep fewer than this many pieces resident, whatever the budget says.
203
+ *
204
+ * Two is the smallest workable number rather than a round one: a piece being
205
+ * read holds its slot, so a second slot must exist for the next piece to land
206
+ * in. With one, a single reader would deadlock the store against itself.
207
+ */
208
+ const MIN_RESIDENT_PIECES = 2;
209
+ /**
210
+ * How long a caller waits for a pinned piece to be released before the store
211
+ * calls it a deadlock. A pin lasts one read of one piece — milliseconds — so
212
+ * anything approaching this is a reader waiting for itself.
213
+ */
214
+ const PINNED_WAIT_MS = 5_000;
215
+ /** How often a wait for a slot looks again when no event is due to wake it. */
216
+ const CLAIM_RETRY_MS = 50;
217
+
218
+ /**
219
+ * A chunk store holding pieces in a `SharedArrayBuffer`, spilling to disk.
220
+ *
221
+ * Implements the `abstract-chunk-store` shape WebTorrent expects — `put`,
222
+ * `get`, `close`, `destroy` — plus {@link locate}, {@link pin} and
223
+ * {@link unpin}, which is how the main thread reads a piece without it being
224
+ * copied or moved.
225
+ */
226
+ export class SharedPieceStore {
227
+ #chunkLength;
228
+ #lastChunkLength;
229
+ #lastChunkIndex;
230
+ #capacity;
231
+ /**
232
+ * How many slots this store may grow into RIGHT NOW, as against the
233
+ * reservation it was born with.
234
+ *
235
+ * The budget used to be settled once, when the store was created, from the
236
+ * memory the machine had at that moment. A store opened on an idle machine
237
+ * therefore kept an idle machine's allowance for the rest of its life, and
238
+ * went on growing into it while everything else on the host competed for what
239
+ * was left — which is how an addon with a positive `oom_score_adj` becomes
240
+ * the kernel's first choice (roadmap item 2). It is revised on the same
241
+ * cadence as the memory report, within the reservation: it can never rise
242
+ * above `#capacity`, because `maxByteLength` was fixed from that and
243
+ * `grow()` cannot pass it.
244
+ */
245
+ #growthCeiling;
246
+ /** @type {SharedArrayBuffer} */
247
+ #shared;
248
+ /** @type {Buffer} A view over the whole pool, for slot arithmetic. */
249
+ #pool;
250
+ /** Piece index → slot number. */
251
+ #slotOf = new Map();
252
+ /** Slot numbers not currently holding a piece. */
253
+ #freeSlots = [];
254
+ /**
255
+ * Pieces being written out to disk right now: index → that write.
256
+ *
257
+ * Such a piece is in neither place — its slot has already been given away,
258
+ * and the disk copy is not finished. A reader arriving in that window must
259
+ * wait for the write instead of concluding the piece is gone.
260
+ *
261
+ * @type {Map<number, Promise<void>>}
262
+ */
263
+ #evicting = new Map();
264
+ /**
265
+ * Slots handed out but not yet recorded against a piece.
266
+ *
267
+ * A slot is claimed before the piece is copied into it, so between those two
268
+ * moments the slot belongs to nobody the books know about. Without counting
269
+ * them, a burst of concurrent puts — which is the normal case, pieces arrive
270
+ * from many peers at once — sees an empty eviction list and concludes the
271
+ * store is exhausted, when in fact it is merely mid-flight.
272
+ */
273
+ #outstandingSlots = 0;
274
+ /** When the wait for a pinned piece began; 0 when nothing is waiting. */
275
+ #pinnedWaitStartedAt = 0;
276
+ /** Resolvers waiting for a slot to become claimable. @type {(() => void)[]} */
277
+ #waiters = [];
278
+ #lru;
279
+ #disk;
280
+ /** Slots backed by memory right now; grows towards {@link capacity}. */
281
+ #allocatedSlots = 0;
282
+ #closed = false;
283
+ #name;
284
+ /**
285
+ * What the store has actually been doing. Reported, not just kept: the
286
+ * balance between memory and disk reads is the number that says whether the
287
+ * budget is right, and it cannot be guessed from outside.
288
+ */
289
+ #counters = {
290
+ fromMemory: 0,
291
+ fromDisk: 0,
292
+ spills: 0,
293
+ revivals: 0,
294
+ blockedByPins: 0,
295
+ waitedForPins: 0
296
+ };
297
+
298
+ /**
299
+ * @param {number} chunkLength - Piece length, and therefore the slot size.
300
+ * @param {object} [options]
301
+ * @param {number} [options.length] - Total torrent length, so the short last piece is sized correctly.
302
+ * @param {number} [options.memoryBytes] - Budget for resident pieces.
303
+ * @param {string} [options.path] - Directory for the spill file.
304
+ * @param {string} [options.name] - Spill file name; must be unique per torrent.
305
+ */
306
+ constructor(chunkLength, options = {}) {
307
+ if (!Number.isInteger(chunkLength) || chunkLength < 1) {
308
+ throw new Error(`Chunk length must be a positive integer, got ${chunkLength}.`);
309
+ }
310
+ this.#chunkLength = chunkLength;
311
+
312
+ const totalLength = Number.isFinite(options.length) ? options.length : 0;
313
+ this.#lastChunkIndex = totalLength > 0 ? Math.ceil(totalLength / chunkLength) - 1 : -1;
314
+ const remainder = totalLength % chunkLength;
315
+ this.#lastChunkLength = remainder === 0 ? chunkLength : remainder;
316
+
317
+ const memoryBytes = Number.isFinite(options.memoryBytes) && options.memoryBytes > 0
318
+ ? options.memoryBytes
319
+ : defaultMemoryBytes();
320
+ this.#capacity = Math.max(MIN_RESIDENT_PIECES, Math.floor(memoryBytes / chunkLength));
321
+
322
+ // Grows into the budget instead of taking it up front. The budget is per
323
+ // torrent, so claiming all of it on `add` would charge a host for pieces
324
+ // nobody has asked for — and a torrent that is merely open, or one being
325
+ // probed for its codecs, needs a handful of slots, not the ceiling.
326
+ this.#shared = new SharedArrayBuffer(MIN_RESIDENT_PIECES * chunkLength, {
327
+ maxByteLength: this.#capacity * chunkLength
328
+ });
329
+ this.#pool = Buffer.from(this.#shared);
330
+ for (let slot = 0; slot < MIN_RESIDENT_PIECES; slot += 1) {
331
+ this.#freeSlots.push(slot);
332
+ }
333
+ this.#allocatedSlots = MIN_RESIDENT_PIECES;
334
+ this.#growthCeiling = this.#capacity;
335
+ this.#lru = new PieceLru(this.#capacity);
336
+ this.#name = options.name ?? "pieces";
337
+ this.#disk = new DiskTier({
338
+ directory: options.path ?? ".",
339
+ name: `${this.#name}.pieces`,
340
+ chunkLength
341
+ });
342
+ liveStores.add(this);
343
+ }
344
+
345
+ /**
346
+ * What this store has been doing, for the periodic report.
347
+ *
348
+ * `residentBytes` is the pieces held right now; `committedBytes` is the
349
+ * memory this store has actually taken from the machine, and the two are not
350
+ * the same number. The pool only ever GROWS — `SharedArrayBuffer` has no
351
+ * shrink — so a piece spilled to disk returns its slot to the free list and
352
+ * its memory to nobody. Reporting only the first is what left 650 MB of a
353
+ * 893 MB process unaccounted for on 2026-08-28 while the store said "144MB"
354
+ * (roadmap item 2).
355
+ *
356
+ * @returns {{ name: string, resident: number, capacity: number, residentBytes: number, committedBytes: number, allocatedSlots: number, budgetBytes: number, spilled: number, spilledBytes: number, fromMemory: number, fromDisk: number, spills: number, revivals: number, blockedByPins: number }}
357
+ */
358
+ stats() {
359
+ return {
360
+ name: this.#name,
361
+ resident: this.#slotOf.size,
362
+ capacity: this.#growthCeiling,
363
+ residentBytes: this.#slotOf.size * this.#chunkLength,
364
+ // The high-water mark, which is what the machine has actually parted
365
+ // with. It never falls while the store lives.
366
+ allocatedSlots: this.#allocatedSlots,
367
+ committedBytes: this.#allocatedSlots * this.#chunkLength,
368
+ budgetBytes: this.#growthCeiling * this.#chunkLength,
369
+ pinned: this.#lru.pinnedCount,
370
+ spilled: this.#disk.size,
371
+ spilledBytes: this.#disk.size * this.#chunkLength,
372
+ ...this.#counters
373
+ };
374
+ }
375
+
376
+ /**
377
+ * Take a new allowance, within the reservation this store was born with.
378
+ *
379
+ * @param {number} allowedBytes
380
+ * @returns {{ name: string, ceilingBytes: number, committedBytes: number }}
381
+ */
382
+ reviseGrowthCeiling(allowedBytes) {
383
+ const wanted = Math.floor(Number(allowedBytes) / this.#chunkLength);
384
+ // Never below what a store needs to work at all — under that it thrashes to
385
+ // disk and the viewer pays for it — and never above the reservation.
386
+ this.#growthCeiling = Math.min(
387
+ this.#capacity,
388
+ Math.max(MIN_RESIDENT_PIECES, Number.isFinite(wanted) ? wanted : this.#capacity)
389
+ );
390
+ return {
391
+ name: this.#name,
392
+ ceilingBytes: this.#growthCeiling * this.#chunkLength,
393
+ committedBytes: this.#allocatedSlots * this.#chunkLength
394
+ };
395
+ }
396
+
397
+ /** `abstract-chunk-store` exposes the piece size under this name. */
398
+ get chunkLength() {
399
+ return this.#chunkLength;
400
+ }
401
+
402
+ /**
403
+ * The pool itself, so another thread can map the same memory and read a piece
404
+ * by the offset {@link locate} reports.
405
+ *
406
+ * @returns {SharedArrayBuffer}
407
+ */
408
+ get sharedBuffer() {
409
+ return this.#shared;
410
+ }
411
+
412
+ /** How many pieces fit in memory at once. */
413
+ get capacity() {
414
+ return this.#capacity;
415
+ }
416
+
417
+ /** How many pieces are resident right now. */
418
+ get residentCount() {
419
+ return this.#slotOf.size;
420
+ }
421
+
422
+ /** How many pieces have been spilled to disk. */
423
+ get spilledCount() {
424
+ return this.#disk.size;
425
+ }
426
+
427
+ /**
428
+ * Length of a given piece — the last one is usually short.
429
+ *
430
+ * @param {number} index
431
+ * @returns {number}
432
+ */
433
+ #lengthOf(index) {
434
+ return index === this.#lastChunkIndex ? this.#lastChunkLength : this.#chunkLength;
435
+ }
436
+
437
+ /**
438
+ * Where a resident piece sits in the shared pool, or `null` if it is not
439
+ * resident.
440
+ *
441
+ * The main thread reads straight from those bytes, so callers MUST hold a pin
442
+ * across the read — see {@link pin}.
443
+ *
444
+ * @param {number} index
445
+ * @returns {{ offset: number, length: number } | null}
446
+ */
447
+ locate(index) {
448
+ const slot = this.#slotOf.get(index);
449
+ if (slot === undefined) {
450
+ return null;
451
+ }
452
+ return { offset: slot * this.#chunkLength, length: this.#lengthOf(index) };
453
+ }
454
+
455
+ /**
456
+ * Hold a piece in memory across a read. Nested; release with {@link unpin}.
457
+ *
458
+ * @param {number} index
459
+ * @returns {void}
460
+ */
461
+ pin(index) {
462
+ this.#lru.pin(index);
463
+ }
464
+
465
+ /**
466
+ * @param {number} index
467
+ * @returns {void}
468
+ */
469
+ unpin(index) {
470
+ this.#lru.unpin(index);
471
+ // A released pin can be exactly what a caller waiting for a slot needs.
472
+ this.#wake();
473
+ }
474
+
475
+ /**
476
+ * Make a slot available, spilling the least recently used piece if need be.
477
+ *
478
+ * @returns {Promise<number>} Slot number.
479
+ */
480
+ async #claimSlot() {
481
+ for (;;) {
482
+ const slot = await this.#claimSlotOnce();
483
+ if (slot !== null) {
484
+ return slot;
485
+ }
486
+ // Nothing claimable this instant, but work is in flight that will make a
487
+ // slot claimable: a spill finishing, or a piece being written into a slot
488
+ // already handed out. Wait for either and look again, rather than failing
489
+ // while the store is in the middle of making room.
490
+ await new Promise((resolve) => {
491
+ this.#waiters.push(resolve);
492
+ for (const spill of this.#evicting.values()) {
493
+ void spill.then(() => this.#wake(), () => this.#wake());
494
+ }
495
+ // A wake is not guaranteed to come. Waiting for a spill is safe one
496
+ // is in flight and will finish but waiting for a PIN to be released
497
+ // is not: if every piece is held and nothing else is happening, there
498
+ // is no event left to fire, and the deadline that gives up cannot be
499
+ // reached because it is only tested inside an attempt. That is a hang,
500
+ // and it hung this store's own test for the full ten minutes a run is
501
+ // allowed. So the wait also re-checks on a timer.
502
+ const retry = setTimeout(() => this.#wake(), CLAIM_RETRY_MS);
503
+ retry.unref?.();
504
+ });
505
+ }
506
+ }
507
+
508
+ /**
509
+ * Record a piece against the slot it now occupies, and let waiters retry.
510
+ *
511
+ * @param {number} index
512
+ * @param {number} slot
513
+ * @returns {void}
514
+ */
515
+ #registerSlot(index, slot) {
516
+ this.#slotOf.set(index, slot);
517
+ this.#lru.touch(index);
518
+ this.#outstandingSlots -= 1;
519
+ this.#wake();
520
+ }
521
+
522
+ /**
523
+ * Release everyone waiting for a slot; each rechecks for itself.
524
+ *
525
+ * @returns {void}
526
+ */
527
+ #wake() {
528
+ const waiting = this.#waiters;
529
+ this.#waiters = [];
530
+ for (const resolve of waiting) {
531
+ resolve();
532
+ }
533
+ }
534
+
535
+ /**
536
+ * One attempt at a slot: a number, or `null` when the caller should wait for
537
+ * an in-flight spill and try again.
538
+ *
539
+ * @returns {Promise<number | null>}
540
+ */
541
+ async #claimSlotOnce() {
542
+ // Every slot handed out below is counted BEFORE this function can suspend.
543
+ // Counting it after an `await` would leave concurrent callers — which is
544
+ // how pieces actually arrive — seeing an idle store and declaring it
545
+ // exhausted while its slots are already spoken for.
546
+ const free = this.#freeSlots.pop();
547
+ if (free !== undefined) {
548
+ this.#outstandingSlots += 1;
549
+ return free;
550
+ }
551
+
552
+ // Room left in the budget: take more memory rather than evicting. Growing
553
+ // replaces the view over the pool, so every slot offset stays valid — the
554
+ // bytes do not move.
555
+ if (this.#allocatedSlots < this.#growthCeiling) {
556
+ this.#allocatedSlots += 1;
557
+ this.#shared.grow(this.#allocatedSlots * this.#chunkLength);
558
+ this.#pool = Buffer.from(this.#shared);
559
+ this.#outstandingSlots += 1;
560
+ return this.#allocatedSlots - 1;
561
+ }
562
+
563
+ const victim = this.#lru.evictionCandidate();
564
+ if (victim === null) {
565
+ if (this.#evicting.size > 0 || this.#outstandingSlots > 0) {
566
+ return null;
567
+ }
568
+ // Every resident piece is being READ right now. That is not a permanent
569
+ // condition: a pin lasts as long as one read of one piece, and the reader
570
+ // releases it a moment later. So wait for that, exactly as the loop above
571
+ // waits for a spill — pins now wake the waiters.
572
+ //
573
+ // It became reachable when a viewer could have three readers on one file
574
+ // (2026-08-15: picture, the audio track chosen and the one left behind);
575
+ // failing here ended a read with zero bytes, which ffmpeg reads as the
576
+ // end of the file, so every encoder died and the session answered 500 to
577
+ // everything after that.
578
+ //
579
+ // The deadline is what keeps a genuine deadlock visible: a reader that
580
+ // holds a pin while waiting for a slot would otherwise wait for itself
581
+ // for ever.
582
+ if (this.#pinnedWaitStartedAt === 0) {
583
+ this.#pinnedWaitStartedAt = Date.now();
584
+ }
585
+ if (Date.now() - this.#pinnedWaitStartedAt < PINNED_WAIT_MS) {
586
+ this.#counters.waitedForPins += 1;
587
+ return null;
588
+ }
589
+ this.#pinnedWaitStartedAt = 0;
590
+ this.#counters.blockedByPins += 1;
591
+ throw new Error(
592
+ `Every resident piece is pinned and none was released in ${PINNED_WAIT_MS}ms; no slot can be freed.`
593
+ );
594
+ }
595
+ this.#pinnedWaitStartedAt = 0;
596
+
597
+ const slot = this.#slotOf.get(victim);
598
+
599
+ // Claim the victim NOW, before the write can suspend us. Picking it and
600
+ // releasing it either side of an `await` lets a second claim, arriving in
601
+ // that gap, pick the same victim and be handed the same slot — after which
602
+ // two pieces write over each other, both fail their hash, and the torrent
603
+ // downloads them again, forever. Removing it from the books first makes the
604
+ // choice atomic; `#evicting` keeps readers correct in the meantime.
605
+ this.#slotOf.delete(victim);
606
+ this.#lru.remove(victim);
607
+ this.#outstandingSlots += 1;
608
+
609
+ const bytes = this.#pool.subarray(slot * this.#chunkLength, slot * this.#chunkLength + this.#lengthOf(victim));
610
+ const spill = this.#disk.write(victim, bytes).then(
611
+ () => {
612
+ this.#counters.spills += 1;
613
+ this.#evicting.delete(victim);
614
+ },
615
+ (error) => {
616
+ this.#evicting.delete(victim);
617
+ throw error;
618
+ }
619
+ );
620
+ this.#evicting.set(victim, spill);
621
+ await spill;
622
+ return slot;
623
+ }
624
+
625
+ /**
626
+ * Store a piece.
627
+ *
628
+ * @param {number} index
629
+ * @param {Uint8Array} bytes
630
+ * @param {(error?: Error | null) => void} [callback]
631
+ * @returns {void}
632
+ */
633
+ put(index, bytes, callback = () => undefined) {
634
+ if (this.#closed) {
635
+ queueMicrotask(() => callback(new Error("Piece store is closed.")));
636
+ return;
637
+ }
638
+
639
+ const existing = this.#slotOf.get(index);
640
+ const write = async () => {
641
+ const slot = existing ?? (await this.#claimSlot());
642
+ bytes.copy
643
+ ? bytes.copy(this.#pool, slot * this.#chunkLength)
644
+ : this.#pool.set(bytes, slot * this.#chunkLength);
645
+ if (existing === undefined) {
646
+ this.#registerSlot(index, slot);
647
+ } else {
648
+ this.#lru.touch(index);
649
+ }
650
+ // A newer copy is in memory; whatever is on disk is stale.
651
+ this.#disk.forget(index);
652
+ };
653
+
654
+ write().then(() => callback(null), (error) => callback(error));
655
+ }
656
+
657
+ /**
658
+ * Fetch a piece, or a range within it.
659
+ *
660
+ * Returns a buffer of its own rather than a view into the pool: WebTorrent
661
+ * keeps what it is given — to verify a hash, to serve a peer and the slot
662
+ * underneath may be reused meanwhile. The thread-crossing path avoids this
663
+ * copy entirely by going through {@link locate}.
664
+ *
665
+ * @param {number} index
666
+ * @param {{ offset?: number, length?: number } | ((error: Error | null, bytes?: Buffer) => void)} [options]
667
+ * @param {(error: Error | null, bytes?: Buffer) => void} [callback]
668
+ * @returns {void}
669
+ */
670
+ get(index, options, callback) {
671
+ if (typeof options === "function") {
672
+ return this.get(index, undefined, options);
673
+ }
674
+ const done = callback ?? (() => undefined);
675
+ if (this.#closed) {
676
+ queueMicrotask(() => done(new Error("Piece store is closed.")));
677
+ return;
678
+ }
679
+
680
+ const pieceLength = this.#lengthOf(index);
681
+ const offset = options?.offset ?? 0;
682
+ const length = options?.length ?? pieceLength - offset;
683
+
684
+ const fetch = async () => {
685
+ const slot = this.#slotOf.get(index);
686
+ if (slot !== undefined) {
687
+ this.#lru.touch(index);
688
+ this.#counters.fromMemory += 1;
689
+ const start = slot * this.#chunkLength + offset;
690
+ return Buffer.from(this.#pool.subarray(start, start + length));
691
+ }
692
+
693
+ // Mid-spill: neither in memory nor yet on disk. See `reside` — reporting
694
+ // it missing here would tell WebTorrent to fetch a piece we already have.
695
+ const spill = this.#evicting.get(index);
696
+ if (spill) {
697
+ await spill.catch(() => undefined);
698
+ }
699
+
700
+ if (!this.#disk.has(index)) {
701
+ throw new Error(`Piece ${index} is not in the store.`);
702
+ }
703
+
704
+ // Bring it back into memory: it was just asked for, so it is likely to be
705
+ // asked for again, and the caller may follow up with `locate`.
706
+ const revived = await this.#claimSlot();
707
+ const target = this.#pool.subarray(
708
+ revived * this.#chunkLength,
709
+ revived * this.#chunkLength + pieceLength
710
+ );
711
+ await this.#disk.read(index, target);
712
+ this.#registerSlot(index, revived);
713
+ this.#counters.fromDisk += 1;
714
+ this.#counters.revivals += 1;
715
+ const start = revived * this.#chunkLength + offset;
716
+ return Buffer.from(this.#pool.subarray(start, start + length));
717
+ };
718
+
719
+ fetch().then((bytes) => done(null, bytes), (error) => done(error));
720
+ }
721
+
722
+ /**
723
+ * Ensure a piece is in memory and say where it sits — without copying it.
724
+ *
725
+ * This is {@link get} minus its final copy, and it exists for exactly one
726
+ * caller: the reader that hands pieces to the other thread. That thread maps
727
+ * the same {@link sharedBuffer}, so an offset and a length are all it needs,
728
+ * and the bytes never move. `get` cannot serve that purpose because
729
+ * WebTorrent keeps what `get` returns while the slot underneath may be
730
+ * reused.
731
+ *
732
+ * The caller MUST hold a pin across the whole read — the returned offset
733
+ * stays valid only while the piece is pinned.
734
+ *
735
+ * @param {number} index
736
+ * @returns {Promise<{ offset: number, length: number } | null>} `null` when
737
+ * the store holds no such piece, in memory or on disk.
738
+ */
739
+ /**
740
+ * Declare the pieces a reader is about to need, so eviction takes something
741
+ * else while it can. Replaces that reader's previous declaration.
742
+ *
743
+ * @param {string|number} readerId
744
+ * @param {number} from - First piece, inclusive.
745
+ * @param {number} to - Last piece, inclusive.
746
+ * @returns {void}
747
+ */
748
+ protectRange(readerId, from, to) {
749
+ this.#lru.protect(readerId, from, to);
750
+ }
751
+
752
+ /**
753
+ * Forget a reader's declaration. Call it when the reader ends.
754
+ *
755
+ * @param {string|number} readerId
756
+ * @returns {void}
757
+ */
758
+ /**
759
+ * The windows live readers have declared. See {@link PieceLru.protectedRanges}.
760
+ *
761
+ * @returns {Array<{ from: number, to: number }>}
762
+ */
763
+ protectedRanges() {
764
+ return this.#lru.protectedRanges();
765
+ }
766
+
767
+ releaseProtection(readerId) {
768
+ this.#lru.unprotect(readerId);
769
+ }
770
+
771
+ /**
772
+ * Bring back into memory, in parallel and without waiting, the pieces of a
773
+ * range that have been spilled to disk.
774
+ *
775
+ * A piece is otherwise revived only when the reader arrives at it, one at a
776
+ * time and in step with decoding, so a seek backward into content already
777
+ * downloaded pays a disk round trip per piece. The disk is local; the whole
778
+ * window can be brought back at once while the reader is still on its first
779
+ * piece.
780
+ *
781
+ * Bounded, because each revival needs a slot and unbounded revival of a
782
+ * window larger than the store would simply thrash. Errors are swallowed: a
783
+ * failed warm-up costs nothing, the reader will ask for the piece properly.
784
+ *
785
+ * @param {number} from - First piece, inclusive.
786
+ * @param {number} to - Last piece, inclusive.
787
+ * @param {number} [limit] - Most pieces to revive at once.
788
+ * @returns {number} How many revivals were started.
789
+ */
790
+ warmRange(from, to, limit = Math.max(1, Math.floor(this.#growthCeiling / 4))) {
791
+ if (this.#closed || !Number.isInteger(from) || !Number.isInteger(to)) {
792
+ return 0;
793
+ }
794
+ let started = 0;
795
+ for (let index = from; index <= to && started < limit; index += 1) {
796
+ if (this.#slotOf.has(index) || !this.#disk.has(index)) {
797
+ continue;
798
+ }
799
+ started += 1;
800
+ void this.reside(index).catch(() => undefined);
801
+ }
802
+ return started;
803
+ }
804
+
805
+ async reside(index) {
806
+ if (this.#closed) {
807
+ throw new Error("Piece store is closed.");
808
+ }
809
+
810
+ const slot = this.#slotOf.get(index);
811
+ if (slot !== undefined) {
812
+ this.#lru.touch(index);
813
+ this.#counters.fromMemory += 1;
814
+ return this.locate(index);
815
+ }
816
+
817
+ // Caught mid-spill: the slot is already gone, the disk copy is not there
818
+ // yet. Waiting is the only correct answer — reporting it missing would make
819
+ // the caller re-download a piece we are in the middle of keeping.
820
+ const spill = this.#evicting.get(index);
821
+ if (spill) {
822
+ await spill.catch(() => undefined);
823
+ }
824
+
825
+ if (!this.#disk.has(index)) {
826
+ return null;
827
+ }
828
+
829
+ const pieceLength = this.#lengthOf(index);
830
+ const revived = await this.#claimSlot();
831
+ const target = this.#pool.subarray(
832
+ revived * this.#chunkLength,
833
+ revived * this.#chunkLength + pieceLength
834
+ );
835
+ await this.#disk.read(index, target);
836
+ this.#registerSlot(index, revived);
837
+ this.#counters.fromDisk += 1;
838
+ this.#counters.revivals += 1;
839
+ return this.locate(index);
840
+ }
841
+
842
+ /**
843
+ * Close the store, keeping the spill file.
844
+ *
845
+ * @param {(error?: Error | null) => void} [callback]
846
+ * @returns {void}
847
+ */
848
+ close(callback = () => undefined) {
849
+ this.#closed = true;
850
+ liveStores.delete(this);
851
+ this.#disk.close().then(() => callback(null), (error) => callback(error));
852
+ }
853
+
854
+ /**
855
+ * Close the store and delete everything it wrote.
856
+ *
857
+ * @param {(error?: Error | null) => void} [callback]
858
+ * @returns {void}
859
+ */
860
+ destroy(callback = () => undefined) {
861
+ this.#closed = true;
862
+ liveStores.delete(this);
863
+ this.#slotOf.clear();
864
+ this.#disk.destroy().then(() => callback(null), (error) => callback(error));
865
+ }
866
+ }