@torrent-tv/proxy 2.69.2 → 2.70.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +14 -0
- package/package.json +1 -1
- package/services/piece-store/piece-lru.js +30 -0
- package/services/piece-store/shared-piece-store.js +494 -34
- package/services/torrent-worker/worker.js +52 -11
- package/test/memory-budget.test.js +73 -29
- package/test/piece-store-eviction.test.js +220 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,17 @@
|
|
|
1
|
+
## 2.70.0
|
|
2
|
+
|
|
3
|
+
- **New**: The piece store keeps a pool of memory blocks instead of allocating one per piece. A block is one piece's worth of memory; it is taken from the free list, and put back there when its piece is written out. Field 2026-09-02: 7575 allocations of 4 MiB in 44 minutes, each released only when the collector reached it, which is why the process held 1.86 GB while the store's own accounting said 352 MB. A block goes back for re-use only AFTER the spill write has finished, because that write reads out of it; a block whose piece is re-put while a reader holds it is given up rather than recycled, and a counter says if one ever is not.
|
|
4
|
+
- **Fix**: The store's allowance could only ever fall. `#capacity` was computed once in the constructor and used as an upper bound on every revision, so a torrent opened while the machine was full kept a small allowance for its whole life however much memory was freed afterwards. The field is gone; the opening figure is now only where the store starts.
|
|
5
|
+
- **Fix**: `AVAILABLE_MEMORY_SHARE` (a quarter), `MIN_BUDGET_BYTES` (64 MB) and `MEMORY_BUDGET_CEILING_BYTES` (512 MB) are removed. All three trace to one observation of one host on 2026-08-03 and none was derived. The budget is now the smaller of what the readers have declared — the union of their windows, since picture and sound overlap — and what the machine allows, which is `MemAvailable` plus what the stores already hold, less what other processes have recently been seen to need. That last figure starts at zero and grows only on evidence: the fall in available memory between two readings, beyond what the stores themselves took.
|
|
6
|
+
- **Fix**: A piece no reader has declared, arriving at a store with no room, goes straight to disk instead of pushing out a piece a reader is about to read. It costs the same one write it would have cost when the next arrival evicted it. Before the first read nothing is declared and nothing is refused memory on a guess.
|
|
7
|
+
- **Fix**: Evicting a piece the disk already holds costs no second write. `#revive` reads a piece back and leaves the copy on disk, and only `put` removes it, so a piece revived and not re-put is identical to what is already written. There were 7575 revivals in that one session (roadmap item 66: 14.4 GB written in a single viewing).
|
|
8
|
+
- **Fix**: A store whose readers have GONE asks for nothing, where before it kept asking for what it held. Its torrent sits until the pool's idle timer removes it, and that timer needs a refcount of zero and can be a quarter of an hour away. A store that has never had a reader is a different case and keeps its opening share.
|
|
9
|
+
- **Fix**: A store is never cut below one reader's whole window, whatever the machine's share says. Obeying a smaller share would leave it unable to finish the read it is serving: every resident piece pinned, zero bytes returned, ffmpeg taking that for the end of the file — which killed every encoder on that file on 2026-08-15. The line says when the share was smaller than the window.
|
|
10
|
+
- **Chore**: What other processes need is a window of the last sixty observations, not a high-water. A single spike would otherwise squeeze the stores for the life of the process, which is the same mistake an all-time maximum makes of the block re-use gap.
|
|
11
|
+
- **New**: A spare block is given up once it has sat unused longer than the store's own working rhythm — the longest wait, over recent work, between a block falling free and being wanted again. Measured, not chosen: while a film is being watched a block is taken again within milliseconds, because one is taken for every piece that arrives.
|
|
12
|
+
- **New**: The store line says how many blocks of memory exist and how many are spare, how long a block waits before it is wanted again, how many were given back, how many pieces were admitted that were in nobody's window, and how many evictions needed no write. `committed` now means what the process holds — the blocks — rather than the pieces in them.
|
|
13
|
+
- **Chore**: Thirteen checks across `test/piece-lru.test.js`, `test/piece-store-eviction.test.js` and `test/memory-budget.test.js`, including that a re-used block never carries the previous piece's bytes into the next one.
|
|
14
|
+
|
|
1
15
|
## 2.69.2
|
|
2
16
|
|
|
3
17
|
- **New**: The piece store says WHY it spills, which no reading has ever answered. A session on 2026-09-02 did 6565 spills and 7575 revivals in 44 minutes with only 53.6 % of reads served from memory, and nothing recorded whether that was an eviction order fighting the read order or a working set that simply does not fit. Three figures now settle it, on one line per store: what the live readers between them are asking to keep against what the store may hold; how many evictions had to take a piece a reader had declared it wants; and how long a revived piece had been on disk before it was wanted back.
|
package/package.json
CHANGED
|
@@ -248,6 +248,36 @@ export class PieceLru {
|
|
|
248
248
|
};
|
|
249
249
|
}
|
|
250
250
|
|
|
251
|
+
/**
|
|
252
|
+
* Whether a reader is holding this piece right now.
|
|
253
|
+
*
|
|
254
|
+
* Asked before a block is put back for re-use: a pinned piece has a view onto
|
|
255
|
+
* its memory somewhere, and handing that memory to another piece would let
|
|
256
|
+
* the holder read bytes that are not its own.
|
|
257
|
+
*
|
|
258
|
+
* @param {number} index
|
|
259
|
+
* @returns {boolean}
|
|
260
|
+
*/
|
|
261
|
+
isPinned(index) {
|
|
262
|
+
return this.#pins.has(index);
|
|
263
|
+
}
|
|
264
|
+
|
|
265
|
+
/**
|
|
266
|
+
* Whether any live reader has declared it will want this piece.
|
|
267
|
+
*
|
|
268
|
+
* Asked on admission, not only on eviction: a piece nobody has declared is
|
|
269
|
+
* being downloaded ahead of every reader, and putting it in memory means
|
|
270
|
+
* pushing out one that IS declared — which is then read back from disk
|
|
271
|
+
* moments later. Measured 2026-09-02: 6565 spills and 7575 revivals in 44
|
|
272
|
+
* minutes with 53.6 % of reads served from memory (roadmap item 9).
|
|
273
|
+
*
|
|
274
|
+
* @param {number} index
|
|
275
|
+
* @returns {boolean}
|
|
276
|
+
*/
|
|
277
|
+
wants(index) {
|
|
278
|
+
return this.#isProtected(index);
|
|
279
|
+
}
|
|
280
|
+
|
|
251
281
|
/**
|
|
252
282
|
* Pieces from `index` to the nearest declared window, zero inside one and -1
|
|
253
283
|
* when nothing is declared.
|
|
@@ -68,31 +68,134 @@ export function findSharedStore(torrent) {
|
|
|
68
68
|
return null;
|
|
69
69
|
}
|
|
70
70
|
|
|
71
|
-
|
|
72
|
-
|
|
71
|
+
/**
|
|
72
|
+
* The largest fall in the machine's available memory that this process has
|
|
73
|
+
* seen and did not cause itself.
|
|
74
|
+
*
|
|
75
|
+
* What has to be left for everything else on the machine. It cannot be a share
|
|
76
|
+
* of what is free — a share is a number chosen out of nothing, which is what
|
|
77
|
+
* `AVAILABLE_MEMORY_SHARE = 0.25`, `MIN_BUDGET_BYTES` and
|
|
78
|
+
* `MEMORY_BUDGET_CEILING_BYTES` were until 2026-09-02, all three traceable to
|
|
79
|
+
* one observation of one host on 2026-08-03. It is measured instead: between
|
|
80
|
+
* two readings, how much available memory went away beyond what the stores
|
|
81
|
+
* themselves took. On the field host that quantity is large — while the proxy
|
|
82
|
+
* held 76-133 MB overnight the machine's available memory fell from 2378 MB to
|
|
83
|
+
* 306 MB and came back — and on a quiet host it stays near zero, which is the
|
|
84
|
+
* right answer there.
|
|
85
|
+
*
|
|
86
|
+
* Starts at zero: nothing is reserved until somebody else has been seen to
|
|
87
|
+
* need it.
|
|
88
|
+
*/
|
|
89
|
+
const otherDemand = { falls: [], lastAvailableBytes: 0, lastStoreBytes: 0 };
|
|
90
|
+
|
|
91
|
+
/**
|
|
92
|
+
* How many observations of other processes' demand are kept.
|
|
93
|
+
*
|
|
94
|
+
* A window rather than a high-water, and for the reason the block re-use gap is
|
|
95
|
+
* one too: a single spike would otherwise stand for the life of the process and
|
|
96
|
+
* squeeze the stores against something that happened once, hours ago.
|
|
97
|
+
*/
|
|
98
|
+
const OTHER_DEMAND_SAMPLES = 60;
|
|
99
|
+
|
|
100
|
+
/**
|
|
101
|
+
* Note what the machine had, and how much of the change was not ours.
|
|
102
|
+
*
|
|
103
|
+
* @param {number} availableBytes
|
|
104
|
+
* @param {number} storeBytes - What the live stores hold right now.
|
|
105
|
+
* @returns {number} The reserve, in bytes.
|
|
106
|
+
*/
|
|
107
|
+
export function noteMachineMemory(availableBytes, storeBytes) {
|
|
108
|
+
if (otherDemand.lastAvailableBytes > 0) {
|
|
109
|
+
const fell = otherDemand.lastAvailableBytes - availableBytes;
|
|
110
|
+
const ours = storeBytes - otherDemand.lastStoreBytes;
|
|
111
|
+
otherDemand.falls.push(Math.max(0, fell - ours));
|
|
112
|
+
if (otherDemand.falls.length > OTHER_DEMAND_SAMPLES) {
|
|
113
|
+
otherDemand.falls.shift();
|
|
114
|
+
}
|
|
115
|
+
}
|
|
116
|
+
otherDemand.lastAvailableBytes = availableBytes;
|
|
117
|
+
otherDemand.lastStoreBytes = storeBytes;
|
|
118
|
+
return machineReserveBytes();
|
|
119
|
+
}
|
|
73
120
|
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
return Math.max(
|
|
121
|
+
/** What has recently been observed to be needed by everything that is not us. */
|
|
122
|
+
export function machineReserveBytes() {
|
|
123
|
+
return otherDemand.falls.length === 0 ? 0 : Math.max(...otherDemand.falls);
|
|
77
124
|
}
|
|
78
125
|
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
126
|
+
/** Forget what other processes have needed. For tests, which share a module. */
|
|
127
|
+
export function forgetMachineMemory() {
|
|
128
|
+
otherDemand.falls = [];
|
|
129
|
+
otherDemand.lastAvailableBytes = 0;
|
|
130
|
+
otherDemand.lastStoreBytes = 0;
|
|
131
|
+
}
|
|
132
|
+
|
|
133
|
+
/**
|
|
134
|
+
* How much memory the stores may hold between them.
|
|
135
|
+
*
|
|
136
|
+
* `MemAvailable` is what could be allocated on top of what is already held, so
|
|
137
|
+
* the stores' own bytes are added back: the pair is the ceiling the stores
|
|
138
|
+
* could reach. The reserve is what has been seen to be needed elsewhere.
|
|
139
|
+
*
|
|
140
|
+
* @param {number} availableBytes
|
|
141
|
+
* @param {number} storeBytes
|
|
142
|
+
* @param {number} reserveBytes
|
|
143
|
+
* @returns {number}
|
|
144
|
+
*/
|
|
145
|
+
export function machineAllowanceBytes(availableBytes, storeBytes, reserveBytes) {
|
|
146
|
+
return Math.max(0, Math.max(availableBytes, 0) + Math.max(storeBytes, 0) - Math.max(reserveBytes, 0));
|
|
147
|
+
}
|
|
148
|
+
|
|
149
|
+
/**
|
|
150
|
+
* Divide what the machine allows between the stores, by what each is asking
|
|
151
|
+
* for.
|
|
152
|
+
*
|
|
153
|
+
* A store asks for the pieces its readers have declared. When everyone's ask
|
|
154
|
+
* fits, everyone gets it and the machine's limit never binds — which is the
|
|
155
|
+
* usual case, since two readers of one film declare 32-192 MB against gigabytes
|
|
156
|
+
* of free memory. When the asks do not fit, each store is cut in proportion to
|
|
157
|
+
* what it asked, so a store wanting little is not cut to make room for one
|
|
158
|
+
* wanting much.
|
|
159
|
+
*
|
|
160
|
+
* @param {number[]} wantedBytes - What each store is asking for, in order.
|
|
161
|
+
* @param {number} allowanceBytes
|
|
162
|
+
* @returns {number[]} What each store may hold, in the same order.
|
|
163
|
+
*/
|
|
164
|
+
export function divideAllowance(wantedBytes, allowanceBytes) {
|
|
165
|
+
const total = wantedBytes.reduce((sum, want) => sum + Math.max(0, want), 0);
|
|
166
|
+
if (total <= allowanceBytes || total === 0) {
|
|
167
|
+
return wantedBytes.map((want) => Math.max(0, want));
|
|
168
|
+
}
|
|
169
|
+
return wantedBytes.map((want) => Math.floor(allowanceBytes * (Math.max(0, want) / total)));
|
|
83
170
|
}
|
|
84
171
|
|
|
85
172
|
export function reviseStoreBudgets() {
|
|
86
|
-
const
|
|
173
|
+
const stores = [...liveStores];
|
|
174
|
+
const held = stores.reduce((sum, store) => sum + store.residentBytes, 0);
|
|
175
|
+
const available = availableMemorySync();
|
|
176
|
+
const reserve = noteMachineMemory(available, held);
|
|
177
|
+
const allowance = machineAllowanceBytes(available, held, reserve);
|
|
178
|
+
const shares = divideAllowance(stores.map((store) => store.wantedBytes), allowance);
|
|
87
179
|
const revised = [];
|
|
88
|
-
for (const store of
|
|
89
|
-
revised.push(store.reviseGrowthCeiling(
|
|
180
|
+
for (const [position, store] of stores.entries()) {
|
|
181
|
+
revised.push(store.reviseGrowthCeiling(shares[position]));
|
|
90
182
|
}
|
|
91
183
|
return revised;
|
|
92
184
|
}
|
|
93
185
|
|
|
94
186
|
function defaultMemoryBytes() {
|
|
95
|
-
|
|
187
|
+
const stores = [...liveStores];
|
|
188
|
+
const held = stores.reduce((sum, store) => sum + store.residentBytes, 0);
|
|
189
|
+
const available = availableMemorySync();
|
|
190
|
+
const allowance = machineAllowanceBytes(available, held, machineReserveBytes());
|
|
191
|
+
// A store being created has no readers, so it has no demand to state and no
|
|
192
|
+
// basis for asking for more or less than the others. It opens on an equal
|
|
193
|
+
// share and the first revision — within a minute, and within seconds of a
|
|
194
|
+
// read starting — replaces that with what its readers actually declare.
|
|
195
|
+
// Deliberately not the whole allowance: on a machine with gigabytes free that
|
|
196
|
+
// would let a torrent nobody is reading yet fill memory before the first
|
|
197
|
+
// revision arrives.
|
|
198
|
+
return Math.floor(allowance / (stores.length + 1));
|
|
96
199
|
}
|
|
97
200
|
|
|
98
201
|
function availableMemorySync() {
|
|
@@ -108,8 +211,12 @@ function availableMemorySync() {
|
|
|
108
211
|
}
|
|
109
212
|
|
|
110
213
|
/**
|
|
111
|
-
* How many piece
|
|
112
|
-
* has actually taken back.
|
|
214
|
+
* How many blocks of piece memory this thread has let go of, and how many the
|
|
215
|
+
* collector has actually taken back.
|
|
216
|
+
*
|
|
217
|
+
* Since the store keeps a pool, one block serves many pieces, so this counts
|
|
218
|
+
* blocks and not pieces — and a healthy store allocates only as many as its
|
|
219
|
+
* allowance permits, so both numbers are now small and nearly equal.
|
|
113
220
|
*
|
|
114
221
|
* The one reading that separates the two explanations of the 700 MB nobody can
|
|
115
222
|
* account for (roadmap item 2, field 2026-08-31): if the two numbers track each
|
|
@@ -139,7 +246,6 @@ export function pieceBufferCollection() {
|
|
|
139
246
|
return { released: released.count, collected: released.collected };
|
|
140
247
|
}
|
|
141
248
|
|
|
142
|
-
const MIN_BUDGET_BYTES = 64 * 1024 * 1024;
|
|
143
249
|
const MIN_RESIDENT_PIECES = 2;
|
|
144
250
|
/**
|
|
145
251
|
* How long a claim may go without ANYTHING moving before it gives up.
|
|
@@ -159,6 +265,13 @@ const PINNED_WAIT_MS = 5_000;
|
|
|
159
265
|
*/
|
|
160
266
|
const REVIVAL_AGE_SAMPLES = 200;
|
|
161
267
|
|
|
268
|
+
/**
|
|
269
|
+
* How many block re-use gaps are kept. The same window as the revival ages, and
|
|
270
|
+
* for the same reason: what is wanted is the rhythm of recent work, not a
|
|
271
|
+
* history of it.
|
|
272
|
+
*/
|
|
273
|
+
const REUSE_GAP_SAMPLES = 200;
|
|
274
|
+
|
|
162
275
|
/**
|
|
163
276
|
* The middle value of a sample, or null when there is nothing to take a middle
|
|
164
277
|
* of. Null rather than zero: no revivals and instant revivals are different
|
|
@@ -183,7 +296,6 @@ export class SharedPieceStore {
|
|
|
183
296
|
#chunkLength;
|
|
184
297
|
#lastChunkLength;
|
|
185
298
|
#lastChunkIndex;
|
|
186
|
-
#capacity;
|
|
187
299
|
#growthCeiling;
|
|
188
300
|
/** Piece index → SharedArrayBuffer of that piece */
|
|
189
301
|
#buffers = new Map();
|
|
@@ -224,8 +336,52 @@ export class SharedPieceStore {
|
|
|
224
336
|
// (roadmap item 9).
|
|
225
337
|
evictedProtected: 0,
|
|
226
338
|
evictedDistanceSum: 0,
|
|
227
|
-
evictedWithDistance: 0
|
|
339
|
+
evictedWithDistance: 0,
|
|
340
|
+
// Where an arriving piece went. A piece nobody has declared is being
|
|
341
|
+
// downloaded ahead of every reader; putting it in memory pushes out one
|
|
342
|
+
// that IS declared, which is then read back from disk moments later
|
|
343
|
+
// (roadmap item 9).
|
|
344
|
+
admittedInsideWindow: 0,
|
|
345
|
+
admittedOutsideWindow: 0,
|
|
346
|
+
admittedToDisk: 0,
|
|
347
|
+
/** Blocks given back to the operating system. */
|
|
348
|
+
blocksReleased: 0,
|
|
349
|
+
/**
|
|
350
|
+
* A block put back for re-use while a reader still held the piece. Zero by
|
|
351
|
+
* construction — a pinned piece is never evicted, and a re-put of a pinned
|
|
352
|
+
* one drops its block instead of recycling it. Counted because if it ever
|
|
353
|
+
* stops being zero, a consumer is reading another piece's bytes, and that
|
|
354
|
+
* is invisible from anywhere else.
|
|
355
|
+
*/
|
|
356
|
+
returnedWhilePinned: 0,
|
|
357
|
+
/** Spills that found the disk already holding identical bytes. */
|
|
358
|
+
spillsSkipped: 0
|
|
228
359
|
};
|
|
360
|
+
/**
|
|
361
|
+
* Blocks that hold no piece, most recently freed last.
|
|
362
|
+
*
|
|
363
|
+
* A block is one piece's worth of memory. Allocating a new one for every
|
|
364
|
+
* piece meant 7575 allocations of 4 MiB in 44 minutes on 2026-09-02, each
|
|
365
|
+
* released only when the collector got to it — which is why the process held
|
|
366
|
+
* 1.86 GB while its own accounting said 352 MB. Blocks are taken from here
|
|
367
|
+
* and put back here instead (roadmap item 2).
|
|
368
|
+
*
|
|
369
|
+
* @type {{ buffer: SharedArrayBuffer, freedAt: number }[]}
|
|
370
|
+
*/
|
|
371
|
+
#freeBlocks = [];
|
|
372
|
+
/** Blocks that exist at all: free plus holding a piece. */
|
|
373
|
+
#blocksAllocated = 0;
|
|
374
|
+
/** Whether a reader has ever declared a window here. See `wantedBytes`. */
|
|
375
|
+
#everHadReader = false;
|
|
376
|
+
/**
|
|
377
|
+
* How long a block sat free before it was taken again, in milliseconds.
|
|
378
|
+
* Bounded, because what is wanted is the longest gap of RECENT work: an
|
|
379
|
+
* all-time maximum would be raised by one long pause and then never let a
|
|
380
|
+
* block go again.
|
|
381
|
+
*
|
|
382
|
+
* @type {number[]}
|
|
383
|
+
*/
|
|
384
|
+
#reuseGaps = [];
|
|
229
385
|
/** Piece index → when it was written out, for the age it comes back at. */
|
|
230
386
|
#spilledAt = new Map();
|
|
231
387
|
/**
|
|
@@ -251,10 +407,11 @@ export class SharedPieceStore {
|
|
|
251
407
|
const memoryBytes = Number.isFinite(options.memoryBytes) && options.memoryBytes > 0
|
|
252
408
|
? options.memoryBytes
|
|
253
409
|
: defaultMemoryBytes();
|
|
254
|
-
this
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
this.#
|
|
410
|
+
// Only where this store STARTS. It is not kept, so it cannot come back as a
|
|
411
|
+
// cap on the revision the way `#capacity` did.
|
|
412
|
+
const openingCeiling = Math.max(MIN_RESIDENT_PIECES, Math.floor(memoryBytes / chunkLength));
|
|
413
|
+
this.#growthCeiling = openingCeiling;
|
|
414
|
+
this.#lru = new PieceLru(openingCeiling);
|
|
258
415
|
this.#name = options.name ?? "pieces";
|
|
259
416
|
// `options.disk` exists so a test can hold a write open or make one fail on
|
|
260
417
|
// purpose. Four of the defects fixed here live in what happens when the
|
|
@@ -277,8 +434,11 @@ export class SharedPieceStore {
|
|
|
277
434
|
resident,
|
|
278
435
|
capacity: this.#growthCeiling,
|
|
279
436
|
residentBytes,
|
|
280
|
-
allocatedSlots:
|
|
281
|
-
|
|
437
|
+
allocatedSlots: this.#blocksAllocated,
|
|
438
|
+
// What the process HOLDS, which with a pool is the blocks that exist —
|
|
439
|
+
// not the pieces in them. Holding and using are different quantities and
|
|
440
|
+
// the difference is the point of the reading.
|
|
441
|
+
committedBytes: this.#blocksAllocated * this.#chunkLength,
|
|
282
442
|
budgetBytes: this.#growthCeiling * this.#chunkLength,
|
|
283
443
|
pinned: this.#lru.pinnedCount,
|
|
284
444
|
// Slots claimed and not yet filled. Reported because a reservation that
|
|
@@ -292,6 +452,14 @@ export class SharedPieceStore {
|
|
|
292
452
|
// however the eviction is ordered, and that is the difference between a
|
|
293
453
|
// policy to fix and arithmetic to accept (roadmap item 9).
|
|
294
454
|
demand: this.#lru.demand(),
|
|
455
|
+
// What the process actually holds for this store, which is not the same
|
|
456
|
+
// as what it is using: blocks are kept for re-use. The difference is the
|
|
457
|
+
// spare, and it is the whole of the question whether consumption only
|
|
458
|
+
// grows (roadmap item 2).
|
|
459
|
+
blocksAllocated: this.#blocksAllocated,
|
|
460
|
+
blocksFree: this.#freeBlocks.length,
|
|
461
|
+
blockBytes: this.#blocksAllocated * this.#chunkLength,
|
|
462
|
+
reuseGapMs: this.#reuseGapCeilingMs(),
|
|
295
463
|
revivalAgeMedianMs: median(this.#revivalAges),
|
|
296
464
|
revivalAgeSamples: this.#revivalAges.length,
|
|
297
465
|
revivedWithinFiveSeconds: this.#revivalAges.filter((age) => age <= 5_000).length,
|
|
@@ -299,12 +467,66 @@ export class SharedPieceStore {
|
|
|
299
467
|
};
|
|
300
468
|
}
|
|
301
469
|
|
|
470
|
+
/** What this store holds right now, in bytes. */
|
|
471
|
+
get residentBytes() {
|
|
472
|
+
return this.#buffers.size * this.#chunkLength;
|
|
473
|
+
}
|
|
474
|
+
|
|
475
|
+
/**
|
|
476
|
+
* What this store is asking to be allowed to hold, in bytes.
|
|
477
|
+
*
|
|
478
|
+
* The union of its readers' declared windows — what they have said they will
|
|
479
|
+
* need — never below the widest single window, because a store that cannot
|
|
480
|
+
* hold one reader's window cannot complete that reader's read at all: every
|
|
481
|
+
* resident piece ends up pinned, the read returns zero bytes and ffmpeg takes
|
|
482
|
+
* that for the end of the file (field 2026-08-15, roadmap item 9).
|
|
483
|
+
*
|
|
484
|
+
* With no reader declaring anything there is no demand to speak of, so the
|
|
485
|
+
* store asks for what the machine allows and the first revision after a read
|
|
486
|
+
* begins brings it down to what that read needs.
|
|
487
|
+
*/
|
|
488
|
+
get wantedBytes() {
|
|
489
|
+
const demand = this.#lru.demand();
|
|
490
|
+
if (demand.readers > 0) {
|
|
491
|
+
this.#everHadReader = true;
|
|
492
|
+
const pieces = Math.max(MIN_RESIDENT_PIECES, demand.unionPieces, demand.widestPieces);
|
|
493
|
+
return pieces * this.#chunkLength;
|
|
494
|
+
}
|
|
495
|
+
// Readers that have GONE are not the same as readers that have not arrived.
|
|
496
|
+
// A store whose readers ended has nothing to hold pieces for — its torrent
|
|
497
|
+
// sits until the pool's idle timer removes it, which needs a refcount of
|
|
498
|
+
// zero and can be a quarter of an hour away — so it asks for nothing and
|
|
499
|
+
// its memory goes back to the machine now. A store that has never had a
|
|
500
|
+
// reader is being filled for one that is on its way, and asks for what it
|
|
501
|
+
// was opened with until the first read says what it needs.
|
|
502
|
+
return this.#everHadReader
|
|
503
|
+
? MIN_RESIDENT_PIECES * this.#chunkLength
|
|
504
|
+
: this.#growthCeiling * this.#chunkLength;
|
|
505
|
+
}
|
|
506
|
+
|
|
302
507
|
reviseGrowthCeiling(allowedBytes) {
|
|
508
|
+
// No cap at what the machine could spare when this store was CREATED.
|
|
509
|
+
// `#capacity` was computed once in the constructor and used as an upper
|
|
510
|
+
// bound here, so the allowance could only ever fall: a torrent opened while
|
|
511
|
+
// the machine was full kept a small allowance for its whole life, however
|
|
512
|
+
// much memory was freed afterwards (roadmap item 2, 2026-09-02).
|
|
303
513
|
const wanted = Math.floor(Number(allowedBytes) / this.#chunkLength);
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
514
|
+
// Never below one reader's whole window while a reader exists, even when
|
|
515
|
+
// the machine's share says less. A store that cannot hold the window of the
|
|
516
|
+
// read it is serving cannot complete that read at all: every resident piece
|
|
517
|
+
// ends up pinned, the read returns zero bytes and ffmpeg takes that for the
|
|
518
|
+
// end of the file, which killed every encoder on that file in the field on
|
|
519
|
+
// 2026-08-15. Exceeding the share is the lesser failure, and the line below
|
|
520
|
+
// says when it happens.
|
|
521
|
+
const demand = this.#lru.demand();
|
|
522
|
+
this.#growthCeiling = Math.max(
|
|
523
|
+
MIN_RESIDENT_PIECES,
|
|
524
|
+
demand.readers > 0 ? demand.widestPieces : MIN_RESIDENT_PIECES,
|
|
525
|
+
Number.isFinite(wanted) ? wanted : MIN_RESIDENT_PIECES
|
|
307
526
|
);
|
|
527
|
+
const belowAWindow = demand.readers > 0
|
|
528
|
+
&& Number.isFinite(wanted)
|
|
529
|
+
&& wanted < demand.widestPieces;
|
|
308
530
|
// The LRU is told too. It was constructed with the store's original
|
|
309
531
|
// capacity and never revised, so `isFull()` answered against a number that
|
|
310
532
|
// had not been the limit for some time — dormant only because nothing calls
|
|
@@ -336,11 +558,17 @@ export class SharedPieceStore {
|
|
|
336
558
|
this.#counters.spillFailures += 1;
|
|
337
559
|
});
|
|
338
560
|
}
|
|
561
|
+
// Blocks the store is no longer using and has waited long enough to give
|
|
562
|
+
// up. The allowance falling is the moment to ask, because that is when the
|
|
563
|
+
// machine has been shown to need the memory.
|
|
564
|
+
const releasedBlocks = this.sweepFreeBlocks();
|
|
339
565
|
return {
|
|
340
566
|
name: this.#name,
|
|
341
567
|
ceilingBytes: this.#growthCeiling * this.#chunkLength,
|
|
342
|
-
committedBytes: this.#
|
|
343
|
-
evicted
|
|
568
|
+
committedBytes: this.#blocksAllocated * this.#chunkLength,
|
|
569
|
+
evicted,
|
|
570
|
+
releasedBlocks,
|
|
571
|
+
belowAWindow
|
|
344
572
|
};
|
|
345
573
|
}
|
|
346
574
|
|
|
@@ -469,16 +697,39 @@ export class SharedPieceStore {
|
|
|
469
697
|
* @returns {Promise<void>}
|
|
470
698
|
*/
|
|
471
699
|
#spill(index, buffer) {
|
|
700
|
+
// The disk may already hold these very bytes. `#revive` reads a piece back
|
|
701
|
+
// into memory and leaves the copy on disk, and only `put` removes it — so a
|
|
702
|
+
// piece that was revived and not re-put is identical to what is already
|
|
703
|
+
// written, and writing it again is work for nothing. There were 7575
|
|
704
|
+
// revivals in one session on 2026-09-02, and every later eviction of one of
|
|
705
|
+
// them wrote a second time (roadmap item 66: 14.4 GB written in a single
|
|
706
|
+
// viewing).
|
|
707
|
+
if (this.#disk.has(index)) {
|
|
708
|
+
this.#counters.spills += 1;
|
|
709
|
+
this.#counters.spillsSkipped += 1;
|
|
710
|
+
this.#spilledAt.set(index, Date.now());
|
|
711
|
+
this.#returnBlock(buffer);
|
|
712
|
+
this.#noteProgress();
|
|
713
|
+
return Promise.resolve();
|
|
714
|
+
}
|
|
715
|
+
|
|
472
716
|
const bytes = Buffer.from(buffer, 0, this.#lengthOf(index));
|
|
473
717
|
const spill = this.#disk.write(index, bytes).then(
|
|
474
718
|
() => {
|
|
475
719
|
this.#counters.spills += 1;
|
|
476
720
|
this.#spilledAt.set(index, Date.now());
|
|
477
721
|
this.#evicting.delete(index);
|
|
722
|
+
// Only now. The write reads out of this block, so a block handed to
|
|
723
|
+
// another piece before the write finished would put that piece's bytes
|
|
724
|
+
// into this piece's place in the file.
|
|
725
|
+
this.#returnBlock(buffer);
|
|
478
726
|
this.#noteProgress();
|
|
479
727
|
},
|
|
480
728
|
(error) => {
|
|
481
729
|
this.#evicting.delete(index);
|
|
730
|
+
// The block is no longer holding anything either way; keeping it out of
|
|
731
|
+
// the pool because the write failed would lose it for good.
|
|
732
|
+
this.#returnBlock(buffer);
|
|
482
733
|
this.#noteProgress();
|
|
483
734
|
throw error;
|
|
484
735
|
}
|
|
@@ -493,6 +744,53 @@ export class SharedPieceStore {
|
|
|
493
744
|
this.#wake();
|
|
494
745
|
}
|
|
495
746
|
|
|
747
|
+
/**
|
|
748
|
+
* Whether admitting one more piece would need something evicted first.
|
|
749
|
+
*
|
|
750
|
+
* Reservations count: a slot claimed and not yet filled is as taken as a
|
|
751
|
+
* resident piece.
|
|
752
|
+
*
|
|
753
|
+
* @returns {boolean}
|
|
754
|
+
*/
|
|
755
|
+
#isFullNow() {
|
|
756
|
+
return this.#buffers.size + this.#outstandingPieces >= this.#growthCeiling;
|
|
757
|
+
}
|
|
758
|
+
|
|
759
|
+
/**
|
|
760
|
+
* Write an arriving piece straight to disk, without it ever occupying memory.
|
|
761
|
+
*
|
|
762
|
+
* Registered in `#evicting` like a spill, so a `get` for this index waits for
|
|
763
|
+
* the write instead of finding the piece on neither tier. Deliberately NOT
|
|
764
|
+
* recorded in `#spilledAt`: that clock measures how long an EVICTED piece
|
|
765
|
+
* stayed away, and a piece that was never resident has no such age.
|
|
766
|
+
*
|
|
767
|
+
* @param {number} index
|
|
768
|
+
* @param {Uint8Array} bytes
|
|
769
|
+
* @returns {Promise<void>}
|
|
770
|
+
*/
|
|
771
|
+
#writeThrough(index, bytes) {
|
|
772
|
+
// Copied, not viewed. `DiskTier.write` opens the file before it reads the
|
|
773
|
+
// bytes, so a view onto the caller's buffer could be written to in between
|
|
774
|
+
// and the file would get the wrong data. The spill path may pass a view
|
|
775
|
+
// because that memory is ours; this buffer belongs to the torrent client.
|
|
776
|
+
// It costs nothing extra: the piece was being copied into a shared buffer
|
|
777
|
+
// on this path before, and now it is copied here instead.
|
|
778
|
+
const copy = Buffer.from(bytes.subarray(0, Math.min(bytes.length, this.#lengthOf(index))));
|
|
779
|
+
const write = this.#disk.write(index, copy).then(
|
|
780
|
+
() => {
|
|
781
|
+
this.#evicting.delete(index);
|
|
782
|
+
this.#noteProgress();
|
|
783
|
+
},
|
|
784
|
+
(error) => {
|
|
785
|
+
this.#evicting.delete(index);
|
|
786
|
+
this.#noteProgress();
|
|
787
|
+
throw error;
|
|
788
|
+
}
|
|
789
|
+
);
|
|
790
|
+
this.#evicting.set(index, write);
|
|
791
|
+
return write;
|
|
792
|
+
}
|
|
793
|
+
|
|
496
794
|
/**
|
|
497
795
|
* Record how long a piece stayed on disk before it was wanted again.
|
|
498
796
|
*
|
|
@@ -629,8 +927,16 @@ export class SharedPieceStore {
|
|
|
629
927
|
this.#counters.fromMemory += 1;
|
|
630
928
|
return already;
|
|
631
929
|
}
|
|
632
|
-
const target = this.#
|
|
633
|
-
|
|
930
|
+
const target = this.#takeBlock();
|
|
931
|
+
try {
|
|
932
|
+
// Only this piece's own length: the block is a full piece long and the
|
|
933
|
+
// last piece of a file is shorter, so reading the whole block would ask
|
|
934
|
+
// the file for bytes past its end.
|
|
935
|
+
await this.#disk.read(index, Buffer.from(target, 0, this.#lengthOf(index)));
|
|
936
|
+
} catch (error) {
|
|
937
|
+
this.#returnBlock(target);
|
|
938
|
+
throw error;
|
|
939
|
+
}
|
|
634
940
|
this.#registerPiece(index, target);
|
|
635
941
|
this.#counters.fromDisk += 1;
|
|
636
942
|
this.#counters.revivals += 1;
|
|
@@ -641,6 +947,121 @@ export class SharedPieceStore {
|
|
|
641
947
|
}
|
|
642
948
|
}
|
|
643
949
|
|
|
950
|
+
/**
|
|
951
|
+
* A block to hold one piece: the most recently freed one, or a new one.
|
|
952
|
+
*
|
|
953
|
+
* Every block is a full piece long, whatever piece will live in it. The last
|
|
954
|
+
* piece of a file is shorter, and it occupies a full block with only its own
|
|
955
|
+
* bytes meaningful — `#lengthOf` is what decides how much is ever read out.
|
|
956
|
+
* Uniform blocks are what makes them interchangeable at all.
|
|
957
|
+
*
|
|
958
|
+
* Most recently freed first, deliberately: a few blocks then carry the whole
|
|
959
|
+
* of a busy store's traffic and the rest age out of use, which is what makes
|
|
960
|
+
* {@link SharedPieceStore#sweepFreeBlocks} able to tell a spare block from a
|
|
961
|
+
* working one.
|
|
962
|
+
*
|
|
963
|
+
* @returns {SharedArrayBuffer}
|
|
964
|
+
*/
|
|
965
|
+
#takeBlock() {
|
|
966
|
+
const spare = this.#freeBlocks.pop();
|
|
967
|
+
if (spare !== undefined) {
|
|
968
|
+
this.#noteReuseGap(Date.now() - spare.freedAt);
|
|
969
|
+
return spare.buffer;
|
|
970
|
+
}
|
|
971
|
+
this.#blocksAllocated += 1;
|
|
972
|
+
return this.#watchForCollection(new SharedArrayBuffer(this.#chunkLength));
|
|
973
|
+
}
|
|
974
|
+
|
|
975
|
+
/**
|
|
976
|
+
* Put a block back for re-use, or give it up.
|
|
977
|
+
*
|
|
978
|
+
* Given up when the allowance has fallen below the number of blocks that
|
|
979
|
+
* exist: keeping it would hold memory the machine has just been shown to
|
|
980
|
+
* need. Otherwise it waits in the free list for the next piece.
|
|
981
|
+
*
|
|
982
|
+
* @param {SharedArrayBuffer | undefined} buffer
|
|
983
|
+
* @returns {void}
|
|
984
|
+
*/
|
|
985
|
+
#returnBlock(buffer) {
|
|
986
|
+
if (buffer === undefined) {
|
|
987
|
+
return;
|
|
988
|
+
}
|
|
989
|
+
if (this.#closed || this.#blocksAllocated > this.#growthCeiling) {
|
|
990
|
+
// Never below zero: `close` gives up every block at once, and a spill
|
|
991
|
+
// that was already in flight resolves afterwards and arrives here.
|
|
992
|
+
if (this.#blocksAllocated > 0) {
|
|
993
|
+
this.#blocksAllocated -= 1;
|
|
994
|
+
this.#counters.blocksReleased += 1;
|
|
995
|
+
}
|
|
996
|
+
return;
|
|
997
|
+
}
|
|
998
|
+
this.#freeBlocks.push({ buffer, freedAt: Date.now() });
|
|
999
|
+
}
|
|
1000
|
+
|
|
1001
|
+
/**
|
|
1002
|
+
* Record how long a block waited to be used again.
|
|
1003
|
+
*
|
|
1004
|
+
* @param {number} gapMs
|
|
1005
|
+
* @returns {void}
|
|
1006
|
+
*/
|
|
1007
|
+
#noteReuseGap(gapMs) {
|
|
1008
|
+
this.#reuseGaps.push(Math.max(0, gapMs));
|
|
1009
|
+
if (this.#reuseGaps.length > REUSE_GAP_SAMPLES) {
|
|
1010
|
+
this.#reuseGaps.shift();
|
|
1011
|
+
}
|
|
1012
|
+
}
|
|
1013
|
+
|
|
1014
|
+
/**
|
|
1015
|
+
* The longest a block has recently waited before being wanted again, or null
|
|
1016
|
+
* when no block has yet been re-used.
|
|
1017
|
+
*
|
|
1018
|
+
* This is the store's own working rhythm, measured rather than chosen: while
|
|
1019
|
+
* a film is being watched a block is taken again within milliseconds, because
|
|
1020
|
+
* one is taken for every piece that arrives. A block that has been sitting
|
|
1021
|
+
* longer than the longest of those waits is not part of the work.
|
|
1022
|
+
*
|
|
1023
|
+
* @returns {number | null}
|
|
1024
|
+
*/
|
|
1025
|
+
#reuseGapCeilingMs() {
|
|
1026
|
+
if (this.#reuseGaps.length === 0) {
|
|
1027
|
+
return null;
|
|
1028
|
+
}
|
|
1029
|
+
return Math.max(...this.#reuseGaps);
|
|
1030
|
+
}
|
|
1031
|
+
|
|
1032
|
+
/**
|
|
1033
|
+
* Give up blocks that have sat unused longer than this store's own working
|
|
1034
|
+
* rhythm.
|
|
1035
|
+
*
|
|
1036
|
+
* The case it is for: a torrent whose peers have gone. Its readers are still
|
|
1037
|
+
* attached, so nothing removes the torrent — the pool's idle timer needs a
|
|
1038
|
+
* refcount of zero and never starts. Its allowance falls to what those
|
|
1039
|
+
* readers declared, the pieces beyond it are written out, and their blocks
|
|
1040
|
+
* would otherwise wait in the free list for peers that may not return.
|
|
1041
|
+
*
|
|
1042
|
+
* @param {number} [now]
|
|
1043
|
+
* @returns {number} Blocks given up.
|
|
1044
|
+
*/
|
|
1045
|
+
sweepFreeBlocks(now = Date.now()) {
|
|
1046
|
+
const ceiling = this.#reuseGapCeilingMs();
|
|
1047
|
+
if (ceiling === null) {
|
|
1048
|
+
return 0;
|
|
1049
|
+
}
|
|
1050
|
+
const keeping = [];
|
|
1051
|
+
let released = 0;
|
|
1052
|
+
for (const spare of this.#freeBlocks) {
|
|
1053
|
+
if (now - spare.freedAt <= ceiling) {
|
|
1054
|
+
keeping.push(spare);
|
|
1055
|
+
continue;
|
|
1056
|
+
}
|
|
1057
|
+
this.#blocksAllocated -= 1;
|
|
1058
|
+
this.#counters.blocksReleased += 1;
|
|
1059
|
+
released += 1;
|
|
1060
|
+
}
|
|
1061
|
+
this.#freeBlocks = keeping;
|
|
1062
|
+
return released;
|
|
1063
|
+
}
|
|
1064
|
+
|
|
644
1065
|
/**
|
|
645
1066
|
* Count this buffer as one this thread will have to let go of, and notice
|
|
646
1067
|
* when the collector takes it. See {@link pieceBufferCollection}.
|
|
@@ -663,14 +1084,14 @@ export class SharedPieceStore {
|
|
|
663
1084
|
*/
|
|
664
1085
|
#copyIntoNewBuffer(index, bytes) {
|
|
665
1086
|
const length = this.#lengthOf(index);
|
|
666
|
-
const
|
|
667
|
-
const view = Buffer.from(
|
|
1087
|
+
const block = this.#takeBlock();
|
|
1088
|
+
const view = Buffer.from(block, 0, length);
|
|
668
1089
|
if (bytes.copy) {
|
|
669
1090
|
bytes.copy(view, 0, 0, length);
|
|
670
1091
|
} else {
|
|
671
1092
|
view.set(bytes.subarray(0, length), 0);
|
|
672
1093
|
}
|
|
673
|
-
return
|
|
1094
|
+
return block;
|
|
674
1095
|
}
|
|
675
1096
|
|
|
676
1097
|
/**
|
|
@@ -704,13 +1125,46 @@ export class SharedPieceStore {
|
|
|
704
1125
|
// needed and none is claimed. A fresh buffer rather than a write into the
|
|
705
1126
|
// old one, so a reader holding the old reference cannot see a torn write.
|
|
706
1127
|
if (this.#buffers.has(index)) {
|
|
1128
|
+
const previous = this.#buffers.get(index);
|
|
1129
|
+
// A fresh block rather than a write into the old one, so a reader
|
|
1130
|
+
// holding the old reference cannot see a torn write.
|
|
707
1131
|
this.#buffers.set(index, this.#copyIntoNewBuffer(index, bytes));
|
|
708
1132
|
this.#lru.touch(index);
|
|
1133
|
+
// And the old block goes back for re-use only if nobody is reading it.
|
|
1134
|
+
// A pinned piece has a view onto its memory somewhere; that block is
|
|
1135
|
+
// given up instead, and the pool allocates another when it needs one.
|
|
1136
|
+
if (this.#lru.isPinned(index)) {
|
|
1137
|
+
if (this.#blocksAllocated > 0) {
|
|
1138
|
+
this.#blocksAllocated -= 1;
|
|
1139
|
+
this.#counters.blocksReleased += 1;
|
|
1140
|
+
}
|
|
1141
|
+
} else {
|
|
1142
|
+
this.#returnBlock(previous);
|
|
1143
|
+
}
|
|
709
1144
|
await this.#forgetOnDisk(index);
|
|
710
1145
|
this.#noteProgress();
|
|
711
1146
|
return;
|
|
712
1147
|
}
|
|
713
1148
|
|
|
1149
|
+
const declared = this.#lru.wants(index);
|
|
1150
|
+
if (declared) {
|
|
1151
|
+
this.#counters.admittedInsideWindow += 1;
|
|
1152
|
+
} else {
|
|
1153
|
+
this.#counters.admittedOutsideWindow += 1;
|
|
1154
|
+
}
|
|
1155
|
+
|
|
1156
|
+
// A piece no reader has declared, arriving at a store with no room, goes
|
|
1157
|
+
// straight to disk. It costs the same one write it would have cost when
|
|
1158
|
+
// the next arrival evicted it, and it saves pushing out a piece a reader
|
|
1159
|
+
// is about to read. Only when SOMETHING is declared: before the first
|
|
1160
|
+
// read there is no basis for calling a piece unwanted.
|
|
1161
|
+
if (!declared && this.#lru.protectedCount > 0 && this.#isFullNow()) {
|
|
1162
|
+
this.#counters.admittedToDisk += 1;
|
|
1163
|
+
await this.#writeThrough(index, bytes);
|
|
1164
|
+
this.#noteProgress();
|
|
1165
|
+
return;
|
|
1166
|
+
}
|
|
1167
|
+
|
|
714
1168
|
const release = await this.#claimSlot();
|
|
715
1169
|
try {
|
|
716
1170
|
this.#registerPiece(index, this.#copyIntoNewBuffer(index, bytes));
|
|
@@ -806,6 +1260,9 @@ export class SharedPieceStore {
|
|
|
806
1260
|
liveStores.delete(this);
|
|
807
1261
|
this.#buffers.clear();
|
|
808
1262
|
this.#spilledAt.clear();
|
|
1263
|
+
this.#counters.blocksReleased += this.#blocksAllocated;
|
|
1264
|
+
this.#blocksAllocated = 0;
|
|
1265
|
+
this.#freeBlocks = [];
|
|
809
1266
|
// Whoever is waiting for a slot is woken and finds the store closed, which
|
|
810
1267
|
// is an error they can report. Left asleep they simply never returned.
|
|
811
1268
|
this.#wake();
|
|
@@ -817,6 +1274,9 @@ export class SharedPieceStore {
|
|
|
817
1274
|
liveStores.delete(this);
|
|
818
1275
|
this.#buffers.clear();
|
|
819
1276
|
this.#spilledAt.clear();
|
|
1277
|
+
this.#counters.blocksReleased += this.#blocksAllocated;
|
|
1278
|
+
this.#blocksAllocated = 0;
|
|
1279
|
+
this.#freeBlocks = [];
|
|
820
1280
|
this.#wake();
|
|
821
1281
|
this.#disk.destroy().then(() => callback(null), (error) => callback(error));
|
|
822
1282
|
}
|
|
@@ -39,7 +39,8 @@ import { startMemoryReport, WORKER_MEMORY_SAMPLE_MS } from "../memory-report.js"
|
|
|
39
39
|
// the hook above had a chance to register. Verified the hard way: with a static
|
|
40
40
|
// import the process still aborted, and the stack named the genuine polyfill.
|
|
41
41
|
const { TorrentPool, resolveDhtBootstrap } = await import("../torrent-pool.js");
|
|
42
|
-
const { collectStoreStats, pieceBufferCollection, reviseStoreBudgets } =
|
|
42
|
+
const { collectStoreStats, machineReserveBytes, pieceBufferCollection, reviseStoreBudgets } =
|
|
43
|
+
await import("../piece-store/shared-piece-store.js");
|
|
43
44
|
|
|
44
45
|
// Resolved before the client exists, because the client builds its DHT in its
|
|
45
46
|
// own constructor and the addresses have to be in hand by then. Awaiting here
|
|
@@ -541,6 +542,9 @@ startMemoryReport({
|
|
|
541
542
|
|
|
542
543
|
const STORE_REPORT_INTERVAL_MS = 60_000;
|
|
543
544
|
|
|
545
|
+
/** Last reported reserve, so an unchanged one stays silent. */
|
|
546
|
+
let lastReserveBytes = 0;
|
|
547
|
+
|
|
544
548
|
/** Last reported figures per store, so unchanged ones stay silent. */
|
|
545
549
|
const lastReported = new Map();
|
|
546
550
|
|
|
@@ -548,12 +552,24 @@ setInterval(() => {
|
|
|
548
552
|
// What the machine can spare NOW, not what it could spare when each store was
|
|
549
553
|
// created. With per-piece buffers a lowered ceiling is honoured immediately:
|
|
550
554
|
// excess pieces are evicted to disk and their memory is reclaimable.
|
|
555
|
+
// What the machine has been seen to need for everything that is not us. It
|
|
556
|
+
// starts at nothing and grows only on evidence, so it is worth saying when it
|
|
557
|
+
// moves — it is the one term of the budget that comes from observation of
|
|
558
|
+
// other processes rather than from our own readers.
|
|
559
|
+
const reserveBefore = machineReserveBytes();
|
|
551
560
|
for (const revised of reviseStoreBudgets()) {
|
|
552
561
|
if (revised.evicted > 0) {
|
|
553
562
|
log(
|
|
554
563
|
`piece-store "${revised.name.slice(0, 40)}": allowance is now ` +
|
|
555
|
-
`${Math.round(revised.ceilingBytes / 1048576)}MB, evicted ${revised.evicted} piece(s) to meet it
|
|
556
|
-
|
|
564
|
+
`${Math.round(revised.ceilingBytes / 1048576)}MB, evicted ${revised.evicted} piece(s) to meet it` +
|
|
565
|
+
(revised.releasedBlocks > 0 ? `, gave back ${revised.releasedBlocks} block(s) of memory` : "") +
|
|
566
|
+
` — now ${Math.round(revised.committedBytes / 1048576)}MB committed`
|
|
567
|
+
);
|
|
568
|
+
} else if (revised.belowAWindow) {
|
|
569
|
+
log(
|
|
570
|
+
`piece-store "${revised.name.slice(0, 40)}": the machine's share is smaller than one ` +
|
|
571
|
+
`reader's window, so the allowance is held at ${Math.round(revised.ceilingBytes / 1048576)}MB ` +
|
|
572
|
+
"anyway — a store that cannot hold the window of the read it is serving cannot finish that read"
|
|
557
573
|
);
|
|
558
574
|
} else if (revised.committedBytes > revised.ceilingBytes) {
|
|
559
575
|
log(
|
|
@@ -564,11 +580,20 @@ setInterval(() => {
|
|
|
564
580
|
);
|
|
565
581
|
}
|
|
566
582
|
}
|
|
583
|
+
const reserveNow = machineReserveBytes();
|
|
584
|
+
if (reserveNow !== reserveBefore || reserveNow !== lastReserveBytes) {
|
|
585
|
+
lastReserveBytes = reserveNow;
|
|
586
|
+
log(
|
|
587
|
+
`piece-store: leaving ${Math.round(reserveNow / 1048576)}MB for everything else on this ` +
|
|
588
|
+
"machine — the largest fall in available memory this process has seen and did not cause"
|
|
589
|
+
);
|
|
590
|
+
}
|
|
567
591
|
for (const stats of collectStoreStats()) {
|
|
568
592
|
const signature =
|
|
569
593
|
`${stats.fromMemory}/${stats.fromDisk}/${stats.spills}/${stats.revivals}/` +
|
|
570
594
|
`${stats.blockedByPins}/${stats.evictedOnRevise}/${stats.spillFailures}/` +
|
|
571
|
-
`${stats.evictedProtected}/${stats.demand?.unionPieces ?? 0}
|
|
595
|
+
`${stats.evictedProtected}/${stats.demand?.unionPieces ?? 0}/${stats.admittedToDisk}/` +
|
|
596
|
+
`${stats.blocksAllocated}/${stats.blocksFree}/${stats.blocksReleased}`;
|
|
572
597
|
if (lastReported.get(stats.name) === signature) {
|
|
573
598
|
continue;
|
|
574
599
|
}
|
|
@@ -581,6 +606,7 @@ setInterval(() => {
|
|
|
581
606
|
`(${Math.round((stats.residentBytes || 0) / 1048576)}MB of ` +
|
|
582
607
|
`${Math.round((stats.budgetBytes || 0) / 1048576)}MB allowed) ` +
|
|
583
608
|
`committed=${Math.round((stats.committedBytes || 0) / 1048576)}MB ` +
|
|
609
|
+
`blocks=${stats.blocksAllocated} (${stats.blocksFree} spare) ` +
|
|
584
610
|
`on-disk=${Math.round((stats.spilledBytes || 0) / 1048576)}MB ` +
|
|
585
611
|
`pinned=${stats.pinned} spilled=${stats.spilled} reads=${reads} (${fromMemoryShare}% from memory) ` +
|
|
586
612
|
`spills=${stats.spills} revivals=${stats.revivals}` +
|
|
@@ -615,19 +641,34 @@ setInterval(() => {
|
|
|
615
641
|
(age === null
|
|
616
642
|
? "; nothing has come back from disk yet"
|
|
617
643
|
: `; a revived piece had been on disk ${(age / 1000).toFixed(1)}s (median of ` +
|
|
618
|
-
`${stats.revivalAgeSamples}, ${stats.revivedWithinFiveSeconds} of them within 5s)`)
|
|
644
|
+
`${stats.revivalAgeSamples}, ${stats.revivedWithinFiveSeconds} of them within 5s)`) +
|
|
645
|
+
`; of ${stats.admittedInsideWindow + stats.admittedOutsideWindow} piece(s) admitted ` +
|
|
646
|
+
`${stats.admittedOutsideWindow} were in nobody's window, ${stats.admittedToDisk} of those ` +
|
|
647
|
+
"went straight to disk" +
|
|
648
|
+
`; ${stats.blocksAllocated} block(s) of memory exist, ${stats.blocksFree} of them spare` +
|
|
649
|
+
(stats.reuseGapMs === null
|
|
650
|
+
? ", none re-used yet"
|
|
651
|
+
: `, a block waits up to ${(stats.reuseGapMs / 1000).toFixed(1)}s before it is wanted again`) +
|
|
652
|
+
`, ${stats.blocksReleased} given back` +
|
|
653
|
+
(stats.spillsSkipped > 0
|
|
654
|
+
? `; ${stats.spillsSkipped} of ${stats.spills} eviction(s) needed no write, the disk already had them`
|
|
655
|
+
: "") +
|
|
656
|
+
(stats.returnedWhilePinned > 0
|
|
657
|
+
? `; ${stats.returnedWhilePinned} BLOCK(S) WERE RECYCLED WHILE STILL BEING READ`
|
|
658
|
+
: "")
|
|
619
659
|
);
|
|
620
660
|
}
|
|
621
661
|
}
|
|
622
662
|
}, STORE_REPORT_INTERVAL_MS).unref();
|
|
623
663
|
|
|
624
664
|
/**
|
|
625
|
-
* The piece
|
|
665
|
+
* The blocks of piece memory this thread has allocated against the pieces the
|
|
666
|
+
* stores hold in them.
|
|
626
667
|
*
|
|
627
668
|
* Not per store: the collector is per thread, and the question is about the
|
|
628
|
-
* thread.
|
|
629
|
-
*
|
|
630
|
-
*
|
|
669
|
+
* thread. With a pool one block serves many pieces, so a number of allocations
|
|
670
|
+
* that keeps climbing while the stores hold a steady number of pieces means
|
|
671
|
+
* blocks are being made and thrown away instead of re-used.
|
|
631
672
|
*
|
|
632
673
|
* @returns {string}
|
|
633
674
|
*/
|
|
@@ -639,8 +680,8 @@ function describePieceBuffers() {
|
|
|
639
680
|
const alive = collection.released - collection.collected;
|
|
640
681
|
const held = collectStoreStats().reduce((sum, stats) => sum + (stats.resident || 0), 0);
|
|
641
682
|
return (
|
|
642
|
-
`
|
|
643
|
-
`${alive} still alive against ${held} the
|
|
683
|
+
`memory blocks ${collection.released} allocated, ${collection.collected} collected, ` +
|
|
684
|
+
`${alive} still alive against ${held} piece(s) the stores hold`
|
|
644
685
|
);
|
|
645
686
|
}
|
|
646
687
|
|
|
@@ -8,9 +8,12 @@ import {
|
|
|
8
8
|
watchedFigures
|
|
9
9
|
} from "../services/memory-report.js";
|
|
10
10
|
import {
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
11
|
+
divideAllowance,
|
|
12
|
+
forgetMachineMemory,
|
|
13
|
+
machineAllowanceBytes,
|
|
14
|
+
machineReserveBytes,
|
|
15
|
+
noteMachineMemory,
|
|
16
|
+
SharedPieceStore
|
|
14
17
|
} from "../services/piece-store/shared-piece-store.js";
|
|
15
18
|
import { mkdtemp, rm } from "node:fs/promises";
|
|
16
19
|
import os from "node:os";
|
|
@@ -19,31 +22,69 @@ import path from "node:path";
|
|
|
19
22
|
const MEGABYTE = 1024 * 1024;
|
|
20
23
|
const GIGABYTE = 1024 * MEGABYTE;
|
|
21
24
|
|
|
22
|
-
test("the
|
|
23
|
-
//
|
|
24
|
-
//
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
assert.equal(
|
|
29
|
-
|
|
30
|
-
|
|
25
|
+
test("the stores are allowed what the machine has, less what others were seen to need", () => {
|
|
26
|
+
// Not a share of what is free. A share is a number chosen out of nothing, and
|
|
27
|
+
// the three that were here until 2026-09-02 — a quarter, a 64 MB floor, a
|
|
28
|
+
// 512 MB ceiling — all came from one observation of one host on 2026-08-03.
|
|
29
|
+
// `MemAvailable` is what can be taken ON TOP of what is held, so what the
|
|
30
|
+
// stores already hold is added back to get the ceiling they could reach.
|
|
31
|
+
assert.equal(
|
|
32
|
+
machineAllowanceBytes(2 * GIGABYTE, 300 * MEGABYTE, 0),
|
|
33
|
+
2 * GIGABYTE + 300 * MEGABYTE
|
|
34
|
+
);
|
|
35
|
+
assert.equal(
|
|
36
|
+
machineAllowanceBytes(2 * GIGABYTE, 300 * MEGABYTE, 500 * MEGABYTE),
|
|
37
|
+
2 * GIGABYTE - 200 * MEGABYTE
|
|
38
|
+
);
|
|
39
|
+
assert.equal(
|
|
40
|
+
machineAllowanceBytes(100 * MEGABYTE, 0, 4 * GIGABYTE),
|
|
41
|
+
0,
|
|
42
|
+
"a reserve larger than everything leaves nothing, and says so rather than going negative"
|
|
43
|
+
);
|
|
31
44
|
});
|
|
32
45
|
|
|
33
|
-
test("
|
|
34
|
-
//
|
|
35
|
-
|
|
36
|
-
//
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
assert.equal(
|
|
46
|
+
test("what to leave for everyone else is measured, starts at nothing, and ages out", () => {
|
|
47
|
+
// Field 2026-09-02: while this proxy held 76-133 MB the machine's available
|
|
48
|
+
// memory fell from 2378 MB to 306 MB and came back. That fall was somebody
|
|
49
|
+
// else's, and it is the quantity to leave room for. On a quiet host the same
|
|
50
|
+
// reading stays near zero, which is the right answer there.
|
|
51
|
+
forgetMachineMemory();
|
|
52
|
+
assert.equal(noteMachineMemory(2378 * MEGABYTE, 100 * MEGABYTE), 0, "nothing is reserved on faith");
|
|
53
|
+
|
|
54
|
+
// A fall of 2072 MB while the stores grew by 20 MB: 2052 MB of it was theirs.
|
|
55
|
+
assert.equal(noteMachineMemory(306 * MEGABYTE, 120 * MEGABYTE), 2052 * MEGABYTE);
|
|
56
|
+
|
|
57
|
+
// Memory coming back does not lower what has been seen to be needed — the
|
|
58
|
+
// spike can happen again, and that is what there has to be room for.
|
|
59
|
+
assert.equal(noteMachineMemory(3567 * MEGABYTE, 120 * MEGABYTE), 2052 * MEGABYTE);
|
|
60
|
+
|
|
61
|
+
// But it does not stand for ever either. A window of observations, not a
|
|
62
|
+
// high-water: one spike hours ago must stop squeezing the stores, which is
|
|
63
|
+
// the same mistake the block re-use gap would make with an all-time maximum.
|
|
64
|
+
for (let quiet = 0; quiet < 60; quiet += 1) {
|
|
65
|
+
noteMachineMemory(3567 * MEGABYTE, 120 * MEGABYTE);
|
|
66
|
+
}
|
|
67
|
+
assert.equal(machineReserveBytes(), 0, "an hour of quiet leaves the spike behind");
|
|
40
68
|
});
|
|
41
69
|
|
|
42
|
-
test("
|
|
43
|
-
//
|
|
44
|
-
//
|
|
45
|
-
|
|
46
|
-
assert.
|
|
70
|
+
test("everyone gets what they asked for while the asks fit", () => {
|
|
71
|
+
// The usual case by a wide margin: two readers of one film declare 32-192 MB
|
|
72
|
+
// between them against gigabytes of free memory, so the machine's limit never
|
|
73
|
+
// binds and the demand is what decides.
|
|
74
|
+
assert.deepEqual(
|
|
75
|
+
divideAllowance([100 * MEGABYTE, 50 * MEGABYTE], GIGABYTE),
|
|
76
|
+
[100 * MEGABYTE, 50 * MEGABYTE]
|
|
77
|
+
);
|
|
78
|
+
|
|
79
|
+
// When they do not fit, each is cut in proportion to what it asked, so a
|
|
80
|
+
// store wanting little is not cut to make room for one wanting much.
|
|
81
|
+
const cut = divideAllowance([300 * MEGABYTE, 100 * MEGABYTE], 200 * MEGABYTE);
|
|
82
|
+
assert.equal(cut[0], 150 * MEGABYTE);
|
|
83
|
+
assert.equal(cut[1], 50 * MEGABYTE);
|
|
84
|
+
assert.equal(cut[0] + cut[1], 200 * MEGABYTE);
|
|
85
|
+
|
|
86
|
+
assert.deepEqual(divideAllowance([], GIGABYTE), []);
|
|
87
|
+
assert.deepEqual(divideAllowance([0, 0], 0), [0, 0], "nobody asking takes nothing");
|
|
47
88
|
});
|
|
48
89
|
|
|
49
90
|
test("the memory line says bytes, and names what it could not measure", () => {
|
|
@@ -131,7 +172,7 @@ test("a thread's reading leaves out what belongs to the process", () => {
|
|
|
131
172
|
assert.doesNotMatch(line, /machine has/);
|
|
132
173
|
});
|
|
133
174
|
|
|
134
|
-
test("a store's allowance follows the machine,
|
|
175
|
+
test("a store's allowance follows the machine, in both directions", async () => {
|
|
135
176
|
// The defect: a store created on an idle machine kept an idle machine's
|
|
136
177
|
// allowance for life and went on growing into memory the host no longer had.
|
|
137
178
|
const directory = await mkdtemp(path.join(os.tmpdir(), "budget-revision-"));
|
|
@@ -149,11 +190,14 @@ test("a store's allowance follows the machine, and never passes its reservation"
|
|
|
149
190
|
assert.ok(lowered.ceilingBytes < born, "a busier machine buys fewer slots");
|
|
150
191
|
assert.equal(store.stats().budgetBytes, lowered.ceilingBytes, "the line says what is allowed now");
|
|
151
192
|
|
|
152
|
-
// The machine empties again: the ceiling
|
|
153
|
-
//
|
|
154
|
-
//
|
|
193
|
+
// The machine empties again: the ceiling rises with it, PAST where this
|
|
194
|
+
// store started. Until 2026-09-02 it could not — the ceiling was capped at
|
|
195
|
+
// a figure computed once in the constructor, so a torrent opened while the
|
|
196
|
+
// machine was full kept a small allowance for its whole life however much
|
|
197
|
+
// memory was freed afterwards.
|
|
155
198
|
const raised = store.reviseGrowthCeiling(1024 * 1024 * 1024);
|
|
156
|
-
assert.
|
|
199
|
+
assert.ok(raised.ceilingBytes > born, "an emptier machine buys more slots than it started with");
|
|
200
|
+
assert.equal(store.stats().budgetBytes, raised.ceilingBytes);
|
|
157
201
|
} finally {
|
|
158
202
|
await new Promise((resolve) => store.destroy(resolve));
|
|
159
203
|
await rm(directory, { recursive: true, force: true });
|
|
@@ -181,3 +181,223 @@ test("the store says why it spills: what is asked of it, what it had to take, ho
|
|
|
181
181
|
await fs.rm(directory, { recursive: true, force: true });
|
|
182
182
|
}
|
|
183
183
|
});
|
|
184
|
+
|
|
185
|
+
test("a piece nobody has declared does not push out one that is being read", async () => {
|
|
186
|
+
const capacity = 4;
|
|
187
|
+
const { store, directory } = await makeStore(capacity);
|
|
188
|
+
try {
|
|
189
|
+
// A reader declares the pieces it will need, and they are put in memory.
|
|
190
|
+
store.protectRange("video", 0, 3);
|
|
191
|
+
for (let index = 0; index < 4; index += 1) {
|
|
192
|
+
await put(store, index, pieceOf(index));
|
|
193
|
+
}
|
|
194
|
+
assert.equal(store.stats().resident, capacity, "the declared window fills the store");
|
|
195
|
+
|
|
196
|
+
// Now pieces arrive that the download fetched ahead of every reader. Before
|
|
197
|
+
// 2026-09-02 each of them claimed a slot and evicted one of the four above,
|
|
198
|
+
// which was then read back from disk moments later: 6565 spills and 7575
|
|
199
|
+
// revivals in 44 minutes, 53.6% of reads served from memory.
|
|
200
|
+
for (let index = 100; index < 110; index += 1) {
|
|
201
|
+
await put(store, index, pieceOf(index));
|
|
202
|
+
}
|
|
203
|
+
|
|
204
|
+
const stats = store.stats();
|
|
205
|
+
assert.equal(stats.admittedToDisk, 10, "every undeclared arrival went straight to disk");
|
|
206
|
+
assert.equal(stats.admittedOutsideWindow, 10);
|
|
207
|
+
assert.equal(stats.admittedInsideWindow, 4);
|
|
208
|
+
assert.equal(stats.spills, 0, "and nothing had to be pushed out to make room");
|
|
209
|
+
|
|
210
|
+
// The declared pieces are still in memory, and every piece reads back as
|
|
211
|
+
// itself from whichever tier holds it.
|
|
212
|
+
assert.equal(stats.resident, capacity);
|
|
213
|
+
for (const index of [0, 1, 2, 3, 100, 105, 109]) {
|
|
214
|
+
const bytes = await get(store, index);
|
|
215
|
+
assert.ok(bytes.equals(pieceOf(index)), `piece ${index} came back changed`);
|
|
216
|
+
}
|
|
217
|
+
} finally {
|
|
218
|
+
store.destroy(() => undefined);
|
|
219
|
+
await fs.rm(directory, { recursive: true, force: true });
|
|
220
|
+
}
|
|
221
|
+
});
|
|
222
|
+
|
|
223
|
+
test("before any reader has declared anything, an arriving piece still goes to memory", async () => {
|
|
224
|
+
const { store, directory } = await makeStore(4);
|
|
225
|
+
try {
|
|
226
|
+
// No window declared: there is no basis for calling a piece unwanted, so
|
|
227
|
+
// the store behaves as it always did. This is the initial download, and the
|
|
228
|
+
// warm-up fetches of the header and the tail.
|
|
229
|
+
for (let index = 0; index < 8; index += 1) {
|
|
230
|
+
await put(store, index, pieceOf(index));
|
|
231
|
+
}
|
|
232
|
+
const stats = store.stats();
|
|
233
|
+
assert.equal(stats.admittedToDisk, 0, "nothing was refused memory on a guess");
|
|
234
|
+
assert.equal(stats.admittedOutsideWindow, 8);
|
|
235
|
+
assert.ok(stats.spills > 0, "the store filled and evicted, as it did before");
|
|
236
|
+
} finally {
|
|
237
|
+
store.destroy(() => undefined);
|
|
238
|
+
await fs.rm(directory, { recursive: true, force: true });
|
|
239
|
+
}
|
|
240
|
+
});
|
|
241
|
+
|
|
242
|
+
test("the store asks for what its readers declared, and for a whole window at least", async () => {
|
|
243
|
+
const { store, directory } = await makeStore(64);
|
|
244
|
+
try {
|
|
245
|
+
// With nobody reading there is no demand to speak of, so the store asks for
|
|
246
|
+
// what it is already allowed and the first revision after a read begins
|
|
247
|
+
// brings it down.
|
|
248
|
+
const idle = store.wantedBytes;
|
|
249
|
+
assert.equal(idle, store.stats().budgetBytes);
|
|
250
|
+
|
|
251
|
+
// Two readers of one file — picture and sound — overlapping by
|
|
252
|
+
// construction. The ask is their union, not their sum.
|
|
253
|
+
store.protectRange("video", 10, 29);
|
|
254
|
+
store.protectRange("audio", 25, 44);
|
|
255
|
+
assert.equal(store.wantedBytes, 35 * PIECE, "10..44 is thirty-five pieces, not forty");
|
|
256
|
+
|
|
257
|
+
// One reader with a window wider than the union of nothing else: the ask
|
|
258
|
+
// never falls below a single window, or that reader's read cannot complete
|
|
259
|
+
// at all — every resident piece ends up pinned and the read returns zero
|
|
260
|
+
// bytes.
|
|
261
|
+
store.releaseProtection("audio");
|
|
262
|
+
store.releaseProtection("video");
|
|
263
|
+
store.protectRange("video", 0, 49);
|
|
264
|
+
assert.equal(store.wantedBytes, 50 * PIECE);
|
|
265
|
+
} finally {
|
|
266
|
+
store.destroy(() => undefined);
|
|
267
|
+
await fs.rm(directory, { recursive: true, force: true });
|
|
268
|
+
}
|
|
269
|
+
});
|
|
270
|
+
|
|
271
|
+
test("a block is re-used instead of a new one being allocated for every piece", async () => {
|
|
272
|
+
const capacity = 4;
|
|
273
|
+
const { store, directory } = await makeStore(capacity);
|
|
274
|
+
try {
|
|
275
|
+
// Twenty pieces through a store that may hold four. Before 2026-09-02 that
|
|
276
|
+
// was twenty allocations of a piece each, every one of them released only
|
|
277
|
+
// when the collector got to it — 7575 of them in 44 minutes in the field,
|
|
278
|
+
// and 1.86 GB held while the store's own accounting said 352 MB.
|
|
279
|
+
for (let index = 0; index < 20; index += 1) {
|
|
280
|
+
await put(store, index, pieceOf(index));
|
|
281
|
+
}
|
|
282
|
+
|
|
283
|
+
const stats = store.stats();
|
|
284
|
+
assert.ok(
|
|
285
|
+
stats.blocksAllocated <= capacity,
|
|
286
|
+
`the pool never exceeds the allowance: ${stats.blocksAllocated} blocks for ${capacity} slots`
|
|
287
|
+
);
|
|
288
|
+
assert.equal(stats.committedBytes, stats.blocksAllocated * PIECE);
|
|
289
|
+
assert.equal(stats.returnedWhilePinned, 0);
|
|
290
|
+
assert.ok(stats.reuseGapMs !== null, "blocks were taken from the free list, not freshly made");
|
|
291
|
+
|
|
292
|
+
// And every piece still reads back as itself: a re-used block must not
|
|
293
|
+
// carry the last piece's bytes into the next one.
|
|
294
|
+
for (let index = 0; index < 20; index += 1) {
|
|
295
|
+
const bytes = await get(store, index);
|
|
296
|
+
assert.ok(bytes.equals(pieceOf(index)), `piece ${index} came back changed`);
|
|
297
|
+
}
|
|
298
|
+
} finally {
|
|
299
|
+
store.destroy(() => undefined);
|
|
300
|
+
await fs.rm(directory, { recursive: true, force: true });
|
|
301
|
+
}
|
|
302
|
+
});
|
|
303
|
+
|
|
304
|
+
test("a spare block is given up once it has sat longer than the store's own working rhythm", async () => {
|
|
305
|
+
const { store, directory } = await makeStore(8);
|
|
306
|
+
try {
|
|
307
|
+
for (let index = 0; index < 12; index += 1) {
|
|
308
|
+
await put(store, index, pieceOf(index));
|
|
309
|
+
}
|
|
310
|
+
// The allowance falls: pieces are written out and their blocks fall spare.
|
|
311
|
+
store.reviseGrowthCeiling(2 * PIECE);
|
|
312
|
+
const spare = store.stats().blocksFree;
|
|
313
|
+
|
|
314
|
+
// Nothing is given up while the blocks are younger than the longest wait
|
|
315
|
+
// this store has actually seen between a block falling free and being
|
|
316
|
+
// wanted again.
|
|
317
|
+
assert.equal(store.sweepFreeBlocks(Date.now()), 0, "a block in use moments ago is not spare");
|
|
318
|
+
assert.equal(store.stats().blocksFree, spare);
|
|
319
|
+
|
|
320
|
+
// An hour later they plainly are.
|
|
321
|
+
const released = store.sweepFreeBlocks(Date.now() + 3_600_000);
|
|
322
|
+
assert.equal(released, spare);
|
|
323
|
+
assert.equal(store.stats().blocksFree, 0);
|
|
324
|
+
assert.equal(store.stats().blocksReleased, released);
|
|
325
|
+
} finally {
|
|
326
|
+
store.destroy(() => undefined);
|
|
327
|
+
await fs.rm(directory, { recursive: true, force: true });
|
|
328
|
+
}
|
|
329
|
+
});
|
|
330
|
+
|
|
331
|
+
test("evicting a piece the disk already holds costs no second write", async () => {
|
|
332
|
+
const { store, directory } = await makeStore(2);
|
|
333
|
+
try {
|
|
334
|
+
// Three pieces through two slots: piece 0 is written out.
|
|
335
|
+
for (let index = 0; index < 3; index += 1) {
|
|
336
|
+
await put(store, index, pieceOf(index));
|
|
337
|
+
}
|
|
338
|
+
const written = store.stats().spills;
|
|
339
|
+
assert.ok(written > 0);
|
|
340
|
+
assert.equal(store.stats().spillsSkipped, 0, "the first write of a piece is a real one");
|
|
341
|
+
|
|
342
|
+
// Read it back — it returns to memory and the copy stays on disk. Evicting
|
|
343
|
+
// it again writes bytes that are already there, byte for byte, because only
|
|
344
|
+
// `put` removes the disk copy and no `put` has happened.
|
|
345
|
+
assert.ok((await get(store, 0)).equals(pieceOf(0)));
|
|
346
|
+
for (let index = 10; index < 13; index += 1) {
|
|
347
|
+
await put(store, index, pieceOf(index));
|
|
348
|
+
}
|
|
349
|
+
assert.ok(store.stats().spillsSkipped > 0, "the second write of the same bytes is skipped");
|
|
350
|
+
|
|
351
|
+
// And the piece still comes back correctly from the disk copy.
|
|
352
|
+
assert.ok((await get(store, 0)).equals(pieceOf(0)));
|
|
353
|
+
} finally {
|
|
354
|
+
store.destroy(() => undefined);
|
|
355
|
+
await fs.rm(directory, { recursive: true, force: true });
|
|
356
|
+
}
|
|
357
|
+
});
|
|
358
|
+
|
|
359
|
+
test("a store whose readers have gone asks for nothing, one that never had them keeps its opening", async () => {
|
|
360
|
+
const { store, directory } = await makeStore(16);
|
|
361
|
+
try {
|
|
362
|
+
// Never had a reader: this is the initial download and the warm-up fetches
|
|
363
|
+
// of the header and the tail, with a read on its way.
|
|
364
|
+
const opening = store.wantedBytes;
|
|
365
|
+
assert.equal(opening, store.stats().budgetBytes);
|
|
366
|
+
|
|
367
|
+
store.protectRange("video", 0, 9);
|
|
368
|
+
assert.equal(store.wantedBytes, 10 * PIECE);
|
|
369
|
+
|
|
370
|
+
// The read ends. Its torrent sits until the pool's idle timer removes it,
|
|
371
|
+
// and that timer needs a refcount of zero and can be a quarter of an hour
|
|
372
|
+
// away. Holding the pieces for a reader that has gone is memory taken from
|
|
373
|
+
// the machine for nothing.
|
|
374
|
+
store.releaseProtection("video");
|
|
375
|
+
assert.ok(store.wantedBytes < opening, "a store with no readers left asks for nothing");
|
|
376
|
+
} finally {
|
|
377
|
+
store.destroy(() => undefined);
|
|
378
|
+
await fs.rm(directory, { recursive: true, force: true });
|
|
379
|
+
}
|
|
380
|
+
});
|
|
381
|
+
|
|
382
|
+
test("the allowance is never cut below one reader's whole window", async () => {
|
|
383
|
+
const { store, directory } = await makeStore(16);
|
|
384
|
+
try {
|
|
385
|
+
store.protectRange("video", 0, 9);
|
|
386
|
+
// The machine says this store may have two pieces. Obeying that would leave
|
|
387
|
+
// it unable to finish the read it is serving: every resident piece pinned,
|
|
388
|
+
// zero bytes returned, and ffmpeg taking that for the end of the file —
|
|
389
|
+
// which killed every encoder on that file in the field on 2026-08-15.
|
|
390
|
+
const revised = store.reviseGrowthCeiling(2 * PIECE);
|
|
391
|
+
assert.equal(revised.ceilingBytes, 10 * PIECE, "one whole window is the floor");
|
|
392
|
+
assert.equal(revised.belowAWindow, true, "and the store says the share was smaller than that");
|
|
393
|
+
|
|
394
|
+
// With no reader there is no window to protect and the share is obeyed.
|
|
395
|
+
store.releaseProtection("video");
|
|
396
|
+
const obeyed = store.reviseGrowthCeiling(2 * PIECE);
|
|
397
|
+
assert.equal(obeyed.ceilingBytes, 2 * PIECE);
|
|
398
|
+
assert.equal(obeyed.belowAWindow, false);
|
|
399
|
+
} finally {
|
|
400
|
+
store.destroy(() => undefined);
|
|
401
|
+
await fs.rm(directory, { recursive: true, force: true });
|
|
402
|
+
}
|
|
403
|
+
});
|