@torrent-tv/proxy 2.69.1 → 2.69.2

package/CHANGELOG.md

@@ -1,3 +1,10 @@
+## 2.69.2
+
+- **New**: The piece store says WHY it spills, which no reading has ever answered. A session on 2026-09-02 did 6565 spills and 7575 revivals in 44 minutes with only 53.6 % of reads served from memory, and nothing recorded whether that was an eviction order fighting the read order or a working set that simply does not fit. Three figures now settle it, on one line per store: what the live readers between them are asking to keep against what the store may hold; how many evictions had to take a piece a reader had declared it wants; and how long a revived piece had been on disk before it was wanted back.
+- **Chore**: The demand is the UNION of the readers' windows, not their sum. Two readers of one file — picture and sound — overlap by construction, and summing them would report the store as short when it is not. A union wider than the capacity cannot be held however the eviction is ordered, which is the difference between a policy to fix and arithmetic to accept.
+- **Chore**: `PieceLru.evictionChoice()` returns the victim with the two facts about it — whether protection had to yield, and how many pieces the victim lies from the nearest declared window, zero inside one and -1 when nothing is declared. `evictionCandidate()` is kept and delegates, so nothing else moved.
+- **Chore**: A revived piece's age is kept as a bounded window of the last 200, because the figure wanted is a median rather than a history. The line also says how many of those came back within five seconds — a piece wanted again that soon should not have left. Seven checks in `test/piece-lru.test.js` and `test/piece-store-eviction.test.js`.
+
 ## 2.69.1
 
 - **Fix**: The change trigger on the torrent worker's memory line watched the one figure that does not move. A thread's watched quantity was `heapTotal`, and through the session of 2026-09-02 that stood at 31-173 MB while the same isolate's `arrayBuffers` swung between 130 and 950 MB — so nothing ever earned a line and every reading of the quantity that grew came out on the quiet minute. Each of `heapTotal`, `external` and `arrayBuffers` is now compared against its own last written value and any one of them moving writes the line. Nothing is summed: `arrayBuffers` is documented as part of `external` and reads larger than it here, and this code has no business resolving that.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@torrent-tv/proxy",
-  "version": "2.69.1",
+  "version": "2.69.2",
   "description": "Torrent proxy client that exposes webseed-like HTTP stream endpoint.",
   "license": "GPL-3.0-or-later",
   "publishConfig": {

package/services/piece-store/piece-lru.js

@@ -174,11 +174,31 @@ export class PieceLru {
    * @returns {number | null}
    */
   evictionCandidate() {
+    return this.evictionChoice().index;
+  }
+
+  /**
+   * The same choice, with the two facts that say whether the store is working
+   * or thrashing: whether protection had to yield, and how far the victim was
+   * from the nearest piece a reader declared it wants.
+   *
+   * Evicting a stale piece nobody asked for is the store doing its job.
+   * Evicting a piece inside a reader's own declared window is the store being
+   * asked to hold more than it has room for, and it comes back from disk
+   * moments later — 6565 spills and 7575 revivals in 44 minutes on
+   * 2026-09-02, with only 53.6% of reads served from memory. Nothing recorded
+   * which of the two was happening (roadmap item 9).
+   *
+   * @returns {{ index: number | null, protectionYielded: boolean, distance: number }}
+   *   `distance` is in pieces from the nearest declared window, zero when the
+   *   victim is inside one, and -1 when no reader has declared anything.
+   */
+  evictionChoice() {
     // First choice: the least recently used piece nobody is reading and nobody
     // is about to read.
     for (const index of this.#order) {
       if (!this.#pins.has(index) && !this.#isProtected(index)) {
-        return index;
+        return { index, protectionYielded: false, distance: this.#distanceToWindow(index) };
       }
     }
     // Nothing spare left. Protection yields — it is a preference, and refusing
@@ -186,10 +206,66 @@ export class PieceLru {
     // yield: a piece being read now cannot have its memory taken away.
     for (const index of this.#order) {
       if (!this.#pins.has(index)) {
-        return index;
+        return { index, protectionYielded: true, distance: this.#distanceToWindow(index) };
+      }
+    }
+    return { index: null, protectionYielded: false, distance: -1 };
+  }
+
+  /**
+   * How many pieces the live readers between them are asking to keep, against
+   * how many this store may hold.
+   *
+   * The union, not the sum: two readers of one file overlap, and counting the
+   * overlap twice would say the store is short when it is not. This is the
+   * comparison that decides whether thrashing is a policy fault or arithmetic —
+   * a union wider than the capacity cannot be held however the eviction is
+   * ordered.
+   *
+   * @returns {{ readers: number, unionPieces: number, widestPieces: number, capacity: number }}
+   */
+  demand() {
+    const ranges = [...this.#protected.values()]
+      .map((range) => ({ from: range.from, to: range.to }))
+      .sort((left, right) => left.from - right.from);
+    let unionPieces = 0;
+    let widestPieces = 0;
+    let coveredTo = -Infinity;
+    for (const range of ranges) {
+      const width = range.to - range.from + 1;
+      widestPieces = Math.max(widestPieces, width);
+      const from = Math.max(range.from, coveredTo + 1);
+      if (range.to >= from) {
+        unionPieces += range.to - from + 1;
+        coveredTo = range.to;
+      }
+    }
+    return {
+      readers: ranges.length,
+      unionPieces,
+      widestPieces,
+      capacity: this.#capacity
+    };
+  }
+
+  /**
+   * Pieces from `index` to the nearest declared window, zero inside one and -1
+   * when nothing is declared.
+   *
+   * @param {number} index
+   * @returns {number}
+   */
+  #distanceToWindow(index) {
+    let nearest = -1;
+    for (const range of this.#protected.values()) {
+      const gap = index < range.from
+        ? range.from - index
+        : index > range.to ? index - range.to : 0;
+      if (nearest === -1 || gap < nearest) {
+        nearest = gap;
       }
     }
-    return null;
+    return nearest;
   }
 
   /**

package/services/piece-store/shared-piece-store.js

@@ -151,6 +151,32 @@ const MIN_RESIDENT_PIECES = 2;
  * never failing (field 2026-08-31).
  */
 const PINNED_WAIT_MS = 5_000;
+
+/**
+ * How many revival ages are kept for the median. A window, not a history: two
+ * hundred covers several minutes of the busiest session measured (7575
+ * revivals in 44 minutes) and costs two hundred numbers.
+ */
+const REVIVAL_AGE_SAMPLES = 200;
+
+/**
+ * The middle value of a sample, or null when there is nothing to take a middle
+ * of. Null rather than zero: no revivals and instant revivals are different
+ * facts and must not print the same.
+ *
+ * @param {number[]} values
+ * @returns {number | null}
+ */
+function median(values) {
+  if (values.length === 0) {
+    return null;
+  }
+  const sorted = [...values].sort((left, right) => left - right);
+  const middle = Math.floor(sorted.length / 2);
+  return sorted.length % 2 === 0
+    ? Math.round((sorted[middle - 1] + sorted[middle]) / 2)
+    : sorted[middle];
+}
 const CLAIM_RETRY_MS = 50;
 
 export class SharedPieceStore {
@@ -191,8 +217,25 @@ export class SharedPieceStore {
     blockedByPins: 0,
     waitedForPins: 0,
     evictedOnRevise: 0,
-    spillFailures: 0
+    spillFailures: 0,
+    // Whether the store is doing its job or being asked to hold more than it
+    // has room for. An eviction that had to take a piece a reader declared it
+    // wants is the second, and it comes back from disk moments later
+    // (roadmap item 9).
+    evictedProtected: 0,
+    evictedDistanceSum: 0,
+    evictedWithDistance: 0
   };
+  /** Piece index → when it was written out, for the age it comes back at. */
+  #spilledAt = new Map();
+  /**
+   * Ages, in milliseconds, of the last revivals — bounded, because the figure
+   * wanted is a median and not a history. A piece that comes back seconds
+   * after it left should not have left.
+   *
+   * @type {number[]}
+   */
+  #revivalAges = [];
 
   constructor(chunkLength, options = {}) {
     if (!Number.isInteger(chunkLength) || chunkLength < 1) {
@@ -244,6 +287,14 @@ export class SharedPieceStore {
       outstanding: this.#outstandingPieces,
       spilled: this.#disk.size,
       spilledBytes: this.#disk.size * this.#chunkLength,
+      // What the readers between them are asking this store to keep, against
+      // what it may hold. A union wider than the capacity cannot be held
+      // however the eviction is ordered, and that is the difference between a
+      // policy to fix and arithmetic to accept (roadmap item 9).
+      demand: this.#lru.demand(),
+      revivalAgeMedianMs: median(this.#revivalAges),
+      revivalAgeSamples: this.#revivalAges.length,
+      revivedWithinFiveSeconds: this.#revivalAges.filter((age) => age <= 5_000).length,
       ...this.#counters
     };
   }
@@ -263,7 +314,7 @@ export class SharedPieceStore {
     // old growable pool. Eagerly evict excess to honour the new ceiling.
     let evicted = 0;
     while (this.#buffers.size > this.#growthCeiling) {
-      const victim = this.#lru.evictionCandidate();
+      const { index: victim, protectionYielded, distance } = this.#lru.evictionChoice();
       if (victim === null) {
         break;
       }
@@ -276,6 +327,7 @@ export class SharedPieceStore {
       this.#lru.remove(victim);
       evicted += 1;
       this.#counters.evictedOnRevise += 1;
+      this.#noteEviction(protectionYielded, distance);
       // Nobody awaits this spill, so its failure has to end here. Rethrowing
       // made it an unhandled rejection, and an unhandled rejection in the
       // torrent worker ends the thread — a second way to lose the torrent
@@ -421,6 +473,7 @@ export class SharedPieceStore {
     const spill = this.#disk.write(index, bytes).then(
       () => {
         this.#counters.spills += 1;
+        this.#spilledAt.set(index, Date.now());
         this.#evicting.delete(index);
         this.#noteProgress();
       },
@@ -440,6 +493,48 @@ export class SharedPieceStore {
     this.#wake();
   }
 
+  /**
+   * Record how long a piece stayed on disk before it was wanted again.
+   *
+   * A piece that comes back seconds after it left was evicted from a working
+   * set that does not fit, and the write and the read were both waste. Kept as
+   * a bounded window of ages because the figure wanted is a median, not a
+   * history.
+   *
+   * @param {number} index
+   * @returns {void}
+   */
+  #noteRevival(index) {
+    const spilledAt = this.#spilledAt.get(index);
+    if (spilledAt === undefined) {
+      return;
+    }
+    this.#spilledAt.delete(index);
+    this.#revivalAges.push(Date.now() - spilledAt);
+    if (this.#revivalAges.length > REVIVAL_AGE_SAMPLES) {
+      this.#revivalAges.shift();
+    }
+  }
+
+  /**
+   * Record what an eviction had to take.
+   *
+   * @param {boolean} protectionYielded - The victim was inside a window a
+   *   reader had declared, and was taken anyway because nothing else was free.
+   * @param {number} distance - Pieces from the nearest declared window, -1 when
+   *   no reader declared one.
+   * @returns {void}
+   */
+  #noteEviction(protectionYielded, distance) {
+    if (protectionYielded) {
+      this.#counters.evictedProtected += 1;
+    }
+    if (distance >= 0) {
+      this.#counters.evictedDistanceSum += distance;
+      this.#counters.evictedWithDistance += 1;
+    }
+  }
+
   #wake() {
     const waiting = this.#waiters;
     this.#waiters = [];
@@ -456,7 +551,7 @@ export class SharedPieceStore {
       return true;
     }
 
-    const victim = this.#lru.evictionCandidate();
+    const { index: victim, protectionYielded, distance } = this.#lru.evictionChoice();
     if (victim === null) {
       // Nothing may leave. Wait while the store is still MOVING — a spill
       // completing, a piece admitted, a pin released — and give up when it has
@@ -491,6 +586,7 @@ export class SharedPieceStore {
     this.#buffers.delete(victim);
     this.#lru.remove(victim);
     this.#outstandingPieces += 1;
+    this.#noteEviction(protectionYielded, distance);
 
     try {
       await this.#spill(victim, victimBuffer);
@@ -538,6 +634,7 @@ export class SharedPieceStore {
       this.#registerPiece(index, target);
       this.#counters.fromDisk += 1;
       this.#counters.revivals += 1;
+      this.#noteRevival(index);
       return target;
     } finally {
       release();
@@ -593,6 +690,7 @@ export class SharedPieceStore {
       await spill.catch(() => undefined);
     }
     this.#disk.forget(index);
+    this.#spilledAt.delete(index);
   }
 
   put(index, bytes, callback = () => undefined) {
@@ -707,6 +805,7 @@ export class SharedPieceStore {
     this.#closed = true;
     liveStores.delete(this);
     this.#buffers.clear();
+    this.#spilledAt.clear();
     // Whoever is waiting for a slot is woken and finds the store closed, which
     // is an error they can report. Left asleep they simply never returned.
     this.#wake();
@@ -717,6 +816,7 @@ export class SharedPieceStore {
     this.#closed = true;
     liveStores.delete(this);
     this.#buffers.clear();
+    this.#spilledAt.clear();
     this.#wake();
     this.#disk.destroy().then(() => callback(null), (error) => callback(error));
   }

package/services/torrent-worker/worker.js

@@ -565,7 +565,10 @@ setInterval(() => {
     }
   }
   for (const stats of collectStoreStats()) {
-    const signature = `${stats.fromMemory}/${stats.fromDisk}/${stats.spills}/${stats.revivals}/${stats.blockedByPins}/${stats.evictedOnRevise}/${stats.spillFailures}`;
+    const signature =
+      `${stats.fromMemory}/${stats.fromDisk}/${stats.spills}/${stats.revivals}/` +
+      `${stats.blockedByPins}/${stats.evictedOnRevise}/${stats.spillFailures}/` +
+      `${stats.evictedProtected}/${stats.demand?.unionPieces ?? 0}`;
     if (lastReported.get(stats.name) === signature) {
       continue;
     }
@@ -586,6 +589,35 @@ setInterval(() => {
       (stats.spillFailures > 0 ? ` spill-failures=${stats.spillFailures}` : "") +
       (stats.outstanding > 0 ? ` outstanding=${stats.outstanding}` : "")
     );
+    // Why it spills, on its own line because it is a different question from
+    // how much it holds. Three facts, and between them they say whether the
+    // thrashing is a policy to fix or arithmetic to accept: what the readers
+    // together are asking to keep against what the store may hold; how many
+    // evictions had to take a piece a reader had declared it wants; and how
+    // long a piece stayed on disk before it was wanted back. On 2026-09-02 a
+    // session did 6565 spills and 7575 revivals with 53.6% of reads served
+    // from memory, and nothing recorded which of the three was the cause
+    // (roadmap item 9).
+    const demand = stats.demand;
+    if (demand && demand.readers > 0) {
+      const age = stats.revivalAgeMedianMs;
+      log(
+        `piece-store "${stats.name.slice(0, 40)}" demand: ${demand.readers} reader(s) want ` +
+        `${demand.unionPieces} piece(s) of ${demand.capacity} the store may hold ` +
+        `(widest window ${demand.widestPieces})` +
+        (stats.evictedProtected > 0
+          ? `; ${stats.evictedProtected} of ${stats.spills} eviction(s) took a piece a reader had declared`
+          : "; no eviction has taken a declared piece") +
+        (stats.evictedWithDistance > 0
+          ? `, a victim lay ${(stats.evictedDistanceSum / stats.evictedWithDistance).toFixed(1)} ` +
+            "piece(s) from the nearest window on average"
+          : "") +
+        (age === null
+          ? "; nothing has come back from disk yet"
+          : `; a revived piece had been on disk ${(age / 1000).toFixed(1)}s (median of ` +
+            `${stats.revivalAgeSamples}, ${stats.revivedWithinFiveSeconds} of them within 5s)`)
+      );
+    }
   }
 }, STORE_REPORT_INTERVAL_MS).unref();
 

package/test/piece-lru.test.js

@@ -189,3 +189,66 @@ test("the capacity follows the store's live allowance", () => {
   lru.setCapacity(0);
   assert.equal(lru.capacity, 2, "a capacity below one is refused, not obeyed");
 });
+
+test("the demand is the union of the readers' windows, not their sum", () => {
+  const lru = new PieceLru(88);
+  // Two readers of one file — picture and sound — overlap by construction.
+  // Summing them would say the store is short when it is not.
+  lru.protect("video", 100, 149);
+  lru.protect("audio", 130, 179);
+
+  const demand = lru.demand();
+  assert.equal(demand.readers, 2);
+  assert.equal(demand.unionPieces, 80, "100..179 is eighty pieces, not a hundred");
+  assert.equal(demand.widestPieces, 50);
+  assert.equal(demand.capacity, 88);
+
+  lru.protect("second-viewer", 900, 979);
+  assert.equal(lru.demand().unionPieces, 160, "windows that do not touch add up");
+
+  lru.unprotect("second-viewer");
+  lru.unprotect("audio");
+  lru.unprotect("video");
+  assert.deepEqual(
+    lru.demand(),
+    { readers: 0, unionPieces: 0, widestPieces: 0, capacity: 88 },
+    "no reader asking for anything is not the same as asking for one piece"
+  );
+});
+
+test("an eviction says whether it had to take a piece a reader declared", () => {
+  const lru = new PieceLru(3);
+  lru.touch(10);
+  lru.touch(11);
+  lru.touch(12);
+  lru.protect("video", 11, 12);
+
+  const spare = lru.evictionChoice();
+  assert.equal(spare.index, 10, "the piece outside every window goes first");
+  assert.equal(spare.protectionYielded, false);
+  assert.equal(spare.distance, 1, "one piece away from the window at 11");
+
+  // Nothing spare left: both survivors are inside the declared window.
+  lru.remove(10);
+  const forced = lru.evictionChoice();
+  assert.equal(forced.index, 11);
+  assert.equal(forced.protectionYielded, true, "the store is holding less than it is asked to");
+  assert.equal(forced.distance, 0, "inside a window");
+
+  assert.equal(lru.evictionCandidate(), 11, "the older answer is the same choice");
+});
+
+test("with nothing declared there is no distance to report", () => {
+  const lru = new PieceLru(2);
+  lru.touch(7);
+  const choice = lru.evictionChoice();
+  assert.equal(choice.index, 7);
+  assert.equal(choice.distance, -1, "-1 is absence, and 0 would read as inside a window");
+
+  lru.pin(7);
+  assert.deepEqual(
+    lru.evictionChoice(),
+    { index: null, protectionYielded: false, distance: -1 },
+    "a pinned piece is never a candidate, and says so without a distance"
+  );
+});

package/test/piece-store-eviction.test.js

@@ -132,3 +132,52 @@ test("pinned pieces are never evicted, and the pin count is reported", async ()
     await fs.rm(directory, { recursive: true, force: true });
   }
 });
+
+test("the store says why it spills: what is asked of it, what it had to take, how soon it came back", async () => {
+  const capacity = 4;
+  const { store, directory } = await makeStore(capacity);
+  try {
+    // A reader declaring more than the store may hold. This is the shape the
+    // field session of 2026-09-02 is suspected of — 88 slots against an
+    // encoder running 120-380 s ahead of a viewer, half the reads missing —
+    // and nothing recorded it (roadmap item 9).
+    store.protectRange("video", 0, 9);
+    for (let index = 0; index < 10; index += 1) {
+      await put(store, index, pieceOf(index));
+    }
+
+    const asked = store.stats();
+    assert.equal(asked.demand.readers, 1);
+    assert.equal(asked.demand.unionPieces, 10);
+    assert.equal(asked.demand.capacity, capacity);
+    assert.ok(
+      asked.demand.unionPieces > asked.demand.capacity,
+      "a reader asking for more than the store holds is arithmetic, not a policy fault"
+    );
+    assert.ok(
+      asked.evictedProtected > 0,
+      "every eviction here had to take a piece the reader had declared"
+    );
+
+    // Read back the pieces that were spilled: each one comes home, and the
+    // store says how long it had been away.
+    for (let index = 0; index < 10; index += 1) {
+      const bytes = await get(store, index);
+      assert.ok(bytes.equals(pieceOf(index)), `piece ${index} came back changed`);
+    }
+
+    const after = store.stats();
+    assert.ok(after.revivalAgeSamples > 0, "pieces came back and their age was recorded");
+    assert.equal(typeof after.revivalAgeMedianMs, "number");
+    assert.ok(
+      after.revivedWithinFiveSeconds > 0,
+      "a piece wanted again seconds after it left should not have left"
+    );
+
+    store.releaseProtection("video");
+    assert.equal(store.stats().demand.readers, 0, "a reader that ends stops being counted");
+  } finally {
+    store.destroy(() => undefined);
+    await fs.rm(directory, { recursive: true, force: true });
+  }
+});