@torrent-tv/proxy 2.69.1 → 2.70.0

package/CHANGELOG.md

@@ -1,3 +1,24 @@
+## 2.70.0
+
+- **New**: The piece store keeps a pool of memory blocks instead of allocating one per piece. A block is one piece's worth of memory; it is taken from the free list, and put back there when its piece is written out. Field 2026-09-02: 7575 allocations of 4 MiB in 44 minutes, each released only when the collector reached it, which is why the process held 1.86 GB while the store's own accounting said 352 MB. A block goes back for re-use only AFTER the spill write has finished, because that write reads out of it; a block whose piece is re-put while a reader holds it is given up rather than recycled, and a counter says if one ever is not.
+- **Fix**: The store's allowance could only ever fall. `#capacity` was computed once in the constructor and used as an upper bound on every revision, so a torrent opened while the machine was full kept a small allowance for its whole life however much memory was freed afterwards. The field is gone; the opening figure is now only where the store starts.
+- **Fix**: `AVAILABLE_MEMORY_SHARE` (a quarter), `MIN_BUDGET_BYTES` (64 MB) and `MEMORY_BUDGET_CEILING_BYTES` (512 MB) are removed. All three trace to one observation of one host on 2026-08-03 and none was derived. The budget is now the smaller of what the readers have declared — the union of their windows, since picture and sound overlap — and what the machine allows, which is `MemAvailable` plus what the stores already hold, less what other processes have recently been seen to need. That last figure starts at zero and grows only on evidence: the fall in available memory between two readings, beyond what the stores themselves took.
+- **Fix**: A piece no reader has declared, arriving at a store with no room, goes straight to disk instead of pushing out a piece a reader is about to read. It costs the same one write it would have cost when the next arrival evicted it. Before the first read nothing is declared and nothing is refused memory on a guess.
+- **Fix**: Evicting a piece the disk already holds costs no second write. `#revive` reads a piece back and leaves the copy on disk, and only `put` removes it, so a piece revived and not re-put is identical to what is already written. There were 7575 revivals in that one session (roadmap item 66: 14.4 GB written in a single viewing).
+- **Fix**: A store whose readers have GONE asks for nothing, where before it kept asking for what it held. Its torrent sits until the pool's idle timer removes it, and that timer needs a refcount of zero and can be a quarter of an hour away. A store that has never had a reader is a different case and keeps its opening share.
+- **Fix**: A store is never cut below one reader's whole window, whatever the machine's share says. Obeying a smaller share would leave it unable to finish the read it is serving: every resident piece pinned, zero bytes returned, ffmpeg taking that for the end of the file — which killed every encoder on that file on 2026-08-15. The line says when the share was smaller than the window.
+- **Chore**: What other processes need is a window of the last sixty observations, not a high-water. A single spike would otherwise squeeze the stores for the life of the process, which is the same mistake an all-time maximum makes of the block re-use gap.
+- **New**: A spare block is given up once it has sat unused longer than the store's own working rhythm — the longest wait, over recent work, between a block falling free and being wanted again. Measured, not chosen: while a film is being watched a block is taken again within milliseconds, because one is taken for every piece that arrives.
+- **New**: The store line says how many blocks of memory exist and how many are spare, how long a block waits before it is wanted again, how many were given back, how many pieces were admitted that were in nobody's window, and how many evictions needed no write. `committed` now means what the process holds — the blocks — rather than the pieces in them.
+- **Chore**: Thirteen checks across `test/piece-lru.test.js`, `test/piece-store-eviction.test.js` and `test/memory-budget.test.js`, including that a re-used block never carries the previous piece's bytes into the next one.
+
+## 2.69.2
+
+- **New**: The piece store says WHY it spills, which no reading has ever answered. A session on 2026-09-02 did 6565 spills and 7575 revivals in 44 minutes with only 53.6 % of reads served from memory, and nothing recorded whether that was an eviction order fighting the read order or a working set that simply does not fit. Three figures now settle it, on one line per store: what the live readers between them are asking to keep against what the store may hold; how many evictions had to take a piece a reader had declared it wants; and how long a revived piece had been on disk before it was wanted back.
+- **Chore**: The demand is the UNION of the readers' windows, not their sum. Two readers of one file — picture and sound — overlap by construction, and summing them would report the store as short when it is not. A union wider than the capacity cannot be held however the eviction is ordered, which is the difference between a policy to fix and arithmetic to accept.
+- **Chore**: `PieceLru.evictionChoice()` returns the victim with the two facts about it — whether protection had to yield, and how many pieces the victim lies from the nearest declared window, zero inside one and -1 when nothing is declared. `evictionCandidate()` is kept and delegates, so nothing else moved.
+- **Chore**: A revived piece's age is kept as a bounded window of the last 200, because the figure wanted is a median rather than a history. The line also says how many of those came back within five seconds — a piece wanted again that soon should not have left. Seven checks in `test/piece-lru.test.js` and `test/piece-store-eviction.test.js`.
+
 ## 2.69.1
 
 - **Fix**: The change trigger on the torrent worker's memory line watched the one figure that does not move. A thread's watched quantity was `heapTotal`, and through the session of 2026-09-02 that stood at 31-173 MB while the same isolate's `arrayBuffers` swung between 130 and 950 MB — so nothing ever earned a line and every reading of the quantity that grew came out on the quiet minute. Each of `heapTotal`, `external` and `arrayBuffers` is now compared against its own last written value and any one of them moving writes the line. Nothing is summed: `arrayBuffers` is documented as part of `external` and reads larger than it here, and this code has no business resolving that.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@torrent-tv/proxy",
-  "version": "2.69.1",
+  "version": "2.70.0",
   "description": "Torrent proxy client that exposes webseed-like HTTP stream endpoint.",
   "license": "GPL-3.0-or-later",
   "publishConfig": {

package/services/piece-store/piece-lru.js

@@ -174,11 +174,31 @@ export class PieceLru {
    * @returns {number | null}
    */
   evictionCandidate() {
+    return this.evictionChoice().index;
+  }
+
+  /**
+   * The same choice, with the two facts that say whether the store is working
+   * or thrashing: whether protection had to yield, and how far the victim was
+   * from the nearest piece a reader declared it wants.
+   *
+   * Evicting a stale piece nobody asked for is the store doing its job.
+   * Evicting a piece inside a reader's own declared window is the store being
+   * asked to hold more than it has room for, and it comes back from disk
+   * moments later — 6565 spills and 7575 revivals in 44 minutes on
+   * 2026-09-02, with only 53.6% of reads served from memory. Nothing recorded
+   * which of the two was happening (roadmap item 9).
+   *
+   * @returns {{ index: number | null, protectionYielded: boolean, distance: number }}
+   *   `distance` is in pieces from the nearest declared window, zero when the
+   *   victim is inside one, and -1 when no reader has declared anything.
+   */
+  evictionChoice() {
     // First choice: the least recently used piece nobody is reading and nobody
     // is about to read.
     for (const index of this.#order) {
       if (!this.#pins.has(index) && !this.#isProtected(index)) {
-        return index;
+        return { index, protectionYielded: false, distance: this.#distanceToWindow(index) };
       }
     }
     // Nothing spare left. Protection yields — it is a preference, and refusing
@@ -186,10 +206,96 @@ export class PieceLru {
     // yield: a piece being read now cannot have its memory taken away.
     for (const index of this.#order) {
       if (!this.#pins.has(index)) {
-        return index;
+        return { index, protectionYielded: true, distance: this.#distanceToWindow(index) };
+      }
+    }
+    return { index: null, protectionYielded: false, distance: -1 };
+  }
+
+  /**
+   * How many pieces the live readers between them are asking to keep, against
+   * how many this store may hold.
+   *
+   * The union, not the sum: two readers of one file overlap, and counting the
+   * overlap twice would say the store is short when it is not. This is the
+   * comparison that decides whether thrashing is a policy fault or arithmetic —
+   * a union wider than the capacity cannot be held however the eviction is
+   * ordered.
+   *
+   * @returns {{ readers: number, unionPieces: number, widestPieces: number, capacity: number }}
+   */
+  demand() {
+    const ranges = [...this.#protected.values()]
+      .map((range) => ({ from: range.from, to: range.to }))
+      .sort((left, right) => left.from - right.from);
+    let unionPieces = 0;
+    let widestPieces = 0;
+    let coveredTo = -Infinity;
+    for (const range of ranges) {
+      const width = range.to - range.from + 1;
+      widestPieces = Math.max(widestPieces, width);
+      const from = Math.max(range.from, coveredTo + 1);
+      if (range.to >= from) {
+        unionPieces += range.to - from + 1;
+        coveredTo = range.to;
+      }
+    }
+    return {
+      readers: ranges.length,
+      unionPieces,
+      widestPieces,
+      capacity: this.#capacity
+    };
+  }
+
+  /**
+   * Whether a reader is holding this piece right now.
+   *
+   * Asked before a block is put back for re-use: a pinned piece has a view onto
+   * its memory somewhere, and handing that memory to another piece would let
+   * the holder read bytes that are not its own.
+   *
+   * @param {number} index
+   * @returns {boolean}
+   */
+  isPinned(index) {
+    return this.#pins.has(index);
+  }
+
+  /**
+   * Whether any live reader has declared it will want this piece.
+   *
+   * Asked on admission, not only on eviction: a piece nobody has declared is
+   * being downloaded ahead of every reader, and putting it in memory means
+   * pushing out one that IS declared — which is then read back from disk
+   * moments later. Measured 2026-09-02: 6565 spills and 7575 revivals in 44
+   * minutes with 53.6 % of reads served from memory (roadmap item 9).
+   *
+   * @param {number} index
+   * @returns {boolean}
+   */
+  wants(index) {
+    return this.#isProtected(index);
+  }
+
+  /**
+   * Pieces from `index` to the nearest declared window, zero inside one and -1
+   * when nothing is declared.
+   *
+   * @param {number} index
+   * @returns {number}
+   */
+  #distanceToWindow(index) {
+    let nearest = -1;
+    for (const range of this.#protected.values()) {
+      const gap = index < range.from
+        ? range.from - index
+        : index > range.to ? index - range.to : 0;
+      if (nearest === -1 || gap < nearest) {
+        nearest = gap;
       }
     }
-    return null;
+    return nearest;
   }
 
   /**