@torrent-tv/proxy 2.38.1 → 2.39.1

package/CHANGELOG.md

@@ -1,3 +1,12 @@
+## 2.39.1
+
+- **Fix**: The block duplication of 2.39.0 is removed, because measured against what a viewer actually feels it never paid. 2.39.0 was shipped on a measurement of the median wait for a piece; the quantity that matters is the seconds the picture stands still, and measured on that (2026-08-19, a reader paced at the film's own byte rate with an eight-second lead, arms alternated per position) it is neutral at best and costly at worst. On a well-seeded film every arm read 67 MB in 59 s and stopped for at most 2.6 s — nothing to improve. With the download capped just above the film's rate, which is what a home line IS whenever the swarm can fill it, duplication made the picture stop for **28.8 s against 12.8 s** at one position and left the other unchanged: under one shared budget a duplicate spends the very bytes it is trying to go around. The regime where it could pay — several slow peers, each with capacity of its own — could not be reproduced: the thin-swarm candidate turned out to have no live swarm at all (4 peers, 179.5 s of a 209 s run spent stopped). A lever with no measured gain and a measured cost does not stay on by default. Narrowing the read window to the blocked piece was tried in the same experiment and is not shipped for the same reason: it was never better and reached **44.8 s against 12.8 s**. What stays is the measurement that decided it (`research/tail-duplication-2026-08-19.md`) and the `tail …` line from 2.38.1, which is what will say whether a real thin swarm ever behaves differently.
+
+## 2.39.0
+
+- **New**: When a reader is blocked and nothing can be steered, the blocks it is still waiting on are asked of a second wire as well. WebTorrent reserves each block for exactly one wire, so once `Piece.reserve()` answers -1 the read ends when the holder of the last block delivers it, however fast the rest of the swarm is. The library's own `_hotswap` does exactly the right thing — `piece.cancel(block)` frees the reservation while the first request stays in flight — but only for a wire under 48 KB/s and twice as slow as the asker, and measured on a real swarm the tails a reader waits on sit at 109-886 KB/s. Peers the library rightly calls good, because for bulk downloading they are; the gate is about throughput across a torrent and knows nothing about a reader blocked on one piece now. Speed is not even what is wrong with them: two blocks — 32 KB — on a wire measured at 109 KB/s is 0.3 s of transfer, and that read waited 4.6 s, because the blocks are queued behind that wire's other work. **Measured against itself, same film, same positions, arms alternated, six pairs across two pacing rates: the median wait for a piece fell in all six — 39 %, 48 %, 68 %, 6 %, 50 % and 28 %.** In the one pair where the swarm had spare capacity throughout, the wait fell 7226→5174 ms and the lead the reader kept ended at +2.0 MB instead of +0.3 MB. Cost: one duplicate per candidate wire per attempt, about 3 % extra traffic, and only while a reader is blocked with every block already spoken for. The tail measurement added in 2.38.1 reports what it placed: `duplicated 14 blocks`.
+- **Chore**: The first attempt at this was reverted the same day on a measurement that could not have shown it. That probe read 64 MB flat out, so its total time was bytes ÷ aggregate rate by construction — a quantity no reordering of requests can change. Reordering moves latency between pieces; it does not add throughput. The corrected probe consumes at a film's own byte rate and keeps a lead, which is the state a viewer is in, and there the effect is plain. Recorded in `research/blocked-piece-tail-2026-08-19.md` so the next such decision is not made on the wrong quantity.
+
 ## 2.38.1
 
 - **New**: When a blocked piece cannot be steered anywhere, the wait line says what is holding it. The steering added in 2.29.0 often places nothing — `steered onto 0 of 9 asks (8 peers held it)`, measured 2026-08-18 while eight peers had the piece — because every block is already reserved and WebTorrent will not hand out a second request for the same block (`Piece.reserve()` answers -1; the only mention of an endgame in the library is a commented-out line). Duplicating those blocks is the standard remedy and costs a block's traffic each time, so this measures the tail before anything is built on it: `tail 2/512 blocks missing, held by 1@12KB/s 1@900KB/s`, slowest wire first, and `held by nobody` when the piece has not been asked for at all. Sampled at the instant an attempt placed nothing rather than once at the start, so the numbers and the reason they are printed describe the same moment. If the missing blocks turn out to sit on one slow wire, duplication is aimed at the right thing; if they are spread across fast ones, the wait has another cause and that work should not be done.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@torrent-tv/proxy",
-  "version": "2.38.1",
+  "version": "2.39.1",
   "description": "Torrent proxy client that exposes webseed-like HTTP stream endpoint.",
   "license": "GPL-3.0-or-later",
   "publishConfig": {

package/services/torrent-worker/fastest-wires.js

@@ -1,199 +1,199 @@
-/**
- * @file Put the piece a reader is blocked on onto the fastest wires that hold
- * it.
- *
- * Measured 2026-08-17: the swarm delivered 5.1-5.9 MB/s against a film consumed
- * at about 1 MB/s — a fivefold surplus — and the reader still blocked 47 times
- * in two minutes, 1.0-4.5 s each. So the shortage is not bandwidth. The piece
- * that blocked had been requested from a median of five peers, and what set the
- * tail was WHICH wire held the last outstanding block: a block is reserved for
- * exactly one wire, and the read finishes when the slowest holder delivers.
- *
- * What the library already does, and what it does not: `torrent.critical()`
- * (which the reader already sets over its window) enables HOTSWAP — an idle
- * wire may take a block away from the SLOWEST holder. It does not duplicate:
- * `piece.reserve()` returns -1 once every block is spoken for, and the
- * end-game that would ask a second peer for the same block is commented out in
- * `webtorrent/lib/torrent.js`. Hotswap therefore fires only when the library's
- * own picker happens to visit an idle wire while our piece is critical.
- *
- * This asks for it on purpose, and chooses who: the wires that are unchoked,
- * hold the piece and are measurably fastest are handed the piece through the
- * same entry the library's picker uses, with hotswap enabled. Nothing is
- * duplicated and no protocol rule is bent — the block moves to a faster holder
- * instead of staying with whoever got it first.
- */
-
-/**
- * The library's own request entry. Internal, so its absence must be noticed
- * rather than swallowed: without it this lever silently does nothing.
- *
- * @param {object} torrent
- * @returns {boolean}
- */
-export function canPlaceRequests(torrent) {
-  return typeof torrent?._request === "function";
-}
-
-/**
- * Wires that could deliver this piece right now, fastest first.
- *
- * Excluded, each for its own reason: a wire that is choking us cannot be asked
- * at all; one that does not hold the piece has nothing to give; a destroyed one
- * is a corpse. Speed is the library's own measurement over its own window, so
- * nothing here needs a history of its own.
- *
- * @param {object} torrent
- * @param {number} pieceIndex
- * @returns {object[]}
- */
-export function wiresForPiece(torrent, pieceIndex) {
-  const wires = Array.isArray(torrent?.wires) ? torrent.wires : [];
-  const usable = [];
-  for (const wire of wires) {
-    if (!wire || wire.destroyed || wire.peerChoking) {
-      continue;
-    }
-    if (wire.peerPieces?.get?.(pieceIndex) !== true) {
-      continue;
-    }
-    usable.push(wire);
-  }
-  return usable.sort((left, right) => speedOf(right) - speedOf(left));
-}
-
-/**
- * A wire's measured download speed in bytes per second, or 0 when it has not
- * been measured. Wrapped because `downloadSpeed` is a method on the wire and a
- * throw from a destroyed one must not take the caller with it.
- *
- * @param {object} wire
- * @returns {number}
- */
-function speedOf(wire) {
-  try {
-    const speed = wire?.downloadSpeed?.();
-    return Number.isFinite(speed) ? speed : 0;
-  } catch {
-    // A wire that cannot say how fast it is ranks last, which is the honest
-    // answer — and the caller's own log line reports how many were asked.
-    return 0;
-  }
-}
-
-/**
- * Ask the fastest holders of `pieceIndex` for it.
- *
- * @param {object} torrent
- * @param {number} pieceIndex
- * @param {number} [limit] - How many wires to push it onto.
- * @returns {{ asked: number, attempted: number, considered: number, fastestBytesPerSecond: number }}
- *   `asked` counts requests the library actually placed: it refuses when a
- *   wire's pipeline is full or when nothing can be reserved even with hotswap,
- *   and that refusal is information — a piece nobody can be asked for is
- *   waiting on the wire, not on the picker.
- */
-export function askFastestWiresFor(torrent, pieceIndex, limit = 3) {
-  if (!canPlaceRequests(torrent) || !Number.isInteger(pieceIndex) || pieceIndex < 0) {
-    return { asked: 0, attempted: 0, considered: 0, fastestBytesPerSecond: 0 };
-  }
-  const candidates = wiresForPiece(torrent, pieceIndex);
-  let asked = 0;
-  for (const wire of candidates.slice(0, Math.max(1, limit))) {
-    try {
-      // `true` is hotswap: if every block is reserved, take one from the
-      // slowest holder. That is the whole point — the reader is blocked
-      // precisely because a slow holder has one.
-      if (torrent._request(wire, pieceIndex, true) === true) {
-        asked += 1;
-      }
-    } catch (error) {
-      // Internal call: report it once per attempt rather than letting the lever
-      // fail in silence.
-      throw new Error(`could not place a request for piece ${pieceIndex}: ${error?.message ?? error}`);
-    }
-  }
-  return {
-    asked,
-    considered: candidates.length,
-    // How many of the asks the library placed, against how many it was asked
-    // for. The caller sums these over the whole wait, and summing `asked`
-    // against a `considered` taken from the LAST attempt is how the field log
-    // came to read "steered onto 12 of 6 holders" — a ratio of two different
-    // things. Both halves are returned per attempt so the caller can add each
-    // to its own total.
-    attempted: Math.min(candidates.length, Math.max(1, limit)),
-    fastestBytesPerSecond: candidates.length > 0 ? speedOf(candidates[0]) : 0
-  };
-}
-
-/**
- * What is actually holding up a piece the reader is blocked on.
- *
- * The steering above can only move a block that the library will hand over, and
- * the field says it often hands over nothing: `steered onto 0 of 9 asks (8
- * peers held it)`, recorded 2026-08-18 while eight peers had the piece. When
- * every block of a piece is already reserved, `Piece.reserve()` answers -1 and
- * there is no request left to place — the read then ends when the SLOWEST
- * holder delivers its block, however fast the rest of the swarm is.
- *
- * Duplicating those last blocks onto faster wires is the standard remedy, and
- * it is not free: every duplicate is a block's worth of traffic paid twice. So
- * this describes the tail before anything is built with it — how many blocks
- * are still missing, and on which wires they sit, with each wire's speed. If
- * the missing blocks turn out to sit on one slow wire, duplication is aimed at
- * exactly the right thing; if they are spread across fast ones, the wait has
- * another cause and this work should not be done at all.
- *
- * Reads only: nothing here changes a reservation or places a request.
- *
- * @param {import("webtorrent").Torrent} torrent
- * @param {number} pieceIndex
- * @returns {{ chunks: number, missing: number,
- *   outstanding: Array<{ blocks: number, bytesPerSecond: number, choking: boolean }> } | null}
- *   Null only when the piece object is gone — it completed and was cleared.
- *   A piece nobody has reserved a block of yet is NOT null: `torrent-piece`
- *   creates its buffer lazily on the first reserve, so a piece the picker has
- *   not reached reads as every block missing and nothing outstanding, which is
- *   the most informative answer this can give — the wait is not on a slow
- *   holder, it is on nobody having been asked.
- */
-export function describePieceTail(torrent, pieceIndex) {
-  const piece = torrent?.pieces?.[pieceIndex];
-  if (!piece) {
-    return null;
-  }
-  const buffer = Array.isArray(piece._buffer) ? piece._buffer : null;
-  const chunks = Number.isFinite(piece._chunks)
-    ? piece._chunks
-    : (buffer ? buffer.length : 0);
-  // No buffer means no block of this piece has been reserved yet, so all of it
-  // is missing. Reading that as "no tail" hid the case worth seeing most.
-  let missing = chunks;
-  if (buffer) {
-    missing = 0;
-    for (let index = 0; index < chunks; index += 1) {
-      if (!buffer[index]) {
-        missing += 1;
-      }
-    }
-  }
-
-  const wires = Array.isArray(torrent?.wires) ? torrent.wires : [];
-  const outstanding = [];
-  for (const wire of wires) {
-    const requests = Array.isArray(wire?.requests) ? wire.requests : [];
-    const blocks = requests.filter((request) => request?.piece === pieceIndex).length;
-    if (blocks > 0) {
-      outstanding.push({
-        blocks,
-        bytesPerSecond: speedOf(wire),
-        choking: wire?.peerChoking === true
-      });
-    }
-  }
-  // Slowest first: that is the wire the read is waiting on, and the one a
-  // duplicate would be aimed past.
-  outstanding.sort((left, right) => left.bytesPerSecond - right.bytesPerSecond);
-  return { chunks, missing, outstanding };
-}
+/**
+ * @file Put the piece a reader is blocked on onto the fastest wires that hold
+ * it.
+ *
+ * Measured 2026-08-17: the swarm delivered 5.1-5.9 MB/s against a film consumed
+ * at about 1 MB/s — a fivefold surplus — and the reader still blocked 47 times
+ * in two minutes, 1.0-4.5 s each. So the shortage is not bandwidth. The piece
+ * that blocked had been requested from a median of five peers, and what set the
+ * tail was WHICH wire held the last outstanding block: a block is reserved for
+ * exactly one wire, and the read finishes when the slowest holder delivers.
+ *
+ * What the library already does, and what it does not: `torrent.critical()`
+ * (which the reader already sets over its window) enables HOTSWAP — an idle
+ * wire may take a block away from the SLOWEST holder. It does not duplicate:
+ * `piece.reserve()` returns -1 once every block is spoken for, and the
+ * end-game that would ask a second peer for the same block is commented out in
+ * `webtorrent/lib/torrent.js`. Hotswap therefore fires only when the library's
+ * own picker happens to visit an idle wire while our piece is critical.
+ *
+ * This asks for it on purpose, and chooses who: the wires that are unchoked,
+ * hold the piece and are measurably fastest are handed the piece through the
+ * same entry the library's picker uses, with hotswap enabled. Nothing is
+ * duplicated and no protocol rule is bent — the block moves to a faster holder
+ * instead of staying with whoever got it first.
+ */
+
+/**
+ * The library's own request entry. Internal, so its absence must be noticed
+ * rather than swallowed: without it this lever silently does nothing.
+ *
+ * @param {object} torrent
+ * @returns {boolean}
+ */
+export function canPlaceRequests(torrent) {
+  return typeof torrent?._request === "function";
+}
+
+/**
+ * Wires that could deliver this piece right now, fastest first.
+ *
+ * Excluded, each for its own reason: a wire that is choking us cannot be asked
+ * at all; one that does not hold the piece has nothing to give; a destroyed one
+ * is a corpse. Speed is the library's own measurement over its own window, so
+ * nothing here needs a history of its own.
+ *
+ * @param {object} torrent
+ * @param {number} pieceIndex
+ * @returns {object[]}
+ */
+export function wiresForPiece(torrent, pieceIndex) {
+  const wires = Array.isArray(torrent?.wires) ? torrent.wires : [];
+  const usable = [];
+  for (const wire of wires) {
+    if (!wire || wire.destroyed || wire.peerChoking) {
+      continue;
+    }
+    if (wire.peerPieces?.get?.(pieceIndex) !== true) {
+      continue;
+    }
+    usable.push(wire);
+  }
+  return usable.sort((left, right) => speedOf(right) - speedOf(left));
+}
+
+/**
+ * A wire's measured download speed in bytes per second, or 0 when it has not
+ * been measured. Wrapped because `downloadSpeed` is a method on the wire and a
+ * throw from a destroyed one must not take the caller with it.
+ *
+ * @param {object} wire
+ * @returns {number}
+ */
+function speedOf(wire) {
+  try {
+    const speed = wire?.downloadSpeed?.();
+    return Number.isFinite(speed) ? speed : 0;
+  } catch {
+    // A wire that cannot say how fast it is ranks last, which is the honest
+    // answer — and the caller's own log line reports how many were asked.
+    return 0;
+  }
+}
+
+/**
+ * Ask the fastest holders of `pieceIndex` for it.
+ *
+ * @param {object} torrent
+ * @param {number} pieceIndex
+ * @param {number} [limit] - How many wires to push it onto.
+ * @returns {{ asked: number, attempted: number, considered: number, fastestBytesPerSecond: number }}
+ *   `asked` counts requests the library actually placed: it refuses when a
+ *   wire's pipeline is full or when nothing can be reserved even with hotswap,
+ *   and that refusal is information — a piece nobody can be asked for is
+ *   waiting on the wire, not on the picker.
+ */
+export function askFastestWiresFor(torrent, pieceIndex, limit = 3) {
+  if (!canPlaceRequests(torrent) || !Number.isInteger(pieceIndex) || pieceIndex < 0) {
+    return { asked: 0, attempted: 0, considered: 0, fastestBytesPerSecond: 0 };
+  }
+  const candidates = wiresForPiece(torrent, pieceIndex);
+  let asked = 0;
+  for (const wire of candidates.slice(0, Math.max(1, limit))) {
+    try {
+      // `true` is hotswap: if every block is reserved, take one from the
+      // slowest holder. That is the whole point — the reader is blocked
+      // precisely because a slow holder has one.
+      if (torrent._request(wire, pieceIndex, true) === true) {
+        asked += 1;
+      }
+    } catch (error) {
+      // Internal call: report it once per attempt rather than letting the lever
+      // fail in silence.
+      throw new Error(`could not place a request for piece ${pieceIndex}: ${error?.message ?? error}`);
+    }
+  }
+  return {
+    asked,
+    considered: candidates.length,
+    // How many of the asks the library placed, against how many it was asked
+    // for. The caller sums these over the whole wait, and summing `asked`
+    // against a `considered` taken from the LAST attempt is how the field log
+    // came to read "steered onto 12 of 6 holders" — a ratio of two different
+    // things. Both halves are returned per attempt so the caller can add each
+    // to its own total.
+    attempted: Math.min(candidates.length, Math.max(1, limit)),
+    fastestBytesPerSecond: candidates.length > 0 ? speedOf(candidates[0]) : 0
+  };
+}
+
+/**
+ * What is actually holding up a piece the reader is blocked on.
+ *
+ * The steering above can only move a block that the library will hand over, and
+ * the field says it often hands over nothing: `steered onto 0 of 9 asks (8
+ * peers held it)`, recorded 2026-08-18 while eight peers had the piece. When
+ * every block of a piece is already reserved, `Piece.reserve()` answers -1 and
+ * there is no request left to place — the read then ends when the SLOWEST
+ * holder delivers its block, however fast the rest of the swarm is.
+ *
+ * Duplicating those last blocks onto faster wires is the standard remedy, and
+ * it is not free: every duplicate is a block's worth of traffic paid twice. So
+ * this describes the tail before anything is built with it — how many blocks
+ * are still missing, and on which wires they sit, with each wire's speed. If
+ * the missing blocks turn out to sit on one slow wire, duplication is aimed at
+ * exactly the right thing; if they are spread across fast ones, the wait has
+ * another cause and this work should not be done at all.
+ *
+ * Reads only: nothing here changes a reservation or places a request.
+ *
+ * @param {import("webtorrent").Torrent} torrent
+ * @param {number} pieceIndex
+ * @returns {{ chunks: number, missing: number,
+ *   outstanding: Array<{ blocks: number, bytesPerSecond: number, choking: boolean }> } | null}
+ *   Null only when the piece object is gone — it completed and was cleared.
+ *   A piece nobody has reserved a block of yet is NOT null: `torrent-piece`
+ *   creates its buffer lazily on the first reserve, so a piece the picker has
+ *   not reached reads as every block missing and nothing outstanding, which is
+ *   the most informative answer this can give — the wait is not on a slow
+ *   holder, it is on nobody having been asked.
+ */
+export function describePieceTail(torrent, pieceIndex) {
+  const piece = torrent?.pieces?.[pieceIndex];
+  if (!piece) {
+    return null;
+  }
+  const buffer = Array.isArray(piece._buffer) ? piece._buffer : null;
+  const chunks = Number.isFinite(piece._chunks)
+    ? piece._chunks
+    : (buffer ? buffer.length : 0);
+  // No buffer means no block of this piece has been reserved yet, so all of it
+  // is missing. Reading that as "no tail" hid the case worth seeing most.
+  let missing = chunks;
+  if (buffer) {
+    missing = 0;
+    for (let index = 0; index < chunks; index += 1) {
+      if (!buffer[index]) {
+        missing += 1;
+      }
+    }
+  }
+
+  const wires = Array.isArray(torrent?.wires) ? torrent.wires : [];
+  const outstanding = [];
+  for (const wire of wires) {
+    const requests = Array.isArray(wire?.requests) ? wire.requests : [];
+    const blocks = requests.filter((request) => request?.piece === pieceIndex).length;
+    if (blocks > 0) {
+      outstanding.push({
+        blocks,
+        bytesPerSecond: speedOf(wire),
+        choking: wire?.peerChoking === true
+      });
+    }
+  }
+  // Slowest first: that is the wire the read is waiting on, and the one a
+  // duplicate would be aimed past.
+  outstanding.sort((left, right) => left.bytesPerSecond - right.bytesPerSecond);
+  return { chunks, missing, outstanding };
+}