@torrent-tv/proxy 2.37.1 → 2.38.1

package/CHANGELOG.md

@@ -1,3 +1,11 @@
+## 2.38.1
+
+- **New**: When a blocked piece cannot be steered anywhere, the wait line says what is holding it. The steering added in 2.29.0 often places nothing — `steered onto 0 of 9 asks (8 peers held it)`, measured 2026-08-18 while eight peers had the piece — because every block is already reserved and WebTorrent will not hand out a second request for the same block (`Piece.reserve()` answers -1; the only mention of an endgame in the library is a commented-out line). Duplicating those blocks is the standard remedy and costs a block's traffic each time, so this measures the tail before anything is built on it: `tail 2/512 blocks missing, held by 1@12KB/s 1@900KB/s`, slowest wire first, and `held by nobody` when the piece has not been asked for at all. Sampled at the instant an attempt placed nothing rather than once at the start, so the numbers and the reason they are printed describe the same moment. If the missing blocks turn out to sit on one slow wire, duplication is aimed at the right thing; if they are spread across fast ones, the wait has another cause and that work should not be done.
+
+## 2.38.0
+
+- **Fix**: An MP4's keyframe times are read as composition times, on the track the handler names. Two faults, both measured on real releases over the swarm (`research/mp4-composition-times-2026-08-19.md`). (1) The reader took sample times from `stts`, which is DECODE order, and used neither `ctts` nor `elst`: ISO/IEC 14496-12 says `CT(n) = DT(n) + CTTS(n)` (§8.6.1.3) and the edit list then shifts that (§8.6.6.3). Every LostFilm MP4 measured carries a composition offset AND an edit list cancelling it exactly, which is why decode times had been right on them; `Firefly.S01E03.720p.mp4` carries the same 2002-tick offset with NO edit list, and its times were **62.1 ms early on all 34 keyframes** compared against ffmpeg's own `pts_time` — a constant that closes to four decimals as offset (0.08342 s) minus the container start (0.02133 s). After the fix that file matches ffmpeg to the container start, which `computeSegmentBoundaries` already subtracts, and `Superman.720p` — where the terms cancel — is unchanged and exact to 0.0000 s. Version 1 offsets are read as SIGNED, which is what that version exists for; an empty edit (`media_time = -1`) is skipped rather than treated as a shift. (2) The video track was "the first one carrying sync samples", and the handler was never read. That worked only because all seven measured files put video first; the standard identifies a track by `hdlr`, and a file whose audio track carries sync samples, or one leading with a cover-art video track, would have been read from the wrong place — the same defect fixed in the Matroska reader the day before, arrived at from the other side.
+
 ## 2.37.1
 
 - **Fix**: The cost of a seek no longer counts against the quality offer. `requiredSpeed` — the speed a step must sustain to survive a swarm — is built from the reader's interruptions, and the wait on the first piece after a JUMP is not one of them: those pieces have not been asked for yet and the encoder is restarting, so it measures the move, not the supply. Measured 2026-08-18: `proxy now offers 720p` landed 131 ms after a seek, collapsing a five-rung menu to one while the player was already hunting for a fragment, and another session churned `640p` → `640p 540p` → `640p 240p`. The wait is still reported, saying plainly that it belongs to the jump and is not counted, so a gap in the history cannot be mistaken for a swarm that never made the reader wait.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@torrent-tv/proxy",
-  "version": "2.37.1",
+  "version": "2.38.1",
   "description": "Torrent proxy client that exposes webseed-like HTTP stream endpoint.",
   "license": "GPL-3.0-or-later",
   "publishConfig": {

package/services/container-index/mp4.js

@@ -149,7 +149,7 @@ function findAllBoxes(buffer, start, end, type) {
  * @param {Set<number>} wanted - Sample numbers (1-based).
  * @returns {number[]} Seconds, ascending.
  */
-function resolveSampleTimes(buffer, stts, timescale, wanted) {
+function resolveSampleTimes(buffer, stts, timescale, wanted, offsets = null) {
   const entryCount = buffer.readUInt32BE(stts.dataOffset + 4);
   const times = [];
   let sampleNumber = 1;
@@ -160,7 +160,10 @@ function resolveSampleTimes(buffer, stts, timescale, wanted) {
     const delta = buffer.readUInt32BE(cursor + 4);
     for (let index = 0; index < count; index += 1) {
       if (wanted.has(sampleNumber)) {
-        times.push(ticks / timescale);
+        // `CT(n) = DT(n) + CTTS(n)` — ISO/IEC 14496-12 §8.6.1.3. The offset is
+        // what turns decode order into the order frames are shown in, and it is
+        // the timeline ffmpeg cuts on.
+        times.push((ticks + (offsets?.get(sampleNumber) ?? 0)) / timescale);
       }
       ticks += delta;
       sampleNumber += 1;
@@ -170,6 +173,100 @@ function resolveSampleTimes(buffer, stts, timescale, wanted) {
   return times;
 }
 
+/**
+ * Composition offsets for the sample numbers asked for.
+ *
+ * `ctts` is run-length encoded like `stts`, and version 1 carries SIGNED
+ * offsets — which is what the version exists for: a frame may be shown before
+ * it is decoded. Reading them as unsigned turns a small negative offset into
+ * roughly four billion ticks.
+ *
+ * @param {Buffer} buffer
+ * @param {{ dataOffset: number, end: number }} ctts
+ * @param {Set<number>} wanted - Sample numbers (1-based).
+ * @returns {Map<number, number>} Sample number to offset in media ticks.
+ */
+function readCompositionOffsets(buffer, ctts, wanted) {
+  const version = buffer[ctts.dataOffset];
+  const entryCount = buffer.readUInt32BE(ctts.dataOffset + 4);
+  const offsets = new Map();
+  let sampleNumber = 1;
+  let cursor = ctts.dataOffset + 8;
+  for (let entry = 0; entry < entryCount && cursor + 8 <= ctts.end; entry += 1) {
+    const count = buffer.readUInt32BE(cursor);
+    const offset = version === 1 ? buffer.readInt32BE(cursor + 4) : buffer.readUInt32BE(cursor + 4);
+    for (let index = 0; index < count; index += 1) {
+      if (wanted.has(sampleNumber)) {
+        offsets.set(sampleNumber, offset);
+      }
+      sampleNumber += 1;
+    }
+    cursor += 8;
+  }
+  return offsets;
+}
+
+/**
+ * How far the edit list shifts this track's composition timeline, in media
+ * ticks.
+ *
+ * ISO/IEC 14496-12 §8.6.6.3: `media_time` is the start of the edit within the
+ * media, in the MEDIA timescale and in composition time, while
+ * `segment_duration` is in the MOVIE timescale — two different units in one
+ * structure, which is why only the first is read here. `media_time = -1` is an
+ * empty edit: it inserts blank presentation time and starts no media, so the
+ * first real edit is the one that matters.
+ *
+ * Measured 2026-08-19: every LostFilm MP4 that carries a composition offset
+ * also carries an edit list cancelling it exactly, which is why decode times
+ * have been right on those files. `Firefly.S01E03` has the offset and NO edit
+ * list, and its times were 62.1 ms early on all 34 keyframes checked.
+ *
+ * @param {Buffer} buffer
+ * @param {{ dataOffset: number, end: number }} elst
+ * @returns {number} Ticks to subtract; zero when nothing is shifted.
+ */
+function readEditShift(buffer, elst) {
+  const version = buffer[elst.dataOffset];
+  const entryCount = buffer.readUInt32BE(elst.dataOffset + 4);
+  const wide = version === 1;
+  const entryBytes = wide ? 20 : 12;
+  let cursor = elst.dataOffset + 8;
+  for (let entry = 0; entry < entryCount && cursor + entryBytes <= elst.end; entry += 1) {
+    const mediaTime = wide
+      ? Number(buffer.readBigInt64BE(cursor + 8))
+      : buffer.readInt32BE(cursor + 4);
+    if (mediaTime >= 0) {
+      return mediaTime;
+    }
+    cursor += entryBytes;
+  }
+  return 0;
+}
+
+/**
+ * Whether this track's handler says it carries video.
+ *
+ * The standard identifies a track by its `hdlr`, and nothing else does. Picking
+ * "the first track that happens to carry sync samples" worked only because the
+ * seven releases measured all put video first; a file whose audio track carries
+ * them, or one that leads with a cover-art video track, would be read from the
+ * wrong place. That is the same defect that was fixed in the Matroska reader on
+ * 2026-08-18, arrived at from the other side.
+ *
+ * @param {Buffer} buffer
+ * @param {{ dataOffset: number, end: number }} mdia
+ * @returns {boolean}
+ */
+function isVideoTrack(buffer, mdia) {
+  const hdlr = findBox(buffer, mdia.dataOffset, mdia.end, "hdlr");
+  if (!hdlr || hdlr.dataOffset + 12 > hdlr.end) {
+    return false;
+  }
+  // FullBox header (4) then a reserved pre_defined (4), then the handler type.
+  return buffer.toString("latin1", hdlr.dataOffset + 8, hdlr.dataOffset + 12) === "vide";
+}
+
 /**
  * Read the keyframe times of an MP4/MOV file.
  *
@@ -189,12 +286,12 @@ export async function readMp4KeyframeTimes(readRange, fileSize) {
     return null;
   }
 
-  // Examine every track; the video one is whichever carries sync samples. A
+  // Examine every track, and take the one whose HANDLER says it is video. A
   // track with no `stss` has every sample a keyframe, so it constrains nothing
-  // and is skipped.
+  // and is skipped even when it is the video one.
   for (const trak of findAllBoxes(moov, moovBox.headerBytes, moov.length, "trak")) {
     const mdia = findBox(moov, trak.dataOffset, trak.end, "mdia");
-    if (!mdia) {
+    if (!mdia || !isVideoTrack(moov, mdia)) {
       continue;
     }
     const mdhd = findBox(moov, mdia.dataOffset, mdia.end, "mdhd");
@@ -237,9 +334,24 @@ export async function readMp4KeyframeTimes(readRange, fileSize) {
       continue;
     }
 
-    const times = resolveSampleTimes(moov, stts, timescale, wanted);
+    // The two terms that turn decode times into the timeline ffmpeg cuts on.
+    // Both are optional: a file without them is one whose decode and
+    // composition orders already agree, and then nothing is added or taken.
+    const ctts = findBox(moov, stbl.dataOffset, stbl.end, "ctts");
+    const offsets = ctts ? readCompositionOffsets(moov, ctts, wanted) : null;
+    const edts = findBox(moov, trak.dataOffset, trak.end, "edts");
+    const elst = edts && findBox(moov, edts.dataOffset, edts.end, "elst");
+    const editShift = elst ? readEditShift(moov, elst) : 0;
+
+    const times = resolveSampleTimes(moov, stts, timescale, wanted, offsets);
     if (times.length > 0) {
-      return times;
+      // A shift applied after the division would be in the wrong units: the
+      // edit's `media_time` is in MEDIA ticks, like everything else here.
+      const shifted = editShift === 0 ? times : times.map((time) => time - editShift / timescale);
+      // A negative time is not a position in the file. It happens when an edit
+      // starts later than a keyframe the table lists, and those frames are not
+      // presented at all.
+      return shifted.filter((time) => time >= 0);
     }
   }
   return null;

package/services/torrent-worker/fastest-wires.js

@@ -126,3 +126,74 @@ export function askFastestWiresFor(torrent, pieceIndex, limit = 3) {
     fastestBytesPerSecond: candidates.length > 0 ? speedOf(candidates[0]) : 0
   };
 }
+
+/**
+ * What is actually holding up a piece the reader is blocked on.
+ *
+ * The steering above can only move a block that the library will hand over, and
+ * the field says it often hands over nothing: `steered onto 0 of 9 asks (8
+ * peers held it)`, recorded 2026-08-18 while eight peers had the piece. When
+ * every block of a piece is already reserved, `Piece.reserve()` answers -1 and
+ * there is no request left to place — the read then ends when the SLOWEST
+ * holder delivers its block, however fast the rest of the swarm is.
+ *
+ * Duplicating those last blocks onto faster wires is the standard remedy, and
+ * it is not free: every duplicate is a block's worth of traffic paid twice. So
+ * this describes the tail before anything is built with it — how many blocks
+ * are still missing, and on which wires they sit, with each wire's speed. If
+ * the missing blocks turn out to sit on one slow wire, duplication is aimed at
+ * exactly the right thing; if they are spread across fast ones, the wait has
+ * another cause and this work should not be done at all.
+ *
+ * Reads only: nothing here changes a reservation or places a request.
+ *
+ * @param {import("webtorrent").Torrent} torrent
+ * @param {number} pieceIndex
+ * @returns {{ chunks: number, missing: number,
+ *   outstanding: Array<{ blocks: number, bytesPerSecond: number, choking: boolean }> } | null}
+ *   Null only when the piece object is gone — it completed and was cleared.
+ *   A piece nobody has reserved a block of yet is NOT null: `torrent-piece`
+ *   creates its buffer lazily on the first reserve, so a piece the picker has
+ *   not reached reads as every block missing and nothing outstanding, which is
+ *   the most informative answer this can give — the wait is not on a slow
+ *   holder, it is on nobody having been asked.
+ */
+export function describePieceTail(torrent, pieceIndex) {
+  const piece = torrent?.pieces?.[pieceIndex];
+  if (!piece) {
+    return null;
+  }
+  const buffer = Array.isArray(piece._buffer) ? piece._buffer : null;
+  const chunks = Number.isFinite(piece._chunks)
+    ? piece._chunks
+    : (buffer ? buffer.length : 0);
+  // No buffer means no block of this piece has been reserved yet, so all of it
+  // is missing. Reading that as "no tail" hid the case worth seeing most.
+  let missing = chunks;
+  if (buffer) {
+    missing = 0;
+    for (let index = 0; index < chunks; index += 1) {
+      if (!buffer[index]) {
+        missing += 1;
+      }
+    }
+  }
+
+  const wires = Array.isArray(torrent?.wires) ? torrent.wires : [];
+  const outstanding = [];
+  for (const wire of wires) {
+    const requests = Array.isArray(wire?.requests) ? wire.requests : [];
+    const blocks = requests.filter((request) => request?.piece === pieceIndex).length;
+    if (blocks > 0) {
+      outstanding.push({
+        blocks,
+        bytesPerSecond: speedOf(wire),
+        choking: wire?.peerChoking === true
+      });
+    }
+  }
+  // Slowest first: that is the wire the read is waiting on, and the one a
+  // duplicate would be aimed past.
+  outstanding.sort((left, right) => left.bytesPerSecond - right.bytesPerSecond);
+  return { chunks, missing, outstanding };
+}

package/services/torrent-worker/piece-reader.js

@@ -22,7 +22,7 @@
 
 import { findSharedStore } from "../piece-store/shared-piece-store.js";
 import { logger } from "../../utils/logger.js";
-import { askFastestWiresFor, canPlaceRequests } from "./fastest-wires.js";
+import { askFastestWiresFor, canPlaceRequests, describePieceTail } from "./fastest-wires.js";
 import { minimumBufferFrom, requiredSpeedFrom } from "../supply-margin.js";
 
 /** Only waits at least this long are reported; sequential reading stays silent. */
@@ -566,9 +566,18 @@ export async function* readFragments({
       // of bandwidth and the reader still waited 1.0-4.5 s, 47 times in two
       // minutes, on pieces five peers already had.
       let pushed = { asked: 0, attempted: 0, considered: 0, fastestBytesPerSecond: 0 };
+      // The tail as it stood at an attempt that placed NOTHING — the state the
+      // duplication work has to answer, and the only one worth a line. Sampled
+      // at that instant rather than once up front, because the steering runs
+      // again every half second and the piece changes under it; the last such
+      // reading is kept, so the line describes the most recent failure.
+      let tailWhenNothingPlaced = null;
       const pushToFastest = () => {
         try {
           const result = askFastestWiresFor(torrent, pieceIndex);
+          if (result.asked === 0) {
+            tailWhenNothingPlaced = describePieceTail(torrent, pieceIndex);
+          }
           pushed = {
             asked: pushed.asked + result.asked,
             // Summed like the successes, so the line compares two totals over
@@ -586,6 +595,9 @@ export async function* readFragments({
       if (canPlaceRequests(torrent)) {
         pushToFastest();
       } else {
+        // Nothing can be placed at all on this build, so the tail is the whole
+        // of the answer.
+        tailWhenNothingPlaced = describePieceTail(torrent, pieceIndex);
         logger.warn(
           "piece-reader: this webtorrent build offers no way to place a request; " +
             "the blocked piece cannot be steered onto a faster peer"
@@ -657,6 +669,19 @@ export async function* readFragments({
             `; steered onto ${pushed.asked} of ${pushed.attempted} asks (${pushed.considered} peers held it)` +
             (pushed.fastestBytesPerSecond > 0
               ? `, fastest ${Math.round(pushed.fastestBytesPerSecond / 1024)}KB/s`
+              : "") +
+            // Only when the steering placed nothing, which is the case that
+            // decides whether duplicating the tail is worth building: it says
+            // how much of the piece is still missing and which wires are
+            // holding it, slowest first.
+            (tailWhenNothingPlaced
+              ? `; tail ${tailWhenNothingPlaced.missing}/${tailWhenNothingPlaced.chunks} blocks missing, held by ` +
+                (tailWhenNothingPlaced.outstanding.length > 0
+                  ? tailWhenNothingPlaced.outstanding
+                    .map((wire) => `${wire.blocks}@${Math.round(wire.bytesPerSecond / 1024)}KB/s` +
+                      (wire.choking ? " (choking)" : ""))
+                    .join(" ")
+                  : "nobody")
               : "")
         );
       }

package/test/piece-tail.test.js

@@ -0,0 +1,97 @@
+/**
+ * @file What a blocked piece's tail looks like, before anything is built on it.
+ *
+ * The steering shipped in 2.29.0 often places nothing — `steered onto 0 of 9
+ * asks (8 peers held it)`, measured 2026-08-18 — because every block of the
+ * piece is already reserved and the library will not hand out a second request
+ * for the same block. Duplicating those blocks is the standard remedy and costs
+ * traffic, so the tail is described first: how much is missing, and on which
+ * wires it sits.
+ */
+
+import test from "node:test";
+import assert from "node:assert/strict";
+import { describePieceTail } from "../services/torrent-worker/fastest-wires.js";
+
+/** A torrent whose piece has some blocks in hand and some in flight. */
+function torrentWith({ buffer, wires }) {
+  return {
+    pieces: [{ _buffer: buffer, _chunks: buffer.length }],
+    wires
+  };
+}
+
+function wire({ has = true, blocks = 0, speed = 0, choking = false }) {
+  return {
+    peerPieces: { get: () => has },
+    requests: Array.from({ length: blocks }, () => ({ piece: 0 })),
+    downloadSpeed: () => speed,
+    peerChoking: choking
+  };
+}
+
+test("the tail names how much is missing and who is holding it", () => {
+  const tail = describePieceTail(
+    torrentWith({
+      // Four blocks, two already in hand.
+      buffer: [new Uint8Array(1), new Uint8Array(1), null, null],
+      wires: [
+        wire({ blocks: 1, speed: 900_000 }),
+        wire({ blocks: 1, speed: 12_000 }),
+        wire({ has: true, blocks: 0, speed: 5_000_000 })
+      ]
+    }),
+    0
+  );
+
+  assert.equal(tail.missing, 2);
+  assert.equal(tail.chunks, 4);
+  assert.equal(tail.outstanding.length, 2, "only the wires actually holding a block of it");
+  assert.equal(
+    tail.outstanding[0].bytesPerSecond,
+    12_000,
+    "slowest first — that is the wire the read is waiting on"
+  );
+});
+
+test("a piece nobody is fetching reads as entirely missing, held by nobody", () => {
+  const tail = describePieceTail(
+    torrentWith({ buffer: [null, null], wires: [wire({ has: false, blocks: 0 })] }),
+    0
+  );
+
+  assert.equal(tail.outstanding.length, 0);
+  assert.equal(tail.missing, 2, "and the missing count still says the piece is untouched");
+});
+
+test("a piece no block of which has been reserved is described, not skipped", () => {
+  // `torrent-piece` builds its buffer lazily on the first reserve, so a piece
+  // the picker has not reached has none. Reading that as "no tail" hid the
+  // clearest answer there is: the wait is on nobody having been asked.
+  const tail = describePieceTail(
+    { pieces: [{ _buffer: null, _chunks: 512 }], wires: [wire({ has: true, blocks: 0, speed: 900_000 })] },
+    0
+  );
+
+  assert.equal(tail.missing, 512, "every block of it is missing");
+  assert.equal(tail.chunks, 512);
+  assert.equal(tail.outstanding.length, 0, "and not one of them has been asked for");
+});
+
+test("a completed piece is not described at all", () => {
+  // WebTorrent nulls `pieces[index]` once the piece is verified and stored.
+  assert.equal(describePieceTail({ pieces: [null], wires: [] }, 0), null);
+  assert.equal(describePieceTail({ pieces: [], wires: [] }, 0), null);
+});
+
+test("a wire that is choking us is named as such", () => {
+  const tail = describePieceTail(
+    torrentWith({
+      buffer: [null],
+      wires: [wire({ blocks: 1, speed: 100, choking: true })]
+    }),
+    0
+  );
+
+  assert.equal(tail.outstanding[0].choking, true, "a block reserved by a choking wire is going nowhere");
+});