@torrent-tv/proxy 2.9.98 → 2.9.100

package/CHANGELOG.md

@@ -1,3 +1,14 @@
+## 2.9.100
+
+- **New**: A long wait for a piece now says who was working on it. The open question about a seek is that a single 8 MiB piece takes 3.0-4.6 s while the swarm as a whole moves 4-6 MB/s, so only about 2 MB/s reaches the piece being waited for — and whether that is because few peers hold it, few are being asked, or each is slow could not be told apart from outside. The line now carries the rate achieved on that piece and, sampled at its peak while waiting, how many connected peers had it, how many were asked, and how many blocks were in flight.
+- **New**: The keyframe index is read alongside the codec probe instead of after it. Both wait for the same tail of the file; measured 2026-08-04, a probe of 722-1206 ms was followed by an index read of 311-430 ms, all of it before the first segment could be produced. Started together the second is free. Fire and forget, sharing the cache a session would fill itself.
+- **Chore**: Every encoder run is numbered in the log. A burst of seeks starts several runs within a second and every line about them carries the session id, which is the same for all of them — so the command that failed could not be told from the ones that succeeded around it. That is the state the unexplained `Cannot write moov atom before AC3 packets` was found in.
+
+## 2.9.99
+
+- **New**: A source can be told to start before anyone asks to play it — `POST /api/sources/:sourceKey/warm`. Everything a cold torrent must do first takes seconds and none of it depends on which file is wanted: announce to the trackers, connect to peers, be unchoked by them. Given a file index it also fetches the two pieces at that file's edges, which is what the codec probe reads and what took **6.7 s of the 10.3 s** before playback in the session measured 2026-08-04. All of it used to begin only once a file had been chosen, because it was buried inside the playback plan. The route returns as soon as the work is under way and reports a refusal rather than an error — nothing is broken if a warm-up does not happen, since the ordinary path still does all of it.
+- **Chore**: Two callers asking for the same file's edges at once now share one prefetch instead of opening a second pair of readers, each claiming a window and holding pieces. That happens by design on a single-video torrent, where the warm-up and the playback plan both want them.
+
 ## 2.9.98
 
 - **Fix**: The upload is no longer raised at moments when nobody wants a byte. A torrent with no reader was counted as starving whenever its download read low — which it always does while the encoder is held back for running ahead of the viewer. Measured: four cycles of 512 KB/s and back in three minutes, each reported as `earn unchoke … down=0KB/s`. Starvation now requires somebody to be waiting.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@torrent-tv/proxy",
-  "version": "2.9.98",
+  "version": "2.9.100",
   "description": "Torrent proxy client that exposes webseed-like HTTP stream endpoint.",
   "license": "GPL-3.0-or-later",
   "publishConfig": {

package/routes/api/sources/warm/post.js

@@ -0,0 +1,83 @@
+import { logger } from "../../../../utils/logger.js";
+
+/**
+ * Start fetching a source before anyone asks to play it.
+ *
+ * POST /api/sources/:sourceKey/warm   { fileIndex?: number }
+ *
+ * Everything a torrent must do before the first byte of video can be served
+ * takes seconds and none of it depends on the viewer: announce to the
+ * trackers, connect to peers, be unchoked by them, and fetch the two pieces at
+ * the file's edges that the codec probe reads. Measured 2026-08-04 on a cold
+ * 7.4 GB torrent: 6.7 s of the 10.3 s before playback was those two pieces
+ * arriving, with the swarm ramping from nothing.
+ *
+ * That work used to begin only when a file had been chosen, because it was
+ * buried inside the playback plan. It can begin as soon as the viewer has
+ * picked a TORRENT — while they are still reading the list of episodes — and
+ * then most or all of it has happened by the time they choose.
+ *
+ * Returns as soon as the work is under way. The caller is not waiting for a
+ * result; it is only saying "you may start". Failures are logged and answered
+ * as `started: false` rather than as an error, because nothing is broken if a
+ * warm-up does not happen — the ordinary path still does all of it.
+ *
+ * @param {import("fastify").FastifyRequest} req
+ * @param {import("fastify").FastifyReply} reply
+ * @param {{
+ *   sourceRegistry: ReturnType<import("../../../../store/source-registry.js").createSourceRegistry>,
+ *   torrentPool: import("../../../../services/torrent-pool.js").TorrentPool
+ * }} deps
+ * @returns {Promise<void>}
+ */
+export async function handleApiSourceWarmPost(req, reply, { sourceRegistry, torrentPool }) {
+  const sourceKey = typeof req.params.sourceKey === "string" ? req.params.sourceKey.trim() : "";
+  if (!sourceKey) {
+    return reply.code(400).send({ error: "sourceKey is required." });
+  }
+
+  const sourceRecord = sourceRegistry.get(sourceKey);
+  if (!sourceRecord) {
+    return reply.code(404).send({ error: "Source key was not found." });
+  }
+
+  const body = req.body && typeof req.body === "object" && !Array.isArray(req.body) ? req.body : {};
+  const requestedIndex = Number(body.fileIndex);
+  const fileIndex = Number.isInteger(requestedIndex) && requestedIndex >= 0 ? requestedIndex : null;
+
+  // Adding the torrent is what announces to the trackers and starts connecting
+  // to peers, and it is also what a magnet needs in order to fetch its
+  // metadata. It is awaited because everything else needs the torrent object,
+  // and because until it resolves there is nothing to report.
+  let torrent;
+  try {
+    torrent = await torrentPool.getTorrent(sourceRecord.sourceType, sourceRecord.source);
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    logger.warn(`warm ${sourceKey.slice(0, 8)}: could not add the torrent: ${message}`);
+    return reply.send({ started: false, swarm: false, edges: false });
+  }
+
+  // The edges are only worth fetching once it is known WHICH file will be
+  // played: on a season pack, warming twenty episodes' worth would spend the
+  // pool owner's bandwidth on nineteen files nobody opened. The caller passes
+  // an index when the torrent holds a single video, and again later if it
+  // wants to.
+  let edges = false;
+  if (fileIndex !== null && torrent.files?.[fileIndex]) {
+    edges = true;
+    // Deliberately not awaited: this is the multi-second part, and the point of
+    // the whole route is that the viewer goes on choosing while it happens.
+    Promise.resolve(torrentPool.prefetchFileEdges(torrent, fileIndex)).catch((error) => {
+      const message = error instanceof Error ? error.message : String(error);
+      logger.warn(`warm ${sourceKey.slice(0, 8)}: file edges failed: ${message}`);
+    });
+  }
+
+  logger.info(
+    `warm ${sourceKey.slice(0, 8)}: swarm started for "${torrent.name}"` +
+      (edges ? `, fetching the edges of file ${fileIndex}` : ", file not chosen yet")
+  );
+
+  return reply.send({ started: true, swarm: true, edges });
+}

package/server.js

@@ -19,6 +19,7 @@ import { handleHealthzGet } from "./routes/healthz/get.js";
 import { handleApiSourcesPost } from "./routes/api/sources/post.js";
 import { handleApiSourceStatsGet } from "./routes/api/sources/stats/get.js";
 import { handleApiSourceFilesGet } from "./routes/api/sources/files/get.js";
+import { handleApiSourceWarmPost } from "./routes/api/sources/warm/post.js";
 import { handleApiPlaybackPlanPost } from "./routes/api/playback-plan/post.js";
 import { handleApiSubtitlesGet } from "./routes/api/subtitles/get.js";
 import { handleApiTranscodeSessionsPost } from "./routes/api/transcode-sessions/post.js";
@@ -162,7 +163,8 @@ export async function startProxyServer({ host, port, transcodeAudio, ffmpegBin,
     transcodeAudioEnabled: transcodeAudio,
     localBaseUrl: hlsSessionManager.localBaseUrl,
     sourceRegistry,
-    torrentPool
+    torrentPool,
+    warmKeyframeIndex: (params) => hlsSessionManager.warmKeyframeIndex(params)
   });
 
   app.get("/health", async (req, reply) => handleHealthGet(req, reply, { version }));
@@ -176,6 +178,9 @@ export async function startProxyServer({ host, port, transcodeAudio, ffmpegBin,
   app.get("/api/sources/:sourceKey/files", async (req, reply) =>
     handleApiSourceFilesGet(req, reply, { sourceRegistry, torrentPool })
   );
+  app.post("/api/sources/:sourceKey/warm", async (req, reply) =>
+    handleApiSourceWarmPost(req, reply, { sourceRegistry, torrentPool })
+  );
   app.post("/api/playback-plan", async (req, reply) =>
     handleApiPlaybackPlanPost(req, reply, { playbackPlanner })
   );

package/services/hls-session-manager.js

@@ -1379,6 +1379,29 @@ export class HlsSessionManager {
     return null;
   }
 
+  /**
+   * Read the file's keyframe index into the cache before a session needs it.
+   *
+   * The index lives at the END of a Matroska file, which is also where the
+   * codec probe reads — both wait for the same piece to arrive, and they used
+   * to do it one after the other: measured 2026-08-04, a probe of 722-1206 ms
+   * followed by an index read of 311-430 ms, all of it before the first
+   * segment. Started together, the second costs nothing.
+   *
+   * Never rejects and is never awaited by the caller: a session that finds
+   * nothing cached simply reads it itself, as before.
+   *
+   * @param {{ sourceKey: string, fileIndex: number, inputUrl: URL, logName: string }} params
+   * @returns {Promise<void>}
+   */
+  async warmKeyframeIndex({ sourceKey, fileIndex, inputUrl, logName }) {
+    try {
+      await this.#readContainerKeyframes({ sourceKey, fileIndex, inputUrl, logName });
+    } catch {
+      // Best effort by construction.
+    }
+  }
+
   async #readContainerKeyframes({ sourceKey, fileIndex, inputUrl, logName }) {
     const cacheKey = `${sourceKey}:${fileIndex}`;
     if (this.keyframeIndexCache.has(cacheKey)) {
@@ -2165,7 +2188,15 @@ export class HlsSessionManager {
     // were verified to handle a copied AC-3 track on this very host, so the
     // arguments that run actually received are the missing evidence. One line
     // per run, and a run happens at most every few seconds.
-    logger.info(`transcode ${session.id} ffmpeg ${describeFfmpegArgs(args)}`);
+    // Numbered, because a burst of seeks starts several runs in one second and
+    // every line about them carries the SESSION id, which is the same for all.
+    // Without a run number the command that failed cannot be told from the two
+    // that succeeded around it — which is exactly the state the unexplained
+    // `Cannot write moov atom before AC3 packets` was found in.
+    session.runCounter = (session.runCounter ?? 0) + 1;
+    const runLabel = `run#${session.runCounter}`;
+    session.runLabel = runLabel;
+    logger.info(`transcode ${session.id} ${runLabel} ffmpeg ${describeFfmpegArgs(args)}`);
 
     const ffmpeg = spawn(this.ffmpegBin, args, {
       cwd: session.dirPath,
@@ -2191,7 +2222,7 @@ export class HlsSessionManager {
     session.budgetSlowSince = 0;
 
     logger.info(
-      `transcode ${session.id} encode-run from segment #${safeIndex} ` +
+      `transcode ${session.id} ${session.runLabel} encode-run from segment #${safeIndex} ` +
         `(${formatSeconds(startSeconds)}) "${session.fileName}"`
     );
 
@@ -2379,7 +2410,9 @@ export class HlsSessionManager {
       session.state = "failed";
       session.progress.state = "failed";
       session.progress.updatedAt = Date.now();
-      logger.error(`transcode ${session.id} encode-run failed: ${session.lastError}`);
+      logger.error(
+        `transcode ${session.id} ${session.runLabel ?? "run#?"} encode-run failed: ${session.lastError}`
+      );
     });
   }
 

package/services/playback-planner.js

@@ -263,7 +263,12 @@ export function createPlaybackPlanner({
   transcodeAudioEnabled,
   localBaseUrl,
   sourceRegistry,
-  torrentPool
+  torrentPool,
+  // Optional. Called once the file's edges are downloaded, so the keyframe
+  // index — which reads the same tail of the file — is fetched alongside the
+  // codec probe instead of after it. Late-bound to the HLS session manager,
+  // which owns the cache both of them share.
+  warmKeyframeIndex
 }) {
   /** @type {Map<string, PlaybackPlan>} */
   const cache = new Map();
@@ -365,6 +370,17 @@ export function createPlaybackPlanner({
       // file, and an unsupported codec like xvid gets copied → black video.
       await torrentPool.prefetchFileEdges(torrent, fileIndex);
       edgesReadyMs = Date.now() - planEntryMs;
+      // The keyframe index reads the tail of the file, which the probe has just
+      // waited for as well. Started here it overlaps the probe instead of
+      // following the whole plan — worth 311-430 ms of the time before the
+      // first segment. Fire and forget: the session reads it itself if this has
+      // not finished, and both share one cache entry.
+      warmKeyframeIndex?.({
+        sourceKey,
+        fileIndex,
+        inputUrl: new URL(directUrl),
+        logName: file.name
+      });
       let probe = await probeStreamCodecs({ ffmpegBin, inputUrl: directUrl, userAgent });
       const probeDeadline = Date.now() + Math.max(0, maxWaitMs);
       let attempt = 0;

package/services/torrent-pool.js

@@ -383,6 +383,14 @@ export class TorrentPool {
    */
   #readPositionByTorrent = new Map();
 
+  /**
+   * Edge prefetches currently running, keyed by infoHash and file index, so two
+   * callers asking at the same time share one.
+   *
+   * @type {Map<string, Promise<void>>}
+   */
+  #edgePrefetches = new Map();
+
   /** Global disk cap in bytes (0 = disabled). */
   #maxDiskBytes = 0;
 
@@ -1034,6 +1042,37 @@ export class TorrentPool {
     fileIndex,
     { headBytes = 256 * 1024, tailBytes = 2 * 1024 * 1024, timeoutMs = 300_000 } = {}
   ) {
+    if (!torrent || !Array.isArray(torrent.files)) {
+      return;
+    }
+    // Two callers can ask for the same edges at once: the warm-up that starts
+    // when a torrent is picked, and the playback plan a moment later. Reading
+    // the same two pieces twice costs nothing in bandwidth — the torrent
+    // fetches each piece once — but it does open a second pair of readers, each
+    // claiming a window and holding pieces. One is enough.
+    const inFlightKey = `${torrent.infoHash}:${fileIndex}`;
+    const running = this.#edgePrefetches.get(inFlightKey);
+    if (running) {
+      return running;
+    }
+    const prefetch = this.#prefetchFileEdgesOnce(torrent, fileIndex, { headBytes, tailBytes, timeoutMs });
+    this.#edgePrefetches.set(inFlightKey, prefetch);
+    try {
+      return await prefetch;
+    } finally {
+      this.#edgePrefetches.delete(inFlightKey);
+    }
+  }
+
+  /**
+   * The body of {@link prefetchFileEdges}, without the de-duplication.
+   *
+   * @param {import("webtorrent").Torrent} torrent
+   * @param {number} fileIndex
+   * @param {{ headBytes: number, tailBytes: number, timeoutMs: number }} options
+   * @returns {Promise<void>}
+   */
+  async #prefetchFileEdgesOnce(torrent, fileIndex, { headBytes, tailBytes, timeoutMs }) {
     if (!torrent || !Array.isArray(torrent.files)) {
       return;
     }

package/services/torrent-worker/piece-reader.js

@@ -154,6 +154,41 @@ function clearCritical(torrent, { from, to }) {
   }
 }
 
+/**
+ * Who is working on the piece a reader is blocked on, right now.
+ *
+ * The open question about a seek: a single 8 MiB piece takes 3.0-4.6 s to
+ * arrive while the swarm as a whole is moving 4-6 MB/s, so roughly 2 MB/s is
+ * reaching the piece that is actually being waited for. Whether that is because
+ * few peers hold it, few are being asked, or each is slow cannot be told apart
+ * from the outside — these three counts tell them apart.
+ *
+ * `wire.requests` is what has been asked of that peer and not yet answered; a
+ * block is 16 KB, so `blocks x 16 KB` is the work in flight on this piece.
+ *
+ * @param {import("webtorrent").Torrent} torrent
+ * @param {number} pieceIndex
+ * @returns {{ peers: number, holders: number, askedOf: number, blocks: number }}
+ */
+export function pieceSupply(torrent, pieceIndex) {
+  const wires = Array.isArray(torrent?.wires) ? torrent.wires : [];
+  let holders = 0;
+  let askedOf = 0;
+  let blocks = 0;
+  for (const wire of wires) {
+    if (wire?.peerPieces?.get?.(pieceIndex)) {
+      holders += 1;
+    }
+    const requests = Array.isArray(wire?.requests) ? wire.requests : [];
+    const forThisPiece = requests.filter((request) => request?.piece === pieceIndex).length;
+    if (forThisPiece > 0) {
+      askedOf += 1;
+      blocks += forThisPiece;
+    }
+  }
+  return { peers: wires.length, holders, askedOf, blocks };
+}
+
 /**
  * Wait until a piece has been downloaded and verified.
  *
@@ -421,7 +456,20 @@ export async function* readFragments({
       }
 
       const waitStartedAt = Date.now();
-      await whenPieceReady(torrent, pieceIndex, cancellation);
+      // Sampled while waiting rather than after: once the piece lands, nothing
+      // is outstanding on it any more and every count reads zero.
+      let supply = null;
+      const supplyProbe = setInterval(() => {
+        const sample = pieceSupply(torrent, pieceIndex);
+        if (!supply || sample.blocks > supply.blocks) {
+          supply = sample;
+        }
+      }, 500);
+      try {
+        await whenPieceReady(torrent, pieceIndex, cancellation);
+      } finally {
+        clearInterval(supplyProbe);
+      }
       // What a reader spent waiting for data, attributed to the exact piece. A
       // seek's cost is dominated by the first segment after the encoder
       // restarts (measured 9.2-9.4 s), and without this there is no way to say
@@ -429,10 +477,16 @@ export async function* readFragments({
       // wait is long enough to matter, so ordinary sequential reading is silent.
       const waitedMs = Date.now() - waitStartedAt;
       if (waitedMs >= PIECE_WAIT_LOG_MS) {
+        const rateKbps = Math.round(pieceLength / 1024 / (waitedMs / 1000));
         logger.info(
           `piece-reader: waited ${waitedMs}ms for piece ${pieceIndex} ` +
             `(${pieceIndex - firstPiece + 1} of ${lastPiece - firstPiece + 1} in a read from ` +
-            `${(start / 1024 / 1024).toFixed(0)}MB of "${file.name}")`
+            `${(start / 1024 / 1024).toFixed(0)}MB of "${file.name}") ` +
+            `— ${rateKbps}KB/s on this piece; ` +
+            (supply
+              ? `${supply.holders}/${supply.peers} peers had it, ${supply.askedOf} were asked, ` +
+                `${supply.blocks} blocks (${Math.round((supply.blocks * 16384) / 1024)}KB) in flight at peak`
+              : "no sample taken")
         );
       }
 

package/test/read-window.test.js

@@ -219,7 +219,8 @@ test("criticality marks the window being waited for, not the whole range", async
   try {
     const iterator = readFragments({
       torrent, fileIndex: 0, start: 0, end: 8000 * PIECE - 1,
-      cancellation: { isCancelled: () => false }
+      cancellation: { isCancelled: () => false },
+      windowBytes: WINDOW_PIECES * PIECE
     });
     const pending = iterator.next();
     await new Promise((resolve) => setImmediate(resolve));