@torrent-tv/proxy 2.9.46 → 2.9.48

package/CHANGELOG.md

@@ -1,3 +1,12 @@
+## 2.9.48
+
+- **Fix**: The "bytes still needed to resume" figure shown while buffering could jump UP mid-poll even though nothing regressed, which read as confusing/broken. Root cause: the resume-window progress (`resumeNeededBytes`/`resumeDownloadedBytes`) was always computed against the LIVE read position, which slides forward as the file is read/transcoded further — so when the window moved past an already-downloaded piece into a fresh, never-touched one, "bytes needed" jumped up (a moving reference frame, not a real setback). `getFileStats` now accepts an optional `resumeAnchorByteStart` and always returns the byte offset the window was computed against; the browser client captures that offset on the FIRST poll of a buffering episode and sends it back on every subsequent poll of the SAME episode, so the window stays pinned to a fixed target and the figure only ever decreases as real download progress happens. Verified: with the anchor pinned, repeated polls report the same "needed" while the live read position moves with no new data, and a real download of a piece inside the frozen window correctly decreases it.
+- **Fix**: A rapid sequence of seeks could leave playback permanently stuck — field-diagnosed from a live session (5 seek-restarts in 16 seconds), showing `failed to rename file segment-NNNNN.m4s.tmp` and a zombie ffmpeg still writing a `.tmp` file ~30 seconds after being "killed" by two later restarts, even after the session had already been released. Root cause: `#startEncodeRun` sent `SIGTERM` to the previous ffmpeg process and immediately spawned the replacement into the SAME session directory without waiting for it to actually exit. `ChildProcess.killed` only means a signal was sent, not that the process died — ffmpeg's own blocking read of our torrent-backed `/stream` input can defer signal handling for a long time while starved, so on a rapid sequence of seeks multiple ffmpeg processes ended up alive concurrently, fighting over CPU and racing each other's file writes in the same directory; none of them would finish a segment in time, which is what "stuck at seek" looks like to the viewer. Fixed by awaiting the previous process's exit (escalating from `SIGTERM` to `SIGKILL` if it does not exit within a grace period, reusing the `waitForChildExit` helper `disposeSession` already used correctly) before spawning the replacement. A new per-session generation counter (`encodeRunGeneration`) lets a restart that was superseded by an even newer seek while it was waiting abort instead of also spawning a process — verified with a standalone race simulation: 5 overlapping restarts against a slow-to-die previous process spawn exactly 1 process, matching the LATEST requested target.
+
+## 2.9.47
+
+- **Fix**: Playback could get permanently stuck (hls.js endlessly re-fetching the manifest and the first segment, buffer never advancing) even though the transcode itself was encoding fine, running ahead of realtime. Root cause: ffmpeg creates the fMP4 `init.mp4` file before it finishes writing the codec-header boxes into it (unlike segments, its write is not gated behind an atomic rename), so a request could race a moment where the file exists but is still empty. That empty read was then cached forever as the session's init segment — a zero-length `Buffer` is still a truthy object, so the `if (session.initBytes)` cache guard treated it as "already resolved" and kept serving the empty file for the rest of the session, which hls.js can never initialize a SourceBuffer from. Fixed by treating a zero-byte read as not-yet-ready (keeps the caller's existing long-poll retrying) instead of caching it as final.
+
 ## 2.9.46
 
 - **New**: `getFileStats` now reports `resumeNeededBytes` / `resumeDownloadedBytes` — the bytes still to download in the 16 MB window ahead of the file's current read position (tracked per file by `prioritizeByteRange`, cleared on torrent removal), counted byte-accurately including partial pieces. Lets the browser show how much is left to download and the time to resume while buffering.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@torrent-tv/proxy",
-  "version": "2.9.46",
+  "version": "2.9.48",
   "description": "Torrent proxy client that exposes webseed-like HTTP stream endpoint.",
   "license": "GPL-3.0-or-later",
   "publishConfig": {

package/routes/api/sources/stats/get.js

@@ -40,7 +40,14 @@ export async function handleApiSourceStatsGet(req, reply, { sourceRegistry, torr
   const fileIndexRaw = typeof req.query.fileIndex === "string" ? req.query.fileIndex : "";
   const fileIndex = fileIndexRaw !== "" && /^\d+$/.test(fileIndexRaw) ? Number(fileIndexRaw) : null;
 
-  const stats = torrentPool.getFileStats(torrent, fileIndex);
+  // Optional: pin the resume window to a FIXED byte offset for the duration of
+  // one buffering episode (see getFileStats JSDoc) instead of the live, moving
+  // read position — otherwise "bytes needed" can jump up mid-poll as the window
+  // slides forward with playback/encoding progress.
+  const resumeAnchorRaw = typeof req.query.resumeAnchorByteStart === "string" ? req.query.resumeAnchorByteStart : "";
+  const resumeAnchorByteStart = resumeAnchorRaw !== "" && /^\d+$/.test(resumeAnchorRaw) ? Number(resumeAnchorRaw) : null;
+
+  const stats = torrentPool.getFileStats(torrent, fileIndex, { resumeAnchorByteStart });
 
   // Diagnostic: surface the real swarm state per poll so a cold-start download
   // stall (0 peers / header not advancing → playback-plan blocks on the codec

package/services/hls-session-manager.js

@@ -48,6 +48,15 @@ const MAX_LOOKAHEAD_SEGMENTS = 8;
 // succession (stall-recovery seeks); without a cooldown ffmpeg ping-pongs
 // between positions, restarting endlessly and producing nothing.
 const RESTART_COOLDOWN_MS = 4_000;
+// Encoder stall watchdog. A running ffmpeg emits `-progress` output on stdout
+// continuously while it encodes; when it hangs mid-file (alive, but producing
+// no output and no stderr — a deadlock, e.g. a stalled input read), that output
+// stops and `progress.updatedAt` freezes. If a segment INSIDE the look-ahead
+// window is being demanded but progress has not advanced for this long, the
+// encoder is wedged (observed: the segment 503s forever). Treat it like a seek
+// and restart ffmpeg at the demanded segment. Conservative — a slow-but-moving
+// encode keeps advancing `updatedAt`, so this only fires on a true freeze.
+const ENCODER_STALL_MS = 12_000;
 // Seek debounce. A far (out-of-window) segment request is a server-side seek.
 // Rather than restart ffmpeg on the first one, wait a short quiet period:
 // further far requests re-arm it and update the target to the latest index, so
@@ -59,6 +68,10 @@ const SEEK_SETTLE_MS = 1_200;
 // Hard cap on the total settle wait, measured from the first far request of a
 // burst, so a still-moving scrubber cannot delay a genuine seek forever.
 const SEEK_SETTLE_MAX_MS = 2_500;
+// Grace period to wait for the PREVIOUS ffmpeg process to exit (per signal
+// escalation step: SIGTERM, then SIGKILL) before spawning its replacement into
+// the same session directory. See #startEncodeRun.
+const ENCODE_RUN_TERMINATE_GRACE_MS = 2_000;
 // Idle TTL: a session is disposed this long after the last segment/playlist
 // access. Long enough that a viewer who pauses, backgrounds the tab, or briefly
 // turns the phone off can resume WITHOUT a cold ffmpeg restart (the warm session
@@ -152,6 +165,20 @@ function waitForChildExit(child, timeoutMs = 2_000) {
   });
 }
 
+/**
+ * Whether a child process has genuinely exited. `ChildProcess.killed` only
+ * means `.kill()` was called — the process can stay alive well after that
+ * (blocked in I/O, ignoring/delaying the signal). `exitCode`/`signalCode` are
+ * only set once the `exit` event has actually fired, so this is the reliable
+ * check before treating a directory/file as free for a new process to use.
+ *
+ * @param {import("node:child_process").ChildProcess} child
+ * @returns {boolean}
+ */
+function hasChildExited(child) {
+  return child.exitCode !== null || child.signalCode !== null;
+}
+
 /**
  * Convert a bind-all host address to the loopback address so that
  * the HLS input URL is always reachable from the same machine.
@@ -591,6 +618,9 @@ function normalizeLogFileName(fileName, fileIndex) {
  * @property {string}  lastError
  * @property {Set<string>} consumers  - Consumer IDs currently using this session.
  * @property {object}  progress       - Live progress metrics updated from ffmpeg stdout.
+ * @property {number}  encodeRunGeneration - Bumped on every #startEncodeRun call;
+ *   lets a call that awaited the previous ffmpeg's exit detect it was superseded
+ *   by a newer restart request and abort instead of spawning a second process.
  */
 
 /**
@@ -869,6 +899,7 @@ export class HlsSessionManager {
       startedAt: Date.now(),
       lastAccessedAt: Date.now(),
       ffmpeg: null,
+      encodeRunGeneration: 0,
       lastError: "",
       // Cold-start timing: entry timestamp + a once-guard so the first servable
       // segment logs its latency exactly once.
@@ -970,7 +1001,7 @@ export class HlsSessionManager {
         `duration=${hasDuration ? formatSeconds(durationSeconds) : "unknown"} segments=${segmentCount}`
     );
 
-    this.#startEncodeRun(session, 0);
+    await this.#startEncodeRun(session, 0);
 
     try {
       await this.waitUntilReady(session);
@@ -1220,7 +1251,7 @@ export class HlsSessionManager {
     if (now - session.budgetLastActionAt < BUDGET_ACTION_COOLDOWN_MS) {
       return false; // let the previous action settle
     }
-    this.#applyBudgetDownshift(
+    await this.#applyBudgetDownshift(
       session,
       `link=${report.linkMbps.toFixed(2)}Mbps stream=${observed.toFixed(2)}Mbps buffer=${report.bufferedAheadSec.toFixed(1)}s`,
       "link"
@@ -1292,7 +1323,7 @@ export class HlsSessionManager {
         session.budgetSlowSince = 0; // re-evaluate fresh; don't thrash on this
         continue;
       }
-      this.#applyBudgetDownshift(session, `speed=${speed.toFixed(2)}x`, bound);
+      await this.#applyBudgetDownshift(session, `speed=${speed.toFixed(2)}x`, bound);
     }
   }
 
@@ -1340,9 +1371,9 @@ export class HlsSessionManager {
    * @param {HlsSession} session
    * @param {string} reasonText - Measurement summary for the log line.
    * @param {"cpu" | "unknown" | "link"} bound
-   * @returns {void}
+   * @returns {Promise<void>}
    */
-  #applyBudgetDownshift(session, reasonText, bound) {
+  async #applyBudgetDownshift(session, reasonText, bound) {
     const nextIndex = session.budgetRungIndex + 1;
     const rung = session.budgetLadder[nextIndex];
     if (!rung) {
@@ -1371,22 +1402,66 @@ export class HlsSessionManager {
         `(rung ${nextIndex + 1}/${session.budgetLadder.length}, downshift ${session.budgetDownshifts}/${BUDGET_MAX_DOWNSHIFTS}), ` +
         `restart at segment #${currentSeg} "${session.fileName}"`
     );
-    this.#startEncodeRun(session, currentSeg);
+    await this.#startEncodeRun(session, currentSeg);
   }
 
   /**
    * (Re)start the ffmpeg encode run beginning at segment `startIndex`.
    *
-   * Any ffmpeg process currently running for this session is terminated first.
+   * Any ffmpeg process currently running for this session is terminated FIRST
+   * AND ITS EXIT IS AWAITED before the replacement is spawned into the same
+   * directory. This closes a real incident: a fire-and-forget SIGTERM does not
+   * mean the process is dead — `ChildProcess.killed` reflects only that a
+   * signal was sent, not that the process exited (ffmpeg's own blocking read of
+   * our torrent-backed `/stream` input can defer signal handling for a long
+   * time while starved). On a rapid sequence of seeks this left multiple
+   * ffmpeg processes alive concurrently, all writing into the SAME session
+   * directory — observed as `failed to rename file segment-NNNNN.m4s.tmp`
+   * (a dying process racing a fresh one) and a zombie process still writing a
+   * `.tmp` file ~30s after being "killed" by two LATER restarts, even after the
+   * session had already been released. Multiple ffmpeg processes fighting over
+   * CPU and the same files on a weak host is what a seek could get "stuck" on.
+   *
+   * Because this now awaits, a NEWER restart request can arrive while an OLDER
+   * one is still waiting for the previous process to die. `encodeRunGeneration`
+   * resolves that: each call captures its own generation number, and after the
+   * await, a call whose generation was superseded aborts without spawning —
+   * only the LATEST requested target ever actually starts a process.
+   *
    * Segment files are named with a global index (`-start_number`) so they
    * always line up with the synthetic VOD playlist regardless of where
    * encoding started — this is what makes server-side seeking work.
    *
    * @param {HlsSession} session
    * @param {number} startIndex
-   * @returns {void}
+   * @returns {Promise<void>}
    */
-  #startEncodeRun(session, startIndex) {
+  async #startEncodeRun(session, startIndex) {
+    const generation = ++session.encodeRunGeneration;
+    const previousFfmpeg = session.ffmpeg;
+    if (previousFfmpeg && !hasChildExited(previousFfmpeg)) {
+      try {
+        previousFfmpeg.kill("SIGTERM");
+      } catch {
+        // Best effort.
+      }
+      await waitForChildExit(previousFfmpeg, ENCODE_RUN_TERMINATE_GRACE_MS);
+      if (!hasChildExited(previousFfmpeg)) {
+        try {
+          previousFfmpeg.kill("SIGKILL");
+        } catch {
+          // Best effort.
+        }
+        await waitForChildExit(previousFfmpeg, ENCODE_RUN_TERMINATE_GRACE_MS);
+      }
+    }
+    // A newer restart (or disposal) won the race while we were waiting for the
+    // old process to die — it either already spawned its own replacement or
+    // there is nothing left to start. Do not also spawn from this stale call.
+    if (session.encodeRunGeneration !== generation || session.state === "disposed") {
+      return;
+    }
+
     const safeIndex = Number.isInteger(startIndex) && startIndex > 0 ? startIndex : 0;
     // 0-based output time of this segment, from the boundary table (uniform for
     // re-encode, real keyframe for copy).
@@ -1631,7 +1706,7 @@ export class HlsSessionManager {
           `transcode ${session.id} hardware encoder ${failedEncoder} failed ` +
             `(${session.lastError}); falling back to software libx264 and restarting`
         );
-        this.#startEncodeRun(session, session.encodeStartIndex);
+        void this.#startEncodeRun(session, session.encodeStartIndex);
         return;
       }
       session.state = "failed";
@@ -1713,7 +1788,7 @@ export class HlsSessionManager {
     session.seekTarget = null;
     session.seekFirstFarAt = 0;
     logger.info(`transcode ${session.id} seek settle → restart at segment #${target}`);
-    this.#startEncodeRun(session, target);
+    void this.#startEncodeRun(session, target);
   }
 
   /**
@@ -1804,12 +1879,28 @@ export class HlsSessionManager {
     // and position-independent, but each seek-restart run REWRITES init.mp4, so
     // cache the FIRST one and always serve that — otherwise the init the player
     // fetched could differ from a later run's, breaking playback after a seek.
+    //
+    // ffmpeg creates init.mp4 before it has finished writing the fMP4 header
+    // boxes into it (unlike segments, its write is not gated behind an atomic
+    // rename), so a read can race a moment where the file EXISTS but is still
+    // EMPTY. Root cause of a real incident: that empty read used to be cached
+    // as `session.initBytes` — a zero-length Buffer is still a truthy object,
+    // so `if (session.initBytes)` treated it as "already resolved" and served
+    // the empty file for the rest of the session's life, permanently breaking
+    // playback (hls.js can never initialize its SourceBuffer from an empty
+    // init segment) while the transcode itself kept encoding normally. Guard
+    // on non-empty content on both the cache check and the fresh read, so an
+    // empty read is treated as not-yet-ready and the caller's long-poll keeps
+    // retrying until ffmpeg has actually written the header.
     if (fileName === SEGMENT_INIT_FILE_NAME) {
-      if (session.initBytes) {
+      if (session.initBytes && session.initBytes.length > 0) {
         return { kind: "file", stream: Readable.from([session.initBytes]), contentType: "video/mp4", isPlaylist: false };
       }
       try {
         const bytes = await readFile(path.join(session.dirPath, SEGMENT_INIT_FILE_NAME));
+        if (bytes.length === 0) {
+          return { kind: "warming-up" };
+        }
         session.initBytes = bytes;
         return { kind: "file", stream: Readable.from([bytes]), contentType: "video/mp4", isPlaylist: false };
       } catch {

package/services/torrent-pool.js

@@ -727,6 +727,16 @@ export class TorrentPool {
    *
    * @param {import("webtorrent").Torrent} torrent
    * @param {number | null} [fileIndex] - Zero-based file index, or null for torrent-level only.
+   * @param {{ resumeAnchorByteStart?: number | null }} [options] - `resumeAnchorByteStart`
+   *   pins the resume window to a FIXED byte offset instead of the live (moving)
+   *   read position. Without it, the window is anchored to wherever the file is
+   *   CURRENTLY being read from — which slides forward as playback/encoding
+   *   advances, so "bytes still needed" can jump up mid-poll even though nothing
+   *   regressed (the window just moved past already-downloaded pieces into
+   *   fresh ones). The caller should capture the returned `resumeAnchorByteStart`
+   *   on the FIRST poll of a buffering episode and pass it back on subsequent
+   *   polls of that SAME episode, so "bytes needed" counts down monotonically
+   *   against a fixed target instead of chasing a moving one.
    * @returns {{
    *   numPeers: number,
    *   downloadSpeed: number,
@@ -736,7 +746,7 @@ export class TorrentPool {
    *   fileLength: number | null
    * }}
    */
-  getFileStats(torrent, fileIndex = null) {
+  getFileStats(torrent, fileIndex = null, options = {}) {
     const numPeers = typeof torrent?.numPeers === "number" ? torrent.numPeers : 0;
     const downloadSpeed = typeof torrent?.downloadSpeed === "number" ? torrent.downloadSpeed : 0;
     const uploadSpeed = typeof torrent?.uploadSpeed === "number" ? torrent.uploadSpeed : 0;
@@ -754,10 +764,14 @@ export class TorrentPool {
 
     const header = this.#getHeaderRangeProgress(torrent, file);
 
-    // Bytes still to download in the window ahead of where this file is being
-    // read — "how much left to resume". Null until a read position is known.
+    // Bytes still to download in the window ahead of the anchor point — "how
+    // much left to resume". Null until a read position is known. The anchor is
+    // the caller-supplied FROZEN offset when given (see JSDoc above), otherwise
+    // the live (moving) read position tracked from /stream range requests.
     const readPositions = this.#readPositionByTorrent.get(torrent);
-    const readByteStart = readPositions ? readPositions.get(fileIndex) : undefined;
+    const liveReadByteStart = readPositions ? readPositions.get(fileIndex) : undefined;
+    const requestedAnchor = options?.resumeAnchorByteStart;
+    const readByteStart = Number.isFinite(requestedAnchor) ? requestedAnchor : liveReadByteStart;
     const resume = typeof readByteStart === "number"
       ? this.#getResumeWindowProgress(torrent, file, readByteStart)
       : null;
@@ -772,9 +786,11 @@ export class TorrentPool {
       fileProgress: fileLength > 0 ? Math.max(0, Math.min(1, fileDownloaded / fileLength)) : 0,
       fileDownloaded,
       fileLength,
-      // Resume window (ahead of the read head): bytes needed vs downloaded.
+      // Resume window (ahead of the anchor): bytes needed vs downloaded, plus
+      // the anchor itself so the caller can pin it for the rest of one episode.
       resumeNeededBytes: resume ? resume.totalBytes : null,
       resumeDownloadedBytes: resume ? resume.downloadedBytes : null,
+      resumeAnchorByteStart: typeof readByteStart === "number" ? readByteStart : null,
       // Phase-1 progress: how much of the header/index region (the bytes the
       // codec probe needs before transcoding can start) is downloaded. Counted
       // by whole pieces from the torrent bitfield, so it advances coarsely