@torrent-tv/proxy 2.9.82 → 2.9.84

package/CHANGELOG.md

@@ -1,3 +1,11 @@
+## 2.9.84
+
+- **Fix**: fMP4 now cuts segments where the playlist says too, closing the gap left by 2.9.82 (which covered MPEG-TS only). The muxer that takes explicit cut times writes each fMP4 piece self-contained — `ftyp moov moof mdat … mfra`, confirmed on the field host — which is not what HLS wants, so the pieces are split on serve: the header is lifted out of the first one to become the init segment named by `#EXT-X-MAP`, and removed from every media segment along with the trailing random-access index, whose offsets describe a file that no longer exists. Timestamps still need stamping exactly as before: measured, all pieces of a run report a start of 0.080 s, each carrying its own zero, which is the same defect the existing rewriting already corrects. Verified end to end on a real piece from the field host — split into a 779-byte init and 221 KB of fragments, recombined, and decoded clean.
+
+## 2.9.83
+
+- **Fix**: Playback died a few seconds in after 2.9.82. The previous muxer wrote each segment under a temporary name and renamed it once complete, so a file appearing WAS a finished segment; the one that takes explicit cut times has no such option and creates the file when writing starts. The route kept judging readiness by existence, so the player was handed a segment that was still being written, rejected it and stopped — while the encoder ran happily ahead, which is exactly how it looked in the field: three segments served, then silence with the transcode at 7.5x. A segment is now considered finished once the next one has been started, or once the run producing it has ended.
+
 ## 2.9.82
 
 - **Fix**: The playlist and the real segments now describe the same thing. On the copy path ffmpeg was given only a target duration and chose its own cut points, while the playlist was built from the container keyframe index — two independent calculations tied together by nothing but the assumption that they agree. They do not: the index is a navigation table and is not obliged to list every keyframe. On a field file it held 1902 while ffmpeg found roughly twice as many and cut twice as often, so segment #876 meant 1:26:50 to the player and about minute 58 to ffmpeg. A seek into the middle landed at the end and the reported duration drifted. ffmpeg now receives the very boundaries the playlist was built from, via the `segment` muxer, which takes the list outright — agreement by construction instead of by luck. Verified on deliberately uneven keyframes: cuts requested at 4.44, 10.36, 16.28, 22.2 and 28.12 s landed exactly there. Two measured details are encoded in the code: those times count from the start of the RUN, not of the file (starting at 12 s and asking for 18 s put the cut at 29.4), and a tolerance absorbs rounding so a boundary recorded a hair late cannot skip to the next keyframe and silently double a segment. MPEG-TS only for now — fMP4 can do this too, but only as self-contained fragments, which removes the shared init segment and the `tfdt` rewriting built around it; that is a separate change and not one to make blind.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@torrent-tv/proxy",
-  "version": "2.9.82",
+  "version": "2.9.84",
   "description": "Torrent proxy client that exposes webseed-like HTTP stream endpoint.",
   "license": "GPL-3.0-or-later",
   "publishConfig": {

package/services/hls-session-manager.js

@@ -1222,6 +1222,41 @@ export class HlsSessionManager {
    * @returns {Promise<number[] | null>} Ascending seconds, or null when this
    *   file carries no readable index.
    */
+  /**
+   * The init header, lifted out of the first segment that exists.
+   *
+   * Needed only on the explicit-cut path, where the muxer produces no init file
+   * of its own. Scans rather than assuming segment 0: a run started by a seek
+   * begins at whatever index the viewer asked for.
+   *
+   * @param {HlsSession} session
+   * @returns {Promise<Buffer | null>}
+   */
+  async #initFromFirstSegment(session) {
+    if (typeof this.segmentFormat.extractInit !== "function") {
+      return null;
+    }
+    let names;
+    try {
+      names = (await readdir(session.dirPath))
+        .filter((name) => this.segmentFormat.isSegmentFileName(name))
+        .sort();
+    } catch {
+      return null;
+    }
+    for (const name of names) {
+      try {
+        const init = this.segmentFormat.extractInit(await readFile(path.join(session.dirPath, name)));
+        if (init && init.length > 0) {
+          return init;
+        }
+      } catch {
+        // Being written right now — try the next one.
+      }
+    }
+    return null;
+  }
+
   async #readContainerKeyframes({ sourceKey, fileIndex, inputUrl, logName }) {
     const cacheKey = `${sourceKey}:${fileIndex}`;
     if (this.keyframeIndexCache.has(cacheKey)) {
@@ -1868,6 +1903,9 @@ export class HlsSessionManager {
       stdio: ["ignore", "pipe", "pipe"]
     });
     session.ffmpeg = ffmpeg;
+    // Whether this run cuts at times we gave it. Decides how a segment is
+    // judged finished — see getFileStream.
+    session.usesExplicitCuts = Boolean(cutTimes && cutTimes.length > 0);
     session.encodeStartIndex = safeIndex;
     session.pendingRestartIndex = -1;
     session.lastRestartAt = Date.now();
@@ -2469,8 +2507,13 @@ export class HlsSessionManager {
         };
       }
       try {
-        const bytes = await readFile(path.join(session.dirPath, initFileName));
-        if (bytes.length === 0) {
+        // With explicit cut times there is no init file: that muxer writes each
+        // piece self-contained, header and all. The header is identical in every
+        // piece, so the first one to exist supplies it.
+        const bytes = session.usesExplicitCuts
+          ? await this.#initFromFirstSegment(session)
+          : await readFile(path.join(session.dirPath, initFileName));
+        if (!bytes || bytes.length === 0) {
           return { kind: "warming-up" };
         }
         session.initBytes = bytes;
@@ -2491,6 +2534,27 @@ export class HlsSessionManager {
     const isPlaylist = fileName === PLAYLIST_FILE_NAME;
     try {
       await access(filePath);
+
+      // Existing is not the same as finished. The `hls` muxer wrote each
+      // segment to a temporary name and renamed it once complete, so a file
+      // appearing WAS a finished segment. The `segment` muxer has no such
+      // option: the file appears when writing begins. Serving it then hands the
+      // player a truncated segment, which it rejects and then simply stops —
+      // observed as playback dying a few seconds in with the encoder still
+      // running happily ahead. A segment is finished once the NEXT one has been
+      // started, or once the run producing it has ended.
+      if (!isPlaylist && session.usesExplicitCuts) {
+        const index = this.segmentFormat.segmentIndexFromName(fileName);
+        const isLast = index >= Math.max(0, (session.segmentBoundaries?.length ?? 1) - 2);
+        if (!isLast && session.ffmpeg) {
+          const nextPath = path.join(session.dirPath, this.segmentFormat.segmentFileName(index + 1));
+          try {
+            await access(nextPath);
+          } catch {
+            return { kind: "warming-up" };
+          }
+        }
+      }
       // Cold-start: log the first servable SEGMENT of this session exactly once
       // — the time from session-create entry to a playable first segment.
       if (!isPlaylist && !session.firstSegmentLogged) {
@@ -2504,7 +2568,11 @@ export class HlsSessionManager {
       // module; the rest stream straight off disk.
       if (!isPlaylist && this.segmentFormat.needsSegmentRewrite) {
         const index = this.segmentFormat.segmentIndexFromName(fileName);
-        const bytes = await readFile(filePath);
+        const raw = await readFile(filePath);
+        // Self-contained pieces carry the init header; a media segment must not.
+        const bytes = session.usesExplicitCuts && this.segmentFormat.stripInit
+          ? this.segmentFormat.stripInit(raw)
+          : raw;
         const prepared = this.segmentFormat.prepareSegmentBytes(bytes, {
           startSeconds: this.#segmentStartTime(session, index),
           initBytes: session.initBytes ?? null

package/services/segment-formats/fmp4.js

@@ -12,6 +12,47 @@
 
 import { readTrackTimescales, stampSegmentStartTime } from "./mp4-boxes.js";
 
+/**
+ * Where the fragments begin and where the trailing index starts, as offsets of
+ * whole boxes at the TOP level.
+ *
+ * Deliberately not `walkBoxes`: that reports the start of a box's *body* and
+ * descends into containers, so it cannot say where a box begins — and cutting a
+ * file at the wrong offset by eight bytes produces something that parses as
+ * garbage rather than failing outright.
+ *
+ * @param {Buffer} bytes
+ * @returns {{ firstFragment: number, trailingIndex: number }} Offsets, or -1.
+ */
+function findFragmentBounds(bytes) {
+  let firstFragment = -1;
+  let trailingIndex = -1;
+  let offset = 0;
+  while (offset + 8 <= bytes.length) {
+    let size = bytes.readUInt32BE(offset);
+    const type = bytes.toString("latin1", offset + 4, offset + 8);
+    if (size === 1) {
+      if (offset + 16 > bytes.length) {
+        break;
+      }
+      size = Number(bytes.readBigUInt64BE(offset + 8));
+    } else if (size === 0) {
+      size = bytes.length - offset;
+    }
+    if (size < 8) {
+      break;
+    }
+    if (firstFragment === -1 && (type === "moof" || type === "sidx")) {
+      firstFragment = offset;
+    }
+    if (type === "mfra") {
+      trailingIndex = offset;
+    }
+    offset += size;
+  }
+  return { firstFragment, trailingIndex };
+}
+
 const INIT_FILE_NAME = "init.mp4";
 const SEGMENT_PATTERN = /^segment-(\d{5})\.m4s$/;
 
@@ -38,21 +79,67 @@ export const fmp4Format = {
   },
 
   /**
-   * Not supported on this format — deliberately, for now.
+   * Cut at times we choose rather than times ffmpeg picks.
+   *
+   * The `segment` muxer writes each fMP4 piece as a **self-contained** file:
+   * `ftyp moov moof mdat … mfra`, verified on the field host. That is not what
+   * HLS wants — it wants one init segment named by `#EXT-X-MAP` and media
+   * segments carrying only fragments — so the pieces are split on serve:
+   * {@link extractInit} takes the header off the first one to serve as the init,
+   * and {@link stripInit} removes it from every one.
    *
-   * The `segment` muxer can produce fMP4 (verified: explicit times cut exactly
-   * where asked), but only as self-contained fragments carrying their own
-   * `moov`. That removes the shared init segment this format is built around —
-   * `#EXT-X-MAP`, and with it the whole `tfdt` rewriting that took a field
-   * failure to get right. Changing all of that at once, on a path no current
-   * deployment exercises and that cannot be verified without a real browser, is
-   * how the last round of regressions happened. MPEG-TS, which is what runs in
-   * the field, gets the fix first.
+   * The timestamps still need stamping, exactly as before: measured on the
+   * field host, all three pieces of a run reported a start time of 0.080 s,
+   * i.e. each carries its own zero. That is the same defect the shared-init
+   * path already corrects, so `prepareSegmentBytes` handles both.
    *
-   * @returns {null}
+   * @returns {string[]}
    */
   explicitTimesMuxerArgs() {
-    return null;
+    return [
+      "-segment_format",
+      "mp4",
+      // `empty_moov` is what makes each piece self-describing, which is what
+      // lets the init be lifted out of it; `default_base_moof` keeps fragment
+      // offsets relative, so removing the header does not invalidate them.
+      "-segment_format_options",
+      "movflags=+frag_keyframe+empty_moov+default_base_moof"
+    ];
+  },
+
+  /** The output path template for the `segment` muxer. */
+  segmentFileNameTemplate() {
+    return "segment-%05d.m4s";
+  },
+
+  /**
+   * The init part of a self-contained piece: everything before the first
+   * fragment.
+   *
+   * @param {Buffer} bytes
+   * @returns {Buffer | null} `null` when the piece carries no fragment, which
+   *   means it is not one of these and must not be cut up.
+   */
+  extractInit(bytes) {
+    const { firstFragment } = findFragmentBounds(bytes);
+    return firstFragment > 0 ? bytes.subarray(0, firstFragment) : null;
+  },
+
+  /**
+   * A piece with its init header removed, and the trailing random-access index
+   * dropped — a player reading a media segment has no use for either, and
+   * `mfra` describes offsets that stop being true once the header is gone.
+   *
+   * @param {Buffer} bytes
+   * @returns {Buffer}
+   */
+  stripInit(bytes) {
+    const { firstFragment, trailingIndex } = findFragmentBounds(bytes);
+    if (firstFragment < 0) {
+      return bytes;
+    }
+    const end = trailingIndex > firstFragment ? trailingIndex : bytes.length;
+    return bytes.subarray(firstFragment, end);
   },
 
   playlistHeaderLines() {

package/test/fmp4-self-contained.test.js

@@ -0,0 +1,80 @@
+/**
+ * Splitting a self-contained fMP4 piece into an init segment and a media
+ * segment.
+ *
+ * The muxer that takes explicit cut times writes every piece whole —
+ * `ftyp moov moof mdat … mfra` — but HLS wants one init named by `#EXT-X-MAP`
+ * and media segments carrying only fragments. Cutting at the wrong offset does
+ * not fail loudly; it produces something that parses as garbage, which is why
+ * the boundaries are pinned here rather than trusted.
+ */
+
+import test from "node:test";
+import assert from "node:assert/strict";
+import { fmp4Format } from "../services/segment-formats/fmp4.js";
+
+/**
+ * A piece shaped like the muxer's output. Sizes are what matters, not contents.
+ *
+ * @param {{ withIndex?: boolean }} [options]
+ * @returns {{ piece: Buffer, initLength: number, fragmentsLength: number }}
+ */
+function makePiece({ withIndex = true } = {}) {
+  const box = (type, bodyLength) => {
+    const buffer = Buffer.alloc(8 + bodyLength);
+    buffer.writeUInt32BE(8 + bodyLength, 0);
+    buffer.write(type, 4, "latin1");
+    buffer.fill(0x5a, 8);
+    return buffer;
+  };
+  const ftyp = box("ftyp", 24);
+  const moov = box("moov", 400);
+  const moof = box("moof", 120);
+  const mdat = box("mdat", 900);
+  const mfra = box("mfra", 40);
+  const parts = withIndex ? [ftyp, moov, moof, mdat, mfra] : [ftyp, moov, moof, mdat];
+  return {
+    piece: Buffer.concat(parts),
+    initLength: ftyp.length + moov.length,
+    fragmentsLength: moof.length + mdat.length
+  };
+}
+
+test("the init is everything before the first fragment", () => {
+  const { piece, initLength } = makePiece();
+  const init = fmp4Format.extractInit(piece);
+  assert.ok(init, "no init found in a piece that has one");
+  assert.equal(init.length, initLength, "init must end exactly where the first fragment begins");
+  assert.equal(init.toString("latin1", 4, 8), "ftyp", "init must start at the file header");
+});
+
+test("the media segment is the fragments alone, index dropped", () => {
+  const { piece, initLength, fragmentsLength } = makePiece();
+  const media = fmp4Format.stripInit(piece);
+  assert.equal(media.toString("latin1", 4, 8), "moof", "a media segment must begin at a fragment");
+  assert.equal(
+    media.length,
+    fragmentsLength,
+    "the trailing random-access index must be dropped: its offsets describe a file that no longer exists"
+  );
+  assert.equal(piece.length, initLength + fragmentsLength + 48, "test fixture sanity");
+});
+
+test("a piece without a trailing index keeps everything to the end", () => {
+  const { piece, fragmentsLength } = makePiece({ withIndex: false });
+  assert.equal(fmp4Format.stripInit(piece).length, fragmentsLength);
+});
+
+test("a segment that is already fragments-only is left alone", () => {
+  const alreadyMedia = fmp4Format.stripInit(makePiece().piece);
+  assert.deepEqual(
+    fmp4Format.stripInit(alreadyMedia),
+    alreadyMedia,
+    "stripping twice must not eat the first fragment"
+  );
+  assert.equal(
+    fmp4Format.extractInit(alreadyMedia),
+    null,
+    "there is no init to lift out of a media segment, and inventing one would corrupt playback"
+  );
+});