@torrent-tv/proxy 2.9.0 → 2.9.3

package/CHANGELOG.md

@@ -1,3 +1,8 @@
+## 2.9.3
+
+- **New**: WebRTC data-channel response bodies are now sent as **binary** frames (`sendMessageBinary`) instead of base64-encoded JSON `response-chunk` messages, removing the ~33% base64 overhead and the JSON encode cost. Frame layout: `[flags(1)][idLen(1)][requestId(ASCII)][payload]`. Control messages (`response-start`, `response-error`, `pong`) remain JSON strings. Requires the matching browser client (server ≥ 0.8.0); **deploy the server before the proxy**.
+- **New**: Backpressure on the send loop — `data-channel-handler.js` pauses queuing body chunks once the channel's `bufferedAmount()` exceeds 8 MB and resumes when it drains below 1 MB (`setBufferedAmountLowThreshold` + `onBufferedAmountLow`), with a 5 s timeout fallback. Prevents the SCTP send buffer from ballooning and stalling throughput.
+
 ## 2.6.3
 
 - **Fix**: Data channel handler now logs **all** requests regardless of body presence — `GET /transcode/…`, `GET /api/…/progress`, `GET /api/…/stats` etc. were previously invisible in logs. Non-2xx response statuses and fetch errors are also logged, enabling diagnosis of HLS manifest load failures.

package/CLAUDE.md

@@ -0,0 +1,62 @@
+# proxy — @torrent-tv/proxy (WebTorrent + ffmpeg)
+
+Downloads a torrent and streams the chosen file to the browser, transcoding to
+HLS only when needed. See the parent `../CLAUDE.md` for the overall architecture
+and release process.
+
+## Deployment-agnostic — important
+
+The HA addon is only ONE way to run this; bare npm and Docker are planned. Keep
+all code free of Home-Assistant assumptions. Anything host-specific (GPU
+devices, ffmpeg build, CLI flags) belongs in the `ha-addon` layer. Hardware
+detection must probe at runtime and fall back gracefully; do not assume a
+Linux-only host (e.g. POSIX-only signals must degrade elsewhere).
+
+## Layout
+
+- `bin/cli.js` — CLI entry; resolves `ffmpegBin` (uses `--ffmpeg-bin` if given,
+  else bundled ffmpeg-static, else PATH `ffmpeg`).
+- `server.js` — Fastify setup; detects the video encoder at startup
+  (`detectVideoEncoder`) and passes it to `HlsSessionManager`.
+- `routes/<path>/<method>.js` — same convention as the server repo.
+  - `routes/stream/get.js` — byte-range torrent file streaming (HTTP 206).
+  - `routes/api/playback-plan/post.js` — codec/container/duration probe result.
+  - `routes/transcode/session-file/get.js` — serves HLS playlist/segments;
+    long-polls while a segment is being produced, returns retryable 503 (never
+    202 — hls.js can't consume it).
+- `services/`:
+  - `playback-planner.js` — single ffmpeg probe returns audioCodec, videoCodec,
+    container, durationSeconds. `mode` is advisory; the browser decides.
+  - `hls-session-manager.js` — one ffmpeg per (source, file, settings). Serves a
+    synthetic full-duration VOD playlist; produces segments on demand; restarts
+    ffmpeg at the requested segment for server-side seeking. Short idle TTL.
+    Uses the detected `videoEncoder` for video re-encode (copy otherwise).
+  - `hwaccel.js` — detect best H.264 encoder (NVENC/QSV/VAAPI/V4L2M2M) with a
+    STRICT startup test: encode `testsrc2` through the real HLS pipeline, then
+    verify each segment decodes independently (catches non-IDR/corrupted hw
+    output). Falls back to software libx264. Runtime fallback to software if a
+    hw encode later fails. v4l2m2m is gated by this test (fails on HA Yellow).
+  - `data-channel-handler.js` — forwards WebRTC data-channel requests to the
+    local HTTP server (loopback), so the same routes serve both transports.
+
+## Gotchas
+
+- Do NOT use `-hls_playlist_type event` — it breaks duration/seek. VOD only.
+- A transitive dep (`ip-set`, via webtorrent) ships a hostile
+  `preinstall: npx only-allow pnpm` that breaks `npm install`. The addon works
+  around it with `--ignore-scripts` + a targeted rebuild of `node-datachannel`;
+  if you ever change install flow, keep that in mind.
+
+## Changelog
+
+Every behavioural change must be recorded in `CHANGELOG.md` — add an entry under
+a new `## <version>` heading at the top (the next patch version that
+`npm run patch` will publish), following the existing
+`- **New**/**Fix**/**Chore**:` format. See the parent `../CLAUDE.md`.
+
+## Release
+
+`npm run patch` (publishes to npm + pushes tags). The HA addon then needs its
+own version bump to pull the new package. Publish proxy BEFORE bumping the addon.
+
+**Any proxy change requires bumping the ha-addon version** (`ha-addon/torrent_tv_proxy/config.yaml`). The addon installs the proxy from npm at build time and the build is cached; without a version bump the plugin will NOT update and keeps running the old proxy. So after `npm run patch`, always bump the addon `config.yaml` version, push, and update the addon.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@torrent-tv/proxy",
-  "version": "2.9.0",
+  "version": "2.9.3",
   "description": "Torrent proxy client that exposes webseed-like HTTP stream endpoint.",
   "license": "GPL-3.0-or-later",
   "publishConfig": {

package/services/data-channel-handler.js

@@ -16,12 +16,23 @@
  *
  * Proxy → Browser
  * ```
- * { type: "response-start", requestId, status, headers }
- * { type: "response-chunk", requestId, data: string (base64), done: boolean }
- * { type: "response-error", requestId, error: string }
- * { type: "pong",           id }
+ * { type: "response-start", requestId, status, headers }   (JSON string)
+ * { type: "response-error", requestId, error: string }     (JSON string)
+ * { type: "pong",           id }                            (JSON string)
  * ```
  *
+ * Response bodies are sent as BINARY data-channel messages (not JSON), to
+ * avoid the ~33% base64 overhead and the JSON encode/decode cost. Each binary
+ * frame is laid out as:
+ * ```
+ * byte 0          flags     (bit 0: done)
+ * byte 1          idLen     (length of the requestId in bytes)
+ * bytes 2..2+N    requestId (ASCII)
+ * bytes 2+N..     payload   (raw body bytes; empty on the final done frame)
+ * ```
+ * Control messages stay JSON strings so the browser can distinguish them from
+ * body frames by message type (string vs ArrayBuffer).
+ *
  * The protocol mirrors the tunnel relay protocol so both transports share
  * the same mental model and the same browser-side `WebRtcProxy` implementation.
  */
@@ -177,30 +188,107 @@ export function createDataChannelHandler({ proxyPort, onLog }) {
     send(channel, { type: "response-start", requestId, status: response.status, headers: responseHeaders });
 
     if (!response.body) {
-      send(channel, { type: "response-chunk", requestId, data: "", done: true });
+      sendChunk(channel, requestId, null, true);
       return;
     }
 
     try {
       const reader = response.body.getReader();
+      // [net-debug] TEMPORARY: measure transfer size/time and channel buffering.
+      const sendStartedAt = Date.now();
+      let totalBytes = 0;
+      let maxBuffered = 0;
       while (true) {
         const { done, value } = await reader.read();
         if (done) {
-          send(channel, { type: "response-chunk", requestId, data: "", done: true });
+          sendChunk(channel, requestId, null, true);
+          const elapsedMs = Date.now() - sendStartedAt;
+          let bufferedNow = 0;
+          try { bufferedNow = typeof channel.bufferedAmount === "function" ? channel.bufferedAmount() : 0; } catch { /* ignore */ }
+          log(
+            `[net-debug] sent ${path}${queryInfo} bytes=${totalBytes} ms=${elapsedMs} ` +
+              `maxBuffered=${maxBuffered} bufferedAtEnd=${bufferedNow}`
+          );
           break;
         }
-        send(channel, {
-          type: "response-chunk",
-          requestId,
-          data: Buffer.from(value).toString("base64"),
-          done: false
-        });
+        totalBytes += value.length;
+        try {
+          const b = typeof channel.bufferedAmount === "function" ? channel.bufferedAmount() : 0;
+          if (b > maxBuffered) maxBuffered = b;
+        } catch { /* ignore */ }
+        sendChunk(channel, requestId, value, false);
+        // Backpressure: do not keep queuing chunks once the channel's outgoing
+        // buffer is large — wait for it to drain. Prevents the SCTP send buffer
+        // from ballooning, which stalls throughput.
+        await waitForBufferDrain(channel);
       }
     } catch {
-      send(channel, { type: "response-chunk", requestId, data: "", done: true });
+      sendChunk(channel, requestId, null, true);
+    }
+  }
+
+  /**
+   * Send a response body frame as a BINARY data-channel message.
+   * Layout: [flags(1)][idLen(1)][requestId(ASCII)][payload].
+   *
+   * @param {DataChannel}            channel
+   * @param {string}                 requestId
+   * @param {Uint8Array | null}      bytes - Body bytes, or null/empty for the done frame.
+   * @param {boolean}                done
+   * @returns {void}
+   */
+  function sendChunk(channel, requestId, bytes, done) {
+    try {
+      const idBuf = Buffer.from(requestId, "ascii");
+      const header = Buffer.allocUnsafe(2 + idBuf.length);
+      header[0] = done ? 1 : 0;
+      header[1] = idBuf.length;
+      idBuf.copy(header, 2);
+      const frame =
+        bytes && bytes.length > 0 ? Buffer.concat([header, Buffer.from(bytes)]) : header;
+      channel.sendMessageBinary(frame);
+    } catch {
+      // Channel closed between check and send — safe to ignore.
     }
   }
 
+  /**
+   * Resolve once the channel's outgoing buffer has drained below the low-water
+   * mark. No-op (resolves immediately) when the buffer is already small or the
+   * channel does not expose buffer APIs. A timeout fallback guards against a
+   * missed low-water event so the send loop can never deadlock.
+   *
+   * @param {DataChannel} channel
+   * @returns {Promise<void>}
+   */
+  function waitForBufferDrain(channel) {
+    return new Promise((resolve) => {
+      try {
+        if (typeof channel.bufferedAmount !== "function" || channel.bufferedAmount() <= DC_BUFFER_HIGH_WATER) {
+          resolve();
+          return;
+        }
+        let settled = false;
+        const done = () => {
+          if (settled) return;
+          settled = true;
+          resolve();
+        };
+        channel.setBufferedAmountLowThreshold(DC_BUFFER_LOW_WATER);
+        channel.onBufferedAmountLow(done);
+        // Guard against a race where the buffer drained between the check above
+        // and registering the callback (the low-water event would never fire).
+        if (channel.bufferedAmount() <= DC_BUFFER_LOW_WATER) {
+          done();
+          return;
+        }
+        setTimeout(done, DC_BUFFER_DRAIN_TIMEOUT_MS);
+      } catch {
+        resolve();
+      }
+    });
+  }
+
   /**
    * Serialise `message` to JSON and send it over the data channel.
    * Errors are silently swallowed — the channel may have closed between
@@ -226,3 +314,10 @@ export function createDataChannelHandler({ proxyPort, onLog }) {
  * Only the known proxy API and streaming routes are accepted.
  */
 const PATH_ALLOWLIST_RE = /^(?:\/api\/|\/stream(?:$|\?)|\/?transcode\/|\/health(?:z)?(?:$|\?))/;
+
+/** Pause sending body chunks once the channel buffer exceeds this many bytes. */
+const DC_BUFFER_HIGH_WATER = 8 * 1024 * 1024;
+/** Resume sending once the channel buffer drains to this many bytes. */
+const DC_BUFFER_LOW_WATER = 1 * 1024 * 1024;
+/** Safety fallback so the send loop cannot deadlock on a missed drain event. */
+const DC_BUFFER_DRAIN_TIMEOUT_MS = 5000;

package/services/hls-session-manager.js

@@ -512,7 +512,7 @@ export class HlsSessionManager {
 
     logger.info(
       `transcode ${sessionId} start "${logName}" ` +
-        `video=${transcodeVideo ? "x264" : "copy"} audio=${transcodeAudio ? "aac" : "copy"} ` +
+        `video=${transcodeVideo ? this.videoEncoder.name : "copy"} audio=${transcodeAudio ? "aac" : "copy"} ` +
         `duration=${hasDuration ? formatSeconds(durationSeconds) : "unknown"} segments=${segmentCount}`
     );
 

package/services/hwaccel.js

@@ -21,7 +21,9 @@
  */
 
 import { spawn } from "node:child_process";
-import { readdirSync } from "node:fs";
+import { mkdtempSync, readdirSync, rmSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
 
 const SOFTWARE_PRESET = "superfast";
 const SOFTWARE_CRF = "24";
@@ -141,8 +143,11 @@ function nvencDescriptor() {
 
 /** @returns {import("./hwaccel.js").VideoEncoderDescriptor} */
 function v4l2m2mDescriptor() {
-  // ARM SoC (e.g. Raspberry Pi) stateful M2M encoder. No GPU scaler — scale in
-  // software, then hand YUV420 frames to the hardware encoder.
+  // ARM SoC (e.g. Raspberry Pi / HA Yellow) stateful M2M encoder. No GPU
+  // scaler — scale in software, hand YUV420 frames to the hardware encoder.
+  // `-g` aligns the GOP to the segment length so an IDR lands on every segment
+  // boundary; this is verified by the keyframe-alignment test before use,
+  // because v4l2m2m does not always honour these hints.
   return {
     name: "h264_v4l2m2m",
     kind: "v4l2m2m",
@@ -155,12 +160,14 @@ function v4l2m2mDescriptor() {
         `scale=${w}:${h}:force_original_aspect_ratio=decrease:force_divisible_by=2,fps=${TRANSCODE_FPS},format=yuv420p`,
         "-c:v", "h264_v4l2m2m",
         "-b:v", "3M",
+        "-g", String(TRANSCODE_FPS * segmentDurationSec),
         ...keyFrameArgs(segmentDurationSec)
       ];
     }
   };
 }
 
+
 /**
  * @typedef {Object} VideoEncoderDescriptor
  * @property {string} name
@@ -253,44 +260,89 @@ function hasV4l2Device() {
 }
 
 /**
- * Kind-specific test-encode args that verify the encoder initialises and
- * encodes a few frames from a synthetic source.
+ * Build a full ffmpeg command that encodes a short, *moving* synthetic clip
+ * (testsrc2 — far more representative than a static black frame) through the
+ * candidate encoder into real HLS segments in `outDir`, with keyframes forced
+ * on segment boundaries. Verifying the resulting segments (see
+ * {@link verifySegmentsDecodeCleanly}) catches encoders that silently produce
+ * a corrupted or non-IDR-aligned stream (e.g. some V4L2 M2M builds).
  *
  * @param {VideoEncoderDescriptor} descriptor
+ * @param {number} segmentDurationSec
+ * @param {string} outDir
  * @returns {string[]}
  */
-function testEncodeArgs(descriptor) {
-  const src = ["-f", "lavfi", "-i", "color=c=black:s=320x240:r=15:d=0.4"];
+function buildEncoderTestArgs(descriptor, segmentDurationSec, outDir) {
+  const durationSec = Math.max(8, segmentDurationSec * 3);
+  const source = ["-f", "lavfi", "-i", `testsrc2=s=640x360:r=${TRANSCODE_FPS}:d=${durationSec}`];
+  const kf = keyFrameArgs(segmentDurationSec);
+
+  /** @type {string[]} */
+  let pre = ["-hide_banner", "-loglevel", "error"];
+  /** @type {string[]} */
+  let encode;
   switch (descriptor.kind) {
     case "vaapi":
-      return [
-        "-hide_banner", "-loglevel", "error",
-        "-vaapi_device", String(descriptor.device),
-        ...src,
-        "-vf", "format=nv12,hwupload", "-c:v", "h264_vaapi",
-        "-f", "null", "-"
-      ];
+      pre = [...pre, "-vaapi_device", String(descriptor.device)];
+      encode = ["-vf", "format=nv12,hwupload", "-c:v", "h264_vaapi", "-qp", "24", ...kf];
+      break;
     case "qsv":
-      return [
-        "-hide_banner", "-loglevel", "error",
-        "-qsv_device", String(descriptor.device),
-        ...src,
-        "-vf", "hwupload=extra_hw_frames=16,format=qsv", "-c:v", "h264_qsv",
-        "-f", "null", "-"
-      ];
+      pre = [...pre, "-qsv_device", String(descriptor.device)];
+      encode = ["-vf", "hwupload=extra_hw_frames=16,format=qsv", "-c:v", "h264_qsv", "-global_quality", "24", ...kf];
+      break;
     case "nvenc":
-      return ["-hide_banner", "-loglevel", "error", ...src, "-c:v", "h264_nvenc", "-f", "null", "-"];
+      encode = ["-c:v", "h264_nvenc", "-preset", "p4", "-cq", "24", "-pix_fmt", "yuv420p", ...kf];
+      break;
     case "v4l2m2m":
-      return [
-        "-hide_banner", "-loglevel", "error",
-        ...src, "-pix_fmt", "yuv420p", "-c:v", "h264_v4l2m2m", "-f", "null", "-"
-      ];
+      encode = ["-pix_fmt", "yuv420p", "-c:v", "h264_v4l2m2m", "-b:v", "3M", "-g", String(TRANSCODE_FPS * segmentDurationSec), ...kf];
+      break;
     default:
-      return [
-        "-hide_banner", "-loglevel", "error",
-        ...src, "-c:v", "libx264", "-preset", "ultrafast", "-f", "null", "-"
-      ];
+      encode = ["-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p", ...kf];
+      break;
   }
+
+  const hlsOut = [
+    "-f", "hls",
+    "-hls_time", String(segmentDurationSec),
+    "-hls_list_size", "0",
+    "-hls_flags", "independent_segments",
+    "-hls_segment_filename", path.join(outDir, "seg-%03d.ts"),
+    path.join(outDir, "index.m3u8")
+  ];
+  return [...pre, ...source, ...encode, ...hlsOut];
+}
+
+/**
+ * Verify the HLS segments produced by the test encode are valid: at least two
+ * segments exist, and each decodes standalone without errors. A segment that
+ * does not begin with a keyframe (broken/corrupted output) emits decode errors
+ * when read on its own, which fails this check.
+ *
+ * @param {string} ffmpegBin
+ * @param {string} outDir
+ * @returns {Promise<boolean>}
+ */
+async function verifySegmentsDecodeCleanly(ffmpegBin, outDir) {
+  let files;
+  try {
+    files = readdirSync(outDir).filter((n) => /^seg-\d+\.ts$/.test(n)).sort();
+  } catch {
+    return false;
+  }
+  if (files.length < 2) {
+    return false;
+  }
+  for (const file of files) {
+    const result = await runFfmpeg(
+      ffmpegBin,
+      ["-hide_banner", "-loglevel", "error", "-i", path.join(outDir, file), "-f", "null", "-"],
+      8000
+    );
+    if (result.code !== 0 || result.stderr.trim().length > 0) {
+      return false;
+    }
+  }
+  return true;
 }
 
 /**
@@ -298,10 +350,10 @@ function testEncodeArgs(descriptor) {
  * software libx264). Each hardware candidate is verified with a real
  * test-encode before being selected.
  *
- * @param {{ ffmpegBin: string, logger?: { info: (m: string) => void, warn: (m: string) => void } }} options
+ * @param {{ ffmpegBin: string, logger?: { info: (m: string) => void, warn: (m: string) => void }, segmentDurationSec?: number }} options
  * @returns {Promise<VideoEncoderDescriptor>}
  */
-export async function detectVideoEncoder({ ffmpegBin, logger }) {
+export async function detectVideoEncoder({ ffmpegBin, logger, segmentDurationSec = 4 }) {
   const log = logger ?? { info: () => {}, warn: () => {} };
   const software = softwareDescriptor();
 
@@ -324,20 +376,41 @@ export async function detectVideoEncoder({ ffmpegBin, logger }) {
   if (has("h264_vaapi") && renderNodes.length > 0) {
     candidates.push(vaapiDescriptor(renderNodes[0]));
   }
+  // h264_v4l2m2m (ARM SoC / Raspberry Pi / HA Yellow). It is gated behind the
+  // strict keyframe-alignment test below, because some V4L2 M2M builds silently
+  // emit a corrupted / non-IDR-aligned stream; the test rejects those and the
+  // host falls back to software libx264.
   if (has("h264_v4l2m2m") && hasV4l2Device()) {
     candidates.push(v4l2m2mDescriptor());
   }
 
   for (const candidate of candidates) {
-    const result = await runFfmpeg(ffmpegBin, testEncodeArgs(candidate), 12000);
-    if (result.code === 0) {
+    const dir = mkdtempSync(path.join(os.tmpdir(), "tt-hwtest-"));
+    let ok = false;
+    try {
+      const encoded = await runFfmpeg(
+        ffmpegBin,
+        buildEncoderTestArgs(candidate, segmentDurationSec, dir),
+        25000
+      );
+      if (encoded.code === 0) {
+        ok = await verifySegmentsDecodeCleanly(ffmpegBin, dir);
+      }
+    } finally {
+      try {
+        rmSync(dir, { recursive: true, force: true });
+      } catch {
+        // best effort
+      }
+    }
+    if (ok) {
       log.info(
         `hwaccel: using hardware encoder ${candidate.name}` +
           `${candidate.device ? ` (${candidate.device})` : ""}`
       );
       return candidate;
     }
-    log.warn(`hwaccel: ${candidate.name} present but test-encode failed; skipping`);
+    log.warn(`hwaccel: ${candidate.name} failed the HLS keyframe-alignment test; skipping`);
   }
 
   log.info("hwaccel: no working hardware encoder; using software libx264");