@torrent-tv/proxy 2.9.34 → 2.9.36

package/CHANGELOG.md

@@ -1,3 +1,8 @@
+## 2.9.35
+
+- **New**: Chunked request bodies over the data channel (OpenSpec change `chunked-request-bodies`). Large request bodies — notably the source registration, whose body is the base64 `.torrent` (hundreds of KB for a multi-season pack) — now arrive as bounded binary frames (the response-frame layout) announced by a `request-start` message, and are reassembled and run through the same path as a single-message request. Bounded: 32 MB per-body cap, a 60 s TTL for incomplete bodies, an abort frame that drops partial state at once, and all per-channel state freed on channel close. This removes the single-message size ceiling symmetrically with responses (which already stream in chunks). Logged as `body=<bytes> bytes (chunked)`.
+- **Fix**: Large torrents (many files / seasons) no longer fail with "Trying to send message larger than max-message-size" when a file is picked. The browser sends the source registration body — the base64-encoded `.torrent` — in a single data-channel message; a big multi-season pack's `.torrent` carries thousands of piece hashes (e.g. Poirot, 13 seasons: 420 KB → ~560 KB base64), exceeding libdatachannel's default advertised limit of 256 KB, so the browser's `channel.send()` threw. The proxy now advertises a 16 MB `a=max-message-size`, so a large single send still works while already-open tabs run the old bundle. Verified the SDP now carries `a=max-message-size:16777216` (was `262144`).
+
 ## 2.9.34
 
 - **New**: Cold-start reduction (OpenSpec change `cold-start`). Creating a transcode session no longer runs a second full ffmpeg input scan: the playback planner caches the media info (duration/resolution/fps/start-time/HDR) parsed from the probe it already ran, and `createSession` reuses it (falling back to its own probe only when the cache cannot serve — e.g. after a restart, or a missing critical field). The banner parsers now live in a shared `ffmpeg-banner.js` so both sides parse identically. Once a plan probe succeeds the proxy also warms the START of the file body (~16 MB, fire-and-forget) so the first segment's encode reads downloaded data instead of waiting on pieces. Session startup is now measurable in the log: `cold-start <id>: media-info=<ms> (cached|probed) keyframes=<ms|skipped> create-total=<ms>` and, once per session, `cold-start <id>: first-segment ready +<ms>`.

package/openspec/changes/chunked-request-bodies/design.md

@@ -0,0 +1,81 @@
+# Design: Chunked request bodies (proxy side)
+
+Written to be executed as specified — the wire format, constants, limits
+and do-NOT list are normative. Read before coding:
+
+- `services/data-channel-handler.js`: the whole file — the wire-protocol
+  doc comment (~lines 9–37), `handleChannel` (onMessage string handling),
+  `handleRequest`, `sendChunk` (the response frame writer whose layout the
+  request frames mirror), `send`.
+
+## Wire protocol (additions)
+
+Existing messages are UNCHANGED. New:
+
+    Browser → Proxy, announcing a chunked request:
+      { type: "request-start", requestId, method, path, query, headers,
+        bodyBytes }          // bodyBytes = exact total body size in bytes
+
+    Browser → Proxy, body frames (BINARY messages — today the proxy only
+    ever receives strings, so binary is unambiguous):
+      byte 0        flags     bit 0: done (last frame)
+                              bit 1: aborted (drop this request, no reply)
+      byte 1        idLen     requestId length in bytes
+      bytes 2..2+N  requestId (ASCII)
+      bytes 2+N..   payload   raw body bytes (UTF-8 of the body string;
+                              may be empty on a done/abort frame)
+
+Identical layout to the response frames (`sendChunk`) — one mental model,
+and the browser already has a parser for it (its builder mirrors it).
+
+No capability negotiation: POC, single-proxy pool, lockstep releases
+(proxy first, then server). The 16 MB `maxMessageSize` advertisement in
+webrtc-manager.js (pending 2.9.35) stays as a one-line transition cover
+for tabs still running the single-send bundle.
+
+## Constants
+
+    PROXY_MAX_REQUEST_BODY_BYTES = 32 * 1024 * 1024
+    PARTIAL_REQUEST_TTL_MS = 60_000
+
+## Assembly (all state per channel, inside `handleChannel`'s closure)
+
+    const partials = new Map(); // requestId → { meta, chunks: [], receivedBytes, timer }
+
+- `request-start`: validate like a legacy request (same path allowlist —
+  run the SAME validation before accepting the body; reject with
+  `response-error` immediately on a bad path). Reject `bodyBytes` >
+  PROXY_MAX_REQUEST_BODY_BYTES with `response-error` "Request body too
+  large." and do NOT create state. Otherwise store `{ meta, chunks: [],
+  receivedBytes: 0 }` and arm the TTL timer.
+- Binary frame: parse header; unknown requestId → ignore (stale/aborted).
+  bit 1 (aborted) → clear timer, delete entry, no reply. Append payload,
+  add to receivedBytes; receivedBytes > bodyBytes (or > the cap) →
+  `response-error` + drop. bit 0 (done): concat chunks → body string via
+  `Buffer.concat(...).toString("utf8")`; receivedBytes !== bodyBytes →
+  `response-error` "Request body size mismatch." and drop; else clear
+  timer, delete entry, and execute through the SAME code path a legacy
+  `request` message takes (factor `handleRequest` so both entry points
+  call one function with `{requestId, method, path, query, headers, body}`).
+- TTL fire: delete entry, log
+  `[dc] Session …: dropped stale partial request <id8> (<receivedBytes>B)`.
+- `channel.onClosed`: clear ALL timers and the map (extend the existing
+  handler — do not replace its logging).
+
+## Logging
+
+Chunked request execution logs the SAME `[dc] <method> <path>` line as
+legacy, with `body=<bytes> bytes (chunked)`.
+
+## Rules — do NOT
+
+- Do NOT change the legacy `{type:"request"}` handling, the response
+  framing, ping/pong, or the path allowlist semantics.
+- Do NOT add capability/version negotiation — POC decision, revisit only
+  when the pool has independently-updated proxies.
+- Do NOT revert the 16 MB `maxMessageSize` advertisement (pending 2.9.35).
+- Do NOT hold partial bodies beyond the TTL or channel lifetime; no global
+  (cross-channel) state.
+- Do NOT create a new proxy version: fold into the pending 2.9.35
+  CHANGELOG entry (accumulate bullets, per the versioning rules) and the
+  pending addon 0.2.56 entry.

package/openspec/changes/chunked-request-bodies/proposal.md

@@ -0,0 +1,53 @@
+# Proposal: Chunked request bodies over the data channel (proxy side)
+
+## Why
+
+The browser sends every request as ONE data-channel message, body included.
+Registering a source carries the base64-encoded `.torrent` as that body — a
+big multi-season pack (Poirot, 13 seasons: 420 KB `.torrent` → ~560 KB
+base64) exceeded libdatachannel's default advertised `a=max-message-size`
+of 256 KB, so the browser's `send()` threw "Trying to send message larger
+than max-message-size" and playback dead-ended.
+
+The committed-but-unpublished stopgap (advertise 16 MB, pending 2.9.35)
+lifts the ceiling but keeps the flaw: any single-message body still has a
+hard cap, and a large message is buffered whole on both ends. Responses
+already solved this properly — they stream as small binary frames. Requests
+should be symmetric.
+
+## What Changes
+
+- **Inbound binary body frames.** The proxy accepts request bodies as
+  binary frames with EXACTLY the response-frame layout
+  (`[flags][idLen][requestId][payload]`, bit 0 = done; new bit 1 = aborted),
+  announced by a new `{type:"request-start", …, bodyBytes}` control message.
+  On the done frame the assembled body runs through the SAME request
+  execution path as a legacy request. Bounded: per-body cap 32 MB, partial
+  bodies dropped after a 60 s TTL or an abort frame, all per-channel state
+  freed on channel close.
+- **No capability negotiation.** POC: the pool is one proxy, released in
+  lockstep with the site (proxy first, then server). The browser just uses
+  chunked frames for large bodies; this proxy just understands them.
+- **The 16 MB `max-message-size` advertisement (pending 2.9.35) stays** —
+  a single config value, no logic: it covers the transition window while
+  already-open tabs still run the single-send bundle.
+- **Observability**: the `[dc]` request log line reports chunked bodies
+  (`body=<bytes> bytes (chunked)`).
+
+Browser-side counterpart (chunk writer, threshold, backpressure, abort) is
+the server repo's `chunked-request-bodies` change. Release order: proxy
+(with addon bump) FIRST, then server.
+
+## Capabilities
+
+### New Capabilities
+
+- `chunked-request-bodies`: request-body transport over the data channel.
+
+## Impact
+
+- `services/data-channel-handler.js` — binary inbound frame parsing;
+  per-channel partial-body assembly with caps/TTL; `request-start`
+  handling; shared execution path.
+- Release: folds into the PENDING proxy 2.9.35 (extend its CHANGELOG entry;
+  do not create a new version) + ha-addon 0.2.56 (same rule).

package/openspec/changes/chunked-request-bodies/specs/chunked-request-bodies/spec.md

@@ -0,0 +1,30 @@
+# chunked-request-bodies — delta spec (proxy)
+
+## ADDED Requirements
+
+### Requirement: Request bodies arrive in bounded binary chunks
+
+The proxy SHALL accept a request whose body is delivered as binary frames
+(mirroring the response-frame layout) announced by a `request-start`
+control message, assemble it, and execute it through the same path as a
+single-message request. Assembly SHALL be bounded: a per-body byte cap, a
+TTL for incomplete bodies, an abort flag that drops the partial state, and
+release of all partial state when the channel closes. Legacy single-message
+requests SHALL keep working unchanged.
+
+#### Scenario: Large .torrent registration
+- **WHEN** the browser registers a multi-season torrent whose base64 body
+  exceeds any single-message limit
+- **THEN** the body arrives in frames, the source registers, and the
+  response streams back exactly as for a small request
+
+#### Scenario: Oversized or inconsistent body
+- **WHEN** the announced or delivered size exceeds the cap, or the
+  delivered bytes do not match the announcement
+- **THEN** the proxy replies with a response-error for that request and
+  drops the partial state; the channel and other requests are unaffected
+
+#### Scenario: Sender vanishes mid-body
+- **WHEN** frames stop arriving (tab closed, aborted without a frame)
+- **THEN** the partial body is dropped after the TTL (or immediately on
+  channel close) and its memory is released

package/openspec/changes/chunked-request-bodies/tasks.md

@@ -0,0 +1,40 @@
+# Tasks: Chunked request bodies (proxy side)
+
+Execute in order; design.md is normative. Read the code regions listed at
+its top first.
+
+## 1. Implementation
+
+- [ ] 1.1 `data-channel-handler.js`: factor the execution tail of
+      `handleRequest` so a legacy `request` message and an assembled
+      chunked request run the SAME function. `node --check`.
+- [ ] 1.2 `request-start` + binary inbound frames + per-channel assembly
+      with cap (32 MB), size-mismatch check, abort flag (bit 1), TTL
+      (60 s), cleanup on channel close. Unknown-requestId frames ignored.
+- [ ] 1.3 Logging: `body=<bytes> bytes (chunked)` on execution; stale-drop
+      line on TTL.
+
+## 2. Verification
+
+- [ ] 2.1 Node loopback test (two node-datachannel PeerConnections in one
+      script, like the SDP test): drive a request-start + 64 KB frames of a
+      ~600 KB body through a real channel into a handler instance wired to
+      a stub local server; assert the assembled body bytes match and a
+      response comes back. Also: abort frame → no reply, state dropped;
+      oversized announcement → response-error.
+- [ ] 2.2 Legacy regression: single-message request path byte-identical
+      behaviour (run an existing small request through both entry points).
+- [ ] 2.3 E2E (after the server-side change lands in preview): local stack
+      — preview server + local proxy (`node bin/cli.js --server-url
+      ws://localhost:8080`), register the real Poirot `.torrent`
+      (C:\Users\AntonNemtsev\Downloads\Пуаро_…​.torrent, ~560 KB base64)
+      via the UI; plan returns; `[dc] … body=… (chunked)` in the proxy log.
+
+## 3. Release
+
+- [ ] 3.1 EXTEND the pending 2.9.35 CHANGELOG entry (do not bump again) and
+      the pending addon 0.2.56 entry.
+- [ ] 3.2 `npm run patch` in proxy (publishes 2.9.35), push addon bump,
+      update the addon in HA; verify `Starting @torrent-tv/proxy v2.9.35`
+      in the addon log. Proxy FIRST, then addon, then the server-side
+      change releases independently.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@torrent-tv/proxy",
-  "version": "2.9.34",
+  "version": "2.9.36",
   "description": "Torrent proxy client that exposes webseed-like HTTP stream endpoint.",
   "license": "GPL-3.0-or-later",
   "publishConfig": {

package/services/data-channel-handler.js

@@ -95,35 +95,149 @@ export function createDataChannelHandler({ proxyPort, onLog }) {
    * @returns {void}
    */
   function handleChannel(sessionId, channel) {
-    log(`[dc] Session ${sessionId.slice(0, 8)}: channel open`);
+    const tag = sessionId.slice(0, 8);
+    log(`[dc] Session ${tag}: channel open`);
+
+    // Partial chunked-request bodies in flight on THIS channel, keyed by
+    // requestId. Each entry buffers frames until the done frame, then runs the
+    // assembled request through the same path as a single-message request.
+    /** @type {Map<string, { meta: object, chunks: Buffer[], receivedBytes: number, bodyBytes: number, timer: ReturnType<typeof setTimeout> }>} */
+    const partials = new Map();
+
+    const dropPartial = (requestId) => {
+      const entry = partials.get(requestId);
+      if (entry) {
+        clearTimeout(entry.timer);
+        partials.delete(requestId);
+      }
+    };
+
+    /**
+     * Begin assembling a chunked request. Validates the path and size up front
+     * so an invalid or oversized request never buffers a body.
+     *
+     * @param {any} message - The `request-start` control message.
+     */
+    const startPartialRequest = (message) => {
+      const { requestId, method, path, query, headers, bodyBytes } = message ?? {};
+      if (typeof requestId !== "string" || requestId.length === 0) {
+        return;
+      }
+      if (!isValidRequestPath(path)) {
+        send(channel, { type: "response-error", requestId, error: "Invalid request path." });
+        return;
+      }
+      if (!Number.isInteger(bodyBytes) || bodyBytes < 0 || bodyBytes > PROXY_MAX_REQUEST_BODY_BYTES) {
+        send(channel, { type: "response-error", requestId, error: "Request body too large." });
+        return;
+      }
+      dropPartial(requestId); // replace any stale entry with the same id
+      const timer = setTimeout(() => {
+        const entry = partials.get(requestId);
+        partials.delete(requestId);
+        log(`[dc] Session ${tag}: dropped stale partial request ${requestId.slice(0, 8)} (${entry?.receivedBytes ?? 0}B)`);
+      }, PARTIAL_REQUEST_TTL_MS);
+      partials.set(requestId, {
+        meta: { requestId, method, path, query, headers },
+        chunks: [],
+        receivedBytes: 0,
+        bodyBytes,
+        timer
+      });
+    };
+
+    /**
+     * Handle a binary body frame for a chunked request.
+     * Layout: [flags(1)][idLen(1)][requestId(ASCII)][payload].
+     *
+     * @param {Buffer} buf
+     */
+    const handleBodyFrame = (buf) => {
+      if (buf.length < 2) {
+        return;
+      }
+      const flags = buf[0];
+      const idLen = buf[1];
+      if (buf.length < 2 + idLen) {
+        return;
+      }
+      const requestId = buf.toString("ascii", 2, 2 + idLen);
+      const entry = partials.get(requestId);
+      if (!entry) {
+        return; // stale / already-dropped / aborted
+      }
+      if (flags & 2) {
+        // Aborted by the browser — drop silently, no reply.
+        dropPartial(requestId);
+        return;
+      }
+      if (buf.length > 2 + idLen) {
+        const payload = buf.subarray(2 + idLen);
+        entry.chunks.push(Buffer.from(payload));
+        entry.receivedBytes += payload.length;
+      }
+      if (entry.receivedBytes > entry.bodyBytes || entry.receivedBytes > PROXY_MAX_REQUEST_BODY_BYTES) {
+        dropPartial(requestId);
+        send(channel, { type: "response-error", requestId, error: "Request body size mismatch." });
+        return;
+      }
+      if (flags & 1) {
+        // Done frame — assemble and execute.
+        dropPartial(requestId);
+        if (entry.receivedBytes !== entry.bodyBytes) {
+          send(channel, { type: "response-error", requestId, error: "Request body size mismatch." });
+          return;
+        }
+        const body = Buffer.concat(entry.chunks).toString("utf8");
+        void handleRequest(channel, { ...entry.meta, body }, true).catch((error) => {
+          log(`[dc] Session ${tag}: request error: ${error?.message ?? error}`);
+        });
+      }
+    };
 
     channel.onMessage((raw) => {
-      /** @type {DataChannelRequest | { type: "ping", id: string }} */
+      // Binary messages are chunked-request body frames; the proxy otherwise
+      // only ever receives JSON strings, so the type discriminates cleanly.
+      if (typeof raw !== "string") {
+        handleBodyFrame(Buffer.isBuffer(raw) ? raw : Buffer.from(raw));
+        return;
+      }
+
+      /** @type {DataChannelRequest | { type: string, id?: string }} */
       let message;
       try {
-        message = JSON.parse(typeof raw === "string" ? raw : raw.toString());
+        message = JSON.parse(raw);
       } catch {
         return;
       }
 
       if (message.type === "request") {
         void handleRequest(channel, message).catch((error) => {
-          log(`[dc] Session ${sessionId.slice(0, 8)}: request error: ${error?.message ?? error}`);
+          log(`[dc] Session ${tag}: request error: ${error?.message ?? error}`);
         });
         return;
       }
 
+      if (message.type === "request-start") {
+        startPartialRequest(message);
+        return;
+      }
+
       if (message.type === "ping") {
         send(channel, { type: "pong", id: message.id });
       }
     });
 
     channel.onClosed(() => {
-      log(`[dc] Session ${sessionId.slice(0, 8)}: channel closed`);
+      for (const entry of partials.values()) {
+        clearTimeout(entry.timer);
+      }
+      partials.clear();
+      log(`[dc] Session ${tag}: channel closed`);
     });
 
     channel.onError((err) => {
-      log(`[dc] Session ${sessionId.slice(0, 8)}: channel error: ${err}`);
+      log(`[dc] Session ${tag}: channel error: ${err}`);
     });
   }
 
@@ -138,24 +252,22 @@ export function createDataChannelHandler({ proxyPort, onLog }) {
    * @param {DataChannelRequest} req
    * @returns {Promise<void>}
    */
-  async function handleRequest(channel, req) {
+  async function handleRequest(channel, req, viaChunks = false) {
     const { requestId, method, path, query, headers: forwardedHeaders, body } = req;
 
     // Reject paths that are not absolute, contain traversal sequences, or
     // do not start with a known proxy route prefix.  All valid browser-side
     // requests use /api/*, /stream, /transcode/*, /health, or /healthz.
-    if (
-      typeof path !== "string" ||
-      !path.startsWith("/") ||
-      path.includes("..") ||
-      !PATH_ALLOWLIST_RE.test(path)
-    ) {
+    if (!isValidRequestPath(path)) {
       send(channel, { type: "response-error", requestId, error: "Invalid request path." });
       return;
     }
 
     const queryInfo = query ? `?${query}` : "";
-    const bodyInfo = body != null && typeof body === "string" && body.length > 0 ? ` body=${body.length} bytes` : "";
+    const bodyInfo =
+      body != null && typeof body === "string" && body.length > 0
+        ? ` body=${body.length} bytes${viaChunks ? " (chunked)" : ""}`
+        : "";
     log(`[dc] ${method} ${path}${queryInfo}${bodyInfo}`);
 
     const targetUrl = `http://127.0.0.1:${proxyPort}${path}${query ? `?${query}` : ""}`;
@@ -328,6 +440,27 @@ export function createDataChannelHandler({ proxyPort, onLog }) {
  */
 const PATH_ALLOWLIST_RE = /^(?:\/api\/|\/stream(?:$|\?)|\/?transcode\/|\/health(?:z)?(?:$|\?))/;
 
+/**
+ * True when `path` is an absolute, traversal-free path on a known proxy route.
+ * Shared by the single-message and chunked request entry points.
+ *
+ * @param {unknown} path
+ * @returns {boolean}
+ */
+function isValidRequestPath(path) {
+  return (
+    typeof path === "string" &&
+    path.startsWith("/") &&
+    !path.includes("..") &&
+    PATH_ALLOWLIST_RE.test(path)
+  );
+}
+
+/** Max assembled size of a chunked request body (guards proxy memory). */
+const PROXY_MAX_REQUEST_BODY_BYTES = 32 * 1024 * 1024;
+/** Drop an incomplete chunked body if no further frame arrives within this window. */
+const PARTIAL_REQUEST_TTL_MS = 60_000;
+
 /** Pause sending body chunks once the channel buffer exceeds this many bytes. */
 const DC_BUFFER_HIGH_WATER = 8 * 1024 * 1024;
 /** Resume sending once the channel buffer drains to this many bytes. */

package/services/webrtc-manager.js

@@ -25,6 +25,18 @@ const ICE_SERVERS = ["stun:stun.l.google.com:19302", "stun:stun.cloudflare.com:3
 // values without bloating the SDP.
 const PORT_PREDICTION_WINDOW = 16;
 
+// Max size of a single data-channel message the proxy advertises (SDP
+// `a=max-message-size`) and will accept. The browser caps `channel.send()` at
+// the REMOTE's advertised value, so this is what lets the browser send a large
+// request body in one message — notably registering a source, whose body is
+// the base64-encoded .torrent. A big multi-season pack's .torrent (thousands of
+// piece hashes) can be hundreds of KB (e.g. Poirot: 420 KB → ~560 KB base64),
+// which exceeds libdatachannel's ~256 KB default and made `send()` throw
+// "message larger than max-message-size". 16 MB is a generous ceiling (memory
+// is allocated per actual message, not reserved). Responses (proxy→browser)
+// are already safe — they stream in small reader-sized chunks.
+const MAX_DC_MESSAGE_BYTES = 16 * 1024 * 1024;
+
 /**
  * Build predicted srflx ICE candidates for a symmetric NAT.
  *
@@ -195,7 +207,7 @@ export function createWebRtcManager({ sendSignal, onDataChannel, onLog, udpPort,
   }
 
   // Base PeerConnection config shared by every session.
-  const pcConfig = { iceServers: ICE_SERVERS };
+  const pcConfig = { iceServers: ICE_SERVERS, maxMessageSize: MAX_DC_MESSAGE_BYTES };
 
   // Single-port UDP mux: create ONE persistent listener that owns the shared
   // UDP socket for the proxy's whole lifetime, then have every PeerConnection