@torrent-tv/proxy 2.9.71 → 2.9.73

package/CHANGELOG.md

@@ -1,3 +1,12 @@
+## 2.9.73
+
+- **Fix**: File stats came back as `{}` after the torrent moved to its own thread (2.9.71), which left the loading screen with no peers, no speed and no progress. Two call sites — the stats route and the health report — invoked `getFileStats` **without awaiting**: it used to answer locally and immediately, and now crosses a thread boundary, so the reply was the pending promise itself, serialised to an empty object. Both now await it.
+- **Chore**: Chunk transfer no longer hands over memory the chunk does not own outright. Node allocates small buffers from a shared 8 KB pool — several unrelated buffers occupy one region, each viewing its own slice (verified: a 1 KB buffer reports an 8192-byte region at offset 8) — so transferring that region would detach it from its neighbours. Chunks sourced from the network are small enough to be pooled while local disk reads are not, which is exactly the difference between the field host and the local test. Measured afterwards: pooled chunks cross intact, so this is a correctness guard rather than the cause of the field failure.
+
+## 2.9.72
+
+- **Fix**: Playback broke entirely after the torrent moved to its own thread (2.9.71): the torrent was deleted **with its downloaded data** while still being read, after which every read hung and ffmpeg saw an empty input (`Stream ends prematurely at 0, should be 3303133078`), and the container-index read took 73 s to return nothing. Cause: `acquireFile` was dispatched without awaiting while its release was sent normally, so a release could overtake the acquire it belonged to; the reader count then hit zero mid-read and the idle sweep fired (`removed idle torrent ... and its store`). Two fixes, each sufficient alone: the release is now chained onto the acquire so it can never arrive first, and the worker additionally holds the file for the whole duration of the read — a claim that lives inside the read and so cannot be reordered against it. Not reproducible locally, where the test torrent was fully downloaded and never went idle.
+
 ## 2.9.71
 
 - **New**: The torrent client now runs on its own thread (`services/torrent-worker/`). Profiling a live seek (2026-08-02) found the main thread ~85% occupied by WebTorrent — buffer concatenation in `uint8-util` ~15%, `_updateWire` and its wrapper ~9%, garbage collection ~5%, and **no piece hashing at all**, which had been the standing assumption — while three of four cores idled. Serving a segment shared that thread, so reading an already-finished 10 MB file off SSD took **12-23 s** where handing it to the channel took 125 ms. Measured after the split, through the real `/stream` route: **3 MB in 0.05-0.12 s** (~500 Mbps), roughly a hundredfold improvement, with main-thread event-loop delay down from 300-390 ms to **28-38 ms**.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@torrent-tv/proxy",
-  "version": "2.9.71",
+  "version": "2.9.73",
   "description": "Torrent proxy client that exposes webseed-like HTTP stream endpoint.",
   "license": "GPL-3.0-or-later",
   "publishConfig": {

package/routes/api/sources/stats/get.js

@@ -1,66 +1,69 @@
-import { logger } from "../../../../utils/logger.js";
-
-/**
- * Return download statistics for a registered torrent source.
- *
- * Provides peer count, transfer speeds, and per-file download progress so
- * that the browser client can display meaningful feedback while the proxy is
- * pre-fetching file metadata (MOOV atom / EBML headers) before codec probing.
- *
- * GET /api/sources/:sourceKey/stats?fileIndex=N
- *
- * @param {import("fastify").FastifyRequest} req
- * @param {import("fastify").FastifyReply} reply
- * @param {{
- *   sourceRegistry: ReturnType<import("../../../../store/source-registry.js").createSourceRegistry>,
- *   torrentPool: import("../../../../services/torrent-pool.js").TorrentPool
- * }} deps
- * @returns {Promise<void>}
- */
-export async function handleApiSourceStatsGet(req, reply, { sourceRegistry, torrentPool }) {
-  const sourceKey = typeof req.params.sourceKey === "string" ? req.params.sourceKey.trim() : "";
-  if (!sourceKey) {
-    return reply.code(400).send({ error: "sourceKey is required." });
-  }
-
-  const sourceRecord = sourceRegistry.get(sourceKey);
-  if (!sourceRecord) {
-    return reply.code(404).send({ error: "Source key was not found." });
-  }
-
-  let torrent;
-  try {
-    // getTorrent resolves immediately when the torrent is already loaded.
-    torrent = await torrentPool.getTorrent(sourceRecord.sourceType, sourceRecord.source);
-  } catch (error) {
-    const message = error instanceof Error ? error.message : String(error);
-    return reply.code(500).send({ error: `Failed to load torrent: ${message}` });
-  }
-
-  const fileIndexRaw = typeof req.query.fileIndex === "string" ? req.query.fileIndex : "";
-  const fileIndex = fileIndexRaw !== "" && /^\d+$/.test(fileIndexRaw) ? Number(fileIndexRaw) : null;
-
-  // Optional: pin the resume window to a FIXED byte offset for the duration of
-  // one buffering episode (see getFileStats JSDoc) instead of the live, moving
-  // read position — otherwise "bytes needed" can jump up mid-poll as the window
-  // slides forward with playback/encoding progress.
-  const resumeAnchorRaw = typeof req.query.resumeAnchorByteStart === "string" ? req.query.resumeAnchorByteStart : "";
-  const resumeAnchorByteStart = resumeAnchorRaw !== "" && /^\d+$/.test(resumeAnchorRaw) ? Number(resumeAnchorRaw) : null;
-
-  const stats = torrentPool.getFileStats(torrent, fileIndex, { resumeAnchorByteStart });
-
-  // Diagnostic: surface the real swarm state per poll so a cold-start download
-  // stall (0 peers / header not advancing → playback-plan blocks on the codec
-  // probe → browser timeout) is visible in the proxy log.
-  const downKbps = (stats.downloadSpeed / 1024).toFixed(0);
-  const filePct = stats.fileProgress != null ? `${(stats.fileProgress * 100).toFixed(1)}%` : "n/a";
-  const header =
-    stats.headerBytes != null
-      ? `${stats.headerDownloadedBytes}/${stats.headerBytes}B`
-      : "n/a";
-  logger.info(
-    `[stats] ${sourceKey.slice(0, 8)} peers=${stats.numPeers} down=${downKbps}KB/s file=${filePct} header=${header}`
-  );
-
-  return reply.send(stats);
-}
+import { logger } from "../../../../utils/logger.js";
+
+/**
+ * Return download statistics for a registered torrent source.
+ *
+ * Provides peer count, transfer speeds, and per-file download progress so
+ * that the browser client can display meaningful feedback while the proxy is
+ * pre-fetching file metadata (MOOV atom / EBML headers) before codec probing.
+ *
+ * GET /api/sources/:sourceKey/stats?fileIndex=N
+ *
+ * @param {import("fastify").FastifyRequest} req
+ * @param {import("fastify").FastifyReply} reply
+ * @param {{
+ *   sourceRegistry: ReturnType<import("../../../../store/source-registry.js").createSourceRegistry>,
+ *   torrentPool: import("../../../../services/torrent-pool.js").TorrentPool
+ * }} deps
+ * @returns {Promise<void>}
+ */
+export async function handleApiSourceStatsGet(req, reply, { sourceRegistry, torrentPool }) {
+  const sourceKey = typeof req.params.sourceKey === "string" ? req.params.sourceKey.trim() : "";
+  if (!sourceKey) {
+    return reply.code(400).send({ error: "sourceKey is required." });
+  }
+
+  const sourceRecord = sourceRegistry.get(sourceKey);
+  if (!sourceRecord) {
+    return reply.code(404).send({ error: "Source key was not found." });
+  }
+
+  let torrent;
+  try {
+    // getTorrent resolves immediately when the torrent is already loaded.
+    torrent = await torrentPool.getTorrent(sourceRecord.sourceType, sourceRecord.source);
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    return reply.code(500).send({ error: `Failed to load torrent: ${message}` });
+  }
+
+  const fileIndexRaw = typeof req.query.fileIndex === "string" ? req.query.fileIndex : "";
+  const fileIndex = fileIndexRaw !== "" && /^\d+$/.test(fileIndexRaw) ? Number(fileIndexRaw) : null;
+
+  // Optional: pin the resume window to a FIXED byte offset for the duration of
+  // one buffering episode (see getFileStats JSDoc) instead of the live, moving
+  // read position — otherwise "bytes needed" can jump up mid-poll as the window
+  // slides forward with playback/encoding progress.
+  const resumeAnchorRaw = typeof req.query.resumeAnchorByteStart === "string" ? req.query.resumeAnchorByteStart : "";
+  const resumeAnchorByteStart = resumeAnchorRaw !== "" && /^\d+$/.test(resumeAnchorRaw) ? Number(resumeAnchorRaw) : null;
+
+  // Awaited: with the torrent on its own thread this is a round trip, not a
+  // local lookup. Without the await the reply was the pending promise itself,
+  // which serialises to `{}` — the empty stats seen in the field 2026-08-02.
+  const stats = await torrentPool.getFileStats(torrent, fileIndex, { resumeAnchorByteStart });
+
+  // Diagnostic: surface the real swarm state per poll so a cold-start download
+  // stall (0 peers / header not advancing → playback-plan blocks on the codec
+  // probe → browser timeout) is visible in the proxy log.
+  const downKbps = (stats.downloadSpeed / 1024).toFixed(0);
+  const filePct = stats.fileProgress != null ? `${(stats.fileProgress * 100).toFixed(1)}%` : "n/a";
+  const header =
+    stats.headerBytes != null
+      ? `${stats.headerDownloadedBytes}/${stats.headerBytes}B`
+      : "n/a";
+  logger.info(
+    `[stats] ${sourceKey.slice(0, 8)} peers=${stats.numPeers} down=${downKbps}KB/s file=${filePct} header=${header}`
+  );
+
+  return reply.send(stats);
+}

package/server.js

@@ -144,7 +144,9 @@ export async function startProxyServer({ host, port, transcodeAudio, ffmpegBin,
       }
       try {
         const torrent = await torrentPool.getTorrent(record.sourceType, record.source);
-        return torrentPool.getFileStats(torrent, Number.isInteger(fileIndex) ? fileIndex : null);
+        // Awaited for the same reason as the stats route: this now crosses a
+        // thread boundary and returns a promise.
+        return await torrentPool.getFileStats(torrent, Number.isInteger(fileIndex) ? fileIndex : null);
       } catch {
         return null;
       }

package/services/torrent-worker/channel.js

@@ -1,222 +1,234 @@
-/**
- * @file The transport both sides of the torrent worker share.
- *
- * One place holds the request/reply bookkeeping and the streaming rules, so the
- * worker and its main-thread client cannot drift apart on the details that
- * matter: which side transfers, which side acknowledges, and when a stream is
- * allowed to keep going.
- *
- * See `protocol.js` for why the design is what it is — every number in it came
- * from a measurement, not a preference.
- */
-
-import { Event, STREAM_HIGH_WATER_CHUNKS } from "./protocol.js";
-
-/**
- * Issue request ids that stay unique for the life of a thread.
- *
- * @returns {() => number}
- */
-export function createRequestIds() {
-  let next = 0;
-  return () => (next += 1);
-}
-
-/**
- * Wrap a port's messages into request/reply calls.
- *
- * Callers get promises; the plumbing of matching replies to requests, and of
- * turning a worker-side failure back into a rejection, lives here. `Error`
- * objects do not survive a thread boundary, so failures cross as messages and
- * are rebuilt into `Error`s on arrival — a caller sees an ordinary rejection.
- *
- * @param {import("node:worker_threads").MessagePort | import("node:worker_threads").Worker} port
- * @returns {{
- *   call: (command: string, params?: object) => Promise<unknown>,
- *   handleReply: (message: object) => boolean,
- *   rejectAll: (reason: Error) => void
- * }}
- */
-export function createCaller(port) {
-  const nextId = createRequestIds();
-  const pending = new Map();
-
-  return {
-    call(command, params = {}) {
-      return new Promise((resolve, reject) => {
-        const id = nextId();
-        pending.set(id, { resolve, reject });
-        port.postMessage({ command, id, params });
-      });
-    },
-
-    /**
-     * Feed a message in; returns true when it was a reply this caller owned.
-     */
-    handleReply(message) {
-      const entry = pending.get(message?.id);
-      if (!entry) {
-        return false;
-      }
-      if (message.type === Event.RESULT) {
-        pending.delete(message.id);
-        entry.resolve(message.result);
-        return true;
-      }
-      if (message.type === Event.ERROR) {
-        pending.delete(message.id);
-        entry.reject(new Error(message.error ?? "Torrent worker request failed."));
-        return true;
-      }
-      return false;
-    },
-
-    /**
-     * Fail everything outstanding — the worker died, or is being shut down.
-     */
-    rejectAll(reason) {
-      for (const [, entry] of pending) {
-        entry.reject(reason);
-      }
-      pending.clear();
-    }
-  };
-}
-
-/**
- * Receive a chunked body as an ordinary `ReadableStream`.
- *
- * This is the half that makes the design pay off: chunks arrive as transferred
- * buffers (no copying), and are handed on through a standard stream, so callers
- * treat it exactly like any other body. Each chunk is acknowledged as it is
- * enqueued, which is what lets the worker keep only
- * {@link STREAM_HIGH_WATER_CHUNKS} in flight.
- *
- * Cancelling the stream — a viewer navigating away, a superseded seek — sends
- * the cancel command, so the worker stops reading rather than filling a queue
- * nobody will drain.
- *
- * @param {object} params
- * @param {import("node:worker_threads").Worker} params.port
- * @param {number} params.requestId
- * @param {() => void} params.onCancel - Sends CANCEL_READ for this request.
- * @returns {{ stream: ReadableStream<Uint8Array>, push: (bytes: Uint8Array) => void, close: () => void, fail: (error: Error) => void }}
- */
-export function createReceiveStream({ port, requestId, onCancel }) {
-  let controller = null;
-  let finished = false;
-
-  const stream = new ReadableStream({
-    start(streamController) {
-      controller = streamController;
-    },
-    cancel() {
-      if (!finished) {
-        finished = true;
-        onCancel();
-      }
-    }
-  });
-
-  return {
-    stream,
-
-    push(bytes) {
-      if (finished || !controller) {
-        return;
-      }
-      controller.enqueue(bytes);
-      // Acknowledge only once the data is in the stream's own queue, so the
-      // worker's in-flight count reflects what has actually been taken up.
-      port.postMessage({ type: Event.CHUNK_ACK, id: requestId });
-    },
-
-    close() {
-      if (finished || !controller) {
-        return;
-      }
-      finished = true;
-      controller.close();
-    },
-
-    fail(error) {
-      if (finished || !controller) {
-        return;
-      }
-      finished = true;
-      controller.error(error);
-    }
-  };
-}
-
-/**
- * Send a body as chunks, pausing when too many are unacknowledged.
- *
- * The worker side of the same arrangement. `waitForCapacity` resolves when the
- * main thread has taken up enough of what was sent; without it a fast disk
- * would outrun the channel and rebuild in the message queue exactly the memory
- * the transfers were saving.
- *
- * @param {object} params
- * @param {import("node:worker_threads").MessagePort} params.port
- * @param {number} params.requestId
- * @returns {{ send: (bytes: Buffer) => Promise<void>, end: () => void, ack: () => void, cancel: () => void, isCancelled: () => boolean }}
- */
-export function createSendStream({ port, requestId }) {
-  let inFlight = 0;
-  let cancelled = false;
-  let wake = null;
-
-  const waitForCapacity = () => {
-    if (cancelled || inFlight < STREAM_HIGH_WATER_CHUNKS) {
-      return Promise.resolve();
-    }
-    return new Promise((resolve) => {
-      wake = resolve;
-    });
-  };
-
-  return {
-    async send(bytes) {
-      await waitForCapacity();
-      if (cancelled) {
-        return;
-      }
-      inFlight += 1;
-      // Transfer the underlying memory rather than copying it — the whole point
-      // of the design, and the difference between 4.8 ms and 37 ms per 10 MB.
-      port.postMessage(
-        { type: Event.CHUNK, id: requestId, bytes },
-        [bytes.buffer]
-      );
-    },
-
-    end() {
-      if (!cancelled) {
-        port.postMessage({ type: Event.READ_END, id: requestId });
-      }
-    },
-
-    ack() {
-      inFlight = Math.max(0, inFlight - 1);
-      if (wake) {
-        const resume = wake;
-        wake = null;
-        resume();
-      }
-    },
-
-    cancel() {
-      cancelled = true;
-      if (wake) {
-        const resume = wake;
-        wake = null;
-        resume();
-      }
-    },
-
-    isCancelled() {
-      return cancelled;
-    }
-  };
-}
+/**
+ * @file The transport both sides of the torrent worker share.
+ *
+ * One place holds the request/reply bookkeeping and the streaming rules, so the
+ * worker and its main-thread client cannot drift apart on the details that
+ * matter: which side transfers, which side acknowledges, and when a stream is
+ * allowed to keep going.
+ *
+ * See `protocol.js` for why the design is what it is — every number in it came
+ * from a measurement, not a preference.
+ */
+
+import { Event, STREAM_HIGH_WATER_CHUNKS } from "./protocol.js";
+
+/**
+ * Issue request ids that stay unique for the life of a thread.
+ *
+ * @returns {() => number}
+ */
+export function createRequestIds() {
+  let next = 0;
+  return () => (next += 1);
+}
+
+/**
+ * Wrap a port's messages into request/reply calls.
+ *
+ * Callers get promises; the plumbing of matching replies to requests, and of
+ * turning a worker-side failure back into a rejection, lives here. `Error`
+ * objects do not survive a thread boundary, so failures cross as messages and
+ * are rebuilt into `Error`s on arrival — a caller sees an ordinary rejection.
+ *
+ * @param {import("node:worker_threads").MessagePort | import("node:worker_threads").Worker} port
+ * @returns {{
+ *   call: (command: string, params?: object) => Promise<unknown>,
+ *   handleReply: (message: object) => boolean,
+ *   rejectAll: (reason: Error) => void
+ * }}
+ */
+export function createCaller(port) {
+  const nextId = createRequestIds();
+  const pending = new Map();
+
+  return {
+    call(command, params = {}) {
+      return new Promise((resolve, reject) => {
+        const id = nextId();
+        pending.set(id, { resolve, reject });
+        port.postMessage({ command, id, params });
+      });
+    },
+
+    /**
+     * Feed a message in; returns true when it was a reply this caller owned.
+     */
+    handleReply(message) {
+      const entry = pending.get(message?.id);
+      if (!entry) {
+        return false;
+      }
+      if (message.type === Event.RESULT) {
+        pending.delete(message.id);
+        entry.resolve(message.result);
+        return true;
+      }
+      if (message.type === Event.ERROR) {
+        pending.delete(message.id);
+        entry.reject(new Error(message.error ?? "Torrent worker request failed."));
+        return true;
+      }
+      return false;
+    },
+
+    /**
+     * Fail everything outstanding — the worker died, or is being shut down.
+     */
+    rejectAll(reason) {
+      for (const [, entry] of pending) {
+        entry.reject(reason);
+      }
+      pending.clear();
+    }
+  };
+}
+
+/**
+ * Receive a chunked body as an ordinary `ReadableStream`.
+ *
+ * This is the half that makes the design pay off: chunks arrive as transferred
+ * buffers (no copying), and are handed on through a standard stream, so callers
+ * treat it exactly like any other body. Each chunk is acknowledged as it is
+ * enqueued, which is what lets the worker keep only
+ * {@link STREAM_HIGH_WATER_CHUNKS} in flight.
+ *
+ * Cancelling the stream — a viewer navigating away, a superseded seek — sends
+ * the cancel command, so the worker stops reading rather than filling a queue
+ * nobody will drain.
+ *
+ * @param {object} params
+ * @param {import("node:worker_threads").Worker} params.port
+ * @param {number} params.requestId
+ * @param {() => void} params.onCancel - Sends CANCEL_READ for this request.
+ * @returns {{ stream: ReadableStream<Uint8Array>, push: (bytes: Uint8Array) => void, close: () => void, fail: (error: Error) => void }}
+ */
+export function createReceiveStream({ port, requestId, onCancel }) {
+  let controller = null;
+  let finished = false;
+
+  const stream = new ReadableStream({
+    start(streamController) {
+      controller = streamController;
+    },
+    cancel() {
+      if (!finished) {
+        finished = true;
+        onCancel();
+      }
+    }
+  });
+
+  return {
+    stream,
+
+    push(bytes) {
+      if (finished || !controller) {
+        return;
+      }
+      controller.enqueue(bytes);
+      // Acknowledge only once the data is in the stream's own queue, so the
+      // worker's in-flight count reflects what has actually been taken up.
+      port.postMessage({ type: Event.CHUNK_ACK, id: requestId });
+    },
+
+    close() {
+      if (finished || !controller) {
+        return;
+      }
+      finished = true;
+      controller.close();
+    },
+
+    fail(error) {
+      if (finished || !controller) {
+        return;
+      }
+      finished = true;
+      controller.error(error);
+    }
+  };
+}
+
+/**
+ * Send a body as chunks, pausing when too many are unacknowledged.
+ *
+ * The worker side of the same arrangement. `waitForCapacity` resolves when the
+ * main thread has taken up enough of what was sent; without it a fast disk
+ * would outrun the channel and rebuild in the message queue exactly the memory
+ * the transfers were saving.
+ *
+ * @param {object} params
+ * @param {import("node:worker_threads").MessagePort} params.port
+ * @param {number} params.requestId
+ * @returns {{ send: (bytes: Buffer) => Promise<void>, end: () => void, ack: () => void, cancel: () => void, isCancelled: () => boolean }}
+ */
+export function createSendStream({ port, requestId }) {
+  let inFlight = 0;
+  let cancelled = false;
+  let wake = null;
+
+  const waitForCapacity = () => {
+    if (cancelled || inFlight < STREAM_HIGH_WATER_CHUNKS) {
+      return Promise.resolve();
+    }
+    return new Promise((resolve) => {
+      wake = resolve;
+    });
+  };
+
+  return {
+    async send(bytes) {
+      await waitForCapacity();
+      if (cancelled) {
+        return;
+      }
+      inFlight += 1;
+      // Transfer the underlying memory rather than copying it — the whole point
+      // of the design, and the difference between 4.8 ms and 37 ms per 10 MB.
+      //
+      // But only memory this chunk owns OUTRIGHT may be transferred. Node hands
+      // out small buffers from a shared 8 KB pool: several unrelated buffers sit
+      // in one region, each viewing its own slice (verified: a 1 KB buffer
+      // reports an 8192-byte region at offset 8). Transferring that region
+      // detaches it from every other buffer living there — which is what broke
+      // reads in the field 2026-08-02: headers were produced in 325 ms and the
+      // body never arrived, because chunks from the network are small enough to
+      // be pooled while local disk reads, which is all the local test exercised,
+      // are not.
+      const ownsRegion = bytes.byteOffset === 0 && bytes.byteLength === bytes.buffer.byteLength;
+      const payload = ownsRegion ? bytes : new Uint8Array(bytes);
+      port.postMessage(
+        { type: Event.CHUNK, id: requestId, bytes: payload },
+        [payload.buffer]
+      );
+    },
+
+    end() {
+      if (!cancelled) {
+        port.postMessage({ type: Event.READ_END, id: requestId });
+      }
+    },
+
+    ack() {
+      inFlight = Math.max(0, inFlight - 1);
+      if (wake) {
+        const resume = wake;
+        wake = null;
+        resume();
+      }
+    },
+
+    cancel() {
+      cancelled = true;
+      if (wake) {
+        const resume = wake;
+        wake = null;
+        resume();
+      }
+    },
+
+    isCancelled() {
+      return cancelled;
+    }
+  };
+}