@torrent-tv/proxy 2.9.76 → 2.9.78

package/CHANGELOG.md

@@ -1,3 +1,16 @@
+## 2.9.78
+
+- **New**: Reads cross the thread boundary as **positions instead of bytes**. The pieces already live in a `SharedArrayBuffer`, so the torrent thread now sends an offset and a length and the main thread reads those bytes where they lie. What this removes is the copy that used to sit on the critical path — 18.84 ms per 10 MB segment on the field host, spent in the same thread that runs the torrent, at the moment a viewer is waiting for that segment. A piece is **pinned** for as long as a fragment of it is outstanding, and unpinned only once the main thread confirms it has finished reading, so eviction cannot take the memory out from under a reader; one fragment is in flight at a time, because the store guarantees only two resident pieces at its smallest budget and holding two pins while asking for a third would deadlock it. Verified against a partially downloaded 5.5 GB torrent: a range read whole matches the same range read in parts, a read spanning a piece boundary matches its two halves, and ffmpeg parses the file through this path (`matroska h264/ac3 5939 s`). The arithmetic is covered by tests, including a file that does not start on a piece boundary — the case where treating file offsets as torrent offsets returns the right number of wrong bytes.
+- **Chore**: `SharedPieceStore` gained `reside`, which brings a piece into memory and reports where it sits without the copy `get` has to make (WebTorrent keeps what `get` returns), and `findSharedStore`, which walks WebTorrent's store wrappers to reach ours rather than assuming their number or order.
+- **Known**: the copy is not gone from the system, only from the torrent thread — the main thread still copies each fragment out of the pool before handing it on, because nothing tells us when the socket has finished with those bytes, and releasing the piece earlier would risk serving whatever landed in the slot next. Removing that last copy needs the body write to report completion, which is a change to the stream route rather than to this transport.
+
+## 2.9.77
+
+- **Fix**: Anything naming a source while that source was still being added got `Unknown source` — which is false, because the source exists and is merely not ready. Adding a magnet takes as long as its metadata does, seconds to tens of seconds, and the browser polls stats and asks for a playback plan throughout that window. The worker registered the torrent only once the add had **finished**; it now registers the pending add itself, so callers wait for it. Reproduced with a magnet nobody seeds: stats, the file listing and a read all failed instantly while the add was in flight, and all three now wait. A source that was never added is still an error, and a failed add is forgotten rather than replayed to every later caller.
+- **Fix**: File claims are held per reader instead of per file. The proxy reads one file from several places at once — ffmpeg's input, the keyframe index, the codec probe, a second viewer — and claims keyed by `sourceKey:fileIndex` were therefore shared: the first reader to finish released the hold while the others were still reading, leaving the data free to be evicted under them. Each acquire now returns its own claim identity and a release names exactly that claim, so a duplicate or late release matches nothing, is logged, and harms no one. A counter would have restored the arithmetic but kept the ambiguity.
+- **Fix**: `HEAD /stream` no longer starts a read of the whole file. Fastify serves HEAD from the GET handler, so a HEAD opened a full-file read whose body Node discarded while the read itself ran on, the response never completed, and the next request on that keep-alive connection waited behind it — measured in the field as headers in 23 ms followed by a 15 s stall, which is where a 73 s transcode-session create came from. It also has to report the real size: the keyframe index asks for it with this very request and treats zero as "no index", silently falling back to an invented segment grid, so the response is written to the raw socket rather than through `reply.send()`, which substitutes `content-length: 0` for an empty payload.
+- **Chore**: `prefetchFileEdges` takes an options object at every layer, matching `TorrentPool`. The worker adapter declared positional parameters instead, so the planner's options object arrived as `headBytes` and only worked because it was passed along far enough to be destructured at the far end; anyone calling it as documented silently got the defaults.
+
 ## 2.9.76
 
 - **Fix**: A read that failed inside the torrent thread left the reader waiting forever, and a read that failed part-way looked like a file that had simply ended. Two halves of one hole, both present since the thread split: the worker sent the end-of-read marker from its `finally` even when the read had thrown, and the main thread had no handler for a read error at all — so the report was dropped as unknown. That is why the 2.9.71 defect took three releases to find: every symptom said "empty file", never "this read failed, here is why". Now the marker is sent only on success and the failure fails the caller's stream. Covered end to end by a test that hung before the fix.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@torrent-tv/proxy",
-  "version": "2.9.76",
+  "version": "2.9.78",
   "description": "Torrent proxy client that exposes webseed-like HTTP stream endpoint.",
   "license": "GPL-3.0-or-later",
   "publishConfig": {

package/routes/stream/get.js

@@ -1,116 +1,141 @@
-/**
- * @file Byte-range aware torrent file streaming endpoint.
- *
- * Accepts either a `sourceKey` (registered via POST /api/sources) or a raw
- * `sourceType` + `source` pair.  Responds with HTTP 206 for range requests
- * and HTTP 200 for full-file requests.
- */
-
-import { parseRange } from "../../utils/parse-range.js";
-
-/**
- * Resolve source parameters from the query string.
- * Prefers a registered `sourceKey`; falls back to inline `sourceType`+`source`.
- *
- * @param {import("fastify").FastifyRequest["query"]} query
- * @param {ReturnType<import("../../store/source-registry.js").createSourceRegistry>} sourceRegistry
- * @returns {{ sourceType: string, source: string }}
- */
-function getSourceParams(query, sourceRegistry) {
-  const sourceKey = typeof query.sourceKey === "string" ? query.sourceKey : "";
-  const sourceTypeFromQuery = typeof query.sourceType === "string" ? query.sourceType : "";
-  const sourceFromQuery = typeof query.source === "string" ? query.source : "";
-
-  const sourceRecord = sourceKey ? sourceRegistry.get(sourceKey) : null;
-  const sourceType = sourceRecord?.sourceType ?? sourceTypeFromQuery;
-  const source = sourceRecord?.source ?? sourceFromQuery;
-  return { sourceType, source };
-}
-
-/**
- * Stream a torrent file over HTTP with byte-range support.
- *
- * GET /stream
- *
- * @param {import("fastify").FastifyRequest} req
- * @param {import("fastify").FastifyReply} reply
- * @param {{ sourceRegistry: ReturnType<import("../../store/source-registry.js").createSourceRegistry>, torrentPool: import("../../services/torrent-pool.js").TorrentPool }} deps
- * @returns {Promise<void>}
- */
-export async function handleStreamGet(req, reply, { sourceRegistry, torrentPool }) {
-  const fileIndexRaw = typeof req.query.fileIndex === "string" ? req.query.fileIndex : "";
-  const fileIndex = Number(fileIndexRaw);
-  const { sourceType, source } = getSourceParams(req.query, sourceRegistry);
-
-  if (!sourceType || !source || !Number.isInteger(fileIndex) || fileIndex < 0) {
-    return reply
-      .code(400)
-      .send({ error: "sourceKey or sourceType+source with fileIndex are required." });
-  }
-
-  let torrent;
-  try {
-    torrent = await torrentPool.getTorrent(sourceType, source);
-  } catch (error) {
-    const message = error instanceof Error ? error.message : String(error);
-    return reply.code(500).send({ error: `Failed to load torrent source: ${message}` });
-  }
-
-  const file = torrent.files[fileIndex];
-  if (!file) {
-    return reply.code(404).send({ error: "File index was not found in torrent." });
-  }
-
-  const releaseFile = torrentPool.acquireFile(torrent, fileIndex);
-
-  const range = parseRange(req.headers.range, file.length);
-  // Prioritize the pieces at this read position so a seek (a request at a new
-  // byte offset) downloads first instead of waiting behind the sequential
-  // backlog — this is what caused ~15-18 s stalls when seeking into an
-  // undownloaded region.
-  torrentPool.prioritizeByteRange(torrent, fileIndex, range ? range.start : 0);
-  reply.header("Accept-Ranges", "bytes");
-  reply.header("Content-Type", "application/octet-stream");
-  reply.header("Content-Disposition", `inline; filename*=UTF-8''${encodeURIComponent(file.name)}`);
-
-  if (!range) {
-    reply.header("Content-Length", String(file.length));
-    const stream = file.createReadStream();
-    bindRelease(stream, reply, releaseFile);
-    return reply.send(stream);
-  }
-
-  const contentLength = range.end - range.start + 1;
-  reply.code(206);
-  reply.header("Content-Length", String(contentLength));
-  reply.header("Content-Range", `bytes ${range.start}-${range.end}/${file.length}`);
-  const stream = file.createReadStream({ start: range.start, end: range.end });
-  bindRelease(stream, reply, releaseFile);
-  return reply.send(stream);
-}
-
-/**
- * Attach event listeners that release the file reference exactly once when
- * the stream or the underlying HTTP connection closes.
- *
- * @param {import("node:stream").Readable} stream
- * @param {import("fastify").FastifyReply} reply
- * @param {() => void} release
- * @returns {void}
- */
-function bindRelease(stream, reply, release) {
-  let released = false;
-  const releaseOnce = () => {
-    if (released) {
-      return;
-    }
-    released = true;
-    release();
-  };
-
-  stream.on("close", releaseOnce);
-  stream.on("end", releaseOnce);
-  stream.on("error", releaseOnce);
-  reply.raw.once("close", releaseOnce);
-  reply.raw.once("finish", releaseOnce);
-}
+/**
+ * @file Byte-range aware torrent file streaming endpoint.
+ *
+ * Accepts either a `sourceKey` (registered via POST /api/sources) or a raw
+ * `sourceType` + `source` pair.  Responds with HTTP 206 for range requests
+ * and HTTP 200 for full-file requests.
+ */
+
+import { parseRange } from "../../utils/parse-range.js";
+
+/**
+ * Resolve source parameters from the query string.
+ * Prefers a registered `sourceKey`; falls back to inline `sourceType`+`source`.
+ *
+ * @param {import("fastify").FastifyRequest["query"]} query
+ * @param {ReturnType<import("../../store/source-registry.js").createSourceRegistry>} sourceRegistry
+ * @returns {{ sourceType: string, source: string }}
+ */
+function getSourceParams(query, sourceRegistry) {
+  const sourceKey = typeof query.sourceKey === "string" ? query.sourceKey : "";
+  const sourceTypeFromQuery = typeof query.sourceType === "string" ? query.sourceType : "";
+  const sourceFromQuery = typeof query.source === "string" ? query.source : "";
+
+  const sourceRecord = sourceKey ? sourceRegistry.get(sourceKey) : null;
+  const sourceType = sourceRecord?.sourceType ?? sourceTypeFromQuery;
+  const source = sourceRecord?.source ?? sourceFromQuery;
+  return { sourceType, source };
+}
+
+/**
+ * Stream a torrent file over HTTP with byte-range support.
+ *
+ * GET /stream
+ *
+ * @param {import("fastify").FastifyRequest} req
+ * @param {import("fastify").FastifyReply} reply
+ * @param {{ sourceRegistry: ReturnType<import("../../store/source-registry.js").createSourceRegistry>, torrentPool: import("../../services/torrent-pool.js").TorrentPool }} deps
+ * @returns {Promise<void>}
+ */
+export async function handleStreamGet(req, reply, { sourceRegistry, torrentPool }) {
+  const fileIndexRaw = typeof req.query.fileIndex === "string" ? req.query.fileIndex : "";
+  const fileIndex = Number(fileIndexRaw);
+  const { sourceType, source } = getSourceParams(req.query, sourceRegistry);
+
+  if (!sourceType || !source || !Number.isInteger(fileIndex) || fileIndex < 0) {
+    return reply
+      .code(400)
+      .send({ error: "sourceKey or sourceType+source with fileIndex are required." });
+  }
+
+  let torrent;
+  try {
+    torrent = await torrentPool.getTorrent(sourceType, source);
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    return reply.code(500).send({ error: `Failed to load torrent source: ${message}` });
+  }
+
+  const file = torrent.files[fileIndex];
+  if (!file) {
+    return reply.code(404).send({ error: "File index was not found in torrent." });
+  }
+
+  // HEAD asks what a GET would return, not for the bytes. Fastify serves HEAD
+  // from this same handler, which used to mean a HEAD started a read of the
+  // WHOLE file: the body was discarded by Node, but the read ran on, the
+  // response never finished, and the next request on that keep-alive connection
+  // waited behind it. Measured on the field host: the keyframe-index HEAD
+  // returned headers in 23 ms and then held the connection until its 15 s
+  // timeout, which is where the 73 s transcode-session create went.
+  if (req.method === "HEAD") {
+    // Written to the raw response on purpose. Answering through `reply.send()`
+    // with no payload makes Fastify set `content-length: 0`, which is worse
+    // than useless here: the keyframe index asks for the file size with this
+    // very request and treats 0 as "no index available", silently falling back
+    // to an invented segment grid. Hijacking leaves the response to us, and
+    // Node omits the body for HEAD by itself.
+    reply.hijack();
+    reply.raw.writeHead(200, {
+      "Accept-Ranges": "bytes",
+      "Content-Type": "application/octet-stream",
+      "Content-Length": String(file.length),
+      "Content-Disposition": `inline; filename*=UTF-8''${encodeURIComponent(file.name)}`
+    });
+    reply.raw.end();
+    return;
+  }
+
+  const releaseFile = torrentPool.acquireFile(torrent, fileIndex);
+
+  const range = parseRange(req.headers.range, file.length);
+  // Prioritize the pieces at this read position so a seek (a request at a new
+  // byte offset) downloads first instead of waiting behind the sequential
+  // backlog — this is what caused ~15-18 s stalls when seeking into an
+  // undownloaded region.
+  torrentPool.prioritizeByteRange(torrent, fileIndex, range ? range.start : 0);
+  reply.header("Accept-Ranges", "bytes");
+  reply.header("Content-Type", "application/octet-stream");
+  reply.header("Content-Disposition", `inline; filename*=UTF-8''${encodeURIComponent(file.name)}`);
+
+  if (!range) {
+    reply.header("Content-Length", String(file.length));
+    const stream = file.createReadStream();
+    bindRelease(stream, reply, releaseFile);
+    return reply.send(stream);
+  }
+
+  const contentLength = range.end - range.start + 1;
+  reply.code(206);
+  reply.header("Content-Length", String(contentLength));
+  reply.header("Content-Range", `bytes ${range.start}-${range.end}/${file.length}`);
+  const stream = file.createReadStream({ start: range.start, end: range.end });
+  bindRelease(stream, reply, releaseFile);
+  return reply.send(stream);
+}
+
+/**
+ * Attach event listeners that release the file reference exactly once when
+ * the stream or the underlying HTTP connection closes.
+ *
+ * @param {import("node:stream").Readable} stream
+ * @param {import("fastify").FastifyReply} reply
+ * @param {() => void} release
+ * @returns {void}
+ */
+function bindRelease(stream, reply, release) {
+  let released = false;
+  const releaseOnce = () => {
+    if (released) {
+      return;
+    }
+    released = true;
+    release();
+  };
+
+  stream.on("close", releaseOnce);
+  stream.on("end", releaseOnce);
+  stream.on("error", releaseOnce);
+  reply.raw.once("close", releaseOnce);
+  reply.raw.once("finish", releaseOnce);
+}