@torrent-tv/proxy 2.9.78 → 2.9.80

package/CHANGELOG.md

@@ -1,3 +1,15 @@
+## 2.9.80
+
+- **Fix**: A seek backward could hang forever. `prioritizeByteRange` demotes the pieces behind the playhead with `deselect`, which removes them from the download set — and `critical`, which runs right after, only flags pieces that are already selected, so it never puts them back. A seek forward followed by a seek backward therefore left the target pieces wanted by nobody: the encoder waited on data the torrent had been told to stop fetching, while the swarm ran at full speed on pieces nobody needed. The read position is now re-selected whenever it moves back behind what an earlier seek deselected, tracked per file because WebTorrent does not report its own selection back.
+- **Fix**: Two pieces could be given the same slot in the shared store. Eviction chose a victim, then **awaited** the spill write before removing it from the books, so a second claim arriving in that window chose the same victim and received the same slot — after which two pieces overwrote each other, both failed their hash, and the torrent downloaded them again indefinitely. From outside this looked exactly like a seek that never completes while the download runs at full speed. The victim is now claimed and unbooked in one uninterrupted step, and a reader that arrives mid-spill waits for the write instead of being told the piece is missing.
+- **Fix**: A burst of concurrent `put`s could fail with "every resident piece is pinned" when nothing was pinned at all. Slots are claimed before the piece is copied into them, and pieces arrive from many peers at once, so the store saw an empty eviction list while its slots were already spoken for. Slots handed out but not yet recorded are now counted, and a claim that finds nothing waits for that work to land rather than declaring the store exhausted.
+- **New**: Two figures the last field failure could not be diagnosed without. The store now reports `pinned=` alongside its other counters, so a leaked pin is visible while it is still harmless instead of only when eviction has nothing left to take; and a read position that jumps — a seek — is logged with its offset and percentage through the file, so it can be seen whether a seek reached the torrent at all.
+
+## 2.9.79
+
+- **New**: The last copy is gone from the read path. `/stream` now writes the response straight out of the torrent's shared memory and releases each piece only when the socket write reports completion — which is the one moment that is safe, because a piece released earlier can be evicted and its slot refilled while those exact bytes are still on their way out. Both halves of that were verified before being relied on: a socket accepts a view into a `SharedArrayBuffer`, and overwriting the pool from inside the write callback leaves the client's copy intact while overwriting it before the callback corrupts it silently. Measured on the same host, 24 MB of already-downloaded data read in 2 MB ranges: **298 ms against 1008 ms**, 675 Mbit/s against 200, and far steadier (265-308 ms against 641-1338). Callers that keep what they are given — the subtitle route, anything using the plain stream — still get a copy and are unaffected; a source with no shared pool falls back to the previous path.
+- **Chore**: Reading the response body by hand is what makes the release point observable, so the route writes and ends the response itself rather than handing Fastify a stream. A client that disconnects mid-response cancels the read, so pieces stop being fetched for a viewer who has gone.
+
 ## 2.9.78
 
 - **New**: Reads cross the thread boundary as **positions instead of bytes**. The pieces already live in a `SharedArrayBuffer`, so the torrent thread now sends an offset and a length and the main thread reads those bytes where they lie. What this removes is the copy that used to sit on the critical path — 18.84 ms per 10 MB segment on the field host, spent in the same thread that runs the torrent, at the moment a viewer is waiting for that segment. A piece is **pinned** for as long as a fragment of it is outstanding, and unpinned only once the main thread confirms it has finished reading, so eviction cannot take the memory out from under a reader; one fragment is in flight at a time, because the store guarantees only two resident pieces at its smallest budget and holding two pins while asking for a third would deadlock it. Verified against a partially downloaded 5.5 GB torrent: a range read whole matches the same range read in parts, a read spanning a piece boundary matches its two halves, and ffmpeg parses the file through this path (`matroska h264/ac3 5939 s`). The arithmetic is covered by tests, including a file that does not start on a piece boundary — the case where treating file offsets as torrent offsets returns the right number of wrong bytes.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@torrent-tv/proxy",
-  "version": "2.9.78",
+  "version": "2.9.80",
   "description": "Torrent proxy client that exposes webseed-like HTTP stream endpoint.",
   "license": "GPL-3.0-or-later",
   "publishConfig": {

package/routes/stream/get.js

@@ -1,141 +1,192 @@
-/**
- * @file Byte-range aware torrent file streaming endpoint.
- *
- * Accepts either a `sourceKey` (registered via POST /api/sources) or a raw
- * `sourceType` + `source` pair.  Responds with HTTP 206 for range requests
- * and HTTP 200 for full-file requests.
- */
-
-import { parseRange } from "../../utils/parse-range.js";
-
-/**
- * Resolve source parameters from the query string.
- * Prefers a registered `sourceKey`; falls back to inline `sourceType`+`source`.
- *
- * @param {import("fastify").FastifyRequest["query"]} query
- * @param {ReturnType<import("../../store/source-registry.js").createSourceRegistry>} sourceRegistry
- * @returns {{ sourceType: string, source: string }}
- */
-function getSourceParams(query, sourceRegistry) {
-  const sourceKey = typeof query.sourceKey === "string" ? query.sourceKey : "";
-  const sourceTypeFromQuery = typeof query.sourceType === "string" ? query.sourceType : "";
-  const sourceFromQuery = typeof query.source === "string" ? query.source : "";
-
-  const sourceRecord = sourceKey ? sourceRegistry.get(sourceKey) : null;
-  const sourceType = sourceRecord?.sourceType ?? sourceTypeFromQuery;
-  const source = sourceRecord?.source ?? sourceFromQuery;
-  return { sourceType, source };
-}
-
-/**
- * Stream a torrent file over HTTP with byte-range support.
- *
- * GET /stream
- *
- * @param {import("fastify").FastifyRequest} req
- * @param {import("fastify").FastifyReply} reply
- * @param {{ sourceRegistry: ReturnType<import("../../store/source-registry.js").createSourceRegistry>, torrentPool: import("../../services/torrent-pool.js").TorrentPool }} deps
- * @returns {Promise<void>}
- */
-export async function handleStreamGet(req, reply, { sourceRegistry, torrentPool }) {
-  const fileIndexRaw = typeof req.query.fileIndex === "string" ? req.query.fileIndex : "";
-  const fileIndex = Number(fileIndexRaw);
-  const { sourceType, source } = getSourceParams(req.query, sourceRegistry);
-
-  if (!sourceType || !source || !Number.isInteger(fileIndex) || fileIndex < 0) {
-    return reply
-      .code(400)
-      .send({ error: "sourceKey or sourceType+source with fileIndex are required." });
-  }
-
-  let torrent;
-  try {
-    torrent = await torrentPool.getTorrent(sourceType, source);
-  } catch (error) {
-    const message = error instanceof Error ? error.message : String(error);
-    return reply.code(500).send({ error: `Failed to load torrent source: ${message}` });
-  }
-
-  const file = torrent.files[fileIndex];
-  if (!file) {
-    return reply.code(404).send({ error: "File index was not found in torrent." });
-  }
-
-  // HEAD asks what a GET would return, not for the bytes. Fastify serves HEAD
-  // from this same handler, which used to mean a HEAD started a read of the
-  // WHOLE file: the body was discarded by Node, but the read ran on, the
-  // response never finished, and the next request on that keep-alive connection
-  // waited behind it. Measured on the field host: the keyframe-index HEAD
-  // returned headers in 23 ms and then held the connection until its 15 s
-  // timeout, which is where the 73 s transcode-session create went.
-  if (req.method === "HEAD") {
-    // Written to the raw response on purpose. Answering through `reply.send()`
-    // with no payload makes Fastify set `content-length: 0`, which is worse
-    // than useless here: the keyframe index asks for the file size with this
-    // very request and treats 0 as "no index available", silently falling back
-    // to an invented segment grid. Hijacking leaves the response to us, and
-    // Node omits the body for HEAD by itself.
-    reply.hijack();
-    reply.raw.writeHead(200, {
-      "Accept-Ranges": "bytes",
-      "Content-Type": "application/octet-stream",
-      "Content-Length": String(file.length),
-      "Content-Disposition": `inline; filename*=UTF-8''${encodeURIComponent(file.name)}`
-    });
-    reply.raw.end();
-    return;
-  }
-
-  const releaseFile = torrentPool.acquireFile(torrent, fileIndex);
-
-  const range = parseRange(req.headers.range, file.length);
-  // Prioritize the pieces at this read position so a seek (a request at a new
-  // byte offset) downloads first instead of waiting behind the sequential
-  // backlog — this is what caused ~15-18 s stalls when seeking into an
-  // undownloaded region.
-  torrentPool.prioritizeByteRange(torrent, fileIndex, range ? range.start : 0);
-  reply.header("Accept-Ranges", "bytes");
-  reply.header("Content-Type", "application/octet-stream");
-  reply.header("Content-Disposition", `inline; filename*=UTF-8''${encodeURIComponent(file.name)}`);
-
-  if (!range) {
-    reply.header("Content-Length", String(file.length));
-    const stream = file.createReadStream();
-    bindRelease(stream, reply, releaseFile);
-    return reply.send(stream);
-  }
-
-  const contentLength = range.end - range.start + 1;
-  reply.code(206);
-  reply.header("Content-Length", String(contentLength));
-  reply.header("Content-Range", `bytes ${range.start}-${range.end}/${file.length}`);
-  const stream = file.createReadStream({ start: range.start, end: range.end });
-  bindRelease(stream, reply, releaseFile);
-  return reply.send(stream);
-}
-
-/**
- * Attach event listeners that release the file reference exactly once when
- * the stream or the underlying HTTP connection closes.
- *
- * @param {import("node:stream").Readable} stream
- * @param {import("fastify").FastifyReply} reply
- * @param {() => void} release
- * @returns {void}
- */
-function bindRelease(stream, reply, release) {
-  let released = false;
-  const releaseOnce = () => {
-    if (released) {
-      return;
-    }
-    released = true;
-    release();
-  };
-
-  stream.on("close", releaseOnce);
-  stream.on("end", releaseOnce);
-  stream.on("error", releaseOnce);
-  reply.raw.once("close", releaseOnce);
-  reply.raw.once("finish", releaseOnce);
-}
+/**
+ * @file Byte-range aware torrent file streaming endpoint.
+ *
+ * Accepts either a `sourceKey` (registered via POST /api/sources) or a raw
+ * `sourceType` + `source` pair.  Responds with HTTP 206 for range requests
+ * and HTTP 200 for full-file requests.
+ */
+
+import { parseRange } from "../../utils/parse-range.js";
+
+/**
+ * Resolve source parameters from the query string.
+ * Prefers a registered `sourceKey`; falls back to inline `sourceType`+`source`.
+ *
+ * @param {import("fastify").FastifyRequest["query"]} query
+ * @param {ReturnType<import("../../store/source-registry.js").createSourceRegistry>} sourceRegistry
+ * @returns {{ sourceType: string, source: string }}
+ */
+function getSourceParams(query, sourceRegistry) {
+  const sourceKey = typeof query.sourceKey === "string" ? query.sourceKey : "";
+  const sourceTypeFromQuery = typeof query.sourceType === "string" ? query.sourceType : "";
+  const sourceFromQuery = typeof query.source === "string" ? query.source : "";
+
+  const sourceRecord = sourceKey ? sourceRegistry.get(sourceKey) : null;
+  const sourceType = sourceRecord?.sourceType ?? sourceTypeFromQuery;
+  const source = sourceRecord?.source ?? sourceFromQuery;
+  return { sourceType, source };
+}
+
+/**
+ * Stream a torrent file over HTTP with byte-range support.
+ *
+ * GET /stream
+ *
+ * @param {import("fastify").FastifyRequest} req
+ * @param {import("fastify").FastifyReply} reply
+ * @param {{ sourceRegistry: ReturnType<import("../../store/source-registry.js").createSourceRegistry>, torrentPool: import("../../services/torrent-pool.js").TorrentPool }} deps
+ * @returns {Promise<void>}
+ */
+export async function handleStreamGet(req, reply, { sourceRegistry, torrentPool }) {
+  const fileIndexRaw = typeof req.query.fileIndex === "string" ? req.query.fileIndex : "";
+  const fileIndex = Number(fileIndexRaw);
+  const { sourceType, source } = getSourceParams(req.query, sourceRegistry);
+
+  if (!sourceType || !source || !Number.isInteger(fileIndex) || fileIndex < 0) {
+    return reply
+      .code(400)
+      .send({ error: "sourceKey or sourceType+source with fileIndex are required." });
+  }
+
+  let torrent;
+  try {
+    torrent = await torrentPool.getTorrent(sourceType, source);
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    return reply.code(500).send({ error: `Failed to load torrent source: ${message}` });
+  }
+
+  const file = torrent.files[fileIndex];
+  if (!file) {
+    return reply.code(404).send({ error: "File index was not found in torrent." });
+  }
+
+  // HEAD asks what a GET would return, not for the bytes. Fastify serves HEAD
+  // from this same handler, which used to mean a HEAD started a read of the
+  // WHOLE file: the body was discarded by Node, but the read ran on, the
+  // response never finished, and the next request on that keep-alive connection
+  // waited behind it. Measured on the field host: the keyframe-index HEAD
+  // returned headers in 23 ms and then held the connection until its 15 s
+  // timeout, which is where the 73 s transcode-session create went.
+  if (req.method === "HEAD") {
+    // Written to the raw response on purpose. Answering through `reply.send()`
+    // with no payload makes Fastify set `content-length: 0`, which is worse
+    // than useless here: the keyframe index asks for the file size with this
+    // very request and treats 0 as "no index available", silently falling back
+    // to an invented segment grid. Hijacking leaves the response to us, and
+    // Node omits the body for HEAD by itself.
+    reply.hijack();
+    reply.raw.writeHead(200, {
+      "Accept-Ranges": "bytes",
+      "Content-Type": "application/octet-stream",
+      "Content-Length": String(file.length),
+      "Content-Disposition": `inline; filename*=UTF-8''${encodeURIComponent(file.name)}`
+    });
+    reply.raw.end();
+    return;
+  }
+
+  const releaseFile = torrentPool.acquireFile(torrent, fileIndex);
+
+  const range = parseRange(req.headers.range, file.length);
+  // Prioritize the pieces at this read position so a seek (a request at a new
+  // byte offset) downloads first instead of waiting behind the sequential
+  // backlog — this is what caused ~15-18 s stalls when seeking into an
+  // undownloaded region.
+  torrentPool.prioritizeByteRange(torrent, fileIndex, range ? range.start : 0);
+
+  const start = range ? range.start : 0;
+  const end = range ? range.end : file.length - 1;
+  const contentLength = end - start + 1;
+
+  // Written straight out of the torrent's shared memory when that is available:
+  // no copy on either thread, at the cost of doing the writing by hand, because
+  // only the write callback tells us when a piece may be released. Falls back to
+  // the ordinary stream for sources without a shared pool.
+  const fragments = typeof file.createFragmentReader === "function"
+    ? file.createFragmentReader({ start, end })
+    : null;
+
+  if (fragments) {
+    reply.hijack();
+    reply.raw.writeHead(range ? 206 : 200, {
+      "Accept-Ranges": "bytes",
+      "Content-Type": "application/octet-stream",
+      "Content-Length": String(contentLength),
+      "Content-Disposition": `inline; filename*=UTF-8''${encodeURIComponent(file.name)}`,
+      ...(range ? { "Content-Range": `bytes ${start}-${end}/${file.length}` } : {})
+    });
+
+    // A client that goes away mid-response must stop the read, or pieces keep
+    // being fetched for nobody.
+    reply.raw.once("close", () => fragments.cancel());
+
+    try {
+      for await (const fragment of fragments) {
+        if (reply.raw.writableEnded || reply.raw.destroyed) {
+          fragment.release();
+          break;
+        }
+        await new Promise((resolve, reject) => {
+          reply.raw.write(fragment.bytes, (error) => (error ? reject(error) : resolve()));
+        });
+        // Only now are these bytes gone: the piece can be unpinned, and the
+        // slot it occupies reused. Releasing before this point corrupts the
+        // response silently.
+        fragment.release();
+      }
+      reply.raw.end();
+    } catch {
+      // The body is already committed by its headers, so there is nothing
+      // useful to send instead — drop the connection and let the client retry.
+      reply.raw.destroy();
+    } finally {
+      releaseFile();
+    }
+    return;
+  }
+
+  reply.header("Accept-Ranges", "bytes");
+  reply.header("Content-Type", "application/octet-stream");
+  reply.header("Content-Disposition", `inline; filename*=UTF-8''${encodeURIComponent(file.name)}`);
+
+  if (!range) {
+    reply.header("Content-Length", String(file.length));
+    const stream = file.createReadStream();
+    bindRelease(stream, reply, releaseFile);
+    return reply.send(stream);
+  }
+
+  reply.code(206);
+  reply.header("Content-Length", String(contentLength));
+  reply.header("Content-Range", `bytes ${start}-${end}/${file.length}`);
+  const stream = file.createReadStream({ start, end });
+  bindRelease(stream, reply, releaseFile);
+  return reply.send(stream);
+}
+
+/**
+ * Attach event listeners that release the file reference exactly once when
+ * the stream or the underlying HTTP connection closes.
+ *
+ * @param {import("node:stream").Readable} stream
+ * @param {import("fastify").FastifyReply} reply
+ * @param {() => void} release
+ * @returns {void}
+ */
+function bindRelease(stream, reply, release) {
+  let released = false;
+  const releaseOnce = () => {
+    if (released) {
+      return;
+    }
+    released = true;
+    release();
+  };
+
+  stream.on("close", releaseOnce);
+  stream.on("end", releaseOnce);
+  stream.on("error", releaseOnce);
+  reply.raw.once("close", releaseOnce);
+  reply.raw.once("finish", releaseOnce);
+}

package/services/data-channel-handler.js

@@ -79,7 +79,37 @@
 import { performance } from "node:perf_hooks";
 import { eventLoopDelay, resetEventLoopDelay } from "../utils/perf.js";
 
+/**
+ * Build one body frame: `[flags(1)][idLen(1)][requestId][payload]`.
+ *
+ * One allocation and one copy. The previous version made two of each — a copy
+ * of the chunk into a `Buffer`, then a `concat` that copied it again into the
+ * frame — which measured 75.9 ms per 13 MB segment on the field host against
+ * 40.0 ms this way, and allocated ~600 extra buffers over a segment's 208
+ * chunks. One copy is the floor: chunks arrive from a web stream that allocates
+ * them itself, so there is no buffer of ours to read them into.
+ *
+ * @param {Buffer} idBytes - The request id, already encoded.
+ * @param {Uint8Array | null} bytes - Payload, or nothing for the done frame.
+ * @param {boolean} done
+ * @returns {Buffer}
+ */
+export function encodeFrame(idBytes, bytes, done) {
+  const payloadLength = bytes?.length ?? 0;
+  const frame = Buffer.allocUnsafe(2 + idBytes.length + payloadLength);
+  frame[0] = done ? 1 : 0;
+  frame[1] = idBytes.length;
+  idBytes.copy(frame, 2);
+  if (payloadLength > 0) {
+    frame.set(bytes, 2 + idBytes.length);
+  }
+  return frame;
+}
+
 export function createDataChannelHandler({ proxyPort, onLog }) {
+  /** Request id → its ASCII bytes; see {@link requestIdBytes}. */
+  const requestIdCache = new Map();
+
   /**
    * @param {string} message
    * @returns {void}
@@ -389,16 +419,32 @@ export function createDataChannelHandler({ proxyPort, onLog }) {
    * @param {boolean}                done
    * @returns {void}
    */
+  /**
+   * The request id as bytes, prepared once per request rather than per chunk.
+   *
+   * A segment is a couple of hundred chunks, and each one was re-encoding the
+   * same 32-character string. The map is bounded because request ids are
+   * short-lived and unbounded in number — dropping the whole cache when it
+   * grows costs one re-encode per live request and cannot leak.
+   *
+   * @param {string} requestId
+   * @returns {Buffer}
+   */
+  function requestIdBytes(requestId) {
+    let bytes = requestIdCache.get(requestId);
+    if (!bytes) {
+      if (requestIdCache.size > 64) {
+        requestIdCache.clear();
+      }
+      bytes = Buffer.from(requestId, "ascii");
+      requestIdCache.set(requestId, bytes);
+    }
+    return bytes;
+  }
+
   function sendChunk(channel, requestId, bytes, done) {
     try {
-      const idBuf = Buffer.from(requestId, "ascii");
-      const header = Buffer.allocUnsafe(2 + idBuf.length);
-      header[0] = done ? 1 : 0;
-      header[1] = idBuf.length;
-      idBuf.copy(header, 2);
-      const frame =
-        bytes && bytes.length > 0 ? Buffer.concat([header, Buffer.from(bytes)]) : header;
-      channel.sendMessageBinary(frame);
+      channel.sendMessageBinary(encodeFrame(requestIdBytes(requestId), bytes, done));
     } catch {
       // Channel closed between check and send — safe to ignore.
     }

package/services/piece-store/piece-lru.js

@@ -49,6 +49,18 @@ export class PieceLru {
     return this.#capacity;
   }
 
+  /**
+   * How many pieces are currently held by a reader.
+   *
+   * Reported rather than merely tracked: a pin that is never released is
+   * invisible until eviction has nothing left to take, and by then the store is
+   * already failing. A count that keeps climbing between reports names the leak
+   * long before that.
+   */
+  get pinnedCount() {
+    return this.#pins.size;
+  }
+
   /**
    * Mark a piece as resident, or as used again if it already was.
    *

package/services/piece-store/shared-piece-store.js

@@ -131,6 +131,28 @@ export class SharedPieceStore {
   #slotOf = new Map();
   /** Slot numbers not currently holding a piece. */
   #freeSlots = [];
+  /**
+   * Pieces being written out to disk right now: index → that write.
+   *
+   * Such a piece is in neither place — its slot has already been given away,
+   * and the disk copy is not finished. A reader arriving in that window must
+   * wait for the write instead of concluding the piece is gone.
+   *
+   * @type {Map<number, Promise<void>>}
+   */
+  #evicting = new Map();
+  /**
+   * Slots handed out but not yet recorded against a piece.
+   *
+   * A slot is claimed before the piece is copied into it, so between those two
+   * moments the slot belongs to nobody the books know about. Without counting
+   * them, a burst of concurrent puts — which is the normal case, pieces arrive
+   * from many peers at once — sees an empty eviction list and concludes the
+   * store is exhausted, when in fact it is merely mid-flight.
+   */
+  #outstandingSlots = 0;
+  /** Resolvers waiting for a slot to become claimable. @type {(() => void)[]} */
+  #waiters = [];
   #lru;
   #disk;
   /** Slots backed by memory right now; grows towards {@link capacity}. */
@@ -206,6 +228,7 @@ export class SharedPieceStore {
       name: this.#name,
       resident: this.#slotOf.size,
       capacity: this.#capacity,
+      pinned: this.#lru.pinnedCount,
       spilled: this.#disk.size,
       ...this.#counters
     };
@@ -293,8 +316,65 @@ export class SharedPieceStore {
    * @returns {Promise<number>} Slot number.
    */
   async #claimSlot() {
+    for (;;) {
+      const slot = await this.#claimSlotOnce();
+      if (slot !== null) {
+        return slot;
+      }
+      // Nothing claimable this instant, but work is in flight that will make a
+      // slot claimable: a spill finishing, or a piece being written into a slot
+      // already handed out. Wait for either and look again, rather than failing
+      // while the store is in the middle of making room.
+      await new Promise((resolve) => {
+        this.#waiters.push(resolve);
+        for (const spill of this.#evicting.values()) {
+          void spill.then(() => this.#wake(), () => this.#wake());
+        }
+      });
+    }
+  }
+
+  /**
+   * Record a piece against the slot it now occupies, and let waiters retry.
+   *
+   * @param {number} index
+   * @param {number} slot
+   * @returns {void}
+   */
+  #registerSlot(index, slot) {
+    this.#slotOf.set(index, slot);
+    this.#lru.touch(index);
+    this.#outstandingSlots -= 1;
+    this.#wake();
+  }
+
+  /**
+   * Release everyone waiting for a slot; each rechecks for itself.
+   *
+   * @returns {void}
+   */
+  #wake() {
+    const waiting = this.#waiters;
+    this.#waiters = [];
+    for (const resolve of waiting) {
+      resolve();
+    }
+  }
+
+  /**
+   * One attempt at a slot: a number, or `null` when the caller should wait for
+   * an in-flight spill and try again.
+   *
+   * @returns {Promise<number | null>}
+   */
+  async #claimSlotOnce() {
+    // Every slot handed out below is counted BEFORE this function can suspend.
+    // Counting it after an `await` would leave concurrent callers — which is
+    // how pieces actually arrive — seeing an idle store and declaring it
+    // exhausted while its slots are already spoken for.
     const free = this.#freeSlots.pop();
     if (free !== undefined) {
+      this.#outstandingSlots += 1;
       return free;
     }
 
@@ -305,11 +385,15 @@ export class SharedPieceStore {
       this.#allocatedSlots += 1;
       this.#shared.grow(this.#allocatedSlots * this.#chunkLength);
       this.#pool = Buffer.from(this.#shared);
+      this.#outstandingSlots += 1;
       return this.#allocatedSlots - 1;
     }
 
     const victim = this.#lru.evictionCandidate();
     if (victim === null) {
+      if (this.#evicting.size > 0 || this.#outstandingSlots > 0) {
+        return null;
+      }
       // Every resident piece is being read. Taking one anyway is precisely the
       // failure this store exists to make impossible.
       this.#counters.blockedByPins += 1;
@@ -317,12 +401,30 @@ export class SharedPieceStore {
     }
 
     const slot = this.#slotOf.get(victim);
-    const bytes = this.#pool.subarray(slot * this.#chunkLength, slot * this.#chunkLength + this.#lengthOf(victim));
-    await this.#disk.write(victim, bytes);
-    this.#counters.spills += 1;
 
+    // Claim the victim NOW, before the write can suspend us. Picking it and
+    // releasing it either side of an `await` lets a second claim, arriving in
+    // that gap, pick the same victim and be handed the same slot — after which
+    // two pieces write over each other, both fail their hash, and the torrent
+    // downloads them again, forever. Removing it from the books first makes the
+    // choice atomic; `#evicting` keeps readers correct in the meantime.
     this.#slotOf.delete(victim);
     this.#lru.remove(victim);
+    this.#outstandingSlots += 1;
+
+    const bytes = this.#pool.subarray(slot * this.#chunkLength, slot * this.#chunkLength + this.#lengthOf(victim));
+    const spill = this.#disk.write(victim, bytes).then(
+      () => {
+        this.#counters.spills += 1;
+        this.#evicting.delete(victim);
+      },
+      (error) => {
+        this.#evicting.delete(victim);
+        throw error;
+      }
+    );
+    this.#evicting.set(victim, spill);
+    await spill;
     return slot;
   }
 
@@ -346,8 +448,11 @@ export class SharedPieceStore {
       bytes.copy
         ? bytes.copy(this.#pool, slot * this.#chunkLength)
         : this.#pool.set(bytes, slot * this.#chunkLength);
-      this.#slotOf.set(index, slot);
-      this.#lru.touch(index);
+      if (existing === undefined) {
+        this.#registerSlot(index, slot);
+      } else {
+        this.#lru.touch(index);
+      }
       // A newer copy is in memory; whatever is on disk is stale.
       this.#disk.forget(index);
     };
@@ -391,6 +496,13 @@ export class SharedPieceStore {
         return Buffer.from(this.#pool.subarray(start, start + length));
       }
 
+      // Mid-spill: neither in memory nor yet on disk. See `reside` — reporting
+      // it missing here would tell WebTorrent to fetch a piece we already have.
+      const spill = this.#evicting.get(index);
+      if (spill) {
+        await spill.catch(() => undefined);
+      }
+
       if (!this.#disk.has(index)) {
         throw new Error(`Piece ${index} is not in the store.`);
       }
@@ -403,8 +515,7 @@ export class SharedPieceStore {
         revived * this.#chunkLength + pieceLength
       );
       await this.#disk.read(index, target);
-      this.#slotOf.set(index, revived);
-      this.#lru.touch(index);
+      this.#registerSlot(index, revived);
       this.#counters.fromDisk += 1;
       this.#counters.revivals += 1;
       const start = revived * this.#chunkLength + offset;
@@ -443,6 +554,14 @@ export class SharedPieceStore {
       return this.locate(index);
     }
 
+    // Caught mid-spill: the slot is already gone, the disk copy is not there
+    // yet. Waiting is the only correct answer — reporting it missing would make
+    // the caller re-download a piece we are in the middle of keeping.
+    const spill = this.#evicting.get(index);
+    if (spill) {
+      await spill.catch(() => undefined);
+    }
+
     if (!this.#disk.has(index)) {
       return null;
     }
@@ -454,8 +573,7 @@ export class SharedPieceStore {
       revived * this.#chunkLength + pieceLength
     );
     await this.#disk.read(index, target);
-    this.#slotOf.set(index, revived);
-    this.#lru.touch(index);
+    this.#registerSlot(index, revived);
     this.#counters.fromDisk += 1;
     this.#counters.revivals += 1;
     return this.locate(index);

package/services/torrent-pool.js

@@ -306,6 +306,17 @@ export class TorrentPool {
    */
   #readPositionByTorrent = new Map();
 
+  /**
+   * Lowest piece currently selected for download, per torrent and fileIndex.
+   *
+   * Needed because selection is not readable back from WebTorrent, and a seek
+   * backward has to know whether the pieces it wants were deselected by an
+   * earlier seek forward.
+   *
+   * @type {Map<import("webtorrent").Torrent, Map<number, number>>}
+   */
+  #selectedFromPiece = new Map();
+
   /** Global disk cap in bytes (0 = disabled). */
   #maxDiskBytes = 0;
 
@@ -1068,8 +1079,24 @@ export class TorrentPool {
       readPositions = new Map();
       this.#readPositionByTorrent.set(torrent, readPositions);
     }
+    const previousStart = readPositions.get(fileIndex);
     readPositions.set(fileIndex, safeStart);
 
+    // Log jumps only. Sequential reading calls this on every range request and
+    // would drown the log; a jump is a seek, and a seek that never reaches the
+    // torrent is exactly the failure this line exists to make visible — after a
+    // seek the encoder waits on pieces nobody has been told to fetch.
+    const isJump =
+      previousStart === undefined || Math.abs(safeStart - previousStart) > PRIORITY_WINDOW_BYTES;
+    if (isJump) {
+      const percent = ((safeStart / fileLength) * 100).toFixed(1);
+      logger.info(
+        `torrent-pool: [${String(torrent.infoHash).slice(0, 8)}] read position -> ` +
+        `${(safeStart / 1024 / 1024).toFixed(0)}MB (${percent}% of file ${fileIndex})` +
+        (previousStart === undefined ? " (first)" : ` (was ${(previousStart / 1024 / 1024).toFixed(0)}MB)`)
+      );
+    }
+
     const absStart = fileOffset + safeStart;
     const playheadPiece = Math.floor(absStart / pieceLength);
     const absWindowEnd = Math.min(
@@ -1078,17 +1105,41 @@ export class TorrentPool {
     );
     const windowEndPiece = Math.floor(absWindowEnd / pieceLength);
 
-    // (1) Demote the gap behind the playhead so the picker scans forward from
+    // (1) Re-select from the playhead when it moved BACK behind what an earlier
+    //     seek deselected. `deselect` removes pieces from the download set, and
+    //     `critical` does NOT put them back — it only flags pieces already
+    //     selected. So without this, a seek forward followed by a seek backward
+    //     leaves the target pieces wanted by nobody: the encoder waits on data
+    //     the torrent was told to stop fetching, and waits forever.
+    //     Only on a backward move, so repeat calls do not pile up selections.
+    let selectedFrom = this.#selectedFromPiece.get(torrent)?.get(fileIndex);
+    if (selectedFrom === undefined || playheadPiece < selectedFrom) {
+      try {
+        torrent.select(playheadPiece, fileEndPiece, 1);
+      } catch {
+        // Best effort — never break streaming because selection failed.
+      }
+      let perFile = this.#selectedFromPiece.get(torrent);
+      if (!perFile) {
+        perFile = new Map();
+        this.#selectedFromPiece.set(torrent, perFile);
+      }
+      perFile.set(fileIndex, playheadPiece);
+      selectedFrom = playheadPiece;
+    }
+
+    // (2) Demote the gap behind the playhead so the picker scans forward from
     //     the read position. Only when there IS a gap (not at the file start).
     if (playheadPiece > fileStartPiece && typeof torrent.deselect === "function") {
       try {
         torrent.deselect(fileStartPiece, playheadPiece - 1);
+        this.#selectedFromPiece.get(torrent)?.set(fileIndex, playheadPiece);
       } catch {
         // Best effort — never break streaming because demotion failed.
       }
     }
 
-    // (2) Reset criticality to a moving read-ahead window (hotswap over the near
+    // (3) Reset criticality to a moving read-ahead window (hotswap over the near
     //     pieces), so it does not accumulate over the whole file across seeks.
     if (Array.isArray(torrent._critical)) {
       torrent._critical.length = 0;

package/services/torrent-worker/client.js

@@ -41,6 +41,8 @@ export class TorrentWorkerClient {
   #poolBySource = new Map();
   /** Which pool an in-flight read belongs to, keyed by request id. */
   #poolByRead = new Map();
+  /** Reads consuming fragments in place, keyed by request id. */
+  #fragmentReaders = new Map();
 
   /**
    * @param {{ maxDiskBytes?: number, memoryBytes?: number }} [options]
@@ -64,25 +66,42 @@ export class TorrentWorkerClient {
         read.fail(new Error(message.error ?? "Torrent worker read failed."));
         return;
       }
+      if (message?.type === Event.ERROR && this.#fragmentReaders.has(message.id)) {
+        const reader = this.#fragmentReaders.get(message.id);
+        this.#fragmentReaders.delete(message.id);
+        this.#poolByRead.delete(message.id);
+        reader.fail(new Error(message.error ?? "Torrent worker read failed."));
+        return;
+      }
       if (this.#caller.handleReply(message)) {
         return;
       }
       switch (message?.type) {
         case Event.FRAGMENT: {
-          // The bytes are already here — this thread maps the same pool. Read
-          // them where they lie, then say so, which is what lets the worker
-          // unpin the piece and move on. The copy exists only because the
-          // consumer keeps what it is given while the slot may be reused; it is
-          // one copy on this thread rather than one on the torrent's.
           const pool = this.#poolByRead.get(message.id);
-          const bytes = pool
-            ? Uint8Array.prototype.slice.call(
-                new Uint8Array(pool, message.offset, message.length)
-              )
-            : null;
-          if (bytes) {
-            this.#reads.get(message.id)?.push(bytes);
+          if (!pool) {
+            // No pool means no way to read the fragment; confirm it so the
+            // worker is not left waiting, and let the read end short.
+            this.#worker.postMessage({ type: Event.FRAGMENT_DONE, id: message.id });
+            break;
+          }
+          const view = new Uint8Array(pool, message.offset, message.length);
+
+          const reader = this.#fragmentReaders.get(message.id);
+          if (reader) {
+            // Handed on as a view into the pool — no copy anywhere. The piece
+            // stays pinned until the consumer says it is done with these exact
+            // bytes, which for a response body means the socket write has
+            // completed.
+            reader.push(view, () => {
+              this.#worker.postMessage({ type: Event.FRAGMENT_DONE, id: message.id });
+            });
+            break;
           }
+
+          // Plain-stream consumers keep what they are given while the slot may
+          // be reused, so they get a copy and the piece is released at once.
+          this.#reads.get(message.id)?.push(Uint8Array.prototype.slice.call(view));
           this.#worker.postMessage({ type: Event.FRAGMENT_DONE, id: message.id });
           break;
         }
@@ -96,6 +115,8 @@ export class TorrentWorkerClient {
         case Event.READ_END:
           this.#reads.get(message.id)?.close();
           this.#reads.delete(message.id);
+          this.#fragmentReaders.get(message.id)?.close();
+          this.#fragmentReaders.delete(message.id);
           this.#poolByRead.delete(message.id);
           break;
         case Event.LOG:
@@ -235,6 +256,104 @@ export class TorrentWorkerClient {
     return receive.stream;
   }
 
+  /**
+   * Read a byte range as fragments of shared memory, without copying.
+   *
+   * Each fragment is a view straight into the torrent's piece pool, and the
+   * piece behind it stays pinned until `release()` is called — so the consumer
+   * must call it once it is genuinely finished with those bytes. For a response
+   * body that means after the socket write has completed, not when it was
+   * queued: verified that writing a shared-memory view and then overwriting the
+   * pool from the write callback leaves the client's copy intact, and that
+   * overwriting it earlier corrupts it silently.
+   *
+   * Returns `null` when this source has no shared pool, so the caller can fall
+   * back to {@link createReadStream}.
+   *
+   * @param {{ sourceKey: string, fileIndex: number, start?: number | null, end?: number | null }} params
+   * @returns {{ [Symbol.asyncIterator]: () => AsyncGenerator<{ bytes: Uint8Array, release: () => void }>, cancel: () => void } | null}
+   */
+  createFragmentReader({ sourceKey, fileIndex, start = null, end = null }) {
+    const pool = this.#poolBySource.get(sourceKey);
+    if (!pool) {
+      return null;
+    }
+
+    const readId = this.#caller.nextId();
+    /** @type {{ bytes: Uint8Array, release: () => void }[]} */
+    const queue = [];
+    let wake = null;
+    let finished = false;
+    let failure = null;
+
+    const notify = () => {
+      const resume = wake;
+      wake = null;
+      resume?.();
+    };
+
+    this.#fragmentReaders.set(readId, {
+      push(bytes, confirm) {
+        queue.push({ bytes, release: confirm });
+        notify();
+      },
+      close() {
+        finished = true;
+        notify();
+      },
+      fail(error) {
+        failure = error;
+        finished = true;
+        notify();
+      }
+    });
+    this.#poolByRead.set(readId, pool);
+
+    const cancel = () => {
+      if (this.#fragmentReaders.delete(readId)) {
+        this.#poolByRead.delete(readId);
+        void this.#caller.call(Command.CANCEL_READ, { readId }).catch(() => undefined);
+      }
+      finished = true;
+      notify();
+    };
+
+    this.#worker.postMessage({
+      command: Command.READ_RANGE,
+      id: readId,
+      params: { sourceKey, fileIndex, start, end }
+    });
+
+    return {
+      cancel,
+      async *[Symbol.asyncIterator]() {
+        try {
+          for (;;) {
+            while (queue.length > 0) {
+              yield queue.shift();
+            }
+            if (failure) {
+              throw failure;
+            }
+            if (finished) {
+              return;
+            }
+            await new Promise((resolve) => {
+              wake = resolve;
+            });
+          }
+        } finally {
+          // Covers the consumer breaking out early — a client that hung up, a
+          // superseded seek — which must stop the read rather than leave it
+          // fetching pieces nobody will take.
+          if (!finished) {
+            cancel();
+          }
+        }
+      }
+    };
+  }
+
   /**
    * A stand-in for the WebTorrent torrent object, backed by the worker.
    *
@@ -285,6 +404,22 @@ export class TorrentWorkerClient {
               end: options.end ?? null
             })
           );
+        },
+
+        /**
+         * Fragments of shared memory, for a caller that can say when it has
+         * finished with each one. `null` when this source has no shared pool.
+         *
+         * @param {{ start?: number, end?: number }} [options]
+         * @returns {ReturnType<TorrentWorkerClient["createFragmentReader"]>}
+         */
+        createFragmentReader(options = {}) {
+          return client.createFragmentReader({
+            sourceKey,
+            fileIndex: file.index,
+            start: options.start ?? null,
+            end: options.end ?? null
+          });
         }
       }))
     };

package/services/torrent-worker/worker.js

@@ -398,7 +398,7 @@ setInterval(() => {
     const fromMemoryShare = reads > 0 ? ((stats.fromMemory / reads) * 100).toFixed(1) : "—";
     log(
       `piece-store "${stats.name.slice(0, 40)}": resident=${stats.resident}/${stats.capacity} ` +
-      `spilled=${stats.spilled} reads=${reads} (${fromMemoryShare}% from memory) ` +
+      `pinned=${stats.pinned} spilled=${stats.spilled} reads=${reads} (${fromMemoryShare}% from memory) ` +
       `spills=${stats.spills} revivals=${stats.revivals}` +
       (stats.blockedByPins > 0 ? ` blocked-by-pins=${stats.blockedByPins}` : "")
     );

package/test/data-channel-frame.test.js

@@ -0,0 +1,63 @@
+/**
+ * @file The wire format of a body frame.
+ *
+ * The browser parses these bytes, so the layout is a contract:
+ * `[flags(1)][idLen(1)][requestId][payload]`. The framing was rewritten to stop
+ * copying every chunk twice (75.9 ms per 13 MB segment against 40.0 on the
+ * field host), and a rewrite of something a client parses needs the format
+ * pinned down, not just the timing.
+ */
+
+import test from "node:test";
+import assert from "node:assert/strict";
+import { encodeFrame } from "../services/data-channel-handler.js";
+
+const requestId = Buffer.from("abc123", "ascii");
+
+test("a body frame carries the id and the payload unchanged", () => {
+  const payload = Buffer.from([1, 2, 3, 4, 250, 255]);
+  const frame = encodeFrame(requestId, payload, false);
+
+  assert.equal(frame[0], 0, "flagged as done");
+  assert.equal(frame[1], requestId.length);
+  assert.deepEqual(frame.subarray(2, 2 + requestId.length), requestId);
+  assert.deepEqual(frame.subarray(2 + requestId.length), payload);
+  assert.equal(frame.length, 2 + requestId.length + payload.length);
+});
+
+test("the done frame carries no payload", () => {
+  const frame = encodeFrame(requestId, null, true);
+
+  assert.equal(frame[0], 1);
+  assert.equal(frame.length, 2 + requestId.length);
+});
+
+test("an empty payload is treated as no payload", () => {
+  const frame = encodeFrame(requestId, new Uint8Array(0), false);
+  assert.equal(frame.length, 2 + requestId.length);
+});
+
+test("a payload that is a view into a larger buffer is copied correctly", () => {
+  // Chunks arrive as views into a bigger allocation, so copying the whole
+  // underlying buffer instead of the view would send the wrong bytes at the
+  // wrong length — silently.
+  const backing = Buffer.alloc(64, 9);
+  backing.fill(42, 16, 32);
+  const view = new Uint8Array(backing.buffer, backing.byteOffset + 16, 16);
+
+  const frame = encodeFrame(requestId, view, false);
+
+  assert.equal(frame.length, 2 + requestId.length + 16);
+  assert.deepEqual(frame.subarray(2 + requestId.length), Buffer.alloc(16, 42));
+});
+
+test("a large payload survives framing byte for byte", () => {
+  const payload = Buffer.allocUnsafeSlow(64 * 1024);
+  for (let at = 0; at < payload.length; at += 1) {
+    payload[at] = at % 251;
+  }
+
+  const frame = encodeFrame(requestId, payload, false);
+
+  assert.deepEqual(frame.subarray(2 + requestId.length), payload);
+});

package/test/piece-store-eviction.test.js

@@ -0,0 +1,134 @@
+/**
+ * Eviction under concurrency.
+ *
+ * Pieces arrive from several peers at once, so `put` runs concurrently by
+ * nature. Choosing a victim and releasing it either side of the spill write
+ * therefore let two claims pick the SAME victim and receive the SAME slot, at
+ * which point two pieces overwrite each other, both fail their hash, and the
+ * torrent re-downloads them without end — which looks from outside exactly like
+ * a seek that never completes while the download runs at full speed.
+ */
+
+import test from "node:test";
+import assert from "node:assert/strict";
+import fs from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { SharedPieceStore } from "../services/piece-store/shared-piece-store.js";
+
+const PIECE = 1024;
+
+/**
+ * @param {number} capacityPieces
+ * @returns {Promise<{ store: SharedPieceStore, directory: string }>}
+ */
+async function makeStore(capacityPieces) {
+  const directory = await fs.mkdtemp(path.join(os.tmpdir(), "piece-store-test-"));
+  const store = new SharedPieceStore(PIECE, {
+    length: PIECE * 64,
+    memoryBytes: PIECE * capacityPieces,
+    spillDirectory: directory,
+    name: "eviction-test"
+  });
+  return { store, directory };
+}
+
+/**
+ * @param {number} index
+ * @returns {Buffer} A piece whose every byte identifies it.
+ */
+const pieceOf = (index) => Buffer.alloc(PIECE, index % 251);
+
+/**
+ * @param {SharedPieceStore} store
+ * @param {number} index
+ * @param {Buffer} bytes
+ * @returns {Promise<void>}
+ */
+const put = (store, index, bytes) =>
+  new Promise((resolve, reject) => {
+    store.put(index, bytes, (error) => (error ? reject(error) : resolve()));
+  });
+
+/**
+ * @param {SharedPieceStore} store
+ * @param {number} index
+ * @returns {Promise<Buffer>}
+ */
+const get = (store, index) =>
+  new Promise((resolve, reject) => {
+    store.get(index, (error, bytes) => (error ? reject(error) : resolve(bytes)));
+  });
+
+test("concurrent puts past capacity never hand two pieces the same slot", async () => {
+  const capacity = 4;
+  const { store, directory } = await makeStore(capacity);
+  try {
+    const total = 24;
+
+    // All at once — the interleaving that a sequential test never produces.
+    await Promise.all(
+      Array.from({ length: total }, (unused, index) => put(store, index, pieceOf(index)))
+    );
+
+    // Every piece must read back as itself, from memory or from disk. A shared
+    // slot shows up here as one piece carrying another's bytes.
+    for (let index = 0; index < total; index += 1) {
+      const bytes = await get(store, index);
+      assert.equal(bytes.length, PIECE, `piece ${index} came back the wrong length`);
+      assert.ok(
+        bytes.equals(pieceOf(index)),
+        `piece ${index} came back as piece ${bytes[0]} — two pieces shared a slot`
+      );
+    }
+
+    const stats = store.stats();
+    assert.ok(
+      stats.resident <= stats.capacity,
+      `resident ${stats.resident} exceeds capacity ${stats.capacity}: slots were handed out twice`
+    );
+  } finally {
+    await store.destroy();
+    await fs.rm(directory, { recursive: true, force: true });
+  }
+});
+
+test("a piece caught mid-spill is waited for, not reported missing", async () => {
+  const { store, directory } = await makeStore(2);
+  try {
+    await put(store, 0, pieceOf(0));
+    await put(store, 1, pieceOf(1));
+
+    // Forces piece 0 out while piece 0 is asked for in the same tick.
+    const evicting = put(store, 2, pieceOf(2));
+    const reading = get(store, 0);
+
+    await evicting;
+    const bytes = await reading;
+    assert.ok(bytes.equals(pieceOf(0)), "piece 0 came back wrong while being spilled");
+  } finally {
+    await store.destroy();
+    await fs.rm(directory, { recursive: true, force: true });
+  }
+});
+
+test("pinned pieces are never evicted, and the pin count is reported", async () => {
+  const { store, directory } = await makeStore(2);
+  try {
+    await put(store, 0, pieceOf(0));
+    store.pin(0);
+    assert.equal(store.stats().pinned, 1, "pin is not reflected in the stats");
+
+    await put(store, 1, pieceOf(1));
+    await put(store, 2, pieceOf(2));
+
+    // Piece 0 is pinned, so it must still be the one in memory, not on disk.
+    assert.ok(store.locate(0), "a pinned piece was evicted");
+
+    store.unpin(0);
+    assert.equal(store.stats().pinned, 0, "unpin is not reflected in the stats");
+  } finally {
+    await store.destroy();
+    await fs.rm(directory, { recursive: true, force: true });
+  }
+});

package/test/worker-channel.test.js

@@ -156,6 +156,11 @@ test("a read of an unknown source fails the stream rather than hanging", async (
   // the reader waited forever. A unit test on either half alone passes happily.
   const client = new TorrentWorkerClient({ memoryBytes: 8 * 1024 * 1024 });
   try {
+    // Starting the worker takes several seconds — it builds a torrent client
+    // and a DHT — so the first request would measure startup, not the failure
+    // path under test.
+    await client.listFiles("warm-up").catch(() => undefined);
+
     const stream = client.createReadStream({ sourceKey: "no-such-source", fileIndex: 0 });
     const reader = stream.getReader();