@torrent-tv/proxy 2.9.72 → 2.9.74

package/CHANGELOG.md

@@ -1,3 +1,15 @@
+## 2.9.74
+
+- **Fix**: Playback works again. Since 2.9.71 every read answered with headers and an empty body — ffmpeg reported `Stream ends prematurely at 0, should be <size>` and the loading screen sat on "Preparing HLS transcode" until it gave up. Root cause, reproduced locally on two different torrents once the right conditions were used (a large, **partially downloaded** file — a complete one never shows it): the worker transferred ownership of a buffer belonging to WebTorrent's piece cache, the cache's memory was detached, and from that moment every read failed with `Cannot perform %TypedArray%.prototype.slice on a detached ArrayBuffer`. Nobody saw that error, because the worker sent the end-of-read marker from its `finally` before posting the failure and the main thread had no handler for a read error at all — so a broken read was indistinguishable from an empty file. Fixed at the root by owning the memory (the new piece store) and at the boundary by having the transport copy into memory it allocated itself rather than trying to guess whether the caller's buffer was safe to take. Verified end to end on both an almost-complete and a freshly-added torrent: playback plan, byte range, transcode session, init segment and first media segment all produced.
+- **New**: A piece store of our own (`services/piece-store/`): pieces live in a `SharedArrayBuffer`, spill to a single sparse file when the memory budget is full, and come back from it on demand. Owning the memory is what makes the thread split safe — WebTorrent's own cache hands out buffers it keeps using, which is why transferring one detached the cache and killed every subsequent read. Two properties are enforced rather than hoped for: a piece being read is **pinned** and cannot be evicted (with every piece pinned the store refuses to make room instead of taking memory from under a reader — the exact failure of 2.9.71), and a buffer handed to a caller is never invalidated by later writes. Sized by measurement on the field host: a piece copy costs 3.64 ms, reading one back from disk into a buffer we already own 7.63 ms, and re-downloading it from the swarm ~1430 ms — so memory first, disk under it, the swarm never twice. Found while testing: opening the spill file in append mode makes POSIX ignore the write position entirely, so pieces piled up in arrival order and reads returned whichever piece happened to sit at that offset.
+- **Fix**: A torrent carrying a `wss://` tracker took the **whole proxy process** down from 2.9.71 — including the demo magnet on the site's own button. node-datachannel is native and cannot be used from two V8 isolates at once (`HandleScope: Entering the V8 API without proper locking in place`); measured identically on win32/x64 and linux/arm64, with one isolate fine either way, two fatal, and `preload()` in both no help. Before the thread split both users lived in one isolate; afterwards the browser's video channel sat on the main thread while the torrent's tracker announces created peer connections on the worker. Fixed by leaving the native stack where it carries video and giving the torrent thread a JavaScript one (`werift`) through a module-resolution hook scoped to that worker — no dependency is patched, which matters because the addon installs with `--ignore-scripts`. The shim supplies the three things werift's data channel lacks and `simple-peer` depends on: `binaryType` (without it every payload goes through a text decoder and arrives corrupted), the buffered-amount-low event (its backpressure never resumes without it), and a session description built from one object rather than two positional arguments (werift's own signature is `(sdp, type)`, so `{ type, sdp }` was rejected as `invalid sessionDescription`). Verified end to end: a magnet with **only** wss trackers now connects to browser peers and downloads the file completely.
+- **Chore**: The proxy has tests, and publishing runs them. There were none before, and nothing stood between writing code and `npm publish` — which is how the 2.9.71 thread split reached the field with a defect that stopped every read. `npm test` (Node's own runner, no new dependencies) plus `prepublishOnly`, so an unproven package cannot be published. The first cases cover the transport's memory contract and are written to FAIL on the current code: sending a chunk must leave the source buffer usable by its owner, and a second read of the same piece must still return its bytes. Both fail today, which is the point — they describe the shipped defect.
+
+## 2.9.73
+
+- **Fix**: File stats came back as `{}` after the torrent moved to its own thread (2.9.71), which left the loading screen with no peers, no speed and no progress. Two call sites — the stats route and the health report — invoked `getFileStats` **without awaiting**: it used to answer locally and immediately, and now crosses a thread boundary, so the reply was the pending promise itself, serialised to an empty object. Both now await it.
+- **Chore**: Chunk transfer no longer hands over memory the chunk does not own outright. Node allocates small buffers from a shared 8 KB pool — several unrelated buffers occupy one region, each viewing its own slice (verified: a 1 KB buffer reports an 8192-byte region at offset 8) — so transferring that region would detach it from its neighbours. Chunks sourced from the network are small enough to be pooled while local disk reads are not, which is exactly the difference between the field host and the local test. Measured afterwards: pooled chunks cross intact, so this is a correctness guard rather than the cause of the field failure.
+
 ## 2.9.72
 
 - **Fix**: Playback broke entirely after the torrent moved to its own thread (2.9.71): the torrent was deleted **with its downloaded data** while still being read, after which every read hung and ffmpeg saw an empty input (`Stream ends prematurely at 0, should be 3303133078`), and the container-index read took 73 s to return nothing. Cause: `acquireFile` was dispatched without awaiting while its release was sent normally, so a release could overtake the acquire it belonged to; the reader count then hit zero mid-read and the idle sweep fired (`removed idle torrent ... and its store`). Two fixes, each sufficient alone: the release is now chained onto the acquire so it can never arrive first, and the worker additionally holds the file for the whole duration of the read — a claim that lives inside the read and so cannot be reordered against it. Not reproducible locally, where the test torrent was fully downloaded and never went idle.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@torrent-tv/proxy",
-  "version": "2.9.72",
+  "version": "2.9.74",
   "description": "Torrent proxy client that exposes webseed-like HTTP stream endpoint.",
   "license": "GPL-3.0-or-later",
   "publishConfig": {
@@ -15,7 +15,9 @@
     "minor": "npm whoami && npm version minor && npm publish && git push --follow-tags",
     "major": "npm whoami && npm version major && npm publish && git push --follow-tags",
     "start": "node ./bin/cli.js",
-    "dev": "node --inspect=0 --experimental-network-inspection ./bin/cli.js"
+    "dev": "node --inspect=0 --experimental-network-inspection ./bin/cli.js",
+    "test": "node --test",
+    "prepublishOnly": "npm test"
   },
   "dependencies": {
     "@fastify/cors": "^11.2.0",
@@ -31,6 +33,7 @@
     "node-datachannel": "^0.32.0",
     "uint8-util": "2.2.6",
     "webtorrent": "2.8.5",
+    "werift": "^0.24.2",
     "ws": "^8.18.2"
   }
 }

package/routes/api/sources/stats/get.js

@@ -1,66 +1,69 @@
-import { logger } from "../../../../utils/logger.js";
-
-/**
- * Return download statistics for a registered torrent source.
- *
- * Provides peer count, transfer speeds, and per-file download progress so
- * that the browser client can display meaningful feedback while the proxy is
- * pre-fetching file metadata (MOOV atom / EBML headers) before codec probing.
- *
- * GET /api/sources/:sourceKey/stats?fileIndex=N
- *
- * @param {import("fastify").FastifyRequest} req
- * @param {import("fastify").FastifyReply} reply
- * @param {{
- *   sourceRegistry: ReturnType<import("../../../../store/source-registry.js").createSourceRegistry>,
- *   torrentPool: import("../../../../services/torrent-pool.js").TorrentPool
- * }} deps
- * @returns {Promise<void>}
- */
-export async function handleApiSourceStatsGet(req, reply, { sourceRegistry, torrentPool }) {
-  const sourceKey = typeof req.params.sourceKey === "string" ? req.params.sourceKey.trim() : "";
-  if (!sourceKey) {
-    return reply.code(400).send({ error: "sourceKey is required." });
-  }
-
-  const sourceRecord = sourceRegistry.get(sourceKey);
-  if (!sourceRecord) {
-    return reply.code(404).send({ error: "Source key was not found." });
-  }
-
-  let torrent;
-  try {
-    // getTorrent resolves immediately when the torrent is already loaded.
-    torrent = await torrentPool.getTorrent(sourceRecord.sourceType, sourceRecord.source);
-  } catch (error) {
-    const message = error instanceof Error ? error.message : String(error);
-    return reply.code(500).send({ error: `Failed to load torrent: ${message}` });
-  }
-
-  const fileIndexRaw = typeof req.query.fileIndex === "string" ? req.query.fileIndex : "";
-  const fileIndex = fileIndexRaw !== "" && /^\d+$/.test(fileIndexRaw) ? Number(fileIndexRaw) : null;
-
-  // Optional: pin the resume window to a FIXED byte offset for the duration of
-  // one buffering episode (see getFileStats JSDoc) instead of the live, moving
-  // read position — otherwise "bytes needed" can jump up mid-poll as the window
-  // slides forward with playback/encoding progress.
-  const resumeAnchorRaw = typeof req.query.resumeAnchorByteStart === "string" ? req.query.resumeAnchorByteStart : "";
-  const resumeAnchorByteStart = resumeAnchorRaw !== "" && /^\d+$/.test(resumeAnchorRaw) ? Number(resumeAnchorRaw) : null;
-
-  const stats = torrentPool.getFileStats(torrent, fileIndex, { resumeAnchorByteStart });
-
-  // Diagnostic: surface the real swarm state per poll so a cold-start download
-  // stall (0 peers / header not advancing → playback-plan blocks on the codec
-  // probe → browser timeout) is visible in the proxy log.
-  const downKbps = (stats.downloadSpeed / 1024).toFixed(0);
-  const filePct = stats.fileProgress != null ? `${(stats.fileProgress * 100).toFixed(1)}%` : "n/a";
-  const header =
-    stats.headerBytes != null
-      ? `${stats.headerDownloadedBytes}/${stats.headerBytes}B`
-      : "n/a";
-  logger.info(
-    `[stats] ${sourceKey.slice(0, 8)} peers=${stats.numPeers} down=${downKbps}KB/s file=${filePct} header=${header}`
-  );
-
-  return reply.send(stats);
-}
+import { logger } from "../../../../utils/logger.js";
+
+/**
+ * Return download statistics for a registered torrent source.
+ *
+ * Provides peer count, transfer speeds, and per-file download progress so
+ * that the browser client can display meaningful feedback while the proxy is
+ * pre-fetching file metadata (MOOV atom / EBML headers) before codec probing.
+ *
+ * GET /api/sources/:sourceKey/stats?fileIndex=N
+ *
+ * @param {import("fastify").FastifyRequest} req
+ * @param {import("fastify").FastifyReply} reply
+ * @param {{
+ *   sourceRegistry: ReturnType<import("../../../../store/source-registry.js").createSourceRegistry>,
+ *   torrentPool: import("../../../../services/torrent-pool.js").TorrentPool
+ * }} deps
+ * @returns {Promise<void>}
+ */
+export async function handleApiSourceStatsGet(req, reply, { sourceRegistry, torrentPool }) {
+  const sourceKey = typeof req.params.sourceKey === "string" ? req.params.sourceKey.trim() : "";
+  if (!sourceKey) {
+    return reply.code(400).send({ error: "sourceKey is required." });
+  }
+
+  const sourceRecord = sourceRegistry.get(sourceKey);
+  if (!sourceRecord) {
+    return reply.code(404).send({ error: "Source key was not found." });
+  }
+
+  let torrent;
+  try {
+    // getTorrent resolves immediately when the torrent is already loaded.
+    torrent = await torrentPool.getTorrent(sourceRecord.sourceType, sourceRecord.source);
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    return reply.code(500).send({ error: `Failed to load torrent: ${message}` });
+  }
+
+  const fileIndexRaw = typeof req.query.fileIndex === "string" ? req.query.fileIndex : "";
+  const fileIndex = fileIndexRaw !== "" && /^\d+$/.test(fileIndexRaw) ? Number(fileIndexRaw) : null;
+
+  // Optional: pin the resume window to a FIXED byte offset for the duration of
+  // one buffering episode (see getFileStats JSDoc) instead of the live, moving
+  // read position — otherwise "bytes needed" can jump up mid-poll as the window
+  // slides forward with playback/encoding progress.
+  const resumeAnchorRaw = typeof req.query.resumeAnchorByteStart === "string" ? req.query.resumeAnchorByteStart : "";
+  const resumeAnchorByteStart = resumeAnchorRaw !== "" && /^\d+$/.test(resumeAnchorRaw) ? Number(resumeAnchorRaw) : null;
+
+  // Awaited: with the torrent on its own thread this is a round trip, not a
+  // local lookup. Without the await the reply was the pending promise itself,
+  // which serialises to `{}` — the empty stats seen in the field 2026-08-02.
+  const stats = await torrentPool.getFileStats(torrent, fileIndex, { resumeAnchorByteStart });
+
+  // Diagnostic: surface the real swarm state per poll so a cold-start download
+  // stall (0 peers / header not advancing → playback-plan blocks on the codec
+  // probe → browser timeout) is visible in the proxy log.
+  const downKbps = (stats.downloadSpeed / 1024).toFixed(0);
+  const filePct = stats.fileProgress != null ? `${(stats.fileProgress * 100).toFixed(1)}%` : "n/a";
+  const header =
+    stats.headerBytes != null
+      ? `${stats.headerDownloadedBytes}/${stats.headerBytes}B`
+      : "n/a";
+  logger.info(
+    `[stats] ${sourceKey.slice(0, 8)} peers=${stats.numPeers} down=${downKbps}KB/s file=${filePct} header=${header}`
+  );
+
+  return reply.send(stats);
+}

package/server.js

@@ -144,7 +144,9 @@ export async function startProxyServer({ host, port, transcodeAudio, ffmpegBin,
       }
       try {
         const torrent = await torrentPool.getTorrent(record.sourceType, record.source);
-        return torrentPool.getFileStats(torrent, Number.isInteger(fileIndex) ? fileIndex : null);
+        // Awaited for the same reason as the stats route: this now crosses a
+        // thread boundary and returns a promise.
+        return await torrentPool.getFileStats(torrent, Number.isInteger(fileIndex) ? fileIndex : null);
       } catch {
         return null;
       }

package/services/piece-store/disk-tier.js

@@ -0,0 +1,151 @@
+/**
+ * @file The second tier: pieces that no longer fit in memory.
+ *
+ * One sparse file per torrent, a piece at `index * chunkLength`. Not the
+ * torrent's real files — nobody reads those directly; playback goes through
+ * `/stream`, and the data is discarded when the torrent goes idle. A single
+ * file by piece index keeps the mapping arithmetic instead of bookkeeping, and
+ * sparseness means the untouched gaps cost nothing.
+ *
+ * Reads take a destination buffer rather than returning a fresh one. That is
+ * not a style preference — measured on the field host, an 8 MB piece costs
+ * 22.08 ms via `readFile`, which allocates, and **7.63 ms** read into a buffer
+ * we already hold. Two thirds of the apparent "disk" cost was allocation.
+ */
+
+import fs from "node:fs/promises";
+import { constants } from "node:fs";
+import path from "node:path";
+
+export class DiskTier {
+  #filePath;
+  #chunkLength;
+  /** @type {import("node:fs/promises").FileHandle | null} */
+  #handle = null;
+  /** Piece indices known to be on disk. */
+  #stored = new Set();
+  /** @type {Promise<void> | null} */
+  #opening = null;
+
+  /**
+   * @param {object} params
+   * @param {string} params.directory - Where the backing file lives.
+   * @param {string} params.name - File name, unique per torrent.
+   * @param {number} params.chunkLength - Piece size; fixes the stride on disk.
+   */
+  constructor({ directory, name, chunkLength }) {
+    this.#filePath = path.join(directory, name);
+    this.#chunkLength = chunkLength;
+  }
+
+  /** Where the backing file lives, for logging and cleanup. */
+  get filePath() {
+    return this.#filePath;
+  }
+
+  /** How many pieces are currently held on disk. */
+  get size() {
+    return this.#stored.size;
+  }
+
+  /**
+   * Open the backing file, creating it and its directory if needed.
+   *
+   * Concurrent callers share one open — several evictions can start at once.
+   *
+   * @returns {Promise<import("node:fs/promises").FileHandle>}
+   */
+  async #open() {
+    if (this.#handle) {
+      return this.#handle;
+    }
+    if (!this.#opening) {
+      this.#opening = (async () => {
+        await fs.mkdir(path.dirname(this.#filePath), { recursive: true });
+        // Read/write, created if absent — NOT append mode. Under `a+` POSIX
+        // ignores the position on every write and puts the bytes at the end of
+        // the file, so pieces would pile up in arrival order and each read
+        // would return whichever piece happened to land at that offset.
+        this.#handle = await fs.open(this.#filePath, constants.O_RDWR | constants.O_CREAT);
+      })();
+    }
+    await this.#opening;
+    if (!this.#handle) {
+      throw new Error(`Could not open the piece file at ${this.#filePath}.`);
+    }
+    return this.#handle;
+  }
+
+  /**
+   * @param {number} index
+   * @returns {boolean}
+   */
+  has(index) {
+    return this.#stored.has(index);
+  }
+
+  /**
+   * Write a piece out.
+   *
+   * @param {number} index
+   * @param {Uint8Array} bytes
+   * @returns {Promise<void>}
+   */
+  async write(index, bytes) {
+    const handle = await this.#open();
+    await handle.write(bytes, 0, bytes.length, index * this.#chunkLength);
+    this.#stored.add(index);
+  }
+
+  /**
+   * Read a piece back into a buffer the caller already owns.
+   *
+   * @param {number} index
+   * @param {Uint8Array} target - Destination; its length is what gets read.
+   * @returns {Promise<number>} Bytes read.
+   */
+  async read(index, target) {
+    if (!this.#stored.has(index)) {
+      throw new Error(`Piece ${index} is not on disk.`);
+    }
+    const handle = await this.#open();
+    const { bytesRead } = await handle.read(target, 0, target.length, index * this.#chunkLength);
+    return bytesRead;
+  }
+
+  /**
+   * Forget a piece. The bytes stay on disk until the file is removed — there is
+   * nothing to gain from punching them out, since the file is discarded whole.
+   *
+   * @param {number} index
+   * @returns {void}
+   */
+  forget(index) {
+    this.#stored.delete(index);
+  }
+
+  /**
+   * Close the file, leaving its contents in place.
+   *
+   * @returns {Promise<void>}
+   */
+  async close() {
+    const handle = this.#handle;
+    this.#handle = null;
+    this.#opening = null;
+    if (handle) {
+      await handle.close();
+    }
+  }
+
+  /**
+   * Close and delete the backing file.
+   *
+   * @returns {Promise<void>}
+   */
+  async destroy() {
+    await this.close();
+    this.#stored.clear();
+    await fs.rm(this.#filePath, { force: true });
+  }
+}

package/services/piece-store/piece-lru.js

@@ -0,0 +1,150 @@
+/**
+ * @file Which piece leaves memory next — and which one may not.
+ *
+ * Two responsibilities, deliberately kept apart from any storage:
+ *
+ *  - **recency**, so the piece evicted is the one least likely to be wanted;
+ *  - **pinning**, so a piece being read cannot be evicted at all.
+ *
+ * The second is not a refinement of the first. webtor's seeder relies on recency
+ * alone and guards the actual eviction with a read/write lock per piece
+ * (`mmap.go`, 1024 shards) — because ordering says a piece is *unlikely* to be
+ * in use, never that it is not. We have already paid for that difference once:
+ * proxy 2.9.71 removed a torrent, and its data, out from under an active reader,
+ * after which every read hung and ffmpeg saw an empty input. Here the guarantee
+ * is explicit: a pinned piece is never returned as an eviction candidate, and
+ * pins nest, because a piece can be read by several sessions at once.
+ */
+
+/**
+ * Recency and pin bookkeeping for one torrent's pieces.
+ *
+ * Holds no data — it answers "what may go" and nothing else, which is what
+ * makes it testable without a torrent, a disk or a thread.
+ */
+export class PieceLru {
+  /** Insertion-ordered: the first key is the least recently used. */
+  #order = new Set();
+  /** Piece index → number of readers currently holding it. */
+  #pins = new Map();
+  #capacity;
+
+  /**
+   * @param {number} capacity - How many pieces may be resident at once.
+   */
+  constructor(capacity) {
+    if (!Number.isInteger(capacity) || capacity < 1) {
+      throw new Error(`Piece capacity must be a positive integer, got ${capacity}.`);
+    }
+    this.#capacity = capacity;
+  }
+
+  /** How many pieces are resident. */
+  get size() {
+    return this.#order.size;
+  }
+
+  /** How many pieces may be resident at once. */
+  get capacity() {
+    return this.#capacity;
+  }
+
+  /**
+   * Mark a piece as resident, or as used again if it already was.
+   *
+   * A `Set` preserves insertion order, so deleting and re-adding is what moves
+   * a piece to the most-recent end.
+   *
+   * @param {number} index
+   * @returns {void}
+   */
+  touch(index) {
+    this.#order.delete(index);
+    this.#order.add(index);
+  }
+
+  /**
+   * @param {number} index
+   * @returns {boolean}
+   */
+  has(index) {
+    return this.#order.has(index);
+  }
+
+  /**
+   * Stop tracking a piece — it is no longer resident.
+   *
+   * @param {number} index
+   * @returns {void}
+   */
+  remove(index) {
+    this.#order.delete(index);
+  }
+
+  /**
+   * Hold a piece in memory for the duration of a read.
+   *
+   * Nested: two readers of the same piece take two pins, and the piece stays
+   * held until both let go.
+   *
+   * @param {number} index
+   * @returns {void}
+   */
+  pin(index) {
+    this.#pins.set(index, (this.#pins.get(index) ?? 0) + 1);
+  }
+
+  /**
+   * Release one pin taken with {@link pin}.
+   *
+   * @param {number} index
+   * @returns {void}
+   */
+  unpin(index) {
+    const held = this.#pins.get(index);
+    if (held === undefined) {
+      return;
+    }
+    if (held <= 1) {
+      this.#pins.delete(index);
+      return;
+    }
+    this.#pins.set(index, held - 1);
+  }
+
+  /**
+   * @param {number} index
+   * @returns {boolean}
+   */
+  isPinned(index) {
+    return this.#pins.has(index);
+  }
+
+  /**
+   * The least recently used piece that is free to go, or `null` when every
+   * resident piece is pinned.
+   *
+   * Returning `null` rather than evicting a pinned piece is the whole point:
+   * the caller must then wait or fail, never take memory out from under a
+   * reader.
+   *
+   * @returns {number | null}
+   */
+  evictionCandidate() {
+    for (const index of this.#order) {
+      if (!this.#pins.has(index)) {
+        return index;
+      }
+    }
+    return null;
+  }
+
+  /**
+   * Whether admitting one more piece would exceed the capacity.
+   *
+   * @returns {boolean}
+   */
+  isFull() {
+    return this.#order.size >= this.#capacity;
+  }
+}