@torrent-tv/proxy 2.9.13 → 2.9.15

package/CHANGELOG.md

@@ -1,3 +1,11 @@
+## 2.9.15
+
+- **Fix**: Torrent data is now cleaned up on graceful shutdown. `TorrentPool.destroyAll()` removes every torrent **with its on-disk store** (`torrent.destroy({ destroyStore: true })`) and then tears down the WebTorrent client; it is wired into the Fastify `onClose` hook (after `hlsSessionManager.disposeAll()`, so ffmpeg readers stop before their source files are removed). Previously nothing called `client.remove()`/`torrent.destroy()` anywhere, so downloaded files accumulated under `os.tmpdir()` until the process was killed — and even a clean SIGTERM/SIGINT left them behind. (First step of disk-hygiene Level 1; refcount/TTL removal and the startup orphan sweep are separate, still pending.)
+
+## 2.9.14
+
+- **New**: `GET /api/sources/:sourceKey/stats` now reports `headerBytes` / `headerDownloadedBytes` — how much of the file's header/index region (leading 256 KB + trailing 2 MB, the bytes the codec probe needs) is downloaded, counted by whole torrent pieces from the bitfield. Lets the browser show the download phase's progress and ETA toward the next (transcode) phase. Coarse by design (piece granularity).
+
 ## 2.9.13
 
 - **Fix**: Video-copy path (`video=copy`, audio transcoded or copied) no longer drops video / desyncs audio at the start. The output timeline is now forced 0-based: the container `start_time` (parsed from the probe; many MKVs report ~0.1 s) is subtracted via `-output_ts_offset -start_time` together with `-copyts`, so segment 0 begins exactly at 0 with audio and video aligned (previously `-copyts` preserved the non-zero start, leaving a hole at the beginning where video was blank but audio played).

package/CLAUDE.md

@@ -47,6 +47,58 @@ Linux-only host (e.g. POSIX-only signals must degrade elsewhere).
   around it with `--ignore-scripts` + a targeted rebuild of `node-datachannel`;
   if you ever change install flow, keep that in mind.
 
+## Planned: public reachability (remote access)
+
+Decided direction — full plan in the parent `../CLAUDE.md`. Proxy-side pieces:
+
+- **Auto port mapping** at startup via UPnP IGD / NAT-PMP / PCP (`nat-api` /
+  `@silentbot1/nat-api`; WebTorrent already does this for the torrent port).
+  Always request a lease time and renew it while running; remove the mapping
+  on graceful shutdown — lease expiry cleans up after crashes. Zero user
+  action, nothing left behind on the router.
+- Report the mapped external endpoint (and local addresses) to the server over
+  the tunnel; the server dial-back-verifies reachability before use.
+- **HTTPS listener**: serve the existing routes over TLS with a per-proxy
+  certificate delivered by the server through the tunnel (persist cert+key
+  locally; ~90-day renewals are pushed the same way). Add CORS headers for the
+  web-app origin so hls.js / `<video>` can fetch cross-origin.
+- Plain HTTPS becomes the preferred video transport; WebRTC data channel stays
+  as fallback for hosts where no port could be opened.
+- **NAT-traversal toolbox** (staged, see parent `../CLAUDE.md`): birthday-
+  paradox port prediction (open ~256 UDP sockets, inject predicted-port ICE
+  candidates) for symmetric NAT; IPv6-first (audit the candidate filter — do
+  not drop *global* v6); STUN-based NAT pre-classification at startup reported
+  to the registry; relay-then-upgrade later.
+- Future: ed25519 proxy identity (sign announcements), BEP 44 endpoint
+  announcements via the `bittorrent-dht` already bundled with WebTorrent.
+
+All of this must stay deployment-agnostic (HA addon, bare npm, Docker).
+
+## Disk hygiene (open item — torrent data is NOT cleaned up today)
+
+HLS segments are handled (`hls-session-manager.js`: idle TTL, `disposeSession`,
+`disposeAll`). Torrent data is NOT: `new WebTorrent()` in `torrent-pool.js` uses
+the default FS store under `os.tmpdir()`, there is no `client.remove()` /
+`torrent.destroy()`, `deselect()` only stops further download, nothing sweeps
+orphans at startup, and `TorrentPool` is not wired into the `onClose` shutdown
+hook (`server.js` only disposes HLS sessions on close — see `cli.js` shutdown →
+`app.close()` → `onClose`).
+
+Level 1 (do first): `client.remove(torrent, { destroyStore: true })` on last-
+file refcount 0 + idle TTL; startup sweep of orphaned store dirs; wire
+`TorrentPool` teardown into the `onClose` hook so closing the proxy cleans up
+(not only startup); global disk cap with LRU eviction of whole torrents.
+Level 2 (research): sliding-window chunk store. Full rationale in the parent
+`../CLAUDE.md` "Disk hygiene" section.
+
+## Cloud proxy
+
+The same proxy code also runs as the company-hosted fallback when the user
+pool can't serve a viewer. Keep the proxy host-agnostic so it runs unchanged on
+rented infra (flat-rate/unmetered bandwidth — Hetzner dedicated / OVH; NOT
+metered-egress clouds). Provider/economics analysis in the parent
+`../CLAUDE.md` "Cloud proxy" section.
+
 ## Changelog
 
 Every behavioural change must be recorded in `CHANGELOG.md` — add an entry under

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@torrent-tv/proxy",
-  "version": "2.9.13",
+  "version": "2.9.15",
   "description": "Torrent proxy client that exposes webseed-like HTTP stream endpoint.",
   "license": "GPL-3.0-or-later",
   "publishConfig": {

package/server.js

@@ -154,7 +154,10 @@ export async function startProxyServer({ host, port, transcodeAudio, ffmpegBin }
   });
 
   app.addHook("onClose", async () => {
+    // Order matters: stop the ffmpeg readers (HLS sessions) before destroying
+    // the torrents whose files they read from, then remove the torrent data.
     await hlsSessionManager.disposeAll();
+    await torrentPool.destroyAll();
   });
 
   await app.listen({ host, port: selectedPort });

package/services/torrent-pool.js

@@ -15,6 +15,11 @@ import { logger } from "../utils/logger.js";
 // small enough not to make "everything critical" (which defeats prioritization).
 const PRIORITY_WINDOW_BYTES = 8 * 1024 * 1024;
 
+// The file's header/index region the codec probe needs (phase 1). Must match
+// the ranges prefetchFileEdges fetches: leading bytes + trailing bytes.
+const HEADER_HEAD_BYTES = 256 * 1024;
+const HEADER_TAIL_BYTES = 2 * 1024 * 1024;
+
 /**
  * Decode a raw torrent source value into the format expected by WebTorrent.
  *
@@ -217,14 +222,68 @@ export class TorrentPool {
       return { ...base, fileProgress: null, fileDownloaded: null, fileLength: null };
     }
 
+    const header = this.#getHeaderRangeProgress(torrent, file);
+
     return {
       ...base,
       fileProgress: typeof file.progress === "number" ? file.progress : 0,
       fileDownloaded: typeof file.downloaded === "number" ? file.downloaded : 0,
-      fileLength: typeof file.length === "number" ? file.length : 0
+      fileLength: typeof file.length === "number" ? file.length : 0,
+      // Phase-1 progress: how much of the header/index region (the bytes the
+      // codec probe needs before transcoding can start) is downloaded. Counted
+      // by whole pieces from the torrent bitfield, so it advances coarsely
+      // (piece granularity). Null when the bitfield/piece info is unavailable.
+      headerBytes: header ? header.totalBytes : null,
+      headerDownloadedBytes: header ? header.downloadedBytes : null
     };
   }
 
+  /**
+   * Count, by whole torrent pieces, how many bytes of a file's header/index
+   * region (leading {@link HEADER_HEAD_BYTES} + trailing {@link HEADER_TAIL_BYTES})
+   * are downloaded. Used to show progress toward the codec-probe phase.
+   *
+   * @param {import("webtorrent").Torrent} torrent
+   * @param {import("webtorrent").TorrentFile} file
+   * @returns {{ totalBytes: number, downloadedBytes: number } | null}
+   */
+  #getHeaderRangeProgress(torrent, file) {
+    const pieceLength = Number(torrent?.pieceLength);
+    const bitfield = torrent?.bitfield;
+    const fileLength = Number(file?.length);
+    if (
+      !Number.isFinite(pieceLength) || pieceLength <= 0 ||
+      !bitfield || typeof bitfield.get !== "function" ||
+      !Number.isFinite(fileLength) || fileLength <= 0
+    ) {
+      return null;
+    }
+    const fileOffset = Number.isFinite(file.offset) ? file.offset : 0;
+    const headEnd = Math.min(HEADER_HEAD_BYTES, fileLength) - 1;
+    const ranges = [[0, headEnd]];
+    const tailStart = Math.max(headEnd + 1, fileLength - HEADER_TAIL_BYTES);
+    if (tailStart <= fileLength - 1) {
+      ranges.push([tailStart, fileLength - 1]);
+    }
+    const pieces = new Set();
+    for (const [start, end] of ranges) {
+      const first = Math.floor((fileOffset + start) / pieceLength);
+      const last = Math.floor((fileOffset + end) / pieceLength);
+      for (let piece = first; piece <= last; piece += 1) {
+        pieces.add(piece);
+      }
+    }
+    let totalBytes = 0;
+    let downloadedBytes = 0;
+    for (const piece of pieces) {
+      totalBytes += pieceLength;
+      if (bitfield.get(piece)) {
+        downloadedBytes += pieceLength;
+      }
+    }
+    return { totalBytes, downloadedBytes };
+  }
+
   /**
    * Pre-fetch the leading and trailing bytes of a torrent file so that
    * WebTorrent prioritises the pieces that contain file headers and footers.
@@ -364,4 +423,56 @@ export class TorrentPool {
       // Best effort — never break streaming because prioritization failed.
     }
   }
+
+  /**
+   * Destroy every torrent together with its on-disk store, then tear down the
+   * WebTorrent client. Called from the proxy's graceful-shutdown `onClose`
+   * hook so downloaded torrent data does not linger under `os.tmpdir()` after
+   * the process stops.
+   *
+   * `client.destroy()` on its own destroys the torrents but only *closes* their
+   * stores (data stays on disk), so each torrent is removed explicitly with
+   * `{ destroyStore: true }` first. Best-effort: never rejects and never hangs
+   * on a single store-removal error during shutdown.
+   *
+   * @returns {Promise<void>}
+   */
+  async destroyAll() {
+    if (!this.client || this.client.destroyed) {
+      return;
+    }
+
+    // Destroy each torrent with its store so downloaded pieces are removed
+    // from disk. This also removes the torrent from `client.torrents`, so the
+    // subsequent `client.destroy()` only tears down the client internals.
+    const torrents = [...this.client.torrents];
+    await Promise.all(
+      torrents.map(
+        (torrent) =>
+          new Promise((resolve) => {
+            try {
+              torrent.destroy({ destroyStore: true }, () => resolve());
+            } catch (error) {
+              const message = error instanceof Error ? error.message : String(error);
+              logger.warn(`failed to destroy torrent store: ${message}`);
+              resolve();
+            }
+          })
+      )
+    );
+
+    // Tear down the client itself (DHT, connection pool, TCP server).
+    await new Promise((resolve) => {
+      try {
+        this.client.destroy(() => resolve());
+      } catch (error) {
+        const message = error instanceof Error ? error.message : String(error);
+        logger.warn(`failed to destroy WebTorrent client: ${message}`);
+        resolve();
+      }
+    });
+
+    this.torrents.clear();
+    this.#pending.clear();
+  }
 }