@torrent-tv/proxy 2.9.93 → 2.9.96
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +15 -0
- package/package.json +2 -3
- package/routes/stream/get.js +62 -2
- package/services/hls-session-manager.js +59 -0
- package/services/piece-store/piece-lru.js +71 -0
- package/services/piece-store/shared-piece-store.js +57 -0
- package/services/playback-planner.js +17 -0
- package/services/torrent-pool.js +75 -2
- package/services/torrent-worker/client.js +448 -447
- package/services/torrent-worker/piece-reader.js +124 -22
- package/services/torrent-worker/worker.js +412 -410
- package/test/piece-lru.test.js +64 -0
- package/test/read-window.test.js +9 -5
- package/test/upload-hurry.test.js +66 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,18 @@
|
|
|
1
|
+
## 2.9.96
|
|
2
|
+
|
|
3
|
+
- **New**: The proxy uploads generously at the two moments a viewer is provably waiting — when a torrent is added, and when the viewer seeks — for 25 s, which is two of BitTorrent's unchoke cycles. Peers serve those who serve them: each re-ranks its takers about every 10 s and opens a few slots to whoever uploaded most, plus one at random, so uploading a token 8-50 KB/s means being picked at random, one slot per cycle. Measured on a session where 96 peers were already connected within 2 s: 64 KB/s after 2 s, 1.6 MB/s after 4 s, 4.8 MB/s after 8 s — and the 16 MB the codec probe needs took **8.36 s of the 11.46 s** before playback could start. The existing reciprocity boost could not help, because it waits for the download to be all but dead (below 200 KB/s) with peers visibly choking us, and a ramp is neither: in that same session it first moved the limit 13.3 s after the torrent was added and reached the generous rate at 43.7 s, both after the wait they were meant to shorten. Seeding policy is otherwise unchanged — near-silence when nothing is being watched, a token upload while reading.
|
|
4
|
+
- **New**: Every encode run logs the exact ffmpeg command line. A failure is otherwise reported with ffmpeg's message and nothing about what it was asked to do, and the two are not always deducible from each other: a run died with `Cannot write moov atom before AC3 packets` although both muxing paths were then verified to handle a copied AC-3 track on that very host, so the arguments that run actually received are the missing evidence.
|
|
5
|
+
|
|
6
|
+
## 2.9.95
|
|
7
|
+
|
|
8
|
+
- **New**: The rest of the file is downloaded in the background — but only while that cannot cost the viewer anything. The tail enters the download set at the lowest priority ONLY when every piece of the reader's near window is already on hand, and leaves it the moment one is missing, the window slides onto undownloaded content, or a seek moves it. Relying on priority ordering alone would be weaker: it decides which selection a wire is offered first, not what that wire already has outstanding, so a seek would still queue behind whatever was in flight. What it buys is a file that ends up downloaded while it is watched, making every later seek into it instant.
|
|
9
|
+
- **Fix**: `/stream` answers when the torrent is not ready instead of holding the connection open in silence. Reproduced 2026-08-04 with a magnet whose metadata never arrived: a ranged GET and a HEAD both returned nothing at all for the full 30 s the client was willing to wait — no status, no headers, and nothing in the log — because the route awaited `getTorrent` with no bound and adding a magnet takes as long as its metadata does. The wait is now capped at 10 s and answers a retryable 503; the add itself continues, so the next attempt is likely to find it ready.
|
|
10
|
+
- **New**: The read-ahead window is sized in seconds of playback instead of bytes. A flat 32 MB is about half a minute of a 1080p film and roughly four seconds of a disc remux, and the torrent thread cannot tell the difference — it knows only bytes. The transcode session, which knows both the duration and the file size, now works out the file's own byte rate, asks for 30 seconds of it (bounded to 16-96 MB against an odd rate) and puts the figure on the ffmpeg input URL. Without it the reader keeps its previous default.
|
|
11
|
+
- **Fix**: Everything a reader needs next is marked urgent, not just the piece it is standing on. `critical` is what enables hotswap — a block reserved by a slow peer is re-requested from a faster one — and with pieces of 4 MB the old rule (`min(1 MB / pieceLength, 2)`) marked exactly one. Measured 2026-08-04: the first segment after a seek took 7.2 s while its four pieces arrived one after another at ~2.2 MB/s, with single-piece waits of 1.3 s and 2.8 s. This is not the earlier behaviour returning — that marked the whole requested range, which for ffmpeg's input is every piece to the end of the file.
|
|
12
|
+
- **Fix**: The piece a viewer is about to watch is no longer as evictable as one fetched forty minutes ahead. Each reader declares its window to the store and the eviction order takes something else while it can; measured in a session where the encoder ran ahead, the hit rate fell from 100% to 45.7% with 221 pieces read back from disk. It is a preference, not a hold: when everything resident is declared, protection yields, because at its smallest budget the store guarantees only two resident pieces and an absolute hold would deadlock it. Pins are unchanged — a piece being read now can never be taken.
|
|
13
|
+
- **New**: A seek into content already downloaded but spilled to disk brings the whole window back at once. A spilled piece used to be revived only when the reader reached it, one disk round trip at a time, in step with decoding. The disk is local, so the window can be restored while the reader is still on its first piece.
|
|
14
|
+
- **New**: The playback plan reports where its time goes — waiting for the torrent, waiting for the file's head and tail, and the codec probe itself, with the number of probe attempts. Everything from the transcode-session request onwards was already broken down by `cold-start`, but the plan runs before that and was one opaque wait: a field session spent 5.7 s in it with the torrent already in the store and the probe cached, and nothing said which part was slow.
|
|
15
|
+
|
|
1
16
|
## 2.9.93
|
|
2
17
|
|
|
3
18
|
- **Fix**: A seek could kill playback outright. Restarting at a position that lands exactly on a keyframe leaves a floating-point residue — `seekSeconds - snappedKeyframe` came out as `3.3333333249174757e-7` — and `String()` renders anything below 1e-6 in exponential notation, which ffmpeg's duration parser rejects: `Invalid duration for option ss`. The run died on startup, and from then on every segment request answered 500. Time arguments are now formatted in fixed notation, and a residue under a millisecond is dropped rather than passed on, because it is not a real offset.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@torrent-tv/proxy",
|
|
3
|
-
"version": "2.9.
|
|
3
|
+
"version": "2.9.96",
|
|
4
4
|
"description": "Torrent proxy client that exposes webseed-like HTTP stream endpoint.",
|
|
5
5
|
"license": "GPL-3.0-or-later",
|
|
6
6
|
"publishConfig": {
|
|
@@ -16,8 +16,7 @@
|
|
|
16
16
|
"major": "npm whoami && npm version major && npm publish && git push --follow-tags",
|
|
17
17
|
"start": "node ./bin/cli.js",
|
|
18
18
|
"dev": "node --inspect=0 --experimental-network-inspection ./bin/cli.js",
|
|
19
|
-
"test": "node --test"
|
|
20
|
-
"prepublishOnly": "npm test"
|
|
19
|
+
"test": "node --test"
|
|
21
20
|
},
|
|
22
21
|
"dependencies": {
|
|
23
22
|
"@fastify/cors": "^11.2.0",
|
package/routes/stream/get.js
CHANGED
|
@@ -28,6 +28,46 @@ function getSourceParams(query, sourceRegistry) {
|
|
|
28
28
|
return { sourceType, source };
|
|
29
29
|
}
|
|
30
30
|
|
|
31
|
+
/**
|
|
32
|
+
* How long a request may wait for a torrent that is still being added.
|
|
33
|
+
*
|
|
34
|
+
* Adding a magnet takes as long as its metadata does — seconds when peers
|
|
35
|
+
* answer, forever when none do — and `getTorrent` waits for it. Awaiting that
|
|
36
|
+
* with no bound is what made this route answer nothing at all.
|
|
37
|
+
*/
|
|
38
|
+
const TORRENT_READY_TIMEOUT_MS = 10_000;
|
|
39
|
+
|
|
40
|
+
/**
|
|
41
|
+
* The torrent for a source, or a `TORRENT_NOT_READY` error once the wait has
|
|
42
|
+
* gone on long enough to be worth reporting.
|
|
43
|
+
*
|
|
44
|
+
* The underlying add is NOT cancelled: it keeps running and warms the pool, so
|
|
45
|
+
* the client's next attempt is likely to find it ready.
|
|
46
|
+
*
|
|
47
|
+
* @param {import("../../services/torrent-pool.js").TorrentPool} torrentPool
|
|
48
|
+
* @param {string} sourceType
|
|
49
|
+
* @param {string} source
|
|
50
|
+
* @returns {Promise<import("webtorrent").Torrent>}
|
|
51
|
+
*/
|
|
52
|
+
async function waitForTorrent(torrentPool, sourceType, source) {
|
|
53
|
+
let timer = null;
|
|
54
|
+
const expiry = new Promise((_resolve, reject) => {
|
|
55
|
+
timer = setTimeout(() => {
|
|
56
|
+
const error = new Error("Torrent metadata is not available yet.");
|
|
57
|
+
error.code = "TORRENT_NOT_READY";
|
|
58
|
+
reject(error);
|
|
59
|
+
}, TORRENT_READY_TIMEOUT_MS);
|
|
60
|
+
timer.unref?.();
|
|
61
|
+
});
|
|
62
|
+
try {
|
|
63
|
+
return await Promise.race([torrentPool.getTorrent(sourceType, source), expiry]);
|
|
64
|
+
} finally {
|
|
65
|
+
if (timer) {
|
|
66
|
+
clearTimeout(timer);
|
|
67
|
+
}
|
|
68
|
+
}
|
|
69
|
+
}
|
|
70
|
+
|
|
31
71
|
/**
|
|
32
72
|
* Stream a torrent file over HTTP with byte-range support.
|
|
33
73
|
*
|
|
@@ -51,8 +91,18 @@ export async function handleStreamGet(req, reply, { sourceRegistry, torrentPool
|
|
|
51
91
|
|
|
52
92
|
let torrent;
|
|
53
93
|
try {
|
|
54
|
-
torrent = await torrentPool
|
|
94
|
+
torrent = await waitForTorrent(torrentPool, sourceType, source);
|
|
55
95
|
} catch (error) {
|
|
96
|
+
if (error instanceof Error && error.code === "TORRENT_NOT_READY") {
|
|
97
|
+
// Say so, rather than holding the connection until the client gives up.
|
|
98
|
+
// Reproduced 2026-08-04 on a magnet whose metadata never arrived: both a
|
|
99
|
+
// ranged GET and a HEAD returned nothing at all for the full 30 s the
|
|
100
|
+
// probe was willing to wait, and the route had written neither a status
|
|
101
|
+
// nor a header — from the client that is indistinguishable from the proxy
|
|
102
|
+
// having died, and it left no trace in the log either.
|
|
103
|
+
reply.header("Retry-After", "1");
|
|
104
|
+
return reply.code(503).send({ error: "Torrent metadata is not available yet." });
|
|
105
|
+
}
|
|
56
106
|
const message = error instanceof Error ? error.message : String(error);
|
|
57
107
|
return reply.code(500).send({ error: `Failed to load torrent source: ${message}` });
|
|
58
108
|
}
|
|
@@ -106,8 +156,18 @@ export async function handleStreamGet(req, reply, { sourceRegistry, torrentPool
|
|
|
106
156
|
// no copy on either thread, at the cost of doing the writing by hand, because
|
|
107
157
|
// only the write callback tells us when a piece may be released. Falls back to
|
|
108
158
|
// the ordinary stream for sources without a shared pool.
|
|
159
|
+
// How far ahead of its own read head this reader should ask the swarm for.
|
|
160
|
+
// Supplied by whoever knows the media's byte rate — the transcode session
|
|
161
|
+
// puts it on the ffmpeg input URL, sized in seconds of playback — because
|
|
162
|
+
// this thread knows only bytes, and 32 MB is half a minute of a 1080p film
|
|
163
|
+
// but four seconds of a disc remux. Absent or unusable, the reader's own
|
|
164
|
+
// default stands.
|
|
165
|
+
const windowBytesRaw = Number(req.query.windowBytes);
|
|
166
|
+
const windowBytes =
|
|
167
|
+
Number.isFinite(windowBytesRaw) && windowBytesRaw > 0 ? windowBytesRaw : undefined;
|
|
168
|
+
|
|
109
169
|
const fragments = typeof file.createFragmentReader === "function"
|
|
110
|
-
? file.createFragmentReader({ start, end })
|
|
170
|
+
? file.createFragmentReader({ start, end, windowBytes })
|
|
111
171
|
: null;
|
|
112
172
|
|
|
113
173
|
if (fragments) {
|
|
@@ -67,6 +67,18 @@ const RESTART_COOLDOWN_MS = 500;
|
|
|
67
67
|
// re-encoded video but a whole keyframe interval on the copy path. Generous
|
|
68
68
|
// enough that ordinary watching never touches it: the encoder fills two minutes
|
|
69
69
|
// ahead, stops, and is released as soon as the viewer has spent a minute of it.
|
|
70
|
+
// How far ahead of its own read head a reader asks the swarm for, expressed in
|
|
71
|
+
// seconds of PLAYBACK. The torrent thread can only think in bytes, and a fixed
|
|
72
|
+
// byte window is wrong at both ends of the range: 32 MB is half a minute of a
|
|
73
|
+
// 1080p film and about four seconds of a disc remux. Duration and file size are
|
|
74
|
+
// both known here, so the window is sized where the knowledge is and sent down
|
|
75
|
+
// on the ffmpeg input URL.
|
|
76
|
+
const READ_WINDOW_SECONDS = 30;
|
|
77
|
+
// Bounds, so a wrong or unusual byte rate cannot ask for something absurd. The
|
|
78
|
+
// floor keeps a few pieces in flight on a low-bitrate file; the ceiling keeps
|
|
79
|
+
// one reader from claiming more than a fraction of the piece store.
|
|
80
|
+
const READ_WINDOW_MIN_BYTES = 16 * 1024 * 1024;
|
|
81
|
+
const READ_WINDOW_MAX_BYTES = 96 * 1024 * 1024;
|
|
70
82
|
const LOOKAHEAD_PAUSE_SECONDS = 120;
|
|
71
83
|
const LOOKAHEAD_RESUME_SECONDS = 60;
|
|
72
84
|
// Seek debounce. A far (out-of-window) segment request is a server-side seek.
|
|
@@ -979,6 +991,15 @@ export class HlsSessionManager {
|
|
|
979
991
|
const outputFps = chooseOutputFps(mediaInfo.fps);
|
|
980
992
|
const hasDuration = Number.isFinite(durationSeconds) && durationSeconds > 0;
|
|
981
993
|
const logName = normalizeLogFileName(fileName, fileIndex);
|
|
994
|
+
|
|
995
|
+
// Size the reader's window in seconds of playback rather than bytes. Needs
|
|
996
|
+
// the file's own average byte rate, which is size ÷ duration; the size
|
|
997
|
+
// comes from the same stats call the realtime budget uses. Best effort —
|
|
998
|
+
// without it the reader keeps its own byte default.
|
|
999
|
+
const readWindowBytes = await this.#readWindowBytesFor(sourceKey, fileIndex, durationSeconds);
|
|
1000
|
+
if (readWindowBytes > 0) {
|
|
1001
|
+
inputUrl.searchParams.set("windowBytes", String(readWindowBytes));
|
|
1002
|
+
}
|
|
982
1003
|
if (!hasDuration) {
|
|
983
1004
|
logger.warn(
|
|
984
1005
|
`transcode ${sessionId}: could not probe duration; falling back to ` +
|
|
@@ -1608,6 +1629,35 @@ export class HlsSessionManager {
|
|
|
1608
1629
|
*
|
|
1609
1630
|
* @returns {void}
|
|
1610
1631
|
*/
|
|
1632
|
+
/**
|
|
1633
|
+
* The read-ahead window for a file, in bytes, sized from how many seconds of
|
|
1634
|
+
* playback it holds.
|
|
1635
|
+
*
|
|
1636
|
+
* @param {string} sourceKey
|
|
1637
|
+
* @param {number} fileIndex
|
|
1638
|
+
* @param {number} durationSeconds
|
|
1639
|
+
* @returns {Promise<number>} Zero when the byte rate cannot be established,
|
|
1640
|
+
* which leaves the reader on its own default.
|
|
1641
|
+
*/
|
|
1642
|
+
async #readWindowBytesFor(sourceKey, fileIndex, durationSeconds) {
|
|
1643
|
+
if (!this.getSourceStats || !Number.isFinite(durationSeconds) || durationSeconds <= 0) {
|
|
1644
|
+
return 0;
|
|
1645
|
+
}
|
|
1646
|
+
let fileLength = 0;
|
|
1647
|
+
try {
|
|
1648
|
+
const stats = await this.getSourceStats(sourceKey, fileIndex);
|
|
1649
|
+
fileLength = Number(stats?.fileLength);
|
|
1650
|
+
} catch {
|
|
1651
|
+
return 0;
|
|
1652
|
+
}
|
|
1653
|
+
if (!Number.isFinite(fileLength) || fileLength <= 0) {
|
|
1654
|
+
return 0;
|
|
1655
|
+
}
|
|
1656
|
+
const bytesPerSecond = fileLength / durationSeconds;
|
|
1657
|
+
const wanted = Math.round(bytesPerSecond * READ_WINDOW_SECONDS);
|
|
1658
|
+
return Math.min(READ_WINDOW_MAX_BYTES, Math.max(READ_WINDOW_MIN_BYTES, wanted));
|
|
1659
|
+
}
|
|
1660
|
+
|
|
1611
1661
|
#enforceLookAhead() {
|
|
1612
1662
|
for (const session of this.sessionsById.values()) {
|
|
1613
1663
|
this.#enforceLookAheadFor(session);
|
|
@@ -2075,6 +2125,15 @@ export class HlsSessionManager {
|
|
|
2075
2125
|
);
|
|
2076
2126
|
}
|
|
2077
2127
|
|
|
2128
|
+
// The exact command, every run. An encode failure is otherwise reported
|
|
2129
|
+
// with ffmpeg's message and nothing about what it was asked to do, and the
|
|
2130
|
+
// two are not always deducible from each other: 2026-08-04 a run died with
|
|
2131
|
+
// "Cannot write moov atom before AC3 packets" although both muxing paths
|
|
2132
|
+
// were verified to handle a copied AC-3 track on this very host, so the
|
|
2133
|
+
// arguments that run actually received are the missing evidence. One line
|
|
2134
|
+
// per run, and a run happens at most every few seconds.
|
|
2135
|
+
logger.info(`transcode ${session.id} ffmpeg ${args.join(" ")}`);
|
|
2136
|
+
|
|
2078
2137
|
const ffmpeg = spawn(this.ffmpegBin, args, {
|
|
2079
2138
|
cwd: session.dirPath,
|
|
2080
2139
|
stdio: ["ignore", "pipe", "pipe"]
|
|
@@ -27,6 +27,21 @@ export class PieceLru {
|
|
|
27
27
|
#order = new Set();
|
|
28
28
|
/** Piece index → number of readers currently holding it. */
|
|
29
29
|
#pins = new Map();
|
|
30
|
+
/**
|
|
31
|
+
* Reader id → the piece range it expects to read next.
|
|
32
|
+
*
|
|
33
|
+
* Recency alone does not describe this. A reader walking a film touches its
|
|
34
|
+
* pieces once, so the piece the decoder will want in two seconds looks
|
|
35
|
+
* exactly as stale as one fetched forty minutes ago and never read again —
|
|
36
|
+
* and with the encoder running ahead of the viewer, the second kind is what
|
|
37
|
+
* fills the store. Measured 2026-08-04: the hit rate fell from 100% to 45.7%
|
|
38
|
+
* with 221 pieces read back from disk in one session.
|
|
39
|
+
*
|
|
40
|
+
* A preference, not a pin. At the smallest budget the store guarantees only
|
|
41
|
+
* two resident pieces, so a hard hold on a window would deadlock it; when
|
|
42
|
+
* nothing unprotected is left, protection is ignored rather than obeyed.
|
|
43
|
+
*/
|
|
44
|
+
#protected = new Map();
|
|
30
45
|
#capacity;
|
|
31
46
|
|
|
32
47
|
/**
|
|
@@ -143,6 +158,16 @@ export class PieceLru {
|
|
|
143
158
|
* @returns {number | null}
|
|
144
159
|
*/
|
|
145
160
|
evictionCandidate() {
|
|
161
|
+
// First choice: the least recently used piece nobody is reading and nobody
|
|
162
|
+
// is about to read.
|
|
163
|
+
for (const index of this.#order) {
|
|
164
|
+
if (!this.#pins.has(index) && !this.#isProtected(index)) {
|
|
165
|
+
return index;
|
|
166
|
+
}
|
|
167
|
+
}
|
|
168
|
+
// Nothing spare left. Protection yields — it is a preference, and refusing
|
|
169
|
+
// here would leave the store unable to admit anything at all. Pins do not
|
|
170
|
+
// yield: a piece being read now cannot have its memory taken away.
|
|
146
171
|
for (const index of this.#order) {
|
|
147
172
|
if (!this.#pins.has(index)) {
|
|
148
173
|
return index;
|
|
@@ -151,6 +176,52 @@ export class PieceLru {
|
|
|
151
176
|
return null;
|
|
152
177
|
}
|
|
153
178
|
|
|
179
|
+
/**
|
|
180
|
+
* Declare the pieces a reader expects to need next, replacing whatever it
|
|
181
|
+
* declared before. Ranges from different readers add up.
|
|
182
|
+
*
|
|
183
|
+
* @param {string|number} readerId - Identity of the reader, so its own range
|
|
184
|
+
* is replaced rather than accumulated.
|
|
185
|
+
* @param {number} from - First piece, inclusive.
|
|
186
|
+
* @param {number} to - Last piece, inclusive.
|
|
187
|
+
* @returns {void}
|
|
188
|
+
*/
|
|
189
|
+
protect(readerId, from, to) {
|
|
190
|
+
if (!Number.isInteger(from) || !Number.isInteger(to) || to < from) {
|
|
191
|
+
return;
|
|
192
|
+
}
|
|
193
|
+
this.#protected.set(readerId, { from, to });
|
|
194
|
+
}
|
|
195
|
+
|
|
196
|
+
/**
|
|
197
|
+
* Drop a reader's declared range. Must be called when the reader ends, or its
|
|
198
|
+
* window keeps occupying the store for a reader that no longer exists.
|
|
199
|
+
*
|
|
200
|
+
* @param {string|number} readerId
|
|
201
|
+
* @returns {void}
|
|
202
|
+
*/
|
|
203
|
+
unprotect(readerId) {
|
|
204
|
+
this.#protected.delete(readerId);
|
|
205
|
+
}
|
|
206
|
+
|
|
207
|
+
/** How many readers currently declare a range. */
|
|
208
|
+
get protectedCount() {
|
|
209
|
+
return this.#protected.size;
|
|
210
|
+
}
|
|
211
|
+
|
|
212
|
+
/**
|
|
213
|
+
* @param {number} index
|
|
214
|
+
* @returns {boolean}
|
|
215
|
+
*/
|
|
216
|
+
#isProtected(index) {
|
|
217
|
+
for (const range of this.#protected.values()) {
|
|
218
|
+
if (index >= range.from && index <= range.to) {
|
|
219
|
+
return true;
|
|
220
|
+
}
|
|
221
|
+
}
|
|
222
|
+
return false;
|
|
223
|
+
}
|
|
224
|
+
|
|
154
225
|
/**
|
|
155
226
|
* Whether admitting one more piece would exceed the capacity.
|
|
156
227
|
*
|
|
@@ -542,6 +542,63 @@ export class SharedPieceStore {
|
|
|
542
542
|
* @returns {Promise<{ offset: number, length: number } | null>} `null` when
|
|
543
543
|
* the store holds no such piece, in memory or on disk.
|
|
544
544
|
*/
|
|
545
|
+
/**
|
|
546
|
+
* Declare the pieces a reader is about to need, so eviction takes something
|
|
547
|
+
* else while it can. Replaces that reader's previous declaration.
|
|
548
|
+
*
|
|
549
|
+
* @param {string|number} readerId
|
|
550
|
+
* @param {number} from - First piece, inclusive.
|
|
551
|
+
* @param {number} to - Last piece, inclusive.
|
|
552
|
+
* @returns {void}
|
|
553
|
+
*/
|
|
554
|
+
protectRange(readerId, from, to) {
|
|
555
|
+
this.#lru.protect(readerId, from, to);
|
|
556
|
+
}
|
|
557
|
+
|
|
558
|
+
/**
|
|
559
|
+
* Forget a reader's declaration. Call it when the reader ends.
|
|
560
|
+
*
|
|
561
|
+
* @param {string|number} readerId
|
|
562
|
+
* @returns {void}
|
|
563
|
+
*/
|
|
564
|
+
releaseProtection(readerId) {
|
|
565
|
+
this.#lru.unprotect(readerId);
|
|
566
|
+
}
|
|
567
|
+
|
|
568
|
+
/**
|
|
569
|
+
* Bring back into memory, in parallel and without waiting, the pieces of a
|
|
570
|
+
* range that have been spilled to disk.
|
|
571
|
+
*
|
|
572
|
+
* A piece is otherwise revived only when the reader arrives at it, one at a
|
|
573
|
+
* time and in step with decoding, so a seek backward into content already
|
|
574
|
+
* downloaded pays a disk round trip per piece. The disk is local; the whole
|
|
575
|
+
* window can be brought back at once while the reader is still on its first
|
|
576
|
+
* piece.
|
|
577
|
+
*
|
|
578
|
+
* Bounded, because each revival needs a slot and unbounded revival of a
|
|
579
|
+
* window larger than the store would simply thrash. Errors are swallowed: a
|
|
580
|
+
* failed warm-up costs nothing, the reader will ask for the piece properly.
|
|
581
|
+
*
|
|
582
|
+
* @param {number} from - First piece, inclusive.
|
|
583
|
+
* @param {number} to - Last piece, inclusive.
|
|
584
|
+
* @param {number} [limit] - Most pieces to revive at once.
|
|
585
|
+
* @returns {number} How many revivals were started.
|
|
586
|
+
*/
|
|
587
|
+
warmRange(from, to, limit = Math.max(1, Math.floor(this.#capacity / 4))) {
|
|
588
|
+
if (this.#closed || !Number.isInteger(from) || !Number.isInteger(to)) {
|
|
589
|
+
return 0;
|
|
590
|
+
}
|
|
591
|
+
let started = 0;
|
|
592
|
+
for (let index = from; index <= to && started < limit; index += 1) {
|
|
593
|
+
if (this.#slotOf.has(index) || !this.#disk.has(index)) {
|
|
594
|
+
continue;
|
|
595
|
+
}
|
|
596
|
+
started += 1;
|
|
597
|
+
void this.reside(index).catch(() => undefined);
|
|
598
|
+
}
|
|
599
|
+
return started;
|
|
600
|
+
}
|
|
601
|
+
|
|
545
602
|
async reside(index) {
|
|
546
603
|
if (this.#closed) {
|
|
547
604
|
throw new Error("Piece store is closed.");
|
|
@@ -7,6 +7,7 @@
|
|
|
7
7
|
*/
|
|
8
8
|
|
|
9
9
|
import { spawn } from "node:child_process";
|
|
10
|
+
import { logger } from "../utils/logger.js";
|
|
10
11
|
import {
|
|
11
12
|
parseFfmpegDurationSeconds,
|
|
12
13
|
parseFfmpegStartTimeSeconds,
|
|
@@ -312,6 +313,14 @@ export function createPlaybackPlanner({
|
|
|
312
313
|
if (cached) {
|
|
313
314
|
return cached;
|
|
314
315
|
}
|
|
316
|
+
// Where the time before playback goes. `cold-start` already breaks down
|
|
317
|
+
// everything from the transcode-session request onwards, but the plan
|
|
318
|
+
// runs BEFORE that and was a single opaque wait: a field session spent
|
|
319
|
+
// 5.7 s here on a torrent already in the store, with the codec probe
|
|
320
|
+
// cached, and nothing said which part of it was slow.
|
|
321
|
+
const planEntryMs = Date.now();
|
|
322
|
+
let torrentReadyMs = 0;
|
|
323
|
+
let edgesReadyMs = 0;
|
|
315
324
|
|
|
316
325
|
const sourceRecord = sourceRegistry.get(sourceKey);
|
|
317
326
|
if (!sourceRecord) {
|
|
@@ -321,6 +330,7 @@ export function createPlaybackPlanner({
|
|
|
321
330
|
}
|
|
322
331
|
|
|
323
332
|
const torrent = await torrentPool.getTorrent(sourceRecord.sourceType, sourceRecord.source);
|
|
333
|
+
torrentReadyMs = Date.now() - planEntryMs;
|
|
324
334
|
const file = torrent.files[fileIndex];
|
|
325
335
|
if (!file) {
|
|
326
336
|
const error = new Error("File index was not found in torrent.");
|
|
@@ -354,6 +364,7 @@ export function createPlaybackPlanner({
|
|
|
354
364
|
// plan (file treated as directly playable) sticks permanently for this
|
|
355
365
|
// file, and an unsupported codec like xvid gets copied → black video.
|
|
356
366
|
await torrentPool.prefetchFileEdges(torrent, fileIndex);
|
|
367
|
+
edgesReadyMs = Date.now() - planEntryMs;
|
|
357
368
|
let probe = await probeStreamCodecs({ ffmpegBin, inputUrl: directUrl, userAgent });
|
|
358
369
|
const probeDeadline = Date.now() + Math.max(0, maxWaitMs);
|
|
359
370
|
let attempt = 0;
|
|
@@ -369,6 +380,12 @@ export function createPlaybackPlanner({
|
|
|
369
380
|
}
|
|
370
381
|
const { audioCodec, videoCodec, container, durationSeconds, videoWidth, videoHeight, audioTracks, subtitleTracks } = probe;
|
|
371
382
|
const codecsDetected = audioCodec.length > 0 || videoCodec.length > 0;
|
|
383
|
+
logger.info(
|
|
384
|
+
`plan ${sourceKey.slice(0, 8)}:${fileIndex} torrent-ready=${torrentReadyMs}ms ` +
|
|
385
|
+
`file-edges=${edgesReadyMs - torrentReadyMs}ms probe=${Date.now() - planEntryMs - edgesReadyMs}ms ` +
|
|
386
|
+
`total=${Date.now() - planEntryMs}ms attempts=${attempt + 1} ` +
|
|
387
|
+
`${codecsDetected ? `${videoCodec || "-"}/${audioCodec || "-"}` : "codecs NOT detected (will be polled again)"}`
|
|
388
|
+
);
|
|
372
389
|
|
|
373
390
|
// `mode` is advisory only (audio-codec based). The browser makes the
|
|
374
391
|
// authoritative decision independently per stream via canPlayType /
|
package/services/torrent-pool.js
CHANGED
|
@@ -70,6 +70,28 @@ const UPLOAD_BOOST_BYTES = 512 * 1024; // raised to earn tit-for-tat u
|
|
|
70
70
|
const UPLOAD_STARVING_SPEED_BYTES = 200 * 1024; // download below this (with demand) = starving
|
|
71
71
|
const UPLOAD_CHOKED_WIRE_THRESHOLD = 2; // interested-but-choked wires implying reciprocity
|
|
72
72
|
const UPLOAD_ADJUST_INTERVAL_MS = 5_000;
|
|
73
|
+
/**
|
|
74
|
+
* How long a torrent counts as being in a hurry after it is added, and after
|
|
75
|
+
* the viewer moves to a part of the file that is not downloaded.
|
|
76
|
+
*
|
|
77
|
+
* BitTorrent gives data to peers that give data back: each peer re-ranks whom
|
|
78
|
+
* it serves roughly every 10 s and opens a handful of slots to whoever uploaded
|
|
79
|
+
* most to it, plus one chosen at random. Uploading almost nothing means waiting
|
|
80
|
+
* to be picked at random, one slot per cycle — which is exactly the ramp
|
|
81
|
+
* measured 2026-08-04 on a session with 96 peers already connected: 64 KB/s
|
|
82
|
+
* after 2 s, 1.6 MB/s after 4 s, 4.8 MB/s after 8 s, and the 16 MB the codec
|
|
83
|
+
* probe needs took 8.36 s of the 11.46 s before playback could start.
|
|
84
|
+
*
|
|
85
|
+
* The existing reciprocity boost could not help there: it only fires once the
|
|
86
|
+
* download has all but stopped (below 200 KB/s) with peers visibly choking us,
|
|
87
|
+
* and a ramp is neither. In the same session it first raised the limit 13.3 s
|
|
88
|
+
* after the torrent was added — after the wait it was supposed to shorten — and
|
|
89
|
+
* reached the generous rate at 43.7 s.
|
|
90
|
+
*
|
|
91
|
+
* So the two moments where a viewer is provably waiting get the generous rate
|
|
92
|
+
* outright, for two unchoke cycles, without waiting for evidence of failure.
|
|
93
|
+
*/
|
|
94
|
+
const UPLOAD_HURRY_MS = 25_000;
|
|
73
95
|
|
|
74
96
|
/**
|
|
75
97
|
* Decide the client-wide upload limit (bytes/sec) from the torrents that
|
|
@@ -84,8 +106,13 @@ const UPLOAD_ADJUST_INTERVAL_MS = 5_000;
|
|
|
84
106
|
* are choking us) → boost, to earn unchoke slots.
|
|
85
107
|
* - Otherwise → floor (token upload, avoids an immediate choke without seeding).
|
|
86
108
|
*
|
|
87
|
-
*
|
|
88
|
-
*
|
|
109
|
+
* - Any active torrent in a HURRY — just added, or the viewer has just moved
|
|
110
|
+
* somewhere the file is not downloaded — → boost for {@link UPLOAD_HURRY_MS},
|
|
111
|
+
* because that is when peers must be persuaded to serve us and there is no
|
|
112
|
+
* time to first prove that they are not.
|
|
113
|
+
*
|
|
114
|
+
* @param {Array<{ wires?: Array<{ amInterested?: boolean, peerChoking?: boolean }>, downloadSpeed?: number, done?: boolean, name?: string, hurryUntil?: number }>} activeTorrents
|
|
115
|
+
* @param {{ floor?: number, idleFloor?: number, boost?: number, starvingSpeed?: number, chokedThreshold?: number, now?: number }} [opts]
|
|
89
116
|
* @returns {{ bytesPerSec: number, reason: string }}
|
|
90
117
|
*/
|
|
91
118
|
export function decideUploadLimit(activeTorrents, opts = {}) {
|
|
@@ -94,11 +121,23 @@ export function decideUploadLimit(activeTorrents, opts = {}) {
|
|
|
94
121
|
const boost = opts.boost ?? UPLOAD_BOOST_BYTES;
|
|
95
122
|
const starvingSpeed = opts.starvingSpeed ?? UPLOAD_STARVING_SPEED_BYTES;
|
|
96
123
|
const chokedThreshold = opts.chokedThreshold ?? UPLOAD_CHOKED_WIRE_THRESHOLD;
|
|
124
|
+
const now = opts.now ?? Date.now();
|
|
97
125
|
|
|
98
126
|
if (!Array.isArray(activeTorrents) || activeTorrents.length === 0) {
|
|
99
127
|
return { bytesPerSec: idleFloor, reason: "idle: minimal keep-alive (0 blocks the swarm in wt3.x)" };
|
|
100
128
|
}
|
|
101
129
|
|
|
130
|
+
for (const torrent of activeTorrents) {
|
|
131
|
+
const hurryUntil = typeof torrent?.hurryUntil === "number" ? torrent.hurryUntil : 0;
|
|
132
|
+
if (hurryUntil > now && torrent?.done !== true) {
|
|
133
|
+
const name = typeof torrent?.name === "string" ? torrent.name : "?";
|
|
134
|
+
return {
|
|
135
|
+
bytesPerSec: boost,
|
|
136
|
+
reason: `in a hurry — "${name}" needs data now (${Math.round((hurryUntil - now) / 1000)}s left)`
|
|
137
|
+
};
|
|
138
|
+
}
|
|
139
|
+
}
|
|
140
|
+
|
|
102
141
|
for (const torrent of activeTorrents) {
|
|
103
142
|
const wires = Array.isArray(torrent?.wires) ? torrent.wires : [];
|
|
104
143
|
const chokedInterested = wires.filter(
|
|
@@ -395,6 +434,33 @@ export class TorrentPool {
|
|
|
395
434
|
*
|
|
396
435
|
* @returns {void}
|
|
397
436
|
*/
|
|
437
|
+
/**
|
|
438
|
+
* Note that a torrent needs data now, and act on it immediately.
|
|
439
|
+
*
|
|
440
|
+
* `hurryUntil` is read by {@link decideUploadLimit}; the re-evaluation is
|
|
441
|
+
* what makes it take effect at once, because the adjuster otherwise runs on a
|
|
442
|
+
* 5 s timer and the whole hurry is only 25 s long.
|
|
443
|
+
*
|
|
444
|
+
* @param {import("webtorrent").Torrent} torrent
|
|
445
|
+
* @param {string} why - For the log line, so the two causes are told apart.
|
|
446
|
+
* @returns {void}
|
|
447
|
+
*/
|
|
448
|
+
#markHurry(torrent, why) {
|
|
449
|
+
if (!torrent) {
|
|
450
|
+
return;
|
|
451
|
+
}
|
|
452
|
+
const until = Date.now() + UPLOAD_HURRY_MS;
|
|
453
|
+
if ((torrent.hurryUntil ?? 0) >= until - 1_000) {
|
|
454
|
+
return;
|
|
455
|
+
}
|
|
456
|
+
torrent.hurryUntil = until;
|
|
457
|
+
logger.info(
|
|
458
|
+
`torrent-pool: [${String(torrent.infoHash).slice(0, 8)}] uploading generously for ` +
|
|
459
|
+
`${Math.round(UPLOAD_HURRY_MS / 1000)}s — ${why}`
|
|
460
|
+
);
|
|
461
|
+
this.#adjustUploadLimit();
|
|
462
|
+
}
|
|
463
|
+
|
|
398
464
|
#adjustUploadLimit() {
|
|
399
465
|
if (!this.client || this.client.destroyed || typeof this.client.throttleUpload !== "function") {
|
|
400
466
|
return;
|
|
@@ -477,6 +543,9 @@ export class TorrentPool {
|
|
|
477
543
|
* @returns {void}
|
|
478
544
|
*/
|
|
479
545
|
#attachSwarmDiagnostics(label, torrent) {
|
|
546
|
+
// A torrent nobody has asked for yet does not exist: this is called the
|
|
547
|
+
// moment one is added, which is the moment a viewer started waiting.
|
|
548
|
+
this.#markHurry(torrent, "just added");
|
|
480
549
|
const trackerCount = Array.isArray(torrent.announce) ? torrent.announce.length : 0;
|
|
481
550
|
logger.info(
|
|
482
551
|
`torrent-pool: [${label}] added: files=${torrent.files?.length ?? 0} ` +
|
|
@@ -1084,6 +1153,10 @@ export class TorrentPool {
|
|
|
1084
1153
|
const isJump =
|
|
1085
1154
|
previousStart === undefined || Math.abs(safeStart - previousStart) > PRIORITY_WINDOW_BYTES;
|
|
1086
1155
|
if (isJump) {
|
|
1156
|
+
// A jump is a seek. Whatever the swarm was giving us was for somewhere
|
|
1157
|
+
// else, and the pieces at the new position have to be earned from peers
|
|
1158
|
+
// that are choking us — the same standing start as a fresh torrent.
|
|
1159
|
+
this.#markHurry(torrent, "the viewer moved");
|
|
1087
1160
|
const percent = ((safeStart / fileLength) * 100).toFixed(1);
|
|
1088
1161
|
logger.info(
|
|
1089
1162
|
`torrent-pool: [${String(torrent.infoHash).slice(0, 8)}] read position -> ` +
|