@torrent-tv/proxy 2.71.1 → 2.72.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +26 -0
- package/bin/cli.js +8 -0
- package/package.json +1 -1
- package/server.js +4 -0
- package/services/container/Container.js +35 -0
- package/services/container/MatroskaContainer.js +42 -0
- package/services/container/Mp4Container.js +22 -1
- package/services/container/SubtitleFileContainer.js +261 -0
- package/services/container/index.js +1 -0
- package/services/container-index/matroska-subtitles.js +6 -1
- package/services/controllers/SubtitleController.js +6 -25
- package/services/health-collector.js +37 -3
- package/services/hls-session-manager.js +10584 -10243
- package/services/piece-store/piece-lru.js +43 -0
- package/services/piece-store/shared-piece-store.js +214 -11
- package/services/playback-planner.js +7 -0
- package/services/subtitle-convert.js +74 -80
- package/services/torrent-worker/subtitle-cues.js +18 -53
- package/services/tracks/TextSubtitleTrack.js +18 -0
- package/services/tracks/index.js +1 -0
- package/services/tracks/subtitle-markup.js +104 -0
- package/services/tunnel-client.js +27 -0
- package/test/health-metrics.test.js +37 -0
- package/test/piece-store-eviction.test.js +98 -10
- package/test/piece-store-slow-disk.test.js +114 -0
- package/test/produced-copy-choice.test.js +361 -0
- package/test/subtitle-cue-framing.test.js +202 -0
- package/test/subtitle-language.test.js +19 -8
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,29 @@
|
|
|
1
|
+
## 2.72.1
|
|
2
|
+
|
|
3
|
+
- **Fix**: Subtitles on an embedded ASS track were shown to the viewer as the whole dialogue row — `21,0,Default,,0000,0000,0000,,I am the powerful Demon King of the Sixth Heaven.` — because one function both unwrapped the container's framing and stripped the codec's markup, and decided which framing it held by counting commas. Matroska writes eight fields before the text and takes the two timing fields out into the block's own timestamp (`matroska.org/technical/subtitles.html`); a row in a `.ass` FILE has nine and states its own column order in `[Events]`. The function expected the file's shape, found nine fields where it wanted more than nine, and returned the row untouched. The test that covered this passed because its fixture carried the file's two timestamps — a shape that never occurs on that path.
|
|
4
|
+
- **New**: `Container.cueTextOf` — each container answers for its own framing, because how a cue's bytes are wrapped is stated by the container's specification and not by the subtitle format's. `MatroskaContainer` carries the eight-field rule with the quotation it comes from, `Mp4Container` the length-prefixed sample, and `AviContainer` throws rather than guessing at somebody else's framing. The unwrapping now happens where the cue is READ, which is the only place the container is known.
|
|
5
|
+
- **New**: `SubtitleFileContainer` — a file of subtitles is a container of its own. It reads the `Format:` line of `[Events]` for ASS, which is where a FILE states its field order, and SubRip's positional form; `.vtt` is passed through unparsed, since taking it apart to write it back would drop its styles, regions and cue identifiers for nothing.
|
|
6
|
+
- **New**: `services/tracks/subtitle-markup.js` — the other axis: `{\pos(…)}`, `\N`, `\n`, `\h`, the same wherever ASS is stored. One table for every name a codec has — Matroska CodecID, MP4 sample entry, file extension — because the answer for all of them is the same. `\h` reached the viewer as a backslash and an h on embedded tracks until now; the file path had always handled it.
|
|
7
|
+
- **Chore**: `finalizeCues` and the WebVTT writing move to `subtitle-convert.js` and exist once. The writer was duplicated between the controller and the file conversion, so a pushed cue and a pulled one were formatted by two different pieces of code.
|
|
8
|
+
- **Chore**: `test/subtitle-cue-framing.test.js`, 17 checks across both axes; three of them fail on the old code, including the field case itself and "one line of dialogue, two framings, one result". Measurements and the spec quotations: `research/subtitle-cue-framing-2026-09-03.md`.
|
|
9
|
+
|
|
10
|
+
- **Fix**: A file's NAME was being read as proof that a segment exists. The `segment` muxer creates its output the moment it OPENS it, so a run killed for a seek leaves a file of zero bytes behind whose name is indistinguishable from a finished piece's — measured 2026-09-03: ffmpeg exited 19 ms after SIGTERM leaving `segment-00025.mp4` empty, and that empty file closed the only hole in the numbering. The two sides of the session then deadlocked on it without either being able to see the other's reason: the look-ahead read `420s ahead of the viewer` and kept the encoder stopped because the segment was on disk, while the serving path refused the very same file for carrying no track. Both sides now ask `usableSegmentIndices`, where a number counts only when some run holds a NON-EMPTY copy of it — which is exactly the condition under which the serving path can answer.
|
|
11
|
+
- **Fix**: The piece a run had open when it ended is removed once that run's process is gone, and only when it is unusable — a run stopped between two cuts leaves a finished file, and deleting good output would mean encoding it a second time. The current run's own unfinished piece is waited for, never deleted.
|
|
12
|
+
- **Fix**: A segment is served from the newest copy that carries every track, across every run the session has had, instead of from whichever run wrote the name last. Where nothing can judge the copies — before the session has an init segment there is nothing to compare a piece against — the newest copy with bytes in it stands, and a piece that no run holds servably is still handed back so the readiness path can say WHY rather than answering "not produced".
|
|
13
|
+
- **Fix**: Two more places asked a file's name where they meant its contents. That a segment is finished is proved by the NEXT one existing, and the proof was taken from whichever run happened to hold that number — a run that has ended closed everything it wrote and needs no such proof, so the evidence now has to come from the same run. And the session's header is derived from the first piece that has bytes in it rather than the first name, since an empty file skips a number whose header is sitting in the run before it.
|
|
14
|
+
- **Chore**: Segment sizes are asked of the filesystem once per file. A piece that has bytes never loses them and a run rewriting a number writes into its own directory, so without that memory every request walked every segment of every run — 1350 of them for a 90-minute film — on the thread that also carries the data channel.
|
|
15
|
+
- **Chore**: `test/produced-copy-choice.test.js`, 9 checks: an empty file is not counted by the look-ahead, a killed run leaves nothing behind, a run that finished its last piece keeps it, a last piece short of a track goes even though it has bytes, and the sizes are read once.
|
|
16
|
+
|
|
17
|
+
## 2.72.0
|
|
18
|
+
|
|
19
|
+
- **Fix**: The proxy was killed by the machine's out-of-memory killer at 4.37 GB, twenty minutes after 2.71.0 went out, and the cause was 2.71.0's own budget. A piece being written out to disk leaves the store's count of what it holds the moment the eviction begins, while its memory stays held until the write — which reads from that very block — has finished. The allowance counted resident pieces, so a block held by a pending write was counted nowhere: every admission turned one resident block into one held by the disk and took a fresh block for the arrival, and memory in use rose by one block per admission for as long as the disk was behind. It was behind by a factor of two: 233 evictions started against about 119 writes completed in the same minute. The store reported 203 blocks held with THREE pieces resident against 68 MB allowed. The allowance now bounds blocks in use — resident, reserved, and held by writes that have not finished — so a full store waits for the disk instead of evicting another piece, using the wait and the wake that were already there.
|
|
20
|
+
- **Fix**: And the reason there were so many evictions: 2.71.0 made the allowance equal to what the readers ask for, exactly. `6 reader(s) want 23 piece(s) of 23 the store may hold` — no free place ever exists, so every arriving piece must evict a wanted one. The allowance now includes room for what arrives while one write is finishing, measured from the store's own median write duration and its own arrival rate, and zero until both have been seen rather than invented in advance.
|
|
21
|
+
- **Chore**: `test/piece-store-slow-disk.test.js` drives a disk that answers only when the test says so, which is the field condition — writes slower than arrivals — and it fails without the fix. The check that shipped with the first attempt at this did not: it passed with the defect in place, which is worth recording, because a test that cannot fail proves nothing.
|
|
22
|
+
|
|
23
|
+
- **Fix**: A proxy reported how much memory it had free with `os.freemem()`, and on Linux that counts only the pages free at this instant — the kernel keeps that number low on purpose and fills the rest with cache, which it hands back the moment anything asks. A host with 4 GB of cache and 200 MB genuinely free called itself nearly full while it had 4.2 GB to give. That figure weighs 0.4 of every proxy's score, so every Linux proxy in the pool understated itself, each by a different amount according to how much cache it happened to hold. It reads `MemAvailable` now — the same fix the piece store's budget got on 2026-08-27, which had stayed in this file until today.
|
|
24
|
+
- **New**: A proxy can answer whether it could sustain a file it is only told ABOUT. The expensive half of that question is finding out what the file IS — add the torrent, wait for metadata, fetch the header, run ffmpeg — and it has already been paid by whichever proxy probed it. Its answer is a handful of numbers; every other proxy answers by arithmetic against its own startup benchmarks in milliseconds, without adding the torrent or fetching a byte. Asked over the tunnel as `can-serve-request`.
|
|
25
|
+
- **New**: The refusal added in 2.71.1 now carries that description, so a viewer whose proxy cannot keep up is moved to one that can instead of being shown an error. A viewer is given a proxy BEFORE the file is known, by a score that reads processor load, free memory and round-trip time — none of which can answer a question about a particular source — and this is where that ordering is repaired, after the fact and only when it went wrong.
|
|
26
|
+
|
|
1
27
|
## 2.71.1
|
|
2
28
|
|
|
3
29
|
- **Fix**: A reader stated the same thing twice — `protectRange` to the piece store for memory, and a window to the torrent for download — and the two were separate lists that could drift. There is one statement now: `SwarmSelection.reconcile` derives both views from the register, so the swarm and the store are told what to do from the same words. Only the urgent levels reach memory: it holds what will be READ soon, and protecting the speculative tail would push out a piece the decoder is about to want.
|
package/bin/cli.js
CHANGED
|
@@ -476,6 +476,14 @@ try {
|
|
|
476
476
|
onHealthRequest() {
|
|
477
477
|
return collectHealthMetrics();
|
|
478
478
|
},
|
|
479
|
+
// Whether this host could sustain a file it has only been told about. The
|
|
480
|
+
// same arithmetic the first offer uses, against this host's own startup
|
|
481
|
+
// benchmarks — no torrent, no bytes, no ffmpeg — so the browser can ask
|
|
482
|
+
// every proxy in the pool and be sent to one that will work instead of
|
|
483
|
+
// being shown an error on the one it happened to land on.
|
|
484
|
+
onCanServeRequest(mediaInfo) {
|
|
485
|
+
return started?.hlsSessionManager?.predictOfferedHeights?.(mediaInfo) ?? null;
|
|
486
|
+
},
|
|
479
487
|
onConnect() {
|
|
480
488
|
// Re-register on every tunnel connect/reconnect so the server's
|
|
481
489
|
// in-memory store stays consistent after server restarts.
|
package/package.json
CHANGED
package/server.js
CHANGED
|
@@ -318,6 +318,10 @@ export async function startProxyServer({
|
|
|
318
318
|
return {
|
|
319
319
|
app,
|
|
320
320
|
port: selectedPort,
|
|
321
|
+
// Asked over the tunnel when the proxy a viewer landed on has refused their
|
|
322
|
+
// file: could THIS host sustain it? Answered from the startup benchmarks
|
|
323
|
+
// and a description, so it needs no torrent and costs milliseconds.
|
|
324
|
+
hlsSessionManager,
|
|
321
325
|
// The browser only ever knows a source by its REGISTRY key (a hash of the
|
|
322
326
|
// raw request bytes, scoped to one API session) — never the torrent
|
|
323
327
|
// pool's own key (the content's infohash, shared across a magnet and a
|
|
@@ -56,4 +56,39 @@ export class Container {
|
|
|
56
56
|
* Subtitle-specific: where cues live (Matroska cluster positions or MP4 sample ranges).
|
|
57
57
|
* Returned via track objects' clusterPositions/samples, so base has no extra method — tracks carry it.
|
|
58
58
|
*/
|
|
59
|
+
|
|
60
|
+
/**
|
|
61
|
+
* The TEXT FIELD of one subtitle cue, taken out of this container's framing.
|
|
62
|
+
*
|
|
63
|
+
* How a cue's bytes are wrapped is stated by the container's own
|
|
64
|
+
* specification, so each subclass answers for itself: Matroska reorders an ASS
|
|
65
|
+
* dialogue row, drops its two timing fields and prepends a read order
|
|
66
|
+
* (`matroska.org/technical/subtitles.html`); an MP4 prefixes a `tx3g` sample
|
|
67
|
+
* with its length (ISO/IEC 14496-12 §12.6); a subtitle FILE states its own
|
|
68
|
+
* field order in `[Events]`. None of that is a fact about the subtitle format,
|
|
69
|
+
* and the format's own markup — `{\pos(…)}`, `\N` — is not a fact about the
|
|
70
|
+
* container. The second half is `tracks/subtitle-markup.js`; this is the
|
|
71
|
+
* first, and the two are applied in that order.
|
|
72
|
+
*
|
|
73
|
+
* Static because de-framing reads no instance state: a caller that has bytes
|
|
74
|
+
* and knows the format needs no container built over the whole file. The
|
|
75
|
+
* instance form below exists so a caller that DOES hold a container gets the
|
|
76
|
+
* right answer without naming the subclass.
|
|
77
|
+
*
|
|
78
|
+
* @param {Buffer} _payload - The cue's bytes as the container stores them.
|
|
79
|
+
* @param {string} _codecId - CodecID / sample entry type / file extension.
|
|
80
|
+
* @returns {string} The text field, markup still in place.
|
|
81
|
+
*/
|
|
82
|
+
static cueTextOf(_payload, _codecId) {
|
|
83
|
+
throw new Error("cueTextOf not implemented");
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
/**
|
|
87
|
+
* @param {Buffer} payload
|
|
88
|
+
* @param {string} codecId
|
|
89
|
+
* @returns {string}
|
|
90
|
+
*/
|
|
91
|
+
cueTextOf(payload, codecId) {
|
|
92
|
+
return /** @type {typeof Container} */ (this.constructor).cueTextOf(payload, codecId);
|
|
93
|
+
}
|
|
59
94
|
}
|
|
@@ -50,6 +50,12 @@ const ID_DISPLAY_WIDTH = 0x54b0;
|
|
|
50
50
|
const ID_DISPLAY_HEIGHT = 0x54ba;
|
|
51
51
|
const ID_SAMPLING_FREQUENCY = 0xb5;
|
|
52
52
|
const ID_CHANNELS = 0x9f;
|
|
53
|
+
/**
|
|
54
|
+
* ReadOrder, Layer, Style, Name, MarginL, MarginR, MarginV, Effect — the eight
|
|
55
|
+
* fields Matroska writes before the text of an SSA/ASS event. See
|
|
56
|
+
* {@link MatroskaContainer.cueTextOf} for the quotation this comes from.
|
|
57
|
+
*/
|
|
58
|
+
const ASS_FIELDS_BEFORE_TEXT = 8;
|
|
53
59
|
|
|
54
60
|
function readString(buf, el) {
|
|
55
61
|
return buf.toString("utf8", el.dataOffset, el.dataOffset + el.size).replace(/\0+$/, "");
|
|
@@ -280,6 +286,42 @@ export class MatroskaContainer extends Container {
|
|
|
280
286
|
return result;
|
|
281
287
|
}
|
|
282
288
|
|
|
289
|
+
/**
|
|
290
|
+
* The text field of one cue as Matroska frames it.
|
|
291
|
+
*
|
|
292
|
+
* Two rules, both from `matroska.org/technical/subtitles.html`, "Now, how are
|
|
293
|
+
* they stored in Matroska?":
|
|
294
|
+
*
|
|
295
|
+
* 1. "All text is converted to UTF-8", so the block is decoded as UTF-8 and
|
|
296
|
+
* no other encoding is guessed at. A subtitle FILE is a different matter —
|
|
297
|
+
* there the bytes may be Windows-1251 and `decodeSubtitleBytes` sniffs for
|
|
298
|
+
* it — but a muxer had to convert before writing the block.
|
|
299
|
+
* 2. "Events are stored in the Block in this order: ReadOrder, Layer, Style,
|
|
300
|
+
* Name, MarginL, MarginR, MarginV, Effect, Text", and "Start & End field
|
|
301
|
+
* are used to set TimeStamp and the BlockDuration element". So eight fields
|
|
302
|
+
* stand before the text, the two timing fields of the file's own row are
|
|
303
|
+
* NOT among them, and a read order takes their place at the front. The text
|
|
304
|
+
* itself may hold commas, so everything from the ninth field on is joined
|
|
305
|
+
* back together.
|
|
306
|
+
*
|
|
307
|
+
* `S_TEXT/UTF8` and `S_TEXT/WEBVTT` have no such framing: the block holds the
|
|
308
|
+
* cue text and nothing else. (A WebVTT cue's settings, identifier and
|
|
309
|
+
* preceding comments live in a BlockAddition, which this proxy does not read;
|
|
310
|
+
* losing them costs positioning, not words.)
|
|
311
|
+
*
|
|
312
|
+
* @param {Buffer} payload - The block's own bytes.
|
|
313
|
+
* @param {string} codecId - Matroska CodecID of the track the block belongs to.
|
|
314
|
+
* @returns {string}
|
|
315
|
+
*/
|
|
316
|
+
static cueTextOf(payload, codecId) {
|
|
317
|
+
const text = Buffer.isBuffer(payload) ? payload.toString("utf8") : String(payload ?? "");
|
|
318
|
+
if (codecId !== "S_TEXT/ASS" && codecId !== "S_TEXT/SSA") {
|
|
319
|
+
return text;
|
|
320
|
+
}
|
|
321
|
+
const fields = text.split(",");
|
|
322
|
+
return fields.length > ASS_FIELDS_BEFORE_TEXT ? fields.slice(ASS_FIELDS_BEFORE_TEXT).join(",") : "";
|
|
323
|
+
}
|
|
324
|
+
|
|
283
325
|
async readKeyframeIndex() {
|
|
284
326
|
const times = await readMatroskaKeyframeTimes(this.readRange, this.fileSize);
|
|
285
327
|
if (!times) return null;
|
|
@@ -16,7 +16,7 @@
|
|
|
16
16
|
|
|
17
17
|
import { Container } from "./Container.js";
|
|
18
18
|
import { isMp4, readMp4KeyframeTimes } from "../container-index/mp4.js";
|
|
19
|
-
import { readMp4SubtitlePlan } from "../container-index/mp4-subtitles.js";
|
|
19
|
+
import { decodeSubtitleSample, readMp4SubtitlePlan } from "../container-index/mp4-subtitles.js";
|
|
20
20
|
import { VideoTrack } from "../tracks/VideoTrack.js";
|
|
21
21
|
import { AudioTrack } from "../tracks/AudioTrack.js";
|
|
22
22
|
import { TextSubtitleTrack, TEXT_FORMATS_MP4 } from "../tracks/TextSubtitleTrack.js";
|
|
@@ -233,6 +233,27 @@ export class Mp4Container extends Container {
|
|
|
233
233
|
return result;
|
|
234
234
|
}
|
|
235
235
|
|
|
236
|
+
/**
|
|
237
|
+
* The text field of one cue as MP4 frames it.
|
|
238
|
+
*
|
|
239
|
+
* A `tx3g`/`text` sample is a 16-bit big-endian length followed by that many
|
|
240
|
+
* bytes of UTF-8 (ISO/IEC 14496-12 §12.6.3 and Apple's text sample format); a
|
|
241
|
+
* `wvtt` sample is a sequence of boxes whose `vttc`/`payl` holds the cue text
|
|
242
|
+
* (§12.6.3.2). Neither carries the subtitle format's own markup, so the
|
|
243
|
+
* markup step that follows has nothing to take off — it is applied all the
|
|
244
|
+
* same, because which step applies is decided by the codec and not here.
|
|
245
|
+
*
|
|
246
|
+
* The byte reading itself stays in `container-index/mp4-subtitles.js`,
|
|
247
|
+
* alongside the sample-table walk that found the range.
|
|
248
|
+
*
|
|
249
|
+
* @param {Buffer} payload - The sample's own bytes.
|
|
250
|
+
* @param {string} codecId - Sample entry type: `tx3g`, `text` or `wvtt`.
|
|
251
|
+
* @returns {string}
|
|
252
|
+
*/
|
|
253
|
+
static cueTextOf(payload, codecId) {
|
|
254
|
+
return decodeSubtitleSample(payload, codecId);
|
|
255
|
+
}
|
|
256
|
+
|
|
236
257
|
async readKeyframeIndex() {
|
|
237
258
|
const r = await readMp4KeyframeTimes(this.readRange, this.fileSize);
|
|
238
259
|
if (!r) return null;
|
|
@@ -0,0 +1,261 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* @file A file of subtitles as a container of its own — `.srt`, `.ass`, `.ssa`,
|
|
3
|
+
* `.vtt` shipped beside the film.
|
|
4
|
+
*
|
|
5
|
+
* It belongs in this folder for the same reason `.mka` does: "a file of its
|
|
6
|
+
* own" is not a KIND of track, only the answer to where a track's bytes are,
|
|
7
|
+
* and the question this folder answers is how a format frames what it carries.
|
|
8
|
+
* SubRip frames a cue as an ordinal, a timing line and the lines under it; ASS
|
|
9
|
+
* frames one as a `Dialogue:` row whose FIELD ORDER the file itself states in
|
|
10
|
+
* `[Events]`; WebVTT is already what a browser reads.
|
|
11
|
+
*
|
|
12
|
+
* That last point about ASS is what makes the file different from Matroska and
|
|
13
|
+
* the reason both need their own answer. Matroska fixes the order in its own
|
|
14
|
+
* specification, so eight fields always stand before the text. A file does not:
|
|
15
|
+
* `Format:` may list the columns in any order, and the specification is that
|
|
16
|
+
* the reader obeys it. Two framings of one subtitle format, each stated by
|
|
17
|
+
* whoever stores it.
|
|
18
|
+
*
|
|
19
|
+
* What this file does NOT do is take off ASS's own markup — `{\pos(…)}`, `\N`,
|
|
20
|
+
* `\h`. That is the same wherever ASS is stored and lives in
|
|
21
|
+
* `tracks/subtitle-markup.js`.
|
|
22
|
+
*/
|
|
23
|
+
|
|
24
|
+
import { Container } from "./Container.js";
|
|
25
|
+
import { TextSubtitleTrack } from "../tracks/TextSubtitleTrack.js";
|
|
26
|
+
|
|
27
|
+
/** The extensions this reads. `.vtt` is included and passes through unparsed. */
|
|
28
|
+
const EXTENSIONS = new Set([".srt", ".ass", ".ssa", ".vtt", ".webvtt"]);
|
|
29
|
+
|
|
30
|
+
/**
|
|
31
|
+
* A SubRip timing line. The specification writes a comma before the
|
|
32
|
+
* milliseconds; WebVTT writes a dot, which is the only difference between the
|
|
33
|
+
* two lines.
|
|
34
|
+
*/
|
|
35
|
+
const SRT_TIMING = /^(\d{1,2}):(\d{2}):(\d{2})[,.](\d{1,3})\s*-->\s*(\d{1,2}):(\d{2}):(\d{2})[,.](\d{1,3})/;
|
|
36
|
+
|
|
37
|
+
/** An ASS timing field: `h:mm:ss.cc`, centiseconds. */
|
|
38
|
+
const ASS_TIMING = /^(\d+):(\d{2}):(\d{2})\.(\d{1,2})$/;
|
|
39
|
+
|
|
40
|
+
/**
|
|
41
|
+
* Seconds from a SubRip timing line's four captured parts.
|
|
42
|
+
*
|
|
43
|
+
* @param {string[]} parts - [hours, minutes, seconds, fraction]
|
|
44
|
+
* @returns {number}
|
|
45
|
+
*/
|
|
46
|
+
function srtSeconds([hours, minutes, seconds, fraction]) {
|
|
47
|
+
const ms = Number(String(fraction).padEnd(3, "0"));
|
|
48
|
+
return Number(hours) * 3600 + Number(minutes) * 60 + Number(seconds) + ms / 1000;
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
/**
|
|
52
|
+
* Seconds from an ASS timing field.
|
|
53
|
+
*
|
|
54
|
+
* @param {string} field
|
|
55
|
+
* @returns {number | null} Null when the field is not a timing at all, which is
|
|
56
|
+
* a malformed row and not a cue at zero.
|
|
57
|
+
*/
|
|
58
|
+
function assSeconds(field) {
|
|
59
|
+
const match = ASS_TIMING.exec(String(field ?? "").trim());
|
|
60
|
+
if (!match) {
|
|
61
|
+
return null;
|
|
62
|
+
}
|
|
63
|
+
const centiseconds = Number(String(match[4]).padEnd(2, "0"));
|
|
64
|
+
return Number(match[1]) * 3600 + Number(match[2]) * 60 + Number(match[3]) + centiseconds / 100;
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
export class SubtitleFileContainer extends Container {
|
|
68
|
+
/**
|
|
69
|
+
* @param {object} params
|
|
70
|
+
* @param {string} params.extension - Lowercase, including the dot.
|
|
71
|
+
* @param {string} [params.label]
|
|
72
|
+
*/
|
|
73
|
+
constructor({ extension, label = "" }) {
|
|
74
|
+
// A subtitle file is read whole — it is kilobytes — so there is no range
|
|
75
|
+
// reading and no size to bound it by. `readTracks` and `readCues` take the
|
|
76
|
+
// decoded text directly, which is why the base's `readRange` is unused.
|
|
77
|
+
super({ readRange: null, fileSize: 0, label });
|
|
78
|
+
this.extension = String(extension ?? "").toLowerCase();
|
|
79
|
+
/** Column order from `[Events]`'s `Format:`, once a file has been read. */
|
|
80
|
+
this.eventColumns = null;
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
get formatName() {
|
|
84
|
+
return "subtitle-file";
|
|
85
|
+
}
|
|
86
|
+
|
|
87
|
+
/**
|
|
88
|
+
* @param {string} extension - Lowercase, including the dot.
|
|
89
|
+
* @returns {boolean}
|
|
90
|
+
*/
|
|
91
|
+
static detect(extension) {
|
|
92
|
+
return EXTENSIONS.has(String(extension ?? "").toLowerCase());
|
|
93
|
+
}
|
|
94
|
+
|
|
95
|
+
/**
|
|
96
|
+
* The single track a subtitle file carries.
|
|
97
|
+
*
|
|
98
|
+
* Its `codecId` is the extension, which is the only thing the file says about
|
|
99
|
+
* its own format, and `subtitle-markup.js` accepts extensions alongside
|
|
100
|
+
* container codec names for exactly this reason.
|
|
101
|
+
*
|
|
102
|
+
* @returns {Promise<TextSubtitleTrack[]>}
|
|
103
|
+
*/
|
|
104
|
+
async readTracks() {
|
|
105
|
+
return [new TextSubtitleTrack({
|
|
106
|
+
trackNumber: 0,
|
|
107
|
+
declaredIndex: 0,
|
|
108
|
+
codecId: this.extension,
|
|
109
|
+
language: "",
|
|
110
|
+
languageBcp47: "",
|
|
111
|
+
name: this.label,
|
|
112
|
+
isEnabled: true,
|
|
113
|
+
isDefault: false,
|
|
114
|
+
declaresDefault: false
|
|
115
|
+
})];
|
|
116
|
+
}
|
|
117
|
+
|
|
118
|
+
/**
|
|
119
|
+
* The text field of one ASS `Dialogue:` row, by the order the file declared.
|
|
120
|
+
*
|
|
121
|
+
* Not static, unlike the other containers': the order is read out of the
|
|
122
|
+
* file's own header, so the answer depends on which file this is. A row
|
|
123
|
+
* arriving before any `Format:` line has been seen has no declared order and
|
|
124
|
+
* is not guessed at.
|
|
125
|
+
*
|
|
126
|
+
* @param {string} row - The row after `Dialogue:`.
|
|
127
|
+
* @param {string} [codecId]
|
|
128
|
+
* @returns {string}
|
|
129
|
+
*/
|
|
130
|
+
cueTextOf(row, codecId = this.extension) {
|
|
131
|
+
if (codecId !== ".ass" && codecId !== ".ssa") {
|
|
132
|
+
return String(row ?? "");
|
|
133
|
+
}
|
|
134
|
+
const at = this.eventColumns ? this.eventColumns.indexOf("text") : -1;
|
|
135
|
+
if (at < 0) {
|
|
136
|
+
return "";
|
|
137
|
+
}
|
|
138
|
+
// The text field is last by the specification and may hold commas, so
|
|
139
|
+
// everything from its column on is joined back together.
|
|
140
|
+
return String(row ?? "").split(",").slice(at).join(",");
|
|
141
|
+
}
|
|
142
|
+
|
|
143
|
+
/**
|
|
144
|
+
* Every cue in the file, with its markup still in place.
|
|
145
|
+
*
|
|
146
|
+
* @param {string} text - The file, already decoded to characters.
|
|
147
|
+
* @returns {{ startSeconds: number, endSeconds: number, text: string }[] | null}
|
|
148
|
+
* Null for WebVTT, which is not parsed: it is already what a browser reads,
|
|
149
|
+
* and taking it apart to write it back would drop its styles, its regions
|
|
150
|
+
* and its cue identifiers for nothing.
|
|
151
|
+
*/
|
|
152
|
+
readCues(text) {
|
|
153
|
+
const lines = String(text ?? "").replace(/\r\n/g, "\n").replace(/\r/g, "\n").split("\n");
|
|
154
|
+
if (this.extension === ".ass" || this.extension === ".ssa") {
|
|
155
|
+
return this.#assCues(lines);
|
|
156
|
+
}
|
|
157
|
+
if (this.extension === ".srt") {
|
|
158
|
+
return this.#srtCues(lines);
|
|
159
|
+
}
|
|
160
|
+
return null;
|
|
161
|
+
}
|
|
162
|
+
|
|
163
|
+
/**
|
|
164
|
+
* @param {string[]} lines
|
|
165
|
+
* @returns {{ startSeconds: number, endSeconds: number, text: string }[]}
|
|
166
|
+
*/
|
|
167
|
+
#assCues(lines) {
|
|
168
|
+
const cues = [];
|
|
169
|
+
let inEvents = false;
|
|
170
|
+
this.eventColumns = null;
|
|
171
|
+
for (const line of lines) {
|
|
172
|
+
const trimmed = line.trim();
|
|
173
|
+
if (/^\[.*\]$/.test(trimmed)) {
|
|
174
|
+
inEvents = trimmed === "[Events]";
|
|
175
|
+
continue;
|
|
176
|
+
}
|
|
177
|
+
if (!inEvents) {
|
|
178
|
+
continue;
|
|
179
|
+
}
|
|
180
|
+
if (trimmed.startsWith("Format:")) {
|
|
181
|
+
this.eventColumns = trimmed
|
|
182
|
+
.slice("Format:".length)
|
|
183
|
+
.split(",")
|
|
184
|
+
.map((column) => column.trim().toLowerCase());
|
|
185
|
+
continue;
|
|
186
|
+
}
|
|
187
|
+
if (!trimmed.startsWith("Dialogue:") || !this.eventColumns) {
|
|
188
|
+
continue;
|
|
189
|
+
}
|
|
190
|
+
const row = trimmed.slice("Dialogue:".length);
|
|
191
|
+
const fields = row.split(",");
|
|
192
|
+
const startAt = this.eventColumns.indexOf("start");
|
|
193
|
+
const endAt = this.eventColumns.indexOf("end");
|
|
194
|
+
if (startAt < 0 || endAt < 0) {
|
|
195
|
+
continue;
|
|
196
|
+
}
|
|
197
|
+
const startSeconds = assSeconds(fields[startAt]);
|
|
198
|
+
const endSeconds = assSeconds(fields[endAt]);
|
|
199
|
+
const cueText = this.cueTextOf(row);
|
|
200
|
+
if (startSeconds === null || endSeconds === null || !cueText) {
|
|
201
|
+
continue;
|
|
202
|
+
}
|
|
203
|
+
cues.push({ startSeconds, endSeconds, text: cueText });
|
|
204
|
+
}
|
|
205
|
+
return cues;
|
|
206
|
+
}
|
|
207
|
+
|
|
208
|
+
/**
|
|
209
|
+
* @param {string[]} lines
|
|
210
|
+
* @returns {{ startSeconds: number, endSeconds: number, text: string }[]}
|
|
211
|
+
*/
|
|
212
|
+
#srtCues(lines) {
|
|
213
|
+
const cues = [];
|
|
214
|
+
/** @type {{ startSeconds: number, endSeconds: number, text: string[] } | null} */
|
|
215
|
+
let open = null;
|
|
216
|
+
const close = () => {
|
|
217
|
+
if (!open) {
|
|
218
|
+
return;
|
|
219
|
+
}
|
|
220
|
+
// A file with no blank line between cues leaves the next cue's ordinal as
|
|
221
|
+
// the last line of this one. It is SubRip's own numbering, never
|
|
222
|
+
// dialogue, so it goes rather than being shown.
|
|
223
|
+
while (open.text.length > 0 && /^\d+$/.test(open.text[open.text.length - 1].trim())) {
|
|
224
|
+
open.text.pop();
|
|
225
|
+
}
|
|
226
|
+
const text = open.text.join("\n").trim();
|
|
227
|
+
if (text) {
|
|
228
|
+
cues.push({ startSeconds: open.startSeconds, endSeconds: open.endSeconds, text });
|
|
229
|
+
}
|
|
230
|
+
open = null;
|
|
231
|
+
};
|
|
232
|
+
for (const line of lines) {
|
|
233
|
+
const timing = SRT_TIMING.exec(line.trim());
|
|
234
|
+
if (timing) {
|
|
235
|
+
// A timing line opens a cue and closes the one before it. The ordinal
|
|
236
|
+
// above it is SubRip's own numbering and carries nothing a player needs,
|
|
237
|
+
// so it is dropped rather than carried into the cue's text — which is
|
|
238
|
+
// what taking the lines between timings would do.
|
|
239
|
+
close();
|
|
240
|
+
open = {
|
|
241
|
+
startSeconds: srtSeconds(timing.slice(1, 5)),
|
|
242
|
+
endSeconds: srtSeconds(timing.slice(5, 9)),
|
|
243
|
+
text: []
|
|
244
|
+
};
|
|
245
|
+
continue;
|
|
246
|
+
}
|
|
247
|
+
if (!open) {
|
|
248
|
+
continue;
|
|
249
|
+
}
|
|
250
|
+
if (line.trim() === "") {
|
|
251
|
+
close();
|
|
252
|
+
continue;
|
|
253
|
+
}
|
|
254
|
+
open.text.push(line);
|
|
255
|
+
}
|
|
256
|
+
close();
|
|
257
|
+
return cues;
|
|
258
|
+
}
|
|
259
|
+
}
|
|
260
|
+
|
|
261
|
+
export { EXTENSIONS as SUBTITLE_FILE_EXTENSIONS };
|
|
@@ -2,4 +2,5 @@ export { Container } from "./Container.js";
|
|
|
2
2
|
export { MatroskaContainer } from "./MatroskaContainer.js";
|
|
3
3
|
export { Mp4Container } from "./Mp4Container.js";
|
|
4
4
|
export { AviContainer } from "./AviContainer.js";
|
|
5
|
+
export { SubtitleFileContainer, SUBTITLE_FILE_EXTENSIONS } from "./SubtitleFileContainer.js";
|
|
5
6
|
export { ContainerFactory } from "./ContainerFactory.js";
|
|
@@ -359,9 +359,14 @@ export function harvestCluster(bytes, trackNumber, secondsPerTick) {
|
|
|
359
359
|
trackNumber,
|
|
360
360
|
secondsPerTick
|
|
361
361
|
);
|
|
362
|
+
// The payload is handed on as BYTES. What those bytes mean — which of them
|
|
363
|
+
// are the text and which are the eight fields Matroska puts before it — is
|
|
364
|
+
// stated by the container's specification and answered by
|
|
365
|
+
// `MatroskaContainer.cueTextOf`, not here: this function's subject is where a
|
|
366
|
+
// block sits and how long it lasts.
|
|
362
367
|
return blocks.map((block) => ({
|
|
363
368
|
startSeconds: block.startSeconds,
|
|
364
369
|
endSeconds: block.durationSeconds === null ? null : block.startSeconds + block.durationSeconds,
|
|
365
|
-
|
|
370
|
+
payload: block.payload
|
|
366
371
|
}));
|
|
367
372
|
}
|
|
@@ -8,30 +8,11 @@
|
|
|
8
8
|
*/
|
|
9
9
|
|
|
10
10
|
import { subtitleOrchestrator } from "../orchestrators/SubtitleOrchestrator.js";
|
|
11
|
-
import { convertSubtitleToVtt, decodeSubtitleBytes } from "../subtitle-convert.js";
|
|
11
|
+
import { convertSubtitleToVtt, cuesToVtt, decodeSubtitleBytes, finalizeCues } from "../subtitle-convert.js";
|
|
12
12
|
import { detectLanguage, detectLanguageFromVtt } from "../language-detect.js";
|
|
13
|
-
import { finalizeCues } from "../torrent-worker/subtitle-cues.js";
|
|
14
13
|
|
|
15
14
|
const EXTERNAL_MAX_BYTES = 8 * 1024 * 1024;
|
|
16
15
|
|
|
17
|
-
function vttTime(s) {
|
|
18
|
-
const safe = Math.max(0, s);
|
|
19
|
-
const h = Math.floor(safe / 3600);
|
|
20
|
-
const m = Math.floor((safe % 3600) / 60);
|
|
21
|
-
const r = safe % 60;
|
|
22
|
-
return `${String(h).padStart(2, "0")}:${String(m).padStart(2, "0")}:${r.toFixed(3).padStart(6, "0")}`;
|
|
23
|
-
}
|
|
24
|
-
|
|
25
|
-
function cuesToVtt(cues, codecId) {
|
|
26
|
-
const lines = ["WEBVTT", ""];
|
|
27
|
-
for (const c of finalizeCues(cues, codecId)) {
|
|
28
|
-
lines.push(`${vttTime(c.startSeconds)} --> ${vttTime(c.endSeconds)}`);
|
|
29
|
-
lines.push(c.text);
|
|
30
|
-
lines.push("");
|
|
31
|
-
}
|
|
32
|
-
return lines.join("\n");
|
|
33
|
-
}
|
|
34
|
-
|
|
35
16
|
function readFileFully(file, maxBytes) {
|
|
36
17
|
return new Promise((resolve, reject) => {
|
|
37
18
|
const stream = file.createReadStream();
|
|
@@ -114,11 +95,11 @@ export class SubtitleController {
|
|
|
114
95
|
const vtt = cuesToVtt(fresh, codecId);
|
|
115
96
|
// Two things this reads, and each of them was wrong before 2.68.1.
|
|
116
97
|
//
|
|
117
|
-
// It reads the cues through `finalizeCues`,
|
|
118
|
-
//
|
|
119
|
-
//
|
|
120
|
-
//
|
|
121
|
-
//
|
|
98
|
+
// It reads the cues through `finalizeCues`, so what reaches the detector
|
|
99
|
+
// is the words and not ASS's `{\…}` override groups, which are Latin on a
|
|
100
|
+
// Russian track. (The dialogue row's own fields are gone earlier now, in
|
|
101
|
+
// the container that framed them — before 2.72.1 they were not gone at
|
|
102
|
+
// all, and the detector was reading them too.)
|
|
122
103
|
//
|
|
123
104
|
// And it reads EVERY cue held so far, not the `fresh` subset that is
|
|
124
105
|
// being sent. A re-subscription after a reconnect asks only for what this
|
|
@@ -6,6 +6,7 @@
|
|
|
6
6
|
* All values are cheap to read and require no background work.
|
|
7
7
|
*/
|
|
8
8
|
|
|
9
|
+
import { readFileSync } from "node:fs";
|
|
9
10
|
import os from "node:os";
|
|
10
11
|
|
|
11
12
|
/**
|
|
@@ -16,17 +17,50 @@ import os from "node:os";
|
|
|
16
17
|
* Suitable as input to `Math.max(0, 1 - Math.min(1, cpuLoad))` for a
|
|
17
18
|
* normalised "CPU availability" score.
|
|
18
19
|
*
|
|
19
|
-
* `memFree` — fraction of total system RAM that
|
|
20
|
+
* `memFree` — fraction of total system RAM that could still be given out
|
|
21
|
+
* (0–1). See {@link availableMemoryBytes} for why that is not the same as
|
|
22
|
+
* free memory.
|
|
20
23
|
*
|
|
21
24
|
* `uptime` — process uptime in whole seconds (useful for preferring
|
|
22
25
|
* already-warmed proxies over freshly started ones).
|
|
23
26
|
*
|
|
24
27
|
* @typedef {Object} HealthMetrics
|
|
25
28
|
* @property {number} cpuLoad - 1-min load avg / cpu-count. 0 = idle, 1 = saturated, >1 = overloaded.
|
|
26
|
-
* @property {number} memFree -
|
|
29
|
+
* @property {number} memFree - Memory an allocation could obtain, as a fraction of total RAM (0–1).
|
|
27
30
|
* @property {number} uptime - Process uptime in seconds.
|
|
28
31
|
*/
|
|
29
32
|
|
|
33
|
+
/**
|
|
34
|
+
* How much memory the machine could still give out, in bytes.
|
|
35
|
+
*
|
|
36
|
+
* NOT `os.freemem()`. On Linux that counts only the pages free at this
|
|
37
|
+
* instant, and the kernel keeps that number low on purpose: what is not in use
|
|
38
|
+
* is filled with cache, which is handed back the moment anything asks. A host
|
|
39
|
+
* with 4 GB of cache and 200 MB genuinely free reports 200 MB and looks full
|
|
40
|
+
* while it has 4.2 GB to give.
|
|
41
|
+
*
|
|
42
|
+
* The kernel publishes its own estimate as `MemAvailable`, and that is what is
|
|
43
|
+
* read here. The same mistake was fixed in the piece store's budget on
|
|
44
|
+
* 2026-08-27 and stayed in this file until 2026-09-02, where it weighed 0.4 of
|
|
45
|
+
* every proxy's score — so every Linux proxy in the pool understated itself,
|
|
46
|
+
* and by a different amount each, according to how much cache it happened to
|
|
47
|
+
* hold.
|
|
48
|
+
*
|
|
49
|
+
* @returns {number}
|
|
50
|
+
*/
|
|
51
|
+
export function availableMemoryBytes() {
|
|
52
|
+
try {
|
|
53
|
+
const match = /^MemAvailable:\s+(\d+)\s+kB$/m.exec(readFileSync("/proc/meminfo", "utf8"));
|
|
54
|
+
if (match) {
|
|
55
|
+
return Number(match[1]) * 1024;
|
|
56
|
+
}
|
|
57
|
+
} catch {
|
|
58
|
+
// silent-ok: not Linux, or /proc is not readable. `os.freemem()` is then
|
|
59
|
+
// the best available answer and on those systems it is not misleading.
|
|
60
|
+
}
|
|
61
|
+
return os.freemem();
|
|
62
|
+
}
|
|
63
|
+
|
|
30
64
|
/**
|
|
31
65
|
* Collect current system health metrics.
|
|
32
66
|
*
|
|
@@ -38,7 +72,7 @@ import os from "node:os";
|
|
|
38
72
|
export function collectHealthMetrics() {
|
|
39
73
|
const cpuCount = os.cpus().length || 1;
|
|
40
74
|
const cpuLoad = os.loadavg()[0] / cpuCount;
|
|
41
|
-
const memFree =
|
|
75
|
+
const memFree = availableMemoryBytes() / os.totalmem();
|
|
42
76
|
|
|
43
77
|
return {
|
|
44
78
|
cpuLoad: Math.round(cpuLoad * 1000) / 1000,
|