@torrent-tv/proxy 2.80.18 → 2.80.19
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +10 -0
- package/docs/encode-architecture.md +51 -2
- package/package.json +1 -1
- package/services/encode/EncodeRun.js +12 -6
- package/services/encode/SegmentStore.js +147 -223
- package/services/encode/run-command.js +16 -1
- package/services/hls-session-manager.js +5 -83
- package/services/orchestrators/EncodeOrchestrator.js +48 -37
- package/services/segment-formats/fmp4.js +54 -0
- package/services/segment-formats/mpegts.js +54 -0
- package/test/coverage-follows-the-disk.test.js +5 -1
- package/test/orchestrator-wired.test.js +5 -1
- package/test/produced-copy-choice.test.js +258 -358
- package/test/segment-serve-wiring.test.js +8 -9
- package/test/segment-store.test.js +238 -216
- package/services/encode/open-piece.js +0 -135
- package/test/open-piece.test.js +0 -152
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,13 @@
|
|
|
1
|
+
## 2.80.19
|
|
2
|
+
|
|
3
|
+
- **Fix**: HALF A SEGMENT WAS SERVED TO THE PLAYER, AND THE SESSION NEVER RECOVERED. A piece was taken as finished when the NEXT number existed. That is sound for one writer walking forward and false the moment two runs share an output — which is what the plan gives an output whenever it places a second encoder, and what a one-segment interval guarantees. Field 2026-09-08: `segment-00057.mp4` was served at 2 268 361 bytes and then at 4 510 940, exactly half; `segment-00055.mp4` at 211 957 and then 2 620 617. The browser appended the half, refused the whole one for the rest of the session and repeated `bufferAppendError` to the end of the log with the picture frozen at 319.66 s.
|
|
4
|
+
**A piece's NAME is the proof now, and there is no second one.** It is written as `making-<from>-00057.mp4` — tagged with the first number of its run's stretch — and takes `segment-00057.mp4` when the encoder says on its own channel that it has closed it, which is one rename inside one directory. The `hls` branch needed nothing added: its muxer already writes through a temporary name of its own. One rule for both branches, and true whether or not this process is alive.
|
|
5
|
+
- **Fix**: The last piece of every run is provable. Under the successor rule nothing followed it, so it never was — which is the resume case that held a segment for 46 s and then answered 404 to a browser that had given up.
|
|
6
|
+
- **Fix**: Clearing up after a dead run is a name match against that run's own tag: no stretch to search, no bytes to judge, and no way to remove a complete piece another run closed. `services/encode/open-piece.js` did all three by guessing, and under the naming rule its guess would have removed a finished segment — the highest served name in a dead run's stretch is a piece that run closed. It is deleted, together with the session manager's own copy of it, which fetched the output's init bytes to judge a file by its contents. 135 lines of service code and 152 of tests for the guessing are gone; the manager is 83 lines shorter and 5 longer.
|
|
7
|
+
- **Fix**: One owner for "this piece is closed". A set of statements was kept beside the disk and told by whoever noticed a piece being produced — the same two-owner fault 2.80.11 removed from the coverage map, one layer down. A rename cannot go out of step with itself.
|
|
8
|
+
- **New**: `encode-plan on <output>` carries every term the decision was made from: `[speed=4.45x firstByte=1.26s kill=0.04s refetch=0.000s/s maxRuns=2 live=1]`. An interval says what was decided; only these say why. A decision of this plan is `delay + (index - at) / rate + madeBetween * refetch` against a deadline, so a recorded decision without the rate can be re-read and not recomputed — which is exactly what happened with the one-piece intervals of 2026-09-08, where the rate was substituted six times from the speeds the session reported elsewhere and none of them gave the answer the plan had given. A zero in the last three is a measurement nobody has taken, not a free operation, and it is printed so that reading it as free is a choice.
|
|
9
|
+
- **Chore**: 965 checks pass, biome clean. NOT yet seen in the field. What the next session must show: `making-*` files present in an output's directory while it is being written and none left after it; no `bufferAppendError` following a segment served twice at different sizes; and the terms above beside every placement.
|
|
10
|
+
|
|
1
11
|
## 2.80.18
|
|
2
12
|
|
|
3
13
|
- **Fix**: The `Infinity` is gone, and with it a double count I had introduced. The measured "time to a first piece" already CONTAINS one piece's encoding, and I was adding another; separated, the two scale differently — a piece costs more when encoders share the machine, a spawn does not. So a fresh encoder owes `spawn overhead + the piece at the rate in force`, a moved one owes the kill and then the same. With nothing measured the overhead is zero and a fresh encoder owes exactly one piece, which is the floor and is DERIVED rather than chosen: a piece cannot appear before it is encoded, and how fast this host encodes is measured before any viewer exists. The `Infinity` was an exception in a model that needs none. Verified by simulation over eighty ticks against the map's real shape — zero moves and zero one-piece intervals at a budget of one encoder and of three.
|
|
@@ -321,10 +321,25 @@ places three.
|
|
|
321
321
|
|
|
322
322
|
```
|
|
323
323
|
encode-plan on <output>: start #58..#481, stop #?..#?
|
|
324
|
+
[speed=4.45x firstByte=1.26s kill=0.04s refetch=0.000s/s maxRuns=2 live=1]
|
|
324
325
|
```
|
|
325
326
|
|
|
326
|
-
Every action with its INTERVAL, which is what a run is
|
|
327
|
-
does something — a session
|
|
327
|
+
Every action with its INTERVAL, which is what a run is, and then every term the
|
|
328
|
+
decision was made from. Printed on any pass that does something — a session
|
|
329
|
+
where nothing changes says nothing.
|
|
330
|
+
|
|
331
|
+
The terms are there because an interval says WHAT was decided and only these say
|
|
332
|
+
WHY. A decision of this plan is
|
|
333
|
+
|
|
334
|
+
delay + (index - at) / rate + madeBetween * refetch against a deadline
|
|
335
|
+
|
|
336
|
+
so a recorded decision without the rate can be re-read and not recomputed. That
|
|
337
|
+
is not hypothetical: the one-piece intervals of 2026-09-08 were diagnosed by
|
|
338
|
+
substituting the rate from the speeds the session reported elsewhere — six
|
|
339
|
+
different figures, none of which reproduced the answer the plan had given.
|
|
340
|
+
|
|
341
|
+
A zero in `firstByte`, `kill` or `refetch` is a measurement nobody has taken, not
|
|
342
|
+
a free operation. It is printed so that reading it as free is a choice.
|
|
328
343
|
|
|
329
344
|
It was missing, and its absence cost three wrong diagnoses of one field session.
|
|
330
345
|
The line printed the windows, the budget and where the live runs stood; the
|
|
@@ -336,6 +351,35 @@ reasons printed beside them read as moves back and forth, so the fault was read
|
|
|
336
351
|
as an oscillating placement three times over. An interval of one segment turns
|
|
337
352
|
the protection against two encoders writing one name into a mill for processes.
|
|
338
353
|
|
|
354
|
+
## What proves a segment is finished
|
|
355
|
+
|
|
356
|
+
Its NAME, and there is nothing else. A piece being written is called
|
|
357
|
+
`making-<from>-00042.mp4` — the tag is the first number of the stretch its run
|
|
358
|
+
was given — and it takes `segment-00042.mp4` when the encoder says it has closed
|
|
359
|
+
it, which it does on a channel of its own (`-segment_list pipe:3`). Making it
|
|
360
|
+
servable is therefore one rename inside one directory, performed by the store
|
|
361
|
+
because the store owns the disk. The `hls` branch needs nothing extra: its muxer
|
|
362
|
+
writes through a temporary name of its own, so its files appear under their final
|
|
363
|
+
name whole.
|
|
364
|
+
|
|
365
|
+
Three things follow, and each replaced a guess:
|
|
366
|
+
|
|
367
|
+
1. **a request can never reach a half-written piece.** Closure used to be
|
|
368
|
+
inferred from the NEXT number existing — sound for one writer walking forward,
|
|
369
|
+
false the moment two runs share an output, which is what the plan gives an
|
|
370
|
+
output whenever it places a second encoder. Field 2026-09-08:
|
|
371
|
+
`segment-00057.mp4` served at 2 268 361 bytes and then at 4 510 940, exactly
|
|
372
|
+
half; the browser appended the half and refused the whole for the rest of the
|
|
373
|
+
session, with the picture frozen at 319.66 s;
|
|
374
|
+
2. **the last piece of a run is provable.** Under the successor rule nothing
|
|
375
|
+
followed it, so it never was — the resume case that held one segment for 46 s
|
|
376
|
+
and then answered 404;
|
|
377
|
+
3. **clearing up after a dead run is a name match.** Its unfinished pieces are
|
|
378
|
+
the ones carrying its own tag: no stretch to search, no bytes to judge, and no
|
|
379
|
+
way to remove a complete piece somebody else closed. `services/encode/
|
|
380
|
+
open-piece.js` did all three of those by guessing and is gone, along with the
|
|
381
|
+
session manager's copy of it.
|
|
382
|
+
|
|
339
383
|
## What is checked
|
|
340
384
|
|
|
341
385
|
`test/one-authority.test.js` holds the shape: one caller of `#startEncodeRun`,
|
|
@@ -348,3 +392,8 @@ viewer registry, the real `LiveOutputs` and the real `PriorityOrchestrator`.
|
|
|
348
392
|
|
|
349
393
|
`test/encode-plan.test.js` holds the arithmetic, including that every encoder
|
|
350
394
|
stops when nobody is watching the output.
|
|
395
|
+
|
|
396
|
+
`test/segment-store.test.js` holds the naming rule: a piece under its served
|
|
397
|
+
name is finished — the last one of a run included — one under a working name is
|
|
398
|
+
not and cannot be reached, closing it is one rename, and clearing up after one
|
|
399
|
+
run leaves every other run's work alone.
|
package/package.json
CHANGED
|
@@ -181,9 +181,11 @@ export class EncodeRun {
|
|
|
181
181
|
* @param {{ info: (line: string) => void, warn: (line: string) => void, error?: (line: string) => void }} params.logger
|
|
182
182
|
* @param {() => number} [params.now]
|
|
183
183
|
* @param {(ended: RunEnded) => void} [params.onEnded]
|
|
184
|
-
* @param {(name: string) =>
|
|
185
|
-
* of every piece the encoder has
|
|
186
|
-
* names it on its own channel
|
|
184
|
+
* @param {(name: string) => string | null} [params.onClosed] - Called with the
|
|
185
|
+
* WORKING name of every piece the encoder has finished writing, as the
|
|
186
|
+
* encoder itself names it on its own channel, and answers with the name that
|
|
187
|
+
* piece is served under — because making it servable is a rename, and only
|
|
188
|
+
* whoever owns the disk can perform one.
|
|
187
189
|
* @param {(progress: { processedSeconds: number | null, speed: string | null }) => void} [params.onProgress]
|
|
188
190
|
* Called for every `-progress` report. Seconds count from the START OF THIS
|
|
189
191
|
* RUN on both branches — neither `-output_ts_offset` nor `-copyts` changes
|
|
@@ -473,8 +475,13 @@ export class EncodeRun {
|
|
|
473
475
|
if (name.length === 0) {
|
|
474
476
|
continue;
|
|
475
477
|
}
|
|
478
|
+
// ITS SERVED NAME, which is what whoever owns the disk gives it in answer.
|
|
479
|
+
// ffmpeg writes a piece under a working name and reports that; the piece
|
|
480
|
+
// becomes servable by being renamed, and everything below works in the
|
|
481
|
+
// name a request can actually ask for.
|
|
482
|
+
const served = this.onClosed(name) ?? name;
|
|
476
483
|
if (!this.#stopping) {
|
|
477
|
-
this.#provenName =
|
|
484
|
+
this.#provenName = served;
|
|
478
485
|
}
|
|
479
486
|
// WHAT THIS RUN HAS MADE IS THIS RUN'S OWN FACT, and this channel is where
|
|
480
487
|
// it learns it. It used to be told from outside, by whoever listed the
|
|
@@ -484,11 +491,10 @@ export class EncodeRun {
|
|
|
484
491
|
// (483 segment(s))", having produced none of them, and its head therefore
|
|
485
492
|
// described somebody else's work. Both the claim it holds and the cleanup
|
|
486
493
|
// after it read that head.
|
|
487
|
-
const index = this.indexOfName(
|
|
494
|
+
const index = this.indexOfName(served);
|
|
488
495
|
if (Number.isInteger(index)) {
|
|
489
496
|
this.noteProduced(index);
|
|
490
497
|
}
|
|
491
|
-
this.onClosed(name);
|
|
492
498
|
}
|
|
493
499
|
}
|
|
494
500
|
|
|
@@ -19,28 +19,32 @@
|
|
|
19
19
|
* killed, work out what it is looking at — without it, everything on disk after
|
|
20
20
|
* a kill is unidentifiable and can only be thrown away.
|
|
21
21
|
*
|
|
22
|
-
* **What proves a segment is closed.**
|
|
23
|
-
*
|
|
24
|
-
*
|
|
25
|
-
*
|
|
26
|
-
*
|
|
27
|
-
*
|
|
28
|
-
*
|
|
29
|
-
*
|
|
22
|
+
* **What proves a segment is closed: its NAME.** A piece being written is called
|
|
23
|
+
* something else — `making-40-00057.mp4`, tagged with the run writing it — and
|
|
24
|
+
* takes its served name only when the
|
|
25
|
+
* encoder has said it is closed, which it does on a channel of its own
|
|
26
|
+
* (`-segment_list pipe:3`). The `hls` branch needs nothing extra: its muxer
|
|
27
|
+
* writes through a temporary name of its own, so its files appear under their
|
|
28
|
+
* final name whole. One rule for both, and true whether or not this process is
|
|
29
|
+
* alive: **a file under its served name is complete.**
|
|
30
30
|
*
|
|
31
|
-
*
|
|
32
|
-
*
|
|
33
|
-
*
|
|
31
|
+
* It was "a segment is closed when the NEXT number exists". That is true of one
|
|
32
|
+
* writer walking forward and false the moment two runs share an output, because
|
|
33
|
+
* the next file is then written by another process while this one is still open
|
|
34
|
+
* — and two runs on one output is not a rare state, it is what the plan gives an
|
|
35
|
+
* output whenever it places a second encoder. Field 2026-09-08:
|
|
36
|
+
* `segment-00057.mp4` served at 2 268 361 bytes and then at 4 510 940, exactly
|
|
37
|
+
* half; the browser appended the half and refused the whole for the rest of the
|
|
38
|
+
* session, `bufferAppendError` fourteen times with the picture frozen at
|
|
39
|
+
* 319.66 s. `segment-00055.mp4` the same, 211 957 against 2 620 617.
|
|
34
40
|
*/
|
|
35
41
|
|
|
36
42
|
import { createHash } from "node:crypto";
|
|
37
|
-
import { existsSync, mkdirSync, readdirSync, readFileSync, statSync, writeFileSync } from "node:fs";
|
|
43
|
+
import { existsSync, mkdirSync, readdirSync, readFileSync, renameSync, statSync, writeFileSync } from "node:fs";
|
|
38
44
|
import { rmSync } from "node:fs";
|
|
39
45
|
import os from "node:os";
|
|
40
46
|
import path from "node:path";
|
|
41
47
|
|
|
42
|
-
import { discardOpenPiece } from "./open-piece.js";
|
|
43
|
-
|
|
44
48
|
/** Where every output's segments live. One root for the process. */
|
|
45
49
|
export const DEFAULT_STORE_ROOT = path.join(os.tmpdir(), "torrent-tv-hls");
|
|
46
50
|
|
|
@@ -86,22 +90,6 @@ export class SegmentStore {
|
|
|
86
90
|
/** Output key → when it was last asked for. @type {Map<string, number>} */
|
|
87
91
|
#touched = new Map();
|
|
88
92
|
|
|
89
|
-
/**
|
|
90
|
-
* Numbers known closed for a reason other than a successor on the disk.
|
|
91
|
-
*
|
|
92
|
-
* Two things fill it. A live run says what it has finished as it finishes it.
|
|
93
|
-
* And adoption records what the successor rule proved BEFORE it removes the
|
|
94
|
-
* unproven piece — otherwise removing that piece would un-prove the segment
|
|
95
|
-
* below it, which is a file that was demonstrably closed a moment earlier.
|
|
96
|
-
*
|
|
97
|
-
* @type {Map<string, Set<number>>}
|
|
98
|
-
*/
|
|
99
|
-
#closed = new Map();
|
|
100
|
-
|
|
101
|
-
/** Pieces already reported as taken on the successor rule, so one is said
|
|
102
|
-
* once. @type {Map<string, Set<number>>} */
|
|
103
|
-
#unreportedSaid = new Map();
|
|
104
|
-
|
|
105
93
|
/** @type {{ info: Function, warn: Function }} */
|
|
106
94
|
#logger;
|
|
107
95
|
|
|
@@ -250,42 +238,28 @@ export class SegmentStore {
|
|
|
250
238
|
}
|
|
251
239
|
|
|
252
240
|
/**
|
|
253
|
-
* The segment numbers this output holds that are
|
|
254
|
-
*
|
|
255
|
-
*
|
|
256
|
-
*
|
|
257
|
-
*
|
|
258
|
-
*
|
|
259
|
-
*
|
|
260
|
-
*
|
|
261
|
-
*
|
|
262
|
-
*
|
|
263
|
-
*
|
|
264
|
-
*
|
|
265
|
-
*
|
|
266
|
-
*
|
|
267
|
-
*
|
|
268
|
-
*
|
|
269
|
-
* a LIVE RUN IS REWRITING. Several runs share one directory, so a file left by
|
|
270
|
-
* an earlier one is a successor to a name the run working now has just
|
|
271
|
-
* reopened, and the disk cannot know the difference. Whether that has ever
|
|
272
|
-
* moved a decision is not established from any log we hold, and every remedy
|
|
273
|
-
* for it changes what "ready" means for five readers with different questions
|
|
274
|
-
* — so it waits for a session that shows it, rather than being guessed at.
|
|
241
|
+
* The segment numbers this output holds that are finished.
|
|
242
|
+
*
|
|
243
|
+
* ONE PROOF, AND IT IS THE PIECE'S OWN NAME. A piece being written is called
|
|
244
|
+
* `making-40-00042.mp4`; it is renamed to `segment-00042.mp4` when its writer
|
|
245
|
+
* says it has closed it, and on the `hls` branch — which has no such channel —
|
|
246
|
+
* the muxer's own `+temp_file` does the same rename for the same reason. So a
|
|
247
|
+
* file under the served name is complete, whoever made it and whenever.
|
|
248
|
+
*
|
|
249
|
+
* WHAT THIS REPLACED, because the difference is what a viewer felt. Closure
|
|
250
|
+
* used to be inferred from the NEXT number existing, which is sound for one
|
|
251
|
+
* writer walking forward and false the moment two runs share an output — and
|
|
252
|
+
* one-piece intervals guarantee that. Field 2026-09-08:
|
|
253
|
+
* `segment-00057.mp4` was served at 2 268 361 bytes and then at 4 510 940, the
|
|
254
|
+
* browser appended the truncated body, and `bufferAppendError` repeated to the
|
|
255
|
+
* end of the log with the picture frozen at 319.66 s. It also left the last
|
|
256
|
+
* piece of every run unprovable for ever, since nothing follows it.
|
|
275
257
|
*
|
|
276
258
|
* @param {string} key
|
|
277
259
|
* @returns {number[]}
|
|
278
260
|
*/
|
|
279
261
|
provenNumbers(key) {
|
|
280
|
-
|
|
281
|
-
const stated = this.#closed.get(key);
|
|
282
|
-
const proven = [];
|
|
283
|
-
for (const index of contents.byNumber.keys()) {
|
|
284
|
-
if (contents.byNumber.has(index + 1) || stated?.has(index)) {
|
|
285
|
-
proven.push(index);
|
|
286
|
-
}
|
|
287
|
-
}
|
|
288
|
-
return proven.sort((left, right) => left - right);
|
|
262
|
+
return [...this.refresh(key).byNumber.keys()].sort((left, right) => left - right);
|
|
289
263
|
}
|
|
290
264
|
|
|
291
265
|
/**
|
|
@@ -317,189 +291,151 @@ export class SegmentStore {
|
|
|
317
291
|
return this.refresh(key).largest ?? { index: -1, size: 0 };
|
|
318
292
|
}
|
|
319
293
|
|
|
320
|
-
/**
|
|
321
|
-
* A run is about to write these numbers again: forget that they were closed.
|
|
322
|
-
*
|
|
323
|
-
* A number closed once is not closed for ever. An encoder started at #N
|
|
324
|
-
* rewrites #N and everything after it, and while it is doing so the file
|
|
325
|
-
* under that name is half a segment — but the store remembered the earlier
|
|
326
|
-
* closing and would call it whole. Field 2026-09-05: seventeen runs were
|
|
327
|
-
* stopped and none ended normally, so numbers were being rewritten
|
|
328
|
-
* constantly, and the player met a fatal append error it never recovered
|
|
329
|
-
* from — an empty picture for the six minutes that followed.
|
|
330
|
-
*
|
|
331
|
-
* @param {string} key
|
|
332
|
-
* @param {number} from - First number the run will write.
|
|
333
|
-
* @param {number} [to] - Last one, inclusive. Infinite for a run given no end,
|
|
334
|
-
* which does walk to the end of the film.
|
|
335
|
-
*/
|
|
336
|
-
forgetClosed(key, from, to = Number.POSITIVE_INFINITY) {
|
|
337
|
-
const known = this.#closed.get(key);
|
|
338
|
-
if (!known || !Number.isInteger(from)) {
|
|
339
|
-
return;
|
|
340
|
-
}
|
|
341
|
-
// BOUNDED BY THE RUN'S OWN STRETCH, because that is what it will rewrite.
|
|
342
|
-
//
|
|
343
|
-
// It used to forget everything from `from` upwards, on the reading that a
|
|
344
|
-
// run has no end — which was true until runs were given intervals. A run of
|
|
345
|
-
// #0..#0 then unproved the whole rest of the film, and with readiness a
|
|
346
|
-
// projection of what is proven that is an output declaring itself unmade
|
|
347
|
-
// every time an encoder starts anywhere near the beginning.
|
|
348
|
-
const last = Number.isFinite(to) ? Math.max(from, Math.trunc(to)) : Number.POSITIVE_INFINITY;
|
|
349
|
-
for (const index of known) {
|
|
350
|
-
if (index >= from && index <= last) {
|
|
351
|
-
known.delete(index);
|
|
352
|
-
}
|
|
353
|
-
}
|
|
354
|
-
// What the directory says has to be read again too, so that the size of a
|
|
355
|
-
// reopened piece is the size it has now and not the one it had before.
|
|
356
|
-
this.#held.delete(key);
|
|
357
|
-
}
|
|
358
294
|
|
|
359
295
|
/**
|
|
360
296
|
* Whether this piece is finished, and may therefore be served.
|
|
361
297
|
*
|
|
362
|
-
*
|
|
363
|
-
*
|
|
364
|
-
*
|
|
365
|
-
*
|
|
366
|
-
*
|
|
367
|
-
* 2. **the next file exists** — the only proof available on the `hls` branch,
|
|
368
|
-
* which has no such channel, and for pieces left by an earlier life of this
|
|
369
|
-
* process. On that branch it is sound: the muxer renames into place on
|
|
370
|
-
* close, so a file that exists is finished.
|
|
371
|
-
*
|
|
372
|
-
* KNOWN AND LEFT ALONE HERE: on the `segment` branch this second proof can
|
|
373
|
-
* still pass a piece a run is halfway through rewriting, which is how 110 698
|
|
374
|
-
* bytes came to be served under a name whose neighbours are 12 MB (field
|
|
375
|
-
* 2026-09-06). Telling the two branches apart is a fact of how a run writes,
|
|
376
|
-
* it needs a field session of its own to verify, and it is not what stopped
|
|
377
|
-
* playback on 2026-09-07 — so it stays open rather than being changed blind in
|
|
378
|
-
* the path that hands bytes to a player. What the PLAN believes is a different
|
|
379
|
-
* question and is answered: a live run's claim outranks readiness there.
|
|
298
|
+
* Its NAME is the proof, and there is no second one: a piece being written is
|
|
299
|
+
* called something else until whoever writes it says it is closed. That holds
|
|
300
|
+
* for a piece a live run is rewriting — the file standing there was closed by
|
|
301
|
+
* somebody, and it is replaced whole or not at all — and for a piece left by an
|
|
302
|
+
* earlier life of this process, which the startup sweep answers the same way.
|
|
380
303
|
*
|
|
381
304
|
* @param {string} key
|
|
382
305
|
* @param {number} index
|
|
383
306
|
* @returns {boolean}
|
|
384
307
|
*/
|
|
385
308
|
isClosed(key, index) {
|
|
386
|
-
|
|
387
|
-
return true;
|
|
388
|
-
}
|
|
389
|
-
const bySuccessor = this.refresh(key).byNumber.has(index + 1);
|
|
390
|
-
if (bySuccessor) {
|
|
391
|
-
this.#noteUnreported(key, index);
|
|
392
|
-
}
|
|
393
|
-
return bySuccessor;
|
|
309
|
+
return this.refresh(key).byNumber.has(index);
|
|
394
310
|
}
|
|
395
311
|
|
|
396
312
|
/**
|
|
397
|
-
*
|
|
398
|
-
*
|
|
399
|
-
*
|
|
400
|
-
*
|
|
401
|
-
*
|
|
402
|
-
*
|
|
403
|
-
* anything wrote one, and then its name promises a whole span while it holds
|
|
404
|
-
* a fraction. Field 2026-09-06: 110 698 bytes served under a name whose
|
|
405
|
-
* neighbours are 12 MB, 40 ms of film where the playlist declared 10.4 s, and
|
|
406
|
-
* the player jumped the hole it left.
|
|
407
|
-
*
|
|
408
|
-
* That cannot arise from two encoders any more — their stretches no longer
|
|
409
|
-
* overlap — so what is left is a piece from a process that died without
|
|
410
|
-
* clearing up. Said once per piece, with its size, so a return of it is
|
|
411
|
-
* visible rather than inferred.
|
|
313
|
+
* Remove the pieces an output was in the middle of writing.
|
|
314
|
+
*
|
|
315
|
+
* They are under working names, so they were never servable and nothing has
|
|
316
|
+
* to be un-proven — this is disk, not correctness. A process killed by the
|
|
317
|
+
* kernel leaves one per live run, and the kernel takes this process often
|
|
318
|
+
* enough for that to matter.
|
|
412
319
|
*
|
|
413
320
|
* @param {string} key
|
|
414
|
-
* @param {
|
|
321
|
+
* @param {string} dir
|
|
322
|
+
* @param {{ makingTagOf?: (name: string) => string | null }} format
|
|
323
|
+
* @param {string | null} [tag] - One run's own tag, or null for every run's.
|
|
324
|
+
* @returns {number} How many were removed.
|
|
415
325
|
*/
|
|
416
|
-
#
|
|
417
|
-
let
|
|
418
|
-
|
|
419
|
-
said = new Set();
|
|
420
|
-
this.#unreportedSaid.set(key, said);
|
|
421
|
-
}
|
|
422
|
-
if (said.has(index)) {
|
|
423
|
-
return;
|
|
424
|
-
}
|
|
425
|
-
said.add(index);
|
|
426
|
-
let bytes = -1;
|
|
326
|
+
#sweepUnfinished(key, dir, format, tag = null) {
|
|
327
|
+
let removed = 0;
|
|
328
|
+
let names = [];
|
|
427
329
|
try {
|
|
428
|
-
|
|
330
|
+
names = readdirSync(dir);
|
|
429
331
|
} catch {
|
|
430
|
-
|
|
431
|
-
return;
|
|
332
|
+
return 0;
|
|
432
333
|
}
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
|
|
437
|
-
|
|
334
|
+
for (const name of names) {
|
|
335
|
+
const wroteIt = format?.makingTagOf?.(name) ?? null;
|
|
336
|
+
if (wroteIt === null || (tag !== null && wroteIt !== tag)) {
|
|
337
|
+
continue;
|
|
338
|
+
}
|
|
339
|
+
try {
|
|
340
|
+
rmSync(path.join(dir, name), { force: true });
|
|
341
|
+
removed += 1;
|
|
342
|
+
} catch {
|
|
343
|
+
// Then it stays, costing disk and nothing else.
|
|
344
|
+
}
|
|
345
|
+
}
|
|
346
|
+
if (removed > 0) {
|
|
347
|
+
this.#held.delete(key);
|
|
348
|
+
}
|
|
349
|
+
return removed;
|
|
438
350
|
}
|
|
439
351
|
|
|
440
352
|
/**
|
|
441
|
-
*
|
|
442
|
-
*
|
|
443
|
-
*
|
|
444
|
-
*
|
|
353
|
+
* The encoder has closed a piece: give it the name it is served under.
|
|
354
|
+
*
|
|
355
|
+
* One rename inside the output's own directory — one filesystem operation, and
|
|
356
|
+
* atomic there. Before it the file is not a segment and no request can reach
|
|
357
|
+
* it; after it, its existence IS the proof that it is whole, and that is one
|
|
358
|
+
* rule for every branch whether or not our own process is alive.
|
|
359
|
+
*
|
|
360
|
+
* It replaced a rule that served half a segment: a piece was taken as finished
|
|
361
|
+
* when the NEXT file existed. That is true of one writer walking forward and
|
|
362
|
+
* false the moment two runs share an output, because the next file is then
|
|
363
|
+
* written by another process while this one is still open — and two runs on one
|
|
364
|
+
* output is not a rare state, it is what the plan gives an output whenever it
|
|
365
|
+
* places a second encoder.
|
|
366
|
+
*
|
|
367
|
+
* Field 2026-09-08: `segment-00057.mp4` was served at 2 268 361 bytes and then
|
|
368
|
+
* at 4 510 940 — exactly half of it. The browser appended the half and refused
|
|
369
|
+
* the whole for the rest of the session, `bufferAppendError` fourteen times
|
|
370
|
+
* over with the picture frozen at 319.66 s. `segment-00055.mp4` went the same
|
|
371
|
+
* way, 211 957 against 2 620 617.
|
|
445
372
|
*
|
|
446
373
|
* @param {string} key
|
|
447
|
-
* @param {
|
|
374
|
+
* @param {string} makingName - What the encoder called it while writing.
|
|
375
|
+
* @param {{ servedNameOf?: (name: string) => string | null }} format
|
|
376
|
+
* @returns {string | null} The served name, or null where nothing was renamed.
|
|
448
377
|
*/
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
378
|
+
publish(key, makingName, format) {
|
|
379
|
+
const served = format?.servedNameOf?.(makingName) ?? null;
|
|
380
|
+
if (!served) {
|
|
381
|
+
return null;
|
|
452
382
|
}
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
|
|
383
|
+
const dir = path.join(this.#root, directoryNameFor(key));
|
|
384
|
+
try {
|
|
385
|
+
renameSync(path.join(dir, makingName), path.join(dir, served));
|
|
386
|
+
} catch (error) {
|
|
387
|
+
// The file may already be gone — a process killed between closing the
|
|
388
|
+
// piece and this line. Said rather than swallowed: a piece the encoder
|
|
389
|
+
// reported and the disk does not have is worth knowing about.
|
|
390
|
+
this.#logger?.warn?.(
|
|
391
|
+
`segment store: could not publish ${makingName} of ${key.slice(0, 60)}: ` +
|
|
392
|
+
`${error instanceof Error ? error.message : String(error)}`
|
|
393
|
+
);
|
|
394
|
+
return null;
|
|
457
395
|
}
|
|
458
|
-
|
|
396
|
+
// What the directory holds has changed, so the memory of it is stale.
|
|
397
|
+
this.#held.delete(key);
|
|
398
|
+
return served;
|
|
459
399
|
}
|
|
460
400
|
|
|
461
401
|
/**
|
|
462
|
-
*
|
|
402
|
+
* Clear up after a run that has ended: remove what it left unfinished.
|
|
403
|
+
*
|
|
404
|
+
* Every file it left open carries its own tag, so this is a name match and
|
|
405
|
+
* nothing else — no stretch to search, no bytes to judge, and no chance of
|
|
406
|
+
* removing a piece somebody else closed.
|
|
463
407
|
*
|
|
464
|
-
*
|
|
465
|
-
*
|
|
466
|
-
*
|
|
467
|
-
*
|
|
468
|
-
*
|
|
469
|
-
*
|
|
470
|
-
* whose name then read as a segment made).
|
|
408
|
+
* WHAT IT REPLACED, because the difference is the whole of the rename design.
|
|
409
|
+
* It used to take the highest SERVED name inside the stretch the ended run was
|
|
410
|
+
* given and judge whether its bytes looked usable — a guess, needed only
|
|
411
|
+
* because an unfinished piece was indistinguishable from a finished one. Under
|
|
412
|
+
* the naming rule it would now remove a complete segment: the highest served
|
|
413
|
+
* name in a dead run's stretch is a piece it closed.
|
|
471
414
|
*
|
|
472
415
|
* @param {string} key
|
|
473
|
-
* @param {
|
|
474
|
-
*
|
|
475
|
-
* has to be looked for inside the stretch the ended run was given.
|
|
476
|
-
* @param {((raw: Buffer) => boolean) | null} [judgeUsable]
|
|
477
|
-
* @returns {Promise<number | null>} The segment number removed, or null.
|
|
416
|
+
* @param {number} startedAt - The run's first segment number, which is its tag.
|
|
417
|
+
* @returns {number} How many unfinished pieces were removed.
|
|
478
418
|
*/
|
|
479
|
-
|
|
419
|
+
clearUpAfter(key, startedAt) {
|
|
480
420
|
const format = this.#formats.get(key);
|
|
481
421
|
if (!format) {
|
|
482
|
-
return
|
|
422
|
+
return 0;
|
|
483
423
|
}
|
|
484
|
-
const removed =
|
|
485
|
-
|
|
486
|
-
this
|
|
424
|
+
const removed = this.#sweepUnfinished(
|
|
425
|
+
key,
|
|
426
|
+
this.directoryFor(key),
|
|
427
|
+
format,
|
|
428
|
+
String(Number.isInteger(startedAt) && startedAt > 0 ? startedAt : 0)
|
|
429
|
+
);
|
|
430
|
+
if (removed > 0) {
|
|
487
431
|
this.#logger?.info?.(
|
|
488
|
-
`segment store:
|
|
432
|
+
`segment store: cleared up ${removed} unfinished piece(s) of the run at ` +
|
|
433
|
+
`#${startedAt} on ${key.slice(0, 60)}`
|
|
489
434
|
);
|
|
490
435
|
}
|
|
491
436
|
return removed;
|
|
492
437
|
}
|
|
493
438
|
|
|
494
|
-
/**
|
|
495
|
-
* The one number in this output whose closure nothing on disk proves.
|
|
496
|
-
*
|
|
497
|
-
* @param {string} key
|
|
498
|
-
* @returns {number} -1 when the directory holds no segments.
|
|
499
|
-
*/
|
|
500
|
-
unprovenNumber(key) {
|
|
501
|
-
return this.refresh(key).unproven;
|
|
502
|
-
}
|
|
503
439
|
|
|
504
440
|
/**
|
|
505
441
|
* Where a segment is, or null when this output does not hold it.
|
|
@@ -542,7 +478,6 @@ export class SegmentStore {
|
|
|
542
478
|
this.#held.delete(key);
|
|
543
479
|
this.#formats.delete(key);
|
|
544
480
|
this.#touched.delete(key);
|
|
545
|
-
this.#closed.delete(key);
|
|
546
481
|
this.#logger.info(`segment-store dropped ${directoryNameFor(key)} (${because})`);
|
|
547
482
|
}
|
|
548
483
|
|
|
@@ -712,30 +647,19 @@ export class SegmentStore {
|
|
|
712
647
|
continue;
|
|
713
648
|
}
|
|
714
649
|
this.#formats.set(entry.key, format);
|
|
715
|
-
//
|
|
716
|
-
//
|
|
717
|
-
//
|
|
718
|
-
|
|
719
|
-
|
|
720
|
-
|
|
721
|
-
|
|
722
|
-
|
|
723
|
-
|
|
724
|
-
const filePath = held.byNumber.get(unproven);
|
|
725
|
-
if (filePath) {
|
|
726
|
-
try {
|
|
727
|
-
rmSync(filePath, { force: true });
|
|
728
|
-
unprovenRemoved += 1;
|
|
729
|
-
} catch {
|
|
730
|
-
// Then it stays unproven and is simply never served.
|
|
731
|
-
}
|
|
732
|
-
}
|
|
733
|
-
this.#held.delete(entry.key);
|
|
734
|
-
}
|
|
650
|
+
// EVERY SEGMENT FOUND IS COMPLETE, because a piece is given its served
|
|
651
|
+
// name only once the encoder has said it is closed. So there is nothing to
|
|
652
|
+
// prove here and nothing to un-prove: what the directory holds under
|
|
653
|
+
// served names is what a killed process finished.
|
|
654
|
+
//
|
|
655
|
+
// What it may also hold is pieces it was in the middle of, under their
|
|
656
|
+
// working names, and those are swept — the file a run was writing when the
|
|
657
|
+
// kernel took the process is exactly this.
|
|
658
|
+
unprovenRemoved += this.#sweepUnfinished(entry.key, entry.dir, format);
|
|
735
659
|
adopted += 1;
|
|
736
660
|
this.#logger.info(
|
|
737
661
|
`segment-store adopted ${path.basename(entry.dir)}: ${this.provenNumbers(entry.key).length} ` +
|
|
738
|
-
|
|
662
|
+
"segments a killed process had already finished"
|
|
739
663
|
);
|
|
740
664
|
}
|
|
741
665
|
return { adopted, dropped, unprovenRemoved };
|