@torrent-tv/proxy 2.82.0 → 2.83.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +9 -0
- package/CLAUDE.md +11 -0
- package/docs/disk-architecture.md +161 -0
- package/docs/encode-architecture.md +36 -7
- package/package.json +1 -1
- package/services/disk/keep.js +48 -0
- package/services/disk/returns.js +103 -0
- package/services/encode/Encoder.js +15 -0
- package/services/encode/QsvEncoder.js +5 -0
- package/services/encode/SegmentStore.js +14 -0
- package/services/encode/VaapiEncoder.js +5 -0
- package/services/encode/start-stop-cost.js +6 -2
- package/services/hls-session-manager.js +17 -1
- package/services/hwaccel.js +4 -0
- package/services/piece-store/piece-disk-store.js +71 -6
- package/services/piece-store/piece-lru.js +17 -0
- package/services/piece-store/shared-piece-store.js +1556 -1549
- package/services/torrent-pool.js +8 -7
- package/test/keeping-period.test.js +83 -0
- package/test/piece-disk-store.test.js +88 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,12 @@
|
|
|
1
|
+
## 2.83.0
|
|
2
|
+
|
|
3
|
+
- **Fix**: THE SPILLED PIECES HAD ONE RULE AND IT NEVER BOUND. The ceiling 2.81.0 gave them is a share of free space, which on the addon host is 52 292 MB against a measured growth of 14 400 MB in one viewing — so nothing was ever removed until the torrent itself went. They have the pair the segments have now: a piece behind every read head has been read and will not be read again unless somebody seeks back, and a seek back re-downloads it, which is the bargain this tier already makes whenever it drops a piece for room. Under pressure the order is the same as the segments’ — behind the earliest reader first, furthest behind first of all — instead of by which piece happened to be touched longest ago, which says nothing about what anybody is about to read.
|
|
4
|
+
- **Fix**: THREE PERIODS FOR ONE UNKNOWN, AND THEY CONTRADICTED EACH OTHER. A torrent and its downloaded bytes went at fifteen minutes, a session lived to thirty, the produced segments to six hours — so between the first two there was a session with no source: a viewer returning at the twentieth minute got a session that could not make a single new segment until the torrent was added again. All three stand for the same unmeasured thing, whether the viewer comes back, so they are ONE number in one place (`services/disk/keep.js`, one hour) and the session now dies first, which is the right order and needs no rule of its own.
|
|
5
|
+
- **New**: The thing those periods stand for is being measured, and the proxy is the only place it can be. A session opened on an output whose segments are still on disk IS a return, and its age is exactly what the store recorded. `returns: 14 session(s) opened on material still held, 3 on material gone; median 12min after the last read, longest 51min — kept for 60min` — said beside the disk figures. The median and the longest are the two the period has to sit between, and a week of ordinary use replaces the hour with what viewers actually do.
|
|
6
|
+
- **Fix**: The benchmark of a device-backed encoder could not have run. VAAPI and QSV take their device in the arguments before the input, and what is there for a session sets up hardware DECODING — which a benchmark fed raw frames must not have. `Encoder.benchmarkInputArgs` states the device and nothing else; without it `h264_vaapi` and `h264_qsv` fail to open and the reading would have said "this host cannot encode" about a host that encodes perfectly well. Found by reviewing 2.82.0 rather than by running it.
|
|
7
|
+
- **Chore**: `docs/disk-architecture.md` — the three things that write to one disk, the two rules that remove material, the order the viewers give it, and what is still not solved. 1009 checks pass, biome clean; the new eviction order was checked by breaking it on purpose.
|
|
8
|
+
- **Chore**: NOT yet seen in the field. What the next session must show: a `returns:` line once material has been reopened, `disk:` with both claimants, and the spill falling behind a viewer rather than only growing.
|
|
9
|
+
|
|
1
10
|
## 2.82.0
|
|
2
11
|
|
|
3
12
|
- **Fix**: THE ENCODER PING-PONGED BECAUSE NOTHING MEASURED WHAT A START COSTS. The plan compares when the wanted pieces appear under each arrangement, and for that it needs how long a fresh encoder takes to produce anything and how long killing one takes. Both were learned only from runs that had ENDED, so at a cold open both were zero — and zero does not read as “not measured”, it reads as “free”. Keeping a warming encoder then cost one piece and moving it cost `0 + 0 + one piece`: the same figure to the millisecond, so the tie fell to position and any advantage however small won. Field 2026-09-08: start at #68, a second later kill and start at #69, half a second later kill and start at #68 again, each dying having produced nothing. Over two days 153 runs were stopped that way and 68 of them made no segment at all.
|
package/CLAUDE.md
CHANGED
|
@@ -72,6 +72,17 @@ Linux-only host (e.g. POSIX-only signals must degrade elsewhere).
|
|
|
72
72
|
wanted (`demand/`) against what the swarm is told (`download/`), why urgency
|
|
73
73
|
is not a number given to the library, and why the speculative levels are
|
|
74
74
|
withdrawn rather than lowered.
|
|
75
|
+
- `disk/` — one owner of the disk, read by everything that takes any of it.
|
|
76
|
+
`DiskSpace` reads the free space once and divides it between claimants by
|
|
77
|
+
the rule memory already uses; `wire.js` says who the claimants are (the
|
|
78
|
+
produced segments, and the pieces the memory store spills, which live on the
|
|
79
|
+
torrent thread and are told their share over its channel); `free.js` reads
|
|
80
|
+
the nearest directory that exists, because the segments' own is made at the
|
|
81
|
+
first session and removed at a clean stop.
|
|
82
|
+
- `docs/disk-architecture.md` — the three things that write to one disk, the
|
|
83
|
+
two rules that remove material (nobody needs it, or there is no room), the
|
|
84
|
+
order the viewers give that removal, why a spilled piece is a file of its
|
|
85
|
+
own, and what is still not solved.
|
|
75
86
|
- `docs/encode-architecture.md` — who decides where encoders go, and the
|
|
76
87
|
answer is one authority: `EncodePlan`, from what is made, what is being
|
|
77
88
|
made, what is wanted and what the host can hold. Why no viewer reaches it,
|
|
@@ -0,0 +1,161 @@
|
|
|
1
|
+
# Disk architecture — one owner, and what goes first when it is short
|
|
2
|
+
|
|
3
|
+
Three things on this proxy write to the same disk. Until 2026-09-10 each read
|
|
4
|
+
the free space as though it were the only claimant, so there were three
|
|
5
|
+
ceilings, each standing for the whole disk.
|
|
6
|
+
|
|
7
|
+
| what | where | what bounded it before |
|
|
8
|
+
|---|---|---|
|
|
9
|
+
| segments an encoder produced | `services/encode/SegmentStore.js` | a quarter of what was free, plus a 2 GB floor — both numbers chosen by hand |
|
|
10
|
+
| pieces the memory store spilled | `services/piece-store/piece-disk-store.js` | **nothing at all** |
|
|
11
|
+
| diagnostics kept on purpose | `/data` — core dumps, heap snapshots, packet captures | a count of files, never a size |
|
|
12
|
+
|
|
13
|
+
The middle row is what it cost: one fifty-minute viewing wrote **14 400 MB** to
|
|
14
|
+
the spill file (field 2026-08-31) while the store held 312-424 MB, and free
|
|
15
|
+
space on the host fell by every megabyte of it. The bottom row is 2.9 GB of one
|
|
16
|
+
core dump on the addon host, inside a budget of "two dumps".
|
|
17
|
+
|
|
18
|
+
## One owner
|
|
19
|
+
|
|
20
|
+
`services/disk/DiskSpace.js` reads the free space once and divides it. The rule
|
|
21
|
+
is the one memory already uses, on the other reading:
|
|
22
|
+
|
|
23
|
+
```
|
|
24
|
+
allowance = free now + what we already hold - what everything that is not us has been seen to need
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
Nothing in it is a fraction chosen by hand. The reserve is measured: between two
|
|
28
|
+
readings, how much free space went away beyond what our own consumers took.
|
|
29
|
+
|
|
30
|
+
`services/disk/wire.js` says who the claimants are. The segments are told their
|
|
31
|
+
share by a call; the spilled pieces live on the torrent thread, so the share
|
|
32
|
+
travels the channel that already carries everything else and the reply says what
|
|
33
|
+
they hold — one exchange, both directions.
|
|
34
|
+
|
|
35
|
+
The reading is said every pass, in the series beside the memory one:
|
|
36
|
+
|
|
37
|
+
```
|
|
38
|
+
disk: 104584MB free; segments 0MB of 52292MB, spilled pieces 0MB of 52292MB
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
## Two rules, answering two questions
|
|
42
|
+
|
|
43
|
+
They are kept apart because they were briefly proposed as one, and that was
|
|
44
|
+
wrong.
|
|
45
|
+
|
|
46
|
+
**Time.** Material nobody needs should not sit on the owner's disk merely
|
|
47
|
+
because there is room for it. An output nobody has read for long enough goes
|
|
48
|
+
whole, whatever the free space is.
|
|
49
|
+
|
|
50
|
+
**Space.** Material everyone needs must still go when there is no room.
|
|
51
|
+
|
|
52
|
+
A ceiling alone would keep 5 GB on a household disk for six hours because the
|
|
53
|
+
disk is large. An idle rule alone would keep everything until its clock ran out,
|
|
54
|
+
however tight the disk had become.
|
|
55
|
+
|
|
56
|
+
## What goes first, and it is the viewers who decide
|
|
57
|
+
|
|
58
|
+
For the segments, the order is the priority map's own read from the other end:
|
|
59
|
+
|
|
60
|
+
1. outputs nobody is watching at all;
|
|
61
|
+
2. what lies behind the earliest viewer — furthest behind first;
|
|
62
|
+
3. what lies ahead of the furthest viewer — furthest ahead first.
|
|
63
|
+
|
|
64
|
+
A segment a viewer is standing on is never a victim. It used to be whole outputs
|
|
65
|
+
by when their directory was last read, which says nothing about what anybody is
|
|
66
|
+
about to watch.
|
|
67
|
+
|
|
68
|
+
## A piece is a file
|
|
69
|
+
|
|
70
|
+
`DiskTier` wrote every spilled piece into ONE sparse file at
|
|
71
|
+
`index * chunkLength` and answered `forget(index)` by dropping the number from a
|
|
72
|
+
set. The bytes stayed: a sparse file's blocks come back only by hole punching,
|
|
73
|
+
which Node exposes no binding for, so nothing this process could do returned a
|
|
74
|
+
single block before the whole file was removed.
|
|
75
|
+
|
|
76
|
+
`PieceDiskStore` gives each piece its own file. Removing one returns exactly its
|
|
77
|
+
blocks, needs no binding this runtime lacks, and makes the unit of eviction the
|
|
78
|
+
same as the unit of storage — so the order pieces leave in is the order we
|
|
79
|
+
choose. Over its allowance the least recently used goes, and a piece being read
|
|
80
|
+
is never the victim. A piece thrown away is answered as absent, so the torrent
|
|
81
|
+
fetches it again, which is the same bargain the memory tier makes when it
|
|
82
|
+
spills.
|
|
83
|
+
|
|
84
|
+
The read this replaced was chosen for its cost — 22.08 ms via `readFile` against
|
|
85
|
+
**7.63 ms** into a buffer we already hold, measured on the field host — and that
|
|
86
|
+
is unchanged: it is still one `read` into the caller's buffer. What it adds is
|
|
87
|
+
an `open` per read, tens of microseconds against those milliseconds.
|
|
88
|
+
|
|
89
|
+
## A clean exit leaves nothing of ours
|
|
90
|
+
|
|
91
|
+
The segments' root was removed only when it happened to be empty — from the
|
|
92
|
+
first commit of this repository, never a decision. What it protected against is
|
|
93
|
+
a second proxy sharing the root; what it did was leave every directory alone,
|
|
94
|
+
including this process's own orphans. A directory adopted at startup, owned by
|
|
95
|
+
no session, therefore survived the exit and was adopted again at the next start.
|
|
96
|
+
|
|
97
|
+
That loop is what made an orphan permanent, and it is why 5.0 GB of segments
|
|
98
|
+
from sessions that had ended hours before were on the addon host on 2026-09-10.
|
|
99
|
+
`SegmentStore.dropAll` now removes everything this process owns, root included,
|
|
100
|
+
which also gives the startup sweep its meaning back: whatever is found then is
|
|
101
|
+
from a kill.
|
|
102
|
+
|
|
103
|
+
Both kinds have both rules. For the spilled pieces the time rule is the same
|
|
104
|
+
statement one layer down: a piece behind every read head has been read and will
|
|
105
|
+
not be read again unless somebody seeks back, and a seek back re-downloads it —
|
|
106
|
+
the bargain this tier already makes whenever it drops a piece for room.
|
|
107
|
+
`PieceLru.readHeads` says where the readers stand, `PieceDiskStore.forgetBehind`
|
|
108
|
+
acts on it, and the eviction order under pressure is the same: behind the
|
|
109
|
+
earliest reader first, furthest behind first of all. With no reader at all
|
|
110
|
+
nothing is removed — a store between reads is not a store nobody wants, and the
|
|
111
|
+
torrent going idle is what empties it whole.
|
|
112
|
+
|
|
113
|
+
## How long material nobody is using is kept
|
|
114
|
+
|
|
115
|
+
One number, in one place — `services/disk/keep.js`, one hour — because
|
|
116
|
+
everything it governs stands for the same unmeasured thing: whether the viewer
|
|
117
|
+
comes back.
|
|
118
|
+
|
|
119
|
+
It was three, and they contradicted each other:
|
|
120
|
+
|
|
121
|
+
| what | was | now |
|
|
122
|
+
|---|---|---|
|
|
123
|
+
| a torrent and its downloaded bytes | 15 minutes | one hour |
|
|
124
|
+
| a session | 30 minutes | unchanged — a session is a record, not material |
|
|
125
|
+
| the segments an encoder produced | 6 hours | one hour |
|
|
126
|
+
|
|
127
|
+
The torrent went at fifteen minutes while the session it feeds lived to thirty,
|
|
128
|
+
so between them there was a session with no source: a viewer returning at the
|
|
129
|
+
twentieth minute got a session that could not make a single new segment. With
|
|
130
|
+
one hour for both kinds of material the session dies first, which is the right
|
|
131
|
+
order and needs no rule of its own to enforce.
|
|
132
|
+
|
|
133
|
+
Nothing derives the hour. Everything else in the decision is measured — a piece
|
|
134
|
+
comes back from the swarm in ~1430 ms, a segment in its own encode time, and the
|
|
135
|
+
disk has an owner that prices holding it — and only the return is unknown.
|
|
136
|
+
|
|
137
|
+
**And the return is measurable here.** A session opened on an output whose
|
|
138
|
+
segments are still on disk IS a return, and its age is exactly what the store
|
|
139
|
+
recorded. `services/disk/returns.js` keeps them and says so beside the disk
|
|
140
|
+
figures:
|
|
141
|
+
|
|
142
|
+
```
|
|
143
|
+
returns: 14 session(s) opened on material still held, 3 on material gone;
|
|
144
|
+
median 12min after the last read, longest 51min — kept for 60min
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
The median and the longest are the two the period has to sit between: shorter
|
|
148
|
+
than the median throws away material half the returns wanted, longer than the
|
|
149
|
+
longest keeps material no return has ever reached. A week of ordinary use and
|
|
150
|
+
the hour is replaced by what viewers actually do — and then the two kinds can
|
|
151
|
+
have different numbers, since their costs of coming back differ.
|
|
152
|
+
|
|
153
|
+
## What is NOT solved
|
|
154
|
+
|
|
155
|
+
**The measurement that would replace the hour has not been taken yet.** The
|
|
156
|
+
counter is in place and says nothing until a week of ordinary use has produced a
|
|
157
|
+
distribution.
|
|
158
|
+
|
|
159
|
+
**The diagnostics are not a claimant yet.** They are bounded by a count of files
|
|
160
|
+
and never by a size, and they cannot simply be thrown away when space is short: a
|
|
161
|
+
dump is the only evidence of the death it records. That needs a rule of its own.
|
|
@@ -252,13 +252,42 @@ a fresh encoder = spawn overhead + the piece at the rate in force
|
|
|
252
252
|
a moved one = the kill, and then the same
|
|
253
253
|
```
|
|
254
254
|
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
255
|
+
**Both figures are measured before any viewer exists**, and until 2026-09-10
|
|
256
|
+
neither was. They were learned only from runs that had ENDED, so at a cold open
|
|
257
|
+
both read zero — and zero does not mean "unknown" here, it means "free".
|
|
258
|
+
|
|
259
|
+
That is not a small bias, it is the whole comparison. Subtract keeping from
|
|
260
|
+
moving and what is left is exactly the killing plus the time the run has already
|
|
261
|
+
lived:
|
|
262
|
+
|
|
263
|
+
```
|
|
264
|
+
keep a warming run = firstByteWait - elapsed
|
|
265
|
+
move it = kill + spawn overhead + one piece
|
|
266
|
+
= kill + firstByteWait (spawn overhead = firstByteWait - one piece)
|
|
267
|
+
move - keep = kill + elapsed
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
`elapsed` is the warm-up a move throws away — a process started, an input
|
|
271
|
+
opened, the first bytes fetched, a decoder filled — and it is the term that
|
|
272
|
+
makes moving cost something. Set `firstByteWait` to zero and it cancels out of
|
|
273
|
+
both sides along with the kill: keeping and moving then cost the same figure to
|
|
274
|
+
the millisecond, the tie falls to position, and any advantage however small wins.
|
|
275
|
+
|
|
276
|
+
Field 2026-09-08, the first fifteen seconds of a session: start at #68, a second
|
|
277
|
+
later kill and start at #69, half a second later kill and start at #68 again,
|
|
278
|
+
each dying having produced nothing. Over two days 153 runs were stopped by the
|
|
279
|
+
plan and 68 of them made no segment at all, median life 9.6 s.
|
|
280
|
+
|
|
281
|
+
`services/encode/start-stop-cost.js` measures both from **one ffmpeg run at
|
|
282
|
+
startup**: spawn to the first piece the encoder itself announces closed, then
|
|
283
|
+
SIGTERM to exit. 0.68 s and 0.02 s on the developer's desktop. `RunCosts` starts
|
|
284
|
+
from those and replaces them with readings from real runs as they arrive.
|
|
285
|
+
|
|
286
|
+
There was an `Infinity` here once for the cost of a move, on the reasoning that
|
|
287
|
+
an unmeasured price must not license an irreversible act. It was removed as an
|
|
288
|
+
exception in a model that needs none — correctly, but what replaced it was a
|
|
289
|
+
floor of zero, which licenses the act rather than forbidding it. A measurement
|
|
290
|
+
is what a model like this needs, not an exception and not a floor.
|
|
262
291
|
|
|
263
292
|
And a run killed before producing anything is a measurement too — a lower bound
|
|
264
293
|
on the first output, and the only reading a thrash can supply, since every run in
|
package/package.json
CHANGED
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* @file How long material nobody is using is kept.
|
|
3
|
+
*
|
|
4
|
+
* ONE NUMBER, IN ONE PLACE, because everything it governs stands for the same
|
|
5
|
+
* unmeasured thing: whether the viewer comes back. Everything else in the
|
|
6
|
+
* decision is measured or measurable — re-downloading a piece from the swarm is
|
|
7
|
+
* ~1430 ms on the field host, re-making a segment is its own encode time, and
|
|
8
|
+
* the disk now has an owner that prices holding it. Only the return is unknown,
|
|
9
|
+
* and a period is what stands in for it.
|
|
10
|
+
*
|
|
11
|
+
* IT WAS THREE NUMBERS AND THEY CONTRADICTED EACH OTHER (read 2026-09-10):
|
|
12
|
+
*
|
|
13
|
+
* torrent and its downloaded bytes 15 minutes
|
|
14
|
+
* session 30 minutes
|
|
15
|
+
* produced segments 6 hours
|
|
16
|
+
*
|
|
17
|
+
* The torrent therefore went at fifteen minutes while the session it feeds
|
|
18
|
+
* lived to thirty, so between them there was a session with no source: a viewer
|
|
19
|
+
* returning at the twentieth minute got a session that could not make a single
|
|
20
|
+
* new segment until the torrent was added again. That is not two answers to one
|
|
21
|
+
* question, it is a contradiction — and it is what two independent guesses about
|
|
22
|
+
* one unknown produce.
|
|
23
|
+
*
|
|
24
|
+
* With one hour for both kinds of material the session dies first, which is the
|
|
25
|
+
* right order and needs no rule of its own to enforce.
|
|
26
|
+
*
|
|
27
|
+
* WHY AN HOUR, honestly: nothing derives it. It is a stand-in, chosen to be
|
|
28
|
+
* long enough that an interruption — a phone call, a meal — does not cost the
|
|
29
|
+
* film, and short enough that a household disk is not held for a day by
|
|
30
|
+
* somebody who is not coming back. It is the OWNER'S disk, and holding
|
|
31
|
+
* gigabytes on it because there happens to be room is taking something that is
|
|
32
|
+
* not ours.
|
|
33
|
+
*
|
|
34
|
+
* WHAT REPLACES IT, and the proxy is already in a position to measure it: every
|
|
35
|
+
* session opened on an output whose segments are still on disk IS a return, and
|
|
36
|
+
* its age is known. `services/disk/returns.js` records them. A week of those and
|
|
37
|
+
* the distribution answers this directly — keep material for as long as returns
|
|
38
|
+
* actually happen — and then the two kinds can have different numbers, since
|
|
39
|
+
* their costs of coming back differ.
|
|
40
|
+
*/
|
|
41
|
+
|
|
42
|
+
/**
|
|
43
|
+
* How long material nobody has read is kept, in milliseconds.
|
|
44
|
+
*
|
|
45
|
+
* Read by the torrent pool for a torrent and its downloaded bytes, and by the
|
|
46
|
+
* session manager for the segments an encoder produced.
|
|
47
|
+
*/
|
|
48
|
+
export const IDLE_KEEP_MS = 60 * 60 * 1000;
|
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* @file How long after material stops being read somebody asks for it again.
|
|
3
|
+
*
|
|
4
|
+
* The one term of "how long do we keep this" that nothing measures. Everything
|
|
5
|
+
* else in that decision is a measured quantity — re-downloading a piece from
|
|
6
|
+
* the swarm is ~1430 ms on the field host, re-making a segment is its own
|
|
7
|
+
* encode time, and the disk has an owner that prices holding it. Only the
|
|
8
|
+
* return is unknown, and `IDLE_KEEP_MS` is a guess standing in for it.
|
|
9
|
+
*
|
|
10
|
+
* IT IS MEASURABLE HERE AND NOWHERE ELSE. A session opened on an output whose
|
|
11
|
+
* segments are still on disk IS a return, and its age is known exactly: the
|
|
12
|
+
* store records when each output was last read. Nothing needs to be inferred.
|
|
13
|
+
*
|
|
14
|
+
* This changes no behaviour. It records, and says what it has seen once in a
|
|
15
|
+
* while, so that after a week of ordinary use the period can be derived from
|
|
16
|
+
* what viewers actually do instead of from what a period felt like.
|
|
17
|
+
*/
|
|
18
|
+
|
|
19
|
+
/** How many returns are kept. Enough to see a shape, few enough to say in a line. */
|
|
20
|
+
const KEPT = 200;
|
|
21
|
+
|
|
22
|
+
export class Returns {
|
|
23
|
+
/** Ages in milliseconds, newest last. @type {number[]} */
|
|
24
|
+
#ages = [];
|
|
25
|
+
|
|
26
|
+
/** Sessions opened on material this proxy no longer had. */
|
|
27
|
+
#cold = 0;
|
|
28
|
+
|
|
29
|
+
/** Sessions opened on material that was still there. */
|
|
30
|
+
#warm = 0;
|
|
31
|
+
|
|
32
|
+
/**
|
|
33
|
+
* Note a session being opened on an output.
|
|
34
|
+
*
|
|
35
|
+
* @param {object} params
|
|
36
|
+
* @param {number | null} params.lastReadAt - When that output was last read,
|
|
37
|
+
* or null where this proxy has never held it.
|
|
38
|
+
* @param {number} params.now
|
|
39
|
+
* @returns {void}
|
|
40
|
+
*/
|
|
41
|
+
note({ lastReadAt, now }) {
|
|
42
|
+
if (!Number.isFinite(lastReadAt) || lastReadAt === null || lastReadAt <= 0) {
|
|
43
|
+
this.#cold += 1;
|
|
44
|
+
return;
|
|
45
|
+
}
|
|
46
|
+
this.#warm += 1;
|
|
47
|
+
this.#ages.push(Math.max(0, now - /** @type {number} */ (lastReadAt)));
|
|
48
|
+
while (this.#ages.length > KEPT) {
|
|
49
|
+
this.#ages.shift();
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
/**
|
|
54
|
+
* What the returns look like, or null while there have been none.
|
|
55
|
+
*
|
|
56
|
+
* The MEDIAN and the LONGEST, because those are the two the period has to sit
|
|
57
|
+
* between: shorter than the median throws away material half the returns
|
|
58
|
+
* wanted, and longer than the longest keeps material no return has ever
|
|
59
|
+
* reached.
|
|
60
|
+
*
|
|
61
|
+
* @returns {{ warm: number, cold: number, medianMs: number, longestMs: number } | null}
|
|
62
|
+
*/
|
|
63
|
+
shape() {
|
|
64
|
+
if (this.#ages.length === 0) {
|
|
65
|
+
return null;
|
|
66
|
+
}
|
|
67
|
+
const sorted = [...this.#ages].sort((left, right) => left - right);
|
|
68
|
+
const middle = Math.floor(sorted.length / 2);
|
|
69
|
+
return {
|
|
70
|
+
warm: this.#warm,
|
|
71
|
+
cold: this.#cold,
|
|
72
|
+
medianMs: sorted.length % 2 === 0 ? (sorted[middle - 1] + sorted[middle]) / 2 : sorted[middle],
|
|
73
|
+
longestMs: sorted[sorted.length - 1]
|
|
74
|
+
};
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
/**
|
|
78
|
+
* One line, for the log, or null while there is nothing to say.
|
|
79
|
+
*
|
|
80
|
+
* @param {number} keepMs - What is being kept for now, so the reading and the
|
|
81
|
+
* guess it will replace stand side by side.
|
|
82
|
+
* @returns {string | null}
|
|
83
|
+
*/
|
|
84
|
+
describe(keepMs) {
|
|
85
|
+
const shape = this.shape();
|
|
86
|
+
if (shape === null) {
|
|
87
|
+
return null;
|
|
88
|
+
}
|
|
89
|
+
return (
|
|
90
|
+
`returns: ${shape.warm} session(s) opened on material still held, ` +
|
|
91
|
+
`${shape.cold} on material gone; median ${minutes(shape.medianMs)} after the last read, ` +
|
|
92
|
+
`longest ${minutes(shape.longestMs)} — kept for ${minutes(keepMs)}`
|
|
93
|
+
);
|
|
94
|
+
}
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
/**
|
|
98
|
+
* @param {number} ms
|
|
99
|
+
* @returns {string}
|
|
100
|
+
*/
|
|
101
|
+
function minutes(ms) {
|
|
102
|
+
return `${Math.round(ms / 60000)}min`;
|
|
103
|
+
}
|
|
@@ -82,6 +82,21 @@ export class Encoder {
|
|
|
82
82
|
throw new Error(`${this.name} does not say how to build its video arguments.`);
|
|
83
83
|
}
|
|
84
84
|
|
|
85
|
+
/**
|
|
86
|
+
* What this kind needs BEFORE the input when it is being benchmarked.
|
|
87
|
+
*
|
|
88
|
+
* Not `inputArgs`: those set up hardware DECODING, and a benchmark is fed raw
|
|
89
|
+
* frames — there is nothing to decode. What a device-backed encoder still
|
|
90
|
+
* needs is the device itself, and without it `h264_vaapi` and `h264_qsv` fail
|
|
91
|
+
* to open at all, so the benchmark would answer "this host cannot encode"
|
|
92
|
+
* about a host that encodes perfectly well.
|
|
93
|
+
*
|
|
94
|
+
* @returns {string[]}
|
|
95
|
+
*/
|
|
96
|
+
benchmarkInputArgs() {
|
|
97
|
+
return [];
|
|
98
|
+
}
|
|
99
|
+
|
|
85
100
|
/**
|
|
86
101
|
* How to encode raw frames at one rung of this kind's speed ladder, for the
|
|
87
102
|
* startup benchmark and nothing else.
|
|
@@ -35,6 +35,11 @@ export class QsvEncoder extends Encoder {
|
|
|
35
35
|
};
|
|
36
36
|
}
|
|
37
37
|
|
|
38
|
+
/** @returns {string[]} */
|
|
39
|
+
benchmarkInputArgs() {
|
|
40
|
+
return this.device ? ["-qsv_device", this.device] : [];
|
|
41
|
+
}
|
|
42
|
+
|
|
38
43
|
/** @param {string | null} rung @returns {string[]} */
|
|
39
44
|
benchmarkArgs(rung = null) {
|
|
40
45
|
const preset = rung ? ["-preset", rung] : [];
|
|
@@ -481,6 +481,20 @@ export class SegmentStore {
|
|
|
481
481
|
this.#logger.info(`segment-store dropped ${directoryNameFor(key)} (${because})`);
|
|
482
482
|
}
|
|
483
483
|
|
|
484
|
+
/**
|
|
485
|
+
* When this output was last asked for, or null where it has never been.
|
|
486
|
+
*
|
|
487
|
+
* The one reading that makes the keeping period measurable rather than
|
|
488
|
+
* guessed: a session opened on an output this answers for IS a return, and
|
|
489
|
+
* this is its age.
|
|
490
|
+
*
|
|
491
|
+
* @param {string} key
|
|
492
|
+
* @returns {number | null}
|
|
493
|
+
*/
|
|
494
|
+
lastReadAt(key) {
|
|
495
|
+
return this.#touched.get(key) ?? null;
|
|
496
|
+
}
|
|
497
|
+
|
|
484
498
|
/**
|
|
485
499
|
* Throw away everything this store owns, and the root with it.
|
|
486
500
|
*
|
|
@@ -40,6 +40,11 @@ export class VaapiEncoder extends Encoder {
|
|
|
40
40
|
|
|
41
41
|
// No fps filter: VAAPI inherits the source rate and keeps keyframes on the
|
|
42
42
|
// grid via time-based -force_key_frames, so it already honours source fps.
|
|
43
|
+
/** @returns {string[]} */
|
|
44
|
+
benchmarkInputArgs() {
|
|
45
|
+
return this.device ? ["-vaapi_device", this.device] : [];
|
|
46
|
+
}
|
|
47
|
+
|
|
43
48
|
/** @param {string | null} rung @returns {string[]} */
|
|
44
49
|
benchmarkArgs(rung = null) {
|
|
45
50
|
// Raw frames live in this process; VAAPI encodes what is in the device, so
|
|
@@ -70,6 +70,8 @@ export async function measureStartAndStop({
|
|
|
70
70
|
"-nostats",
|
|
71
71
|
"-loglevel",
|
|
72
72
|
"error",
|
|
73
|
+
// The device, where the kind takes one.
|
|
74
|
+
...(typeof encoder?.benchmarkInputArgs === "function" ? encoder.benchmarkInputArgs() : []),
|
|
73
75
|
// A generated picture: the reading is of this host's encoder and muxer,
|
|
74
76
|
// and a file would add its own reading and its own download.
|
|
75
77
|
"-f",
|
|
@@ -81,8 +83,10 @@ export async function measureStartAndStop({
|
|
|
81
83
|
// The encoder this proxy has chosen, asked for its own arguments: the
|
|
82
84
|
// reading must be of the thing that will actually run, since what a start
|
|
83
85
|
// costs is mostly the encoder opening.
|
|
84
|
-
|
|
85
|
-
|
|
86
|
+
// The same arguments the throughput benchmark uses, for the same reason:
|
|
87
|
+
// a start is timed on the encoder itself, not on a scaler in front of it.
|
|
88
|
+
...(typeof encoder?.benchmarkArgs === "function"
|
|
89
|
+
? encoder.benchmarkArgs(null)
|
|
86
90
|
: ["-c:v", "libx264", "-preset", "ultrafast"]),
|
|
87
91
|
"-an",
|
|
88
92
|
"-f",
|
|
@@ -94,6 +94,8 @@ import { LiveOutputs } from "./output/LiveOutputs.js";
|
|
|
94
94
|
import { variantHeightsFor } from "./output/ladder.js";
|
|
95
95
|
import { EncodeOrchestrator } from "./orchestrators/EncodeOrchestrator.js";
|
|
96
96
|
import { wireDiskSpace } from "./disk/wire.js";
|
|
97
|
+
import { IDLE_KEEP_MS } from "./disk/keep.js";
|
|
98
|
+
import { Returns } from "./disk/returns.js";
|
|
97
99
|
import { freeBytesFor } from "./disk/free.js";
|
|
98
100
|
|
|
99
101
|
/**
|
|
@@ -474,7 +476,7 @@ const DEFAULT_SESSION_TTL_MS = 30 * 60 * 1000;
|
|
|
474
476
|
* not this; this only stops something nobody has touched all day from sitting
|
|
475
477
|
* there for the life of the process.
|
|
476
478
|
*/
|
|
477
|
-
const SEGMENT_STORE_IDLE_MS =
|
|
479
|
+
const SEGMENT_STORE_IDLE_MS = IDLE_KEEP_MS;
|
|
478
480
|
const DEFAULT_STARTUP_WAIT_MS = 5_000;
|
|
479
481
|
// Realtime budget — runtime downswitch (software encoder only). Periodically
|
|
480
482
|
// check each active software-transcode session's ffmpeg `speed`; when it stays
|
|
@@ -1605,6 +1607,10 @@ export class HlsSessionManager {
|
|
|
1605
1607
|
void this.cleanupExpired();
|
|
1606
1608
|
}, CLEANUP_INTERVAL_MS);
|
|
1607
1609
|
this.cleanupTimer.unref();
|
|
1610
|
+
// How long after material stops being read somebody asks for it again — the
|
|
1611
|
+
// one term of the keeping period that is guessed rather than measured, and
|
|
1612
|
+
// the only place it can be measured from.
|
|
1613
|
+
this.returns = new Returns();
|
|
1608
1614
|
// One owner of the disk, and the list of what takes it lives with the owner.
|
|
1609
1615
|
this.diskSpace = wireDiskSpace({
|
|
1610
1616
|
segmentStore: this.segmentStore,
|
|
@@ -2257,6 +2263,10 @@ export class HlsSessionManager {
|
|
|
2257
2263
|
// session was never registered, and no sweep looks for one. Proxy
|
|
2258
2264
|
// 2.9.101-2.9.102 failed here on every single request and the leftovers
|
|
2259
2265
|
// were the only trace of it on disk.
|
|
2266
|
+
// A RETURN, if this output was held before — and its age, which is the one
|
|
2267
|
+
// term of the keeping period that nothing measures. Read BEFORE the
|
|
2268
|
+
// directory is claimed, since claiming it is what marks it read.
|
|
2269
|
+
this.returns.note({ lastReadAt: this.segmentStore.lastReadAt(spec.toKey()), now: Date.now() });
|
|
2260
2270
|
this.segmentStore.directoryFor(spec.toKey());
|
|
2261
2271
|
this.segmentStore.useFormat(spec.toKey(), segmentFormat);
|
|
2262
2272
|
|
|
@@ -9678,6 +9688,12 @@ export class HlsSessionManager {
|
|
|
9678
9688
|
// last read, and how much room the disk has for the lot.
|
|
9679
9689
|
// The room is the disk owner's to divide; this asks what the share is now.
|
|
9680
9690
|
await this.diskSpace.revise();
|
|
9691
|
+
// What viewers actually do, beside the period that stands in for it. Said
|
|
9692
|
+
// where it can be read against the disk figures rather than on its own.
|
|
9693
|
+
const returns = this.returns.describe(IDLE_KEEP_MS);
|
|
9694
|
+
if (returns !== null) {
|
|
9695
|
+
logger.info(returns);
|
|
9696
|
+
}
|
|
9681
9697
|
this.segmentStore.enforce({
|
|
9682
9698
|
idleMs: SEGMENT_STORE_IDLE_MS,
|
|
9683
9699
|
maxBytes: this.diskSpace.segmentBytes(),
|
package/services/hwaccel.js
CHANGED
|
@@ -1434,6 +1434,10 @@ function measureEncodeSlope(ffmpegBin, encoder, rung, rawFramesPath) {
|
|
|
1434
1434
|
return new Promise((resolve) => {
|
|
1435
1435
|
const args = [
|
|
1436
1436
|
"-hide_banner", "-loglevel", "error", "-nostats",
|
|
1437
|
+
// What a device-backed encoder needs before the input — the device. Not
|
|
1438
|
+
// its decoding setup: this is fed raw frames and there is nothing to
|
|
1439
|
+
// decode, and `-hwaccel vaapi` over rawvideo fails to open.
|
|
1440
|
+
...(typeof encoder?.benchmarkInputArgs === "function" ? encoder.benchmarkInputArgs() : []),
|
|
1437
1441
|
"-stream_loop", "-1",
|
|
1438
1442
|
"-f", "rawvideo", "-pix_fmt", "yuv420p",
|
|
1439
1443
|
"-s", `${BENCHMARK_REF_W}x${BENCHMARK_REF_H}`, "-r", String(TRANSCODE_FPS),
|