@torrent-tv/proxy 2.81.2 → 2.83.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,3 +1,20 @@
1
+ ## 2.83.0
2
+
3
+ - **Fix**: THE SPILLED PIECES HAD ONE RULE AND IT NEVER BOUND. The ceiling 2.81.0 gave them is a share of free space, which on the addon host is 52 292 MB against a measured growth of 14 400 MB in one viewing — so nothing was ever removed until the torrent itself went. They have the pair the segments have now: a piece behind every read head has been read and will not be read again unless somebody seeks back, and a seek back re-downloads it, which is the bargain this tier already makes whenever it drops a piece for room. Under pressure the order is the same as the segments’ — behind the earliest reader first, furthest behind first of all — instead of by which piece happened to be touched longest ago, which says nothing about what anybody is about to read.
4
+ - **Fix**: THREE PERIODS FOR ONE UNKNOWN, AND THEY CONTRADICTED EACH OTHER. A torrent and its downloaded bytes went at fifteen minutes, a session lived to thirty, the produced segments to six hours — so between the first two there was a session with no source: a viewer returning at the twentieth minute got a session that could not make a single new segment until the torrent was added again. All three stand for the same unmeasured thing, whether the viewer comes back, so they are ONE number in one place (`services/disk/keep.js`, one hour) and the session now dies first, which is the right order and needs no rule of its own.
5
+ - **New**: The thing those periods stand for is being measured, and the proxy is the only place it can be. A session opened on an output whose segments are still on disk IS a return, and its age is exactly what the store recorded. `returns: 14 session(s) opened on material still held, 3 on material gone; median 12min after the last read, longest 51min — kept for 60min` — said beside the disk figures. The median and the longest are the two the period has to sit between, and a week of ordinary use replaces the hour with what viewers actually do.
6
+ - **Fix**: The benchmark of a device-backed encoder could not have run. VAAPI and QSV take their device in the arguments before the input, and what is there for a session sets up hardware DECODING — which a benchmark fed raw frames must not have. `Encoder.benchmarkInputArgs` states the device and nothing else; without it `h264_vaapi` and `h264_qsv` fail to open and the reading would have said "this host cannot encode" about a host that encodes perfectly well. Found by reviewing 2.82.0 rather than by running it.
7
+ - **Chore**: `docs/disk-architecture.md` — the three things that write to one disk, the two rules that remove material, the order the viewers give it, and what is still not solved. 1009 checks pass, biome clean; the new eviction order was checked by breaking it on purpose.
8
+ - **Chore**: NOT yet seen in the field. What the next session must show: a `returns:` line once material has been reopened, `disk:` with both claimants, and the spill falling behind a viewer rather than only growing.
9
+
10
+ ## 2.82.0
11
+
12
+ - **Fix**: THE ENCODER PING-PONGED BECAUSE NOTHING MEASURED WHAT A START COSTS. The plan compares when the wanted pieces appear under each arrangement, and for that it needs how long a fresh encoder takes to produce anything and how long killing one takes. Both were learned only from runs that had ENDED, so at a cold open both were zero — and zero does not read as “not measured”, it reads as “free”. Keeping a warming encoder then cost one piece and moving it cost `0 + 0 + one piece`: the same figure to the millisecond, so the tie fell to position and any advantage however small won. Field 2026-09-08: start at #68, a second later kill and start at #69, half a second later kill and start at #68 again, each dying having produced nothing. Over two days 153 runs were stopped that way and 68 of them made no segment at all.
13
+ **Both are measured at startup now, from one ffmpeg run**: spawn to the first piece the encoder itself announces closed, then SIGTERM to exit. 0.68 s and 0.02 s on the developer’s desktop. Readings from real runs replace them as they arrive. Worth stating because it is what the arithmetic already assumed: subtract keeping from moving and what is left is the killing plus the time the run has already lived — the warm-up a move throws away. That term exists in the formula and vanishes with the measurement, which is why the plan behaved as though a move were free.
14
+ - **Fix**: The decode cost and the contention penalty were asked only where the chosen encoder was SOFTWARE, and neither is about the encoder. A host with a GPU decodes in software just the same — no hardware decoder is asked for anywhere — and what a second job costs is a property of the machine. So a GPU host had no decode model, no contention penalty and no encoder throughput at all, and the quality offer, which is arithmetic over those three, had nothing to compute from.
15
+ - **New**: The throughput benchmark walks the CHOSEN encoder’s own speed ladder — libx264’s presets, NVENC’s `p1`…`p7`, QSV’s `veryfast`…`veryslow`, VAAPI’s quality levels. Each kind says how to measure itself (`Encoder.benchmarkArgs`), which is what the ladders declared and nothing ever ran. A kind with no ladder is measured once, which is still a reading where there was none.
16
+ - **Chore**: 1000 checks pass, biome clean. NOT yet seen in the field. What the next session must show: `a start costs …s to a first piece and a stop …s on this host` once at startup, `firstByte=` and `kill=` non-zero in the very first `encode-plan` line, and no run stopped with “scores worse than standing at” an adjacent number.
17
+
1
18
  ## 2.81.2
2
19
 
3
20
  - **Fix**: `disk: 0MB free` on a host with 103 GB. The free space was read from the segments’ own directory, which is made when the first session starts and removed when the proxy stops — so at every start, and after every clean exit, the reading failed and answered zero, which means “no room” to everything downstream. It reads the nearest ancestor that exists; the disk is the same disk either way.
package/CLAUDE.md CHANGED
@@ -72,6 +72,17 @@ Linux-only host (e.g. POSIX-only signals must degrade elsewhere).
72
72
  wanted (`demand/`) against what the swarm is told (`download/`), why urgency
73
73
  is not a number given to the library, and why the speculative levels are
74
74
  withdrawn rather than lowered.
75
+ - `disk/` — one owner of the disk, read by everything that takes any of it.
76
+ `DiskSpace` reads the free space once and divides it between claimants by
77
+ the rule memory already uses; `wire.js` says who the claimants are (the
78
+ produced segments, and the pieces the memory store spills, which live on the
79
+ torrent thread and are told their share over its channel); `free.js` reads
80
+ the nearest directory that exists, because the segments' own is made at the
81
+ first session and removed at a clean stop.
82
+ - `docs/disk-architecture.md` — the three things that write to one disk, the
83
+ two rules that remove material (nobody needs it, or there is no room), the
84
+ order the viewers give that removal, why a spilled piece is a file of its
85
+ own, and what is still not solved.
75
86
  - `docs/encode-architecture.md` — who decides where encoders go, and the
76
87
  answer is one authority: `EncodePlan`, from what is made, what is being
77
88
  made, what is wanted and what the host can hold. Why no viewer reaches it,
@@ -0,0 +1,161 @@
1
+ # Disk architecture — one owner, and what goes first when it is short
2
+
3
+ Three things on this proxy write to the same disk. Until 2026-09-10 each read
4
+ the free space as though it were the only claimant, so there were three
5
+ ceilings, each standing for the whole disk.
6
+
7
+ | what | where | what bounded it before |
8
+ |---|---|---|
9
+ | segments an encoder produced | `services/encode/SegmentStore.js` | a quarter of what was free, plus a 2 GB floor — both numbers chosen by hand |
10
+ | pieces the memory store spilled | `services/piece-store/piece-disk-store.js` | **nothing at all** |
11
+ | diagnostics kept on purpose | `/data` — core dumps, heap snapshots, packet captures | a count of files, never a size |
12
+
13
+ The middle row is what it cost: one fifty-minute viewing wrote **14 400 MB** to
14
+ the spill file (field 2026-08-31) while the store held 312-424 MB, and free
15
+ space on the host fell by every megabyte of it. The bottom row is 2.9 GB of one
16
+ core dump on the addon host, inside a budget of "two dumps".
17
+
18
+ ## One owner
19
+
20
+ `services/disk/DiskSpace.js` reads the free space once and divides it. The rule
21
+ is the one memory already uses, on the other reading:
22
+
23
+ ```
24
+ allowance = free now + what we already hold - what everything that is not us has been seen to need
25
+ ```
26
+
27
+ Nothing in it is a fraction chosen by hand. The reserve is measured: between two
28
+ readings, how much free space went away beyond what our own consumers took.
29
+
30
+ `services/disk/wire.js` says who the claimants are. The segments are told their
31
+ share by a call; the spilled pieces live on the torrent thread, so the share
32
+ travels the channel that already carries everything else and the reply says what
33
+ they hold — one exchange, both directions.
34
+
35
+ The reading is said every pass, in the series beside the memory one:
36
+
37
+ ```
38
+ disk: 104584MB free; segments 0MB of 52292MB, spilled pieces 0MB of 52292MB
39
+ ```
40
+
41
+ ## Two rules, answering two questions
42
+
43
+ They are kept apart because they were briefly proposed as one, and that was
44
+ wrong.
45
+
46
+ **Time.** Material nobody needs should not sit on the owner's disk merely
47
+ because there is room for it. An output nobody has read for long enough goes
48
+ whole, whatever the free space is.
49
+
50
+ **Space.** Material everyone needs must still go when there is no room.
51
+
52
+ A ceiling alone would keep 5 GB on a household disk for six hours because the
53
+ disk is large. An idle rule alone would keep everything until its clock ran out,
54
+ however tight the disk had become.
55
+
56
+ ## What goes first, and it is the viewers who decide
57
+
58
+ For the segments, the order is the priority map's own read from the other end:
59
+
60
+ 1. outputs nobody is watching at all;
61
+ 2. what lies behind the earliest viewer — furthest behind first;
62
+ 3. what lies ahead of the furthest viewer — furthest ahead first.
63
+
64
+ A segment a viewer is standing on is never a victim. It used to be whole outputs
65
+ by when their directory was last read, which says nothing about what anybody is
66
+ about to watch.
67
+
68
+ ## A piece is a file
69
+
70
+ `DiskTier` wrote every spilled piece into ONE sparse file at
71
+ `index * chunkLength` and answered `forget(index)` by dropping the number from a
72
+ set. The bytes stayed: a sparse file's blocks come back only by hole punching,
73
+ which Node exposes no binding for, so nothing this process could do returned a
74
+ single block before the whole file was removed.
75
+
76
+ `PieceDiskStore` gives each piece its own file. Removing one returns exactly its
77
+ blocks, needs no binding this runtime lacks, and makes the unit of eviction the
78
+ same as the unit of storage — so the order pieces leave in is the order we
79
+ choose. Over its allowance the least recently used goes, and a piece being read
80
+ is never the victim. A piece thrown away is answered as absent, so the torrent
81
+ fetches it again, which is the same bargain the memory tier makes when it
82
+ spills.
83
+
84
+ The read this replaced was chosen for its cost — 22.08 ms via `readFile` against
85
+ **7.63 ms** into a buffer we already hold, measured on the field host — and that
86
+ is unchanged: it is still one `read` into the caller's buffer. What it adds is
87
+ an `open` per read, tens of microseconds against those milliseconds.
88
+
89
+ ## A clean exit leaves nothing of ours
90
+
91
+ The segments' root was removed only when it happened to be empty — from the
92
+ first commit of this repository, never a decision. What it protected against is
93
+ a second proxy sharing the root; what it did was leave every directory alone,
94
+ including this process's own orphans. A directory adopted at startup, owned by
95
+ no session, therefore survived the exit and was adopted again at the next start.
96
+
97
+ That loop is what made an orphan permanent, and it is why 5.0 GB of segments
98
+ from sessions that had ended hours before were on the addon host on 2026-09-10.
99
+ `SegmentStore.dropAll` now removes everything this process owns, root included,
100
+ which also gives the startup sweep its meaning back: whatever is found then is
101
+ from a kill.
102
+
103
+ Both kinds have both rules. For the spilled pieces the time rule is the same
104
+ statement one layer down: a piece behind every read head has been read and will
105
+ not be read again unless somebody seeks back, and a seek back re-downloads it —
106
+ the bargain this tier already makes whenever it drops a piece for room.
107
+ `PieceLru.readHeads` says where the readers stand, `PieceDiskStore.forgetBehind`
108
+ acts on it, and the eviction order under pressure is the same: behind the
109
+ earliest reader first, furthest behind first of all. With no reader at all
110
+ nothing is removed — a store between reads is not a store nobody wants, and the
111
+ torrent going idle is what empties it whole.
112
+
113
+ ## How long material nobody is using is kept
114
+
115
+ One number, in one place — `services/disk/keep.js`, one hour — because
116
+ everything it governs stands for the same unmeasured thing: whether the viewer
117
+ comes back.
118
+
119
+ It was three, and they contradicted each other:
120
+
121
+ | what | was | now |
122
+ |---|---|---|
123
+ | a torrent and its downloaded bytes | 15 minutes | one hour |
124
+ | a session | 30 minutes | unchanged — a session is a record, not material |
125
+ | the segments an encoder produced | 6 hours | one hour |
126
+
127
+ The torrent went at fifteen minutes while the session it feeds lived to thirty,
128
+ so between them there was a session with no source: a viewer returning at the
129
+ twentieth minute got a session that could not make a single new segment. With
130
+ one hour for both kinds of material the session dies first, which is the right
131
+ order and needs no rule of its own to enforce.
132
+
133
+ Nothing derives the hour. Everything else in the decision is measured — a piece
134
+ comes back from the swarm in ~1430 ms, a segment in its own encode time, and the
135
+ disk has an owner that prices holding it — and only the return is unknown.
136
+
137
+ **And the return is measurable here.** A session opened on an output whose
138
+ segments are still on disk IS a return, and its age is exactly what the store
139
+ recorded. `services/disk/returns.js` keeps them and says so beside the disk
140
+ figures:
141
+
142
+ ```
143
+ returns: 14 session(s) opened on material still held, 3 on material gone;
144
+ median 12min after the last read, longest 51min — kept for 60min
145
+ ```
146
+
147
+ The median and the longest are the two the period has to sit between: shorter
148
+ than the median throws away material half the returns wanted, longer than the
149
+ longest keeps material no return has ever reached. A week of ordinary use and
150
+ the hour is replaced by what viewers actually do — and then the two kinds can
151
+ have different numbers, since their costs of coming back differ.
152
+
153
+ ## What is NOT solved
154
+
155
+ **The measurement that would replace the hour has not been taken yet.** The
156
+ counter is in place and says nothing until a week of ordinary use has produced a
157
+ distribution.
158
+
159
+ **The diagnostics are not a claimant yet.** They are bounded by a count of files
160
+ and never by a size, and they cannot simply be thrown away when space is short: a
161
+ dump is the only evidence of the death it records. That needs a rule of its own.
@@ -252,13 +252,42 @@ a fresh encoder = spawn overhead + the piece at the rate in force
252
252
  a moved one = the kill, and then the same
253
253
  ```
254
254
 
255
- With nothing measured the overhead is zero and a fresh encoder owes exactly one
256
- piece. **That is the floor, and it is derived rather than chosen:** a piece
257
- cannot appear before it is encoded, and how fast this host encodes is measured
258
- before any viewer exists. There was an `Infinity` here for the cost of a move,
259
- on the reasoning that an unmeasured price must not license an irreversible act
260
- an exception in a model that needs none, and this is the same statement made by
261
- arithmetic.
255
+ **Both figures are measured before any viewer exists**, and until 2026-09-10
256
+ neither was. They were learned only from runs that had ENDED, so at a cold open
257
+ both read zero and zero does not mean "unknown" here, it means "free".
258
+
259
+ That is not a small bias, it is the whole comparison. Subtract keeping from
260
+ moving and what is left is exactly the killing plus the time the run has already
261
+ lived:
262
+
263
+ ```
264
+ keep a warming run = firstByteWait - elapsed
265
+ move it = kill + spawn overhead + one piece
266
+ = kill + firstByteWait (spawn overhead = firstByteWait - one piece)
267
+ move - keep = kill + elapsed
268
+ ```
269
+
270
+ `elapsed` is the warm-up a move throws away — a process started, an input
271
+ opened, the first bytes fetched, a decoder filled — and it is the term that
272
+ makes moving cost something. Set `firstByteWait` to zero and it cancels out of
273
+ both sides along with the kill: keeping and moving then cost the same figure to
274
+ the millisecond, the tie falls to position, and any advantage however small wins.
275
+
276
+ Field 2026-09-08, the first fifteen seconds of a session: start at #68, a second
277
+ later kill and start at #69, half a second later kill and start at #68 again,
278
+ each dying having produced nothing. Over two days 153 runs were stopped by the
279
+ plan and 68 of them made no segment at all, median life 9.6 s.
280
+
281
+ `services/encode/start-stop-cost.js` measures both from **one ffmpeg run at
282
+ startup**: spawn to the first piece the encoder itself announces closed, then
283
+ SIGTERM to exit. 0.68 s and 0.02 s on the developer's desktop. `RunCosts` starts
284
+ from those and replaces them with readings from real runs as they arrive.
285
+
286
+ There was an `Infinity` here once for the cost of a move, on the reasoning that
287
+ an unmeasured price must not license an irreversible act. It was removed as an
288
+ exception in a model that needs none — correctly, but what replaced it was a
289
+ floor of zero, which licenses the act rather than forbidding it. A measurement
290
+ is what a model like this needs, not an exception and not a floor.
262
291
 
263
292
  And a run killed before producing anything is a measurement too — a lower bound
264
293
  on the first output, and the only reading a thrash can supply, since every run in
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@torrent-tv/proxy",
3
- "version": "2.81.2",
3
+ "version": "2.83.0",
4
4
  "description": "Torrent proxy client that exposes webseed-like HTTP stream endpoint.",
5
5
  "license": "GPL-3.0-or-later",
6
6
  "publishConfig": {
package/server.js CHANGED
@@ -40,6 +40,7 @@ import { WorkerTorrentPool } from "./services/torrent-worker/pool-adapter.js";
40
40
  import { HlsSessionManager } from "./services/hls-session-manager.js";
41
41
  import { createPlaybackPlanner } from "./services/playback-planner.js";
42
42
  import { detectVideoEncoder, benchmarkSoftwarePresets, benchmarkDecodeCost, benchmarkContention, benchmarkCopySpeed, detectTonemapSupport } from "./services/hwaccel.js";
43
+ import { measureStartAndStop } from "./services/encode/start-stop-cost.js";
43
44
  import { logger } from "./utils/logger.js";
44
45
 
45
46
  const __filename = fileURLToPath(import.meta.url);
@@ -126,23 +127,31 @@ export async function startProxyServer({
126
127
  const videoEncoder = transcodeAudio
127
128
  ? await detectVideoEncoder({ ffmpegBin, logger })
128
129
  : null;
129
- // For software libx264, benchmark preset throughput once at startup so the
130
- // session manager can pick the highest-quality preset that still encodes each
131
- // stream faster than realtime. Hardware encoders use their own fixed preset.
132
- // The decode model first, and the preset benchmark secondthey are
133
- // independent now (the presets are timed on raw frames), but the order costs
134
- // nothing and keeps the two figures side by side in the log.
135
- const decodeCostModel = videoEncoder?.kind === "software"
130
+ // WHAT THIS HOST DOES, measured before any viewer exists. Every one of these
131
+ // was gated on the chosen encoder being SOFTWARE, and two of the three are
132
+ // not about the encoder at all: a host with a GPU decodes in software just
133
+ // the same no hardware decoder is asked for anywhere and what a second
134
+ // job costs is a property of the machine. So a GPU host had no decode cost,
135
+ // no contention penalty and no encoder throughput, and the quality offer,
136
+ // which is arithmetic over those three, had nothing to compute from.
137
+ //
138
+ // The decode model first and the throughput second — they are independent
139
+ // (the rungs are timed on raw frames), but the order keeps the two figures
140
+ // side by side in the log.
141
+ const decodeCostModel = transcodeAudio
136
142
  ? await benchmarkDecodeCost({ ffmpegBin, logger })
137
143
  : null;
138
144
  // What a second job costs on this host. Measured because the budget adds
139
145
  // independent prices and this host says two jobs that each fit alone do not
140
146
  // fit together — 2.6× on the addon box (2026-08-18).
141
- const contentionPenalties = videoEncoder?.kind === "software"
147
+ const contentionPenalties = transcodeAudio
142
148
  ? await benchmarkContention({ ffmpegBin, logger })
143
149
  : null;
144
- const softwarePresetBenchmark = videoEncoder?.kind === "software"
145
- ? await benchmarkSoftwarePresets({ ffmpegBin, logger })
150
+ // The chosen encoder walked over its OWN speed ladder: libx264's presets,
151
+ // NVENC's p1…p7, QSV's veryfast…veryslow, VAAPI's quality levels. A kind with
152
+ // no ladder is measured once, which is still a reading where there was none.
153
+ const softwarePresetBenchmark = videoEncoder
154
+ ? await benchmarkSoftwarePresets({ ffmpegBin, logger, encoder: videoEncoder })
146
155
  : null;
147
156
  // What this host does with a picture it does NOT re-encode. Every other
148
157
  // startup measurement prices encoding or decoding, and a copied picture does
@@ -155,6 +164,15 @@ export async function startProxyServer({
155
164
  const copySpeedX = transcodeAudio
156
165
  ? await benchmarkCopySpeed({ ffmpegBin, logger })
157
166
  : null;
167
+ // WHAT A START AND A STOP COST HERE, before any viewer exists. Both decide one
168
+ // thing — leave an encoder where it stands, or kill it and start another —
169
+ // and both used to be learned only from runs that had ENDED, so at a cold
170
+ // open they were zero. Zero does not read as "not measured": it reads as
171
+ // "free", and a free move is always taken. Field 2026-09-08: an encoder moved
172
+ // between two adjacent numbers every half second and produced nothing.
173
+ const startStopCost = transcodeAudio
174
+ ? await measureStartAndStop({ ffmpegBin, encoder: videoEncoder, logger })
175
+ : null;
158
176
  // Whether this ffmpeg build can tone-map HDR→SDR (zscale + tonemap filters).
159
177
  // Detected once; the session manager applies the tonemap chain only for HDR
160
178
  // sources on the software path when available.
@@ -171,6 +189,7 @@ export async function startProxyServer({
171
189
  decodeCostModel,
172
190
  contentionPenalties,
173
191
  copySpeedX,
192
+ startStopCost,
174
193
  tonemapSupported,
175
194
  segmentFormatId: segmentFormat,
176
195
  stateDir,
@@ -178,15 +197,15 @@ export async function startProxyServer({
178
197
  // CPU-bound transcode from a download-starved input before downscaling.
179
198
  // What every torrent here has moved, so the proxy can price its own
180
199
  // downloading, hashing and delivery against the machine (roadmap item 7).
181
- // What the spilled pieces weigh on the torrent thread, and how to tell them
182
- // their share of the disk. The owner of the disk is on this side, where the
183
- // segments are; the pieces are on the other.
184
- spillDisk: typeof torrentPool.allowSpillBytes === "function"
185
- ? {
186
- held: () => torrentPool.spilledBytes ?? 0,
187
- allow: (bytes) => torrentPool.allowSpillBytes(bytes)
188
- }
189
- : null,
200
+ // What the spilled pieces weigh on the torrent thread, and how to tell them
201
+ // their share of the disk. The owner of the disk is on this side, where the
202
+ // segments are; the pieces are on the other.
203
+ spillDisk: typeof torrentPool.allowSpillBytes === "function"
204
+ ? {
205
+ held: () => torrentPool.spilledBytes ?? 0,
206
+ allow: (bytes) => torrentPool.allowSpillBytes(bytes)
207
+ }
208
+ : null,
190
209
  getTorrentTotals: async () => {
191
210
  if (typeof torrentPool.getTorrentTotals !== "function") {
192
211
  return null;
@@ -0,0 +1,48 @@
1
+ /**
2
+ * @file How long material nobody is using is kept.
3
+ *
4
+ * ONE NUMBER, IN ONE PLACE, because everything it governs stands for the same
5
+ * unmeasured thing: whether the viewer comes back. Everything else in the
6
+ * decision is measured or measurable — re-downloading a piece from the swarm is
7
+ * ~1430 ms on the field host, re-making a segment is its own encode time, and
8
+ * the disk now has an owner that prices holding it. Only the return is unknown,
9
+ * and a period is what stands in for it.
10
+ *
11
+ * IT WAS THREE NUMBERS AND THEY CONTRADICTED EACH OTHER (read 2026-09-10):
12
+ *
13
+ * torrent and its downloaded bytes 15 minutes
14
+ * session 30 minutes
15
+ * produced segments 6 hours
16
+ *
17
+ * The torrent therefore went at fifteen minutes while the session it feeds
18
+ * lived to thirty, so between them there was a session with no source: a viewer
19
+ * returning at the twentieth minute got a session that could not make a single
20
+ * new segment until the torrent was added again. That is not two answers to one
21
+ * question, it is a contradiction — and it is what two independent guesses about
22
+ * one unknown produce.
23
+ *
24
+ * With one hour for both kinds of material the session dies first, which is the
25
+ * right order and needs no rule of its own to enforce.
26
+ *
27
+ * WHY AN HOUR, honestly: nothing derives it. It is a stand-in, chosen to be
28
+ * long enough that an interruption — a phone call, a meal — does not cost the
29
+ * film, and short enough that a household disk is not held for a day by
30
+ * somebody who is not coming back. It is the OWNER'S disk, and holding
31
+ * gigabytes on it because there happens to be room is taking something that is
32
+ * not ours.
33
+ *
34
+ * WHAT REPLACES IT, and the proxy is already in a position to measure it: every
35
+ * session opened on an output whose segments are still on disk IS a return, and
36
+ * its age is known. `services/disk/returns.js` records them. A week of those and
37
+ * the distribution answers this directly — keep material for as long as returns
38
+ * actually happen — and then the two kinds can have different numbers, since
39
+ * their costs of coming back differ.
40
+ */
41
+
42
+ /**
43
+ * How long material nobody has read is kept, in milliseconds.
44
+ *
45
+ * Read by the torrent pool for a torrent and its downloaded bytes, and by the
46
+ * session manager for the segments an encoder produced.
47
+ */
48
+ export const IDLE_KEEP_MS = 60 * 60 * 1000;
@@ -0,0 +1,103 @@
1
+ /**
2
+ * @file How long after material stops being read somebody asks for it again.
3
+ *
4
+ * The one term of "how long do we keep this" that nothing measures. Everything
5
+ * else in that decision is a measured quantity — re-downloading a piece from
6
+ * the swarm is ~1430 ms on the field host, re-making a segment is its own
7
+ * encode time, and the disk has an owner that prices holding it. Only the
8
+ * return is unknown, and `IDLE_KEEP_MS` is a guess standing in for it.
9
+ *
10
+ * IT IS MEASURABLE HERE AND NOWHERE ELSE. A session opened on an output whose
11
+ * segments are still on disk IS a return, and its age is known exactly: the
12
+ * store records when each output was last read. Nothing needs to be inferred.
13
+ *
14
+ * This changes no behaviour. It records, and says what it has seen once in a
15
+ * while, so that after a week of ordinary use the period can be derived from
16
+ * what viewers actually do instead of from what a period felt like.
17
+ */
18
+
19
+ /** How many returns are kept. Enough to see a shape, few enough to say in a line. */
20
+ const KEPT = 200;
21
+
22
+ export class Returns {
23
+ /** Ages in milliseconds, newest last. @type {number[]} */
24
+ #ages = [];
25
+
26
+ /** Sessions opened on material this proxy no longer had. */
27
+ #cold = 0;
28
+
29
+ /** Sessions opened on material that was still there. */
30
+ #warm = 0;
31
+
32
+ /**
33
+ * Note a session being opened on an output.
34
+ *
35
+ * @param {object} params
36
+ * @param {number | null} params.lastReadAt - When that output was last read,
37
+ * or null where this proxy has never held it.
38
+ * @param {number} params.now
39
+ * @returns {void}
40
+ */
41
+ note({ lastReadAt, now }) {
42
+ if (!Number.isFinite(lastReadAt) || lastReadAt === null || lastReadAt <= 0) {
43
+ this.#cold += 1;
44
+ return;
45
+ }
46
+ this.#warm += 1;
47
+ this.#ages.push(Math.max(0, now - /** @type {number} */ (lastReadAt)));
48
+ while (this.#ages.length > KEPT) {
49
+ this.#ages.shift();
50
+ }
51
+ }
52
+
53
+ /**
54
+ * What the returns look like, or null while there have been none.
55
+ *
56
+ * The MEDIAN and the LONGEST, because those are the two the period has to sit
57
+ * between: shorter than the median throws away material half the returns
58
+ * wanted, and longer than the longest keeps material no return has ever
59
+ * reached.
60
+ *
61
+ * @returns {{ warm: number, cold: number, medianMs: number, longestMs: number } | null}
62
+ */
63
+ shape() {
64
+ if (this.#ages.length === 0) {
65
+ return null;
66
+ }
67
+ const sorted = [...this.#ages].sort((left, right) => left - right);
68
+ const middle = Math.floor(sorted.length / 2);
69
+ return {
70
+ warm: this.#warm,
71
+ cold: this.#cold,
72
+ medianMs: sorted.length % 2 === 0 ? (sorted[middle - 1] + sorted[middle]) / 2 : sorted[middle],
73
+ longestMs: sorted[sorted.length - 1]
74
+ };
75
+ }
76
+
77
+ /**
78
+ * One line, for the log, or null while there is nothing to say.
79
+ *
80
+ * @param {number} keepMs - What is being kept for now, so the reading and the
81
+ * guess it will replace stand side by side.
82
+ * @returns {string | null}
83
+ */
84
+ describe(keepMs) {
85
+ const shape = this.shape();
86
+ if (shape === null) {
87
+ return null;
88
+ }
89
+ return (
90
+ `returns: ${shape.warm} session(s) opened on material still held, ` +
91
+ `${shape.cold} on material gone; median ${minutes(shape.medianMs)} after the last read, ` +
92
+ `longest ${minutes(shape.longestMs)} — kept for ${minutes(keepMs)}`
93
+ );
94
+ }
95
+ }
96
+
97
+ /**
98
+ * @param {number} ms
99
+ * @returns {string}
100
+ */
101
+ function minutes(ms) {
102
+ return `${Math.round(ms / 60000)}min`;
103
+ }
@@ -81,4 +81,46 @@ export class Encoder {
81
81
  buildVideoArgs(_options) {
82
82
  throw new Error(`${this.name} does not say how to build its video arguments.`);
83
83
  }
84
+
85
+ /**
86
+ * What this kind needs BEFORE the input when it is being benchmarked.
87
+ *
88
+ * Not `inputArgs`: those set up hardware DECODING, and a benchmark is fed raw
89
+ * frames — there is nothing to decode. What a device-backed encoder still
90
+ * needs is the device itself, and without it `h264_vaapi` and `h264_qsv` fail
91
+ * to open at all, so the benchmark would answer "this host cannot encode"
92
+ * about a host that encodes perfectly well.
93
+ *
94
+ * @returns {string[]}
95
+ */
96
+ benchmarkInputArgs() {
97
+ return [];
98
+ }
99
+
100
+ /**
101
+ * How to encode raw frames at one rung of this kind's speed ladder, for the
102
+ * startup benchmark and nothing else.
103
+ *
104
+ * SEPARATE FROM `buildVideoArgs` on purpose. That one produces a picture from
105
+ * a film: it scales, it tone-maps, it forces keyframes onto a grid, it caps a
106
+ * bitrate. The benchmark is fed raw frames of a known size and wants the
107
+ * encoder and the rung with nothing else in the way — otherwise the reading
108
+ * prices a scaler as though it were the encoder.
109
+ *
110
+ * WHY EVERY KIND MUST ANSWER IT. Until now the throughput benchmark existed
111
+ * only for libx264 and was gated on the chosen encoder being software, so a
112
+ * host with NVENC, QSV, VAAPI or V4L2M2M measured its encoder not at all —
113
+ * and the quality offer, which is arithmetic over pixels per second, had no
114
+ * pixels per second to work with. NVENC's own ladder is `p1`…`p7` and QSV's
115
+ * is `veryfast`…`veryslow`; both are declared here and neither was ever run.
116
+ *
117
+ * @param {string | null} rung - One value of {@link speedLadder}, or null
118
+ * where the kind has no ladder and there is one thing to measure.
119
+ * @returns {string[]}
120
+ */
121
+ benchmarkArgs(rung = null) {
122
+ const ladder = this.speedLadder;
123
+ const setting = ladder?.flag && rung ? [ladder.flag, rung] : [];
124
+ return ["-c:v", this.name, ...setting];
125
+ }
84
126
  }
@@ -35,6 +35,17 @@ export class QsvEncoder extends Encoder {
35
35
  };
36
36
  }
37
37
 
38
+ /** @returns {string[]} */
39
+ benchmarkInputArgs() {
40
+ return this.device ? ["-qsv_device", this.device] : [];
41
+ }
42
+
43
+ /** @param {string | null} rung @returns {string[]} */
44
+ benchmarkArgs(rung = null) {
45
+ const preset = rung ? ["-preset", rung] : [];
46
+ return ["-c:v", "h264_qsv", "-global_quality", "24", ...preset];
47
+ }
48
+
38
49
  buildVideoArgs({ targetWidth, targetHeight, segmentDurationSec, forcedKeyframeTimes }) {
39
50
  const { w, h } = safeDimensions(targetWidth, targetHeight);
40
51
  return [
@@ -481,6 +481,20 @@ export class SegmentStore {
481
481
  this.#logger.info(`segment-store dropped ${directoryNameFor(key)} (${because})`);
482
482
  }
483
483
 
484
+ /**
485
+ * When this output was last asked for, or null where it has never been.
486
+ *
487
+ * The one reading that makes the keeping period measurable rather than
488
+ * guessed: a session opened on an output this answers for IS a return, and
489
+ * this is its age.
490
+ *
491
+ * @param {string} key
492
+ * @returns {number | null}
493
+ */
494
+ lastReadAt(key) {
495
+ return this.#touched.get(key) ?? null;
496
+ }
497
+
484
498
  /**
485
499
  * Throw away everything this store owns, and the root with it.
486
500
  *
@@ -42,6 +42,11 @@ export class SoftwareEncoder extends Encoder {
42
42
  };
43
43
  }
44
44
 
45
+ /** @param {string | null} rung @returns {string[]} */
46
+ benchmarkArgs(rung = null) {
47
+ return ["-c:v", "libx264", "-preset", rung ?? SOFTWARE_PRESET, "-crf", SOFTWARE_CRF, "-pix_fmt", "yuv420p"];
48
+ }
49
+
45
50
  buildVideoArgs({
46
51
  targetWidth,
47
52
  targetHeight,
@@ -40,6 +40,19 @@ export class VaapiEncoder extends Encoder {
40
40
 
41
41
  // No fps filter: VAAPI inherits the source rate and keeps keyframes on the
42
42
  // grid via time-based -force_key_frames, so it already honours source fps.
43
+ /** @returns {string[]} */
44
+ benchmarkInputArgs() {
45
+ return this.device ? ["-vaapi_device", this.device] : [];
46
+ }
47
+
48
+ /** @param {string | null} rung @returns {string[]} */
49
+ benchmarkArgs(rung = null) {
50
+ // Raw frames live in this process; VAAPI encodes what is in the device, so
51
+ // the upload is part of what this kind costs and belongs in the reading.
52
+ const quality = rung ? ["-quality", rung] : [];
53
+ return ["-vf", "format=nv12,hwupload", "-c:v", "h264_vaapi", "-qp", "24", ...quality];
54
+ }
55
+
43
56
  buildVideoArgs({ targetWidth, targetHeight, segmentDurationSec, forcedKeyframeTimes }) {
44
57
  const { w, h } = safeDimensions(targetWidth, targetHeight);
45
58
  return [