@torrent-tv/proxy 2.31.0 → 2.33.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +10 -0
- package/assets/calibration/NOTICE.md +49 -16
- package/assets/calibration/cal-h264-1080-hi.mp4 +0 -0
- package/assets/calibration/cal-h264-1080-lo.mp4 +0 -0
- package/assets/calibration/cal-h264-480-hi.mp4 +0 -0
- package/assets/calibration/cal-h264-480-lo.mp4 +0 -0
- package/assets/calibration/cal-h264-720-hi.mp4 +0 -0
- package/assets/calibration/cal-h264-720-lo.mp4 +0 -0
- package/package.json +1 -1
- package/services/available-share.js +89 -0
- package/services/decode-cost-fit.js +331 -0
- package/services/hls-session-manager.js +72 -4
- package/services/hwaccel.js +35 -100
- package/test/available-share.test.js +76 -0
- package/test/decode-cost-fit.test.js +145 -0
- package/test/decode-cost.test.js +347 -347
- package/assets/calibration/cal-1080-hi.mp4 +0 -0
- package/assets/calibration/cal-1080-lo.mp4 +0 -0
- package/assets/calibration/cal-720.mp4 +0 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,13 @@
|
|
|
1
|
+
## 2.33.0
|
|
2
|
+
|
|
3
|
+
- **New**: A quality step is judged on the machine it will actually run on. The encoder benchmark measures a QUIET host — one ffmpeg and nothing else — while the addon host was measured 99 % busy, and a step predicted at 1.83x ran at 1.01-1.12x (2026-08-17). The offer now multiplies each prediction by the share of the machine that is free, taken from the same `host-load` reading that is already printed every five seconds. What is subtracted is ONLY the work nobody has been charged for — the kernel, the container, whatever else the owner runs — because our own encoders are already priced by the concurrency arithmetic and the proxy's own work per megabyte moved. Charging those here as well is what shipped in 2.21.0 and emptied the quality menu down to a single copied height. On the field reading the correction is about 0.77, and the "not offering" line now says what the machine had to spare when it decided.
|
|
4
|
+
- **New**: Each step reports what its prediction was worth. When a step runs with the machine to itself, the log states the speed it was predicted at, the speed it measured, and the ratio — so the error that REMAINS after the availability correction is a number in the field rather than an argument. It is written when it moves by more than a tenth, so a steady step says it once. On the field case that correction takes 1.83x to 1.41x against 1.01-1.12x measured: part of the gap, not all of it, and this line is how the rest gets found.
|
|
5
|
+
|
|
6
|
+
## 2.32.0
|
|
7
|
+
|
|
8
|
+
- **New**: The decode cost is fitted from a clip set that can be checked, and a term the measurements do not determine is refused instead of published as a zero. The set that shipped until now was three clips for three unknowns — an EXACT system, with two of the clips at the same pixel rate — and such a system cannot fail visibly: it returns whatever satisfies its equations. On 2026-08-17 it returned `0.007542 × Mpx/s + 0.000000 × Mbit/s + 0.0000 s/s`, so a film's own bitrate never entered its price, and the prediction built on it was 1.8-2.2x optimistic against the same file measured while playing. The new set is six clips — three sizes × two bitrates, the axes varied INDEPENDENTLY — cut from the same Netflix Open Content "Meridian" footage (CC BY 4.0, `assets/calibration/NOTICE.md`), 7.7 MB against 8.8 MB before. Three spare measurements give the fit a residual, and with it two questions it could not ask before: whether a term's whole effect across the measured range exceeds the scatter, and whether the coefficient exceeds its own standard error. A term that fails either is dropped, the rest are fitted again, and the log names it — a zero now means "not measured" only when it says so. A NEGATIVE coefficient is dropped too rather than clamped to zero: more pixels cannot cost less work, so a negative fit is noise beating an effect, not a discovery about the host. Measured on the developer's machine, the new set determines all three terms (`0.000520 × Mpx/s + 0.002086 × Mbit/s + 0.0033 s/s`, typical disagreement 0.0012 s/s), and the bitrate term it recovers matches the difference between the two 1080p clips to 15 %. The arithmetic is a pure module with the degenerate case as a test (`services/decode-cost-fit.js`).
|
|
9
|
+
- **Chore**: What the calibration costs at startup, measured rather than assumed: six clips take 14.8 s on the developer's Windows box, of which the decoding is about 50 ms per clip — an empty ffmpeg spawn there costs 774 ms, and opening each file most of the rest. Shortening the clips would therefore save nothing; the cost is spawning ffmpeg once per clip, and it is paid before any viewer exists.
|
|
10
|
+
|
|
1
11
|
## 2.31.0
|
|
2
12
|
|
|
3
13
|
- **Chore**: The encoder run's two status strings are gone; both are now outputs of the state table shipped in 2.23.0. `session.progress.state` was maintained by hand at seven sites and `session.state` at nine, and neither could answer on its own — the warm-up test had to read both under an `||`, because one said "starting" from the first spawn until something overwrote it while the other said it again on its own schedule. What the browser is told is computed where it is sent (`wireState(runState)`), and `session.state` is reduced to the session's own lifetime: it exists, or it has been disposed. That deletes the line in the spawn path that read `state === "disposed" ? "disposed" : "starting"` — two lifetimes in one variable, which is what it was there to paper over. Verified before the change that nothing in the browser reads the wire string, so the value set is unchanged and unobserved either way; the four values it can take are the same four as before.
|
|
@@ -1,30 +1,63 @@
|
|
|
1
1
|
# Calibration clips — attribution
|
|
2
2
|
|
|
3
|
-
The
|
|
3
|
+
The clips in this folder are excerpts from **"Meridian"**, part of
|
|
4
4
|
[Netflix Open Content](https://opencontent.netflix.com/), licensed under the
|
|
5
5
|
**Creative Commons Attribution 4.0 International (CC BY 4.0)** licence:
|
|
6
6
|
<https://creativecommons.org/licenses/by/4.0/>.
|
|
7
7
|
|
|
8
8
|
They were cut from `Meridian/Meridian_UHD4k5994_HDR_P3PQ.mp4` and re-encoded to
|
|
9
|
-
H.264 High, 24 fps,
|
|
9
|
+
H.264 High, 24 fps, 2 s each, no audio:
|
|
10
10
|
|
|
11
11
|
| file | resolution | bitrate |
|
|
12
12
|
|---|---|---|
|
|
13
|
-
| `cal-1080-hi.mp4` | 1920×1080 |
|
|
14
|
-
| `cal-1080-lo.mp4` | 1920×1080 |
|
|
15
|
-
| `cal-720.mp4` | 1280×720 |
|
|
13
|
+
| `cal-h264-1080-hi.mp4` | 1920×1080 | 9.36 Mbit/s |
|
|
14
|
+
| `cal-h264-1080-lo.mp4` | 1920×1080 | 1.00 Mbit/s |
|
|
15
|
+
| `cal-h264-720-hi.mp4` | 1280×720 | 9.95 Mbit/s |
|
|
16
|
+
| `cal-h264-720-lo.mp4` | 1280×720 | 1.04 Mbit/s |
|
|
17
|
+
| `cal-h264-480-hi.mp4` | 854×480 | 9.48 Mbit/s |
|
|
18
|
+
| `cal-h264-480-lo.mp4` | 854×480 | 1.25 Mbit/s |
|
|
16
19
|
|
|
17
|
-
## Why
|
|
20
|
+
## Why six, and why this grid
|
|
18
21
|
|
|
19
|
-
`services/hwaccel.js` decodes them at startup and
|
|
20
|
-
`a × Mpixel/s + b × Mbit/s + c`, from the
|
|
21
|
-
|
|
22
|
-
pixel count. Three points, three unknowns.
|
|
22
|
+
`services/hwaccel.js` decodes them at startup and fits the host's decode cost,
|
|
23
|
+
`a × Mpixel/s + b × Mbit/s + c`, from the measurements
|
|
24
|
+
(`services/decode-cost-fit.js`).
|
|
23
25
|
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
26
|
+
The set before this one was three clips for three unknowns — an EXACT system,
|
|
27
|
+
with two of the clips sharing a pixel count. Such a system cannot fail visibly:
|
|
28
|
+
it returns whatever satisfies its three equations, and on 2026-08-17 it returned
|
|
29
|
+
`0.007542 × Mpx/s + 0.000000 × Mbit/s + 0.0000 s/s` — the bitrate term and the
|
|
30
|
+
constant exactly zero, so a film's own bitrate never entered its price, and the
|
|
31
|
+
prediction built on it was 1.8-2.2x optimistic against the same file measured
|
|
32
|
+
while playing.
|
|
27
33
|
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
34
|
+
Six clips are three sizes × two bitrates, with the two axes varied
|
|
35
|
+
**independently**. That gives three spare measurements, so the fit has a
|
|
36
|
+
residual — something to notice a degeneracy WITH — and lets a term be refused
|
|
37
|
+
when the data does not determine it, instead of being published as a zero that
|
|
38
|
+
looks measured.
|
|
39
|
+
|
|
40
|
+
A 480p clip at 9.5 Mbit/s is not content anyone ships; that is the point. The
|
|
41
|
+
bitrate has to vary at every size, or it cannot be told apart from the size.
|
|
42
|
+
|
|
43
|
+
## Why real footage
|
|
44
|
+
|
|
45
|
+
Grainy live action rather than a generated pattern: measured 2026-08-14 on the
|
|
46
|
+
addon host, this material decodes 11 % away from the film being served, where a
|
|
47
|
+
generated `testsrc2` clip is 158 % away. It also matches most of what the
|
|
48
|
+
product actually plays — a survey of 133 releases (2026-07-10) found ~67 %
|
|
49
|
+
H.264 live action, the rest HEVC, XviD and animation.
|
|
50
|
+
|
|
51
|
+
## Replacing or extending the set
|
|
52
|
+
|
|
53
|
+
Allowed: the benchmark reads each clip's dimensions, frame rate and bitrate from
|
|
54
|
+
ffmpeg's own output rather than from this table. Two rules hold, though:
|
|
55
|
+
|
|
56
|
+
- **more measurements than terms**, or there is no residual and no way to see a
|
|
57
|
+
degenerate fit;
|
|
58
|
+
- **the axes stay independent** — every size at both bitrates. Dropping one cell
|
|
59
|
+
reintroduces the collinearity that produced the zeros above.
|
|
60
|
+
|
|
61
|
+
Codecs other than H.264 (HEVC, AV1, 10-bit) decode dearer per pixel, and the
|
|
62
|
+
model fitted here does not describe them. Covering them means a set per codec
|
|
63
|
+
family and constants chosen by the source's own codec — roadmap item 3(b).
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
package/package.json
CHANGED
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* @file How much of the machine a new encoder can actually have.
|
|
3
|
+
*
|
|
4
|
+
* The encoder benchmark measures a QUIET host: one ffmpeg, nothing else. A real
|
|
5
|
+
* encode runs on a machine that is also downloading, hashing and serving, and
|
|
6
|
+
* on the addon host that machine was measured 99 % busy — `ffmpeg=52-60%
|
|
7
|
+
* proxy=17-24% system=99%` — while a step predicted at 1.83x ran at 1.01-1.12x
|
|
8
|
+
* (2026-08-17).
|
|
9
|
+
*
|
|
10
|
+
* What this corrects is ONLY the part nobody has been charged for. That
|
|
11
|
+
* distinction is the whole of the file, because the alternative has already
|
|
12
|
+
* shipped once and broke the product: in 2.21.0 the budget ADDED what else was
|
|
13
|
+
* running while the per-file costs were being LEARNED from runs that already
|
|
14
|
+
* contained that other work, so every cost was counted twice, every re-encoded
|
|
15
|
+
* step was refused, and the quality menu emptied itself down to the one copied
|
|
16
|
+
* height (fixed in 2.21.1).
|
|
17
|
+
*
|
|
18
|
+
* So: our own encoders are priced by the concurrency arithmetic, and the
|
|
19
|
+
* proxy's own work — the torrent, the hashing, the delivery — is priced per
|
|
20
|
+
* megabyte moved. Both are already in the budget. What is NOT in it is
|
|
21
|
+
* everything else the machine does: the kernel, the container runtime, whatever
|
|
22
|
+
* else the owner runs on their box. That is what is subtracted here, and
|
|
23
|
+
* nothing more.
|
|
24
|
+
*/
|
|
25
|
+
|
|
26
|
+
/**
|
|
27
|
+
* The share of the machine available to a new encoder.
|
|
28
|
+
*
|
|
29
|
+
* @param {object} reading - Fractions of the WHOLE machine (all cores), as
|
|
30
|
+
* `shareOfMachine` reports them.
|
|
31
|
+
* @param {number | null} reading.systemBusy - Everything the machine is doing.
|
|
32
|
+
* @param {number | null} reading.encoderShare - Our own ffmpeg processes.
|
|
33
|
+
* @param {number | null} reading.proxyShare - The proxy process itself.
|
|
34
|
+
* @returns {{ share: number, unattributed: number, known: boolean }}
|
|
35
|
+
* `known` is false when the host does not report its own load — then the
|
|
36
|
+
* share is 1 and the caller must say it is uncorrected rather than pretend.
|
|
37
|
+
*/
|
|
38
|
+
export function availableShareFrom(reading = {}) {
|
|
39
|
+
const systemBusy = finite(reading.systemBusy);
|
|
40
|
+
if (systemBusy === null) {
|
|
41
|
+
// No reading at all: not every host has /proc. An uncorrected prediction is
|
|
42
|
+
// the honest answer, and the caller says so.
|
|
43
|
+
return { share: 1, unattributed: 0, known: false };
|
|
44
|
+
}
|
|
45
|
+
const ours = (finite(reading.encoderShare) ?? 0) + (finite(reading.proxyShare) ?? 0);
|
|
46
|
+
// Rounding, and the two readings being taken microseconds apart, can put our
|
|
47
|
+
// own share fractionally above the system total. Below zero is not a
|
|
48
|
+
// measurement of anything.
|
|
49
|
+
const unattributed = clamp(systemBusy - ours, 0, 1);
|
|
50
|
+
return { share: clamp(1 - unattributed, 0, 1), unattributed, known: true };
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
/**
|
|
54
|
+
* Apply the correction to a predicted speed.
|
|
55
|
+
*
|
|
56
|
+
* A speed is work per unit time, so a machine that can give a new encoder only
|
|
57
|
+
* `share` of itself produces `share ×` the speed the benchmark measured alone.
|
|
58
|
+
*
|
|
59
|
+
* @param {number} predictedSpeed - From the quiet-host benchmark.
|
|
60
|
+
* @param {{ share: number, known: boolean }} availability
|
|
61
|
+
* @returns {number}
|
|
62
|
+
*/
|
|
63
|
+
export function correctForAvailability(predictedSpeed, availability) {
|
|
64
|
+
if (!Number.isFinite(predictedSpeed) || predictedSpeed <= 0) {
|
|
65
|
+
return predictedSpeed;
|
|
66
|
+
}
|
|
67
|
+
if (!availability?.known) {
|
|
68
|
+
return predictedSpeed;
|
|
69
|
+
}
|
|
70
|
+
return predictedSpeed * availability.share;
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
/**
|
|
74
|
+
* @param {unknown} value
|
|
75
|
+
* @returns {number | null}
|
|
76
|
+
*/
|
|
77
|
+
function finite(value) {
|
|
78
|
+
return Number.isFinite(value) ? Number(value) : null;
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
/**
|
|
82
|
+
* @param {number} value
|
|
83
|
+
* @param {number} low
|
|
84
|
+
* @param {number} high
|
|
85
|
+
* @returns {number}
|
|
86
|
+
*/
|
|
87
|
+
function clamp(value, low, high) {
|
|
88
|
+
return Math.min(high, Math.max(low, value));
|
|
89
|
+
}
|
|
@@ -0,0 +1,331 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* @file Fit the cost of decoding from measured clips — and say which terms the
|
|
3
|
+
* measurements actually determine.
|
|
4
|
+
*
|
|
5
|
+
* The model is `cost = pixelTerm × Mpixel/s + bitrateTerm × Mbit/s +
|
|
6
|
+
* constantTerm`, in seconds of work per second of content.
|
|
7
|
+
*
|
|
8
|
+
* What went wrong with the fit this replaces, measured 2026-08-17: three clips
|
|
9
|
+
* for three unknowns is an EXACT system, and two of those clips shared a pixel
|
|
10
|
+
* rate and differed only in bitrate. A system that near-singular does not fail
|
|
11
|
+
* loudly — it returns whatever satisfies the three equations, and on that boot
|
|
12
|
+
* it returned `0.007542 × Mpx/s + 0.000000 × Mbit/s + 0.0000 s/s`. The bitrate
|
|
13
|
+
* term and the constant were zero, so a film's own bitrate never entered its
|
|
14
|
+
* price, and the whole prediction was 1.8-2.2x optimistic against what the same
|
|
15
|
+
* file measured while playing.
|
|
16
|
+
*
|
|
17
|
+
* Two things are different here, and both are about honesty rather than
|
|
18
|
+
* accuracy:
|
|
19
|
+
*
|
|
20
|
+
* - the system is OVER-determined (six clips, three unknowns) and solved by
|
|
21
|
+
* least squares, so no single noisy reading decides a term;
|
|
22
|
+
* - a term is published only if the measurements determine it. The test is
|
|
23
|
+
* the coefficient against its OWN standard error: a term smaller than the
|
|
24
|
+
* uncertainty in it has not been measured, and stating it as a number —
|
|
25
|
+
* zero or otherwise — claims knowledge that is not there. It is dropped,
|
|
26
|
+
* the remaining terms are fitted again, and the caller is told which shape
|
|
27
|
+
* survived.
|
|
28
|
+
*
|
|
29
|
+
* There is no threshold to tune in that rule and no coefficient anywhere in
|
|
30
|
+
* this file. "Smaller than its own uncertainty" is a statement about the data,
|
|
31
|
+
* not a setting.
|
|
32
|
+
*/
|
|
33
|
+
|
|
34
|
+
/**
|
|
35
|
+
* One measured clip.
|
|
36
|
+
*
|
|
37
|
+
* @typedef {object} DecodeSample
|
|
38
|
+
* @property {number} megapixelsPerSecond - Width × height × frame rate ÷ 1e6.
|
|
39
|
+
* @property {number} megabitsPerSecond - The clip's own measured bitrate.
|
|
40
|
+
* @property {number} costSecondsPerSecond - Seconds of work per second of
|
|
41
|
+
* content, from ffmpeg's own progress.
|
|
42
|
+
*/
|
|
43
|
+
|
|
44
|
+
/**
|
|
45
|
+
* The fitted model, with the shape the data supported.
|
|
46
|
+
*
|
|
47
|
+
* @typedef {object} DecodeCostModel
|
|
48
|
+
* @property {number} pixelTerm
|
|
49
|
+
* @property {number} bitrateTerm
|
|
50
|
+
* @property {number} constantTerm
|
|
51
|
+
* @property {string} shape - Which terms were determined.
|
|
52
|
+
* @property {string[]} dropped - Terms the measurements could not determine.
|
|
53
|
+
* @property {number} samples
|
|
54
|
+
* @property {number} residualRms - Typical disagreement between the fit and a
|
|
55
|
+
* measurement, in the same units as the cost.
|
|
56
|
+
*/
|
|
57
|
+
|
|
58
|
+
/** Every term the model can carry, in column order. */
|
|
59
|
+
const TERMS = ["pixels", "bitrate", "constant"];
|
|
60
|
+
|
|
61
|
+
/**
|
|
62
|
+
* Fit the model, dropping any term the measurements do not determine.
|
|
63
|
+
*
|
|
64
|
+
* @param {DecodeSample[]} samples
|
|
65
|
+
* @returns {DecodeCostModel | null} Null when nothing at all can be said: fewer
|
|
66
|
+
* measurements than terms, or no measurable dependence on the source.
|
|
67
|
+
*/
|
|
68
|
+
export function fitDecodeCost(samples) {
|
|
69
|
+
const usable = (Array.isArray(samples) ? samples : []).filter(
|
|
70
|
+
(sample) =>
|
|
71
|
+
Number.isFinite(sample?.megapixelsPerSecond) &&
|
|
72
|
+
sample.megapixelsPerSecond > 0 &&
|
|
73
|
+
Number.isFinite(sample?.megabitsPerSecond) &&
|
|
74
|
+
sample.megabitsPerSecond >= 0 &&
|
|
75
|
+
Number.isFinite(sample?.costSecondsPerSecond) &&
|
|
76
|
+
sample.costSecondsPerSecond > 0
|
|
77
|
+
);
|
|
78
|
+
// Fitting three terms to three points is what produced the degenerate answer
|
|
79
|
+
// this file exists to prevent: it has no residual and therefore no way to
|
|
80
|
+
// notice that two of the points said the same thing.
|
|
81
|
+
if (usable.length < TERMS.length + 1) {
|
|
82
|
+
return null;
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
let columns = [...TERMS];
|
|
86
|
+
let dropped = [];
|
|
87
|
+
for (let attempt = 0; attempt < TERMS.length; attempt += 1) {
|
|
88
|
+
const fit = leastSquares(usable, columns);
|
|
89
|
+
if (!fit) {
|
|
90
|
+
return null;
|
|
91
|
+
}
|
|
92
|
+
// Which terms the data did not determine: a coefficient no larger than the
|
|
93
|
+
// uncertainty in it. Only one is dropped per pass — removing a column
|
|
94
|
+
// changes the uncertainty in the others, so the question has to be asked
|
|
95
|
+
// again of the smaller model.
|
|
96
|
+
const undetermined = columns
|
|
97
|
+
.map((name, index) => ({
|
|
98
|
+
name,
|
|
99
|
+
coefficient: fit.coefficients[index],
|
|
100
|
+
ratio: determination(usable, columns, name, fit.coefficients[index], fit.standardErrors[index], fit.residualRms)
|
|
101
|
+
}))
|
|
102
|
+
// The pixel term is the one relationship every measurement agrees on, and
|
|
103
|
+
// a model without it says nothing at all. If IT is undetermined the
|
|
104
|
+
// answer is no model, which the caller handles by pricing the encoder
|
|
105
|
+
// alone and refusing nothing.
|
|
106
|
+
// A NEGATIVE coefficient is undetermined whatever its ratio says. Every
|
|
107
|
+
// term here is physically non-negative — more pixels cannot cost less
|
|
108
|
+
// work, and neither can more bits — so a negative fit is noise winning
|
|
109
|
+
// over an effect, not a discovery about the host. Clamping it to zero
|
|
110
|
+
// instead, which the first version of this file did, publishes a zero
|
|
111
|
+
// that looks measured: exactly the dishonesty the module exists to
|
|
112
|
+
// remove.
|
|
113
|
+
.filter((term) => term.ratio < 1 || term.coefficient < 0)
|
|
114
|
+
.sort((left, right) => {
|
|
115
|
+
const negative = (term) => (term.coefficient < 0 ? 0 : 1);
|
|
116
|
+
return negative(left) - negative(right) || left.ratio - right.ratio;
|
|
117
|
+
});
|
|
118
|
+
|
|
119
|
+
if (undetermined.length === 0) {
|
|
120
|
+
return published(columns, fit, dropped, usable.length);
|
|
121
|
+
}
|
|
122
|
+
const weakest = undetermined[0].name;
|
|
123
|
+
if (weakest === "pixels") {
|
|
124
|
+
return null;
|
|
125
|
+
}
|
|
126
|
+
columns = columns.filter((name) => name !== weakest);
|
|
127
|
+
dropped = [...dropped, weakest];
|
|
128
|
+
if (columns.length === 0) {
|
|
129
|
+
return null;
|
|
130
|
+
}
|
|
131
|
+
}
|
|
132
|
+
return null;
|
|
133
|
+
}
|
|
134
|
+
|
|
135
|
+
|
|
136
|
+
/**
|
|
137
|
+
* How well one term is determined, as a ratio against 1.
|
|
138
|
+
*
|
|
139
|
+
* Two ways a term can fail to be measured, and both have to be asked, because
|
|
140
|
+
* each is blind where the other sees:
|
|
141
|
+
*
|
|
142
|
+
* - **against the noise.** The term's effect across the range the clips
|
|
143
|
+
* actually cover — its coefficient times the spread of its own column — has
|
|
144
|
+
* to be larger than the typical disagreement between the fit and the
|
|
145
|
+
* measurements. A term whose whole contribution is smaller than the scatter
|
|
146
|
+
* was not measured; it was fitted to the scatter.
|
|
147
|
+
* - **against the arithmetic.** When a fit happens to be exact the scatter is
|
|
148
|
+
* zero, every standard error vanishes, and a coefficient of 7e-18 — which
|
|
149
|
+
* is how double-precision writes "nothing" — passes any test framed as a
|
|
150
|
+
* ratio. So the effect must also be larger than what the arithmetic itself
|
|
151
|
+
* can represent at this scale. That floor is a property of the numbers, not
|
|
152
|
+
* a setting: it moves with the size of the measurements.
|
|
153
|
+
*
|
|
154
|
+
* @param {DecodeSample[]} samples
|
|
155
|
+
* @param {string[]} columns
|
|
156
|
+
* @param {string} name
|
|
157
|
+
* @param {number} coefficient
|
|
158
|
+
* @param {number} standardError
|
|
159
|
+
* @param {number} residualRms
|
|
160
|
+
* @returns {number} Below 1 means the measurements do not determine this term.
|
|
161
|
+
*/
|
|
162
|
+
function determination(samples, columns, name, coefficient, standardError, residualRms) {
|
|
163
|
+
const values = samples.map((sample) => {
|
|
164
|
+
if (name === "pixels") {
|
|
165
|
+
return sample.megapixelsPerSecond;
|
|
166
|
+
}
|
|
167
|
+
if (name === "bitrate") {
|
|
168
|
+
return sample.megabitsPerSecond;
|
|
169
|
+
}
|
|
170
|
+
return 1;
|
|
171
|
+
});
|
|
172
|
+
const spread = name === "constant" ? 1 : Math.max(...values) - Math.min(...values);
|
|
173
|
+
const effect = Math.abs(coefficient) * spread;
|
|
174
|
+
const meanCost = samples.reduce((sum, sample) => sum + sample.costSecondsPerSecond, 0) / samples.length;
|
|
175
|
+
const arithmeticFloor = Math.abs(meanCost) * Number.EPSILON * samples.length * columns.length;
|
|
176
|
+
const noiseFloor = Math.max(residualRms, arithmeticFloor);
|
|
177
|
+
const againstNoise = noiseFloor > 0 ? effect / noiseFloor : Number.POSITIVE_INFINITY;
|
|
178
|
+
const againstError = standardError > 0 ? Math.abs(coefficient) / standardError : Number.POSITIVE_INFINITY;
|
|
179
|
+
return Math.min(againstNoise, againstError);
|
|
180
|
+
}
|
|
181
|
+
|
|
182
|
+
/**
|
|
183
|
+
* Assemble the answer, with absent terms as zero — and `dropped` saying that
|
|
184
|
+
* the zero means "not measured" rather than "measured to be nothing".
|
|
185
|
+
*
|
|
186
|
+
* @param {string[]} columns
|
|
187
|
+
* @param {{ coefficients: number[], residualRms: number }} fit
|
|
188
|
+
* @param {string[]} dropped
|
|
189
|
+
* @param {number} samples
|
|
190
|
+
* @returns {DecodeCostModel | null}
|
|
191
|
+
*/
|
|
192
|
+
function published(columns, fit, dropped, samples) {
|
|
193
|
+
const value = (name) => {
|
|
194
|
+
const index = columns.indexOf(name);
|
|
195
|
+
return index < 0 ? 0 : fit.coefficients[index];
|
|
196
|
+
};
|
|
197
|
+
const pixelTerm = value("pixels");
|
|
198
|
+
// A negative price for pixels is not a host being unusual — it is noise
|
|
199
|
+
// larger than the effect, and carrying it would price a bigger picture as
|
|
200
|
+
// cheaper than a smaller one.
|
|
201
|
+
if (!(pixelTerm > 0)) {
|
|
202
|
+
return null;
|
|
203
|
+
}
|
|
204
|
+
const constantTerm = value("constant");
|
|
205
|
+
return {
|
|
206
|
+
pixelTerm,
|
|
207
|
+
bitrateTerm: Math.max(0, value("bitrate")),
|
|
208
|
+
// A negative constant would make a small enough source free, which no
|
|
209
|
+
// measurement here supports: it is the fitted line crossing below zero
|
|
210
|
+
// outside the range that was measured.
|
|
211
|
+
constantTerm: Math.max(0, constantTerm),
|
|
212
|
+
shape: columns.join("+"),
|
|
213
|
+
dropped,
|
|
214
|
+
samples,
|
|
215
|
+
residualRms: fit.residualRms
|
|
216
|
+
};
|
|
217
|
+
}
|
|
218
|
+
|
|
219
|
+
/**
|
|
220
|
+
* Least squares over the named columns, with the standard error of each
|
|
221
|
+
* coefficient.
|
|
222
|
+
*
|
|
223
|
+
* Solved through the normal equations. The matrix is at most 3×3 and the
|
|
224
|
+
* columns are of similar magnitude once a clip set varies them deliberately,
|
|
225
|
+
* which is exactly what the set is designed for.
|
|
226
|
+
*
|
|
227
|
+
* @param {DecodeSample[]} samples
|
|
228
|
+
* @param {string[]} columns
|
|
229
|
+
* @returns {{ coefficients: number[], standardErrors: number[], residualRms: number } | null}
|
|
230
|
+
*/
|
|
231
|
+
function leastSquares(samples, columns) {
|
|
232
|
+
const rowOf = (sample) =>
|
|
233
|
+
columns.map((name) => {
|
|
234
|
+
if (name === "pixels") {
|
|
235
|
+
return sample.megapixelsPerSecond;
|
|
236
|
+
}
|
|
237
|
+
if (name === "bitrate") {
|
|
238
|
+
return sample.megabitsPerSecond;
|
|
239
|
+
}
|
|
240
|
+
return 1;
|
|
241
|
+
});
|
|
242
|
+
const width = columns.length;
|
|
243
|
+
const normal = Array.from({ length: width }, () => new Array(width).fill(0));
|
|
244
|
+
const right = new Array(width).fill(0);
|
|
245
|
+
for (const sample of samples) {
|
|
246
|
+
const row = rowOf(sample);
|
|
247
|
+
for (let i = 0; i < width; i += 1) {
|
|
248
|
+
right[i] += row[i] * sample.costSecondsPerSecond;
|
|
249
|
+
for (let j = 0; j < width; j += 1) {
|
|
250
|
+
normal[i][j] += row[i] * row[j];
|
|
251
|
+
}
|
|
252
|
+
}
|
|
253
|
+
}
|
|
254
|
+
const inverse = invert(normal);
|
|
255
|
+
if (!inverse) {
|
|
256
|
+
return null;
|
|
257
|
+
}
|
|
258
|
+
const coefficients = inverse.map((row) => row.reduce((sum, value, index) => sum + value * right[index], 0));
|
|
259
|
+
|
|
260
|
+
let residualSquares = 0;
|
|
261
|
+
for (const sample of samples) {
|
|
262
|
+
const row = rowOf(sample);
|
|
263
|
+
const predicted = row.reduce((sum, value, index) => sum + value * coefficients[index], 0);
|
|
264
|
+
residualSquares += (sample.costSecondsPerSecond - predicted) ** 2;
|
|
265
|
+
}
|
|
266
|
+
const degreesOfFreedom = samples.length - width;
|
|
267
|
+
const variance = degreesOfFreedom > 0 ? residualSquares / degreesOfFreedom : 0;
|
|
268
|
+
const standardErrors = inverse.map((row, index) => Math.sqrt(Math.max(0, variance * row[index])));
|
|
269
|
+
return {
|
|
270
|
+
coefficients,
|
|
271
|
+
standardErrors,
|
|
272
|
+
residualRms: Math.sqrt(residualSquares / samples.length)
|
|
273
|
+
};
|
|
274
|
+
}
|
|
275
|
+
|
|
276
|
+
/**
|
|
277
|
+
* Invert a small symmetric matrix by Gauss-Jordan, or null when it is singular.
|
|
278
|
+
*
|
|
279
|
+
* @param {number[][]} matrix
|
|
280
|
+
* @returns {number[][] | null}
|
|
281
|
+
*/
|
|
282
|
+
function invert(matrix) {
|
|
283
|
+
const size = matrix.length;
|
|
284
|
+
const work = matrix.map((row, index) => [
|
|
285
|
+
...row,
|
|
286
|
+
...Array.from({ length: size }, (_unused, column) => (column === index ? 1 : 0))
|
|
287
|
+
]);
|
|
288
|
+
for (let column = 0; column < size; column += 1) {
|
|
289
|
+
let pivot = column;
|
|
290
|
+
for (let row = column + 1; row < size; row += 1) {
|
|
291
|
+
if (Math.abs(work[row][column]) > Math.abs(work[pivot][column])) {
|
|
292
|
+
pivot = row;
|
|
293
|
+
}
|
|
294
|
+
}
|
|
295
|
+
// Singular to the precision of the arithmetic: the columns are not
|
|
296
|
+
// independent, which is a statement about the clip set, not about the host.
|
|
297
|
+
if (Math.abs(work[pivot][column]) < 1e-12) {
|
|
298
|
+
return null;
|
|
299
|
+
}
|
|
300
|
+
[work[column], work[pivot]] = [work[pivot], work[column]];
|
|
301
|
+
const scale = work[column][column];
|
|
302
|
+
for (let k = 0; k < 2 * size; k += 1) {
|
|
303
|
+
work[column][k] /= scale;
|
|
304
|
+
}
|
|
305
|
+
for (let row = 0; row < size; row += 1) {
|
|
306
|
+
if (row === column) {
|
|
307
|
+
continue;
|
|
308
|
+
}
|
|
309
|
+
const factor = work[row][column];
|
|
310
|
+
for (let k = 0; k < 2 * size; k += 1) {
|
|
311
|
+
work[row][k] -= factor * work[column][k];
|
|
312
|
+
}
|
|
313
|
+
}
|
|
314
|
+
}
|
|
315
|
+
return work.map((row) => row.slice(size));
|
|
316
|
+
}
|
|
317
|
+
|
|
318
|
+
/**
|
|
319
|
+
* What a model prices one source at.
|
|
320
|
+
*
|
|
321
|
+
* @param {DecodeCostModel} model
|
|
322
|
+
* @param {{ megapixelsPerSecond: number, megabitsPerSecond: number }} source
|
|
323
|
+
* @returns {number} Seconds of work per second of content.
|
|
324
|
+
*/
|
|
325
|
+
export function decodeCostOf(model, source) {
|
|
326
|
+
return (
|
|
327
|
+
model.pixelTerm * source.megapixelsPerSecond +
|
|
328
|
+
model.bitrateTerm * source.megabitsPerSecond +
|
|
329
|
+
model.constantTerm
|
|
330
|
+
);
|
|
331
|
+
}
|