@woosh/meep-engine 3.60.4 → 3.60.5

package/src/engine/graphics/texture/sampler/compare/FLIP_IMPLEMENTATION_PLAN.md

@@ -1,466 +0,0 @@
-# FLIP on the CPU — implementation plan
-
-**Status:** LDR-FLIP implemented and conformance-tested against NVIDIA's `flip-evaluator` 1.7.
-HDR-FLIP and the Shade compute port remain unbuilt.
-**Scope:** CPU only, `Sampler2D` in / `Sampler2D` out.
-
-Sections below describe what was built; where the implementation departed from the original
-proposal, the departure is called out inline.
-
-FLIP is NVIDIA's perceptual image difference evaluator (Andersson et al., HPG 2020; HDR
-extension 2021). It answers "would a person flipping back and forth between these two images
-notice a difference, and where" — which is exactly the question a renderer regression gate
-wants answered, and exactly the question RMSE/PSNR answer badly.
-
-Reference: <https://github.com/NVlabs/flip> (BSD-3-Clause),
-<https://research.nvidia.com/publication/flip>.
-
-The output is a per-pixel error image in `[0, 1]` plus a scalar pooled score. It is symmetric,
-parameter-free apart from the viewing condition (pixels per degree), and it degrades
-gracefully: a uniform tint scores low, a displaced edge scores high.
-
----
-
-## 1. What already exists, and what has to be written
-
-The bulk of the pipeline is colour-space plumbing and separable convolution. meep already owns
-most of the former.
-
-### Reused verbatim
-
-| Need | Module |
-| --- | --- |
-| sRGB u8 RGBA -> linear f32 RGBA | `engine/graphics/texture/sampler/sampler2d_srgb_to_linear.js` |
-| linear sRGB -> CIE XYZ (D65) | `core/color/xyz/rgb_to_xyz.js` |
-| CIE XYZ -> linear sRGB (D65) | `core/color/xyz/xyz_to_rgb.js` |
-| clamp to gamut | `core/math/clamp01.js` (`saturate`) |
-| `Y/116 + 16/116` channel remap | `sampler2d_channel_linear_transform.js` (gradient/intercept fit it exactly) |
-| replicate a channel into a wider image | `sampler2d_copy_channel_data.js` |
-| float image container | `Sampler2D.float32(...)`, `Sampler2D.uint8(...)` |
-| base64 for the conformance fixture and the colour map | `core/binary/base64/Base64.js` |
-| integer-backing check on the public entry point | `core/collection/array/typed/typed_array_is_integer.js` |
-
-The existing `rgb_to_xyz` / `xyz_to_rgb` matrices are the sRGB D65 primaries, which is what
-FLIP uses. `convert_channel_sRGB_to_linear` is the standard piecewise EOTF with the same
-`0.04045` breakpoint FLIP uses. Both drop in without modification.
-
-Downstream of the module, `sampler2d_to_data_url.js` and `downloadSamplerAsPNG.js` take the magma
-output straight to a browser harness. Nothing in the module calls them.
-
-### Deliberately *not* reused
-
-- **`filter/sampler2d_blur_gaussian.js`** — cannot be used. It samples taps with
-  `sampleChannelBilinear` at fractional offsets (an approximation of the filter, not the
-  filter), applies one kernel to every channel, and only does Gaussians. FLIP needs a distinct
-  kernel per channel, a two-Gaussian channel, and first/second derivative kernels.
-- **`filter/sampler2d_scale_down_generic.js`** — 2D radial weight function tied to a resampling
-  ratio. Wrong shape.
-- **`sampler2d_combine.js`** — the proposal expected to use it for the two per-pixel difference
-  passes, and the implementation does not. It marshals each texel into a plain array before handing
-  it to the callback, whereas `lab_distance_hyab` already reads `(array, offset)` directly; going
-  through it would add a copy per pixel and an indirection without shortening the call site.
-  (Surveying it for this did turn up a defect, since fixed — see §7.)
-- **`computeStatisticalMean` / `computeStatisticalPartialMedian`** — the mean is accumulated in the
-  same pass that already copies the map and finds its extremes, so routing it through a helper that
-  takes `number[]` would add a pass and a conversion. The median helper is the wrong statistic
-  outright; see §3.7.
-- **`PNGReader`** — the conformance fixture inlines raw RGB bytes rather than PNGs, which are
-  smaller once base64'd and cannot be edited out of step with the expectation.
-
-### New code
-
-| Module | Purpose |
-| --- | --- |
-| `core/color/ycxcz/xyz_to_ycxcz.js` | CIE XYZ -> YyCxCz opponent space |
-| `core/color/ycxcz/ycxcz_to_xyz.js` | inverse |
-| `core/color/lab/xyz_to_lab.js` | CIE XYZ -> CIELab (D65) |
-| `core/color/lab/lab_apply_hunt_adjustment.js` | FLIP's Hunt effect scaling of `a`,`b` by `0.01 * L` |
-| `core/color/lab/lab_distance_hyab.js` | HyAB colour distance |
-| `core/color/illuminant/D65_TRISTIMULUS_XYZ.js` | the white point the above three share |
-| `engine/graphics/texture/sampler/filter/SeparableKernel.js` | per-channel, per-axis tap storage |
-| `engine/graphics/texture/sampler/filter/sampler2d_convolve_separable.js` | **generic** separable convolution, per-channel kernels, clamp-to-edge |
-| `.../compare/flip/flip_constants.js` | the published constants |
-| `.../compare/flip/FlipParameters.js` | viewing condition |
-| `.../compare/flip/flip_compute_pixels_per_degree.js` | ppd from monitor geometry |
-| `.../compare/flip/flip_build_spatial_kernel.js` | CSF kernel, 4 channels |
-| `.../compare/flip/flip_build_feature_kernel.js` | point/edge kernels, 4 channels |
-| `.../compare/flip/flip_srgb_to_ycxcz.js` | decode and move into the opponent space |
-| `.../compare/flip/flip_filter_spatial.js` | CSF filter and the trip to Hunt-adjusted CIELab |
-| `.../compare/flip/flip_filter_feature.js` | point and edge detection |
-| `.../compare/flip/flip_compute_color_difference.js` | HyAB plus the redistribution curve |
-| `.../compare/flip/flip_compute_feature_difference.js` | the stronger of the two structural disagreements |
-| `.../compare/flip/sampler2d_flip_ldr.js` | the pipeline; returns the error map |
-| `.../compare/flip/FlipPooling.js` + `flip_pool.js` | mean / weighted median / min / max |
-| `engine/graphics/texture/sampler/sampler2d_channel_apply_lut.js` | **generic** channel-through-a-lookup-table mapping |
-| `core/color/colormap/MAGMA_LUT.js` | the colour map the error visualization uses |
-
-The magma table is matplotlib's, which is public domain; nothing is copied from the FLIP sources.
-
-The last two started as a single `flip_error_to_magma` and were promoted out: neither the lookup
-nor the colour map has anything to do with FLIP. What remained FLIP-specific after the split was
-nothing at all, so no wrapper was kept.
-
-`Sampler2D` lives under `engine/`, so anything touching it must live under `engine/` too — the
-colour primitives are pure math and belong in `core/color/`, per the core/engine decoupling
-rule.
-
----
-
-## 2. The one new generic primitive
-
-Everything convolutional in FLIP reduces to a single operation, and it is worth having in
-`filter/` on its own merits:
-
-```js
-sampler2d_convolve_separable(output, input, kernel)   // kernel: SeparableKernel
-```
-
-The four loose parameters of the proposal became one `SeparableKernel` — taps, radius and channel
-count travel together, and callers stop being able to pair the wrong radius with the wrong taps.
-
-Notes on the shape:
-
-- **Per-channel kernels** are the whole reason this is new. FLIP's achromatic, red-green and
-  blue-yellow channels have different spatial extents, and the feature stage wants a derivative
-  along one axis and a smoothing kernel along the other *in the same pass*.
-- **Separate `kernel_x` / `kernel_y`** buys the feature stage for free: pack
-  `[edge_x, edge_y, point_x, point_y]` as four channels with `kernel_x = [dg, g, ddg, g]` and
-  `kernel_y = [g, dg, g, ddg]`, and one call produces all four responses.
-- **Clamp-to-edge** by clamping the tap index. At integer coordinates `sampleChannelBilinear`
-  already degenerates to a clamped texel fetch, so this is the same operator written plainly;
-  `readChannel` does *not* clamp, so it cannot be used directly.
-- Intermediate buffer: a module-static `Float64Array` grown on demand, following the pattern
-  already in `sampler2d_blur_gaussian.js`. f64 accumulation, f32 storage.
-- **Loop order**: taps outside, channels inside. The border clamp is then evaluated once per tap
-  rather than once per tap per channel, and a texel's channels accumulate in memory order. Measured
-  at 2.3x on the four channel images this pipeline uses — see §8.
-
----
-
-## 3. Pipeline
-
-All stages operate on full images; no tiling in v1.
-
-### 3.0 Decode
-
-`sampler2d_srgb_to_linear(reference)` and the same for `test`. 4x u8 -> 4x f32. Alpha is carried
-but ignored — FLIP is defined on RGB.
-
-Precondition: both images share dimensions. Assert it; do not resample.
-
-### 3.1 Linear RGB -> YyCxCz
-
-Per pixel: `rgb_to_xyz` then `xyz_to_ycxcz`, against the D65 reference white:
-
-```
-Y  = 116 * (y / Yn) - 16
-Cx = 500 * (x / Xn - y / Yn)
-Cz = 200 * (y / Yn - z / Zn)
-```
-
-Output: 3x f32 `Sampler2D` per image. This image feeds *both* the spatial stage and the feature
-stage — the feature detectors read the **unfiltered** achromatic channel.
-
-**On the white point.** `D65_TRISTIMULUS_XYZ` is derived by summing the rows of `rgb_to_xyz`, not
-quoted from a table. The reference implementation quotes `(0.950428545, 1, 1.088900371)`, which
-differs from meep's matrix in the fifth digit — enough that its own reference white does not land
-exactly on `L* = 100, a* = 0, b* = 0`. Deriving the white from the matrix it is used with makes that
-round trip exact, which is a property `xyz_to_lab.spec.js` asserts, and the conformance spec shows
-the choice costs nothing in agreement: the residual is below half a quantization step regardless.
-
-### 3.2 Spatial contrast sensitivity filter
-
-The CSF is fitted in the frequency domain as a sum of exponentials, giving a spatial-domain
-kernel that is a sum of Gaussians. Constants (Y, Cx, Cz):
-
-```
-a1 = [ 1.0,    1.0,    34.1  ]
-b1 = [ 0.0047, 0.0053,  0.04  ]
-a2 = [ 0.0,    0.0,    13.5  ]
-b2 = [ 1.0e-5, 1.0e-5,  0.025 ]
-```
-
-`a2 = 0` for Y and Cx, so those are single Gaussians and separable outright. Cz is a sum of two
-Gaussians, which is *not* separable as one kernel — but each term is, so the reference
-implementation runs both terms as separate channels and sums them after the vertical pass. That
-maps onto our primitive directly:
-
-1. Build a **4-channel** input `[Y, Cx, Cz, Cz]` by copying channel 2 twice
-   (`sampler2d_copy_channel_data`).
-2. `flip_build_spatial_kernel(ppd)` returns a 4-channel kernel `[gY, gCx, gCz1, gCz2]` and a
-   radius, with
-
-   ```
-   radius   = ceil(3 * sqrt(max(b) / (2 * PI^2)) * ppd)      // max(b) = 0.04
-   delta_x  = 1 / ppd
-   ix       = (i - radius) * delta_x
-
-   gaussian(x2, a, b)      = a * sqrt(PI / b) * exp(-PI^2 * x2 / b)
-   gaussian_sqrt(x2, a, b) = sqrt(a * sqrt(PI / b)) * exp(-PI^2 * x2 / b)
-
-   gY[i]   = gaussian(ix*ix, a1.Y,  b1.Y)
-   gCx[i]  = gaussian(ix*ix, a1.Cx, b1.Cx)
-   gCz1[i] = gaussian_sqrt(ix*ix, a1.Cz, b1.Cz)
-   gCz2[i] = gaussian_sqrt(ix*ix, a2.Cz, b2.Cz)
-   ```
-
-   `gaussian_sqrt` takes the square root of the *amplitude only*, so that the outer product
-   `g(x) * g(y)` reproduces the intended 2D Gaussian.
-
-   Normalization: `gY` and `gCx` are scaled to sum to 1 (which makes their 2D outer products sum
-   to 1). The two Cz terms are scaled *jointly* by `1 / sqrt(S1^2 + S2^2)`, because the 2D mass
-   of the summed filter is `S1^2 + S2^2`.
-
-3. One `sampler2d_convolve_separable` call with the same kernel on both axes.
-4. Fold: `Cz = ch2 + ch3`, producing a 3-channel filtered YCxCz.
-
-At the reference default (ppd 67.02) the radius is 10, so a 21-tap kernel; at a more typical
-1080p desktop condition (ppd ~47) it is 7, so 15 taps.
-
-### 3.3 Filtered YCxCz -> Hunt-adjusted CIELab
-
-Per pixel:
-
-```
-ycxcz_to_xyz -> xyz_to_rgb -> saturate each channel -> rgb_to_xyz -> xyz_to_lab -> hunt
-```
-
-The round trip through linear RGB is not redundant: the filtered signal can leave the sRGB
-gamut, and the clamp has to happen in **linear RGB** before Lab, exactly as the reference does.
-
-CIELab uses the same D65 white and the standard `delta = 6/29` piecewise nonlinearity.
-
-Hunt adjustment (this is FLIP's own lightness-dependent chroma scaling, unrelated to the
-existing `core/color/hunt/xyz_to_hpe.js` matrix):
-
-```
-a' = 0.01 * L * a
-b' = 0.01 * L * b
-```
-
-Output: 3x f32 per image.
-
-### 3.4 Colour difference
-
-```
-d = |L_r - L_t| + sqrt((a_r - a_t)^2 + (b_r - b_t)^2)     // HyAB
-d = d ^ qc                                                // qc = 0.7
-
-cmax   = HyAB(hunt(lab(green)), hunt(lab(blue))) ^ qc     // green/blue sRGB primaries
-pccmax = pc * cmax                                        // pc = 0.4
-
-dc = d < pccmax
-   ? (pt / pccmax) * d                                    // pt = 0.95
-   : pt + ((d - pccmax) / (cmax - pccmax)) * (1 - pt)
-```
-
-`cmax` is computed once at module load from the primaries rather than hard-coded — it is a
-derived value, not a magic number.
-
-Output: 1x f32.
-
-### 3.5 Feature difference
-
-1. Normalized achromatic channel from the **unfiltered** YCxCz:
-   `y_n = Y * (1/116) + (16/116)` — one `sampler2d_channel_linear_transform` call.
-2. Broadcast into a 4-channel image (four `sampler2d_copy_channel_data` calls).
-3. `flip_build_feature_kernel(ppd)`:
-
-   ```
-   std_dev = 0.5 * w * ppd            // w = 0.082
-   radius  = ceil(3 * std_dev)
-
-   g(x)   = exp(-x^2 / (2 * std_dev^2))
-   dg(x)  = -x * g(x)                          // 1st derivative -> edge
-   ddg(x) = (x^2 / std_dev^2 - 1) * g(x)       // 2nd derivative -> point
-   ```
-
-   `g` is normalized to sum 1; `dg` and `ddg` have their positive and negative lobes normalized
-   separately to `+1` and `-1`. Because `g` is non-negative and sums to 1, doing this in 1D
-   gives the same lobe sums in the 2D outer product, so the separable build is equivalent to the
-   reference's 2D normalization.
-
-   `kernel_x = [dg, g, ddg, g]`, `kernel_y = [g, dg, g, ddg]`.
-4. One `sampler2d_convolve_separable` call per image.
-5. Per pixel:
-
-   ```
-   edge  = hypot(ch0, ch1)
-   point = hypot(ch2, ch3)
-   df = ( (1 / sqrt(2)) * max(|edge_r - edge_t|, |point_r - point_t|) ) ^ qf   // qf = 0.5
-   ```
-
-Output: 1x f32.
-
-### 3.6 Combine
-
-```
-error = dc ^ (1 - df)
-```
-
-1x f32 `Sampler2D`, values in `[0, 1]`. This is the deliverable.
-
-### 3.7 Pooling
-
-`flip_pool(error)` -> `FlipPooling { mean, weighted_median, minimum, maximum }`.
-
-**Correction to the proposal.** It claimed the reference tool derives its weighted median from a
-100-bin histogram. It does not — the statistic is exact. Walk the error values in ascending order
-accumulating *error*, and return the value at which half the total has been reached. Checked
-against the tool: it reports `0.969462` for the fixture in §5, the exact walk gives
-`0.9694619774818420`, and a 100-bin histogram reading of the same data gives `0.965`. The spec pins
-this, because the number alone does not reveal which definition produced it.
-
-`computeStatisticalPartialMedian` is not reusable here — it is the median of the pixels, not of the
-error, and the two differ by a lot on any image whose disagreement is localized. Sorting is done on
-a `Float64Array` copy, which sorts numerically without a comparator and leaves the caller's map
-untouched.
-
-### 3.8 Visualization
-
-`sampler2d_channel_apply_lut(error, 0, MAGMA_LUT, 3)` -> 3x u8 `Sampler2D`, then the existing
-`sampler2d_to_data_url` / `downloadSamplerAsPNG` for browser harnesses. This is what makes the
-metric usable — the error map is the point, the scalar is the summary.
-
-The lookup is a general sampler operation rather than a FLIP one. It maps a chosen channel onto
-table entries linearly across a range (defaulting to `[0, 1]`, which suits normalized data; pass
-`0, 255` to index a palette by a raw 8-bit value), and the result takes its channel count and its
-backing array type from the table. Magma is worth the table lookup over a greyscale ramp because it
-is monotonic in *perceptual lightness* — `MAGMA_LUT.spec.js` asserts exactly that, in CIELAB. Rec.709
-luma over the gamma-encoded bytes reverses at three of the 256 entries, which is a good illustration
-of why the property has to be stated in the right space to mean anything.
-
----
-
-## 4. Public surface
-
-```js
-const params = new FlipParameters();
-params.pixels_per_degree = flip_compute_pixels_per_degree(0.7, 0.7, 3840); // distance m, width m, px
-
-const error = sampler2d_flip_ldr(reference, test, params);   // 1x f32 Sampler2D
-const pooled = flip_pool(error);                             // FlipPooling
-
-const preview = sampler2d_channel_apply_lut(error, 0, MAGMA_LUT, 3);   // 3x u8 Sampler2D
-```
-
-`FlipParameters` is a class with typed fields and a sensible default (ppd 67.0206, the reference
-tool's default viewing condition), not an options bag.
-
----
-
-## 5. Verification
-
-`sampler2d_flip_ldr.spec.js` is the gate. Black-box only — observable behaviour of the error
-map, never internal call structure.
-
-**Structural, cheap, catches most breakage:**
-
-- `flip(A, A)` is zero at every pixel, for several images including hard edges and noise.
-- Symmetry: `flip(A, B)` equals `flip(B, A)` pixel for pixel. FLIP is symmetric by construction
-  (HyAB and both feature differences are absolute), so any asymmetry is a bug.
-- Monotonicity: increasing a uniform luminance offset increases the mean, without exception.
-- A one-pixel shift of a high-contrast edge scores materially higher than a uniform tint of
-  equal RMSE — the property that motivates using FLIP at all.
-- Kernel specs: spatial kernels sum to 1 per channel (Cz jointly), derivative lobes sum to
-  `+/-1`, radius matches the published formulas at several ppd values.
-
-**Colour primitives** get their own specs against published values: D65 white -> `L*=100, a=0,
-b=0`; a YCxCz round trip returns the input; sRGB primaries land on their tabulated Lab
-coordinates.
-
-**Conformance** against the reference implementation is the one that actually pins the constants,
-and it is in `sampler2d_flip_ldr.conformance.spec.js`. A 24x24 quadrant fixture — one quadrant per
-path through the metric — is inlined as base64 along with the full error map and the pooled
-statistics that `flip-evaluator` 1.7 produces for it, at two viewing conditions. `PNGReader` turned
-out to be unnecessary: raw RGB bytes inline more compactly than a PNG and cannot drift from the
-expectation.
-
-Result: **the worst pixel in the map diverges by 0.498 of one 8-bit step**, which is below the
-half-step floor of what a quantized expectation can resolve — the two implementations agree to the
-last digit the fixture can express. The pooled mean differs by `4e-6` at the default condition and
-`1e-5` at the coarse one, consistent with the reference accumulating in float32 where this
-accumulates in float64.
-
-To regenerate the fixture — only needed if the fixture images themselves change — install the
-reference tool and read the values back out of it:
-
-```bash
-pip install flip-evaluator
-```
-
-`flip.evaluate(reference, test, "LDR", applyMagma=False)` returns the raw error map and the mean;
-the CLI (`flip -r ref.png -t test.png`) is the only source for the pooled median and quartiles.
-Neither is a dependency of the build or the test suite.
-
----
-
-## 6. Phasing
-
-1. ~~Colour primitives in `core/color/` + specs.~~ **Done.** Independently useful; CIELab and HyAB
-   were missing from meep regardless of FLIP.
-2. ~~`sampler2d_convolve_separable` + specs.~~ **Done.** Also independently useful.
-3. ~~LDR-FLIP pipeline + the gate spec.~~ **Done.**
-4. ~~Pooling and magma visualization.~~ **Done.** Wiring into the `probeShade*` harnesses so a
-   device screenshot can be diffed against a golden is **not** done.
-5. *Not built:* HDR-FLIP.
-6. *Not built:* Shade compute port. The algorithm is embarrassingly parallel and the WGSL test
-   infrastructure exists, so this is a natural follow-up — but the CPU path is what a vitest gate
-   can call, so it comes first.
-
-Built: 23 implementation modules, 12 spec files, 67 tests.
-
----
-
-## 7. Risks and open decisions
-
-1. ~~**Conformance fixtures.**~~ **Resolved:** taken, and it was the right call — see §5. The tool
-   was run once offline to generate the fixture; nothing from it is a runtime or test dependency.
-
-2. ~~**Float precision.**~~ **Measured:** `4e-6` to `1e-5` on the pooled mean, so the spec asserts
-   to 4 decimal places. Storage is f32 and accumulation f64, as proposed.
-
-3. ~~**`sampler2d_combine` has a latent bug.**~~ Its read loop for `input1` was bounded by
-   `input0.itemSize`, so it silently truncated when the second input was wider. Found while
-   surveying it for reuse here, and **fixed separately** in `0bc24b72a`.
-
-4. **HDR-FLIP must not reuse `tonemap_aces`.** The reference uses the Narkowicz rational fit with
-   a 0.6 pre-exposure cancellation baked into the coefficients; meep's `tonemap_aces` is Stephen
-   Hill's matrix fit. They are different operators and swapping them breaks parity. HDR-FLIP also
-   needs `computeExposureParameters`, whose constants should be lifted verbatim from `FLIP.h`
-   rather than re-derived.
-
-5. **Provenance.** FLIP is BSD-3-Clause. Constants and the magma LUT are fine to port with the
-   notice retained in the module headers; confirm before copying the LUT table verbatim.
-
-6. ~~**Two formulas to re-check at implementation time.**~~ Both are confirmed correct by the
-   conformance spec — a wrong amplitude on the two-lobe blue-yellow filter or a wrong lobe
-   normalization would not agree to within half a quantization step.
-
-7. **Redundant work in the feature pass.** The four detector channels are convolved from four
-   copies of the same signal, and in the horizontal pass two of them (the two whose horizontal
-   kernel is the plain gaussian) compute an identical result. Removing the duplicate would mean a
-   convolution primitive whose second pass can read a different channel than its first, which is a
-   worse primitive; the waste is roughly 12% of the feature filter. Left as is deliberately.
-
----
-
-## 8. Cost
-
-Measured at 1920x1080, default viewing condition (spatial radius 10, feature radius 9):
-
-| Stage | Cost |
-| --- | --- |
-| decode + opponent space | ~0.3s per image |
-| spatial filter + trip to CIELab | ~1.7s per image |
-| feature filter | ~1.2s per image |
-| both difference passes + pooling | ~0.45s total |
-
-**Roughly 6 seconds for a 1080p pair warm, 10 cold.** The proposal guessed "low single-digit
-seconds" and was out by about 3x: the first working version measured 16s, and reordering the
-convolution loops (§2) brought it to 6. The two separable passes are ~90% of what remains.
-
-Fine for an offline gate and for a harness button; not fine for a per-frame in-engine overlay. The
-next lever, if one is wanted, is splitting the convolution into border and interior ranges so that
-the ~99% of pixels which cannot reach an edge skip the clamp entirely. After that the honest answer
-is the GPU, which is phase 6.