@woosh/meep-engine 3.60.4 → 3.60.6

package/src/core/binary/lz4/LZ4_HIGH_COMPRESSION_PLAN_2026_09_14.md

@@ -1,542 +0,0 @@
-# LZ4_HIGH_COMPRESSION_PLAN — `src/core/binary/lz4`
-
-Date: 2026-09-14 · Adds a compression level to the encoder. The format does not change and
-[lz4_decompress_block.js](lz4_decompress_block.js) is not touched.
-
-Every number below was measured on this machine on 2026-09-14, against real data. The measurement
-recipes are in §9 so any of it can be re-run.
-
----
-
-## 1 · The question, and the answer
-
-> Add a configurable compression level that gets us to the current state-of-the-art maximum, without
-> changing the format.
-
-**The state of the art for the LZ4 block format is `lz4hc` level 12, and there is nothing
-meaningful past it.** An encoder built with an *uncapped* match search and a shortest-path parse over
-the whole block — strictly more search than level 12 is allowed — was measured against level 12:
-
-| corpus | level 12 | uncapped search + whole-block parse | time |
-|---|---|---|---|
-| 60 real `.vgeo` pages (1.77 MB) | 1,531,612 B | **1,531,612 B** — identical | 1.6× slower |
-| 24 text chunks (1.57 MB) | 356,238 B | 356,233 B — **0.0014% smaller** | 7.9× slower |
-| 18 runs and periodic blocks (74 KB) | 4,225 B | **4,225 B** — identical | 110× slower |
-
-Five bytes in a third of a megabyte, for eight times the work — and on repetitive data the uncapped
-parse is 110× level 12 and 3,500× level 9, because dropping the reference's `sufficient_len` early
-exit is exactly what lets a run of zeros cost quadratic time. The ceiling is level 12; the plan is to
-reach it and stop.
-
-A JS port of `lz4hc` was written and validated before this plan was drafted: **20,976 blocks over
-1,947 inputs, at every level from 2 to 12, byte for byte identical to liblz4**, first try, including
-runs, periodic data, matches at the far edge of the 64KB window, engine source, glTF JSON and real
-geometry pages. The work is a known quantity, not an experiment.
-
----
-
-## 2 · What it is worth
-
-Corpus: **2,636 decoded pages, 140,268,332 bytes**, taken from 120 of the 2,975 `.vgeo` containers
-the Zorah build wrote on 2026-09-13. These are the exact bytes
-[vgeo_encode_frame_blob.js](../../../shade/renderer/geometry/virtual/format/frame/vgeo_encode_frame_blob.js)
-is handed in production. Baseline is what [lz4_compress_block.js](lz4_compress_block.js) ships today.
-
-| level | ratio | vs. today | JS throughput |
-|---|---|---|---|
-| fast (today) | 0.89470 | — | 412–436 MB/s |
-| 3 | 0.87272 | −2.457% | 51–53 MB/s |
-| 4 | 0.87239 | −2.494% | 51 MB/s |
-| 6 | 0.87214 | −2.522% | 51 MB/s |
-| **9** | **0.87209** | **−2.528%** | **52–55 MB/s** |
-| 10 | 0.87204 | −2.533% | 46 MB/s |
-| 11 | 0.87204 | −2.533% | 45 MB/s |
-| 12 | 0.87202 | −2.535% | 42 MB/s |
-
-Two things fall out of that table.
-
-**Level 3 already gets 97% of what is available on geometry.** The whole ladder from 3 to 12 is
-0.078 percentage points — 41 MiB on the 51 GiB Zorah build. Meshlet pages are quantised, densely
-packed bytes; the matches that exist are short and near, and a four-deep chain walk finds them.
-
-**Level 9 is both smaller and slightly faster than levels 3–6** — repeatably, three paired runs,
-always in the same direction. Level 9 is the only hash-chain level that turns on pattern analysis
-(the reference enables it at `nbSearches > 128`), and on data with runs in it that pays for itself.
-There is no reason to ship 3–8 as the default when 9 dominates them.
-
-### It is not the same story for every payload
-
-The codec is generic core code, so the ladder has to be there even though vgeo barely uses it.
-Measured on 64KB chunks, same baseline:
-
-| payload | fast | level 3 | level 9 | level 12 |
-|---|---|---|---|---|
-| engine source + glTF JSON | 0.30153 | 0.23931 (−20.6%) | 0.23164 (−23.2%) | 0.22972 (−23.8%) |
-| runs and periodic data | 0.05763 | 0.05759 (−0.07%) | 0.05731 (−0.57%) | 0.05731 (−0.57%) |
-| real `.vgeo` pages | 0.89470 | −2.46% | −2.53% | −2.54% |
-
-On text the top of the ladder is worth 3.2 points over level 3 and costs 8× the time (113 MB/s at
-level 3, 14 MB/s at level 12). That is the payload class that justifies building levels 10–12 at all.
-
-### What it does to a Zorah build
-
-From the 2026-09-13 conversion log: pages **51.38 GiB stored, 56.38 GiB decoded**, built in
-**4 h 8 m wall / 17 h 9 m CPU** across ten workers.
-
-| level | pages would be | saved | compression CPU | share of build CPU |
-|---|---|---|---|---|
-| fast | 51.38 GiB | — | 2.4 min | 0.2% |
-| 3 | 50.12 GiB | 1.26 GiB | 19.4 min | 1.9% |
-| **9** | **50.08 GiB** | **1.30 GiB** | **18.9 min** | **1.8%** |
-| 12 | 50.08 GiB | 1.30 GiB | 24.0 min | 2.3% |
-
-**The whole ladder is free.** Sixteen extra minutes of CPU on a seventeen-hour build, about two
-minutes of wall clock across ten workers, for 1.3 GiB. The interesting decision is not whether to
-pay for high compression — it is which level, and the answer is 9 because 10–12 cost 15–40% more
-time for 0.007 points.
-
----
-
-## 3 · What it costs the reader
-
-The format does not change, so nothing has to be done. But the *shape* of the blocks changes, and
-that is worth saying out loud because it does not go the way one would guess.
-
-Measured with [lz4_decompress_block.js](lz4_decompress_block.js) over 250 real pages:
-
-| | ratio | sequences | bytes/sequence | decode |
-|---|---|---|---|---|
-| fast | 0.89372 | 208,503 | 61.9 | 1,174 MB/s |
-| level 9 | 0.87087 | 336,176 | 38.4 | 1,059 MB/s |
-| level 12 | 0.87081 | 339,644 | 38.0 | 999 MB/s |
-
-On geometry, HC finds **more** matches, not longer ones — 61% more sequences — and the decoder's
-per-sequence cost is a byte loop, so pages decode **10–15% slower**. On text the same measurement
-inverts: 115,657 sequences down to 81,431, and decode goes from 705 to 828 MB/s (**+18%**).
-
-This lands in a worker ([vgeo_page_worker_serve.js](../../../shade/renderer/geometry/virtual/format/read/vgeo_page_worker_serve.js)),
-off the main thread, and it buys 2.5% fewer bytes to fetch on every page of every session. It is a
-fair trade as it stands. It is also **entirely recoverable**, for an unrelated reason — see §8.
-
----
-
-## 4 · What the reference actually is
-
-`lz4hc.c` at 1.10.0. The level table is identical in 1.9.4 and 1.10.0 for levels 3–12, so targeting
-that range is version-stable:
-
-| level | strategy | chain attempts | sufficient length |
-|---|---|---|---|
-| 3–9 | hash chain, lazy parse | 4, 8, 16, 32, 64, 128, 256 | 16 |
-| 10 | optimal parse | 96 | 64 |
-| 11 | optimal parse | 512 | 128 |
-| 12 | optimal parse, full update | 16,384 | 4,096 |
-
-**Level 2 is deliberately not on that list.** 1.9.4 gives it the hash-chain strategy; 1.10.0 gives it
-`lz4mid`, a different compressor added in that release. "Level 2" does not name one thing across
-reference versions, so this encoder will not have one. Levels 1 (fast) and 3–12 are the ladder.
-
-Four pieces of the reference are load-bearing and none of them can be dropped:
-
-- **The hash chain.** `hashTable` (32,768 × u32) holds the newest position per 4-byte hash;
-  `chainTable` (65,536 × u16) holds, per position, the *delta* back to the previous position with the
-  same hash, capped at 65,535. Positions are inserted lazily, from `nextToUpdate` up to the current
-  position, so nothing is built that is never searched.
-- **Pattern analysis.** When the chain shows consecutive positions (delta 1) and the 4 bytes are a
-  repeat of a single byte, the walk jumps straight to the best position in the run instead of
-  stepping through thousands of links. Without it, levels 9–12 go quadratic on a page of zeros.
-  This is what makes level 9 *faster* than level 6 on real data.
-- **Chain swap.** On a tie, scan the chain deltas inside the match for the largest step and follow
-  that instead. Only the optimal parser uses it.
-- **The three-match lazy parse** (levels 3–9) and the **windowed shortest path** (10–12). The price
-  model is exact in bytes — a token, a 2-byte offset, and the continuation bytes each length field
-  spills into — which is why the parse is a genuine shortest path and not a heuristic.
-
-Everything else in `lz4hc.c` is dictionary and streaming support: `extDict`, `dictCtx`,
-`LZ4HC_protectDictEnd`, the pattern rotation across a dictionary boundary, `favorDecSpeed`,
-`fillOutput`. meep compresses one self-contained block with no history. **All of it drops**, and with
-it about half the file. The port that was validated is ~700 lines against the reference's ~1,400.
-
----
-
-## 5 · Design
-
-### 5.1 Files
-
-Four new files in this directory. **[lz4_compress_block.js](lz4_compress_block.js) is not edited.**
-
-- **`LZ4_LEVEL.js`** — `{ FAST: 1, HC_MIN: 3, HC_DEFAULT: 9, HC_MAX: 12 }`, with the note about 2.
-- **`lz4_compress_block_hc.js`** — the high-compression encoder. Same parameter list as the fast one
-  with `level` appended:
-  `lz4_compress_block_hc(destination, destination_offset, destination_end, source, source_offset, source_end, level)`.
-- **`lz4_compress_block_at_level.js`** — dispatch. `level <= 1` goes to `lz4_compress_block`,
-  anything else to `lz4_compress_block_hc`. This is what callers with a knob call; it is the only
-  new thing `vgeo_encode_frame_blob` needs to know about.
-- **`lz4_compress_block_hc.spec.js`** — §6.
-
-### 5.2 Why the fast encoder is not touched
-
-The obvious tidy — lift `write_token_and_literals` and `write_continuation` out of
-`lz4_compress_block.js` into a module both encoders import — was built and measured. Across five
-paired runs it cost the fast encoder between 0% and 8% of its throughput and never once helped.
-The HC encoder sequences its writes differently anyway (`LZ4HC_encodeSequence` emits token, literals,
-offset and match continuation as one unit, where the fast encoder splits it), so the two are not the
-same function. **The HC module carries its own writers.** The fast encoder's pinned bytes and its
-435 MB/s cannot regress if nothing edits it.
-
-### 5.3 Memory, and when it is taken
-
-| table | type | bytes | needed from |
-|---|---|---|---|
-| hash | `Int32Array(32768)` | 128 KB | level 3 |
-| chain | `Uint16Array(65536)` | 128 KB | level 3 |
-| price / offset / length / literal-length | 4 × `Int32Array(4099)` | 64 KB | level 10 |
-
-320 KB, against the 32 KB the fast encoder holds today. meep ships raw source, so a module-level
-`new Int32Array(1 << 15)` would cost that to every consumer whether or not they ever compress —
-including the runtime, which only ever *decodes*. **Allocate lazily**: the chain tables on the first
-HC call, the parser tables on the first level-10-or-higher call. Module-static after that, like the
-fast encoder's, and safe for the same reason — the encoder is straight-line synchronous code that
-resets its tables on entry.
-
-### 5.4 Table lifetime
-
-`hash_table.fill(0)` and `chain_table.fill(0xFFFF)` on entry, per call. This looked like the obvious
-thing to optimise — 256 KB of fill for a page that may be 1.4 KB — so it was measured.
-
-*Advancing* that bias from call to call so the tables never need clearing at all — which is what the
-reference does across streamed blocks, and which was verified byte-exact over 6,600 fuzz blocks —
-came out at **0.95–1.01×: no gain**, because turning `IDX_BASE` into a mutable module binding costs
-the inner loop more than the fill saves. Dropping only the chain-table clear (which is provably unnecessary here: every position's
-delta is written by the insert before anything can walk to it) also measured as noise.
-
-**Ship the plain per-call fill.** It matches the fast encoder's precedent and it is not the cost.
-
-### 5.5 Two invariants the port must keep
-
-- **Self-contained blocks.** No match may reach before `source_offset`. With an index space biased by
-  one window, an empty hash slot reads as "before the block" for free and the existing
-  "a block compressed after another does not carry the previous one's positions" test passes without
-  a special case.
-- **The output bound is checked, not assumed.** The reference has `limitedOutput`; this codebase
-  throws, with message shapes the existing spec asserts (`/needs \d+ bytes, and only \d+ are left/`).
-  The HC writers must do the same checks at the same two points — before the token-and-literals run,
-  and before the offset-and-match-length. The prototype has neither yet; this is the one piece of the
-  port that is not already written.
-
----
-
-## 6 · Tests
-
-Following the idiom already in [lz4_compress_block.spec.js](lz4_compress_block.spec.js): pinned
-reference blocks first, then shapes too large to write down driven through and back.
-
-**Pinned vectors.** These are what liblz4's own `LZ4_compress_HC` writes, and the prototype
-reproduces them exactly:
-
-```
-// the greedy pass leaves bytes on the table
-//   "acddafadcacddaacddafadcacddaacdadcacbddafadcacddaacgdccaafadcacddaacgdadcacdfbeagdda"  (84 bytes)
-//   fast=47  hc3=42  hc9=42  hc12=42
-//   hc3/hc9/hc12: 91 61 63 64 64 61 66 61 64 63 09 00 0d 0e 00 01 0b 00 1a 62 15 00 5a 67 64
-//                 63 63 61 11 00 02 0c 00 80 66 62 65 61 67 64 64 61
-
-// the optimal parser earns its own row
-//   "ffafdfdefafdfdeeahadefehahadefeeahadefehfd"  (42 bytes)
-//   fast=35  hc3=35  hc9=35  hc12=32
-//   hc3/hc9: 83 66 66 61 66 64 66 64 65 07 00 93 65 61 68 61 64 65 66 65 68 08 00 b0 65 61 68
-//            61 64 65 66 65 68 66 64
-//   hc12   : 83 66 66 61 66 64 66 64 65 07 00 92 65 61 68 61 64 65 66 65 68 08 00 03 10 00 50
-//            66 65 68 66 64
-```
-
-The first pins that HC finds matches the greedy pass walks past; the second pins that levels 10–12
-are a different parser and not just a longer search.
-
-**Properties, over the existing `CORPUS` at every level:**
-
-1. Round trip through `lz4_decompress_block` — bytes back exactly.
-2. The encoder restrictions, using the spec's existing `read_sequences`: last five bytes literal, no
-   match starting within twelve bytes of the end, no offset reaching before the block start, the
-   block closes with a literals-only sequence.
-3. **Aggregate** monotonicity: summed over the corpus, level *n+1* is never larger than level *n*.
-   **Not per block** — 54 non-monotone steps were found across 275 inputs, concentrated at the 9→10
-   boundary where the parser changes, the worst being a 64KB JSON chunk going 16,741 → 16,993 bytes.
-   Asserting per-block monotonicity would be asserting something false.
-4. A destination one byte short of the block is rejected, at every level, with the same message
-   shapes the fast encoder produces.
-5. `source_offset > 0`: the same payload compressed at offsets 0, 7, 32, 33 and 4096 inside buffers
-   filled with different padding bytes produces **identical blocks**. This is the real guard on the
-   one latent hazard in the port (§7), and it holds today for both encoders — 72 pages × 5 paddings.
-6. Compressing a second block after a first does not carry the first's positions (the existing test,
-   extended across levels).
-7. The level argument is validated: 2 and anything outside 1…12 is refused rather than silently
-   clamped to something the caller did not ask for.
-
----
-
-## 7 · What the port has to be careful about
-
-Found while building and validating the prototype. None of these are speculative.
-
-**The cheap rejection gate can read outside the block.** Before a full compare, the reference tests
-the two bytes at the current best match end: `read16(iLowLimit + longest - 1)` against
-`read16(matchPtr - lookBackLength + longest - 1)`. When the search is allowed to start behind the
-current position — which is exactly what the lazy parser's second and third searches do — the second
-address can fall before `source_offset`, and at `source_offset === 0` before the array. The C reads
-whatever is in the buffer; JS reads `undefined`, which `| 0` turns into a zero. The gate is only a
-heuristic, so either behaviour produces a *valid* block, but they can produce *different* blocks.
-Clamp it explicitly and pin it with test 5 above. Over 72 real pages × 5 paddings, and 1,200 fuzz
-inputs, no divergence was reachable — it is latent, not live, and it should stay that way by
-construction rather than by luck.
-
-**The level table, not the levels.** Levels 3–12 map onto `{ attempts, sufficient_length }` pairs and
-a strategy; the strategy boundary at 10 and the pattern-analysis boundary at `attempts > 128` are
-where behaviour changes, not the level numbers. Write the table, not a switch.
-
-**`LZ4_MAX_INPUT_SIZE` still applies.** Indices are biased by one window and held in an `Int32Array`,
-which is fine to 2^31, and the existing assert already caps input well below that.
-
----
-
-## 8 · Adjacent, and out of scope
-
-**The decoder's byte loops.** [lz4_decompress_block.js](lz4_decompress_block.js) copies literals and
-matches one byte at a time. Splicing in `destination.set` for literal runs of 16 or more and
-`copyWithin` for non-overlapping matches of 16 or more — nothing else changed, output verified
-identical — measured on 250 real pages:
-
-| | shipped decoder | with bulk copies |
-|---|---|---|
-| fast blocks | 1,206 MB/s | **2,034 MB/s** |
-| level 9 blocks | 1,036 MB/s | **1,361 MB/s** |
-
-HC blocks under a bulk-copy decoder (1,361 MB/s) decode **faster than fast blocks do today**
-(1,206 MB/s). The decode cost in §3 is a property of the decoder's inner loop, not of high
-compression. This is a separate change to a file this plan deliberately does not touch, and it is
-worth making on its own merits.
-
-**Done, 2026-09-14**, in the commit that added this paragraph. The threshold went in at
-thirty-two rather than sixteen: sweeping both crossovers over 2,267 real pages put the top of the
-curve between twenty-four and thirty-two, and thirty-two is where
-[lz4_compress_block.js](lz4_compress_block.js) had already put its own. Measured the way §10 says
-to, twenty-one interleaved rounds alternating which decoder went first:
-
-| | shipped decoder | with bulk copies |
-|---|---|---|
-| fast blocks | 1,205 MB/s | **2,237 MB/s** |
-| level 9 blocks | 1,109 MB/s | **1,592 MB/s** |
-
-The claim survives with room to spare: level 9 blocks now decode 32% *faster* than fast blocks
-managed before the change, and 44% faster than they manage today. What does **not** close is the
-gap between the two shapes — bulk copies help fast blocks more, because their literal runs are
-58.8 bytes against level 9's 34.7, so level 9 goes from 8% behind level 1 to 29% behind it. The
-reader ends up far ahead in absolute terms and §3's trade is paid for several times over; the
-*ratio* §3 quoted gets wider, not narrower, and a later reader should not expect otherwise.
-
-Output is pinned against the old decoder over **57,938 stored pages of 150 Zorah containers,
-3,489,392,680 decoded bytes, not one byte different**, and over the same 2,267 pages re-encoded at
-both levels. The overlapping match stays a byte loop and has its own vectors in the spec.
-
-What the sweep also showed is that the win is the **literal** path, which §3's text figures
-predict but do not say outright. Four fifths of a decoded page is literal runs averaging 34.7
-bytes at level 9, and its matches average 8.7 — so `set` moves 68% of the output and `copyWithin`
-4%. On engine source, where literal runs average 1.1 bytes, the change is a wash at any threshold.
-
-**python-lz4 is not a usable reference for the fast encoder.** The 4.4.5 wheel (liblz4 1.9.4) fails
-to find matches whose candidate was recorded during the search loop: `"QabcdXXabcdZZZZZZZZ"` codes
-as all literals, while the same input with the repeat at offset 0 codes the match. Its HC path is
-correct. Consequence: **[lz4_compress_block.js](lz4_compress_block.js) is right and that build is
-wrong** — meep's blocks are 1.43% *smaller* than its `mode='default'` output over the corpus, and a
-line-by-line transliteration of `LZ4_compress_generic`'s `byU16` path agrees with meep on all 152
-inputs tried. Validate HC against python-lz4; validate the fast encoder against the lz4 CLI or a
-transliteration.
-
----
-
-## 9 · The plan
-
-Each stage is independently shippable and leaves the tree green.
-
-### Stage A — the hash chain, levels 3–9
-
-**Files:** `LZ4_LEVEL.js`, `lz4_compress_block_hc.js`, `lz4_compress_block_at_level.js`,
-`lz4_compress_block_hc.spec.js`
-
-Port, from `lz4hc.c`, with no dictionary and no streaming: the lazy insert, the match finder with
-pattern analysis, `LZ4HC_countBack`, and the three-match lazy parser. Add the output-bound checks
-(§5.5) and the clamped gate (§7). Levels 10–12 assert as not yet available.
-
-**Proves:** −2.53% on vgeo pages, −23.2% on text, at 52 MB/s. That is 97% of everything this plan can
-deliver.
-
-**Test:** all of §6 except the level-12 vector and the 9→10 aggregate step.
-
-### Stage B — the optimal parser, levels 10–12
-
-**Files:** `lz4_compress_block_hc.js` (the parser and its lazily-allocated tables), spec additions.
-
-Port `LZ4HC_compress_optimal` with `favorDecSpeed` fixed off: the exact byte price model, the
-4,096-position window, `sufficient_len`, and full update at level 12. Add chain swap to the finder.
-
-**Proves:** the ladder reaches the documented maximum, and §1's measurement says there is nothing
-past it.
-
-**Test:** the level-12 vector, and aggregate monotonicity across the whole ladder.
-
-### Stage C — the level reaches the build
-
-**Files:**
-[VGeoBuildOptions.js](../../../shade/renderer/geometry/virtual/build/VGeoBuildOptions.js) ·
-[VGeoWriteOptions.js](../../../shade/renderer/geometry/virtual/format/VGeoWriteOptions.js) ·
-[vgeo_encode_frame_blob.js](../../../shade/renderer/geometry/virtual/format/frame/vgeo_encode_frame_blob.js) ·
-[vgeo_write_container.js](../../../shade/renderer/geometry/virtual/format/vgeo_write_container.js) ·
-[gltf_to_vgeo.js](../../../../tools/geometry/virtual/gltf_to_vgeo.js)
-
-A `codec_level` beside the existing `codec` at each layer, defaulting to `LZ4_LEVEL.HC_DEFAULT` (9),
-and a `--codec-level` flag on the tool. `vgeo_encode_frame_blob` already declines compression per
-page when it would not shrink the page, so the level cannot make a file larger — nothing about that
-contract changes.
-
-**Test:** [vgeo_frame_blob.spec.js](../../../shade/renderer/geometry/virtual/format/frame/vgeo_frame_blob.spec.js)
-gains a level case; the container round trip runs at the default.
-
-Nothing in the runtime changes. `codec` on the frame is still `VGEO_CODEC_LZ4`, the reader still
-reads it, and a container built at level 12 is readable by every build of the reader that exists.
-
-### Stage D — confirm on a real conversion
-
-Re-run a Zorah subset at the new default and check the two numbers this plan promised: pages about
-2.5% smaller, build CPU up under 2%. Record it next to the 2026-09-13 log.
-
----
-
-## 10 · How to re-measure
-
-Everything above came from a scratchpad at
-`C:\Users\Alex\AppData\Local\Temp\claude\H--git-CompanyNamed-meep\f41c851a-bc82-447f-be15-80a25e108370\scratchpad`,
-which is session-local and will not survive. The recipes are short:
-
-- **Corpus** — walk a `.vgeo` with `vgeo_read_header` + `vgeo_read_directory`, slice each frame at
-  `frame_offset`/`fetch_size`, and decode the blob past
-  `VGEO_FRAME_HEADER_SIZE + 48 × child_page_count`. Note that `vgeo_read_header` wants more than
-  `VGEO_HEADER_SIZE` bytes — the level table follows the header, so hand it 8 KB.
-- **Ground truth** — `python -c "import lz4.block as B; B.compress(data, mode='high_compression', compression=N, store_size=False)"`.
-  Python has `lz4` installed here (4.4.5, liblz4 1.9.4). For HC only — see §8.
-- **Timing** — interleave A and B in one process and alternate which runs first; nineteen node
-  processes is a normal background load on this machine and a single-shot number means nothing.
-- **Build cost** — `H:\vgeo_scratch\zorah-2026-13-09-console.log` carries the stored/decoded page
-  totals and the CPU time the projections in §2 are scaled against.
-
----
-
-## 11 · What it came to
-
-Carried out 2026-09-14, in four commits: the encoder and its spec, the plumbing through the build,
-one profiling pass, and this. Every number below was measured on this machine after the work, the
-same way §10 says to measure it.
-
-### The port
-
-`lz4hc.c` at 1.10.0, with the dictionary and streaming half dropped as §4 said. One thing dropped
-that §4 did not name: **`LZ4HC_protectDictEnd` goes too.** It guards against a candidate whose four
-bytes would run off the end of a dictionary, and with no dictionary every candidate the loop reaches
-already satisfies it — `matchCandidateIdx >= lowestMatchIndex` implies it, and so does every index
-the pattern-analysis branch assigns. 733 lines of code against the reference's 2,192 of file.
-
-Byte for byte identical to liblz4 1.9.4's `LZ4_compress_HC` over **26,093 blocks**:
-
-| corpus | inputs | levels | blocks |
-|---|---|---|---|
-| synthetic: runs, periods, window edges, structured and small fuzz | 1,182 | 3–12 | 11,820 |
-| the same, capped at 20 KB | 961 | 3–12 | 9,610 |
-| the same, over 20 KB | 221 | 10–12 | 663 |
-| real `.vgeo` pages from the 2026-09-13 Zorah build | 400 | 3–12 | 4,000 |
-
-First try, as §1 said it would be. §6's two pinned vectors reproduce exactly, level-12 row included.
-
-### The ladder, on 5,802 real pages
-
-The pages of a 507-geometry Zorah subset, 190,044,904 bytes decoded — the exact bytes
-`vgeo_encode_frame_blob` was handed:
-
-| level | stored | ratio | vs. level 1 | encoder |
-|---|---|---|---|---|
-| 1 | 166,692,697 | 0.87712 | — | 340–390 MB/s |
-| 3 | 162,743,403 | 0.85634 | −2.369% | 44–53 MB/s |
-| **9** | **162,604,364** | **0.85561** | **−2.453%** | **42–50 MB/s** |
-| 12 | 162,586,298 | 0.85552 | −2.463% | 31–38 MB/s |
-
-The stored byte counts are exact and repeat to the byte. The throughputs are ranges because that is
-what this machine supports: two best-of-nine runs of the same measurement, minutes apart, came out
-20% apart, which is wider than most of the differences anyone would want to read off the column.
-What *is* stable is the shape within a run — level 1 is about eight times level 9, and level 9 is
-about 5% slower than level 3 — because those are ratios taken between interleaved passes.
-
-§2's shape holds exactly: level 3 is 96.2% of everything the ladder has on this payload, and the
-whole run from 3 to 12 is 0.082 percentage points. On a wider sample — the first 25 pages of each of
-120 containers, 2,718 pages and 122,283,932 bytes — the same ladder runs 0.90483 to 0.87733, which
-is **−3.040%**, so 2.5% is the conservative end of what a whole build gets rather than the middle.
-
-Throughput came out below §2's 52 MB/s at first. Profiling put 16.3% of all ticks in the guarded
-gate read of §7, which V8 was not inlining: it is called once per link of every chain walked. Taking
-the guard only on addresses that need it, and reading the call's context into locals once rather
-than reloading module bindings inside that loop, bought 15% in a paired in-process comparison and
-left nothing outside the search loop above 14% of the profile.
-
-### The two numbers Stage D promised
-
-**Pages about 2.5% smaller — confirmed.** The paired conversions, `--max-triangles 30000` on
-`zorah_main_public.v2.gltf`, 507 geometries and 5.25 M triangles, identical in every run:
-
-| | pages stored | decoded |
-|---|---|---|
-| level 1 | 159.0 MiB | 181.2 MiB |
-| level 9 | 155.1 MiB | 181.2 MiB |
-
-**Build CPU up under 2% — confirmed for the Zorah build, but not for this subset.** Six runs,
-alternating which level went first:
-
-| | CPU | wall |
-|---|---|---|
-| level 1 | 139.3 s · 139.7 s · **119.5 s** | 128.8 · 129.4 · **110.8** |
-| level 9 | 143.0 s · 147.6 s · **124.1 s** | 131.6 · 133.8 · **114.9** |
-
-The last pair ran back to back on a quiet machine and is the one to read: **+4.6 s of CPU, +3.85%**.
-Measuring the compression alone over the same build's own pages — five interleaved passes, so no
-noise at all — gives 0.51 s against 4.03 s, a delta of **+3.52 s, or +2.95%** of that build.
-
-That is over 2%, and it is over 2% *because of what the subset is*. `--max-triangles 30000` selects
-the small geometries, and they are cheap: 190 MB of pages out of 119.5 s of build CPU, which is
-1.59 MB of page bytes per CPU-second. The 2026-09-13 build produced 60,536 MB of pages from 61,740 s
-(17 h 9 m, validation excluded), which is **0.98** — it spends 62% more CPU per page byte, because
-it is building the large meshes too. Scaling this subset's measured delta by that gives **about 2%**
-of the Zorah build's CPU.
-
-**About 2% is as fine as this machine resolves it, and the honest answer is that it straddles the
-line.** The three paired conversions give deltas of +3.7 s, +7.9 s and +4.6 s on a build of 190 MB
-of pages, which scale to 1.9%, 4.1% and 2.4% of the Zorah build; the direct compression measurement,
-taken twice on a quiet machine, gives 1.7% and 2.1%. Every one of those is a difference of a few
-seconds measured against a two-minute build on a box whose own run-to-run spread is 20%, so §2's
-1.8% is not *confirmed* so much as *not contradicted*. What can be said without a measurement error
-swallowing it: the level costs a couple of percent of a build and a couple of minutes of wall clock
-across ten workers, and it is about 3% on a build made only of small meshes.
-
-### What it costs the reader, re-measured
-
-§3 said the blocks change shape and the plan was right about that, on 2,500 real pages:
-
-| | ratio | bytes/sequence | decode |
-|---|---|---|---|
-| level 1 | 0.87252 | 55.3 | 1,197 MB/s |
-| level 9 | 0.85228 | 38.4 | 1,080 MB/s |
-| level 12 | 0.85220 | 38.1 | 1,077 MB/s |
-
-38.4 bytes a sequence at level 9, which is §3's figure to the decimal, and decode 9.8% slower rather
-than 10–15%. §8's bulk-copy decoder is still the answer to that, and still out of scope here.
-
-### And it reads
-
-507 Zorah containers built at level 9 pass the tool's whole post-build audit — every build
-invariant, checksum and directory record, in every file — which is the claim that the format does
-not change, made against real geometry rather than argued.

package/src/core/bvh2/bvh3/NOTES.md

@@ -1,2 +0,0 @@
-# References:
-* "Fast and Simple Agglomerative LBVH Construction" by Ciprian Apetrei, 2014

package/src/core/bvh8/NOTES.md

@@ -1,63 +0,0 @@
-References:
-
-* "Efficient Incoherent Ray Traversal on GPUs Through Compressed Wide BVHs" 2017, NVIDIA
-* https://github.com/jan-van-bergen/GPU-Raytracer
-* https://github.com/WeiPhil/nbvh/blob/0a7f8e0e09898ba6e161cd7433325368355e2d2a/include/cuda/accel/bvh8.cuh#L35
-* https://github.com/DGriffin91/obvhs/tree/main/src/cwbvh
-*
-
-Build
-
-* Consider triangles in groups of up-to 32 (bitmask)
-    * Find "lowest" index
-    * Walk up, and mark bitmask
-
-Explore additional metrics such as:
-
-* % of empty nodes
-* % of utilized triangle slots (up to 3 per child)
-
-Runtime
-
-* Debug shader for raycast, visualize:
-    * traversal depth (number of node intersection tests)
-    * number of triangle tests
-    * amount of memory bandwidth (estimate based on size of what we have to load)
-
-# Findings [2024/12/27]:
-
-> [2026-06-11] The numbers below predate the review-01 fixes (see REVIEW_01_ACTION_PLAN.md) —
-> notably the morton sort keyed on box *size* instead of box *center* (F4) and the converter's
-> corrupted SAH cost table (F3), both of which directly degrade build quality and time.
-> Re-measure before drawing conclusions from this section.
-
-Conclusion for now - not worth it. Too slow to build, and runtime performance inadequate.
-
-// RUNGHOLT, ORIGINAL build mechanism
-BVH8 triangles = 24, size = 480 bytes, quality = 88.16 (total=53.92515886274739)
-Geometry build: 27701.5078125 ms
-BVH2 size total: 465,232,160
-BLAS size: 134,670,480 bytes
-
-// RUNGHOLT, NEW (by A.G.)
-BVH8 triangles = 24, size = 240 bytes, quality = 145.27999999999997 (total=594.418961181131)
-Geometry build: 315166.4069824219 ms
-BVH2 size total: 465,232,160
-BLAS size: 302,576,640 bytes
-
-## Findings
-
-* Build times are far too long for runtime.
-* Binary sizes are considerably smaller than BVH2
-* Render times are much worse, in some cases 10x on shadows alone
-* Sorting children during traversal does help, as paper claims, by as much as 10%
-* packing a lot of triangles is tricky, due to restriction of only being able to address 24 contiunuous primitives per
-  node
-* you have to try and pack at least 1 triangle per node, otherwise you're wasting the 4 bytes reserved for base triangle
-  offset
-* original paper requires an extra triangle index array, which is hugely wasteful and creates an extra indirection
-  table, which is bad for perf
-
-# microsoft's DXR fallback implementation
-
-https://github.com/microsoft/DirectX-Graphics-Samples/tree/master/Libraries/D3D12RaytracingFallback