@woosh/meep-engine 3.26.0 → 3.28.0

package/package.json

@@ -6,7 +6,7 @@
   "description": "Advanced 3D game engine",
   "type": "module",
   "author": "Alexander Goldring",
-  "version": "3.26.0",
+  "version": "3.28.0",
   "exports": {
     "./build/*": "./build/*",
     "./src/*": "./src/*",

package/src/core/binary/lz4/LZ4_HIGH_COMPRESSION_PLAN_2026_09_14.md

@@ -0,0 +1,542 @@
+# LZ4_HIGH_COMPRESSION_PLAN — `src/core/binary/lz4`
+
+Date: 2026-09-14 · Adds a compression level to the encoder. The format does not change and
+[lz4_decompress_block.js](lz4_decompress_block.js) is not touched.
+
+Every number below was measured on this machine on 2026-09-14, against real data. The measurement
+recipes are in §9 so any of it can be re-run.
+
+---
+
+## 1 · The question, and the answer
+
+> Add a configurable compression level that gets us to the current state-of-the-art maximum, without
+> changing the format.
+
+**The state of the art for the LZ4 block format is `lz4hc` level 12, and there is nothing
+meaningful past it.** An encoder built with an *uncapped* match search and a shortest-path parse over
+the whole block — strictly more search than level 12 is allowed — was measured against level 12:
+
+| corpus | level 12 | uncapped search + whole-block parse | time |
+|---|---|---|---|
+| 60 real `.vgeo` pages (1.77 MB) | 1,531,612 B | **1,531,612 B** — identical | 1.6× slower |
+| 24 text chunks (1.57 MB) | 356,238 B | 356,233 B — **0.0014% smaller** | 7.9× slower |
+| 18 runs and periodic blocks (74 KB) | 4,225 B | **4,225 B** — identical | 110× slower |
+
+Five bytes in a third of a megabyte, for eight times the work — and on repetitive data the uncapped
+parse is 110× level 12 and 3,500× level 9, because dropping the reference's `sufficient_len` early
+exit is exactly what lets a run of zeros cost quadratic time. The ceiling is level 12; the plan is to
+reach it and stop.
+
+A JS port of `lz4hc` was written and validated before this plan was drafted: **20,976 blocks over
+1,947 inputs, at every level from 2 to 12, byte for byte identical to liblz4**, first try, including
+runs, periodic data, matches at the far edge of the 64KB window, engine source, glTF JSON and real
+geometry pages. The work is a known quantity, not an experiment.
+
+---
+
+## 2 · What it is worth
+
+Corpus: **2,636 decoded pages, 140,268,332 bytes**, taken from 120 of the 2,975 `.vgeo` containers
+the Zorah build wrote on 2026-09-13. These are the exact bytes
+[vgeo_encode_frame_blob.js](../../../shade/renderer/geometry/virtual/format/frame/vgeo_encode_frame_blob.js)
+is handed in production. Baseline is what [lz4_compress_block.js](lz4_compress_block.js) ships today.
+
+| level | ratio | vs. today | JS throughput |
+|---|---|---|---|
+| fast (today) | 0.89470 | — | 412–436 MB/s |
+| 3 | 0.87272 | −2.457% | 51–53 MB/s |
+| 4 | 0.87239 | −2.494% | 51 MB/s |
+| 6 | 0.87214 | −2.522% | 51 MB/s |
+| **9** | **0.87209** | **−2.528%** | **52–55 MB/s** |
+| 10 | 0.87204 | −2.533% | 46 MB/s |
+| 11 | 0.87204 | −2.533% | 45 MB/s |
+| 12 | 0.87202 | −2.535% | 42 MB/s |
+
+Two things fall out of that table.
+
+**Level 3 already gets 97% of what is available on geometry.** The whole ladder from 3 to 12 is
+0.078 percentage points — 41 MiB on the 51 GiB Zorah build. Meshlet pages are quantised, densely
+packed bytes; the matches that exist are short and near, and a four-deep chain walk finds them.
+
+**Level 9 is both smaller and slightly faster than levels 3–6** — repeatably, three paired runs,
+always in the same direction. Level 9 is the only hash-chain level that turns on pattern analysis
+(the reference enables it at `nbSearches > 128`), and on data with runs in it that pays for itself.
+There is no reason to ship 3–8 as the default when 9 dominates them.
+
+### It is not the same story for every payload
+
+The codec is generic core code, so the ladder has to be there even though vgeo barely uses it.
+Measured on 64KB chunks, same baseline:
+
+| payload | fast | level 3 | level 9 | level 12 |
+|---|---|---|---|---|
+| engine source + glTF JSON | 0.30153 | 0.23931 (−20.6%) | 0.23164 (−23.2%) | 0.22972 (−23.8%) |
+| runs and periodic data | 0.05763 | 0.05759 (−0.07%) | 0.05731 (−0.57%) | 0.05731 (−0.57%) |
+| real `.vgeo` pages | 0.89470 | −2.46% | −2.53% | −2.54% |
+
+On text the top of the ladder is worth 3.2 points over level 3 and costs 8× the time (113 MB/s at
+level 3, 14 MB/s at level 12). That is the payload class that justifies building levels 10–12 at all.
+
+### What it does to a Zorah build
+
+From the 2026-09-13 conversion log: pages **51.38 GiB stored, 56.38 GiB decoded**, built in
+**4 h 8 m wall / 17 h 9 m CPU** across ten workers.
+
+| level | pages would be | saved | compression CPU | share of build CPU |
+|---|---|---|---|---|
+| fast | 51.38 GiB | — | 2.4 min | 0.2% |
+| 3 | 50.12 GiB | 1.26 GiB | 19.4 min | 1.9% |
+| **9** | **50.08 GiB** | **1.30 GiB** | **18.9 min** | **1.8%** |
+| 12 | 50.08 GiB | 1.30 GiB | 24.0 min | 2.3% |
+
+**The whole ladder is free.** Sixteen extra minutes of CPU on a seventeen-hour build, about two
+minutes of wall clock across ten workers, for 1.3 GiB. The interesting decision is not whether to
+pay for high compression — it is which level, and the answer is 9 because 10–12 cost 15–40% more
+time for 0.007 points.
+
+---
+
+## 3 · What it costs the reader
+
+The format does not change, so nothing has to be done. But the *shape* of the blocks changes, and
+that is worth saying out loud because it does not go the way one would guess.
+
+Measured with [lz4_decompress_block.js](lz4_decompress_block.js) over 250 real pages:
+
+| | ratio | sequences | bytes/sequence | decode |
+|---|---|---|---|---|
+| fast | 0.89372 | 208,503 | 61.9 | 1,174 MB/s |
+| level 9 | 0.87087 | 336,176 | 38.4 | 1,059 MB/s |
+| level 12 | 0.87081 | 339,644 | 38.0 | 999 MB/s |
+
+On geometry, HC finds **more** matches, not longer ones — 61% more sequences — and the decoder's
+per-sequence cost is a byte loop, so pages decode **10–15% slower**. On text the same measurement
+inverts: 115,657 sequences down to 81,431, and decode goes from 705 to 828 MB/s (**+18%**).
+
+This lands in a worker ([vgeo_page_worker_serve.js](../../../shade/renderer/geometry/virtual/format/read/vgeo_page_worker_serve.js)),
+off the main thread, and it buys 2.5% fewer bytes to fetch on every page of every session. It is a
+fair trade as it stands. It is also **entirely recoverable**, for an unrelated reason — see §8.
+
+---
+
+## 4 · What the reference actually is
+
+`lz4hc.c` at 1.10.0. The level table is identical in 1.9.4 and 1.10.0 for levels 3–12, so targeting
+that range is version-stable:
+
+| level | strategy | chain attempts | sufficient length |
+|---|---|---|---|
+| 3–9 | hash chain, lazy parse | 4, 8, 16, 32, 64, 128, 256 | 16 |
+| 10 | optimal parse | 96 | 64 |
+| 11 | optimal parse | 512 | 128 |
+| 12 | optimal parse, full update | 16,384 | 4,096 |
+
+**Level 2 is deliberately not on that list.** 1.9.4 gives it the hash-chain strategy; 1.10.0 gives it
+`lz4mid`, a different compressor added in that release. "Level 2" does not name one thing across
+reference versions, so this encoder will not have one. Levels 1 (fast) and 3–12 are the ladder.
+
+Four pieces of the reference are load-bearing and none of them can be dropped:
+
+- **The hash chain.** `hashTable` (32,768 × u32) holds the newest position per 4-byte hash;
+  `chainTable` (65,536 × u16) holds, per position, the *delta* back to the previous position with the
+  same hash, capped at 65,535. Positions are inserted lazily, from `nextToUpdate` up to the current
+  position, so nothing is built that is never searched.
+- **Pattern analysis.** When the chain shows consecutive positions (delta 1) and the 4 bytes are a
+  repeat of a single byte, the walk jumps straight to the best position in the run instead of
+  stepping through thousands of links. Without it, levels 9–12 go quadratic on a page of zeros.
+  This is what makes level 9 *faster* than level 6 on real data.
+- **Chain swap.** On a tie, scan the chain deltas inside the match for the largest step and follow
+  that instead. Only the optimal parser uses it.
+- **The three-match lazy parse** (levels 3–9) and the **windowed shortest path** (10–12). The price
+  model is exact in bytes — a token, a 2-byte offset, and the continuation bytes each length field
+  spills into — which is why the parse is a genuine shortest path and not a heuristic.
+
+Everything else in `lz4hc.c` is dictionary and streaming support: `extDict`, `dictCtx`,
+`LZ4HC_protectDictEnd`, the pattern rotation across a dictionary boundary, `favorDecSpeed`,
+`fillOutput`. meep compresses one self-contained block with no history. **All of it drops**, and with
+it about half the file. The port that was validated is ~700 lines against the reference's ~1,400.
+
+---
+
+## 5 · Design
+
+### 5.1 Files
+
+Four new files in this directory. **[lz4_compress_block.js](lz4_compress_block.js) is not edited.**
+
+- **`LZ4_LEVEL.js`** — `{ FAST: 1, HC_MIN: 3, HC_DEFAULT: 9, HC_MAX: 12 }`, with the note about 2.
+- **`lz4_compress_block_hc.js`** — the high-compression encoder. Same parameter list as the fast one
+  with `level` appended:
+  `lz4_compress_block_hc(destination, destination_offset, destination_end, source, source_offset, source_end, level)`.
+- **`lz4_compress_block_at_level.js`** — dispatch. `level <= 1` goes to `lz4_compress_block`,
+  anything else to `lz4_compress_block_hc`. This is what callers with a knob call; it is the only
+  new thing `vgeo_encode_frame_blob` needs to know about.
+- **`lz4_compress_block_hc.spec.js`** — §6.
+
+### 5.2 Why the fast encoder is not touched
+
+The obvious tidy — lift `write_token_and_literals` and `write_continuation` out of
+`lz4_compress_block.js` into a module both encoders import — was built and measured. Across five
+paired runs it cost the fast encoder between 0% and 8% of its throughput and never once helped.
+The HC encoder sequences its writes differently anyway (`LZ4HC_encodeSequence` emits token, literals,
+offset and match continuation as one unit, where the fast encoder splits it), so the two are not the
+same function. **The HC module carries its own writers.** The fast encoder's pinned bytes and its
+435 MB/s cannot regress if nothing edits it.
+
+### 5.3 Memory, and when it is taken
+
+| table | type | bytes | needed from |
+|---|---|---|---|
+| hash | `Int32Array(32768)` | 128 KB | level 3 |
+| chain | `Uint16Array(65536)` | 128 KB | level 3 |
+| price / offset / length / literal-length | 4 × `Int32Array(4099)` | 64 KB | level 10 |
+
+320 KB, against the 32 KB the fast encoder holds today. meep ships raw source, so a module-level
+`new Int32Array(1 << 15)` would cost that to every consumer whether or not they ever compress —
+including the runtime, which only ever *decodes*. **Allocate lazily**: the chain tables on the first
+HC call, the parser tables on the first level-10-or-higher call. Module-static after that, like the
+fast encoder's, and safe for the same reason — the encoder is straight-line synchronous code that
+resets its tables on entry.
+
+### 5.4 Table lifetime
+
+`hash_table.fill(0)` and `chain_table.fill(0xFFFF)` on entry, per call. This looked like the obvious
+thing to optimise — 256 KB of fill for a page that may be 1.4 KB — so it was measured.
+
+*Advancing* that bias from call to call so the tables never need clearing at all — which is what the
+reference does across streamed blocks, and which was verified byte-exact over 6,600 fuzz blocks —
+came out at **0.95–1.01×: no gain**, because turning `IDX_BASE` into a mutable module binding costs
+the inner loop more than the fill saves. Dropping only the chain-table clear (which is provably unnecessary here: every position's
+delta is written by the insert before anything can walk to it) also measured as noise.
+
+**Ship the plain per-call fill.** It matches the fast encoder's precedent and it is not the cost.
+
+### 5.5 Two invariants the port must keep
+
+- **Self-contained blocks.** No match may reach before `source_offset`. With an index space biased by
+  one window, an empty hash slot reads as "before the block" for free and the existing
+  "a block compressed after another does not carry the previous one's positions" test passes without
+  a special case.
+- **The output bound is checked, not assumed.** The reference has `limitedOutput`; this codebase
+  throws, with message shapes the existing spec asserts (`/needs \d+ bytes, and only \d+ are left/`).
+  The HC writers must do the same checks at the same two points — before the token-and-literals run,
+  and before the offset-and-match-length. The prototype has neither yet; this is the one piece of the
+  port that is not already written.
+
+---
+
+## 6 · Tests
+
+Following the idiom already in [lz4_compress_block.spec.js](lz4_compress_block.spec.js): pinned
+reference blocks first, then shapes too large to write down driven through and back.
+
+**Pinned vectors.** These are what liblz4's own `LZ4_compress_HC` writes, and the prototype
+reproduces them exactly:
+
+```
+// the greedy pass leaves bytes on the table
+//   "acddafadcacddaacddafadcacddaacdadcacbddafadcacddaacgdccaafadcacddaacgdadcacdfbeagdda"  (84 bytes)
+//   fast=47  hc3=42  hc9=42  hc12=42
+//   hc3/hc9/hc12: 91 61 63 64 64 61 66 61 64 63 09 00 0d 0e 00 01 0b 00 1a 62 15 00 5a 67 64
+//                 63 63 61 11 00 02 0c 00 80 66 62 65 61 67 64 64 61
+
+// the optimal parser earns its own row
+//   "ffafdfdefafdfdeeahadefehahadefeeahadefehfd"  (42 bytes)
+//   fast=35  hc3=35  hc9=35  hc12=32
+//   hc3/hc9: 83 66 66 61 66 64 66 64 65 07 00 93 65 61 68 61 64 65 66 65 68 08 00 b0 65 61 68
+//            61 64 65 66 65 68 66 64
+//   hc12   : 83 66 66 61 66 64 66 64 65 07 00 92 65 61 68 61 64 65 66 65 68 08 00 03 10 00 50
+//            66 65 68 66 64
+```
+
+The first pins that HC finds matches the greedy pass walks past; the second pins that levels 10–12
+are a different parser and not just a longer search.
+
+**Properties, over the existing `CORPUS` at every level:**
+
+1. Round trip through `lz4_decompress_block` — bytes back exactly.
+2. The encoder restrictions, using the spec's existing `read_sequences`: last five bytes literal, no
+   match starting within twelve bytes of the end, no offset reaching before the block start, the
+   block closes with a literals-only sequence.
+3. **Aggregate** monotonicity: summed over the corpus, level *n+1* is never larger than level *n*.
+   **Not per block** — 54 non-monotone steps were found across 275 inputs, concentrated at the 9→10
+   boundary where the parser changes, the worst being a 64KB JSON chunk going 16,741 → 16,993 bytes.
+   Asserting per-block monotonicity would be asserting something false.
+4. A destination one byte short of the block is rejected, at every level, with the same message
+   shapes the fast encoder produces.
+5. `source_offset > 0`: the same payload compressed at offsets 0, 7, 32, 33 and 4096 inside buffers
+   filled with different padding bytes produces **identical blocks**. This is the real guard on the
+   one latent hazard in the port (§7), and it holds today for both encoders — 72 pages × 5 paddings.
+6. Compressing a second block after a first does not carry the first's positions (the existing test,
+   extended across levels).
+7. The level argument is validated: 2 and anything outside 1…12 is refused rather than silently
+   clamped to something the caller did not ask for.
+
+---
+
+## 7 · What the port has to be careful about
+
+Found while building and validating the prototype. None of these are speculative.
+
+**The cheap rejection gate can read outside the block.** Before a full compare, the reference tests
+the two bytes at the current best match end: `read16(iLowLimit + longest - 1)` against
+`read16(matchPtr - lookBackLength + longest - 1)`. When the search is allowed to start behind the
+current position — which is exactly what the lazy parser's second and third searches do — the second
+address can fall before `source_offset`, and at `source_offset === 0` before the array. The C reads
+whatever is in the buffer; JS reads `undefined`, which `| 0` turns into a zero. The gate is only a
+heuristic, so either behaviour produces a *valid* block, but they can produce *different* blocks.
+Clamp it explicitly and pin it with test 5 above. Over 72 real pages × 5 paddings, and 1,200 fuzz
+inputs, no divergence was reachable — it is latent, not live, and it should stay that way by
+construction rather than by luck.
+
+**The level table, not the levels.** Levels 3–12 map onto `{ attempts, sufficient_length }` pairs and
+a strategy; the strategy boundary at 10 and the pattern-analysis boundary at `attempts > 128` are
+where behaviour changes, not the level numbers. Write the table, not a switch.
+
+**`LZ4_MAX_INPUT_SIZE` still applies.** Indices are biased by one window and held in an `Int32Array`,
+which is fine to 2^31, and the existing assert already caps input well below that.
+
+---
+
+## 8 · Adjacent, and out of scope
+
+**The decoder's byte loops.** [lz4_decompress_block.js](lz4_decompress_block.js) copies literals and
+matches one byte at a time. Splicing in `destination.set` for literal runs of 16 or more and
+`copyWithin` for non-overlapping matches of 16 or more — nothing else changed, output verified
+identical — measured on 250 real pages:
+
+| | shipped decoder | with bulk copies |
+|---|---|---|
+| fast blocks | 1,206 MB/s | **2,034 MB/s** |
+| level 9 blocks | 1,036 MB/s | **1,361 MB/s** |
+
+HC blocks under a bulk-copy decoder (1,361 MB/s) decode **faster than fast blocks do today**
+(1,206 MB/s). The decode cost in §3 is a property of the decoder's inner loop, not of high
+compression. This is a separate change to a file this plan deliberately does not touch, and it is
+worth making on its own merits.
+
+**Done, 2026-09-14**, in the commit that added this paragraph. The threshold went in at
+thirty-two rather than sixteen: sweeping both crossovers over 2,267 real pages put the top of the
+curve between twenty-four and thirty-two, and thirty-two is where
+[lz4_compress_block.js](lz4_compress_block.js) had already put its own. Measured the way §10 says
+to, twenty-one interleaved rounds alternating which decoder went first:
+
+| | shipped decoder | with bulk copies |
+|---|---|---|
+| fast blocks | 1,205 MB/s | **2,237 MB/s** |
+| level 9 blocks | 1,109 MB/s | **1,592 MB/s** |
+
+The claim survives with room to spare: level 9 blocks now decode 32% *faster* than fast blocks
+managed before the change, and 44% faster than they manage today. What does **not** close is the
+gap between the two shapes — bulk copies help fast blocks more, because their literal runs are
+58.8 bytes against level 9's 34.7, so level 9 goes from 8% behind level 1 to 29% behind it. The
+reader ends up far ahead in absolute terms and §3's trade is paid for several times over; the
+*ratio* §3 quoted gets wider, not narrower, and a later reader should not expect otherwise.
+
+Output is pinned against the old decoder over **57,938 stored pages of 150 Zorah containers,
+3,489,392,680 decoded bytes, not one byte different**, and over the same 2,267 pages re-encoded at
+both levels. The overlapping match stays a byte loop and has its own vectors in the spec.
+
+What the sweep also showed is that the win is the **literal** path, which §3's text figures
+predict but do not say outright. Four fifths of a decoded page is literal runs averaging 34.7
+bytes at level 9, and its matches average 8.7 — so `set` moves 68% of the output and `copyWithin`
+4%. On engine source, where literal runs average 1.1 bytes, the change is a wash at any threshold.
+
+**python-lz4 is not a usable reference for the fast encoder.** The 4.4.5 wheel (liblz4 1.9.4) fails
+to find matches whose candidate was recorded during the search loop: `"QabcdXXabcdZZZZZZZZ"` codes
+as all literals, while the same input with the repeat at offset 0 codes the match. Its HC path is
+correct. Consequence: **[lz4_compress_block.js](lz4_compress_block.js) is right and that build is
+wrong** — meep's blocks are 1.43% *smaller* than its `mode='default'` output over the corpus, and a
+line-by-line transliteration of `LZ4_compress_generic`'s `byU16` path agrees with meep on all 152
+inputs tried. Validate HC against python-lz4; validate the fast encoder against the lz4 CLI or a
+transliteration.
+
+---
+
+## 9 · The plan
+
+Each stage is independently shippable and leaves the tree green.
+
+### Stage A — the hash chain, levels 3–9
+
+**Files:** `LZ4_LEVEL.js`, `lz4_compress_block_hc.js`, `lz4_compress_block_at_level.js`,
+`lz4_compress_block_hc.spec.js`
+
+Port, from `lz4hc.c`, with no dictionary and no streaming: the lazy insert, the match finder with
+pattern analysis, `LZ4HC_countBack`, and the three-match lazy parser. Add the output-bound checks
+(§5.5) and the clamped gate (§7). Levels 10–12 assert as not yet available.
+
+**Proves:** −2.53% on vgeo pages, −23.2% on text, at 52 MB/s. That is 97% of everything this plan can
+deliver.
+
+**Test:** all of §6 except the level-12 vector and the 9→10 aggregate step.
+
+### Stage B — the optimal parser, levels 10–12
+
+**Files:** `lz4_compress_block_hc.js` (the parser and its lazily-allocated tables), spec additions.
+
+Port `LZ4HC_compress_optimal` with `favorDecSpeed` fixed off: the exact byte price model, the
+4,096-position window, `sufficient_len`, and full update at level 12. Add chain swap to the finder.
+
+**Proves:** the ladder reaches the documented maximum, and §1's measurement says there is nothing
+past it.
+
+**Test:** the level-12 vector, and aggregate monotonicity across the whole ladder.
+
+### Stage C — the level reaches the build
+
+**Files:**
+[VGeoBuildOptions.js](../../../shade/renderer/geometry/virtual/build/VGeoBuildOptions.js) ·
+[VGeoWriteOptions.js](../../../shade/renderer/geometry/virtual/format/VGeoWriteOptions.js) ·
+[vgeo_encode_frame_blob.js](../../../shade/renderer/geometry/virtual/format/frame/vgeo_encode_frame_blob.js) ·
+[vgeo_write_container.js](../../../shade/renderer/geometry/virtual/format/vgeo_write_container.js) ·
+[gltf_to_vgeo.js](../../../../tools/geometry/virtual/gltf_to_vgeo.js)
+
+A `codec_level` beside the existing `codec` at each layer, defaulting to `LZ4_LEVEL.HC_DEFAULT` (9),
+and a `--codec-level` flag on the tool. `vgeo_encode_frame_blob` already declines compression per
+page when it would not shrink the page, so the level cannot make a file larger — nothing about that
+contract changes.
+
+**Test:** [vgeo_frame_blob.spec.js](../../../shade/renderer/geometry/virtual/format/frame/vgeo_frame_blob.spec.js)
+gains a level case; the container round trip runs at the default.
+
+Nothing in the runtime changes. `codec` on the frame is still `VGEO_CODEC_LZ4`, the reader still
+reads it, and a container built at level 12 is readable by every build of the reader that exists.
+
+### Stage D — confirm on a real conversion
+
+Re-run a Zorah subset at the new default and check the two numbers this plan promised: pages about
+2.5% smaller, build CPU up under 2%. Record it next to the 2026-09-13 log.
+
+---
+
+## 10 · How to re-measure
+
+Everything above came from a scratchpad at
+`C:\Users\Alex\AppData\Local\Temp\claude\H--git-CompanyNamed-meep\f41c851a-bc82-447f-be15-80a25e108370\scratchpad`,
+which is session-local and will not survive. The recipes are short:
+
+- **Corpus** — walk a `.vgeo` with `vgeo_read_header` + `vgeo_read_directory`, slice each frame at
+  `frame_offset`/`fetch_size`, and decode the blob past
+  `VGEO_FRAME_HEADER_SIZE + 48 × child_page_count`. Note that `vgeo_read_header` wants more than
+  `VGEO_HEADER_SIZE` bytes — the level table follows the header, so hand it 8 KB.
+- **Ground truth** — `python -c "import lz4.block as B; B.compress(data, mode='high_compression', compression=N, store_size=False)"`.
+  Python has `lz4` installed here (4.4.5, liblz4 1.9.4). For HC only — see §8.
+- **Timing** — interleave A and B in one process and alternate which runs first; nineteen node
+  processes is a normal background load on this machine and a single-shot number means nothing.
+- **Build cost** — `H:\vgeo_scratch\zorah-2026-13-09-console.log` carries the stored/decoded page
+  totals and the CPU time the projections in §2 are scaled against.
+
+---
+
+## 11 · What it came to
+
+Carried out 2026-09-14, in four commits: the encoder and its spec, the plumbing through the build,
+one profiling pass, and this. Every number below was measured on this machine after the work, the
+same way §10 says to measure it.
+
+### The port
+
+`lz4hc.c` at 1.10.0, with the dictionary and streaming half dropped as §4 said. One thing dropped
+that §4 did not name: **`LZ4HC_protectDictEnd` goes too.** It guards against a candidate whose four
+bytes would run off the end of a dictionary, and with no dictionary every candidate the loop reaches
+already satisfies it — `matchCandidateIdx >= lowestMatchIndex` implies it, and so does every index
+the pattern-analysis branch assigns. 733 lines of code against the reference's 2,192 of file.
+
+Byte for byte identical to liblz4 1.9.4's `LZ4_compress_HC` over **26,093 blocks**:
+
+| corpus | inputs | levels | blocks |
+|---|---|---|---|
+| synthetic: runs, periods, window edges, structured and small fuzz | 1,182 | 3–12 | 11,820 |
+| the same, capped at 20 KB | 961 | 3–12 | 9,610 |
+| the same, over 20 KB | 221 | 10–12 | 663 |
+| real `.vgeo` pages from the 2026-09-13 Zorah build | 400 | 3–12 | 4,000 |
+
+First try, as §1 said it would be. §6's two pinned vectors reproduce exactly, level-12 row included.
+
+### The ladder, on 5,802 real pages
+
+The pages of a 507-geometry Zorah subset, 190,044,904 bytes decoded — the exact bytes
+`vgeo_encode_frame_blob` was handed:
+
+| level | stored | ratio | vs. level 1 | encoder |
+|---|---|---|---|---|
+| 1 | 166,692,697 | 0.87712 | — | 340–390 MB/s |
+| 3 | 162,743,403 | 0.85634 | −2.369% | 44–53 MB/s |
+| **9** | **162,604,364** | **0.85561** | **−2.453%** | **42–50 MB/s** |
+| 12 | 162,586,298 | 0.85552 | −2.463% | 31–38 MB/s |
+
+The stored byte counts are exact and repeat to the byte. The throughputs are ranges because that is
+what this machine supports: two best-of-nine runs of the same measurement, minutes apart, came out
+20% apart, which is wider than most of the differences anyone would want to read off the column.
+What *is* stable is the shape within a run — level 1 is about eight times level 9, and level 9 is
+about 5% slower than level 3 — because those are ratios taken between interleaved passes.
+
+§2's shape holds exactly: level 3 is 96.2% of everything the ladder has on this payload, and the
+whole run from 3 to 12 is 0.082 percentage points. On a wider sample — the first 25 pages of each of
+120 containers, 2,718 pages and 122,283,932 bytes — the same ladder runs 0.90483 to 0.87733, which
+is **−3.040%**, so 2.5% is the conservative end of what a whole build gets rather than the middle.
+
+Throughput came out below §2's 52 MB/s at first. Profiling put 16.3% of all ticks in the guarded
+gate read of §7, which V8 was not inlining: it is called once per link of every chain walked. Taking
+the guard only on addresses that need it, and reading the call's context into locals once rather
+than reloading module bindings inside that loop, bought 15% in a paired in-process comparison and
+left nothing outside the search loop above 14% of the profile.
+
+### The two numbers Stage D promised
+
+**Pages about 2.5% smaller — confirmed.** The paired conversions, `--max-triangles 30000` on
+`zorah_main_public.v2.gltf`, 507 geometries and 5.25 M triangles, identical in every run:
+
+| | pages stored | decoded |
+|---|---|---|
+| level 1 | 159.0 MiB | 181.2 MiB |
+| level 9 | 155.1 MiB | 181.2 MiB |
+
+**Build CPU up under 2% — confirmed for the Zorah build, but not for this subset.** Six runs,
+alternating which level went first:
+
+| | CPU | wall |
+|---|---|---|
+| level 1 | 139.3 s · 139.7 s · **119.5 s** | 128.8 · 129.4 · **110.8** |
+| level 9 | 143.0 s · 147.6 s · **124.1 s** | 131.6 · 133.8 · **114.9** |
+
+The last pair ran back to back on a quiet machine and is the one to read: **+4.6 s of CPU, +3.85%**.
+Measuring the compression alone over the same build's own pages — five interleaved passes, so no
+noise at all — gives 0.51 s against 4.03 s, a delta of **+3.52 s, or +2.95%** of that build.
+
+That is over 2%, and it is over 2% *because of what the subset is*. `--max-triangles 30000` selects
+the small geometries, and they are cheap: 190 MB of pages out of 119.5 s of build CPU, which is
+1.59 MB of page bytes per CPU-second. The 2026-09-13 build produced 60,536 MB of pages from 61,740 s
+(17 h 9 m, validation excluded), which is **0.98** — it spends 62% more CPU per page byte, because
+it is building the large meshes too. Scaling this subset's measured delta by that gives **about 2%**
+of the Zorah build's CPU.
+
+**About 2% is as fine as this machine resolves it, and the honest answer is that it straddles the
+line.** The three paired conversions give deltas of +3.7 s, +7.9 s and +4.6 s on a build of 190 MB
+of pages, which scale to 1.9%, 4.1% and 2.4% of the Zorah build; the direct compression measurement,
+taken twice on a quiet machine, gives 1.7% and 2.1%. Every one of those is a difference of a few
+seconds measured against a two-minute build on a box whose own run-to-run spread is 20%, so §2's
+1.8% is not *confirmed* so much as *not contradicted*. What can be said without a measurement error
+swallowing it: the level costs a couple of percent of a build and a couple of minutes of wall clock
+across ten workers, and it is about 3% on a build made only of small meshes.
+
+### What it costs the reader, re-measured
+
+§3 said the blocks change shape and the plan was right about that, on 2,500 real pages:
+
+| | ratio | bytes/sequence | decode |
+|---|---|---|---|
+| level 1 | 0.87252 | 55.3 | 1,197 MB/s |
+| level 9 | 0.85228 | 38.4 | 1,080 MB/s |
+| level 12 | 0.85220 | 38.1 | 1,077 MB/s |
+
+38.4 bytes a sequence at level 9, which is §3's figure to the decimal, and decode 9.8% slower rather
+than 10–15%. §8's bulk-copy decoder is still the answer to that, and still out of scope here.
+
+### And it reads
+
+507 Zorah containers built at level 9 pass the tool's whole post-build audit — every build
+invariant, checksum and directory record, in every file — which is the claim that the format does
+not change, made against real geometry rather than argued.

package/src/core/binary/lz4/LZ4_LEVEL.d.ts

@@ -0,0 +1,33 @@
+/**
+ * The compression levels {@link lz4_compress_block_at_level} accepts.
+ *
+ * Level 1 is the format's fast encoder, {@link lz4_compress_block}. Levels 3 to 12 are the
+ * high-compression encoder, {@link lz4_compress_block_hc}: 3 to 9 walk a hash chain with a lazy
+ * parse of ever more candidates, and 10 to 12 replace that parse with a shortest path over a window
+ * of the block. Twelve is the top of the ladder in the reference and, measured against an encoder
+ * with an uncapped search and a whole-block parse, the top of what the block format allows at all.
+ *
+ * **There is no level 2.** The reference gives the number different algorithms in different
+ * releases — 1.9.4 compresses it on the hash chain, 1.10.0 hands it to `lz4mid`, a compressor that
+ * release added — so "level 2" does not name one thing, and this encoder will not pretend it does.
+ *
+ * Nine is the default because the ladder above it is not worth its time: on real geometry pages the
+ * whole run from 3 to 12 is under a tenth of a percentage point of ratio, and 10 to 12 cost 15-40%
+ * more time for the last thousandth of it. Nine is the smallest of the hash-chain levels and costs
+ * about 5% more time than 3 — the reference's own claim that 9 comes out *faster* than the levels
+ * below it is a property of its eight-byte pattern scan, which JavaScript has no cheap way to do,
+ * and it does not reproduce here. The case for 9 over 3 is ratio, and it is a small one.
+ *
+ * @see https://github.com/lz4/lz4/blob/dev/lib/lz4hc.h — `LZ4HC_CLEVEL_MIN` and friends
+ * @type {{FAST: number, HC_MIN: number, HC_DEFAULT: number, HC_MAX: number}}
+ *
+ * @author Alex Goldring
+ * @copyright Company Named Limited (c) 2026
+ */
+export const LZ4_LEVEL: {
+    FAST: number;
+    HC_MIN: number;
+    HC_DEFAULT: number;
+    HC_MAX: number;
+};
+//# sourceMappingURL=LZ4_LEVEL.d.ts.map

package/src/core/binary/lz4/LZ4_LEVEL.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"LZ4_LEVEL.d.ts","sourceRoot":"","sources":["../../../../../src/core/binary/lz4/LZ4_LEVEL.js"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;GAyBG;AACH,wBALU;IAAC,IAAI,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,MAAM,CAAC;IAAC,UAAU,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,MAAM,CAAA;CAAC,CAU1E"}

package/src/core/binary/lz4/LZ4_LEVEL.js

@@ -0,0 +1,32 @@
+/**
+ * The compression levels {@link lz4_compress_block_at_level} accepts.
+ *
+ * Level 1 is the format's fast encoder, {@link lz4_compress_block}. Levels 3 to 12 are the
+ * high-compression encoder, {@link lz4_compress_block_hc}: 3 to 9 walk a hash chain with a lazy
+ * parse of ever more candidates, and 10 to 12 replace that parse with a shortest path over a window
+ * of the block. Twelve is the top of the ladder in the reference and, measured against an encoder
+ * with an uncapped search and a whole-block parse, the top of what the block format allows at all.
+ *
+ * **There is no level 2.** The reference gives the number different algorithms in different
+ * releases — 1.9.4 compresses it on the hash chain, 1.10.0 hands it to `lz4mid`, a compressor that
+ * release added — so "level 2" does not name one thing, and this encoder will not pretend it does.
+ *
+ * Nine is the default because the ladder above it is not worth its time: on real geometry pages the
+ * whole run from 3 to 12 is under a tenth of a percentage point of ratio, and 10 to 12 cost 15-40%
+ * more time for the last thousandth of it. Nine is the smallest of the hash-chain levels and costs
+ * about 5% more time than 3 — the reference's own claim that 9 comes out *faster* than the levels
+ * below it is a property of its eight-byte pattern scan, which JavaScript has no cheap way to do,
+ * and it does not reproduce here. The case for 9 over 3 is ratio, and it is a small one.
+ *
+ * @see https://github.com/lz4/lz4/blob/dev/lib/lz4hc.h — `LZ4HC_CLEVEL_MIN` and friends
+ * @type {{FAST: number, HC_MIN: number, HC_DEFAULT: number, HC_MAX: number}}
+ *
+ * @author Alex Goldring
+ * @copyright Company Named Limited (c) 2026
+ */
+export const LZ4_LEVEL = {
+    FAST: 1,
+    HC_MIN: 3,
+    HC_DEFAULT: 9,
+    HC_MAX: 12
+};

package/src/core/binary/lz4/lz4_compress_block_at_level.d.ts

@@ -0,0 +1,33 @@
+/**
+ * Encode one LZ4 block, at whichever level the caller asks for.
+ *
+ * This is what a caller with a knob calls. The two encoders behind it are not variants of one
+ * another — {@link lz4_compress_block} makes a single greedy pass over a table that holds one
+ * position per hash, and {@link lz4_compress_block_hc} walks a chain of every position carrying the
+ * same hash and parses the matches against each other — so they are two functions, and this is the
+ * one line that chooses between them.
+ *
+ * **Every level writes the same format.** A block is a block: the level changes how long the encoder
+ * spent looking for matches, not what a match is coded as. So the level is a build-time decision
+ * with no reader consequence at all, and one that can be changed on an existing pipeline without
+ * anything downstream knowing.
+ *
+ * `destination` and `source` must not overlap.
+ *
+ * @param {Uint8Array} destination
+ * @param {number} destination_offset where this block's output starts
+ * @param {number} destination_end one past the last byte the block may write; give it
+ *                                 {@link lz4_compress_bound} bytes and it cannot run out
+ * @param {Uint8Array} source
+ * @param {number} source_offset the first byte of input, and the floor a match may reach back to
+ * @param {number} source_end one past the last byte of input
+ * @param {number} level {@link LZ4_LEVEL.FAST}, or one of 3 to {@link LZ4_LEVEL.HC_MAX} — there is
+ *                       no level 2, and {@link LZ4_LEVEL} says why
+ * @returns {number} the offset just past the last byte written
+ * @throws {Error} if the block does not fit the space given for it
+ *
+ * @author Alex Goldring
+ * @copyright Company Named Limited (c) 2026
+ */
+export function lz4_compress_block_at_level(destination: Uint8Array, destination_offset: number, destination_end: number, source: Uint8Array, source_offset: number, source_end: number, level: number): number;
+//# sourceMappingURL=lz4_compress_block_at_level.d.ts.map

package/src/core/binary/lz4/lz4_compress_block_at_level.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"lz4_compress_block_at_level.d.ts","sourceRoot":"","sources":["../../../../../src/core/binary/lz4/lz4_compress_block_at_level.js"],"names":[],"mappings":"AAKA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA8BG;AACH,yDAfW,UAAU,sBACV,MAAM,mBACN,MAAM,UAEN,UAAU,iBACV,MAAM,cACN,MAAM,SACN,MAAM,GAEJ,MAAM,CAgBlB"}