@woosh/meep-engine 3.21.0 → 3.23.0

package/src/shade/renderer/geometry/virtual/VIRTUAL_GEOMETRY_DESIGN.md

@@ -1,10 +1,20 @@
 # Virtual Geometry — Design
 
-Status: **draft, pre-implementation**. This document pins
-only what is certain; everything else is listed explicitly under *Open*. Runtime home is
+Status: **builder, container and reader implemented; the GPU runtime is not**. This document
+pins only what is certain; everything else is listed explicitly under *Open*. Runtime home is
 this directory (`shade/renderer/geometry/virtual/`, mirroring `texture/virtual/`); builder
 lives with `core/geom/3d/topology/struct/binary/` and `core/graph/metis/`.
 
+What exists, in the order a newcomer meets it: `build/` turns a `Geometry` into a cluster DAG,
+`format/` writes and validates the `.vgeo` container ([VGEO_FORMAT.md](./VGEO_FORMAT.md)),
+`format/read/` reads one back a page at a time with residency and activation, and
+`shade/playground/vgeo_viewer/` puts a clamped CPU cut of a *streaming* container on screen.
+What does not exist is everything §7 describes: the selection buffer, the compute cut pass, the
+VRAM page pool, and the seam through which VG instances enter the meshlet pipeline. The
+playground draws through Shade's direct geometry path instead, which is exactly why it proves
+the DAG and not the pipeline. §10 says which milestone that leaves us in, and the order the work
+actually arrived in.
+
 ## 1. What and why
 
 Cluster-DAG continuous LOD (Nanite-family): the builder turns a source mesh into a DAG of
@@ -28,10 +38,14 @@ hands-off principle argues for anyway.
 - **Mesh structure**: `BinaryTopology` (BMesh-style, binary pools, float32 coordinates),
   with `bt_mesh_validate`, `bt_merge_vertices_by_distance` (float32 weld quantum floor;
   its contract requires a short-edge kill/collapse pre-pass — §8), boundary queries.
-- **Simplifier**: `bt_mesh_simplify(mesh, target_face_count, restricted_vertices)` —
-  greedy quadric edge collapse, optimal placement, pinned vertices neither moved nor
-  removed. Required extensions (all builder-side, none exist yet):
-  1. report the max/accumulated quadric residual of a pass (§4);
+- **Simplifier**: `bt_mesh_simplify(mesh, target_face_count, restricted_vertices,
+  protected_faces)` — greedy quadric edge collapse, optimal placement, pinned vertices neither
+  moved nor removed, protected faces never destroyed. Required extensions (all builder-side;
+  three of the four have since landed, and the list is kept because the reasoning is what makes
+  the fourth findable):
+  1. ~~report the max/accumulated quadric residual of a pass (§4)~~ **done**. It returns the
+     object-space distance the surface moved, measured *before* the cleanup pass — see item 2,
+     where that ordering turned out to be most of the fix;
   2. ~~boundary constraint quadrics~~ **done**. Open and non-manifold edges now carry a
      constraint plane through the edge perpendicular to its face, so a border can no
      longer retreat at a recorded cost of zero — which it could, and which the round loop
@@ -43,19 +57,27 @@ hands-off principle argues for anyway.
      object-space distance rather than a raw residual. A flat sheet still reports exactly
      zero, which is the other half of the contract. The same mechanism later serves UV
      seams (§9);
-  3. triangle-inversion (flip) rejection — absent today, and pin-forced placement along
-     locked group perimeters is exactly where greedy QEM folds over;
-  4. group-boundary **polyline** preservation — pinning vertices does not by itself
-     protect the boundary *edges* between them (face-kill paths can strip them); this is
-     a guarantee to enforce and validate, not a property to assume.
+  3. triangle-inversion (flip) rejection — **still open, and the only one of the four that
+     is**. Pin-forced placement along locked group perimeters is exactly where greedy QEM folds
+     over. Nothing in `bt_mesh_simplify` tests the sign of a face normal before or after a
+     collapse, and no build validation catches a fold, so this is a real gap rather than a
+     latent one;
+  4. ~~group-boundary **polyline** preservation~~ **done**. Pinning vertices does not by itself
+     protect the boundary *edges* between them: a collapse elsewhere in the fan can take away
+     the only face holding one, and the cleanup pass then removes it. `protected_faces` names
+     the faces carrying a group's boundary polyline, and `vgeo_build_levels` re-walks every
+     boundary edge after the call and throws if one is gone — a missing boundary edge is a crack
+     between two LODs that never closes, so the build stops rather than shipping it.
 - **Partitioner**: the ported METIS — `metis_partition_kway` (CSR in, deterministic via
   fixed seed, single balance constraint, **unit vertex weights, no contiguity
   guarantee**) and `cluster_mesh_metis(result, mesh, patch_size)` (face clustering with a
   hard size ceiling, minimized edge cut, dense renumbering). Disconnected parts are
   possible and matter (§3).
-- **Bounds**: `Miniball` (minimal bounding sphere, iteration-capped);
-  `compute_triangle_cluster_normal_bounding_cone` is legacy-`TopoMesh`-typed and needs a
-  `BinaryTopology`/index port before §8 can use it.
+- **Bounds**: `Miniball` (minimal bounding sphere, iteration-capped). That is the whole list —
+  **normal cones are not part of this design**. Backface cone culling is declined outright (§9,
+  plan §5), so `cone_reserved` stays zero, and
+  `bt_face_cluster_compute_normal_bounding_cone` — the `BinaryTopology` port of the old
+  `TopoMesh` cone builder — has no consumer and no prospect of one.
 - **Renderer formats (the real seam)**: meshlets capped at 128 triangles / 128 vertices
   (8-bit local indices); `MESHLET_METADATA_STRUCT`; the draw record
   (`MESHLET_DEFINITION_STRUCT` = meshlet index + mesh); bucket pipeline issuing one
@@ -66,13 +88,19 @@ hands-off principle argues for anyway.
   cluster-local position quantization is new work spanning encode, metadata struct, and
   every decode shader, not a finished capability.
 - **Renderer managers (need a VG layer, not reuse as-is)**: `GPUMeshletManager` pools
-  metadata+data via OffsetAllocator, but its compaction/growth is a CPU-side rebuild —
-  new buffer pair (transient ~2× VRAM), CPU-retained metadata mirrors, full re-upload;
-  the GPU intra-copy + address-patch path exists only for clone flushes. Downstream,
-  meshlet-address migration is an acknowledged unhandled TODO in `GPUGeometryManager`.
-  Mesh→meshlet expansion is inlined per material bucket at four call sites and assumes
-  contiguous static per-geometry meshlet ranges — VG's scattered, dynamic residency does
-  not fit those assumptions (§6, §7 state what replaces them).
+  metadata+data via OffsetAllocator. **Growth no longer re-packs**: it appends, files the new
+  tail as free space through `OffsetAllocator#grow`, copies each old buffer into the larger one
+  at offset 0, and moves no record — so no address is patched and no `changed` fires.
+  **Compaction still is a CPU-side rebuild** — new buffer pair (transient ~2× VRAM),
+  CPU-retained metadata mirrors, full re-upload — and it is what closes the holes growth leaves
+  behind, which under VG's churn is the case that matters rather than the one that got cheaper.
+  Downstream, meshlet-address migration is still an acknowledged unhandled TODO in
+  `GPUGeometryManager` (it logs a warning and continues); it is now reachable only through
+  compaction, which narrows the hole without closing it.
+  Mesh→meshlet expansion is one shared FrameGraph helper — `graph_expand_meshes_to_meshlets`,
+  called from four rasterization paths — and it reads a geometry's `meshlets_address` and
+  `meshlets_count` as one contiguous static range. VG's scattered, dynamic residency does not
+  fit that (§6, §7 state what replaces them).
 - **Readback**: `graph_read_buffer` (MAP_READ scratch from the general buffer pool,
   `mapAsync`, ≥1 frame latency). The virtual-texture system is the shipped precedent for
   the whole feedback loop and pins its contract: in-flight readbacks capped, feedback
@@ -84,7 +112,7 @@ Pinned:
 
 - **Leaf clusters = meshlets** (≤ 128 triangles), from `cluster_mesh_metis` on the source
   mesh, followed by a **connected-components split** (metis parts can be disconnected;
-  disconnected clusters/groups inflate spheres and cones and force the exact LOD-lockstep
+  disconnected clusters/groups inflate spheres and force the exact LOD-lockstep
   over-rendering this design criticizes fixed-32 grouping for).
 - **The round loop** (not a level sweep): hold a *frontier* of every cluster nothing has
   merged yet; select a **band** of it by error; partition the band's *cluster adjacency
@@ -238,6 +266,17 @@ Pinned:
   *before* it is fetched. Below page granularity, each page carries a tree over its own groups
   (format §6.1–6.2). Cost, measured: 0.3%–1.6% of asset size.
 
+  *What exists.* `playground/vgeo_viewer/select_cut.js` is this rule on the CPU, against a
+  partially resident container: descend from the root's parentless groups, refine where
+  `project(parent) > τ`, take the cluster where it does not, and stop where the child group is
+  not selectable — recording the want that would unblock it. It is the reference the GPU pass
+  has to agree with, and it is where the clamp was found to need *selectable* rather than
+  *resident* (§6). What it does **not** do is use either accelerator this section just
+  described: it descends group by group, and reads neither the child-table traversal metric nor
+  the page node tree. Both are written by the builder and checked by the validator, and nothing
+  reads either — the largest piece of built-and-unused machinery in the system, and the cut pass
+  is what it was built for.
+
   *Per view.* **Virtual geometry is view-dependent by construction.** A cut is selected at one
   camera's distances and pixel budget; it is the wrong cut for a shadow view, whose light may
   sit a hundred metres behind the camera, and wrong again for each cascade. What is shared
@@ -300,14 +339,18 @@ Pinned:
   in the file (coarse-first order is topological); non-root pages hold clusters of a
   single level band. Build-validated.
 - **Compression must not break random access.** Transparent HTTP compression does not
-  compose with range requests, so compression is per-page *inside* the container. Codec
-  open; `raw` is a valid v1 codec — but note quantization is *new work* (§2), so raw v1
-  pages are position-heavy.
+  compose with range requests, so compression is per-page *inside* the container. Two codecs
+  are defined and both shipped — `raw` and a bare LZ4 block — chosen **per frame** rather than
+  per file, so a page that would not shrink is stored verbatim and says so. Measured, LZ4 is
+  worth 1.12x–1.18x on the Stanford models, and the reason it is not more is the reason to stop
+  here: the payload is float32 positions and packed attribute words, and it does not compress.
+  Quantization (§2) is the lever, not the codec.
 - **Coarse-first layout**: pages ordered so a plain prefix read of the file yields a
   complete coarse asset — progressive load without a feedback loop.
 
-Open: page size; page-internal codec; per-attribute quantization widths (cluster-local
-grids); relationship to `MeshletGeometrySerializationAdapter` (extend vs parallel).
+Open: page size; a denser page codec than LZ4, if quantization leaves one worth having;
+per-attribute quantization widths (cluster-local grids); relationship to
+`MeshletGeometrySerializationAdapter` (extend vs parallel).
 
 ## 6. Streaming and residency
 
@@ -319,8 +362,16 @@ Pinned:
 - **The VRAM pool's hard ceiling is the adapter's `maxStorageBufferBindingSize`**, not
   free configuration — pools bind as single storage buffers (spec default 128 MiB; Shade
   requests adapter max, but low-end adapters stay low). Growth must clamp against device
-  limits with a defined failure policy (today's `grow()` doesn't check); the low-limit
-  fallback is a smaller pool + higher τ. Multi-buffer sharding is explicit future work.
+  limits with a defined failure policy; the low-limit fallback is a smaller pool + higher τ.
+  Multi-buffer sharding is explicit future work.
+  `GPUMeshletManager.grow` clamps now, having not before — but against **`maxBufferSize`**,
+  which is the wrong limit for an arena that binds as one storage buffer. Spec defaults are
+  256 MiB and 128 MiB respectively, so a buffer between the two is creatable and not bindable.
+  `GPUDatabase` clamps against the binding limit; `GPUMeshletManager` and
+  `GPUGeometrySDFManager` clamp against `maxBufferSize`. A VG pool has to pick deliberately,
+  and the reason this is worth stating rather than assuming is that `createBuffer` answers an
+  over-limit size with an *invalid buffer* rather than an exception — the failure is silent at
+  the point it happens and loud several frames later.
 - **The VRAM paging mechanism is new work**, not `GPUMeshletManager` as-is (§2): v1
   intent is **fixed-size page-slot allocation** (no compaction, no CPU metadata mirror,
   no transient 2× rebuild), plus propagation of address migration to every consumer —
@@ -337,11 +388,52 @@ Pinned:
   recheck, never a fetch. With the §4 clamped rule, any miss (or resident-but-inactive
   group) has a coarser stand-in that *is selected* — streaming latency degrades
   quality, never correctness. No holes, no double-draw, in any arrival order.
+  **Implemented**, in `VGeoContainerReader`: a live-reference counter per resident group, a
+  reverse index for pages not yet installed, cascades on both install and evict, and
+  `is_selectable(page, group)`. The double-draw half of that guarantee is not theoretical —
+  clamping on residency instead drew two levels over the same surface on 355 of 576 camera
+  positions of a 19-page bunny, up to 10.7% of drawn triangles in a doubly-covered region, and
+  it reads as corrupt geometry rather than as a missing page, which is what makes it hard to
+  attribute. 21 of that asset's 362 non-root groups have parents in more than one page, which
+  is all it takes.
 - **Feedback carries touches as well as wants.** Wants = pages whose refinement was
   blocked; touches = pages actually selected this frame. Eviction is LRU on
   **last-touched** with a protect window ≥ feedback latency — LRU on last-*wanted* would
   freeze stamps at fetch time and evict exactly what is on screen (the classic
   fallback-chain bug; the VT system already keys on touch for this reason).
+  **Implemented**, in `virtual/residency/`: `VGeoResidencyManager` holds the RAM tier's byte budget
+  across every registered reader, `touch(reader, page)` takes the stamp, `advance_frame()` moves the
+  protect window, and `evict_to_budget()` drops on the format §10 rule ordered by coldest touch.
+  It is a sibling of `format/` and not part of it, because a container cannot answer a question
+  about a scene. The reader stayed mechanics: it reports `resident_bytes` and enforces nothing, and
+  the manager is written entirely against its public surface.
+  The parenthetical above is load-bearing and should be followed rather than taken on trust:
+  `VTPhysicalCache` is the working implementation of this same rule, with a `#last_touch`
+  `Uint32Array` per slot, `touch`/`last_touch`, a lock flag for slots that may never be evicted,
+  and `find_eviction_candidate(current_frame, protect_window)` whose doc gives the window the
+  identical justification. The similarity is therefore convergent *by instruction* — this bullet
+  told the implementation to key on touch because VT does — and the only real difference is that
+  VT budgets a count of fixed-size slots, which is what lets it pick by linear scan at ≤ ~1k
+  slots. Budgeting **bytes** across **several** readers, and admitting a page against that budget
+  before the fetch out of the `decoded_size` its parent already carries, is the part with no
+  precedent in this tree; so are per-page retry-into-quarantine and per-fetch `AbortSignal`.
+- **The fetch cap has to be shared, and per-reader caps do not compose.**
+  `VGeoReadOptions.max_concurrent_fetches` bounds one reader, which is all a reader can see: four
+  readers at a cap of four each were measured issuing sixteen concurrent range requests, eight at
+  the default six issuing forty-eight. `VGeoResidencyManager` holds one total across every reader
+  as one priority queue, so a want's priority — pixels of projected error, comparable across assets
+  by construction — decides globally rather than within whichever reader asked first.
+  A shared cap is not itself a new idea here: `AssetManager.load_concurrency` bounds concurrent
+  top-level loads for the same reason and orders its queue through a negated priority into a
+  `BinaryHeap`, and `VirtualTextureManager` bounds `max_concurrent_loads` at 32. Writing that
+  negation a second time is what moved the queue out of this directory: it is
+  `core/process/ConcurrencyGate.js` now — a free-standing "run at most N of these at a time,
+  highest priority first" — and the `VGeoFetchScheduler` that shipped first was folded into it.
+  The other callers kept their own queues, each recording why where its cap is declared. What none
+  of them can do is compare two containers' wants — VT sorts its queue coarse-mip first, a proxy
+  that only orders within one texture, and `AssetRequestScope.final_priority` is hierarchical
+  rather than a common scale. The cut hands this one, so the global order is the half that is
+  actually new.
 - **Wants are occlusion-aware in v1** (HZB test in the want path, or previous-frame
   feedback à la VT — which is occlusion-aware by construction). Frustum-only wants plus
   error-priority would page in and pin entire occluded interiors, breaking both budgets.
@@ -364,36 +456,55 @@ Pinned:
   child pages are resident in that tier (checked against its own child table —
   leaves-first order emerges from the rule).
 
-Open: page store API choice; prefetch heuristics (camera velocity); RAM eviction details;
-priority scoring and hysteresis constants.
+Open: page store API choice; prefetch heuristics (camera velocity); priority scoring and hysteresis
+constants; the in-flight pin, which is now blocked on a `VGeoPageWant` that names the page it was
+discovered *through* rather than on nobody owning the budget (format §10 — it is an efficiency item,
+because activation is safe at every arrival order). RAM eviction is no longer open; the numbers in
+`VGeoResidencyOptions` are defaults to be overridden by whoever knows the device, not measurements.
 
 ## 7. Runtime pipeline (per frame)
 
 1. Instance-level culling — existing.
-2. **VG cut pass (new, compute)**: evaluates the §4 rule over the **VG selection buffer**
-   — a VG-owned, densely maintained table of resident clusters (two (sphere, error)
-   pairs, geometry id, own-page and child-page/group indices, child-group selectable
-   bit — the page indices are what touch and want emission stamp), plus per-geometry
-   resident-cluster index tables; expansion over visible instances via prefix sums.
-   Emits meshlet draw records (existing record type) and wants/touches. VG carries
-   geometry only — materials are orthogonal and engine-side: a VG instance binds one
-   geometry to one material through its mesh exactly like any other geometry (a
-   multi-material source splits into one geometry per material upstream of the
-   builder), so the mesh-keyed material buckets downstream stay unchanged. Cost is
-   `O(Σ over visible instances of resident(geometry))` — per-instance, not per-pool;
-   the existing metadata buffer is *not* iterable for this (freed slices hold garbage,
-   and it carries no error pairs), which is why the selection buffer exists.
+2. **VG cut pass (new, compute)**: a §4 traversal, one per view. It descends the page DAG
+   and, inside a page, that page's node tree, from a queue seeded with the root cluster of
+   each visible instance; a node or cluster fine enough for τ terminates the descent, and so
+   does one whose child group is not selectable — which emits a want. It emits meshlet draw
+   records (existing record type) and wants/touches. VG carries geometry only — materials are
+   orthogonal and engine-side: a VG instance binds one geometry to one material through its
+   mesh exactly like any other geometry (a multi-material source splits into one geometry per
+   material upstream of the builder), so the mesh-keyed material buckets downstream stay
+   unchanged.
+
+   What it reads is the **VG selection buffer** — a VG-owned, densely maintained table of
+   resident clusters (two (sphere, error) pairs, geometry id, own-page and child-page/group
+   indices, child-group selectable bit — the page indices are what touch and want emission
+   stamp), plus per-geometry resident-cluster index tables. The existing metadata buffer is
+   *not* usable for this (freed slices hold garbage, and it carries no error pairs), which is
+   why the selection buffer exists.
+
+   Superseded: this step previously specified a **flat** evaluation of the rule over every
+   row of that buffer, at a cost of `O(Σ over visible instances of resident(geometry))`. §4
+   replaced it, and the reason is worth keeping: that cost is the resident set rather than the
+   cut, and it is the resident set that grows without bound at the scale this design targets.
+   The buffer survives the change; the sweep over it does not.
 3. Existing pipeline: HZB meshlet filtering (two-pass, per view — this is where frustum
    and occlusion happen), material bucket sort, indirect draws, G-buffer, deferred
    shading. The *record formats* flow unchanged; the *injection point* is new work — the
-   current mesh→meshlet expansion is inlined per material bucket and assumes contiguous
-   static meshlet ranges, so VG instances enter through their own expansion step (exact
-   seam fixed in M0/M1).
-
-Open: flat evaluation vs hierarchy-accelerated selection once resident counts are large
-(decide on measurement — prior-implementation data says selection is cheap; triangle
-throughput and screen-size uniformity dominate); shadow views (v1 reuses the
-one cut per view per §4, sharing setup rather than results); DRS τ semantics.
+   current mesh→meshlet expansion (`graph_expand_meshes_to_meshlets`, one helper shared by
+   four rasterization paths) reads one contiguous static meshlet range per geometry, so VG
+   instances enter through their own expansion step. M0 was where that seam was to be fixed
+   and it was not (§10); it is the first thing M1 has to do.
+
+Open: shadow views (v1 takes the one cut per view of §4, sharing setup rather than results,
+and a proxy where a view does not justify a cut); DRS τ semantics; the traversal's GPU shape —
+persistent threads over a queue, per nvpro, against what WebGPU without 64-bit atomics actually
+makes affordable.
+
+Settled since: **flat evaluation versus a hierarchy is no longer a measurement to take.** §4
+pins the traversal, and the hierarchy it descends is already written into every built asset
+(plan §2.3). The prior-implementation data that made flat evaluation look adequate — selection
+cheap, triangle throughput and screen-size uniformity dominant — was taken at that
+implementation's scale, and is not evidence about millions of resident clusters.
 
 ## 8. Builder pipeline (offline)
 
@@ -402,9 +513,12 @@ Pinned:
 - Stages: load → short-edge pre-pass (`bt_mesh_kill_short_edges` on soups / edge collapse
   on closed surfaces — required by the weld contract) → weld
   (`bt_merge_vertices_by_distance`; float32 quantum floor constrains tolerance) →
-  `bt_mesh_validate` → level-0 clustering (+ CC split) → the §3 level loop → per-cluster
-  encode (miniball sphere, normal cone, quantized attributes) → page assignment
-  (level-banded, DAG + spatial locality, acyclicity check) → container emit.
+  `bt_mesh_validate` → level-0 clustering (+ CC split) → the §3 round loop → per-cluster
+  encode → page assignment (level-banded, DAG + spatial locality, acyclicity check) →
+  container emit. The encode is the miniball sphere, the AABB, and `SHADE_MESHLET_V1`
+  attributes at their shipped widths (format §8): `cone_reserved` is zero because cone culling
+  is declined (§9), and positions are raw float32 because quantization is a `payload_encoding`
+  that does not exist yet.
 - **The weld tolerance is scale-relative**, a fraction of each geometry's bounding-box
   diagonal (`bt_mesh_relative_merge_distance`), resolved once off the source's bounds
   before any pass has removed anything. A conversion is many geometries at many sizes and
@@ -446,13 +560,19 @@ canonicalizing them is a preprocessor's job).
 
 ## 9. Non-goals (v1)
 
+- **Backface normal-cone culling — declined outright, not deferred.** The two-pass HZB already
+  rejects what a cone would, on evidence rather than on a bound; see plan §5 for the mechanism
+  and the reports behind it. `cone_reserved` stays a zeroed reserved field (format §7). This is
+  the one entry here that is not a "v1" qualifier.
 - Skinned / morphing / WPO meshes through the VG path (static geometry only).
 - Compute/software rasterization — WebGPU lacks portable 64-bit atomics; the hardware
   raster path through the existing pipeline is the only raster path. Deferred G-buffer
   shading already bounds the overshading cost of small triangles.
 - Ray tracing over the VG cut (`blas_from_meshlets` exists for static meshes; VG×RT is
   future work).
-- One cut per view, and proxy meshes for views that do not justify one (§4; plan §3.3).
+- *Sharing* one cut between views. §4 pins a cut per view and a proxy mesh where a view does
+  not justify one (plan §3.2, §3.3); what v1 does not attempt is a single cut made to serve the
+  camera and a shadow cascade at once, which was this document's earlier position and is wrong.
 - Attribute-quadric placement and UV-seam constraint planes (the boundary-plane-penalty
   mechanism §2 requires for open boundaries is the same one UV seams will use; v1
   targets position/normal-fidelity content).
@@ -461,20 +581,40 @@ canonicalizing them is a preprocessor's job).
 
 ## 10. Milestones
 
-- **M0 — DAG proof**: builder (in-memory, no format) → flat CPU-side cut → existing
-  meshlet pipeline draws it in a playground; the VG expansion seam is fixed here.
+These have not been taken in order. The format and the RAM-tier pager were built before the
+GPU cut, because the container is what the builder had to produce anyway and a reader is what
+proves it; the consequence is that M0's one *engine*-facing criterion — the expansion seam —
+is the piece still missing, and M1 is where the work now starts.
+
+- **M0 — DAG proof** — *the DAG half is done; the engine half is not*: builder (in-memory, no
+  format) → flat CPU-side cut → existing meshlet pipeline draws it in a playground; the VG
+  expansion seam is fixed here.
   Acceptance: crack-free under τ sweep and flythrough (incl. group-boundary polyline and
   flip validation); drawn-triangle count tracks resolution, not source size; §3 build
   validations green.
+  **Done**: the builder; a CPU cut (`playground/vgeo_viewer/select_cut.js`) that is a traversal
+  rather than the flat pass planned here, and that runs against a partially resident container,
+  which was M2's job; τ-sweep watertightness as a spec (`build/vgeo_cut.spec.js` counts edges
+  used by one triangle instead of two, at a spread of thresholds, and wants zero); the
+  group-boundary polyline guarantee, enforced by the builder rather than merely validated.
+  **Not done**: the expansion seam — the playground draws through Shade's direct geometry path
+  (a `DynamicMesh` from raw attributes, no meshlet encode), so nothing has yet entered the
+  meshlet pipeline through a VG-shaped door. Flip validation is not done either (§2, extension
+  3).
 - **M1 — GPU cut**: selection buffer + compute cut pass; wants/touches readback wired
   (still fully resident). Acceptance: parity with M0, selection cost measured
   (flat-vs-hierarchy decision data).
-- **M2 — format + streaming**: `.vgeo` container, pager, residency tiers with the §6
-  invariants (page-slot pool, address-migration propagation, occlusion-aware wants,
-  touch-LRU, in-flight pinning), budgets, eviction. Position quantization lands here
-  (encode + struct + decode shaders). Acceptance: cold-load TTFP = header + root page
-  regardless of asset size; streaming never produces holes; page size/codec chosen on
-  measurement.
+- **M2 — format + streaming** — *the RAM tier landed early*: `.vgeo` container, pager,
+  residency tiers with the §6 invariants (page-slot pool, address-migration propagation,
+  occlusion-aware wants, touch-LRU, in-flight pinning), budgets, eviction. Position
+  quantization lands here (encode + struct + decode shaders). Acceptance: cold-load TTFP =
+  header + root page regardless of asset size; streaming never produces holes; page
+  size/codec chosen on measurement.
+  **Done ahead of M1**: the container and its validator; the RAM-tier pager with residency,
+  activation and eviction (`format/read/`); TTFP, which is header + level table + root page and
+  measured to hold on a 27M-triangle asset; the page codec, chosen on measurement (§5).
+  **Still M2's**: the VRAM page-slot pool, address-migration propagation, occlusion-aware
+  wants, touch-LRU, in-flight pinning, budgets, and position quantization.
 - **M3 — hardening**: prefetch, disk page store decision, budget defaults, low-limit
   adapters (τ fallback), stress on CAD-class input; browser builder-harness proof
   (build a mid-size asset in-tab).

package/src/shade/renderer/geometry/virtual/VIRTUAL_GEOMETRY_PLAN.md

@@ -6,8 +6,10 @@ This file holds what is *next*, and — more usefully — the reasoning and the
 ideas that are accepted but not yet scheduled, so that picking one up does not mean
 rediscovering it.
 
-Current scope is **the format and the builder**. There is no runtime yet; everything under
-§3 waits on one.
+Current scope is **the GPU runtime**. The builder, the container and an incremental reader are
+built — design §10 says which milestone that leaves us in and in what order the work arrived —
+so §3 no longer waits on "a runtime" as a whole. Each item there now names what is actually
+blocking it.
 
 Items carry their provenance. `[M §x]` is a section of
 [MICRON_COMPARISON_2026_08_20.md](./MICRON_COMPARISON_2026_08_20.md), the review of the
@@ -38,8 +40,49 @@ accepted-and-waiting one.
   an error this file previously made.
 - **Boundary constraint quadrics** — see §4.1, which was the blocking item.
 - **A partitioning fallback** `[M §3.3]`, `[V §3.2]` — see §2.1.
-
-## 2. Next, in the current scope
+- **An incremental reader, and residency with it** — `format/read/`. A container opens in its
+  header, level table and root frame; after that a page arrives only because something asked for
+  it, from exactly the byte range the page pointing at it wrote down. `VGeoContainerReader`
+  holds residency, activation and eviction; `VGeoByteSource` is the seam — an `ArrayBuffer`, a
+  dropped `Blob`, or HTTP ranges that refuse a `200` where they asked for a range. The
+  whole-file audit is a tool in `format/read/tool/` and nothing on the runtime path can reach it;
+  format §4/§5's reject rules always run.
+  VGEO_FORMAT §10 marks, requirement by requirement, what this covers and what it leaves to a
+  budget owner.
+- **The reader hardened for 1000 assets, and the budget owner it was waiting for** —
+  `virtual/residency/`. Measured first, and three of the audit's suspicions did not survive the
+  measurement, which is the more useful half of the result: the per-page allocation the reader adds
+  on top of a decoded page is 6.4–11.9 KB, which is 1.2% of a 544 KB page and not worth a rewrite;
+  the reverse-reference index tracks the residency frontier rather than the session's history, so it
+  does not leak; and the pinned root page is one cluster by invariant §11.7, measured at 1,472 and
+  1,732 B, so a thousand of them is under 2 MB and no builder-side cap is wanted. What *was* real:
+  a 256 KiB head prefix retained for the reader's life outside every budget (released now, and the
+  default cut to 64 KiB), no retry/backoff/quarantine so one bad page threw (now §10's full
+  sequence, with telemetry), no cancellation at all (now a per-read `AbortSignal` end to end), and a
+  whole-file audit and install-every-page sweep reachable from a reader a renderer holds (now tools,
+  with a 1 GiB bound and a message instead of an allocator failure). Above the reader,
+  `VGeoResidencyManager` owns the byte budget, the touch stamps, the eviction order and — through
+  one `core/process/ConcurrencyGate` — one fetch cap across every reader, because a per-reader cap
+  does not compose, measured at sixteen concurrent requests from four readers capped at four.
+- **Activation, and the double-draw it prevents** — format §10's selectable invariant, as one
+  live-reference counter per resident group with cascades on install and evict. The clamp reads
+  *selectable*, not *resident*: a group whose parents are split across pages must not be refined
+  into through the parents that happen to have arrived. Measured on a 19-page bunny, 21 of whose
+  362 non-root groups have split parents: 355 of 576 camera positions double-drew, up to 10.7%
+  of drawn triangles, before. Zero after. It reads as corrupt geometry rather than as a missing
+  page, which is why the tool that found it could not name it.
+- **A CPU cut against a partially resident container** — `playground/vgeo_viewer/`.
+  `select_cut.js` is the runtime's rule rather than a debug approximation of it; the viewer is
+  what makes clamping, streaming and eviction watchable, and `__state()` is what makes it
+  assertable from a headless browser. It draws through Shade's direct geometry path, not the
+  meshlet pipeline — so it proves the DAG and the format, and proves nothing about the engine
+  seam.
+
+## 2. Next
+
+§2.1 and §2.2 both landed. They are kept because their notes are the reasoning behind decisions
+that are now only visible in code, and rediscovering them is the expensive part. **§2.3 is the
+open one**, and it is the one to read if you are about to start on the cut pass.
 
 ### 2.1 A partitioning fallback for when METIS fails `[M §3.3]`, `[V §3.2]` — LANDED
 
@@ -161,10 +204,38 @@ instance count first and from fan-out immediately after — measured out-degree
 the middle bands of the 98k sphere. It bites only for a single instance in close-up, and only
 for the first few bands, which are also the cheapest to walk.
 
-## 3. Waiting on a runtime
+### 2.3 Nothing reads the traversal metric or the node tree
+
+§2.2 built both, and the format specifies both: every child-table entry carries the child page's
+LOD-metric sphere and error range (VGEO_FORMAT §5, §6.2), and every page carries a fanout-4 tree
+over its own groups (§6.1). The writer emits them, `vgeo_validate_container` checks them
+(§11.10–11.11) with corruption tests behind both checks, and **nothing reads either**.
+`VGeoPage` records `node_count` and stops; `select_cut.js` descends group by group.
+
+This is not a defect to fix on its own account. Building a consumer for its own sake would fix
+the interface before the pass that has to live with it, and the CPU traversal does not need the
+acceleration — it walks a resident page set on a machine with branches. It is written down here
+for two reasons: because "the validator checks it" reads exactly like "something uses it", and
+because the cost is already paid — 0.3%–1.6% of asset size, on disk, in every asset already
+built, earning nothing.
+
+What it buys when the pass arrives, and what to hold it to:
 
-Accepted, documented here so the reasoning is not lost, and out of scope until there is
-something to run.
+- **Before the fetch.** A page can be frustum- and LOD-culled from its parent's child-table
+  entry, so a want list is exactly the pages that passed the test and are not resident, rather
+  than everything a descent happened to reach. Today the only way to learn a page was not wanted
+  is to fetch and decode it, which is backwards.
+- **Below the page.** A node discards a region rather than a cluster, and gives the top of the
+  descent width it does not otherwise have.
+
+Neither is measurable until there is a GPU traversal to measure, which is the honest reason it
+has waited.
+
+## 3. Waiting on a GPU runtime
+
+Accepted, documented here so the reasoning is not lost. "Waiting on a runtime" used to mean
+waiting on all of it; a CPU traversal and a RAM-tier pager now exist (§1), so what each item is
+actually blocked on is named individually.
 
 ### 3.1 Object-space refinement `[M §2.7]`
 
@@ -174,6 +245,11 @@ and run every cull and error test in **object space** against untransformed clus
 One matrix multiply per instance instead of transforming every cluster's bounds to world
 space. With thousands of instances sharing one geometry that difference is the entire cost.
 
+**Blocked on**: instances. The CPU traversal already tests in object space — but only because
+the viewer draws one geometry at identity and never had to decide, so the rule is written the
+right way round and the multiply that makes it pay has never been exercised. This becomes real
+work at the same moment the cut pass does.
+
 ### 3.2 Proxy geometry from a DAG cut `[M §2.6]`
 
 `Micron/format/micron_build_proxy_geometry.js`: refine from the root with a heap keyed on
@@ -220,8 +296,10 @@ free-gap list maintained on the GPU —
 `shaders/stream_allocator_unload_groups.comp.glsl`.
 
 The mirror-image problem on our side is recorded in design §2: `GPUMeshletManager`'s
-compaction and growth is a CPU-side rebuild costing ~2× VRAM transiently, and
-meshlet-address migration is an acknowledged unhandled TODO in `GPUGeometryManager`.
+*compaction* is a CPU-side rebuild costing ~2× VRAM transiently, and meshlet-address migration
+is an acknowledged unhandled TODO in `GPUGeometryManager`. Growth has stopped being a rebuild
+since this was written — it appends through `OffsetAllocator#grow` and moves no record — which
+narrows the problem to exactly the case VG churn produces: closing holes, not adding room.
 
 ## 4. Was blocking
 
@@ -274,12 +352,34 @@ attacked:
   mantissa bits of positions and UVs ahead of its lossless group compression. It keeps the
   float32 layout, the metadata struct and every decode shader exactly as they are, and simply
   makes the bytes compress. A fraction of the work of quantization for a real share of the
-  benefit, and it composes with a page codec, which VGEO_FORMAT §12 still lists as open.
+  benefit, and it composes with the page codec, which has since shipped: LZ4 finds only
+  1.12x–1.18x on today's payloads *because* float32 positions give an LZ matcher nothing, and
+  dropping mantissa bits is how you give it something.
 - **Cluster-local position quantization**, as design §2 describes — invasive, spanning
   encode, the metadata struct and every decode shader.
 
 ## 5. Considered and declined
 
+- **Backface normal-cone culling** `[M §2.3]` — the `cone_reserved` field, a producer for it,
+  and the cull pass that would read it. **Declined: the HZB already rejects what a cone would,
+  and it does it on evidence rather than on a bound.** On a closed surface backfacing *is*
+  occluded, and §7's two-pass HZB sees that; what a cone adds over it is pre-emption by one
+  pass, bought with per-cluster data and a per-cluster test in the cull pass, every frame and
+  per view. It pays best where cones are tight — flat clusters, which are also the cheapest
+  to raster — and degrades exactly where a cut spends its clusters, because a group's cone
+  widens toward hemispherical as the levels coarsen. Micron paid for tight cones by scoring
+  merges on cone angle (`[M §2.3]`), which is the real price: grouping stops being a purely
+  spatial criterion in order to feed a cull that the HZB was already doing.
+
+  This is not our finding alone, which is why it is a decline rather than an experiment: Epic
+  did not ship it in Nanite, meshoptimizer's author argues against the payoff despite the
+  library exposing cone data, and Remedy report the same for Northlight. Taken together with
+  the mechanism above, there is nothing left to measure.
+
+  Consequences, so that nobody re-derives them: `cone_reserved` stays a zeroed reserved u32
+  (format §7) — reclaiming 4 bytes of 112 is not worth a `format_version` move, and the page
+  codec compresses a constant-zero column away. `bt_face_cluster_compute_normal_bounding_cone`
+  has no consumer and no prospect of one.
 - **Fixed-size residency slots** `[M §2.5]`. Micron gives every resident patch one slot of
   `MICRON_PATCH_SIZE_MAX × 3` vertices with occupancy in a BitSet: no allocator, no
   fragmentation, no address migration, at the cost of internal fragmentation proportional to

package/src/shade/renderer/geometry/virtual/VK_LOD_CLUSTERS_COMPARISON_2026_08_20.md

@@ -14,6 +14,11 @@ The build side is a fork of meshoptimizer's single-header `clusterlod.h`
 > scheduled but not yet built, all live in
 > [VIRTUAL_GEOMETRY_PLAN.md](./VIRTUAL_GEOMETRY_PLAN.md) — that file is the one to act on.
 > This one is kept as the analysis behind those decisions.
+>
+> It is a **snapshot dated in its filename** and is not maintained against the code. Statements
+> about meep that have since gone stale are corrected inline in block quotes like this one,
+> rather than edited away — the argument that was made at the time is the reason the decision
+> was taken, and rewriting it hides that.
 
 ## 1. Where it confirms decisions already made
 
@@ -154,6 +159,13 @@ cases where it currently gives up — and those cases are real: the boundary-qua
 design §2 means open geometry reduces at recorded-zero error today, which is worse than
 reducing at an inflated one.
 
+> **Since this was written**: boundary constraint quadrics landed (plan §4.1), so that closing
+> argument is spent — open geometry no longer reduces at recorded-zero error, and the choice is
+> no longer between an honest inflated error and a dishonest zero. The case for a middle rung
+> survives it, but it is now the plain one: a group that cannot be reduced is carried forward
+> untouched, and enough of those is what makes a build fail to converge and throw. Plan §2.1
+> records that the sloppy-mode escalation is *safe* to build now for the same reason.
+
 ### 3.3 A minimum cluster size
 
 `meshopt_buildMeshletsFlex` takes `min_triangles` as well as `max_triangles`, so a cluster
@@ -163,6 +175,13 @@ meep's splitter caps and never floors. Measured fill on the current builder runs
 tris/cluster at the leaves down to 67 at the coarse levels. A floor, enforced by merging
 under-filled pieces the way `pack_pieces` already does inside a group, would close that.
 
+> **Since this was written**: the leaf half landed (plan §1). A group of leaves with room to
+> spare between them is *re-cut* into fewer and fuller leaves rather than simplified —
+> `try_repack` in `vgeo_build_levels.js` — and the re-cut **replaces** them rather than
+> becoming their parent, because a step of zero error is a step the cut rule can never take.
+> Only leaves: above them the same re-cut would orphan a subtree. So the 120-at-the-leaves half
+> of that measurement is addressed and the 67-at-the-coarse-levels half still stands.
+
 ### 3.4 Spatial variants at every partitioning step
 
 `clusterize()` picks `meshopt_buildMeshletsSpatial()` or `meshopt_buildMeshletsFlex()` on