@woosh/meep-engine 3.22.0 → 3.23.0

package/src/shade/renderer/geometry/virtual/VGEO_FORMAT.md

@@ -1,6 +1,6 @@
 # `.vgeo` — Virtual Geometry Container Format
 
-Status: **writer and validator implemented; no reader yet.** Companion to
+Status: **writer, validator and reader implemented; no GPU runtime.** Companion to
 [VIRTUAL_GEOMETRY_DESIGN.md](./VIRTUAL_GEOMETRY_DESIGN.md); every requirement pinned in
 its §5/§6 is realized here as a concrete field or invariant. Pinned structure below;
 open details are listed at the end. One `.vgeo` file = one unique geometry's complete
@@ -8,10 +8,21 @@ cluster DAG; instances share the file.
 
 The implementation lives beside this document: `format/` is the container — what a file is,
 how one is written, and `vgeo_validate_container.js`, which audits every §11 invariant in
-one — and `build/` is the pipeline that produces it from a `Geometry`, glTF front end
-included. The node command line over both is at `tools/geometry/virtual/`. One glTF yields **one file per
+one; `format/read/` reads one a page at a time; and `build/` is the pipeline that produces
+it from a `Geometry`, glTF front end included. The node command line over the writer and the
+validator is at `meep-engine/tools/geometry/virtual/`. One glTF yields **one file per
 primitive**, in object space, with the scene's placements written to a sidecar manifest
-rather than baked in. The runtime side (pager, residency, cut pass) is still design only.
+rather than baked in.
+
+**Where the reader stops**, because it is the first thing a runtime needs to know. `format/read/`
+implements the *mechanics* of §10 — cold start, one fetch per page from the byte range its parent
+wrote down, want dedup, activation, eviction, and the mandatory reject rules of §4 and §5. It
+implements none of the *policy*: no fetch scheduler, no budgets, no touch-LRU, no quarantine, and
+the pinning rules are left to whoever manages residency. It also decodes neither the node table
+(§6.1) nor the child-table traversal metric (§5, §6.2) — `VGeoPage` records `node_count` and stops
+— because nothing consumes them yet; a GPU cut pass is the first thing that would.
+`shade/playground/vgeo_viewer/` drives the reader end to end, and its `select_cut.js` is a CPU
+reference traversal of the cut rule this format was shaped around.
 
 ## 1. Design constraints (inherited, binding)
 
@@ -98,8 +109,8 @@ All u64/u128 fields are naturally 8-byte aligned.
 | offset | type      | field                    | notes |
 |-------:|-----------|--------------------------|-------|
 | 0      | u32       | magic                    | `'VGEO'` = 0x4F454756 |
-| 4      | u16       | format_version           | breaking layout changes only |
-| 6      | u16       | min_reader_version       | reader MUST reject if its version is lower |
+| 4      | u16       | format_version           | 3 today; breaking layout changes only |
+| 6      | u16       | min_reader_version       | 3 today; reader MUST reject if its version is lower |
 | 8      | u16       | header_size              | = 256 |
 | 10     | u8        | hash_algo                | 0 = xxHash3; readers MUST reject unknown values |
 | 11     | u8        | reserved                 | |
@@ -378,7 +389,7 @@ in the cut test):
 | 56     | [f32; 6] | aabb            | this cluster's object-space AABB — culling, GPU metadata `bounds_box`, and the (future) dequantization grid |
 | 80     | u16      | child_ref       | index into this frame's child table; `SAME_PAGE` when the child group is in this page; `NO_CHILD` for leaves |
 | 82     | u16      | child_group     | group index within the child page (or this page, for `SAME_PAGE`); `NO_GROUP` for leaves |
-| 84     | u32      | cone_reserved   | reserved for backface-cone culling — **write 0**. No cull pass consumes a cone today and the producer needs a `BinaryTopology` port; the sub-encoding is pinned together with that pass, not before |
+| 84     | u32      | cone_reserved   | reserved — **write 0**. Named for a backface-cone cull pass that is now **declined outright** (design §9, plan §5), so no sub-encoding will ever be pinned here. Kept rather than reclaimed: removing it moves `format_version`, a constant-zero column costs nothing after the page codec, and there are ten more reserved bytes beside it at 102 and 104 anyway. A candidate to rename and reuse the next time the version moves for some other reason |
 | 88     | u32      | encoding_flags  | frozen bit table in §8 |
 | 92     | u32      | payload_offset  | into this page's payload section; ≡ 0 (mod 4) |
 | 96     | u32      | payload_size    | bytes; ≡ 0 (mod 4) |
@@ -462,21 +473,46 @@ the directory fetch it at block granularity and MUST verify.
 
 ## 10. Reader requirements
 
-Standard runtime operation is **pure traversal** — the directory is not involved:
+Standard runtime operation is **pure traversal** — the directory is not involved. Every item
+below is marked with what `format/read/` does about it: *(shipped)*, *(open)*, or a split where
+the mechanics are there and the policy is the caller's. Where a requirement is *policy*, the
+implementation is `renderer/geometry/virtual/residency/` rather than `format/read/` — a sibling of
+the format, not part of it, because what may be resident and what may be in flight are questions
+about a scene and not about a container.
 
-- Cold start: fetch `[0, 256)`, verify `header_checksum`, magic, versions (§4 reject
+- **Cold start** *(shipped)*: fetch `[0, 256)`, verify `header_checksum`, magic, versions (§4 reject
   rules), then fetch `[root_page_offset, root_page_offset + root_page_fetch_size)` — the
   level table (§4.1) sits between the two, so a head prefix covering both brings the whole
   always-resident set in one request. The root frame's `decoded_size` MUST equal
   `root_page_decoded_size` and its frame extent MUST match `root_page_fetch_size`, else
   reject. Or speculatively fetch one larger head prefix — coarse-first layout makes every
   prefix byte useful.
-- **Refinement**: a want is a `child_ref` in an installed page; its child-table entry
+
+  **How much of that prefix to keep, and for how long, is not a free choice at scale.** The
+  always-resident set is small: invariant §11.7 puts exactly one root cluster in the file and the
+  assembler gives the root page the complete topmost level, so a measured root page is one cluster —
+  1,732 B and 1,472 B on two torus builds, header and level table included coming to about 2.4 KB.
+  A prefix past that is speculation about the *next* band, and at the hundreds-of-KB pages design §5
+  targets it buys one page. `VGeoReadOptions.head_prefix_bytes` therefore defaults to 64 KiB rather
+  than the 256 KiB it did, and the reader **releases the prefix** the first time a read falls
+  outside it (`VGeoContainerReader.release_head`, and `head_bytes` to see it). Measured, at the old
+  default a 256 KiB prefix over a 22 MB container covered 105 of its 4,000 pages and served nothing
+  after those installed — while sitting outside `resident_bytes` and outside anything `evict` could
+  reach, which across a thousand open assets was a quarter of a gigabyte no budget could see.
+- **Refinement** *(shipped)*: a want is a `child_ref` in an installed page; its child-table entry
   carries the exact byte range — one round-trip per cold page, no metadata fetch ever.
   Wants dedup on `page_index` across clusters, groups, and instances. A want whose
   target page is **already installed** resolves as an activation recheck of the
   referenced group (below), never a fetch.
-- **Activation (closure without dependency lists)** — the selectable invariant:
+
+  A want that goes stale mid-flight is cancellable: `VGeoByteSource.read` takes a per-read
+  `AbortSignal`, `vgeo_fetch_byte_source` passes it to `fetch` alongside the source's own, and
+  `VGeoContainerReader.cancel` abandons one page's request without touching the others.
+  **What a stale want does, decided:** it is aborted while it can still be aborted, and installed
+  once it cannot — the bytes are already paid for, and refusing to keep them costs the fetch *and*
+  the page. An arrival nobody wanted enters residency untouched, which under LRU on last-touched
+  makes it the first eviction candidate and free if the camera comes back.
+- **Activation (closure without dependency lists)** *(shipped)* — the selectable invariant:
 
   *A group `g` is selectable iff every one of its `parent_cluster_count` references is
   live — the referencing cluster's page is installed and that cluster's own group is
@@ -497,30 +533,112 @@ Standard runtime operation is **pure traversal** — the directory is not involv
   ancestors keep covering that surface — **no holes and no double-draw, in any arrival
   order**, because a group never draws before all pages holding its parents are
   installed and selectable.
-- **In-flight protection**: a fetch discovered through parent page P pins P until the
+- **In-flight protection** *(open)*: a fetch discovered through parent page P pins P until the
   child installs — known at queue time, purely local. Arrivals verify the
   identity-seeded frame checksum; activation makes any arrival order safe beyond that.
-- **Eviction**: within each residency tier, a page is evictable only while it has no
-  resident child pages in that tier — checked against its own child table (`SAME_PAGE`
-  references never pin). Leaves-first order emerges from the rule; the root page is
-  pinned.
-- Verify every ranged response is `206` with the requested `Content-Range`; a `200`
+  The reader dedups in-flight fetches by page index but pins nothing.
+  Still open, and now for a nameable reason rather than for want of an owner: a `VGeoPageWant`
+  carries the page it names and not the page it was *discovered through*, so neither the reader nor
+  the manager can pin P without the cut telling them which P it was. It is an efficiency item and not
+  a correctness one — evicting P while its child is on the wire wastes the fetch, because the child
+  installs non-selectable and activation is safe at every arrival order — and closing it means adding
+  the source page to the want, which is a change to the cut as much as to the reader.
+- **Eviction** *(mechanics shipped in the reader, policy shipped above it)*: within each residency tier, a page is
+  evictable only while it has no resident child pages in that tier — checked against its own
+  child table (`SAME_PAGE` references never pin). Leaves-first order emerges from the rule;
+  the root page is pinned. `VGeoContainerReader.evict` does the dropping and the activation
+  retraction that goes with it — deliberately including the case where a caller evicts against
+  this rule, where the cascade is what stops an orphaned child page from staying drawable — but
+  it chooses nothing: which page to drop, and when, is the budget owner's.
+  `VGeoContainerReader.resident_child_count` is the lookup the rule is written in terms of, and it is
+  the whole of what the reader contributes.
+- **The budget owner** *(shipped, above the reader)*: `VGeoResidencyManager`. Readers register with
+  one; it holds the RAM tier's byte budget across every registered reader, admits a want against it
+  *before* the fetch out of the `decoded_size` the child table already carries, and evicts on the
+  rule above ordered by design §6's LRU on **last-touched** with a protect window no shorter than
+  feedback latency. It reaches nothing private: it is written entirely against the reader's public
+  surface, which is the test that the split is real rather than nominal.
+
+  A shared **fetch scheduler** comes with it, and has to: `VGeoReadOptions.max_concurrent_fetches`
+  bounds one reader and cannot do better, and four readers at that cap were measured issuing sixteen
+  concurrent range requests, eight readers at the default 6 issuing 48. `VGeoResidencyManager`
+  holds one total across every reader in a single `ConcurrencyGate` (`core/process/`), as one
+  priority queue — so a want's priority, which is pixels of projected error and therefore
+  comparable across assets by construction, is compared globally rather than within whichever
+  reader happened to ask first.
+
+  **Most of that shape already exists in this engine twice, and the bullet is more useful read as
+  the third one than as new work.** `renderer/texture/virtual/VTPhysicalCache` keeps a
+  `#last_touch` stamp per slot, a lock flag for slots that may never go, and
+  `find_eviction_candidate(current_frame, protect_window)` — LRU on last-touch under a protect
+  window justified by feedback latency, which is the rule above. Design §6 said so when it pinned
+  that rule ("the VT system already keys on touch for this reason"), so the resemblance is
+  convergent by instruction rather than accidental. `engine/asset/AssetManager` caps concurrent
+  top-level loads on `load_concurrency` for the reason the gate holds a total, and inverts
+  priority into a `BinaryHeap` the same way in `get_pending_asset_priority_score` — being the
+  third place to want that, and the second to write that inversion, is why the gate is now
+  `core/process/ConcurrencyGate` rather than a file in this directory, and `AssetManager` keeps
+  its own queue anyway for the reasons its `load_concurrency` doc records.
+  `VirtualTextureManager` caps `max_concurrent_loads` at 32 and drops queued requests untouched
+  for `queue_stale_window` frames, which is a stale-want policy of the kind `cancel` answers here.
+
+  **The narrow claim that survives that** is a *byte* budget across *several* readers, admitted
+  against before the fetch: `VTPhysicalCache` budgets a count of fixed-size slots and
+  `AssetManager` budgets nothing and evicts nothing. The retry-into-quarantine and the per-fetch
+  `AbortSignal` above are likewise new here — `AssetManager` records a failure and lets a later
+  `get` retry, and neither precedent can abort one request. Granularity is why the pages are not
+  simply `AssetManager` entries: an `AssetDescription` is a path and a type, where a page is a
+  byte range in a container the format gives no global id to (a cluster is a page plus an index),
+  at the 4,000 and 20,000 pages per container measured above and a thousand assets open at once.
+- *(shipped)* Verify every ranged response is `206` with the requested `Content-Range`; a `200`
   means the server ignored the range — abort the transfer immediately (design §6).
-- Verify frame checksums (identity-seeded, §5) before decode, and `decoded_size`
+  `vgeo_fetch_byte_source` checks the status *and* the returned range: a proxy can answer 206
+  with a range other than the one asked for.
+- *(shipped)* Verify frame checksums (identity-seeded, §5) before decode, and `decoded_size`
   against the child-table entry used to fetch and against the codec's decode bound
   (§5) before allocating for it. On mismatch: bounded retries with
   backoff, then **quarantine** the page — the clamped cut keeps a coarse stand-in
   correct indefinitely — and surface telemetry. Never retry-loop: corruption at a
   content-addressed URL is deterministic.
-- After install, stamp each cluster's selection row with its own page index and its
-  child page index/group (touch and want emission, design §6/§7).
-- Fetch batching: coalesce adjacent wanted pages into contiguous single ranges; ranges
+  The decode bound and the exact-length decode are unconditional. The frame checksum is
+  `VGeoReadOptions.verify_frames`, off by default because a tool reading a file it just built is not
+  defending against a network — a runtime registering a reader with `VGeoResidencyManager` should
+  turn it on, and it is what lets the retries see corruption rather than only outages.
+  Retry and backoff are `fetch_retries` (2) and `fetch_retry_backoff_ms` (50, doubling); after them
+  the page enters `VGeoContainerReader.quarantine()` with its reason and is never fetched again, and
+  `install_wants` skips it rather than failing the batch — one bad page out of a thousand assets'
+  worth costs exactly that page. *When* to lift a quarantine is policy and is not the reader's:
+  a checksum that failed at a content-addressed URL will fail again, while a run of 503s from one
+  edge node is a different thing that a manager watching other assets arrive can tell apart, so
+  `clear_quarantine` exists and `VGeoResidencyManager.clear_quarantines` is who calls it.
+- *(open)* After install, stamp each cluster's selection row with its own page index and its
+  child page index/group (touch and want emission, design §6/§7). There is no selection buffer
+  yet; `VGeoPage` keeps a page's tables and the traversal reads them directly.
+- *(open)* Fetch batching: coalesce adjacent wanted pages into contiguous single ranges; ranges
   never cross a band boundary by construction (coarse-first ordering), so coalescing
   lives within bands — the within-band spatial curve (§12) is what carries fetch
   locality and must be chosen with that as a criterion. Fetching through small unwanted
   gaps is permitted under an implementation waste threshold.
-- Never assume `nominal_page_size`; per-page sizes come from child tables (traversal),
-  frames (prefix scan), or the directory (special tasks).
+- *(shipped)* Never assume `nominal_page_size`; per-page sizes come from child tables
+  (traversal), frames (prefix scan), or the directory (special tasks).
+- *(shipped)* **Nothing a runtime holds may be able to read the whole file.** The whole-file audit
+  (§11, every checksum, the directory against a frame scan) and the install-every-page sweep are in
+  `format/read/tool/` as `vgeo_audit_container` and `vgeo_install_all_pages`, not on the reader —
+  `VGeoContainerReader.validate` and `.install_all_pages` remain as deprecated shims for the
+  playground and the specs, and they reach the tools through a dynamic import so that a bundle that
+  only opens containers does not carry the validator at all. The audit takes the container as one
+  `ArrayBuffer`, because every §11 invariant is a cross-check between two places in the file; that is
+  a ceiling and not a slope, so it refuses above `VGEO_MAX_AUDIT_BYTES` (1 GiB) naming the size,
+  rather than failing in the allocator. Measured: a 514 MiB container audited in 19.4 s, and
+  installing every page of a 2,000-page container of 544 KB pages made 514 MiB resident in 210 ms.
+- *(shipped)* The reverse-reference index of the activation rule is bounded by the **residency
+  frontier**, not by how many pages a session has visited, and it is worth knowing which because it
+  has the shape of an index that leaks. An entry exists only while a resident, selectable cluster
+  refers into an absent page, and install/activation and evict/deactivation are symmetric. Measured
+  on a 20,000-page container with the 3,280 pages of its eight coarse bands pinned resident:
+  `pending_reference_page_count` was 6,561 after pinning and still 6,561 after touring all 6,561
+  pages of the band below them a window at a time and evicting each window. Dropping the pinned
+  pages too leaves 3.
 
 Build consequence (accepted): child tables hold downstream offsets — the format is
 optimized for reading, and the builder pays for it. Preferred strategy: emit frames
@@ -603,5 +721,4 @@ clustering and simplification dominate regardless.
 - Spatial ordering curve within a level band (Morton over group centroids is the
   default candidate; chosen with fetch-coalescing locality as an explicit criterion,
   §10).
-- Backface-cone cull pass and, with it, the `cone_reserved` sub-encoding.
 - Multi-geometry containers (v1: one geometry per file; scenes reference many files).