@woosh/meep-engine 2.168.1 → 2.169.0

package/src/engine/graphics/CONTEXT_LOSS_RECOVERY_PLAN.md

@@ -0,0 +1,446 @@
+# WebGL context loss recovery — investigation, plan, and results
+
+Date: 2026-07-30. Scope: `meep` graphics stack (three.js r136, WebGL2) as consumed by the Askara app
+(`app/src/mir-engine/model/game/makeMirEngineConfig.js`).
+
+Goal: when the browser drops the WebGL context, detect it, recover in place, and let the player continue —
+no application restart, no reload, no lost session state.
+
+**Status: phases 0–3 implemented and verified.** Phases 4–6 remain (§11). Claims below are tagged
+**[src]** (read from three.js / meep source) or **[measured]** (observed in the browser harness, §10).
+
+---
+
+## 1. TL;DR
+
+* **The engine was much closer to recoverable than it looked.** three.js r136 rebuilds all of its
+  GPU-object bookkeeping on `webglcontextrestored` (`initGLContext()` re-instantiates `WebGLProperties`),
+  so every texture, geometry, render target and shader program is re-created from CPU-side data on first
+  use after restore. meep keeps CPU-side sources for essentially everything it uploads. **[src]**
+  **[measured]**: on a fresh context the same scene renders with identical draw calls (7) and triangle
+  count (412,999), and pixel-identical output once one known defect is accounted for.
+* **That one defect was the environment map.** `load_and_set_cubemap_v0` baked PMREM into a render target
+  and dropped the source texture, so IBL died with the context — worth **59% of scene luminance** in the
+  test scene (83.8 → 34.7). It is now re-baked on restore from a retained source. **[measured]**
+* **Everything else needed was state, not content**: three silently resets the clear colour on restore
+  (`WebGLBackground` is re-instantiated), rendering was not gated while the context was gone, and nothing
+  told the user or the metrics pipeline anything.
+* **Recovery is now pixel-exact.** Repeated loss/restore cycles on the harness scene come back
+  bit-identical to the baseline, with no resource growth across cycles. **[measured]**
+* **Cost**: steady-state overhead is one boolean compare per frame and no allocations — the design is
+  push-based (signals), never polled. The unavoidable cost is the first frame after restore:
+  **~37 ms** on the harness scene, **~214 ms** on the game's title scene, against ~0.6 ms steady state.
+  **[measured]**
+* **What is not yet proven**: that the *browser* will hand the context back. The restore path only runs
+  while the page is visible, and the harness available here runs in a hidden tab (§10.4). The failure
+  ladder exists precisely because that decision is not ours to make.
+
+---
+
+## 2. What happened before this change
+
+| Step | Old behaviour | File |
+| --- | --- | --- |
+| Context lost | three set `_isContextLost = true`; meep logged a typo'd warning | three.module.js:26236, GraphicsEngine.js |
+| During loss | `Engine.render()` kept calling `graphics.render()` every rAF. three guards only `render()`; `setRenderTarget` / `clear` / `copyTextureToTexture` / `readRenderTargetPixels` are unguarded | three.module.js:26555, [Engine.js:438](src/engine/Engine.js:438) |
+| Context restored | three ran `initGLContext()`, meep logged and did nothing | three.module.js:26246 |
+| Result | scene mostly returned by accident, with black IBL, a reset clear colour, a multi-second hitch, and no diagnostics | — |
+| User feedback | none, ever, including on unrecoverable failure | — |
+
+---
+
+## 3. Why most of it recovers for free
+
+`onContextRestore` → `initGLContext()` replaces `properties`, `textures`, `attributes`, `geometries`,
+`bindingStates`, `programCache`, `state`, `background`, `info`, `renderLists`, `renderStates`
+(three.module.js:25921, 26246). Because `WebGLProperties` is a **fresh `WeakMap`**:
+
+* **Textures** — `initTexture()` sees no `__webglInit`, `uploadTexture()` sees `__version === undefined`,
+  so every texture re-uploads from `texture.image` / `texture.mipmaps` (three.module.js:22505).
+* **Render targets** — `setRenderTarget()` sees `__webglFramebuffer === undefined` and calls
+  `setupRenderTarget()` (three.module.js:27473). The objects and their `texture` identities survive, so
+  uniforms pointing at them stay valid. Their *contents* are gone (§4.2).
+* **Geometry / VAOs** — re-uploaded from the `BufferAttribute` arrays; meep never uses
+  `BufferAttribute.onUpload` to release them.
+* **Programs** — `WebGLPrograms` cache is new, so every material recompiles on first draw.
+* **Extensions** — `extensions.init()` re-requests `EXT_color_buffer_float` on WebGL2, which is what the
+  float/half-float buffers need (three.module.js:16065).
+* **Shadow maps** — re-rendered every frame from `postVisibilityBuild`
+  ([ThreeLightBinding.js:291](src/engine/graphics/ecs/light/binding/three/ThreeLightBinding.js:291)).
+* **CPU-sourced data textures** — FoW (`Sampler2D` → `DataTexture`), Forward+ cluster tables, terrain
+  splat/height/AO maps, LPG SH textures. All keep their typed arrays, so all re-upload.
+* **Extra memory cost of relying on this**: none. Those CPU-side sources are retained today regardless.
+
+**Corollary that shaped the design:** recovery must be *narrow*. Re-running startup would be actively
+harmful — `FrameBuffer.initialize()` allocates a **new** render target per call
+([ColorAndDepthFrameBuffer.js:31](src/engine/graphics/render/buffer/buffers/ColorAndDepthFrameBuffer.js:31))
+and plugins allocate their own
+([AmbientOcclusionPostProcessEffect.js:91](src/engine/graphics/render/buffer/simple-fx/ao/AmbientOcclusionPostProcessEffect.js:91)),
+so "just restart graphics" would leak a full G-buffer set per loss and turn a recoverable event into an
+out-of-memory crash.
+
+---
+
+## 4. What is lost and does not come back by itself
+
+### 4.1 Renderer configuration dropped by `initGLContext()` — FIXED
+
+`WebGLBackground` is re-instantiated and owns the clear colour/alpha (`clearColor = 0x000000`,
+`clearAlpha = 0`; three.module.js:13615), and `setClearColor` delegates to it (three.module.js:26154).
+**[measured]** counterfactual on a bare renderer: `setClearColor(0xBBBBFF, 0)` then a restore event →
+`getClearColor` returns `(0, 0, 0)`. Plain renderer properties (`autoClear`, `toneMapping`,
+`outputEncoding`, `sortObjects`, `info.autoReset`, `shadowMap.*`) do survive — three preserves the
+shadow-map and `info.autoReset` fields explicitly (three.module.js:26250).
+
+Fix: `configureThreeRenderer` is exported and re-applied on restore, along with `enableExtensions()`,
+`debug.checkShaderErrors` and `updateSize()`.
+
+### 4.2 Render-target contents that are not rebuilt every frame — FIXED
+
+Every `WebGLRenderTarget` construction site (16 files) was audited. In the shipped configuration all are
+re-rendered each frame (G-buffer, normal buffer, composit layers, outline, AO, shadow maps) **except** the
+PMREM environment map: `load_and_set_cubemap_v0` → `load_environment_map` baked a `WebGLCubeRenderTarget`
+once, assigned `scene.environment`, and then `dispose()`d the source cube texture, dropping the only
+reference — so the bake could not be repeated. **[measured]**: losing it costs 59% of scene luminance, and
+`scene.environment = null` on a live renderer is pixel-identical to a fresh context, which is how the
+defect was isolated.
+
+Fix: the graphics engine owns the bake. `set_environment_map(texture, projection)` retains the source and
+produces the prefiltered radiance; `set_environment_texture` adds the skybox on top; the restore handler
+re-bakes and disposes the previous target. `load_and_set_cubemap_v0` now loads the source and hands it
+over instead of baking behind the engine's back.
+
+### 4.3 Raw GL handles and residency bookkeeping — library tier, not in the shipped game
+
+These are `meep` public surface but are not reachable from `makeMirEngineConfig`, so they are correctness
+debt rather than shipping blockers (Phase 6):
+
+| Subsystem | What breaks | File |
+| --- | --- | --- |
+| Virtual texture | page atlas written with `copyTextureToTexture`; CPU residency map still claims pages are valid → garbage tiles forever | [VirtualTexturePage.js:389](src/engine/graphics/texture/virtual/VirtualTexturePage.js:389) |
+| `WebGLTextureAtlas` | same pattern | [WebGLTextureAtlas.js:178](src/engine/graphics/texture/atlas/gpu/WebGLTextureAtlas.js:178) |
+| Node-based particles | raw `gl.createBuffer()` handles + cached `gl`, no invalidation | [GLDataBuffer.js:48](src/engine/graphics/particles/node-based/simulation/GLDataBuffer.js:48), [TransformFeedback.js:414](src/engine/graphics/particles/node-based/simulation/TransformFeedback.js:414) |
+| Hi-Z occlusion | readback of a lost target yields zeroes → whole-frame visibility misclassification | [BatchOcclusionQuery.js:149](src/engine/graphics/render/visibility/hiz/query/BatchOcclusionQuery.js:149) |
+| Impostors | baked atlases are GPU-only | [ImpostorBaker.js](src/engine/graphics/impostors/octahedral/ImpostorBaker.js) |
+
+### 4.4 Secondary contexts — Phase 4
+
+A GPU-process crash kills every context on the page at once, so recovery is ultimately a page-level
+concern. The secondary contexts, in priority order:
+
+* **`computeImageBitmapEquality`** — caches a module-level WebGL2 context plus **one texture and one
+  framebuffer created once** ([computeImageBitmapEquality.js:43](src/engine/asset/loaders/material/computeImageBitmapEquality.js:43)).
+  After a loss those handles are dead and never recreated, so `bitmap2sampler_gl` silently reads zeroes and
+  texture dedup starts declaring unrelated textures identical. This is the worst failure mode found:
+  silent, wrong, and it corrupts CPU-side results rather than just pixels. Worth fixing on its own merits.
+* **`MinimapWorldGL`** — holds a pooled renderer for the lifetime of the minimap
+  ([MinimapWorldGL.js:124](src/view/minimap/gl/MinimapWorldGL.js:124)); it has a full re-render path, so
+  recovery is a re-render, not a rebuild.
+* **`MeshPreview`** — takes a renderer per linked view. Editor-level feature, low priority.
+
+`WebGLRendererPool` itself is not a significant risk: most of the engine takes a supplied renderer rather
+than reaching into the pool, and the remaining callers are short-lived bakes. (Noted in passing while
+reading it: `useAsync()` drops the callback's return value — `await` without `return`,
+[RendererPool.js:74](src/engine/graphics/render/RendererPool.js:74). `buildLightTexture` happens to rely
+only on the side effect, so nothing is broken today.)
+
+### 4.5 Async work in flight across a loss — partly addressed
+
+Bakes that straddle a loss complete with garbage and nobody notices: terrain AO
+([BuildLightTexture.js:31](src/engine/ecs/terrain/ecs/BuildLightTexture.js:31)), impostor bakes,
+`compileAllMaterials` during `transitionToScene`. The environment bake is now immune because it is repeated
+on restore from a retained source; the general mechanism is the context generation counter
+(`graphics.context.generation`), which async work can compare against the generation it started in. No
+existing bake uses it yet — wiring them is Phase 4 work, and there is no way to detect the problem by
+catching, because a lost context does not throw.
+
+---
+
+## 5. What was built
+
+### 5.1 `WebGLContextMonitor` — the state machine
+
+`src/engine/graphics/context/WebGLContextMonitor.js`, with `WebGLContextState` (Live / Lost / Failed) and
+`WebGLContextFailureReason` (`restore-timeout` / `too-many-losses`).
+
+* **Push, not poll.** No `gl.isContextLost()` per frame, no per-resource generation comparison in hot
+  loops. Consumers subscribe; the per-frame cost is one field compare (`monitor.is_live`).
+* **No separate `Restoring` state.** The monitor always calls `preventDefault()`, so `Lost` already means
+  "gone, waiting for the browser", and the fix-ups run synchronously inside the restore handler. Three
+  states, no observable in-between.
+* **Generation counter** — monotonic, bumped per restore, never reset (even when re-attaching to a new
+  canvas), so stale async results are always recognisable.
+* **Bounded recovery**: `restore_timeout` (default 5 s) escalates to `Failed`; `max_loss_count` (default 3)
+  within `loss_window` (default 60 s) does too. The window matters: a session that hits one unrelated loss
+  every couple of hours should keep recovering, one that dies four times in a minute should not.
+* **Terminal means terminal** — a restore that arrives after `Failed` is ignored, because the application
+  has already been told the device is gone and may have torn down presentation state.
+* **No new virtual hooks.** Deliberately not adding `onContextRestored()` no-ops to `FrameBuffer`,
+  `EnginePlugin`, `System` and friends: that would spread an empty method across dozens of classes for a
+  handful of real subscribers.
+
+### 5.2 Graphics engine wiring
+
+`GraphicsEngine` owns the monitor (`graphics.context`) and re-emits ordered engine-level signals
+`on.contextLost` / `on.contextRestored(generation)` / `on.contextFailed(reason)` — dispatched *after* its
+own fix-ups, so subscribers can render immediately. `render()` returns early while the context is not
+live, which is also what keeps meep's unguarded `setRenderTarget`/`clear`/readback calls away from a dead
+context. Loss, restore and failure are recorded through `globalMetrics`, next to the existing `gpu-type`
+metric, because without telemetry we will never learn the real-world loss rate.
+
+Restore sequence, in order: `configureThreeRenderer` → `debug.checkShaderErrors` → `enableExtensions()` →
+`updateSize()` → re-bake environment → `on.contextRestored`. Explicitly *not* included:
+`initializeFrameBuffers()` or any plugin startup (§3 corollary).
+
+### 5.3 Simulation and failure policy
+
+`Engine` subscribes to the three signals. Rendering stops immediately, but the simulation keeps running for
+the first `CONTEXT_LOSS_SIMULATION_PAUSE_DELAY` (2 s) — most losses resolve in well under a second and
+stopping the world for those is more disruptive than letting it run. A longer outage pauses the ticker via
+`Ticker.pause()`, which freezes the clock so the first step after resuming is normal-sized (`Clock.getDelta`
+does not clamp, so pausing without that would hand physics one enormous step). A ticker that somebody else
+already paused is left alone, and never resumed by us — hence the new `Ticker.is_active` / `Ticker.is_paused`
+accessors, since `pause()`/`resume()` throw when the loop is not cycling.
+
+On `Failed`: the simulation is paused, application state is left intact (so a save-on-failure handler
+stays viable), and `WebGLContextFailureView` explains the situation with a reload button, instead of
+leaving the player looking at a black canvas.
+
+---
+
+## 6. Performance impact
+
+**Steady state (no loss):**
+
+| Cost | Magnitude |
+| --- | --- |
+| Per-frame gate in `GraphicsEngine.render()` | one field compare; unmeasurable |
+| Signal subscriptions | 4 subscribers, invoked only on context events |
+| Extra retained memory | the environment source texture (~24 KB for the shipped 32² cubemap) |
+| Polling / `isContextLost()` | none, by design |
+| Hot-path branches added elsewhere | none |
+
+**During recovery [measured]** — intrinsic to context loss, not to this design:
+
+| Scene | First frame after restore | Steady state |
+| --- | --- | --- |
+| Harness (terrain + metal sphere + sun/shadows, 5 programs) | 35–48 ms | ~0.6 ms |
+| Game title scene (14 programs) | 214 ms | — |
+
+The dominant term is shader recompilation plus lazy texture re-upload, both driven by three.js as materials
+are drawn. A one-off couple of hundred milliseconds is well inside what the failure overlay's absence
+implies (nothing is shown; the frame simply arrives late). If a larger scene turns out to stall long enough
+to be objectionable, the existing time-sliced `compileAllMaterials` task is the lever — it is already used
+on scene transitions — but on this evidence pre-warming is not worth its complexity yet.
+
+---
+
+## 7. Complexity
+
+~640 lines added across 8 files, of which ~250 are the state machine plus its 12 tests and ~340 are the
+harness. No new abstractions imposed on unrelated code, no base-class churn, no per-resource generation
+plumbing:
+
+| File | Change |
+| --- | --- |
+| `graphics/context/WebGLContextState.js` | new, enum |
+| `graphics/context/WebGLContextFailureReason.js` | new, enum |
+| `graphics/context/WebGLContextMonitor.js` | new, state machine |
+| `graphics/context/WebGLContextMonitor.spec.js` | new, 12 tests |
+| `graphics/context/testWebGLContextLoss.js` | new, browser harness |
+| `graphics/GraphicsEngine.js` | monitor wiring, render gate, restore fix-ups, environment retention/re-bake |
+| `graphics/load_and_set_cubemap_v0.js` | hand the source to the engine instead of baking |
+| `Engine.js` | ticker policy, failure view, logging |
+| `simulation/Ticker.js` | `is_active` / `is_paused` accessors |
+| `view/graphics/WebGLContextFailureView.js` | new, failure overlay |
+
+---
+
+## 8. Adversarial review
+
+Each claim challenged, with what would falsify it, and what the experiments actually showed.
+
+**#1 "three re-uploads everything automatically, so most of the engine is already recoverable."**
+This was the load-bearing claim and it rested on reading `initGLContext()`. It could have failed if a
+texture's CPU source had been released, if a `Texture` subclass cached upload state outside `properties`,
+or if three left a dangling internal reference across restore. `_transmissionRenderTarget` is a closure
+variable *not* reset by `initGLContext()` — inspected, and it is re-allocated lazily through the fresh
+properties map, so it is fine, but it is exactly the shape of bug that would have invalidated the claim
+elsewhere. **Outcome: confirmed by pixels**, twice (fresh context, and synthetic restore). A grep would
+not have been enough; the fresh-context experiment is what turned it into evidence.
+
+**#2 "Nothing in the shipped game holds GPU-only content except PMREM."**
+Established by grepping construction sites and reading each hit — a static audit that could have missed a
+dirty-flag-driven cache that looks per-frame in code and behaves as cached at runtime. Shadow maps were
+specifically checked for this. **Outcome: confirmed for the harness scene**, and confirmed sharply: the
+fresh-context delta was *exactly* reproduced by setting `scene.environment = null` on the live renderer
+(34.70 vs 34.71 luminance, identical region breakdown), which leaves no room for a second hidden defect in
+that scene. Still unproven for content the harness scene does not exercise — FoW, decals, particles,
+Forward+ clusters, water — because the game's own scenes cannot be driven in a hidden tab (§10.4). Those
+are all CPU-sourced by inspection, which is weaker than a measurement.
+
+**#3 "In-place restore is sufficient."**
+Still the weakest assumption, and the experiments *sharpened* the worry rather than settling it:
+`preventDefault()` makes the browser *allowed* to restore, not obliged to, and in this environment it
+never did (§10.4) — for an environmental reason (hidden tab), but the failure mode was real and the code
+had to handle it. **Outcome: the fallback ladder is baseline, and it fired correctly** — timeout →
+`Failed` → overlay → simulation paused. Whatever the browser decides, the player ends up with a coherent
+state rather than a black screen.
+
+**#4 "Gating `GraphicsEngine.render()` stops all GL traffic during loss."**
+Not strictly: ECS-tick-driven work (Forward+ cluster uploads, trail/geometry updates) can still touch the
+renderer, and only three's `render()` is guarded internally. Analysis said this is benign because GL calls
+on a lost context are no-ops — **[measured]** `gl.getError()` returns `CONTEXT_LOST_WEBGL` (0x9242) and
+five driven frames threw nothing. There *was* a latent trap here: a `setupRenderTarget` during loss writes
+`__webglFramebuffer = null` into three's properties map, and `setRenderTarget` tests `=== undefined`, so
+that target would never be re-set-up again. It is unreachable today because `initGLContext()` replaces the
+map wholesale, and now doubly unreachable because the gate stops us drawing at all. Recorded so nobody
+"optimises" the gate away.
+
+**#5 "Readbacks during loss are harmless."**
+True for the shipped set (no production `readRenderTargetPixels`), false in general — §4.4's
+`bitmap2sampler_gl` turns a lost context into silently wrong *CPU* data. The general rule adopted: any
+GPU→CPU readback must be generation-checked, because "returns zeroes" is indistinguishable from "returns a
+valid black frame". This is also why there is no `try`/`catch` anywhere in the recovery path: a lost
+context does not throw, so exception handling would buy nothing.
+
+**#6 "The renderer pool is a likely cause of loss."**
+Corrected by the code owner: most of the engine works with a supplied renderer rather than reaching into
+the pool, and `MeshPreview` — the one per-instance consumer — is an editor feature. The context-cap theory
+was reasoned from documented Chrome behaviour, not observed, and it is not where the risk lives. Telemetry
+(§5.2) ships with detection so the real loss causes can be measured instead of guessed.
+
+**#7 "Recovery cost is dominated by shader recompilation."**
+**Outcome: consistent with measurement but not isolated.** First-frame cost scales with program count
+(5 programs → ~37 ms, 14 → ~214 ms), which fits, but texture upload was not separately instrumented, so
+this remains an inference. It matters only if pre-warming is ever built, which the numbers do not
+currently justify.
+
+**#8 "Re-running startup would be a simpler design."**
+Tempting and wrong: `FrameBuffer.initialize()` and plugin `startup()` allocate per call, so per-loss leaks
+would accumulate and a loss loop would exhaust GPU memory — turning a recoverable event into a hard crash.
+**[measured]** with the narrow sequence: three consecutive cycles held frame buffers at 2, programs at 5,
+textures at 6, and the environment target was replaced (id 31 → 33 → 35) rather than accumulated.
+
+**#9 "Test strategy is weak by nature."**
+Partly resolved. Jest cannot exercise WebGL, so the 12 unit tests cover the state machine only — using a
+real `HTMLCanvasElement` and real event dispatch, so nothing is mocked. Pixel-level behaviour is covered by
+the harness. The residual gap is honest and stated in §10.4: `WEBGL_lose_context` is a *cooperative* loss
+and does not reproduce a real GPU-process crash (all contexts dying at once, driver reset, restore refusal).
+
+**#10 "The `computeImageBitmapEquality` context produces silent garbage after a loss."**
+Still untested — it needs its own harness pass, deferred with the rest of Phase 4. If the browser instead
+fails loudly there, the item drops in priority.
+
+**#11 New: the loss budget was wrong when first written.**
+The original policy failed permanently after 3 losses *per session*, which would punish a long session
+that hit four unrelated hiccups hours apart. Caught while reviewing the 3-cycle measurement (which ended
+one loss short of tripping it) and replaced with a 3-per-60-seconds window. Worth recording as an argument
+for measuring your own defaults: the bug was invisible in code review and obvious in a graph.
+
+---
+
+## 9. Non-goals
+
+* Recovering the *contents* of anything genuinely transient (one frame of temporal history, in-flight
+  bakes) — re-derive, do not restore.
+* WebGPU migration, or making the engine context-agnostic.
+* Surviving a browser tab crash or an OOM abort.
+* Deleting the unused GL-direct subsystems in §4.3 — they are public library surface; they get fixed, not
+  removed.
+
+---
+
+## 10. Verification
+
+### 10.1 Unit tests
+
+`WebGLContextMonitor.spec.js`, 12 tests, real canvas and real event dispatch: initial state, loss →
+`preventDefault` + `Lost`, restore → `Live` + generation increment, restore-timeout escalation, deadline
+cancellation, burst budget exhaustion, budget decay over an hour of spaced losses, re-attach semantics,
+terminal-state behaviour on late restore and on further losses, detach.
+
+```bash
+npx jest -c jest.conf.json app/src/mir-engine/meep/src/engine/graphics/context/WebGLContextMonitor.spec.js
+```
+
+### 10.2 Browser harness
+
+`src/engine/graphics/context/testWebGLContextLoss.js`, run by pointing the webpack entry at it (the entry
+line is already present, commented, in `webpack.config.js`). Static scene — terrain, sun with shadows, and
+a mirror-metal sphere whose shading is almost entirely IBL, which is what makes the environment map visible
+in a pixel A/B. Two consecutive frames are bit-identical, so the noise floor is zero and any difference is
+real. Because the harness is expected to run in a hidden tab it drives frames by hand
+(`engine.ticker.onTick.send1(dt)` + `engine.render()`), reads the framebuffer in the same task as the draw,
+and can print a coarse ASCII luminance map — which is how the environment defect was localised without
+screenshots.
+
+| API | Scenario | Runs in a hidden tab |
+| --- | --- | --- |
+| `__context_loss_test.synthetic()` | loss/restore lifecycle via synthetic canvas events; exercises the gate, the fix-up sequence, the environment re-bake and three's own restore path | yes |
+| `__context_loss_test.fresh()` | same scene through a brand new renderer/context; measures what has to come back from CPU-side data | yes |
+| `__context_loss_test.run()` | real `WEBGL_lose_context` loss and restore | needs a visible tab |
+| `__context_loss_test.repeat(n)` | n real cycles, watching for resource growth | needs a visible tab |
+| `image()`, `capture()`, `state()`, `frame(n)` | inspection helpers | yes |
+
+### 10.3 Results
+
+| Check | Result |
+| --- | --- |
+| Frames drawn while lost | 0, no exceptions, `gl.getError()` = `CONTEXT_LOST_WEBGL` |
+| Synthetic restore, harness scene | bit-identical to baseline, including IBL |
+| Three consecutive cycles | cycles 2 and 3 bit-identical; frame buffers 2, programs 5, textures 6 unchanged; environment target replaced, not accumulated |
+| Clear colour after restore | 0xBBBBFF preserved (counterfactual on a bare renderer: resets to black) |
+| Fresh context | identical draw calls / triangles; only delta is the environment map, reproduced exactly by `scene.environment = null` |
+| Real loss, no restore possible (hidden tab) | escalated to `Failed` after the 5 s deadline, overlay shown, no exceptions. Superseded: the deadline now only runs while the page is visible (see the note below the table), so this scenario stays `Lost` until the page is revealed, then fails 5 s later |
+| Production game bundle, synthetic cycle | frame counter frozen while lost; generation 0 → 1; environment re-baked (texture id 24 → 77); clear colour preserved; no GL errors; first frame 214 ms |
+
+Follow-up (post-review): the restore deadline is now suspended while the page is hidden. Chrome does not
+restore a context in a hidden tab, so counting hidden time turned any loss in a backgrounded tab — the
+place where the browser prefers to evict contexts — into a terminal `Failed`, shown to the user on their
+return even though the context was recoverable. `WebGLContextMonitor` now arms the watchdog only while
+the page is visible; each return to visibility restarts the full `restore_timeout`. Covered by three
+visibility tests in `WebGLContextMonitor.spec.js`.
+
+### 10.4 What remains unverified, and why
+
+The Browser pane available here does not composite, and Chrome only restores a lost context while the page
+is visible, so `webglcontextrestored` never arrives — a real loss escalates to `Failed` instead of
+recovering. Consequently these need one run in a visible tab:
+
+1. **A real browser-driven restore** (`__context_loss_test.run()` and `.repeat(3)`): does the browser hand
+   the context back, and do stale GL handles from the dead context misbehave? Everything downstream of the
+   restore event is already proven by the synthetic path; what is untested is the event itself.
+2. **The prolonged-outage ticker pause.** In a hidden tab the ticker loop never bootstraps
+   (`requestAnimationFrame` never fires, so `is_active` is false) and the guard correctly declines to pause
+   — which exercised the guard but not the pause.
+3. **The game's own scenes.** The game boots in a hidden tab but never renders, so per-frame GPU state in
+   FoW, decals, particles, Forward+ and water is covered by inspection only.
+
+To run them: point the webpack entry at the harness, open the page with the pane visible, then
+
+```bash
+npx jest -c jest.conf.json app/src/mir-engine/meep/src/engine/graphics/context/WebGLContextMonitor.spec.js
+```
+
+and in the page console: `await __context_loss_test.run()`, then `await __context_loss_test.repeat(3)`.
+Expect `restored: true`, `settled_vs_baseline` near zero, and no growth in `textures` / `geometries` /
+`programs` across cycles.
+
+---
+
+## 11. Remaining phases
+
+**Phase 4 — secondary contexts and async generation checks.** `computeImageBitmapEquality` handle reset
+(§4.4, the silent-corruption one, worth doing on its own), `MinimapWorldGL` re-render on restore, and
+generation checks on the long-running bakes (§4.5). `MeshPreview` is editor-only and can trail. Estimate
+1–2 days.
+
+**Phase 5 (conditional) — renderer rebuild fallback.** If telemetry shows browsers refusing to restore in
+practice, replace the terminal `Failed` state with a rebuild: dispose the renderer, create a new canvas,
+re-attach `viewport.el` / `graphics.domElement` / styles / size, and re-run frame-buffer init *with*
+disposal of the old targets. The monitor already supports it — `attach()` resets the loss budget and returns
+to `Live` while keeping the generation counter monotonic. Only worth building on evidence.
+
+**Phase 6 (deferred) — library tier.** VT and GPU-atlas residency invalidation, node-based particle GL
+buffers, Hi-Z, impostor atlases (§4.3). Each is self-contained; do them when the subsystem is next touched.

package/src/engine/graphics/GraphicsEngine.d.ts

@@ -1,3 +1,14 @@
+/**
+ * Applies the renderer configuration the engine relies on.
+ *
+ * Must be re-applied after a context restore: three.js rebuilds its internal modules from scratch in
+ * that case, and some of the settings live inside those modules rather than on the renderer itself -
+ * most notably the clear colour and alpha, which are owned by `WebGLBackground` and silently revert to
+ * transparent black.
+ *
+ * @param {WebGLRenderer} webGLRenderer
+ */
+export function configureThreeRenderer(webGLRenderer: WebGLRenderer): void;
 export class GraphicsEngine {
     /**
      *
@@ -6,6 +17,13 @@ export class GraphicsEngine {
      * @constructor
      */
     constructor({ camera, debug }: Camera);
+    /**
+     * Tracks WebGL context loss. Rendering is suppressed while the context is gone, and GPU-side state
+     * that is not re-derived every frame is rebuilt when it comes back.
+     * @readonly
+     * @type {WebGLContextMonitor}
+     */
+    readonly context: WebGLContextMonitor;
     get isGraphicsEngine(): boolean;
     /**
      *
@@ -29,6 +47,25 @@ export class GraphicsEngine {
         buffersRendered: Signal;
         visibilityConstructionStarted: Signal;
         visibilityConstructionEnded: Signal;
+        /**
+         * WebGL context has been lost. No rendering happens until the context comes back; GPU-side
+         * resources must be treated as gone.
+         * @type {Signal}
+         */
+        contextLost: Signal;
+        /**
+         * WebGL context has been restored, carries the context generation number. Dispatched after
+         * the graphics engine has re-applied its own renderer state, so subscribers can render.
+         * Anything held on the GPU that is not rebuilt every frame has to be rebuilt here.
+         * @type {Signal<number>}
+         */
+        contextRestored: any;
+        /**
+         * Context recovery was abandoned, carries a {@link WebGLContextFailureReason}. Terminal -
+         * there will be no further rendering.
+         * @type {Signal<string>}
+         */
+        contextFailed: any;
     };
     /**
      * @type {Vector1}
@@ -124,19 +161,30 @@ export class GraphicsEngine {
      */
     getRenderer(): WebGLRenderer;
     /**
-     * Set the scene's environment map. The supplied texture is treated as BOTH:
-     *   - the skybox background (shown directly), and
-     *   - the image-based-lighting (IBL) source for physically based materials.
+     * Set the scene's image-based-lighting (IBL) source.
      *
-     * The texture is run through three.js' PMREMGenerator to produce the
-     * prefiltered radiance assigned to `scene.environment`, while the original
-     * texture is assigned to `scene.background`.
+     * The texture is run through three.js' PMREMGenerator to produce the prefiltered radiance assigned
+     * to `scene.environment`. The scene background is left alone, use
+     * {@link GraphicsEngine#set_environment_texture} to show the same texture as a skybox as well.
+     *
+     * The source texture is retained: the prefiltered radiance lives in a render target, which means its
+     * contents are lost along with the WebGL context, so the bake has to be repeatable. The bake is
+     * re-run automatically on context restore.
      *
      * Must be called after the renderer has started up.
      *
      * @param {THREE.Texture} texture environment texture (equirectangular or cube)
      * @param {EnvironmentTextureProjection} [projection=EnvironmentTextureProjection.Equirectangular] how the texture is projected
      */
+    set_environment_map(texture: THREE.Texture, projection?: EnvironmentTextureProjection): void;
+    /**
+     * Set the scene's environment map. The supplied texture is treated as BOTH:
+     *   - the skybox background (shown directly), and
+     *   - the image-based-lighting (IBL) source for physically based materials.
+     *
+     * @param {THREE.Texture} texture environment texture (equirectangular or cube)
+     * @param {EnvironmentTextureProjection} [projection=EnvironmentTextureProjection.Equirectangular] how the texture is projected
+     */
     set_environment_texture(texture: THREE.Texture, projection?: EnvironmentTextureProjection): void;
     updateSize(): void;
     /**
@@ -207,13 +255,14 @@ export class GraphicsEngine {
     render(): void;
     #private;
 }
+import { WebGLRenderer } from "three";
+import { WebGLContextMonitor } from "./context/WebGLContextMonitor.js";
 import Signal from "../../core/events/signal/Signal.js";
 import Vector1 from "../../core/geom/Vector1.js";
 import { Vector2 } from "../../core/geom/Vector2.js";
 import { RenderLayerManager } from "./render/layers/RenderLayerManager.js";
 import { Group } from "three";
 import { Camera } from "./ecs/camera/Camera.js";
-import { WebGLRenderer } from "three";
 import LayerCompositer from "./composit/LayerCompositer.js";
 import { FrameBufferManager } from "./render/buffer/FrameBufferManager.js";
 import { ShadowMapRenderer } from "./shadows/ShadowMapRenderer.js";

package/src/engine/graphics/GraphicsEngine.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"GraphicsEngine.d.ts","sourceRoot":"","sources":["../../../../src/engine/graphics/GraphicsEngine.js"],"names":[],"mappings":"AAiEA;IAOI;;;;;OAKG;IACH,+BAJW,MAAM,EAoLhB;IA1LD,gCAEC;IAgBG;;;;OAIG;IACH,2BAA+C;IAK/C;;;;;QAOI;;WAEG;;QAEH;;WAEG;;;;;MAMN;IAED;;OAEG;IACH,YAFU,OAAO,CAEe;IAEhC;;;;;;;;;;OAUG;IACH,4BAFU,OAAO,CAEyB;IAE1C;;;OAGG;IACH,QAFU,kBAAkB,CAEU;IAWtC;;;OAGG;IACH,aAAkB;IAGlB;;;OAGG;IACH,cAFU,KAAK,CAEgB;IAK/B;;;OAGG;IACH,QAFU,MAAM,CAEI;IAEpB;;;OAGG;IACH,UAFU,aAAa,CAEH;IAGpB,+BAA0C;IAU1C;;OAEG;IACH,eAA+B;IAI/B;;;OAGG;IACH,uBAFU,kBAAkB,CAEgB;IAE5C;;;OAGG;IACH,oBAFU,iBAAiB,CAEsB;IAEjD;;;OAGG;IACH,gBAFU,iBAAiB,CAES;IAEpC;;;OAGG;IACH,sBAAoC;IAGpC;;;;OAIG;IACH,UAFU,OAAO,CAEG;IAEpB;;;OAGG;IACH,UAFU,OAAO,CAEG;IAEpB;;;OAGG;IACH,YAFU,MAAM,CAEG;IACnB,kKAcI;IAGR;;;OAGG;IACH,sBAFa,eAAe,CAI3B;IAED;;;OAGG;IACH,eAFa,aAAa,CAIzB;IAED;;;;;;;;;;;;;OAaG;IACH,iCAHW,MAAM,OAAO,eACb,4BAA4B,QAkCtC;IAED,mBAqCC;IAED;;;;OAIG;IACH,iCAEC;IAED;;;OAGG;IACH,0BAFa,MAAM,CAIlB;IAED,+BAcC;IAED,cAgEC;IAzBsB,8BAA0C;IA2BjE;;;OAGG;IACH,cAFa,MAAM,CAYlB;IAED;;OAEG;IACH,aAQC;IAED;;OAEG;IACH,yBAQC;IAED;;;;;;OAMG;IACH,yBALW,MAAM,KACN,MAAM,6CAMhB;IAED;;;;OAIG;IACH,8BAHW,OAAO,UAAQ,UACf,OAAO,UAAQ,QAczB;IAED;;;;;OAKG;IACH,8BAQC;IAED,0BAEC;IAED;;;OAGG;IACH,4BAFW,cAAc,QA8BxB;IAED;;OAEG;IACH,qBAqBC;IAED;;OAEG;IACH,0BAeC;IAED;;OAEG;IACH,eA8CC;;CACJ;mBAxpBkB,oCAAoC;oBACnC,4BAA4B;wBACxB,4BAA4B;mCAajB,uCAAuC;sBAlBnE,OAAO;uBAaS,wBAAwB;8BAbxC,OAAO;4BAYc,+BAA+B;mCAKxB,uCAAuC;kCAKxC,gCAAgC;kCADhC,oCAAoC;gCAPtC,uCAAuC;6CAW1B,2CAA2C;+BANzD,4BAA4B"}
+{"version":3,"file":"GraphicsEngine.d.ts","sourceRoot":"","sources":["../../../../src/engine/graphics/GraphicsEngine.js"],"names":[],"mappings":"AAyCA;;;;;;;;;GASG;AACH,sDAFW,aAAa,QAoBvB;AAGD;IAyCI;;;;;OAKG;IACH,+BAJW,MAAM,EAwMhB;IA7ND;;;;;OAKG;IACH,kBAFU,mBAAmB,CAEO;IASpC,gCAEC;IAgBG;;;;OAIG;IACH,2BAA+C;IAK/C;;;;;QAOI;;WAEG;;QAEH;;WAEG;;;;;QAOH;;;;WAIG;qBADO,MAAM;QAGhB;;;;;WAKG;;QAEH;;;;WAIG;;MAEN;IAED;;OAEG;IACH,YAFU,OAAO,CAEe;IAEhC;;;;;;;;;;OAUG;IACH,4BAFU,OAAO,CAEyB;IAE1C;;;OAGG;IACH,QAFU,kBAAkB,CAEU;IAWtC;;;OAGG;IACH,aAAkB;IAGlB;;;OAGG;IACH,cAFU,KAAK,CAEgB;IAK/B;;;OAGG;IACH,QAFU,MAAM,CAEI;IAEpB;;;OAGG;IACH,UAFU,aAAa,CAEH;IAGpB,+BAA0C;IAU1C;;OAEG;IACH,eAA+B;IAI/B;;;OAGG;IACH,uBAFU,kBAAkB,CAEgB;IAE5C;;;OAGG;IACH,oBAFU,iBAAiB,CAEsB;IAEjD;;;OAGG;IACH,gBAFU,iBAAiB,CAES;IAEpC;;;OAGG;IACH,sBAAoC;IAGpC;;;;OAIG;IACH,UAFU,OAAO,CAEG;IAEpB;;;OAGG;IACH,UAFU,OAAO,CAEG;IAEpB;;;OAGG;IACH,YAFU,MAAM,CAEG;IACnB,kKAcI;IAGR;;;OAGG;IACH,sBAFa,eAAe,CAI3B;IAED;;;OAGG;IACH,eAFa,aAAa,CAIzB;IAED;;;;;;;;;;;;;;;OAeG;IACH,6BAHW,MAAM,OAAO,eACb,4BAA4B,QAStC;IAED;;;;;;;OAOG;IACH,iCAHW,MAAM,OAAO,eACb,4BAA4B,QAQtC;IA8CD,mBAqCC;IAED;;;;OAIG;IACH,iCAEC;IAED;;;OAGG;IACH,0BAFa,MAAM,CAIlB;IAED,+BAcC;IAED,cA6DC;IAzBsB,8BAA0C;IAgGjE;;;OAGG;IACH,cAFa,MAAM,CAYlB;IAED;;OAEG;IACH,aAUC;IAED;;OAEG;IACH,yBAQC;IAED;;;;;;OAMG;IACH,yBALW,MAAM,KACN,MAAM,6CAMhB;IAED;;;;OAIG;IACH,8BAHW,OAAO,UAAQ,UACf,OAAO,UAAQ,QAczB;IAED;;;;;OAKG;IACH,8BAQC;IAED,0BAEC;IAED;;;OAGG;IACH,4BAFW,cAAc,QA8BxB;IAED;;OAEG;IACH,qBAqBC;IAED;;OAEG;IACH,0BAeC;IAED;;OAEG;IACH,eAoDC;;CACJ;8BAv0BM,OAAO;oCAasB,kCAAkC;mBAVnD,oCAAoC;oBACnC,4BAA4B;wBACxB,4BAA4B;mCAcjB,uCAAuC;sBAnBnE,OAAO;uBAcS,wBAAwB;4BAFnB,+BAA+B;mCAMxB,uCAAuC;kCAKxC,gCAAgC;kCADhC,oCAAoC;gCAPtC,uCAAuC;6CAW1B,2CAA2C;+BANzD,4BAA4B"}