@woosh/meep-engine 2.169.4 → 2.170.0

package/src/engine/ecs/ASYNC_SYSTEMS_PROPOSAL.md

@@ -0,0 +1,531 @@
+# Asynchronous Systems — Design Proposal
+
+**Status:** proposal, 2026-08-14
+**Scope:** ECS `System`s whose work executes outside the main-thread tick — Web Workers and WebGPU — while keeping the authoring interface as small as possible.
+
+> **What was actually built.** The design space was narrowed after this document
+> to *shared memory* (`SharedArrayBuffer` + `Atomics`) rather than `postMessage`,
+> and that variant is implemented and shipping — see
+> [`async/WORKER_SYSTEM_PLAN.md`](async/WORKER_SYSTEM_PLAN.md), `async/WorkerSystem.js`,
+> and its first consumer `physics/fluid/ecs/WorkerFluidSystem.js`.
+>
+> Three things changed versus the sketch below, all for the better:
+>
+> - **`AsyncPump` collapsed into four shared integers.** With shared memory the
+>   pump's whole bookkeeping — in-flight, capacity, completion — is the control
+>   header, so there is no separate object to own it.
+> - **The gate became real, and load-bearing.** §7 deferred the fixed-latency
+>   contract as future work. Shared memory makes "is step T done?" a synchronous
+>   question, so `EntityManager` gained a fixed-step gate and the deterministic
+>   mode is the *only* mode rather than an upgrade path.
+> - **Capacity is fixed at one, not configurable.** Under in-place mutation the
+>   main thread has nowhere to build a second step's input, so a deeper pipeline
+>   is a different design rather than a knob.
+>
+> The invariant that motivated the whole document is unchanged and is what the
+> implementation enforces: **async transports, synchronous integration.**
+>
+> Everything below remains the reference for the parts not yet built — the
+> `postMessage` transport (`WorkerChannel`/`WorkerHost`) and the WebGPU tier.
+
+---
+
+## 1. Summary
+
+meep's `System` contract is already asynchronous where it needs to be for *lifecycle*: `startup`/`shutdown` return Promises, the `EntityManager` gates `RUNNING` on them, and entity linking only begins after startup resolves. What is missing is a story for *per-tick* asynchrony — systems whose computation is a round trip (postMessage → worker → response; submit → GPU → `mapAsync`) and therefore cannot complete inside the synchronous `simulate()` pass.
+
+The proposal is deliberately **not** to make the tick asynchronous. `EntityManager.simulate` stays exactly as it is. Instead:
+
+1. **One invariant**: async continuations never touch the `EntityComponentDataset`. They only enqueue results. All dataset reads and writes happen inside the system's normal synchronous slot in the tick, in system order.
+2. **One primitive**: `AsyncPump` — bookkeeping for in-flight round trips. Each tick it *drains* completed results (applying them), then *launches* new work if capacity allows. It owns tick-stamping, dataset-epoch invalidation, entity-generation staleness, backpressure, and shutdown settling.
+3. **One convenience base**: `AsyncSystem extends System` — wires a pump to `update()` and exposes a three-method contract: `collect()` → `process(request)` (async, off-tick) → `apply(result, request)` (sync, on-tick). Systems that already extend another base (`AbstractContextSystem` etc.) use the pump directly; the base class is ten lines of sugar.
+4. **Two transports**, both just "things `process()` can await":
+   - `WorkerChannel`/`WorkerHost` — a modernized worker RPC (module workers, transferables in both directions, ready handshake, injectable port for tests), superseding the codegen `WorkerBuilder` path for new code.
+   - WebGPU — a `WebGPUDeviceSystem` that owns adapter/device, plus a `GpuReadbackRing` for the only genuinely async part (readback). GPU-resident compute that feeds rendering needs **no new machinery at all** — async `startup` for pipeline creation plus synchronous `queue.submit` in `update` already works today.
+
+**Zero changes to `System`, `EntityManager`, or `EntityComponentDataset` are required.** Everything is additive library code under `engine/ecs/async/` (plus `core/process/worker/` for the channel). The execution-order scheduler keeps working unchanged, because an async system's dataset writes still happen inside its own `update` slot — `components_used` declarations remain truthful.
+
+---
+
+## 2. What exists today
+
+Survey of the current codebase (paths relative to `meep/src/`).
+
+### 2.1 The System contract is already async where it matters
+
+`engine/ecs/System.js` — `async startup(entityManager)` / `async shutdown(entityManager)`. `engine/ecs/EntityManager.js` (`startSystem`, ~line 682) asserts the returned thenable, arms a 5 s deadlock watchdog, and only sets `RUNNING` / attaches the `EntityObserver` / calls `handleDatasetAttached` after the promise resolves. The callback form was already deprecated in meep 2.128.0 in favor of `async`.
+
+This means **spawning a worker, handshaking, requesting a GPU device, and compiling pipelines already have a correct home**: `startup`. Nothing new is needed for lifecycle.
+
+### 2.2 The tick is synchronous — and must stay so
+
+`EntityManager.simulate` (~line 423): a lock-step fixed-update loop over all systems on a single shared accumulator (with `fixedStepTick`, the monotonic simulation tick id, incremented inside the loop), followed by the variable-rate `update` pass. Per-system try/catch, wall-clock runaway guard. Simulation and rendering run on **separate loops** (`engine/simulation/Ticker.js` drives `simulate`; `Engine.#animation_frame` drives `render`).
+
+Two consequences worth stating explicitly, because the design leans on them:
+
+- Because `simulate()` is one synchronous call, **promise continuations can never run between two systems within a pass**, nor between fixed steps. The JS microtask queue only drains when the stack empties.
+- Continuations **can** run between `simulate` and `render`, and between frames. Today, any code that applies worker results inside a `.then(...)` mutates components at an uncontrolled point relative to both loops.
+
+### 2.3 Worker infrastructure exists, one generation old
+
+`core/process/worker/`:
+
+- `WorkerProxy` — RPC client over `{methodName, id, parameters}` / `{methodName, id, result|error}`. Queues requests while stopped, rejects everything on worker error (`FAILED` is terminal). `cancelRequest` throws "not implemented". **Transferables are only used on the response path** — requests always structured-clone.
+- `WorkerBuilder` — builds the worker by **string-concatenating code** into a Blob URL (`codeToURL`), with an inlined duplicate of `extractTransferables`. Generated workers use `importScripts` (classic worker semantics) to load rollup bundles that are emitted as `format: 'esm'` — this works only because the entry scripts happen to export nothing.
+- `OnDemandWorkerManager` — idle-termination wrapper (start on demand, stop after a grace period).
+
+Consumers: PNG decode (`ThreadedImageDecoder`), terrain tile builds (`makeTerrainWorkerProxy`, one worker **per `Terrain` component**), and the deprecated `Metis` binding. A fourth worker entry (`sh3/lpv/worker/lpv_build_worker.js`) is dead/unfinished.
+
+`SharedArrayBuffer` has a seam (`core/binary/makeArrayBuffer.js`, gated on `crossOriginIsolated`) but is never actually used with a worker. No `Atomics`, no `OffscreenCanvas`.
+
+### 2.4 The scheduler and its Promise bridge
+
+`core/process/executor/ConcurrentExecutor.js` + `core/process/task/Task.js` — cooperative, single-threaded, time-sliced (`setTimeout`-driven, not frame-synced; `Engine.executor = new ConcurrentExecutor(1, 10)`). This is meep's answer to *main-thread* long-running work, and it is the dominant pattern: the lightmap baker, LPG builder, `ThemeEngine`, and `ShadedGeometrySystem`'s standing optimization/maintenance tasks all run as `Task`/`TaskGroup` cycles.
+
+`promiseTask` (`core/process/task/util/promiseTask.js`) is the existing Promise→Task bridge — it polls with `TaskSignal.Yield` and discards the resolved value. It bridges *completion*, not *results*.
+
+### 2.5 Result-delivery precedents — the design in embryo
+
+Two places already solve pieces of this problem, differently:
+
+- **`AssetManager`** (`engine/asset/AssetManager.js`): async results are never delivered from whatever stack settled them. They go into `#response_queue`, drained by a `#response_processor` Task on the executor, with a microtask-deferred wake. This is precisely the "continuations only enqueue" invariant — applied to asset loads.
+- **`TerrainTileManager.build`** (~line 748): the worker round trip resolves into a `.then` that hand-rolls three staleness guards — tile identity (`this.tiles[tile_index] !== tile`), a monotonic `tile.version`, and an `isBuildInProgress` flag — then mutates main-thread objects **directly in the promise callback**. It works, but every guard is bespoke, and delivery timing is uncontrolled (§2.2).
+
+The proposal is essentially: take what `AssetManager` and `TerrainTileManager` each got right, and make it the standard, so the next bridged system doesn't reinvent either half.
+
+### 2.6 WebGPU: greenfield
+
+There is **no WebGPU code** in `meep/src/` — no `navigator.gpu`, no WGSL, nothing. (The build already has a `.wgsl` raw-loader rule in the app webpack config, so shader imports are ready.) All existing GPU→CPU readback is synchronous WebGL (`readPixels`, `getBufferSubData`) inside utilities, not systems. WebGPU support is a clean-slate design, constrained only by three.js interop at the rendering seam.
+
+---
+
+## 3. Problem statement
+
+Bridging a System to a Worker or a WebGPU queue means every tick-relevant computation becomes a **round trip with ≥ 1 frame of latency**. Concretely, an author today has to solve, per system, with no engine help:
+
+1. **When do results land?** Ad hoc `.then(apply)` mutates the dataset between `simulate` and `render`, or mid-frame relative to other systems' expectations. Nothing guarantees writes happen at the system's slot in execution order, so `components_used`-based ordering silently stops describing reality.
+2. **Is the result still valid?** Entities die, datasets get detached/attached (dataset switching is a supported scenario), systems get removed. The dataset has an entity **generation table** (`EntityComponentDataset.getEntityGeneration`) — ideal for this — but nothing uses it for async staleness; terrain invented three parallel guards instead.
+3. **Backpressure.** Nothing stops a system from launching a new request every tick while the previous one is still in flight. A slow worker turns into an unbounded queue (the network review found this exact failure mode in the pending-action log).
+4. **Cancellation/shutdown.** `WorkerProxy.cancelRequest` is unimplemented; nothing drains in-flight work at system shutdown; `Engine.stop()` has a standing `// TODO shutdown executor`.
+5. **Transport boilerplate.** Codegen workers, transferables on one path only, per-component worker instances, no ready handshake, no typed wrapper convention.
+
+And for WebGPU specifically: no device ownership story, no readback pattern, no device-loss handling.
+
+---
+
+## 4. Design principles
+
+1. **The tick stays synchronous.** Determinism, ordering, and the mental model of `simulate` are load-bearing (physics determinism gate, network lockstep design). Async work is *pipelined around* the tick, never awaited inside it.
+2. **Continuations never touch the dataset.** The single rule that eliminates the whole "results land at random times" bug class. Continuations enqueue; ticks apply.
+3. **Composition first.** Systems already extend intermediate bases (`AbstractContextSystem`, `HighlightSystem`). The primitive must be an object a system *owns*, with the base class as optional sugar.
+4. **Zero core changes.** `System`/`EntityManager`/`EntityComponentDataset` are untouched in v1. Everything is additive library code. (This also keeps the door open for a job-graph scheduler later — §11.)
+5. **Uniform control flow.** One delivery order (launch order), one staleness mechanism (epoch + generation), one failure path (reject → discard + loud log). No auto-retry, no auto-restart, no "null means auto" modes.
+6. **Transports are just Promises.** The pump doesn't know about workers or GPUs. `process()` returns a Promise; where the work happens is the subclass's business. This is what makes Worker and WebGPU the *same* feature.
+7. **Fail loudly.** Rejections and dead workers surface with system names via `console.error`, matching `simulate`'s per-system error style. No silent catch.
+
+---
+
+## 5. Proposal
+
+New package: `engine/ecs/async/`. Transport: `core/process/worker/v2/` (or sibling files; naming below is provisional).
+
+### 5.1 `AsyncPump` — the tick-integration primitive
+
+Owns the in-flight set and the settled-but-unapplied queue. All bookkeeping, no policy about *what* the work is.
+
+```js
+export class AsyncPump {
+
+    /** max round trips in flight; default 1 = coalescing pipeline */
+    capacity = 1;
+
+    #inflight = [];      // [{request, context, epoch, settled, result, error}] in launch order
+    #epoch = 0;
+
+    /**
+     * Launch a round trip. `promise` must not touch the dataset in its
+     * continuations — the pump only records settlement.
+     * @param {Promise} promise
+     * @param {*} context  opaque; returned to apply/discard (tick stamp, entity handles, ...)
+     */
+    launch(promise, context) { /* record; .then only sets settled/result flags */ }
+
+    /** @returns {boolean} capacity available for another launch */
+    get is_ready() { }
+
+    /**
+     * Deliver settled results IN LAUNCH ORDER to `apply(result, context)`.
+     * A settled head unblocks delivery; an unsettled head holds younger settled
+     * entries back (single-worker FIFO and mapAsync both settle in launch order
+     * anyway, so head-of-line blocking is theoretical for one transport).
+     * Entries from an older epoch, and rejected entries, go to
+     * `discard(context, error)` instead. Called from the system's update slot.
+     */
+    drain(apply, discard) { }
+
+    /** Invalidate everything currently in flight (dataset detached, reset). */
+    invalidate() { this.#epoch++; }
+
+    /**
+     * Stop accepting launches and resolve when all in-flight work settles
+     * (results are discarded — the dataset is already detached by the time
+     * shutdown runs, see §5.7). For use in System.shutdown.
+     * @returns {Promise<void>}
+     */
+    settle() { }
+}
+```
+
+Notes:
+
+- **Launch-order delivery** is a deliberate choice over completion-order: it makes multi-request pipelines reason like FIFO queues, and it costs nothing for the two real transports (a single worker responds in request order; `mapAsync` resolves in submit order). One rule, no configuration.
+- `capacity = 1` is the default and the most important policy: while a request is in flight, no new one launches; when it completes, the *next* launch snapshots *fresh* state. That is natural coalescing — the pattern `TerrainTileManager` approximates with `isBuildInProgress`, and the right default for "recompute X from current world state" workloads (path fields, GI, LODs).
+- The pump is deliberately dumb: no request queue (pending work is regenerated from world state by the next `collect`), no retry, no timers.
+
+### 5.2 `AsyncSystem` — the convenience base
+
+```js
+export class AsyncSystem extends System {
+
+    /** @readonly */
+    pump = new AsyncPump();
+
+    /**
+     * Build a request from current world state. Called only when the pump has
+     * capacity. Return undefined to launch nothing this tick.
+     * Runs on-tick: full dataset access.
+     * @returns {*} request
+     */
+    collect() { }
+
+    /**
+     * Perform the round trip. Runs OFF-tick: must not touch the dataset or
+     * entityManager. Typically: channel.request(...) or submit+mapAsync.
+     * @param {*} request
+     * @returns {Promise<*>} result
+     */
+    async process(request) { }
+
+    /**
+     * Integrate a completed result. Runs on-tick, inside this system's update
+     * slot, in execution order. Full dataset access. Validate entity handles
+     * here (§5.3).
+     * @param {*} result
+     * @param {*} request
+     */
+    apply(result, request) { }
+
+    /**
+     * A request whose result will never be applied: rejected, or invalidated
+     * by dataset detach. Default: log rejections, drop invalidations.
+     * @param {*} request
+     * @param {Error} [error]
+     */
+    discard(request, error) { }
+
+    update(dt) {
+        this.pump.drain(
+            (result, req) => this.apply(result, req),
+            (req, err) => this.discard(req, err),
+        );
+
+        if (this.pump.is_ready) {
+            const request = this.collect();
+            if (request !== undefined) {
+                this.pump.launch(this.process(request), request);
+            }
+        }
+    }
+
+    handleDatasetDetached(dataset) {
+        this.pump.invalidate();
+    }
+}
+```
+
+The authoring surface for a bridged system is **three methods** (`collect`/`process`/`apply`), all of which are independently testable — `process` doesn't need an ECS at all.
+
+Systems that can't extend `AsyncSystem` (e.g. `AbstractContextSystem` descendants) own an `AsyncPump` directly and put the same six lines in their `update`. Systems with several independent streams own several pumps. Systems that want fixed-cadence launching call the drain/launch pair from `fixedUpdate` instead — the pump doesn't care.
+
+Drain-then-launch order matters: results integrate first, so the subsequent `collect` sees post-apply state, and downstream systems in this tick's pass see the newly written components.
+
+### 5.3 Staleness: epochs and entity handles
+
+Two mechanisms, two failure modes:
+
+- **Dataset epoch** (pump-level, automatic): `handleDatasetDetached` bumps the epoch; anything launched under an older epoch drains to `discard`. Covers dataset switching, `clear()`-style resets (systems can call `invalidate()` manually), and shutdown.
+- **Entity generation** (request-level, author's job, one helper): requests that reference entities stamp `(entity_id, generation)` pairs at `collect` time using the dataset's existing generation table. `apply` revalidates:
+
+```js
+// engine/ecs/async/entity_handle_is_live.js
+export function entity_handle_is_live(ecd, entity, generation) {
+    return ecd.entityExists(entity) && ecd.getEntityGeneration(entity) === generation;
+}
+```
+
+This replaces terrain's three bespoke guards with the mechanism the dataset already maintains. Per-request domain versions (like `tile.version`) remain domain logic when finer granularity is needed — the pump's `context` carries them.
+
+Tick stamping: the pump records `entityManager.fixedStepTick` at launch time into the context (the base class does this), so `apply` always knows the age of what it is integrating — needed for interpolation-aware consumers and for the deterministic mode later (§7).
+
+### 5.4 `WorkerChannel` / `WorkerHost` — transport v2
+
+A small, modern replacement for the `WorkerBuilder` codegen path. Real worker *files* instead of string concatenation — bundlers understand `new Worker(new URL(...), { type: 'module' })` statically (webpack 5 in the app today; the library's prebuilt-bundle path keeps working because the channel takes a factory).
+
+```js
+// ---- main thread ----
+const channel = await WorkerChannel.start(
+    () => new Worker(new URL('./path_field.worker.js', import.meta.url), { type: 'module' })
+);
+
+const reply  = await channel.request('find_path', payload, [payload.buffer]);  // transfers OUT
+channel.notify('set_costs', costs, [costs.buffer]);                            // one-way
+channel.on('progress', p => this.progress.set(p));                             // worker-initiated
+await channel.stop();    // settle in-flight, then terminate
+
+// ---- path_field.worker.js ----
+import { WorkerHost } from '.../worker/v2/WorkerHost.js';
+import { find_path } from '../nav/find_path.js';   // plain engine imports — it's a module worker
+
+WorkerHost.serve({
+    async find_path(payload) {
+        const data = find_path(payload);
+        return WorkerHost.result(data, [data.path.buffer]);   // transfers BACK
+    },
+    set_costs(costs) { /* update local state */ },
+});
+```
+
+Contract:
+
+- **Wire format** stays `{ id, name, payload }` / `{ id, result | error: {message, stack} }` — same spirit as `WorkerProxy`, plus `{ type: 'ready' }` handshake so `start()` resolves only when the worker booted (and boot errors reject startup instead of hanging until the 5 s watchdog).
+- **Transferables both directions, always explicit.** Fixes the current request-path gap. `extractTransferables` remains available as an opt-in convenience for callers who want the recursive walk, but the default is explicit lists — moving a buffer the caller still needs is too easy to do by accident.
+- **Transport-injectable**: the channel core operates on anything with `postMessage`/`message` events. `WorkerChannel.start(factory)` wraps a real `Worker`; tests construct it over a `MessageChannel` port pair (a real platform object, available in Node/Jest) with `WorkerHost.serve` on the other port — no mocks.
+- **Failure**: worker `error`/`messageerror` rejects all in-flight and all future requests and fires `channel.on_dead`. No auto-restart — the owning system decides (usually: log loudly; recovery is `removeSystem`/`addSystem`, which the `EntityManager` already supports at runtime).
+- **Typed wrappers over string calls**: systems wrap `channel.request('find_path', ...)` in real methods (as `makeTerrainWorkerProxy` already does), keeping string names confined to the transport seam.
+
+The old `WorkerProxy`/`WorkerBuilder`/`OnDemandWorkerManager` stack stays — meep is a library, its consumers keep working. New code targets the channel; migrating image-decode/terrain is optional follow-up (§10). `OnDemandWorkerManager` remains the right tool for *shared utility* workers (decode pools); an `AsyncSystem`'s channel is instead owned 1:1 by the system, started in `startup`, stopped in `shutdown` — one worker per system, not per component (terrain's per-`Terrain` worker is a consolidation candidate when it migrates).
+
+### 5.5 Payloads: three tiers
+
+The pump/channel don't care what a payload is; pick per system:
+
+- **Tier 0 — structured clone.** POJOs / `toJSON` output. Right for prototyping and small requests (a path query). This is what everything uses today.
+- **Tier 1 — binary + transfer.** Serialize into an `ArrayBuffer` (`core/binary/BinaryBuffer`), transfer it (zero-copy move), deserialize on the other side. For component-shaped data, the object-v2 serializer (`engine/ecs/storage/binary/object-v2/`) and the network package's per-component adapters (`engine/network/adapters/`) already define exactly this discipline — a worker mirror is "replication to a peer that happens to share your address space". Transferred buffers make a natural **ping-pong**: the worker sends the buffer back with the response, so steady-state runs allocation-free.
+- **Tier 2 — SharedArrayBuffer.** For high-frequency bulk state (a physics mirror's transform block). `makeArrayBuffer(size, true)` already gates on `crossOriginIsolated`; the dev server and any host must send COOP/COEP headers. Explicitly **optional and last** — Tier 1 with ping-pong buffers covers most workloads without the deployment constraint, and no current code ships SAB.
+
+### 5.6 WebGPU
+
+Three distinct shapes, only one of which needs the pump:
+
+**a) Device ownership — `WebGPUDeviceSystem`.** One system owns the adapter/device:
+
+```js
+class WebGPUDeviceSystem extends System {
+    /** @type {GPUDevice} */ device = null;
+
+    async startup(em) {
+        const adapter = await navigator.gpu.requestAdapter({ powerPreference: 'high-performance' });
+        this.device = await adapter.requestDevice({ /* limits/features */ });
+        this.device.lost.then(info => this.on_device_lost.send1(info));   // enqueue-only, like everything else
+    }
+    async shutdown(em) { this.device.destroy(); }
+}
+```
+
+Consumers resolve it in their own startup via the **existing** facility — no new dependency mechanism:
+
+```js
+async startup(em) {
+    const gpu = await em.promiseSystemInState(WebGPUDeviceSystem, SystemState.RUNNING);
+    this.#pipeline = gpu.device.createComputePipeline({ ... });
+}
+```
+
+Device loss v1 = surface the signal and log; full restart-all-GPU-systems recovery mirrors the WebGL context-loss monitor work and is follow-up, not blocking.
+
+**b) GPU-resident compute — needs nothing new.** A particle/skinning/fluid step whose results are consumed by rendering: `startup` (async) builds pipelines; `update`/`fixedUpdate` encode and `queue.submit()` — which is synchronous fire-and-forget. No readback, no pump, no core changes. This works **today** and the proposal's only contribution is writing the pattern down. (Interop caveat: sharing buffers with a three.js/WebGL renderer isn't possible — GPU-resident systems make sense once rendering itself has a WebGPU path, or when the system renders via its own WebGPU pass. Until then, GPU compute for the WebGL renderer must round-trip through the readback shape below.)
+
+**c) Readback — the pump again.** Culling/picking/GPGPU queries where the CPU needs the numbers. The only async part of WebGPU is `mapAsync`; wrap the staging discipline once:
+
+```js
+export class GpuReadbackRing {
+    constructor(device, { byte_size, depth = 3 }) { /* depth staging buffers */ }
+    get is_ready() { }                    // a free staging buffer exists
+    /** copy src into a free staging buffer inside the caller's encoder pass,
+     *  returns a Promise<ArrayBuffer copy> that settles after mapAsync+copy+unmap */
+    read(encoder, src_buffer, byte_offset, byte_length) { }
+}
+```
+
+An occlusion-query system is then a plain `AsyncSystem`: `collect` gathers query AABBs, `process` encodes the compute pass + `ring.read(...)` + `queue.submit()` and awaits the mapped copy, `apply` writes visibility components. `capacity` = ring depth. `mapAsync` resolves in submit order, matching the pump's launch-order delivery exactly.
+
+Worker + WebGPU compose for free: WebGPU is available inside workers, so a worker-side `process` may itself own a device (with `OffscreenCanvas` if it presents). Nothing in the design distinguishes the cases.
+
+### 5.7 Failure and shutdown semantics
+
+- **Rejected round trip** → drains to `discard(request, error)`; default implementation `console.error`s with the system name (same voice as `simulate`'s per-system catch). No retry machinery: if the work is still wanted, the next tick's `collect` regenerates it from current state. Uniform flow, no hidden loops.
+- **Dead worker** → all in-flight reject (flows through the same discard path), `on_dead` fires, system stays `RUNNING` but launches will reject immediately. Escalation is the owner's choice; the engine offers `removeSystem`/`addSystem` for a clean restart.
+- **Shutdown ordering is already safe.** Both teardown paths (`EntityManager.shutdown` and `removeSystem`) detach the dataset / call `handleDatasetDetached` **before** invoking `System.shutdown` — so by the time an `AsyncSystem.shutdown` runs, the epoch is already bumped and nothing can be applied. Shutdown is therefore just:
+
+```js
+async shutdown(em) {
+    await this.pump.settle();     // in-flight work resolves into discard
+    await this.#channel.stop();   // then tear down the transport
+}
+```
+
+  A settle timeout (reusing the 5 s startup-watchdog convention, warn-not-kill) covers a hung worker.
+
+---
+
+## 6. Worked examples
+
+### 6.1 Path planning in a worker (capacity-1 coalescing)
+
+```js
+class PathPlannerSystem extends AsyncSystem {
+    dependencies = [PathRequest, Transform];
+    components_used = [ResourceAccessSpecification.from(PathPlan, ResourceAccessKind.Create)];
+
+    #channel = null;
+    #dirty = [];                       // entities with un-serviced requests, maintained by link/unlink
+
+    async startup(em) {
+        this.#channel = await WorkerChannel.start(
+            () => new Worker(new URL('./path_planner.worker.js', import.meta.url), { type: 'module' })
+        );
+    }
+
+    async shutdown(em) {
+        await this.pump.settle();
+        await this.#channel.stop();
+    }
+
+    link(request, transform, entity) { this.#dirty.push(entity); }
+    unlink(request, transform, entity) { /* drop from #dirty */ }
+
+    collect() {
+        if (this.#dirty.length === 0) return undefined;
+
+        const ecd = this.entityManager.dataset;
+        const queries = this.#dirty.map(entity => ({
+            entity,
+            generation: ecd.getEntityGeneration(entity),
+            from: ecd.getComponent(entity, Transform).position.toJSON(),
+            to: ecd.getComponent(entity, PathRequest).target.toJSON(),
+        }));
+        this.#dirty.length = 0;
+        return { queries };
+    }
+
+    process(request) {
+        return this.#channel.request('plan', request);
+    }
+
+    apply(result, request) {
+        const ecd = this.entityManager.dataset;
+        for (const path of result.paths) {
+            if (!entity_handle_is_live(ecd, path.entity, path.generation)) continue;  // died mid-flight
+            ecd.addComponentToEntity(path.entity, PathPlan.fromFlatArray(path.points));
+        }
+    }
+}
+```
+
+Behavior that falls out for free: while the worker chews on a batch, new requests pile into `#dirty`; the moment the pump frees up, the *next* batch snapshots all of them (coalescing); a slow worker means bigger batches, never a deeper queue; entities that died mid-flight are skipped by the generation check; shutdown drains cleanly.
+
+### 6.2 Terrain tiles, rebuilt on the primitives
+
+Not a commitment to migrate — a demonstration that the bespoke guards collapse. `tile.version`/identity/`isBuildInProgress` become: epoch (dataset detach), `(entity, generation)` (terrain entity death), and `capacity` (in-flight limit). One system-owned channel replaces one worker per `Terrain` component. The result-application moves from a bare `.then` into `apply`, which lands at the system's slot instead of between `simulate` and `render`. Delivery of built geometry stays a transferred `ArrayBuffer` (Tier 1) exactly as today.
+
+### 6.3 GPU-resident compute needs nothing new
+
+```js
+class GpuParticleSystem extends System {
+    async startup(em) {
+        const gpu = await em.promiseSystemInState(WebGPUDeviceSystem, SystemState.RUNNING);
+        this.#device = gpu.device;
+        this.#pipeline = this.#device.createComputePipeline({ /* WGSL via .wgsl import */ });
+    }
+
+    update(dt) {
+        const encoder = this.#device.createCommandEncoder();
+        // ... dispatchWorkgroups ...
+        this.#device.queue.submit([encoder.finish()]);   // synchronous; done
+    }
+}
+```
+
+Included to make the point explicit: for the fire-and-forget half of WebGPU, the current `System` contract is already sufficient.
+
+### 6.4 Heavy domain mirror (physics-in-worker) — the pattern, not a commitment
+
+For domains too big to snapshot per tick, the worker holds a **persistent mirror** and the protocol becomes *commands in, state out* — the same bet the network package already made (action-log replication, not snapshot diffs), with the same serialization discipline (`Replicator`-style per-component binary adapters, object-v2 for schema-ful payloads):
+
+- `link`/`unlink` and component events accumulate a command list (spawn body, remove body, set gravity…).
+- `collect` = drain the command list + the current `fixedStepTick` horizon; `process` = `channel.request('step', commands, transfers)`; the worker steps its world and returns packed transforms (Tier 1 ping-pong buffer, or Tier 2 SAB).
+- `apply` = bulk-write `Transform`s for live entities.
+
+**Flagged constraint:** the physics determinism gate (`determinism.spec.js`) and the network lockstep design mean a worker-side physics step may not ship on the default latency-tolerant pump — it needs the fixed-latency contract from §7. That is why this is an illustration of scale, not a v1 target.
+
+---
+
+## 7. Determinism
+
+The default `AsyncSystem` is **latency-tolerant**: results apply at whatever tick they drain on, which depends on frame timing. That is correct for the large class of cosmetic/asset/derived-data workloads — terrain tiles, GI/probe updates, impostor and LOD bakes, occlusion results, path *suggestions*, audio probe refreshes — and wrong for anything inside the deterministic simulation envelope (lockstep-replicated state).
+
+For sim-affecting async work, the contract has to be **fixed latency**: a request launched at fixed tick `T` applies at exactly tick `T + K`, and if the result hasn't arrived when the fixed loop reaches `T + K`, the simulation *waits* (the same shape as the network layer waiting on remote inputs; GGPO-style frame delay). That requires a small `EntityManager` hook (the fixed loop asking "may I advance past tick X?") and is deliberately **out of scope for v1** — but v1 is forward-compatible with it: requests already carry launch-tick stamps, and delivery is already launch-ordered. It becomes a `capacity`/`deadline` policy on the pump, not a redesign.
+
+---
+
+## 8. What does not change
+
+- `System`, `EntityManager`, `EntityComponentDataset`: untouched. `simulate` stays synchronous; the noop-gate, try/catch, and runaway guard are unaffected.
+- **Execution-order scheduling stays truthful.** Because an async system's writes happen inside its own `update` slot, its `components_used` declarations (e.g. `PathPlan: Create`) describe exactly when downstream systems see the data — the scheduler's scoring keeps working without knowing async exists.
+- The `ConcurrentExecutor`/`Task` world is unaffected and remains the right tool for *main-thread* chunked work (lightmap bakes). `promiseTask` remains the bridge for awaiting a pump-less one-shot inside a task graph.
+- Existing worker consumers keep running on `WorkerProxy` until individually migrated.
+
+---
+
+## 9. Testing plan
+
+All black-box, no mocks beyond genuine platform boundaries:
+
+- **`AsyncPump.spec.js`** — real Promises: launch-order delivery with out-of-order settlement, capacity gating, epoch invalidation routing to discard, rejection routing, `settle()` semantics, re-launch after settle refused.
+- **`WorkerChannel.spec.js`** — channel over a real `MessageChannel` port pair (global in Node ≥ 15, available under Jest's node environment), `WorkerHost.serve` on the far port: request/response, transferable round trip (`ArrayBuffer.byteLength === 0` after transfer proves the move), error serialization, `on_dead` rejection sweep, ready handshake. Spawning a *real* `Worker` is covered by a browser harness prototype (`prototypeAsyncSystem.js` webpack entry), not Jest.
+- **`AsyncSystem.spec.js`** — integration with a **real** `EntityManager` + `EntityComponentDataset`: results only ever apply during `simulate` (assert dataset untouched between simulate calls while promises settle), generation-check drops on entity removal, epoch drop on `detachDataset`, drain-then-launch ordering visible to a downstream system in the same pass, clean `settle` on `removeSystem`.
+- **`GpuReadbackRing`** — behavioral spec over a stub device (WebGPU is a genuine platform boundary, like `MockAudioContext`); real-GPU verification via the browser harness.
+
+---
+
+## 10. Phasing
+
+| Phase | Contents | Size (rough) |
+|---|---|---|
+| **P0** | `AsyncPump`, `AsyncSystem`, `entity_handle_is_live`, specs | ~300 LOC + specs |
+| **P1** | `WorkerChannel`/`WorkerHost` (+ transferables both ways, handshake, injectable port), specs, browser harness with a real worker `AsyncSystem` | ~400 LOC + specs |
+| **P2** | `WebGPUDeviceSystem`, `GpuReadbackRing`, `.wgsl` sample system in the harness | ~300 LOC |
+| **P3** (opt) | Migrate `ThreadedImageDecoder`, then terrain, off `WorkerBuilder`; consolidate per-component terrain workers | migration |
+| **P4** (opt) | Deterministic fixed-latency mode (fixed-loop gate in `EntityManager`); SAB tier for a physics mirror | design + core hook |
+
+P0 is useful alone (any promise-shaped work source, including `fetch`-based systems). P1 and P2 are independent of each other.
+
+---
+
+## 11. Alternatives considered
+
+- **`async update()` awaited by the manager.** Rejected. `simulate` would stall on the slowest system or require overlapping passes; per-system try/catch, the runaway guard, lock-step fixed ticks, and execution-order semantics all break; frame pacing becomes hostage to worker latency. The entire value of a worker is *not* waiting for it.
+- **Fire-and-forget promises with `.then(apply)` (status quo).** This is what §2.5/§3 documents: uncontrolled delivery timing, bespoke staleness guards per site, no backpressure, no shutdown story. The proposal is this pattern with the sharp edges standardized away.
+- **Full job-graph scheduler (Unity-DOTS-style)** — systems return handles, the manager tracks per-component access and inserts sync points. Attractive long-term (and `components_used` + `ResourceAccessKind` are exactly the metadata it needs), but it is a rewrite of `simulate`'s contract, far beyond "keep the interface simple". The pump neither blocks nor anticipates it wrongly: if a job graph ever lands, pumps become leaf nodes that publish at declared slots.
+- **Running systems themselves inside workers (ECS-in-worker).** Moving the *dataset* across threads makes every component access a marshalling question and breaks the library's synchronous component API. The mirror pattern (§6.4) achieves the win — heavy domain off-thread — while the ECS stays single-threaded and simple.
+- **SharedArrayBuffer-first transport.** Requires cross-origin isolation everywhere the game ships; ping-ponged transferables get most of the win with zero deployment constraints. SAB stays an opt-in tier.
+
+---
+
+## 12. Open questions
+
+1. **Naming.** `AsyncPump`/`AsyncSystem`/`WorkerChannel`/`WorkerHost` are provisional; `AsyncPipe`, `RoundTrip`, `BridgeSystem` were the runners-up.
+2. **Where worker entries get bundled for library consumers** — keep the meep rollup worker-entry convention (channel takes a URL/factory, so prebuilt bundles keep working), or document webpack-native `new URL` as the only supported path for apps?
+3. **Migration appetite** — is P3 (image decoder + terrain onto the channel) wanted soon, or do the old consumers ride `WorkerProxy` indefinitely?
+4. **Deterministic mode priority** — does the network lockstep roadmap need the §7 fixed-latency contract early enough to pull the `EntityManager` gate hook into P0–P2?
+5. **Device-loss policy for WebGPU** — signal-and-log (proposed v1) vs. automatic restart of GPU systems, mirroring the WebGL context-loss recovery work.

package/src/engine/ecs/EntityManager.d.ts

@@ -131,6 +131,21 @@ export class EntityManager {
      * @private
      */
     private updateExecutionOrder;
+    /**
+     * Register a gate consulted before each fixed step. Idempotent.
+     * @param {{may_advance: (next_tick: number) => boolean}} gate
+     * @see #fixed_step_gates
+     */
+    registerFixedStepGate(gate: {
+        may_advance: (next_tick: number) => boolean;
+    }): void;
+    /**
+     * @param {{may_advance: (next_tick: number) => boolean}} gate
+     * @returns {boolean} whether the gate was registered
+     */
+    unregisterFixedStepGate(gate: {
+        may_advance: (next_tick: number) => boolean;
+    }): boolean;
     /**
      * Get list of all components referenced by active systems
      * @returns {Class[]}
@@ -244,6 +259,7 @@ export class EntityManager {
      * @deprecated use lowercase spelling `detachDataset` instead
      */
     detachDataSet: () => void;
+    #private;
 }
 import { System } from "./System.js";
 import Signal from "../../core/events/signal/Signal.js";

package/src/engine/ecs/EntityManager.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"EntityManager.d.ts","sourceRoot":"","sources":["../../../../src/engine/ecs/EntityManager.js"],"names":[],"mappings":"iCAgBU,MAAM;;;;;;;;;AAiBhB;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA4BG;AACH;IAEI;;;;OAIG;IACH,kBAFU,aAAQ,CAEL;IAEb;;;;;;OAMG;IACH,uCAA2B;IAE3B;;;;;OAKG;IACH,iCAA4B;IAE5B;;OAEG;IACH;;;QAGI;;WAEG;;;MAGL;IAEF;;;OAGG;IACH,OAFU,kBAAkB,CAEO;IAEnC;;;;;;;;;OASG;IACH,qBAFU,MAAM,CAEqB;IAErC;;;;;;;;OAQG;IACH,wCAFU,MAAM,CAE4B;IAE5C;;;;;;;;OAQG;IACH,eAFU,MAAM,CAEE;IAElB;;;;;OAKG;IACH,+BAA2B;IAE3B;;;;OAIG;IACH,SAFU,sBAAsB,CAEjB;IAEf;;;;;OAKG;IACH,uCAAsC;IAEtC;;;OAGG;IACH,6BAmFC;IAED;;;OAGG;IACH,uBAFa,OAAO,CAuBnB;IAED;;;;;OAKG;IACH,uBAJW,sBAAsB,QA4ChC;IAGD;;;OAGG;IACH,sBA2BC;IAED;;;;OAIG;IACH,qCAFa,OAAO,CAInB;IAED;;;;OAIG;IACH,6CAgBC;IAED;;;;;OAKG;IACH,wCAHW,MAAM,GACJ,IAAI,aAAS,CAiBzB;IAGD;;;OAGG;IACH,6BAFW,MAAM,QA8EhB;IAED;;;;;;;;;OASG;IACH,qBAFa,MAAM,CAKlB;IAED;;;;;OAKG;IACH,6CA6DC;IAED;;;;OAIG;IACH,mCAFa,QAAQ,OAAO,CAAC,CA2C5B;IAED;;;;;OAKG;IACH,mBAmCC;IAED;;;;;OAKG;IACH,oBAqFC;IAED;;;;;OAKG;IACH,kEAyEC;IAED;;;;OAIG;IACH,mCAFa,oBAAgB,CA0B5B;IAED;;;;OAIG;IACH,gDAHW,WAAW,GACT,oBAAgB,CA0B5B;IAED;;;;;OAKG;IACH,mEAoFC;IAGL;;OAEG;IACH,yBArtBe,sBAAsB,UAqtBA;IACrC;;OAEG;IACH,0BAAqC;CATpC;uBAh+BmC,aAAa;mBAR9B,oCAAoC;uCAMhB,6BAA6B;4BAEhC,aAAa"}
+{"version":3,"file":"EntityManager.d.ts","sourceRoot":"","sources":["../../../../src/engine/ecs/EntityManager.js"],"names":[],"mappings":"iCAgBU,MAAM;;;;;;;;;AAiBhB;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA4BG;AACH;IAEI;;;;OAIG;IACH,kBAFU,aAAQ,CAEL;IAEb;;;;;;OAMG;IACH,uCAA2B;IAE3B;;;;;OAKG;IACH,iCAA4B;IAE5B;;OAEG;IACH;;;QAGI;;WAEG;;;MAGL;IAEF;;;OAGG;IACH,OAFU,kBAAkB,CAEO;IAEnC;;;;;;;;;OASG;IACH,qBAFU,MAAM,CAEqB;IAErC;;;;;;;;OAQG;IACH,wCAFU,MAAM,CAE4B;IAE5C;;;;;;;;OAQG;IACH,eAFU,MAAM,CAEE;IAElB;;;;;OAKG;IACH,+BAA2B;IAE3B;;;;OAIG;IACH,SAFU,sBAAsB,CAEjB;IAuBf;;;;;OAKG;IACH,uCAAsC;IAEtC;;;OAGG;IACH,6BAmFC;IAED;;;;OAIG;IACH;iCAHqC,MAAM,KAAK,OAAO;aAatD;IAED;;;OAGG;IACH;iCAHqC,MAAM,KAAK,OAAO;QAC1C,OAAO,CAYnB;IAoBD;;;OAGG;IACH,uBAFa,OAAO,CAuBnB;IAED;;;;;OAKG;IACH,uBAJW,sBAAsB,QA4ChC;IAGD;;;OAGG;IACH,sBA2BC;IAED;;;;OAIG;IACH,qCAFa,OAAO,CAInB;IAED;;;;OAIG;IACH,6CAgBC;IAED;;;;;OAKG;IACH,wCAHW,MAAM,GACJ,IAAI,aAAS,CAiBzB;IAGD;;;OAGG;IACH,6BAFW,MAAM,QAuFhB;IAED;;;;;;;;;OASG;IACH,qBAFa,MAAM,CAKlB;IAED;;;;;OAKG;IACH,6CA6DC;IAED;;;;OAIG;IACH,mCAFa,QAAQ,OAAO,CAAC,CA2C5B;IAED;;;;;OAKG;IACH,mBAmCC;IAED;;;;;OAKG;IACH,oBAqFC;IAED;;;;;OAKG;IACH,kEAyEC;IAED;;;;OAIG;IACH,mCAFa,oBAAgB,CA0B5B;IAED;;;;OAIG;IACH,gDAHW,WAAW,GACT,oBAAgB,CA0B5B;IAED;;;;;OAKG;IACH,mEAoFC;IAGL;;OAEG;IACH,yBA9tBe,sBAAsB,UA8tBA;IACrC;;OAEG;IACH,0BAAqC;;CATpC;uBAjjCmC,aAAa;mBAR9B,oCAAoC;uCAMhB,6BAA6B;4BAEhC,aAAa"}