@woosh/meep-engine 3.46.0 → 3.49.0

package/src/engine/network/MESSAGE_TRANSPORT.md

@@ -0,0 +1,364 @@
+# Message transport
+
+Status: LANDED, design record. Planned 2026-10-15, revised after two adversarial reviews of the plan (§9),
+implemented, then reviewed adversarially again (§10). Replaced `transport/Channel.js`, `transport/fragments/*`
+and `transport/ReliableCommandPipeline.js` with one connection layer that carries messages of any size, for
+every message type the engine sends and for user messages.
+
+## 1. Why
+
+All engine traffic leaves `NetworkPeer` through one function, `#send_to_peer`. A message was one datagram, or
+it was cut into fragments by `fragment_send.js`. That path had five independent ceilings, all measured
+(2026-10-14, probes `zz_probe_join` / `zz_probe_wire`):
+
+| Ceiling | Effect |
+|---|---|
+| `FragmentAssembler.max_message_size` 65,536 B | larger messages dropped silently: a world of 2,990 one-component entities could never be joined |
+| uint8 chunk count, 255 × 1,186 B ≈ 302 KB | send throws; for INITIAL_SYNC on every tick forever, for a 20k-entity spawn frame every tick in `closeFixedStep` |
+| `Channel` acks: latest + 32-bit bitfield, only on packets the receiver sends anyway | a message of 34+ fragments sent in one tick is never fully acked; its fragments age out as lost though they arrived (one spawn frame re-sent 32×) |
+| NACK / retention timers on `performance.now()` | repair ignores the simulation clock; a spec ticking faster than wall time gets no repair |
+| no pacing | a 447 KB snapshot leaves in one tick |
+
+Steady-state wire cost was also poor: host → client sent `2 + owned entities` datagrams per tick (each
+AUTH_STATE, TIME_DILATION and FRAME_MARKER its own datagram with a 9 B channel header). 16 owned entities: 17
+datagrams, 346 B payload, 822 B on the wire with UDP/IPv4 headers.
+
+**Measured after** (same probe, 16 owned entities): host → client 1 datagram, 237 B payload, 265 B on the wire
+per tick (3.1× less); client → host 2 datagrams → 1. A 5,000-entity world joins over Loopback; 20,000 joins
+paced, including while it grows by 1,000 entities a tick (`initial_sync_large_world.spec.js`).
+
+## 2. Requirements
+
+1. Any message type, engine or user, up to a configured size, survives loss and arrives intact.
+2. Receiver memory per connection is bounded by configuration, never by what a peer claims; so is the
+   sender's (queues are capped; a join's pages are built only as the window frees). So is receiver time (§3.9).
+3. What a peer can make the sender do is bounded by what the sender chose to send (README, "Connecting").
+4. One datagram per peer per tick in steady state; per-message overhead of a few bytes.
+5. No per-packet or per-message allocation in steady state (buffers grow to a high-watermark and stay; §4).
+6. Every timer runs on a clock the caller supplies; the network stack never reads wall time itself.
+7. Large transfers never delay latency-critical traffic.
+8. The action stream keeps its contract: confirm a slice only when every packet carrying it is acked, abandon
+   it when any is lost (never credit what was not delivered).
+
+## 3. Design
+
+One class per peer, `Connection` (`transport/connection/`), used by `NetworkPeer` in place of `Channel` +
+fragments + `ReliableCommandPipeline`. A message is still `[NetworkPacketType byte][payload]`; `NetworkPeer`'s
+dispatch is unchanged except that FRAGMENT and NACK_FRAGMENT are gone, RELIABLE_COMMAND lost its logical seq,
+INITIAL_SYNC is a page, and TIME_DILATION carries the host's frame.
+
+### 3.1 Packet layer (after QUIC, RFC 9000 §12–13, §17.1, §19.3; RFC 9002)
+
+```
+u8      flags           bit0 ACK present, bit1 4-byte packet number, bits 4–7 protocol version (1)
+u16|u32 packet number   truncated; the full number increases monotonically and is never reused
+[ACK]   varint largest acknowledged (full number)
+        varint ack delay (8 µs units)
+        varint additional range count (≤ 63)
+        varint first range
+        count × (varint gap, varint range length)
+frames  to the end of the datagram
+```
+
+- **Packet numbers are never reused**; a retransmission is new frames in a new packet. Encoded in 2 bytes while
+  `number − largest acknowledged < 16,384`, else 4 (RFC 9000 §17.1), expanded by Appendix A.3.
+- **Acks are ranges**: one ACK covers any burst. The receiver tracks at most `max_ack_ranges` (32, at most 64);
+  everything below its floor is dropped as a duplicate or too old. The floor rises when an ACK frame of its own
+  is acknowledged (§13.2.4) or when the oldest range is pushed out by the cap. A side that sends only ACKs adds
+  a PING once per RTT so that it ever hears which ACKs arrived. The ACK takes at most half a datagram; ranges
+  that do not fit wait for a later ACK.
+- **Loss** (RFC 9002 §6.1): packet threshold 3, time threshold `9/8 × max(srtt, latest_rtt)`, floored at the
+  timer granularity. **Lost means not confirmed**, as in QUIC: a packet that arrived but whose acknowledgement
+  never got through — every ACK carrying it lost, or crowded out by newer ranges past `max_ack_ranges` holes —
+  is reported lost. The action stream's contract (requirement 8) is exactly this: never credit the unconfirmed.
+- **Probe timeout** (§6.2) doubles per expiry. On expiry the oldest outstanding packet whose reliable bytes are
+  still unacknowledged has them queued again (never declared lost); a packet whose bytes were all acknowledged
+  by another — the one it re-sent, an earlier probe — is passed over; with none, a PING. A probe may pass an
+  empty token bucket once.
+- **Notices are deferred.** `onMessageAcked` / `onMessageLost` are told after the walk that settled them —
+  the end of a flush, or of an ACK's processing — never in the middle of one. A handler may send, flush
+  (which tells the rest, in order) or dispose (after which nothing more is told).
+- **RTT** from ack delay, clamped to `max_ack_delay`; Karn-free since numbers are never reused.
+- **Timer granularity and `max_ack_delay` are one tick.** All sends and receives happen at fixed-step
+  boundaries, so nothing finer is real.
+- **Validation before action.** A datagram is parsed whole before any of it is acted on; a malformed one is
+  dropped unacknowledged (`onMalformedDatagram`). An ACK naming a packet never sent is a protocol error (§3.9).
+- **Transports that are already reliable** (`transport.reliable`: WebSocket) never declare loss and never
+  probe: on TCP a re-send queues behind the original and makes a stall worse. Loopback is *not* reliable: its
+  `drop_next` loses packets, and the specs need the loss path.
+- **Datagram size per transport**: `Transport.max_datagram_bytes`, in [64, 16,383]. 1,200 by default; WebRTC
+  1,150 so a full datagram fits one SCTP packet (§8); WebTransport reads `datagrams.maxDatagramSize` once.
+
+### 3.2 Messages
+
+```
+0x01 MESSAGE   varint length, bytes                   unreliable, at most one datagram
+0x02 PING                                             asks for an ACK
+0x10|lane      varint offset, varint length, bytes    STREAM, reliable lane 0..3
+```
+
+**Unreliable messages fit one datagram** (1,192 B at the default size). There is no segmented unreliable tier:
+a segmented message evicted from reassembly after its packets were acked would be credited by the action
+stream without being applied — exactly the old fragment-eviction defect. Larger is refused (throws); a caller
+with more goes reliable. Each carries an optional token, and the connection reports `acked(token)` or
+`lost(token)` with its packet, exactly once. When the ACK opening a flush leaves too little room for the
+largest message, the ACK goes alone.
+
+**Reliable lanes**, after QUIC streams: each lane is an ordered byte stream of `[varint length][bytes]`.
+- Sender: the message is copied once into the lane's buffer; segments are ranges of it; each packet records the
+  ranges it carried; a lost range is queued again minus anything acknowledged since. Bytes are released when
+  room is needed, not per ack (a large message's tail is not slid down once per packet).
+- **Flow control without frames**: a sender never sends a byte at or past `acked_prefix + receive_window`. The
+  receiver delivers each message the moment its bytes are contiguous, so its contiguous point is never behind
+  the sender's acked prefix, and what it holds is at most `max_message_bytes + receive_window`. This holds only
+  if both ends use the same `max_message_bytes` and `receive_window_bytes`; a segment past the receiver's bound,
+  a message declaring more than its maximum, or more holes than an honest sender makes (§3.9) is a **protocol
+  error that ends the connection** (QUIC's FLOW_CONTROL_ERROR), never a silent drop of acknowledged data.
+- **Queue cap**: `max_queued_bytes` per lane; `send_reliable` throws past it.
+- **Delivery mark**: `send_reliable` returns the offset past the message; `is_delivered(lane, mark)` asks
+  whether the acked prefix has reached it.
+
+Lanes, in scheduling order:
+
+| Lane | Carries | Paced |
+|---|---|---|
+| CONTROL | RESUME_HELLO/ACCEPT/REJECT, DISCONNECT, RECOVERY_REQUEST — engine-only, one datagram each | no |
+| (unreliable) | ACTION_STREAM slices, AUTH_STATE, TIME_DILATION, FRAME_MARKER | no |
+| FRAMES | action-stream frames larger than a datagram | no — the stream waits on them |
+| USER | `send_reliable_command`, up to `max_message_bytes` | yes |
+| BULK | INITIAL_SYNC pages, STATE_BURST | yes |
+
+### 3.3 Scheduler
+
+Every message for a peer is queued; `flush` sends them packed into as few datagrams as fit, in the order above,
+the ACK in the first datagram. A token bucket per connection at `send_rate_bytes_per_second` (default 256 KiB/s)
+paces USER and BULK; unpaced traffic is counted against it and may take it at most one burst below zero. The
+burst is two ticks of rate (at least two datagrams). Paced lanes always get at least one datagram a flush, so a
+peer that owns many entities cannot starve a join. Congestion control, when built (`CONGESTION_CONTROL.md`),
+sets the rate; transport backpressure (`bufferedAmount`) is a follow-up.
+
+**When things go out.** `NetworkPeer.send_*` only queue. `NetworkPeer.send_queued()` flushes every connection.
+`NetworkPeer.end_tick`, `ServerAuthoritativeServer.end_tick` (after `onTickComplete`) and
+`ServerAuthoritativeClient.end_tick` call it last, so one tick's traffic — the action stream, AUTH_STATE,
+TIME_DILATION, join pages; the client's FRAME_MARKER, queued before `end_tick` — leaves in one flush. A send
+outside a tick is flushed where it is made: RESUME_HELLO by `connect()`, a kick's last word by the kick.
+`flush_outbound` only queues.
+
+**Closing.** `disconnect_peer` never sends. An explicit close (`drop_peer`, `disconnect`, a kick) calls
+`NetworkPeer.flush_peer(id)` after queueing DISCONNECT and tears down in `finally`: one best-effort datagram.
+A transport that has already died is never flushed. Teardown from inside a connection's own dispatch is safe:
+the receive loop stops at the first message after disposal.
+
+### 3.4 Clock
+
+`NetworkPeer` takes a required `clock: () => number` (ms) and `tick_period_ms`. `NetworkSession` passes its own
+`#clock_ms`, the render-paced clock that already drives timeouts, grace and reconnects: one clock, and the fixed
+steps of one update all see the same time, so a burst of catch-up steps cannot age packets whose acks could not
+have arrived.
+
+### 3.5 What moved where
+
+| Message | Before | After |
+|---|---|---|
+| ACTION_STREAM slice ≤ datagram | own datagram + notice | unreliable message + notice, coalesced |
+| ACTION_STREAM frame > datagram | fragments, ≤ 64 KiB, 33-window | FRAMES, reliable, once (§3.6) |
+| AUTH_STATE, TIME_DILATION, FRAME_MARKER | own datagram each | unreliable, coalesced; AUTH_STATE > datagram throws |
+| RECOVERY_REQUEST, RESUME_*, DISCONNECT | unreliable datagram | CONTROL |
+| STATE_BURST | fragments | BULK |
+| INITIAL_SYNC | one message ≤ 64 KiB, sent twice | lazily built pages on BULK, catch-up rounds, then FINAL (§3.7) |
+| RELIABLE_COMMAND | `ReliableCommandPipeline`, ≤ 1,178 B | USER, ≤ `max_message_bytes` |
+
+### 3.6 Oversized action-stream frames
+
+A frame whose group does not fit one datagram goes on FRAMES, unpaced, once, keeping the head flag it was
+packed with. While it is in flight the pack skips it and sends what follows as further slices, which the
+receiver holds until it lands; it does not stall the stream. It is credited when delivered, like any slice,
+through `#credit_confirmed`. Unpaced because the action stream cannot outlast the 32-frame ring: a frame that
+takes longer to deliver than the ring holds loses the frames after it.
+
+A frame larger than `max_message_bytes` can never be delivered. Once it is the pack start it is given up and
+said so (`NetworkPeer.onFrameTooLarge(peer_id, frame_start, frame_end, bytes)`), credited, and the stream goes
+on behind it — never a throw, which would stop the tick for every peer.
+
+### 3.7 Joining a world of any size
+
+Host, per joining peer (`orchestrator/HostJoin.js`):
+1. **Hold** the peer's action stream (`hold_outbound`); skip its AUTH_STATE.
+2. **First round.** At the first tick, collect the networked entities. If the whole world fits
+   `JOIN_FINAL_ROUND_BYTES` (256 KiB) and the lane's room, it goes in that tick and the round is the last:
+   every entity is at the same frame. Otherwise pages of whole entities (`JOIN_PAGE_BYTES`, 16 KiB; a bigger
+   entity is a page by itself) are built only as BULK's window (128 KiB for a join) frees, each at the current
+   frame, noting each entity's frame sent. Host memory per joiner: the window plus 12 bytes per entity (its
+   frame sent, and its place in the list).
+3. **Catch-up rounds.** When every page is delivered, compare each networked entity's last-changed frame
+   (`NetworkPeer.last_changed_frame`, stamped for everything its executor executes or records — a
+   direct write reported with `net_mutate_component` included; a write nobody reports is not a
+   change the join can know of) with the frame its state
+   was sent; entities changed since, or created since, go out as a round at the current frame. A catch-up
+   round always tries to finish in its tick, with what BULK may still hold (`max_queued_bytes − unacked −
+   max_message_bytes`) as its budget: what changed while a world went out is a fraction of it, and a round
+   that may only page would never shrink for a world that changes faster than a tick's budget.
+4. **Release.** A round that finishes in its tick leaves the client with the whole world at that frame D: every
+   entity it carries is at D, every entity it leaves out unchanged since its state went out. The FINAL page
+   (frame D) goes after it, and the peer's stream is released from D + 1 (still in the ring — the next frame).
+   The join is over once FINAL is *delivered*: until then a peer that drops holds part of a world and is synced
+   afresh rather than resumed.
+5. **Bounded.** At most `JOIN_MAX_ROUNDS` (8) rounds; a join that has not converged is dropped
+   (`join_did_not_converge`). Pages are only built as the window frees, so a peer that never acknowledges costs
+   the window and nothing else.
+
+Client:
+- From `connect()` until its binding's answer is whole, the client **holds inbound** action stream and
+  AUTH_STATE (`hold_inbound`): a stream that overtakes the pages is about a world the client does not hold yet.
+  Bounded by `max_queued_bytes`; past it the connection is broken (§3.9) and the client joins again.
+- The FIRST page issues the token, seeks the frame counter, clears prediction state and forgets the timeline,
+  and puts the binding in **Syncing**, which counts as the binding's answer: a later RESUME_ACCEPT/REJECT is
+  stale, the reconnect ladder stops.
+- Each page is applied as it lands (creating entities as INITIAL_SYNC did).
+- FINAL seeks again, repeats the clears and `forget_timeline` (prediction ran on a partial world meanwhile),
+  sets the world start frame, releases outbound, and makes the binding **Synced**. The held inbound is applied
+  once the step's inbound is done (not inside the page's dispatch, which shares the receive buffer), then
+  `NetworkSession.onSynced` fires. RESUME_ACCEPT releases the hold the same way.
+- **Seeking.** A page's frame can be old when it lands — a late FINAL — so the client seeks to
+  `max(page frame, host frame seen) + 1 + depth`, where the host's frame comes with every TIME_DILATION.
+
+### 3.8 Deleted
+
+`Channel.js`, `ReliableCommandPipeline.js`, `fragments/*`, FRAGMENT / NACK_FRAGMENT, `NetworkPeer.channel_for`
+(now `connection_for`), `#pending_initial_sync`'s resend count. Hard break, no shims.
+
+### 3.9 Trust, and a connection that breaks
+
+The connection does not authenticate datagrams; its transport delivers only the peer's. WebRTC (DTLS) and
+WebTransport / WebSocket (TLS) do; `NodeUDPTransport` drops datagrams from any source but its remote (by port
+alone when the remote is a host name); Loopback and Simulated are in-process. Before this was stated, one stray
+datagram numbered ahead of the peer's wedged a connection for good: the receiver acknowledged it, and the
+sender dropped every datagram carrying that ACK, silently (measured: 0 of 300 messages through, for a stray
+1,000 ahead or 2^30 ahead).
+
+`Connection.onProtocolError` says the connection is no longer whole:
+- a reliable lane broke its limits (past the window, a message over `max_message_bytes`, a bad length);
+- more than `max(1,024, window / 256)` holes held out of order on a lane: an honest sender's holes are packets
+  it lost and has not re-sent, a few hundred at most in a 256 KiB window, while a peer sending one-byte
+  segments two bytes apart had the receiver spend 46 ms a datagram at 512k ranges (measured; now refused after
+  6 datagrams, the worst taking 1.5 ms; ranges are added by binary search);
+- the peer acknowledged a packet this side never sent: it took in a datagram that was not ours.
+
+`NetworkPeer.onProtocolError` adds the client's inbound hold overflowing (§3.7). `NetworkSession` handles
+every one as a dropped transport and closes the transport so the other end hears of it at once: on the host
+the peer goes into the grace window; on the client the reconnect ladder resumes on a new connection. A stray
+datagram on either side now costs a resume (the specs allow 30 ticks), where it cost the connection.
+
+## 4. Allocation and copies
+
+- Outbound: unreliable bytes are copied once into a staging buffer, reliable bytes once into the lane buffer, and
+  from either straight into the datagram. The transport's own copy is the last.
+- Inbound: a datagram is parsed in place; an unreliable message is dispatched as a view into it; stream bytes are
+  copied once into the lane's receive buffer and dispatched as a view.
+- Per-packet state is struct-of-arrays (packet table, record ring, notice queue) grown to a high-watermark.
+- `NetworkPeer`'s per-slice `PendingAck` objects are a typed-array pool keyed by token (`PendingSlices`).
+- Copies go through `bytes_copy`: a loop up to 64 bytes, `set` over a `subarray` view past that. Measured (Node
+  22): the loop costs ~1.2 ns a byte, the view 50–70 ns flat, so a loop for all sizes made 1 KB copies 20×
+  slower. Two connections exchanging ten tokened messages a tick plus ACKs, 200k ticks: with `subarray` on every
+  copy, 236 scavenges at 100 B messages; now 13 at 20 B, 212 at 100 B and 226 at 1,000 B, where each copy is one
+  short-lived view.
+
+## 5. Phases
+
+All landed: **P1** Connection; **P2** NetworkPeer on Connection, old layers deleted; **P3** oversized frames,
+the join, resume/disconnect paths, forest specs; **P4** README "Connecting", the packet table, this record.
+
+## 6. Defaults
+
+| Option | Default | Why |
+|---|---|---|
+| `send_rate_bytes_per_second` | 262,144 | GNS's default clamp; a 447 KB join in ~1.8 s |
+| `max_message_bytes` | 1 MiB | largest USER/BULK/FRAMES message; pages are far smaller |
+| `receive_window_bytes` | 256 KiB | ≥ rate × 1 s RTT, so the window never limits the paced rate |
+| `max_queued_bytes` | 4 MiB per lane | a lane this far behind is a peer that is gone |
+| `max_ack_ranges` | 32 (≤ 64) | QUIC implementations keep a few dozen |
+| `JOIN_PAGE_BYTES` | 16 KiB | small enough that a page is one tick of rate |
+| `JOIN_FINAL_ROUND_BYTES` | 256 KiB | a first round this size finishes in one tick |
+| `JOIN_MAX_ROUNDS` | 8 | a join that needs more is a world changing faster than a peer can be sent it |
+
+## 7. Not in scope
+
+Congestion control and transport backpressure; encryption, handshakes and anti-spoofing tokens (§3.9 states
+what is assumed of transports); level-baseline identity; the open action-stream issues in
+`netcode-open-defects` (silent watermark skip at `max_packets_per_tick = 1`, silent loss past the ring floor).
+
+## 8. Open
+
+Stated, not measured:
+- A client that spawns networked entities locally *during* a join can take a network id a later page carries.
+  Clients pre-build pools before connecting (GAP-038); spawning mid-join is not guarded.
+- The WebRTC datagram size (1,150) is from Chrome's source as remembered, not a device measurement.
+- A stray from the remote's own address and port — a previous connection's packet on a reused socket pair — is
+  not filtered by `NodeUDPTransport`; it costs a resume (§3.9), and a lower-numbered one can mark a lost packet
+  acknowledged. A connection id would close it.
+
+From the implementation review, kept open (§10):
+- **I5** An oversized non-head frame behind a frame lost past the ring floor is never applied while the stream
+  stays quiet. The same class as the ring-floor loss in §7.
+- **I7** A tokenless reconnect after an incomplete join is refused (NoSessionToken). Tokenless claims are
+  refused on purpose: they are what an id-squatter looks like.
+
+Measured in passing: 1,500 entities created a tick outrun the action stream itself (the forest spec runs 1,000);
+not a join defect.
+
+## 9. Plan review dispositions
+
+Protocol review (R1) and integration review (R2) of the plan, 2026-10-15.
+
+| # | Finding | Disposition |
+|---|---|---|
+| R1.1, R2.8 | segmented unreliable message evicted after its packets were acked is credited | **removed the tier**: unreliable ≤ one datagram |
+| R1.2 | 16-bit packet numbers wrap under a silent peer; ring overwrites live entries | 2/4-byte encoding by distance; packet table grows, never wraps; ACK of an unsent number refused (now a protocol error, §3.9) |
+| R1.3, R2.13 | steps × period clock: catch-up steps fake losses; RTT quantised | session `#clock_ms`; granularity and `max_ack_delay` one tick |
+| R1.4 | probe timeout without backoff, behind the bucket | doubles per expiry; one probe passes an empty bucket; never declares loss |
+| R1.5 | frameless flow control assumes matching config | violation is a connection-ending protocol error, documented; no silent drop |
+| R1.6, R2.10 | unbounded queues; unpaced traffic starves BULK; user messages unpaced | per-lane queue cap; bucket floor; paced lanes get ≥ 1 datagram a flush; USER lane paced |
+| R1.7, R2.3 | frozen snapshot = world-sized memory per joiner; ledger missing, retention-bound, empty-burst deadlock | **lazy pages + per-entity change stamps**; no ledger; catch-up as pages; FINAL page even when nothing changed |
+| R1.8 | one code path misbehaves on TCP | reliable transports never declare loss or probe |
+| R1.9 | 1,200 B datagram = two SCTP packets on WebRTC | WebRTC declares 1,150 (§8) |
+| R1.10, R2.2 | oversized frame behind BULK; stall past ring; re-pack re-sends it | FRAMES lane ahead of paced lanes, unpaced; skipped while in flight |
+| R1.11 | ack-only side can never prune ranges | PING once per RTT in ack-only packets; floor rule stated |
+| R1.12 | ACK frames unvalidated | range count ≤ 63, numbers < next sent, delay clamped |
+| R1.13 | no version | 4-bit version in every datagram; anti-spoofing out of scope (§7) |
+| R1.14 | burst of 22 datagrams | burst = two ticks of rate |
+| R1.15 | exactly one outcome per packet | first wins; a late ACK of a lost packet is ignored |
+| R2.1 | paged join is not the binding's answer; ladder restarts it; stale reject acted on | **Syncing** binding state on page 0 |
+| R2.4 | page-by-page apply breaks clear-on-sync | clears and `forget_timeline` repeated at FINAL; host holds AUTH_STATE until release |
+| R2.5 | id collision from client spawns mid-join | stated (§8) |
+| R2.6 | flush placement; spec breaks | single flush at the end of each `end_tick`; FRAME_MARKER queued before it |
+| R2.7 | flushing in `disconnect_peer` sends on dead transports | `disconnect_peer` never sends; explicit `flush_peer` on explicit close only |
+| R2.9 | peer withholds BULK acks to make the host re-serialize forever | at most 8 rounds; pages only as the window frees |
+| R2.11 | resume accepted over a half-applied world | incomplete join → StaleFrame, fresh sync |
+| R2.12 | release checked outside the flush that applies it | release from D + 1, the next frame, by construction |
+| R2.14 | forest spec budget too small for the default rate | spec budget states the rate |
+| R2.15 | README contradictions | P4 |
+
+## 10. Implementation review dispositions
+
+Connection review (C) and peer/session integration review (I) of the implementation, 2026-10-15. Each fix was
+pinned by a spec that failed first.
+
+| # | Finding | Disposition |
+|---|---|---|
+| C1 | one stray datagram wedges the connection for good | ACK of an unsent packet is a protocol error; session resumes on a new connection; NodeUDP filters by source (§3.9) |
+| C2 | `flush` throws `RangeError` for a full-size unreliable message when an ACK is owed | the ACK goes alone |
+| C3 | a handler flushing from an ACK notice made acked messages report lost (23 of 200) | notices deferred to the end of the walk |
+| C4 | `dispose()` in `onMessageLost` did not stop the walk; later notices still told | deferred notices stop at dispose |
+| C5 | a probe whose bytes were acked elsewhere sent nothing; a tail loss then never recovered (> 1 h) | probes pass over such packets; PING when none has bytes |
+| C6 | one-byte segments two bytes apart: 46 ms a datagram at 512k ranges | hole cap, protocol error; binary-search insert |
+| C7 | delivered messages reported lost past `max_ack_ranges` holes | lost = not confirmed, documented (§3.1); the action stream's contract |
+| C8 | `max_ack_ranges` above 64 made the peer reject every ACK | asserted ≤ 64 |
+| I1 | actions arriving before the FIRST page applied to the old world | inbound held from `connect()` |
+| I2 | a frame over `max_message_bytes` threw every tick, for every peer | given up, `onFrameTooLarge` |
+| I3 | join counted complete when FINAL was queued, not delivered | FINAL_SENT until delivered |
+| I4 | a world changing more than 256 KiB a round could never be joined | catch-up budget is the lane's room |
+| I5 | non-head oversized frame behind a frame lost past the ring floor | open (§8) |
+| I6 | inbound-hold overflow dropped acknowledged frames silently | protocol error; session wiring (§3.9) |
+| I7 | tokenless reconnect after an incomplete join refused | open, by design (§8) |
+| I8 | `release_inbound` ran inside the INITIAL_SYNC dispatch | released after the step's inbound (with I1) |
+| — | `subarray` views were 2.3 of 2.5 KB a tick of the layer's allocation | `bytes_copy` (§4) |

package/src/engine/network/NetworkSession.d.ts

@@ -71,8 +71,17 @@ export const NetworkSessionReconcile: Readonly<{
  *
  * Replication is component-driven: attaching a {@link NetworkIdentity}
  * to an entity registers it with the session via an `EntityObserver`.
- * Mutation is event-driven: callers fire
- * `dataset.sendEvent(entity, "net_mutate_component", payload)`.
+ *
+ * **What travels is what goes through the action executor**, and nothing
+ * else: actions (`send`, a client's input), and direct writes their writer
+ * reports — `dataset.sendEvent(entity, "net_mutate_component", payload)`.
+ * The session never looks for changes by itself. That is what keeps a tick's
+ * cost in what changed and what is owned, never in the size of the world;
+ * the stack is built for worlds of millions of networked objects. A
+ * replicated component written directly and not reported stays where it was
+ * written. Ownership is set when an identity attaches, or by the join, and
+ * `owner_peer_id` / `network_id` are not written afterwards. The README's
+ * "What the session assumes" has the whole contract and its measured costs.
  *
  * **The session runs on the engine's clock.** The `EntityManager`'s fixed
  * step is the network tick: each step opens a network frame before any
@@ -155,12 +164,6 @@ export class NetworkSession {
      * - `reconnect` (client role): policy for the reconnect ladder. See
      *   {@link DEFAULT_RECONNECT_POLICY}. Set `enabled: false` to opt
      *   out entirely.
-     * - `publish_changes` (host role, default true): at the end of every
-     *   step, send each replicated component the simulation changed — physics,
-     *   controllers, anything that wrote it directly. Components an action
-     *   wrote are already on the wire and are not sent twice. Publishing never
-     *   writes back into the host's own state. Set false to publish by hand
-     *   (`net_mutate_component`).
      * - `reconcile` (client role): see {@link NetworkSessionReconcile}. Default
      *   `'correct'`.
      * - `late_input` (host role): what to do with client input that arrives
@@ -168,7 +171,7 @@ export class NetworkSession {
      *   `'apply'`: fold it into the current frame. `'rollback'` rewinds and
      *   replays, which re-executes actions only, never the engine's systems.
      */
-    constructor({ entity_manager, transport, transport_factory, role, local_peer_id, simulation_delay_ticks, tick_rate_hz, frame_capacity, binary_registry, scope_filter, time_dilation, adaptive_render_delay, server_resume_grace_ms, connection_timeout_ms, reconnect, publish_changes, late_input, reconcile, }?: {
+    constructor({ entity_manager, transport, transport_factory, role, local_peer_id, simulation_delay_ticks, tick_rate_hz, frame_capacity, binary_registry, scope_filter, time_dilation, adaptive_render_delay, server_resume_grace_ms, connection_timeout_ms, reconnect, late_input, reconcile, }?: {
         entity_manager: EntityManager;
         transport?: object;
         transport_factory?: (() => object);
@@ -209,6 +212,15 @@ export class NetworkSession {
      * @type {number}
      */
     correction_skipped_count: number;
+    /**
+     * Client: how many times it found itself more than the host's input window behind the frame the
+     * host last named, or was told by the host's depth feedback that its input lands below the
+     * host's window, and re-took the host's time. A tab in the background, a debugger, a laptop
+     * asleep — see {@link #retake_host_time_if_stalled}. Diagnostic: a game that sees these on a
+     * machine that never left the foreground has a client that cannot keep the step.
+     * @type {number}
+     */
+    stall_count: number;
     /**
      * Client, {@link NetworkSessionReconcile.Correct}: a component of an authoritative state
      * differed from the prediction, and the present was corrected. Once per component corrected.
@@ -260,6 +272,13 @@ export class NetworkSession {
      * @type {Signal}
      */
     onReconnected: Signal;
+    /**
+     * Client: a join has completed — the host's FINAL page has landed, every networked entity's
+     * state is here, and from now on this client's input is sent. No arguments. A world is sent
+     * as pages over as many ticks as its size takes; until this fires it is still arriving.
+     * @type {Signal}
+     */
+    onSynced: Signal;
     /**
      * Client: the reconnect ladder is exhausted, or the host explicitly
      * kicked us with a DISCONNECT packet, or `reconnect.enabled` is
@@ -400,7 +419,7 @@ export class NetworkSession {
     /**
      * Wire the session to the engine. After this, the EntityObserver
      * is active, the right orchestrator is constructed, and a
-     * {@link NetworkSystem} is attached. Subsequent `replicate_*`
+     * {@link NetworkSystem} is attached until {@link stop}. Subsequent `replicate_*`
      * / {@link defineAction} / {@link defineInputSampler} calls throw.
      *
      * Pre-condition: every replicated component class must already have a
@@ -464,8 +483,18 @@ export class NetworkSession {
     disconnect(reason?: string): void;
     /**
      * Tear down. Idempotent — safe to call from cleanup paths.
+     *
+     * Takes back from the engine what {@link start} put in it: the {@link NetworkSystem} is out of
+     * the engine's loop when this returns, so another session can start on the same engine; this one
+     * does not start again. The
+     * networked entities stay in the world, and their `network_id`s are released — an id means
+     * nothing without the session that gave it. The {@link InterpolationSystem} stays too, whoever
+     * added it: it is the engine's, and blends whatever else is interpolated; the session only
+     * withdraws its own source from it.
+     *
+     * @returns {Promise} resolves once the engine has shut the {@link NetworkSystem} down
      */
-    stop(): void;
+    stop(): Promise<any>;
     /**
      * **Deprecated; does nothing.** The session used to run its own fixed-step clock here, next to
      * the engine's: two clocks, so network frame N and the engine's step N were unrelated, and
@@ -590,9 +619,9 @@ export class NetworkSession {
      * frame 6,000; a client has to start tagging its inputs near 6,000 or the host discards every
      * one of them — `ServerAuthoritativeServer.tick` trims pending actions older than
      * `sim_frame - frame_capacity + 1`, and a client tagging frame 0 against a host at 6,000 has
-     * every input dropped **silently**. It moves on its own screen, never moves on the host's, and
-     * no signal fires: `onPendingActionDropped` covers the future-lead and capacity bounds, not
-     * this one.
+     * every input dropped. It moves on its own screen and never moves on the host's — and until
+     * 2026-10-15 no signal fired; the host's `onPendingActionDropped` now reports each such drop
+     * as `'window'`.
      *
      * Everything needed was already there except the setter. INITIAL_SYNC carries the host's frame,
      * `NetworkPeer` writes it, and `onInitialSync` dispatches it — and the handler named it
@@ -601,8 +630,10 @@ export class NetworkSession {
      * with the input sampler silenced until its counter caught up: 28,801 calls and 62 ms of wall
      * clock for an hour-old host. Reported as GAP-042.
      *
-     * A client now seeks automatically on INITIAL_SYNC. This is the manual door, for a caller that
-     * computes its own lead.
+     * A client now seeks automatically on INITIAL_SYNC, and again whenever it finds itself more than
+     * the host's input window behind the frame the host last named — a stall, see
+     * {@link #retake_host_time_if_stalled}. This is the manual door, for a caller that computes its
+     * own lead.
      *
      * A client sends the host nothing it predicted before its last sync, and knows it by frame: its
      * input starts at the frame the sync sought to. A caller whose own lead is shorter moves that

package/src/engine/network/NetworkSession.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"NetworkSession.d.ts","sourceRoot":"","sources":["../../../../src/engine/network/NetworkSession.js"],"names":[],"mappings":";;;;;;;;;;;;;;;;iCA6CoB,MAAM;AAhB1B;;;;;;;;;;;;;;;;;GAiBG;AACH;;;GAGG;;;;sCAKiB,MAAM;AAH1B;;;;GAIG;AACH;IACI;;;;;;;OAOG;;IAEH;;;;OAIG;;GAEJ;AAgLH;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAoCG;AACH;IAwaI;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;OAkEG;IACH,kTAlEW;QACV,cAAkB,gBAAgB;QAClC,SAAa,CAAC,EAAE,MAAM,CAAC;QACvB,iBAAqB,CAAC,EAAE,CAAC,MAAM,MAAM,CAAC,CAAC;QACvC,IAAQ,CAAC,EAAE,QAAQ,GAAC,MAAM,CAAC;QAC3B,aAAiB,CAAC,EAAE,MAAM,CAAC;QAC3B,sBAA0B,CAAC,EAAE,MAAM,CAAC;QACpC,YAAgB,CAAC,EAAE,MAAM,CAAC;QAC1B,eAAmB,CAAC,EAAE,2BAA2B,CAAC;QAClD,YAAgB,CAAC,EAAE,MAAM,CAAC;QAC1B,aAAiB,CAAC,EAAE,YAAY,CAAC;QACjC,qBAAyB,CAAC,EAAE,mBAAmB,CAAC;QAChD,sBAA0B,CAAC,EAAE,MAAM,CAAC;QACpC,qBAAyB,CAAC,EAAE,MAAM,CAAC;QACnC,SAAa,CAAC,EAAE;YAChB,OAAa,CAAC,EAAE,OAAO,CAAC;YACxB,YAAkB,CAAC,EAAE,MAAM,CAAC;YAC5B,aAAmB,CAAC,EAAE,MAAM,CAAC;YAC7B,YAAkB,CAAC,EAAE,MAAM,CAAC;YAC5B,kBAAwB,CAAC,EAAE,MAAM,CAAC;YAClC,gBAAsB,CAAC,EAAE,MAAM,CAAC;YAChC,mBAAyB,CAAC,EAAE,OAAO,CAAC;SAC/B,CAAC;KACH,EAkLH;IAriBD;;;;OAIG;IACH,cAFU,OAAO,CAEI;IAgHrB;;;OAGG;IACH,kBAFU,MAAM,CAEK;IAErB;;;;OAIG;IACH,0BAFU,MAAM,CAEa;IAE7B;;;;;;;;OAQG;IACH,cAFU,MAAM,CAEY;IA6E5B;;;;;;OAMG;IACH,eAFU,IAAI,GAAC,IAAI,CAEE;IAiDrB;;;;OAIG;IACH,YAFU,MAAM,CAEU;IAE1B;;;;;OAKG;IACH,0BAFU,MAAM,CAEwB;IAExC;;;;OAIG;IACH,kBAFU,MAAM,CAEgB;IAEhC;;;;OAIG;IACH,oBAFU,MAAM,CAEkB;IAElC;;;;;;OAMG;IACH,eAFU,MAAM,CAEa;IAE7B;;;;;OAKG;IACH,6BAFU,MAAM,CAE2B;IAkIvC;;;OAGG;IACH,uCAAoC;IAEpC;;;OAGG;IACH,uCAAmC;IAEnC;;;OAGG;IACH,eAFU,kBAAkB,GAAC,MAAM,CAEnB;IAQhB;;;;OAIG;IACH,yBAFU,MAAM,CAE+C;IAE/D;;;OAGG;IACH,iCAFU,MAAM,CAIT;IAEP;;;;;;OAMG;IACH,yBAFU,MAAM,CAEoB;IAEpC;;;OAGG;IACH,wBAFU,MAAM,CAIgC;IAEhD;;;OAGG;IACH,0BAFU,2BAA2B,CAEsC;IAiD/E;;;;;;;;;;;;;;;;;;;;;;;;;;OA0BG;IACH,8DAHW,WAAW,gCAkBrB;IAED;;;;;;;;;;;;;;OAcG;IACH,2EAMC;IAED;;;;;;;;;;;;OAYG;IACH,mDAFW,0BAA0B,GAAC,IAAI,QAsBzC;IA0BD;;;;;;;;OAQG;IACH,6CAIC;IAED;;;;;;;;;;;;;;;;OAgBG;IACH,+BAFW,CAAC,KAAK,EAAE,MAAM,KAAK,MAAM,SAAS,CAAC,GAAC,IAAI,QAWlD;IAMD;;;;;;;;;OASG;IACH,uBAsSC;IAED;;;;;;;;;;;;;;;;;;;;;;OAsBG;IACH,wBAHW,MAAM,aACN,MAAM,QAsFhB;IAED;;;;;;;;;;;;;;;;;;;;OAoBG;IACH,mBAHW,MAAM,WACN,MAAM,QA8BhB;IAED;;;;;;OAMG;IACH,oBAFW,MAAM,QAkChB;IAED;;OAEG;IACH,aA0CC;IAMD;;;;;;;;;;;OAWG;IACH,kBAFW,MAAM,QAOhB;IA2OD;;;;;;;;;OASG;IACH,2BAKC;IAED;;;;;;;;;;;;;;;;;;;;;;;;;;;OA2BG;IACH,sBAFW,SAAS,QAYnB;IAMD;;OAEG;IACH,wCAEC;IAED;;OAEG;IACH,wCAEC;IAED;;OAEG;IACH,wBAEC;IAED;;OAEG;IACH,0CAEC;IAED;;OAEG;IACH,iDAEC;IAED;;OAEG;IACH,kCAEC;IAED;;;;;;;;;;OAUG;IACH,kCAEC;IAED;;;;;;;;;;;;;;;;;;;OAmBG;IACH,wBAHW,MAAM,GACJ;QAAC,iBAAiB,EAAE,MAAM,CAAC;QAAC,iBAAiB,EAAE,MAAM,CAAA;KAAC,CAIlE;IAED;;OAEG;IACH,4BAGC;IAED;;;;;;;;;;;;;;;;;;;;;;;;;;OA0BG;IACH,qBAFW,MAAM,QAahB;;CA43CJ;mBAn6GkB,oCAAoC;qBAElC,qBAAqB;4CACG,sDAAsD;4BAWvE,iCAAiC;2CADE,gDAAgD;0BAKrF,oBAAoB;0CAPO,6CAA6C;0CADxD,6CAA6C;4BADtC,+BAA+B;iCAO/C,sCAAsC;oCAKnC,+BAA+B;6BAEtC,wBAAwB"}
+{"version":3,"file":"NetworkSession.d.ts","sourceRoot":"","sources":["../../../../src/engine/network/NetworkSession.js"],"names":[],"mappings":";;;;;;;;;;;;;;;;iCA6CoB,MAAM;AAhB1B;;;;;;;;;;;;;;;;;GAiBG;AACH;;;GAGG;;;;sCAKiB,MAAM;AAH1B;;;;GAIG;AACH;IACI;;;;;;;OAOG;;IAEH;;;;OAIG;;GAEJ;AA8LH;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6CG;AACH;IA0eI;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;OA4DG;IACH,iSA5DW;QACV,cAAkB,gBAAgB;QAClC,SAAa,CAAC,EAAE,MAAM,CAAC;QACvB,iBAAqB,CAAC,EAAE,CAAC,MAAM,MAAM,CAAC,CAAC;QACvC,IAAQ,CAAC,EAAE,QAAQ,GAAC,MAAM,CAAC;QAC3B,aAAiB,CAAC,EAAE,MAAM,CAAC;QAC3B,sBAA0B,CAAC,EAAE,MAAM,CAAC;QACpC,YAAgB,CAAC,EAAE,MAAM,CAAC;QAC1B,eAAmB,CAAC,EAAE,2BAA2B,CAAC;QAClD,YAAgB,CAAC,EAAE,MAAM,CAAC;QAC1B,aAAiB,CAAC,EAAE,YAAY,CAAC;QACjC,qBAAyB,CAAC,EAAE,mBAAmB,CAAC;QAChD,sBAA0B,CAAC,EAAE,MAAM,CAAC;QACpC,qBAAyB,CAAC,EAAE,MAAM,CAAC;QACnC,SAAa,CAAC,EAAE;YAChB,OAAa,CAAC,EAAE,OAAO,CAAC;YACxB,YAAkB,CAAC,EAAE,MAAM,CAAC;YAC5B,aAAmB,CAAC,EAAE,MAAM,CAAC;YAC7B,YAAkB,CAAC,EAAE,MAAM,CAAC;YAC5B,kBAAwB,CAAC,EAAE,MAAM,CAAC;YAClC,gBAAsB,CAAC,EAAE,MAAM,CAAC;YAChC,mBAAyB,CAAC,EAAE,OAAO,CAAC;SAC/B,CAAC;KACH,EA0KH;IA1kBD;;;;OAIG;IACH,cAFU,OAAO,CAEI;IAiGrB;;;OAGG;IACH,kBAFU,MAAM,CAEK;IAErB;;;;OAIG;IACH,0BAFU,MAAM,CAEa;IAE7B;;;;;;;OAOG;IACH,aAFU,MAAM,CAEA;IAEhB;;;;;;;;OAQG;IACH,cAFU,MAAM,CAEY;IAsF5B;;;;;;OAMG;IACH,eAFU,IAAI,GAAC,IAAI,CAEE;IAkFrB;;;;OAIG;IACH,YAFU,MAAM,CAEU;IAE1B;;;;;OAKG;IACH,0BAFU,MAAM,CAEwB;IAExC;;;;OAIG;IACH,kBAFU,MAAM,CAEgB;IAEhC;;;;OAIG;IACH,oBAFU,MAAM,CAEkB;IAElC;;;;;;OAMG;IACH,eAFU,MAAM,CAEa;IAE7B;;;;;OAKG;IACH,UAFU,MAAM,CAEQ;IAExB;;;;;OAKG;IACH,6BAFU,MAAM,CAE2B;IA2HvC;;;OAGG;IACH,uCAAoC;IAEpC;;;OAGG;IACH,uCAAmC;IAEnC;;;OAGG;IACH,eAFU,kBAAkB,GAAC,MAAM,CAEnB;IAQhB;;;;OAIG;IACH,yBAFU,MAAM,CAE+C;IAE/D;;;OAGG;IACH,iCAFU,MAAM,CAIT;IAEP;;;;;;OAMG;IACH,yBAFU,MAAM,CAEoB;IAEpC;;;OAGG;IACH,wBAFU,MAAM,CAIgC;IAEhD;;;OAGG;IACH,0BAFU,2BAA2B,CAEsC;IAgD/E;;;;;;;;;;;;;;;;;;;;;;;;;;OA0BG;IACH,8DAHW,WAAW,gCAkBrB;IAED;;;;;;;;;;;;;;OAcG;IACH,2EAMC;IAED;;;;;;;;;;;;OAYG;IACH,mDAFW,0BAA0B,GAAC,IAAI,QAsBzC;IA0BD;;;;;;;;OAQG;IACH,6CAIC;IAED;;;;;;;;;;;;;;;;OAgBG;IACH,+BAFW,CAAC,KAAK,EAAE,MAAM,KAAK,MAAM,SAAS,CAAC,GAAC,IAAI,QAWlD;IAMD;;;;;;;;;OASG;IACH,uBAyQC;IAED;;;;;;;;;;;;;;;;;;;;;;OAsBG;IACH,wBAHW,MAAM,aACN,MAAM,QAiGhB;IAED;;;;;;;;;;;;;;;;;;;;OAoBG;IACH,mBAHW,MAAM,WACN,MAAM,QA+BhB;IAeD;;;;;;OAMG;IACH,oBAFW,MAAM,QAmChB;IAED;;;;;;;;;;;;OAYG;IACH,qBAmDC;IAMD;;;;;;;;;;;OAWG;IACH,kBAFW,MAAM,QAOhB;IAiJD;;;;;;;;;OASG;IACH,2BAKC;IAED;;;;;;;;;;;;;;;;;;;;;;;;;;;OA2BG;IACH,sBAFW,SAAS,QAYnB;IAMD;;OAEG;IACH,wCAEC;IAED;;OAEG;IACH,wCAEC;IAED;;OAEG;IACH,wBAEC;IAED;;OAEG;IACH,0CAEC;IAED;;OAEG;IACH,iDAEC;IAED;;OAEG;IACH,kCAEC;IAED;;;;;;;;;;OAUG;IACH,kCAEC;IAED;;;;;;;;;;;;;;;;;;;OAmBG;IACH,wBAHW,MAAM,GACJ;QAAC,iBAAiB,EAAE,MAAM,CAAC;QAAC,iBAAiB,EAAE,MAAM,CAAA;KAAC,CAIlE;IAED;;OAEG;IACH,4BAGC;IAED;;;;;;;;;;;;;;;;;;;;;;;;;;;;OA4BG;IACH,qBAFW,MAAM,QAahB;;CA2nDJ;mBA3qHkB,oCAAoC;qBAElC,qBAAqB;4CACG,sDAAsD;4BAYvE,iCAAiC;2CADE,gDAAgD;0BAKrF,oBAAoB;0CAPO,6CAA6C;0CADxD,6CAA6C;4BAFrB,+BAA+B;iCAQhE,sCAAsC;oCAInC,+BAA+B;6BAEtC,wBAAwB"}