@statewalker/webrun-msgpack 0.3.2 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +242 -307
  2. package/package.json +9 -6
package/README.md CHANGED
@@ -1,89 +1,77 @@
1
1
  # @statewalker/webrun-msgpack
2
2
 
3
- MessagePack on the wire, in **two distinct shapes**:
3
+ ## What it is
4
4
 
5
- - a **stream codec** — `encodeMsgpack` / `decodeMsgpack`, length-prefixed, for a transport with no
6
- message boundaries (plus specialisations for `Float32Array`);
7
- - a **message codec** — `msgpackCodec`, a `PortCodec` for `@statewalker/webrun-rpc`'s
8
- `multiplexPort`, with no length prefix, for a transport that already frames.
5
+ MessagePack for the wire, in two shapes. A **stream codec** (`encodeMsgpack` / `decodeMsgpack`,
6
+ plus `Float32Array` variants) turns an iterable of values into length-prefixed frames and back, for
7
+ a transport with no message boundaries. A **message codec** (`msgpackCodec`) is a `PortCodec` for
8
+ `@statewalker/webrun-rpc`'s `multiplexPort`: one envelope becomes one bare MessagePack document and
9
+ one `postMessage`, for a transport that already frames. Both sit on the package's own MessagePack
10
+ implementation, `serialize` / `deserialize`, a TypeScript port of Yves Goergen's
11
+ [msgpack.js](https://github.com/ygoe/msgpack.js), which is exported too.
9
12
 
10
- Both sit on the package's own MessagePack implementation, `serialize` / `deserialize`, which is
11
- exported too. It is a TypeScript port of Yves Goergen's
12
- [msgpack.js](https://github.com/ygoe/msgpack.js) with a handful of fixes — see
13
- [Provenance and credits](#provenance-and-credits). The package has no runtime dependencies.
13
+ ## Why it exists
14
14
 
15
- The two codecs are not interchangeable, and reaching for the wrong one is easy. The table below is
16
- the whole decision.
15
+ A raw MessagePack stream has no frame boundaries. A decoder fed a byte stream only succeeds if the
16
+ chunk boundaries happen to line up with the document boundaries. Code that pipes values across a
17
+ transport — a scanner writing chunks to a store, a pipeline streaming embeddings — needs to
18
+ serialise a stream of objects into bytes and get exactly those objects back on the other side,
19
+ without truncation surprises. The stream codec does that with a length prefix.
17
20
 
18
- ## Which codec
21
+ `multiplexPort` runs many virtual ports over one transport and needs a codec to put its envelopes
22
+ on the wire. `structuredCodec` (in `webrun-rpc`) passes envelopes through unencoded, which only
23
+ works where messages are structured values: a `MessagePort`, a worker, an iframe. `msgpackCodec`
24
+ is the codec for transports whose messages are bytes.
19
25
 
20
- | | `encodeMsgpack` / `decodeMsgpack` | `msgpackCodec` |
21
- | --- | --- | --- |
22
- | Kind | stream codec | message codec (`PortCodec`) |
23
- | Shape | `AsyncIterable<T>` ⇄ `AsyncIterable<Uint8Array>` | one `PortEnvelope` ⇄ one `postMessage` |
24
- | Framing | **4-byte big-endian length prefix**, added by this package | **none** — the transport's own message boundaries are the framing |
25
- | Use it when | the transport is a byte *stream*: a TCP-like socket, a `ReadableStream`, a file, anything where chunk boundaries are arbitrary | the transport preserves *message* boundaries: a WebSocket, an `RTCDataChannel`, a LiveKit data packet |
26
- | Malformed input | a truncated trailing frame is never emitted; the consumer ends without yielding a partial value | dropped, never thrown — a bad frame from a peer cannot take the multiplexer down |
27
- | Depends on | nothing | **type-only** `@statewalker/webrun-rpc` |
28
-
29
- Adding a length prefix on a transport that already frames is redundant framing; relying on message
30
- boundaries where there are none is the truncation bug the stream codec exists to prevent. Pick by
31
- the transport, not by taste.
32
-
33
- ## Why the stream codec exists
34
-
35
- Consumers that pipe values across transports (scanners writing chunks to a store, chat pipelines streaming embeddings, etc.) need a way to serialise a stream of objects into a byte stream and reassemble it on the other side without truncation surprises.
36
-
37
- A raw MessagePack stream has no frame boundaries: a decoder can only succeed if the chunk boundaries happen to line up with the payload boundaries. Length-prefix framing fixes this — the decoder buffers incoming bytes and only yields when a complete `[length][payload]` pair is available. Partial trailing frames are NEVER emitted, so callers can detect truncation by comparing observed count to expected.
38
-
39
- Previously the codec lived inside `@repo/streams` (private, unpublished). It's been extracted here so (a) consumers that only need framing don't pull in the broader `webrun-streams` surface, and (b) MessagePack lives in exactly one place.
40
-
41
- ## Why the port codec exists
26
+ The package carries its own MessagePack implementation so that its bundle imports nothing at
27
+ runtime, and so that its behaviour on truncated or hostile input is defined by tests in this
28
+ package (see [Internals](#internals)).
42
29
 
43
- `@statewalker/webrun-rpc`'s `multiplexPort` runs many virtual ports over one transport, and it needs
44
- a codec to put its envelopes on the wire. `structuredCodec` (in `webrun-rpc`) passes them through
45
- unencoded, which works only where messages are *structured values* — a `MessagePort`, a worker, an
46
- iframe. `msgpackCodec` is the byte-transport sibling: one envelope becomes one msgpack frame and one
47
- `postMessage`.
48
-
49
- ## Install
30
+ ## How to use
50
31
 
51
32
  ```sh
52
- npm install @statewalker/webrun-msgpack
33
+ pnpm add @statewalker/webrun-msgpack
53
34
  ```
54
35
 
55
- No runtime dependencies **in the emitted bundle** — `dist/index.js` imports nothing — and no peer
56
- dependencies. ESM only (`"type": "module"`). That is not the same as the install cost:
57
- `@statewalker/webrun-rpc` is a declared `dependency`, so `npm install` also pulls it and,
58
- transitively, `@statewalker/webrun-streams` into `node_modules` — even for a consumer who uses only
59
- the stream codec or `serialize` / `deserialize`.
36
+ No peer dependencies. ESM only. One entry point, `.`, which works in browsers, Node, Deno, Bun and
37
+ workers; the runtime needs `TextDecoder`.
60
38
 
61
- `@statewalker/webrun-rpc` is declared as a dependency but is **type-only**: `msgpackCodec` imports
62
- the `PortCodec` interface from it and no runtime code, so nothing of `webrun-rpc` is in the built
63
- bundle and `webrun-rpc` gains no msgpack dependency in either direction.
39
+ `@statewalker/webrun-rpc` is a declared dependency but is used for **types only** (`PortCodec`,
40
+ `PortEnvelope`): `dist/index.js` imports nothing. The install still pulls `webrun-rpc` and,
41
+ through it, `@statewalker/webrun-streams` into `node_modules`, even if you use only the stream
42
+ codec or `serialize` / `deserialize`.
64
43
 
65
- The runtime needs `TextDecoder`, which every current browser, Node, Deno and Bun provide.
44
+ ### Pick the codec by the transport
66
45
 
67
- ## How to use
46
+ | | `encodeMsgpack` / `decodeMsgpack` | `msgpackCodec` |
47
+ | --- | --- | --- |
48
+ | Kind | stream codec | message codec (`PortCodec`) |
49
+ | Shape | `Iterable<T> \| AsyncIterable<T>` ⇄ `AsyncGenerator<Uint8Array>` | one `PortEnvelope` ⇄ one `postMessage` |
50
+ | Framing | 4-byte big-endian length prefix, added here | none: the transport's message boundaries are the framing |
51
+ | Use it when | the transport is a byte stream: a TCP-like socket, a `ReadableStream`, a file | the transport preserves message boundaries: a WebSocket, an `RTCDataChannel`, a LiveKit data packet |
52
+ | Malformed input | a truncated trailing frame is never emitted; the iteration just ends | dropped, never thrown |
53
+
54
+ A length prefix on a transport that already frames is redundant. Relying on message boundaries
55
+ where there are none is the truncation bug the stream codec exists to prevent.
68
56
 
69
57
  ### Exports
70
58
 
71
- | Export | Direction | Use case |
59
+ | Export | Direction | Use |
72
60
  | --- | --- | --- |
73
- | `encodeMsgpack<T>(src: Iterable<T> \| AsyncIterable<T>)` | values → frames | generic JSON-ish values |
74
- | `decodeMsgpack<T>(src: Iterable<Uint8Array> \| AsyncIterable<Uint8Array>)` | frames → values | inverse of `encodeMsgpack` |
75
- | `encodeFloat32Arrays(src: Iterable<Float32Array> \| AsyncIterable<Float32Array>)` | arrays → frames | float streaming, no per-element conversion |
76
- | `decodeFloat32Arrays(src: Iterable<Uint8Array> \| AsyncIterable<Uint8Array>)` | frames → arrays | inverse of `encodeFloat32Arrays` |
77
- | `msgpackCodec: PortCodec` | envelope ⇄ one framed message | `multiplexPort` over a byte transport |
61
+ | `encodeMsgpack<T>(input: Iterable<T> \| AsyncIterable<T>)` | values → frames | generic JSON-like values |
62
+ | `decodeMsgpack<T>(input: Iterable<Uint8Array> \| AsyncIterable<Uint8Array>)` | frames → values | inverse of `encodeMsgpack` |
63
+ | `encodeFloat32Arrays(input: Iterable<Float32Array> \| AsyncIterable<Float32Array>)` | arrays → frames | float vectors, no per-element conversion |
64
+ | `decodeFloat32Arrays(input: Iterable<Uint8Array> \| AsyncIterable<Uint8Array>)` | frames → arrays | inverse of `encodeFloat32Arrays` |
65
+ | `msgpackCodec: PortCodec` | envelope ⇄ one message | `multiplexPort` over a byte transport |
78
66
  | `serialize(value, options?)` | one value → one MessagePack document | the format itself, no framing |
79
67
  | `deserialize(bytes, options?)` | one document → one value | inverse of `serialize` |
80
68
 
81
- Types: `SerializeOptions`, `DeserializeOptions`, `MsgpackInput` (what `deserialize` accepts:
82
- `Uint8Array`, `ArrayBuffer` or an array of byte values) and `MsgpackExtension` (an extension value
83
- other than a timestamp: `{ type, data }`).
69
+ Types: `SerializeOptions` (`multiple?: boolean`, `invalidTypeReplacement?: unknown`),
70
+ `DeserializeOptions` (`multiple?: boolean`), `MsgpackInput` (`Uint8Array | ArrayBuffer |
71
+ ArrayLike<number>`) and `MsgpackExtension` (`{ type: number; data: Uint8Array }`, an extension
72
+ value other than a timestamp).
84
73
 
85
- All four stream functions take synchronous iterables too — an array, a generator — so a fixed
86
- list needs no async wrapper.
74
+ The four stream functions accept synchronous iterables too, so an array needs no async wrapper.
87
75
 
88
76
  ## Examples
89
77
 
@@ -98,12 +86,15 @@ const value = deserialize(bytes); // same shape; `at` is a Date, `raw` a Uint8Ar
98
86
  // Several documents back to back:
99
87
  const three = serialize([1, "two", { three: 3 }], { multiple: true });
100
88
  deserialize(three, { multiple: true }); // [1, "two", { three: 3 }]
89
+
90
+ // A value that cannot be encoded, replaced instead of throwing:
91
+ serialize({ f: () => 1 }, { invalidTypeReplacement: "unsupported" }); // encodes { f: "unsupported" }
101
92
  ```
102
93
 
103
- ### Stream of values
94
+ ### A stream of values
104
95
 
105
96
  ```ts
106
- import { encodeMsgpack, decodeMsgpack } from "@statewalker/webrun-msgpack";
97
+ import { decodeMsgpack, encodeMsgpack } from "@statewalker/webrun-msgpack";
107
98
 
108
99
  async function* events() {
109
100
  yield { type: "start" };
@@ -111,55 +102,49 @@ async function* events() {
111
102
  yield { type: "done" };
112
103
  }
113
104
 
114
- // encode
115
105
  const bytes: AsyncIterable<Uint8Array> = encodeMsgpack(events());
116
106
 
117
- // decode on the other side — handles arbitrary chunk boundaries
107
+ // The decoder handles arbitrary chunk boundaries.
118
108
  for await (const msg of decodeMsgpack<{ type: string; text?: string }>(bytes)) {
119
109
  console.log(msg);
120
110
  }
121
111
  ```
122
112
 
123
- ### Embeddings pipeline
113
+ ### Re-framing across arbitrary chunks
124
114
 
125
115
  ```ts
126
- import {
127
- decodeFloat32Arrays,
128
- encodeFloat32Arrays,
129
- } from "@statewalker/webrun-msgpack";
116
+ import { decodeMsgpack, encodeMsgpack } from "@statewalker/webrun-msgpack";
130
117
 
131
- async function* chunks() {
132
- yield new Float32Array([0.1, 0.2, 0.3, 0.4]);
133
- yield new Float32Array([0.5, 0.6, 0.7, 0.8]);
134
- }
118
+ const frames: Uint8Array[] = [];
119
+ for await (const f of encodeMsgpack([{ a: 1, b: "hi" }])) frames.push(f);
135
120
 
136
- // wire-efficient: msgpack `bin` type reinterpreted byte-for-byte as Float32.
137
- const pipe = decodeFloat32Arrays(encodeFloat32Arrays(chunks()));
138
- for await (const arr of pipe) console.log(arr.length); // 4, 4
121
+ // Feed the decoder one byte at a time; it buffers until a frame is complete.
122
+ const oneByteAtATime = frames.flatMap((f) => [...f].map((byte) => new Uint8Array([byte])));
123
+ for await (const v of decodeMsgpack<{ a: number; b: string }>(oneByteAtATime)) {
124
+ console.log(v); // { a: 1, b: "hi" }
125
+ }
139
126
  ```
140
127
 
141
- ### Re-framing across transport chunks
128
+ ### Float32 vectors
142
129
 
143
130
  ```ts
144
- import { decodeMsgpack, encodeMsgpack } from "@statewalker/webrun-msgpack";
131
+ import { decodeFloat32Arrays, encodeFloat32Arrays } from "@statewalker/webrun-msgpack";
145
132
 
146
- // Produce one frame (a plain array is a valid input), then split the bytes any way you like:
147
- const bytes: Uint8Array[] = [];
148
- for await (const f of encodeMsgpack([{ a: 1, b: "hi" }])) bytes.push(f);
149
- // Hand the decoder arbitrarily small slices — it buffers until complete:
150
- async function* byOne() {
151
- for (const b of bytes) for (const byte of b) yield new Uint8Array([byte]);
133
+ async function* chunks() {
134
+ yield new Float32Array([0.1, 0.2, 0.3, 0.4]);
135
+ yield new Float32Array([0.5, 0.6, 0.7, 0.8]);
152
136
  }
153
- for await (const v of decodeMsgpack<{ a: number; b: string }>(byOne())) {
154
- console.log(v); // { a: 1, b: "hi" }
137
+
138
+ // Each array travels as a msgpack `bin`, reinterpreted byte for byte as Float32 on arrival.
139
+ for await (const arr of decodeFloat32Arrays(encodeFloat32Arrays(chunks()))) {
140
+ console.log(arr.length); // 4, 4
155
141
  }
156
142
  ```
157
143
 
158
144
  ### An RPC stream over a byte transport
159
145
 
160
- `msgpackCodec` on both ends of a transport that carries `Uint8Array`s, one virtual port, one
161
- `duplexOverPort` round trip. The pipe below stands in for the real thing — replace it with a
162
- WebSocket pair, an `RTCDataChannel`, or a LiveKit packet stream and nothing else changes.
146
+ `msgpackCodec` on both ends of a transport that carries `Uint8Array`s. The in-process pipe below
147
+ stands in for a WebSocket pair, an `RTCDataChannel` or a LiveKit packet stream.
163
148
 
164
149
  ```js
165
150
  import { duplexOverPort, multiplexPort, serveDuplexOverPort } from "@statewalker/webrun-rpc";
@@ -213,124 +198,72 @@ await client.close();
213
198
  await server.close();
214
199
  ```
215
200
 
216
- The same stack passes the unmodified `webrun-streams-conformance` L0–L6 suite over a byte pipe, in
217
- both framing regimes — unlimited, and with frames capped at 64 KiB
218
- (`tests/conformance-bytes.test.ts`).
219
-
220
- **Read that green narrowly: an in-process pipe is not a transport.** The pipe hands `Uint8Array`s
221
- straight from one object to another inside one process, so what the suite covers is this codec's own
222
- contract end to end, including under chunking. It covers *none* of what a real byte transport brings:
223
- framing, message-size ceilings and what a transport does when you exceed one, backpressure,
224
- reconnection, close codes and error semantics. A WebSocket, an `RTCDataChannel` and a LiveKit data
225
- track each need their own run before anything here is claimed of them.
226
-
227
- ## `maxMessageSize` bounds the payload, not the frame
228
-
229
- **Leave a margin of at least 256 bytes below your transport's hard limit.** This is the one thing
230
- that will bite you when wiring `msgpackCodec` to a capped transport, and the frame sizes below are
231
- measured rather than cautious:
232
-
233
- `duplexOverPort` applies `toChunks(maxMessageSize)` to the *payload*. The envelope framing —
234
- `WireChunk`, `callPort`'s `{type, channelName, callId, params}`, the mux's `{type, id, payload}`,
235
- then this codec — is added **on top, afterwards**. Over `msgpackCodec` that overhead is
236
- `87 + len(callId)` bytes and it is **not constant**: `callId` is
237
- `` `call-${Date.now()}-${String(Math.random()).substring(2)}` ``, whose length varies **31–40**
238
- characters *per chunk* because `Math.random()` drops trailing zeros; the port id's integer width
239
- adds 0–4; the channel name adds 1 for `"out"` over `"in"`; and a chunk at or above 64 KiB adds 2 as
240
- the payload's `bin` header widens.
241
-
242
- Two numbers, and the difference between them matters: adding those terms up gives a **modelled
243
- ceiling of 134 bytes**, while the overheads *actually observed* span **123–128 bytes**. The 134 is
244
- arithmetic; the 123–128 is measurement.
245
-
246
- The largest, 128, comes from a **64 KiB** cap — a 65,664-byte frame in the capped conformance run
247
- over a 10 MiB body — which is the regime where the `bin`-header term applies. A separate sweep, eight
248
- runs at a **16 KiB** cap with a 1 MiB body (several thousand chunks, so several thousand `callId`s),
249
- never exceeded **126** at that cap; the table below is one run per cap and is not that sweep. A
250
- 256-byte margin covers all of it, which is why the advice is a round number rather than a tight one.
251
-
252
- Measured, with a 512 KiB body through the stack above:
253
-
254
- | `maxMessageSize` | intent | largest frame actually posted | overhead |
255
- | --- | --- | --- | --- |
256
- | `16 * 1024` | an `RTCDataChannel`'s conservative ceiling | **16,508 bytes** | 124 |
257
- | `12 * 1024` | LiveKit's safe packet size | **12,413 bytes** | 125 |
258
- | `64 * 1024` | | **65,662 bytes** | 126 |
201
+ The same stack passes the unmodified `@statewalker/webrun-streams-conformance` L0–L6 suite over a
202
+ byte pipe, both unlimited and with frames capped at 64 KiB (`tests/conformance-bytes.test.ts`).
203
+ Read that narrowly: an in-process pipe covers this codec's own contract, including chunking. It
204
+ covers none of what a real transport adds — message-size ceilings and what happens past them,
205
+ backpressure, reconnection, close codes. Each real transport needs its own run.
206
+
207
+ ## Internals
208
+
209
+ ### `maxMessageSize` bounds the payload, not the frame
210
+
211
+ **Set `maxMessageSize` at least 256 bytes below the transport's hard limit.** `duplexOverPort`
212
+ splits the *payload* to `maxMessageSize`. The framing — the chunk wrapper, `callPort`'s
213
+ `{type, channelName, callId, params}`, the mux's `{type, id, payload}`, then this codec — is added
214
+ on top afterwards. Over `msgpackCodec` that overhead is `87 + len(callId)` bytes and it is not
215
+ constant: `callId` is `` `call-${Date.now()}-${String(Math.random()).substring(2)}` ``, 31–40
216
+ characters per chunk because `Math.random()` drops trailing zeros; the port id's integer width adds
217
+ 0–4; the channel name `"out"` adds 1 over `"in"`; a chunk of 64 KiB or more adds 2 as the `bin`
218
+ header widens. Summing the terms gives a modelled ceiling of 134 bytes. The overheads actually
219
+ observed span 123–128 bytes.
259
220
 
260
- So setting `maxMessageSize` to the transport's hard limit **overruns it on the first full-size
261
- chunk** — and a transport that silently drops an oversized message (LiveKit does; the body arrives
262
- as zero bytes with no error on either side) gives you no signal at all. Set it to
263
- `limit - 256` and the arithmetic stops mattering.
221
+ Measured with a 512 KiB body through the stack in the example above:
264
222
 
265
- This is spec D10's correction, recorded in
266
- `docs/superpowers/specs/2026-09-05-port-multiplexer-design.md`.
223
+ | `maxMessageSize` | intent | largest frame posted | overhead |
224
+ | --- | --- | --- | --- |
225
+ | `16 * 1024` | an `RTCDataChannel`'s conservative ceiling | 16,508 bytes | 124 |
226
+ | `12 * 1024` | LiveKit's safe packet size | 12,413 bytes | 125 |
227
+ | `64 * 1024` | | 65,662 bytes | 126 |
228
+
229
+ The 128-byte maximum came from a 64 KiB cap in the capped conformance run (a 65,664-byte frame over
230
+ a 10 MiB body). Setting `maxMessageSize` to the transport's hard limit overruns it on the first
231
+ full-size chunk, and a transport that silently drops oversized messages gives no signal. On LiveKit
232
+ the body arrives as zero bytes with no error on either side. `limit - 256` covers every measured
233
+ case with room to spare.
267
234
 
268
- ## Two things `msgpackCodec` does that `structuredCodec` does not
235
+ ### Two ways `msgpackCodec` differs from `structuredCodec`
269
236
 
270
237
  **The transfer list is ignored.** `PortCodec.post` receives an optional `Transferable[]`;
271
- `msgpackCodec` drops it. After encoding, the payload is *inside* the bytes — there is no live
272
- `ArrayBuffer` left on the far side of the call to hand over, and passing the caller's original
273
- buffers as transferables would detach buffers the caller still owns. `structuredCodec` forwards the
274
- list, because there the objects themselves cross.
275
-
276
- **msgpack drops object keys whose value is explicitly `undefined`**; structured clone keeps them. So
277
- `{ result: undefined }` arrives as `{}` over this codec and as `{ result: undefined }` over
278
- `structuredCodec`. Nothing `webrun-rpc`'s layer 2 sends depends on the difference — every wire shape
279
- it produces is pinned against both codecs in `tests/codec-equivalence.test.ts` — and per **spec
280
- D16** it must not come to. If you build a payload where `"key" in obj` means something different
281
- from `obj.key === undefined`, it will not survive this codec.
282
-
283
- D16's reach has one open edge, worth knowing before you put arbitrary application errors on a byte
284
- transport: `serializeError` copies **every own enumerable property** off a thrown `Error` onto the
285
- wire. Whether the resulting `error` payload is msgpack-expressible therefore depends on what your
286
- code throws — a `cause` holding a `Map` or a class instance collapses to `{}`, and a circular
287
- reference makes `serialize` throw.
288
-
289
- ## What `deserialize` refuses, and how
290
-
291
- `deserialize` throws on input that is not MessagePack; it never logs. Specifically:
292
-
293
- - **Truncated input** — any read that would run past the end — throws a `RangeError`
294
- (`Insufficient data: …`). Every proper prefix of a valid encoding is refused this way, so a cut
295
- integer can no longer come back as `NaN`, nor a cut `bin` as a shorter array.
296
- - **The never-used type byte `0xc1`**, an empty input, a timestamp extension of an unknown size,
297
- and a non-byte argument throw an `Error`.
298
- - **Malformed UTF-8 inside a string is not an error**: each malformed sequence decodes to U+FFFD,
299
- exactly as `TextDecoder` does, and the values after the string are read normally. Overlong
300
- forms are malformed — `C0 AF` does not decode to `/`.
301
- - **Bytes after the first document are ignored** unless `{ multiple: true }` asks for all of them.
302
-
303
- `msgpackCodec` turns every one of these throws into a dropped message. (Before this package
304
- carried its own implementation, `@ygoe/msgpack` printed the whole buffer with `console.debug` on
305
- some truncated input before throwing. That is gone.)
306
-
307
- ## The value model
308
-
309
- What each JavaScript value becomes on the wire, and what comes back:
238
+ `msgpackCodec` drops it. After encoding, the payload is inside the bytes, so there is nothing left
239
+ to hand over, and transferring the caller's buffers would detach buffers the caller still owns.
310
240
 
311
- | Written | As | Read back as |
312
- | --- | --- | --- |
313
- | `null`, `undefined` | nil | `null` — but an object key whose value is `undefined` is **dropped** |
314
- | `boolean` | bool | `boolean` |
315
- | safe integer | the narrowest int / uint | `number`; `-0` is written as `0` and loses its sign |
316
- | any other number (fraction, beyond ±2⁵³, `NaN`, `±Infinity`) | float 64 | `number` |
317
- | `string` | str, UTF-8; a lone surrogate is written as U+FFFD, as `TextEncoder` does | `string` |
318
- | `Uint8Array`, `Uint8ClampedArray` | bin | `Uint8Array` — a **view into the input**, not a copy |
319
- | other typed arrays (`Float32Array`, `Int16Array`, …) | array of numbers | `number[]` |
320
- | `Array` | array | `unknown[]` |
321
- | `Date` | timestamp extension (type -1), 32/64/96-bit as the instant needs | `Date`, floored to the millisecond; an invalid `Date` throws |
322
- | any other object | map of its **own** enumerable string keys | plain object; every key, `__proto__` included, is an own property |
323
- | `bigint`, `function`, `symbol` | — throws, unless `invalidTypeReplacement` supplies a stand-in | |
241
+ **Object keys whose value is `undefined` are dropped**; structured clone keeps them.
242
+ `{ result: undefined }` arrives as `{}`. Nothing `webrun-rpc`'s stream and RPC layers send depends
243
+ on the difference: every wire shape they produce is pinned against both codecs in
244
+ `tests/codec-equivalence.test.ts`. A payload where `"key" in obj` means something different from
245
+ `obj.key === undefined` will not survive this codec.
324
246
 
325
- Reading also accepts what other encoders write: float 32, int 64 / uint 64 (as the nearest
326
- `number` — precision beyond 2⁵³ is lost), maps with non-string keys (the key is converted with
327
- `String()`), and extension types other than timestamps, which come back as
328
- `{ type, data }` with `type` as the unsigned byte (so type `-2` reads as `254`). A `Map` or `Set`
329
- has no own enumerable keys and is written as an empty map.
247
+ One edge: `webrun-streams`' `serializeError` copies every own enumerable property of a thrown
248
+ `Error` onto the wire. A `cause` holding a `Map` or a class instance collapses to `{}`, and a
249
+ circular reference makes `serialize` throw.
330
250
 
331
- ## Internals
251
+ ### What `msgpackCodec.read` accepts, and why it never throws
332
252
 
333
- ### Frame layout
253
+ It accepts a `Uint8Array`, an `ArrayBuffer`, or any `ArrayBufferView` (a `DataView` included,
254
+ offset and length honoured). Everything else returns `undefined`, which the multiplexer treats as
255
+ "not ours": a non-byte value, empty bytes, malformed MessagePack, and valid MessagePack that is not
256
+ a `PortEnvelope` (`id` must be a non-negative integer, `type` one of `open` / `message` / `close`).
257
+ A shared transport may carry other traffic, and the multiplexer must not mistake it for an
258
+ envelope.
259
+
260
+ Refusal is a dropped message, never a throw. A throw here would escape inside the raw port's own
261
+ listener, outside any consumer's reach, and one hostile frame could take the multiplexer down.
262
+ `read` can still throw for three inputs, all in the byte-shape check before the `try`: a detached
263
+ `ArrayBuffer`, a `DataView` over a detached buffer, and a `Proxy` with a throwing
264
+ `getPrototypeOf`. None can arrive over a wire.
265
+
266
+ ### Stream frame layout and decoder
334
267
 
335
268
  ```
336
269
  ┌──────────────┬────────────────────────────┐
@@ -339,141 +272,143 @@ has no own enumerable keys and is written as an empty map.
339
272
  └──────────────┴────────────────────────────┘
340
273
  ```
341
274
 
342
- - Big-endian 32-bit length prefix — same convention as Java `DataOutputStream` and most wire protocols.
343
- - Max payload per frame: 2³²−1 bytes. No fragmentation within a frame (a single call to `serialize` produces the whole payload up-front); very large values will allocate proportionally.
344
- - **This layout is the stream codec's only.** `msgpackCodec` writes a bare msgpack document per
345
- message and prefixes nothing.
346
-
347
- ### Decoder state machine
348
-
349
- The decoder keeps a rolling `Uint8Array` buffer. Each incoming chunk is appended (single allocation per chunk); then:
350
-
351
- 1. If buffer is shorter than 4 bytes — wait for more.
352
- 2. Read the 32-bit BE length.
353
- 3. If buffer doesn't hold `4 + length` bytes — wait for more.
354
- 4. Copy the payload out, `deserialize` it, yield.
355
- 5. Advance the buffer past this frame; repeat step 1.
356
-
357
- Zero-length chunks are tolerated and simply no-op through the loop. Truncated trailing frames are silently dropped — the buffer retains them but the consuming `for await` ends without yielding a partial value.
358
-
359
- ### What `msgpackCodec.read` accepts, and what it refuses
360
-
361
- It accepts whatever byte shape a transport pump hands over — a `Uint8Array`, a bare `ArrayBuffer`,
362
- or any `ArrayBufferView` (a `DataView` included, offset and length honoured). Everything else is
363
- refused by returning `undefined`: a non-byte value, empty bytes, malformed msgpack, and well-formed
364
- msgpack that decodes to something that is not a `PortEnvelope` (the `id` must be a non-negative
365
- integer and the `type` one of `open` / `message` / `close`). A shared transport carries traffic that
366
- is not ours, and layer 1 must not mistake it for an envelope.
367
-
368
- Refusal is always a dropped message, never a throw: a throw here would escape inside the raw port's
369
- own listener, outside any consumer's reach, and would let one hostile frame take the multiplexer
370
- down.
275
+ Big-endian 32-bit length, so at most 2³²−1 bytes per frame. A frame is built whole: one
276
+ `serialize` call per value, no fragmentation. This layout belongs to the stream codec only;
277
+ `msgpackCodec` prefixes nothing.
371
278
 
372
- There are three inputs for which `read` *can* still throw, all inside the byte-shape check that runs
373
- before the `try`: a detached `ArrayBuffer`, a `DataView` over a detached buffer, and a `Proxy` with
374
- a throwing `getPrototypeOf`. None is producible by a remote peer sending bytes — each needs
375
- same-process JavaScript already holding the backing memory — so the guarantee is "cannot throw for
376
- anything that arrives over a wire", not "cannot throw for any JavaScript value".
279
+ The decoder keeps a rolling buffer. Each chunk is appended (one allocation per chunk), then: if
280
+ fewer than 4 bytes are buffered, wait; read the length; if `4 + length` bytes are not buffered,
281
+ wait; copy the payload out, `deserialize` it, yield; advance and repeat. Zero-length chunks are
282
+ harmless. A truncated trailing frame stays in the buffer and the iteration ends without yielding
283
+ it, so a caller detects truncation by comparing the count received with the count expected.
377
284
 
378
- ### Float32Array: no per-element conversion
285
+ ### Float32: no per-element conversion, but not zero-copy
379
286
 
380
- `encodeFloat32Arrays` constructs a `Uint8Array` view over the `Float32Array`'s underlying buffer and serialises it as a msgpack `bin` payload — no float-by-float conversion. `decodeFloat32Arrays` reinterprets the decoded `Uint8Array` as a `Float32Array`, copying it first when its `byteOffset` is not 4-byte aligned.
287
+ `encodeFloat32Arrays` wraps the array's bytes in a `Uint8Array` view and serialises it as a `bin`.
288
+ `decodeFloat32Arrays` reinterprets the decoded bytes as a `Float32Array`, copying first when the
289
+ `byteOffset` is not 4-byte aligned. In practice the copy always happens: the decoded `bin` is a
290
+ view starting after a 2-, 3- or 5-byte header, never a multiple of 4. `serialize` and frame
291
+ assembly also copy on the way out. Element type is strictly Float32; there is no negotiation.
381
292
 
382
- **In practice that copy always happens**, and the bytes are copied on the way out as well:
383
- `serialize` copies the view into its output, and the frame is assembled by another copy. The
384
- decoded `bin` is a view into the payload, starting just after its 2-, 3- or 5-byte header, and
385
- none of those offsets is a multiple of 4. So "no per-element conversion" is the whole claim, not
386
- "zero-copy".
293
+ ### Send the view `serialize` returns, not its buffer
387
294
 
388
- ### A trap when writing a transport pump
295
+ `serialize` returns a view over an internal buffer that is not trimmed: a 10 MiB envelope measures
296
+ about 10,485,800 bytes over a 16,777,216-byte `ArrayBuffer`. A transport pump that sends
297
+ `frame.buffer` instead of `frame` puts about 6 MiB of trailing zeros on the wire per message.
389
298
 
390
- `serialize` returns a **view** over an internal buffer that is not trimmed: a 10 MiB envelope
391
- measures ~10,485,800 bytes over a 16,777,216-byte `ArrayBuffer` (~1.6× slack, `byteOffset` 0). Send
392
- the view. A pump that reaches for `frame.buffer` instead would put ~6 MiB of trailing zeros on the
393
- wire per message.
299
+ ### What `deserialize` refuses, and how
394
300
 
395
- ### Dependencies
301
+ `deserialize` throws on input that is not MessagePack, and never logs:
396
302
 
397
- - [`@statewalker/webrun-rpc`](../webrun-rpc) — **types only** (`PortCodec`, `PortEnvelope`); no runtime import is emitted.
398
- - MessagePack itself is `src/msgpack-core.ts`, in this package — see below.
303
+ - Truncated input — any read past the end — throws a `RangeError`:
304
+ `Insufficient data: N byte(s) needed at index I in the MessagePack binary data (length L).`
305
+ Every proper prefix of a valid encoding is refused this way.
306
+ - `Invalid byte code 0xc1 found.` for the never-used type byte.
307
+ - `Invalid argument: The byte array to deserialize is empty.` for empty input.
308
+ - `Invalid argument type: Expected a byte array (Array or Uint8Array) to deserialize.` for a
309
+ non-byte argument.
310
+ - `Invalid data length for a date value.` for a timestamp extension of unknown size.
311
+ - Malformed UTF-8 inside a string is not an error: each bad sequence decodes to U+FFFD, as
312
+ `TextDecoder` does. Overlong forms are malformed, so `C0 AF` does not decode to `/`.
313
+ - Bytes after the first document are ignored unless `{ multiple: true }` is passed.
399
314
 
400
- Dev: TypeScript, vitest, rolldown, rimraf (catalog versions from the monorepo root).
401
- `@statewalker/webrun-streams` and `@statewalker/webrun-streams-conformance` are dev-only, for the
402
- conformance run.
315
+ `serialize` throws `Invalid argument type: The type '<type>' cannot be serialized.` for a
316
+ `bigint`, function or symbol when `invalidTypeReplacement` is not set. A falsy replacement (`null`,
317
+ `0`, `false`, `""`) counts as not set and still throws; to write nil, pass a function such as
318
+ `() => null`. It also throws `Invalid argument: an
319
+ invalid Date cannot be serialized.` for an invalid `Date`, and `Invalid argument type: Expected an
320
+ Array to serialize multiple values.` when `multiple` is set on a non-array.
403
321
 
404
- ### Constraints
322
+ `msgpackCodec` turns every `deserialize` throw into a dropped message.
405
323
 
406
- - Big-endian length prefix only — no little-endian variant.
407
- - `decodeMsgpack` allocates one `Uint8Array` per incoming chunk for the `concat`; long streams with many tiny chunks may benefit from a batched source upstream.
408
- - `Float32Array` codec is strictly `Float32` — no element-size negotiation.
409
- - `msgpackCodec` carries no length prefix, so it is **unusable** on a transport without message
410
- boundaries. Use the stream codec there.
324
+ ### The value model
411
325
 
412
- ## Scripts
326
+ | Written | As | Read back as |
327
+ | --- | --- | --- |
328
+ | `null`, `undefined` | nil | `null`; an object key whose value is `undefined` is dropped |
329
+ | `boolean` | bool | `boolean` |
330
+ | safe integer | narrowest int / uint | `number`; `-0` is written as `0` |
331
+ | any other number (fraction, beyond ±2⁵³, `NaN`, `±Infinity`) | float 64 | `number` |
332
+ | `string` | str, UTF-8; a lone surrogate is written as U+FFFD | `string` |
333
+ | `Uint8Array`, `Uint8ClampedArray` | bin | `Uint8Array`, a view into the input, not a copy |
334
+ | other typed arrays | array of numbers | `number[]` |
335
+ | `Array` | array | `unknown[]` |
336
+ | `Date` | timestamp extension (type -1), 32/64/96-bit as needed | `Date`, floored to the millisecond |
337
+ | any other object | map of its own enumerable string keys | plain object; every key, `__proto__` included, is an own property |
338
+ | `bigint`, `function`, `symbol` | throws, unless `invalidTypeReplacement` supplies a stand-in | |
413
339
 
414
- ```sh
415
- pnpm test # vitest run (406 tests / 7 files)
416
- pnpm run build # rolldown + tsc --emitDeclarationOnly
417
- pnpm lint # biome check src tests
418
- pnpm typecheck # tsc --noEmit (src)
419
- pnpm typecheck:tests # tsc -p tsconfig.tests.json — needs the sibling packages built
420
- ```
340
+ Reading also accepts what other encoders write: float 32, int 64 / uint 64 (as the nearest
341
+ `number`, losing precision beyond 2⁵³), maps with non-string keys (converted with `String()`), and
342
+ extension types other than timestamps, returned as `{ type, data }` with `type` as the unsigned
343
+ byte (type `-2` reads as `254`). A `Map` or `Set` has no own enumerable keys and is written as an
344
+ empty map.
421
345
 
422
- ## Provenance and credits
346
+ ### The MessagePack implementation is a port of msgpack.js
423
347
 
424
348
  `src/msgpack-core.ts` is a TypeScript port of **[msgpack.js](https://github.com/ygoe/msgpack.js)**
425
- by **Yves Goergen**, © 2019, MIT license — the library this package depended on, as
426
- [`@ygoe/msgpack`](https://www.npmjs.com/package/@ygoe/msgpack), until 0.3.0. Thank you.
349
+ by **Yves Goergen**, © 2019, MIT license. Thank you.
427
350
 
428
351
  It is ported from commit
429
352
  [`05733cf`](https://github.com/ygoe/msgpack.js/tree/05733cfb43a2974cf669f0eb8693f43b548bdcd4)
430
- (2024-04-16) on `master`, not from the npm release 1.0.3. `master` carries three fixes that 1.0.3
431
- lacks, and they change what goes on the wire:
353
+ (2024-04-16) on `master`, not from the npm release 1.0.3, because `master` carries three fixes
354
+ that change what goes on the wire:
432
355
 
433
356
  - [#34](https://github.com/ygoe/msgpack.js/pull/34): an integer beyond the safe range is written
434
- as a float 64. 1.0.3 wrote `2 ** 100` as the maximal uint 64, which reads back as ~1.8e19.
435
- - [#32](https://github.com/ygoe/msgpack.js/issues/32): a positive integer above uint 32 is written
436
- with the uint 64 prefix `0xcf`, not int 64's `0xd3`.
357
+ as a float 64, not as the maximal uint 64.
358
+ - [#32](https://github.com/ygoe/msgpack.js/issues/32): a positive integer above uint 32 uses the
359
+ uint 64 prefix `0xcf`, not int 64's `0xd3`.
437
360
  - [#33](https://github.com/ygoe/msgpack.js/issues/33): a 16–255-byte `bin` gets the one-byte bin 8
438
- header, not bin 16's two bytes.
361
+ header.
439
362
 
440
363
  The port keeps upstream's structure, comments and error messages. Before any change it was checked
441
- against upstream `msgpack.js` itself: byte-identical output for 3,000 randomised values (226 MB
442
- encoded), and the same value or the same error message for 20,000 random garbage inputs.
443
-
444
- **Changes from upstream**, each marked `Modified from upstream` in the source and made where a
445
- test adopted from another implementation failed:
446
-
447
- 1. **Truncated input throws a `RangeError`** instead of decoding to `NaN` or a short `bin`, and
448
- nothing is logged — upstream called `console.debug` with the whole input.
449
- 2. **Timestamps are floored to the millisecond**, both ways. Upstream rounded
450
- `…:07.999999999Z` up to the next second, read pre-1970 instants toward zero, and wrote
451
- `new Date(-1002)` as second -1. An invalid `Date` throws instead of being written as second -1.
452
- 3. **Strings follow the WHATWG Encoding standard.** Decoding goes through `TextDecoder`: malformed
453
- sequences, overlong forms included, become U+FFFD, where upstream decoded `C0 AF` to `/` and
454
- threw on a sequence cut by the end of the string. Encoding writes a lone surrogate as U+FFFD,
455
- where upstream threw on a high one and wrote a low one as invalid UTF-8.
456
- 4. **Object keys**: a `__proto__` map key is decoded as an own property — upstream's assignment
457
- replaced the decoded object's prototype — and only own enumerable keys are written, where
458
- upstream's `for…in` also wrote inherited ones.
364
+ against upstream itself: byte-identical output for 3,000 randomised values, and the same value or
365
+ the same error message for 20,000 random garbage inputs. Each change below is marked
366
+ `Modified from upstream` in the source and was made where a test adopted from another
367
+ implementation failed:
368
+
369
+ 1. Truncated input throws a `RangeError` instead of decoding to `NaN` or a short `bin`, and
370
+ nothing is logged (upstream called `console.debug` with the whole input).
371
+ 2. Timestamps are floored to the millisecond both ways. Upstream rounded `…:07.999999999Z` up to
372
+ the next second, read pre-1970 instants toward zero, and wrote `new Date(-1002)` as second -1.
373
+ An invalid `Date` throws.
374
+ 3. Strings follow the WHATWG Encoding standard: decoding goes through `TextDecoder`, and encoding
375
+ writes a lone surrogate as U+FFFD, where upstream threw on a high one and wrote a low one as
376
+ invalid UTF-8.
377
+ 4. A `__proto__` map key is decoded as an own property (upstream's assignment replaced the
378
+ prototype), and only own enumerable keys are written (upstream's `for…in` also wrote inherited
379
+ ones).
459
380
  5. TypeScript types, `unknown` in place of `any`, and the bounds `0xffffffffffffffff` /
460
381
  `0x7fffffffffffffff` spelled `2 ** 64` / `2 ** 63` (the same doubles).
461
382
 
462
- **Tests.** Besides this package's own, the suite runs:
383
+ Besides this package's own tests, the suite runs upstream's test page ported to vitest
384
+ (`tests/msgpack-core.upstream.test.ts`);
385
+ [kawanet/msgpack-test-suite](https://github.com/kawanet/msgpack-test-suite) (MIT, © Yusuke
386
+ Kawasaki), vendored (`tests/msgpack-core.test-suite.test.ts`); and cases adopted from
387
+ [msgpack/msgpack-javascript](https://github.com/msgpack/msgpack-javascript) (ISC, © The
388
+ MessagePack Community) and [kriszyp/msgpackr](https://github.com/kriszyp/msgpackr) (MIT, © Kris
389
+ Zyp), with msgpackr's sample documents vendored (`tests/msgpack-core.adopted.test.ts`). Sources,
390
+ commits and licenses of the vendored fixtures are in
391
+ [`tests/fixtures/README.md`](./tests/fixtures/README.md). Outside the suite, the implementation was
392
+ checked for interoperability with `@msgpack/msgpack` 3.1.3 and `msgpackr` 2.1.0: 2,000 random
393
+ values encoded by each side decode identically on the other, in both directions.
394
+
395
+ ### Dependencies
396
+
397
+ - Runtime: none. `dist/index.js` imports nothing.
398
+ - `@statewalker/webrun-rpc` (`dependency`, type-only): `PortCodec`, `PortEnvelope`.
399
+ - Dev only: `@statewalker/webrun-streams` and `@statewalker/webrun-streams-conformance` for the
400
+ conformance run; TypeScript, vitest, rolldown, rimraf, Biome.
463
401
 
464
- - upstream's test page, ported to vitest — `tests/msgpack-core.upstream.test.ts`;
465
- - [kawanet/msgpack-test-suite](https://github.com/kawanet/msgpack-test-suite) (MIT, © Yusuke
466
- Kawasaki), vendored — `tests/msgpack-core.test-suite.test.ts`;
467
- - cases adopted from [msgpack/msgpack-javascript](https://github.com/msgpack/msgpack-javascript)
468
- (ISC, © The MessagePack Community) and [kriszyp/msgpackr](https://github.com/kriszyp/msgpackr)
469
- (MIT, © Kris Zyp), with msgpackr's sample documents vendored —
470
- `tests/msgpack-core.adopted.test.ts`.
402
+ ### Scripts
471
403
 
472
- Sources, commits and licenses of everything vendored are in
473
- [`tests/fixtures/README.md`](./tests/fixtures/README.md). Outside the suite, the final
474
- implementation was checked for interoperability with `@msgpack/msgpack` 3.1.3 and `msgpackr` 2.1.0:
475
- 2,000 random values encoded by each side decode identically on the other, in both directions.
404
+ ```sh
405
+ pnpm test # vitest run
406
+ pnpm run build # rolldown + tsc --emitDeclarationOnly
407
+ pnpm lint # biome check src tests
408
+ pnpm typecheck # tsc --noEmit
409
+ pnpm typecheck:tests # tsc -p tsconfig.tests.json; needs the workspace packages built
410
+ ```
476
411
 
477
412
  ## License
478
413
 
479
- MIT © statewalker — see [LICENSE](./LICENSE), which also carries msgpack.js's MIT notice.
414
+ MIT © statewalker. See [LICENSE](./LICENSE), which also carries msgpack.js's MIT notice.
package/package.json CHANGED
@@ -1,10 +1,13 @@
1
1
  {
2
2
  "name": "@statewalker/webrun-msgpack",
3
- "version": "0.3.2",
3
+ "version": "0.4.0",
4
4
  "private": false,
5
5
  "type": "module",
6
6
  "description": "MessagePack for the wire, with no runtime dependencies: serialize/deserialize (a TypeScript port of msgpack.js), a length-prefixed stream codec for iterables (values and Float32Arrays), and msgpackCodec, a PortCodec that carries webrun-rpc port envelopes over a byte transport.",
7
- "homepage": "https://github.com/statewalker/webrun-wire",
7
+ "bugs": {
8
+ "url": "https://github.com/statewalker/webrun-wire/issues"
9
+ },
10
+ "homepage": "https://github.com/statewalker/webrun-wire/tree/main/packages/webrun-msgpack#readme",
8
11
  "author": {
9
12
  "name": "Mikhail Kotelnikov",
10
13
  "email": "mikhail.kotelnikov@gmail.com"
@@ -12,7 +15,7 @@
12
15
  "license": "MIT",
13
16
  "repository": {
14
17
  "type": "git",
15
- "url": "git@github.com:statewalker/webrun-wire.git",
18
+ "url": "git+https://github.com/statewalker/webrun-wire.git",
16
19
  "directory": "packages/webrun-msgpack"
17
20
  },
18
21
  "exports": {
@@ -27,7 +30,7 @@
27
30
  "src"
28
31
  ],
29
32
  "dependencies": {
30
- "@statewalker/webrun-rpc": "^0.4.2"
33
+ "@statewalker/webrun-rpc": "^0.5.0"
31
34
  },
32
35
  "devDependencies": {
33
36
  "@biomejs/biome": "^2.5.15",
@@ -35,8 +38,8 @@
35
38
  "rolldown": "^1.2.12",
36
39
  "typescript": "^7.0.2",
37
40
  "vitest": "^5.0.3",
38
- "@statewalker/webrun-streams": "^0.2.1",
39
- "@statewalker/webrun-streams-conformance": "^0.3.1"
41
+ "@statewalker/webrun-streams-conformance": "^0.3.2",
42
+ "@statewalker/webrun-streams": "^0.2.2"
40
43
  },
41
44
  "sideEffects": false,
42
45
  "publishConfig": {