@sofa-buffers/corelib 0.10.0 → 0.11.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -22,20 +22,24 @@ JavaScript does (Node.js, browsers, Electron, Deno, Bun, a `<script>` tag).
22
22
 
23
23
  Like protobuf's `CodedInputStream` / `CodedOutputStream`, it is meant to be
24
24
  driven by generated code: the `sofabgen` generator emits one class per message
25
- with marshal / unmarshal methods that call these primitives. Two decode models
26
- are offered — a resumable push / visitor decoder for streaming, and a
27
- monomorphic pull cursor (`Cursor`) driven by a single `switch` over the field id.
25
+ with `serialize` / `decode` methods that call these primitives. Decoding has one
26
+ surface, the **visitor** (CORELIB_PLAN §5.3.1): a resumable push decoder that takes
27
+ chunks of any size and calls one method per field.
28
28
 
29
29
  ### Requirements
30
30
 
31
- Node.js 20+ (CI runs 20 / 24), or any modern browser / Electron / Deno /
32
- Bun. Built with TypeScript 6.x; targets ES2020 (`bigint` required).
31
+ Node.js 20+ — CI runs 20 / 22 / 24 / 26 — or any modern browser / Electron /
32
+ Deno / Bun. Built with TypeScript 6.x; targets ES2020 (`bigint` required).
33
33
 
34
34
  ### Dependencies
35
35
 
36
36
  None. Zero runtime dependencies; uses only standard JS / Web APIs
37
37
  (`Uint8Array`, `DataView`, `TextEncoder` / `TextDecoder`).
38
38
 
39
+ ### Feature flags
40
+
41
+ None — the build always ships every wire type.
42
+
39
43
  ### Packaging
40
44
 
41
45
  Published as `@sofa-buffers/corelib`:
@@ -52,43 +56,85 @@ full type declarations.
52
56
  | Goal | How |
53
57
  |------|-----|
54
58
  | Runs everywhere | Pure TypeScript over `Uint8Array` / `DataView` / `TextEncoder`, no Node built-ins on the hot path. |
55
- | Streaming **out** | `OStream` writes into a small caller buffer and calls a `FlushSink` when it fills, so a message can exceed the buffer. |
59
+ | Streaming **out** | `OStream` writes into a small caller buffer and calls a `FlushSink` when it fills, so a message can exceed the buffer — by any amount, down to a one-byte buffer: a value too large for the buffer is split across flushes. |
56
60
  | Streaming **in** | `IStream` is a resumable state machine fed arbitrary chunks; large string / blob payloads arrive in pieces. |
57
- | Fast whole-buffer decode | With the whole message in one buffer, `decode()` (push) and `Cursor` (pull) advance a single cursor. |
58
- | Full 64-bit fidelity | Scalars round-trip the entire `uint64` / `int64` range: `number` when exact, `bigint` beyond `2^53-1` (`Long` offers a `bigint`-free array path). |
59
- | Generated-code friendly | The pull `Cursor` gives a monomorphic `readHeader()` + typed `read*` loop; the push `Visitor` has all-optional methods. |
60
- | Reserve-offset | `new OStream(buf, offset)` leaves room at the front for a lower-layer header, saving a copy. |
61
- | Explicit endianness | IEEE-754 values are read / written little-endian via `DataView`, identical on every engine. |
62
- | Pluggable acceleration | The encoder's bulk array paths run through a swappable `Kernel`; the default is pure TypeScript. |
61
+ | One decode surface | The visitor, and nothing beside it (§5.3.1). `decode()` is that decoder fed once, so a whole-buffer decode runs the same code and the same rules as a chunked one. |
62
+ | Full 64-bit fidelity | Scalars round-trip the entire `uint64` / `int64` range: `number` when exact, `bigint` beyond `2^53-1`, and every integer callback carries the exact `lo` / `hi` halves beside the value for a `bigint`-free consumer (`Long`). |
63
+ | Generated-code friendly | One flat `Visitor` per message, all methods optional; nesting arrives as `sequenceBegin` / `sequenceEnd` events carrying id and depth, which generated code routes on. |
64
+ | Reserve-offset | `new OStream(buf, offset)` leaves room at the front for a lower-layer header, saving a copy. The offset belongs to that installation and is consumed by the flush that hands the unit over; `setBuffer(buf, offset)` from inside the sink re-arms it, for header room in every packet. |
65
+ | Caller-owned buffers | The encoder allocates no output buffer, grows none, and has no hook that could grow one for it: it writes into yours, and when it fills it flushes to your sink, which may install the next buffer. `growingOStream()` is that caller ready-made — a scratch buffer with a sink that accumulates the result. |
66
+ | No payload storage in the codec | After construction the encoder and decoder allocate no storage a wire number sizes (§6.6) — no views, no scratch, no growable state — apart from the **language-forced handles** of §6.6.2, itemised under [Memory handling](#memory-handling). Verified two ways: no allocation primitive on a codec path, and a flat heap over a complete encode and decode. |
67
+ | No views | Nothing the decoder hands over aliases anything it owns (§6.7). A payload arrives as a range of the chunk **you** fed, so what you keep, you copied. |
68
+ | Explicit endianness | IEEE-754 values are read / written little-endian — bit-for-bit identical on every engine, big-endian hosts included. |
69
+ | Pluggable acceleration | The encoder's bulk array paths run through a swappable `Kernel`, and that interface is the entire seam: `setKernel(yourKernel)`. **No accelerated backend exists today** — the kernel is the pure-TypeScript one on every host unless you build and install your own (native addon or WASM); the library ships no loader for one. |
63
70
 
64
71
  ## Usage
65
72
 
66
73
  The codec has four use cases — serialize a message that fits in one buffer,
67
74
  serialize one too large for the buffer (streamed out in chunks), deserialize a
68
75
  whole message, and deserialize one arriving in chunks — plus the generated-code
69
- path that wraps them. Problems are reported by throwing `SofabError`; the cause is
70
- on `SofabError.code` (`ARGUMENT`, `USAGE`, `BUFFER_FULL`, `INVALID_MSG`,
71
- `INCOMPLETE`). The decoder splits its two failure kinds (MESSAGE_SPEC §7):
72
- `INVALID_MSG` is a message malformed regardless of what follows, while
73
- `INCOMPLETE` means the bytes merely ended *inside* a field — a truncation more
74
- bytes could complete, which is not an error the caller must treat as one. There
75
- is no finish/finalize step: a streaming decode reports `INCOMPLETE` from `end()`
76
- (see below), never by promoting it to a throw.
76
+ path that wraps them.
77
+
78
+ Problems are reported by throwing `SofabError`; the cause is on
79
+ `SofabError.code` (`ARGUMENT`, `BUFFER_FULL`, `INVALID_MSG`, `INCOMPLETE`,
80
+ `LIMIT_EXCEEDED`), and that is the whole set. A read whose declared type
81
+ contradicts the field on the wire is not an error at all: the field is *skipped*
82
+ like an unknown id, the destination is left untouched and the decode stays
83
+ `COMPLETE`. `INVALID_MSG` is a message malformed regardless of what follows;
84
+ `INCOMPLETE` means the bytes merely ended *inside* a field, and is reported by
85
+ what `feed()` returns rather than thrown — there is no finish/finalize step.
86
+ `LIMIT_EXCEEDED` is neither: it is a receiver-local *policy* rejection, a field
87
+ larger than a cap **you** configured (see [Receiver limits](#receiver-limits)).
77
88
 
78
89
  ### Serialize
79
90
 
80
- Write fields into an in-memory `OStream` and take a view of the finished bytes:
91
+ `OStream` writes into the buffer **you** hand it: the library allocates no output
92
+ buffer and never grows one it was given. Where the schema bounds the message,
93
+ that is one buffer of `MAX_SIZE` bytes:
81
94
 
82
95
  ```ts
83
96
  import { OStream } from "@sofa-buffers/corelib";
84
97
 
85
- const os = new OStream(); // in-memory, auto-growing buffer
98
+ const os = new OStream(new Uint8Array(MAX_SIZE)); // your buffer, sized from the schema
86
99
  os.writeUnsigned(1, 42);
87
100
  os.writeSigned(2, -7);
88
101
  os.writeString(3, "hi");
89
102
  const bytes = os.bytes(); // Uint8Array view of the finished message
90
103
  ```
91
104
 
105
+ Where it does not — no `maxlen` / `count` to size from — `growingOStream()` owns
106
+ a buffer, hands it to the encoder like any other caller and replaces it with a
107
+ bigger one of its own as the message grows:
108
+
109
+ ```ts
110
+ import { growingOStream } from "@sofa-buffers/corelib";
111
+
112
+ const os = growingOStream(); // the accumulator owns the buffer
113
+ os.writeUnsigned(1, 42);
114
+ os.writeSigned(2, -7);
115
+ os.writeString(3, "hi");
116
+ const bytes = os.bytes(); // the whole message; never throws BUFFER_FULL
117
+ ```
118
+
119
+ Everything below is written against `OStream`, and every one of its `write*`
120
+ methods works the same on the stream `growingOStream()` returns.
121
+
122
+ Every integer written — scalar **or array element** — is checked against the
123
+ 64-bit value domains: unsigned `0 .. 2^64 - 1`, signed `-2^63 .. 2^63 - 1`.
124
+ Anything outside them, and any `number` that is not an integer at all (a
125
+ fraction, `NaN`, `±Infinity`), throws `SofabError` with code `ARGUMENT` rather
126
+ than a bare `RangeError`; the encoder never reduces a value modulo 2^64 and never
127
+ puts a wrapped one on the wire. That answer does not depend on how the encoder
128
+ was constructed, nor on the installed `Kernel`, which carries the same
129
+ obligation.
130
+
131
+ The byte-level `writeFixlen(id, data, subtype)` is checked the same way against
132
+ the fixlen domain: subtypes `0x4`–`0x7` are **reserved**, and an `fp32` / `fp64`
133
+ payload is **exactly** 4 / 8 bytes. Either mistake throws `ARGUMENT` before a
134
+ byte is written. `String` and `Blob` still take any length up to `FIXLEN_MAX`
135
+ (`0x7fffffff`); the typed `writeFp32` / `writeFp64` / `writeString` are correct
136
+ by construction.
137
+
92
138
  ### Serialize stream
93
139
 
94
140
  Constructed over a caller-owned buffer with a `FlushSink`, `OStream` drains that
@@ -98,7 +144,11 @@ small buffer whenever it fills, so the buffer never has to be message-sized:
98
144
  import { OStream, type FlushSink } from "@sofa-buffers/corelib";
99
145
 
100
146
  const out: number[] = [];
101
- const sink: FlushSink = (chunk) => out.push(...chunk); // or socket / file / stream
147
+ // The sink is handed the installed buffer and the region's bounds — never memory
148
+ // from anywhere else (§5.1.6), and never a view the encoder built (§6.6).
149
+ const sink: FlushSink = (buf, start, end) => { // or socket / file / stream
150
+ for (let i = start; i < end; i++) out.push(buf[i]!);
151
+ };
102
152
  const os = new OStream(new Uint8Array(16), 0, sink); // tiny 16-byte buffer
103
153
  for (let i = 0; i < 1000; i++) os.writeUnsigned(i, BigInt(i));
104
154
  os.flush(); // push the tail
@@ -107,13 +157,12 @@ os.flush(); // push the tail
107
157
  ### Nested sequences
108
158
 
109
159
  A nested message is a *sequence*: a fresh id scope between a begin header and the
110
- `0x07` end marker. MESSAGE_SPEC §2 omits a sequence-typed **field** whose value
111
- equals its declared default, so the encoder holds the begin header back until the
112
- sequence proves it has content — no buffering of the sub-message, and nothing to
113
- compare byte images against:
160
+ `0x07` end marker. A sequence-typed **field** whose value equals its declared
161
+ default is omitted from the wire, so the encoder holds the begin header back
162
+ until the sequence proves it has content — no buffering of the sub-message:
114
163
 
115
164
  ```ts
116
- const os = new OStream();
165
+ const os = growingOStream();
117
166
  os.writeUnsigned(1, 42);
118
167
  os.writeSequenceBeginLazy(2); // a nested field...
119
168
  os.writeSequenceEnd(); // ...that got no content: header and end both vanish
@@ -132,16 +181,14 @@ by the value:
132
181
  | wrapper-array **element**, or an array field differing from a non-empty declared default | `writeSequenceEndKeep()` — always emits `begin` + `end` |
133
182
 
134
183
  An element keeps its frame because element presence is what carries a dynamic
135
- array's length (highest present id + 1, §5.1); dropping an all-default element
136
- would change the decoded length, not just the bytes. The failure directions are
137
- not symmetric, so `writeSequenceEndKeep()` is the safe choice when in doubt: a
138
- needless one costs a non-canonical empty frame that a decoder normalizes away,
139
- while a wrong `writeSequenceEnd()` shortens an array. Raw transcoding — replaying
140
- bytes rather than encoding a schema value — should use `writeSequenceEndKeep()`
141
- throughout, so the output reproduces the input frame for frame.
184
+ array's length (highest present id + 1): dropping an all-default element would
185
+ shorten the array. `writeSequenceEndKeep()` is the safe choice when in doubt — a
186
+ needless one costs only a non-canonical empty frame that a decoder normalizes
187
+ away — and raw transcoding, replaying bytes rather than encoding a schema value,
188
+ uses it throughout so the output reproduces the input frame for frame.
142
189
 
143
190
  ```ts
144
- const os = new OStream();
191
+ const os = growingOStream();
145
192
  os.writeSequenceBeginLazy(4); // the wrapper array
146
193
  os.writeSequenceBeginLazy(0); // element 0 — has content
147
194
  os.writeUnsigned(0, 7);
@@ -155,14 +202,15 @@ os.bytes(); // 26 06 00 07 07 0e 07 07
155
202
  Decoding is unaffected by the distinction: an empty frame is valid input that the
156
203
  message layer normalizes to the default, and an absent sequence field is
157
204
  reconstructed from the schema default. Nesting is capped at `MAX_DEPTH` (255) on
158
- both sides; the encoder holds headers back to that full depth, so its output is
159
- canonical however deep a message nests. A held-back header is encoder state, never
160
- buffer content, so streaming through a small buffer produces the same bytes.
205
+ both sides, and the encoder holds headers back to that full depth. A held-back
206
+ header is encoder state, never buffer content, so streaming through a small
207
+ buffer produces the same bytes.
161
208
 
162
209
  ### Deserialize
163
210
 
164
- `decode()` walks a whole buffer and calls one optional `Visitor` method per field;
165
- unhandled fields are silently skipped:
211
+ `decode()` walks a whole buffer and calls one optional `Visitor` method per field; a
212
+ field whose callback you did not implement is skipped. The visitor is **flat**: one
213
+ object receives the whole message, nested scopes included.
166
214
 
167
215
  ```ts
168
216
  import { decode, type Visitor } from "@sofa-buffers/corelib";
@@ -178,152 +226,704 @@ class My implements Visitor {
178
226
  decode(bytes, new My());
179
227
  ```
180
228
 
229
+ There are exactly two things to do with a field — **read** it or **skip** it
230
+ (§6.7.2) — and not implementing a callback is how you say the second.
231
+
232
+ `fieldBegin(id, wire)` is announced first for every field — right after the header
233
+ varint, before the value and before the value's *own* header word (a fixlen length
234
+ word, an array count word, a nested sequence's fields). It gives a reader the field
235
+ stream in wire order without writing the eight value callbacks. The sequence-*end*
236
+ marker gets none: it closes a scope rather than opening a field, and its id is
237
+ discarded (§4.9).
238
+
239
+ **Do not apply a schema bound from it.** An element id past the declared `count`
240
+ looks decidable from the id alone, and is not: that bound applies only to a field
241
+ whose *subtype* has confirmed it is the declared one, so it belongs on
242
+ `fixlenBegin`. A message ending inside the fixlen word is `INCOMPLETE` even when
243
+ the id would violate the bound. Throwing from `fieldBegin` is still how you
244
+ reject a field the header alone settles — an id you will not accept in any shape.
245
+ Everything schema-shaped stays on the later, more informative hook: a fixlen
246
+ subtype and a declared length on `fixlenBegin`, a declared element count on
247
+ `arrayBegin`.
248
+
249
+ An array's **elements** arrive through `arrayBulk(id, kind, count)`, and only
250
+ there: return the destination they should be written into — a `number[]`, an
251
+ exact-width typed array (`Uint16Array`, `Int32Array`, …), a `Long[]`, a pair of
252
+ `Uint32Array` halves, a `Float32Array` / `Float64Array`, or the raw `fp32` words —
253
+ together with the schema's element bound as `min`/`max` 32-bit halves, and the
254
+ decoder fills it directly. Return `null` (or declare no `arrayBulk`) and the
255
+ elements are walked over without being decoded at all. There is no callback per
256
+ element: one array is one call.
257
+
258
+ The **exact-width** destination (`typed`) is for a field whose declared width is a
259
+ typed array's own — a `u16` array into a `Uint16Array`, a `u64` array into a
260
+ `BigUint64Array`. It stores unboxed, it costs
261
+ two bytes an element rather than a tagged slot, and it lets the *encoder* know the
262
+ width too: `writeUnsignedArray` reads a `Uint16Array` element without the type and
263
+ range guards a `number[]` element needs, and reserves three bytes an element rather
264
+ than ten, which is what a caller-sized buffer has room for.
265
+
266
+ A **64-bit** destination is filled through the two 32-bit halves of each element,
267
+ written into a `Uint32Array` over the array's own buffer — `b[i] = 5n` would demand
268
+ a `bigint` per element, and this builds none. The same view reads them back when
269
+ such an array is encoded. A **`boolean`** array has a destination of its own,
270
+ `bool`, one byte per element: §4.4 gives a boolean no width bound, so a wire value
271
+ of 256 is `true` and must NOT mask to `0` — the decoder normalizes every non-zero
272
+ to `1`, which is also the only value an encoder may write back.
273
+
274
+ The element bound is still compared, and that is not a formality: a typed array
275
+ *masks* on store (`a[0] = 70000` in a `Uint16Array` is 4464), while an element
276
+ outside the declared width is a malformed message. So the width is matched against
277
+ the bound once, at the hand-off — a destination narrower than the bound is refused
278
+ with `Argument`, never silently truncated — and the fill then compares exactly as
279
+ `values` does.
280
+
281
+ ```ts
282
+ const big: number[] = [];
283
+ const v: Visitor = {
284
+ arrayBulk: (id) =>
285
+ id === 6 ? { values: big, minLo: 0, minHi: 0, maxLo: 0xffff, maxHi: 0 } : null,
286
+ };
287
+ ```
288
+
289
+ `sequenceBegin(id, depth)` opens a nested scope and `sequenceEnd(id, depth)` closes
290
+ it — the same visitor receives the scope's fields, with their own ids and
291
+ `depth + 1`. Route on the `(id, depth)` pair, which a schema fixes statically:
292
+
293
+ ```ts
294
+ let inChild = false;
295
+ const v: Visitor = {
296
+ sequenceBegin: (id, depth) => { if (id === 3 && depth === 1) inChild = true; },
297
+ sequenceEnd: (id, depth) => { if (id === 3 && depth === 1) inChild = false; },
298
+ unsigned: (id, value) => { /* `inChild` says which scope this id belongs to */ },
299
+ };
300
+ ```
301
+
302
+ Return **`false`** from `sequenceBegin` to decline the whole subtree: no callback of
303
+ any kind fires inside it, a scope opened within it is never offered either, and no
304
+ `sequenceEnd` arrives for what was declined.
305
+
306
+ A declined subtree is still parsed — a sequence is framed by markers rather than by
307
+ a length, so its end has to be found — but nothing is decoded into existence for
308
+ it, and no receiver cap fires inside it: the callback that would have compared one
309
+ is never called. Format ceilings (`ARRAY_MAX`, `FIXLEN_MAX`, `MAX_DEPTH`, the varint
310
+ bound) apply inside a declined subtree exactly as outside it.
311
+
181
312
  ### Deserialize stream
182
313
 
183
- `IStream` resumes across chunk boundaries, so feed it whatever the transport hands
184
- you — from any source — and read the outcome from `end()`. String / blob payloads
185
- arrive as one or more chunks tagged with the field's `total` length and byte
186
- `offset`:
314
+ `IStream` resumes across chunk boundaries: feed it whatever the transport hands
315
+ you and read the outcome from what `feed()` **returns** — `COMPLETE` or
316
+ `INCOMPLETE` for the bytes consumed so far, the third outcome being thrown rather
317
+ than returned (below). There is no end / finalize step and no second way to ask.
318
+ The visitor is bound at construction. String / blob payloads arrive in one or more pieces, each a range
319
+ `[start, end)` of the chunk **you** fed, tagged with the field's `total` length and
320
+ the piece's `offset` within it:
187
321
 
188
322
  ```ts
189
323
  import { IStream, DecodeStatus, type Visitor } from "@sofa-buffers/corelib";
190
324
 
191
325
  const visitor: Visitor = {
192
- blob(id, total, offset, chunk) {
193
- /* append `chunk` at `offset`; the field is `total` bytes */
326
+ blob(id, total, offset, src, start, end) {
327
+ /* copy `src[start..end)` to `offset` of a `total`-byte destination of yours */
194
328
  },
195
329
  };
196
330
 
197
- const is = new IStream();
198
- for await (const chunk of source) is.feed(chunk, visitor); // any async byte source
199
- // end() is a pure accessor — it never throws and never promotes an incomplete
200
- // decode to an error (MESSAGE_SPEC §7). The caller owns end-of-input.
201
- if (is.end() !== DecodeStatus.Complete) {
331
+ const is = new IStream(visitor);
332
+ let status = DecodeStatus.Complete; // zero bytes end on a boundary
333
+ for await (const chunk of source) {
334
+ status = is.feed(chunk); // any async byte source
335
+ }
336
+ // feed() never throws for a merely incomplete decode and never promotes one to
337
+ // an error (MESSAGE_SPEC §7). The caller owns end-of-input.
338
+ if (status !== DecodeStatus.Complete) {
202
339
  // stream ended inside a field (INCOMPLETE) — wait for more bytes, or treat
203
340
  // the truncation as an error if this really was the end of input.
204
341
  }
205
342
  ```
206
343
 
344
+ The chunk is borrowed **only for the duration of `feed`** (§6.0): once it returns
345
+ you may reuse, overwrite or free it, and what you decoded is unaffected — the
346
+ decoder retains nothing that points into it. Copy what you want to keep, during the
347
+ call; `PayloadAcc` and `decodeUtf8` are the ready-made way.
348
+
349
+ **`feed()` is the only way to ask.** There is no `status()` accessor and no end
350
+ step: what a `feed` returns, or what it throws, is the whole answer, so you are
351
+ never one call short of knowing where you stand and never holding two answers that
352
+ could disagree. (This library shipped that disagreement once — a `status()` that
353
+ answered `COMPLETE` for a message `feed` had already refused — which is why the
354
+ second way to ask is gone rather than repaired.) If you want the outcome again
355
+ without keeping it, feed an empty chunk: it consumes nothing and returns the same
356
+ value.
357
+
358
+ `INVALID` is **terminal**, and is the outcome `feed()` never returns: it travels on
359
+ the error channel, as a thrown `INVALID_MSG`. A stream that has thrown it is
360
+ poisoned for good — every further `feed` re-throws it without consuming a byte or
361
+ calling the visitor, so a refused stream can never hand back a status at all:
362
+
363
+ ```ts
364
+ import { SofabError, SofabErrorCode } from "@sofa-buffers/corelib";
365
+
366
+ const is = new IStream(visitor);
367
+ try {
368
+ for await (const chunk of source) is.feed(chunk);
369
+ } catch (e) {
370
+ if ((e as SofabError).code !== SofabErrorCode.InvalidMsg) throw e;
371
+ // The verdict is in hand: `code` says INVALID_MSG, and nothing further needs
372
+ // asking. Feeding on would only raise the same error again.
373
+ }
374
+ ```
375
+
376
+ A receiver-side cap (`LIMIT_EXCEEDED`, see [Receiver limits](#receiver-limits)) is
377
+ **not** the `INVALID` outcome and is never folded into one: the bytes are well-formed
378
+ and decode under a looser cap. It travels the same error channel under its own code,
379
+ and is terminal in the same way — the code you catch is what tells the two apart.
380
+
381
+ ### 64-bit values without `bigint`
382
+
383
+ The default 64-bit surface is *number-first*: a value that fits exactly comes
384
+ back as a `number`, and only past `2^53-1` is a `bigint` materialised — so the
385
+ runtime type of a `u64` / `i64` depends on the value. `Long`, a value carried as two
386
+ unsigned 32-bit halves (`.low` / `.high`), is the fixed-type alternative on the
387
+ **encode** side. It is **representation-only**: the wire is identical to the
388
+ `number | bigint` path, byte for byte.
389
+
390
+ ```ts
391
+ import { Long, growingOStream } from "@sofa-buffers/corelib";
392
+
393
+ const os = growingOStream();
394
+ os.writeUnsignedLong(1, Long.fromValue(2n ** 63n)); // scalar
395
+ os.writeSignedLong(2, Long.fromValue(-(2n ** 62n)));
396
+ os.writeUnsignedArrayLong(3, [1n, 2n].map(Long.fromValue)); // array
397
+ ```
398
+
399
+ On the decode side there is no channel to switch on: **every** integer callback
400
+ carries the exact 64 bits as two unsigned 32-bit halves, beside the number-first
401
+ value, and an array hands its elements over as `Long`s or as raw halves. Read
402
+ whichever you want — the halves cost nothing to pass and nothing to ignore, and a
403
+ `Long` built from them never goes through `bigint` arithmetic:
404
+
405
+ ```ts
406
+ import { ArrayKind, decode, Long, type Visitor } from "@sofa-buffers/corelib";
407
+
408
+ const longs: Long[] = [];
409
+ const v: Visitor = {
410
+ unsigned(id, value, lo, hi) { const x = Long.fromBits(lo, hi); },
411
+ signed(id, value, lo, hi) { /* lo/hi are the decoded two's-complement halves */ },
412
+ // One call per array, not per element: the decoder fills `longs` itself. A
413
+ // destination has to match the element kind, so decline the ones it does not:
414
+ // `null` costs nothing but the call.
415
+ arrayBulk: (id, kind) =>
416
+ kind === ArrayKind.Unsigned
417
+ ? { longs, minLo: 0, minHi: 0, maxLo: 0xffffffff, maxHi: 0xffffffff }
418
+ : null,
419
+ };
420
+ decode(bytes, v);
421
+ ```
422
+
423
+ Narrowing back is exact:
424
+ `value.low` for `u8`..`u32`, and `value.low | 0` for `i8`..`i32`.
425
+
207
426
  ### Code generator
208
427
 
209
- `sofabgen` compiles a schema to one class per message with a `marshal` (chaining
210
- `OStream` writes) and a `static decode` driven by a monomorphic pull `Cursor` —
211
- one `switch` over `c.id`. A hand-written stand-in, encoded then decoded:
428
+ `sofabgen` compiles a schema to one class per message with a `serialize` (chaining
429
+ `OStream` writes) and two decode entry points: a `static decode` for a message
430
+ already in one buffer, and a `static decoder()` bound to `IStream` for one arriving
431
+ in chunks — the same generated type driven whole-buffer or incrementally. Both
432
+ drive the same visitor, because there is only one decode surface (§5.3.1): what
433
+ changes is the drive, not the reader. A hand-written stand-in of both halves,
434
+ encoded and decoded each way:
212
435
 
213
436
  ```ts
214
- import { OStream, Cursor } from "@sofa-buffers/corelib";
437
+ import {
438
+ OStream,
439
+ growingOStream,
440
+ IStream,
441
+ DecodeStatus,
442
+ decode,
443
+ type FeedStatus,
444
+ type FlushSink,
445
+ type Visitor,
446
+ } from "@sofa-buffers/corelib";
215
447
 
216
448
  // generated by: sofabgen --lang typescript
217
449
  class Point {
218
450
  x = 0;
219
451
  y = 0;
220
452
 
221
- marshal(os: OStream): void {
453
+ serialize(os: OStream): void {
222
454
  os.writeSigned(1, this.x);
223
455
  os.writeSigned(2, this.y);
224
456
  }
225
457
 
226
458
  static decode(bytes: Uint8Array): Point {
227
- return Point.decodeFrom(new Cursor(bytes));
459
+ return _decodeIntoPoint(bytes, new Point());
228
460
  }
229
461
 
230
- static decodeFrom(c: Cursor): Point {
231
- const p = new Point();
232
- while (c.readHeader()) {
233
- switch (c.id) {
234
- case 1: p.x = Number(c.readSigned()); break;
235
- case 2: p.y = Number(c.readSigned()); break;
236
- // case 3: p.child = Child.decodeFrom(c); break; // nested sequence
237
- default: c.skip(c.wire); break; // forward-compatible
238
- }
239
- }
240
- return p;
462
+ /** The streaming half: a reader bound to the corelib's resumable IStream. */
463
+ static decoder(): PointDecoder {
464
+ return new PointDecoder();
241
465
  }
242
466
  }
243
467
 
468
+ // The decode-into step sits beside the class, not on it: CORELIB_PLAN §6.1.1
469
+ // closes the generated object's surface to encode / decode / try_decode /
470
+ // serialize / deserialize / decoder, and `decode_from` / `decode_into` are two of
471
+ // the spellings it names as forbidden. It stays module-private — reachable from
472
+ // the sibling classes that decode into one another, and from nowhere else.
473
+
474
+ // Decodes into `o`, so a re-opened sequence continues the scope an earlier
475
+ // opening populated (MESSAGE_SPEC §7.4).
476
+ function _decodeIntoPoint(bytes: Uint8Array, o: Point): Point {
477
+ decode(bytes, new PointVisitor(o));
478
+ return o;
479
+ }
480
+
481
+ // generated alongside it: the visitor that fills a Point — the library's only
482
+ // decode surface (§5.3.1), one callback per wire type instead of one `case` per id.
483
+ // A visitor *is* the decode-into step: it writes into the object it was handed.
484
+ class PointVisitor implements Visitor {
485
+ private readonly out: Point;
486
+ constructor(out: Point) { this.out = out; }
487
+
488
+ signed(id: number, v: number | bigint): void {
489
+ if (id === 1) this.out.x = Number(v);
490
+ else if (id === 2) this.out.y = Number(v);
491
+ // no branch for an unknown id — or for a field whose wire type is not
492
+ // `signed` — so it is skipped and the decode stays COMPLETE (§7.3)
493
+ }
494
+ }
495
+
496
+ // ...and the handle Point.decoder() returns: an IStream plus its destination
497
+ class PointDecoder {
498
+ readonly message = new Point();
499
+ private readonly is = new IStream(new PointVisitor(this.message));
500
+
501
+ // The one place the answer is: what feed() returns, or what it throws. A
502
+ // status() accessor here would be a second way to learn the same fact, which
503
+ // is the second way it can be learned wrong — so the generated handle does not
504
+ // grow one either.
505
+ feed(chunk: Uint8Array): FeedStatus { return this.is.feed(chunk); }
506
+ }
507
+
244
508
  const p = new Point(); p.x = 3; p.y = 4;
245
- const os = new OStream(); p.marshal(os);
246
- const wire = os.bytes();
247
- const got = Point.decode(wire); // got.x === 3, got.y === 4
509
+
510
+ // one-shot: encode into memory, decode a whole buffer
511
+ const os = growingOStream(); p.serialize(os);
512
+ const wire = os.bytes().slice();
513
+ const got = Point.decode(wire); // got.x === 3, got.y === 4
514
+
515
+ // streaming out: the same serialize(), over a 4-byte buffer with a sink. The sink
516
+ // is handed the installed buffer and the region's bounds — never memory from
517
+ // anywhere else — so it copies out what it wants to keep.
518
+ const parts: Uint8Array[] = [];
519
+ const sink: FlushSink = (buf, start, end) => { parts.push(buf.slice(start, end)); };
520
+ const so = new OStream(new Uint8Array(4), 0, sink);
521
+ p.serialize(so); so.flush(); // the same bytes, in pieces
522
+
523
+ // streaming in: feed those pieces — or any other chunking — to the decoder
524
+ const dec = Point.decoder();
525
+ let st: FeedStatus = DecodeStatus.Complete; // zero bytes end on a boundary
526
+ for (const part of parts) st = dec.feed(part);
527
+
528
+ // COMPLETE says the bytes so far ended on a field boundary, not that the
529
+ // message is over — the caller's framing decides that, and a still-INCOMPLETE
530
+ // status once the input really has ended is truncation (§5.2.4).
531
+ const streamed = st === DecodeStatus.Complete ? dec.message : null;
248
532
  ```
249
533
 
534
+ A generated visitor takes the nested cases too: a nested message switches the
535
+ router into the child's fields on `sequenceBegin(id, depth)`, and a compact scalar
536
+ array is filled straight into the destination the router hands over at
537
+ `arrayBegin` / `arrayBulk`, so no part of the message is ever buffered whole. Nothing from a fed chunk is retained either — a
538
+ string is decoded and a blob copied on the way into the destination — so a chunk is
539
+ reusable the moment `feed` returns.
540
+
541
+ This example is compiled and executed by the test suite
542
+ (`test/helpers/readme-generator-example.ts`), so it cannot drift from the API.
543
+
250
544
  ## Memory handling
251
545
 
252
546
  Who owns the bytes:
253
547
 
254
- - **Encode (`OStream`).** In-memory `new OStream()` — the library allocates and
255
- auto-grows an internal buffer (never throws `BUFFER_FULL`); `bytes()` returns a
256
- **view** of the finished message, so `.slice()` it if it must outlive the next
257
- write or grow. Streaming `new OStream(buf, offset?, flush?)` — writes into the
258
- caller-owned buffer and never grows; when it fills it drains a view to the
259
- `flush` sink (valid only during that callback) and, with no sink, throws
260
- `BUFFER_FULL`.
261
- - **Decode (`decode()` / `Cursor` / `IStream`).** Input payload bytes are
262
- zero-copy: string / blob chunks and `Cursor.readBlob` are `subarray` **views**
263
- aliasing the input (or, for `IStream`, the chunk you fed). A visitor chunk is
264
- valid **only during that callback**; a `Cursor` view lasts as long as the source
265
- buffer lives. Scalars are delivered by value. Copy (`.slice()`) or decode
266
- (`Cursor.readString` decodes for you) to retain a payload.
267
-
268
- ### Decode limits
269
-
270
- For a schema whose `count` / `maxlen` bounds are omitted, the decoder otherwise
271
- accepts whatever count / length the received message claims. Pass an optional
272
- `DecodeLimits` object to cap that and protect a receiver from a hostile oversized
273
- field:
548
+ - **Encode (`OStream`).** Every buffer the encoder writes into is
549
+ **caller-supplied**: the library allocates none of its own and never grows or
550
+ reallocates one it was handed — `new OStream(buf, offset?, flush?)` writes into
551
+ `buf` and into nothing else. When it fills it calls the `flush` sink with **that
552
+ buffer** and the region's bounds — `(buffer, start, end)`, never memory from
553
+ anywhere else and never a view the encoder built, since pass-through is forbidden
554
+ (§5.1.6) — and continues; the region is valid for the duration of that call. With
555
+ no sink it throws `BUFFER_FULL`. `bytes()` returns a **view** of what is in the
556
+ buffer — with a sink, only the not-yet-flushed tail — so `.slice()` it if it
557
+ must outlive the next write.
558
+ - **The `offset` belongs to the installation, not to the buffer.** It reserves
559
+ room at the front of the unit the buffer-set begins — the constructor or
560
+ `setBuffer` — and handing that unit to the sink **consumes** it: a sink that
561
+ returns without installing a buffer has *copied*, so the encoder keeps writing
562
+ into the same buffer and resumes at `0`, with the whole buffer usable from
563
+ there. A sink that wants header room in **every** flushed unit — one framing
564
+ header per packet — re-arms it by calling `setBuffer(buf, offset)` from inside
565
+ the callback; passing the buffer it already has counts. A sink that *takes* the
566
+ buffer (hands it to a transport, queues it, gives it to DMA) must install a
567
+ replacement before returning. Either way the bytes are the same — only the unit
568
+ sizes differ. `reset()` and `bytes()` follow the current installation, so after
569
+ a flush they are relative to `0`; on a sink-less stream, which can never flush,
570
+ the reservation stands for the life of the encode.
571
+ - **Encode into memory (`growingOStream()`).** The allocating half is the
572
+ caller's role. `growingOStream(initialCapacity?)` is that caller ready-made: a
573
+ scratch buffer installed **with a sink** that accumulates the result (§5.1.2).
574
+ It never throws `BUFFER_FULL`, its `bytes()` is the **whole** message (a view —
575
+ `.slice()` it if it must outlive the next write or growth), and `reset()` keeps
576
+ the buffer it grew to, so a pooled encoder stops allocating. It is an ordinary
577
+ streaming stream otherwise, so `setBuffer` works and means what it always means:
578
+ the not-yet-flushed bytes are dropped and encoding continues into your buffer.
579
+ Pass an `initialCapacity` when you know roughly how large the message is: a
580
+ message built from many small fields grows by doubling, so 100 KB of them costs
581
+ nine enlargements from the 256-byte default. A single large field does not — a
582
+ bulk write tells the accumulator how much contiguous room it wants, so the
583
+ buffer reaches that size in one step and the write keeps its bulk route.
584
+
585
+ It reaches the encoder through `setBuffer(buffer, offset, carried)`, the third
586
+ argument being how many bytes of the message the replacement already holds
587
+ before `offset`. That is what keeps `bytes()` meaning "the message" across an
588
+ enlargement, and it is available to any caller that keeps a message in one
589
+ growing store.
590
+
591
+ Its storage is **carved from a shared slab** while it is small enough (up to
592
+ 4 KiB of an 8 KiB slab). A carve is handed out once and never recycled, so no
593
+ two encoders ever share bytes and no message can read another's; what it
594
+ changes is *lifetime* — a retained `bytes()` view keeps its slab alive, so
595
+ `.slice()` (already the advice for a view that outlives the next write) is also
596
+ what releases it.
597
+ - **`MIN_OUTPUT_BUFFER` = `1`.** The smallest buffer this port accepts *for
598
+ streaming*, exported from the package so a caller can size from it. It is `1`
599
+ because the encoder splits every atomic unit — field header, fixlen word,
600
+ element count, a scalar or array element varint, an `fp32` / `fp64` element —
601
+ across a flush, so a message of any size encodes through a one-byte buffer and
602
+ the bytes produced are identical at every size. It binds a buffer installed
603
+ **with** a sink, at construction and at every mid-stream `setBuffer`:
604
+ `buf.length - offset` must be at least `MIN_OUTPUT_BUFFER`, and a smaller
605
+ window is rejected right there with `ARGUMENT` — never partway through a
606
+ message — leaving the encoder on the buffer it already had. A buffer installed
607
+ **without** a sink has no minimum: no flush can occur, so nothing can be split,
608
+ and a two-byte message encodes into a two-byte buffer.
609
+ - **Decode (`decode()` / `IStream`).** You own the bytes being parsed, and they
610
+ must stay valid only for the duration of the `feed` (or `decode`) call. After it
611
+ returns, reuse, overwrite or free them freely: **nothing the decoder produced
612
+ points into them**.
613
+ - **No views.** The decoder exposes no zero-copy view of a decoded value, no
614
+ payload-position getter and no borrowed value (§6.7) — on the one-shot path
615
+ exactly as on the streaming one, with no option that reinstates one. A `string` /
616
+ `blob` payload is reported in pieces as `(src, start, end)`, where `src` **is**
617
+ the chunk you fed: the decoder builds no view over it and keeps no storage of its
618
+ own, so whatever you want to keep, you copy out of memory you already own, during
619
+ the call. Scalars are delivered by value. If any of this README ever describes a
620
+ borrowed decoded value, either the README or the port is wrong.
621
+ - **No wire value decides an allocation in the codec.** After construction the
622
+ encoder and decoder allocate no payload storage (§6.6), and nothing at all except
623
+ the itemised handles below: no per-message, per-field or per-chunk allocation,
624
+ no growable state, and no
625
+ accumulator for a payload that straddles a chunk — a decoder's whole memory is
626
+ fixed-size state sized from this format's constants (a `MAX_DEPTH` scope stack, a
627
+ partial varint, an 8-byte float landing zone). Constructing an `OStream` /
628
+ `IStream` is the one allocating step, and `decode()` reuses one decoder across
629
+ calls so a one-shot caller does not pay it per message. A `bigint` for an integer
630
+ past `2^53` is not an exception: it is a *value*, not storage, and the `lo` / `hi`
631
+ halves beside it are there for a consumer that would rather not have one. A `Long`
632
+ written into a `longs` or `typed` bulk destination is the same kind of thing — a value, placed
633
+ in storage you supplied.
634
+ - **The language-forced handles, itemised** (§6.6.2). JavaScript will not let a codec
635
+ place or take an IEEE-754 value at a byte offset, or copy a *range* of bytes,
636
+ without building an object first: `TypedArray.set` — the only `memcpy` there is —
637
+ takes a typed array as its source, and a float needs a `DataView`. These are all
638
+ of them:
639
+
640
+ | handle | where | how many |
641
+ |---|---|---|
642
+ | `DataView` over the **output buffer** | `Kernel`, bulk `fp32` / `fp64` arrays | one per bulk call, and only from 64 `fp32` / 16 `fp64` elements up |
643
+ | `DataView` over the **fed chunk** | `IStream`, bulk float array reads | one per chunk, on the first run in it that clears the same thresholds |
644
+ | `subarray` of the caller's payload | `OStream.writeRaw`, as `set`'s source | one per copied piece, only when the payload does not fit the buffer |
645
+ | `Uint32Array` over a **64-bit destination** | `IStream`, an array handed a `BigUint64Array` / `BigInt64Array` | one per array |
646
+ | `Uint32Array` over a **64-bit source** | `Kernel`, `writeUnsignedArray` / `writeSignedArray` from one | one per bulk call |
647
+ | `Uint32Array` + `DataView` over an **`fp32` source** and the output | `Kernel`, `writeFp32Array` from a `Float32Array` | one pair per bulk call, at **every** length |
648
+ | `Uint32Array` over an **`fp32` source** | `OStream`, `writeFp32Array` from a `Float32Array` that does not fit the buffer | one per call, however many flushes it spans |
649
+
650
+ Each addresses storage **you** supplied, each is sized by that storage and never by
651
+ a number from the wire, and none of them leaves the codec. The 64-bit `Uint32Array`
652
+ rows are what a 64-bit element costs instead of a `bigint`: its halves are already
653
+ in hand, and a view over the caller's own array is how they are written without
654
+ building one.
655
+
656
+ A **scalar** float, a float array below the element threshold, and a float array fed
657
+ in chunks too small to hold a long run all build no handle at all: they go through a
658
+ shared 8-byte scratch word, which is fixed state. The one exception is the two `fp32`-source
659
+ rows — an `fp32` array whose source *is* a `Float32Array` already holds the wire words,
660
+ and copying them is what keeps a signaling NaN intact (§4.6/§6.5), so those handles
661
+ are built at any length rather than past a threshold. Streamed through a buffer too
662
+ small for the whole array, the words still go out as words — through the
663
+ `Uint32Array` alone, stored with shifts — so a small buffer gives the same bytes as
664
+ a large one (§5.1.4). `heap-free-codec.test.ts` asserts
665
+ every count in this table exactly, including the short runs that allocate nothing,
666
+ the element one under the threshold, and the two-element `Float32Array` that builds
667
+ the pair anyway. The thresholds and what they were derived from are on
668
+ `FP32_HANDLE_MIN` / `FP64_HANDLE_MIN` in the API documentation.
669
+ - **The bulk array hand-off borrows your destination until `arrayEnd`.**
670
+ `Visitor.arrayBulk` hands the decoder the array, `Long[]` or typed array it should
671
+ fill for one array field; the decoder writes into it ascending from index 0 and
672
+ holds it from the hand-off until that array ends, which on a chunked decode spans
673
+ several `feed` calls. It is dropped there, and on `decode()`'s pooled machine when
674
+ the call returns. The object must stay the same one for the whole array. A typed
675
+ destination must already hold `count` elements; a plain `number[]` / `Long[]`
676
+ grows as it fills and is **cut to the elements written** when the array ends —
677
+ including to zero for an array that is empty on the wire — so reusing one across
678
+ arrays or messages never leaves the previous array's tail behind and its `length`
679
+ after `arrayEnd` is exactly that array's element count. An element the target's bound rejects stops the fill: everything before
680
+ it is written, it and everything after it are not. This is the only reference the
681
+ decoder keeps into your storage between calls.
682
+ - **The static helper layer allocates, on your behalf.** `PayloadAcc`,
683
+ `ElementSeq`, `FramedSeq`, `StringSeq`, `BlobSeq`, `decodeUtf8`, `elementsEqual`,
684
+ `longElementsEqual` and `fp32RawBytes` are the generated layer's
685
+ code shipped here for reuse (ARCHITECTURE §8), not part of the codec: the codec
686
+ never calls them, and they allocate the values they build.
687
+ - **String validity is checked where a string is materialized** (§6.4.5).
688
+ JavaScript strings are a Unicode type, so this port is always strict — but a
689
+ `string` payload piece is *raw wire bytes* and is not validated (it may end
690
+ mid-code-point), so whoever materializes one owns the check.
691
+ `decodeUtf8(bytes, start?, end?)` is that check, exported for exactly this: it
692
+ rejects malformed bytes as `INVALID_MSG` rather than as a platform `TypeError`.
693
+ Rolling your own instead
694
+ means `new TextDecoder("utf-8", { fatal: true })` — the default `TextDecoder`
695
+ silently substitutes `U+FFFD`, which the format forbids in either direction,
696
+ and `TextEncoder` does the same to an unpaired surrogate where this encoder
697
+ refuses it with `ARGUMENT`.
698
+ - **Reassembly is the caller's, with a helper.** The codec holds no payload across
699
+ `feed` calls. `PayloadAcc.take(total, offset, src, start, end)` joins the pieces —
700
+ one accumulator per decoder, since only one payload is ever in flight — and
701
+ returns storage of its own that aliases nothing, on the whole-payload path exactly
702
+ as on the split one. `StringSeq` / `BlobSeq` collect the elements of a `string` /
703
+ `blob` wrapper array; `ElementSeq` holds the index rules for any element kind
704
+ (index bound, gap fill, last-write-wins) and `FramedSeq` is its twin for an element
705
+ whose default is a fresh object — a `struct`, a `union`, a nested row — where one
706
+ shared default would alias every gap of the array onto a single instance;
707
+ `elementsEqual` (and `longElementsEqual`, for `Long`-backed 64-bit arrays, whose
708
+ elements are object identities) is the array form of the omit-if-default test an
709
+ encoder applies before writing a field; `fp32RawBytes` turns the
710
+ 32-bit word `Visitor.fp32` hands over back into the four wire bytes a generated
711
+ message keeps beside an `fp32` it cannot re-encode from a `number` (§6.5).
712
+
713
+ ### Receiver limits
714
+
715
+ **This corelib holds none, by design.** A field the schema leaves unbounded is still
716
+ bounded by the receiver — CORELIB_PLAN §6.2.1 admits no unset state and no unlimited
717
+ mode — but the *numbers* belong to generated code, which knows the schema and the
718
+ deployment, and §6.2.1 is explicit that a codec
719
+
720
+ > **MUST NOT** hold a limit of its own, **MUST NOT** supply a default for one it was
721
+ > not given, **MUST NOT** read an omitted argument as *unlimited*, and **MUST NOT**
722
+ > clamp to one
723
+
724
+ and that a format ceiling reached because no cap was stated
725
+
726
+ > is the **format's** bound, not a receiver cap, and a port **MUST NOT** present it
727
+ > as one.
728
+
729
+ So `decode(bytes, visitor)` and `new IStream(visitor)` take no limits argument. Up to
730
+ v0.10.0 they took a `DecodeLimits` whose absent members fell back to `ARRAY_MAX` /
731
+ `FIXLEN_MAX`; that object is gone, along with the `LIMIT_EXCEEDED` rejections it
732
+ raised against ceilings nobody had configured.
733
+
734
+ | the cap | who states it | who compares it |
735
+ |---|---|---|
736
+ | `max_dyn_array_count` on an array field | generated code | its own `arrayBegin` |
737
+ | `max_dyn_string_len` / `max_dyn_blob_len` on a `string` / `blob` field | generated code | its own `fixlenBegin` |
738
+ | the element **index** of a wrapper array | generated code | `StringSeq` / `BlobSeq` / `ElementSeq` / `FramedSeq`, from `receiverCap` |
739
+ | the element **byte length** of a wrapper array | generated code | `StringSeq` / `BlobSeq`, from `receiverElemMax` |
740
+
741
+ §6.2.1 permits the comparison to run inside the corelib — "A corelib **MAY** take a
742
+ limit as an argument and perform the check itself" — and the collectors do exactly
743
+ that, for the one shape a visitor cannot see: a wrapper array's element length words
744
+ go to the collector, never to the generated visitor. Every one of their bounds is a
745
+ **required** constructor argument with no default, the schema halves included
746
+ (`UNBOUNDED`, `-1`, is the explicit "the schema declared none"):
274
747
 
275
748
  ```ts
276
- const limits = { maxArrayCount: 65536, maxStringLen: 1 << 20, maxBlobLen: 1 << 20 };
277
- decode(bytes, visitor, limits); // one-shot push
278
- new Cursor(bytes, limits); // pull
279
- new IStream(limits); // streaming
749
+ // out, acc, count, maxlen, name, receiverCap, receiverElemMax
750
+ new StringSeq( out, new PayloadAcc(), UNBOUNDED, UNBOUNDED, "tags", 65_536, 1 << 20);
751
+ new BlobSeq( out, new PayloadAcc(), 8, 4096, "parts", 65_536, 1 << 20);
752
+ new ElementSeq( out, defaultElem, UNBOUNDED, "rows", 65_536);
753
+ // out, make, count, name, receiverCap
754
+ new FramedSeq( out, () => new Elem(), 8, "codes", 65_536);
280
755
  ```
281
756
 
282
- An over-limit array count or string / blob length is rejected at the field's
283
- header — **before** the array is sized or any payload is decoded or streamed to
284
- the visitor — by throwing `SofabError` with code
285
- `SofabErrorCode.LimitExceeded`. The decoder never clamps or truncates. Each limit
286
- is independent, and an omitted one means **no cap** (the default is today's
287
- unlimited behavior — the corelib invents no default). `LimitExceeded` is distinct
288
- from `INVALID_MSG`: exceeding a receiver-configured limit is policy, not a
289
- malformed message. Generated code supplies these values from the sofabgen config.
290
-
291
- ## Feature flags
292
-
293
- None — the build always ships every wire type.
757
+ Each pair is **exclusive**, never additive (§6.2.1: a cap "**MUST NOT** be applied to
758
+ a field the schema already bounds"): where the schema declared `count` / `maxlen`
759
+ that bound governs and its violation is `INVALID_MSG`, a statement about *validity*;
760
+ where it declared none the receiver cap governs and its violation is
761
+ `LIMIT_EXCEEDED`, a *policy* rejection on well-formed bytes.
762
+
763
+ A receiver bound that is about to govern and **states no cap** — negative, `NaN`
764
+ (what an omitted argument becomes in JavaScript), `Infinity` — is refused at
765
+ construction with `SofabErrorCode.Argument`. It is neither of the two categories
766
+ above: no receiver policy was set, so there is no `LIMIT_EXCEEDED` to raise
767
+ ("a format ceiling reached because no cap was stated is the **format's** bound …
768
+ and a port **MUST NOT** present it as one"), and no unlimited mode to fall back to
769
+ ("**MUST NOT** read an omitted argument as *unlimited*"). It is a mistake in the
770
+ call, which §6.3 makes `InvalidArgument`. The refusal is fail-closed: nothing is
771
+ decoded through a collector that could not be built. A bound the schema half makes
772
+ inert is not checked — §6.2.1 forbids applying it at all.
773
+
774
+ **What this decoder still owes the layer that holds the numbers** is the enforcement
775
+ point §6.2.1 requires — the count / length header, before the allocation the cap
776
+ exists to prevent, and behind the MESSAGE_SPEC §7.3 tag test:
777
+
778
+ * `arrayBegin(id, kind, count)` is raised at the **count word**, before any element
779
+ is delivered (for a fixlen array, at its element-length word — the element kind is
780
+ unknown until then, and still before any element);
781
+ * `fixlenBegin(id, subtype, total)` is raised at the **length word**, before any
782
+ payload piece;
783
+ * both carry the number a destination gets sized from, so a rejection there costs no
784
+ allocation at all. Reject, never clamp: materialising `limit` elements where the
785
+ wire said more is data corruption wearing a safety jacket.
786
+
787
+ A cap therefore applies **only to a field you read**, and that falls out of the
788
+ structure rather than needing a rule: a field the visitor steps over never reaches
789
+ the callback that holds the number, so a decode that walks past an over-cap field it
790
+ was never going to read stays `COMPLETE` (§6.2.1's "a skipped field is never
791
+ capped"). The **format ceilings** are not yours to waive this way: a count above
792
+ `ARRAY_MAX` or a length above `FIXLEN_MAX` stays `INVALID` whether anyone reads the
793
+ field, because it bounds what the *wire* may express.
794
+
795
+ A cap rejection is raised by throwing `SofabError` with code
796
+ `SofabErrorCode.LimitExceeded`, which propagates out of `feed` / `decode`. It is
797
+ **not** the `INVALID` outcome and is never folded into one — the same bytes decode
798
+ under a looser cap. The error channel is the only place it appears, and that is not
799
+ a gap: the three-valued outcome has no value for "valid, but more than I am
800
+ configured to accept", so there is nothing about it a returned status could have
801
+ said. Catch it and read its `code`.
294
802
 
295
803
  ## Build & test
296
804
 
297
805
  ```bash
298
806
  npm ci
299
807
  npm run typecheck # tsc --noEmit (strict)
300
- npm test # vitest run: vectors, chunked feeding, cursor, errors, round-trips
808
+ npm test # vitest run: vectors, chunked feeding, memory rules, round-trips
301
809
  npm run coverage # vitest run --coverage (v8)
302
810
  npm run build # tsup -> ESM + CJS + IIFE + .d.ts in dist/
303
811
  npm run smoke # cross-runtime smoke test of the built bundle
304
812
  ```
305
813
 
306
- Tests live in `test/` as focused suites, including `vectors.test.ts` (encode +
307
- decode every shared conformance vector), `istream.chunked.test.ts` (every vector
308
- fed one byte at a time), `cursor.test.ts`, `errors.test.ts`, `ostream.test.ts`,
309
- `roundtrip.test.ts` and more. CI type-checks, tests and builds on Node 20 /
310
- 24, smoke-tests the bundle on Node, Deno and Bun, and publishes coverage badges;
311
- a separate `docs.yml` deploys the TypeDoc API reference to GitHub Pages.
814
+ Tests live in `test/` as focused vitest suites, including `vectors.test.ts` (encode
815
+ + decode every shared conformance vector), `istream.chunked.test.ts` (every vector
816
+ fed one byte at a time), `skip-ids.test.ts` (every vector that carries
817
+ `skip_ids`, decoded by a receiver that ignores those ids at every nesting level —
818
+ contiguous, one byte at a time, and split in two at every byte boundary),
819
+ `heap-free-codec.test.ts` (no allocation primitive on a codec path, a flat heap
820
+ over encode and decode, no view into a fed or one-shot buffer) and `pooled-decoder-state.test.ts` (a decode aborted at every cut point
821
+ leaves nothing behind for the next one).
822
+
823
+ The vector-driven suites each print one summary line — `[vectors] 131
824
+ vectors, none gated out by requires, 524 checks` — so a run says how much of the
825
+ shared suite it actually executed, and a file that arrived truncated or a group
826
+ gated out by `requires` shows up as a smaller number rather than as silence. This
827
+ port compiles no feature out, so nothing is ever gated.
828
+
829
+ `assets/test_vectors.json` carries six blocks and this port runs all six:
830
+ `vectors`, `invalid_utf8`, `sequence_growth`, `header_limits`,
831
+ `header_limits_nested` and `boolean_tolerant`. The file is a
832
+ **verbatim** copy of the one in `corelib-c-cpp`, which authors it. A daily CI job (`.github/workflows/shared-vectors.yml`) compares this copy's sha256 against that file on `corelib-c-cpp@main`, so a copy left behind by an upstream change is reported rather than going unnoticed.
833
+
834
+ `sequence_growth` holds the wrapper-array growth cases of §7.2 item 8, replayed by
835
+ `sequence-growth.test.ts` for both element kinds at three chunkings. This port
836
+ declares `dynamic_arrays`: its wrapper-array containers are JS arrays that grow at
837
+ decode time, so the block applies. The cases are cap-relative and the run installs
838
+ `max_dyn_array_count = 8`. Growth **geometry** splits in two: the backing store's
839
+ reallocation strategy is the engine's amortised doubling, which is not this port's to
840
+ pin, while the fill is, and is asserted as one write per slot in a single pass.
841
+
842
+ `header_limits` holds the truncated over-ceiling headers of §6.2.1 / §6.3 — bytes
843
+ that declare a length or an element count and then end, with no payload behind them —
844
+ replayed by `header-limits.test.ts`. The ceiling answers **at that word**, before the
845
+ payload is asked for, so the verdict is terminal and never `INCOMPLETE`; which
846
+ ceiling the case configures decides the category, `INVALID` for a schema `maxlen` and
847
+ `LIMIT_EXCEEDED` for a §6.2.1 receiver cap. This port declares `receiver_caps`: its
848
+ generated layer carries receiver caps distinct from schema bounds, compared inside the
849
+ visitor's own `fixlenBegin` / `arrayBegin`, which this library raises at the header
850
+ word. Every rejection case is paired with an in-cap control that must still answer
851
+ `incomplete`, and the block is also run with the ceilings lifted — where all six
852
+ rejections fall back to `incomplete`, since this port's `FIXLEN_MAX` (`INT32_MAX`)
853
+ sits above even the amplification case's 1 GiB claim.
854
+
855
+ `header_limits_nested` is that same assertion one or two sequence frames deeper,
856
+ replayed by `header-limits-nested.test.ts` — the axis the flat block leaves
857
+ untested, since every case there sits at the top level. Its cases open a sequence,
858
+ declare the over-ceiling word inside it and end with the frame **still open**, so a
859
+ decoder has a second, independent reason to answer `incomplete` and a port that
860
+ binds its ceiling to the top-level scope looks plausible while capping nothing. The
861
+ runner descends the `frames` chain outermost first, applies the **same leaf** the
862
+ flat block uses (`test/helpers/header-limits.ts`) at the innermost depth, and runs a
863
+ negative control over every rejection with that case's own kind of ceiling lifted to
864
+ 65536: all four then answer `incomplete` instead, which is what shows the verdict was
865
+ the ceiling's rather than an unclosed frame's. All 8 cases run, none gated.
866
+
867
+ `boolean_tolerant` holds §4.4's decode half — bytes carrying `2`, `256` or `2^64-1`
868
+ at a boolean position — replayed by `boolean-tolerant.test.ts`. No conforming
869
+ *encoder* emits such a value, so the positive vectors cannot reach the rule at all:
870
+ those bytes only ever arrive from someone else's encoder. A boolean carries no width
871
+ bound, so every non-zero value is `true` — not `INVALID`, and not a truncation to
872
+ `false` — and the decode is **normalized**, which only the re-encode makes visible.
873
+ Each case is therefore asserted three ways: the outcome is `complete`, the
874
+ destination holds exactly `0` / `1` (poisoned with `0xaa` first, so a decoder that
875
+ never writes cannot pass), and the re-encode of *what was decoded* is byte-compared
876
+ against the block's `reencoded_hex` — `0001`, never `0002`. An array decodes through
877
+ the `bool` destination, which is where this library performs the normalization; a
878
+ scalar boolean has no callback of its own and arrives on `unsigned` with its full 64
879
+ bits, so the `!== 0` test there is the generated layer's and the test performs it,
880
+ after asserting that the delivered value and its two halves agree. In this block an
881
+ unsatisfied `requires` tag means the message must be **rejected**, not skipped
882
+ (§4.4 lifts the width bound the *type* carries, never the one a narrowed *build*
883
+ has) — a path this port never takes, since it compiles no feature out.
884
+
885
+ CI type-checks, tests and builds on Node 20 / 22 / 24 / 26, smoke-tests the
886
+ bundle on Node, Deno and Bun, and publishes coverage badges; a separate
887
+ `docs.yml` deploys the TypeDoc API reference to GitHub Pages.
312
888
 
313
889
  ## Benchmarks
314
890
 
315
- Two standalone tools mirror the other-language ports so implementations can be
316
- compared directly:
891
+ Three standalone tools, specified by `BENCH_SPEC.md` and mirrored in every other
892
+ port — same datasets, same timing rules, same output grammar — so the numbers
893
+ compare directly across languages:
317
894
 
318
895
  ```bash
319
- npm run perf # per-op cost: code-cost figure plus throughput MB/s
320
- npm run bench # throughput table (MB/s) for a u64 array and a mixed message
896
+ npm run perf # per-op cost on the 170-byte perf message
897
+ npm run bench # throughput table (MB/s) over the four shared datasets
321
898
  npm run bench:callgrind # machine-independent instructions/op under Valgrind
322
899
  ```
323
900
 
324
- `perf` and `bench` encode the identical message as their counterparts in the
325
- other ports and print the same report layout. Since JS engines expose no portable
326
- cycle counter, `perf` uses CPU time/op as the code-cost proxy; `bench:callgrind`
327
- counts instructions/op under Valgrind for a fully machine-independent figure.
328
- Running the same tools under Node (V8) and Bun (JavaScriptCore) gives directly
329
- comparable numbers.
901
+ `bench` prints ten rows over four datasets: a 1000-element `u64` array, the small
902
+ `typical` message, an **unbounded 1 MB blob**, and a `composite` message that
903
+ reaches what the flat ones miss — a 64-element wrapper array, 320 bytes of 1-,
904
+ 2-, 3- and 4-byte UTF-8, nesting three deep, a default-valued field the encoder
905
+ must *not* write, and a two-byte field header. Three of the encoded sizes are
906
+ cross-port parity checks (`perf` = 170 bytes, `blob 1MB` = 1,000,005,
907
+ `composite` = 956); `test/bench-datasets.test.ts` holds the datasets to them, and
908
+ `test/bench-grammar.test.ts` holds the tools to the output grammar.
909
+
910
+ Every encode row writes into a **caller-supplied buffer** rather than the
911
+ accumulator. The `blob 1MB` rows are the ones that exercise streaming end to end:
912
+ `one-shot` is a single contiguous write into a 1,000,005-byte buffer, `streaming`
913
+ is the same bytes through a **4096-byte** buffer with a flush sink (~245
914
+ flushes), and `decode: blob 1MB` is fed back in 4096-byte chunks. The
915
+ **difference** between the two encode rows is what the divisible-run flush path
916
+ costs, and it is legible only under `Ir/op`. BENCH_SPEC's optional
917
+ `blob 1MB passthrough` row is absent: pass-through is forbidden (§5.1.6), so every
918
+ `string` / `blob` run is copied through the output buffer. Both copies are
919
+ `TypedArray.set` — the whole payload when it fits, a range per flush when it does
920
+ not, the latter through one of the itemised §6.6.2 handles under
921
+ [Memory handling](#memory-handling).
922
+
923
+ Since JS engines expose no portable cycle counter, `perf` uses CPU time/op as the
924
+ code-cost proxy; `bench:callgrind` counts instructions/op under Valgrind (two rep
925
+ counts per workload, subtracted, on a `--predictable` V8) for a fully
926
+ machine-independent figure. The same tools under Node (V8) and Bun
927
+ (JavaScriptCore) give directly comparable numbers. `tsx bench/bench.ts --smoke`
928
+ runs every row exactly once — a liveness check for the rows, never a
929
+ measurement.