@sofa-buffers/corelib 0.10.0 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +213 -0
- package/README.md +725 -125
- package/dist/index.cjs +2689 -1276
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +1672 -477
- package/dist/index.d.ts +1672 -477
- package/dist/index.global.js +2689 -1276
- package/dist/index.global.js.map +1 -1
- package/dist/index.js +2678 -1273
- package/dist/index.js.map +1 -1
- package/package.json +3 -4
package/README.md
CHANGED
|
@@ -22,20 +22,24 @@ JavaScript does (Node.js, browsers, Electron, Deno, Bun, a `<script>` tag).
|
|
|
22
22
|
|
|
23
23
|
Like protobuf's `CodedInputStream` / `CodedOutputStream`, it is meant to be
|
|
24
24
|
driven by generated code: the `sofabgen` generator emits one class per message
|
|
25
|
-
with
|
|
26
|
-
|
|
27
|
-
|
|
25
|
+
with `serialize` / `decode` methods that call these primitives. Decoding has one
|
|
26
|
+
surface, the **visitor** (CORELIB_PLAN §5.3.1): a resumable push decoder that takes
|
|
27
|
+
chunks of any size and calls one method per field.
|
|
28
28
|
|
|
29
29
|
### Requirements
|
|
30
30
|
|
|
31
|
-
Node.js 20+
|
|
32
|
-
Bun. Built with TypeScript 6.x; targets ES2020 (`bigint` required).
|
|
31
|
+
Node.js 20+ — CI runs 20 / 22 / 24 / 26 — or any modern browser / Electron /
|
|
32
|
+
Deno / Bun. Built with TypeScript 6.x; targets ES2020 (`bigint` required).
|
|
33
33
|
|
|
34
34
|
### Dependencies
|
|
35
35
|
|
|
36
36
|
None. Zero runtime dependencies; uses only standard JS / Web APIs
|
|
37
37
|
(`Uint8Array`, `DataView`, `TextEncoder` / `TextDecoder`).
|
|
38
38
|
|
|
39
|
+
### Feature flags
|
|
40
|
+
|
|
41
|
+
None — the build always ships every wire type.
|
|
42
|
+
|
|
39
43
|
### Packaging
|
|
40
44
|
|
|
41
45
|
Published as `@sofa-buffers/corelib`:
|
|
@@ -52,43 +56,85 @@ full type declarations.
|
|
|
52
56
|
| Goal | How |
|
|
53
57
|
|------|-----|
|
|
54
58
|
| Runs everywhere | Pure TypeScript over `Uint8Array` / `DataView` / `TextEncoder`, no Node built-ins on the hot path. |
|
|
55
|
-
| Streaming **out** | `OStream` writes into a small caller buffer and calls a `FlushSink` when it fills, so a message can exceed the buffer. |
|
|
59
|
+
| Streaming **out** | `OStream` writes into a small caller buffer and calls a `FlushSink` when it fills, so a message can exceed the buffer — by any amount, down to a one-byte buffer: a value too large for the buffer is split across flushes. |
|
|
56
60
|
| Streaming **in** | `IStream` is a resumable state machine fed arbitrary chunks; large string / blob payloads arrive in pieces. |
|
|
57
|
-
|
|
|
58
|
-
| Full 64-bit fidelity | Scalars round-trip the entire `uint64` / `int64` range: `number` when exact, `bigint` beyond `2^53-1`
|
|
59
|
-
| Generated-code friendly |
|
|
60
|
-
| Reserve-offset | `new OStream(buf, offset)` leaves room at the front for a lower-layer header, saving a copy. |
|
|
61
|
-
|
|
|
62
|
-
|
|
|
61
|
+
| One decode surface | The visitor, and nothing beside it (§5.3.1). `decode()` is that decoder fed once, so a whole-buffer decode runs the same code and the same rules as a chunked one. |
|
|
62
|
+
| Full 64-bit fidelity | Scalars round-trip the entire `uint64` / `int64` range: `number` when exact, `bigint` beyond `2^53-1`, and every integer callback carries the exact `lo` / `hi` halves beside the value for a `bigint`-free consumer (`Long`). |
|
|
63
|
+
| Generated-code friendly | One flat `Visitor` per message, all methods optional; nesting arrives as `sequenceBegin` / `sequenceEnd` events carrying id and depth, which generated code routes on. |
|
|
64
|
+
| Reserve-offset | `new OStream(buf, offset)` leaves room at the front for a lower-layer header, saving a copy. The offset belongs to that installation and is consumed by the flush that hands the unit over; `setBuffer(buf, offset)` from inside the sink re-arms it, for header room in every packet. |
|
|
65
|
+
| Caller-owned buffers | The encoder allocates no output buffer, grows none, and has no hook that could grow one for it: it writes into yours, and when it fills it flushes to your sink, which may install the next buffer. `growingOStream()` is that caller ready-made — a scratch buffer with a sink that accumulates the result. |
|
|
66
|
+
| No payload storage in the codec | After construction the encoder and decoder allocate no storage a wire number sizes (§6.6) — no views, no scratch, no growable state — apart from the **language-forced handles** of §6.6.2, itemised under [Memory handling](#memory-handling). Verified two ways: no allocation primitive on a codec path, and a flat heap over a complete encode and decode. |
|
|
67
|
+
| No views | Nothing the decoder hands over aliases anything it owns (§6.7). A payload arrives as a range of the chunk **you** fed, so what you keep, you copied. |
|
|
68
|
+
| Explicit endianness | IEEE-754 values are read / written little-endian — bit-for-bit identical on every engine, big-endian hosts included. |
|
|
69
|
+
| Pluggable acceleration | The encoder's bulk array paths run through a swappable `Kernel`, and that interface is the entire seam: `setKernel(yourKernel)`. **No accelerated backend exists today** — the kernel is the pure-TypeScript one on every host unless you build and install your own (native addon or WASM); the library ships no loader for one. |
|
|
63
70
|
|
|
64
71
|
## Usage
|
|
65
72
|
|
|
66
73
|
The codec has four use cases — serialize a message that fits in one buffer,
|
|
67
74
|
serialize one too large for the buffer (streamed out in chunks), deserialize a
|
|
68
75
|
whole message, and deserialize one arriving in chunks — plus the generated-code
|
|
69
|
-
path that wraps them.
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
`
|
|
73
|
-
`
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
76
|
+
path that wraps them.
|
|
77
|
+
|
|
78
|
+
Problems are reported by throwing `SofabError`; the cause is on
|
|
79
|
+
`SofabError.code` (`ARGUMENT`, `BUFFER_FULL`, `INVALID_MSG`, `INCOMPLETE`,
|
|
80
|
+
`LIMIT_EXCEEDED`), and that is the whole set. A read whose declared type
|
|
81
|
+
contradicts the field on the wire is not an error at all: the field is *skipped*
|
|
82
|
+
like an unknown id, the destination is left untouched and the decode stays
|
|
83
|
+
`COMPLETE`. `INVALID_MSG` is a message malformed regardless of what follows;
|
|
84
|
+
`INCOMPLETE` means the bytes merely ended *inside* a field, and is reported by
|
|
85
|
+
what `feed()` returns rather than thrown — there is no finish/finalize step.
|
|
86
|
+
`LIMIT_EXCEEDED` is neither: it is a receiver-local *policy* rejection, a field
|
|
87
|
+
larger than a cap **you** configured (see [Receiver limits](#receiver-limits)).
|
|
77
88
|
|
|
78
89
|
### Serialize
|
|
79
90
|
|
|
80
|
-
|
|
91
|
+
`OStream` writes into the buffer **you** hand it: the library allocates no output
|
|
92
|
+
buffer and never grows one it was given. Where the schema bounds the message,
|
|
93
|
+
that is one buffer of `MAX_SIZE` bytes:
|
|
81
94
|
|
|
82
95
|
```ts
|
|
83
96
|
import { OStream } from "@sofa-buffers/corelib";
|
|
84
97
|
|
|
85
|
-
const os = new OStream();
|
|
98
|
+
const os = new OStream(new Uint8Array(MAX_SIZE)); // your buffer, sized from the schema
|
|
86
99
|
os.writeUnsigned(1, 42);
|
|
87
100
|
os.writeSigned(2, -7);
|
|
88
101
|
os.writeString(3, "hi");
|
|
89
102
|
const bytes = os.bytes(); // Uint8Array view of the finished message
|
|
90
103
|
```
|
|
91
104
|
|
|
105
|
+
Where it does not — no `maxlen` / `count` to size from — `growingOStream()` owns
|
|
106
|
+
a buffer, hands it to the encoder like any other caller and replaces it with a
|
|
107
|
+
bigger one of its own as the message grows:
|
|
108
|
+
|
|
109
|
+
```ts
|
|
110
|
+
import { growingOStream } from "@sofa-buffers/corelib";
|
|
111
|
+
|
|
112
|
+
const os = growingOStream(); // the accumulator owns the buffer
|
|
113
|
+
os.writeUnsigned(1, 42);
|
|
114
|
+
os.writeSigned(2, -7);
|
|
115
|
+
os.writeString(3, "hi");
|
|
116
|
+
const bytes = os.bytes(); // the whole message; never throws BUFFER_FULL
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
Everything below is written against `OStream`, and every one of its `write*`
|
|
120
|
+
methods works the same on the stream `growingOStream()` returns.
|
|
121
|
+
|
|
122
|
+
Every integer written — scalar **or array element** — is checked against the
|
|
123
|
+
64-bit value domains: unsigned `0 .. 2^64 - 1`, signed `-2^63 .. 2^63 - 1`.
|
|
124
|
+
Anything outside them, and any `number` that is not an integer at all (a
|
|
125
|
+
fraction, `NaN`, `±Infinity`), throws `SofabError` with code `ARGUMENT` rather
|
|
126
|
+
than a bare `RangeError`; the encoder never reduces a value modulo 2^64 and never
|
|
127
|
+
puts a wrapped one on the wire. That answer does not depend on how the encoder
|
|
128
|
+
was constructed, nor on the installed `Kernel`, which carries the same
|
|
129
|
+
obligation.
|
|
130
|
+
|
|
131
|
+
The byte-level `writeFixlen(id, data, subtype)` is checked the same way against
|
|
132
|
+
the fixlen domain: subtypes `0x4`–`0x7` are **reserved**, and an `fp32` / `fp64`
|
|
133
|
+
payload is **exactly** 4 / 8 bytes. Either mistake throws `ARGUMENT` before a
|
|
134
|
+
byte is written. `String` and `Blob` still take any length up to `FIXLEN_MAX`
|
|
135
|
+
(`0x7fffffff`); the typed `writeFp32` / `writeFp64` / `writeString` are correct
|
|
136
|
+
by construction.
|
|
137
|
+
|
|
92
138
|
### Serialize stream
|
|
93
139
|
|
|
94
140
|
Constructed over a caller-owned buffer with a `FlushSink`, `OStream` drains that
|
|
@@ -98,7 +144,11 @@ small buffer whenever it fills, so the buffer never has to be message-sized:
|
|
|
98
144
|
import { OStream, type FlushSink } from "@sofa-buffers/corelib";
|
|
99
145
|
|
|
100
146
|
const out: number[] = [];
|
|
101
|
-
|
|
147
|
+
// The sink is handed the installed buffer and the region's bounds — never memory
|
|
148
|
+
// from anywhere else (§5.1.6), and never a view the encoder built (§6.6).
|
|
149
|
+
const sink: FlushSink = (buf, start, end) => { // or socket / file / stream
|
|
150
|
+
for (let i = start; i < end; i++) out.push(buf[i]!);
|
|
151
|
+
};
|
|
102
152
|
const os = new OStream(new Uint8Array(16), 0, sink); // tiny 16-byte buffer
|
|
103
153
|
for (let i = 0; i < 1000; i++) os.writeUnsigned(i, BigInt(i));
|
|
104
154
|
os.flush(); // push the tail
|
|
@@ -107,13 +157,12 @@ os.flush(); // push the tail
|
|
|
107
157
|
### Nested sequences
|
|
108
158
|
|
|
109
159
|
A nested message is a *sequence*: a fresh id scope between a begin header and the
|
|
110
|
-
`0x07` end marker.
|
|
111
|
-
|
|
112
|
-
sequence proves it has content — no buffering of the sub-message
|
|
113
|
-
compare byte images against:
|
|
160
|
+
`0x07` end marker. A sequence-typed **field** whose value equals its declared
|
|
161
|
+
default is omitted from the wire, so the encoder holds the begin header back
|
|
162
|
+
until the sequence proves it has content — no buffering of the sub-message:
|
|
114
163
|
|
|
115
164
|
```ts
|
|
116
|
-
const os =
|
|
165
|
+
const os = growingOStream();
|
|
117
166
|
os.writeUnsigned(1, 42);
|
|
118
167
|
os.writeSequenceBeginLazy(2); // a nested field...
|
|
119
168
|
os.writeSequenceEnd(); // ...that got no content: header and end both vanish
|
|
@@ -132,16 +181,14 @@ by the value:
|
|
|
132
181
|
| wrapper-array **element**, or an array field differing from a non-empty declared default | `writeSequenceEndKeep()` — always emits `begin` + `end` |
|
|
133
182
|
|
|
134
183
|
An element keeps its frame because element presence is what carries a dynamic
|
|
135
|
-
array's length (highest present id + 1
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
bytes rather than encoding a schema value — should use `writeSequenceEndKeep()`
|
|
141
|
-
throughout, so the output reproduces the input frame for frame.
|
|
184
|
+
array's length (highest present id + 1): dropping an all-default element would
|
|
185
|
+
shorten the array. `writeSequenceEndKeep()` is the safe choice when in doubt — a
|
|
186
|
+
needless one costs only a non-canonical empty frame that a decoder normalizes
|
|
187
|
+
away — and raw transcoding, replaying bytes rather than encoding a schema value,
|
|
188
|
+
uses it throughout so the output reproduces the input frame for frame.
|
|
142
189
|
|
|
143
190
|
```ts
|
|
144
|
-
const os =
|
|
191
|
+
const os = growingOStream();
|
|
145
192
|
os.writeSequenceBeginLazy(4); // the wrapper array
|
|
146
193
|
os.writeSequenceBeginLazy(0); // element 0 — has content
|
|
147
194
|
os.writeUnsigned(0, 7);
|
|
@@ -155,14 +202,15 @@ os.bytes(); // 26 06 00 07 07 0e 07 07
|
|
|
155
202
|
Decoding is unaffected by the distinction: an empty frame is valid input that the
|
|
156
203
|
message layer normalizes to the default, and an absent sequence field is
|
|
157
204
|
reconstructed from the schema default. Nesting is capped at `MAX_DEPTH` (255) on
|
|
158
|
-
both sides
|
|
159
|
-
|
|
160
|
-
buffer
|
|
205
|
+
both sides, and the encoder holds headers back to that full depth. A held-back
|
|
206
|
+
header is encoder state, never buffer content, so streaming through a small
|
|
207
|
+
buffer produces the same bytes.
|
|
161
208
|
|
|
162
209
|
### Deserialize
|
|
163
210
|
|
|
164
|
-
`decode()` walks a whole buffer and calls one optional `Visitor` method per field;
|
|
165
|
-
|
|
211
|
+
`decode()` walks a whole buffer and calls one optional `Visitor` method per field; a
|
|
212
|
+
field whose callback you did not implement is skipped. The visitor is **flat**: one
|
|
213
|
+
object receives the whole message, nested scopes included.
|
|
166
214
|
|
|
167
215
|
```ts
|
|
168
216
|
import { decode, type Visitor } from "@sofa-buffers/corelib";
|
|
@@ -178,152 +226,704 @@ class My implements Visitor {
|
|
|
178
226
|
decode(bytes, new My());
|
|
179
227
|
```
|
|
180
228
|
|
|
229
|
+
There are exactly two things to do with a field — **read** it or **skip** it
|
|
230
|
+
(§6.7.2) — and not implementing a callback is how you say the second.
|
|
231
|
+
|
|
232
|
+
`fieldBegin(id, wire)` is announced first for every field — right after the header
|
|
233
|
+
varint, before the value and before the value's *own* header word (a fixlen length
|
|
234
|
+
word, an array count word, a nested sequence's fields). It gives a reader the field
|
|
235
|
+
stream in wire order without writing the eight value callbacks. The sequence-*end*
|
|
236
|
+
marker gets none: it closes a scope rather than opening a field, and its id is
|
|
237
|
+
discarded (§4.9).
|
|
238
|
+
|
|
239
|
+
**Do not apply a schema bound from it.** An element id past the declared `count`
|
|
240
|
+
looks decidable from the id alone, and is not: that bound applies only to a field
|
|
241
|
+
whose *subtype* has confirmed it is the declared one, so it belongs on
|
|
242
|
+
`fixlenBegin`. A message ending inside the fixlen word is `INCOMPLETE` even when
|
|
243
|
+
the id would violate the bound. Throwing from `fieldBegin` is still how you
|
|
244
|
+
reject a field the header alone settles — an id you will not accept in any shape.
|
|
245
|
+
Everything schema-shaped stays on the later, more informative hook: a fixlen
|
|
246
|
+
subtype and a declared length on `fixlenBegin`, a declared element count on
|
|
247
|
+
`arrayBegin`.
|
|
248
|
+
|
|
249
|
+
An array's **elements** arrive through `arrayBulk(id, kind, count)`, and only
|
|
250
|
+
there: return the destination they should be written into — a `number[]`, an
|
|
251
|
+
exact-width typed array (`Uint16Array`, `Int32Array`, …), a `Long[]`, a pair of
|
|
252
|
+
`Uint32Array` halves, a `Float32Array` / `Float64Array`, or the raw `fp32` words —
|
|
253
|
+
together with the schema's element bound as `min`/`max` 32-bit halves, and the
|
|
254
|
+
decoder fills it directly. Return `null` (or declare no `arrayBulk`) and the
|
|
255
|
+
elements are walked over without being decoded at all. There is no callback per
|
|
256
|
+
element: one array is one call.
|
|
257
|
+
|
|
258
|
+
The **exact-width** destination (`typed`) is for a field whose declared width is a
|
|
259
|
+
typed array's own — a `u16` array into a `Uint16Array`, a `u64` array into a
|
|
260
|
+
`BigUint64Array`. It stores unboxed, it costs
|
|
261
|
+
two bytes an element rather than a tagged slot, and it lets the *encoder* know the
|
|
262
|
+
width too: `writeUnsignedArray` reads a `Uint16Array` element without the type and
|
|
263
|
+
range guards a `number[]` element needs, and reserves three bytes an element rather
|
|
264
|
+
than ten, which is what a caller-sized buffer has room for.
|
|
265
|
+
|
|
266
|
+
A **64-bit** destination is filled through the two 32-bit halves of each element,
|
|
267
|
+
written into a `Uint32Array` over the array's own buffer — `b[i] = 5n` would demand
|
|
268
|
+
a `bigint` per element, and this builds none. The same view reads them back when
|
|
269
|
+
such an array is encoded. A **`boolean`** array has a destination of its own,
|
|
270
|
+
`bool`, one byte per element: §4.4 gives a boolean no width bound, so a wire value
|
|
271
|
+
of 256 is `true` and must NOT mask to `0` — the decoder normalizes every non-zero
|
|
272
|
+
to `1`, which is also the only value an encoder may write back.
|
|
273
|
+
|
|
274
|
+
The element bound is still compared, and that is not a formality: a typed array
|
|
275
|
+
*masks* on store (`a[0] = 70000` in a `Uint16Array` is 4464), while an element
|
|
276
|
+
outside the declared width is a malformed message. So the width is matched against
|
|
277
|
+
the bound once, at the hand-off — a destination narrower than the bound is refused
|
|
278
|
+
with `Argument`, never silently truncated — and the fill then compares exactly as
|
|
279
|
+
`values` does.
|
|
280
|
+
|
|
281
|
+
```ts
|
|
282
|
+
const big: number[] = [];
|
|
283
|
+
const v: Visitor = {
|
|
284
|
+
arrayBulk: (id) =>
|
|
285
|
+
id === 6 ? { values: big, minLo: 0, minHi: 0, maxLo: 0xffff, maxHi: 0 } : null,
|
|
286
|
+
};
|
|
287
|
+
```
|
|
288
|
+
|
|
289
|
+
`sequenceBegin(id, depth)` opens a nested scope and `sequenceEnd(id, depth)` closes
|
|
290
|
+
it — the same visitor receives the scope's fields, with their own ids and
|
|
291
|
+
`depth + 1`. Route on the `(id, depth)` pair, which a schema fixes statically:
|
|
292
|
+
|
|
293
|
+
```ts
|
|
294
|
+
let inChild = false;
|
|
295
|
+
const v: Visitor = {
|
|
296
|
+
sequenceBegin: (id, depth) => { if (id === 3 && depth === 1) inChild = true; },
|
|
297
|
+
sequenceEnd: (id, depth) => { if (id === 3 && depth === 1) inChild = false; },
|
|
298
|
+
unsigned: (id, value) => { /* `inChild` says which scope this id belongs to */ },
|
|
299
|
+
};
|
|
300
|
+
```
|
|
301
|
+
|
|
302
|
+
Return **`false`** from `sequenceBegin` to decline the whole subtree: no callback of
|
|
303
|
+
any kind fires inside it, a scope opened within it is never offered either, and no
|
|
304
|
+
`sequenceEnd` arrives for what was declined.
|
|
305
|
+
|
|
306
|
+
A declined subtree is still parsed — a sequence is framed by markers rather than by
|
|
307
|
+
a length, so its end has to be found — but nothing is decoded into existence for
|
|
308
|
+
it, and no receiver cap fires inside it: the callback that would have compared one
|
|
309
|
+
is never called. Format ceilings (`ARRAY_MAX`, `FIXLEN_MAX`, `MAX_DEPTH`, the varint
|
|
310
|
+
bound) apply inside a declined subtree exactly as outside it.
|
|
311
|
+
|
|
181
312
|
### Deserialize stream
|
|
182
313
|
|
|
183
|
-
`IStream` resumes across chunk boundaries
|
|
184
|
-
you
|
|
185
|
-
|
|
186
|
-
|
|
314
|
+
`IStream` resumes across chunk boundaries: feed it whatever the transport hands
|
|
315
|
+
you and read the outcome from what `feed()` **returns** — `COMPLETE` or
|
|
316
|
+
`INCOMPLETE` for the bytes consumed so far, the third outcome being thrown rather
|
|
317
|
+
than returned (below). There is no end / finalize step and no second way to ask.
|
|
318
|
+
The visitor is bound at construction. String / blob payloads arrive in one or more pieces, each a range
|
|
319
|
+
`[start, end)` of the chunk **you** fed, tagged with the field's `total` length and
|
|
320
|
+
the piece's `offset` within it:
|
|
187
321
|
|
|
188
322
|
```ts
|
|
189
323
|
import { IStream, DecodeStatus, type Visitor } from "@sofa-buffers/corelib";
|
|
190
324
|
|
|
191
325
|
const visitor: Visitor = {
|
|
192
|
-
blob(id, total, offset,
|
|
193
|
-
/*
|
|
326
|
+
blob(id, total, offset, src, start, end) {
|
|
327
|
+
/* copy `src[start..end)` to `offset` of a `total`-byte destination of yours */
|
|
194
328
|
},
|
|
195
329
|
};
|
|
196
330
|
|
|
197
|
-
const is = new IStream();
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
331
|
+
const is = new IStream(visitor);
|
|
332
|
+
let status = DecodeStatus.Complete; // zero bytes end on a boundary
|
|
333
|
+
for await (const chunk of source) {
|
|
334
|
+
status = is.feed(chunk); // any async byte source
|
|
335
|
+
}
|
|
336
|
+
// feed() never throws for a merely incomplete decode and never promotes one to
|
|
337
|
+
// an error (MESSAGE_SPEC §7). The caller owns end-of-input.
|
|
338
|
+
if (status !== DecodeStatus.Complete) {
|
|
202
339
|
// stream ended inside a field (INCOMPLETE) — wait for more bytes, or treat
|
|
203
340
|
// the truncation as an error if this really was the end of input.
|
|
204
341
|
}
|
|
205
342
|
```
|
|
206
343
|
|
|
344
|
+
The chunk is borrowed **only for the duration of `feed`** (§6.0): once it returns
|
|
345
|
+
you may reuse, overwrite or free it, and what you decoded is unaffected — the
|
|
346
|
+
decoder retains nothing that points into it. Copy what you want to keep, during the
|
|
347
|
+
call; `PayloadAcc` and `decodeUtf8` are the ready-made way.
|
|
348
|
+
|
|
349
|
+
**`feed()` is the only way to ask.** There is no `status()` accessor and no end
|
|
350
|
+
step: what a `feed` returns, or what it throws, is the whole answer, so you are
|
|
351
|
+
never one call short of knowing where you stand and never holding two answers that
|
|
352
|
+
could disagree. (This library shipped that disagreement once — a `status()` that
|
|
353
|
+
answered `COMPLETE` for a message `feed` had already refused — which is why the
|
|
354
|
+
second way to ask is gone rather than repaired.) If you want the outcome again
|
|
355
|
+
without keeping it, feed an empty chunk: it consumes nothing and returns the same
|
|
356
|
+
value.
|
|
357
|
+
|
|
358
|
+
`INVALID` is **terminal**, and is the outcome `feed()` never returns: it travels on
|
|
359
|
+
the error channel, as a thrown `INVALID_MSG`. A stream that has thrown it is
|
|
360
|
+
poisoned for good — every further `feed` re-throws it without consuming a byte or
|
|
361
|
+
calling the visitor, so a refused stream can never hand back a status at all:
|
|
362
|
+
|
|
363
|
+
```ts
|
|
364
|
+
import { SofabError, SofabErrorCode } from "@sofa-buffers/corelib";
|
|
365
|
+
|
|
366
|
+
const is = new IStream(visitor);
|
|
367
|
+
try {
|
|
368
|
+
for await (const chunk of source) is.feed(chunk);
|
|
369
|
+
} catch (e) {
|
|
370
|
+
if ((e as SofabError).code !== SofabErrorCode.InvalidMsg) throw e;
|
|
371
|
+
// The verdict is in hand: `code` says INVALID_MSG, and nothing further needs
|
|
372
|
+
// asking. Feeding on would only raise the same error again.
|
|
373
|
+
}
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
A receiver-side cap (`LIMIT_EXCEEDED`, see [Receiver limits](#receiver-limits)) is
|
|
377
|
+
**not** the `INVALID` outcome and is never folded into one: the bytes are well-formed
|
|
378
|
+
and decode under a looser cap. It travels the same error channel under its own code,
|
|
379
|
+
and is terminal in the same way — the code you catch is what tells the two apart.
|
|
380
|
+
|
|
381
|
+
### 64-bit values without `bigint`
|
|
382
|
+
|
|
383
|
+
The default 64-bit surface is *number-first*: a value that fits exactly comes
|
|
384
|
+
back as a `number`, and only past `2^53-1` is a `bigint` materialised — so the
|
|
385
|
+
runtime type of a `u64` / `i64` depends on the value. `Long`, a value carried as two
|
|
386
|
+
unsigned 32-bit halves (`.low` / `.high`), is the fixed-type alternative on the
|
|
387
|
+
**encode** side. It is **representation-only**: the wire is identical to the
|
|
388
|
+
`number | bigint` path, byte for byte.
|
|
389
|
+
|
|
390
|
+
```ts
|
|
391
|
+
import { Long, growingOStream } from "@sofa-buffers/corelib";
|
|
392
|
+
|
|
393
|
+
const os = growingOStream();
|
|
394
|
+
os.writeUnsignedLong(1, Long.fromValue(2n ** 63n)); // scalar
|
|
395
|
+
os.writeSignedLong(2, Long.fromValue(-(2n ** 62n)));
|
|
396
|
+
os.writeUnsignedArrayLong(3, [1n, 2n].map(Long.fromValue)); // array
|
|
397
|
+
```
|
|
398
|
+
|
|
399
|
+
On the decode side there is no channel to switch on: **every** integer callback
|
|
400
|
+
carries the exact 64 bits as two unsigned 32-bit halves, beside the number-first
|
|
401
|
+
value, and an array hands its elements over as `Long`s or as raw halves. Read
|
|
402
|
+
whichever you want — the halves cost nothing to pass and nothing to ignore, and a
|
|
403
|
+
`Long` built from them never goes through `bigint` arithmetic:
|
|
404
|
+
|
|
405
|
+
```ts
|
|
406
|
+
import { ArrayKind, decode, Long, type Visitor } from "@sofa-buffers/corelib";
|
|
407
|
+
|
|
408
|
+
const longs: Long[] = [];
|
|
409
|
+
const v: Visitor = {
|
|
410
|
+
unsigned(id, value, lo, hi) { const x = Long.fromBits(lo, hi); },
|
|
411
|
+
signed(id, value, lo, hi) { /* lo/hi are the decoded two's-complement halves */ },
|
|
412
|
+
// One call per array, not per element: the decoder fills `longs` itself. A
|
|
413
|
+
// destination has to match the element kind, so decline the ones it does not:
|
|
414
|
+
// `null` costs nothing but the call.
|
|
415
|
+
arrayBulk: (id, kind) =>
|
|
416
|
+
kind === ArrayKind.Unsigned
|
|
417
|
+
? { longs, minLo: 0, minHi: 0, maxLo: 0xffffffff, maxHi: 0xffffffff }
|
|
418
|
+
: null,
|
|
419
|
+
};
|
|
420
|
+
decode(bytes, v);
|
|
421
|
+
```
|
|
422
|
+
|
|
423
|
+
Narrowing back is exact:
|
|
424
|
+
`value.low` for `u8`..`u32`, and `value.low | 0` for `i8`..`i32`.
|
|
425
|
+
|
|
207
426
|
### Code generator
|
|
208
427
|
|
|
209
|
-
`sofabgen` compiles a schema to one class per message with a `
|
|
210
|
-
`OStream` writes) and a `static decode`
|
|
211
|
-
one
|
|
428
|
+
`sofabgen` compiles a schema to one class per message with a `serialize` (chaining
|
|
429
|
+
`OStream` writes) and two decode entry points: a `static decode` for a message
|
|
430
|
+
already in one buffer, and a `static decoder()` bound to `IStream` for one arriving
|
|
431
|
+
in chunks — the same generated type driven whole-buffer or incrementally. Both
|
|
432
|
+
drive the same visitor, because there is only one decode surface (§5.3.1): what
|
|
433
|
+
changes is the drive, not the reader. A hand-written stand-in of both halves,
|
|
434
|
+
encoded and decoded each way:
|
|
212
435
|
|
|
213
436
|
```ts
|
|
214
|
-
import {
|
|
437
|
+
import {
|
|
438
|
+
OStream,
|
|
439
|
+
growingOStream,
|
|
440
|
+
IStream,
|
|
441
|
+
DecodeStatus,
|
|
442
|
+
decode,
|
|
443
|
+
type FeedStatus,
|
|
444
|
+
type FlushSink,
|
|
445
|
+
type Visitor,
|
|
446
|
+
} from "@sofa-buffers/corelib";
|
|
215
447
|
|
|
216
448
|
// generated by: sofabgen --lang typescript
|
|
217
449
|
class Point {
|
|
218
450
|
x = 0;
|
|
219
451
|
y = 0;
|
|
220
452
|
|
|
221
|
-
|
|
453
|
+
serialize(os: OStream): void {
|
|
222
454
|
os.writeSigned(1, this.x);
|
|
223
455
|
os.writeSigned(2, this.y);
|
|
224
456
|
}
|
|
225
457
|
|
|
226
458
|
static decode(bytes: Uint8Array): Point {
|
|
227
|
-
return
|
|
459
|
+
return _decodeIntoPoint(bytes, new Point());
|
|
228
460
|
}
|
|
229
461
|
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
switch (c.id) {
|
|
234
|
-
case 1: p.x = Number(c.readSigned()); break;
|
|
235
|
-
case 2: p.y = Number(c.readSigned()); break;
|
|
236
|
-
// case 3: p.child = Child.decodeFrom(c); break; // nested sequence
|
|
237
|
-
default: c.skip(c.wire); break; // forward-compatible
|
|
238
|
-
}
|
|
239
|
-
}
|
|
240
|
-
return p;
|
|
462
|
+
/** The streaming half: a reader bound to the corelib's resumable IStream. */
|
|
463
|
+
static decoder(): PointDecoder {
|
|
464
|
+
return new PointDecoder();
|
|
241
465
|
}
|
|
242
466
|
}
|
|
243
467
|
|
|
468
|
+
// The decode-into step sits beside the class, not on it: CORELIB_PLAN §6.1.1
|
|
469
|
+
// closes the generated object's surface to encode / decode / try_decode /
|
|
470
|
+
// serialize / deserialize / decoder, and `decode_from` / `decode_into` are two of
|
|
471
|
+
// the spellings it names as forbidden. It stays module-private — reachable from
|
|
472
|
+
// the sibling classes that decode into one another, and from nowhere else.
|
|
473
|
+
|
|
474
|
+
// Decodes into `o`, so a re-opened sequence continues the scope an earlier
|
|
475
|
+
// opening populated (MESSAGE_SPEC §7.4).
|
|
476
|
+
function _decodeIntoPoint(bytes: Uint8Array, o: Point): Point {
|
|
477
|
+
decode(bytes, new PointVisitor(o));
|
|
478
|
+
return o;
|
|
479
|
+
}
|
|
480
|
+
|
|
481
|
+
// generated alongside it: the visitor that fills a Point — the library's only
|
|
482
|
+
// decode surface (§5.3.1), one callback per wire type instead of one `case` per id.
|
|
483
|
+
// A visitor *is* the decode-into step: it writes into the object it was handed.
|
|
484
|
+
class PointVisitor implements Visitor {
|
|
485
|
+
private readonly out: Point;
|
|
486
|
+
constructor(out: Point) { this.out = out; }
|
|
487
|
+
|
|
488
|
+
signed(id: number, v: number | bigint): void {
|
|
489
|
+
if (id === 1) this.out.x = Number(v);
|
|
490
|
+
else if (id === 2) this.out.y = Number(v);
|
|
491
|
+
// no branch for an unknown id — or for a field whose wire type is not
|
|
492
|
+
// `signed` — so it is skipped and the decode stays COMPLETE (§7.3)
|
|
493
|
+
}
|
|
494
|
+
}
|
|
495
|
+
|
|
496
|
+
// ...and the handle Point.decoder() returns: an IStream plus its destination
|
|
497
|
+
class PointDecoder {
|
|
498
|
+
readonly message = new Point();
|
|
499
|
+
private readonly is = new IStream(new PointVisitor(this.message));
|
|
500
|
+
|
|
501
|
+
// The one place the answer is: what feed() returns, or what it throws. A
|
|
502
|
+
// status() accessor here would be a second way to learn the same fact, which
|
|
503
|
+
// is the second way it can be learned wrong — so the generated handle does not
|
|
504
|
+
// grow one either.
|
|
505
|
+
feed(chunk: Uint8Array): FeedStatus { return this.is.feed(chunk); }
|
|
506
|
+
}
|
|
507
|
+
|
|
244
508
|
const p = new Point(); p.x = 3; p.y = 4;
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
const
|
|
509
|
+
|
|
510
|
+
// one-shot: encode into memory, decode a whole buffer
|
|
511
|
+
const os = growingOStream(); p.serialize(os);
|
|
512
|
+
const wire = os.bytes().slice();
|
|
513
|
+
const got = Point.decode(wire); // got.x === 3, got.y === 4
|
|
514
|
+
|
|
515
|
+
// streaming out: the same serialize(), over a 4-byte buffer with a sink. The sink
|
|
516
|
+
// is handed the installed buffer and the region's bounds — never memory from
|
|
517
|
+
// anywhere else — so it copies out what it wants to keep.
|
|
518
|
+
const parts: Uint8Array[] = [];
|
|
519
|
+
const sink: FlushSink = (buf, start, end) => { parts.push(buf.slice(start, end)); };
|
|
520
|
+
const so = new OStream(new Uint8Array(4), 0, sink);
|
|
521
|
+
p.serialize(so); so.flush(); // the same bytes, in pieces
|
|
522
|
+
|
|
523
|
+
// streaming in: feed those pieces — or any other chunking — to the decoder
|
|
524
|
+
const dec = Point.decoder();
|
|
525
|
+
let st: FeedStatus = DecodeStatus.Complete; // zero bytes end on a boundary
|
|
526
|
+
for (const part of parts) st = dec.feed(part);
|
|
527
|
+
|
|
528
|
+
// COMPLETE says the bytes so far ended on a field boundary, not that the
|
|
529
|
+
// message is over — the caller's framing decides that, and a still-INCOMPLETE
|
|
530
|
+
// status once the input really has ended is truncation (§5.2.4).
|
|
531
|
+
const streamed = st === DecodeStatus.Complete ? dec.message : null;
|
|
248
532
|
```
|
|
249
533
|
|
|
534
|
+
A generated visitor takes the nested cases too: a nested message switches the
|
|
535
|
+
router into the child's fields on `sequenceBegin(id, depth)`, and a compact scalar
|
|
536
|
+
array is filled straight into the destination the router hands over at
|
|
537
|
+
`arrayBegin` / `arrayBulk`, so no part of the message is ever buffered whole. Nothing from a fed chunk is retained either — a
|
|
538
|
+
string is decoded and a blob copied on the way into the destination — so a chunk is
|
|
539
|
+
reusable the moment `feed` returns.
|
|
540
|
+
|
|
541
|
+
This example is compiled and executed by the test suite
|
|
542
|
+
(`test/helpers/readme-generator-example.ts`), so it cannot drift from the API.
|
|
543
|
+
|
|
250
544
|
## Memory handling
|
|
251
545
|
|
|
252
546
|
Who owns the bytes:
|
|
253
547
|
|
|
254
|
-
- **Encode (`OStream`).**
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
548
|
+
- **Encode (`OStream`).** Every buffer the encoder writes into is
|
|
549
|
+
**caller-supplied**: the library allocates none of its own and never grows or
|
|
550
|
+
reallocates one it was handed — `new OStream(buf, offset?, flush?)` writes into
|
|
551
|
+
`buf` and into nothing else. When it fills it calls the `flush` sink with **that
|
|
552
|
+
buffer** and the region's bounds — `(buffer, start, end)`, never memory from
|
|
553
|
+
anywhere else and never a view the encoder built, since pass-through is forbidden
|
|
554
|
+
(§5.1.6) — and continues; the region is valid for the duration of that call. With
|
|
555
|
+
no sink it throws `BUFFER_FULL`. `bytes()` returns a **view** of what is in the
|
|
556
|
+
buffer — with a sink, only the not-yet-flushed tail — so `.slice()` it if it
|
|
557
|
+
must outlive the next write.
|
|
558
|
+
- **The `offset` belongs to the installation, not to the buffer.** It reserves
|
|
559
|
+
room at the front of the unit the buffer-set begins — the constructor or
|
|
560
|
+
`setBuffer` — and handing that unit to the sink **consumes** it: a sink that
|
|
561
|
+
returns without installing a buffer has *copied*, so the encoder keeps writing
|
|
562
|
+
into the same buffer and resumes at `0`, with the whole buffer usable from
|
|
563
|
+
there. A sink that wants header room in **every** flushed unit — one framing
|
|
564
|
+
header per packet — re-arms it by calling `setBuffer(buf, offset)` from inside
|
|
565
|
+
the callback; passing the buffer it already has counts. A sink that *takes* the
|
|
566
|
+
buffer (hands it to a transport, queues it, gives it to DMA) must install a
|
|
567
|
+
replacement before returning. Either way the bytes are the same — only the unit
|
|
568
|
+
sizes differ. `reset()` and `bytes()` follow the current installation, so after
|
|
569
|
+
a flush they are relative to `0`; on a sink-less stream, which can never flush,
|
|
570
|
+
the reservation stands for the life of the encode.
|
|
571
|
+
- **Encode into memory (`growingOStream()`).** The allocating half is the
|
|
572
|
+
caller's role. `growingOStream(initialCapacity?)` is that caller ready-made: a
|
|
573
|
+
scratch buffer installed **with a sink** that accumulates the result (§5.1.2).
|
|
574
|
+
It never throws `BUFFER_FULL`, its `bytes()` is the **whole** message (a view —
|
|
575
|
+
`.slice()` it if it must outlive the next write or growth), and `reset()` keeps
|
|
576
|
+
the buffer it grew to, so a pooled encoder stops allocating. It is an ordinary
|
|
577
|
+
streaming stream otherwise, so `setBuffer` works and means what it always means:
|
|
578
|
+
the not-yet-flushed bytes are dropped and encoding continues into your buffer.
|
|
579
|
+
Pass an `initialCapacity` when you know roughly how large the message is: a
|
|
580
|
+
message built from many small fields grows by doubling, so 100 KB of them costs
|
|
581
|
+
nine enlargements from the 256-byte default. A single large field does not — a
|
|
582
|
+
bulk write tells the accumulator how much contiguous room it wants, so the
|
|
583
|
+
buffer reaches that size in one step and the write keeps its bulk route.
|
|
584
|
+
|
|
585
|
+
It reaches the encoder through `setBuffer(buffer, offset, carried)`, the third
|
|
586
|
+
argument being how many bytes of the message the replacement already holds
|
|
587
|
+
before `offset`. That is what keeps `bytes()` meaning "the message" across an
|
|
588
|
+
enlargement, and it is available to any caller that keeps a message in one
|
|
589
|
+
growing store.
|
|
590
|
+
|
|
591
|
+
Its storage is **carved from a shared slab** while it is small enough (up to
|
|
592
|
+
4 KiB of an 8 KiB slab). A carve is handed out once and never recycled, so no
|
|
593
|
+
two encoders ever share bytes and no message can read another's; what it
|
|
594
|
+
changes is *lifetime* — a retained `bytes()` view keeps its slab alive, so
|
|
595
|
+
`.slice()` (already the advice for a view that outlives the next write) is also
|
|
596
|
+
what releases it.
|
|
597
|
+
- **`MIN_OUTPUT_BUFFER` = `1`.** The smallest buffer this port accepts *for
|
|
598
|
+
streaming*, exported from the package so a caller can size from it. It is `1`
|
|
599
|
+
because the encoder splits every atomic unit — field header, fixlen word,
|
|
600
|
+
element count, a scalar or array element varint, an `fp32` / `fp64` element —
|
|
601
|
+
across a flush, so a message of any size encodes through a one-byte buffer and
|
|
602
|
+
the bytes produced are identical at every size. It binds a buffer installed
|
|
603
|
+
**with** a sink, at construction and at every mid-stream `setBuffer`:
|
|
604
|
+
`buf.length - offset` must be at least `MIN_OUTPUT_BUFFER`, and a smaller
|
|
605
|
+
window is rejected right there with `ARGUMENT` — never partway through a
|
|
606
|
+
message — leaving the encoder on the buffer it already had. A buffer installed
|
|
607
|
+
**without** a sink has no minimum: no flush can occur, so nothing can be split,
|
|
608
|
+
and a two-byte message encodes into a two-byte buffer.
|
|
609
|
+
- **Decode (`decode()` / `IStream`).** You own the bytes being parsed, and they
|
|
610
|
+
must stay valid only for the duration of the `feed` (or `decode`) call. After it
|
|
611
|
+
returns, reuse, overwrite or free them freely: **nothing the decoder produced
|
|
612
|
+
points into them**.
|
|
613
|
+
- **No views.** The decoder exposes no zero-copy view of a decoded value, no
|
|
614
|
+
payload-position getter and no borrowed value (§6.7) — on the one-shot path
|
|
615
|
+
exactly as on the streaming one, with no option that reinstates one. A `string` /
|
|
616
|
+
`blob` payload is reported in pieces as `(src, start, end)`, where `src` **is**
|
|
617
|
+
the chunk you fed: the decoder builds no view over it and keeps no storage of its
|
|
618
|
+
own, so whatever you want to keep, you copy out of memory you already own, during
|
|
619
|
+
the call. Scalars are delivered by value. If any of this README ever describes a
|
|
620
|
+
borrowed decoded value, either the README or the port is wrong.
|
|
621
|
+
- **No wire value decides an allocation in the codec.** After construction the
|
|
622
|
+
encoder and decoder allocate no payload storage (§6.6), and nothing at all except
|
|
623
|
+
the itemised handles below: no per-message, per-field or per-chunk allocation,
|
|
624
|
+
no growable state, and no
|
|
625
|
+
accumulator for a payload that straddles a chunk — a decoder's whole memory is
|
|
626
|
+
fixed-size state sized from this format's constants (a `MAX_DEPTH` scope stack, a
|
|
627
|
+
partial varint, an 8-byte float landing zone). Constructing an `OStream` /
|
|
628
|
+
`IStream` is the one allocating step, and `decode()` reuses one decoder across
|
|
629
|
+
calls so a one-shot caller does not pay it per message. A `bigint` for an integer
|
|
630
|
+
past `2^53` is not an exception: it is a *value*, not storage, and the `lo` / `hi`
|
|
631
|
+
halves beside it are there for a consumer that would rather not have one. A `Long`
|
|
632
|
+
written into a `longs` or `typed` bulk destination is the same kind of thing — a value, placed
|
|
633
|
+
in storage you supplied.
|
|
634
|
+
- **The language-forced handles, itemised** (§6.6.2). JavaScript will not let a codec
|
|
635
|
+
place or take an IEEE-754 value at a byte offset, or copy a *range* of bytes,
|
|
636
|
+
without building an object first: `TypedArray.set` — the only `memcpy` there is —
|
|
637
|
+
takes a typed array as its source, and a float needs a `DataView`. These are all
|
|
638
|
+
of them:
|
|
639
|
+
|
|
640
|
+
| handle | where | how many |
|
|
641
|
+
|---|---|---|
|
|
642
|
+
| `DataView` over the **output buffer** | `Kernel`, bulk `fp32` / `fp64` arrays | one per bulk call, and only from 64 `fp32` / 16 `fp64` elements up |
|
|
643
|
+
| `DataView` over the **fed chunk** | `IStream`, bulk float array reads | one per chunk, on the first run in it that clears the same thresholds |
|
|
644
|
+
| `subarray` of the caller's payload | `OStream.writeRaw`, as `set`'s source | one per copied piece, only when the payload does not fit the buffer |
|
|
645
|
+
| `Uint32Array` over a **64-bit destination** | `IStream`, an array handed a `BigUint64Array` / `BigInt64Array` | one per array |
|
|
646
|
+
| `Uint32Array` over a **64-bit source** | `Kernel`, `writeUnsignedArray` / `writeSignedArray` from one | one per bulk call |
|
|
647
|
+
| `Uint32Array` + `DataView` over an **`fp32` source** and the output | `Kernel`, `writeFp32Array` from a `Float32Array` | one pair per bulk call, at **every** length |
|
|
648
|
+
| `Uint32Array` over an **`fp32` source** | `OStream`, `writeFp32Array` from a `Float32Array` that does not fit the buffer | one per call, however many flushes it spans |
|
|
649
|
+
|
|
650
|
+
Each addresses storage **you** supplied, each is sized by that storage and never by
|
|
651
|
+
a number from the wire, and none of them leaves the codec. The 64-bit `Uint32Array`
|
|
652
|
+
rows are what a 64-bit element costs instead of a `bigint`: its halves are already
|
|
653
|
+
in hand, and a view over the caller's own array is how they are written without
|
|
654
|
+
building one.
|
|
655
|
+
|
|
656
|
+
A **scalar** float, a float array below the element threshold, and a float array fed
|
|
657
|
+
in chunks too small to hold a long run all build no handle at all: they go through a
|
|
658
|
+
shared 8-byte scratch word, which is fixed state. The one exception is the two `fp32`-source
|
|
659
|
+
rows — an `fp32` array whose source *is* a `Float32Array` already holds the wire words,
|
|
660
|
+
and copying them is what keeps a signaling NaN intact (§4.6/§6.5), so those handles
|
|
661
|
+
are built at any length rather than past a threshold. Streamed through a buffer too
|
|
662
|
+
small for the whole array, the words still go out as words — through the
|
|
663
|
+
`Uint32Array` alone, stored with shifts — so a small buffer gives the same bytes as
|
|
664
|
+
a large one (§5.1.4). `heap-free-codec.test.ts` asserts
|
|
665
|
+
every count in this table exactly, including the short runs that allocate nothing,
|
|
666
|
+
the element one under the threshold, and the two-element `Float32Array` that builds
|
|
667
|
+
the pair anyway. The thresholds and what they were derived from are on
|
|
668
|
+
`FP32_HANDLE_MIN` / `FP64_HANDLE_MIN` in the API documentation.
|
|
669
|
+
- **The bulk array hand-off borrows your destination until `arrayEnd`.**
|
|
670
|
+
`Visitor.arrayBulk` hands the decoder the array, `Long[]` or typed array it should
|
|
671
|
+
fill for one array field; the decoder writes into it ascending from index 0 and
|
|
672
|
+
holds it from the hand-off until that array ends, which on a chunked decode spans
|
|
673
|
+
several `feed` calls. It is dropped there, and on `decode()`'s pooled machine when
|
|
674
|
+
the call returns. The object must stay the same one for the whole array. A typed
|
|
675
|
+
destination must already hold `count` elements; a plain `number[]` / `Long[]`
|
|
676
|
+
grows as it fills and is **cut to the elements written** when the array ends —
|
|
677
|
+
including to zero for an array that is empty on the wire — so reusing one across
|
|
678
|
+
arrays or messages never leaves the previous array's tail behind and its `length`
|
|
679
|
+
after `arrayEnd` is exactly that array's element count. An element the target's bound rejects stops the fill: everything before
|
|
680
|
+
it is written, it and everything after it are not. This is the only reference the
|
|
681
|
+
decoder keeps into your storage between calls.
|
|
682
|
+
- **The static helper layer allocates, on your behalf.** `PayloadAcc`,
|
|
683
|
+
`ElementSeq`, `FramedSeq`, `StringSeq`, `BlobSeq`, `decodeUtf8`, `elementsEqual`,
|
|
684
|
+
`longElementsEqual` and `fp32RawBytes` are the generated layer's
|
|
685
|
+
code shipped here for reuse (ARCHITECTURE §8), not part of the codec: the codec
|
|
686
|
+
never calls them, and they allocate the values they build.
|
|
687
|
+
- **String validity is checked where a string is materialized** (§6.4.5).
|
|
688
|
+
JavaScript strings are a Unicode type, so this port is always strict — but a
|
|
689
|
+
`string` payload piece is *raw wire bytes* and is not validated (it may end
|
|
690
|
+
mid-code-point), so whoever materializes one owns the check.
|
|
691
|
+
`decodeUtf8(bytes, start?, end?)` is that check, exported for exactly this: it
|
|
692
|
+
rejects malformed bytes as `INVALID_MSG` rather than as a platform `TypeError`.
|
|
693
|
+
Rolling your own instead
|
|
694
|
+
means `new TextDecoder("utf-8", { fatal: true })` — the default `TextDecoder`
|
|
695
|
+
silently substitutes `U+FFFD`, which the format forbids in either direction,
|
|
696
|
+
and `TextEncoder` does the same to an unpaired surrogate where this encoder
|
|
697
|
+
refuses it with `ARGUMENT`.
|
|
698
|
+
- **Reassembly is the caller's, with a helper.** The codec holds no payload across
|
|
699
|
+
`feed` calls. `PayloadAcc.take(total, offset, src, start, end)` joins the pieces —
|
|
700
|
+
one accumulator per decoder, since only one payload is ever in flight — and
|
|
701
|
+
returns storage of its own that aliases nothing, on the whole-payload path exactly
|
|
702
|
+
as on the split one. `StringSeq` / `BlobSeq` collect the elements of a `string` /
|
|
703
|
+
`blob` wrapper array; `ElementSeq` holds the index rules for any element kind
|
|
704
|
+
(index bound, gap fill, last-write-wins) and `FramedSeq` is its twin for an element
|
|
705
|
+
whose default is a fresh object — a `struct`, a `union`, a nested row — where one
|
|
706
|
+
shared default would alias every gap of the array onto a single instance;
|
|
707
|
+
`elementsEqual` (and `longElementsEqual`, for `Long`-backed 64-bit arrays, whose
|
|
708
|
+
elements are object identities) is the array form of the omit-if-default test an
|
|
709
|
+
encoder applies before writing a field; `fp32RawBytes` turns the
|
|
710
|
+
32-bit word `Visitor.fp32` hands over back into the four wire bytes a generated
|
|
711
|
+
message keeps beside an `fp32` it cannot re-encode from a `number` (§6.5).
|
|
712
|
+
|
|
713
|
+
### Receiver limits
|
|
714
|
+
|
|
715
|
+
**This corelib holds none, by design.** A field the schema leaves unbounded is still
|
|
716
|
+
bounded by the receiver — CORELIB_PLAN §6.2.1 admits no unset state and no unlimited
|
|
717
|
+
mode — but the *numbers* belong to generated code, which knows the schema and the
|
|
718
|
+
deployment, and §6.2.1 is explicit that a codec
|
|
719
|
+
|
|
720
|
+
> **MUST NOT** hold a limit of its own, **MUST NOT** supply a default for one it was
|
|
721
|
+
> not given, **MUST NOT** read an omitted argument as *unlimited*, and **MUST NOT**
|
|
722
|
+
> clamp to one
|
|
723
|
+
|
|
724
|
+
and that a format ceiling reached because no cap was stated
|
|
725
|
+
|
|
726
|
+
> is the **format's** bound, not a receiver cap, and a port **MUST NOT** present it
|
|
727
|
+
> as one.
|
|
728
|
+
|
|
729
|
+
So `decode(bytes, visitor)` and `new IStream(visitor)` take no limits argument. Up to
|
|
730
|
+
v0.10.0 they took a `DecodeLimits` whose absent members fell back to `ARRAY_MAX` /
|
|
731
|
+
`FIXLEN_MAX`; that object is gone, along with the `LIMIT_EXCEEDED` rejections it
|
|
732
|
+
raised against ceilings nobody had configured.
|
|
733
|
+
|
|
734
|
+
| the cap | who states it | who compares it |
|
|
735
|
+
|---|---|---|
|
|
736
|
+
| `max_dyn_array_count` on an array field | generated code | its own `arrayBegin` |
|
|
737
|
+
| `max_dyn_string_len` / `max_dyn_blob_len` on a `string` / `blob` field | generated code | its own `fixlenBegin` |
|
|
738
|
+
| the element **index** of a wrapper array | generated code | `StringSeq` / `BlobSeq` / `ElementSeq` / `FramedSeq`, from `receiverCap` |
|
|
739
|
+
| the element **byte length** of a wrapper array | generated code | `StringSeq` / `BlobSeq`, from `receiverElemMax` |
|
|
740
|
+
|
|
741
|
+
§6.2.1 permits the comparison to run inside the corelib — "A corelib **MAY** take a
|
|
742
|
+
limit as an argument and perform the check itself" — and the collectors do exactly
|
|
743
|
+
that, for the one shape a visitor cannot see: a wrapper array's element length words
|
|
744
|
+
go to the collector, never to the generated visitor. Every one of their bounds is a
|
|
745
|
+
**required** constructor argument with no default, the schema halves included
|
|
746
|
+
(`UNBOUNDED`, `-1`, is the explicit "the schema declared none"):
|
|
274
747
|
|
|
275
748
|
```ts
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
new
|
|
279
|
-
new
|
|
749
|
+
// out, acc, count, maxlen, name, receiverCap, receiverElemMax
|
|
750
|
+
new StringSeq( out, new PayloadAcc(), UNBOUNDED, UNBOUNDED, "tags", 65_536, 1 << 20);
|
|
751
|
+
new BlobSeq( out, new PayloadAcc(), 8, 4096, "parts", 65_536, 1 << 20);
|
|
752
|
+
new ElementSeq( out, defaultElem, UNBOUNDED, "rows", 65_536);
|
|
753
|
+
// out, make, count, name, receiverCap
|
|
754
|
+
new FramedSeq( out, () => new Elem(), 8, "codes", 65_536);
|
|
280
755
|
```
|
|
281
756
|
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
757
|
+
Each pair is **exclusive**, never additive (§6.2.1: a cap "**MUST NOT** be applied to
|
|
758
|
+
a field the schema already bounds"): where the schema declared `count` / `maxlen`
|
|
759
|
+
that bound governs and its violation is `INVALID_MSG`, a statement about *validity*;
|
|
760
|
+
where it declared none the receiver cap governs and its violation is
|
|
761
|
+
`LIMIT_EXCEEDED`, a *policy* rejection on well-formed bytes.
|
|
762
|
+
|
|
763
|
+
A receiver bound that is about to govern and **states no cap** — negative, `NaN`
|
|
764
|
+
(what an omitted argument becomes in JavaScript), `Infinity` — is refused at
|
|
765
|
+
construction with `SofabErrorCode.Argument`. It is neither of the two categories
|
|
766
|
+
above: no receiver policy was set, so there is no `LIMIT_EXCEEDED` to raise
|
|
767
|
+
("a format ceiling reached because no cap was stated is the **format's** bound …
|
|
768
|
+
and a port **MUST NOT** present it as one"), and no unlimited mode to fall back to
|
|
769
|
+
("**MUST NOT** read an omitted argument as *unlimited*"). It is a mistake in the
|
|
770
|
+
call, which §6.3 makes `InvalidArgument`. The refusal is fail-closed: nothing is
|
|
771
|
+
decoded through a collector that could not be built. A bound the schema half makes
|
|
772
|
+
inert is not checked — §6.2.1 forbids applying it at all.
|
|
773
|
+
|
|
774
|
+
**What this decoder still owes the layer that holds the numbers** is the enforcement
|
|
775
|
+
point §6.2.1 requires — the count / length header, before the allocation the cap
|
|
776
|
+
exists to prevent, and behind the MESSAGE_SPEC §7.3 tag test:
|
|
777
|
+
|
|
778
|
+
* `arrayBegin(id, kind, count)` is raised at the **count word**, before any element
|
|
779
|
+
is delivered (for a fixlen array, at its element-length word — the element kind is
|
|
780
|
+
unknown until then, and still before any element);
|
|
781
|
+
* `fixlenBegin(id, subtype, total)` is raised at the **length word**, before any
|
|
782
|
+
payload piece;
|
|
783
|
+
* both carry the number a destination gets sized from, so a rejection there costs no
|
|
784
|
+
allocation at all. Reject, never clamp: materialising `limit` elements where the
|
|
785
|
+
wire said more is data corruption wearing a safety jacket.
|
|
786
|
+
|
|
787
|
+
A cap therefore applies **only to a field you read**, and that falls out of the
|
|
788
|
+
structure rather than needing a rule: a field the visitor steps over never reaches
|
|
789
|
+
the callback that holds the number, so a decode that walks past an over-cap field it
|
|
790
|
+
was never going to read stays `COMPLETE` (§6.2.1's "a skipped field is never
|
|
791
|
+
capped"). The **format ceilings** are not yours to waive this way: a count above
|
|
792
|
+
`ARRAY_MAX` or a length above `FIXLEN_MAX` stays `INVALID` whether anyone reads the
|
|
793
|
+
field, because it bounds what the *wire* may express.
|
|
794
|
+
|
|
795
|
+
A cap rejection is raised by throwing `SofabError` with code
|
|
796
|
+
`SofabErrorCode.LimitExceeded`, which propagates out of `feed` / `decode`. It is
|
|
797
|
+
**not** the `INVALID` outcome and is never folded into one — the same bytes decode
|
|
798
|
+
under a looser cap. The error channel is the only place it appears, and that is not
|
|
799
|
+
a gap: the three-valued outcome has no value for "valid, but more than I am
|
|
800
|
+
configured to accept", so there is nothing about it a returned status could have
|
|
801
|
+
said. Catch it and read its `code`.
|
|
294
802
|
|
|
295
803
|
## Build & test
|
|
296
804
|
|
|
297
805
|
```bash
|
|
298
806
|
npm ci
|
|
299
807
|
npm run typecheck # tsc --noEmit (strict)
|
|
300
|
-
npm test # vitest run: vectors, chunked feeding,
|
|
808
|
+
npm test # vitest run: vectors, chunked feeding, memory rules, round-trips
|
|
301
809
|
npm run coverage # vitest run --coverage (v8)
|
|
302
810
|
npm run build # tsup -> ESM + CJS + IIFE + .d.ts in dist/
|
|
303
811
|
npm run smoke # cross-runtime smoke test of the built bundle
|
|
304
812
|
```
|
|
305
813
|
|
|
306
|
-
Tests live in `test/` as focused suites, including `vectors.test.ts` (encode
|
|
307
|
-
decode every shared conformance vector), `istream.chunked.test.ts` (every vector
|
|
308
|
-
fed one byte at a time), `
|
|
309
|
-
`
|
|
310
|
-
|
|
311
|
-
|
|
814
|
+
Tests live in `test/` as focused vitest suites, including `vectors.test.ts` (encode
|
|
815
|
+
+ decode every shared conformance vector), `istream.chunked.test.ts` (every vector
|
|
816
|
+
fed one byte at a time), `skip-ids.test.ts` (every vector that carries
|
|
817
|
+
`skip_ids`, decoded by a receiver that ignores those ids at every nesting level —
|
|
818
|
+
contiguous, one byte at a time, and split in two at every byte boundary),
|
|
819
|
+
`heap-free-codec.test.ts` (no allocation primitive on a codec path, a flat heap
|
|
820
|
+
over encode and decode, no view into a fed or one-shot buffer) and `pooled-decoder-state.test.ts` (a decode aborted at every cut point
|
|
821
|
+
leaves nothing behind for the next one).
|
|
822
|
+
|
|
823
|
+
The vector-driven suites each print one summary line — `[vectors] 131
|
|
824
|
+
vectors, none gated out by requires, 524 checks` — so a run says how much of the
|
|
825
|
+
shared suite it actually executed, and a file that arrived truncated or a group
|
|
826
|
+
gated out by `requires` shows up as a smaller number rather than as silence. This
|
|
827
|
+
port compiles no feature out, so nothing is ever gated.
|
|
828
|
+
|
|
829
|
+
`assets/test_vectors.json` carries six blocks and this port runs all six:
|
|
830
|
+
`vectors`, `invalid_utf8`, `sequence_growth`, `header_limits`,
|
|
831
|
+
`header_limits_nested` and `boolean_tolerant`. The file is a
|
|
832
|
+
**verbatim** copy of the one in `corelib-c-cpp`, which authors it. A daily CI job (`.github/workflows/shared-vectors.yml`) compares this copy's sha256 against that file on `corelib-c-cpp@main`, so a copy left behind by an upstream change is reported rather than going unnoticed.
|
|
833
|
+
|
|
834
|
+
`sequence_growth` holds the wrapper-array growth cases of §7.2 item 8, replayed by
|
|
835
|
+
`sequence-growth.test.ts` for both element kinds at three chunkings. This port
|
|
836
|
+
declares `dynamic_arrays`: its wrapper-array containers are JS arrays that grow at
|
|
837
|
+
decode time, so the block applies. The cases are cap-relative and the run installs
|
|
838
|
+
`max_dyn_array_count = 8`. Growth **geometry** splits in two: the backing store's
|
|
839
|
+
reallocation strategy is the engine's amortised doubling, which is not this port's to
|
|
840
|
+
pin, while the fill is, and is asserted as one write per slot in a single pass.
|
|
841
|
+
|
|
842
|
+
`header_limits` holds the truncated over-ceiling headers of §6.2.1 / §6.3 — bytes
|
|
843
|
+
that declare a length or an element count and then end, with no payload behind them —
|
|
844
|
+
replayed by `header-limits.test.ts`. The ceiling answers **at that word**, before the
|
|
845
|
+
payload is asked for, so the verdict is terminal and never `INCOMPLETE`; which
|
|
846
|
+
ceiling the case configures decides the category, `INVALID` for a schema `maxlen` and
|
|
847
|
+
`LIMIT_EXCEEDED` for a §6.2.1 receiver cap. This port declares `receiver_caps`: its
|
|
848
|
+
generated layer carries receiver caps distinct from schema bounds, compared inside the
|
|
849
|
+
visitor's own `fixlenBegin` / `arrayBegin`, which this library raises at the header
|
|
850
|
+
word. Every rejection case is paired with an in-cap control that must still answer
|
|
851
|
+
`incomplete`, and the block is also run with the ceilings lifted — where all six
|
|
852
|
+
rejections fall back to `incomplete`, since this port's `FIXLEN_MAX` (`INT32_MAX`)
|
|
853
|
+
sits above even the amplification case's 1 GiB claim.
|
|
854
|
+
|
|
855
|
+
`header_limits_nested` is that same assertion one or two sequence frames deeper,
|
|
856
|
+
replayed by `header-limits-nested.test.ts` — the axis the flat block leaves
|
|
857
|
+
untested, since every case there sits at the top level. Its cases open a sequence,
|
|
858
|
+
declare the over-ceiling word inside it and end with the frame **still open**, so a
|
|
859
|
+
decoder has a second, independent reason to answer `incomplete` and a port that
|
|
860
|
+
binds its ceiling to the top-level scope looks plausible while capping nothing. The
|
|
861
|
+
runner descends the `frames` chain outermost first, applies the **same leaf** the
|
|
862
|
+
flat block uses (`test/helpers/header-limits.ts`) at the innermost depth, and runs a
|
|
863
|
+
negative control over every rejection with that case's own kind of ceiling lifted to
|
|
864
|
+
65536: all four then answer `incomplete` instead, which is what shows the verdict was
|
|
865
|
+
the ceiling's rather than an unclosed frame's. All 8 cases run, none gated.
|
|
866
|
+
|
|
867
|
+
`boolean_tolerant` holds §4.4's decode half — bytes carrying `2`, `256` or `2^64-1`
|
|
868
|
+
at a boolean position — replayed by `boolean-tolerant.test.ts`. No conforming
|
|
869
|
+
*encoder* emits such a value, so the positive vectors cannot reach the rule at all:
|
|
870
|
+
those bytes only ever arrive from someone else's encoder. A boolean carries no width
|
|
871
|
+
bound, so every non-zero value is `true` — not `INVALID`, and not a truncation to
|
|
872
|
+
`false` — and the decode is **normalized**, which only the re-encode makes visible.
|
|
873
|
+
Each case is therefore asserted three ways: the outcome is `complete`, the
|
|
874
|
+
destination holds exactly `0` / `1` (poisoned with `0xaa` first, so a decoder that
|
|
875
|
+
never writes cannot pass), and the re-encode of *what was decoded* is byte-compared
|
|
876
|
+
against the block's `reencoded_hex` — `0001`, never `0002`. An array decodes through
|
|
877
|
+
the `bool` destination, which is where this library performs the normalization; a
|
|
878
|
+
scalar boolean has no callback of its own and arrives on `unsigned` with its full 64
|
|
879
|
+
bits, so the `!== 0` test there is the generated layer's and the test performs it,
|
|
880
|
+
after asserting that the delivered value and its two halves agree. In this block an
|
|
881
|
+
unsatisfied `requires` tag means the message must be **rejected**, not skipped
|
|
882
|
+
(§4.4 lifts the width bound the *type* carries, never the one a narrowed *build*
|
|
883
|
+
has) — a path this port never takes, since it compiles no feature out.
|
|
884
|
+
|
|
885
|
+
CI type-checks, tests and builds on Node 20 / 22 / 24 / 26, smoke-tests the
|
|
886
|
+
bundle on Node, Deno and Bun, and publishes coverage badges; a separate
|
|
887
|
+
`docs.yml` deploys the TypeDoc API reference to GitHub Pages.
|
|
312
888
|
|
|
313
889
|
## Benchmarks
|
|
314
890
|
|
|
315
|
-
|
|
316
|
-
|
|
891
|
+
Three standalone tools, specified by `BENCH_SPEC.md` and mirrored in every other
|
|
892
|
+
port — same datasets, same timing rules, same output grammar — so the numbers
|
|
893
|
+
compare directly across languages:
|
|
317
894
|
|
|
318
895
|
```bash
|
|
319
|
-
npm run perf # per-op cost
|
|
320
|
-
npm run bench # throughput table (MB/s)
|
|
896
|
+
npm run perf # per-op cost on the 170-byte perf message
|
|
897
|
+
npm run bench # throughput table (MB/s) over the four shared datasets
|
|
321
898
|
npm run bench:callgrind # machine-independent instructions/op under Valgrind
|
|
322
899
|
```
|
|
323
900
|
|
|
324
|
-
`
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
901
|
+
`bench` prints ten rows over four datasets: a 1000-element `u64` array, the small
|
|
902
|
+
`typical` message, an **unbounded 1 MB blob**, and a `composite` message that
|
|
903
|
+
reaches what the flat ones miss — a 64-element wrapper array, 320 bytes of 1-,
|
|
904
|
+
2-, 3- and 4-byte UTF-8, nesting three deep, a default-valued field the encoder
|
|
905
|
+
must *not* write, and a two-byte field header. Three of the encoded sizes are
|
|
906
|
+
cross-port parity checks (`perf` = 170 bytes, `blob 1MB` = 1,000,005,
|
|
907
|
+
`composite` = 956); `test/bench-datasets.test.ts` holds the datasets to them, and
|
|
908
|
+
`test/bench-grammar.test.ts` holds the tools to the output grammar.
|
|
909
|
+
|
|
910
|
+
Every encode row writes into a **caller-supplied buffer** rather than the
|
|
911
|
+
accumulator. The `blob 1MB` rows are the ones that exercise streaming end to end:
|
|
912
|
+
`one-shot` is a single contiguous write into a 1,000,005-byte buffer, `streaming`
|
|
913
|
+
is the same bytes through a **4096-byte** buffer with a flush sink (~245
|
|
914
|
+
flushes), and `decode: blob 1MB` is fed back in 4096-byte chunks. The
|
|
915
|
+
**difference** between the two encode rows is what the divisible-run flush path
|
|
916
|
+
costs, and it is legible only under `Ir/op`. BENCH_SPEC's optional
|
|
917
|
+
`blob 1MB passthrough` row is absent: pass-through is forbidden (§5.1.6), so every
|
|
918
|
+
`string` / `blob` run is copied through the output buffer. Both copies are
|
|
919
|
+
`TypedArray.set` — the whole payload when it fits, a range per flush when it does
|
|
920
|
+
not, the latter through one of the itemised §6.6.2 handles under
|
|
921
|
+
[Memory handling](#memory-handling).
|
|
922
|
+
|
|
923
|
+
Since JS engines expose no portable cycle counter, `perf` uses CPU time/op as the
|
|
924
|
+
code-cost proxy; `bench:callgrind` counts instructions/op under Valgrind (two rep
|
|
925
|
+
counts per workload, subtracted, on a `--predictable` V8) for a fully
|
|
926
|
+
machine-independent figure. The same tools under Node (V8) and Bun
|
|
927
|
+
(JavaScriptCore) give directly comparable numbers. `tsx bench/bench.ts --smoke`
|
|
928
|
+
runs every row exactly once — a liveness check for the rows, never a
|
|
929
|
+
measurement.
|