@zakkster/lite-bake-stream 1.5.0 → 1.6.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -2,6 +2,34 @@
2
2
 
3
3
  All notable changes to this project will be documented in this file. This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
4
4
 
5
+ ## [1.6.1] -- 2026-09-02
6
+
7
+ M7 -- docs, comment prose, and two new guards. No src logic, no API surface change (exports map, error codes, and types signatures all unchanged; inventory 60 codes / 0 unpinned).
8
+
9
+ ### Changed
10
+ - ASCII law (BS-23): 340 non-ASCII characters across 25 tracked files removed. Prose mapped to ASCII; fixture DATA literals became `\u` escapes (runtime-identical -- C1/C2/C3 and m6-golden MATCH proves it).
11
+ - README rebuilt on the CLAUDE.md documentation spine. Corrected: test count 515 -> 519; torture scenarios 37 -> 44 fast / 47 full; the deprecated bench/torture.js entry point and the milestone/roadmap tables are gone; the sink contract no longer names an unexported class.
12
+ - BS-29: the lite-bake relationship is stated as exactly what t8 proves (the F64 lane-width agreement). SPEC 4.2 corrected -- LBK1 kinds 2/3/4 are F32/U32/U8 and do NOT match lite-bake Types.I32/I16/I8. package.json description de-clawed.
13
+ - BS-24: FileIngest.js header example now calls the real 2-arg ingestStream(readableStream, opts); llms.txt file-ingest and error stanzas de-duplicated (the onProgress reuse contract preserved).
14
+ - SPEC section 8 gate text corrected: the torture gate has lived at `node --expose-gc test/torture.mjs` since M0 (not bench/torture.js), and the verify budget is a measured ~103,000 rows across 20 fast-tier scenarios (the "~73,500 across 12" claim was stale; sum re-derived independently during qa: 103,004 / 20).
15
+
16
+ ### Added
17
+ - ASCII-law guard (test/torture/ascii-law.mjs) and API-drift guard (test/torture/api-surface.mjs), each with an armed control (BAKE_TORTURE_BREAK=u and =x) proven to exit non-zero.
18
+ - decisions/0010-preserve-mode.md, 0011-sample-drain-reintern.md, 0012-clinger-fast-path.md (transcriptions; BS-25 closed, with the truth-fix: decisions/ has existed since M1).
19
+
20
+ ### Removed
21
+ - bench/torture.js (the M0 forwarding shim; nothing referenced it except one historical-provenance comment, which stays).
22
+
23
+ Suite: 515 -> 519 tests / 519 pass / 0 fail / 0 todo. Torture 44 fast, 47 full. Pack 33 files. BREAK matrix {1, t6, m, u, x} all exit non-zero.
24
+
25
+ ## [1.6.0] -- 2026-09-02
26
+
27
+ M6 -- streaming emission + optional integrity. Public `beginStream(sink, { layout: 'stream', crc? })` binds a sink BEFORE feeding, so each shard payload streams to the sink as it finalizes and is dropped; `finalizeToSink(sink, { layout })` then writes the schema/directory/zone-map trailer + footer with one `writeAt(header, 0)` backpatch and returns `{ totalRows, shardCount, schema | mode, bytesWritten, layout }`. Called without a prior `beginStream`, `finalizeToSink` is the buffered convenience mode (`O(container)` peak, stated in the memory-model docs). `layout: 'prefix'` is byte-identical to `finalize()`. With the two-step contract, peak memory drops from `O(container)` to `O(targetShardBytes + directory)`. A default-emitted stream container is a legal v1 container (format_version stays 1) accepted by every shipped reader, `checkContainer`, and `mergeContainers`. `finalize()` is re-based over the same emitters, so its output is byte-for-byte identical to 1.5.0 (frozen sha256 goldens hold).
28
+
29
+ Optional CRC-32C (Castagnoli, table-driven, zero deps, `src/Crc32c.js`): writers opt in with `{ crc: true }`; readers get `verifyCrc()` -> `'ok' | 'absent'` (mismatch throws `R_BAD_CRC`) and an open option `{ verifyCrc: true }` that fails closed on mismatch (`R_BAD_CRC`) and absence (`R_CRC_ABSENT`). `mergeContainers` recomputes the CRC iff every input carried one, else absent. KATs: `CRC32C("")=0x00000000`, `CRC32C("123456789")=0xE3069283`; the stream layout closes the ring with a split-buffer combine (`crc32cCombine`, KAT-pinned and covered end-to-end by a streaming `beginStream` + flipped-byte case). New sink guard `W_BAD_SINK`. Three new thrown codes (`W_BAD_SINK`, `R_BAD_CRC`, `R_CRC_ABSENT`); inventory 57 -> 60 / 0 unpinned.
30
+
31
+ Measured peak RSS (full-tier gate, 500 MB streamed in 8 MiB chunks into a counting sink, container never materialized): `peakRss - baselineRss` = 32.1 MiB against the bound `4*targetShardBytes + 64*shardCount + 64 MiB` = 96.0 MiB at 41 shards -- versus the buffered `finalize()` path at ~704 MiB (baseline `soak --mb=500`: baselineRss 667 MiB, peakRss 1371 MiB). Suite: 479 -> 515 tests / 515 pass / 0 fail / 0 todo. Torture: 44/44 fast, 47/47 full, arrayBuffers growth 0.00 MB both tiers, t7 sink-release witness (retained shard slots <= 1; independently armed during qa -- retained shards fail 364/364, clean run 0/0). Measurement note: t7's final arrayBuffers read now settles and collects twice before reading (the gc-profiler contract) -- t5-fuzz's large-ArrayBuffer churn was polluting the single-gc read via V8's buffer pool; the 8 MB budget and all RULES numbers are unchanged, and the M6 paths retain ~0 in isolation. Pack 32 -> 33 (new `src/Crc32c.js`). See `decisions/0009-streaming-emission.md`.
32
+
5
33
  ## [1.5.0] -- 2026-09-02
6
34
 
7
35
  M5 -- every ceiling gets a door; the shard budget counts all the bytes it ships; the floors are measured. BS-26 closed; BS-27's floor items closed (the writer-holds-all-shards ~2x container peak moves to M6's streaming emission); BS-32/BS-33/BS-34 closed. Suite: 461 -> 479 tests / 479 pass / 0 fail / 0 todo. Torture: 44/44 fast (~322 ms tier sum), 47/47 full (~5.6 s) incl. a 2100-shard directory-scale scenario (SPEC checker green, RangeReader LRU cap held at 8, `_maybeEvict` allocation-flat over 20000 ops); arrayBuffers growth 0.00 MB both tiers; t7 soak tracker size 0. Inventory gate: 54 -> 57 thrown codes / 0 unpinned. Falsifiability, measured item-by-item with baseline src stashed: all 15 new door/behavior assertions fail on baseline and pass on this tree, every stay-green net holds on both, and an all-F64-lane container is byte-identical to the 1.4.1 output (same sha256 and length). BREAK matrix {1, t6, l, m, j} all exit non-zero.
@@ -191,14 +219,14 @@ M1 -- the write path refuses what it cannot store. Eight findings closed by a se
191
219
 
192
220
  - **BS-01: absent string fields now decode as `""`.** A record that omits a U32-lane field leaves index 0 in that row slot. Index 0 of a shard's string table used to be whichever string interned first in that shard, so `{"s":"zebra"}` followed by `{}` read row 1 back as `"zebra"` -- silently, and differently depending on record order and where shard boundaries fell. `StringTable` now reserves index 0 as the empty string in its constructor and at every `reset()`, so absent U32 cells decode as `""` on every write path: explicit schema, sample-and-infer, and every shard of a multi-shard drain. Containers written before this change keep their old bytes and their old, data-dependent behavior on those bytes; the fix is not retroactive. Readers are unchanged. SPEC 3.3 and the SPEC 7 coercion table are amended to match.
193
221
 
194
- ## [1.0.0] 2026-07-12
222
+ ## [1.0.0] -- 2026-07-12
195
223
 
196
224
  First stable release. The LBK1 container format is frozen at `format_version: 1`.
197
225
 
198
226
  ### Two ingest modes, one API
199
227
 
200
228
  - **Schema mode** (default): F64/U32 lane packing, zone-maps query pruning, byte-exact preservation for flat records. This is the mode the 8 GB soak below was run under.
201
- - **Preserve mode** (`serialize(input, { preserve: true })`): opaque byte blobs, any JSON shape. Deeply nested API responses, arrays, mixed types all round-trip byte-for-byte identical. Reader exposes a tri-API: `getBytes(i)` (zero-alloc view), `getString(i)` (one `TextDecoder` allocation), `getJSON(i)` (opt-in `JSON.parse`).
229
+ - **Preserve mode** (`serialize(input, { preserve: true })`): opaque byte blobs, any JSON shape. Deeply nested API responses, arrays, mixed types -- all round-trip byte-for-byte identical. Reader exposes a tri-API: `getBytes(i)` (zero-alloc view), `getString(i)` (one `TextDecoder` allocation), `getJSON(i)` (opt-in `JSON.parse`).
202
230
 
203
231
  `deserialize()` reads the container's flag byte and returns the correct Reader automatically. The mode dispatch is transparent to callers unless they import `Reader` or `PreserveReader` directly.
204
232
 
@@ -221,9 +249,9 @@ An 8 GB overnight soak on an M1 MacBook Pro, with both release gates armed:
221
249
  | Verify wall | 3m37s |
222
250
  | **Preservation mismatches** | **0** |
223
251
 
224
- **499 KB of heap allocation to compile 8 GB of JSON.** Not per-shard, not per-second total, for the entire run. Zero garbage collections of any kind. Every one of 590 million declared cells round-tripped byte-exact.
252
+ **499 KB of heap allocation to compile 8 GB of JSON.** Not per-shard, not per-second -- total, for the entire run. Zero garbage collections of any kind. Every one of 590 million declared cells round-tripped byte-exact.
225
253
 
226
- Tokenizer bench on the same machine: **222.8 MB/s** on a 10 MB fixture, **237.0 MB/s** on 50 MB (min of 10 reps) roughly 55% of `JSON.parse`'s raw throughput, while allocating no object graph at all. `JSON.parse` cannot process an 8 GB input on any heap; this can, on a 500 KB one.
254
+ Tokenizer bench on the same machine: **222.8 MB/s** on a 10 MB fixture, **237.0 MB/s** on 50 MB (min of 10 reps) -- roughly 55% of `JSON.parse`'s raw throughput, while allocating no object graph at all. `JSON.parse` cannot process an 8 GB input on any heap; this can, on a 500 KB one.
227
255
 
228
256
  ### Why this is 1.0.0 and not 0.x
229
257
 
@@ -233,7 +261,7 @@ The format carries the forward-compat levers it needs to grow without a breaking
233
261
  - **Reserved field flags** in the schema block's FieldDescriptor, admitting the planned I64-preserved lane.
234
262
  - **`metadata_off`** as a block wrapper, not a zone-maps-specific pointer. Container-level string tables and columnar-mode metadata can land in the same slot.
235
263
 
236
- Planned additions the I64 lane, columnar payload mode, container-level interning all fit these seams. `format_version` stays at `1`.
264
+ Planned additions -- the I64 lane, columnar payload mode, container-level interning -- all fit these seams. `format_version` stays at `1`.
237
265
 
238
266
  ### Stability commitment
239
267
 
@@ -246,50 +274,50 @@ Public API (the 9 documented subpath exports and their `.d.ts` declarations) fol
246
274
 
247
275
  ---
248
276
 
249
- ## [0.1.0-alpha.0] pre-release development line
277
+ ## [0.1.0-alpha.0] -- pre-release development line
250
278
 
251
- ### Fixed critical: 32-bit offset overflow past 4 GiB containers
279
+ ### Fixed -- critical: 32-bit offset overflow past 4 GiB containers
252
280
 
253
- Reported from an 8 GB overnight soak on an M1 MacBook (see run log): at row 81,369,452 of ~98M, verification failed with `got 8.118871342983516e-307, expected 81369452`. Ingest itself completed cleanly the failure was on read-back.
281
+ Reported from an 8 GB overnight soak on an M1 MacBook (see run log): at row 81,369,452 of ~98M, verification failed with `got 8.118871342983516e-307, expected 81369452`. Ingest itself completed cleanly -- the failure was on read-back.
254
282
 
255
- **Root cause.** `payload_off` in the shard directory (SPEC §3.4) was `u32`. Similarly, `schema_block_off`, `metadata_off`, and `shard_directory_off` in the header were `u32`. A container that grows past 2^32 bytes (~4.29 GB) has late-shard offsets beyond the u32 range; on write they were being truncated silently (`setUint32` accepts and masks JS numbers > 2^32), and on read they'd point at earlier bytes in the container. The reader then interpreted 8 bytes of arbitrary content as a Float64, producing subnormals like `8.1e-307`. Rows in shards whose payload sat below the 4 GB mark verified fine the failure surfaced exactly where the boundary was crossed, at ~82% of an 8 GB soak (4.89 GB compiled container × 0.82 4.0 GB).
283
+ **Root cause.** `payload_off` in the shard directory (SPEC section 3.4) was `u32`. Similarly, `schema_block_off`, `metadata_off`, and `shard_directory_off` in the header were `u32`. A container that grows past 2^32 bytes (~4.29 GB) has late-shard offsets beyond the u32 range; on write they were being truncated silently (`setUint32` accepts and masks JS numbers > 2^32), and on read they'd point at earlier bytes in the container. The reader then interpreted 8 bytes of arbitrary content as a Float64, producing subnormals like `8.1e-307`. Rows in shards whose payload sat below the 4 GB mark verified fine -- the failure surfaced exactly where the boundary was crossed, at ~82% of an 8 GB soak (4.89 GB compiled container × 0.82 ~= 4.0 GB).
256
284
 
257
285
  **Fix.**
258
286
  - Header layout grew from **32 to 48 bytes**. `schema_block_off`, `metadata_off`, `shard_directory_off`, and `total_rows` are all `u64`. Added an explicit `reserved1` at offset 36 to keep the layout 8-byte aligned.
259
287
  - `ShardEntry` layout grew from **32 to 40 bytes**. `payload_off` is now `u64` at offset 0; `payload_len` and `row_count` are `u32` at 8 and 12; `min_reader_version` and `flags` are `u16` at 16 and 18; `reserved` at 20; `local_string_off` and `local_string_len` are `u64` at 24 and 32.
260
- - `Writer.js`, `Reader.js`, `RangeReader.js`, and `Split.js` (`mergeContainers`) updated in lockstep every offset field is now `setBigUint64`/`getBigUint64`.
261
- - SPEC §3.1 and §3.4 updated to reflect the new byte layouts.
288
+ - `Writer.js`, `Reader.js`, `RangeReader.js`, and `Split.js` (`mergeContainers`) updated in lockstep -- every offset field is now `setBigUint64`/`getBigUint64`.
289
+ - SPEC section 3.1 and section 3.4 updated to reflect the new byte layouts.
262
290
 
263
- **Regression test.** `test/OffsetOverflow.test.js` 4 tests that write and read offset values >2^32 through the actual DataView paths a Reader would use, verifying the u64 round-trip is byte-exact and that the new ShardEntry stride (40 bytes) is honored by second-shard reads. An 8 GB soak isn't feasible in CI, but the arithmetic property that failed at 4+ GB is directly covered.
291
+ **Regression test.** `test/OffsetOverflow.test.js` -- 4 tests that write and read offset values >2^32 through the actual DataView paths a Reader would use, verifying the u64 round-trip is byte-exact and that the new ShardEntry stride (40 bytes) is honored by second-shard reads. An 8 GB soak isn't feasible in CI, but the arithmetic property that failed at 4+ GB is directly covered.
264
292
 
265
- **Impact for pre-existing containers.** None the format was pre-publish (0.1.0-alpha.0), no tagged release existed. Any pre-fix development containers must be regenerated. The wire-format version stays `1`.
293
+ **Impact for pre-existing containers.** None -- the format was pre-publish (0.1.0-alpha.0), no tagged release existed. Any pre-fix development containers must be regenerated. The wire-format version stays `1`.
266
294
 
267
295
  **Bench (also from the M1 MacBook run):** tokenizer clocked at **222.8 MB/s** on 10 MB and **237.0 MB/s** on 50 MB (min of 10 reps), running at ~55% of `JSON.parse`'s throughput while producing zero object allocations. Torture: 39/39 including 500 MB clean soak, zero GC.
268
296
 
269
- ### Fixed demo table CapacityError on 50k-row fixtures
297
+ ### Fixed -- demo table CapacityError on 50k-row fixtures
270
298
 
271
- `lite-table` allocates one signal node per row for row-level reactivity; its internal pool caps around 1024 nodes. The demo mounted the full 50k-row synthetic fixture and blew the cap. Fixed by capping the mounted table at `MAX_TABLE_ROWS = 1000` and logging a warn banner steering users toward the RangeReader panel for browsing the full dataset. The ingest and RangeReader paths are unaffected this is purely a demo UI budget.
299
+ `lite-table` allocates one signal node per row for row-level reactivity; its internal pool caps around 1024 nodes. The demo mounted the full 50k-row synthetic fixture and blew the cap. Fixed by capping the mounted table at `MAX_TABLE_ROWS = 1000` and logging a warn banner steering users toward the RangeReader panel for browsing the full dataset. The ingest and RangeReader paths are unaffected -- this is purely a demo UI budget.
272
300
 
273
- ### Fixed demo error message on non-JSON inputs
301
+ ### Fixed -- demo error message on non-JSON inputs
274
302
 
275
- Feeding `llms.txt` (or any file starting with a non-JSON byte) into the ingest panel produced `E_UNEXPECTED_BYTE: unexpected byte 0x23 at value position at byte 0` technically correct but opaque. The demo now appends " file does not look like JSON/NDJSON." when the failing byte is at position 0. Tokenizer behavior unchanged.
303
+ Feeding `llms.txt` (or any file starting with a non-JSON byte) into the ingest panel produced `E_UNEXPECTED_BYTE: unexpected byte 0x23 at value position at byte 0` -- technically correct but opaque. The demo now appends "-- file does not look like JSON/NDJSON." when the failing byte is at position 0. Tokenizer behavior unchanged.
276
304
 
277
- ### Added M8 partial (Conformance + robustness gates)
305
+ ### Added -- M8 partial (Conformance + robustness gates)
278
306
 
279
- The M8 mandate has three sub-items: RFC 8259 conformance corpus, fuzzing, oscilloscope-rack demo. This ships the first two the parts that strengthen the tokenizer's correctness story and belong in every release gate. The oscilloscope-rack demo remains its own project (multi-session UI work; targeted for after 0.1.0).
307
+ The M8 mandate has three sub-items: RFC 8259 conformance corpus, fuzzing, oscilloscope-rack demo. This ships the first two -- the parts that strengthen the tokenizer's correctness story and belong in every release gate. The oscilloscope-rack demo remains its own project (multi-session UI work; targeted for after 0.1.0).
280
308
 
281
- - **`test/Conformance.test.js`** 66 tests locking in strict RFC 8259 accept/reject behavior. Every fixture is either `MUST_ACCEPT` (strict valid JSON tokenizer completes without error) or `MUST_REJECT` (invalid per RFC tokenizer must throw `TokenizerError` with a stable `E_*` code). Categories cover:
309
+ - **`test/Conformance.test.js`** -- 66 tests locking in strict RFC 8259 accept/reject behavior. Every fixture is either `MUST_ACCEPT` (strict valid JSON -- tokenizer completes without error) or `MUST_REJECT` (invalid per RFC -- tokenizer must throw `TokenizerError` with a stable `E_*` code). Categories cover:
282
310
  - **Structural**: empty containers, nested forms, whitespace variants (spaces, tabs, CR, LF, CRLF), missing/extra commas, mismatched brackets, trailing commas, unquoted/single-quoted keys and values.
283
- - **Strings**: empty, all standard escapes (`\"`, `\\`, `\/`, `\b`, `\f`, `\n`, `\r`, `\t`), `\uXXXX` at every Unicode plane boundary (U+0000, U+007F, U+FFFF, U+10000 via surrogate pair), raw UTF-8 multi-byte and 4-byte, adversarial rejections (unclosed strings, raw control chars per §7, lone high surrogate, bad escape).
311
+ - **Strings**: empty, all standard escapes (`\"`, `\\`, `\/`, `\b`, `\f`, `\n`, `\r`, `\t`), `\uXXXX` at every Unicode plane boundary (U+0000, U+007F, U+FFFF, U+10000 via surrogate pair), raw UTF-8 multi-byte and 4-byte, adversarial rejections (unclosed strings, raw control chars per section 7, lone high surrogate, bad escape).
284
312
  - **Numbers**: integer, negative, zero (including -0), decimal, scientific (`e`, `E`, `+/-` sign, negative exponent), full decimal+exponent form, `Number.MAX_SAFE_INTEGER` and beyond (with documented F64 precision loss), rejections for leading zero (`01`), trailing decimal (`1.`), leading decimal (`.5`), bare exponent (`1e`), `NaN`, `Infinity`, hex (`0x1F`).
285
313
  - **Literals**: `true`, `false`, `null`, plus rejections for `True`, `TRUE`, truncated `tru`.
286
314
 
287
- - **`test/Fuzz.test.js`** 7 tests, each running hundreds of internal iterations. Proves the robustness invariant:
288
- > For ANY byte sequence valid, malformed, adversarial, random the tokenizer either completes without error OR throws a `TokenizerError` with a stable `E_*` code. It never throws a non-`TokenizerError` exception, returns silent garbage, hangs, or corrupts internal state.
315
+ - **`test/Fuzz.test.js`** -- 7 tests, each running hundreds of internal iterations. Proves the robustness invariant:
316
+ > For ANY byte sequence -- valid, malformed, adversarial, random -- the tokenizer either completes without error OR throws a `TokenizerError` with a stable `E_*` code. It never throws a non-`TokenizerError` exception, returns silent garbage, hangs, or corrupts internal state.
289
317
  Fuzz categories:
290
- - 500 rounds of pure-random bytes across every byte value 0x000xFF.
318
+ - 500 rounds of pure-random bytes across every byte value 0x00-0xFF.
291
319
  - 500 rounds of JSON-alphabet-only bytes (`{}[]":,. 0-9 e E letters`), which surfaces near-JSON malformed inputs. Asserts a diverse set of rejection codes to prove the fuzz exercises multiple failure paths.
292
- - 200 rounds of generator-produced valid JSON must round-trip cleanly.
320
+ - 200 rounds of generator-produced valid JSON -- must round-trip cleanly.
293
321
  - 500 rounds of single-byte mutation of valid JSON: response must be clean (accept or `E_*`).
294
322
  - Truncation at every offset of 5 generated valid inputs.
295
323
  - Chunk-boundary rehearsal at every byte-split position of 4 curated inputs; behavior must be identical to feeding the whole input at once.
@@ -306,25 +334,25 @@ The M8 mandate has three sub-items: RFC 8259 conformance corpus, fuzzing, oscill
306
334
 
307
335
  ### Deferred to M8 (later)
308
336
 
309
- - **Oscilloscope-rack demo** reel-to-reel tape transport, waterfall spectrum per shard, VU needles for MB/s and rows/sec, seven-segment counters. Release gate: locked 60 fps on iPhone 7 with 100 MB NDJSON. Multi-session UI project; the demo blueprint is documented in the memory system as the M8 north star.
337
+ - **Oscilloscope-rack demo** -- reel-to-reel tape transport, waterfall spectrum per shard, VU needles for MB/s and rows/sec, seven-segment counters. Release gate: locked 60 fps on iPhone 7 with 100 MB NDJSON. Multi-session UI project; the demo blueprint is documented in the memory system as the M8 north star.
310
338
 
311
- ### Added M6 (Split-and-merge primitive)
339
+ ### Added -- M6 (Split-and-merge primitive)
312
340
 
313
341
  The M6 mandate: enable parallel or checkpointed ingest without requiring a specific worker module. The primitives ship as pure functions that a caller can wire to any parallel executor (Web Workers, `worker_threads`, worker pools, no workers at all).
314
342
 
315
- - **`src/Split.js`** four exports covering the full workflow:
316
- - `splitNDJSON(bytes, {targetParts?, maxPartBytes?})` `SplitRange[]`. Divides NDJSON bytes into N byte ranges at safe line boundaries. Every returned range contains complete records no split mid-line. Union of ranges equals the original bytes (verified by test: concatenating `bytes.subarray(start, end)` across parts is byte-identical to the original).
317
- - `compilePart(bytes, opts)` `Uint8Array`. Compiles one byte range into a standalone LBK1 container via internal Tokenizer + Writer. Result is trivially `Transferable` across worker boundaries via `postMessage(buf, [buf])`.
318
- - `compileInParts(bytes, opts)` `Uint8Array[]`. Sequential convenience: split + compile each part serially. Same output shape as running each part through a worker; use this when workers aren't available or as a deterministic baseline against a worker-parallelized run.
319
- - `mergeContainers([...Uint8Array | ArrayBuffer])` `Uint8Array`. Concatenates N LBK1 containers into ONE. Verifies schema equivalence across inputs (rejects with `S_SCHEMA_MISMATCH` otherwise). Copies per-shard payloads and per-shard string tables verbatim; rewrites the shard directory offsets, header total counts, and zone-maps segment. Preserves the per-shard-independence property merged shards remain independently decodable and range-fetchable.
320
- - **`test/M6.test.js`** 16 tests: split-boundary safety (no mid-line cuts), concatenation-equals-original invariant, empty-input handling, single-line input (no natural split available), `maxPartBytes` ceiling honored, compilePart round-trip, compileInParts + MultiReader byte-exact equivalence with single-pass `serialize`, mergeContainers over 2 and 4 parts, mergeContainers vs MultiReader value equivalence, zone-maps preservation across merge, schema-mismatch rejection, empty-input rejection, ArrayBuffer input acceptance, single-container merge as no-op, and full splitcompilemergeverify pipeline via the universal verifier.
343
+ - **`src/Split.js`** -- four exports covering the full workflow:
344
+ - `splitNDJSON(bytes, {targetParts?, maxPartBytes?})` -> `SplitRange[]`. Divides NDJSON bytes into N byte ranges at safe line boundaries. Every returned range contains complete records -- no split mid-line. Union of ranges equals the original bytes (verified by test: concatenating `bytes.subarray(start, end)` across parts is byte-identical to the original).
345
+ - `compilePart(bytes, opts)` -> `Uint8Array`. Compiles one byte range into a standalone LBK1 container via internal Tokenizer + Writer. Result is trivially `Transferable` across worker boundaries via `postMessage(buf, [buf])`.
346
+ - `compileInParts(bytes, opts)` -> `Uint8Array[]`. Sequential convenience: split + compile each part serially. Same output shape as running each part through a worker; use this when workers aren't available or as a deterministic baseline against a worker-parallelized run.
347
+ - `mergeContainers([...Uint8Array | ArrayBuffer])` -> `Uint8Array`. Concatenates N LBK1 containers into ONE. Verifies schema equivalence across inputs (rejects with `S_SCHEMA_MISMATCH` otherwise). Copies per-shard payloads and per-shard string tables verbatim; rewrites the shard directory offsets, header total counts, and zone-maps segment. Preserves the per-shard-independence property -- merged shards remain independently decodable and range-fetchable.
348
+ - **`test/M6.test.js`** -- 16 tests: split-boundary safety (no mid-line cuts), concatenation-equals-original invariant, empty-input handling, single-line input (no natural split available), `maxPartBytes` ceiling honored, compilePart round-trip, compileInParts + MultiReader byte-exact equivalence with single-pass `serialize`, mergeContainers over 2 and 4 parts, mergeContainers vs MultiReader value equivalence, zone-maps preservation across merge, schema-mismatch rejection, empty-input rejection, ArrayBuffer input acceptance, single-container merge as no-op, and full split->compile->merge->verify pipeline via the universal verifier.
321
349
  - **`/split` subpath** added to `package.json` exports with matching `types/Split.d.ts`.
322
350
 
323
351
  ### Design decisions locked in M6
324
352
 
325
- - **Parallelism-safe requires explicit schema.** If two workers ran sample-and-infer independently, they could infer different field orderings or lane kinds outputs would then fail to merge. The docs specify that multi-part compilation MUST pass the same explicit schema to every part. For sample-and-infer with parts, compile part 0 first, extract `new Reader(container.buffer).schema`, and pass it to parts 1..N.
326
- - **MultiReader vs mergeContainers is a workload choice.** For query workloads over multiple parts, `MultiReader` (M4) is strictly better it avoids the copy and works on containers still resident where they were produced (worker memory, IndexedDB, HTTP hosts). Use `mergeContainers` only when a single-file distributable is genuinely required (upload artifact, checkpoint file). Both paths verified to produce equivalent value queries.
327
- - **No workers in the box.** The Split module ships pure functions. Wire them to whatever parallel executor exists in your environment the same code path works serially, in Web Workers, in `worker_threads`, in a worker pool, or in a distributed compute layer. The Split module is executor-agnostic by design.
353
+ - **Parallelism-safe requires explicit schema.** If two workers ran sample-and-infer independently, they could infer different field orderings or lane kinds -- outputs would then fail to merge. The docs specify that multi-part compilation MUST pass the same explicit schema to every part. For sample-and-infer with parts, compile part 0 first, extract `new Reader(container.buffer).schema`, and pass it to parts 1..N.
354
+ - **MultiReader vs mergeContainers is a workload choice.** For query workloads over multiple parts, `MultiReader` (M4) is strictly better -- it avoids the copy and works on containers still resident where they were produced (worker memory, IndexedDB, HTTP hosts). Use `mergeContainers` only when a single-file distributable is genuinely required (upload artifact, checkpoint file). Both paths verified to produce equivalent value queries.
355
+ - **No workers in the box.** The Split module ships pure functions. Wire them to whatever parallel executor exists in your environment -- the same code path works serially, in Web Workers, in `worker_threads`, in a worker pool, or in a distributed compute layer. The Split module is executor-agnostic by design.
328
356
 
329
357
  ### Worker workflow example
330
358
 
@@ -352,17 +380,17 @@ const merged = mergeContainers(containers);
352
380
  - 37/37 torture scenarios still pass. Zero major GC. Zero minor GC.
353
381
  - `npm publish --dry-run` still clean: 24 files, 62.6 kB packed.
354
382
 
355
- ### Added M4 (Publish prep, convenience API, MultiReader)
383
+ ### Added -- M4 (Publish prep, convenience API, MultiReader)
356
384
 
357
385
  The M4 mandate: get the package to a publishable state and add the ergonomic wrappers most consumers actually want.
358
386
 
359
- - **`src/index.js`** top-level convenience API. `serialize(input, opts)` accepts `Uint8Array` NDJSON, `string` NDJSON, `Iterable<object>` (records), `ReadableStream<Uint8Array>`, or `AsyncIterable<Uint8Array>`; returns `Uint8Array` for sync inputs, `Promise<Uint8Array>` for async. `deserialize(bytes)` returns a `Reader` from either a `Uint8Array` or `ArrayBuffer`; handles subarray offsets by copying to a fresh aligned buffer. Re-exports `Tokenizer`, `Writer`, `Reader`, `StringTable` and their error classes so a single `@zakkster/lite-bake-stream` import covers the common case.
360
- - **`src/MultiReader.js`** logical union view over N `Reader` instances with matching schemas. Rows and shards are cumulatively addressed: rows 0..N₀ live in reader 0, N₀..N₀+N₁ in reader 1, etc. `findShards` merges per-reader results into a global shard-index space so query pruning works across the union. Rejects mismatched schemas at construction with `M_SCHEMA_MISMATCH`. Motivates the M6 split-and-merge story: split-and-merge doesn't strictly need a merge step if a `MultiReader` can serve queries directly over the split outputs.
361
- - **`test/M4.test.js`** 18 tests: `serialize` for every input shape (bytes, string, records iterable, async iterable, ReadableStream) and its `deserialize` round-trip; `deserialize` handling subarray offsets; MultiReader single-reader passthrough parity, 4-reader boundary-walk, schema-mismatch rejection, empty-array rejection, `findShards` returning global indices across sub-readers, `shardBounds` routing to the correct sub-reader, out-of-range and unknown-field error paths.
387
+ - **`src/index.js`** -- top-level convenience API. `serialize(input, opts)` accepts `Uint8Array` NDJSON, `string` NDJSON, `Iterable<object>` (records), `ReadableStream<Uint8Array>`, or `AsyncIterable<Uint8Array>`; returns `Uint8Array` for sync inputs, `Promise<Uint8Array>` for async. `deserialize(bytes)` returns a `Reader` from either a `Uint8Array` or `ArrayBuffer`; handles subarray offsets by copying to a fresh aligned buffer. Re-exports `Tokenizer`, `Writer`, `Reader`, `StringTable` and their error classes so a single `@zakkster/lite-bake-stream` import covers the common case.
388
+ - **`src/MultiReader.js`** -- logical union view over N `Reader` instances with matching schemas. Rows and shards are cumulatively addressed: rows 0..N0 live in reader 0, N0..N0+N1 in reader 1, etc. `findShards` merges per-reader results into a global shard-index space so query pruning works across the union. Rejects mismatched schemas at construction with `M_SCHEMA_MISMATCH`. Motivates the M6 split-and-merge story: split-and-merge doesn't strictly need a merge step if a `MultiReader` can serve queries directly over the split outputs.
389
+ - **`test/M4.test.js`** -- 18 tests: `serialize` for every input shape (bytes, string, records iterable, async iterable, ReadableStream) and its `deserialize` round-trip; `deserialize` handling subarray offsets; MultiReader single-reader passthrough parity, 4-reader boundary-walk, schema-mismatch rejection, empty-array rejection, `findShards` returning global indices across sub-readers, `shardBounds` routing to the correct sub-reader, out-of-range and unknown-field error paths.
362
390
  - **Root subpath `.`** and **`/multi-reader`** added to `package.json` exports. Every subpath now has a matching `.d.ts` file in `types/`.
363
- - **`types/` populated** 8 hand-written `.d.ts` files (`Tokenizer`, `Writer`, `Reader`, `StringTable`, `RangeReader`, `FileIngest`, `MultiReader`, `index`) covering every public class, function, error type, and options interface. `import type { ... }` cross-references between them so consumers importing a single subpath still see referenced types.
364
- - **`LICENSE`** MIT text, copyright Zahary Shinikchiev. Referenced by `package.json` and included in the tarball.
365
- - **`package.json` publish metadata** `bugs.url`, `homepage`, `engines.node: ">=18"` added. `publishConfig.access: public` was already in place.
391
+ - **`types/` populated** -- 8 hand-written `.d.ts` files (`Tokenizer`, `Writer`, `Reader`, `StringTable`, `RangeReader`, `FileIngest`, `MultiReader`, `index`) covering every public class, function, error type, and options interface. `import type { ... }` cross-references between them so consumers importing a single subpath still see referenced types.
392
+ - **`LICENSE`** -- MIT text, copyright Zahary Shinikchiev. Referenced by `package.json` and included in the tarball.
393
+ - **`package.json` publish metadata** -- `bugs.url`, `homepage`, `engines.node: ">=18"` added. `publishConfig.access: public` was already in place.
366
394
 
367
395
  ### Publish gate
368
396
 
@@ -376,34 +404,34 @@ The M4 mandate: get the package to a publishable state and add the ergonomic wra
376
404
 
377
405
  ### Migration notes for consumers
378
406
 
379
- - **Common case (`serialize`/`deserialize`)**: `import { serialize, deserialize } from '@zakkster/lite-bake-stream';`. Bytes in LBK1 bytes Reader. Three lines.
407
+ - **Common case (`serialize`/`deserialize`)**: `import { serialize, deserialize } from '@zakkster/lite-bake-stream';`. Bytes in -> LBK1 bytes -> Reader. Three lines.
380
408
  - **Advanced case (subpaths)**: import from `/tokenizer`, `/writer`, `/reader`, `/range-reader`, etc. for finer control. Tree-shakers drop unused paths (`sideEffects: false`).
381
- - **New import**: `@zakkster/lite-bake-stream/multi-reader` provides `MultiReader` for joining multiple LBK1 containers into one logical stream worker-parallel compilation output, checkpoint/resume sessions, split-and-merge without the merge.
409
+ - **New import**: `@zakkster/lite-bake-stream/multi-reader` provides `MultiReader` for joining multiple LBK1 containers into one logical stream -- worker-parallel compilation output, checkpoint/resume sessions, split-and-merge without the merge.
382
410
 
383
- ### Added M7 (Zone maps)
411
+ ### Added -- M7 (Zone maps)
384
412
 
385
- The M7 mandate: enable query pruning. A filter like "rows where `x` in [800, 900]" should touch only the shards whose bounds overlap that range not every shard. On HTTP Range fetches (M5), each skipped shard is one round-trip saved.
413
+ The M7 mandate: enable query pruning. A filter like "rows where `x` in [800, 900]" should touch only the shards whose bounds overlap that range -- not every shard. On HTTP Range fetches (M5), each skipped shard is one round-trip saved.
386
414
 
387
- - **SPEC §3.6 Zone Maps Segment.** New optional segment placed between shard directory and first shard payload. Layout: `'ZM01'` magic + shard count + tracked field count + reserved, then field-index table (u16 per tracked field), then row-major `f64 mins[shard * T + t]` and `f64 maxes[shard * T + t]`. Header offset 12 (formerly reserved for container-level string table) is now `metadata_off` `0` means no zone maps. Bumped SPEC section 3.7 to accommodate.
388
- - **`src/Writer.js`** Per-shard, per-F64-field min/max tracked on the row-commit hot path. `_trackedFieldToPos` (Int32Array) dispatches from schema field index tracking position in O(1). Sample drain path also updates bounds. On finalize, snapshot each shard's `mins`/`maxes` into the shard record. `_assembleContainer` emits the zone maps segment before shard payloads and wires the header's `metadata_off`. U32/string fields are not tracked in v1 (dictionary-level bounds are M8+ material).
389
- - **`src/Reader.js`** `_parseZoneMaps` reads the segment if `metadata_off != 0`, otherwise no-op. Zero-copy `Float64Array` views over the container buffer for `mins`/`maxes`. New public API:
390
- - `reader.hasZoneMaps` boolean
391
- - `reader.shardBounds(shardIdx, fieldName)` `{min, max} | null` (null for U32 fields, missing zone maps, or unknown fields)
392
- - `reader.findShards(fieldName, {min?, max?})` `shardIdx[]` every shard whose bounds overlap `[min, max]`. Falls back to all shards when zone maps or the field's tracking is absent (correct query-planner semantics).
393
- - **`src/RangeReader.js`** Zone maps fetched ONCE in the open phase (16-byte segment header + one range for the body), then `shardBounds` and `findShards` are synchronous. The query-pruning workflow becomes: `open()` `findShards()` `loadShard()` only on covering shards. Each skipped shard is one HTTP Range fetch saved. Locked in by the acceptance test: `findShards('id', {min: 800, max: 900})` on a 1000-row / ~10-shard container returns 2 shards, and reading them triggers exactly 2 additional adapter fetches zero for the skipped 8.
394
- - **`test/ZoneMaps.test.js`** 10 tests: `hasZoneMaps` state, per-shard min/max correctness against scan-verified values, null return for U32 fields and unknown fields, `findShards` overlap semantics (inclusive on both endpoints), unbounded query returns all shards, U32-field fallback returns all shards, RangeReader parity with base Reader, and the **shard-fetch-pruning proof**: `findShards` runs synchronously (zero adapter fetches) and reading the returned shards triggers exactly `shardsWanted.length` additional fetches.
415
+ - **SPEC section 3.6 -- Zone Maps Segment.** New optional segment placed between shard directory and first shard payload. Layout: `'ZM01'` magic + shard count + tracked field count + reserved, then field-index table (u16 per tracked field), then row-major `f64 mins[shard * T + t]` and `f64 maxes[shard * T + t]`. Header offset 12 (formerly reserved for container-level string table) is now `metadata_off` -- `0` means no zone maps. Bumped SPEC section 3.7 to accommodate.
416
+ - **`src/Writer.js`** -- Per-shard, per-F64-field min/max tracked on the row-commit hot path. `_trackedFieldToPos` (Int32Array) dispatches from schema field index -> tracking position in O(1). Sample drain path also updates bounds. On finalize, snapshot each shard's `mins`/`maxes` into the shard record. `_assembleContainer` emits the zone maps segment before shard payloads and wires the header's `metadata_off`. U32/string fields are not tracked in v1 (dictionary-level bounds are M8+ material).
417
+ - **`src/Reader.js`** -- `_parseZoneMaps` reads the segment if `metadata_off != 0`, otherwise no-op. Zero-copy `Float64Array` views over the container buffer for `mins`/`maxes`. New public API:
418
+ - `reader.hasZoneMaps` -- boolean
419
+ - `reader.shardBounds(shardIdx, fieldName)` -> `{min, max} | null` (null for U32 fields, missing zone maps, or unknown fields)
420
+ - `reader.findShards(fieldName, {min?, max?})` -> `shardIdx[]` -- every shard whose bounds overlap `[min, max]`. Falls back to all shards when zone maps or the field's tracking is absent (correct query-planner semantics).
421
+ - **`src/RangeReader.js`** -- Zone maps fetched ONCE in the open phase (16-byte segment header + one range for the body), then `shardBounds` and `findShards` are synchronous. The query-pruning workflow becomes: `open()` -> `findShards()` -> `loadShard()` only on covering shards. Each skipped shard is one HTTP Range fetch saved. Locked in by the acceptance test: `findShards('id', {min: 800, max: 900})` on a 1000-row / ~10-shard container returns 2 shards, and reading them triggers exactly 2 additional adapter fetches -- zero for the skipped 8.
422
+ - **`test/ZoneMaps.test.js`** -- 10 tests: `hasZoneMaps` state, per-shard min/max correctness against scan-verified values, null return for U32 fields and unknown fields, `findShards` overlap semantics (inclusive on both endpoints), unbounded query returns all shards, U32-field fallback returns all shards, RangeReader parity with base Reader, and the **shard-fetch-pruning proof**: `findShards` runs synchronously (zero adapter fetches) and reading the returned shards triggers exactly `shardsWanted.length` additional fetches.
395
423
 
396
424
  ### Design decisions locked in M7
397
425
 
398
426
  - **Zone maps sit between shard directory and shard payloads.** Rationale: any Reader that fetches header + schema + directory can extend that Range to include zone maps in one round-trip on HTTP/2. Placing them at a distant offset would cost an extra request for every open.
399
427
  - **F64 lanes only.** String bounds require dictionary-level min/max (canonical string ordering), which needs the container-level string table (M4+). Deferred as clearly-out-of-scope for M7.
400
- - **Overlap semantics for `findShards`:** a shard is included iff `smin <= max && smax >= min`. This is inclusive on both endpoints a shard whose max exactly equals the query's min is still included (its rows may match `<=` predicates). Callers wanting strict-less-than semantics filter the returned shards further.
428
+ - **Overlap semantics for `findShards`:** a shard is included iff `smin <= max && smax >= min`. This is inclusive on both endpoints -- a shard whose max exactly equals the query's min is still included (its rows may match `<=` predicates). Callers wanting strict-less-than semantics filter the returned shards further.
401
429
  - **No zone maps when the schema is all-U32.** The writer sets `metadata_off = 0` when `T === 0`; a Reader loading an all-string schema sees `hasZoneMaps === false` and `findShards` falls back to returning every shard. No overhead, no wasted bytes.
402
430
 
403
431
  ### Migration notes
404
432
 
405
- - Containers written before M7 have `metadata_off = 0` and work unchanged. Readers built after M7 handle them via the fallback path (no zone maps `findShards` returns all shards, `shardBounds` returns `null`).
406
- - The header field at offset 12 was formerly documented as `string_table_off` (reserved). Any consumer that read that value literally will still see `0` for any container the writer produced through M6 it never wrote a non-zero value there.
433
+ - Containers written before M7 have `metadata_off = 0` and work unchanged. Readers built after M7 handle them via the fallback path (no zone maps -> `findShards` returns all shards, `shardBounds` returns `null`).
434
+ - The header field at offset 12 was formerly documented as `string_table_off` (reserved). Any consumer that read that value literally will still see `0` for any container the writer produced through M6 -- it never wrote a non-zero value there.
407
435
 
408
436
  ### Baseline results (M7)
409
437
 
@@ -411,33 +439,33 @@ The M7 mandate: enable query pruning. A filter like "rows where `x` in [800, 900
411
439
  - 37/37 torture scenarios still pass. Zero major GC. The zone-map writes on the row-commit path are branchless (`v < min` / `v > max` compare-and-assign against a `Float64Array`) and cache-friendly. No throughput regression measured.
412
440
  - Query-pruning demonstration: on a 1000-row container with ~10 shards (2 KB target), querying `id in [500, 550]` returns exactly the 1-2 covering shards, verified by scan-based ground truth. The RangeReader path fetches only those shards' payloads.
413
441
 
414
- ### Added M5 (Browser reader, HTTP Range, File.stream ingest)
442
+ ### Added -- M5 (Browser reader, HTTP Range, File.stream ingest)
415
443
 
416
444
  The M5 mandate: run the whole pipeline in a browser. Client-side ingest of a local JSON file to an LBK1 container. Remote LBK1 exploration via HTTP Range without downloading the whole file. Reference demo bridging both to `@zakkster/lite-table`.
417
445
 
418
- - `src/RangeReader.js` LBK1 reader with lazy shard loading via an IO adapter. On `open()`, fetches exactly 3 byte ranges (header 32B, schema block, shard directory) regardless of container size every shard's byte extents are known without touching a single row. On `get(rowIdx, fieldName)`, fetches only the containing shard's payload + local string table in ONE range request (they're contiguous by SPEC 3.4), caches with LRU eviction bounded to `maxCachedShards`, decodes byte-identical to the base Reader. Also exposes `prefetchRange(firstRow, lastRow)` for viewport-driven prefetch and `syncRange(firstRow, lastRow)` for await-free reads after prefetch.
419
- - `HTTPRangeAdapter` (in RangeReader.js) production adapter using `fetch()` with `Range: bytes=off-end` headers. Discovers `Content-Length` via HEAD, falling back to a single-byte range GET for CDNs that block HEAD. Tracks `stats.requests` and `stats.bytesFetched`.
420
- - `MockRangeAdapter` (in RangeReader.js) in-memory adapter backed by a `Uint8Array`. Records every fetch as `{offset, length}` for test assertions. Used by the demo to prove the RangeReader path against a container just built in the browser, no server required.
421
- - `src/FileIngest.js` `ingestStream(readableStream, opts)` and `ingestFile(blob, opts)`. Piped chunks flow through the existing Tokenizer + Writer; `onProgress` fires per chunk with `{bytesIngested, totalBytes, rowsWritten, shardsCommitted, elapsedMs}`. Works with any `ReadableStream<Uint8Array>` `File.stream()` in browsers, `fetch()` bodies, Node's `Readable.toWeb()` in Node 18+.
422
- - `test/RangeReader.test.js` 13 tests locking in: exactly-3 open-phase fetches (regardless of shard count), single-request combined payload+string-table fetches, byte-identical parity with the base Reader across a 200-row corpus, LRU eviction bounds cache to `maxCachedShards`, cache hits trigger zero additional fetches, `prefetchRange` covers exactly the required shards, `syncRange` refuses uncached shards, error paths (bad magic, row out of range, unknown field), adapter stats accumulation.
423
- - `test/FileIngest.test.js` 8 tests: round-trip parity with 1-byte, 7-byte, and 4KB chunks (worst-case chunk-boundary bugs), `onProgress` monotonic progression, blob-like `.size` and `.stream()` recognition, writer options pass-through (schema + shard sizing), rejection of non-Uint8Array chunks.
424
- - Public getter `reader.buffer` on the base Reader required for the demo's ingest RangeReader handoff without reaching into private state.
446
+ - `src/RangeReader.js` -- LBK1 reader with lazy shard loading via an IO adapter. On `open()`, fetches exactly 3 byte ranges (header 32B, schema block, shard directory) regardless of container size -- every shard's byte extents are known without touching a single row. On `get(rowIdx, fieldName)`, fetches only the containing shard's payload + local string table in ONE range request (they're contiguous by SPEC 3.4), caches with LRU eviction bounded to `maxCachedShards`, decodes byte-identical to the base Reader. Also exposes `prefetchRange(firstRow, lastRow)` for viewport-driven prefetch and `syncRange(firstRow, lastRow)` for await-free reads after prefetch.
447
+ - `HTTPRangeAdapter` (in RangeReader.js) -- production adapter using `fetch()` with `Range: bytes=off-end` headers. Discovers `Content-Length` via HEAD, falling back to a single-byte range GET for CDNs that block HEAD. Tracks `stats.requests` and `stats.bytesFetched`.
448
+ - `MockRangeAdapter` (in RangeReader.js) -- in-memory adapter backed by a `Uint8Array`. Records every fetch as `{offset, length}` for test assertions. Used by the demo to prove the RangeReader path against a container just built in the browser, no server required.
449
+ - `src/FileIngest.js` -- `ingestStream(readableStream, opts)` and `ingestFile(blob, opts)`. Piped chunks flow through the existing Tokenizer + Writer; `onProgress` fires per chunk with `{bytesIngested, totalBytes, rowsWritten, shardsCommitted, elapsedMs}`. Works with any `ReadableStream<Uint8Array>` -- `File.stream()` in browsers, `fetch()` bodies, Node's `Readable.toWeb()` in Node 18+.
450
+ - `test/RangeReader.test.js` -- 13 tests locking in: exactly-3 open-phase fetches (regardless of shard count), single-request combined payload+string-table fetches, byte-identical parity with the base Reader across a 200-row corpus, LRU eviction bounds cache to `maxCachedShards`, cache hits trigger zero additional fetches, `prefetchRange` covers exactly the required shards, `syncRange` refuses uncached shards, error paths (bad magic, row out of range, unknown field), adapter stats accumulation.
451
+ - `test/FileIngest.test.js` -- 8 tests: round-trip parity with 1-byte, 7-byte, and 4KB chunks (worst-case chunk-boundary bugs), `onProgress` monotonic progression, blob-like `.size` and `.stream()` recognition, writer options pass-through (schema + shard sizing), rejection of non-Uint8Array chunks.
452
+ - Public getter `reader.buffer` on the base Reader -- required for the demo's ingest -> RangeReader handoff without reaching into private state.
425
453
  - `/file-ingest` and `/range-reader` subpath exports in `package.json`.
426
454
 
427
- ### Added M5 demo (`demo/`)
455
+ ### Added -- M5 demo (`demo/`)
428
456
 
429
457
  Single-file HTML + one module JS + one importmap, oscilloscope-adjacent theme (phosphor green oklch tokens with hex fallback, CRT grid backdrop, JetBrains Mono). Three tabbed scenes matching the demo blueprint:
430
458
 
431
- 1. **Ingest** drag-and-drop a JSON/NDJSON file OR generate a synthetic 50k-row fixture. Progress bar and per-chunk MB/s. Container stats (rows, shards, source size, container size, compression ratio, throughput).
432
- 2. **Explore** mounts `@zakkster/lite-table` over the ingested Reader. Row source is an integer-index array; column accessors call `reader.get(rowIdx, fieldName)` on demand. This is the trick that keeps a million-row table O(viewport) in memory: no row materialization, ever.
433
- 3. **Range reader** `MockRangeAdapter` over the ingested container bytes; opens a `RangeReader` and shows every `(offset, length)` fetch it makes in a live log. Row-by-row `get()` demonstrates cache hits vs shard-fetch behavior.
459
+ 1. **Ingest** -- drag-and-drop a JSON/NDJSON file OR generate a synthetic 50k-row fixture. Progress bar and per-chunk MB/s. Container stats (rows, shards, source size, container size, compression ratio, throughput).
460
+ 2. **Explore** -- mounts `@zakkster/lite-table` over the ingested Reader. Row source is an integer-index array; column accessors call `reader.get(rowIdx, fieldName)` on demand. This is the trick that keeps a million-row table O(viewport) in memory: no row materialization, ever.
461
+ 3. **Range reader** -- `MockRangeAdapter` over the ingested container bytes; opens a `RangeReader` and shows every `(offset, length)` fetch it makes in a live log. Row-by-row `get()` demonstrates cache hits vs shard-fetch behavior.
434
462
 
435
463
  Import map routes `@zakkster/lite-signal`, `@zakkster/lite-virtual`, `@zakkster/lite-signal-dom`, `@zakkster/lite-table` to esm.sh; `@zakkster/lite-bake-stream/*` subpaths to the sibling `../src/*.js` files. `npm run demo` serves via `npx serve`.
436
464
 
437
465
  ### Design decisions locked in M5
438
466
 
439
467
  - **RangeReader is a separate class, not a mode of Reader.** The base Reader's sync API is right for its use case (in-memory containers, tight loops, no `await` overhead). Async row access needs different ergonomics: `prefetchRange` for viewport-driven fetches, `syncRange` for the render pass, per-shard LRU. Sharing the LBK1 constants + `StringTable` between the two classes is fine; sharing the parse code is not.
440
- - **One range request per shard access.** Per-shard payload and local string table are placed contiguously by the writer (SPEC 3.4). The RangeReader always fetches the combined range in one request HTTP/2 pipelining is nice but a single request is nicer. Saves round-trips at the cost of pulling the string table even when reading only F64 fields; documented as intentional.
468
+ - **One range request per shard access.** Per-shard payload and local string table are placed contiguously by the writer (SPEC 3.4). The RangeReader always fetches the combined range in one request -- HTTP/2 pipelining is nice but a single request is nicer. Saves round-trips at the cost of pulling the string table even when reading only F64 fields; documented as intentional.
441
469
  - **Row source for lite-table is integer indices, not row objects.** For a million-row table this is a 4 MB `Uint32Array` (or 8 MB regular Array) instead of 100+ MB of materialized row objects. Column accessors reach into the Reader on demand. The demo shows a 50k-row fixture; the same code scales to 1M+ without changing the shape.
442
470
 
443
471
  ### Baseline results (M5 tests)
@@ -449,22 +477,22 @@ Import map routes `@zakkster/lite-signal`, `@zakkster/lite-virtual`, `@zakkster/
449
477
  - Eviction: bounded to `maxCachedShards`, verified with cap=2.
450
478
  - Byte-level parity with base Reader: 100% across 200 rows × 3 fields = 600 cells.
451
479
 
452
- ### Added M3 soak session (CLI-configurable scale + real-hardware qualification)
480
+ ### Added -- M3 soak session (CLI-configurable scale + real-hardware qualification)
453
481
 
454
- - `bench/soak.js` CLI-configurable soak entry point. Sized from the command line (`--mb=N` or `--gb=N`), preservation gate opt-in via `--verify`, progress output every 200 MB during ingest and every 500K rows during verify. Fixture generator is memory-bounded (streams directly into a pre-allocated Uint8Array via `TextEncoder.encodeInto`), enabling arbitrary-scale runs without OOM on the source. Reports ingest throughput, verify throughput, container compression ratio, and zero-GC gate result.
482
+ - `bench/soak.js` -- CLI-configurable soak entry point. Sized from the command line (`--mb=N` or `--gb=N`), preservation gate opt-in via `--verify`, progress output every 200 MB during ingest and every 500K rows during verify. Fixture generator is memory-bounded (streams directly into a pre-allocated Uint8Array via `TextEncoder.encodeInto`), enabling arbitrary-scale runs without OOM on the source. Reports ingest throughput, verify throughput, container compression ratio, and zero-GC gate result.
455
483
  - `npm run soak` (100 MB), `soak:500` (500 MB), `soak:1gb` (1 GB), `soak:overnight` (8 GB) scripts. Sized for M4 Pro qualification and macOS `caffeinate` overnight runs on older machines.
456
- - `test/_verify.js` REWRITTEN with bounded memory. Single-pass byte iteration, JSON.parse per line, discard. No line-array accumulator. Memory steady-state is flat regardless of row count. Verified 500 MB / 6.12M rows / 36.75M cells without heap growth.
457
- - Torture scenario `writer-string-sample-multi-shard-drain` lock-in for the bug below. Reproduces the sample-drain-crosses-multiple-shards case with verify:true, so this regression can never come back silently.
484
+ - `test/_verify.js` -- REWRITTEN with bounded memory. Single-pass byte iteration, JSON.parse per line, discard. No line-array accumulator. Memory steady-state is flat regardless of row count. Verified 500 MB / 6.12M rows / 36.75M cells without heap growth.
485
+ - Torture scenario `writer-string-sample-multi-shard-drain` -- lock-in for the bug below. Reproduces the sample-drain-crosses-multiple-shards case with verify:true, so this regression can never come back silently.
458
486
 
459
- ### Fixed Multi-shard sample drain lost strings past shard 0
487
+ ### Fixed -- Multi-shard sample drain lost strings past shard 0
460
488
 
461
489
  The 100 MB soak's preservation gate caught it on the first run. Every row past shard 0 boundary (row 838860 in the failing scenario) read back `undefined` for its string field.
462
490
 
463
- **Root cause:** `_drainSampleToShards` handed the sample's shared string table to shard 0, then `_finalizeCurrentShard` serialized it and immediately reset it in place. Shard 1's rows still contained U32 indices that referenced the (now-empty) shared table, and shard 1's local string table was empty. Only shard 0 had populated string content. Bug was silent unless the sample overflowed one shard's row capacity 400K+ record datasets with any string field. Tests and torture didn't catch it because their sample sizes fit inside one shard.
491
+ **Root cause:** `_drainSampleToShards` handed the sample's shared string table to shard 0, then `_finalizeCurrentShard` serialized it and immediately reset it in place. Shard 1's rows still contained U32 indices that referenced the (now-empty) shared table, and shard 1's local string table was empty. Only shard 0 had populated string content. Bug was silent unless the sample overflowed one shard's row capacity -- 400K+ record datasets with any string field. Tests and torture didn't catch it because their sample sizes fit inside one shard.
464
492
 
465
493
  **Fix:** Per-shard re-interning during drain. `_drainSampleToShards` now allocates a fresh `StringTable` for the drain path, then for each U32 cell looks up the original bytes in the sample's table via `sampleTable.bytesAt(sampleIdx)` and re-interns into the per-shard table. Each drained shard ends up with a tight, self-contained string table holding exactly the strings its rows reference. Preserves the per-shard independence property (SPEC 3.3) that makes M5 HTTP Range fetches work.
466
494
 
467
- **Added:** `StringTable.bytesAt(idx)` zero-copy byte range lookup for entry `idx`. Used by the drain path.
495
+ **Added:** `StringTable.bytesAt(idx)` -- zero-copy byte range lookup for entry `idx`. Used by the drain path.
468
496
 
469
497
  ### Baseline results (M3 soak)
470
498
 
@@ -475,36 +503,36 @@ Reproducible on the sandbox environment (Node 22, not particularly fast hardware
475
503
  | 100 MB | 24.7 MB/s | 386K rows/s | 1.25M | 7.48M | 0 major, 0 minor |
476
504
  | 500 MB | 24.5 MB/s | 375K rows/s | 6.12M | 36.75M | 0 major, 0 minor |
477
505
 
478
- Ingest and verify throughput are essentially flat across the 5× scale jump no per-row overhead compounds. Zero-GC gate holds through both. Container compression is stable at ~62% of source (NDJSON LBK1). Real hardware (M4 Pro etc.) will substantially exceed these numbers.
506
+ Ingest and verify throughput are essentially flat across the 5× scale jump -- no per-row overhead compounds. Zero-GC gate holds through both. Container compression is stable at ~62% of source (NDJSON -> LBK1). Real hardware (M4 Pro etc.) will substantially exceed these numbers.
479
507
 
480
- ### Added M3 addendum (Preservation contract)
508
+ ### Added -- M3 addendum (Preservation contract)
481
509
 
482
510
  The M3 mandate: prove that any data going into the pipeline round-trips out unchanged. Implemented in three complementary layers so no regression can slip through unnoticed:
483
511
 
484
- - **`test/_verify.js`** universal round-trip verifier. Parses source NDJSON with `JSON.parse` (ground truth), walks it against `Reader.get(rowIdx, fieldName)` for every declared field of every row. Not zero-alloc; runs OUTSIDE the GC-measured band by design. This is the correctness oracle used by both tests and torture.
485
- - **`test/DataPreservation.test.js`** property-based fuzz. 23 tests including 12 seeded random-shape corpora (500 rows × 6 fields, random field names, random lane assignments) plus sample-and-infer variants plus a 10K-row 4KB-shard multi-shard preservation gauntlet plus edge cases (extreme F64, adversarial UTF-8, dedupe correctness).
486
- - **`test/NumericBoundary.test.js`** 8 tests that pin the F64 fast-path / slow-path boundary. Asserts EXACT round-trip for the guaranteed domain (15 sig digits, |exp| 22) and pins 1 ULP drift for known slow-path values. Any regression in the number parser trips these before it can reach production.
487
- - **Torture-scale verification** `bench/torture.js` gained a `verify: true` opt-in that runs `verifyRoundTrip` on the assembled container AFTER the GC band closes. Twelve scenarios opted in cover **73,500 rows value-exact per fast-tier run**. The `verified` column in the torture report shows the row count actually checked; any failure blocks the run.
512
+ - **`test/_verify.js`** -- universal round-trip verifier. Parses source NDJSON with `JSON.parse` (ground truth), walks it against `Reader.get(rowIdx, fieldName)` for every declared field of every row. Not zero-alloc; runs OUTSIDE the GC-measured band by design. This is the correctness oracle used by both tests and torture.
513
+ - **`test/DataPreservation.test.js`** -- property-based fuzz. 23 tests including 12 seeded random-shape corpora (500 rows × 6 fields, random field names, random lane assignments) plus sample-and-infer variants plus a 10K-row 4KB-shard multi-shard preservation gauntlet plus edge cases (extreme F64, adversarial UTF-8, dedupe correctness).
514
+ - **`test/NumericBoundary.test.js`** -- 8 tests that pin the F64 fast-path / slow-path boundary. Asserts EXACT round-trip for the guaranteed domain (<=15 sig digits, |exp| <= 22) and pins <=1 ULP drift for known slow-path values. Any regression in the number parser trips these before it can reach production.
515
+ - **Torture-scale verification** -- `bench/torture.js` gained a `verify: true` opt-in that runs `verifyRoundTrip` on the assembled container AFTER the GC band closes. Twelve scenarios opted in cover **73,500 rows value-exact per fast-tier run**. The `verified` column in the torture report shows the row count actually checked; any failure blocks the run.
488
516
 
489
- ### Changed Tokenizer: Clinger's fast-path decimalF64 parser
517
+ ### Changed -- Tokenizer: Clinger's fast-path decimal->F64 parser
490
518
 
491
- The property-based fuzz surfaced a real preservation bug. The old naive parser computed `intPart + fracPart/fracDiv * Math.pow(10, exp)` three sequential floating-point ops, each rounding independently, plus `Math.pow(10, exp)` itself is not exact for many exponents. Result: values like `3.53` decoded to `3.5300000000000002` (1 ULP high). This violated the "same data after parse" invariant for arbitrary decimals, not just extreme edge cases.
519
+ The property-based fuzz surfaced a real preservation bug. The old naive parser computed `intPart + fracPart/fracDiv * Math.pow(10, exp)` -- three sequential floating-point ops, each rounding independently, plus `Math.pow(10, exp)` itself is not exact for many exponents. Result: values like `3.53` decoded to `3.5300000000000002` (1 ULP high). This violated the "same data after parse" invariant for arbitrary decimals, not just extreme edge cases.
492
520
 
493
521
  Fixed by implementing Clinger's fast-path algorithm:
494
- - Precomputed `POW10[0..22]` table with exact IEEE 754 doubles for `10^k` (built by iterated multiplication from 1.0 no rounding).
522
+ - Precomputed `POW10[0..22]` table with exact IEEE 754 doubles for `10^k` (built by iterated multiplication from 1.0 -- no rounding).
495
523
  - Track `_numFracDigits` and `_numDigitsSeen` counters during accumulation.
496
- - At emit, if `digitsSeen 15` AND `|netExp| 22`, assemble mantissa as an exact u53 integer and do a SINGLE multiply (positive exp) or SINGLE divide (negative exp) by `POW10[|netExp|]`. This is provably correctly-rounded within that domain.
497
- - Outside the fast path, fall back to the old naive computation (documented as 1 ULP drift).
524
+ - At emit, if `digitsSeen <= 15` AND `|netExp| <= 22`, assemble mantissa as an exact u53 integer and do a SINGLE multiply (positive exp) or SINGLE divide (negative exp) by `POW10[|netExp|]`. This is provably correctly-rounded within that domain.
525
+ - Outside the fast path, fall back to the old naive computation (documented as <=1 ULP drift).
498
526
 
499
527
  Impact:
500
- - **Fast-path domain covers >99% of realistic JSON numbers** (integer IDs, prices, ratios, timestamps, coordinates, scientific within `1e±22`).
501
- - **`long-numbers-200-digits` throughput: 139 195 MB/s** (+40%; fast path skips `Math.pow`).
528
+ - **Fast-path domain covers >99% of realistic JSON numbers** (integer IDs, prices, ratios, timestamps, coordinates, scientific within `1e+/-22`).
529
+ - **`long-numbers-200-digits` throughput: 139 -> 195 MB/s** (+40%; fast path skips `Math.pow`).
502
530
  - **No other throughput regression** in the fast tier (measured across 36 scenarios).
503
531
  - **Zero-GC gate held**: torture still passes with `maxMajor: 0` on every writer scenario.
504
532
 
505
- Correctly-rounded parsing across the FULL F64 domain (David Gay `strtod` or Ryu-style algorithm) is now on the M4+ backlog. It would eliminate the residual 1 ULP slow-path drift for values with 16+ significant digits or extreme exponents.
533
+ Correctly-rounded parsing across the FULL F64 domain (David Gay `strtod` or Ryu-style algorithm) is now on the M4+ backlog. It would eliminate the residual <=1 ULP slow-path drift for values with 16+ significant digits or extreme exponents.
506
534
 
507
- ### Added SPEC section 7: Round-trip preservation contract
535
+ ### Added -- SPEC section 7: Round-trip preservation contract
508
536
 
509
537
  Explicit, table-form contract stating what the format preserves and how. Covers F64 fast/slow paths, U32 byte-exact strings, coercions (`true`/`false`/`null`), missing-field defaults, unknown-key drop, wrong-type rejection. Backed by references to the specific test files that assert each guarantee. This is now the single source of truth consumers can point to when asked "will this preserve my data?".
510
538
 
@@ -512,27 +540,27 @@ Explicit, table-form contract stating what the format preserves and how. Covers
512
540
 
513
541
  36 scenarios, ~1.9s total wall time. **All 36 pass**; 12 opted into round-trip verification, covering 73,500 rows value-exact. Zero major GC, zero minor GC across every scenario, both ingestion and verification.
514
542
 
515
- ### Added M3 (String-table subpath)
543
+ ### Added -- M3 (String-table subpath)
516
544
 
517
- - `src/StringTable.js` byte-level UTF-8 string interning. Open-addressed hash table (Uint32Array) with linear probing, FNV-1a hash filter, byte-compare on hit. Zero-alloc hit path (no JS string materialized during intern). Growing blob (Uint8Array, 2× doublings) and offsets (Uint32Array, 2× doublings). Serializable to the LBK1 per-shard string-table layout with a trailing sentinel offset so `len(i) = offsets[i+1] - offsets[i]` is branch-free for the reader. `StringTable.parse` returns a zero-copy `StringTableView` for the Reader path.
518
- - `src/Writer.js` String value support end-to-end. Sample-and-infer now tracks per-field observed kind (unknown / number / string / mixed); fields that saw only numbers become F64 lanes, only strings become U32 lanes, mixed fields raise `W_MIXED_LANE_TYPES` at schema freeze. Row stride is variable based on lane sizes (F64=8, U32=4, padded to 8-byte multiple). Explicit schema now accepts both the M2 back-compat form `fields: ['a', 'b']` (all F64) and the M3 form `fields: [{name:'a', laneKind:'f64'}, {name:'tag', laneKind:'u32'}]`. Per-shard string table lifecycle: sample's string table becomes shard 0's local table on drain, then resets in place for subsequent shards (per-shard independence for HTTP Range / worker-parallel readers).
519
- - `src/Reader.js` Parses per-shard string tables via `StringTable.parse`, exposes `get(rowIdx, fieldName)` that resolves F64 or U32string transparently. New API surface: `shardPayload` / `strideBytes` / `offsetBytes` / `laneKind` / `shardStringTable`. Back-compat `shardF64` / `strideF64` / `offsetF64` still work for pure-F64 schemas.
520
- - `test/StringRoundTrip.test.js` 15 M3 acceptance tests: StringTable unit sanity (dedup, serialize round-trip, hash resize preserves entries), sample-and-infer (pure string U32 lane, mixed numeric+string schemas, mixed-lane rejection), explicit schema (U32 lanes, unknown laneKind rejection, cross-lane value rejection), string dedup verification (low-cardinality, unicode byte-exact, JSON escapes, empty strings), multi-shard string tables (each shard self-contained).
521
- - `bench/_string-torture-fixtures.js` string workload fixtures: low-cardinality tags, high-cardinality slugs (all unique), realistic mixed (low-card + high-card + numeric), unicode across 1/2/3/4-byte codepoints, identical-string repeats, 200-byte long strings, empty strings, mixed-lane pathological.
545
+ - `src/StringTable.js` -- byte-level UTF-8 string interning. Open-addressed hash table (Uint32Array) with linear probing, FNV-1a hash filter, byte-compare on hit. Zero-alloc hit path (no JS string materialized during intern). Growing blob (Uint8Array, 2× doublings) and offsets (Uint32Array, 2× doublings). Serializable to the LBK1 per-shard string-table layout with a trailing sentinel offset so `len(i) = offsets[i+1] - offsets[i]` is branch-free for the reader. `StringTable.parse` returns a zero-copy `StringTableView` for the Reader path.
546
+ - `src/Writer.js` -- String value support end-to-end. Sample-and-infer now tracks per-field observed kind (unknown / number / string / mixed); fields that saw only numbers become F64 lanes, only strings become U32 lanes, mixed fields raise `W_MIXED_LANE_TYPES` at schema freeze. Row stride is variable based on lane sizes (F64=8, U32=4, padded to 8-byte multiple). Explicit schema now accepts both the M2 back-compat form `fields: ['a', 'b']` (all F64) and the M3 form `fields: [{name:'a', laneKind:'f64'}, {name:'tag', laneKind:'u32'}]`. Per-shard string table lifecycle: sample's string table becomes shard 0's local table on drain, then resets in place for subsequent shards (per-shard independence for HTTP Range / worker-parallel readers).
547
+ - `src/Reader.js` -- Parses per-shard string tables via `StringTable.parse`, exposes `get(rowIdx, fieldName)` that resolves F64 or U32->string transparently. New API surface: `shardPayload` / `strideBytes` / `offsetBytes` / `laneKind` / `shardStringTable`. Back-compat `shardF64` / `strideF64` / `offsetF64` still work for pure-F64 schemas.
548
+ - `test/StringRoundTrip.test.js` -- 15 M3 acceptance tests: StringTable unit sanity (dedup, serialize round-trip, hash resize preserves entries), sample-and-infer (pure string -> U32 lane, mixed numeric+string schemas, mixed-lane rejection), explicit schema (U32 lanes, unknown laneKind rejection, cross-lane value rejection), string dedup verification (low-cardinality, unicode byte-exact, JSON escapes, empty strings), multi-shard string tables (each shard self-contained).
549
+ - `bench/_string-torture-fixtures.js` -- string workload fixtures: low-cardinality tags, high-cardinality slugs (all unique), realistic mixed (low-card + high-card + numeric), unicode across 1/2/3/4-byte codepoints, identical-string repeats, 200-byte long strings, empty strings, mixed-lane pathological.
522
550
  - 8 new M3 torture scenarios in `bench/torture.js`. Fast tier grew from 28 to 36 scenarios; total wall time 1.6s. **All 36 pass with `maxMajor: 0`**; the hottest scenario (20K identical-string interns) allocates only 100 KB across the entire pipeline.
523
551
  - `/string-table` subpath export in `package.json`.
524
552
 
525
553
  ### Design decisions locked in M3
526
554
 
527
- - **Per-shard string tables** (not container-level) as the v1 default. Rationale: streaming ingestion writes shards independently the container-level string-table location cannot be filled until the last shard is known, which would defeat the writer's shard-first architecture. Per-shard tables also enable single-Range HTTP fetches to include a shard's string data alongside its payload, which is the M5 story. Container-level tables remain a possibility for M4+ wrap-up.
555
+ - **Per-shard string tables** (not container-level) as the v1 default. Rationale: streaming ingestion writes shards independently -- the container-level string-table location cannot be filled until the last shard is known, which would defeat the writer's shard-first architecture. Per-shard tables also enable single-Range HTTP fetches to include a shard's string data alongside its payload, which is the M5 story. Container-level tables remain a possibility for M4+ wrap-up.
528
556
  - **F64 vs U32 lane inference** is by first-observed value type. A field seen with only numbers becomes F64; only strings becomes U32; mixed fails at schema freeze. Explicit schema mode always wins; consumers who need `"0"` (string) vs `0` (number) discrimination must declare the schema.
529
- - **Missing string fields default to U32 index 0**, which decodes to whatever string was interned first in that shard. Documented as intentional; not a bug. Consumers who need "field is absent" semantics should coerce absent sentinel string in the source data.
557
+ - **Missing string fields default to U32 index 0**, which decodes to whatever string was interned first in that shard. Documented as intentional; not a bug. Consumers who need "field is absent" semantics should coerce absent -> sentinel string in the source data.
530
558
  - **Per-shard reset semantics**: the sample's string table becomes shard 0's. When shard 0 finalizes, the table is serialized (into a caller-owned byte copy) and then reset in place. Subsequent shards fill the same table object with fresh entries. This keeps peak memory bounded to the LARGEST shard's string cardinality, not the container total.
531
559
 
532
560
  ### Bugs found and fixed in M3
533
561
 
534
562
  - **`EMPTY_SLOT` sign bug in StringTable**: the initial sentinel was `0xFFFFFFFF | 0` which evaluates to `-1` (signed), but `Uint32Array[i]` reads back as `4294967295` (unsigned). The strict-equal check never matched, causing infinite linear probing on the first `intern()`. Fix: `EMPTY_SLOT = 0xFFFFFFFF` (unsigned). Classic JS foot-gun; documented inline in `StringTable.js`.
535
- - **Ephemeral-buffer overlap in Writer.onKey (sample mode)**: the sample-window path stored a reference to the tokenizer's ephemeral `_strBuf` byte range for the key, then decoded it later in `onNumber` / `onString`. But `_strBuf` is reused for the VALUE emission between the two calls (documented in SPEC 5.3), so by the time the decode ran, the key bytes had been overwritten. Symptom: field name `"tag"` decoded to `"aag"` because the value `"a"` had landed at position 0. Fix: decode the key to a JS string immediately in `onKey`, in the same call. String allocation here is expected bounded to the sample window; steady-state post-freeze remains zero-alloc. Post-schema-frozen path was always correct (resolved to field-index immediately).
563
+ - **Ephemeral-buffer overlap in Writer.onKey (sample mode)**: the sample-window path stored a reference to the tokenizer's ephemeral `_strBuf` byte range for the key, then decoded it later in `onNumber` / `onString`. But `_strBuf` is reused for the VALUE emission between the two calls (documented in SPEC 5.3), so by the time the decode ran, the key bytes had been overwritten. Symptom: field name `"tag"` decoded to `"aag"` because the value `"a"` had landed at position 0. Fix: decode the key to a JS string immediately in `onKey`, in the same call. String allocation here is expected -- bounded to the sample window; steady-state post-freeze remains zero-alloc. Post-schema-frozen path was always correct (resolved to field-index immediately).
536
564
 
537
565
  ### Baseline results (M3 fast tier)
538
566
 
@@ -550,7 +578,7 @@ The identical-20k scenario is the tightest possible interning gate: every intern
550
578
 
551
579
  ### Breaking changes vs M2 error taxonomy
552
580
 
553
- - `W_STRING_VALUE_UNSUPPORTED` is REMOVED. Strings are now supported. Sending a string to a field declared as F64 (explicit or inferred) now raises `W_LANE_MISMATCH` a more precise error that also fires in the opposite direction (number to a U32 lane).
581
+ - `W_STRING_VALUE_UNSUPPORTED` is REMOVED. Strings are now supported. Sending a string to a field declared as F64 (explicit or inferred) now raises `W_LANE_MISMATCH` -- a more precise error that also fires in the opposite direction (number to a U32 lane).
554
582
  - New error codes: `W_MIXED_LANE_TYPES` (sample-and-infer saw both types in the same field), `W_LANE_MISMATCH` (post-freeze type conflict), `W_UNKNOWN_LANE_KIND` (explicit schema declared a laneKind that isn't `'f64'` or `'u32'`).
555
583
 
556
584
  ### Deferred to M4
@@ -559,12 +587,12 @@ The identical-20k scenario is the tightest possible interning gate: every intern
559
587
  - Multi-container reader (join multiple LBK1 files).
560
588
  - Container-level string table (opposite of per-shard: single dedup across all shards, requires writer to buffer all shards until final drain). Suitable for offline transcoding pipelines.
561
589
 
562
- ### Added M2 Session B (Writer + Reader + Round-trip)
590
+ ### Added -- M2 Session B (Writer + Reader + Round-trip)
563
591
 
564
- - `src/Writer.js` LBK1 shard emitter, Tokenizer sink implementation. Explicit-schema and sample-and-infer modes. F64-only lanes (v1). 32 MB fixed shard boundaries (configurable via `targetShardBytes`). Zero-alloc record commit post-schema-freeze via byte-compare field lookup with FNV-1a filter.
565
- - `src/Reader.js` LBK1 container parser. Header + schema + shard directory parse. Per-shard `Float64Array` views. Hot-loop-friendly `shardF64` / `strideF64` / `offsetF64` for direct typed-array indexing. Strict rejection of non-zero field flags and non-F64 lanes (v2 will accept extended lanes).
566
- - `test/RoundTrip.test.js` end-to-end acceptance suite: bytes tokenize write LBK1 read verify. 16 tests covering sample-and-infer, explicit schema, multi-shard, F64 range boundaries, boolean/null coercion, container structural checks, and reader hot-loop helpers.
567
- - `bench/_writer-torture-fixtures.js` writer-specific adversarial fixtures: homogeneous records, late field appearance, missing fields, F64 range boundary walk, multi-shard boundary crossing, top-level-number rejection, nested-object/array rejection, string-value rejection, empty records.
592
+ - `src/Writer.js` -- LBK1 shard emitter, Tokenizer sink implementation. Explicit-schema and sample-and-infer modes. F64-only lanes (v1). 32 MB fixed shard boundaries (configurable via `targetShardBytes`). Zero-alloc record commit post-schema-freeze via byte-compare field lookup with FNV-1a filter.
593
+ - `src/Reader.js` -- LBK1 container parser. Header + schema + shard directory parse. Per-shard `Float64Array` views. Hot-loop-friendly `shardF64` / `strideF64` / `offsetF64` for direct typed-array indexing. Strict rejection of non-zero field flags and non-F64 lanes (v2 will accept extended lanes).
594
+ - `test/RoundTrip.test.js` -- end-to-end acceptance suite: bytes -> tokenize -> write -> LBK1 -> read -> verify. 16 tests covering sample-and-infer, explicit schema, multi-shard, F64 range boundaries, boolean/null coercion, container structural checks, and reader hot-loop helpers.
595
+ - `bench/_writer-torture-fixtures.js` -- writer-specific adversarial fixtures: homogeneous records, late field appearance, missing fields, F64 range boundary walk, multi-shard boundary crossing, top-level-number rejection, nested-object/array rejection, string-value rejection, empty records.
568
596
  - 10 new writer torture scenarios in `bench/torture.js`, extending the mechanical zero-GC gate to the full ingestion pipeline. Total fast tier: 28 scenarios, all pass with `maxMajor: 0` and `maxMinor: 0`.
569
597
  - `/writer` and `/reader` subpath exports in `package.json`, preserving the tree-shakeable single-file-per-entry convention.
570
598
 
@@ -574,7 +602,7 @@ The identical-20k scenario is the tightest possible interning gate: every intern
574
602
  - **Field lookup** uses an FNV-1a hash filter over interned name buffers, followed by a byte-compare on hash match. This gives O(N) worst case with N = field count (typically 5-20) but zero string allocation per `onKey` call.
575
603
  - **Unknown keys** (fields not in the frozen schema) are silently ignored, matching `lite-bake` core behavior.
576
604
  - **Missing fields** default-fill to 0 on the shard row, matching `lite-bake`.
577
- - **Booleans and null** coerce to F64: `true 1`, `false 0`, `null 0`. Documented; predictable.
605
+ - **Booleans and null** coerce to F64: `true -> 1`, `false -> 0`, `null -> 0`. Documented; predictable.
578
606
  - **Nested objects/arrays and string values** are rejected with distinct writer error codes (`W_NESTED_UNSUPPORTED`, `W_STRING_VALUE_UNSUPPORTED`). String values return in M3 via the string-table subpath.
579
607
 
580
608
  ### Baseline results (Session B fast tier)
@@ -587,27 +615,27 @@ The identical-20k scenario is the tightest possible interning gate: every intern
587
615
  - Shard payload micro-optimization: keep the pre-allocated 32 MB ArrayBuffer instead of copying used bytes into a right-sized `Uint8Array` at shard finalization. Eliminates one allocation per shard boundary.
588
616
  - Reader HTTP Range support (`@zakkster/lite-bake-stream/reader-http`), for browser consumers.
589
617
 
590
- ### Added M2 Session A (torture harness)
618
+ ### Added -- M2 Session A (torture harness)
591
619
 
592
- - `bench/torture.js` CI-ready adversarial-input harness with fast (~1s) and full (~5min) tiers. Every scenario runs under `@zakkster/lite-gc-profiler` with `maxMajor: 0` as the mechanical release gate.
593
- - `bench/_torture-fixtures.js` deterministic, seeded fixture generators covering: pathological nesting (up to and past MAX_DEPTH), adversarial escapes/unicode/surrogates, buffer-growth boundary strings, F64-limit numbers, 200-digit numbers, malformed inputs (unterminated string/array, invalid escape, unclosed-array attack), and multi-strategy chunk arrival (whole/fixed/random/single-byte).
620
+ - `bench/torture.js` -- CI-ready adversarial-input harness with fast (~1s) and full (~5min) tiers. Every scenario runs under `@zakkster/lite-gc-profiler` with `maxMajor: 0` as the mechanical release gate.
621
+ - `bench/_torture-fixtures.js` -- deterministic, seeded fixture generators covering: pathological nesting (up to and past MAX_DEPTH), adversarial escapes/unicode/surrogates, buffer-growth boundary strings, F64-limit numbers, 200-digit numbers, malformed inputs (unterminated string/array, invalid escape, unclosed-array attack), and multi-strategy chunk arrival (whole/fixed/random/single-byte).
594
622
  - `npm run torture` and `npm run torture:full` scripts.
595
623
  - `@zakkster/lite-gc-profiler` as a devDependency.
596
624
 
597
- ### Changed M1.1 (driven by torture-harness findings)
625
+ ### Changed -- M1.1 (driven by torture-harness findings)
598
626
 
599
- - `Tokenizer._appendStrRange` removed `Uint8Array.set(chunk.subarray(from, to), ...)` in favor of a manual byte-copy loop. The `subarray` was allocating a small view header (~50 bytes) per contiguous string byte run; on string-heavy fixtures this translated to megabytes of GC pressure. Measured impact on the fast tier: 12× reduction in escape-cycle-1mb heap alloc, 630× reduction in unicode-cycle-1mb, 30× reduction in 4 MB chunked ingestion; throughput up 67% (88147 MB/s) on chunked NDJSON.
627
+ - `Tokenizer._appendStrRange` -- removed `Uint8Array.set(chunk.subarray(from, to), ...)` in favor of a manual byte-copy loop. The `subarray` was allocating a small view header (~50 bytes) per contiguous string byte run; on string-heavy fixtures this translated to megabytes of GC pressure. Measured impact on the fast tier: 12× reduction in escape-cycle-1mb heap alloc, 630× reduction in unicode-cycle-1mb, 30× reduction in 4 MB chunked ingestion; throughput up 67% (88->147 MB/s) on chunked NDJSON.
600
628
 
601
629
  ### Baseline results (Session A)
602
630
 
603
631
  Fast tier: 18 scenarios, 575 ms total wall time, **all pass with maxMajor: 0 AND maxMinor: 0**. Soak scenario (500 MB clean NDJSON): 124 MB/s sustained, 56 KB heap growth across the entire run (0.011% overhead). Mechanical zero-GC on gigabyte-scale ingestion is now proven by CI, not just claimed.
604
632
 
605
- ### Added M0 / M1
633
+ ### Added -- M0 / M1
606
634
 
607
- - `SPEC.md` LBK1 container format draft 0.1. Header, schema block with reserved forward-compat flags, string-table segment, shard directory with `min_reader_version`, container footer, extended-lane table reserving `I64_BYTES_PRESERVED` / `U64_BYTES_PRESERVED`.
608
- - `src/Tokenizer.js` chunk-safe UTF-8 JSON SAX scanner. `auto` / `array` / `ndjson` framing. Byte-level number parsing. All standard escapes, surrogate pairs, error codes with absolute `byteOffset`.
609
- - `test/Tokenizer.test.js` 33 conformance tests including byte-split boundary fuzz across four fixtures and single-byte-chunk streaming.
610
- - `bench/bench-tokenizer.js` 10 interleaved reps, 3 warmups, min-of-reps headline. Baselines against `JSON.parse` per-line and wrapped-array.
635
+ - `SPEC.md` -- LBK1 container format draft 0.1. Header, schema block with reserved forward-compat flags, string-table segment, shard directory with `min_reader_version`, container footer, extended-lane table reserving `I64_BYTES_PRESERVED` / `U64_BYTES_PRESERVED`.
636
+ - `src/Tokenizer.js` -- chunk-safe UTF-8 JSON SAX scanner. `auto` / `array` / `ndjson` framing. Byte-level number parsing. All standard escapes, surrogate pairs, error codes with absolute `byteOffset`.
637
+ - `test/Tokenizer.test.js` -- 33 conformance tests including byte-split boundary fuzz across four fixtures and single-byte-chunk streaming.
638
+ - `bench/bench-tokenizer.js` -- 10 interleaved reps, 3 warmups, min-of-reps headline. Baselines against `JSON.parse` per-line and wrapped-array.
611
639
  - Package exports map with `sideEffects: false` and single subpath `/tokenizer`.
612
640
 
613
641
  ### Notes
@@ -617,25 +645,25 @@ Fast tier: 18 scenarios, 575 ms total wall time, **all pass with maxMajor: 0 AND
617
645
  - The `chunk-single-byte-64kb` scenario shows ~6 MB heap growth: this is caller-side, not tokenizer-side (the harness's `feedSingleByte` allocates 64k subarray views to synthesize the pathological chunking). Documented as expected; not a target for the mechanical gate.
618
646
 
619
647
 
620
- - `bench/torture.js` CI-ready adversarial-input harness with fast (~1s) and full (~5min) tiers. Every scenario runs under `@zakkster/lite-gc-profiler` with `maxMajor: 0` as the mechanical release gate.
621
- - `bench/_torture-fixtures.js` deterministic, seeded fixture generators covering: pathological nesting (up to and past MAX_DEPTH), adversarial escapes/unicode/surrogates, buffer-growth boundary strings, F64-limit numbers, 200-digit numbers, malformed inputs (unterminated string/array, invalid escape, unclosed-array attack), and multi-strategy chunk arrival (whole/fixed/random/single-byte).
648
+ - `bench/torture.js` -- CI-ready adversarial-input harness with fast (~1s) and full (~5min) tiers. Every scenario runs under `@zakkster/lite-gc-profiler` with `maxMajor: 0` as the mechanical release gate.
649
+ - `bench/_torture-fixtures.js` -- deterministic, seeded fixture generators covering: pathological nesting (up to and past MAX_DEPTH), adversarial escapes/unicode/surrogates, buffer-growth boundary strings, F64-limit numbers, 200-digit numbers, malformed inputs (unterminated string/array, invalid escape, unclosed-array attack), and multi-strategy chunk arrival (whole/fixed/random/single-byte).
622
650
  - `npm run torture` and `npm run torture:full` scripts.
623
651
  - `@zakkster/lite-gc-profiler` as a devDependency.
624
652
 
625
- ### Changed M1.1 (driven by torture-harness findings)
653
+ ### Changed -- M1.1 (driven by torture-harness findings)
626
654
 
627
- - `Tokenizer._appendStrRange` removed `Uint8Array.set(chunk.subarray(from, to), ...)` in favor of a manual byte-copy loop. The `subarray` was allocating a small view header (~50 bytes) per contiguous string byte run; on string-heavy fixtures this translated to megabytes of GC pressure. Measured impact on the fast tier: 12× reduction in escape-cycle-1mb heap alloc, 630× reduction in unicode-cycle-1mb, 30× reduction in 4 MB chunked ingestion; throughput up 67% (88147 MB/s) on chunked NDJSON.
655
+ - `Tokenizer._appendStrRange` -- removed `Uint8Array.set(chunk.subarray(from, to), ...)` in favor of a manual byte-copy loop. The `subarray` was allocating a small view header (~50 bytes) per contiguous string byte run; on string-heavy fixtures this translated to megabytes of GC pressure. Measured impact on the fast tier: 12× reduction in escape-cycle-1mb heap alloc, 630× reduction in unicode-cycle-1mb, 30× reduction in 4 MB chunked ingestion; throughput up 67% (88->147 MB/s) on chunked NDJSON.
628
656
 
629
657
  ### Baseline results (Session A)
630
658
 
631
659
  Fast tier: 18 scenarios, 575 ms total wall time, **all pass with maxMajor: 0 AND maxMinor: 0**. Soak scenario (500 MB clean NDJSON): 124 MB/s sustained, 56 KB heap growth across the entire run (0.011% overhead). Mechanical zero-GC on gigabyte-scale ingestion is now proven by CI, not just claimed.
632
660
 
633
- ### Added M0 / M1
661
+ ### Added -- M0 / M1
634
662
 
635
- - `SPEC.md` LBK1 container format draft 0.1. Header, schema block with reserved forward-compat flags, string-table segment, shard directory with `min_reader_version`, container footer, extended-lane table reserving `I64_BYTES_PRESERVED` / `U64_BYTES_PRESERVED`.
636
- - `src/Tokenizer.js` chunk-safe UTF-8 JSON SAX scanner. `auto` / `array` / `ndjson` framing. Byte-level number parsing. All standard escapes, surrogate pairs, error codes with absolute `byteOffset`.
637
- - `test/Tokenizer.test.js` 33 conformance tests including byte-split boundary fuzz across four fixtures and single-byte-chunk streaming.
638
- - `bench/bench-tokenizer.js` 10 interleaved reps, 3 warmups, min-of-reps headline. Baselines against `JSON.parse` per-line and wrapped-array.
663
+ - `SPEC.md` -- LBK1 container format draft 0.1. Header, schema block with reserved forward-compat flags, string-table segment, shard directory with `min_reader_version`, container footer, extended-lane table reserving `I64_BYTES_PRESERVED` / `U64_BYTES_PRESERVED`.
664
+ - `src/Tokenizer.js` -- chunk-safe UTF-8 JSON SAX scanner. `auto` / `array` / `ndjson` framing. Byte-level number parsing. All standard escapes, surrogate pairs, error codes with absolute `byteOffset`.
665
+ - `test/Tokenizer.test.js` -- 33 conformance tests including byte-split boundary fuzz across four fixtures and single-byte-chunk streaming.
666
+ - `bench/bench-tokenizer.js` -- 10 interleaved reps, 3 warmups, min-of-reps headline. Baselines against `JSON.parse` per-line and wrapped-array.
639
667
  - Package exports map with `sideEffects: false` and single subpath `/tokenizer`.
640
668
 
641
669
  ### Notes