@zakkster/lite-bake-stream 1.0.0

package/README.md

@@ -0,0 +1,150 @@
+# @zakkster/lite-bake-stream
+
+[![npm version](https://img.shields.io/npm/v/@zakkster/lite-bake-stream.svg?style=for-the-badge&color=latest)](https://www.npmjs.com/package/@zakkster/lite-bake-stream)
+[![sponsor](https://img.shields.io/badge/sponsor-PeshoVurtoleta-ea4aaa.svg?logo=github)](https://github.com/sponsors/PeshoVurtoleta)
+![Zero-GC](https://img.shields.io/badge/Zero--GC-Engine-00C853?style=for-the-badge&logo=leaf&logoColor=white)
+[![npm bundle size](https://img.shields.io/bundlephobia/minzip/@zakkster/lite-bake-stream?style=for-the-badge)](https://bundlephobia.com/result?p=@zakkster/lite-bake-stream)
+[![npm downloads](https://img.shields.io/npm/dm/@zakkster/lite-bake-stream?style=for-the-badge&color=blue)](https://www.npmjs.com/package/@zakkster/lite-bake-stream)
+[![npm total downloads](https://img.shields.io/npm/dt/@zakkster/lite-bake-stream?style=for-the-badge&color=blue)](https://www.npmjs.com/package/@zakkster/lite-bake-stream)
+![Tree-Shakeable](https://img.shields.io/badge/tree--shakeable-yes-brightgreen)
+![TypeScript](https://img.shields.io/badge/TypeScript-Types-informational)
+![Dependencies](https://img.shields.io/badge/dependencies-0-brightgreen)
+[![license](https://img.shields.io/badge/license-MIT-blue?style=flat-square)](./LICENSE)
+
+> Streaming byte-level JSON compiler for [`@zakkster/lite-bake`](https://github.com/PeshoVurtoleta/lite-bake). Zero-GC, tree-shakeable, gigabyte-scale.
+
+**Status:** v1.0.0. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.
+
+## Two modes, one API
+
+- **Schema mode** (default): F64/U32 lane packing, zone-maps query pruning, byte-exact preservation for flat records. Millions of rows/sec random access, zero-GC hot path.
+- **Preserve mode**: `serialize(input, { preserve: true })`. Opaque byte blobs — the module doesn't crack open records, it just moves them intact. Deeply nested API JSON, arrays, mixed types — any valid JSON round-trips byte-for-byte identical. Reader exposes `getBytes(i)` (zero-alloc), `getString(i)`, `getJSON(i)`.
+
+`deserialize()` auto-dispatches on the container's flag bit. If you're compiling gigabytes of API responses with 2-3 levels of nesting, preserve mode is what you want. If the data is flat and you want columnar-style range queries, schema mode is what you want.
+
+## Why this exists
+
+`lite-bake` compiles a JS array into a flat binary buffer. That's the destination format you want. But if your input is a gigabyte of JSON, you can't call `JSON.parse` on it — V8's string limit is ~512 MB, and even under that, the intermediate object graph will trash the heap and destroy your process before `bake()` ever runs.
+
+`lite-bake-stream` is the front door: a chunk-safe UTF-8 JSON SAX scanner that reads bytes off a stream and emits `lite-bake`-shaped output shards, without ever materializing the object graph.
+
+## Qualified at 8 GB
+
+Overnight soak, M1 MacBook Pro, both release gates armed:
+
+| | |
+| :-- | --: |
+| Source | 8.00 GB NDJSON |
+| Container | 4.89 GB (61.1% of source) |
+| Rows | 98,367,702 |
+| Ingest throughput | 110.0 MB/s |
+| **Major GC** | **0** |
+| **Minor GC** | **0** |
+| **Total heap allocation** | **499.2 KB** |
+| Cells verified | 590,206,212 |
+| **Preservation mismatches** | **0** |
+
+**499 KB of heap allocation to compile 8 GB of JSON.** Total, for the whole run — not per shard, not per second. Zero garbage collections of any kind. Every one of 590 million declared cells round-tripped byte-exact.
+
+Tokenizer throughput is ~55% of `JSON.parse` (237 MB/s vs 416 MB/s on a 50 MB fixture). That's the trade: you give up some raw speed and you get an input size ceiling that's bounded by your disk instead of your heap.
+
+## Quickstart
+
+**Schema mode** (flat records, columnar queries):
+
+```js
+import { serialize, deserialize } from '@zakkster/lite-bake-stream';
+
+const container = serialize(ndjsonBytes);
+const reader = deserialize(container);
+reader.get(0, 'id');                     // 1
+reader.findShards('id', { min: 500 });   // shard pruning via zone maps
+```
+
+**Preserve mode** (any JSON shape, byte-exact passthrough):
+
+```js
+import { serialize, deserialize } from '@zakkster/lite-bake-stream';
+
+// Handles arbitrarily nested API JSON
+const container = serialize(apiResponseBytes, { preserve: true });
+const reader = deserialize(container);
+reader.getBytes(0);    // Uint8Array view, zero allocation
+reader.getString(0);   // TextDecoder allocation
+reader.getJSON(0);     // full JSON.parse, allocates the parsed value graph
+```
+
+For streaming ingest with progress, use `/file-ingest`. For HTTP Range-fetched containers (schema mode), `/range-reader`. For joining multiple containers into one logical view (schema mode), `/multi-reader`. For parallel or checkpointed compilation (schema mode), `/split`.
+
+## What shipped in 1.0
+
+| Milestone | Deliverable                                                                 |
+| :-------- | :-------------------------------------------------------------------------- |
+| ✅ **M0** | LBK1 container format spec, forward-compat seams for the I64 lane.          |
+| ✅ **M1** | Chunk-safe tokenizer; zero-GC on the hot path after warmup.                 |
+| ✅ **M2** | Schema-driven writer; sample-and-infer path; typed-lane shard buffers.      |
+| ✅ **M3** | Per-shard UTF-8 string table with pre-alloc typed-array hash-map interning. |
+| ✅ **M4** | `serialize()`/`deserialize()`, MultiReader, full type declarations.         |
+| ✅ **M5** | Browser reader, HTTP Range fetch, `File.stream()` client-side ingest.       |
+| ✅ **M6** | NDJSON splitter + per-part compilation + container merge.                   |
+| ✅ **M7** | Per-shard zone maps (min/max per F64 column) + query-pruning APIs.          |
+| ✅ **M8** | RFC 8259 conformance corpus + robustness fuzz.                              |
+
+## Roadmap beyond 1.0
+
+All of these fit the format's existing forward-compat seams — `format_version` stays at `1`.
+
+| | |
+| :-- | :-- |
+| **I64 lane** | Exact 64-bit integer preservation. FieldDescriptor flags are already reserved for it; shards using it set `min_reader_version: 2`. |
+| **Columnar payload mode** | Column-contiguous shard payloads instead of interleaved rows. Better compression, SIMD-friendly scans. Opt-in via a shard flag. |
+| **Container-level string table** | Cross-shard interning for high-repetition corpora. Lands in the `metadata_off` block alongside zone maps. |
+| **Oscilloscope-rack demo** | Reel-to-reel tape transport at actual MB/s, waterfall spectrum per shard, VU needles, seven-segment counters. Locked 60fps on an iPhone 7 with 100 MB NDJSON. |
+
+## What's here today
+
+- `SPEC.md` — LBK1 container spec draft 0.1, per-shard string table (§3.3), preservation contract (§7).
+- `src/index.js` — top-level convenience API: `serialize(input)` and `deserialize(bytes)` plus class re-exports.
+- `src/Tokenizer.js` — chunk-safe SAX tokenizer with Clinger's fast-path decimal→F64 conversion.
+- `src/Writer.js` — LBK1 shard emitter with F64 and U32 lanes, sample-and-infer schema, per-shard string tables, per-shard zone maps.
+- `src/Reader.js` — LBK1 container parser with typed-array views, transparent string-field resolution, and zone-map query APIs.
+- `src/StringTable.js` — open-addressed byte-level interning with zero-alloc hit path.
+- `src/RangeReader.js` — LBK1 reader with lazy shard loading via an IO adapter (HTTP Range or in-memory mock); zone maps loaded once at open enable synchronous query pruning.
+- `src/MultiReader.js` — union view over N Reader instances with matching schemas; merged shard indices for query pruning across containers.
+- `src/Split.js` — NDJSON splitter, per-part compilation, and container merge for worker-parallel or checkpointed ingest.
+- `src/FileIngest.js` — browser helper: `File.stream()` → Reader. Any `ReadableStream<Uint8Array>` works.
+- `types/*.d.ts` — 8 hand-written TypeScript declaration files, one per subpath.
+- `demo/` — single-file HTML + module JS. Ingest a JSON file → LBK1 → browse in `@zakkster/lite-table`. Zone-maps query-pruning panel visualizes the fetch-set reduction. `npm run demo`.
+- `test/*.test.js` — 233 tests across 12 files, including a 66-fixture RFC 8259 conformance corpus and a 7-scenario robustness fuzz (~2,300 tokenizer invocations per run).
+- `test/_verify.js` — universal round-trip verifier used by tests, torture, and soak.
+- `bench/bench-tokenizer.js` — interleaved-rep bench vs `JSON.parse`.
+- `bench/torture.js` — 37-scenario CI harness. Two release gates: `maxMajor: 0` (zero-GC) AND `verify: true` (value-exact round-trip on 77,500+ rows per run).
+- `bench/soak.js` — CLI-configurable soak with the preservation gate wired in.
+
+Two release gates. Zero major GC AND every declared cell round-trips exactly. If either fails, no publish.
+
+## Release testing tiers
+
+The gates run at four scales, chosen to fit different hardware and time budgets:
+
+| Command | Scale | Use |
+| :-- | :-- | :-- |
+| `npm test` | 96 tests | Dev loop, every save |
+| `npm run torture` | 37 scenarios (~1 MB each) | Before every commit |
+| `npm run torture:full` | + 500 MB full-tier soak scenarios | Before every publish |
+| `npm run soak` | 100 MB with preservation gate | Sanity check |
+| `npm run soak:500` | 500 MB | Pre-publish scale check |
+| `npm run soak:1gb` | 1 GB | Real-hardware qualification (M4 Pro etc.) |
+| `npm run soak:overnight` | 8 GB | Overnight `caffeinate` run |
+
+Custom sizes:
+```
+node --expose-gc bench/soak.js --gb=4 --verify
+caffeinate node --expose-gc bench/soak.js --gb=16 --verify   # macOS overnight
+```
+
+Every soak run reports ingest throughput (MB/s), verification throughput (rows/s), the zero-GC gate result, container compression ratio, and progress lines so long runs are observable. The preservation gate proves — cell by cell — that every declared field of every row round-trips exactly.
+
+## License
+
+MIT © 2026 Zahary Shinikchiev

package/SPEC.md

@@ -0,0 +1,364 @@
+# LBK1 Format Specification
+
+> Version 1 — frozen as of `@zakkster/lite-bake-stream` v1.0.0.
+> Future lane kinds and payload modes land via the forward-compat seams
+> (`min_reader_version`, reserved FieldDescriptor flags, the `metadata_off`
+> block wrapper) without a `format_version` bump.
+> Copyright (c) 2026 Zahary Shinikchiev. MIT.
+
+## 0. Scope
+
+LBK1 is the on-disk / on-wire container for streaming-compiled `lite-bake` datasets. It carries one or more shards of interleaved binary rows plus the metadata needed to reconstruct a Reader over the whole dataset without re-parsing JSON.
+
+The tokenizer specified in section 5 is the reference producer path from UTF-8 JSON (top-level array or NDJSON) into an LBK1 container. Non-JSON producers are allowed as long as they emit a well-formed container.
+
+Numbers on the hot path are F64. Section 4.3 reserves the schema-level bit that will admit a future `I64_BYTES_PRESERVED` lane without a container format-version bump.
+
+## 1. Design invariants
+
+These are load-bearing; every other decision follows from them.
+
+1. **Streaming-first.** A producer must be able to emit a valid, seekable container without buffering the entire input. Readers must be able to consume shard N without having seen shards 0..N-1.
+2. **Forward-compatible via strict decoders.** Unknown flags or lane kinds are rejected, not ignored. Silent corruption is the failure mode we refuse.
+3. **Shard-scoped versioning.** A shard header carries the minimum reader version required to decode that shard. Old readers refuse new shards cleanly; new readers read old shards forever.
+4. **Zero-copy where the CPU allows it.** Row payloads are aligned so that typed-array views over the shard buffer read fields with native alignment.
+5. **Zero runtime dependencies.** Reference producer and reader share the `@zakkster` no-dep constraint.
+
+## 2. Notation
+
+- `u8`, `u16`, `u32`, `u64`, `f32`, `f64` — little-endian unless stated.
+- `bytes(n)` — a raw byte run of length n.
+- `varlen_utf8` — u32 length prefix followed by that many UTF-8 bytes.
+- Offsets are relative to the start of the container unless stated.
+- All multi-byte integers in headers are little-endian. Row payloads use native endianness with a byte-order marker in the container header (section 3.1); non-native containers require a byteswap on load or must be rejected.
+
+## 3. Container layout
+
+```
++---------------------+  offset 0
+| Container Header    |
++---------------------+
+| Schema Block        |
++---------------------+
+| String Table Segment| (optional; length 0 = absent)
++---------------------+
+| Shard Directory     |
++---------------------+
+| Shard 0 payload     |
+| Shard 1 payload     |
+| ...                 |
++---------------------+
+| Container Footer    |
++---------------------+
+```
+
+### 3.1 Container Header (48 bytes, fixed)
+
+| Offset | Size | Field                | Notes                                                     |
+| -----: | ---: | -------------------- | --------------------------------------------------------- |
+|      0 |    4 | `magic`              | ASCII `L B K 1` = `0x4C 0x42 0x4B 0x31`                   |
+|      4 |    2 | `format_version`     | u16. This spec = `1`.                                     |
+|      6 |    1 | `endian`             | `0x01` = little, `0x02` = big. Determines row payload byte order. |
+|      7 |    1 | `flags`              | bit 0 = **preserve mode** (opaque byte blobs, any JSON shape). bits 1-7 reserved, must be 0. |
+|      8 |    8 | `schema_block_off`   | u64 byte offset to schema block.                          |
+|     16 |    8 | `metadata_off`       | u64 byte offset to an optional metadata block (M7+). `0` = none. Currently carries zone maps only; wrapper block reserved for M4+ container-level string table extensions. |
+|     24 |    8 | `shard_directory_off`| u64.                                                      |
+|     32 |    4 | `shard_count`        | u32.                                                      |
+|     36 |    4 | `reserved1`          | Must be `0x00000000`.                                     |
+|     40 |    8 | `total_rows`         | u64. Sum of `row_count` across all shards.                |
+
+Header uses u64 offsets throughout so containers can safely exceed 4 GiB. A container that stores 8 GB of data across ~120 shards would have late-shard offsets past the u32 boundary; truncating to u32 would point the reader at arbitrary earlier bytes (regression covered by `test/OffsetOverflow.test.js`). Header stays 8-byte aligned; readers reject `reserved0`/`reserved1` non-zero.
+
+### 3.2 Schema Block
+
+```
+u32   field_count
+u32   row_stride_bytes       // padded to 8
+FieldDescriptor[field_count]
+```
+
+Padded with zero bytes so the block ends on an 8-byte boundary.
+
+### 3.3 String Table Segment
+
+There are two string-table locations in an LBK1 container:
+
+- **Container-level** at `string_table_off` (v1 reserved; not written by the current Writer). Reserved for future container-wide interning across shards.
+- **Per-shard** at each ShardEntry's `local_string_off` (M3, implemented). Every shard with U32-lane fields has its own local string table so shards remain independently decodable — critical for HTTP Range fetches (M5+) and worker-parallel readers (M6+).
+
+Both use the same on-disk layout:
+
+```
+u32   entry_count
+u32   blob_length_bytes
+u32   offsets[entry_count + 1]   // byte offsets into blob, INCLUDING a trailing sentinel = blob_length_bytes
+u8    blob[blob_length_bytes]
+```
+
+Padded to 8. Strings are UTF-8, not null-terminated. Length of string `i` is `offsets[i+1] - offsets[i]`; the trailing sentinel makes this valid for `i = entry_count - 1` without a branch. Entry index 0 is not reserved; the empty string is a legitimate entry.
+
+Per-shard tables are placed immediately after their shard payload in the container, so a single HTTP Range fetch covering `[payload_off, payload_off + payload_len + local_string_len)` retrieves both.
+
+### 3.4 Shard Directory
+
+```
+ShardEntry[shard_count]
+```
+
+Each entry (40 bytes):
+
+| Offset | Size | Field                | Notes                                                     |
+| -----: | ---: | -------------------- | --------------------------------------------------------- |
+|      0 |    8 | `payload_off`        | u64 offset to shard payload.                              |
+|      8 |    4 | `payload_len_bytes`  | u32.                                                      |
+|     12 |    4 | `row_count`          | u32.                                                      |
+|     16 |    2 | `min_reader_version` | u16. Reader with lower version must refuse this shard.    |
+|     18 |    2 | `flags`              | u16. Must be `0` in v1. Reserved.                         |
+|     20 |    4 | `reserved`           | Must be `0`. Pads to next u64 alignment.                  |
+|     24 |    8 | `local_string_off`   | u64. Absolute byte offset to this shard's local string table (M3+). `0` if the shard has no U32-lane fields. |
+|     32 |    8 | `local_string_len`   | u64. Byte length of the local string table. `0` if none.  |
+
+`min_reader_version` is the load-bearing forward-compat lever. A shard that uses a lane kind introduced in reader v2 sets `min_reader_version = 2`; a v1 reader detects the mismatch at directory load and refuses cleanly (not on first bad read).
+
+### 3.5 Shard Payload
+
+Interleaved rows, `row_count` many, each `row_stride_bytes` wide. Field byte offsets within a row come from the schema block. No per-row header.
+
+Payloads are padded to a multiple of 8 bytes so the last row's F64 fields are safely readable.
+
+### 3.6 Zone Maps Segment (M7, optional)
+
+Present iff header's `metadata_off != 0`. Carries per-shard min/max for every F64-lane field, enabling query pruning: a Reader can skip shards whose bounds cannot contain a filter's target range, without fetching the shard payload at all. String (U32) fields are not tracked in v1 zone maps; dictionary-level bounds are M8+ material.
+
+Layout at `metadata_off`:
+
+```
+u32  magic                'ZM01'   (0x314D5A30 in little-endian byte order)
+u32  shard_count                   Must equal the container's shard_count.
+u32  tracked_field_count           Number of F64 fields with zone maps.
+u32  reserved0                     Must be 0.
+u16  field_indices[tracked_field_count]   Indices into the schema's fields array.
+u8   pad                           to next 8-byte boundary.
+f64  mins[shard_count * tracked_field_count]   Row-major: mins[s*T + t].
+f64  maxes[shard_count * tracked_field_count]  Same layout.
+```
+
+- `tracked_field_count` MAY be less than the schema's F64 field count (the writer chooses which fields to track). For fields not present in `field_indices`, `shardBounds` returns `null`.
+- `mins[s*T + t]` and `maxes[s*T + t]` are the exact min/max of the tracked field's values across the rows in shard `s`. For shards where the field is missing (defaults to `0`), the min/max collapse to `[0, 0]` — still valid input to range predicates.
+- Zone maps segment is placed between the shard directory and the first shard payload. This keeps it inside a single Range fetch alongside the header/schema/directory during Reader initialization.
+
+Reader query APIs (see `Reader.js`):
+
+```
+reader.shardBounds(shardIdx, fieldName)     → {min, max} | null
+reader.findShards(fieldName, {min, max})    → shardIdx[]     // shards that MAY contain the range
+```
+
+`findShards` returns exactly the shards whose bounds overlap `[min, max]`; a shard whose bounds are disjoint is skipped, saving one range fetch per skipped shard in the RangeReader path.
+
+### 3.7 Container Footer (16 bytes)
+
+| Offset | Size | Field       | Notes                                                     |
+| -----: | ---: | ----------- | --------------------------------------------------------- |
+|      0 |    4 | `crc32`     | CRC-32C over bytes [0, footer_off). `0xFFFFFFFF` = absent.|
+|      4 |    4 | `reserved1` | Zero.                                                     |
+|      8 |    4 | `magic_end` | ASCII `1 K B L` = `0x31 0x4B 0x42 0x4C`.                  |
+|     12 |    4 | `footer_len`| u32. Currently `16`. Lets future versions grow the footer without breaking magic detection. |
+
+CRC is optional in M4; producers that omit it write `0xFFFFFFFF`.
+
+## 4. Schema and field descriptors
+
+### 4.1 FieldDescriptor (24 bytes, fixed)
+
+| Offset | Size | Field                | Notes                                                     |
+| -----: | ---: | -------------------- | --------------------------------------------------------- |
+|      0 |    2 | `name_len_bytes`     | u16. Field name UTF-8 length. Max 255 in v1.              |
+|      2 |    2 | `offset_in_row`      | u16. Byte offset within a row.                            |
+|      4 |    1 | `lane_kind`          | u8. See 4.2.                                              |
+|      5 |    1 | `flags`              | u8. See 4.3. Reserved in v1; readers reject non-zero.     |
+|      6 |    2 | `reserved2`          | u16. Zero.                                                |
+|      8 |    8 | `name_str_off`       | u64. Offset into per-schema name blob (see below).        |
+|     16 |    8 | `reserved3`          | u64. Zero.                                                |
+
+After the descriptor array, a `u32 name_blob_len` is followed by concatenated UTF-8 field names, padded to 8.
+
+Fixed-width descriptors let a Reader load the schema with a single `Uint32Array` cast plus a name-blob view.
+
+### 4.2 `lane_kind` values (v1)
+
+| Value | Name    | Bytes | Notes                                                                    |
+| ----: | ------- | ----: | ------------------------------------------------------------------------ |
+|   `1` | `F64`   |     8 | The only numeric lane in M4. Matches lite-bake's `Types.F64`.            |
+|   `2` | `F32`   |     4 | Reserved for future producer opt-in via schema override.                 |
+|   `3` | `U32`   |     4 | String-table index (M3+). Row cell holds a u32 index into the shard's `local_string_off` table. |
+|   `4` | `U8`    |     1 | Reserved.                                                                |
+
+Values 5..127 reserved for future lite-bake type lanes (matching `Types.*`).
+
+Values 128..255 reserved for stream-specific lanes; see 4.4.
+
+### 4.3 `flags` (per-field, v1)
+
+Currently all bits are reserved and must be zero. Reserved bit assignments (v2+) will use:
+
+- Bit 0: `LANE_KIND_EXT` — if set, the field uses an extended lane; consult 4.4.
+- Bits 1..7: unassigned.
+
+This is the seam Option (c) — `I64_BYTES_PRESERVED` — slides into without a container format-version bump. A shard using it sets `min_reader_version = 2` at the directory level; v1 readers refuse the shard, not the whole container.
+
+### 4.4 Extended lane kinds (reserved for v2)
+
+When `flags` bit 0 is set on a FieldDescriptor, `lane_kind` is interpreted from the extended-lane table:
+
+| Value | Name                    | Bytes | Notes                                                     |
+| ----: | ----------------------- | ----: | --------------------------------------------------------- |
+| `128` | `I64_BYTES_PRESERVED`   |     8 | Two-`u32` little-endian pair. Bytes preserved verbatim from source. |
+| `129` | `U64_BYTES_PRESERVED`   |     8 | As above, unsigned.                                       |
+
+All other values in the extended range are reserved and must be rejected.
+
+## 5. Reference producer: tokenizer contract
+
+The M1 tokenizer is a chunk-safe UTF-8 JSON SAX scanner. It does not itself write LBK1 shards — that is the M2 writer's job. This section specifies the contract between them so M1 can be built and benched in isolation.
+
+### 5.1 Input framing
+
+Two top-level modes, auto-detected by the first non-whitespace byte:
+
+- `[` — Top-level JSON array. Records are the array's elements.
+- Any other JSON value byte (`{`, `"`, `-`, `0`..`9`, `t`, `f`, `n`) — NDJSON. Records are separated by JSON whitespace (`0x20`, `0x09`, `0x0A`, `0x0D`).
+
+Records themselves must be JSON objects for the reference writer path. Non-object records are accepted by the tokenizer (it is generic) but rejected by the writer.
+
+### 5.2 Chunk safety
+
+`feed(chunk)` accepts a `Uint8Array` of any length, including chunks that split a token, an escape sequence, a UTF-8 code point, or a number across the boundary. `end()` signals input completion. The tokenizer never assumes chunk-aligned tokens.
+
+### 5.3 Sink interface
+
+The tokenizer emits events to a caller-provided sink object:
+
+```
+sink.onStartObject()
+sink.onEndObject()
+sink.onStartArray()
+sink.onEndArray()
+sink.onKey(bytes, start, end)      // UTF-8 range; ephemeral, copy if retained
+sink.onString(bytes, start, end)   // ephemeral
+sink.onNumber(value)               // F64
+sink.onTrue()
+sink.onFalse()
+sink.onNull()
+sink.onEnd()
+```
+
+Ranges reference the tokenizer's internal accumulation buffer. The bytes at `[start, end)` are valid only until control returns from the sink call. Sinks that need the bytes past that point must copy them.
+
+For a string that spans multiple chunks, the tokenizer accumulates into an internal buffer and emits a single `onString` at the closing quote. Numbers are parsed to F64 incrementally and emitted as a single `onNumber`.
+
+### 5.4 Number semantics (v1)
+
+- `NaN`, `Infinity`, `-Infinity` are not valid JSON per RFC 8259 and are rejected.
+- Numbers are parsed to IEEE 754 double. Values exceeding representable magnitude round to `+/-Infinity` inside the parser and are rejected as a "number overflow" error before reaching the sink.
+- Integers above 2^53 lose precision silently. Producers concerned with 64-bit ID fidelity should target v2 with the `I64_BYTES_PRESERVED` lane (4.4). The M0/M1 spec commits to F64 semantics on the hot path.
+
+#### 5.4.1 F64 preservation contract (M3)
+
+The tokenizer implements Clinger's fast-path decimal→F64 conversion with an exact `POW10[0..22]` table. This gives a two-tier preservation guarantee that consumers can rely on:
+
+- **Fast path (guaranteed correctly-rounded):** any numeric literal whose decimal representation has ≤ 15 significant digits AND absolute net exponent ≤ 22 produces the SAME IEEE 754 double as JavaScript's `Number(str)`. This covers all integer IDs within `Number.MAX_SAFE_INTEGER`, all "typical" decimals (prices, ratios, timestamps, coordinates), and scientific notation within `1e±22`. This is the domain the round-trip verifier (`test/_verify.js`) asserts bit-exact across every torture scenario.
+- **Slow path (may drift ≤ 1 ULP):** literals outside the fast path (16+ significant digits, or `|exponent| > 22`) fall through to a naive `intPart + fracPart/fracDiv * Math.pow(10, exp)` computation. The result is within 1 ULP of the correctly-rounded value. Correctly-rounded parsing across the full F64 domain (David Gay's `strtod` or equivalent) is planned for M4+.
+
+The boundary is exercised explicitly by `test/NumericBoundary.test.js`, which pins expected drift so any regression is caught.
+
+### 5.5 String semantics (v1)
+
+- Input MUST be valid UTF-8. Invalid sequences are a parse error.
+- All standard JSON escapes are supported: `\"`, `\\`, `\/`, `\b`, `\f`, `\n`, `\r`, `\t`, `\uXXXX`.
+- Surrogate pairs (`\uD800`..`\uDBFF` followed by `\uDC00`..`\uDFFF`) are combined and emitted as UTF-8.
+- Unpaired surrogates are a parse error.
+
+### 5.6 Errors
+
+Errors are surfaced synchronously as thrown `TokenizerError` instances with a `code`, `byteOffset` (absolute across chunks), and human-readable message. Codes are stable and enumerated in `Tokenizer.js` header comment.
+
+## 6. Package exports map (M4 shipped, publish-ready)
+
+Subpath entries carved so consumers pay only for what they import. Each entry is a self-contained single file to preserve the one-file-per-unit convention per entry. `sideEffects: false` in `package.json` so tree-shakers drop everything not explicitly imported.
+
+| Subpath | Purpose |
+| :-- | :-- |
+| `@zakkster/lite-bake-stream` | Root convenience API: `serialize(input, opts)`, `deserialize(bytes)`, plus re-exports of `Tokenizer`, `Writer`, `Reader`, `StringTable`. |
+| `@zakkster/lite-bake-stream/tokenizer` | Chunk-safe UTF-8 JSON SAX tokenizer. Zero-alloc post-warmup. |
+| `@zakkster/lite-bake-stream/writer` | LBK1 shard emitter. Explicit or sample-and-infer schema. Emits per-shard zone maps. |
+| `@zakkster/lite-bake-stream/reader` | LBK1 parser over an in-memory `ArrayBuffer`. Synchronous. Zone-maps query APIs. |
+| `@zakkster/lite-bake-stream/string-table` | Byte-level UTF-8 interning primitive. |
+| `@zakkster/lite-bake-stream/file-ingest` | Browser helper: `File.stream()` → Reader. Any `ReadableStream<Uint8Array>` works. |
+| `@zakkster/lite-bake-stream/range-reader` | LBK1 parser with lazy shard loading via an IO adapter. HTTP Range fetch or in-memory mock. Synchronous zone-maps pruning. |
+| `@zakkster/lite-bake-stream/multi-reader` | Logical union over N Reader instances. Rows and shards cumulatively addressed; findShards merges across sub-readers. |
+| `@zakkster/lite-bake-stream/split` | NDJSON splitter + per-part compile + container merge. Executor-agnostic — wire to any worker pool or run serially. |
+
+Every entry works in browsers (evergreen Chromium / Firefox / Safari) and Node 18+; no Node-specific APIs are used in the source. Every subpath has a matching `.d.ts` file in `types/` for TypeScript consumers.
+
+### 3.8 Preserve-Mode Containers (`flags` bit 0 = 1)
+
+A preserve-mode container carries opaque byte blobs -- one per record -- with no schema, no lane packing, no interning, no zone maps. Bytes in, same bytes out. Reader-side query surface is `getBytes(i) / getString(i) / getJSON(i)`.
+
+Motivation: real-world API JSON is often 2-3 levels deep with mixed nested objects and arrays. The schema-mode F64/U32 lane model rejects nested records by design. Preserve mode is the escape hatch: a fast mediator for arbitrary JSON payloads that the frontend/backend already knows how to parse.
+
+The container reuses the same 48-byte header and 40-byte ShardEntry, with these differences:
+
+- `flags` byte at offset 7 has bit 0 set to `1`.
+- `schema_block_off` (header offset 8) must be `0`.
+- `metadata_off` (header offset 16) must be `0` -- preserve mode carries no zone maps.
+- Each ShardEntry's `local_string_off` and `local_string_len` are `0`.
+- Each shard payload has this layout:
+
+```
+[ blob_0 | blob_1 | ... | blob_{N-1} | u32 offsets[N] ]
+^payload_off                          ^offset_table_off = payload_off + payload_len - N*4
+```
+
+The trailing `u32 offsets[N]` are little-endian, one per record. `offsets[i]` is the byte position of record `i` relative to `payload_off`. Record `i`'s byte range is `[offsets[i], offsets[i+1])`, or `[offsets[N-1], payload_len - N*4)` for the last record.
+
+Dispatch: a schema-mode `Reader` opening a preserve container throws `R_WRONG_MODE`. A `PreserveReader` opening a schema-mode container also throws `R_WRONG_MODE`. `deserialize()` reads the flag byte and returns the right one.
+
+Not currently supported in preserve mode: `RangeReader` (HTTP-Range lazy loading), `MultiReader` (unifying multiple containers), and `Split.mergeContainers`. These are additive future work; preserve containers work with `serialize`, `deserialize`, `ingestStream`, and `ingestFile` today.
+
+## 7. Round-trip preservation contract
+
+Given source NDJSON `S` and a schema `Σ` (either explicit or inferred), the LBK1 container `C` produced by the reference writer satisfies the following for every row `i ∈ [0, rowCount(S))` and every field `f ∈ Σ`:
+
+| Source value at (i, f) | Lane | Round-trip result |
+| :-- | :-- | :-- |
+| JSON number in fast-path domain (≤15 sig digits, `|exp| ≤ 22`) | F64 | **bit-exact IEEE 754 double** |
+| JSON number in slow-path domain | F64 | within 1 ULP of the correctly-rounded value |
+| JSON string (any valid UTF-8) | U32 | **byte-exact UTF-8 sequence** |
+| JSON `true` / `false` / `null` | F64 | `1` / `0` / `0` (documented coercion) |
+| absent field (missing in source) | F64 | `0` (default) |
+| absent field | U32 | string at index 0 of the shard's local table |
+| field in source but NOT in `Σ` | — | silently dropped (matches lite-bake core) |
+| field in `Σ` with wrong value type (post-freeze) | — | `W_LANE_MISMATCH` error, container not produced |
+
+The preservation contract is asserted mechanically by three complementary layers:
+
+1. **Unit tests** — `test/RoundTrip.test.js`, `test/StringRoundTrip.test.js` verify individual known-value scenarios.
+2. **Property-based fuzz** — `test/DataPreservation.test.js` generates arbitrary random shapes across 12 seeds and asserts exact preservation on every cell of every row.
+3. **Torture-scale verification** — `bench/torture.js` runs the `verifyRoundTrip` helper (`test/_verify.js`) OUTSIDE the GC-measured band, asserting that every scenario opting into `verify: true` produces a container whose Reader output matches its source NDJSON byte-for-byte. The verification budget currently covers ~73,500 rows across 12 scenarios per fast-tier run; the `verified` column in the torture report shows the row count actually checked.
+
+The boundary between "bit-exact" and "1 ULP" for F64 is pinned by `test/NumericBoundary.test.js`. Any drift in the tokenizer's number parser trips these tests before it can reach production.
+
+Correctness of the preservation contract is a release gate on par with the zero-GC gate. If either fails, no publish.
+
+## 8. Release gates
+
+- **M4 GA gate.** JSON conformance corpus (RFC 8259 + JSONTestSuite), fuzzed chunk-boundary splits at every byte position, F64 round-trip parity, container round-trip parity, zero-alloc-per-row proven by lite-gc-profiler after warmup.
+- **M8 demo gate.** Locked 60 fps on iPhone 7 (A10, iOS 15 Safari) while ingesting 100 MB of synthetic NDJSON from a fetch stream. Under-featured demo beats a laggy full-featured one.
+
+## 9. Non-goals (v1)
+
+- JSON5, JSONC, trailing commas, comments. Not accepted.
+- Streaming output of individual field updates (LBK1 is a write-once container).
+- Compression. Compose gzip/zstd around the container at the transport layer.
+- Multi-language producer libraries. JS-first; formats are portable, ports welcome.

package/llms.txt

@@ -0,0 +1,81 @@
+# @zakkster/lite-bake-stream
+
+Streaming byte-level JSON to lite-bake binary compiler for JS runtimes.
+
+## Purpose
+
+Ingest gigabyte-scale JSON (top-level array or NDJSON) into the `lite-bake` LBK1 binary format without materializing the intermediate object graph. Reference producer path for downstream consumers of the flat, interleaved, zero-GC `lite-bake` reader API.
+
+## Status
+
+v1.0.0 — stable. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak (M1 MacBook Pro): 98.37M rows, 4.89 GB container, zero major GC, zero minor GC, 499 KB total heap allocation, 590.21M cells verified byte-exact, zero mismatches. Tokenizer benches at 222-237 MB/s (~55% of JSON.parse, with no object graph allocated).
+
+Public API follows semver from 1.0.0. Future additions (I64 lane, columnar payload mode, container-level string table) land via the format's forward-compat seams -- `min_reader_version` on ShardEntry, reserved FieldDescriptor flags, the `metadata_off` block wrapper -- without a format_version bump.
+
+See SPEC.md for the LBK1 container format, section 3.6 for zone maps, section 4.3 for the reserved field flags.
+
+## Public API (v1.0.0)
+
+Two ingest modes share one top-level API:
+
+- **Schema mode** (default): F64/U32 lane packing, zone-maps query pruning. Flat records only.
+- **Preserve mode** (`{ preserve: true }`): opaque byte blobs, any JSON shape. Deeply nested API responses, arrays, mixed types round-trip byte-for-byte identical.
+
+Root import (`@zakkster/lite-bake-stream`): `serialize(input, opts)` and `deserialize(bytes)`. `deserialize` auto-dispatches on the container's flag bit. Plus class re-exports.
+
+Schema-mode classes:
+- `Tokenizer` (`/tokenizer`): chunk-safe UTF-8 JSON SAX scanner.
+- `Writer` (`/writer`): LBK1 shard emitter, plugs into Tokenizer. F64/U32 lanes, per-shard string tables and zone maps.
+- `Reader` (`/reader`): container parser, sync, zero-alloc `get(row, field)` accessors + zone-maps query APIs (`shardBounds`, `findShards`).
+- `RangeReader` (`/range-reader`): HTTP Range lazy shard loading, synchronous query pruning after open.
+- `MultiReader` (`/multi-reader`): logical union over N Readers sharing a schema.
+- `splitNDJSON`, `compilePart`, `compileInParts`, `mergeContainers` (`/split`): worker-agnostic split/compile/merge.
+
+Preserve-mode classes:
+- `PreserveTokenizer` (`/preserve-tokenizer`): NDJSON record-boundary scanner, JSON-aware depth tracking, chunk-safe.
+- `PreserveWriter` (`/preserve-writer`): opaque byte-blob sink, pre-allocated shard buffer, zero-GC record path.
+- `PreserveReader` (`/preserve-reader`): tri-API — `getBytes(i)` (zero-alloc view), `getString(i)`, `getJSON(i)`.
+
+Shared:
+- `StringTable` (`/string-table`): byte-level UTF-8 interning primitive.
+- `ingestStream`, `ingestFile` (`/file-ingest`): browser helpers piping a `ReadableStream<Uint8Array>` through the pipeline. `preserve` option dispatches to the right writer.
+
+Error classes with stable `code`: `TokenizerError`, `WriterError`, `ReaderError`, `RangeReaderError`, `MultiReaderError`, `SplitError`, `PreserveTokenizerError`, `PreserveWriterError`, `PreserveReaderError`.
+- `ingestStream`, `ingestFile` (`/file-ingest`): browser helpers piping a `ReadableStream<Uint8Array>` (e.g. `File.stream()`) through the Tokenizer + Writer, returning a Reader. Per-chunk `onProgress` callback.
+- `TokenizerError`, `WriterError`, `ReaderError`, `RangeReaderError`: thrown on parse/write/read errors with stable `code`.
+- `VERSION` const per subpath.
+
+## Schema forms
+
+- `{ fields: ['id', 'x', 'y'] }` — all-F64 shorthand, back-compat with M2.
+- `{ fields: [{name:'id', laneKind:'f64'}, {name:'tag', laneKind:'u32'}] }` — mixed lanes (M3).
+- Sample-and-infer (default when no schema is passed): first `sampleBytes` of input observed; fields with only numbers become F64, only strings become U32, mixed types raise `W_MIXED_LANE_TYPES` at freeze.
+
+## Contract
+
+- `feed(chunk: Uint8Array)`: consumes bytes; sink events fire synchronously.
+- `end()`: signals input completion. Errors if input ends mid-token.
+- Sink methods: `onStartObject`, `onEndObject`, `onStartArray`, `onEndArray`, `onKey`, `onString`, `onNumber`, `onTrue`, `onFalse`, `onNull`, `onEnd`.
+- Key/String byte ranges are ephemeral views into the tokenizer's internal buffer; copy if retained past the sink call.
+
+## Preservation contract
+
+Every declared field of every row round-trips. F64 lanes: bit-exact for numeric literals with ≤15 significant digits and |exponent| ≤ 22 (Clinger's fast path); ≤1 ULP drift outside that domain (documented, pinned by NumericBoundary.test.js). U32 lanes: byte-exact UTF-8, unconditional. Missing fields → documented defaults. Unknown keys → silently dropped. Wrong-type value on post-freeze schema → W_LANE_MISMATCH error, no corrupt container produced. See SPEC section 7 for the full table. Asserted by tests, property-based fuzz, AND torture-scale verification (~73,500 rows per fast-tier run).
+
+## Numbers
+
+F64 only. Values exceeding IEEE 754 double range are rejected as `E_NUMBER_OVERFLOW`. 64-bit integer IDs above 2^53 lose precision silently (documented in SPEC 5.4); v2 will introduce an opt-in bytes-preserved lane.
+
+## Framing
+
+- `framing: 'auto'` (default): first non-ws byte decides. `[` = top-level array, else NDJSON.
+- `framing: 'array'`: enforces `[ ... ]` wrapper.
+- `framing: 'ndjson'`: whitespace-separated top-level values.
+
+## Tree-shaking
+
+Subpath entries per SPEC section 6. `sideEffects: false`. Consumers import only the path they need; the browser reader never pulls the writer.
+
+## Non-goals
+
+JSON5, JSONC, comments, trailing commas, streaming field updates, compression, native (non-JS) producers.