@zakkster/lite-bake-stream 1.5.0 → 1.6.1

package/llms.txt

@@ -1,20 +1,20 @@
 # @zakkster/lite-bake-stream
 
-Streaming byte-level JSON to lite-bake binary compiler for JS runtimes.
+Streaming byte-level JSON to LBK1 binary containers for JS runtimes.
 
 ## Purpose
 
-Ingest gigabyte-scale JSON (top-level array or NDJSON) into the `lite-bake` LBK1 binary format without materializing the intermediate object graph. Reference producer path for downstream consumers of the flat, interleaved, zero-GC `lite-bake` reader API.
+Ingest gigabyte-scale JSON (top-level array or NDJSON) into the LBK1 binary container format without materializing the intermediate object graph. The container is flat, interleaved, and zero-GC to read; a Reader gives random access to any row without re-parsing JSON.
 
 ## Status
 
-v1.5.0 — stable. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak (M1 MacBook Pro): 98.37M rows, 4.89 GB container, zero major GC, zero minor GC, 499 KB total heap allocation, 590.21M cells verified byte-exact, zero mismatches. Tokenizer benches at 222-237 MB/s (~55% of JSON.parse, with no object graph allocated).
+v1.6.1 -- stable. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak (M1 MacBook Pro): 98.37M rows, 4.89 GB container, zero major GC, zero minor GC, 499 KB total heap allocation, 590.21M cells verified byte-exact, zero mismatches. Tokenizer benches at 222-237 MB/s (~55% of JSON.parse, with no object graph allocated).
 
 Public API follows semver from 1.0.0. Future additions (I64 lane, columnar payload mode, container-level string table) land via the format's forward-compat seams -- `min_reader_version` on ShardEntry, reserved FieldDescriptor flags, the `metadata_off` block wrapper -- without a format_version bump.
 
 See SPEC.md for the LBK1 container format, section 3.6 for zone maps, section 4.3 for the reserved field flags.
 
-## Public API (v1.5.0)
+## Public API (v1.6.1)
 
 Two ingest modes share one top-level API:
 
@@ -34,21 +34,18 @@ Schema-mode classes:
 Preserve-mode classes:
 - `PreserveTokenizer` (`/preserve-tokenizer`): NDJSON record-boundary scanner, JSON-aware depth tracking, chunk-safe.
 - `PreserveWriter` (`/preserve-writer`): opaque byte-blob sink, pre-allocated shard buffer, zero-GC record path.
-- `PreserveReader` (`/preserve-reader`): tri-API — `getBytes(i)` (zero-alloc view), `getString(i)`, `getJSON(i)`.
+- `PreserveReader` (`/preserve-reader`): tri-API -- `getBytes(i)` (zero-alloc view), `getString(i)`, `getJSON(i)`. A returned view pins the container's ArrayBuffer (plain subarray semantics); copy the bytes out if you need the container to be collectable.
 
 Shared:
 - `StringTable` (`/string-table`): byte-level UTF-8 interning primitive.
-- `ingestStream`, `ingestFile` (`/file-ingest`): browser helpers piping a `ReadableStream<Uint8Array>` through the pipeline. `preserve` option dispatches to the right writer.
+- `ingestStream`, `ingestFile` (`/file-ingest`): browser helpers piping a `ReadableStream<Uint8Array>` (e.g. `File.stream()`) through the pipeline. `preserve` option dispatches to the right writer. Per-chunk `onProgress` callback; its state object is reused across calls (mutated in place, zero per-chunk allocation) -- copy it if retained past the callback.
 
 Error classes with stable `code`: `TokenizerError`, `WriterError`, `ReaderError`, `RangeReaderError`, `MultiReaderError`, `SplitError`, `PreserveTokenizerError`, `PreserveWriterError`, `PreserveReaderError`.
-- `ingestStream`, `ingestFile` (`/file-ingest`): browser helpers piping a `ReadableStream<Uint8Array>` (e.g. `File.stream()`) through the Tokenizer + Writer, returning a Reader. Per-chunk `onProgress` callback; its state object is reused across calls (mutated in place, zero per-chunk allocation) -- copy it if retained past the callback.
-- `TokenizerError`, `WriterError`, `ReaderError`, `RangeReaderError`: thrown on parse/write/read errors with stable `code`.
-- `VERSION` const per subpath.
 
 ## Schema forms
 
-- `{ fields: ['id', 'x', 'y'] }` — all-F64 shorthand, back-compat with M2.
-- `{ fields: [{name:'id', laneKind:'f64'}, {name:'tag', laneKind:'u32'}] }` — mixed lanes (M3).
+- `{ fields: ['id', 'x', 'y'] }` -- all-F64 shorthand, back-compat with M2.
+- `{ fields: [{name:'id', laneKind:'f64'}, {name:'tag', laneKind:'u32'}] }` -- mixed lanes (M3).
 - Sample-and-infer (default when no schema is passed): the first `sampleBytes` of INPUT are observed, byte-true (BS-07) -- the window ends at the first record boundary at or after `sampleBytes` input bytes, read from the Tokenizer's `absOffset`, independent of chunking and record count (a hand-driven Writer with no wired source samples by record count). Fields with only numbers become F64, only strings become U32. null is lane-neutral (BS-20): it sets no kind, so null+string infers U32 (no longer "mixed") and null+number infers F64; post-freeze null reads 0 on an F64 lane and "" on a U32 lane. A field that saw both a real number and a real string raises `W_MIXED_LANE_TYPES` at freeze. Sample memory is O(`sampleBytes`): columnar staging plus a shared string table, independent of total input size.
 
 ## Contract
@@ -60,7 +57,7 @@ Error classes with stable `code`: `TokenizerError`, `WriterError`, `ReaderError`
 
 ## Preservation contract
 
-Every declared field of every row round-trips. F64 lanes: bit-exact for numeric literals with ≤15 significant digits and |exponent| ≤ 22 (Clinger's fast path); ≤1 ULP drift outside that domain (documented, pinned by NumericBoundary.test.js). U32 lanes: byte-exact UTF-8, unconditional. Missing fields → documented defaults. Unknown keys → silently dropped. Wrong-type value on post-freeze schema → W_LANE_MISMATCH error, no corrupt container produced. See SPEC section 7 for the full table. Asserted by tests, property-based fuzz, AND torture-scale verification (~73,500 rows per fast-tier run).
+Every declared field of every row round-trips. F64 lanes: bit-exact for numeric literals with <=15 significant digits and |exponent| <= 22 (Clinger's fast path); <=1 ULP drift outside that domain (documented, pinned by NumericBoundary.test.js). U32 lanes: byte-exact UTF-8, unconditional. Missing fields -> documented defaults. Unknown keys -> silently dropped. Wrong-type value on post-freeze schema -> W_LANE_MISMATCH error, no corrupt container produced. See SPEC section 7 for the full table. Asserted by tests, property-based fuzz, AND torture-scale verification (~103,000 rows across 20 scenarios per fast-tier run).
 
 ## Row-index and refusal codes
 
@@ -85,7 +82,15 @@ F64 only. Values exceeding IEEE 754 double range are rejected as `E_NUMBER_OVERF
 
 ## Tree-shaking
 
-Subpath entries per SPEC section 6. `sideEffects: false`. Consumers import only the path they need; the browser reader never pulls the writer.
+Subpath entries per SPEC section 6. `sideEffects: false`. Consumers import only the path they need; the browser reader never pulls the writer. Every subpath exports a `VERSION` const.
+
+## Streaming emission and integrity (M6)
+
+`finalize()` returns the whole container as one ArrayBuffer (peak memory O(container)). `writer.finalizeToSink(sink, { layout })` emits to a caller sink instead and returns `{ totalRows, shardCount, schema|mode, bytesWritten, layout }` (no buffer). A sink is any object with a synchronous `write(bytes)`; `layout: 'stream'` also needs `writeAt(bytes, position)` for one header backpatch. Two ways to drive a stream with different peaks: (1) TWO-STEP bounded RAM -- `writer.beginStream(sink, { layout: 'stream' })` BEFORE feeding, feed the tokenizer, then `writer.finalizeToSink(sink, { layout: 'stream' })`; each shard streams to the sink as it finalizes and its bytes drop, so peak is `peak(stream) = 2*targetShardBytes + shardCount*(40 + 16*T) + schemaBlockBytes` = O(targetShardBytes + directory). (2) ONE-SHOT buffered -- `finalizeToSink(sink, opts)` alone (no beginStream); correct but the shards are buffered first, so peak is `peak(prefix) = containerBytes + sum(shard bytes)` = O(container), same as `finalize()`. `layout: 'prefix'` is the classic layout, byte-identical to `finalize()`. A default-emitted stream container is a legal v1 container (format_version stays 1) that every shipped reader, `checkContainer`, and `mergeContainers` accept. A malformed sink (non-object, missing `write`, missing `writeAt` for stream, or an async/thenable return) throws `W_BAD_SINK`; a sink that throws mid-emission fails the writer closed and rethrows the source error verbatim (a retry then hits `W_FINALIZED`).
+
+Memory model. `peak(prefix) = containerBytes + sum(shard bytes)`; `peak(stream) = 2*targetShardBytes + shardCount*(40 + 16*T) + schemaBlockBytes`. The full-tier gate streams 500 MB in 8 MiB chunks into a counting sink and asserts `peakRss - baselineRss <= 4*targetShardBytes + 64*shardCount + 64 MiB` (measured 32.1 MiB vs the 96.0 MiB bound at 41 shards; the buffered prefix path is ~704 MiB at 500 MB).
+
+Optional CRC-32C (Castagnoli, table-driven, zero deps). Writers opt in with `{ crc: true }` (constructor or serialize `writer` opts); coverage is [0, footer_off), folded per emitted chunk. Readers expose `verifyCrc()` -> `'ok' | 'absent'` (a mismatch throws `R_BAD_CRC`) on `Reader`/`PreserveReader` (sync) and `RangeReader` (async). The open option `{ verifyCrc: true }` (also `deserialize(bytes, opts)`) fails closed on both mismatch (`R_BAD_CRC`) and absence (`R_CRC_ABSENT`). `0xFFFFFFFF` means absent and stays legal (SPEC 3.7). `mergeContainers` recomputes the CRC iff every input carried one, else emits absent. KAT: `CRC32C("123456789") = 0xE3069283`. See decisions/0009-streaming-emission.md.
 
 ## Non-goals
 

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@zakkster/lite-bake-stream",
-  "version": "1.5.0",
-  "description": "Streaming byte-level JSON to lite-bake binary compiler. Zero-GC, tree-shakeable, gigabyte-scale.",
+  "version": "1.6.1",
+  "description": "Streaming byte-level JSON to LBK1 binary containers. Zero-GC, tree-shakeable, gigabyte-scale.",
   "type": "module",
   "sideEffects": false,
   "exports": {

package/src/Crc32c.js

@@ -0,0 +1,96 @@
+// @zakkster/lite-bake-stream / Crc32c (internal)
+// Copyright (c) 2026 Zahary Shinikchiev. MIT.
+//
+// CRC-32C (Castagnoli, bit-reflected polynomial 0x82F63B78) over container
+// bodies. Backs the optional footer integrity field (SPEC 3.7). The 256-entry
+// lookup table is built ONCE at import (cold); the update loop is scalar and
+// allocation-free, folding one byte per iteration:
+//     crc = (crc >>> 8) ^ TABLE[(crc ^ b) & 0xFF]
+// It is called per emitted CHUNK, never per record.
+//
+// The running CRC is carried as a signed int32 (init = 0xFFFFFFFF, i.e. -1);
+// crc32cFinal applies the trailing one-complement and returns an unsigned u32.
+//
+// crc32cCombine folds the finalized CRC of a suffix onto the finalized CRC of a
+// prefix (GF(2) matrix method, as in zlib's crc32_combine but with the CRC-32C
+// polynomial). The streaming writer uses it ONCE at end-of-input to prepend the
+// 48-byte header CRC onto the running body CRC without re-reading the dropped
+// shard payloads. Cold path; not on any per-chunk loop.
+//
+// Not a public export. No subpath. Internal to the package.
+
+const POLY = 0x82F63B78;   // CRC-32C, bit-reflected
+const TABLE = buildTable();
+
+function buildTable() {
+  const t = new Int32Array(256);
+  for (let n = 0; n < 256; n++) {
+    let c = n;
+    for (let k = 0; k < 8; k++) {
+      c = (c & 1) ? (POLY ^ (c >>> 1)) : (c >>> 1);
+    }
+    t[n] = c | 0;
+  }
+  return t;
+}
+
+export function crc32cInit() { return 0xFFFFFFFF | 0; }
+
+export function crc32cUpdate(crc, bytes, off, len) {
+  let c = crc | 0;
+  const end = off + len;
+  for (let i = off; i < end; i++) {
+    c = (c >>> 8) ^ TABLE[(c ^ bytes[i]) & 0xFF];
+  }
+  return c | 0;
+}
+
+export function crc32cFinal(crc) {
+  return (crc ^ 0xFFFFFFFF) >>> 0;
+}
+
+// ---- GF(2) matrix helpers for crc32cCombine (cold) --------------------------
+
+const _even = new Int32Array(32);
+const _odd = new Int32Array(32);
+
+function gf2Times(mat, vec) {
+  let sum = 0;
+  let v = vec >>> 0;
+  let i = 0;
+  while (v !== 0) {
+    if (v & 1) sum ^= mat[i];
+    v >>>= 1;
+    i++;
+  }
+  return sum | 0;
+}
+
+function gf2Square(square, mat) {
+  for (let n = 0; n < 32; n++) square[n] = gf2Times(mat, mat[n]);
+}
+
+// crcA = finalized CRC of prefix A; crcB = finalized CRC of suffix B; len2 =
+// byte length of B. Returns the finalized CRC of A || B. len2 may exceed 2^31,
+// so it is halved by division and tested by remainder.
+export function crc32cCombine(crcA, crcB, len2) {
+  if (len2 === 0) return crcA >>> 0;
+  const odd = _odd, even = _even;
+  odd[0] = POLY | 0;               // operator for a single zero bit
+  let row = 1;
+  for (let n = 1; n < 32; n++) { odd[n] = row; row = (row << 1) | 0; }
+  gf2Square(even, odd);            // 2 zero bits
+  gf2Square(odd, even);            // 4 zero bits
+  let crc = crcA >>> 0;
+  let len = len2;
+  do {
+    gf2Square(even, odd);
+    if (len % 2) crc = gf2Times(even, crc) >>> 0;
+    len = Math.floor(len / 2);
+    if (len === 0) break;
+    gf2Square(odd, even);
+    if (len % 2) crc = gf2Times(odd, crc) >>> 0;
+    len = Math.floor(len / 2);
+  } while (len !== 0);
+  return (crc ^ (crcB >>> 0)) >>> 0;
+}

package/src/FileIngest.js

@@ -2,11 +2,15 @@
 // Browser-side helper: File.stream() -> Tokenizer -> Writer -> Reader.
 // Copyright (c) 2026 Zahary Shinikchiev. MIT.
 //
+// Error codes:
+//   E_OPTION_CONFLICT  - preserve mode requested with a non-ndjson framing
+//
 // The natural browser ingest path is:
 //
 //   const file = fileInputEl.files[0];
-//   const reader = await ingestStream(file.stream(), file.size, {
+//   const reader = await ingestStream(file.stream(), {
 //     framing: 'ndjson',
+//     totalBytes: file.size,
 //     onProgress: ({bytesIngested, totalBytes, rowsWritten, shardsCommitted}) => {...}
 //   });
 //   // reader is a fully-materialized in-memory LBK1 Reader.
@@ -27,7 +31,7 @@ import { PreserveWriter } from './PreserveWriter.js';
 import { PreserveReader } from './PreserveReader.js';
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.5.0';
+export const VERSION = '1.6.1';
 
 const U32_MAX = 4294967295;
 const INGEST_OPTS = {

package/src/MultiReader.js

@@ -25,7 +25,7 @@
 //   M_ROW_OUT_OF_RANGE    - rowIdx >= totalRows
 //   M_TOO_MANY_ROWS       - cumulative row count exceeds Number.MAX_SAFE_INTEGER
 
-export const VERSION = '1.5.0';
+export const VERSION = '1.6.1';
 
 export class MultiReaderError extends Error {
   constructor(code, msg) { super(msg); this.code = code; this.name = 'MultiReaderError'; }

package/src/Opts.js

@@ -1,6 +1,10 @@
 // @zakkster/lite-bake-stream / Opts (internal, not exported from index.js)
 // Copyright (c) 2026 Zahary Shinikchiev. MIT.
 //
+// Error codes:
+//   E_UNKNOWN_OPTION  - an unknown option key (fails closed with a did-you-mean hint)
+//   E_OPTION_VALUE    - an option value falls outside its declared domain
+//
 // Shared construction-time options validator. Every public constructor and
 // entry function that takes an opts object runs checkOpts once, at construction
 // or in a synchronous prologue -- never per byte, never per record. An unknown

package/src/PreserveReader.js

@@ -29,15 +29,23 @@
 //   R_INVALID                - a structure is internally inconsistent but in-bounds
 //   R_SHARD_VERSION_TOO_NEW  - a shard's min_reader_version > this reader
 //   R_ROW_OUT_OF_RANGE       - rowIdx negative, fractional, NaN, or >= totalRows (BS-32)
+//   R_BAD_CRC                - CRC-32C verify requested and the stored checksum mismatched
+//   R_CRC_ABSENT             - CRC verify requested but the container carries no checksum
 //   R_OFFSET_TOO_LARGE       - a u64 header/directory offset exceeds 2^53-1
 
 import { toContainerBuffer } from './Views.js';
+import { checkOpts } from './Opts.js';
+import { crc32cInit, crc32cUpdate, crc32cFinal } from './Crc32c.js';
 
-export const VERSION = '1.5.0';
+export const VERSION = '1.6.1';
 
 const CONTAINER_HEADER_BYTES = 48;
 const SHARD_ENTRY_BYTES = 40;
 const FOOTER_BYTES = 16;
+const CRC_ABSENT = 0xFFFFFFFF;
+
+const PRESERVE_READER_OPTS = { verifyCrc: { t: 'bool' } };
+function raisePreserveReaderOpt(code, msg) { throw new PreserveReaderError(code, msg); }
 
 export class PreserveReaderError extends Error {
   constructor(code, msg) { super(msg); this.code = code; this.name = 'PreserveReaderError'; }
@@ -55,14 +63,15 @@ function u64(dv, off, what) {
 }
 
 export class PreserveReader {
-  static fromBuffer(input) {
-    return new PreserveReader(toContainerBuffer(input, 'PreserveReader.fromBuffer'));
+  static fromBuffer(input, opts) {
+    return new PreserveReader(toContainerBuffer(input, 'PreserveReader.fromBuffer'), opts);
   }
 
-  constructor(buffer) {
+  constructor(buffer, opts) {
     if (!(buffer instanceof ArrayBuffer)) {
       throw new TypeError('PreserveReader: expected ArrayBuffer');
     }
+    checkOpts('PreserveReader', opts, PRESERVE_READER_OPTS, raisePreserveReaderOpt);
     this._buffer = buffer;
     this._dv = new DataView(buffer);
     this._bytes = new Uint8Array(buffer);
@@ -70,6 +79,22 @@ export class PreserveReader {
     this._parseHeader();
     this._parseFooter();
     this._parseShardDirectory();
+    if (opts && opts.verifyCrc === true) {
+      const status = this.verifyCrc();
+      if (status === 'absent')
+        throw new PreserveReaderError('R_CRC_ABSENT', 'verifyCrc:true but the container carries no CRC (footer CRC is absent, 0xFFFFFFFF)');
+    }
+  }
+
+  // SPEC 3.7 integrity: 'ok' | 'absent'; a mismatch throws R_BAD_CRC.
+  verifyCrc() {
+    const footerOff = this._buffer.byteLength - FOOTER_BYTES;
+    const stored = this._dv.getUint32(footerOff, true) >>> 0;
+    if (stored === CRC_ABSENT) return 'absent';
+    const actual = crc32cFinal(crc32cUpdate(crc32cInit(), this._bytes, 0, footerOff));
+    if (actual !== stored)
+      throw new PreserveReaderError('R_BAD_CRC', 'container CRC mismatch: stored 0x' + stored.toString(16) + ' != computed 0x' + actual.toString(16));
+    return 'ok';
   }
 
   _parseHeader() {

package/src/PreserveTokenizer.js

@@ -1,11 +1,20 @@
 // @zakkster/lite-bake-stream / PreserveTokenizer
 // NDJSON record-boundary scanner for preserve-mode. Copyright (c) 2026 Zahary Shinikchiev. MIT.
 //
+// Error codes:
+//   E_UNSUPPORTED_FRAMING  - preserve mode fed a framing other than ndjson
+//   E_RECORD_TOO_LARGE     - a single record exceeds maxRecordBytes
+//   E_UNBALANCED           - a record's JSON structure never closes
+//   E_TRUNCATED            - input ends mid-record
+//   E_ENDED                - feed() called after end()
+//   E_POISONED             - reuse after a prior throw
+//
 // The schema-mode Tokenizer walks the full JSON AST (numbers, strings, keys,
 // object/array structure) so the Writer can pack values into typed lanes.
-// Preserve-mode doesn't crack open records — it just needs to find where one
+// Preserve-mode doesn't crack open records -- it just needs to find where one
 // record ends and the next begins, so the record's original bytes can be
 // shoved into a shard intact.
+// See decisions/0010-preserve-mode.md.
 //
 // Contract:
 //   - Input:  NDJSON bytes. One JSON value per line, delimited by 0x0A.
@@ -32,7 +41,7 @@
 
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.5.0';
+export const VERSION = '1.6.1';
 
 const U32_MAX = 4294967295;
 const PRESERVE_TOKENIZER_OPTS = {
@@ -57,7 +66,7 @@ const BYTE_RBRACK = 0x5D;
 const BYTE_SPACE  = 0x20;
 const BYTE_TAB    = 0x09;
 
-const INITIAL_BUF = 1 << 16;   // 64 KiB — grows on demand for larger records
+const INITIAL_BUF = 1 << 16;   // 64 KiB -- grows on demand for larger records
 
 export class PreserveTokenizer {
   constructor(sink, opts) {