@zakkster/lite-bake-stream 1.5.0 → 1.6.0

package/CHANGELOG.md

@@ -2,6 +2,14 @@
 
 All notable changes to this project will be documented in this file. This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.6.0] -- 2026-09-02
+
+M6 -- streaming emission + optional integrity. Public `beginStream(sink, { layout: 'stream', crc? })` binds a sink BEFORE feeding, so each shard payload streams to the sink as it finalizes and is dropped; `finalizeToSink(sink, { layout })` then writes the schema/directory/zone-map trailer + footer with one `writeAt(header, 0)` backpatch and returns `{ totalRows, shardCount, schema | mode, bytesWritten, layout }`. Called without a prior `beginStream`, `finalizeToSink` is the buffered convenience mode (`O(container)` peak, stated in the memory-model docs). `layout: 'prefix'` is byte-identical to `finalize()`. With the two-step contract, peak memory drops from `O(container)` to `O(targetShardBytes + directory)`. A default-emitted stream container is a legal v1 container (format_version stays 1) accepted by every shipped reader, `checkContainer`, and `mergeContainers`. `finalize()` is re-based over the same emitters, so its output is byte-for-byte identical to 1.5.0 (frozen sha256 goldens hold).
+
+Optional CRC-32C (Castagnoli, table-driven, zero deps, `src/Crc32c.js`): writers opt in with `{ crc: true }`; readers get `verifyCrc()` -> `'ok' | 'absent'` (mismatch throws `R_BAD_CRC`) and an open option `{ verifyCrc: true }` that fails closed on mismatch (`R_BAD_CRC`) and absence (`R_CRC_ABSENT`). `mergeContainers` recomputes the CRC iff every input carried one, else absent. KATs: `CRC32C("")=0x00000000`, `CRC32C("123456789")=0xE3069283`; the stream layout closes the ring with a split-buffer combine (`crc32cCombine`, KAT-pinned and covered end-to-end by a streaming `beginStream` + flipped-byte case). New sink guard `W_BAD_SINK`. Three new thrown codes (`W_BAD_SINK`, `R_BAD_CRC`, `R_CRC_ABSENT`); inventory 57 -> 60 / 0 unpinned.
+
+Measured peak RSS (full-tier gate, 500 MB streamed in 8 MiB chunks into a counting sink, container never materialized): `peakRss - baselineRss` = 32.1 MiB against the bound `4*targetShardBytes + 64*shardCount + 64 MiB` = 96.0 MiB at 41 shards -- versus the buffered `finalize()` path at ~704 MiB (baseline `soak --mb=500`: baselineRss 667 MiB, peakRss 1371 MiB). Suite: 479 -> 515 tests / 515 pass / 0 fail / 0 todo. Torture: 44/44 fast, 47/47 full, arrayBuffers growth 0.00 MB both tiers, t7 sink-release witness (retained shard slots <= 1; independently armed during qa -- retained shards fail 364/364, clean run 0/0). Measurement note: t7's final arrayBuffers read now settles and collects twice before reading (the gc-profiler contract) -- t5-fuzz's large-ArrayBuffer churn was polluting the single-gc read via V8's buffer pool; the 8 MB budget and all RULES numbers are unchanged, and the M6 paths retain ~0 in isolation. Pack 32 -> 33 (new `src/Crc32c.js`). See `decisions/0009-streaming-emission.md`.
+
 ## [1.5.0] -- 2026-09-02
 
 M5 -- every ceiling gets a door; the shard budget counts all the bytes it ships; the floors are measured. BS-26 closed; BS-27's floor items closed (the writer-holds-all-shards ~2x container peak moves to M6's streaming emission); BS-32/BS-33/BS-34 closed. Suite: 461 -> 479 tests / 479 pass / 0 fail / 0 todo. Torture: 44/44 fast (~322 ms tier sum), 47/47 full (~5.6 s) incl. a 2100-shard directory-scale scenario (SPEC checker green, RangeReader LRU cap held at 8, `_maybeEvict` allocation-flat over 20000 ops); arrayBuffers growth 0.00 MB both tiers; t7 soak tracker size 0. Inventory gate: 54 -> 57 thrown codes / 0 unpinned. Falsifiability, measured item-by-item with baseline src stashed: all 15 new door/behavior assertions fail on baseline and pass on this tree, every stay-green net holds on both, and an all-F64-lane container is byte-identical to the 1.4.1 output (same sha256 and length). BREAK matrix {1, t6, l, m, j} all exit non-zero.

package/README.md

@@ -13,7 +13,7 @@
 
 > Streaming byte-level JSON compiler for [`@zakkster/lite-bake`](https://github.com/PeshoVurtoleta/lite-bake). Zero-GC, tree-shakeable, gigabyte-scale.
 
-**Status:** v1.5.0. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.
+**Status:** v1.6.0. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.
 
 ## Two modes, one API
 
@@ -123,6 +123,23 @@ All of these fit the format's existing forward-compat seams — `format_version`
 
 Two release gates. Zero major GC AND every declared cell round-trips exactly. If either fails, no publish.
 
+## Streaming emission and integrity
+
+`finalize()` returns the whole container as one `ArrayBuffer` -- peak memory is `O(container)`. For bounded-memory output, `writer.finalizeToSink(sink, { layout })` emits to a caller sink and returns `{ totalRows, shardCount, schema | mode, bytesWritten, layout }` (no buffer). A sink is any object with a synchronous `write(bytes)`; `layout: 'stream'` also needs `writeAt(bytes, position)` for one header backpatch -- satisfiable by `fs.write(fd, buf, 0, len, pos)`, the FS Access API's `createWritable().write({ type: 'write', position })`, and the in-memory `MemorySink`.
+
+There are two ways to drive a stream, and they deliver different peaks:
+
+- **Two-step, bounded RAM** -- call `writer.beginStream(sink, { layout: 'stream' })` BEFORE feeding, feed the tokenizer, then `writer.finalizeToSink(sink, { layout: 'stream' })`. Each shard is written to the sink as it finalizes and its bytes are dropped, so peak memory is `peak(stream) = 2*targetShardBytes + shardCount*(40 + 16*T) + schemaBlockBytes` -- `O(targetShardBytes + directory)`, never `O(container)`. This is the mode the 500 MB gate proves.
+- **One-shot, buffered** -- call `writer.finalizeToSink(sink, opts)` alone (no `beginStream`). Correct and simplest, but the shards are buffered first, so peak is `peak(prefix) = containerBytes + sum(shard bytes)` -- `O(container)`, the same as `finalize()`.
+
+`layout: 'prefix'` is the classic layout, **byte-identical** to `finalize()`. `layout: 'stream'` places the schema/directory/zone-map trailer and footer after the payloads with one header backpatch; a default-emitted stream container is a legal v1 container (format_version stays 1) that every shipped reader, `checkContainer`, and `mergeContainers` accept.
+
+A malformed sink (non-object, missing `write`, missing `writeAt` for stream, or an async/thenable return) throws `W_BAD_SINK`; a sink that throws mid-emission fails the writer closed and rethrows the source error verbatim (a retry then hits `W_FINALIZED`).
+
+**Memory model.** `peak(prefix) = containerBytes + sum(shard bytes)`; `peak(stream) = 2*targetShardBytes + shardCount*(40 + 16*T) + schemaBlockBytes`. The full-tier gate drives the two-step bounded path, streaming 500 MB in 8 MiB chunks into a counting sink, and asserts `peakRss - baselineRss <= 4*targetShardBytes + 64*shardCount + 64 MiB` (measured 32.1 MiB against the 96.0 MiB bound at 41 shards; the buffered path is ~704 MiB at 500 MB).
+
+**Optional CRC-32C** (Castagnoli, table-driven, zero deps). Opt in with `{ crc: true }` on the writer (or serialize `writer` opts); coverage is `[0, footer_off)`, folded per emitted chunk. Readers expose `verifyCrc()` -> `'ok' | 'absent'` (a mismatch throws `R_BAD_CRC`) on `Reader`/`PreserveReader` (sync) and `RangeReader` (async). The open option `{ verifyCrc: true }` (also `deserialize(bytes, opts)`) fails closed on both mismatch (`R_BAD_CRC`) and absence (`R_CRC_ABSENT`) -- `null` is not zero. `0xFFFFFFFF` means absent and stays legal. `mergeContainers` recomputes the CRC iff every input carried one, else emits absent. See `decisions/0009-streaming-emission.md`.
+
 ## Release testing tiers
 
 The gates run at four scales, chosen to fit different hardware and time budgets:

package/SPEC.md

@@ -165,7 +165,9 @@ reader.findShards(fieldName, {min, max})    → shardIdx[]     // shards that MA
 |      8 |    4 | `magic_end` | ASCII `1 K B L` = `0x31 0x4B 0x42 0x4C`.                  |
 |     12 |    4 | `footer_len`| u32. Currently `16`. Lets future versions grow the footer without breaking magic detection. |
 
-CRC is optional in M4; producers that omit it write `0xFFFFFFFF`.
+CRC is optional; producers that omit it write `0xFFFFFFFF` (absent). Producers opt in with the writer option `{ crc: true }`. The CRC-32C uses the Castagnoli polynomial (bit-reflected `0x82F63B78`); check value `CRC32C("123456789") = 0xE3069283`. Readers verify with `verifyCrc()` (returns `'ok' | 'absent'`, throws `R_BAD_CRC` on mismatch) or the open option `{ verifyCrc: true }`, which additionally throws `R_CRC_ABSENT` when the caller demanded verification but the CRC is absent.
+
+Note (SPEC 3 / 3.6): container offsets are ABSOLUTE, so two conformant orderings exist for a v1 container. The classic PREFIX ordering places the schema block, shard directory, and zone maps before the shard payloads. The STREAMING ordering (`layout: 'stream'`) places the shard payloads first, then the schema block, shard directory, and zone maps as a trailer, then the footer; the schema block still immediately precedes the shard directory (adjacency preserved). Both are valid v1 containers; format_version is unchanged.
 
 ## 4. Schema and field descriptors
 

package/llms.txt

@@ -8,13 +8,13 @@ Ingest gigabyte-scale JSON (top-level array or NDJSON) into the `lite-bake` LBK1
 
 ## Status
 
-v1.5.0 — stable. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak (M1 MacBook Pro): 98.37M rows, 4.89 GB container, zero major GC, zero minor GC, 499 KB total heap allocation, 590.21M cells verified byte-exact, zero mismatches. Tokenizer benches at 222-237 MB/s (~55% of JSON.parse, with no object graph allocated).
+v1.6.0 — stable. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak (M1 MacBook Pro): 98.37M rows, 4.89 GB container, zero major GC, zero minor GC, 499 KB total heap allocation, 590.21M cells verified byte-exact, zero mismatches. Tokenizer benches at 222-237 MB/s (~55% of JSON.parse, with no object graph allocated).
 
 Public API follows semver from 1.0.0. Future additions (I64 lane, columnar payload mode, container-level string table) land via the format's forward-compat seams -- `min_reader_version` on ShardEntry, reserved FieldDescriptor flags, the `metadata_off` block wrapper -- without a format_version bump.
 
 See SPEC.md for the LBK1 container format, section 3.6 for zone maps, section 4.3 for the reserved field flags.
 
-## Public API (v1.5.0)
+## Public API (v1.6.0)
 
 Two ingest modes share one top-level API:
 
@@ -87,6 +87,14 @@ F64 only. Values exceeding IEEE 754 double range are rejected as `E_NUMBER_OVERF
 
 Subpath entries per SPEC section 6. `sideEffects: false`. Consumers import only the path they need; the browser reader never pulls the writer.
 
+## Streaming emission and integrity (M6)
+
+`finalize()` returns the whole container as one ArrayBuffer (peak memory O(container)). `writer.finalizeToSink(sink, { layout })` emits to a caller sink instead and returns `{ totalRows, shardCount, schema|mode, bytesWritten, layout }` (no buffer). A sink is any object with a synchronous `write(bytes)`; `layout: 'stream'` also needs `writeAt(bytes, position)` for one header backpatch. Two ways to drive a stream with different peaks: (1) TWO-STEP bounded RAM -- `writer.beginStream(sink, { layout: 'stream' })` BEFORE feeding, feed the tokenizer, then `writer.finalizeToSink(sink, { layout: 'stream' })`; each shard streams to the sink as it finalizes and its bytes drop, so peak is `peak(stream) = 2*targetShardBytes + shardCount*(40 + 16*T) + schemaBlockBytes` = O(targetShardBytes + directory). (2) ONE-SHOT buffered -- `finalizeToSink(sink, opts)` alone (no beginStream); correct but the shards are buffered first, so peak is `peak(prefix) = containerBytes + sum(shard bytes)` = O(container), same as `finalize()`. `layout: 'prefix'` is the classic layout, byte-identical to `finalize()`. A default-emitted stream container is a legal v1 container (format_version stays 1) that every shipped reader, `checkContainer`, and `mergeContainers` accept. A malformed sink (non-object, missing `write`, missing `writeAt` for stream, or an async/thenable return) throws `W_BAD_SINK`; a sink that throws mid-emission fails the writer closed and rethrows the source error verbatim (a retry then hits `W_FINALIZED`).
+
+Memory model. `peak(prefix) = containerBytes + sum(shard bytes)`; `peak(stream) = 2*targetShardBytes + shardCount*(40 + 16*T) + schemaBlockBytes`. The full-tier gate streams 500 MB in 8 MiB chunks into a counting sink and asserts `peakRss - baselineRss <= 4*targetShardBytes + 64*shardCount + 64 MiB` (measured 32.1 MiB vs the 96.0 MiB bound at 41 shards; the buffered prefix path is ~704 MiB at 500 MB).
+
+Optional CRC-32C (Castagnoli, table-driven, zero deps). Writers opt in with `{ crc: true }` (constructor or serialize `writer` opts); coverage is [0, footer_off), folded per emitted chunk. Readers expose `verifyCrc()` -> `'ok' | 'absent'` (a mismatch throws `R_BAD_CRC`) on `Reader`/`PreserveReader` (sync) and `RangeReader` (async). The open option `{ verifyCrc: true }` (also `deserialize(bytes, opts)`) fails closed on both mismatch (`R_BAD_CRC`) and absence (`R_CRC_ABSENT`). `0xFFFFFFFF` means absent and stays legal (SPEC 3.7). `mergeContainers` recomputes the CRC iff every input carried one, else emits absent. KAT: `CRC32C("123456789") = 0xE3069283`. See decisions/0009-streaming-emission.md.
+
 ## Non-goals
 
 JSON5, JSONC, comments, trailing commas, streaming field updates, compression, native (non-JS) producers.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zakkster/lite-bake-stream",
-  "version": "1.5.0",
+  "version": "1.6.0",
   "description": "Streaming byte-level JSON to lite-bake binary compiler. Zero-GC, tree-shakeable, gigabyte-scale.",
   "type": "module",
   "sideEffects": false,

package/src/Crc32c.js

@@ -0,0 +1,96 @@
+// @zakkster/lite-bake-stream / Crc32c (internal)
+// Copyright (c) 2026 Zahary Shinikchiev. MIT.
+//
+// CRC-32C (Castagnoli, bit-reflected polynomial 0x82F63B78) over container
+// bodies. Backs the optional footer integrity field (SPEC 3.7). The 256-entry
+// lookup table is built ONCE at import (cold); the update loop is scalar and
+// allocation-free, folding one byte per iteration:
+//     crc = (crc >>> 8) ^ TABLE[(crc ^ b) & 0xFF]
+// It is called per emitted CHUNK, never per record.
+//
+// The running CRC is carried as a signed int32 (init = 0xFFFFFFFF, i.e. -1);
+// crc32cFinal applies the trailing one-complement and returns an unsigned u32.
+//
+// crc32cCombine folds the finalized CRC of a suffix onto the finalized CRC of a
+// prefix (GF(2) matrix method, as in zlib's crc32_combine but with the CRC-32C
+// polynomial). The streaming writer uses it ONCE at end-of-input to prepend the
+// 48-byte header CRC onto the running body CRC without re-reading the dropped
+// shard payloads. Cold path; not on any per-chunk loop.
+//
+// Not a public export. No subpath. Internal to the package.
+
+const POLY = 0x82F63B78;   // CRC-32C, bit-reflected
+const TABLE = buildTable();
+
+function buildTable() {
+  const t = new Int32Array(256);
+  for (let n = 0; n < 256; n++) {
+    let c = n;
+    for (let k = 0; k < 8; k++) {
+      c = (c & 1) ? (POLY ^ (c >>> 1)) : (c >>> 1);
+    }
+    t[n] = c | 0;
+  }
+  return t;
+}
+
+export function crc32cInit() { return 0xFFFFFFFF | 0; }
+
+export function crc32cUpdate(crc, bytes, off, len) {
+  let c = crc | 0;
+  const end = off + len;
+  for (let i = off; i < end; i++) {
+    c = (c >>> 8) ^ TABLE[(c ^ bytes[i]) & 0xFF];
+  }
+  return c | 0;
+}
+
+export function crc32cFinal(crc) {
+  return (crc ^ 0xFFFFFFFF) >>> 0;
+}
+
+// ---- GF(2) matrix helpers for crc32cCombine (cold) --------------------------
+
+const _even = new Int32Array(32);
+const _odd = new Int32Array(32);
+
+function gf2Times(mat, vec) {
+  let sum = 0;
+  let v = vec >>> 0;
+  let i = 0;
+  while (v !== 0) {
+    if (v & 1) sum ^= mat[i];
+    v >>>= 1;
+    i++;
+  }
+  return sum | 0;
+}
+
+function gf2Square(square, mat) {
+  for (let n = 0; n < 32; n++) square[n] = gf2Times(mat, mat[n]);
+}
+
+// crcA = finalized CRC of prefix A; crcB = finalized CRC of suffix B; len2 =
+// byte length of B. Returns the finalized CRC of A || B. len2 may exceed 2^31,
+// so it is halved by division and tested by remainder.
+export function crc32cCombine(crcA, crcB, len2) {
+  if (len2 === 0) return crcA >>> 0;
+  const odd = _odd, even = _even;
+  odd[0] = POLY | 0;               // operator for a single zero bit
+  let row = 1;
+  for (let n = 1; n < 32; n++) { odd[n] = row; row = (row << 1) | 0; }
+  gf2Square(even, odd);            // 2 zero bits
+  gf2Square(odd, even);            // 4 zero bits
+  let crc = crcA >>> 0;
+  let len = len2;
+  do {
+    gf2Square(even, odd);
+    if (len % 2) crc = gf2Times(even, crc) >>> 0;
+    len = Math.floor(len / 2);
+    if (len === 0) break;
+    gf2Square(odd, even);
+    if (len % 2) crc = gf2Times(odd, crc) >>> 0;
+    len = Math.floor(len / 2);
+  } while (len !== 0);
+  return (crc ^ (crcB >>> 0)) >>> 0;
+}

package/src/FileIngest.js

@@ -27,7 +27,7 @@ import { PreserveWriter } from './PreserveWriter.js';
 import { PreserveReader } from './PreserveReader.js';
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.5.0';
+export const VERSION = '1.6.0';
 
 const U32_MAX = 4294967295;
 const INGEST_OPTS = {

package/src/MultiReader.js

@@ -25,7 +25,7 @@
 //   M_ROW_OUT_OF_RANGE    - rowIdx >= totalRows
 //   M_TOO_MANY_ROWS       - cumulative row count exceeds Number.MAX_SAFE_INTEGER
 
-export const VERSION = '1.5.0';
+export const VERSION = '1.6.0';
 
 export class MultiReaderError extends Error {
   constructor(code, msg) { super(msg); this.code = code; this.name = 'MultiReaderError'; }

package/src/PreserveReader.js

@@ -32,12 +32,18 @@
 //   R_OFFSET_TOO_LARGE       - a u64 header/directory offset exceeds 2^53-1
 
 import { toContainerBuffer } from './Views.js';
+import { checkOpts } from './Opts.js';
+import { crc32cInit, crc32cUpdate, crc32cFinal } from './Crc32c.js';
 
-export const VERSION = '1.5.0';
+export const VERSION = '1.6.0';
 
 const CONTAINER_HEADER_BYTES = 48;
 const SHARD_ENTRY_BYTES = 40;
 const FOOTER_BYTES = 16;
+const CRC_ABSENT = 0xFFFFFFFF;
+
+const PRESERVE_READER_OPTS = { verifyCrc: { t: 'bool' } };
+function raisePreserveReaderOpt(code, msg) { throw new PreserveReaderError(code, msg); }
 
 export class PreserveReaderError extends Error {
   constructor(code, msg) { super(msg); this.code = code; this.name = 'PreserveReaderError'; }
@@ -55,14 +61,15 @@ function u64(dv, off, what) {
 }
 
 export class PreserveReader {
-  static fromBuffer(input) {
-    return new PreserveReader(toContainerBuffer(input, 'PreserveReader.fromBuffer'));
+  static fromBuffer(input, opts) {
+    return new PreserveReader(toContainerBuffer(input, 'PreserveReader.fromBuffer'), opts);
   }
 
-  constructor(buffer) {
+  constructor(buffer, opts) {
     if (!(buffer instanceof ArrayBuffer)) {
       throw new TypeError('PreserveReader: expected ArrayBuffer');
     }
+    checkOpts('PreserveReader', opts, PRESERVE_READER_OPTS, raisePreserveReaderOpt);
     this._buffer = buffer;
     this._dv = new DataView(buffer);
     this._bytes = new Uint8Array(buffer);
@@ -70,6 +77,22 @@ export class PreserveReader {
     this._parseHeader();
     this._parseFooter();
     this._parseShardDirectory();
+    if (opts && opts.verifyCrc === true) {
+      const status = this.verifyCrc();
+      if (status === 'absent')
+        throw new PreserveReaderError('R_CRC_ABSENT', 'verifyCrc:true but the container carries no CRC (footer CRC is absent, 0xFFFFFFFF)');
+    }
+  }
+
+  // SPEC 3.7 integrity: 'ok' | 'absent'; a mismatch throws R_BAD_CRC.
+  verifyCrc() {
+    const footerOff = this._buffer.byteLength - FOOTER_BYTES;
+    const stored = this._dv.getUint32(footerOff, true) >>> 0;
+    if (stored === CRC_ABSENT) return 'absent';
+    const actual = crc32cFinal(crc32cUpdate(crc32cInit(), this._bytes, 0, footerOff));
+    if (actual !== stored)
+      throw new PreserveReaderError('R_BAD_CRC', 'container CRC mismatch: stored 0x' + stored.toString(16) + ' != computed 0x' + actual.toString(16));
+    return 'ok';
   }
 
   _parseHeader() {

package/src/PreserveTokenizer.js

@@ -32,7 +32,7 @@
 
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.5.0';
+export const VERSION = '1.6.0';
 
 const U32_MAX = 4294967295;
 const PRESERVE_TOKENIZER_OPTS = {

package/src/PreserveWriter.js

@@ -17,8 +17,10 @@
 // container with R_WRONG_MODE.
 
 import { checkOpts } from './Opts.js';
+import { crc32cInit, crc32cUpdate, crc32cFinal, crc32cCombine } from './Crc32c.js';
+import { validateSink, isThenable } from './Views.js';
 
-export const VERSION = '1.5.0';
+export const VERSION = '1.6.0';
 
 export class PreserveWriterError extends Error {
   constructor(code, msg) { super(msg); this.code = code; this.name = 'PreserveWriterError'; }
@@ -28,6 +30,11 @@ function raisePreserveWriter(code, msg) { throw new PreserveWriterError(code, ms
 const CONTAINER_HEADER_BYTES = 48;
 const SHARD_ENTRY_BYTES = 40;
 const FOOTER_BYTES = 16;
+const CRC_ABSENT = 0xFFFFFFFF;
+const FINALIZE_TO_SINK_OPTS = {
+  layout: { t: 'enum', values: ['prefix', 'stream'] },
+  crc: { t: 'bool' },
+};
 const DEFAULT_TARGET_SHARD_BYTES = 8 * 1024 * 1024;   // 8 MiB — smaller than schema mode
 const INITIAL_OFFSETS_CAP = 4096;
 const U32_MAX = 4294967295;
@@ -38,6 +45,7 @@ const U32_MAX = 4294967295;
 const PRESERVE_WRITER_OPTS = {
   targetShardBytes: { t: 'int', min: 1, max: U32_MAX },
   maxRecordBytes: { t: 'int', min: 0, max: U32_MAX },
+  crc: { t: 'bool' },
 };
 
 export class PreserveWriter {
@@ -45,9 +53,16 @@ export class PreserveWriter {
     checkOpts('PreserveWriter', opts, PRESERVE_WRITER_OPTS, raisePreserveWriter);
     opts = opts || {};
     this._targetShardBytes = opts.targetShardBytes !== undefined ? opts.targetShardBytes : DEFAULT_TARGET_SHARD_BYTES;
+    this._crc = opts.crc === true;
     this._shards = [];
     this._totalRows = 0;
     this._finalized = false;
+    // Streaming-emission state (M6); see Writer for the model. A bound sink makes
+    // _finalizeCurrentShard emit + drop each shard, retaining only a descriptor.
+    this._sink = null;
+    this._sinkCrcOn = false;
+    this._sinkPos = 0;
+    this._sinkCrc = 0;
     this._currentBlobs = null;   // null => never allocated; _allocateShard allocates once
     this._allocateShard();
   }
@@ -162,11 +177,15 @@ export class PreserveWriter {
     for (let i = 0; i < rowCount; i++) {
       dv.setUint32(blobLen + i * 4, this._currentOffsets[i], true);
     }
-    this._shards.push({
-      bytes: shardBytes,
-      rowCount,
-      blobLen,
-    });
+    if (this._sink !== null) {
+      this._streamEmitShard(shardBytes, rowCount);
+    } else {
+      this._shards.push({
+        bytes: shardBytes,
+        rowCount,
+        blobLen,
+      });
+    }
     // Reuse the working buffers for the next shard: reset the cursors only, keep
     // the allocations (BS-27). No stale bytes can leak -- every read into these
     // buffers is bounded by _currentBlobBytes / _currentRowCount (both zeroed
@@ -175,87 +194,219 @@ export class PreserveWriter {
     this._currentRowCount = 0;
   }
 
-  finalize() {
+  _completeInput() {
     if (this._finalized) throw new PreserveWriterError('W_FINALIZED', 'writer already finalized');
     if (this._currentRowCount > 0) this._finalizeCurrentShard();
     if (this._shards.length === 0) {
-      // Emit an empty container? Or error? Match schema-mode: throw.
+      // Match schema-mode: an empty container is an error, not a silent no-op.
       throw new PreserveWriterError('W_EMPTY_INPUT', 'no records written; nothing to finalize');
     }
+  }
+
+  finalize() {
+    this._completeInput();
+    const bytes = this._buildPrefixBytes(this._crc);
     this._finalized = true;
-    return this._assembleContainer();
+    return { buffer: bytes.buffer, totalRows: this._totalRows, shardCount: this._shards.length, mode: 'preserve' };
   }
 
   get totalRows()  { return this._totalRows; }
   get shardCount() { return this._shards.length; }
 
-  _assembleContainer() {
+  // -------- container assembly (placement-free emitters) --------
+  // Preserve mode has no schema block and no zone maps: schema_block_off and
+  // metadata_off are 0. Classic prefix layout is header | directory | payloads |
+  // footer; the streaming layout (O9) is header | payloads | directory | footer.
+
+  _emitHeaderInto(dv, bytes, off, shardDirOff, shardCount, totalRows) {
+    bytes[off + 0] = 0x4C; bytes[off + 1] = 0x42; bytes[off + 2] = 0x4B; bytes[off + 3] = 0x31;   // 'LBK1'
+    dv.setUint16(off + 4, 1, true);       // format_version
+    bytes[off + 6] = 1;                   // endian LE
+    bytes[off + 7] = 0x01;                // flags: bit 0 = preserve mode
+    dv.setBigUint64(off + 8,  0n, true);  // schema_block_off = 0
+    dv.setBigUint64(off + 16, 0n, true);  // metadata_off = 0
+    dv.setBigUint64(off + 24, BigInt(shardDirOff), true);
+    dv.setUint32(off + 32, shardCount, true);
+    dv.setUint32(off + 36, 0, true);      // reserved1
+    dv.setBigUint64(off + 40, BigInt(totalRows), true);
+  }
+
+  _emitDirEntryInto(dv, entryOff, payloadOff, payloadLen, rowCount) {
+    dv.setBigUint64(entryOff + 0,  BigInt(payloadOff), true);
+    dv.setUint32(entryOff + 8,  payloadLen, true);
+    dv.setUint32(entryOff + 12, rowCount, true);
+    dv.setUint16(entryOff + 16, 1, true);   // min_reader_version
+    dv.setUint16(entryOff + 18, 0, true);   // shard flags
+    dv.setUint32(entryOff + 20, 0, true);   // reserved
+    dv.setBigUint64(entryOff + 24, 0n, true);   // no string table
+    dv.setBigUint64(entryOff + 32, 0n, true);
+  }
+
+  _emitFooterInto(dv, bytes, footerOff, crcVal) {
+    dv.setUint32(footerOff + 0, crcVal >>> 0, true);
+    dv.setUint32(footerOff + 4, 0, true);
+    bytes[footerOff + 8]  = 0x31;   // '1'
+    bytes[footerOff + 9]  = 0x4B;   // 'K'
+    bytes[footerOff + 10] = 0x42;   // 'B'
+    bytes[footerOff + 11] = 0x4C;   // 'L'
+    dv.setUint32(footerOff + 12, FOOTER_BYTES, true);
+  }
+
+  // Classic prefix layout, byte-for-byte identical to the pre-M6 assembler when
+  // crc is off; the optional CRC-32C covers [0, footer_off).
+  _buildPrefixBytes(crcOn) {
     const shardCount = this._shards.length;
     const shardDirBytes = shardCount * SHARD_ENTRY_BYTES;
-
-    // Layout: header (48) | shard directory | shard payloads | footer (16)
-    // No schema block, no zone maps in preserve mode; schema_block_off = 0.
     const shardDirOff = CONTAINER_HEADER_BYTES;
     let cursor = shardDirOff + shardDirBytes;
-    const shardPayloadOffsets = new Array(shardCount);
+    const payloadOffs = new Array(shardCount);
     for (let i = 0; i < shardCount; i++) {
-      shardPayloadOffsets[i] = cursor;
+      payloadOffs[i] = cursor;
       cursor += this._shards[i].bytes.length;
     }
+    const footerOff = cursor;
     const totalBytes = cursor + FOOTER_BYTES;
+    const buffer = new ArrayBuffer(totalBytes);
+    const dv = new DataView(buffer);
+    const bytes = new Uint8Array(buffer);
 
-    const container = new ArrayBuffer(totalBytes);
-    const dv = new DataView(container);
-    const bytes = new Uint8Array(container);
-
-    // Header
-    bytes[0] = 0x4C; bytes[1] = 0x42; bytes[2] = 0x4B; bytes[3] = 0x31;   // 'LBK1'
-    dv.setUint16(4, 1, true);            // format_version
-    bytes[6] = 1;                         // endian LE
-    bytes[7] = 0x01;                      // flags: bit 0 = preserve mode
-    dv.setBigUint64(8,  0n, true);        // schema_block_off = 0 (no schema)
-    dv.setBigUint64(16, 0n, true);        // metadata_off = 0 (no zone maps)
-    dv.setBigUint64(24, BigInt(shardDirOff), true);
-    dv.setUint32(32, shardCount, true);
-    dv.setUint32(36, 0, true);            // reserved1
-    dv.setBigUint64(40, BigInt(this._totalRows), true);
-
-    // Shard directory (40 bytes per entry, u64 payload_off, u32 payload_len,
-    // u32 row_count, u16 min_reader_version, u16 flags, u32 reserved,
-    // u64 local_string_off=0, u64 local_string_len=0)
+    this._emitHeaderInto(dv, bytes, 0, shardDirOff, shardCount, this._totalRows);
     for (let i = 0; i < shardCount; i++) {
-      const entryOff = shardDirOff + i * SHARD_ENTRY_BYTES;
       const s = this._shards[i];
-      dv.setBigUint64(entryOff + 0,  BigInt(shardPayloadOffsets[i]), true);
-      dv.setUint32(entryOff + 8,  s.bytes.length, true);
-      dv.setUint32(entryOff + 12, s.rowCount, true);
-      dv.setUint16(entryOff + 16, 1, true);   // min_reader_version
-      dv.setUint16(entryOff + 18, 0, true);   // shard flags
-      dv.setUint32(entryOff + 20, 0, true);   // reserved
-      dv.setBigUint64(entryOff + 24, 0n, true);   // no string table
-      dv.setBigUint64(entryOff + 32, 0n, true);
+      this._emitDirEntryInto(dv, shardDirOff + i * SHARD_ENTRY_BYTES, payloadOffs[i], s.bytes.length, s.rowCount);
+    }
+    for (let i = 0; i < shardCount; i++) bytes.set(this._shards[i].bytes, payloadOffs[i]);
+
+    let crcVal = CRC_ABSENT;
+    if (crcOn) crcVal = crc32cFinal(crc32cUpdate(crc32cInit(), bytes, 0, footerOff));
+    this._emitFooterInto(dv, bytes, footerOff, crcVal);
+    return bytes;
+  }
+
+  // PUBLIC. Bind a sink and stream shards as they finalize (bounded RAM). Call
+  // before feeding, then finalizeToSink(sink, opts) writes the trailer. Calling
+  // finalizeToSink alone is BUFFERED mode (O(container) peak). opts.layout must
+  // be 'stream'; opts.crc defaults to the constructor crc.
+  beginStream(sink, opts) {
+    if (this._finalized) throw new PreserveWriterError('W_FINALIZED', 'writer already finalized');
+    if (this._sink !== null) throw new PreserveWriterError('W_FINALIZED', 'writer is already streaming to a sink');
+    if (this._shards.length > 0 || this._currentRowCount > 0)
+      throw new PreserveWriterError('W_FINALIZED', 'beginStream must be called before the first record');
+    checkOpts('PreserveWriter.beginStream', opts, FINALIZE_TO_SINK_OPTS, raisePreserveWriter);
+    opts = opts || {};
+    if (opts.layout !== undefined && opts.layout !== 'stream')
+      raisePreserveWriter('W_BAD_SINK', "beginStream requires layout:'stream'");
+    validateSink(sink, true, raisePreserveWriter);
+    this._sink = sink;
+    this._sinkCrcOn = opts.crc !== undefined ? opts.crc === true : this._crc;
+    this._sinkCrc = crc32cInit();
+    this._sinkWrite(new Uint8Array(CONTAINER_HEADER_BYTES), false);
+    this._sinkPos = CONTAINER_HEADER_BYTES;
+  }
+
+  _sinkWrite(bytes, fold) {
+    if (fold && this._sinkCrcOn) this._sinkCrc = crc32cUpdate(this._sinkCrc, bytes, 0, bytes.length);
+    let ret;
+    // A throwing sink fails the writer closed and rethrows verbatim; retry -> W_FINALIZED.
+    try { ret = this._sink.write(bytes); }
+    catch (e) { this._finalized = true; throw e; }
+    if (isThenable(ret)) { this._finalized = true; raisePreserveWriter('W_BAD_SINK', 'sink.write returned a thenable; sinks must be synchronous'); }
+  }
+
+  // Preserve payloads carry an internal offset table and are NOT 8-padded (the
+  // classic layout packs them back-to-back), so the streaming layout does the
+  // same: no inter-shard padding.
+  _streamEmitShard(shardBytes, rowCount) {
+    const payloadOff = this._sinkPos;
+    const payloadLen = shardBytes.length;
+    this._sinkWrite(shardBytes, true);
+    this._sinkPos += payloadLen;
+    this._shards.push({ rowCount, payloadOff, payloadLen });
+  }
+
+  // PUBLIC. Emit to a caller sink. Returns { totalRows, shardCount, mode,
+  // bytesWritten, layout }.
+  finalizeToSink(sink, opts) {
+    if (this._finalized) throw new PreserveWriterError('W_FINALIZED', 'writer already finalized');
+    checkOpts('PreserveWriter.finalizeToSink', opts, FINALIZE_TO_SINK_OPTS, raisePreserveWriter);
+    opts = opts || {};
+
+    if (this._sink !== null) {
+      if (sink !== this._sink) raisePreserveWriter('W_BAD_SINK', 'finalizeToSink sink differs from the streaming sink');
+      if (opts.layout !== undefined && opts.layout !== 'stream')
+        raisePreserveWriter('W_BAD_SINK', "a streaming writer must be finalized with layout:'stream'");
+      const crcOn = opts.crc !== undefined ? opts.crc === true : this._sinkCrcOn;
+      this._sinkCrcOn = crcOn;
+      this._completeInput();
+      return this._finishStream(sink, crcOn);
+    }
+
+    const layout = opts.layout !== undefined ? opts.layout : 'stream';
+    const crcOn = opts.crc !== undefined ? opts.crc === true : this._crc;
+    validateSink(sink, layout === 'stream', raisePreserveWriter);
+    this._completeInput();
+
+    if (layout === 'prefix') {
+      const bytes = this._buildPrefixBytes(crcOn);
+      let ret;
+      try { ret = sink.write(bytes); }
+      catch (e) { this._finalized = true; throw e; }
+      if (isThenable(ret)) { this._finalized = true; raisePreserveWriter('W_BAD_SINK', 'sink.write returned a thenable; sinks must be synchronous'); }
+      this._finalized = true;
+      return { totalRows: this._totalRows, shardCount: this._shards.length, mode: 'preserve', bytesWritten: bytes.length, layout: 'prefix' };
+    }
+
+    const buffered = this._shards;
+    this._shards = [];
+    this._sink = sink;
+    this._sinkCrcOn = crcOn;
+    this._sinkCrc = crc32cInit();
+    this._sinkWrite(new Uint8Array(CONTAINER_HEADER_BYTES), false);
+    this._sinkPos = CONTAINER_HEADER_BYTES;
+    for (let i = 0; i < buffered.length; i++) {
+      this._streamEmitShard(buffered[i].bytes, buffered[i].rowCount);
+      buffered[i] = null;
     }
+    return this._finishStream(sink, crcOn);
+  }
+
+  _finishStream(sink, crcOn) {
+    const shardCount = this._shards.length;
+    const shardDirBytes = shardCount * SHARD_ENTRY_BYTES;
+    const shardDirOff = this._sinkPos;   // directory follows the payloads (O9)
+    const footerOff = shardDirOff + shardDirBytes;
 
-    // Shard payloads
+    const dir = new Uint8Array(shardDirBytes);
+    const ddv = new DataView(dir.buffer);
     for (let i = 0; i < shardCount; i++) {
-      bytes.set(this._shards[i].bytes, shardPayloadOffsets[i]);
+      const d = this._shards[i];
+      this._emitDirEntryInto(ddv, i * SHARD_ENTRY_BYTES, d.payloadOff, d.payloadLen, d.rowCount);
     }
+    this._sinkWrite(dir, true);
+    this._sinkPos += shardDirBytes;
 
-    // Footer
-    const footerOff = totalBytes - FOOTER_BYTES;
-    dv.setUint32(footerOff + 0, 0xFFFFFFFF, true);   // CRC absent
-    dv.setUint32(footerOff + 4, 0, true);
-    bytes[footerOff + 8]  = 0x31;   // '1'
-    bytes[footerOff + 9]  = 0x4B;   // 'K'
-    bytes[footerOff + 10] = 0x42;   // 'B'
-    bytes[footerOff + 11] = 0x4C;   // 'L'
-    dv.setUint32(footerOff + 12, FOOTER_BYTES, true);
+    const header = new Uint8Array(CONTAINER_HEADER_BYTES);
+    const hdv = new DataView(header.buffer);
+    this._emitHeaderInto(hdv, header, 0, shardDirOff, shardCount, this._totalRows);
 
-    return {
-      buffer: container,
-      totalRows: this._totalRows,
-      shardCount,
-      mode: 'preserve',
-    };
+    let crcVal = CRC_ABSENT;
+    if (crcOn) {
+      const headerCrc = crc32cFinal(crc32cUpdate(crc32cInit(), header, 0, CONTAINER_HEADER_BYTES));
+      const suffixCrc = crc32cFinal(this._sinkCrc);
+      crcVal = crc32cCombine(headerCrc, suffixCrc, footerOff - CONTAINER_HEADER_BYTES);
+    }
+    const footer = new Uint8Array(FOOTER_BYTES);
+    const fdv = new DataView(footer.buffer);
+    this._emitFooterInto(fdv, footer, 0, crcVal);
+    this._sinkWrite(footer, false);
+    this._sinkPos += FOOTER_BYTES;
+
+    let ret;
+    try { ret = sink.writeAt(header, 0); }
+    catch (e) { this._finalized = true; throw e; }
+    if (isThenable(ret)) { this._finalized = true; raisePreserveWriter('W_BAD_SINK', 'sink.writeAt returned a thenable; sinks must be synchronous'); }
+
+    this._finalized = true;
+    return { totalRows: this._totalRows, shardCount, mode: 'preserve', bytesWritten: this._sinkPos, layout: 'stream' };
   }
 }