@zakkster/lite-bake-stream 1.5.0 → 1.6.1

package/README.md

@@ -1,5 +1,7 @@
 # @zakkster/lite-bake-stream
 
+> Streaming byte-level JSON to LBK1 binary containers. Zero-GC, tree-shakeable, gigabyte-scale.
+
 [![npm version](https://img.shields.io/npm/v/@zakkster/lite-bake-stream.svg?style=for-the-badge&color=latest)](https://www.npmjs.com/package/@zakkster/lite-bake-stream)
 [![sponsor](https://img.shields.io/badge/sponsor-PeshoVurtoleta-ea4aaa.svg?logo=github)](https://github.com/sponsors/PeshoVurtoleta)
 ![Zero-GC](https://img.shields.io/badge/Zero--GC-Engine-00C853?style=for-the-badge&logo=leaf&logoColor=white)
@@ -11,26 +13,257 @@
 ![Dependencies](https://img.shields.io/badge/dependencies-0-brightgreen)
 [![license](https://img.shields.io/badge/license-MIT-blue?style=flat-square)](./LICENSE)
 
-> Streaming byte-level JSON compiler for [`@zakkster/lite-bake`](https://github.com/PeshoVurtoleta/lite-bake). Zero-GC, tree-shakeable, gigabyte-scale.
+**Status:** v1.6.1. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.
+
+## The gigabyte-scale JSON front door the ecosystem was missing
+
+You have a gigabyte of JSON and a program that has to read it. `JSON.parse`
+cannot get you there: V8 caps a single string near 512 MB, and well under that
+ceiling the intermediate object graph shreds your heap. This package is the
+front door -- a chunk-safe UTF-8 JSON SAX scanner that reads bytes off a stream
+and emits a flat, seekable LBK1 binary container, without ever materializing the
+object graph.
 
-**Status:** v1.5.0. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.
+```bash
+npm install @zakkster/lite-bake-stream
+```
 
-## Two modes, one API
+```js
+import { serialize, deserialize } from '@zakkster/lite-bake-stream';
+
+const container = serialize(ndjsonBytes);
+const reader = deserialize(container);
+reader.get(0, 'id');                     // 1
+reader.findShards('id', { min: 500 });   // shard pruning via zone maps
+```
 
-- **Schema mode** (default): F64/U32 lane packing, zone-maps query pruning, byte-exact preservation for flat records. Millions of rows/sec random access, zero-GC hot path.
-- **Preserve mode**: `serialize(input, { preserve: true })`. Opaque byte blobs — the module doesn't crack open records, it just moves them intact. Deeply nested API JSON, arrays, mixed types — any valid JSON round-trips byte-for-byte identical. Reader exposes `getBytes(i)` (zero-alloc), `getString(i)`, `getJSON(i)`.
+## Table of contents
 
-`deserialize()` auto-dispatches on the container's flag bit. If you're compiling gigabytes of API responses with 2-3 levels of nesting, preserve mode is what you want. If the data is flat and you want columnar-style range queries, schema mode is what you want.
+- [Why this exists](#why-this-exists)
+- [What you get](#what-you-get)
+- [The core surface](#the-core-surface)
+- [API reference](#api-reference)
+- [Composability](#composability)
+- [Zero-GC design notes](#zero-gc-design-notes)
+- [Design decisions worth knowing](#design-decisions-worth-knowing)
+- [Testing](#testing)
+- [What this is not](#what-this-is-not)
+- [Ecosystem](#ecosystem)
+- [License](#license)
 
 ## Why this exists
 
-`lite-bake` compiles a JS array into a flat binary buffer. That's the destination format you want. But if your input is a gigabyte of JSON, you can't call `JSON.parse` on it — V8's string limit is ~512 MB, and even under that, the intermediate object graph will trash the heap and destroy your process before `bake()` ever runs.
+The enemy is `JSON.parse`, not any one downstream format. Two hard walls stop it
+on large input:
+
+- **The string ceiling.** V8 refuses to hold a single string beyond roughly
+  512 MB. A gigabyte file cannot even be read into memory as text, let alone
+  parsed.
+- **Object-graph heap death.** Even under the ceiling, `JSON.parse` allocates a
+  fully-realized object graph -- every object, array, string, and boxed number.
+  For a gigabyte of records that graph is many gigabytes of live heap, and the
+  garbage collector spends the run trying (and failing) to keep up.
+
+`lite-bake-stream` never builds the graph. It scans bytes as they arrive, packs
+each record into typed-lane shard buffers (or preserves it as an opaque blob),
+and writes an LBK1 container whose Reader gives you random access without
+re-parsing. The input-size ceiling becomes your disk, not your heap.
+
+## What you get
+
+Two ingest modes share one top-level API. `deserialize()` auto-dispatches on the
+container's flag bit, so a consumer never has to know which mode wrote the bytes.
+
+- **Schema mode** (default): F64/U32 lane packing, per-shard zone-maps for query
+  pruning, byte-exact preservation for flat records. Random access to millions
+  of rows per second on a zero-GC hot path.
+- **Preserve mode** (`serialize(input, { preserve: true })`): opaque byte blobs.
+  The module does not crack open records, it moves them intact. Deeply nested
+  API JSON, arrays, mixed types -- any valid JSON round-trips byte-for-byte
+  identical. The reader exposes `getBytes(i)` (zero-alloc view), `getString(i)`,
+  and `getJSON(i)`.
+
+Compiling gigabytes of API responses with two-to-three levels of nesting?
+Preserve mode. Flat records that want columnar-style range queries? Schema mode.
+
+## The core surface
+
+<details>
+<summary>Streaming emission, bounded memory, and integrity</summary>
+
+`finalize()` returns the whole container as one `ArrayBuffer` -- peak memory is
+`O(container)`. For bounded-memory output, `writer.finalizeToSink(sink, { layout })`
+emits to a caller sink and returns `{ totalRows, shardCount, schema | mode,
+bytesWritten, layout }` (no buffer). A sink is any object with a synchronous
+`write(bytes)`; `layout: 'stream'` also needs `writeAt(bytes, position)` for one
+header backpatch. The contract is satisfiable by `fs.write(fd, buf, 0, len, pos)`
+and the File System Access API's `createWritable().write({ type: 'write',
+position })`; the test suite uses an in-memory sink of the same shape. No sink
+class is exported -- the contract is public, the convenience implementation is
+internal.
+
+Two ways to drive a stream, with different peaks:
+
+- **Two-step, bounded RAM** -- call `writer.beginStream(sink, { layout: 'stream' })`
+  BEFORE feeding, feed the tokenizer, then `writer.finalizeToSink(sink, { layout:
+  'stream' })`. Each shard is written to the sink as it finalizes and its bytes
+  are dropped, so peak memory is `2*targetShardBytes + shardCount*(40 + 16*T) +
+  schemaBlockBytes` -- `O(targetShardBytes + directory)`, never `O(container)`.
+  This is the mode the 500 MB gate proves.
+- **One-shot, buffered** -- call `writer.finalizeToSink(sink, opts)` alone (no
+  `beginStream`). Correct and simplest, but the shards are buffered first, so
+  peak is `containerBytes + sum(shard bytes)` -- `O(container)`, the same as
+  `finalize()`.
+
+`layout: 'prefix'` is the classic layout, byte-identical to `finalize()`.
+`layout: 'stream'` places the schema/directory/zone-map trailer and footer after
+the payloads with one header backpatch; a default-emitted stream container is a
+legal v1 container (`format_version` stays 1) that every shipped reader,
+`checkContainer`, and `mergeContainers` accept. A malformed sink (non-object,
+missing `write`, missing `writeAt` for stream, or an async/thenable return)
+throws `W_BAD_SINK`; a sink that throws mid-emission fails the writer closed and
+rethrows the source error verbatim (a retry then hits `W_FINALIZED`).
+
+**Optional CRC-32C** (Castagnoli, table-driven, zero deps). Opt in with
+`{ crc: true }` on the writer (or serialize `writer` opts); coverage is
+`[0, footer_off)`, folded per emitted chunk. Readers expose `verifyCrc()` ->
+`'ok' | 'absent'` (a mismatch throws `R_BAD_CRC`) on `Reader`/`PreserveReader`
+(sync) and `RangeReader` (async). The open option `{ verifyCrc: true }` (also
+`deserialize(bytes, opts)`) fails closed on both mismatch (`R_BAD_CRC`) and
+absence (`R_CRC_ABSENT`) -- `null` is not zero. `0xFFFFFFFF` means absent and
+stays legal. `mergeContainers` recomputes the CRC iff every input carried one,
+else emits absent. See `decisions/0009-streaming-emission.md`.
+
+Future lane kinds and payload modes (an I64 exact-integer lane, a columnar
+payload mode, a container-level string table) land via the format's
+forward-compat seams -- `min_reader_version` on ShardEntry, reserved
+FieldDescriptor flags, the `metadata_off` block wrapper -- without a
+`format_version` bump.
+
+</details>
+
+## API reference
+
+Every subpath is a standalone import; the browser reader never pulls the writer.
+Every subpath also exports a `VERSION` const.
+
+**Root** (`@zakkster/lite-bake-stream`)
+
+- `serialize(input, opts?) -> Uint8Array` -- ingest bytes into an LBK1 container.
+  `opts.preserve` selects preserve mode; `opts.writer` forwards writer options.
+- `deserialize(bytes, opts?) -> Reader | PreserveReader` -- auto-dispatch on the
+  container flag bit. Plus class re-exports and `StringTableError`.
+
+**Schema mode**
+
+- `Tokenizer` (`/tokenizer`) -- chunk-safe UTF-8 JSON SAX scanner. `feed(chunk)`,
+  `end()`, sink callbacks fire synchronously.
+- `Writer` (`/writer`) -- LBK1 shard emitter; plugs into the Tokenizer. F64/U32
+  lanes, per-shard string tables and zone maps, `beginStream`/`finalizeToSink`.
+- `Reader` (`/reader`) -- container parser, sync, zero-alloc `get(row, field)`
+  plus zone-map query APIs `shardBounds(shardIdx, field)` and
+  `findShards(field, range)`.
+- `RangeReader` (`/range-reader`) -- HTTP Range lazy shard loading; synchronous
+  query pruning after open.
+- `MultiReader` (`/multi-reader`) -- logical union over N Readers sharing a
+  schema.
+- `splitNDJSON`, `compilePart`, `compileInParts`, `mergeContainers` (`/split`) --
+  worker-agnostic split, compile, and merge.
+
+**Preserve mode**
+
+- `PreserveTokenizer` (`/preserve-tokenizer`) -- NDJSON record-boundary scanner
+  with JSON-aware depth tracking, chunk-safe.
+- `PreserveWriter` (`/preserve-writer`) -- opaque byte-blob sink, pre-allocated
+  shard buffer, zero-GC record path.
+- `PreserveReader` (`/preserve-reader`) -- `getBytes(i)` (zero-alloc view),
+  `getString(i)`, `getJSON(i)`.
+
+**Shared**
+
+- `StringTable` (`/string-table`) -- byte-level UTF-8 interning primitive.
+- `ingestStream(readableStream, opts?)`, `ingestFile(file, opts?)`
+  (`/file-ingest`) -- browser helpers piping a `ReadableStream<Uint8Array>` (e.g.
+  `File.stream()`) through the Tokenizer + Writer, returning a Reader. The
+  per-chunk `onProgress` callback's state object is reused across calls (mutated
+  in place, zero per-chunk allocation) -- copy it if retained past the callback.
+
+**Zero-copy view contract.** A byte view returned by `getBytes(i)` (and any
+shard raw-access getter) is a plain `subarray` over the container's
+`ArrayBuffer`: it pins that buffer for as long as the view is reachable. Copy the
+bytes out (`getBytes(i).slice()`) if you need the container itself to be
+collectable.
+
+### Constants
+
+Lane kinds (SPEC 4.2):
+
+| Kind | Name  | Bytes |
+| ---: | :---- | ----: |
+| `1`  | `F64` | 8     |
+| `2`  | `F32` | 4     |
+| `3`  | `U32` | 4     |
+| `4`  | `U8`  | 1     |
+
+Ceilings:
+
+| Limit                                | Value        |
+| :----------------------------------- | -----------: |
+| String-table blob bytes (u32)        | `4294967295` |
+| String-table entry count             | `4294967294` |
+| u64 offset safe `Number()` cast      | `2^53 - 1`   |
+
+Error-code families (thrown with a stable `code`; the full inventory is pinned by
+the torture gate):
+
+| Prefix | Class                                                              |
+| :----- | :----------------------------------------------------------------- |
+| `E_`   | `TokenizerError` (e.g. `E_NUMBER_OVERFLOW`)                        |
+| `W_`   | `WriterError` (`W_MIXED_LANE_TYPES`, `W_LANE_MISMATCH`, `W_BAD_SINK`, `W_FINALIZED`) |
+| `R_`   | `ReaderError` / `RangeReaderError` (`R_ROW_OUT_OF_RANGE`, `R_OFFSET_TOO_LARGE`, `R_BAD_CRC`, `R_CRC_ABSENT`, ...) |
+| `M_`   | `MultiReaderError` (`M_ROW_OUT_OF_RANGE`)                          |
+| `S_`   | `SplitError`                                                       |
+| `ST_`  | `StringTableError` (`ST_BLOB_OVERFLOW`)                            |
+| `P*_`  | `PreserveTokenizerError` / `PreserveWriterError` / `PreserveReaderError` |
+
+## Composability
+
+The `/split` subpath turns one logical dataset into worker-parallel or
+checkpointed parts, and the readers merge them back into a single view:
+
+```js
+import { splitNDJSON, compileInParts, mergeContainers } from '@zakkster/lite-bake-stream/split';
+import { RangeReader } from '@zakkster/lite-bake-stream/range-reader';
+
+// 1. split NDJSON into byte-aligned parts (record boundaries preserved)
+const parts = splitNDJSON(ndjsonBytes, { partBytes: 64 * 1024 * 1024 });
+
+// 2. compile each part (fan out to workers if you like), then merge
+const containers = compileInParts(parts);
+const merged = mergeContainers(containers);
 
-`lite-bake-stream` is the front door: a chunk-safe UTF-8 JSON SAX scanner that reads bytes off a stream and emits `lite-bake`-shaped output shards, without ever materializing the object graph.
+// 3. open the merged container and prune shards before fetching them
+const reader = await RangeReader.open(adapterOver(merged));
+const shards = reader.findShards('id', { min: 500, max: 999 });
+for (const s of shards) reader.get(s.firstRow, 'id');
+```
 
-## Qualified at 8 GB
+## Zero-GC design notes
 
-Overnight soak, M1 MacBook Pro, both release gates armed:
+<details>
+<summary>Allocation table and the 8 GB soak</summary>
+
+The hot path allocates nothing steady-state. Buffers are pre-allocated and
+reused across the churn:
+
+| Operation             | Steady-state allocation |
+| :-------------------- | :---------------------- |
+| `feed(chunk)`         | 0 (views into the tokenizer's internal buffer) |
+| per row               | 0 (typed-lane staging is pre-allocated) |
+| per shard roll        | 0 (working buffers reused across rolls) |
+| `Reader.get(row, f)`  | 0 for F64; `U32` string fields resolve through a cached decoder |
+
+**Qualified at 8 GB.** Overnight soak, M1 MacBook Pro, both release gates armed:
 
 | | |
 | :-- | --: |
@@ -44,107 +277,73 @@ Overnight soak, M1 MacBook Pro, both release gates armed:
 | Cells verified | 590,206,212 |
 | **Preservation mismatches** | **0** |
 
-**499 KB of heap allocation to compile 8 GB of JSON.** Total, for the whole run — not per shard, not per second. Zero garbage collections of any kind. Every one of 590 million declared cells round-tripped byte-exact.
+499 KB of heap allocation to compile 8 GB of JSON -- total, for the whole run,
+not per shard, not per second. Zero garbage collections of any kind. Every one of
+590 million declared cells round-tripped byte-exact.
 
-Tokenizer throughput is ~55% of `JSON.parse` (237 MB/s vs 416 MB/s on a 50 MB fixture). That's the trade: you give up some raw speed and you get an input size ceiling that's bounded by your disk instead of your heap.
+**Bounded streaming, measured.** The full-tier gate drives the two-step bounded
+path, streaming 500 MB in 8 MiB chunks into a counting sink, and asserts
+`peakRss - baselineRss <= 4*targetShardBytes + 64*shardCount + 64 MiB` -- measured
+32.1 MiB against the 96.0 MiB bound at 41 shards (the buffered prefix path is
+~704 MiB at the same 500 MB).
 
-## Quickstart
+Tokenizer throughput is ~55% of `JSON.parse` (237 MB/s vs 416 MB/s on a 50 MB
+fixture). That is the trade: you give up some raw speed and you get an input-size
+ceiling bounded by your disk instead of your heap.
 
-**Schema mode** (flat records, columnar queries):
+</details>
 
-```js
-import { serialize, deserialize } from '@zakkster/lite-bake-stream';
+## Design decisions worth knowing
 
-const container = serialize(ndjsonBytes);
-const reader = deserialize(container);
-reader.get(0, 'id');                     // 1
-reader.findShards('id', { min: 500 });   // shard pruning via zone maps
-```
-
-**Preserve mode** (any JSON shape, byte-exact passthrough):
+Each call is an on-disk record under `decisions/`:
 
-```js
-import { serialize, deserialize } from '@zakkster/lite-bake-stream';
-
-// Handles arbitrarily nested API JSON
-const container = serialize(apiResponseBytes, { preserve: true });
-const reader = deserialize(container);
-reader.getBytes(0);    // Uint8Array view, zero allocation
-reader.getString(0);   // TextDecoder allocation
-reader.getJSON(0);     // full JSON.parse, allocates the parsed value graph
-```
+- [0001](decisions/0001-reserved-empty-string.md) -- reserve string-table index 0 as the empty string.
+- [0002](decisions/0002-record-shape-policy.md) -- depth-0 non-object records are refused, not dropped.
+- [0003](decisions/0003-duplicate-field-names.md) -- duplicate schema field names are refused at freeze.
+- [0004](decisions/0004-container-doors-and-lying-metadata.md) -- a container is verified at the door; a lying pointer is corruption.
+- [0005](decisions/0005-one-unknown-field-policy.md) -- absent names are refused, untracked names fall back.
+- [0006](decisions/0006-utf8-door.md) -- the UTF-8 door is documented-permissive, not a tokenizer gate.
+- [0007](decisions/0007-sample-window-bytes.md) -- the sample window is byte-true, not record-count.
+- [0008](decisions/0008-null-policy.md) -- JSON null is lane-neutral.
+- [0009](decisions/0009-streaming-emission.md) -- streaming emission, sinks, and optional CRC-32C.
+- [0010](decisions/0010-preserve-mode.md) -- why a second (preserve) mode exists at all.
+- [0011](decisions/0011-sample-drain-reintern.md) -- per-shard re-intern on drain and the byte-true sample window.
+- [0012](decisions/0012-clinger-fast-path.md) -- the Clinger fast path and the two-tier F64 guarantee.
 
-For streaming ingest with progress, use `/file-ingest`. For HTTP Range-fetched containers (schema mode), `/range-reader`. For joining multiple containers into one logical view (schema mode), `/multi-reader`. For parallel or checkpointed compilation (schema mode), `/split`.
+## Testing
 
-## What shipped in 1.0
-
-| Milestone | Deliverable                                                                 |
-| :-------- | :-------------------------------------------------------------------------- |
-| ✅ **M0** | LBK1 container format spec, forward-compat seams for the I64 lane.          |
-| ✅ **M1** | Chunk-safe tokenizer; zero-GC on the hot path after warmup.                 |
-| ✅ **M2** | Schema-driven writer; sample-and-infer path; typed-lane shard buffers.      |
-| ✅ **M3** | Per-shard UTF-8 string table with pre-alloc typed-array hash-map interning. |
-| ✅ **M4** | `serialize()`/`deserialize()`, MultiReader, full type declarations.         |
-| ✅ **M5** | Browser reader, HTTP Range fetch, `File.stream()` client-side ingest.       |
-| ✅ **M6** | NDJSON splitter + per-part compilation + container merge.                   |
-| ✅ **M7** | Per-shard zone maps (min/max per F64 column) + query-pruning APIs.          |
-| ✅ **M8** | RFC 8259 conformance corpus + robustness fuzz.                              |
-
-## Roadmap beyond 1.0
-
-All of these fit the format's existing forward-compat seams — `format_version` stays at `1`.
-
-| | |
-| :-- | :-- |
-| **I64 lane** | Exact 64-bit integer preservation. FieldDescriptor flags are already reserved for it; shards using it set `min_reader_version: 2`. |
-| **Columnar payload mode** | Column-contiguous shard payloads instead of interleaved rows. Better compression, SIMD-friendly scans. Opt-in via a shard flag. |
-| **Container-level string table** | Cross-shard interning for high-repetition corpora. Lands in the `metadata_off` block alongside zone maps. |
-| **Oscilloscope-rack demo** | Reel-to-reel tape transport at actual MB/s, waterfall spectrum per shard, VU needles, seven-segment counters. Locked 60fps on an iPhone 7 with 100 MB NDJSON. |
-
-## What's here today
-
-- `SPEC.md` — LBK1 container spec draft 0.1, per-shard string table (§3.3), preservation contract (§7).
-- `src/index.js` — top-level convenience API: `serialize(input)` and `deserialize(bytes)` plus class re-exports.
-- `src/Tokenizer.js` — chunk-safe SAX tokenizer with Clinger's fast-path decimal→F64 conversion.
-- `src/Writer.js` — LBK1 shard emitter with F64 and U32 lanes, sample-and-infer schema, per-shard string tables, per-shard zone maps.
-- `src/Reader.js` — LBK1 container parser with typed-array views, transparent string-field resolution, and zone-map query APIs.
-- `src/StringTable.js` — open-addressed byte-level interning with zero-alloc hit path.
-- `src/RangeReader.js` — LBK1 reader with lazy shard loading via an IO adapter (HTTP Range or in-memory mock); zone maps loaded once at open enable synchronous query pruning.
-- `src/MultiReader.js` — union view over N Reader instances with matching schemas; merged shard indices for query pruning across containers.
-- `src/Split.js` — NDJSON splitter, per-part compilation, and container merge for worker-parallel or checkpointed ingest.
-- `src/FileIngest.js` — browser helper: `File.stream()` → Reader. Any `ReadableStream<Uint8Array>` works.
-- `types/*.d.ts` — 8 hand-written TypeScript declaration files, one per subpath.
-- `demo/` — single-file HTML + module JS. Ingest a JSON file → LBK1 → browse in `@zakkster/lite-table`. Zone-maps query-pruning panel visualizes the fetch-set reduction. `npm run demo`.
-- `test/*.test.js` — 233 tests across 12 files, including a 66-fixture RFC 8259 conformance corpus and a 7-scenario robustness fuzz (~2,300 tokenizer invocations per run).
-- `test/_verify.js` — universal round-trip verifier used by tests, torture, and soak.
-- `bench/bench-tokenizer.js` — interleaved-rep bench vs `JSON.parse`.
-- `bench/torture.js` — 37-scenario CI harness. Two release gates: `maxMajor: 0` (zero-GC) AND `verify: true` (value-exact round-trip on 77,500+ rows per run).
-- `bench/soak.js` — CLI-configurable soak with the preservation gate wired in.
-
-Two release gates. Zero major GC AND every declared cell round-trips exactly. If either fails, no publish.
-
-## Release testing tiers
-
-The gates run at four scales, chosen to fit different hardware and time budgets:
+The gates run at scales chosen to fit different hardware and time budgets:
 
 | Command | Scale | Use |
 | :-- | :-- | :-- |
-| `npm test` | 96 tests | Dev loop, every save |
-| `npm run torture` | 37 scenarios (~1 MB each) | Before every commit |
-| `npm run torture:full` | + 500 MB full-tier soak scenarios | Before every publish |
+| `npm test` | 519 tests across 36 files | Dev loop, every save |
+| `npm run torture` | 44 fast scenarios (~1 MB each) | Before every commit |
+| `npm run torture:full` | 47 scenarios incl. the 500 MB soak | Before every publish |
 | `npm run soak` | 100 MB with preservation gate | Sanity check |
 | `npm run soak:500` | 500 MB | Pre-publish scale check |
-| `npm run soak:1gb` | 1 GB | Real-hardware qualification (M4 Pro etc.) |
+| `npm run soak:1gb` | 1 GB | Real-hardware qualification |
 | `npm run soak:overnight` | 8 GB | Overnight `caffeinate` run |
 
-Custom sizes:
-```
-node --expose-gc bench/soak.js --gb=4 --verify
-caffeinate node --expose-gc bench/soak.js --gb=16 --verify   # macOS overnight
-```
+The test tree includes a 66-fixture RFC 8259 conformance corpus, a property-based
+robustness fuzz, and two structural guards: an ASCII-law scanner
+(`test/AsciiLaw.test.js`) and an API-surface drift guard
+(`test/ApiSurface.test.js`), each also wired into the torture tier with an armed
+control that proves it can fail. Two release gates decide a publish: zero major
+GC AND every declared cell round-trips exactly. If either fails, no publish.
+
+## What this is not
+
+No JSON5, JSONC, comments, or trailing commas. No streaming field updates, no
+compression, no native (non-JS) producers. LBK1 is this package's own container
+format; non-JSON producers are welcome as long as they emit a well-formed
+container.
+
+## Ecosystem
 
-Every soak run reports ingest throughput (MB/s), verification throughput (rows/s), the zero-GC gate result, container compression ratio, and progress lines so long runs are observable. The preservation gate proves — cell by cell — that every declared field of every row round-trips exactly.
+The `demo/` directory is a single-file HTML + module-JS app: ingest a JSON file,
+compile it to LBK1, and browse the result, with a zone-maps panel that visualizes
+the query-pruning fetch-set reduction. Run it with `npm run demo`.
 
 ## License
 
-MIT © 2026 Zahary Shinikchiev
+MIT (c) 2026 Zahary Shinikchiev

package/SPEC.md

@@ -1,6 +1,6 @@
 # LBK1 Format Specification
 
-> Version 1 — frozen as of `@zakkster/lite-bake-stream` v1.0.0.
+> Version 1 -- frozen as of `@zakkster/lite-bake-stream` v1.0.0.
 > Future lane kinds and payload modes land via the forward-compat seams
 > (`min_reader_version`, reserved FieldDescriptor flags, the `metadata_off`
 > block wrapper) without a `format_version` bump.
@@ -10,7 +10,7 @@
 
 LBK1 is the on-disk / on-wire container for streaming-compiled `lite-bake` datasets. It carries one or more shards of interleaved binary rows plus the metadata needed to reconstruct a Reader over the whole dataset without re-parsing JSON.
 
-The tokenizer specified in section 5 is the reference producer path from UTF-8 JSON (top-level array or NDJSON) into an LBK1 container. Non-JSON producers are allowed as long as they emit a well-formed container.
+The tokenizer specified in section 5 is the canonical producer path from UTF-8 JSON (top-level array or NDJSON) into an LBK1 container. Non-JSON producers are allowed as long as they emit a well-formed container.
 
 Numbers on the hot path are F64. Section 4.3 reserves the schema-level bit that will admit a future `I64_BYTES_PRESERVED` lane without a container format-version bump.
 
@@ -22,13 +22,13 @@ These are load-bearing; every other decision follows from them.
 2. **Forward-compatible via strict decoders.** Unknown flags or lane kinds are rejected, not ignored. Silent corruption is the failure mode we refuse.
 3. **Shard-scoped versioning.** A shard header carries the minimum reader version required to decode that shard. Old readers refuse new shards cleanly; new readers read old shards forever.
 4. **Zero-copy where the CPU allows it.** Row payloads are aligned so that typed-array views over the shard buffer read fields with native alignment.
-5. **Zero runtime dependencies.** Reference producer and reader share the `@zakkster` no-dep constraint.
+5. **Zero runtime dependencies.** Producer and reader share the `@zakkster` no-dep constraint.
 
 ## 2. Notation
 
-- `u8`, `u16`, `u32`, `u64`, `f32`, `f64` — little-endian unless stated.
-- `bytes(n)` — a raw byte run of length n.
-- `varlen_utf8` — u32 length prefix followed by that many UTF-8 bytes.
+- `u8`, `u16`, `u32`, `u64`, `f32`, `f64` -- little-endian unless stated.
+- `bytes(n)` -- a raw byte run of length n.
+- `varlen_utf8` -- u32 length prefix followed by that many UTF-8 bytes.
 - Offsets are relative to the start of the container unless stated.
 - All multi-byte integers in headers are little-endian. Row payloads use native endianness with a byte-order marker in the container header (section 3.1); non-native containers require a byteswap on load or must be rejected.
 
@@ -84,7 +84,7 @@ Padded with zero bytes so the block ends on an 8-byte boundary.
 There are two string-table locations in an LBK1 container:
 
 - **Container-level** at `string_table_off` (v1 reserved; not written by the current Writer). Reserved for future container-wide interning across shards.
-- **Per-shard** at each ShardEntry's `local_string_off` (M3, implemented). Every shard with U32-lane fields has its own local string table so shards remain independently decodable — critical for HTTP Range fetches (M5+) and worker-parallel readers (M6+).
+- **Per-shard** at each ShardEntry's `local_string_off` (M3, implemented). Every shard with U32-lane fields has its own local string table so shards remain independently decodable -- critical for HTTP Range fetches (M5+) and worker-parallel readers (M6+).
 
 Both use the same on-disk layout:
 
@@ -144,14 +144,14 @@ f64  maxes[shard_count * tracked_field_count]  Same layout.
 ```
 
 - `tracked_field_count` MAY be less than the schema's F64 field count (the writer chooses which fields to track). For fields not present in `field_indices`, `shardBounds` returns `null`.
-- `mins[s*T + t]` and `maxes[s*T + t]` are the exact min/max of the tracked field's values across the rows in shard `s`. For shards where the field is missing (defaults to `0`), the min/max collapse to `[0, 0]` — still valid input to range predicates.
+- `mins[s*T + t]` and `maxes[s*T + t]` are the exact min/max of the tracked field's values across the rows in shard `s`. For shards where the field is missing (defaults to `0`), the min/max collapse to `[0, 0]` -- still valid input to range predicates.
 - Zone maps segment is placed between the shard directory and the first shard payload. This keeps it inside a single Range fetch alongside the header/schema/directory during Reader initialization.
 
 Reader query APIs (see `Reader.js`):
 
 ```
-reader.shardBounds(shardIdx, fieldName)     → {min, max} | null
-reader.findShards(fieldName, {min, max})    → shardIdx[]     // shards that MAY contain the range
+reader.shardBounds(shardIdx, fieldName)     -> {min, max} | null
+reader.findShards(fieldName, {min, max})    -> shardIdx[]     // shards that MAY contain the range
 ```
 
 `findShards` returns exactly the shards whose bounds overlap `[min, max]`; a shard whose bounds are disjoint is skipped, saving one range fetch per skipped shard in the RangeReader path.
@@ -165,7 +165,9 @@ reader.findShards(fieldName, {min, max})    → shardIdx[]     // shards that MA
 |      8 |    4 | `magic_end` | ASCII `1 K B L` = `0x31 0x4B 0x42 0x4C`.                  |
 |     12 |    4 | `footer_len`| u32. Currently `16`. Lets future versions grow the footer without breaking magic detection. |
 
-CRC is optional in M4; producers that omit it write `0xFFFFFFFF`.
+CRC is optional; producers that omit it write `0xFFFFFFFF` (absent). Producers opt in with the writer option `{ crc: true }`. The CRC-32C uses the Castagnoli polynomial (bit-reflected `0x82F63B78`); check value `CRC32C("123456789") = 0xE3069283`. Readers verify with `verifyCrc()` (returns `'ok' | 'absent'`, throws `R_BAD_CRC` on mismatch) or the open option `{ verifyCrc: true }`, which additionally throws `R_CRC_ABSENT` when the caller demanded verification but the CRC is absent.
+
+Note (SPEC 3 / 3.6): container offsets are ABSOLUTE, so two conformant orderings exist for a v1 container. The classic PREFIX ordering places the schema block, shard directory, and zone maps before the shard payloads. The STREAMING ordering (`layout: 'stream'`) places the shard payloads first, then the schema block, shard directory, and zone maps as a trailer, then the footer; the schema block still immediately precedes the shard directory (adjacency preserved). Both are valid v1 containers; format_version is unchanged.
 
 ## 4. Schema and field descriptors
 
@@ -189,12 +191,12 @@ Fixed-width descriptors let a Reader load the schema with a single `Uint32Array`
 
 | Value | Name    | Bytes | Notes                                                                    |
 | ----: | ------- | ----: | ------------------------------------------------------------------------ |
-|   `1` | `F64`   |     8 | The only numeric lane in M4. Matches lite-bake's `Types.F64`.            |
+|   `1` | `F64`   |     8 | The only numeric lane in M4.                                            |
 |   `2` | `F32`   |     4 | Reserved for future producer opt-in via schema override.                 |
 |   `3` | `U32`   |     4 | String-table index (M3+). Row cell holds a u32 index into the shard's `local_string_off` table. |
 |   `4` | `U8`    |     1 | Reserved.                                                                |
 
-Values 5..127 reserved for future lite-bake type lanes (matching `Types.*`).
+Values 5..127 reserved for future lane kinds defined by this spec.
 
 Values 128..255 reserved for stream-specific lanes; see 4.4.
 
@@ -202,10 +204,10 @@ Values 128..255 reserved for stream-specific lanes; see 4.4.
 
 Currently all bits are reserved and must be zero. Reserved bit assignments (v2+) will use:
 
-- Bit 0: `LANE_KIND_EXT` — if set, the field uses an extended lane; consult 4.4.
+- Bit 0: `LANE_KIND_EXT` -- if set, the field uses an extended lane; consult 4.4.
 - Bits 1..7: unassigned.
 
-This is the seam Option (c) — `I64_BYTES_PRESERVED` — slides into without a container format-version bump. A shard using it sets `min_reader_version = 2` at the directory level; v1 readers refuse the shard, not the whole container.
+This is the seam Option (c) -- `I64_BYTES_PRESERVED` -- slides into without a container format-version bump. A shard using it sets `min_reader_version = 2` at the directory level; v1 readers refuse the shard, not the whole container.
 
 ### 4.4 Extended lane kinds (reserved for v2)
 
@@ -218,16 +220,16 @@ When `flags` bit 0 is set on a FieldDescriptor, `lane_kind` is interpreted from
 
 All other values in the extended range are reserved and must be rejected.
 
-## 5. Reference producer: tokenizer contract
+## 5. Producer: tokenizer contract
 
-The M1 tokenizer is a chunk-safe UTF-8 JSON SAX scanner. It does not itself write LBK1 shards — that is the M2 writer's job. This section specifies the contract between them so M1 can be built and benched in isolation.
+The M1 tokenizer is a chunk-safe UTF-8 JSON SAX scanner. It does not itself write LBK1 shards -- that is the M2 writer's job. This section specifies the contract between them so M1 can be built and benched in isolation.
 
 ### 5.1 Input framing
 
 Two top-level modes, auto-detected by the first non-whitespace byte:
 
-- `[` — Top-level JSON array. Records are the array's elements.
-- Any other JSON value byte (`{`, `"`, `-`, `0`..`9`, `t`, `f`, `n`) — NDJSON. Records are separated by JSON whitespace (`0x20`, `0x09`, `0x0A`, `0x0D`).
+- `[` -- Top-level JSON array. Records are the array's elements.
+- Any other JSON value byte (`{`, `"`, `-`, `0`..`9`, `t`, `f`, `n`) -- NDJSON. Records are separated by JSON whitespace (`0x20`, `0x09`, `0x0A`, `0x0D`).
 
 Records themselves must be JSON objects for the reference writer path. Non-object records are accepted by the tokenizer (it is generic) but rejected by the writer.
 
@@ -265,10 +267,10 @@ For a string that spans multiple chunks, the tokenizer accumulates into an inter
 
 #### 5.4.1 F64 preservation contract (M3)
 
-The tokenizer implements Clinger's fast-path decimal→F64 conversion with an exact `POW10[0..22]` table. This gives a two-tier preservation guarantee that consumers can rely on:
+The tokenizer implements Clinger's fast-path decimal->F64 conversion with an exact `POW10[0..22]` table. This gives a two-tier preservation guarantee that consumers can rely on:
 
-- **Fast path (guaranteed correctly-rounded):** any numeric literal whose decimal representation has ≤ 15 significant digits AND absolute net exponent ≤ 22 produces the SAME IEEE 754 double as JavaScript's `Number(str)`. This covers all integer IDs within `Number.MAX_SAFE_INTEGER`, all "typical" decimals (prices, ratios, timestamps, coordinates), and scientific notation within `1e±22`. This is the domain the round-trip verifier (`test/_verify.js`) asserts bit-exact across every torture scenario.
-- **Slow path (may drift ≤ 1 ULP):** literals outside the fast path (16+ significant digits, or `|exponent| > 22`) fall through to a naive `intPart + fracPart/fracDiv * Math.pow(10, exp)` computation. The result is within 1 ULP of the correctly-rounded value. Correctly-rounded parsing across the full F64 domain (David Gay's `strtod` or equivalent) is planned for M4+.
+- **Fast path (guaranteed correctly-rounded):** any numeric literal whose decimal representation has <= 15 significant digits AND absolute net exponent <= 22 produces the SAME IEEE 754 double as JavaScript's `Number(str)`. This covers all integer IDs within `Number.MAX_SAFE_INTEGER`, all "typical" decimals (prices, ratios, timestamps, coordinates), and scientific notation within `1e+/-22`. This is the domain the round-trip verifier (`test/_verify.js`) asserts bit-exact across every torture scenario.
+- **Slow path (may drift <= 1 ULP):** literals outside the fast path (16+ significant digits, or `|exponent| > 22`) fall through to a naive `intPart + fracPart/fracDiv * Math.pow(10, exp)` computation. The result is within 1 ULP of the correctly-rounded value. Correctly-rounded parsing across the full F64 domain (David Gay's `strtod` or equivalent) is planned for M4+.
 
 The boundary is exercised explicitly by `test/NumericBoundary.test.js`, which pins expected drift so any regression is caught.
 
@@ -294,10 +296,10 @@ Subpath entries carved so consumers pay only for what they import. Each entry is
 | `@zakkster/lite-bake-stream/writer` | LBK1 shard emitter. Explicit or sample-and-infer schema. Emits per-shard zone maps. |
 | `@zakkster/lite-bake-stream/reader` | LBK1 parser over an in-memory `ArrayBuffer`. Synchronous. Zone-maps query APIs. |
 | `@zakkster/lite-bake-stream/string-table` | Byte-level UTF-8 interning primitive. |
-| `@zakkster/lite-bake-stream/file-ingest` | Browser helper: `File.stream()` → Reader. Any `ReadableStream<Uint8Array>` works. |
+| `@zakkster/lite-bake-stream/file-ingest` | Browser helper: `File.stream()` -> Reader. Any `ReadableStream<Uint8Array>` works. |
 | `@zakkster/lite-bake-stream/range-reader` | LBK1 parser with lazy shard loading via an IO adapter. HTTP Range fetch or in-memory mock. Synchronous zone-maps pruning. |
 | `@zakkster/lite-bake-stream/multi-reader` | Logical union over N Reader instances. Rows and shards cumulatively addressed; findShards merges across sub-readers. |
-| `@zakkster/lite-bake-stream/split` | NDJSON splitter + per-part compile + container merge. Executor-agnostic — wire to any worker pool or run serially. |
+| `@zakkster/lite-bake-stream/split` | NDJSON splitter + per-part compile + container merge. Executor-agnostic -- wire to any worker pool or run serially. |
 
 Every entry works in browsers (evergreen Chromium / Firefox / Safari) and Node 18+; no Node-specific APIs are used in the source. Every subpath has a matching `.d.ts` file in `types/` for TypeScript consumers.
 
@@ -328,11 +330,11 @@ Not currently supported in preserve mode: `RangeReader` (HTTP-Range lazy loading
 
 ## 7. Round-trip preservation contract
 
-Given source NDJSON `S` and a schema `Σ` (either explicit or inferred), the LBK1 container `C` produced by the reference writer satisfies the following for every row `i ∈ [0, rowCount(S))` and every field `f ∈ Σ`:
+Given source NDJSON `S` and a schema `Sigma` (either explicit or inferred), the LBK1 container `C` produced by the reference writer satisfies the following for every row `i in [0, rowCount(S))` and every field `f in Sigma`:
 
 | Source value at (i, f) | Lane | Round-trip result |
 | :-- | :-- | :-- |
-| JSON number in fast-path domain (≤15 sig digits, `|exp| ≤ 22`) | F64 | **bit-exact IEEE 754 double** |
+| JSON number in fast-path domain (<=15 sig digits, `|exp| <= 22`) | F64 | **bit-exact IEEE 754 double** |
 | JSON number in slow-path domain | F64 | within 1 ULP of the correctly-rounded value |
 | JSON string (any bytes, valid UTF-8 or not) | U32 | **byte-exact bytes; validity not asserted** |
 | JSON `true` / `false` | F64 | `1` / `0` (documented coercion) |
@@ -340,14 +342,14 @@ Given source NDJSON `S` and a schema `Σ` (either explicit or inferred), the LBK
 | JSON `null` | U32 | `""` (index 0, the reserved empty-string entry; BS-20: null is lane-neutral) |
 | absent field (missing in source) | F64 | `0` (default) |
 | absent field | U32 | `""` (index 0 is the reserved empty-string entry) |
-| field in source but NOT in the schema | -- | silently dropped (matches lite-bake core) |
+| field in source but NOT in the schema | -- | silently dropped (see decisions/0005-one-unknown-field-policy.md) |
 | field in the schema with wrong value type (post-freeze), except null | -- | `W_LANE_MISMATCH` error, container not produced; null is lane-legal on both lanes (F64 -> 0, U32 -> "") |
 
 The preservation contract is asserted mechanically by three complementary layers:
 
-1. **Unit tests** — `test/RoundTrip.test.js`, `test/StringRoundTrip.test.js` verify individual known-value scenarios.
-2. **Property-based fuzz** — `test/DataPreservation.test.js` generates arbitrary random shapes across 12 seeds and asserts exact preservation on every cell of every row.
-3. **Torture-scale verification** — `bench/torture.js` runs the `verifyRoundTrip` helper (`test/_verify.js`) OUTSIDE the GC-measured band, asserting that every scenario opting into `verify: true` produces a container whose Reader output matches its source NDJSON byte-for-byte. The verification budget currently covers ~73,500 rows across 12 scenarios per fast-tier run; the `verified` column in the torture report shows the row count actually checked.
+1. **Unit tests** -- `test/RoundTrip.test.js`, `test/StringRoundTrip.test.js` verify individual known-value scenarios.
+2. **Property-based fuzz** -- `test/DataPreservation.test.js` generates arbitrary random shapes across 12 seeds and asserts exact preservation on every cell of every row.
+3. **Torture-scale verification** -- `node --expose-gc test/torture.mjs` (44 fast / 47 full scenarios) runs the `verifyRoundTrip` helper (`test/_verify.js`) OUTSIDE the GC-measured band, asserting that every scenario opting into `verify: true` produces a container whose Reader output matches its source NDJSON byte-for-byte. The verification budget currently covers ~103,000 rows across 20 scenarios per fast-tier run; the `verified` column in the torture report shows the row count actually checked.
 
 The boundary between "bit-exact" and "1 ULP" for F64 is pinned by `test/NumericBoundary.test.js`. Any drift in the tokenizer's number parser trips these tests before it can reach production.