npm - @zakkster/lite-bake-stream - Versions diffs - 1.6.0 → 1.7.0 - Mend

@zakkster/lite-bake-stream 1.6.0 → 1.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (25) hide show

package/README.md CHANGED Viewed

@@ -1,5 +1,7 @@
 # @zakkster/lite-bake-stream
+> Streaming byte-level JSON to LBK1 binary containers. Zero-GC, tree-shakeable, gigabyte-scale.
 [![npm version](https://img.shields.io/npm/v/@zakkster/lite-bake-stream.svg?style=for-the-badge&color=latest)](https://www.npmjs.com/package/@zakkster/lite-bake-stream)
 [![sponsor](https://img.shields.io/badge/sponsor-PeshoVurtoleta-ea4aaa.svg?logo=github)](https://github.com/sponsors/PeshoVurtoleta)
 ![Zero-GC](https://img.shields.io/badge/Zero--GC-Engine-00C853?style=for-the-badge&logo=leaf&logoColor=white)
@@ -11,46 +13,20 @@
 ![Dependencies](https://img.shields.io/badge/dependencies-0-brightgreen)
 [![license](https://img.shields.io/badge/license-MIT-blue?style=flat-square)](./LICENSE)
-> Streaming byte-level JSON compiler for [`@zakkster/lite-bake`](https://github.com/PeshoVurtoleta/lite-bake). Zero-GC, tree-shakeable, gigabyte-scale.
-**Status:** v1.6.0. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.
-## Two modes, one API
-- **Schema mode** (default): F64/U32 lane packing, zone-maps query pruning, byte-exact preservation for flat records. Millions of rows/sec random access, zero-GC hot path.
-- **Preserve mode**: `serialize(input, { preserve: true })`. Opaque byte blobs — the module doesn't crack open records, it just moves them intact. Deeply nested API JSON, arrays, mixed types — any valid JSON round-trips byte-for-byte identical. Reader exposes `getBytes(i)` (zero-alloc), `getString(i)`, `getJSON(i)`.
-`deserialize()` auto-dispatches on the container's flag bit. If you're compiling gigabytes of API responses with 2-3 levels of nesting, preserve mode is what you want. If the data is flat and you want columnar-style range queries, schema mode is what you want.
-## Why this exists
+**Status:** v1.7.0. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.
-`lite-bake` compiles a JS array into a flat binary buffer. That's the destination format you want. But if your input is a gigabyte of JSON, you can't call `JSON.parse` on it — V8's string limit is ~512 MB, and even under that, the intermediate object graph will trash the heap and destroy your process before `bake()` ever runs.
+## The gigabyte-scale JSON front door the ecosystem was missing
-`lite-bake-stream` is the front door: a chunk-safe UTF-8 JSON SAX scanner that reads bytes off a stream and emits `lite-bake`-shaped output shards, without ever materializing the object graph.
+You have a gigabyte of JSON and a program that has to read it. `JSON.parse`
+cannot get you there: V8 caps a single string near 512 MB, and well under that
+ceiling the intermediate object graph shreds your heap. This package is the
+front door -- a chunk-safe UTF-8 JSON SAX scanner that reads bytes off a stream
+and emits a flat, seekable LBK1 binary container, without ever materializing the
+object graph.
-## Qualified at 8 GB
-Overnight soak, M1 MacBook Pro, both release gates armed:
-| | |
-| :-- | --: |
-| Source | 8.00 GB NDJSON |
-| Container | 4.89 GB (61.1% of source) |
-| Rows | 98,367,702 |
-| Ingest throughput | 110.0 MB/s |
-| **Major GC** | **0** |
-| **Minor GC** | **0** |
-| **Total heap allocation** | **499.2 KB** |
-| Cells verified | 590,206,212 |
-| **Preservation mismatches** | **0** |
-**499 KB of heap allocation to compile 8 GB of JSON.** Total, for the whole run — not per shard, not per second. Zero garbage collections of any kind. Every one of 590 million declared cells round-tripped byte-exact.
-Tokenizer throughput is ~55% of `JSON.parse` (237 MB/s vs 416 MB/s on a 50 MB fixture). That's the trade: you give up some raw speed and you get an input size ceiling that's bounded by your disk instead of your heap.
-## Quickstart
-**Schema mode** (flat records, columnar queries):
+```bash
+npm install @zakkster/lite-bake-stream
+```
 ```js
 import { serialize, deserialize } from '@zakkster/lite-bake-stream';
@@ -61,107 +37,315 @@ reader.get(0, 'id');                     // 1
 reader.findShards('id', { min: 500 });   // shard pruning via zone maps
 ```
-**Preserve mode** (any JSON shape, byte-exact passthrough):
+## Table of contents
-```js
-import { serialize, deserialize } from '@zakkster/lite-bake-stream';
+- [Why this exists](#why-this-exists)
+- [What you get](#what-you-get)
+- [The core surface](#the-core-surface)
+- [API reference](#api-reference)
+- [Composability](#composability)
+- [Zero-GC design notes](#zero-gc-design-notes)
+- [Design decisions worth knowing](#design-decisions-worth-knowing)
+- [Testing](#testing)
+- [What this is not](#what-this-is-not)
+- [Ecosystem](#ecosystem)
+- [License](#license)
-// Handles arbitrarily nested API JSON
-const container = serialize(apiResponseBytes, { preserve: true });
-const reader = deserialize(container);
-reader.getBytes(0);    // Uint8Array view, zero allocation
-reader.getString(0);   // TextDecoder allocation
-reader.getJSON(0);     // full JSON.parse, allocates the parsed value graph
-```
+## Why this exists
-For streaming ingest with progress, use `/file-ingest`. For HTTP Range-fetched containers (schema mode), `/range-reader`. For joining multiple containers into one logical view (schema mode), `/multi-reader`. For parallel or checkpointed compilation (schema mode), `/split`.
+The enemy is `JSON.parse`, not any one downstream format. Two hard walls stop it
+on large input:
+- **The string ceiling.** V8 refuses to hold a single string beyond roughly
+  512 MB. A gigabyte file cannot even be read into memory as text, let alone
+  parsed.
+- **Object-graph heap death.** Even under the ceiling, `JSON.parse` allocates a
+  fully-realized object graph -- every object, array, string, and boxed number.
+  For a gigabyte of records that graph is many gigabytes of live heap, and the
+  garbage collector spends the run trying (and failing) to keep up.
+`lite-bake-stream` never builds the graph. It scans bytes as they arrive, packs
+each record into typed-lane shard buffers (or preserves it as an opaque blob),
+and writes an LBK1 container whose Reader gives you random access without
+re-parsing. The input-size ceiling becomes your disk, not your heap.
+## What you get
+Two ingest modes share one top-level API. `deserialize()` auto-dispatches on the
+container's flag bit, so a consumer never has to know which mode wrote the bytes.
+- **Schema mode** (default): F64/U32 lane packing, per-shard zone-maps for query
+  pruning, byte-exact preservation for flat records. Random access to millions
+  of rows per second on a zero-GC hot path.
+- **Preserve mode** (`serialize(input, { preserve: true })`): opaque byte blobs.
+  The module does not crack open records, it moves them intact. Deeply nested
+  API JSON, arrays, mixed types -- any valid JSON round-trips byte-for-byte
+  identical. The reader exposes `getBytes(i)` (zero-alloc view), `getString(i)`,
+  and `getJSON(i)`.
+Compiling gigabytes of API responses with two-to-three levels of nesting?
+Preserve mode. Flat records that want columnar-style range queries? Schema mode.
+## The core surface
+<details>
+<summary>Streaming emission, bounded memory, and integrity</summary>
+`finalize()` returns the whole container as one `ArrayBuffer` -- peak memory is
+`O(container)`. For bounded-memory output, `writer.finalizeToSink(sink, { layout })`
+emits to a caller sink and returns `{ totalRows, shardCount, schema | mode,
+bytesWritten, layout }` (no buffer). A sink is any object with a synchronous
+`write(bytes)`; `layout: 'stream'` also needs `writeAt(bytes, position)` for one
+header backpatch. The contract is satisfiable by `fs.write(fd, buf, 0, len, pos)`
+and the File System Access API's `createWritable().write({ type: 'write',
+position })`; the test suite uses an in-memory sink of the same shape. No sink
+class is exported -- the contract is public, the convenience implementation is
+internal.
+Two ways to drive a stream, with different peaks:
+- **Two-step, bounded RAM** -- call `writer.beginStream(sink, { layout: 'stream' })`
+  BEFORE feeding, feed the tokenizer, then `writer.finalizeToSink(sink, { layout:
+  'stream' })`. Each shard is written to the sink as it finalizes and its bytes
+  are dropped, so peak memory is `2*targetShardBytes + shardCount*(40 + 16*T) +
+  schemaBlockBytes` -- `O(targetShardBytes + directory)`, never `O(container)`.
+  This is the mode the 500 MB gate proves.
+- **One-shot, buffered** -- call `writer.finalizeToSink(sink, opts)` alone (no
+  `beginStream`). Correct and simplest, but the shards are buffered first, so
+  peak is `containerBytes + sum(shard bytes)` -- `O(container)`, the same as
+  `finalize()`.
+`layout: 'prefix'` is the classic layout, byte-identical to `finalize()`.
+`layout: 'stream'` places the schema/directory/zone-map trailer and footer after
+the payloads with one header backpatch; a default-emitted stream container is a
+legal v1 container (`format_version` stays 1) that every shipped reader,
+`checkContainer`, and `mergeContainers` accept. A malformed sink (non-object,
+missing `write`, missing `writeAt` for stream, or an async/thenable return)
+throws `W_BAD_SINK`; a sink that throws mid-emission fails the writer closed and
+rethrows the source error verbatim (a retry then hits `W_FINALIZED`).
+**Optional CRC-32C** (Castagnoli, table-driven, zero deps). Opt in with
+`{ crc: true }` on the writer (or serialize `writer` opts); coverage is
+`[0, footer_off)`, folded per emitted chunk. Readers expose `verifyCrc()` ->
+`'ok' | 'absent'` (a mismatch throws `R_BAD_CRC`) on `Reader`/`PreserveReader`
+(sync) and `RangeReader` (async). The open option `{ verifyCrc: true }` (also
+`deserialize(bytes, opts)`) fails closed on both mismatch (`R_BAD_CRC`) and
+absence (`R_CRC_ABSENT`) -- `null` is not zero. `0xFFFFFFFF` means absent and
+stays legal. `mergeContainers` recomputes the CRC iff every input carried one,
+else emits absent. See `decisions/0009-streaming-emission.md`.
+Future lane kinds and payload modes (an I64 exact-integer lane, a columnar
+payload mode, a container-level string table) land via the format's
+forward-compat seams -- `min_reader_version` on ShardEntry, reserved
+FieldDescriptor flags, the `metadata_off` block wrapper -- without a
+`format_version` bump.
+</details>
+## API reference
+Every subpath is a standalone import; the browser reader never pulls the writer.
+Every subpath also exports a `VERSION` const.
+**Root** (`@zakkster/lite-bake-stream`)
+- `serialize(input, opts?) -> Uint8Array` -- ingest bytes into an LBK1 container.
+  `opts.preserve` selects preserve mode; `opts.writer` forwards writer options.
+- `deserialize(bytes, opts?) -> Reader | PreserveReader` -- auto-dispatch on the
+  container flag bit. Plus class re-exports and `StringTableError`.
+**Schema mode**
+- `Tokenizer` (`/tokenizer`) -- chunk-safe UTF-8 JSON SAX scanner. `feed(chunk)`,
+  `end()`, sink callbacks fire synchronously.
+- `Writer` (`/writer`) -- LBK1 shard emitter; plugs into the Tokenizer. F64/U32
+  lanes, per-shard string tables and zone maps, `beginStream`/`finalizeToSink`.
+- `Reader` (`/reader`) -- container parser, sync, zero-alloc `get(row, field)`
+  plus zone-map query APIs `shardBounds(shardIdx, field)` and
+  `findShards(field, range)`.
+- `RangeReader` (`/range-reader`) -- HTTP Range lazy shard loading; synchronous
+  query pruning after open. Optional `{ signal }` cancels in-flight range I/O
+  (`R_ABORTED`); once it fires the reader is dead for new I/O.
+- `MultiReader` (`/multi-reader`) -- logical union over N Readers sharing a
+  schema.
+- `splitNDJSON`, `compilePart`, `compileInParts`, `mergeContainers` (`/split`) --
+  worker-agnostic split, compile, and merge.
+**Preserve mode**
+- `PreserveTokenizer` (`/preserve-tokenizer`) -- NDJSON record-boundary scanner
+  with JSON-aware depth tracking, chunk-safe.
+- `PreserveWriter` (`/preserve-writer`) -- opaque byte-blob sink, pre-allocated
+  shard buffer, zero-GC record path.
+- `PreserveReader` (`/preserve-reader`) -- `getBytes(i)` (zero-alloc view),
+  `getString(i)`, `getJSON(i)`.
+**Shared**
+- `StringTable` (`/string-table`) -- byte-level UTF-8 interning primitive.
+- `ingestStream(readableStream, opts?)`, `ingestFile(file, opts?)`
+  (`/file-ingest`) -- browser helpers piping a `ReadableStream<Uint8Array>` (e.g.
+  `File.stream()`) through the Tokenizer + Writer, returning a Reader. The
+  per-chunk `onProgress` callback's state object is reused across calls (mutated
+  in place, zero per-chunk allocation) -- copy it if retained past the callback.
+**Zero-copy view contract.** A byte view returned by `getBytes(i)` (and any
+shard raw-access getter) is a plain `subarray` over the container's
+`ArrayBuffer`: it pins that buffer for as long as the view is reachable. Copy the
+bytes out (`getBytes(i).slice()`) if you need the container itself to be
+collectable.
+### Constants
+Lane kinds (SPEC 4.2):
+| Kind | Name  | Bytes |
+| ---: | :---- | ----: |
+| `1`  | `F64` | 8     |
+| `2`  | `F32` | 4     |
+| `3`  | `U32` | 4     |
+| `4`  | `U8`  | 1     |
+Ceilings:
+| Limit                                | Value        |
+| :----------------------------------- | -----------: |
+| String-table blob bytes (u32)        | `4294967295` |
+| String-table entry count             | `4294967294` |
+| u64 offset safe `Number()` cast      | `2^53 - 1`   |
+Error-code families (thrown with a stable `code`; the full inventory is pinned by
+the torture gate):
+| Prefix | Class                                                              |
+| :----- | :----------------------------------------------------------------- |
+| `E_`   | `TokenizerError` (e.g. `E_NUMBER_OVERFLOW`)                        |
+| `W_`   | `WriterError` (`W_MIXED_LANE_TYPES`, `W_LANE_MISMATCH`, `W_BAD_SINK`, `W_FINALIZED`) |
+| `R_`   | `ReaderError` / `RangeReaderError` (`R_ROW_OUT_OF_RANGE`, `R_OFFSET_TOO_LARGE`, `R_BAD_CRC`, `R_CRC_ABSENT`, `R_ABORTED`, ...) |
+| `M_`   | `MultiReaderError` (`M_ROW_OUT_OF_RANGE`)                          |
+| `S_`   | `SplitError`                                                       |
+| `ST_`  | `StringTableError` (`ST_BLOB_OVERFLOW`)                            |
+| `P*_`  | `PreserveTokenizerError` / `PreserveWriterError` / `PreserveReaderError` |
+## Composability
+The `/split` subpath turns one logical dataset into worker-parallel or
+checkpointed parts, and the readers merge them back into a single view:
-## What shipped in 1.0
+```js
+import { splitNDJSON, compileInParts, mergeContainers } from '@zakkster/lite-bake-stream/split';
+import { RangeReader } from '@zakkster/lite-bake-stream/range-reader';
-| Milestone | Deliverable                                                                 |
-| :-------- | :-------------------------------------------------------------------------- |
-| ✅ **M0** | LBK1 container format spec, forward-compat seams for the I64 lane.          |
-| ✅ **M1** | Chunk-safe tokenizer; zero-GC on the hot path after warmup.                 |
-| ✅ **M2** | Schema-driven writer; sample-and-infer path; typed-lane shard buffers.      |
-| ✅ **M3** | Per-shard UTF-8 string table with pre-alloc typed-array hash-map interning. |
-| ✅ **M4** | `serialize()`/`deserialize()`, MultiReader, full type declarations.         |
-| ✅ **M5** | Browser reader, HTTP Range fetch, `File.stream()` client-side ingest.       |
-| ✅ **M6** | NDJSON splitter + per-part compilation + container merge.                   |
-| ✅ **M7** | Per-shard zone maps (min/max per F64 column) + query-pruning APIs.          |
-| ✅ **M8** | RFC 8259 conformance corpus + robustness fuzz.                              |
+// 1. split NDJSON into byte-aligned parts (record boundaries preserved)
+const parts = splitNDJSON(ndjsonBytes, { partBytes: 64 * 1024 * 1024 });
-## Roadmap beyond 1.0
+// 2. compile each part (fan out to workers if you like), then merge
+const containers = compileInParts(parts);
+const merged = mergeContainers(containers);
-All of these fit the format's existing forward-compat seams — `format_version` stays at `1`.
+// 3. open the merged container and prune shards before fetching them
+//    (pass { signal } to cancel in-flight range I/O when the consumer detaches)
+const reader = await RangeReader.open(adapterOver(merged), { signal: controller.signal });
+const shards = reader.findShards('id', { min: 500, max: 999 });
+for (const s of shards) reader.get(s.firstRow, 'id');
+```
-| | |
-| :-- | :-- |
-| **I64 lane** | Exact 64-bit integer preservation. FieldDescriptor flags are already reserved for it; shards using it set `min_reader_version: 2`. |
-| **Columnar payload mode** | Column-contiguous shard payloads instead of interleaved rows. Better compression, SIMD-friendly scans. Opt-in via a shard flag. |
-| **Container-level string table** | Cross-shard interning for high-repetition corpora. Lands in the `metadata_off` block alongside zone maps. |
-| **Oscilloscope-rack demo** | Reel-to-reel tape transport at actual MB/s, waterfall spectrum per shard, VU needles, seven-segment counters. Locked 60fps on an iPhone 7 with 100 MB NDJSON. |
+## Zero-GC design notes
-## What's here today
+<details>
+<summary>Allocation table and the 8 GB soak</summary>
-- `SPEC.md` — LBK1 container spec draft 0.1, per-shard string table (§3.3), preservation contract (§7).
-- `src/index.js` — top-level convenience API: `serialize(input)` and `deserialize(bytes)` plus class re-exports.
-- `src/Tokenizer.js` — chunk-safe SAX tokenizer with Clinger's fast-path decimal→F64 conversion.
-- `src/Writer.js` — LBK1 shard emitter with F64 and U32 lanes, sample-and-infer schema, per-shard string tables, per-shard zone maps.
-- `src/Reader.js` — LBK1 container parser with typed-array views, transparent string-field resolution, and zone-map query APIs.
-- `src/StringTable.js` — open-addressed byte-level interning with zero-alloc hit path.
-- `src/RangeReader.js` — LBK1 reader with lazy shard loading via an IO adapter (HTTP Range or in-memory mock); zone maps loaded once at open enable synchronous query pruning.
-- `src/MultiReader.js` — union view over N Reader instances with matching schemas; merged shard indices for query pruning across containers.
-- `src/Split.js` — NDJSON splitter, per-part compilation, and container merge for worker-parallel or checkpointed ingest.
-- `src/FileIngest.js` — browser helper: `File.stream()` → Reader. Any `ReadableStream<Uint8Array>` works.
-- `types/*.d.ts` — 8 hand-written TypeScript declaration files, one per subpath.
-- `demo/` — single-file HTML + module JS. Ingest a JSON file → LBK1 → browse in `@zakkster/lite-table`. Zone-maps query-pruning panel visualizes the fetch-set reduction. `npm run demo`.
-- `test/*.test.js` — 233 tests across 12 files, including a 66-fixture RFC 8259 conformance corpus and a 7-scenario robustness fuzz (~2,300 tokenizer invocations per run).
-- `test/_verify.js` — universal round-trip verifier used by tests, torture, and soak.
-- `bench/bench-tokenizer.js` — interleaved-rep bench vs `JSON.parse`.
-- `bench/torture.js` — 37-scenario CI harness. Two release gates: `maxMajor: 0` (zero-GC) AND `verify: true` (value-exact round-trip on 77,500+ rows per run).
-- `bench/soak.js` — CLI-configurable soak with the preservation gate wired in.
+The hot path allocates nothing steady-state. Buffers are pre-allocated and
+reused across the churn:
-Two release gates. Zero major GC AND every declared cell round-trips exactly. If either fails, no publish.
+| Operation             | Steady-state allocation |
+| :-------------------- | :---------------------- |
+| `feed(chunk)`         | 0 (views into the tokenizer's internal buffer) |
+| per row               | 0 (typed-lane staging is pre-allocated) |
+| per shard roll        | 0 (working buffers reused across rolls) |
+| `Reader.get(row, f)`  | 0 for F64; `U32` string fields resolve through a cached decoder |
-## Streaming emission and integrity
+**Qualified at 8 GB.** Overnight soak, M1 MacBook Pro, both release gates armed:
-`finalize()` returns the whole container as one `ArrayBuffer` -- peak memory is `O(container)`. For bounded-memory output, `writer.finalizeToSink(sink, { layout })` emits to a caller sink and returns `{ totalRows, shardCount, schema | mode, bytesWritten, layout }` (no buffer). A sink is any object with a synchronous `write(bytes)`; `layout: 'stream'` also needs `writeAt(bytes, position)` for one header backpatch -- satisfiable by `fs.write(fd, buf, 0, len, pos)`, the FS Access API's `createWritable().write({ type: 'write', position })`, and the in-memory `MemorySink`.
+| | |
+| :-- | --: |
+| Source | 8.00 GB NDJSON |
+| Container | 4.89 GB (61.1% of source) |
+| Rows | 98,367,702 |
+| Ingest throughput | 110.0 MB/s |
+| **Major GC** | **0** |
+| **Minor GC** | **0** |
+| **Total heap allocation** | **499.2 KB** |
+| Cells verified | 590,206,212 |
+| **Preservation mismatches** | **0** |
-There are two ways to drive a stream, and they deliver different peaks:
+499 KB of heap allocation to compile 8 GB of JSON -- total, for the whole run,
+not per shard, not per second. Zero garbage collections of any kind. Every one of
+590 million declared cells round-tripped byte-exact.
-- **Two-step, bounded RAM** -- call `writer.beginStream(sink, { layout: 'stream' })` BEFORE feeding, feed the tokenizer, then `writer.finalizeToSink(sink, { layout: 'stream' })`. Each shard is written to the sink as it finalizes and its bytes are dropped, so peak memory is `peak(stream) = 2*targetShardBytes + shardCount*(40 + 16*T) + schemaBlockBytes` -- `O(targetShardBytes + directory)`, never `O(container)`. This is the mode the 500 MB gate proves.
-- **One-shot, buffered** -- call `writer.finalizeToSink(sink, opts)` alone (no `beginStream`). Correct and simplest, but the shards are buffered first, so peak is `peak(prefix) = containerBytes + sum(shard bytes)` -- `O(container)`, the same as `finalize()`.
+**Bounded streaming, measured.** The full-tier gate drives the two-step bounded
+path, streaming 500 MB in 8 MiB chunks into a counting sink, and asserts
+`peakRss - baselineRss <= 4*targetShardBytes + 64*shardCount + 64 MiB` -- measured
+32.1 MiB against the 96.0 MiB bound at 41 shards (the buffered prefix path is
+~704 MiB at the same 500 MB).
-`layout: 'prefix'` is the classic layout, **byte-identical** to `finalize()`. `layout: 'stream'` places the schema/directory/zone-map trailer and footer after the payloads with one header backpatch; a default-emitted stream container is a legal v1 container (format_version stays 1) that every shipped reader, `checkContainer`, and `mergeContainers` accept.
+Tokenizer throughput is ~55% of `JSON.parse` (237 MB/s vs 416 MB/s on a 50 MB
+fixture). That is the trade: you give up some raw speed and you get an input-size
+ceiling bounded by your disk instead of your heap.
-A malformed sink (non-object, missing `write`, missing `writeAt` for stream, or an async/thenable return) throws `W_BAD_SINK`; a sink that throws mid-emission fails the writer closed and rethrows the source error verbatim (a retry then hits `W_FINALIZED`).
+</details>
-**Memory model.** `peak(prefix) = containerBytes + sum(shard bytes)`; `peak(stream) = 2*targetShardBytes + shardCount*(40 + 16*T) + schemaBlockBytes`. The full-tier gate drives the two-step bounded path, streaming 500 MB in 8 MiB chunks into a counting sink, and asserts `peakRss - baselineRss <= 4*targetShardBytes + 64*shardCount + 64 MiB` (measured 32.1 MiB against the 96.0 MiB bound at 41 shards; the buffered path is ~704 MiB at 500 MB).
+## Design decisions worth knowing
-**Optional CRC-32C** (Castagnoli, table-driven, zero deps). Opt in with `{ crc: true }` on the writer (or serialize `writer` opts); coverage is `[0, footer_off)`, folded per emitted chunk. Readers expose `verifyCrc()` -> `'ok' | 'absent'` (a mismatch throws `R_BAD_CRC`) on `Reader`/`PreserveReader` (sync) and `RangeReader` (async). The open option `{ verifyCrc: true }` (also `deserialize(bytes, opts)`) fails closed on both mismatch (`R_BAD_CRC`) and absence (`R_CRC_ABSENT`) -- `null` is not zero. `0xFFFFFFFF` means absent and stays legal. `mergeContainers` recomputes the CRC iff every input carried one, else emits absent. See `decisions/0009-streaming-emission.md`.
+Each call is an on-disk record under `decisions/`:
-## Release testing tiers
+- [0001](decisions/0001-reserved-empty-string.md) -- reserve string-table index 0 as the empty string.
+- [0002](decisions/0002-record-shape-policy.md) -- depth-0 non-object records are refused, not dropped.
+- [0003](decisions/0003-duplicate-field-names.md) -- duplicate schema field names are refused at freeze.
+- [0004](decisions/0004-container-doors-and-lying-metadata.md) -- a container is verified at the door; a lying pointer is corruption.
+- [0005](decisions/0005-one-unknown-field-policy.md) -- absent names are refused, untracked names fall back.
+- [0006](decisions/0006-utf8-door.md) -- the UTF-8 door is documented-permissive, not a tokenizer gate.
+- [0007](decisions/0007-sample-window-bytes.md) -- the sample window is byte-true, not record-count.
+- [0008](decisions/0008-null-policy.md) -- JSON null is lane-neutral.
+- [0009](decisions/0009-streaming-emission.md) -- streaming emission, sinks, and optional CRC-32C.
+- [0010](decisions/0010-preserve-mode.md) -- why a second (preserve) mode exists at all.
+- [0011](decisions/0011-sample-drain-reintern.md) -- per-shard re-intern on drain and the byte-true sample window.
+- [0012](decisions/0012-clinger-fast-path.md) -- the Clinger fast path and the two-tier F64 guarantee.
-The gates run at four scales, chosen to fit different hardware and time budgets:
+## Testing
+The gates run at scales chosen to fit different hardware and time budgets:
 | Command | Scale | Use |
 | :-- | :-- | :-- |
-| `npm test` | 96 tests | Dev loop, every save |
-| `npm run torture` | 37 scenarios (~1 MB each) | Before every commit |
-| `npm run torture:full` | + 500 MB full-tier soak scenarios | Before every publish |
+| `npm test` | 530 tests across 37 files | Dev loop, every save |
+| `npm run torture` | 44 fast scenarios (~1 MB each) | Before every commit |
+| `npm run torture:full` | 47 scenarios incl. the 500 MB soak | Before every publish |
 | `npm run soak` | 100 MB with preservation gate | Sanity check |
 | `npm run soak:500` | 500 MB | Pre-publish scale check |
-| `npm run soak:1gb` | 1 GB | Real-hardware qualification (M4 Pro etc.) |
+| `npm run soak:1gb` | 1 GB | Real-hardware qualification |
 | `npm run soak:overnight` | 8 GB | Overnight `caffeinate` run |
-Custom sizes:
-```
-node --expose-gc bench/soak.js --gb=4 --verify
-caffeinate node --expose-gc bench/soak.js --gb=16 --verify   # macOS overnight
-```
+The test tree includes a 66-fixture RFC 8259 conformance corpus, a property-based
+robustness fuzz, and two structural guards: an ASCII-law scanner
+(`test/AsciiLaw.test.js`) and an API-surface drift guard
+(`test/ApiSurface.test.js`), each also wired into the torture tier with an armed
+control that proves it can fail. Two release gates decide a publish: zero major
+GC AND every declared cell round-trips exactly. If either fails, no publish.
+## What this is not
+No JSON5, JSONC, comments, or trailing commas. No streaming field updates, no
+compression, no native (non-JS) producers. LBK1 is this package's own container
+format; non-JSON producers are welcome as long as they emit a well-formed
+container.
+## Ecosystem
-Every soak run reports ingest throughput (MB/s), verification throughput (rows/s), the zero-GC gate result, container compression ratio, and progress lines so long runs are observable. The preservation gate proves — cell by cell — that every declared field of every row round-trips exactly.
+The `demo/` directory is a single-file HTML + module-JS app: ingest a JSON file,
+compile it to LBK1, and browse the result, with a zone-maps panel that visualizes
+the query-pruning fetch-set reduction. Run it with `npm run demo`.
 ## License
-MIT © 2026 Zahary Shinikchiev
+MIT (c) 2026 Zahary Shinikchiev

package/SPEC.md CHANGED Viewed

@@ -1,6 +1,6 @@
 # LBK1 Format Specification
-> Version 1 — frozen as of `@zakkster/lite-bake-stream` v1.0.0.
+> Version 1 -- frozen as of `@zakkster/lite-bake-stream` v1.0.0.
 > Future lane kinds and payload modes land via the forward-compat seams
 > (`min_reader_version`, reserved FieldDescriptor flags, the `metadata_off`
 > block wrapper) without a `format_version` bump.
@@ -10,7 +10,7 @@
 LBK1 is the on-disk / on-wire container for streaming-compiled `lite-bake` datasets. It carries one or more shards of interleaved binary rows plus the metadata needed to reconstruct a Reader over the whole dataset without re-parsing JSON.
-The tokenizer specified in section 5 is the reference producer path from UTF-8 JSON (top-level array or NDJSON) into an LBK1 container. Non-JSON producers are allowed as long as they emit a well-formed container.
+The tokenizer specified in section 5 is the canonical producer path from UTF-8 JSON (top-level array or NDJSON) into an LBK1 container. Non-JSON producers are allowed as long as they emit a well-formed container.
 Numbers on the hot path are F64. Section 4.3 reserves the schema-level bit that will admit a future `I64_BYTES_PRESERVED` lane without a container format-version bump.
@@ -22,13 +22,13 @@ These are load-bearing; every other decision follows from them.
 2. **Forward-compatible via strict decoders.** Unknown flags or lane kinds are rejected, not ignored. Silent corruption is the failure mode we refuse.
 3. **Shard-scoped versioning.** A shard header carries the minimum reader version required to decode that shard. Old readers refuse new shards cleanly; new readers read old shards forever.
 4. **Zero-copy where the CPU allows it.** Row payloads are aligned so that typed-array views over the shard buffer read fields with native alignment.
-5. **Zero runtime dependencies.** Reference producer and reader share the `@zakkster` no-dep constraint.
+5. **Zero runtime dependencies.** Producer and reader share the `@zakkster` no-dep constraint.
 ## 2. Notation
-- `u8`, `u16`, `u32`, `u64`, `f32`, `f64` — little-endian unless stated.
-- `bytes(n)` — a raw byte run of length n.
-- `varlen_utf8` — u32 length prefix followed by that many UTF-8 bytes.
+- `u8`, `u16`, `u32`, `u64`, `f32`, `f64` -- little-endian unless stated.
+- `bytes(n)` -- a raw byte run of length n.
+- `varlen_utf8` -- u32 length prefix followed by that many UTF-8 bytes.
 - Offsets are relative to the start of the container unless stated.
 - All multi-byte integers in headers are little-endian. Row payloads use native endianness with a byte-order marker in the container header (section 3.1); non-native containers require a byteswap on load or must be rejected.
@@ -84,7 +84,7 @@ Padded with zero bytes so the block ends on an 8-byte boundary.
 There are two string-table locations in an LBK1 container:
 - **Container-level** at `string_table_off` (v1 reserved; not written by the current Writer). Reserved for future container-wide interning across shards.
-- **Per-shard** at each ShardEntry's `local_string_off` (M3, implemented). Every shard with U32-lane fields has its own local string table so shards remain independently decodable — critical for HTTP Range fetches (M5+) and worker-parallel readers (M6+).
+- **Per-shard** at each ShardEntry's `local_string_off` (M3, implemented). Every shard with U32-lane fields has its own local string table so shards remain independently decodable -- critical for HTTP Range fetches (M5+) and worker-parallel readers (M6+).
 Both use the same on-disk layout:
@@ -144,14 +144,14 @@ f64  maxes[shard_count * tracked_field_count]  Same layout.
 ```
 - `tracked_field_count` MAY be less than the schema's F64 field count (the writer chooses which fields to track). For fields not present in `field_indices`, `shardBounds` returns `null`.
-- `mins[s*T + t]` and `maxes[s*T + t]` are the exact min/max of the tracked field's values across the rows in shard `s`. For shards where the field is missing (defaults to `0`), the min/max collapse to `[0, 0]` — still valid input to range predicates.
+- `mins[s*T + t]` and `maxes[s*T + t]` are the exact min/max of the tracked field's values across the rows in shard `s`. For shards where the field is missing (defaults to `0`), the min/max collapse to `[0, 0]` -- still valid input to range predicates.
 - Zone maps segment is placed between the shard directory and the first shard payload. This keeps it inside a single Range fetch alongside the header/schema/directory during Reader initialization.
 Reader query APIs (see `Reader.js`):
 ```
-reader.shardBounds(shardIdx, fieldName)     → {min, max} | null
-reader.findShards(fieldName, {min, max})    → shardIdx[]     // shards that MAY contain the range
+reader.shardBounds(shardIdx, fieldName)     -> {min, max} | null
+reader.findShards(fieldName, {min, max})    -> shardIdx[]     // shards that MAY contain the range
 ```
 `findShards` returns exactly the shards whose bounds overlap `[min, max]`; a shard whose bounds are disjoint is skipped, saving one range fetch per skipped shard in the RangeReader path.
@@ -191,12 +191,12 @@ Fixed-width descriptors let a Reader load the schema with a single `Uint32Array`
 | Value | Name    | Bytes | Notes                                                                    |
 | ----: | ------- | ----: | ------------------------------------------------------------------------ |
-|   `1` | `F64`   |     8 | The only numeric lane in M4. Matches lite-bake's `Types.F64`.            |
+|   `1` | `F64`   |     8 | The only numeric lane in M4.                                            |
 |   `2` | `F32`   |     4 | Reserved for future producer opt-in via schema override.                 |
 |   `3` | `U32`   |     4 | String-table index (M3+). Row cell holds a u32 index into the shard's `local_string_off` table. |
 |   `4` | `U8`    |     1 | Reserved.                                                                |
-Values 5..127 reserved for future lite-bake type lanes (matching `Types.*`).
+Values 5..127 reserved for future lane kinds defined by this spec.
 Values 128..255 reserved for stream-specific lanes; see 4.4.
@@ -204,10 +204,10 @@ Values 128..255 reserved for stream-specific lanes; see 4.4.
 Currently all bits are reserved and must be zero. Reserved bit assignments (v2+) will use:
-- Bit 0: `LANE_KIND_EXT` — if set, the field uses an extended lane; consult 4.4.
+- Bit 0: `LANE_KIND_EXT` -- if set, the field uses an extended lane; consult 4.4.
 - Bits 1..7: unassigned.
-This is the seam Option (c) — `I64_BYTES_PRESERVED` — slides into without a container format-version bump. A shard using it sets `min_reader_version = 2` at the directory level; v1 readers refuse the shard, not the whole container.
+This is the seam Option (c) -- `I64_BYTES_PRESERVED` -- slides into without a container format-version bump. A shard using it sets `min_reader_version = 2` at the directory level; v1 readers refuse the shard, not the whole container.
 ### 4.4 Extended lane kinds (reserved for v2)
@@ -220,16 +220,16 @@ When `flags` bit 0 is set on a FieldDescriptor, `lane_kind` is interpreted from
 All other values in the extended range are reserved and must be rejected.
-## 5. Reference producer: tokenizer contract
+## 5. Producer: tokenizer contract
-The M1 tokenizer is a chunk-safe UTF-8 JSON SAX scanner. It does not itself write LBK1 shards — that is the M2 writer's job. This section specifies the contract between them so M1 can be built and benched in isolation.
+The M1 tokenizer is a chunk-safe UTF-8 JSON SAX scanner. It does not itself write LBK1 shards -- that is the M2 writer's job. This section specifies the contract between them so M1 can be built and benched in isolation.
 ### 5.1 Input framing
 Two top-level modes, auto-detected by the first non-whitespace byte:
-- `[` — Top-level JSON array. Records are the array's elements.
-- Any other JSON value byte (`{`, `"`, `-`, `0`..`9`, `t`, `f`, `n`) — NDJSON. Records are separated by JSON whitespace (`0x20`, `0x09`, `0x0A`, `0x0D`).
+- `[` -- Top-level JSON array. Records are the array's elements.
+- Any other JSON value byte (`{`, `"`, `-`, `0`..`9`, `t`, `f`, `n`) -- NDJSON. Records are separated by JSON whitespace (`0x20`, `0x09`, `0x0A`, `0x0D`).
 Records themselves must be JSON objects for the reference writer path. Non-object records are accepted by the tokenizer (it is generic) but rejected by the writer.
@@ -267,10 +267,10 @@ For a string that spans multiple chunks, the tokenizer accumulates into an inter
 #### 5.4.1 F64 preservation contract (M3)
-The tokenizer implements Clinger's fast-path decimal→F64 conversion with an exact `POW10[0..22]` table. This gives a two-tier preservation guarantee that consumers can rely on:
+The tokenizer implements Clinger's fast-path decimal->F64 conversion with an exact `POW10[0..22]` table. This gives a two-tier preservation guarantee that consumers can rely on:
-- **Fast path (guaranteed correctly-rounded):** any numeric literal whose decimal representation has ≤ 15 significant digits AND absolute net exponent ≤ 22 produces the SAME IEEE 754 double as JavaScript's `Number(str)`. This covers all integer IDs within `Number.MAX_SAFE_INTEGER`, all "typical" decimals (prices, ratios, timestamps, coordinates), and scientific notation within `1e±22`. This is the domain the round-trip verifier (`test/_verify.js`) asserts bit-exact across every torture scenario.
-- **Slow path (may drift ≤ 1 ULP):** literals outside the fast path (16+ significant digits, or `|exponent| > 22`) fall through to a naive `intPart + fracPart/fracDiv * Math.pow(10, exp)` computation. The result is within 1 ULP of the correctly-rounded value. Correctly-rounded parsing across the full F64 domain (David Gay's `strtod` or equivalent) is planned for M4+.
+- **Fast path (guaranteed correctly-rounded):** any numeric literal whose decimal representation has <= 15 significant digits AND absolute net exponent <= 22 produces the SAME IEEE 754 double as JavaScript's `Number(str)`. This covers all integer IDs within `Number.MAX_SAFE_INTEGER`, all "typical" decimals (prices, ratios, timestamps, coordinates), and scientific notation within `1e+/-22`. This is the domain the round-trip verifier (`test/_verify.js`) asserts bit-exact across every torture scenario.
+- **Slow path (may drift <= 1 ULP):** literals outside the fast path (16+ significant digits, or `|exponent| > 22`) fall through to a naive `intPart + fracPart/fracDiv * Math.pow(10, exp)` computation. The result is within 1 ULP of the correctly-rounded value. Correctly-rounded parsing across the full F64 domain (David Gay's `strtod` or equivalent) is planned for M4+.
 The boundary is exercised explicitly by `test/NumericBoundary.test.js`, which pins expected drift so any regression is caught.
@@ -296,10 +296,10 @@ Subpath entries carved so consumers pay only for what they import. Each entry is
 | `@zakkster/lite-bake-stream/writer` | LBK1 shard emitter. Explicit or sample-and-infer schema. Emits per-shard zone maps. |
 | `@zakkster/lite-bake-stream/reader` | LBK1 parser over an in-memory `ArrayBuffer`. Synchronous. Zone-maps query APIs. |
 | `@zakkster/lite-bake-stream/string-table` | Byte-level UTF-8 interning primitive. |
-| `@zakkster/lite-bake-stream/file-ingest` | Browser helper: `File.stream()` → Reader. Any `ReadableStream<Uint8Array>` works. |
-| `@zakkster/lite-bake-stream/range-reader` | LBK1 parser with lazy shard loading via an IO adapter. HTTP Range fetch or in-memory mock. Synchronous zone-maps pruning. |
+| `@zakkster/lite-bake-stream/file-ingest` | Browser helper: `File.stream()` -> Reader. Any `ReadableStream<Uint8Array>` works. |
+| `@zakkster/lite-bake-stream/range-reader` | LBK1 parser with lazy shard loading via an IO adapter. HTTP Range fetch or in-memory mock. Synchronous zone-maps pruning. Optional reader-level `{ signal }` cancels in-flight range I/O (`R_ABORTED`); the adapter `fetch(byteOffset, byteLength, signal?)` contract is additive (a 2-arg adapter degrades cancellation only). |
 | `@zakkster/lite-bake-stream/multi-reader` | Logical union over N Reader instances. Rows and shards cumulatively addressed; findShards merges across sub-readers. |
-| `@zakkster/lite-bake-stream/split` | NDJSON splitter + per-part compile + container merge. Executor-agnostic — wire to any worker pool or run serially. |
+| `@zakkster/lite-bake-stream/split` | NDJSON splitter + per-part compile + container merge. Executor-agnostic -- wire to any worker pool or run serially. |
 Every entry works in browsers (evergreen Chromium / Firefox / Safari) and Node 18+; no Node-specific APIs are used in the source. Every subpath has a matching `.d.ts` file in `types/` for TypeScript consumers.
@@ -330,11 +330,11 @@ Not currently supported in preserve mode: `RangeReader` (HTTP-Range lazy loading
 ## 7. Round-trip preservation contract
-Given source NDJSON `S` and a schema `Σ` (either explicit or inferred), the LBK1 container `C` produced by the reference writer satisfies the following for every row `i ∈ [0, rowCount(S))` and every field `f ∈ Σ`:
+Given source NDJSON `S` and a schema `Sigma` (either explicit or inferred), the LBK1 container `C` produced by the reference writer satisfies the following for every row `i in [0, rowCount(S))` and every field `f in Sigma`:
 | Source value at (i, f) | Lane | Round-trip result |
 | :-- | :-- | :-- |
-| JSON number in fast-path domain (≤15 sig digits, `|exp| ≤ 22`) | F64 | **bit-exact IEEE 754 double** |
+| JSON number in fast-path domain (<=15 sig digits, `|exp| <= 22`) | F64 | **bit-exact IEEE 754 double** |
 | JSON number in slow-path domain | F64 | within 1 ULP of the correctly-rounded value |
 | JSON string (any bytes, valid UTF-8 or not) | U32 | **byte-exact bytes; validity not asserted** |
 | JSON `true` / `false` | F64 | `1` / `0` (documented coercion) |
@@ -342,14 +342,14 @@ Given source NDJSON `S` and a schema `Σ` (either explicit or inferred), the LBK
 | JSON `null` | U32 | `""` (index 0, the reserved empty-string entry; BS-20: null is lane-neutral) |
 | absent field (missing in source) | F64 | `0` (default) |
 | absent field | U32 | `""` (index 0 is the reserved empty-string entry) |
-| field in source but NOT in the schema | -- | silently dropped (matches lite-bake core) |
+| field in source but NOT in the schema | -- | silently dropped (see decisions/0005-one-unknown-field-policy.md) |
 | field in the schema with wrong value type (post-freeze), except null | -- | `W_LANE_MISMATCH` error, container not produced; null is lane-legal on both lanes (F64 -> 0, U32 -> "") |
 The preservation contract is asserted mechanically by three complementary layers:
-1. **Unit tests** — `test/RoundTrip.test.js`, `test/StringRoundTrip.test.js` verify individual known-value scenarios.
-2. **Property-based fuzz** — `test/DataPreservation.test.js` generates arbitrary random shapes across 12 seeds and asserts exact preservation on every cell of every row.
-3. **Torture-scale verification** — `bench/torture.js` runs the `verifyRoundTrip` helper (`test/_verify.js`) OUTSIDE the GC-measured band, asserting that every scenario opting into `verify: true` produces a container whose Reader output matches its source NDJSON byte-for-byte. The verification budget currently covers ~73,500 rows across 12 scenarios per fast-tier run; the `verified` column in the torture report shows the row count actually checked.
+1. **Unit tests** -- `test/RoundTrip.test.js`, `test/StringRoundTrip.test.js` verify individual known-value scenarios.
+2. **Property-based fuzz** -- `test/DataPreservation.test.js` generates arbitrary random shapes across 12 seeds and asserts exact preservation on every cell of every row.
+3. **Torture-scale verification** -- `node --expose-gc test/torture.mjs` (44 fast / 47 full scenarios) runs the `verifyRoundTrip` helper (`test/_verify.js`) OUTSIDE the GC-measured band, asserting that every scenario opting into `verify: true` produces a container whose Reader output matches its source NDJSON byte-for-byte. The verification budget currently covers ~103,000 rows across 20 scenarios per fast-tier run; the `verified` column in the torture report shows the row count actually checked.
 The boundary between "bit-exact" and "1 ULP" for F64 is pinned by `test/NumericBoundary.test.js`. Any drift in the tokenizer's number parser trips these tests before it can reach production.