@zakkster/lite-bake-stream 1.1.0 → 1.3.0

package/CHANGELOG.md

@@ -2,6 +2,64 @@
 
 All notable changes to this project will be documented in this file. This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.3.0] -- 2026-09-01
+
+M2 -- a container is verified at the door or refused at the door. The four readers (`Reader`, `PreserveReader`, `RangeReader`, `MultiReader`) now agree with each other and with the `Writer` about the bytes between them; every malformed container names its refusal with a code, and a standalone SPEC byte-checker (reads SPEC.md offsets only, imports nothing from `src/`) is the contract's executable form. All new validation runs at open/parse/construction time -- the `get`/`syncRange`/`findShards` row loops and both tokenizer byte loops are byte-identical. Suite: 432 tests / 431 pass / 0 fail / 1 todo. Torture: 37/37 fast, 39/39 full, arrayBuffers growth 0. Mutation check: all 52 door tests fail on source revert; all 23 behavior-unchanged pins hold.
+
+### Added
+
+- New stable refusal codes: `R_RESERVED_NONZERO` (a reserved header/descriptor/shard field is non-zero), `R_BAD_FOOTER` (footer `magic_end` or `footer_len` malformed), `R_BAD_METADATA` (a non-zero `metadata_off` pointing at an unparseable zone-map segment), and `M_TOO_MANY_ROWS` (a `MultiReader` union whose cumulative row count exceeds 2^53).
+- `src/Views.js` (internal, not exported): a shared `toContainerBuffer` helper so `deserialize`, `Reader.fromBuffer`, and `PreserveReader.fromBuffer` resolve caller bytes by one rule -- a full-buffer view is unwrapped in place, a partial view is copied.
+
+### Changed
+
+- Malformed containers that previously opened, half-opened, or failed later with a raw `RangeError` now refuse at the door with a code. Reserved bytes, footer shape, structural bounds (`row_count * row_stride <= payload_len`, `offset_in_row + lane_bytes <= row_stride`, name-blob and payload extents, monotonic string-table offsets with a `blob_length` sentinel, `total_rows == sum(row_count)`) are all checked at parse time. Every byte-prefix of a valid container either opens fully or throws an `R_`/`M_` code -- no raw `RangeError`, no clamped subarray, no silent short read.
+- `findShards`/`shardBounds` with a field name absent from the schema now throw `R_UNKNOWN_FIELD` (`M_UNKNOWN_FIELD` on `MultiReader`) instead of silently returning all shards / `null`. A field present in the schema but untracked by zone maps still returns all shards / `null` (decisions/0005).
+- F64-only shards no longer emit a 16-byte empty local string table -- such containers are `16 * shardCount` bytes smaller and their ShardEntries carry `local_string_off = 0` / `local_string_len = 0` (SPEC 3.4). `mergeContainers` applies the same rule to its output even when fed a pre-M2 container that violated it.
+- `RangeReader.open` performs one additional 16-byte footer fetch (open-phase fetch count 5 -> 6 for a container with zone maps), so a wrong-footer or short container is refused before the schema fetch.
+- `MultiReader` cumulative row/shard offsets moved `Uint32Array` -> `Float64Array`, so a union totaling more than 2^32 rows locates rows correctly instead of wrapping to `null`.
+
+### Fixed
+
+- BS-06: a truncated preserve container no longer returns a clamped short `subarray`; the shard directory is validated at open.
+- BS-08: reserved header/descriptor/shard bytes and the footer are now enforced; a lying `metadata_off` throws `R_BAD_METADATA` instead of silently degrading or throwing a raw `RangeError`.
+- BS-09: `Reader.fromBuffer` now respects a `Uint8Array`'s `byteOffset`/`byteLength`, copying a partial view instead of reading a sibling's bytes from a pooled buffer.
+- BS-10: `RangeReader.open` on a preserve-mode container throws `R_WRONG_MODE` naming `PreserveReader`, instead of misreading it as schema mode.
+- BS-15: F64-only shards emit no local string table, matching SPEC 3.4's 0/0 rule.
+- BS-16: `MultiReader` over sub-readers totaling more than 2^32 rows addresses every row exactly; beyond 2^53 it refuses with `M_TOO_MANY_ROWS`.
+- BS-17: one unknown-field policy across `get`/`findShards`/`shardBounds` on all readers (decisions/0005).
+- Sink-poison rider (M1 deferred): a sink callback that throws mid-`feed()`/`end()` now leaves the tokenizer poisoned; the next `feed()`/`end()` throws `E_POISONED`. Both tokenizers, via an armed-flag pattern with zero `try`/`catch` and two boolean stores per chunk.
+
+## [1.2.0] -- 2026-08-31
+
+M1 -- the write path refuses what it cannot store. Eight findings closed by a set of doors that all run at construction, freeze, or terminal transition; the byte and record hot paths are unchanged (proven by the torture t6 gate, numbers within noise of M0). Suite: 359 tests / 358 pass / 0 fail / 1 todo. Torture: 37/37 fast, 39/39 full, arrayBuffers growth 0. Mutation check: all 63 door tests fail on source revert; all 13 behavior-unchanged pins hold.
+
+### Added
+
+- `src/Opts.js` (internal, not exported): a shared construction-time options validator wired into `Tokenizer`, `Writer`, `PreserveTokenizer`, `PreserveWriter`, `serialize`, `ingestStream`/`ingestFile`, `RangeReader`, `HTTPRangeAdapter.open`, and the `./split` subpath (`splitNDJSON`, `compilePart`, `compileInParts`). Unknown keys throw with a did-you-mean hint (Levenshtein distance <= 2); out-of-domain values throw naming the key, the received value, and the domain.
+- New stable error codes: `E_UNKNOWN_OPTION`, `E_OPTION_VALUE`, `E_OPTION_CONFLICT` (options); `W_DUPLICATE_FIELD`, `W_SCHEMA_TOO_WIDE`, `W_FIELD_NAME_INVALID` (schema freeze); `W_FINALIZED` on the schema `Writer` (already present on `PreserveWriter`); `E_ENDED`, `E_POISONED` (tokenizer terminal state); `E_STRING_TOO_LONG` (maxStringBytes cap).
+
+### Changed
+
+- Unknown or typo'd option keys now throw `E_UNKNOWN_OPTION` instead of being silently ignored.
+- Explicit `0` now means 0 for `sampleBytes` (freeze from the first record), `maxCachedShards` (retain no shard), and `maxRecordBytes` (unlimited) instead of silently becoming the default. `null` is out-of-domain and throws (null is not zero).
+- `Writer.finalize()` called twice now throws `W_FINALIZED` instead of returning the stale container.
+- Over-cap strings now throw the dedicated `E_STRING_TOO_LONG` instead of the wrong `E_UNEXPECTED_BYTE`, and the cap is enforced exactly (no longer only when the 4096-byte buffer grows).
+- `Tokenizer` no longer writes NOOP methods into the caller's sink object; a frozen sink now constructs and drives a full feed/end.
+- `splitNDJSON`/`compileInParts`: an explicit `targetParts:0` or `maxPartBytes:0` now throws instead of silently becoming the default 4 / no-cap, and a negative `targetParts` now throws instead of clamping to 1.
+- `compilePart`/`compileInParts`: `preserve` was never supported and was silently ignored; it is now an unknown-key error (use `serialize(..., {preserve:true})` for preserve mode).
+
+### Fixed
+
+- **BS-02:** an over-wide schema (a field offset past the u16 `offset_in_row` ceiling of 65535) now throws `W_SCHEMA_TOO_WIDE` at freeze instead of wrapping the offset and cross-writing fields. 8192 all-F64 fields freeze; 8193 throw.
+- **BS-03:** duplicate schema field names now throw `W_DUPLICATE_FIELD` at freeze instead of writing a container the Writer reads from one lane and the Reader from another. See `decisions/0003-duplicate-field-names.md`.
+- **BS-04:** a depth-0 non-object record (number, string, true, false, null) now throws `W_TOP_LEVEL_NOT_OBJECT` in schema mode instead of silently vanishing. Preserve mode is unchanged. See `decisions/0002-record-shape-policy.md`.
+- **BS-11:** every construction-time options object is validated at every entry point that takes one; unknown keys and out-of-domain values fail closed.
+- **BS-12:** `maxStringBytes` is enforced exactly against the running string length, so a cap below the initial 4096-byte buffer is honored, and over-cap strings throw `E_STRING_TOO_LONG`.
+- **BS-13:** the schema `Writer` refuses any sink event or a second `finalize()` after `finalize()` with `W_FINALIZED`, matching `PreserveWriter`.
+- **BS-14:** `Tokenizer` and `PreserveTokenizer` refuse `feed()`/`end()` after `end()` (`E_ENDED`) and after any thrown error (`E_POISONED`); a terminal instance never resurrects.
+- **BS-19:** `Tokenizer` reads each sink method once at construction into a per-instance dispatch field (bound to the sink), so the caller's sink object is never mutated and a frozen sink works.
+
 ## [1.1.0] -- 2026-08-31
 
 ### Added

package/README.md

@@ -13,7 +13,7 @@
 
 > Streaming byte-level JSON compiler for [`@zakkster/lite-bake`](https://github.com/PeshoVurtoleta/lite-bake). Zero-GC, tree-shakeable, gigabyte-scale.
 
-**Status:** v1.1.0. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.
+**Status:** v1.3.0. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.
 
 ## Two modes, one API
 

package/llms.txt

@@ -8,13 +8,13 @@ Ingest gigabyte-scale JSON (top-level array or NDJSON) into the `lite-bake` LBK1
 
 ## Status
 
-v1.1.0 — stable. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak (M1 MacBook Pro): 98.37M rows, 4.89 GB container, zero major GC, zero minor GC, 499 KB total heap allocation, 590.21M cells verified byte-exact, zero mismatches. Tokenizer benches at 222-237 MB/s (~55% of JSON.parse, with no object graph allocated).
+v1.3.0 — stable. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak (M1 MacBook Pro): 98.37M rows, 4.89 GB container, zero major GC, zero minor GC, 499 KB total heap allocation, 590.21M cells verified byte-exact, zero mismatches. Tokenizer benches at 222-237 MB/s (~55% of JSON.parse, with no object graph allocated).
 
 Public API follows semver from 1.0.0. Future additions (I64 lane, columnar payload mode, container-level string table) land via the format's forward-compat seams -- `min_reader_version` on ShardEntry, reserved FieldDescriptor flags, the `metadata_off` block wrapper -- without a format_version bump.
 
 See SPEC.md for the LBK1 container format, section 3.6 for zone maps, section 4.3 for the reserved field flags.
 
-## Public API (v1.1.0)
+## Public API (v1.3.0)
 
 Two ingest modes share one top-level API:
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zakkster/lite-bake-stream",
-  "version": "1.1.0",
+  "version": "1.3.0",
   "description": "Streaming byte-level JSON to lite-bake binary compiler. Zero-GC, tree-shakeable, gigabyte-scale.",
   "type": "module",
   "sideEffects": false,
@@ -109,6 +109,10 @@
   "engines": {
     "node": ">=18"
   },
+  "funding": {
+    "type": "github",
+    "url": "https://github.com/sponsors/PeshoVurtoleta"
+  },
   "publishConfig": {
     "access": "public"
   },

package/src/FileIngest.js

@@ -20,13 +20,24 @@
 // most engines), use server-side pre-compilation and RangeReader instead.
 
 import { Tokenizer } from './Tokenizer.js';
-import { Writer } from './Writer.js';
+import { Writer, WriterError } from './Writer.js';
 import { Reader } from './Reader.js';
 import { PreserveTokenizer } from './PreserveTokenizer.js';
 import { PreserveWriter } from './PreserveWriter.js';
 import { PreserveReader } from './PreserveReader.js';
+import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.1.0';
+export const VERSION = '1.3.0';
+
+const U32_MAX = 4294967295;
+const INGEST_OPTS = {
+  preserve: { t: 'bool' },
+  framing: { t: 'enum', values: ['auto', 'array', 'ndjson'] },
+  writer: { t: 'obj' },
+  onProgress: { t: 'fn' },
+  totalBytes: { t: 'int', min: 0, max: Number.MAX_SAFE_INTEGER },
+};
+function raiseIngest(code, msg) { throw new WriterError(code, msg); }
 
 // Ingest a ReadableStream<Uint8Array> and return a Reader over the resulting
 // LBK1 container. Options:
@@ -42,16 +53,23 @@ export const VERSION = '1.1.0';
 //   totalBytes:         optional total for progress percentage. Pass file.size
 //                       when ingesting a File; omit for indeterminate streams.
 export async function ingestStream(readableStream, opts) {
+  // Validated once, here, after ingestFile's totalBytes merge (R6): a single
+  // label, no double report.
+  checkOpts('ingestStream', opts, INGEST_OPTS, raiseIngest);
   opts = opts || {};
   const preserve = opts.preserve === true;
-  const framing  = opts.framing || 'ndjson';
-  const writerOpts = opts.writer || {};
-  const onProgress = typeof opts.onProgress === 'function' ? opts.onProgress : null;
-  const totalBytes = typeof opts.totalBytes === 'number' ? opts.totalBytes : -1;
+  const framing  = opts.framing !== undefined ? opts.framing : 'ndjson';
+  const writerOpts = opts.writer !== undefined ? opts.writer : {};
+  const onProgress = opts.onProgress !== undefined ? opts.onProgress : null;
+  const totalBytes = opts.totalBytes !== undefined ? opts.totalBytes : -1;
+  if (preserve && opts.framing !== undefined && opts.framing !== 'ndjson') {
+    throw new WriterError('E_OPTION_CONFLICT',
+      "ingestStream: preserve mode is NDJSON-only; remove framing:'" + opts.framing + "' or set preserve:false");
+  }
 
   const writer = preserve ? new PreserveWriter(writerOpts) : new Writer(writerOpts);
   const tokenizer = preserve
-    ? new PreserveTokenizer(writer, { framing, maxRecordBytes: writerOpts.maxRecordBytes || 0 })
+    ? new PreserveTokenizer(writer, { framing, maxRecordBytes: writerOpts.maxRecordBytes !== undefined ? writerOpts.maxRecordBytes : 0 })
     : new Tokenizer(writer, { framing });
 
   const reader = readableStream.getReader();

package/src/MultiReader.js

@@ -21,10 +21,11 @@
 // Error codes:
 //   M_EMPTY               - no readers passed
 //   M_SCHEMA_MISMATCH     - schemas differ across readers
-//   M_UNKNOWN_FIELD       - .get() / .fieldIndex() called with unknown name
+//   M_UNKNOWN_FIELD       - .get() / .fieldIndex() / findShards / shardBounds with unknown name
 //   M_ROW_OUT_OF_RANGE    - rowIdx >= totalRows
+//   M_TOO_MANY_ROWS       - cumulative row count exceeds Number.MAX_SAFE_INTEGER
 
-export const VERSION = '1.1.0';
+export const VERSION = '1.3.0';
 
 export class MultiReaderError extends Error {
   constructor(code, msg) { super(msg); this.code = code; this.name = 'MultiReaderError'; }
@@ -60,15 +61,26 @@ export class MultiReader {
 
     // Cumulative row offsets and shard offsets. Row rowIdx belongs to reader
     // r iff _rowOffsets[r] <= rowIdx < _rowOffsets[r+1].
+    // Float64Array (not Uint32Array): cumulative row/shard offsets can exceed
+    // 2^32 across many large sub-readers. F64 is exact to 2^53 and stays flat
+    // and monotonic, so the linear scan in _locateRow/_locateShard is unchanged
+    // apart from the element type (BS-16).
     let cumRows = 0, cumShards = 0;
-    this._rowOffsets   = new Uint32Array(readers.length + 1);
-    this._shardOffsets = new Uint32Array(readers.length + 1);
+    this._rowOffsets   = new Float64Array(readers.length + 1);
+    this._shardOffsets = new Float64Array(readers.length + 1);
     for (let i = 0; i < readers.length; i++) {
       this._rowOffsets[i]   = cumRows;
       this._shardOffsets[i] = cumShards;
       cumRows   += readers[i].totalRows;
       cumShards += readers[i].shardCount;
     }
+    // Boundary is 2^53 (BRIEF T6 / A8): total_rows == 2^53 keeps the largest
+    // row index at 2^53-1 (= MAX_SAFE_INTEGER), still exactly representable;
+    // beyond 2^53 an index collides with its neighbour in F64.
+    if (cumRows > 2 ** 53) {
+      throw new MultiReaderError('M_TOO_MANY_ROWS',
+        'cumulative row count ' + cumRows + ' exceeds 2^53; row indices past 2^53 are not exactly addressable in F64');
+    }
     this._rowOffsets[readers.length]   = cumRows;
     this._shardOffsets[readers.length] = cumShards;
     this._totalRows = cumRows;
@@ -129,6 +141,8 @@ export class MultiReader {
   // reader; results are concatenated and re-numbered into the global shard
   // index space.
   findShards(fieldName, opts) {
+    if (!this._fieldIndex.has(fieldName))
+      throw new MultiReaderError('M_UNKNOWN_FIELD', 'no field named ' + JSON.stringify(fieldName));
     const out = [];
     for (let r = 0; r < this._readers.length; r++) {
       const localShards = this._readers[r].findShards(fieldName, opts);
@@ -139,6 +153,8 @@ export class MultiReader {
   }
 
   shardBounds(globalShardIdx, fieldName) {
+    if (!this._fieldIndex.has(fieldName))
+      throw new MultiReaderError('M_UNKNOWN_FIELD', 'no field named ' + JSON.stringify(fieldName));
     const loc = this._locateShard(globalShardIdx);
     if (!loc) return null;
     return this._readers[loc.readerIdx].shardBounds(loc.localShard, fieldName);

package/src/Opts.js

@@ -0,0 +1,189 @@
+// @zakkster/lite-bake-stream / Opts (internal, not exported from index.js)
+// Copyright (c) 2026 Zahary Shinikchiev. MIT.
+//
+// Shared construction-time options validator. Every public constructor and
+// entry function that takes an opts object runs checkOpts once, at construction
+// or in a synchronous prologue -- never per byte, never per record. An unknown
+// key is an error with a did-you-mean hint (suite law: no silent ignore). An
+// out-of-domain value is an error (suite law: null is not zero; explicit 0 is 0
+// where 0 is in-domain). This file allocates only on the throw path.
+//
+// Each host declares a module-level frozen spec object literal mapping every
+// known key to a domain descriptor:
+//   { t: 'enum', values: [...] }        - string, must be a listed member
+//   { t: 'int',  min, max }             - integer in [min, max] inclusive
+//   { t: 'bool' }                       - boolean
+//   { t: 'fn' }                         - function
+//   { t: 'obj', nullable }              - object (contents validated downstream)
+// A value of `undefined` for any key means "use the default" and is skipped,
+// matching each host's `opts.k !== undefined ? opts.k : DEFAULT` defaulting.
+//
+// OPTS DOMAIN TABLE (ships with the package; the M1 plan is the source of truth)
+//
+// Unknown-key rule (all entry points): throw E_UNKNOWN_OPTION. Levenshtein
+// distance from the offending key to every known key; if the minimum is <= 2,
+// message is "unknown option 'framming' -- did you mean 'framing'?"; else
+// "unknown option 'zzz' (known: framing, maxStringBytes)". Ties break by spec
+// declaration order.
+//
+//   Tokenizer(sink,opts)
+//     framing         enum 'auto'|'array'|'ndjson'      def 'auto'
+//     maxStringBytes  int [1, 4294967295]               def 1048576   (0 forbidden)
+//   Writer(opts)
+//     schema          object|null with array fields     def null
+//     targetShardBytes int [1, 4294967295]              def 33554432  (0 forbidden)
+//     sampleBytes     int [0, 4294967295]               def targetShardBytes  (0 = freeze on record 1)
+//   PreserveTokenizer(sink,opts)
+//     framing         enum 'ndjson'                      def 'ndjson'
+//     maxRecordBytes  int [0, 4294967295]               def 0         (0 = unlimited)
+//   PreserveWriter(opts)
+//     targetShardBytes int [1, 4294967295]              def 8388608   (0 forbidden)
+//     maxRecordBytes  int [0, 4294967295]               declared, consumed by PreserveTokenizer
+//   serialize(input,opts) / ingestStream / ingestFile
+//     preserve        bool                               def false
+//     framing         enum 'auto'|'array'|'ndjson'       def 'ndjson'  (conflict if preserve && != 'ndjson')
+//     writer          object                             def {}
+//     onProgress      function                           (ingest only)
+//     totalBytes      int [0, 9007199254740991]          def -1        (ingest only)
+//   RangeReader(adapter,opts) / .open
+//     maxCachedShards int [0, 4294967295]               def 8         (0 = retain no shard)
+//   HTTPRangeAdapter.open(url,opts)
+//     fetch           function                           def globalThis.fetch
+//   splitNDJSON(bytes,opts) / compileInParts
+//     targetParts     int [1, 4294967295]               def 4         (0 forbidden)
+//     maxPartBytes    int [1, 4294967295] or Infinity   def Infinity  (0 forbidden; Infinity = no cap)
+//   compilePart(bytes,opts) / compileInParts
+//     framing         enum 'auto'|'array'|'ndjson'       def 'ndjson'
+//     writer          object                             def {}
+//
+// Non-integer, NaN, Infinity, negative, null, and wrong-typeof all land on
+// E_OPTION_VALUE with a message naming key, received value, and the domain.
+
+export function checkOpts(label, opts, spec, raise) {
+  if (opts === undefined || opts === null || typeof opts !== 'object') return;
+  for (const key in opts) {
+    if (!Object.prototype.hasOwnProperty.call(opts, key)) continue;
+    const d = spec[key];
+    if (d === undefined) {
+      const keys = Object.keys(spec);
+      const near = nearestKey(key, keys);
+      if (near !== null) {
+        raise('E_UNKNOWN_OPTION',
+          label + ": unknown option '" + key + "' -- did you mean '" + near + "'?");
+      } else {
+        raise('E_UNKNOWN_OPTION',
+          label + ": unknown option '" + key + "' (known: " + keys.join(', ') + ')');
+      }
+      continue;
+    }
+    const v = opts[key];
+    if (v === undefined) continue;   // explicit undefined == use default
+    _checkValue(label, key, v, d, raise);
+  }
+}
+
+function _checkValue(label, key, v, d, raise) {
+  const t = d.t;
+  if (t === 'int') {
+    if (d.inf && v === Infinity) return;   // Infinity is a legal explicit value for this key
+    if (typeof v !== 'number' || !Number.isInteger(v) || v < d.min || v > d.max) {
+      raise('E_OPTION_VALUE',
+        label + ": option '" + key + "' must be an integer in [" +
+        d.min + ', ' + d.max + ']' + (d.inf ? ' or Infinity' : '') + '; got ' + _show(v));
+    }
+    return;
+  }
+  if (t === 'enum') {
+    if (typeof v !== 'string' || d.values.indexOf(v) < 0) {
+      raise('E_OPTION_VALUE',
+        label + ": option '" + key + "' must be one of " +
+        _quoteList(d.values) + '; got ' + _show(v));
+    }
+    return;
+  }
+  if (t === 'bool') {
+    if (typeof v !== 'boolean') {
+      raise('E_OPTION_VALUE',
+        label + ": option '" + key + "' must be a boolean; got " + _show(v));
+    }
+    return;
+  }
+  if (t === 'fn') {
+    if (typeof v !== 'function') {
+      raise('E_OPTION_VALUE',
+        label + ": option '" + key + "' must be a function; got " + _show(v));
+    }
+    return;
+  }
+  if (t === 'obj') {
+    if (v === null) {
+      if (!d.nullable) {
+        raise('E_OPTION_VALUE',
+          label + ": option '" + key + "' must be an object; got null");
+      }
+      return;
+    }
+    if (typeof v !== 'object' || Array.isArray(v)) {
+      raise('E_OPTION_VALUE',
+        label + ": option '" + key + "' must be an object; got " + _show(v));
+    }
+  }
+}
+
+function _show(v) {
+  if (typeof v === 'string') return "'" + v + "'";
+  if (v === null) return 'null';
+  if (typeof v === 'object') return Array.isArray(v) ? 'array' : 'object';
+  return String(v);
+}
+
+function _quoteList(values) {
+  let s = '';
+  for (let i = 0; i < values.length; i++) {
+    if (i > 0) s += ', ';
+    s += "'" + values[i] + "'";
+  }
+  return s;
+}
+
+// Nearest known key by Levenshtein distance, returning it only when the minimum
+// distance is <= 2. Ties break by declaration order (strict < keeps the first).
+export function nearestKey(key, knownKeys) {
+  let best = null;
+  let bestDist = 3;
+  const klen = key.length;
+  for (let k = 0; k < knownKeys.length; k++) {
+    const cand = knownKeys[k];
+    const diff = klen - cand.length;
+    if (diff > 2 || diff < -2) continue;   // length delta alone exceeds the cap
+    const dist = _lev(key, cand);
+    if (dist < bestDist) { bestDist = dist; best = cand; }
+  }
+  return bestDist <= 2 ? best : null;
+}
+
+// Two-row Levenshtein. Allocates two small arrays; only reached on the throw
+// path (nearestKey is called only when an unknown key is being reported).
+function _lev(a, b) {
+  const n = a.length, m = b.length;
+  if (n === 0) return m;
+  if (m === 0) return n;
+  let prev = new Array(m + 1);
+  let curr = new Array(m + 1);
+  for (let j = 0; j <= m; j++) prev[j] = j;
+  for (let i = 1; i <= n; i++) {
+    curr[0] = i;
+    const ca = a.charCodeAt(i - 1);
+    for (let j = 1; j <= m; j++) {
+      const cost = ca === b.charCodeAt(j - 1) ? 0 : 1;
+      let v = prev[j] + 1;
+      const del = curr[j - 1] + 1;
+      if (del < v) v = del;
+      const sub = prev[j - 1] + cost;
+      if (sub < v) v = sub;
+      curr[j] = v;
+    }
+    const tmp = prev; prev = curr; curr = tmp;
+  }
+  return prev[m];
+}

package/src/PreserveReader.js

@@ -16,11 +16,27 @@
 // The Uint8Array returned by getBytes is a live subarray view of the container
 // buffer. It is valid for as long as the Reader instance holds the container.
 // Do NOT mutate it, and do NOT retain it past the Reader's lifetime.
+//
+// Error codes:
+//   R_BAD_MAGIC              - header magic mismatch
+//   R_UNSUPPORTED_VERSION    - format_version > 1
+//   R_UNSUPPORTED_ENDIAN     - endian byte is not LE
+//   R_WRONG_MODE             - schema-mode container fed to the PreserveReader
+//   R_BAD_FLAGS              - unknown header flag bits set
+//   R_TRUNCATED              - a structure claims bytes past the buffer end
+//   R_RESERVED_NONZERO       - a reserved header/shard field is non-zero
+//   R_BAD_FOOTER             - footer magic_end or footer_len is malformed
+//   R_INVALID                - a structure is internally inconsistent but in-bounds
+//   R_SHARD_VERSION_TOO_NEW  - a shard's min_reader_version > this reader
+//   R_ROW_OUT_OF_RANGE       - rowIdx >= totalRows
+
+import { toContainerBuffer } from './Views.js';
 
-export const VERSION = '1.1.0';
+export const VERSION = '1.3.0';
 
 const CONTAINER_HEADER_BYTES = 48;
 const SHARD_ENTRY_BYTES = 40;
+const FOOTER_BYTES = 16;
 
 export class PreserveReaderError extends Error {
   constructor(code, msg) { super(msg); this.code = code; this.name = 'PreserveReaderError'; }
@@ -28,16 +44,7 @@ export class PreserveReaderError extends Error {
 
 export class PreserveReader {
   static fromBuffer(input) {
-    if (input instanceof ArrayBuffer) return new PreserveReader(input);
-    if (input instanceof Uint8Array) {
-      if (input.byteOffset === 0 && input.byteLength === input.buffer.byteLength) {
-        return new PreserveReader(input.buffer);
-      }
-      const copy = new Uint8Array(input.byteLength);
-      copy.set(input);
-      return new PreserveReader(copy.buffer);
-    }
-    throw new TypeError('PreserveReader.fromBuffer: expected ArrayBuffer or Uint8Array');
+    return new PreserveReader(toContainerBuffer(input, 'PreserveReader.fromBuffer'));
   }
 
   constructor(buffer) {
@@ -49,6 +56,7 @@ export class PreserveReader {
     this._bytes = new Uint8Array(buffer);
     this._decoder = new TextDecoder('utf-8', { fatal: false });
     this._parseHeader();
+    this._parseFooter();
     this._parseShardDirectory();
   }
 
@@ -72,17 +80,40 @@ export class PreserveReader {
     if (flags & ~0x01) {
       throw new PreserveReaderError('R_BAD_FLAGS', 'unknown flag bits set: 0x' + flags.toString(16));
     }
+    const reserved1 = this._dv.getUint32(36, true);
+    if (reserved1 !== 0)
+      throw new PreserveReaderError('R_RESERVED_NONZERO', 'header reserved1 at offset 36 must be 0, got ' + reserved1);
 
     this._schemaBlockOff = Number(this._dv.getBigUint64(8, true));
     this._metadataOff    = Number(this._dv.getBigUint64(16, true));
     this._shardDirOff    = Number(this._dv.getBigUint64(24, true));
     this._shardCount     = this._dv.getUint32(32, true);
-    // 4 bytes reserved at 36
     this._totalRows      = Number(this._dv.getBigUint64(40, true));
 
     if (this._schemaBlockOff !== 0) {
       throw new PreserveReaderError('R_INVALID', 'preserve container has non-zero schema_block_off');
     }
+    if (this._metadataOff !== 0) {
+      throw new PreserveReaderError('R_INVALID', 'preserve container has non-zero metadata_off (SPEC 3.8)');
+    }
+    const len = this._buffer.byteLength;
+    if (this._shardDirOff < CONTAINER_HEADER_BYTES || this._shardDirOff >= len)
+      throw new PreserveReaderError('R_TRUNCATED', 'shard_directory_off ' + this._shardDirOff + ' out of range [48, ' + len + ')');
+  }
+
+  // Footer (SPEC 3.7): magic_end '1KBL' + footer_len >= 16.
+  _parseFooter() {
+    const len = this._buffer.byteLength;
+    const footerOff = len - FOOTER_BYTES;
+    const b = this._bytes;
+    if (b[footerOff + 8] !== 0x31 || b[footerOff + 9] !== 0x4B ||
+        b[footerOff + 10] !== 0x42 || b[footerOff + 11] !== 0x4C)
+      throw new PreserveReaderError('R_BAD_FOOTER', 'footer magic_end at offset ' + (footerOff + 8) + ' is not 1KBL');
+    const footerLen = this._dv.getUint32(footerOff + 12, true);
+    if (footerLen < FOOTER_BYTES)
+      throw new PreserveReaderError('R_BAD_FOOTER', 'footer_len ' + footerLen + ' is less than the minimum ' + FOOTER_BYTES);
+    if (footerLen > len - CONTAINER_HEADER_BYTES)
+      throw new PreserveReaderError('R_BAD_FOOTER', 'footer_len ' + footerLen + ' exceeds the container body size');
   }
 
   _parseShardDirectory() {
@@ -100,21 +131,46 @@ export class PreserveReader {
       const payloadLen = this._dv.getUint32(entryOff + 8, true);
       const rowCount   = this._dv.getUint32(entryOff + 12, true);
       const minReaderVer = this._dv.getUint16(entryOff + 16, true);
+      const shardFlags   = this._dv.getUint16(entryOff + 18, true);
+      const shardReserved = this._dv.getUint32(entryOff + 20, true);
+      const localStrOff  = Number(this._dv.getBigUint64(entryOff + 24, true));
+      const localStrLen  = Number(this._dv.getBigUint64(entryOff + 32, true));
       if (minReaderVer > 1) {
         throw new PreserveReaderError('R_SHARD_VERSION_TOO_NEW',
           'shard ' + i + ' requires reader version ' + minReaderVer);
       }
+      if (shardFlags !== 0) throw new PreserveReaderError('R_RESERVED_NONZERO', 'shard ' + i + ' flags must be 0, got ' + shardFlags);
+      if (shardReserved !== 0) throw new PreserveReaderError('R_RESERVED_NONZERO', 'shard ' + i + ' reserved must be 0, got ' + shardReserved);
+      if (localStrOff !== 0 || localStrLen !== 0)
+        throw new PreserveReaderError('R_INVALID', 'shard ' + i + ' local_string_* must be 0 in preserve mode (SPEC 3.8)');
+      if (payloadOff < CONTAINER_HEADER_BYTES || payloadOff + payloadLen > this._buffer.byteLength)
+        throw new PreserveReaderError('R_TRUNCATED', 'shard ' + i + ' payload [' + payloadOff + ', ' + (payloadOff + payloadLen) + ') out of range');
       // Trailing u32 offset table.
       const offsetTableOff = payloadOff + payloadLen - rowCount * 4;
       if (offsetTableOff < payloadOff) {
         throw new PreserveReaderError('R_INVALID',
           'shard ' + i + ' payload_len (' + payloadLen + ') too small for offset table of ' + rowCount + ' rows');
       }
+      const blobLen = offsetTableOff - payloadOff;
+      // Offset-table walk: offsets[0] must be 0, non-decreasing, last <= blobLen.
+      if (rowCount > 0) {
+        let prev = this._dv.getUint32(offsetTableOff, true);
+        if (prev !== 0)
+          throw new PreserveReaderError('R_INVALID', 'shard ' + i + ' offset table[0] is ' + prev + ', must be 0');
+        for (let r = 1; r < rowCount; r++) {
+          const cur = this._dv.getUint32(offsetTableOff + r * 4, true);
+          if (cur < prev)
+            throw new PreserveReaderError('R_INVALID', 'shard ' + i + ' offset table not monotonic at record ' + r);
+          prev = cur;
+        }
+        if (prev > blobLen)
+          throw new PreserveReaderError('R_INVALID', 'shard ' + i + ' offset table last entry ' + prev + ' exceeds blob length ' + blobLen);
+      }
       shards[i] = {
         payloadOff,
         payloadLen,
         rowCount,
-        blobLen: offsetTableOff - payloadOff,
+        blobLen,
         offsetTableOff,
         firstRow: cumulativeRow,
         endRow: cumulativeRow + rowCount,