@zakkster/lite-bake-stream 1.0.0 → 1.1.0

package/CHANGELOG.md

@@ -2,6 +2,26 @@
 
 All notable changes to this project will be documented in this file. This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.1.0] -- 2026-08-31
+
+### Added
+
+- Torture gate at the pipeline's canonical path: `node --expose-gc test/torture.mjs` prints `ok` on success, exit 0/1. Tiers: t5 (the 37-scenario stress suite, relocated from bench/ with behavior unchanged), t6 (zero-alloc gates with `maxArrayBuffersGrowth: 0` via measureOps `stabilize: 'deep'` over the tokenizer steady state, the StringTable intern hit path, and the Reader cell loop, each with structural buffer asserts), t7 (lite-leak retention soak: 4096 full-tier cycles, tracker size 0, arrayBuffers delta 0.00 MB), t9 (four in-process controls plus the `BAKE_TORTURE_BREAK=1` whole-suite control, proven to exit non-zero). Tiers t0-t4/t8 are registered reserved stubs. Preflight exits 2 with a remedy line when devDependencies are not installed.
+- `decisions/` directory (repo-only, not shipped): 0001-reserved-empty-string.md records the BS-01 fix and its two rejected alternatives.
+- Round-trip regression suite hardening: 284 tests (282 pass / 2 todo). Five permanent guards pin the BS-01 fix; the two remaining todos are registered known issues (BS-05 invalid UTF-8 acceptance, BS-03 duplicate schema field names). Absent-field fuzz generation, -0 sign, duplicate-key last-wins, null/bool lane-policy codes, and preserve-mode byte-exactness pins added.
+- `npm run verify` (test + torture).
+
+### Changed
+
+- `npm test` now globs `test/*.test.js`; a new test file runs without editing package.json.
+- `bench/torture.js` is a deprecation shim forwarding to `test/torture.mjs` (same exit codes, same `--tier` flag).
+- devDependencies: added `@zakkster/lite-leak` `^1.10.0`; `@zakkster/lite-gc-profiler` `^1.0.0` -> `^1.16.0` (the `stabilize: 'deep'` option used by the t6 gate landed in 1.16.0).
+- SPEC 3.3: string-table entry 0 is reserved as the empty string (writers MUST emit it); SPEC 7 absent-U32 row now reads `""`. See Fixed.
+
+### Fixed
+
+- **BS-01: absent string fields now decode as `""`.** A record that omits a U32-lane field leaves index 0 in that row slot. Index 0 of a shard's string table used to be whichever string interned first in that shard, so `{"s":"zebra"}` followed by `{}` read row 1 back as `"zebra"` -- silently, and differently depending on record order and where shard boundaries fell. `StringTable` now reserves index 0 as the empty string in its constructor and at every `reset()`, so absent U32 cells decode as `""` on every write path: explicit schema, sample-and-infer, and every shard of a multi-shard drain. Containers written before this change keep their old bytes and their old, data-dependent behavior on those bytes; the fix is not retroactive. Readers are unchanged. SPEC 3.3 and the SPEC 7 coercion table are amended to match.
+
 ## [1.0.0] — 2026-07-12
 
 First stable release. The LBK1 container format is frozen at `format_version: 1`.

package/README.md

@@ -13,7 +13,7 @@
 
 > Streaming byte-level JSON compiler for [`@zakkster/lite-bake`](https://github.com/PeshoVurtoleta/lite-bake). Zero-GC, tree-shakeable, gigabyte-scale.
 
-**Status:** v1.0.0. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.
+**Status:** v1.1.0. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.
 
 ## Two modes, one API
 

package/SPEC.md

@@ -95,7 +95,7 @@ u32   offsets[entry_count + 1]   // byte offsets into blob, INCLUDING a trailing
 u8    blob[blob_length_bytes]
 ```
 
-Padded to 8. Strings are UTF-8, not null-terminated. Length of string `i` is `offsets[i+1] - offsets[i]`; the trailing sentinel makes this valid for `i = entry_count - 1` without a branch. Entry index 0 is not reserved; the empty string is a legitimate entry.
+Padded to 8. Strings are UTF-8, not null-terminated. Length of string `i` is `offsets[i+1] - offsets[i]`; the trailing sentinel makes this valid for `i = entry_count - 1` without a branch. Entry index 0 is reserved: it is always the empty string. Writers MUST emit "" as entry 0 of every shard's local table, including tables that hold no other entries. Readers MAY rely on this for containers written at or after this rule; containers written before it may hold any string at index 0.
 
 Per-shard tables are placed immediately after their shard payload in the container, so a single HTTP Range fetch covering `[payload_off, payload_off + payload_len + local_string_len)` retrieves both.
 
@@ -337,7 +337,7 @@ Given source NDJSON `S` and a schema `Σ` (either explicit or inferred), the LBK
 | JSON string (any valid UTF-8) | U32 | **byte-exact UTF-8 sequence** |
 | JSON `true` / `false` / `null` | F64 | `1` / `0` / `0` (documented coercion) |
 | absent field (missing in source) | F64 | `0` (default) |
-| absent field | U32 | string at index 0 of the shard's local table |
+| absent field | U32 | `""` (index 0 is the reserved empty-string entry) |
 | field in source but NOT in `Σ` | — | silently dropped (matches lite-bake core) |
 | field in `Σ` with wrong value type (post-freeze) | — | `W_LANE_MISMATCH` error, container not produced |
 

package/llms.txt

@@ -8,13 +8,13 @@ Ingest gigabyte-scale JSON (top-level array or NDJSON) into the `lite-bake` LBK1
 
 ## Status
 
-v1.0.0 — stable. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak (M1 MacBook Pro): 98.37M rows, 4.89 GB container, zero major GC, zero minor GC, 499 KB total heap allocation, 590.21M cells verified byte-exact, zero mismatches. Tokenizer benches at 222-237 MB/s (~55% of JSON.parse, with no object graph allocated).
+v1.1.0 — stable. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak (M1 MacBook Pro): 98.37M rows, 4.89 GB container, zero major GC, zero minor GC, 499 KB total heap allocation, 590.21M cells verified byte-exact, zero mismatches. Tokenizer benches at 222-237 MB/s (~55% of JSON.parse, with no object graph allocated).
 
 Public API follows semver from 1.0.0. Future additions (I64 lane, columnar payload mode, container-level string table) land via the format's forward-compat seams -- `min_reader_version` on ShardEntry, reserved FieldDescriptor flags, the `metadata_off` block wrapper -- without a format_version bump.
 
 See SPEC.md for the LBK1 container format, section 3.6 for zone maps, section 4.3 for the reserved field flags.
 
-## Public API (v1.0.0)
+## Public API (v1.1.0)
 
 Two ingest modes share one top-level API:
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zakkster/lite-bake-stream",
-  "version": "1.0.0",
+  "version": "1.1.0",
   "description": "Streaming byte-level JSON to lite-bake binary compiler. Zero-GC, tree-shakeable, gigabyte-scale.",
   "type": "module",
   "sideEffects": false,
@@ -76,10 +76,11 @@
     "LICENSE"
   ],
   "scripts": {
-    "test": "node --test test/Tokenizer.test.js test/RoundTrip.test.js test/StringRoundTrip.test.js test/DataPreservation.test.js test/NumericBoundary.test.js test/RangeReader.test.js test/FileIngest.test.js test/ZoneMaps.test.js test/M4.test.js test/M6.test.js test/Conformance.test.js test/Fuzz.test.js test/OffsetOverflow.test.js test/Preserve.test.js",
+    "test": "node --test test/*.test.js",
     "bench": "node --expose-gc bench/bench-tokenizer.js",
-    "torture": "node --expose-gc bench/torture.js --tier=fast",
-    "torture:full": "node --expose-gc bench/torture.js --tier=full",
+    "torture": "node --expose-gc test/torture.mjs",
+    "torture:full": "node --expose-gc test/torture.mjs --tier=full",
+    "verify": "npm test && npm run torture",
     "soak": "node --expose-gc bench/soak.js --mb=100 --verify",
     "soak:500": "node --expose-gc bench/soak.js --mb=500 --verify",
     "soak:1gb": "node --expose-gc bench/soak.js --gb=1 --verify",
@@ -112,6 +113,7 @@
     "access": "public"
   },
   "devDependencies": {
-    "@zakkster/lite-gc-profiler": "^1.0.0"
+    "@zakkster/lite-gc-profiler": "^1.16.0",
+    "@zakkster/lite-leak": "^1.10.0"
   }
 }

package/src/FileIngest.js

@@ -26,7 +26,7 @@ import { PreserveTokenizer } from './PreserveTokenizer.js';
 import { PreserveWriter } from './PreserveWriter.js';
 import { PreserveReader } from './PreserveReader.js';
 
-export const VERSION = '1.0.0';
+export const VERSION = '1.1.0';
 
 // Ingest a ReadableStream<Uint8Array> and return a Reader over the resulting
 // LBK1 container. Options:

package/src/MultiReader.js

@@ -24,7 +24,7 @@
 //   M_UNKNOWN_FIELD       - .get() / .fieldIndex() called with unknown name
 //   M_ROW_OUT_OF_RANGE    - rowIdx >= totalRows
 
-export const VERSION = '1.0.0';
+export const VERSION = '1.1.0';
 
 export class MultiReaderError extends Error {
   constructor(code, msg) { super(msg); this.code = code; this.name = 'MultiReaderError'; }

package/src/PreserveReader.js

@@ -17,7 +17,7 @@
 // buffer. It is valid for as long as the Reader instance holds the container.
 // Do NOT mutate it, and do NOT retain it past the Reader's lifetime.
 
-export const VERSION = '1.0.0';
+export const VERSION = '1.1.0';
 
 const CONTAINER_HEADER_BYTES = 48;
 const SHARD_ENTRY_BYTES = 40;

package/src/PreserveTokenizer.js

@@ -26,7 +26,7 @@
 //   E_TRUNCATED         - end of input reached mid-record (depth != 0 or in-string)
 //   E_RECORD_TOO_LARGE  - single record exceeds maxRecordBytes (default: none)
 
-export const VERSION = '1.0.0';
+export const VERSION = '1.1.0';
 
 export class PreserveTokenizerError extends Error {
   constructor(code, msg) { super(msg); this.code = code; this.name = 'PreserveTokenizerError'; }

package/src/PreserveWriter.js

@@ -16,7 +16,7 @@
 // Reader dispatch happens via the flag bit; a schema-mode Reader refuses this
 // container with R_WRONG_MODE.
 
-export const VERSION = '1.0.0';
+export const VERSION = '1.1.0';
 
 export class PreserveWriterError extends Error {
   constructor(code, msg) { super(msg); this.code = code; this.name = 'PreserveWriterError'; }

package/src/RangeReader.js

@@ -35,7 +35,7 @@
 
 import { StringTable } from './StringTable.js';
 
-export const VERSION = '1.0.0';
+export const VERSION = '1.1.0';
 
 const CONTAINER_HEADER_BYTES = 48;
 const SHARD_ENTRY_BYTES = 40;

package/src/Reader.js

@@ -18,7 +18,7 @@
 
 import { StringTable } from './StringTable.js';
 
-export const VERSION = '1.0.0';
+export const VERSION = '1.1.0';
 
 const CONTAINER_HEADER_BYTES = 48;
 const SHARD_ENTRY_BYTES = 40;

package/src/Split.js

@@ -36,7 +36,7 @@ import { Writer, WriterError } from './Writer.js';
 import { Reader, ReaderError } from './Reader.js';
 import { StringTable } from './StringTable.js';
 
-export const VERSION = '1.0.0';
+export const VERSION = '1.1.0';
 
 const LF = 0x0A;
 const CONTAINER_HEADER_BYTES = 48;

package/src/StringTable.js

@@ -15,8 +15,12 @@
 //
 // The trailing sentinel makes `len(i) = offsets[i+1] - offsets[i]` uniform for
 // all i including the last — no branch in the reader hot path.
+//
+// Entry 0 is always the empty string: the table reserves it in the constructor
+// and at every reset(), so an absent U32 row cell (which is 0) decodes as ""
+// rather than aliasing the shard's first-interned string (SPEC 3.3, SPEC 7).
 
-export const VERSION = '1.0.0';
+export const VERSION = '1.1.0';
 
 const EMPTY_SLOT = 0xFFFFFFFF;   // MUST be unsigned; typed-array reads are unsigned
 const INITIAL_BLOB_BYTES  = 64 * 1024;
@@ -25,6 +29,10 @@ const INITIAL_HASH_CAP    = 2048;      // load factor target 50%
 const HASH_MAX_LOAD_NUM   = 1;         // 50% load factor: num/den = 1/2
 const HASH_MAX_LOAD_DEN   = 2;
 
+// Reserved entry 0. A zero-length range: no blob bytes, no growth, one hash
+// slot. Module-level so reset() reserves without allocating.
+const EMPTY = new Uint8Array(0);
+
 // FNV-1a 32-bit over a byte range.
 function fnv1a(bytes, from, to) {
   let h = 0x811c9dc5 | 0;
@@ -44,6 +52,7 @@ export class StringTable {
     this._hashCap = INITIAL_HASH_CAP;
     this._hashMask = INITIAL_HASH_CAP - 1;
     this._hashKeys = new Uint32Array(INITIAL_HASH_CAP).fill(EMPTY_SLOT);
+    this._reserveEmpty();
   }
 
   // Returns u32 index of the string with these bytes, adding a new entry if
@@ -76,6 +85,13 @@ export class StringTable {
     return true;
   }
 
+  // Class invariant: index 0 is the empty string. Zero-alloc -- _insertNew with
+  // n === 0 appends no bytes, offsets[0]/[1] already exist, and the load-factor
+  // check cannot trip at count 1.
+  _reserveEmpty() {
+    this.intern(EMPTY, 0, 0);
+  }
+
   _insertNew(bytes, from, to, slot) {
     const n = to - from;
     // Grow blob if needed
@@ -158,6 +174,7 @@ export class StringTable {
     // (we only touched [0..count], so the fill is bounded)
     this._offsets[0] = 0;
     this._hashKeys.fill(EMPTY_SLOT);
+    this._reserveEmpty();
   }
 
   // Serialize into the LBK1 per-shard string-table byte layout described at the

package/src/Tokenizer.js

@@ -17,7 +17,7 @@
 //   E_TRAILING_INPUT       - non-whitespace bytes after top-level value(s) (array mode only)
 //   E_DEPTH_LIMIT          - nesting depth exceeded MAX_DEPTH
 
-export const VERSION = '1.0.0';
+export const VERSION = '1.1.0';
 
 // ---------- byte constants ----------
 const B_SPACE = 0x20, B_TAB = 0x09, B_LF = 0x0A, B_CR = 0x0D;

package/src/Writer.js

@@ -29,7 +29,7 @@
 
 import { StringTable } from './StringTable.js';
 
-export const VERSION = '1.0.0';
+export const VERSION = '1.1.0';
 
 const CONTAINER_HEADER_BYTES = 48;
 const SHARD_ENTRY_BYTES = 40;

package/src/index.js

@@ -26,7 +26,7 @@ export {
   PreserveWriter, PreserveWriterError,
   PreserveReader, PreserveReaderError,
 };
-export const VERSION = '1.0.0';
+export const VERSION = '1.1.0';
 
 const encoder = new TextEncoder();