@zakkster/lite-bake-stream 1.3.0 → 1.3.1

package/CHANGELOG.md

@@ -2,6 +2,18 @@
 
 All notable changes to this project will be documented in this file. This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.3.1] -- 2026-09-01
+
+M3 remainder (BS-05) -- the UTF-8 door ends the promise-A/behavior-B split with a measured decision. Option A (validate at the tokenizer, refuse invalid UTF-8 with `E_INVALID_UTF8`) was built, proven correctness-green (460/460, torture fast 42/42), and rejected on its pre-committed budget: unicode-heavy throughput regressed 229.7 -> 213.1 MB/s (-7.2%, over the 3% ceiling), because the baseline scan bulk-copies `>= 0x80` bytes with zero inspection so validation is pure added cost, while ASCII stayed within noise. SPEC 5.5 is instead rewritten to match the long-standing behavior (documented-permissive), and the dead code is removed. No bytes move at runtime. See decisions/0006-utf8-door.md for the before/after numbers and the rejected option. Suite: 453 tests / 453 pass / 0 fail / 0 todo. Torture: 42/42 fast, 44/44 full, arrayBuffers growth 0. Inverted mutation check: all 49 battery and pin tests pass on baseline src as well -- zero behavior change, proven.
+
+### Changed
+
+- SPEC 5.5 and SPEC 7's U32 row rewritten to documented-permissive: string bytes between the quotes are stored byte-exact through U32 lanes regardless of UTF-8 validity; invalid sequences are not a parse error. `get()`/`getString()` decode with replacement (U+FFFD); `shardStringTable(s).bytesAt(i)` returns the exact stored bytes, the schema-mode twin of `PreserveReader.getBytes(i)`. This is a documentation-and-dead-code change only -- no runtime behavior changed, no new public surface (the accessor already existed).
+
+### Fixed
+
+- BS-05: the SPEC no longer promises a UTF-8 parse error the tokenizer never enforced, and `Tokenizer.js` no longer declares an unreachable `E_INVALID_UTF8` code. The byte-exact string contract is now stated truthfully: invalid bytes round-trip byte-exact and decode with U+FFFD, in schema mode exactly as in preserve mode. Lone-surrogate escapes still refuse with `E_UNPAIRED_SURROGATE` (an escape-grammar rule, unrelated to input-UTF-8 validity).
+
 ## [1.3.0] -- 2026-09-01
 
 M2 -- a container is verified at the door or refused at the door. The four readers (`Reader`, `PreserveReader`, `RangeReader`, `MultiReader`) now agree with each other and with the `Writer` about the bytes between them; every malformed container names its refusal with a code, and a standalone SPEC byte-checker (reads SPEC.md offsets only, imports nothing from `src/`) is the contract's executable form. All new validation runs at open/parse/construction time -- the `get`/`syncRange`/`findShards` row loops and both tokenizer byte loops are byte-identical. Suite: 432 tests / 431 pass / 0 fail / 1 todo. Torture: 37/37 fast, 39/39 full, arrayBuffers growth 0. Mutation check: all 52 door tests fail on source revert; all 23 behavior-unchanged pins hold.

package/README.md

@@ -13,7 +13,7 @@
 
 > Streaming byte-level JSON compiler for [`@zakkster/lite-bake`](https://github.com/PeshoVurtoleta/lite-bake). Zero-GC, tree-shakeable, gigabyte-scale.
 
-**Status:** v1.3.0. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.
+**Status:** v1.3.1. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.
 
 ## Two modes, one API
 

package/SPEC.md

@@ -274,7 +274,7 @@ The boundary is exercised explicitly by `test/NumericBoundary.test.js`, which pi
 
 ### 5.5 String semantics (v1)
 
-- Input MUST be valid UTF-8. Invalid sequences are a parse error.
+- String bytes between the quotes are stored byte-exact through U32 lanes, regardless of UTF-8 validity. Invalid sequences are **not** a parse error; the prior rule declaring them one (and its `E_INVALID_UTF8` code) is withdrawn. `get()` / `getString()` decode with replacement (U+FFFD); `shardStringTable(s).bytesAt(i)` returns the exact stored bytes, as `PreserveReader.getBytes(i)` does in preserve mode.
 - All standard JSON escapes are supported: `\"`, `\\`, `\/`, `\b`, `\f`, `\n`, `\r`, `\t`, `\uXXXX`.
 - Surrogate pairs (`\uD800`..`\uDBFF` followed by `\uDC00`..`\uDFFF`) are combined and emitted as UTF-8.
 - Unpaired surrogates are a parse error.
@@ -334,7 +334,7 @@ Given source NDJSON `S` and a schema `Σ` (either explicit or inferred), the LBK
 | :-- | :-- | :-- |
 | JSON number in fast-path domain (≤15 sig digits, `|exp| ≤ 22`) | F64 | **bit-exact IEEE 754 double** |
 | JSON number in slow-path domain | F64 | within 1 ULP of the correctly-rounded value |
-| JSON string (any valid UTF-8) | U32 | **byte-exact UTF-8 sequence** |
+| JSON string (any bytes, valid UTF-8 or not) | U32 | **byte-exact bytes; validity not asserted** |
 | JSON `true` / `false` / `null` | F64 | `1` / `0` / `0` (documented coercion) |
 | absent field (missing in source) | F64 | `0` (default) |
 | absent field | U32 | `""` (index 0 is the reserved empty-string entry) |

package/llms.txt

@@ -8,13 +8,13 @@ Ingest gigabyte-scale JSON (top-level array or NDJSON) into the `lite-bake` LBK1
 
 ## Status
 
-v1.3.0 — stable. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak (M1 MacBook Pro): 98.37M rows, 4.89 GB container, zero major GC, zero minor GC, 499 KB total heap allocation, 590.21M cells verified byte-exact, zero mismatches. Tokenizer benches at 222-237 MB/s (~55% of JSON.parse, with no object graph allocated).
+v1.3.1 — stable. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak (M1 MacBook Pro): 98.37M rows, 4.89 GB container, zero major GC, zero minor GC, 499 KB total heap allocation, 590.21M cells verified byte-exact, zero mismatches. Tokenizer benches at 222-237 MB/s (~55% of JSON.parse, with no object graph allocated).
 
 Public API follows semver from 1.0.0. Future additions (I64 lane, columnar payload mode, container-level string table) land via the format's forward-compat seams -- `min_reader_version` on ShardEntry, reserved FieldDescriptor flags, the `metadata_off` block wrapper -- without a format_version bump.
 
 See SPEC.md for the LBK1 container format, section 3.6 for zone maps, section 4.3 for the reserved field flags.
 
-## Public API (v1.3.0)
+## Public API (v1.3.1)
 
 Two ingest modes share one top-level API:
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zakkster/lite-bake-stream",
-  "version": "1.3.0",
+  "version": "1.3.1",
   "description": "Streaming byte-level JSON to lite-bake binary compiler. Zero-GC, tree-shakeable, gigabyte-scale.",
   "type": "module",
   "sideEffects": false,

package/src/FileIngest.js

@@ -27,7 +27,7 @@ import { PreserveWriter } from './PreserveWriter.js';
 import { PreserveReader } from './PreserveReader.js';
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.3.1';
 
 const U32_MAX = 4294967295;
 const INGEST_OPTS = {

package/src/MultiReader.js

@@ -25,7 +25,7 @@
 //   M_ROW_OUT_OF_RANGE    - rowIdx >= totalRows
 //   M_TOO_MANY_ROWS       - cumulative row count exceeds Number.MAX_SAFE_INTEGER
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.3.1';
 
 export class MultiReaderError extends Error {
   constructor(code, msg) { super(msg); this.code = code; this.name = 'MultiReaderError'; }

package/src/PreserveReader.js

@@ -32,7 +32,7 @@
 
 import { toContainerBuffer } from './Views.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.3.1';
 
 const CONTAINER_HEADER_BYTES = 48;
 const SHARD_ENTRY_BYTES = 40;

package/src/PreserveTokenizer.js

@@ -32,7 +32,7 @@
 
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.3.1';
 
 const U32_MAX = 4294967295;
 const PRESERVE_TOKENIZER_OPTS = {

package/src/PreserveWriter.js

@@ -18,7 +18,7 @@
 
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.3.1';
 
 export class PreserveWriterError extends Error {
   constructor(code, msg) { super(msg); this.code = code; this.name = 'PreserveWriterError'; }

package/src/RangeReader.js

@@ -42,7 +42,7 @@
 import { StringTable } from './StringTable.js';
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.3.1';
 
 const CONTAINER_HEADER_BYTES = 48;
 const SHARD_ENTRY_BYTES = 40;

package/src/Reader.js

@@ -25,7 +25,7 @@
 import { StringTable } from './StringTable.js';
 import { toContainerBuffer } from './Views.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.3.1';
 
 const CONTAINER_HEADER_BYTES = 48;
 const SHARD_ENTRY_BYTES = 40;

package/src/Split.js

@@ -37,7 +37,7 @@ import { Reader, ReaderError } from './Reader.js';
 import { StringTable } from './StringTable.js';
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.3.1';
 
 const LF = 0x0A;
 const CONTAINER_HEADER_BYTES = 48;

package/src/StringTable.js

@@ -20,7 +20,7 @@
 // and at every reset(), so an absent U32 row cell (which is 0) decodes as ""
 // rather than aliasing the shard's first-interned string (SPEC 3.3, SPEC 7).
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.3.1';
 
 const EMPTY_SLOT = 0xFFFFFFFF;   // MUST be unsigned; typed-array reads are unsigned
 const INITIAL_BLOB_BYTES  = 64 * 1024;

package/src/Tokenizer.js

@@ -10,7 +10,6 @@
 //   E_INVALID_ESCAPE       - unknown \ sequence
 //   E_INVALID_HEX          - non-hex digit in \uXXXX
 //   E_UNPAIRED_SURROGATE   - lone high/low surrogate in \u escape
-//   E_INVALID_UTF8         - malformed UTF-8 byte sequence
 //   E_NUMBER_OVERFLOW      - number magnitude exceeds F64 range
 //   E_NUMBER_INVALID       - malformed number literal
 //   E_KEYWORD_MISMATCH     - true/false/null spelled wrong
@@ -24,7 +23,7 @@
 
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.3.1';
 
 const U32_MAX = 4294967295;
 const TOKENIZER_OPTS = {

package/src/Writer.js

@@ -36,7 +36,7 @@
 import { StringTable } from './StringTable.js';
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.3.1';
 
 const U32_MAX = 4294967295;
 // Post-finalize sentinel for _recordDepth. Chosen = 2 so every post-finalize

package/src/index.js

@@ -28,7 +28,7 @@ export {
   PreserveWriter, PreserveWriterError,
   PreserveReader, PreserveReaderError,
 };
-export const VERSION = '1.3.0';
+export const VERSION = '1.3.1';
 
 const encoder = new TextEncoder();