@zakkster/lite-bake-stream 1.3.0 → 1.4.0

package/CHANGELOG.md

@@ -2,6 +2,32 @@
 
 All notable changes to this project will be documented in this file. This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.4.0] -- 2026-09-01
+
+M4 -- the sample window does what its name says, and null stops eating lanes. Two inference-window bugs fixed together (BS-07, BS-20). Suite: 461 tests / 461 pass / 0 fail / 0 todo. Torture: 44/44 fast, 47/47 full (incl. the 64 MB structural scenario), arrayBuffers growth 0. Mutation measurement: the 8-test manifest failed exactly and the 5 canaries passed on baseline src. Findings probes: 19 NOT-REPRODUCED / 0 REPRODUCED / 0 PARTIAL -- the ledger's first all-clear; the probe file is retired.
+
+### Changed
+
+- BS-07: the sample-and-infer window is now byte-true. The Tokenizer exposes `absOffset` (a getter over the cumulative consumed-input-byte counter that already survived chunk boundaries), and the Writer reads it at record boundaries via a new optional `setInputSource(tokenizer)` hook the Tokenizer calls at construction. The window ends at the first record boundary at or after `sampleBytes` INPUT bytes -- independent of chunking and record count. Previously `setInputByteOffset` had zero callers, so the window measured record count ("first `sampleBytes` records") and `serialize()` on a large input buffered the whole thing columnar until finalize (SPEC invariant 1 violated on the default path). `setInputByteOffset` is now a documented no-op retained until 2.0. A hand-driven Writer with no wired source samples by record count (documented fallback; `null` is not a 0-byte estimate). No sink-protocol change: the hook is a call probed with `typeof`, never a field write, so a frozen sink is untouched (BS-19). See decisions/0007-sample-window-bytes.md.
+- BS-20: JSON `null` is now lane-neutral. In the sample window `onNull` marks the field but sets no kind, so `null + string` infers U32 (was `W_MIXED_LANE_TYPES`) and `null + number` infers F64; an only-null field still infers F64. Post-freeze, `null` is legal on both lanes -- F64 -> `0` (unchanged), U32 -> `""` (index 0, the reserved empty-string entry; was `W_LANE_MISMATCH`). Booleans are UNCHANGED: `true`/`false` on a U32 lane still throw `W_LANE_MISMATCH` (pinned). A field that is null through the whole sample window then sees a string post-freeze still throws `W_LANE_MISMATCH` (the pinned sharp edge; no post-freeze schema evolution). The `test/_verify.js` oracle and the t5 scenarios adopt the same policy in the same change. See decisions/0008-null-policy.md.
+
+### Fixed
+
+- BS-07: `sampleBytes` now bounds INPUT bytes as documented; sample memory is O(`sampleBytes`) (columnar staging plus the shared string table), independent of total input size. Proven structurally at full tier (64 MB default-opts stream: shards commit mid-ingest; the sample buffer is nulled after the window drains) and by a t7 SampleBuffer retention witness with a t9 BREAK control.
+- BS-20: a single `null` in a string field no longer poisons inference or throws on an explicit U32 lane -- real-world API JSON that nulls string fields now compiles.
+
+## [1.3.1] -- 2026-09-01
+
+M3 remainder (BS-05) -- the UTF-8 door ends the promise-A/behavior-B split with a measured decision. Option A (validate at the tokenizer, refuse invalid UTF-8 with `E_INVALID_UTF8`) was built, proven correctness-green (460/460, torture fast 42/42), and rejected on its pre-committed budget: unicode-heavy throughput regressed 229.7 -> 213.1 MB/s (-7.2%, over the 3% ceiling), because the baseline scan bulk-copies `>= 0x80` bytes with zero inspection so validation is pure added cost, while ASCII stayed within noise. SPEC 5.5 is instead rewritten to match the long-standing behavior (documented-permissive), and the dead code is removed. No bytes move at runtime. See decisions/0006-utf8-door.md for the before/after numbers and the rejected option. Suite: 453 tests / 453 pass / 0 fail / 0 todo. Torture: 42/42 fast, 44/44 full, arrayBuffers growth 0. Inverted mutation check: all 49 battery and pin tests pass on baseline src as well -- zero behavior change, proven.
+
+### Changed
+
+- SPEC 5.5 and SPEC 7's U32 row rewritten to documented-permissive: string bytes between the quotes are stored byte-exact through U32 lanes regardless of UTF-8 validity; invalid sequences are not a parse error. `get()`/`getString()` decode with replacement (U+FFFD); `shardStringTable(s).bytesAt(i)` returns the exact stored bytes, the schema-mode twin of `PreserveReader.getBytes(i)`. This is a documentation-and-dead-code change only -- no runtime behavior changed, no new public surface (the accessor already existed).
+
+### Fixed
+
+- BS-05: the SPEC no longer promises a UTF-8 parse error the tokenizer never enforced, and `Tokenizer.js` no longer declares an unreachable `E_INVALID_UTF8` code. The byte-exact string contract is now stated truthfully: invalid bytes round-trip byte-exact and decode with U+FFFD, in schema mode exactly as in preserve mode. Lone-surrogate escapes still refuse with `E_UNPAIRED_SURROGATE` (an escape-grammar rule, unrelated to input-UTF-8 validity).
+
 ## [1.3.0] -- 2026-09-01
 
 M2 -- a container is verified at the door or refused at the door. The four readers (`Reader`, `PreserveReader`, `RangeReader`, `MultiReader`) now agree with each other and with the `Writer` about the bytes between them; every malformed container names its refusal with a code, and a standalone SPEC byte-checker (reads SPEC.md offsets only, imports nothing from `src/`) is the contract's executable form. All new validation runs at open/parse/construction time -- the `get`/`syncRange`/`findShards` row loops and both tokenizer byte loops are byte-identical. Suite: 432 tests / 431 pass / 0 fail / 1 todo. Torture: 37/37 fast, 39/39 full, arrayBuffers growth 0. Mutation check: all 52 door tests fail on source revert; all 23 behavior-unchanged pins hold.

package/README.md

@@ -13,7 +13,7 @@
 
 > Streaming byte-level JSON compiler for [`@zakkster/lite-bake`](https://github.com/PeshoVurtoleta/lite-bake). Zero-GC, tree-shakeable, gigabyte-scale.
 
-**Status:** v1.3.0. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.
+**Status:** v1.4.0. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.
 
 ## Two modes, one API
 

package/SPEC.md

@@ -274,7 +274,7 @@ The boundary is exercised explicitly by `test/NumericBoundary.test.js`, which pi
 
 ### 5.5 String semantics (v1)
 
-- Input MUST be valid UTF-8. Invalid sequences are a parse error.
+- String bytes between the quotes are stored byte-exact through U32 lanes, regardless of UTF-8 validity. Invalid sequences are **not** a parse error; the prior rule declaring them one (and its `E_INVALID_UTF8` code) is withdrawn. `get()` / `getString()` decode with replacement (U+FFFD); `shardStringTable(s).bytesAt(i)` returns the exact stored bytes, as `PreserveReader.getBytes(i)` does in preserve mode.
 - All standard JSON escapes are supported: `\"`, `\\`, `\/`, `\b`, `\f`, `\n`, `\r`, `\t`, `\uXXXX`.
 - Surrogate pairs (`\uD800`..`\uDBFF` followed by `\uDC00`..`\uDFFF`) are combined and emitted as UTF-8.
 - Unpaired surrogates are a parse error.
@@ -334,12 +334,14 @@ Given source NDJSON `S` and a schema `Σ` (either explicit or inferred), the LBK
 | :-- | :-- | :-- |
 | JSON number in fast-path domain (≤15 sig digits, `|exp| ≤ 22`) | F64 | **bit-exact IEEE 754 double** |
 | JSON number in slow-path domain | F64 | within 1 ULP of the correctly-rounded value |
-| JSON string (any valid UTF-8) | U32 | **byte-exact UTF-8 sequence** |
-| JSON `true` / `false` / `null` | F64 | `1` / `0` / `0` (documented coercion) |
+| JSON string (any bytes, valid UTF-8 or not) | U32 | **byte-exact bytes; validity not asserted** |
+| JSON `true` / `false` | F64 | `1` / `0` (documented coercion) |
+| JSON `null` | F64 | `0` (documented coercion) |
+| JSON `null` | U32 | `""` (index 0, the reserved empty-string entry; BS-20: null is lane-neutral) |
 | absent field (missing in source) | F64 | `0` (default) |
 | absent field | U32 | `""` (index 0 is the reserved empty-string entry) |
-| field in source but NOT in `Σ` | — | silently dropped (matches lite-bake core) |
-| field in `Σ` with wrong value type (post-freeze) | — | `W_LANE_MISMATCH` error, container not produced |
+| field in source but NOT in the schema | -- | silently dropped (matches lite-bake core) |
+| field in the schema with wrong value type (post-freeze), except null | -- | `W_LANE_MISMATCH` error, container not produced; null is lane-legal on both lanes (F64 -> 0, U32 -> "") |
 
 The preservation contract is asserted mechanically by three complementary layers:
 
@@ -351,6 +353,14 @@ The boundary between "bit-exact" and "1 ULP" for F64 is pinned by `test/NumericB
 
 Correctness of the preservation contract is a release gate on par with the zero-GC gate. If either fails, no publish.
 
+### 7.1 Sample window (byte-true)
+
+In sample-and-infer mode (no explicit schema) the Writer observes the first `sampleBytes` of INPUT before it freezes an inferred schema. The window is byte-true: it ends at the first record boundary AT OR AFTER `sampleBytes` input bytes have been consumed (BS-07). "Input bytes" is the Tokenizer's `absOffset` -- the cumulative count of bytes consumed across every `feed()` chunk, correct across chunk boundaries -- read by the Writer at record boundaries, never in the byte scan. The freeze point is therefore independent of how the input is chunked and of record count: 1000 tiny records or one 5 KB record both freeze at ~`sampleBytes` input bytes, not at a record count.
+
+The Writer receives the live Tokenizer through an OPTIONAL capability hook: at construction the Tokenizer calls `sink.setInputSource(tokenizer)` if the method exists. This is a call, never a field write onto the sink, so a frozen sink is untouched (BS-19) and a sink without the method is skipped. The required sink event set is unchanged. A hand-driven Writer with no wired source samples by record count (documented fallback; never a 0-byte estimate). `setInputByteOffset` is a deprecated no-op.
+
+Sample memory is O(`sampleBytes`): the columnar staging area plus the shared string table (bounded by string cardinality within the window), independent of total input size. The staging buffer drains to shards at freeze and is released.
+
 ## 8. Release gates
 
 - **M4 GA gate.** JSON conformance corpus (RFC 8259 + JSONTestSuite), fuzzed chunk-boundary splits at every byte position, F64 round-trip parity, container round-trip parity, zero-alloc-per-row proven by lite-gc-profiler after warmup.

package/llms.txt

@@ -8,13 +8,13 @@ Ingest gigabyte-scale JSON (top-level array or NDJSON) into the `lite-bake` LBK1
 
 ## Status
 
-v1.3.0 — stable. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak (M1 MacBook Pro): 98.37M rows, 4.89 GB container, zero major GC, zero minor GC, 499 KB total heap allocation, 590.21M cells verified byte-exact, zero mismatches. Tokenizer benches at 222-237 MB/s (~55% of JSON.parse, with no object graph allocated).
+v1.4.0 — stable. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak (M1 MacBook Pro): 98.37M rows, 4.89 GB container, zero major GC, zero minor GC, 499 KB total heap allocation, 590.21M cells verified byte-exact, zero mismatches. Tokenizer benches at 222-237 MB/s (~55% of JSON.parse, with no object graph allocated).
 
 Public API follows semver from 1.0.0. Future additions (I64 lane, columnar payload mode, container-level string table) land via the format's forward-compat seams -- `min_reader_version` on ShardEntry, reserved FieldDescriptor flags, the `metadata_off` block wrapper -- without a format_version bump.
 
 See SPEC.md for the LBK1 container format, section 3.6 for zone maps, section 4.3 for the reserved field flags.
 
-## Public API (v1.3.0)
+## Public API (v1.4.0)
 
 Two ingest modes share one top-level API:
 
@@ -49,7 +49,7 @@ Error classes with stable `code`: `TokenizerError`, `WriterError`, `ReaderError`
 
 - `{ fields: ['id', 'x', 'y'] }` — all-F64 shorthand, back-compat with M2.
 - `{ fields: [{name:'id', laneKind:'f64'}, {name:'tag', laneKind:'u32'}] }` — mixed lanes (M3).
-- Sample-and-infer (default when no schema is passed): first `sampleBytes` of input observed; fields with only numbers become F64, only strings become U32, mixed types raise `W_MIXED_LANE_TYPES` at freeze.
+- Sample-and-infer (default when no schema is passed): the first `sampleBytes` of INPUT are observed, byte-true (BS-07) -- the window ends at the first record boundary at or after `sampleBytes` input bytes, read from the Tokenizer's `absOffset`, independent of chunking and record count (a hand-driven Writer with no wired source samples by record count). Fields with only numbers become F64, only strings become U32. null is lane-neutral (BS-20): it sets no kind, so null+string infers U32 (no longer "mixed") and null+number infers F64; post-freeze null reads 0 on an F64 lane and "" on a U32 lane. A field that saw both a real number and a real string raises `W_MIXED_LANE_TYPES` at freeze. Sample memory is O(`sampleBytes`): columnar staging plus a shared string table, independent of total input size.
 
 ## Contract
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zakkster/lite-bake-stream",
-  "version": "1.3.0",
+  "version": "1.4.0",
   "description": "Streaming byte-level JSON to lite-bake binary compiler. Zero-GC, tree-shakeable, gigabyte-scale.",
   "type": "module",
   "sideEffects": false,

package/src/FileIngest.js

@@ -27,7 +27,7 @@ import { PreserveWriter } from './PreserveWriter.js';
 import { PreserveReader } from './PreserveReader.js';
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.4.0';
 
 const U32_MAX = 4294967295;
 const INGEST_OPTS = {

package/src/MultiReader.js

@@ -25,7 +25,7 @@
 //   M_ROW_OUT_OF_RANGE    - rowIdx >= totalRows
 //   M_TOO_MANY_ROWS       - cumulative row count exceeds Number.MAX_SAFE_INTEGER
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.4.0';
 
 export class MultiReaderError extends Error {
   constructor(code, msg) { super(msg); this.code = code; this.name = 'MultiReaderError'; }

package/src/PreserveReader.js

@@ -32,7 +32,7 @@
 
 import { toContainerBuffer } from './Views.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.4.0';
 
 const CONTAINER_HEADER_BYTES = 48;
 const SHARD_ENTRY_BYTES = 40;

package/src/PreserveTokenizer.js

@@ -32,7 +32,7 @@
 
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.4.0';
 
 const U32_MAX = 4294967295;
 const PRESERVE_TOKENIZER_OPTS = {

package/src/PreserveWriter.js

@@ -18,7 +18,7 @@
 
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.4.0';
 
 export class PreserveWriterError extends Error {
   constructor(code, msg) { super(msg); this.code = code; this.name = 'PreserveWriterError'; }

package/src/RangeReader.js

@@ -42,7 +42,7 @@
 import { StringTable } from './StringTable.js';
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.4.0';
 
 const CONTAINER_HEADER_BYTES = 48;
 const SHARD_ENTRY_BYTES = 40;

package/src/Reader.js

@@ -25,7 +25,7 @@
 import { StringTable } from './StringTable.js';
 import { toContainerBuffer } from './Views.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.4.0';
 
 const CONTAINER_HEADER_BYTES = 48;
 const SHARD_ENTRY_BYTES = 40;

package/src/Split.js

@@ -37,7 +37,7 @@ import { Reader, ReaderError } from './Reader.js';
 import { StringTable } from './StringTable.js';
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.4.0';
 
 const LF = 0x0A;
 const CONTAINER_HEADER_BYTES = 48;

package/src/StringTable.js

@@ -20,7 +20,7 @@
 // and at every reset(), so an absent U32 row cell (which is 0) decodes as ""
 // rather than aliasing the shard's first-interned string (SPEC 3.3, SPEC 7).
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.4.0';
 
 const EMPTY_SLOT = 0xFFFFFFFF;   // MUST be unsigned; typed-array reads are unsigned
 const INITIAL_BLOB_BYTES  = 64 * 1024;

package/src/Tokenizer.js

@@ -4,13 +4,19 @@
 //
 // Contract: see /SPEC.md section 5.
 //
+// Byte offset (BS-07): `absOffset` (getter) is the cumulative count of input
+// bytes consumed across every feed() chunk; it survives chunk boundaries. A
+// sink may expose setInputSource(tokenizer) to receive the live Tokenizer at
+// construction and read absOffset at record boundaries (the Writer uses this
+// for byte-true sampleBytes). Purely optional; a sink without the method is
+// never called and never mutated (BS-19).
+//
 // Error codes (stable):
 //   E_UNEXPECTED_BYTE      - byte does not belong in current state
 //   E_UNEXPECTED_EOF       - end() called mid-token
 //   E_INVALID_ESCAPE       - unknown \ sequence
 //   E_INVALID_HEX          - non-hex digit in \uXXXX
 //   E_UNPAIRED_SURROGATE   - lone high/low surrogate in \u escape
-//   E_INVALID_UTF8         - malformed UTF-8 byte sequence
 //   E_NUMBER_OVERFLOW      - number magnitude exceeds F64 range
 //   E_NUMBER_INVALID       - malformed number literal
 //   E_KEYWORD_MISMATCH     - true/false/null spelled wrong
@@ -24,7 +30,7 @@
 
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.4.0';
 
 const U32_MAX = 4294967295;
 const TOKENIZER_OPTS = {
@@ -142,6 +148,12 @@ export class Tokenizer {
     this._onNull        = sink.onNull        ? sink.onNull.bind(sink)        : NOOP;
     this._onEnd         = sink.onEnd         ? sink.onEnd.bind(sink)         : NOOP;
 
+    // Optional capability hook (BS-07): a sink may expose setInputSource to
+    // receive this Tokenizer and read `absOffset` at record boundaries. This is
+    // a CALL, never a field write onto the sink, so a frozen sink is untouched
+    // (BS-19); a sink without the method is skipped. Cold path (construction).
+    if (typeof sink.setInputSource === 'function') sink.setInputSource(this);
+
     // Terminal state (BS-14): once ended or poisoned, the instance is dead.
     this._ended = false;
     this._poisoned = false;
@@ -369,6 +381,11 @@ export class Tokenizer {
     this._err('E_UNEXPECTED_EOF', 'input ended mid-token (state=' + this._state + ')');
   }
 
+  // Cumulative input bytes consumed across every feed() chunk (survives chunk
+  // boundaries). Read by a Writer sink at record boundaries for byte-true
+  // sampleBytes (BS-07); never read on the byte scan.
+  get absOffset() { return this._absOffset; }
+
   // -------- top-level handling --------
 
   _handleTopByte(b) {

package/src/Writer.js

@@ -5,10 +5,15 @@
 // Two modes:
 //   - Explicit schema: consumer passes { fields: [names...] } or
 //     { fields: [{name, laneKind}...] }; no sampling.
-//   - Sample-and-infer (default): first `sampleBytes` of input buffered in a
-//     columnar staging area, per-field lane kind inferred from observed
-//     value types (number->F64, string->U32-into-string-table). Fields that
-//     saw both raise W_MIXED_LANE_TYPES at freeze.
+//   - Sample-and-infer (default): the first `sampleBytes` of INPUT (byte-true,
+//     BS-07: the window ends at the first record boundary at or after
+//     sampleBytes input bytes, read from the Tokenizer's absOffset via
+//     setInputSource; hand-driven Writers with no source sample by record
+//     count) buffered in a columnar staging area, per-field lane kind inferred
+//     from observed value types (number->F64, string->U32-into-string-table).
+//     null is lane-neutral (BS-20): it marks the field but sets no kind, so
+//     null+string infers U32 and null+number infers F64. A field that saw both
+//     a real number and a real string raises W_MIXED_LANE_TYPES at freeze.
 //
 // v1 lane kinds: F64 (numeric) and U32 (string-table index). Per-shard string
 // tables ride immediately after each shard's payload; the shard directory
@@ -23,8 +28,10 @@
 //   W_NESTED_UNSUPPORTED       - nested object/array inside a record
 //   W_EMPTY_INPUT              - finalize() with zero records seen
 //   W_ROW_STRIDE_ZERO          - schema has zero fields (nothing to store)
-//   W_MIXED_LANE_TYPES         - a field saw both number and string values
+//   W_MIXED_LANE_TYPES         - a field saw both real number and string values
+//                                (null excluded -- it is lane-neutral, BS-20)
 //   W_LANE_MISMATCH            - post-freeze: value type doesn't match schema lane
+//                                (null is lane-legal on both lanes: F64->0, U32->"")
 //   W_UNKNOWN_LANE_KIND        - explicit schema declares an unknown laneKind
 //   W_DUPLICATE_FIELD          - schema declares the same field name twice
 //   W_SCHEMA_TOO_WIDE          - a field offset exceeds the u16 offset_in_row ceiling
@@ -36,7 +43,7 @@
 import { StringTable } from './StringTable.js';
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.0';
+export const VERSION = '1.4.0';
 
 const U32_MAX = 4294967295;
 // Post-finalize sentinel for _recordDepth. Chosen = 2 so every post-finalize
@@ -145,6 +152,15 @@ class SampleBuffer {
     meta.columnU32[this._rowCount] = idx;
   }
 
+  // BS-20: null is lane-neutral. setNull ensures the field exists but leaves its
+  // kind untouched, so an only-null field stays K_UNKNOWN (-> F64, reads 0),
+  // null+string infers U32, and null+number infers F64. The row's column slot is
+  // left at its default 0, which reads back as 0 on an F64 lane and the reserved
+  // "" on a U32 lane. No per-field flag: K_UNKNOWN already carries the only-null case.
+  setNull(name) {
+    this._ensureField(name);
+  }
+
   _ensureField(name) {
     let meta = this._fieldMeta.get(name);
     if (!meta) {
@@ -218,6 +234,11 @@ export class Writer {
     this._currentFieldIdx = -1;
     this._recordStartByteOffset = 0;
     this._absOffset = 0;
+    // BS-07: the live Tokenizer, handed over via setInputSource at construction.
+    // When present, the sample window measures INPUT bytes (byte-true) by reading
+    // _source.absOffset at record boundaries. null => hand-driven Writer: fall
+    // back to record-count sampling (Math.max(1, ...) below). null is not zero.
+    this._source = null;
 
     this._finalized = false;
 
@@ -237,7 +258,9 @@ export class Writer {
       const f = this._rowValueSlotsF64, u = this._rowValueSlotsU32;
       for (let i = 0; i < f.length; i++) { f[i] = 0; u[i] = 0; }
     }
-    this._recordStartByteOffset = this._absOffset;
+    // Record-boundary read (BS-07): byte-true when a source is wired, else the
+    // dead hand-driven offset (0) that yields record-count sampling. Not the byte scan.
+    this._recordStartByteOffset = this._source !== null ? this._source.absOffset : this._absOffset;
   }
 
   onEndObject() {
@@ -286,7 +309,28 @@ export class Writer {
 
   onTrue()  { this.onNumber(1); }
   onFalse() { this.onNumber(0); }
-  onNull()  { this.onNumber(0); }
+
+  // BS-20: null is lane-neutral. Mirrors onString's guard structure (depth /
+  // finalized doors, unknown-field idx<0 return) but dispatches per lane instead
+  // of throwing: F64 slot -> 0, U32 slot -> 0 = the reserved "" (index 0, real in
+  // every shard since BS-01). onTrue/onFalse stay delegating to onNumber, so a
+  // bool on a U32 lane still throws W_LANE_MISMATCH. Non-null values never reach
+  // here -- zero added cost on the numeric/string hot path.
+  onNull() {
+    if (this._recordDepth !== 1) {
+      if (this._finalized) throw new WriterError('W_FINALIZED', 'writer already finalized');
+      throw new WriterError('W_TOP_LEVEL_NOT_OBJECT', 'top-level null is not an object; LBK1 schema mode stores objects only (SPEC 5.1)');
+    }
+    if (this._schema) {
+      const idx = this._currentFieldIdx;
+      if (idx < 0) return;
+      if (this._fieldLaneKinds[idx] === LANE_F64) this._rowValueSlotsF64[idx] = 0;
+      else this._rowValueSlotsU32[idx] = 0;
+    } else {
+      if (!this._currentKeyName) return;
+      this._sample.setNull(this._currentKeyName);
+    }
+  }
 
   onString(bytes, from, to) {
     if (this._recordDepth !== 1) {
@@ -469,7 +513,8 @@ export class Writer {
       this._totalRows++;
       if (this._currentShardRowCount >= this._currentShardMaxRows) this._finalizeCurrentShard();
     } else {
-      const consumed = Math.max(1, this._absOffset - this._recordStartByteOffset);
+      const endOff = this._source !== null ? this._source.absOffset : this._absOffset;
+      const consumed = Math.max(1, endOff - this._recordStartByteOffset);
       this._sample.commitRow(consumed);
       if (this._sample.byteEstimate >= this._sampleBytes) {
         this._freezeSchemaFromSample();
@@ -611,7 +656,16 @@ export class Writer {
     return this._container;
   }
 
-  setInputByteOffset(n) { this._absOffset = n; }
+  // Optional capability hook (BS-07): the Tokenizer calls this at construction
+  // to hand over the live tokenizer. The Writer then reads src.absOffset at
+  // record boundaries so the sample window is byte-true. A CALL only; the
+  // Tokenizer is never mutated.
+  setInputSource(src) { this._source = src; }
+
+  // Deprecated (BS-07): superseded by setInputSource. The Writer now reads the
+  // live Tokenizer's absOffset at record boundaries, so this hand-driven offset
+  // dance is a documented no-op retained for source compatibility until 2.0.
+  setInputByteOffset() {}
 
   // -------- container assembly --------
 

package/src/index.js

@@ -28,7 +28,7 @@ export {
   PreserveWriter, PreserveWriterError,
   PreserveReader, PreserveReaderError,
 };
-export const VERSION = '1.3.0';
+export const VERSION = '1.4.0';
 
 const encoder = new TextEncoder();
 

package/types/Tokenizer.d.ts

@@ -33,6 +33,12 @@ export class Tokenizer {
   feed(chunk: Uint8Array): void;
   /** Signal end of input. Any pending record is flushed to the sink. */
   end(): void;
+  /**
+   * Cumulative count of input bytes consumed across every `feed()` chunk;
+   * survives chunk boundaries. A sink reads this at record boundaries for
+   * byte-true `sampleBytes` (BS-07); it is never read on the byte scan.
+   */
+  readonly absOffset: number;
 }
 
 export class TokenizerError extends Error {

package/types/Writer.d.ts

@@ -22,8 +22,12 @@ export interface WriterOptions {
   /** Target output bytes per shard. Default 32 MB. */
   targetShardBytes?: number;
   /**
-   * Bytes of input to observe before freezing an inferred schema. Only used
-   * in sample-and-infer mode (no explicit schema). Default = targetShardBytes.
+   * Bytes of INPUT to observe before freezing an inferred schema. Only used in
+   * sample-and-infer mode (no explicit schema). Byte-true (BS-07): the window
+   * ends at the first record boundary at or after `sampleBytes` input bytes,
+   * read from the Tokenizer's `absOffset` -- independent of chunking and record
+   * count. A hand-driven Writer with no wired source samples by record count.
+   * Default = targetShardBytes.
    */
   sampleBytes?: number;
 }
@@ -63,7 +67,18 @@ export class Writer implements TokenizerSinkForWriter {
   onFalse(): void;
   onNull(): void;
   onEnd(): void;
-  /** Optional: set the current input byte offset for byte-estimate tracking. */
+  /**
+   * Optional capability hook (BS-07): the Tokenizer calls this at construction
+   * to hand over the live tokenizer. The Writer then reads `src.absOffset` at
+   * record boundaries so the sample window is byte-true. A call only; the
+   * Tokenizer is never mutated.
+   */
+  setInputSource(src: { readonly absOffset: number }): void;
+  /**
+   * @deprecated Superseded by `setInputSource` (BS-07). The Writer now reads the
+   * live Tokenizer's `absOffset` at record boundaries, so this is a no-op
+   * retained for source compatibility until 2.0.
+   */
   setInputByteOffset(n: number): void;
   /**
    * Seal the container and return the LBK1 binary. Call once, after the