@zakkster/lite-bake-stream 1.3.1 → 1.4.1

package/CHANGELOG.md

@@ -2,6 +2,43 @@
 
 All notable changes to this project will be documented in this file. This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.4.1] -- 2026-09-02
+
+M8 -- the four reserved torture cells become real; BS-30 and BS-31 close. No tier prints "reserved" any more. Suite: 461 tests / 461 pass / 0 fail / 0 todo (unchanged -- no named-suite edits). Torture: 44/44 fast (~1.3 s wall), 47/47 full, arrayBuffers growth 0. BREAK matrix: ten runs (`=1`, eight registry ids, one unknown id) all exit non-zero; the unknown id is refused at import. Inventory gate: 54 thrown codes / 0 unpinned. Falsifiability: with baseline src stashed, exactly one new assertion fails (the onProgress-identity pin), both tiers. Test-only except one hot-path reuse in `src/FileIngest.js` (below).
+
+### Changed
+
+- BS-31 (hot path): `ingestStream`'s per-chunk `onProgress` state object is now allocated ONCE and mutated in place, removing a per-chunk object literal from the ingest read loop. The state object is a reused ephemeral view -- valid only during the callback; copy it if retained. Contract documented in `src/FileIngest.js`, `types/FileIngest.d.ts`, and `llms.txt`. Falsifiability: with baseline src stashed, exactly the new onProgress-identity pin fails (baseline allocates a fresh object per chunk); every other new cell passes on both trees.
+- Torture cells t0/t1/t2/t4 filled in. t0: the re-chunk determinism law (byte-identical container across whole/string/1/7/64/4096-byte chunkings x both modes on a shared degenerate corpus) plus mergeContainers/compileInParts/MultiReader query-equivalence, findShards superset+point-exact, and splitNDJSON reassembly. t1: degenerate-value and refusal pins over the shared corpus (`test/torture/fixtures/degenerate.mjs`, reused by t0/t2 and MQ2). t2: every-byte-position split-invariance fuzz over both tokenizers. t4: API-abuse decided-policy pins plus the R5 adversarial doors.
+- BREAK selector: `BAKE_TORTURE_BREAK` now accepts a registry id (t6, g, i, j, k, l, m, n) to arm exactly one break site -- each control proves its own exit-non-zero path -- alongside legacy `=1` (arm all, halt at first). An unknown value fails closed at import. Registry documented in `test/torture/harness.mjs`.
+- Demo (repo-only, never shipped in the tarball): refreshed from its pre-1.0 state to the current release. The synthetic fixture now exercises the v1.4 inference-window semantics (~8% null string cells read back `""` with the lane still U32; ~8% absent numeric cells read `0`); schema-mode ingest uses `framing: 'auto'` so a dropped top-level `[...]` array works; error handling matches on stable codes (`W_NESTED_UNSUPPORTED`) instead of message text; the onProgress reuse contract is stated where the demo destructures the state object; both demo files are ASCII-clean. Every library call the demo makes was verified headless against `src/` (24 checks green, incl. both handled error paths).
+
+### Added
+
+- BS-30: the code-inventory gate (`test/torture/inventory.mjs`, run from t8). It scans `src/*.js` for every thrown stable code (constructed-error and `_err`/`raise*` families), asserts each is pinned by at least one test, and prints its collected-code count (54). The six previously-open doors are pinned in t3/t4 (`R_UNSUPPORTED_VERSION`, `R_UNSUPPORTED_ENDIAN`, `R_ADAPTER_SHORT_READ`, `W_ROW_STRIDE_ZERO` x2 sites, `E_RECORD_TOO_LARGE`, `E_UNSUPPORTED_FRAMING`); t1 pins the two remaining unpinned degenerate codes (`E_INVALID_HEX`, `E_TRAILING_INPUT`). A t9 control feeds the scanner a synthetic unpinned code to prove the gate can flag one.
+- BS-31: a t7 mid-stream-failure witness -- a ReadableStream whose pull throws makes `ingestStream` reject with the source error (no half-built Reader), and the lite-leak tracker returns to size 0. t6 gates the ingest sync core (feed + reused-progress mutation + `onProgress`) at zero allocation; the async await-read loop is covered by the t7 witness (runOpsGate rejects async fns).
+- t9 controls (i)-(n): one in-run rehearsal per newly-real cell plus the inventory gate, so a plain `npm run torture` proves every new gate can fail.
+
+### Findings (ledgered, no ride-along fix per session non-goals)
+
+- `Reader.get(1.5)` returns garbage (a fractional-byte-offset read) instead of a decided policy (throw / no-op / undefined) -- t4 records it, does not pin it as correct; the t9 (l) control proves the gate would catch a garbage return.
+- `RangeReader.syncRange` without a prior `prefetchRange` throws the shared `R_TRUNCATED`; the ROADMAP calls for a distinct code. t4 records it, does not pin the shared code.
+- `RangeReader` never length-validates an adapter's `fetch()` return: an adapter that returns fewer bytes than requested surfaces a raw downstream `RangeError`, not `R_ADAPTER_SHORT_READ` (whose two real sites are the Mock adapter's own past-size check and `HTTPRangeAdapter`'s 200-not-206 check). Ledgered as a reader-guard gap.
+
+## [1.4.0] -- 2026-09-01
+
+M4 -- the sample window does what its name says, and null stops eating lanes. Two inference-window bugs fixed together (BS-07, BS-20). Suite: 461 tests / 461 pass / 0 fail / 0 todo. Torture: 44/44 fast, 47/47 full (incl. the 64 MB structural scenario), arrayBuffers growth 0. Mutation measurement: the 8-test manifest failed exactly and the 5 canaries passed on baseline src. Findings probes: 19 NOT-REPRODUCED / 0 REPRODUCED / 0 PARTIAL -- the ledger's first all-clear; the probe file is retired.
+
+### Changed
+
+- BS-07: the sample-and-infer window is now byte-true. The Tokenizer exposes `absOffset` (a getter over the cumulative consumed-input-byte counter that already survived chunk boundaries), and the Writer reads it at record boundaries via a new optional `setInputSource(tokenizer)` hook the Tokenizer calls at construction. The window ends at the first record boundary at or after `sampleBytes` INPUT bytes -- independent of chunking and record count. Previously `setInputByteOffset` had zero callers, so the window measured record count ("first `sampleBytes` records") and `serialize()` on a large input buffered the whole thing columnar until finalize (SPEC invariant 1 violated on the default path). `setInputByteOffset` is now a documented no-op retained until 2.0. A hand-driven Writer with no wired source samples by record count (documented fallback; `null` is not a 0-byte estimate). No sink-protocol change: the hook is a call probed with `typeof`, never a field write, so a frozen sink is untouched (BS-19). See decisions/0007-sample-window-bytes.md.
+- BS-20: JSON `null` is now lane-neutral. In the sample window `onNull` marks the field but sets no kind, so `null + string` infers U32 (was `W_MIXED_LANE_TYPES`) and `null + number` infers F64; an only-null field still infers F64. Post-freeze, `null` is legal on both lanes -- F64 -> `0` (unchanged), U32 -> `""` (index 0, the reserved empty-string entry; was `W_LANE_MISMATCH`). Booleans are UNCHANGED: `true`/`false` on a U32 lane still throw `W_LANE_MISMATCH` (pinned). A field that is null through the whole sample window then sees a string post-freeze still throws `W_LANE_MISMATCH` (the pinned sharp edge; no post-freeze schema evolution). The `test/_verify.js` oracle and the t5 scenarios adopt the same policy in the same change. See decisions/0008-null-policy.md.
+
+### Fixed
+
+- BS-07: `sampleBytes` now bounds INPUT bytes as documented; sample memory is O(`sampleBytes`) (columnar staging plus the shared string table), independent of total input size. Proven structurally at full tier (64 MB default-opts stream: shards commit mid-ingest; the sample buffer is nulled after the window drains) and by a t7 SampleBuffer retention witness with a t9 BREAK control.
+- BS-20: a single `null` in a string field no longer poisons inference or throws on an explicit U32 lane -- real-world API JSON that nulls string fields now compiles.
+
 ## [1.3.1] -- 2026-09-01
 
 M3 remainder (BS-05) -- the UTF-8 door ends the promise-A/behavior-B split with a measured decision. Option A (validate at the tokenizer, refuse invalid UTF-8 with `E_INVALID_UTF8`) was built, proven correctness-green (460/460, torture fast 42/42), and rejected on its pre-committed budget: unicode-heavy throughput regressed 229.7 -> 213.1 MB/s (-7.2%, over the 3% ceiling), because the baseline scan bulk-copies `>= 0x80` bytes with zero inspection so validation is pure added cost, while ASCII stayed within noise. SPEC 5.5 is instead rewritten to match the long-standing behavior (documented-permissive), and the dead code is removed. No bytes move at runtime. See decisions/0006-utf8-door.md for the before/after numbers and the rejected option. Suite: 453 tests / 453 pass / 0 fail / 0 todo. Torture: 42/42 fast, 44/44 full, arrayBuffers growth 0. Inverted mutation check: all 49 battery and pin tests pass on baseline src as well -- zero behavior change, proven.

package/README.md

@@ -13,7 +13,7 @@
 
 > Streaming byte-level JSON compiler for [`@zakkster/lite-bake`](https://github.com/PeshoVurtoleta/lite-bake). Zero-GC, tree-shakeable, gigabyte-scale.
 
-**Status:** v1.3.1. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.
+**Status:** v1.4.1. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.
 
 ## Two modes, one API
 

package/SPEC.md

@@ -335,11 +335,13 @@ Given source NDJSON `S` and a schema `Σ` (either explicit or inferred), the LBK
 | JSON number in fast-path domain (≤15 sig digits, `|exp| ≤ 22`) | F64 | **bit-exact IEEE 754 double** |
 | JSON number in slow-path domain | F64 | within 1 ULP of the correctly-rounded value |
 | JSON string (any bytes, valid UTF-8 or not) | U32 | **byte-exact bytes; validity not asserted** |
-| JSON `true` / `false` / `null` | F64 | `1` / `0` / `0` (documented coercion) |
+| JSON `true` / `false` | F64 | `1` / `0` (documented coercion) |
+| JSON `null` | F64 | `0` (documented coercion) |
+| JSON `null` | U32 | `""` (index 0, the reserved empty-string entry; BS-20: null is lane-neutral) |
 | absent field (missing in source) | F64 | `0` (default) |
 | absent field | U32 | `""` (index 0 is the reserved empty-string entry) |
-| field in source but NOT in `Σ` | — | silently dropped (matches lite-bake core) |
-| field in `Σ` with wrong value type (post-freeze) | — | `W_LANE_MISMATCH` error, container not produced |
+| field in source but NOT in the schema | -- | silently dropped (matches lite-bake core) |
+| field in the schema with wrong value type (post-freeze), except null | -- | `W_LANE_MISMATCH` error, container not produced; null is lane-legal on both lanes (F64 -> 0, U32 -> "") |
 
 The preservation contract is asserted mechanically by three complementary layers:
 
@@ -351,6 +353,14 @@ The boundary between "bit-exact" and "1 ULP" for F64 is pinned by `test/NumericB
 
 Correctness of the preservation contract is a release gate on par with the zero-GC gate. If either fails, no publish.
 
+### 7.1 Sample window (byte-true)
+
+In sample-and-infer mode (no explicit schema) the Writer observes the first `sampleBytes` of INPUT before it freezes an inferred schema. The window is byte-true: it ends at the first record boundary AT OR AFTER `sampleBytes` input bytes have been consumed (BS-07). "Input bytes" is the Tokenizer's `absOffset` -- the cumulative count of bytes consumed across every `feed()` chunk, correct across chunk boundaries -- read by the Writer at record boundaries, never in the byte scan. The freeze point is therefore independent of how the input is chunked and of record count: 1000 tiny records or one 5 KB record both freeze at ~`sampleBytes` input bytes, not at a record count.
+
+The Writer receives the live Tokenizer through an OPTIONAL capability hook: at construction the Tokenizer calls `sink.setInputSource(tokenizer)` if the method exists. This is a call, never a field write onto the sink, so a frozen sink is untouched (BS-19) and a sink without the method is skipped. The required sink event set is unchanged. A hand-driven Writer with no wired source samples by record count (documented fallback; never a 0-byte estimate). `setInputByteOffset` is a deprecated no-op.
+
+Sample memory is O(`sampleBytes`): the columnar staging area plus the shared string table (bounded by string cardinality within the window), independent of total input size. The staging buffer drains to shards at freeze and is released.
+
 ## 8. Release gates
 
 - **M4 GA gate.** JSON conformance corpus (RFC 8259 + JSONTestSuite), fuzzed chunk-boundary splits at every byte position, F64 round-trip parity, container round-trip parity, zero-alloc-per-row proven by lite-gc-profiler after warmup.

package/llms.txt

@@ -8,13 +8,13 @@ Ingest gigabyte-scale JSON (top-level array or NDJSON) into the `lite-bake` LBK1
 
 ## Status
 
-v1.3.1 — stable. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak (M1 MacBook Pro): 98.37M rows, 4.89 GB container, zero major GC, zero minor GC, 499 KB total heap allocation, 590.21M cells verified byte-exact, zero mismatches. Tokenizer benches at 222-237 MB/s (~55% of JSON.parse, with no object graph allocated).
+v1.4.1 — stable. LBK1 format frozen at `format_version: 1`. Qualified on an 8 GB soak (M1 MacBook Pro): 98.37M rows, 4.89 GB container, zero major GC, zero minor GC, 499 KB total heap allocation, 590.21M cells verified byte-exact, zero mismatches. Tokenizer benches at 222-237 MB/s (~55% of JSON.parse, with no object graph allocated).
 
 Public API follows semver from 1.0.0. Future additions (I64 lane, columnar payload mode, container-level string table) land via the format's forward-compat seams -- `min_reader_version` on ShardEntry, reserved FieldDescriptor flags, the `metadata_off` block wrapper -- without a format_version bump.
 
 See SPEC.md for the LBK1 container format, section 3.6 for zone maps, section 4.3 for the reserved field flags.
 
-## Public API (v1.3.1)
+## Public API (v1.4.1)
 
 Two ingest modes share one top-level API:
 
@@ -41,7 +41,7 @@ Shared:
 - `ingestStream`, `ingestFile` (`/file-ingest`): browser helpers piping a `ReadableStream<Uint8Array>` through the pipeline. `preserve` option dispatches to the right writer.
 
 Error classes with stable `code`: `TokenizerError`, `WriterError`, `ReaderError`, `RangeReaderError`, `MultiReaderError`, `SplitError`, `PreserveTokenizerError`, `PreserveWriterError`, `PreserveReaderError`.
-- `ingestStream`, `ingestFile` (`/file-ingest`): browser helpers piping a `ReadableStream<Uint8Array>` (e.g. `File.stream()`) through the Tokenizer + Writer, returning a Reader. Per-chunk `onProgress` callback.
+- `ingestStream`, `ingestFile` (`/file-ingest`): browser helpers piping a `ReadableStream<Uint8Array>` (e.g. `File.stream()`) through the Tokenizer + Writer, returning a Reader. Per-chunk `onProgress` callback; its state object is reused across calls (mutated in place, zero per-chunk allocation) -- copy it if retained past the callback.
 - `TokenizerError`, `WriterError`, `ReaderError`, `RangeReaderError`: thrown on parse/write/read errors with stable `code`.
 - `VERSION` const per subpath.
 
@@ -49,7 +49,7 @@ Error classes with stable `code`: `TokenizerError`, `WriterError`, `ReaderError`
 
 - `{ fields: ['id', 'x', 'y'] }` — all-F64 shorthand, back-compat with M2.
 - `{ fields: [{name:'id', laneKind:'f64'}, {name:'tag', laneKind:'u32'}] }` — mixed lanes (M3).
-- Sample-and-infer (default when no schema is passed): first `sampleBytes` of input observed; fields with only numbers become F64, only strings become U32, mixed types raise `W_MIXED_LANE_TYPES` at freeze.
+- Sample-and-infer (default when no schema is passed): the first `sampleBytes` of INPUT are observed, byte-true (BS-07) -- the window ends at the first record boundary at or after `sampleBytes` input bytes, read from the Tokenizer's `absOffset`, independent of chunking and record count (a hand-driven Writer with no wired source samples by record count). Fields with only numbers become F64, only strings become U32. null is lane-neutral (BS-20): it sets no kind, so null+string infers U32 (no longer "mixed") and null+number infers F64; post-freeze null reads 0 on an F64 lane and "" on a U32 lane. A field that saw both a real number and a real string raises `W_MIXED_LANE_TYPES` at freeze. Sample memory is O(`sampleBytes`): columnar staging plus a shared string table, independent of total input size.
 
 ## Contract
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zakkster/lite-bake-stream",
-  "version": "1.3.1",
+  "version": "1.4.1",
   "description": "Streaming byte-level JSON to lite-bake binary compiler. Zero-GC, tree-shakeable, gigabyte-scale.",
   "type": "module",
   "sideEffects": false,

package/src/FileIngest.js

@@ -27,7 +27,7 @@ import { PreserveWriter } from './PreserveWriter.js';
 import { PreserveReader } from './PreserveReader.js';
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.1';
+export const VERSION = '1.4.1';
 
 const U32_MAX = 4294967295;
 const INGEST_OPTS = {
@@ -49,7 +49,12 @@ function raiseIngest(code, msg) { throw new WriterError(code, msg); }
 //                       sampleBytes }. Preserve: { targetShardBytes, maxRecordBytes }.
 //   onProgress(state):  called after each chunk with { bytesIngested,
 //                       totalBytes, rowsWritten, shardsCommitted, chunkBytes,
-//                       chunkCount, elapsedMs }
+//                       chunkCount, elapsedMs }. The state object is REUSED
+//                       across calls -- one object is mutated in place per chunk
+//                       (zero per-chunk allocation), so it is valid only during
+//                       the callback; copy it if you retain it past the call.
+//                       Same ephemeral-view contract as the tokenizer's byte
+//                       ranges.
 //   totalBytes:         optional total for progress percentage. Pass file.size
 //                       when ingesting a File; omit for indeterminate streams.
 export async function ingestStream(readableStream, opts) {
@@ -73,10 +78,20 @@ export async function ingestStream(readableStream, opts) {
     : new Tokenizer(writer, { framing });
 
   const reader = readableStream.getReader();
-  const t0 = (globalThis.performance && performance.now) ? performance.now() : Date.now();
+  const nowFn = (globalThis.performance && performance.now)
+    ? () => performance.now() : () => Date.now();
+  const t0 = nowFn();
   let bytesIngested = 0;
   let chunkCount = 0;
 
+  // One progress object, hoisted and mutated per chunk -- the ingest loop
+  // allocates nothing per chunk. Documented as a reused ephemeral view above.
+  const progress = onProgress ? {
+    bytesIngested: 0, totalBytes,
+    rowsWritten: 0, shardsCommitted: 0,
+    chunkBytes: 0, chunkCount: 0, elapsedMs: 0,
+  } : null;
+
   try {
     while (true) {
       const { value, done } = await reader.read();
@@ -87,16 +102,14 @@ export async function ingestStream(readableStream, opts) {
       tokenizer.feed(value);
       bytesIngested += value.length;
       chunkCount++;
-      if (onProgress) {
-        const now = (globalThis.performance && performance.now) ? performance.now() : Date.now();
-        onProgress({
-          bytesIngested, totalBytes,
-          rowsWritten: writer.totalRows,
-          shardsCommitted: writer.shardCount,
-          chunkBytes: value.length,
-          chunkCount,
-          elapsedMs: now - t0,
-        });
+      if (progress) {
+        progress.bytesIngested = bytesIngested;
+        progress.rowsWritten = writer.totalRows;
+        progress.shardsCommitted = writer.shardCount;
+        progress.chunkBytes = value.length;
+        progress.chunkCount = chunkCount;
+        progress.elapsedMs = nowFn() - t0;
+        onProgress(progress);
       }
     }
   } finally {

package/src/MultiReader.js

@@ -25,7 +25,7 @@
 //   M_ROW_OUT_OF_RANGE    - rowIdx >= totalRows
 //   M_TOO_MANY_ROWS       - cumulative row count exceeds Number.MAX_SAFE_INTEGER
 
-export const VERSION = '1.3.1';
+export const VERSION = '1.4.1';
 
 export class MultiReaderError extends Error {
   constructor(code, msg) { super(msg); this.code = code; this.name = 'MultiReaderError'; }

package/src/PreserveReader.js

@@ -32,7 +32,7 @@
 
 import { toContainerBuffer } from './Views.js';
 
-export const VERSION = '1.3.1';
+export const VERSION = '1.4.1';
 
 const CONTAINER_HEADER_BYTES = 48;
 const SHARD_ENTRY_BYTES = 40;

package/src/PreserveTokenizer.js

@@ -32,7 +32,7 @@
 
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.1';
+export const VERSION = '1.4.1';
 
 const U32_MAX = 4294967295;
 const PRESERVE_TOKENIZER_OPTS = {

package/src/PreserveWriter.js

@@ -18,7 +18,7 @@
 
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.1';
+export const VERSION = '1.4.1';
 
 export class PreserveWriterError extends Error {
   constructor(code, msg) { super(msg); this.code = code; this.name = 'PreserveWriterError'; }

package/src/RangeReader.js

@@ -42,7 +42,7 @@
 import { StringTable } from './StringTable.js';
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.1';
+export const VERSION = '1.4.1';
 
 const CONTAINER_HEADER_BYTES = 48;
 const SHARD_ENTRY_BYTES = 40;

package/src/Reader.js

@@ -25,7 +25,7 @@
 import { StringTable } from './StringTable.js';
 import { toContainerBuffer } from './Views.js';
 
-export const VERSION = '1.3.1';
+export const VERSION = '1.4.1';
 
 const CONTAINER_HEADER_BYTES = 48;
 const SHARD_ENTRY_BYTES = 40;

package/src/Split.js

@@ -37,7 +37,7 @@ import { Reader, ReaderError } from './Reader.js';
 import { StringTable } from './StringTable.js';
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.1';
+export const VERSION = '1.4.1';
 
 const LF = 0x0A;
 const CONTAINER_HEADER_BYTES = 48;

package/src/StringTable.js

@@ -20,7 +20,7 @@
 // and at every reset(), so an absent U32 row cell (which is 0) decodes as ""
 // rather than aliasing the shard's first-interned string (SPEC 3.3, SPEC 7).
 
-export const VERSION = '1.3.1';
+export const VERSION = '1.4.1';
 
 const EMPTY_SLOT = 0xFFFFFFFF;   // MUST be unsigned; typed-array reads are unsigned
 const INITIAL_BLOB_BYTES  = 64 * 1024;

package/src/Tokenizer.js

@@ -4,6 +4,13 @@
 //
 // Contract: see /SPEC.md section 5.
 //
+// Byte offset (BS-07): `absOffset` (getter) is the cumulative count of input
+// bytes consumed across every feed() chunk; it survives chunk boundaries. A
+// sink may expose setInputSource(tokenizer) to receive the live Tokenizer at
+// construction and read absOffset at record boundaries (the Writer uses this
+// for byte-true sampleBytes). Purely optional; a sink without the method is
+// never called and never mutated (BS-19).
+//
 // Error codes (stable):
 //   E_UNEXPECTED_BYTE      - byte does not belong in current state
 //   E_UNEXPECTED_EOF       - end() called mid-token
@@ -23,7 +30,7 @@
 
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.1';
+export const VERSION = '1.4.1';
 
 const U32_MAX = 4294967295;
 const TOKENIZER_OPTS = {
@@ -141,6 +148,12 @@ export class Tokenizer {
     this._onNull        = sink.onNull        ? sink.onNull.bind(sink)        : NOOP;
     this._onEnd         = sink.onEnd         ? sink.onEnd.bind(sink)         : NOOP;
 
+    // Optional capability hook (BS-07): a sink may expose setInputSource to
+    // receive this Tokenizer and read `absOffset` at record boundaries. This is
+    // a CALL, never a field write onto the sink, so a frozen sink is untouched
+    // (BS-19); a sink without the method is skipped. Cold path (construction).
+    if (typeof sink.setInputSource === 'function') sink.setInputSource(this);
+
     // Terminal state (BS-14): once ended or poisoned, the instance is dead.
     this._ended = false;
     this._poisoned = false;
@@ -368,6 +381,11 @@ export class Tokenizer {
     this._err('E_UNEXPECTED_EOF', 'input ended mid-token (state=' + this._state + ')');
   }
 
+  // Cumulative input bytes consumed across every feed() chunk (survives chunk
+  // boundaries). Read by a Writer sink at record boundaries for byte-true
+  // sampleBytes (BS-07); never read on the byte scan.
+  get absOffset() { return this._absOffset; }
+
   // -------- top-level handling --------
 
   _handleTopByte(b) {

package/src/Writer.js

@@ -5,10 +5,15 @@
 // Two modes:
 //   - Explicit schema: consumer passes { fields: [names...] } or
 //     { fields: [{name, laneKind}...] }; no sampling.
-//   - Sample-and-infer (default): first `sampleBytes` of input buffered in a
-//     columnar staging area, per-field lane kind inferred from observed
-//     value types (number->F64, string->U32-into-string-table). Fields that
-//     saw both raise W_MIXED_LANE_TYPES at freeze.
+//   - Sample-and-infer (default): the first `sampleBytes` of INPUT (byte-true,
+//     BS-07: the window ends at the first record boundary at or after
+//     sampleBytes input bytes, read from the Tokenizer's absOffset via
+//     setInputSource; hand-driven Writers with no source sample by record
+//     count) buffered in a columnar staging area, per-field lane kind inferred
+//     from observed value types (number->F64, string->U32-into-string-table).
+//     null is lane-neutral (BS-20): it marks the field but sets no kind, so
+//     null+string infers U32 and null+number infers F64. A field that saw both
+//     a real number and a real string raises W_MIXED_LANE_TYPES at freeze.
 //
 // v1 lane kinds: F64 (numeric) and U32 (string-table index). Per-shard string
 // tables ride immediately after each shard's payload; the shard directory
@@ -23,8 +28,10 @@
 //   W_NESTED_UNSUPPORTED       - nested object/array inside a record
 //   W_EMPTY_INPUT              - finalize() with zero records seen
 //   W_ROW_STRIDE_ZERO          - schema has zero fields (nothing to store)
-//   W_MIXED_LANE_TYPES         - a field saw both number and string values
+//   W_MIXED_LANE_TYPES         - a field saw both real number and string values
+//                                (null excluded -- it is lane-neutral, BS-20)
 //   W_LANE_MISMATCH            - post-freeze: value type doesn't match schema lane
+//                                (null is lane-legal on both lanes: F64->0, U32->"")
 //   W_UNKNOWN_LANE_KIND        - explicit schema declares an unknown laneKind
 //   W_DUPLICATE_FIELD          - schema declares the same field name twice
 //   W_SCHEMA_TOO_WIDE          - a field offset exceeds the u16 offset_in_row ceiling
@@ -36,7 +43,7 @@
 import { StringTable } from './StringTable.js';
 import { checkOpts } from './Opts.js';
 
-export const VERSION = '1.3.1';
+export const VERSION = '1.4.1';
 
 const U32_MAX = 4294967295;
 // Post-finalize sentinel for _recordDepth. Chosen = 2 so every post-finalize
@@ -145,6 +152,15 @@ class SampleBuffer {
     meta.columnU32[this._rowCount] = idx;
   }
 
+  // BS-20: null is lane-neutral. setNull ensures the field exists but leaves its
+  // kind untouched, so an only-null field stays K_UNKNOWN (-> F64, reads 0),
+  // null+string infers U32, and null+number infers F64. The row's column slot is
+  // left at its default 0, which reads back as 0 on an F64 lane and the reserved
+  // "" on a U32 lane. No per-field flag: K_UNKNOWN already carries the only-null case.
+  setNull(name) {
+    this._ensureField(name);
+  }
+
   _ensureField(name) {
     let meta = this._fieldMeta.get(name);
     if (!meta) {
@@ -218,6 +234,11 @@ export class Writer {
     this._currentFieldIdx = -1;
     this._recordStartByteOffset = 0;
     this._absOffset = 0;
+    // BS-07: the live Tokenizer, handed over via setInputSource at construction.
+    // When present, the sample window measures INPUT bytes (byte-true) by reading
+    // _source.absOffset at record boundaries. null => hand-driven Writer: fall
+    // back to record-count sampling (Math.max(1, ...) below). null is not zero.
+    this._source = null;
 
     this._finalized = false;
 
@@ -237,7 +258,9 @@ export class Writer {
       const f = this._rowValueSlotsF64, u = this._rowValueSlotsU32;
       for (let i = 0; i < f.length; i++) { f[i] = 0; u[i] = 0; }
     }
-    this._recordStartByteOffset = this._absOffset;
+    // Record-boundary read (BS-07): byte-true when a source is wired, else the
+    // dead hand-driven offset (0) that yields record-count sampling. Not the byte scan.
+    this._recordStartByteOffset = this._source !== null ? this._source.absOffset : this._absOffset;
   }
 
   onEndObject() {
@@ -286,7 +309,28 @@ export class Writer {
 
   onTrue()  { this.onNumber(1); }
   onFalse() { this.onNumber(0); }
-  onNull()  { this.onNumber(0); }
+
+  // BS-20: null is lane-neutral. Mirrors onString's guard structure (depth /
+  // finalized doors, unknown-field idx<0 return) but dispatches per lane instead
+  // of throwing: F64 slot -> 0, U32 slot -> 0 = the reserved "" (index 0, real in
+  // every shard since BS-01). onTrue/onFalse stay delegating to onNumber, so a
+  // bool on a U32 lane still throws W_LANE_MISMATCH. Non-null values never reach
+  // here -- zero added cost on the numeric/string hot path.
+  onNull() {
+    if (this._recordDepth !== 1) {
+      if (this._finalized) throw new WriterError('W_FINALIZED', 'writer already finalized');
+      throw new WriterError('W_TOP_LEVEL_NOT_OBJECT', 'top-level null is not an object; LBK1 schema mode stores objects only (SPEC 5.1)');
+    }
+    if (this._schema) {
+      const idx = this._currentFieldIdx;
+      if (idx < 0) return;
+      if (this._fieldLaneKinds[idx] === LANE_F64) this._rowValueSlotsF64[idx] = 0;
+      else this._rowValueSlotsU32[idx] = 0;
+    } else {
+      if (!this._currentKeyName) return;
+      this._sample.setNull(this._currentKeyName);
+    }
+  }
 
   onString(bytes, from, to) {
     if (this._recordDepth !== 1) {
@@ -469,7 +513,8 @@ export class Writer {
       this._totalRows++;
       if (this._currentShardRowCount >= this._currentShardMaxRows) this._finalizeCurrentShard();
     } else {
-      const consumed = Math.max(1, this._absOffset - this._recordStartByteOffset);
+      const endOff = this._source !== null ? this._source.absOffset : this._absOffset;
+      const consumed = Math.max(1, endOff - this._recordStartByteOffset);
       this._sample.commitRow(consumed);
       if (this._sample.byteEstimate >= this._sampleBytes) {
         this._freezeSchemaFromSample();
@@ -611,7 +656,16 @@ export class Writer {
     return this._container;
   }
 
-  setInputByteOffset(n) { this._absOffset = n; }
+  // Optional capability hook (BS-07): the Tokenizer calls this at construction
+  // to hand over the live tokenizer. The Writer then reads src.absOffset at
+  // record boundaries so the sample window is byte-true. A CALL only; the
+  // Tokenizer is never mutated.
+  setInputSource(src) { this._source = src; }
+
+  // Deprecated (BS-07): superseded by setInputSource. The Writer now reads the
+  // live Tokenizer's absOffset at record boundaries, so this hand-driven offset
+  // dance is a documented no-op retained for source compatibility until 2.0.
+  setInputByteOffset() {}
 
   // -------- container assembly --------
 

package/src/index.js

@@ -28,7 +28,7 @@ export {
   PreserveWriter, PreserveWriterError,
   PreserveReader, PreserveReaderError,
 };
-export const VERSION = '1.3.1';
+export const VERSION = '1.4.1';
 
 const encoder = new TextEncoder();
 

package/types/FileIngest.d.ts

@@ -6,6 +6,11 @@ import type { WriterOptions } from './Writer.d.ts';
 
 export const VERSION: string;
 
+/**
+ * Progress state passed to onProgress. The SAME object is reused across every
+ * chunk callback (mutated in place, zero per-chunk allocation): it is valid only
+ * during the callback -- copy it if you retain it past the call.
+ */
 export interface IngestProgress {
   bytesIngested: number;
   totalBytes: number;

package/types/Tokenizer.d.ts

@@ -33,6 +33,12 @@ export class Tokenizer {
   feed(chunk: Uint8Array): void;
   /** Signal end of input. Any pending record is flushed to the sink. */
   end(): void;
+  /**
+   * Cumulative count of input bytes consumed across every `feed()` chunk;
+   * survives chunk boundaries. A sink reads this at record boundaries for
+   * byte-true `sampleBytes` (BS-07); it is never read on the byte scan.
+   */
+  readonly absOffset: number;
 }
 
 export class TokenizerError extends Error {

package/types/Writer.d.ts

@@ -22,8 +22,12 @@ export interface WriterOptions {
   /** Target output bytes per shard. Default 32 MB. */
   targetShardBytes?: number;
   /**
-   * Bytes of input to observe before freezing an inferred schema. Only used
-   * in sample-and-infer mode (no explicit schema). Default = targetShardBytes.
+   * Bytes of INPUT to observe before freezing an inferred schema. Only used in
+   * sample-and-infer mode (no explicit schema). Byte-true (BS-07): the window
+   * ends at the first record boundary at or after `sampleBytes` input bytes,
+   * read from the Tokenizer's `absOffset` -- independent of chunking and record
+   * count. A hand-driven Writer with no wired source samples by record count.
+   * Default = targetShardBytes.
    */
   sampleBytes?: number;
 }
@@ -63,7 +67,18 @@ export class Writer implements TokenizerSinkForWriter {
   onFalse(): void;
   onNull(): void;
   onEnd(): void;
-  /** Optional: set the current input byte offset for byte-estimate tracking. */
+  /**
+   * Optional capability hook (BS-07): the Tokenizer calls this at construction
+   * to hand over the live tokenizer. The Writer then reads `src.absOffset` at
+   * record boundaries so the sample window is byte-true. A call only; the
+   * Tokenizer is never mutated.
+   */
+  setInputSource(src: { readonly absOffset: number }): void;
+  /**
+   * @deprecated Superseded by `setInputSource` (BS-07). The Writer now reads the
+   * live Tokenizer's `absOffset` at record boundaries, so this is a no-op
+   * retained for source compatibility until 2.0.
+   */
   setInputByteOffset(n: number): void;
   /**
    * Seal the container and return the LBK1 binary. Call once, after the