pikelet-wasm 0.9.3 → 0.10.1

package/CHANGELOG.md

@@ -8,6 +8,113 @@ Through 0.6.0 these were published as `pancake-wasm` and
 packages are pre-1.0: a minor bump may carry breaking changes, and each entry lists them
 first.
 
+## pikelet-wasm 0.10.1 / pikelet 0.10.1 — 2026-10-11
+
+Layered packs now serve the same grounding and fetch tuning as single
+files, HTTP readers make fewer, larger range requests, and lazily read
+keyword indexes cache what they read. Upgrade if you serve layered packs
+over `pikelet mcp`: under 0.10.0 they returned no `grounding`, the fact
+check a relevance verdict leaves to the reader. Both packages move to
+0.10.1 together, and `pikelet` requires `pikelet-wasm` ^0.10.1.
+
+### Compatibility
+
+- **HTTP range sources coalesce rerank rows within 256 KiB by default**
+  (was 16 KiB), and a pack's `recommendedGap` is a floor under the
+  source's own gap rather than a replacement for it. On the 456,153-record
+  Wikipedia pack a query makes about 38 range requests instead of 61 and
+  transfers about 2.3 MiB instead of 0.9 MiB; results are unchanged. Pass
+  `preferredGapBytes` to `httpRangeSource`, or `gap` to `query()`, to
+  choose otherwise.
+
+### Fixed
+
+- **A layered pack returns `grounding`.** The chain reader kept only the
+  inherited scorer's verdict and confidence, so `grounding.covered` and
+  `grounding.uncovered` were missing from chain queries and from MCP
+  `search` sections as soon as a pack gained a layer. `recordId` is the
+  chain-global id of the passage the coverage was taken from.
+- **A layered pack honors fetch tuning.** `gap`, `parallelism` and
+  `maxRangeBytes` (and the mount-level `rerankGap`, `rerankParallelism`
+  and `rerankMaxRangeBytes`) did not reach the tiers, and a tier's
+  manifest `recommendedGap` never applied. Each tier now fetches as its
+  own file would.
+- **Lazily read keyword indexes cache postings and share reads.** A
+  repeated query re-read every postings list, and query terms whose
+  entries share an uncached term-table window each read it. Postings are
+  now cached (4 MiB, by bytes), pending reads are shared, and a failed
+  read is retried by the next query rather than cached.
+- **`CompleteQueryResult` declares `grounding`** in the TypeScript
+  types; `query()` already returned it.
+
+## pikelet-wasm 0.10.0 / pikelet 0.10.0 — 2026-10-09
+
+`matchQuality` changes meaning for packs compiled with this release: it
+now measures retrieval relevance — whether the pack covers the
+question's topic — instead of trying to say whether the answer is there.
+Packs already built keep their behavior, since the calibration ships
+inside each file. The inline query encoder runs about 1.65× faster in
+Node.js and Chromium and about 2.2× in WebKit with identical embeddings,
+and readers can now run 12-layer and 4-bit encoders. Both packages move
+to 0.10.0 together, and `pikelet` requires `pikelet-wasm` ^0.10.0.
+
+### Compatibility
+
+- **Newly compiled packs score relevance.** `none` means the pack does
+  not cover the question's topic; a question about a covered topic whose
+  specific fact the pack lacks now comes back `strong` or `weak` instead
+  of `none`. Do not read `strong` as "answered": an asked-for value in a
+  result's `grounding.uncovered` is one no passage states. On the
+  repository's labeled questions, answerable questions called `none`
+  went from 12 of 44 to 0 (docs) and 26 of 50 to 0 (Veyra registry).
+  Rebuild a pack to pick this up; packs compiled by 0.9.x keep their
+  calibration.
+- **The calibration summary changes shape.** The evaluation segment's
+  `calibration` block gains `method` (`self-templates-v6`) and `target`
+  (`relevance`); `cvAuc` is now measured against off-topic queries;
+  `cvAucHard`, answerable vs. in-domain-unanswerable, is reported but no
+  longer gates the fit; `stage1FloorAuc` is gone. Corpora that shipped
+  `unscored` for lack of hard negatives or a low hard-negative AUC now
+  calibrate.
+- **MCP wording changes.** The `search` description, the note on a
+  `none` result, the server instructions and `verify_pack`'s calibration
+  note describe relevance; `verify_pack`'s calibration block gains
+  `method` and `target` (null on older packs). Tool names and result
+  shapes are unchanged.
+- **Packs that declare a layer count other than 6, or 4-bit weights,
+  open only in 0.10.0 readers**; 0.9.x readers refuse them at parse. No
+  published pack declares either.
+
+### Added
+
+- **Inline encoders with 1 to 12 layers and u4 weights.** A kind-3
+  declaration may give `layout.L` from 1 to 12 and `layout.Q: 4` with
+  `layout.B: 32` (u4 weights, two per byte, an f32 scale and offset per
+  block of 32); `Q` absent means u8 in blocks of 64, as before. Exported
+  as `KERNEL_MAX_LAYERS`. The parser and the embedder check the layer
+  count, the format and the blob length before the kernel runs. A u4
+  BGE-small-en-v1.5 blob is 23.9 MiB against MiniLM's 24.3 and matches
+  PyTorch at cosine 0.985 (1.000 against the same weights fake-quantized).
+  `compile` has no flag for these layouts yet.
+
+### Changed
+
+- **The inline encoder runs its projections over tiles of four tokens**
+  instead of one token at a time, so each layer's weights stream through
+  memory once per tile. Median query encoding, before → after: about
+  23 → 14 ms for a 10-word question and 213 → 125 ms for 100 words in
+  Chromium; 32 → 14 and 285 → 134 ms in WebKit. Embeddings are
+  bit-identical, so existing packs and their verification vectors are
+  unaffected.
+- **Calibration** (`compile`, `compact` refits) fits verified on-topic
+  probes against off-topic ones — the built-in foreign bank, minus
+  entries the corpus turns out to cover, plus gibberish — and places
+  `none` at the 95th percentile of the off-topic scores. The
+  in-domain-unanswerable probe classes are still generated, for the
+  `cvAucHard` diagnostic only.
+- **Documentation** (both READMEs, `docs/how-a-query-runs.md`,
+  `docs/veyra-ablation.md`) describes `matchQuality` as relevance.
+
 ## pikelet-wasm 0.9.3 / pikelet 0.9.3 — 2026-10-06
 
 Hardening for readers that mount packs from other publishers or hosts:

package/README.md

@@ -133,7 +133,7 @@ claude \
   --mcp-config examples/one-file-search/web/public/veyra.mcp.json
 ```
 
-All three packs derive from the same small synthetic Station Veyra corpus, with controlled differences in the Tovash evidence. Ask where the Tovash project is housed: the full pack supports Chamber 17, the modified pack supports Chamber 43, and the ablated pack should abstain.
+All three packs derive from the same small synthetic Station Veyra corpus, with controlled differences in the Tovash evidence. Ask where the Tovash project is housed: the full pack supports Chamber 17, the modified pack supports Chamber 43, and the ablated pack should not support any chamber.
 
 ---
 
@@ -153,7 +153,7 @@ A layer records its parent's identity, and its location when the parent sits in
 
 ## Match quality and abstention
 
-A nearest neighbour is not automatically evidence that a corpus answers a question. At build time Pikelet can fit a classifier over retrieval signals — distances, and how much of the question the top passages actually contain ([details](docs/how-a-query-runs.md#how-abstention-is-decided)). When the corpus supports a reliable classifier, results carry `matchQuality: strong | weak | none`. When calibration can't separate supported from unsupported reliably — a single novel may be semantically homogeneous enough that the fit isn't trustworthy — Pikelet reports `matchQuality: unscored` and records why calibration was skipped, rather than manufacturing confidence. In the library, a `none` verdict withholds `results` by default; pass `query(text, { showAbstained: true })` to see the raw retrieval anyway — `matchQuality` and `confidence` are unaffected either way. MCP `search` does the opposite: it returns results under `none` with a note telling the model the support is indirect, because the calibrator can misjudge a paraphrase; `showAbstained: false` withholds them.
+A nearest neighbour is not automatically evidence that a corpus covers a question. At build time Pikelet fits a classifier over retrieval signals — distances, and how much of the question the top passages actually contain — against queries the corpus does not cover ([details](docs/how-a-query-runs.md#how-abstention-is-decided)), and results carry `matchQuality: strong | weak | none`. It measures retrieval relevance: `none` means the pack does not cover the question's topic; `strong` means it does, not that a returned passage states the specific fact asked for. For that, each result lists the question's words no top passage contains (`grounding.uncovered`), and an asked-for value in that list is unsupported whatever `matchQuality` says. When calibration can't separate on-topic from off-topic queries reliably, Pikelet reports `matchQuality: unscored` and records why calibration was skipped, rather than manufacturing confidence. In the library, a `none` verdict withholds `results` by default; pass `query(text, { showAbstained: true })` to see the raw retrieval anyway — `matchQuality` and `confidence` are unaffected either way. MCP `search` does the opposite: it returns results under `none` with a note telling the model the support is indirect, because the calibrator can misjudge a paraphrase; `showAbstained: false` withholds them.
 
 `matchQuality` is evidence about retrieval support. It is **not** a guarantee that an LLM will never hallucinate.
 

package/complete/index.d.ts

@@ -41,6 +41,11 @@ export interface HttpRangeSource extends CompleteRangeSource {
 
 export declare function httpRangeSource(url: string, options?: {
   preferredParallelism?: number;
+  /**
+   * Rows of the vector rerank closer than this many bytes are fetched in
+   * one request, reading the bytes between them. Default 256 KiB. A pack's
+   * manifest recommendedGap applies only when it is wider.
+   */
   preferredGapBytes?: number;
   /** Cap on the one-time full download when a host ignores Range, enforced on received bytes. Default 64 MiB. */
   maxFullFallbackBytes?: number;
@@ -67,6 +72,14 @@ export declare function httpRangeSource(url: string, options?: {
 }): HttpRangeSource;
 
 export interface CompleteQueryResult {
+  /**
+   * The pack's calibrated verdict on this retrieval. On packs calibrated
+   * by pikelet 0.10 or later (evaluation calibration.target 'relevance')
+   * it measures retrieval relevance: 'none' means the pack does not appear
+   * to cover the query's topic, 'strong' that it does — not that a
+   * returned passage states the specific fact asked for. 'unscored': the
+   * pack ships no fitted calibrator.
+   */
   matchQuality: 'strong' | 'weak' | 'none' | 'unscored';
   confidence?: number;
   /**
@@ -76,6 +89,22 @@ export interface CompleteQueryResult {
    * other way round.
    */
   results: Array<{ id: number; distance: number } & Record<string, unknown>>;
+  /**
+   * Which passage grounded the verdict and which of the query's content
+   * words it contains. Present when a fitted calibrator scored the query
+   * (absent on 'unscored'). An asked-for value in `uncovered` is one no top
+   * passage states, whatever matchQuality says.
+   */
+  grounding?: {
+    /** Id of the passage the coverage score was taken from. */
+    recordId: number | null;
+    /** Index of that passage in the fused (or distance-ordered) window. */
+    passageIndex: number;
+    /** Weighted share of the query's content words the passage contains, 0-1. */
+    coverage: number;
+    covered: string[];
+    uncovered: string[];
+  };
 }
 
 export interface CompleteSearch {
@@ -297,6 +326,8 @@ export declare function verifyHostEncoder(
 
 /** Kernel layout compiled into the kind-3 encoder (mirrors encoder.cpp). */
 export declare const KERNEL_LAYOUT: Record<'V' | 'P' | 'T' | 'D' | 'F' | 'L' | 'B' | 'H', number>;
+/** Largest layer count (layout L) the kind-3 kernel runs; L may be 1 to this. */
+export declare const KERNEL_MAX_LAYERS: number;
 export declare function expectedBlobBytes(layout?: Record<string, number>): number;
 export declare function parseInlineTransformerEncoder(encoderBytes: Uint8Array): {
   declaration: Record<string, unknown>;

package/complete/index.mjs

@@ -29,7 +29,7 @@ import { PikeletSketchArtifact } from '../src/artifact/pikelet-artifact.js';
 import { MAGIC, HEADER_BYTES, TABLE_ENTRY_BYTES, KINDS, KIND_NAMES } from './format.mjs';
 
 import {
-    KERNEL_LAYOUT, expectedBlobBytes, parseInlineTransformerEncoder,
+    KERNEL_LAYOUT, KERNEL_MAX_LAYERS, expectedBlobBytes, parseInlineTransformerEncoder,
     createInlineTransformerEmbedder, INLINE_TEST_VECTOR_TEXTS,
     buildInlineTestVectors, verifyInlineTestVectors,
     validateExpectedEmbedding, validateTestVectorTolerance,
@@ -44,7 +44,7 @@ export { openLexicalIndex } from './lexical.mjs';
 export { fuseCandidates, FUSION_DEFAULTS } from './fusion.mjs';
 import { fuseCandidates } from './fusion.mjs';
 export {
-    KERNEL_LAYOUT, expectedBlobBytes, parseInlineTransformerEncoder,
+    KERNEL_LAYOUT, KERNEL_MAX_LAYERS, expectedBlobBytes, parseInlineTransformerEncoder,
     createInlineTransformerEmbedder, INLINE_TEST_VECTOR_TEXTS,
     buildInlineTestVectors, verifyInlineTestVectors,
     validateExpectedEmbedding, validateTestVectorTolerance,
@@ -1243,6 +1243,19 @@ export async function openPikeletFile(input, options = {}) {
             if (closed) throw new Error('.pikelet reader is closed');
         };
 
+        // The rerank's default coalescing gap. The identity-verified manifest
+        // may carry a fetch hint: cluster-ordered layouts only coalesce at a
+        // gap at least matched to their cluster geometry (the wiki pack's
+        // k-means layout needs ~16 KiB; a 2 KiB gap squanders it — measured
+        // 440 vs 200 requests/query at C=600). The hint is a floor: a wider
+        // gap merges a superset of the same runs, so a source that prefers
+        // more (HTTP's 256 KiB) keeps its own. undefined leaves the sketch
+        // reader on the source's preference. A chain reads it per tier
+        // through __chainTier.
+        const rerankGapHint = Number.isFinite(manifest.recommendedGap)
+            ? Math.max(manifest.recommendedGap, Number(source.preferredGapBytes) || 0)
+            : undefined;
+
         return {
             info() {
                 return {
@@ -1368,14 +1381,7 @@ export async function openPikeletFile(input, options = {}) {
                     searched = (await sketch.search(context.vector, k, {
                         rerank: queryOptions.rerank,
                         parallelism: queryOptions.parallelism ?? queryOptions.rerankParallelism ?? options.rerankParallelism,
-                        // The identity-verified manifest may carry a fetch
-                        // hint: cluster-ordered layouts only coalesce at a
-                        // gap matched to their cluster geometry (the wiki
-                        // pack's k-means layout needs ~16 KiB; the default
-                        // 2 KiB squanders it — measured 440 vs 200
-                        // requests/query at C=600).
-                        gap: queryOptions.gap ?? queryOptions.rerankGap ?? options.rerankGap
-                            ?? (Number.isFinite(manifest.recommendedGap) ? manifest.recommendedGap : undefined),
+                        gap: queryOptions.gap ?? queryOptions.rerankGap ?? options.rerankGap ?? rerankGapHint,
                         maxRangeBytes: queryOptions.maxRangeBytes ?? options.rerankMaxRangeBytes,
                         // Present once background staging finished; the
                         // sketch reader ignores it for any query whose C
@@ -1588,6 +1594,10 @@ export async function openPikeletFile(input, options = {}) {
                     // the kernel. A getter, not a captured value, because
                     // staging completes after open() returns.
                     get scanner() { return scanner; },
+                    // The rerank's default coalescing gap for this member
+                    // (manifest hint as a floor under the source's own), so
+                    // a chain fetches each tier's rows as this member would.
+                    rerankGap: rerankGapHint,
                     // The base's OWN encoder, for `append` (6.1): "a kind-3
                     // base carries the teacher, and append loads it from the
                     // base's own query-interp segment — no model download, no

package/complete/inline-transformer.mjs

@@ -5,8 +5,13 @@ const decoder = new TextDecoder();
 // The wasm kernel compiles these in (encoder-spike/encoder.cpp `constexpr`
 // block) and walks the weight blob with a bare running cursor — it never
 // reads the declaration. Declaration/blob skew is therefore only catchable
-// here, host-side, before the first forward.
+// here, host-side, before the first forward. Two values are exceptions,
+// taken by the kernel per call: the layer count, any L from 1 to
+// KERNEL_MAX_LAYERS (KERNEL_LAYOUT.L is MiniLM-L6's, the default), and the
+// weight format, Q bits per weight in blocks of B — u8 in blocks of 64
+// (Q absent or 8, every pack before Q existed) or u4 in blocks of 32.
 export const KERNEL_LAYOUT = { V: 30522, P: 512, T: 2, D: 384, F: 1536, L: 6, B: 64, H: 12 };
+export const KERNEL_MAX_LAYERS = 12;
 const KERNEL_MAX_SEQ = 512;
 // Smallest window that still frames interior tokens between [CLS] and [SEP]
 // with room to make progress; see the maxTokens check in
@@ -14,11 +19,11 @@ const KERNEL_MAX_SEQ = 512;
 const MIN_DECLARED_MAX_TOKENS = 8;
 
 // Byte size the kernel's fill_layout() will consume for a given layout:
-// each quantized matrix is rows*cols u8 plus per-block f32 scales and
-// offsets; biases and layer norms are plain f32.
+// each quantized matrix is rows*cols weights of Q bits plus per-block f32
+// scales and offsets; biases and layer norms are plain f32.
 export function expectedBlobBytes(layout = KERNEL_LAYOUT) {
-  const { V, P, T, D, F, L, B } = layout;
-  const quant = (rows, cols) => rows * cols + rows * (cols / B) * 4 * 2;
+  const { V, P, T, D, F, L, B, Q = 8 } = layout;
+  const quant = (rows, cols) => (rows * cols * Q) / 8 + rows * (cols / B) * 4 * 2;
   const layer = 4 * quant(D, D) + 4 * (D * 4)   // wq,wk,wv,wo + their biases
     + 2 * (D * 4)                                // ln1 gamma/beta
     + quant(F, D) + F * 4                        // wu + bu
@@ -27,6 +32,33 @@ export function expectedBlobBytes(layout = KERNEL_LAYOUT) {
   return quant(V, D) + quant(P, D) + quant(T, D) + 2 * (D * 4) + L * layer;
 }
 
+// The declared layer count, which the kernel takes per call. Its
+// Layout struct holds KERNEL_MAX_LAYERS layers (encoder.cpp MAXL), and it
+// returns -3 outside 1..MAXL rather than walking past them.
+function kernelLayers(layout) {
+  const layers = layout.L;
+  if (!Number.isInteger(layers) || layers < 1 || layers > KERNEL_MAX_LAYERS) {
+    throw new Error(`.pikelet inline-encoder declares layout L=${layers}; `
+      + `the kernel runs 1 to ${KERNEL_MAX_LAYERS} layers`);
+  }
+  return layers;
+}
+
+// The declared weight format as the kernel's bits argument. The kernel's
+// block size is fixed per format, so B must be the one Q implies.
+const KERNEL_WEIGHT_FORMATS = { 8: 64, 4: 32 };
+function kernelWeightBits(layout) {
+  // Only an absent Q means u8 (packs from before Q existed); null is not
+  // absent, and expectedBlobBytes() would size it as zero-bit weights.
+  const bits = layout.Q === undefined ? 8 : layout.Q;
+  if (!Object.hasOwn(KERNEL_WEIGHT_FORMATS, bits) || typeof bits !== 'number'
+      || layout.B !== KERNEL_WEIGHT_FORMATS[bits]) {
+    throw new Error(`.pikelet inline-encoder declares layout Q=${layout.Q} B=${layout.B}; `
+      + 'the kernel runs u8 weights in blocks of 64 (Q absent or 8) or u4 in blocks of 32 (Q=4)');
+  }
+  return bits;
+}
+
 export function parseInlineTransformerEncoder(encoderBytes) {
   const view = new DataView(encoderBytes.buffer, encoderBytes.byteOffset, encoderBytes.byteLength);
   const declLen = view.getUint32(0, true);
@@ -38,7 +70,12 @@ export function parseInlineTransformerEncoder(encoderBytes) {
   const declaration = JSON.parse(decoder.decode(encoderBytes.subarray(12, 12 + declLen)));
   const blob = encoderBytes.subarray(12 + declLen + vocabLen);
   const layout = declaration.layout || {};
+  // L, Q and B first: expectedBlobBytes() below scales with them, so they
+  // must be bounded before they size anything.
+  kernelLayers(layout);
+  kernelWeightBits(layout);
   for (const key of Object.keys(KERNEL_LAYOUT)) {
+    if (key === 'L' || key === 'B') continue;
     if (layout[key] !== KERNEL_LAYOUT[key]) {
       throw new Error(`.pikelet inline-encoder declares layout ${key}=${layout[key]}; `
         + `the compiled kernel requires ${key}=${KERNEL_LAYOUT[key]}`);
@@ -80,6 +117,16 @@ export async function createInlineTransformerEmbedder({ declaration, vocabText,
   if (vocabSize > KERNEL_LAYOUT.V) {
     throw new Error(`inline-encoder vocab has ${vocabSize} entries; the kernel's embedding table holds ${KERNEL_LAYOUT.V}`);
   }
+  // Like maxTokens below, checked here too because this function is also
+  // called directly, without parseInlineTransformerEncoder(): the kernel
+  // walks the blob for exactly this many layers in this weight format.
+  const layers = kernelLayers({ L: declaration.layout?.L ?? KERNEL_LAYOUT.L });
+  const blockSize = declaration.layout?.B ?? KERNEL_LAYOUT.B;
+  const bits = kernelWeightBits({ Q: declaration.layout?.Q, B: blockSize });
+  const blobBytes = expectedBlobBytes({ ...KERNEL_LAYOUT, L: layers, Q: bits, B: blockSize });
+  if (blob.length !== blobBytes) {
+    throw new Error(`inline-encoder blob is ${blob.length} bytes but ${layers} layers of u${bits} weights imply ${blobBytes}`);
+  }
   const tokenizer = createWordPiece(vocabText);
   const EM = await createEncoder();
   const dim = declaration.dim;
@@ -121,7 +168,7 @@ export async function createInlineTransformerEmbedder({ declaration, vocabText,
         throw new Error(`inline encoder window is ${tokenIds.length} tokens for a ${maxSeq}-token buffer`);
       }
       new Int32Array(EM.HEAP32.buffer, idsPtr, tokenIds.length).set(tokenIds);
-      const rc = EM._encoder_forward(blobPtr, idsPtr, tokenIds.length, hiddenPtr, 0);
+      const rc = EM._encoder_forward(blobPtr, idsPtr, tokenIds.length, hiddenPtr, 0, layers, bits);
       if (rc !== tokenIds.length) throw new Error(`inline encoder failed: ${rc}`);
       const hidden = new Float32Array(EM.HEAPF32.buffer, hiddenPtr, tokenIds.length * dim);
       if (usesClsPooling) {
@@ -181,7 +228,7 @@ export async function createInlineTransformerEmbedder({ declaration, vocabText,
         throw new Error(`inline encoder window is ${tokenIds.length} tokens for a ${maxSeq}-token buffer`);
       }
       new Int32Array(EM.HEAP32.buffer, idsPtr, tokenIds.length).set(tokenIds);
-      const rc = EM._encoder_forward(blobPtr, idsPtr, tokenIds.length, hiddenPtr, 0);
+      const rc = EM._encoder_forward(blobPtr, idsPtr, tokenIds.length, hiddenPtr, 0, layers, bits);
       if (rc !== tokenIds.length) throw new Error(`inline encoder failed: ${rc}`);
       const hidden = new Float32Array(EM.HEAPF32.buffer, hiddenPtr, tokenIds.length * dim);
       // Offset by 1: tokenIds[0] is [CLS], so idSlice's tokens start at

package/complete/layer-reader.mjs

@@ -465,6 +465,13 @@ export async function openPikeletChain(members, options = {}) {
                     // tier has no live deletion (maskCounts, computed at mount).
                     const out = await t.reader.sketch.search(vector, Math.min(k, t.records), {
                         rerank: queryOptions.rerank,
+                        // The single-file reader's fetch tuning, with the same
+                        // precedence: the query's own value, then the mount's,
+                        // then this tier's default gap (its manifest hint as a
+                        // floor under its source's preference).
+                        parallelism: queryOptions.parallelism ?? queryOptions.rerankParallelism ?? options.rerankParallelism,
+                        gap: queryOptions.gap ?? queryOptions.rerankGap ?? options.rerankGap ?? t.reader.rerankGap,
+                        maxRangeBytes: queryOptions.maxRangeBytes ?? options.rerankMaxRangeBytes,
                         ...(maskCounts[j] > 0 ? { exclude: masks[j] } : {}),
                         // The tier's own staged scan kernel. Read per query, not
                         // captured at mount: staging finishes in the background
@@ -528,6 +535,7 @@ export async function openPikeletChain(members, options = {}) {
                 // to prevent.
                 let matchQuality = 'unscored';
                 let confidence;
+                let grounding = null;
                 if (calibration.status === 'inherited' && base.scoreQuality) {
                     // Coverage reads the fit's passagesNeeded from the FULL
                     // fused window; top.slice(0, 3) gave it one passage at k=1.
@@ -553,6 +561,13 @@ export async function openPikeletChain(members, options = {}) {
                     if (scored && scored.match_quality) {
                         matchQuality = scored.match_quality;
                         if (Number.isFinite(scored.confidence)) confidence = scored.confidence;
+                        // Which passage grounded the verdict and which query
+                        // words no passage contains, as the single-file
+                        // reader returns it; recordId is the chain-global id
+                        // of the passage coverage scored.
+                        if (scored.grounding) {
+                            grounding = { ...scored.grounding, recordId: passageOrder[scored.grounding.passageIndex]?.id ?? null };
+                        }
                     }
                 } else if (calibration.status === 'none') {
                     matchQuality = 'unscored';
@@ -567,6 +582,7 @@ export async function openPikeletChain(members, options = {}) {
                     matchQuality,
                     ...(confidence === undefined ? {} : { confidence }),
                     results: withheld ? [] : results,
+                    ...(grounding ? { grounding } : {}),
                     identity: chainMembers[chainMembers.length - 1].identity,
                     calibration: {
                         status: calibration.status,

package/complete/lexical.mjs

@@ -80,15 +80,58 @@ export async function openLexicalIndexLazy(read, segLength) {
     const doclenOf = (id) => doclenView.getUint32(4 * id, true);
 
     const WINDOW = 512;
-    const windowCache = new Map(); // startIdx -> entry bytes (small LRU)
-    async function readWindow(startIdx, count) {
+    // Both caches hold the pending read, not its result, so concurrent
+    // lookups of one window (two query terms whose hashes land in it) or one
+    // postings list share a single read; a failed read is evicted so the
+    // next query retries rather than re-throwing a cached rejection.
+    const windowCache = new Map(); // `${startIdx}:${count}` -> Promise<entry bytes> (small LRU)
+    function readWindow(startIdx, count) {
         const key = `${startIdx}:${count}`;
         const hit = windowCache.get(key);
-        if (hit) return hit;
-        const bytes = await read(termTableOffset + 24 * startIdx, 24 * count);
+        if (hit) {
+            windowCache.delete(key);
+            windowCache.set(key, hit);
+            return hit;
+        }
+        const pending = read(termTableOffset + 24 * startIdx, 24 * count);
         if (windowCache.size >= 64) windowCache.delete(windowCache.keys().next().value);
-        windowCache.set(key, bytes);
-        return bytes;
+        windowCache.set(key, pending);
+        pending.catch(() => { if (windowCache.get(key) === pending) windowCache.delete(key); });
+        return pending;
+    }
+    // Postings, LRU by bytes: a repeated or overlapping query re-reads only
+    // the terms it has not seen. A list larger than POSTINGS_CACHE_MAX_ENTRY
+    // (a very common term) is read every time rather than evicting
+    // everything else.
+    const POSTINGS_CACHE_BYTES = 4 * 1024 * 1024;
+    const POSTINGS_CACHE_MAX_ENTRY = 1024 * 1024;
+    const postingsCache = new Map(); // postingsRel -> { pending, bytes }
+    let postingsCachedBytes = 0;
+    function readPostings(rel, len) {
+        const hit = postingsCache.get(rel);
+        if (hit && hit.bytes === len) {
+            postingsCache.delete(rel);
+            postingsCache.set(rel, hit);
+            return hit.pending;
+        }
+        const pending = read(postingsOffset + rel, len);
+        if (len > POSTINGS_CACHE_MAX_ENTRY) return pending;
+        const entry = { pending, bytes: len };
+        if (hit) postingsCachedBytes -= hit.bytes;
+        postingsCache.set(rel, entry);
+        postingsCachedBytes += len;
+        while (postingsCachedBytes > POSTINGS_CACHE_BYTES && postingsCache.size > 1) {
+            const [oldRel, old] = postingsCache.entries().next().value;
+            postingsCache.delete(oldRel);
+            postingsCachedBytes -= old.bytes;
+        }
+        pending.catch(() => {
+            if (postingsCache.get(rel) === entry) {
+                postingsCache.delete(rel);
+                postingsCachedBytes -= len;
+            }
+        });
+        return pending;
     }
     const entryAt = (bytes, i) => {
         const view = new DataView(bytes.buffer, bytes.byteOffset + 24 * i, 24);
@@ -195,7 +238,7 @@ export async function openLexicalIndexLazy(read, segLength) {
                     throw new Error('.pikelet lexical postings out of bounds');
                 }
             }
-            const postings = await Promise.all(found.map((e) => read(postingsOffset + e.postingsRel, e.postingsLen)));
+            const postings = await Promise.all(found.map((e) => readPostings(e.postingsRel, e.postingsLen)));
             const scores = new Map();
             found.forEach((e, i) => {
                 const bytes = postings[i];

package/complete/retrieval-abstention.mjs

@@ -4,6 +4,12 @@
 // bloom filter, standardized and passed through the fitted logistic model.
 // Verdict semantics: 'answer' (strong match), 'weak' (closest match is
 // distant — shown with a caveat), 'abstain' (nothing useful in the pack).
+// What "match" means is the asset's: pikelet's self-templates-v6 assets
+// (calibration.target 'relevance') are fit against off-topic queries, so
+// 'abstain' means the pack does not cover the query's topic and 'answer'
+// means it does — not that a passage states the asked-for fact. Earlier
+// assets were fit against in-domain-unanswerable queries instead. The
+// scoring math here is the same for both.
 //
 // Assets calibrated by pikelet's self-templates-v2+ may carry an additional
 // grounding term (asset.coverage): the fraction of the query's content

package/complete/sources.mjs

@@ -102,7 +102,12 @@ export function httpRangeSource(url, options = {}) {
         url,
         size: undefined,
         preferredParallelism: options.preferredParallelism ?? 32,
-        preferredGapBytes: options.preferredGapBytes ?? 16 * 1024,
+        // Rows closer than this are fetched in one request, reading the
+        // bytes between them. Over HTTP a round trip costs far more than
+        // the bytes: on the 648.5 MB wiki pack in Chromium, 256 KiB took
+        // 61 -> 38 requests and about 2.8 -> 2.2 s per query against 16 KiB,
+        // at 0.9 -> 2.3 MiB transferred.
+        preferredGapBytes: options.preferredGapBytes ?? 256 * 1024,
 
         async init() {
             await resolveTarget();

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pikelet-wasm",
-  "version": "0.9.3",
+  "version": "0.10.1",
   "description": "Vector search engine and .pikelet artifact readers/builders, in WebAssembly. Compile a corpus into one self-contained, hash-verified search file, or use the underlying HNSW engine directly. Runs in Node.js, browsers, and Cloudflare Workers with no native dependencies.",
   "main": "src/pikelet.js",
   "types": "src/pikelet.d.ts",