pikelet-wasm 0.9.2 → 0.10.0

package/CHANGELOG.md

@@ -8,6 +8,135 @@ Through 0.6.0 these were published as `pancake-wasm` and
 packages are pre-1.0: a minor bump may carry breaking changes, and each entry lists them
 first.
 
+## pikelet-wasm 0.10.0 / pikelet 0.10.0 — 2026-10-09
+
+`matchQuality` changes meaning for packs compiled with this release: it
+now measures retrieval relevance — whether the pack covers the
+question's topic — instead of trying to say whether the answer is there.
+Packs already built keep their behavior, since the calibration ships
+inside each file. The inline query encoder runs about 1.65× faster in
+Node.js and Chromium and about 2.2× in WebKit with identical embeddings,
+and readers can now run 12-layer and 4-bit encoders. Both packages move
+to 0.10.0 together, and `pikelet` requires `pikelet-wasm` ^0.10.0.
+
+### Compatibility
+
+- **Newly compiled packs score relevance.** `none` means the pack does
+  not cover the question's topic; a question about a covered topic whose
+  specific fact the pack lacks now comes back `strong` or `weak` instead
+  of `none`. Do not read `strong` as "answered": an asked-for value in a
+  result's `grounding.uncovered` is one no passage states. On the
+  repository's labeled questions, answerable questions called `none`
+  went from 12 of 44 to 0 (docs) and 26 of 50 to 0 (Veyra registry).
+  Rebuild a pack to pick this up; packs compiled by 0.9.x keep their
+  calibration.
+- **The calibration summary changes shape.** The evaluation segment's
+  `calibration` block gains `method` (`self-templates-v6`) and `target`
+  (`relevance`); `cvAuc` is now measured against off-topic queries;
+  `cvAucHard`, answerable vs. in-domain-unanswerable, is reported but no
+  longer gates the fit; `stage1FloorAuc` is gone. Corpora that shipped
+  `unscored` for lack of hard negatives or a low hard-negative AUC now
+  calibrate.
+- **MCP wording changes.** The `search` description, the note on a
+  `none` result, the server instructions and `verify_pack`'s calibration
+  note describe relevance; `verify_pack`'s calibration block gains
+  `method` and `target` (null on older packs). Tool names and result
+  shapes are unchanged.
+- **Packs that declare a layer count other than 6, or 4-bit weights,
+  open only in 0.10.0 readers**; 0.9.x readers refuse them at parse. No
+  published pack declares either.
+
+### Added
+
+- **Inline encoders with 1 to 12 layers and u4 weights.** A kind-3
+  declaration may give `layout.L` from 1 to 12 and `layout.Q: 4` with
+  `layout.B: 32` (u4 weights, two per byte, an f32 scale and offset per
+  block of 32); `Q` absent means u8 in blocks of 64, as before. Exported
+  as `KERNEL_MAX_LAYERS`. The parser and the embedder check the layer
+  count, the format and the blob length before the kernel runs. A u4
+  BGE-small-en-v1.5 blob is 23.9 MiB against MiniLM's 24.3 and matches
+  PyTorch at cosine 0.985 (1.000 against the same weights fake-quantized).
+  `compile` has no flag for these layouts yet.
+
+### Changed
+
+- **The inline encoder runs its projections over tiles of four tokens**
+  instead of one token at a time, so each layer's weights stream through
+  memory once per tile. Median query encoding, before → after: about
+  23 → 14 ms for a 10-word question and 213 → 125 ms for 100 words in
+  Chromium; 32 → 14 and 285 → 134 ms in WebKit. Embeddings are
+  bit-identical, so existing packs and their verification vectors are
+  unaffected.
+- **Calibration** (`compile`, `compact` refits) fits verified on-topic
+  probes against off-topic ones — the built-in foreign bank, minus
+  entries the corpus turns out to cover, plus gibberish — and places
+  `none` at the 95th percentile of the off-topic scores. The
+  in-domain-unanswerable probe classes are still generated, for the
+  `cvAucHard` diagnostic only.
+- **Documentation** (both READMEs, `docs/how-a-query-runs.md`,
+  `docs/veyra-ablation.md`) describes `matchQuality` as relevance.
+
+## pikelet-wasm 0.9.3 / pikelet 0.9.3 — 2026-10-06
+
+Hardening for readers that mount packs from other publishers or hosts:
+the index header commitment now covers the bytes actually parsed, and a
+pack can no longer steer lazy reads, demand an exhaustive rerank, or turn
+a malformed abstention asset into confident answers. Upgrade if you mount
+packs you did not build, including over `pikelet mcp`. Both packages move
+to 0.9.3 together, and `pikelet` requires `pikelet-wasm` ^0.9.3 so that
+`pikelet mcp` serves through the hardened reader.
+
+### Compatibility
+
+- **A pack that declares a rerank depth over 4096 is served at 4096 by
+  default**, unless the caller passes `rerank`. No published pack is
+  affected.
+- **A pack whose abstention asset has malformed `thresholds` or
+  `coverage.topK` now reports `unscored`** instead of answering. Packs
+  built by `compile` are not affected.
+- **`compile` declares at most 4096** for a pack over 4096 records whose
+  sweep does not meet its recall target, and warns. Other packs compile
+  as under 0.9.2.
+
+### Fixed
+
+- **The index header commitment binds the bytes the reader parses.** The
+  complete reader checked `manifest.index.headerSha256` on its own read of
+  the sketch header, while the sketch parsed a separate read of the same
+  range. A host answering the two reads differently could pass the check
+  with the genuine header and have a forged one parsed, together with the
+  integrity anchors it carries, replacing the index under a pinned
+  identity. `PikeletSketchArtifact.open()` now takes
+  `expectedHeaderSha256` and checks the header it parses.
+- **Lazily read keyword segments cannot steer reads or inflate them.**
+  Lexical segments over 8 MiB are read on demand without per-read
+  verification. Reads are now confined to the segment's window, and each
+  term entry is checked before its postings are fetched: inside the
+  postings region, `df` no larger than the corpus, and a length a postings
+  list of that `df` can occupy. `info().lexical.integrity` reports
+  `'unverified-lazy-reads'` for such segments and `'segment-sha256'` for
+  eagerly verified ones.
+- **A pack's declared rerank depth is capped at 4096 by default.** The
+  sketch header's `recommendedRerank` is a producer hint; a value near the
+  row count made every query fetch, hash and rerank the whole vectors
+  region. Readers default to at most 4096, keep the declared value as
+  `declaredRecommendedRerank`, and still honor an explicit `rerank`. The
+  published wiki, astro-docs and rust-book packs declare 200, 40 and 30.
+- **`compile` and `append` no longer declare a rerank depth readers will not
+  use.** `compile`'s rerank sweep fell back to the full row count when no
+  depth up to 640 met its recall target; it now stops at the same 4096
+  ceiling, records the recall measured there, and logs a warning when the
+  target was not met. `append` caps a layer's exhaustive hint the same way.
+  `measureRecommendedRerank` returns `targetReached` and `rerankCeiling`,
+  and `pikelet-wasm/artifact` exports `MAX_RECOMMENDED_RERANK`.
+- **A malformed abstention asset serves `unscored`.** Missing or
+  non-numeric `thresholds` compared false against every score, so the pack
+  answered every query as if calibrated; out-of-order or out-of-range ones
+  gave verdicts the fit never defined. A `coverage.topK` outside 1–10 is
+  refused too, since each counted passage is a record read on every query.
+  The scorer now rejects such an asset and the pack reports `unscored`.
+  Assets written by `compile` are unaffected.
+
 ## pikelet-wasm 0.9.2 / pikelet 0.9.2 — 2026-10-01
 
 Makes the layered update path work as documented: `append` on a freshly

package/README.md

@@ -133,7 +133,7 @@ claude \
   --mcp-config examples/one-file-search/web/public/veyra.mcp.json
 ```
 
-All three packs derive from the same small synthetic Station Veyra corpus, with controlled differences in the Tovash evidence. Ask where the Tovash project is housed: the full pack supports Chamber 17, the modified pack supports Chamber 43, and the ablated pack should abstain.
+All three packs derive from the same small synthetic Station Veyra corpus, with controlled differences in the Tovash evidence. Ask where the Tovash project is housed: the full pack supports Chamber 17, the modified pack supports Chamber 43, and the ablated pack should not support any chamber.
 
 ---
 
@@ -153,7 +153,7 @@ A layer records its parent's identity, and its location when the parent sits in
 
 ## Match quality and abstention
 
-A nearest neighbour is not automatically evidence that a corpus answers a question. At build time Pikelet can fit a classifier over retrieval signals — distances, and how much of the question the top passages actually contain ([details](docs/how-a-query-runs.md#how-abstention-is-decided)). When the corpus supports a reliable classifier, results carry `matchQuality: strong | weak | none`. When calibration can't separate supported from unsupported reliably — a single novel may be semantically homogeneous enough that the fit isn't trustworthy — Pikelet reports `matchQuality: unscored` and records why calibration was skipped, rather than manufacturing confidence. In the library, a `none` verdict withholds `results` by default; pass `query(text, { showAbstained: true })` to see the raw retrieval anyway — `matchQuality` and `confidence` are unaffected either way. MCP `search` does the opposite: it returns results under `none` with a note telling the model the support is indirect, because the calibrator can misjudge a paraphrase; `showAbstained: false` withholds them.
+A nearest neighbour is not automatically evidence that a corpus covers a question. At build time Pikelet fits a classifier over retrieval signals — distances, and how much of the question the top passages actually contain — against queries the corpus does not cover ([details](docs/how-a-query-runs.md#how-abstention-is-decided)), and results carry `matchQuality: strong | weak | none`. It measures retrieval relevance: `none` means the pack does not cover the question's topic; `strong` means it does, not that a returned passage states the specific fact asked for. For that, each result lists the question's words no top passage contains (`grounding.uncovered`), and an asked-for value in that list is unsupported whatever `matchQuality` says. When calibration can't separate on-topic from off-topic queries reliably, Pikelet reports `matchQuality: unscored` and records why calibration was skipped, rather than manufacturing confidence. In the library, a `none` verdict withholds `results` by default; pass `query(text, { showAbstained: true })` to see the raw retrieval anyway — `matchQuality` and `confidence` are unaffected either way. MCP `search` does the opposite: it returns results under `none` with a note telling the model the support is indirect, because the calibrator can misjudge a paraphrase; `showAbstained: false` withholds them.
 
 `matchQuality` is evidence about retrieval support. It is **not** a guarantee that an LLM will never hallucinate.
 

package/complete/builder.d.ts

@@ -71,6 +71,10 @@ export declare function measureRecommendedRerank(input: {
 }): Promise<{
   recommendedRerank: number;
   recall: number;
+  /** False when no rung up to rerankCeiling reached targetRecall; recommendedRerank is then the ceiling. */
+  targetReached: boolean;
+  /** min(row count, MAX_RECOMMENDED_RERANK): the deepest rerank the sweep measures. */
+  rerankCeiling: number;
   k: number;
   targetRecall: number;
   queries: number;

package/complete/builder.mjs

@@ -351,6 +351,12 @@ export function buildInlineTransformerEncoderSegment({ declaration, vocabBytes,
 // targetRecall — plus the whole curve for the evaluation segment. Queries
 // are the corpus's own float embeddings when the caller has them, else rows
 // dequantized out of the snapshot.
+//
+// The sweep stops at the reader's default ceiling (MAX_RECOMMENDED_RERANK):
+// it used to fall back to the full row count, a depth readers no longer
+// honor by default, so a pack could record a recall it would not be served
+// at. The last rung is the ceiling, measured like any other, and
+// targetReached says whether the target was met below it.
 export async function measureRecommendedRerank({
   artifactModule,
   sketchBytes,
@@ -387,9 +393,10 @@ export async function measureRecommendedRerank({
     for (const q of queries) {
       truth.push(new Set((await sketch.search(q, topK, { rerank: count })).results.map((r) => r.id)));
     }
+    const rerankCeiling = Math.min(count, artifactModule.MAX_RECOMMENDED_RERANK ?? count);
     const ladder = (sweep || [10, 15, 20, 30, 40, 60, 80, 120, 160, 240, 320, 480, 640])
-      .filter((c) => c >= topK && c < count);
-    ladder.push(count);
+      .filter((c) => c >= topK && c < rerankCeiling);
+    ladder.push(rerankCeiling);
     const curve = [];
     let recommendedRerank = null;
     let recommendedRecall = null;
@@ -409,13 +416,16 @@ export async function measureRecommendedRerank({
         recommendedRecall = recall;
       }
     }
-    if (recommendedRerank === null) {
-      recommendedRerank = count;
+    const targetReached = recommendedRerank !== null;
+    if (!targetReached) {
+      recommendedRerank = rerankCeiling;
       recommendedRecall = curve[curve.length - 1].recall;
     }
     return {
       recommendedRerank,
       recall: recommendedRecall,
+      targetReached,
+      rerankCeiling,
       k: topK,
       targetRecall,
       queries: queries.length,

package/complete/index.d.ts

@@ -67,6 +67,14 @@ export declare function httpRangeSource(url: string, options?: {
 }): HttpRangeSource;
 
 export interface CompleteQueryResult {
+  /**
+   * The pack's calibrated verdict on this retrieval. On packs calibrated
+   * by pikelet 0.10 or later (evaluation calibration.target 'relevance')
+   * it measures retrieval relevance: 'none' means the pack does not appear
+   * to cover the query's topic, 'strong' that it does — not that a
+   * returned passage states the specific fact asked for. 'unscored': the
+   * pack ships no fitted calibrator.
+   */
   matchQuality: 'strong' | 'weak' | 'none' | 'unscored';
   confidence?: number;
   /**
@@ -134,7 +142,13 @@ export interface CompleteSearch {
      * order fuses distance and BM25 rankings by reciprocal rank). Null on
      * vector-only artifacts.
      */
-    lexical: { terms: number; docCount: number; lazy?: true } | null;
+    lexical: {
+        terms: number;
+        docCount: number;
+        /** 'segment-sha256' when read eagerly and verified whole; 'unverified-lazy-reads' for lazily opened segments. */
+        integrity: 'segment-sha256' | 'unverified-lazy-reads';
+        lazy?: true;
+    } | null;
     /**
      * 'engine' once a WASM scan kernel serves the resident scan
      * (auto-staged or injected via options.sketchScanner); 'js' before
@@ -291,6 +305,8 @@ export declare function verifyHostEncoder(
 
 /** Kernel layout compiled into the kind-3 encoder (mirrors encoder.cpp). */
 export declare const KERNEL_LAYOUT: Record<'V' | 'P' | 'T' | 'D' | 'F' | 'L' | 'B' | 'H', number>;
+/** Largest layer count (layout L) the kind-3 kernel runs; L may be 1 to this. */
+export declare const KERNEL_MAX_LAYERS: number;
 export declare function expectedBlobBytes(layout?: Record<string, number>): number;
 export declare function parseInlineTransformerEncoder(encoderBytes: Uint8Array): {
   declaration: Record<string, unknown>;

package/complete/index.mjs

@@ -29,7 +29,7 @@ import { PikeletSketchArtifact } from '../src/artifact/pikelet-artifact.js';
 import { MAGIC, HEADER_BYTES, TABLE_ENTRY_BYTES, KINDS, KIND_NAMES } from './format.mjs';
 
 import {
-    KERNEL_LAYOUT, expectedBlobBytes, parseInlineTransformerEncoder,
+    KERNEL_LAYOUT, KERNEL_MAX_LAYERS, expectedBlobBytes, parseInlineTransformerEncoder,
     createInlineTransformerEmbedder, INLINE_TEST_VECTOR_TEXTS,
     buildInlineTestVectors, verifyInlineTestVectors,
     validateExpectedEmbedding, validateTestVectorTolerance,
@@ -44,7 +44,7 @@ export { openLexicalIndex } from './lexical.mjs';
 export { fuseCandidates, FUSION_DEFAULTS } from './fusion.mjs';
 import { fuseCandidates } from './fusion.mjs';
 export {
-    KERNEL_LAYOUT, expectedBlobBytes, parseInlineTransformerEncoder,
+    KERNEL_LAYOUT, KERNEL_MAX_LAYERS, expectedBlobBytes, parseInlineTransformerEncoder,
     createInlineTransformerEmbedder, INLINE_TEST_VECTOR_TEXTS,
     buildInlineTestVectors, verifyInlineTestVectors,
     validateExpectedEmbedding, validateTestVectorTolerance,
@@ -780,7 +780,12 @@ export async function openPikeletFile(input, options = {}) {
                 }
                 return bytes;
             })(),
-            PikeletSketchArtifact.open(windowSource(source, idx.offset, idx.length), { maxReadBytes }),
+            // Format 2's header commitment is checked inside open(), against
+            // the same bytes the sketch parses (see expectedHeaderSha256).
+            PikeletSketchArtifact.open(windowSource(source, idx.offset, idx.length), {
+                maxReadBytes,
+                ...(perRecord ? { expectedHeaderSha256: manifest.index.headerSha256 } : {}),
+            }),
             readChecked(source, corpus.offset, tablesBytes, 'corpus tables', maxReadBytes, fileBytes),
             // Lexical index (kind 5, OPTIONAL). Small segments: eager
             // whole-segment read, digest-verified like the query-interp
@@ -791,7 +796,15 @@ export async function openPikeletFile(input, options = {}) {
             segments.has('lexical') ? (async () => {
                 const seg = segments.get('lexical');
                 if (seg.length > LAZY_LEXICAL_BYTES) {
-                    const lexRead = (off, len) => readChecked(source, seg.offset + off, len, 'lexical segment', maxReadBytes, fileBytes);
+                    // Confined to the segment's own window: lazy reads are
+                    // not individually verified, so no field they carry may
+                    // steer a read into another segment.
+                    const lexRead = (off, len) => {
+                        if (!Number.isSafeInteger(off) || !Number.isSafeInteger(len) || off < 0 || len < 0 || off + len > seg.length) {
+                            throw new Error('.pikelet lexical segment read is outside the segment window');
+                        }
+                        return readChecked(source, seg.offset + off, len, 'lexical segment', maxReadBytes, fileBytes);
+                    };
                     return { lazyIndex: await openLexicalIndexLazy(lexRead, seg.length) };
                 }
                 const bytes = await readChecked(source, seg.offset, seg.length, 'lexical segment', maxReadBytes, fileBytes);
@@ -800,12 +813,6 @@ export async function openPikeletFile(input, options = {}) {
                 }
                 return { bytes };
             })() : null,
-            perRecord ? (async () => {
-                const headerBytes = await readChecked(source, idx.offset, SKETCH_HEADER_BYTES, 'index header', maxReadBytes, fileBytes);
-                if (await sha256hex(headerBytes) !== manifest.index.headerSha256) {
-                    throw new Error('.pikelet index header failed hash verification against the manifest');
-                }
-            })() : null,
         ]);
 
         const lexicalIndex = lexicalBytes
@@ -1289,6 +1296,9 @@ export async function openPikeletFile(input, options = {}) {
                         ? {
                             terms: lexicalIndex.termCount,
                             docCount: lexicalIndex.docCount,
+                            // Lazy reads are bounded to the segment but not
+                            // individually verified (LAZY_LEXICAL_BYTES).
+                            integrity: lexicalIndex.lazy ? 'unverified-lazy-reads' : 'segment-sha256',
                             ...(lexicalIndex.lazy ? { lazy: true } : {}),
                         }
                         : null,

package/complete/inline-transformer.mjs

@@ -5,8 +5,13 @@ const decoder = new TextDecoder();
 // The wasm kernel compiles these in (encoder-spike/encoder.cpp `constexpr`
 // block) and walks the weight blob with a bare running cursor — it never
 // reads the declaration. Declaration/blob skew is therefore only catchable
-// here, host-side, before the first forward.
+// here, host-side, before the first forward. Two values are exceptions,
+// taken by the kernel per call: the layer count, any L from 1 to
+// KERNEL_MAX_LAYERS (KERNEL_LAYOUT.L is MiniLM-L6's, the default), and the
+// weight format, Q bits per weight in blocks of B — u8 in blocks of 64
+// (Q absent or 8, every pack before Q existed) or u4 in blocks of 32.
 export const KERNEL_LAYOUT = { V: 30522, P: 512, T: 2, D: 384, F: 1536, L: 6, B: 64, H: 12 };
+export const KERNEL_MAX_LAYERS = 12;
 const KERNEL_MAX_SEQ = 512;
 // Smallest window that still frames interior tokens between [CLS] and [SEP]
 // with room to make progress; see the maxTokens check in
@@ -14,11 +19,11 @@ const KERNEL_MAX_SEQ = 512;
 const MIN_DECLARED_MAX_TOKENS = 8;
 
 // Byte size the kernel's fill_layout() will consume for a given layout:
-// each quantized matrix is rows*cols u8 plus per-block f32 scales and
-// offsets; biases and layer norms are plain f32.
+// each quantized matrix is rows*cols weights of Q bits plus per-block f32
+// scales and offsets; biases and layer norms are plain f32.
 export function expectedBlobBytes(layout = KERNEL_LAYOUT) {
-  const { V, P, T, D, F, L, B } = layout;
-  const quant = (rows, cols) => rows * cols + rows * (cols / B) * 4 * 2;
+  const { V, P, T, D, F, L, B, Q = 8 } = layout;
+  const quant = (rows, cols) => (rows * cols * Q) / 8 + rows * (cols / B) * 4 * 2;
   const layer = 4 * quant(D, D) + 4 * (D * 4)   // wq,wk,wv,wo + their biases
     + 2 * (D * 4)                                // ln1 gamma/beta
     + quant(F, D) + F * 4                        // wu + bu
@@ -27,6 +32,33 @@ export function expectedBlobBytes(layout = KERNEL_LAYOUT) {
   return quant(V, D) + quant(P, D) + quant(T, D) + 2 * (D * 4) + L * layer;
 }
 
+// The declared layer count, which the kernel takes per call. Its
+// Layout struct holds KERNEL_MAX_LAYERS layers (encoder.cpp MAXL), and it
+// returns -3 outside 1..MAXL rather than walking past them.
+function kernelLayers(layout) {
+  const layers = layout.L;
+  if (!Number.isInteger(layers) || layers < 1 || layers > KERNEL_MAX_LAYERS) {
+    throw new Error(`.pikelet inline-encoder declares layout L=${layers}; `
+      + `the kernel runs 1 to ${KERNEL_MAX_LAYERS} layers`);
+  }
+  return layers;
+}
+
+// The declared weight format as the kernel's bits argument. The kernel's
+// block size is fixed per format, so B must be the one Q implies.
+const KERNEL_WEIGHT_FORMATS = { 8: 64, 4: 32 };
+function kernelWeightBits(layout) {
+  // Only an absent Q means u8 (packs from before Q existed); null is not
+  // absent, and expectedBlobBytes() would size it as zero-bit weights.
+  const bits = layout.Q === undefined ? 8 : layout.Q;
+  if (!Object.hasOwn(KERNEL_WEIGHT_FORMATS, bits) || typeof bits !== 'number'
+      || layout.B !== KERNEL_WEIGHT_FORMATS[bits]) {
+    throw new Error(`.pikelet inline-encoder declares layout Q=${layout.Q} B=${layout.B}; `
+      + 'the kernel runs u8 weights in blocks of 64 (Q absent or 8) or u4 in blocks of 32 (Q=4)');
+  }
+  return bits;
+}
+
 export function parseInlineTransformerEncoder(encoderBytes) {
   const view = new DataView(encoderBytes.buffer, encoderBytes.byteOffset, encoderBytes.byteLength);
   const declLen = view.getUint32(0, true);
@@ -38,7 +70,12 @@ export function parseInlineTransformerEncoder(encoderBytes) {
   const declaration = JSON.parse(decoder.decode(encoderBytes.subarray(12, 12 + declLen)));
   const blob = encoderBytes.subarray(12 + declLen + vocabLen);
   const layout = declaration.layout || {};
+  // L, Q and B first: expectedBlobBytes() below scales with them, so they
+  // must be bounded before they size anything.
+  kernelLayers(layout);
+  kernelWeightBits(layout);
   for (const key of Object.keys(KERNEL_LAYOUT)) {
+    if (key === 'L' || key === 'B') continue;
     if (layout[key] !== KERNEL_LAYOUT[key]) {
       throw new Error(`.pikelet inline-encoder declares layout ${key}=${layout[key]}; `
         + `the compiled kernel requires ${key}=${KERNEL_LAYOUT[key]}`);
@@ -80,6 +117,16 @@ export async function createInlineTransformerEmbedder({ declaration, vocabText,
   if (vocabSize > KERNEL_LAYOUT.V) {
     throw new Error(`inline-encoder vocab has ${vocabSize} entries; the kernel's embedding table holds ${KERNEL_LAYOUT.V}`);
   }
+  // Like maxTokens below, checked here too because this function is also
+  // called directly, without parseInlineTransformerEncoder(): the kernel
+  // walks the blob for exactly this many layers in this weight format.
+  const layers = kernelLayers({ L: declaration.layout?.L ?? KERNEL_LAYOUT.L });
+  const blockSize = declaration.layout?.B ?? KERNEL_LAYOUT.B;
+  const bits = kernelWeightBits({ Q: declaration.layout?.Q, B: blockSize });
+  const blobBytes = expectedBlobBytes({ ...KERNEL_LAYOUT, L: layers, Q: bits, B: blockSize });
+  if (blob.length !== blobBytes) {
+    throw new Error(`inline-encoder blob is ${blob.length} bytes but ${layers} layers of u${bits} weights imply ${blobBytes}`);
+  }
   const tokenizer = createWordPiece(vocabText);
   const EM = await createEncoder();
   const dim = declaration.dim;
@@ -121,7 +168,7 @@ export async function createInlineTransformerEmbedder({ declaration, vocabText,
         throw new Error(`inline encoder window is ${tokenIds.length} tokens for a ${maxSeq}-token buffer`);
       }
       new Int32Array(EM.HEAP32.buffer, idsPtr, tokenIds.length).set(tokenIds);
-      const rc = EM._encoder_forward(blobPtr, idsPtr, tokenIds.length, hiddenPtr, 0);
+      const rc = EM._encoder_forward(blobPtr, idsPtr, tokenIds.length, hiddenPtr, 0, layers, bits);
       if (rc !== tokenIds.length) throw new Error(`inline encoder failed: ${rc}`);
       const hidden = new Float32Array(EM.HEAPF32.buffer, hiddenPtr, tokenIds.length * dim);
       if (usesClsPooling) {
@@ -181,7 +228,7 @@ export async function createInlineTransformerEmbedder({ declaration, vocabText,
         throw new Error(`inline encoder window is ${tokenIds.length} tokens for a ${maxSeq}-token buffer`);
       }
       new Int32Array(EM.HEAP32.buffer, idsPtr, tokenIds.length).set(tokenIds);
-      const rc = EM._encoder_forward(blobPtr, idsPtr, tokenIds.length, hiddenPtr, 0);
+      const rc = EM._encoder_forward(blobPtr, idsPtr, tokenIds.length, hiddenPtr, 0, layers, bits);
       if (rc !== tokenIds.length) throw new Error(`inline encoder failed: ${rc}`);
       const hidden = new Float32Array(EM.HEAPF32.buffer, hiddenPtr, tokenIds.length * dim);
       // Offset by 1: tokenIds[0] is [CLS], so idSlice's tokens start at

package/complete/lexical.mjs

@@ -181,6 +181,20 @@ export async function openLexicalIndexLazy(read, segLength) {
                 return e && { ...e, term: t };
             }))).filter(Boolean);
             if (!found.length) return [];
+            // Term entries arrive by lazy read, unverified (see the stance
+            // note above), so each one is held to the segment before its
+            // postings are fetched: inside the postings region (spec 3.8, as
+            // the eager reader enforces), a df no larger than the corpus, and
+            // a length a df-entry list can actually occupy (a doc-gap and a
+            // tf varint, each at most 6 bytes). Without these, one forged
+            // entry could steer a read anywhere in the file or ask for
+            // hundreds of MiB per query term.
+            for (const e of found) {
+                if (!Number.isSafeInteger(e.postingsRel) || e.postingsRel + e.postingsLen > postingsBytes
+                    || e.df > docCount || e.postingsLen > 12 * e.df) {
+                    throw new Error('.pikelet lexical postings out of bounds');
+                }
+            }
             const postings = await Promise.all(found.map((e) => read(postingsOffset + e.postingsRel, e.postingsLen)));
             const scores = new Map();
             found.forEach((e, i) => {

package/complete/retrieval-abstention.mjs

@@ -4,6 +4,12 @@
 // bloom filter, standardized and passed through the fitted logistic model.
 // Verdict semantics: 'answer' (strong match), 'weak' (closest match is
 // distant — shown with a caveat), 'abstain' (nothing useful in the pack).
+// What "match" means is the asset's: pikelet's self-templates-v6 assets
+// (calibration.target 'relevance') are fit against off-topic queries, so
+// 'abstain' means the pack does not cover the query's topic and 'answer'
+// means it does — not that a passage states the asked-for fact. Earlier
+// assets were fit against in-domain-unanswerable queries instead. The
+// scoring math here is the same for both.
 //
 // Assets calibrated by pikelet's self-templates-v2+ may carry an additional
 // grounding term (asset.coverage): the fraction of the query's content
@@ -34,11 +40,25 @@
 // one — there is no scenario where re-adding this via a config flag was
 // going to end differently a third time.
 
+// Upper bound on asset.coverage.topK. calibrate.mjs ships 5; every passage
+// counted here is a record hydrated on every query, so a pack must not be
+// able to turn one query into a hydration of its whole rerank window.
+export const MAX_COVERAGE_TOP_K = 10;
+
 export function createAbstentionScorer(asset, bloomBytes) {
     if (!asset || !Array.isArray(asset.weights) || !asset.thresholds) return null;
+    // Thresholds that are missing, non-numeric or out of order compare false
+    // against every p and would answer everything; a malformed asset must
+    // degrade to unscored instead (callers map a null scorer to it).
+    const { hard, weak } = asset.thresholds;
+    if (!Number.isFinite(hard) || !Number.isFinite(weak) || hard < 0 || hard > weak || weak > 1) return null;
     const coverageCfg = asset.coverage && typeof asset.coverage.weight === 'number'
         && Number.isFinite(asset.coverage.mean) && Number.isFinite(asset.coverage.std)
         ? asset.coverage : null;
+    if (coverageCfg && coverageCfg.topK !== undefined
+        && !(Number.isInteger(coverageCfg.topK) && coverageCfg.topK >= 1 && coverageCfg.topK <= MAX_COVERAGE_TOP_K)) {
+        return null;
+    }
     const coverageStopwords = coverageCfg ? new Set(coverageCfg.stopwords || []) : null;
     const coverageMinLen = coverageCfg ? (coverageCfg.minWordLen || 3) : 3;
     // Words the corpus uses everywhere ground any query that mentions them,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pikelet-wasm",
-  "version": "0.9.2",
+  "version": "0.10.0",
   "description": "Vector search engine and .pikelet artifact readers/builders, in WebAssembly. Compile a corpus into one self-contained, hash-verified search file, or use the underlying HNSW engine directly. Runs in Node.js, browsers, and Cloudflare Workers with no native dependencies.",
   "main": "src/pikelet.js",
   "types": "src/pikelet.d.ts",

package/src/artifact/pikelet-artifact-sketch.js

@@ -37,6 +37,12 @@ const {
 const SKETCH_MAGIC = 0x31415350; // PSA1
 const SKETCH_HEADER_BYTES = 256;
 const SKETCH_KIND_U8 = 1;
+// Ceiling on the header's recommendedRerank as a reader default. The field
+// is a producer hint, and a hostile (or exhaustive-fallback) value up to the
+// row count would make every query fetch, hash and rerank the whole vectors
+// region — at wiki scale, minutes per query with every check passing. An
+// explicit options.rerank is the caller's own choice and is not capped here.
+const MAX_RECOMMENDED_RERANK = 4096;
 
 // The affine table feeds every rerank distance; a NaN/Infinity entry makes
 // that row's distance non-finite and lets it sit in the result set
@@ -331,6 +337,28 @@ class PikeletSketchArtifact {
         if (header.byteLength !== SKETCH_HEADER_BYTES) {
             throw pikeletError(PIKELET_ERROR_CODES.SNAPSHOT_INVALID, 'Sketch artifact header is truncated');
         }
+        // options.expectedHeaderSha256 (hex): a container that commits to this
+        // header (the complete profile's manifest.index.headerSha256) checks
+        // it HERE, against the very bytes parsed below. Checking a separate
+        // read instead would let a host serve the genuine header to that read
+        // and a rewritten one (with a self-consistent residentSha256 and page
+        // table) to this one, replacing the index under a pinned identity.
+        if (options.expectedHeaderSha256 !== undefined) {
+            const expected = options.expectedHeaderSha256;
+            if (typeof expected !== 'string' || !/^[0-9a-f]{64}$/.test(expected)) {
+                throw pikeletError(PIKELET_ERROR_CODES.INVALID_ARGUMENT, 'SketchArtifact.open() expectedHeaderSha256 must be 64 lowercase hex characters');
+            }
+            const digest = await sha256BytesAsync(header);
+            if (!digest) {
+                throw pikeletError(PIKELET_ERROR_CODES.SNAPSHOT_INVALID,
+                    'Sketch artifact header verification requested but no crypto backend is available');
+            }
+            let hex = '';
+            for (const b of digest) hex += b.toString(16).padStart(2, '0');
+            if (hex !== expected) {
+                throw pikeletError(PIKELET_ERROR_CODES.SNAPSHOT_INVALID, '.pikelet index header failed hash verification against the manifest');
+            }
+        }
         const view = new DataView(header.buffer, header.byteOffset, header.byteLength);
         if (view.getUint32(0, true) !== SKETCH_MAGIC) {
             throw pikeletError(PIKELET_ERROR_CODES.SNAPSHOT_INVALID, 'Not a Pikelet sketch artifact (bad magic)');
@@ -353,7 +381,8 @@ class PikeletSketchArtifact {
         const sketchesOffset = view.getUint32(40, true);
         artifact.vectorsOffset = view.getUint32(44, true);
         const fileBytes = Number(view.getBigUint64(48, true));
-        artifact.recommendedRerank = view.getUint32(120, true);
+        artifact.declaredRecommendedRerank = view.getUint32(120, true);
+        artifact.recommendedRerank = Math.min(artifact.declaredRecommendedRerank, MAX_RECOMMENDED_RERANK);
         // Retained so the lazy tier stays verifiable after open — see
         // verifyVectors(). Copied out of the header read buffer.
         artifact.vectorsSha256 = new Uint8Array(header.subarray(88, 120));
@@ -1131,6 +1160,7 @@ async function createSketchScanner(loadEngine, artifact, options = {}) {
 
 module.exports = {
     PikeletSketchArtifact,
+    MAX_RECOMMENDED_RERANK,
     createSketchScanner,
     buildSketchArtifact,
     buildSketchArtifactBytes,

package/src/artifact/pikelet-artifact.d.ts

@@ -105,5 +105,12 @@ export declare function exportSketchArtifact(
   options?: SketchArtifactBuildOptions
 ): SketchArtifactBuildManifest;
 
+/**
+ * Ceiling on the header's recommendedRerank that readers use as their default
+ * rerank depth. Producers should not declare more: a reader will not honor it
+ * unless the caller passes an explicit rerank.
+ */
+export declare const MAX_RECOMMENDED_RERANK: number;
+
 /** Parse and validate an untrusted uint8 engine snapshot (fails closed). */
 export declare function parseUint8Snapshot(bytes: Uint8Array | ArrayBufferLike): Uint8SnapshotGraph;

package/src/artifact/pikelet-artifact.js

@@ -12,13 +12,14 @@
 const { NodeFileRangeSource, parseUint8Snapshot } = require('./pikelet-artifact-common.js');
 const { PikeletRangeArtifact, buildRangeArtifact, buildRangeArtifactFile } = require('./pikelet-artifact-range.js');
 const {
-    PikeletSketchArtifact, createSketchScanner,
+    PikeletSketchArtifact, MAX_RECOMMENDED_RERANK, createSketchScanner,
     buildSketchArtifact, buildSketchArtifactBytes, buildSketchArtifactFile, exportSketchArtifact,
 } = require('./pikelet-artifact-sketch.js');
 
 module.exports = {
     PikeletRangeArtifact,
     PikeletSketchArtifact,
+    MAX_RECOMMENDED_RERANK,
     createSketchScanner,
     NodeFileRangeSource,
     buildRangeArtifact,