pikelet-wasm 0.9.3 → 0.10.0

package/CHANGELOG.md

@@ -8,6 +8,74 @@ Through 0.6.0 these were published as `pancake-wasm` and
 packages are pre-1.0: a minor bump may carry breaking changes, and each entry lists them
 first.
 
+## pikelet-wasm 0.10.0 / pikelet 0.10.0 — 2026-10-09
+
+`matchQuality` changes meaning for packs compiled with this release: it
+now measures retrieval relevance — whether the pack covers the
+question's topic — instead of trying to say whether the answer is there.
+Packs already built keep their behavior, since the calibration ships
+inside each file. The inline query encoder runs about 1.65× faster in
+Node.js and Chromium and about 2.2× in WebKit with identical embeddings,
+and readers can now run 12-layer and 4-bit encoders. Both packages move
+to 0.10.0 together, and `pikelet` requires `pikelet-wasm` ^0.10.0.
+
+### Compatibility
+
+- **Newly compiled packs score relevance.** `none` means the pack does
+  not cover the question's topic; a question about a covered topic whose
+  specific fact the pack lacks now comes back `strong` or `weak` instead
+  of `none`. Do not read `strong` as "answered": an asked-for value in a
+  result's `grounding.uncovered` is one no passage states. On the
+  repository's labeled questions, answerable questions called `none`
+  went from 12 of 44 to 0 (docs) and 26 of 50 to 0 (Veyra registry).
+  Rebuild a pack to pick this up; packs compiled by 0.9.x keep their
+  calibration.
+- **The calibration summary changes shape.** The evaluation segment's
+  `calibration` block gains `method` (`self-templates-v6`) and `target`
+  (`relevance`); `cvAuc` is now measured against off-topic queries;
+  `cvAucHard`, answerable vs. in-domain-unanswerable, is reported but no
+  longer gates the fit; `stage1FloorAuc` is gone. Corpora that shipped
+  `unscored` for lack of hard negatives or a low hard-negative AUC now
+  calibrate.
+- **MCP wording changes.** The `search` description, the note on a
+  `none` result, the server instructions and `verify_pack`'s calibration
+  note describe relevance; `verify_pack`'s calibration block gains
+  `method` and `target` (null on older packs). Tool names and result
+  shapes are unchanged.
+- **Packs that declare a layer count other than 6, or 4-bit weights,
+  open only in 0.10.0 readers**; 0.9.x readers refuse them at parse. No
+  published pack declares either.
+
+### Added
+
+- **Inline encoders with 1 to 12 layers and u4 weights.** A kind-3
+  declaration may give `layout.L` from 1 to 12 and `layout.Q: 4` with
+  `layout.B: 32` (u4 weights, two per byte, an f32 scale and offset per
+  block of 32); `Q` absent means u8 in blocks of 64, as before. Exported
+  as `KERNEL_MAX_LAYERS`. The parser and the embedder check the layer
+  count, the format and the blob length before the kernel runs. A u4
+  BGE-small-en-v1.5 blob is 23.9 MiB against MiniLM's 24.3 and matches
+  PyTorch at cosine 0.985 (1.000 against the same weights fake-quantized).
+  `compile` has no flag for these layouts yet.
+
+### Changed
+
+- **The inline encoder runs its projections over tiles of four tokens**
+  instead of one token at a time, so each layer's weights stream through
+  memory once per tile. Median query encoding, before → after: about
+  23 → 14 ms for a 10-word question and 213 → 125 ms for 100 words in
+  Chromium; 32 → 14 and 285 → 134 ms in WebKit. Embeddings are
+  bit-identical, so existing packs and their verification vectors are
+  unaffected.
+- **Calibration** (`compile`, `compact` refits) fits verified on-topic
+  probes against off-topic ones — the built-in foreign bank, minus
+  entries the corpus turns out to cover, plus gibberish — and places
+  `none` at the 95th percentile of the off-topic scores. The
+  in-domain-unanswerable probe classes are still generated, for the
+  `cvAucHard` diagnostic only.
+- **Documentation** (both READMEs, `docs/how-a-query-runs.md`,
+  `docs/veyra-ablation.md`) describes `matchQuality` as relevance.
+
 ## pikelet-wasm 0.9.3 / pikelet 0.9.3 — 2026-10-06
 
 Hardening for readers that mount packs from other publishers or hosts:

package/README.md

@@ -133,7 +133,7 @@ claude \
   --mcp-config examples/one-file-search/web/public/veyra.mcp.json
 ```
 
-All three packs derive from the same small synthetic Station Veyra corpus, with controlled differences in the Tovash evidence. Ask where the Tovash project is housed: the full pack supports Chamber 17, the modified pack supports Chamber 43, and the ablated pack should abstain.
+All three packs derive from the same small synthetic Station Veyra corpus, with controlled differences in the Tovash evidence. Ask where the Tovash project is housed: the full pack supports Chamber 17, the modified pack supports Chamber 43, and the ablated pack should not support any chamber.
 
 ---
 
@@ -153,7 +153,7 @@ A layer records its parent's identity, and its location when the parent sits in
 
 ## Match quality and abstention
 
-A nearest neighbour is not automatically evidence that a corpus answers a question. At build time Pikelet can fit a classifier over retrieval signals — distances, and how much of the question the top passages actually contain ([details](docs/how-a-query-runs.md#how-abstention-is-decided)). When the corpus supports a reliable classifier, results carry `matchQuality: strong | weak | none`. When calibration can't separate supported from unsupported reliably — a single novel may be semantically homogeneous enough that the fit isn't trustworthy — Pikelet reports `matchQuality: unscored` and records why calibration was skipped, rather than manufacturing confidence. In the library, a `none` verdict withholds `results` by default; pass `query(text, { showAbstained: true })` to see the raw retrieval anyway — `matchQuality` and `confidence` are unaffected either way. MCP `search` does the opposite: it returns results under `none` with a note telling the model the support is indirect, because the calibrator can misjudge a paraphrase; `showAbstained: false` withholds them.
+A nearest neighbour is not automatically evidence that a corpus covers a question. At build time Pikelet fits a classifier over retrieval signals — distances, and how much of the question the top passages actually contain — against queries the corpus does not cover ([details](docs/how-a-query-runs.md#how-abstention-is-decided)), and results carry `matchQuality: strong | weak | none`. It measures retrieval relevance: `none` means the pack does not cover the question's topic; `strong` means it does, not that a returned passage states the specific fact asked for. For that, each result lists the question's words no top passage contains (`grounding.uncovered`), and an asked-for value in that list is unsupported whatever `matchQuality` says. When calibration can't separate on-topic from off-topic queries reliably, Pikelet reports `matchQuality: unscored` and records why calibration was skipped, rather than manufacturing confidence. In the library, a `none` verdict withholds `results` by default; pass `query(text, { showAbstained: true })` to see the raw retrieval anyway — `matchQuality` and `confidence` are unaffected either way. MCP `search` does the opposite: it returns results under `none` with a note telling the model the support is indirect, because the calibrator can misjudge a paraphrase; `showAbstained: false` withholds them.
 
 `matchQuality` is evidence about retrieval support. It is **not** a guarantee that an LLM will never hallucinate.
 

package/complete/index.d.ts

@@ -67,6 +67,14 @@ export declare function httpRangeSource(url: string, options?: {
 }): HttpRangeSource;
 
 export interface CompleteQueryResult {
+  /**
+   * The pack's calibrated verdict on this retrieval. On packs calibrated
+   * by pikelet 0.10 or later (evaluation calibration.target 'relevance')
+   * it measures retrieval relevance: 'none' means the pack does not appear
+   * to cover the query's topic, 'strong' that it does — not that a
+   * returned passage states the specific fact asked for. 'unscored': the
+   * pack ships no fitted calibrator.
+   */
   matchQuality: 'strong' | 'weak' | 'none' | 'unscored';
   confidence?: number;
   /**
@@ -297,6 +305,8 @@ export declare function verifyHostEncoder(
 
 /** Kernel layout compiled into the kind-3 encoder (mirrors encoder.cpp). */
 export declare const KERNEL_LAYOUT: Record<'V' | 'P' | 'T' | 'D' | 'F' | 'L' | 'B' | 'H', number>;
+/** Largest layer count (layout L) the kind-3 kernel runs; L may be 1 to this. */
+export declare const KERNEL_MAX_LAYERS: number;
 export declare function expectedBlobBytes(layout?: Record<string, number>): number;
 export declare function parseInlineTransformerEncoder(encoderBytes: Uint8Array): {
   declaration: Record<string, unknown>;

package/complete/index.mjs

@@ -29,7 +29,7 @@ import { PikeletSketchArtifact } from '../src/artifact/pikelet-artifact.js';
 import { MAGIC, HEADER_BYTES, TABLE_ENTRY_BYTES, KINDS, KIND_NAMES } from './format.mjs';
 
 import {
-    KERNEL_LAYOUT, expectedBlobBytes, parseInlineTransformerEncoder,
+    KERNEL_LAYOUT, KERNEL_MAX_LAYERS, expectedBlobBytes, parseInlineTransformerEncoder,
     createInlineTransformerEmbedder, INLINE_TEST_VECTOR_TEXTS,
     buildInlineTestVectors, verifyInlineTestVectors,
     validateExpectedEmbedding, validateTestVectorTolerance,
@@ -44,7 +44,7 @@ export { openLexicalIndex } from './lexical.mjs';
 export { fuseCandidates, FUSION_DEFAULTS } from './fusion.mjs';
 import { fuseCandidates } from './fusion.mjs';
 export {
-    KERNEL_LAYOUT, expectedBlobBytes, parseInlineTransformerEncoder,
+    KERNEL_LAYOUT, KERNEL_MAX_LAYERS, expectedBlobBytes, parseInlineTransformerEncoder,
     createInlineTransformerEmbedder, INLINE_TEST_VECTOR_TEXTS,
     buildInlineTestVectors, verifyInlineTestVectors,
     validateExpectedEmbedding, validateTestVectorTolerance,

package/complete/inline-transformer.mjs

@@ -5,8 +5,13 @@ const decoder = new TextDecoder();
 // The wasm kernel compiles these in (encoder-spike/encoder.cpp `constexpr`
 // block) and walks the weight blob with a bare running cursor — it never
 // reads the declaration. Declaration/blob skew is therefore only catchable
-// here, host-side, before the first forward.
+// here, host-side, before the first forward. Two values are exceptions,
+// taken by the kernel per call: the layer count, any L from 1 to
+// KERNEL_MAX_LAYERS (KERNEL_LAYOUT.L is MiniLM-L6's, the default), and the
+// weight format, Q bits per weight in blocks of B — u8 in blocks of 64
+// (Q absent or 8, every pack before Q existed) or u4 in blocks of 32.
 export const KERNEL_LAYOUT = { V: 30522, P: 512, T: 2, D: 384, F: 1536, L: 6, B: 64, H: 12 };
+export const KERNEL_MAX_LAYERS = 12;
 const KERNEL_MAX_SEQ = 512;
 // Smallest window that still frames interior tokens between [CLS] and [SEP]
 // with room to make progress; see the maxTokens check in
@@ -14,11 +19,11 @@ const KERNEL_MAX_SEQ = 512;
 const MIN_DECLARED_MAX_TOKENS = 8;
 
 // Byte size the kernel's fill_layout() will consume for a given layout:
-// each quantized matrix is rows*cols u8 plus per-block f32 scales and
-// offsets; biases and layer norms are plain f32.
+// each quantized matrix is rows*cols weights of Q bits plus per-block f32
+// scales and offsets; biases and layer norms are plain f32.
 export function expectedBlobBytes(layout = KERNEL_LAYOUT) {
-  const { V, P, T, D, F, L, B } = layout;
-  const quant = (rows, cols) => rows * cols + rows * (cols / B) * 4 * 2;
+  const { V, P, T, D, F, L, B, Q = 8 } = layout;
+  const quant = (rows, cols) => (rows * cols * Q) / 8 + rows * (cols / B) * 4 * 2;
   const layer = 4 * quant(D, D) + 4 * (D * 4)   // wq,wk,wv,wo + their biases
     + 2 * (D * 4)                                // ln1 gamma/beta
     + quant(F, D) + F * 4                        // wu + bu
@@ -27,6 +32,33 @@ export function expectedBlobBytes(layout = KERNEL_LAYOUT) {
   return quant(V, D) + quant(P, D) + quant(T, D) + 2 * (D * 4) + L * layer;
 }
 
+// The declared layer count, which the kernel takes per call. Its
+// Layout struct holds KERNEL_MAX_LAYERS layers (encoder.cpp MAXL), and it
+// returns -3 outside 1..MAXL rather than walking past them.
+function kernelLayers(layout) {
+  const layers = layout.L;
+  if (!Number.isInteger(layers) || layers < 1 || layers > KERNEL_MAX_LAYERS) {
+    throw new Error(`.pikelet inline-encoder declares layout L=${layers}; `
+      + `the kernel runs 1 to ${KERNEL_MAX_LAYERS} layers`);
+  }
+  return layers;
+}
+
+// The declared weight format as the kernel's bits argument. The kernel's
+// block size is fixed per format, so B must be the one Q implies.
+const KERNEL_WEIGHT_FORMATS = { 8: 64, 4: 32 };
+function kernelWeightBits(layout) {
+  // Only an absent Q means u8 (packs from before Q existed); null is not
+  // absent, and expectedBlobBytes() would size it as zero-bit weights.
+  const bits = layout.Q === undefined ? 8 : layout.Q;
+  if (!Object.hasOwn(KERNEL_WEIGHT_FORMATS, bits) || typeof bits !== 'number'
+      || layout.B !== KERNEL_WEIGHT_FORMATS[bits]) {
+    throw new Error(`.pikelet inline-encoder declares layout Q=${layout.Q} B=${layout.B}; `
+      + 'the kernel runs u8 weights in blocks of 64 (Q absent or 8) or u4 in blocks of 32 (Q=4)');
+  }
+  return bits;
+}
+
 export function parseInlineTransformerEncoder(encoderBytes) {
   const view = new DataView(encoderBytes.buffer, encoderBytes.byteOffset, encoderBytes.byteLength);
   const declLen = view.getUint32(0, true);
@@ -38,7 +70,12 @@ export function parseInlineTransformerEncoder(encoderBytes) {
   const declaration = JSON.parse(decoder.decode(encoderBytes.subarray(12, 12 + declLen)));
   const blob = encoderBytes.subarray(12 + declLen + vocabLen);
   const layout = declaration.layout || {};
+  // L, Q and B first: expectedBlobBytes() below scales with them, so they
+  // must be bounded before they size anything.
+  kernelLayers(layout);
+  kernelWeightBits(layout);
   for (const key of Object.keys(KERNEL_LAYOUT)) {
+    if (key === 'L' || key === 'B') continue;
     if (layout[key] !== KERNEL_LAYOUT[key]) {
       throw new Error(`.pikelet inline-encoder declares layout ${key}=${layout[key]}; `
         + `the compiled kernel requires ${key}=${KERNEL_LAYOUT[key]}`);
@@ -80,6 +117,16 @@ export async function createInlineTransformerEmbedder({ declaration, vocabText,
   if (vocabSize > KERNEL_LAYOUT.V) {
     throw new Error(`inline-encoder vocab has ${vocabSize} entries; the kernel's embedding table holds ${KERNEL_LAYOUT.V}`);
   }
+  // Like maxTokens below, checked here too because this function is also
+  // called directly, without parseInlineTransformerEncoder(): the kernel
+  // walks the blob for exactly this many layers in this weight format.
+  const layers = kernelLayers({ L: declaration.layout?.L ?? KERNEL_LAYOUT.L });
+  const blockSize = declaration.layout?.B ?? KERNEL_LAYOUT.B;
+  const bits = kernelWeightBits({ Q: declaration.layout?.Q, B: blockSize });
+  const blobBytes = expectedBlobBytes({ ...KERNEL_LAYOUT, L: layers, Q: bits, B: blockSize });
+  if (blob.length !== blobBytes) {
+    throw new Error(`inline-encoder blob is ${blob.length} bytes but ${layers} layers of u${bits} weights imply ${blobBytes}`);
+  }
   const tokenizer = createWordPiece(vocabText);
   const EM = await createEncoder();
   const dim = declaration.dim;
@@ -121,7 +168,7 @@ export async function createInlineTransformerEmbedder({ declaration, vocabText,
         throw new Error(`inline encoder window is ${tokenIds.length} tokens for a ${maxSeq}-token buffer`);
       }
       new Int32Array(EM.HEAP32.buffer, idsPtr, tokenIds.length).set(tokenIds);
-      const rc = EM._encoder_forward(blobPtr, idsPtr, tokenIds.length, hiddenPtr, 0);
+      const rc = EM._encoder_forward(blobPtr, idsPtr, tokenIds.length, hiddenPtr, 0, layers, bits);
       if (rc !== tokenIds.length) throw new Error(`inline encoder failed: ${rc}`);
       const hidden = new Float32Array(EM.HEAPF32.buffer, hiddenPtr, tokenIds.length * dim);
       if (usesClsPooling) {
@@ -181,7 +228,7 @@ export async function createInlineTransformerEmbedder({ declaration, vocabText,
         throw new Error(`inline encoder window is ${tokenIds.length} tokens for a ${maxSeq}-token buffer`);
       }
       new Int32Array(EM.HEAP32.buffer, idsPtr, tokenIds.length).set(tokenIds);
-      const rc = EM._encoder_forward(blobPtr, idsPtr, tokenIds.length, hiddenPtr, 0);
+      const rc = EM._encoder_forward(blobPtr, idsPtr, tokenIds.length, hiddenPtr, 0, layers, bits);
       if (rc !== tokenIds.length) throw new Error(`inline encoder failed: ${rc}`);
       const hidden = new Float32Array(EM.HEAPF32.buffer, hiddenPtr, tokenIds.length * dim);
       // Offset by 1: tokenIds[0] is [CLS], so idSlice's tokens start at

package/complete/retrieval-abstention.mjs

@@ -4,6 +4,12 @@
 // bloom filter, standardized and passed through the fitted logistic model.
 // Verdict semantics: 'answer' (strong match), 'weak' (closest match is
 // distant — shown with a caveat), 'abstain' (nothing useful in the pack).
+// What "match" means is the asset's: pikelet's self-templates-v6 assets
+// (calibration.target 'relevance') are fit against off-topic queries, so
+// 'abstain' means the pack does not cover the query's topic and 'answer'
+// means it does — not that a passage states the asked-for fact. Earlier
+// assets were fit against in-domain-unanswerable queries instead. The
+// scoring math here is the same for both.
 //
 // Assets calibrated by pikelet's self-templates-v2+ may carry an additional
 // grounding term (asset.coverage): the fraction of the query's content

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pikelet-wasm",
-  "version": "0.9.3",
+  "version": "0.10.0",
   "description": "Vector search engine and .pikelet artifact readers/builders, in WebAssembly. Compile a corpus into one self-contained, hash-verified search file, or use the underlying HNSW engine directly. Runs in Node.js, browsers, and Cloudflare Workers with no native dependencies.",
   "main": "src/pikelet.js",
   "types": "src/pikelet.d.ts",