pikelet-wasm 0.12.0 → 0.13.0

package/CHANGELOG.md

@@ -8,6 +8,84 @@ Through 0.6.0 these were published as `pancake-wasm` and
 packages are pre-1.0: a minor bump may carry breaking changes, and each entry lists them
 first.
 
+## pikelet-wasm 0.13.0 / pikelet 0.13.0 — 2026-11-01
+
+Mounting a pack over HTTP is much cheaper the second time: `pikelet mcp`
+now keeps the two largest mount reads of a URL pack on disk, so a fresh
+process mounting the 649 MiB Wikipedia pack from R2 transfers 9.1 MiB
+instead of 80.1 MiB. A slow remote mount no longer leaves an MCP client
+with no tools. `compile` and `append` now name what they drop instead of
+dropping it silently, and a layered pack keeps its name over MCP. Upgrade
+if you mount packs by URL or update them with `append`. Packs compiled
+before this release still verify and serve, and compiling the same inputs
+gives the same identities as 0.12.0. Both packages move to 0.13.0 together,
+and `pikelet` requires `pikelet-wasm` ^0.13.0.
+
+### Compatibility
+
+- **`pikelet mcp` writes a cache for URL packs.** The resident index prefix
+  and the query encoder of each URL pack mounted (about 71 MiB for the
+  Wikipedia pack) are stored in `~/.cache/pikelet/blobs`, up to 2 GiB,
+  evicting least recently used entries first. `$PIKELET_CACHE_DIR` or
+  `$XDG_CACHE_HOME` moves it, and `--no-cache` turns it off. Local packs
+  never use it.
+- **A layered pack mounts under its pack's name.** A chain is named like a
+  single pack: from the shelf entry, else the base's manifest name, else
+  the file name with the `.0002` layer number dropped. A chain mounted from
+  `docs.0002.pikelet` was served as `docs.0002`; it is now `docs`, the
+  name its base alone mounted under. Tool calls that name the old pack
+  need updating. A single pack keeps a four-digit suffix (`wiki.2025`).
+- **A missing calibration questions file is an error.** A
+  `--calibration-queries` file (or `runtime.calibrationQueries`) that does
+  not exist used to calibrate without the questions and say nothing; it
+  now fails, naming the path.
+- **`tools/list` can answer before the packs have mounted.** After 2
+  seconds it answers with the same tools described without pack names,
+  uncached, and sends `notifications/tools/list_changed` once the mounts
+  settle; `initialize` and `server/discover` advertise
+  `tools.listChanged`. Tool calls still wait for the mounts.
+
+### Added
+
+- **On-disk cache for URL packs (`pikelet-wasm`).** `openPikeletFile`
+  takes `blobCache`, a store keyed by sha256. It holds the sketch's
+  resident prefix and the query-interp segment that carries the encoder.
+  Every hit is hashed against its key, so a corrupt entry is a miss that
+  is deleted and refetched, and cache faults never fail an open.
+  `diskBlobCache()` and `defaultBlobCacheDir()` (Node, from
+  `pikelet-wasm/complete`) provide one on disk, and `pikelet mcp` uses it
+  for URL packs. Measured against the Wikipedia pack on R2, a fresh
+  process went from 80.1 MiB and 11.5 s to its first answer down to
+  9.1 MiB and 6.9 s. Per-query reads are not cached.
+- **A chain reports its pack's name.** `openPikeletChain(...).info().name`
+  is the base's name.
+
+### Fixed
+
+- **A slow remote mount left an MCP client with no tools.** `tools/list`
+  waited for every mount; when a URL mount stalled (6.3 MiB in 37 s from
+  R2), Claude Code gave up on the list and started the session with no
+  tools at all. See Compatibility for how it answers now.
+- **A calibration question naming no record was dropped silently.** Each
+  is now logged by name with the reason, and with the title it probably
+  meant when Markdown formatting is the difference (titles are matched
+  after formatting is stripped). An `expectId` outside the corpus is named
+  the same way.
+- **`append` dropped short pages without saying which.** Documents under
+  25 whitespace-separated tokens are skipped whole; `compile` and `append`
+  now log them with their token counts, and `append`'s refusal when
+  nothing is left names them.
+- **`append`'s closing hint was not a usable next step.** It suggested
+  `openPikeletChain([...])`, which takes a query vector, not text; it now
+  prints `pikelet mcp --pack <layer>#<identity>`.
+
+### Documentation
+
+- **Deleting is not redacting.** `append --remove` and `--supersede` take a
+  record out of search, but earlier files keep its bytes and `get_record`
+  serves it by id, marked `tombstoned`, so old citations resolve.
+  `compact` writes a base without it. Both READMEs say so.
+
 ## pikelet-wasm 0.12.0 / pikelet 0.12.0 — 2026-10-30
 
 Query embedding is about 1.65x faster, because the inline encoder now

package/README.md

@@ -88,7 +88,7 @@ npx pikelet append --parent docs.0001.pikelet --remove 412 --out docs.0002.pikel
 npx pikelet mcp --pack docs.0002.pikelet
 ```
 
-On a 20-record test pack with a 24.6 MiB base, deleting a record took a 1.2 KiB layer and replacing one took 6.9 KiB. Mounting the newest layer finds and verifies the rest of the chain, locally or over HTTP, and `#<sha256>` on it pins the whole history. Layers inherit the base's calibration until the chain drifts past 20% of the base's records; then it serves `unscored` until `compact` folds it into a fresh base. The format is specified in [`LAYERED_PROFILE.md`](LAYERED_PROFILE.md).
+On a 20-record test pack with a 24.6 MiB base, deleting a record took a 1.2 KiB layer and replacing one took 6.9 KiB. Deleting hides a record from search but does not erase it: the older files still hold its bytes, and `get_record` still serves it by id, marked `tombstoned`, so old citations resolve. `compact` writes a new base without it. Mounting the newest layer finds and verifies the rest of the chain, locally or over HTTP, and `#<sha256>` on it pins the whole history. Layers inherit the base's calibration until the chain drifts past 20% of the base's records; then it serves `unscored` until `compact` folds it into a fresh base. The format is specified in [`LAYERED_PROFILE.md`](LAYERED_PROFILE.md).
 
 ## Match quality
 

package/complete/blob-cache.node.mjs

@@ -0,0 +1,111 @@
+// A disk store for openPikeletFile's options.blobCache: whole regions a pack
+// commits to by sha256 (the sketch's resident prefix, the query-interp
+// segment holding the encoder), filed under that digest. Those regions are
+// most of what a URL mount downloads (about 71 of 77 MiB on the 649 MiB
+// Simple English Wikipedia pack), so a second process mounting the same pack
+// reads them from disk instead of the network.
+//
+// The store is untrusted by design: the reader hashes every hit against the
+// digest it asked for and treats a mismatch as a miss (blobCacheGet in
+// pikelet-artifact-common.js), so a torn write or a corrupted file costs one
+// refetch, never a wrong byte. Entries are immutable and never go stale; the
+// only policy is a size budget, enforced by evicting least recently used
+// entries (by mtime, touched on every hit) after each write.
+
+import fs from 'node:fs/promises';
+import os from 'node:os';
+import path from 'node:path';
+
+const KEY = /^[0-9a-f]{64}$/;
+export const DEFAULT_BLOB_CACHE_BYTES = 2 * 1024 * 1024 * 1024;
+
+// $PIKELET_CACHE_DIR, else $XDG_CACHE_HOME/pikelet/blobs, else
+// ~/.cache/pikelet/blobs: the same root the CLI already caches encoder
+// weights under.
+export function defaultBlobCacheDir(env = process.env) {
+    if (env.PIKELET_CACHE_DIR) return path.resolve(env.PIKELET_CACHE_DIR, 'blobs');
+    const base = env.XDG_CACHE_HOME ? path.resolve(env.XDG_CACHE_HOME) : path.join(os.homedir(), '.cache');
+    return path.join(base, 'pikelet', 'blobs');
+}
+
+export function diskBlobCache(dir = defaultBlobCacheDir(), { maxBytes = DEFAULT_BLOB_CACHE_BYTES } = {}) {
+    if (!Number.isSafeInteger(maxBytes) || maxBytes < 0) {
+        throw new Error('diskBlobCache maxBytes must be a non-negative integer');
+    }
+    const root = path.resolve(dir);
+    const fileOf = (key) => {
+        if (!KEY.test(key)) throw new Error(`blob cache key must be a lowercase sha256 hex digest, got ${JSON.stringify(key)}`);
+        return path.join(root, key.slice(0, 2), key);
+    };
+    const pending = new Set();
+    const track = (promise) => {
+        pending.add(promise);
+        promise.finally(() => pending.delete(promise)).catch(() => {});
+        return promise;
+    };
+
+    const evict = async (keep) => {
+        const entries = [];
+        let total = 0;
+        for (const shard of await fs.readdir(root).catch(() => [])) {
+            const shardDir = path.join(root, shard);
+            for (const name of await fs.readdir(shardDir).catch(() => [])) {
+                if (!KEY.test(name)) continue;
+                const file = path.join(shardDir, name);
+                const st = await fs.stat(file).catch(() => null);
+                if (!st) continue;
+                total += st.size;
+                entries.push({ file, size: st.size, mtime: st.mtimeMs });
+            }
+        }
+        entries.sort((a, b) => a.mtime - b.mtime);
+        for (const entry of entries) {
+            if (total <= maxBytes) break;
+            if (entry.file === keep) continue;
+            await fs.rm(entry.file, { force: true });
+            total -= entry.size;
+        }
+    };
+
+    return {
+        dir: root,
+        async get(key) {
+            const file = fileOf(key);
+            let bytes;
+            try { bytes = await fs.readFile(file); } catch (err) {
+                if (err?.code === 'ENOENT') return null;
+                throw err;
+            }
+            // Touch for LRU; best effort.
+            const now = new Date();
+            fs.utimes(file, now, now).catch(() => {});
+            return new Uint8Array(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+        },
+        put(key, bytes) {
+            const file = fileOf(key);
+            return track((async () => {
+                if (bytes.byteLength > maxBytes) return;
+                await fs.mkdir(path.dirname(file), { recursive: true });
+                // Write-then-rename, so a reader in another process sees
+                // either no entry or a whole one.
+                const tmp = `${file}.${process.pid}.${Math.random().toString(36).slice(2)}.tmp`;
+                try {
+                    await fs.writeFile(tmp, bytes);
+                    await fs.rename(tmp, file);
+                } catch (err) {
+                    await fs.rm(tmp, { force: true });
+                    throw err;
+                }
+                await evict(file);
+            })());
+        },
+        async delete(key) {
+            await fs.rm(fileOf(key), { force: true });
+        },
+        // Settles every write started so far: for tests, and for a process
+        // that wants its cache populated before it exits.
+        async flush() {
+            while (pending.size) await Promise.allSettled([...pending]);
+        },
+    };
+}

package/complete/index.d.ts

@@ -96,6 +96,26 @@ export declare function nodeFetch(input: string, init?: {
   inflateGzip?: boolean;
 }): Promise<any>;
 
+/** openPikeletFile's options.blobCache: bytes keyed by their sha256 (lowercase hex). */
+export interface BlobCache {
+  get(key: string): Promise<Uint8Array | null> | Uint8Array | null;
+  put(key: string, bytes: Uint8Array): Promise<void> | void;
+  delete?(key: string): Promise<void> | void;
+}
+
+/**
+ * Node only (the 'node' export condition): a BlobCache on disk, one file per
+ * key under `dir` (default defaultBlobCacheDir()), evicting least recently
+ * used entries past maxBytes (default 2 GiB). flush() settles pending writes.
+ */
+export declare function diskBlobCache(dir?: string, options?: { maxBytes?: number }): BlobCache & {
+  dir: string;
+  flush(): Promise<void>;
+};
+/** $PIKELET_CACHE_DIR/blobs, else $XDG_CACHE_HOME/pikelet/blobs, else ~/.cache/pikelet/blobs. */
+export declare function defaultBlobCacheDir(env?: Record<string, string | undefined>): string;
+export declare const DEFAULT_BLOB_CACHE_BYTES: number;
+
 export interface CompleteQueryResult {
   /**
    * The pack's calibrated verdict on this retrieval. On packs calibrated
@@ -287,6 +307,17 @@ export declare function openPikeletFile(
      * true.
      */
     prefetchEncoder?: boolean;
+    /**
+     * A store for the regions a pack commits to by sha256 that dominate a
+     * cold URL mount: the sketch's resident prefix and the query-interp
+     * segment holding the inline encoder. Keys are those digests (lowercase
+     * hex), so entries never go stale. Every hit is hashed against its key
+     * before use and a mismatch is treated as a miss (and deleted), so the
+     * store need not be trusted; get/put/delete faults never fail an open.
+     * diskBlobCache() (Node) is a ready implementation. Ignored by an open
+     * whose resident verification is off.
+     */
+    blobCache?: BlobCache;
     /**
      * Fully verify the index's lazy vector rows at open (one streamed pass
      * against the identity-anchored vectorsSha256). Default false: without

package/complete/index.mjs

@@ -27,6 +27,7 @@ import { createAbstentionScorer } from './retrieval-abstention.mjs';
 import { openLexicalIndex, openLexicalIndexLazy } from './lexical.mjs';
 import { parseRowMapSegment } from './rowmap.mjs';
 import { PikeletSketchArtifact } from '../src/artifact/pikelet-artifact.js';
+import { blobCacheGet, blobCachePut } from '../src/artifact/pikelet-artifact-common.js';
 import { MAGIC, HEADER_BYTES, TABLE_ENTRY_BYTES, KINDS, KIND_NAMES } from './format.mjs';
 
 import {
@@ -785,6 +786,7 @@ export async function openPikeletFile(input, options = {}) {
             // the same bytes the sketch parses (see expectedHeaderSha256).
             PikeletSketchArtifact.open(windowSource(source, idx.offset, idx.length), {
                 maxReadBytes,
+                ...(options.blobCache ? { blobCache: options.blobCache } : {}),
                 ...(perRecord ? { expectedHeaderSha256: manifest.index.headerSha256 } : {}),
             }),
             readChecked(source, corpus.offset, tablesBytes, 'corpus tables', maxReadBytes, fileBytes),
@@ -1004,9 +1006,16 @@ export async function openPikeletFile(input, options = {}) {
             const loadEmbedder = async () => {
                 let encBytes = encoderBytes;
                 if (qiDeferred) {
-                    const full = await readChecked(source, qi.offset, qi.length, 'query-interp segment', maxReadBytes, fileBytes);
-                    if (await sha256hex(full) !== qi.sha256) {
-                        throw new Error('.pikelet query-interp segment failed hash verification');
+                    // The ~25 MiB encoder is the other bulk of a cold mount;
+                    // options.blobCache serves it by the manifest's segment
+                    // digest (hash-checked on the way out, as a fetch is).
+                    const cached = await blobCacheGet(options.blobCache, qi.sha256, qi.length);
+                    const full = cached || await readChecked(source, qi.offset, qi.length, 'query-interp segment', maxReadBytes, fileBytes);
+                    if (!cached) {
+                        if (await sha256hex(full) !== qi.sha256) {
+                            throw new Error('.pikelet query-interp segment failed hash verification');
+                        }
+                        blobCachePut(options.blobCache, qi.sha256, full);
                     }
                     // The header and calibration slices used at open were
                     // unverified; the digest-verified segment must contain

package/complete/index.node.mjs

@@ -9,6 +9,7 @@ import { nodeFetch } from './node-fetch.mjs';
 
 export * from './index.mjs';
 export { nodeFetch };
+export { diskBlobCache, defaultBlobCacheDir, DEFAULT_BLOB_CACHE_BYTES } from './blob-cache.node.mjs';
 
 export function httpRangeSource(url, options = {}) {
     return portableHttpRangeSource(url, { fetch: nodeFetch, ...options });

package/complete/layer-reader.mjs

@@ -369,6 +369,10 @@ export async function openPikeletChain(members, options = {}) {
                     profile: LAYER_PROFILE,
                     identity: chainMembers[chainMembers.length - 1].identity,
                     baseIdentity: base.identity,
+                    // The pack's name is the base's: layers add records, not a
+                    // new name. Without it a host fell back to naming the
+                    // chain after its newest layer's file.
+                    name: opened[0].info().name ?? null,
                     layers: chainMembers.length,
                     searchTiers: searchTiers.length,
                     records: table.rowTotal,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pikelet-wasm",
-  "version": "0.12.0",
+  "version": "0.13.0",
   "description": "Vector search engine and .pikelet artifact readers/builders, in WebAssembly. Compile a corpus into one self-contained, hash-verified search file, or use the underlying HNSW engine directly. Runs in Node.js, browsers, and Cloudflare Workers with no native dependencies.",
   "main": "src/pikelet.js",
   "types": "src/pikelet.d.ts",

package/src/artifact/pikelet-artifact-common.js

@@ -462,6 +462,36 @@ async function sha256BytesAsync(bytes) {
     }
 }
 
+// Read-through cache for whole regions that carry a sha256 commitment (the
+// sketch's resident prefix, a pack's query-interp segment), keyed by that
+// digest. A hit is hashed exactly as a network read would be, so the cache
+// can only ever hand back the committed bytes: anything else it holds (a
+// partial write, disk corruption, a bug elsewhere) reads as a miss and the
+// entry is dropped. Because the key is the content, entries never go stale
+// and need no invalidation. Cache faults never fail a read.
+async function blobCacheGet(cache, digestHex, length) {
+    if (!cache || typeof cache.get !== 'function') return null;
+    let hit;
+    try {
+        hit = await cache.get(digestHex);
+        if (!hit) return null;
+        hit = asUint8Array(hit);
+    } catch { return null; }
+    const digest = hit.byteLength === length ? await sha256BytesAsync(hit) : null;
+    let hex = '';
+    if (digest) for (const b of digest) hex += b.toString(16).padStart(2, '0');
+    if (hex === digestHex) return hit;
+    try { await cache.delete?.(digestHex); } catch { /* a miss either way */ }
+    return null;
+}
+
+// Store verified bytes. Fire-and-forget: a slow or failing cache must not
+// hold up the open that just fetched them.
+function blobCachePut(cache, digestHex, bytes) {
+    if (!cache || typeof cache.put !== 'function') return;
+    Promise.resolve().then(() => cache.put(digestHex, bytes)).catch(() => {});
+}
+
 // Verify bytes against an expected 32-byte SHA-256 digest. Verification was
 // requested by the caller, so a missing crypto backend fails closed rather
 // than admitting unverified bytes.
@@ -556,6 +586,8 @@ module.exports = {
     parseUint8Snapshot,
     sha256Bytes,
     sha256BytesAsync,
+    blobCacheGet,
+    blobCachePut,
     verifySha256,
     verifySegmentSha256,
 };

package/src/artifact/pikelet-artifact-sketch.js

@@ -17,6 +17,8 @@ const {
     parseUint8Snapshot,
     sha256Bytes,
     sha256BytesAsync,
+    blobCacheGet,
+    blobCachePut,
     verifySegmentSha256,
 } = require('./pikelet-artifact-common.js');
 const {
@@ -28,6 +30,12 @@ const {
 // ============================================================================
 // Sketch artifact profile (.pikelet-sketch) — spec/SKETCH_PROFILE.md
 // ============================================================================
+
+function hexOf(bytes) {
+    let hex = '';
+    for (const b of bytes) hex += b.toString(16).padStart(2, '0');
+    return hex;
+}
 //
 // Layout: 256-byte header | scales f32[count] | offsets f32[count] |
 // packed pooled sketches | zero padding to 16-byte alignment | raw u8 rows.
@@ -467,8 +475,16 @@ class PikeletSketchArtifact {
         artifact.tier = 'full';
         artifact.fullyResident = Promise.resolve(artifact);
 
-        if (!staged) {
-            const resident = await readChecked(source, SKETCH_HEADER_BYTES, vectorsOffset - SKETCH_HEADER_BYTES, 'resident prefix', artifact.maxReadBytes);
+        // options.blobCache: the resident prefix is the bulk of a mount (46 MiB
+        // of a 649 MiB wiki pack) and is committed by residentSha256, so a
+        // cache keyed by that digest serves it to later processes. A hit is
+        // hash-checked inside blobCacheGet; only verified opens consult it,
+        // since verify:false has nothing to check a hit against. A cache hit
+        // also skips staging: the full tier is already local.
+        const cacheKey = verify && options.blobCache ? hexOf(header.subarray(56, 88)) : null;
+        const cached = cacheKey ? await blobCacheGet(options.blobCache, cacheKey, vectorsOffset - SKETCH_HEADER_BYTES) : null;
+        if (!staged || cached) {
+            const resident = cached || await readChecked(source, SKETCH_HEADER_BYTES, vectorsOffset - SKETCH_HEADER_BYTES, 'resident prefix', artifact.maxReadBytes);
             if (resident.byteLength !== vectorsOffset - SKETCH_HEADER_BYTES) {
                 throw pikeletError(PIKELET_ERROR_CODES.SNAPSHOT_INVALID, 'Sketch artifact resident prefix is truncated');
             }
@@ -477,7 +493,9 @@ class PikeletSketchArtifact {
             const residentCopy = new Uint8Array(resident.byteLength);
             residentCopy.set(resident);
             artifact._adoptResident(residentCopy, count, sketchRowBytes, microRowBytes, microOffset);
-            if (verify) await artifact._verifyFullResident(residentCopy, header);
+            if (cached) artifact.residentVerified = true;
+            else if (verify) await artifact._verifyFullResident(residentCopy, header);
+            if (cacheKey && !cached) blobCachePut(options.blobCache, cacheKey, residentCopy);
             return artifact;
         }