pikelet-wasm 0.13.0 → 0.14.0

package/CHANGELOG.md

@@ -8,6 +8,82 @@ Through 0.6.0 these were published as `pancake-wasm` and
 packages are pre-1.0: a minor bump may carry breaking changes, and each entry lists them
 first.
 
+## pikelet-wasm 0.14.0 / pikelet 0.14.0 — 2026-11-04
+
+Asking a URL pack a question an earlier process already asked is now
+mostly local too: `pikelet mcp` keeps the verified rows, records and digest
+pages a query reads in its disk cache, so on the Wikipedia pack over R2 a
+repeated question transferred 178 KiB instead of 1.3 MiB and answered in
+0.65 s instead of 1.45 s. The cache is now private to its owner (0.13.0
+left it readable by other local users under the usual umask), and two
+processes sharing it no longer mis-index each other's writes. Upgrade if
+you mount packs by URL, and especially if other people have accounts on
+the machine. Packs compiled before this release still verify and serve,
+and compiling the same inputs gives the same identities as 0.13.0. Both
+packages move to 0.14.0 together, and `pikelet` requires `pikelet-wasm`
+^0.14.0.
+
+### Compatibility
+
+- **The cache also holds query reads.** On top of the mount entries (up
+  to 2 GiB), URL packs now store what queries read in append-only logs
+  under `~/.cache/pikelet/blobs/small`, up to 64 MiB. `--no-cache` still
+  turns all of it off, and local packs still never use it.
+- **The cache root becomes 0700.** The first write tightens the store's
+  directory to owner-only, including a `$PIKELET_CACHE_DIR/blobs` or a
+  directory passed to `diskBlobCache(dir)` that was opened up on purpose.
+  Nothing above that directory is touched.
+- **Library callers: close a `diskBlobCache()` you are done with.** A store
+  that has served a query entry holds read handles on its log until
+  `close()`. Node 26 treats a file handle left to garbage collection as a
+  fatal error, so a store dropped without `close()` can end the process
+  there. The CLI keeps its store for the life of the process and is not
+  affected.
+
+### Added
+
+- **Queries use the disk cache too.** With `blobCache`, the verified pieces
+  a query reads are stored and served by digest: rerank vector rows (by
+  their row digests; format 2), corpus records (by their record digests),
+  and the digest pages that verify both (by their page-table hashes). Every
+  hit is hash-checked against its key, a truncated row digest of at least
+  128 bits included, and only reads verified on their own use the cache:
+  the lazy lexical index never does. `diskBlobCache()` keeps entries under
+  64 KiB in append-only log generations (64 MiB in total by default,
+  `smallMaxBytes`; the oldest generation dropped whole, hits rewritten to
+  the newest) instead of one file each, and gains `close()`. On the
+  Wikipedia pack over R2, asking a question an earlier process already
+  asked transferred 178 KiB in 11 requests instead of 1.3 MiB in 29, and
+  answered in 0.65 s instead of 1.45 s; a new question costs what it did.
+
+### Fixed
+
+- **Generated configs point at a schema that exists.** The configs the
+  CLI generates carried a `$schema` URL from before the repository moved
+  to `packages/`, which returned 404; it and the schema's own `$id` now name
+  `packages/pikelet/schemas/v1/pikelet.config.schema.json`. Configs
+  generated earlier keep the old URL until it is edited.
+- **Two processes sharing a disk cache no longer mis-index each other's
+  appends.** A store computed a log batch's offset as the file size after
+  its append, so another process appending in between left its index
+  pointing at the wrong record: about a third of each writer's own
+  lookups across three processes. Every hit is hash-checked, so this cost
+  refetches, never a wrong byte, and a fresh process re-indexed the log
+  correctly. A batch is now read back at its offset before it is indexed,
+  and if the end moved, the generation is rescanned to find where it
+  landed.
+
+### Security
+
+- **The disk cache is private to its owner.** `diskBlobCache()` stored
+  entries with the process umask, so under the usual 022 its directories
+  were 0755 and its files 0644: readable by other local users wherever
+  the parent directories allowed it. Entries hold record text, which for a
+  pack of a private repository, a journal, or conversation history is the
+  source itself. Directories are now created 0700 and files 0600, and the
+  first write tightens the store's root to 0700, which also closes off
+  entries a 0.13.0 cache already wrote.
+
 ## pikelet-wasm 0.13.0 / pikelet 0.13.0 — 2026-11-01
 
 Mounting a pack over HTTP is much cheaper the second time: `pikelet mcp`

package/complete/blob-cache.node.mjs

@@ -1,23 +1,46 @@
-// A disk store for openPikeletFile's options.blobCache: whole regions a pack
-// commits to by sha256 (the sketch's resident prefix, the query-interp
-// segment holding the encoder), filed under that digest. Those regions are
-// most of what a URL mount downloads (about 71 of 77 MiB on the 649 MiB
-// Simple English Wikipedia pack), so a second process mounting the same pack
-// reads them from disk instead of the network.
+// A disk store for openPikeletFile's options.blobCache: bytes a pack commits
+// to by sha256, filed under that digest. Two kinds of entry reach it:
+//
+//   - whole regions read at mount (the sketch's resident prefix, the
+//     query-interp segment holding the encoder): most of what a URL mount
+//     downloads, about 71 of 77 MiB on the 649 MiB Simple English Wikipedia
+//     pack. One file each.
+//   - pieces read per query (rerank vector rows, corpus records, and the
+//     digest pages that verify them): a few hundred bytes to a few KiB, and
+//     hundreds per query. One file each would be a poor fit, so entries under
+//     SMALL_ENTRY_BYTES go into append-only log files (generations) with an
+//     in-memory index, and whole generations rotate out under their own
+//     budget.
 //
 // The store is untrusted by design: the reader hashes every hit against the
 // digest it asked for and treats a mismatch as a miss (blobCacheGet in
-// pikelet-artifact-common.js), so a torn write or a corrupted file costs one
-// refetch, never a wrong byte. Entries are immutable and never go stale; the
-// only policy is a size budget, enforced by evicting least recently used
-// entries (by mtime, touched on every hit) after each write.
+// pikelet-artifact-common.js), so a torn write, a corrupted file, or two
+// processes appending to one log at once costs a refetch, never a wrong byte.
+// Entries are immutable and never go stale; the only policy is size.
 
 import fs from 'node:fs/promises';
 import os from 'node:os';
 import path from 'node:path';
 
-const KEY = /^[0-9a-f]{64}$/;
+// A full sha256, or a truncated one of at least 128 bits (sketch row digests
+// may be 16-32 bytes).
+const KEY = /^(?:[0-9a-f]{2}){16,32}$/;
 export const DEFAULT_BLOB_CACHE_BYTES = 2 * 1024 * 1024 * 1024;
+export const DEFAULT_SMALL_CACHE_BYTES = 64 * 1024 * 1024;
+// Entries below this go to the log; at or above it, a file of their own.
+export const SMALL_ENTRY_BYTES = 64 * 1024;
+// The log keeps this many generations; the oldest is deleted whole when the
+// newest fills (smallMaxBytes / GENERATIONS each).
+const GENERATIONS = 4;
+// Log record: magic 'PKB1' (u32 LE), key length (u8), key bytes, data length
+// (u32 LE), data. A scan stops at the first record that does not parse.
+const RECORD_MAGIC = 0x3142_4b50;
+const GEN_FILE = /^gen-(\d{8})\.log$/;
+// Entries are pack content, record text included, and a pack can be a
+// private repository or a conversation history: the store is private to its
+// owner whatever the umask.
+const DIR_MODE = 0o700;
+const FILE_MODE = 0o600;
 
 // $PIKELET_CACHE_DIR, else $XDG_CACHE_HOME/pikelet/blobs, else
 // ~/.cache/pikelet/blobs: the same root the CLI already caches encoder
@@ -28,15 +51,28 @@ export function defaultBlobCacheDir(env = process.env) {
     return path.join(base, 'pikelet', 'blobs');
 }
 
-export function diskBlobCache(dir = defaultBlobCacheDir(), { maxBytes = DEFAULT_BLOB_CACHE_BYTES } = {}) {
-    if (!Number.isSafeInteger(maxBytes) || maxBytes < 0) {
-        throw new Error('diskBlobCache maxBytes must be a non-negative integer');
+export function diskBlobCache(dir = defaultBlobCacheDir(), { maxBytes = DEFAULT_BLOB_CACHE_BYTES, smallMaxBytes = DEFAULT_SMALL_CACHE_BYTES } = {}) {
+    for (const [label, value] of [['maxBytes', maxBytes], ['smallMaxBytes', smallMaxBytes]]) {
+        if (!Number.isSafeInteger(value) || value < 0) {
+            throw new Error(`diskBlobCache ${label} must be a non-negative integer`);
+        }
     }
     const root = path.resolve(dir);
-    const fileOf = (key) => {
-        if (!KEY.test(key)) throw new Error(`blob cache key must be a lowercase sha256 hex digest, got ${JSON.stringify(key)}`);
-        return path.join(root, key.slice(0, 2), key);
+    const checkKey = (key) => {
+        if (typeof key !== 'string' || !KEY.test(key)) {
+            throw new Error(`blob cache key must be a lowercase sha256 hex digest (32-64 hex digits), got ${JSON.stringify(key)}`);
+        }
+        return key;
     };
+    const fileOf = (key) => path.join(root, key.slice(0, 2), checkKey(key));
+    // Once per instance, before the first write: a root made by an older
+    // store (or by mkdir -p under umask 022) is tightened, which also closes
+    // off whatever it already holds. Best effort; on Windows modes are moot.
+    let privateRoot = null;
+    const makeRootPrivate = () => (privateRoot ??= (async () => {
+        await fs.mkdir(root, { recursive: true, mode: DIR_MODE });
+        await fs.chmod(root, DIR_MODE).catch(() => {});
+    })());
     const pending = new Set();
     const track = (promise) => {
         pending.add(promise);
@@ -48,6 +84,7 @@ export function diskBlobCache(dir = defaultBlobCacheDir(), { maxBytes = DEFAULT_
         const entries = [];
         let total = 0;
         for (const shard of await fs.readdir(root).catch(() => [])) {
+            if (shard === 'small') continue;
             const shardDir = path.join(root, shard);
             for (const name of await fs.readdir(shardDir).catch(() => [])) {
                 if (!KEY.test(name)) continue;
@@ -67,9 +104,178 @@ export function diskBlobCache(dir = defaultBlobCacheDir(), { maxBytes = DEFAULT_
         }
     };
 
+    // --- the small-entry log -------------------------------------------------
+    const smallDir = path.join(root, 'small');
+    const genBytes = Math.max(1, Math.floor(smallMaxBytes / GENERATIONS));
+    const genPath = (gen) => path.join(smallDir, `gen-${String(gen).padStart(8, '0')}.log`);
+    const index = new Map(); // key -> { gen, offset, length }
+    const handles = new Map(); // gen -> Promise<FileHandle> (read)
+    let current = 0;
+    // Set when the newest generation ends in bytes that do not parse (a torn
+    // write): a scan stops there, so anything appended after it would never
+    // be indexed again. Writing resumes in a fresh generation instead.
+    let sealed = false;
+    let queue = [];
+    let flushing = null;
+
+    const listGens = async () => (await fs.readdir(smallDir).catch(() => []))
+        .map((name) => GEN_FILE.exec(name)).filter(Boolean).map((m) => Number(m[1])).sort((a, b) => a - b);
+
+    const forgetGen = async (gen) => {
+        for (const [key, at] of index) if (at.gen === gen) index.delete(key);
+        const handle = handles.get(gen);
+        handles.delete(gen);
+        if (handle) await handle.then((h) => h.close(), () => {});
+    };
+
+    const scan = (gen, bytes) => {
+        const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+        let at = 0;
+        while (at + 9 <= bytes.length && view.getUint32(at, true) === RECORD_MAGIC) {
+            const keyLen = bytes[at + 4];
+            if (keyLen < 16 || keyLen > 32 || at + 5 + keyLen + 4 > bytes.length) break;
+            const key = Buffer.from(bytes.subarray(at + 5, at + 5 + keyLen)).toString('hex');
+            const length = view.getUint32(at + 5 + keyLen, true);
+            const offset = at + 9 + keyLen;
+            if (offset + length > bytes.length) break;
+            index.set(key, { gen, offset, length });
+            at = offset + length;
+        }
+        return at === bytes.length;
+    };
+
+    // Built once, on first use: every generation still on disk is read and
+    // indexed (at most smallMaxBytes), later generations overriding earlier.
+    let ready = null;
+    const smallReady = () => (ready ??= (async () => {
+        if (smallMaxBytes === 0) return;
+        const gens = await listGens();
+        for (const gen of gens.slice(0, Math.max(0, gens.length - GENERATIONS))) {
+            await fs.rm(genPath(gen), { force: true });
+        }
+        for (const gen of gens.slice(-GENERATIONS)) {
+            const bytes = await fs.readFile(genPath(gen)).catch(() => null);
+            const whole = bytes ? scan(gen, bytes) : true;
+            if (gen === gens[gens.length - 1]) sealed = !whole;
+        }
+        current = gens.length ? gens[gens.length - 1] : 0;
+    })());
+
+    const readSmall = async (key) => {
+        const at = index.get(key);
+        if (!at) return null;
+        let handle = handles.get(at.gen);
+        if (!handle) {
+            handle = fs.open(genPath(at.gen), 'r');
+            handles.set(at.gen, handle);
+            handle.catch(() => { if (handles.get(at.gen) === handle) handles.delete(at.gen); });
+        }
+        let fh;
+        try { fh = await handle; } catch (err) {
+            // Another process rotated this generation away.
+            if (err?.code === 'ENOENT') { await forgetGen(at.gen); return null; }
+            throw err;
+        }
+        const out = new Uint8Array(at.length);
+        const { bytesRead } = await fh.read(out, 0, at.length, at.offset);
+        if (bytesRead !== at.length) { index.delete(key); return null; }
+        // A hit from an older generation is written again to the newest, so
+        // rotation drops what has not been read lately: LRU by generation.
+        if (at.gen !== current) enqueue(key, out);
+        return out;
+    };
+
+    const rotate = async () => {
+        await makeRootPrivate();
+        await fs.mkdir(smallDir, { recursive: true, mode: DIR_MODE });
+        const gens = await listGens();
+        current = Math.max(current, gens.length ? gens[gens.length - 1] : 0) + 1;
+        // Exclusive create; another process that rotated first owns the
+        // name, and appending after its records is fine.
+        await fs.writeFile(genPath(current), new Uint8Array(0), { flag: 'wx', mode: FILE_MODE }).catch((err) => {
+            if (err?.code !== 'EEXIST') throw err;
+        });
+        for (const gen of (await listGens()).filter((g) => g <= current - GENERATIONS)) {
+            await forgetGen(gen);
+            await fs.rm(genPath(gen), { force: true });
+        }
+    };
+
+    const flushSmall = async () => {
+        while (queue.length) {
+            const batch = queue;
+            queue = [];
+            const total = batch.reduce((n, r) => n + r.record.length, 0);
+            if (!current || sealed) {
+                await rotate();
+                sealed = false;
+            } else {
+                const st = await fs.stat(genPath(current)).catch(() => null);
+                if (!st || st.size + total > genBytes) await rotate();
+            }
+            const buf = Buffer.concat(batch.map((r) => r.record));
+            const fh = await fs.open(genPath(current), 'a+', FILE_MODE);
+            let base;
+            try {
+                await fh.write(buf);
+                // O_APPEND puts the batch at the end; its offset is the size
+                // now minus its length, unless another process appended
+                // between the write and the stat. So the batch is read back
+                // there before it is indexed: records are content-addressed,
+                // so wherever these exact bytes sit is a right offset.
+                base = (await fh.stat()).size - buf.length;
+                if (base >= 0) {
+                    const back = Buffer.alloc(buf.length);
+                    const { bytesRead } = await fh.read(back, 0, buf.length, base);
+                    if (bytesRead !== buf.length || !back.equals(buf)) base = -1;
+                }
+            } finally {
+                await fh.close();
+            }
+            if (base < 0) {
+                // The end moved: find where the batch landed by scanning the
+                // generation, which indexes the other writer's records too.
+                // A record still being written stops the scan; whatever lies
+                // past it is a miss, not a wrong offset.
+                const bytes = await fs.readFile(genPath(current)).catch(() => null);
+                if (bytes) scan(current, bytes);
+                continue;
+            }
+            let at = base;
+            for (const r of batch) {
+                index.set(r.key, { gen: current, offset: at + r.headerBytes, length: r.length });
+                at += r.record.length;
+            }
+        }
+    };
+
+    const enqueue = (key, bytes) => {
+        if (index.has(key) && index.get(key).gen === current) return;
+        const raw = Buffer.from(key, 'hex');
+        const headerBytes = 9 + raw.length;
+        const record = Buffer.alloc(headerBytes + bytes.byteLength);
+        record.writeUInt32LE(RECORD_MAGIC, 0);
+        record[4] = raw.length;
+        raw.copy(record, 5);
+        record.writeUInt32LE(bytes.byteLength, 5 + raw.length);
+        record.set(bytes, headerBytes);
+        queue.push({ key, record, headerBytes, length: bytes.byteLength });
+        // One append per burst: a query stores hundreds of rows at once.
+        flushing ??= track(new Promise((resolve) => setImmediate(resolve))
+            .then(() => flushSmall())
+            .finally(() => { flushing = null; }));
+        return flushing;
+    };
+
     return {
         dir: root,
         async get(key) {
+            checkKey(key);
+            if (smallMaxBytes > 0) {
+                await smallReady();
+                const small = await readSmall(key);
+                if (small) return small;
+            }
             const file = fileOf(key);
             let bytes;
             try { bytes = await fs.readFile(file); } catch (err) {
@@ -82,15 +288,21 @@ export function diskBlobCache(dir = defaultBlobCacheDir(), { maxBytes = DEFAULT_
             return new Uint8Array(bytes.buffer, bytes.byteOffset, bytes.byteLength);
         },
         put(key, bytes) {
+            checkKey(key);
+            if (bytes.byteLength < SMALL_ENTRY_BYTES) {
+                if (smallMaxBytes === 0 || bytes.byteLength > genBytes) return Promise.resolve();
+                return track(smallReady().then(() => enqueue(key, bytes)));
+            }
             const file = fileOf(key);
             return track((async () => {
                 if (bytes.byteLength > maxBytes) return;
-                await fs.mkdir(path.dirname(file), { recursive: true });
+                await makeRootPrivate();
+                await fs.mkdir(path.dirname(file), { recursive: true, mode: DIR_MODE });
                 // Write-then-rename, so a reader in another process sees
                 // either no entry or a whole one.
                 const tmp = `${file}.${process.pid}.${Math.random().toString(36).slice(2)}.tmp`;
                 try {
-                    await fs.writeFile(tmp, bytes);
+                    await fs.writeFile(tmp, bytes, { mode: FILE_MODE });
                     await fs.rename(tmp, file);
                 } catch (err) {
                     await fs.rm(tmp, { force: true });
@@ -100,6 +312,10 @@ export function diskBlobCache(dir = defaultBlobCacheDir(), { maxBytes = DEFAULT_
             })());
         },
         async delete(key) {
+            checkKey(key);
+            // A log entry cannot be removed in place; forgetting it is enough,
+            // and rotation reclaims the bytes.
+            index.delete(key);
             await fs.rm(fileOf(key), { force: true });
         },
         // Settles every write started so far: for tests, and for a process
@@ -107,5 +323,15 @@ export function diskBlobCache(dir = defaultBlobCacheDir(), { maxBytes = DEFAULT_
         async flush() {
             while (pending.size) await Promise.allSettled([...pending]);
         },
+        // Closes the log's read handles (they otherwise live as long as the
+        // process). The store stays usable; handles reopen on demand.
+        async close() {
+            await this.flush();
+            for (const gen of [...handles.keys()]) {
+                const handle = handles.get(gen);
+                handles.delete(gen);
+                await handle.then((h) => h.close(), () => {});
+            }
+        },
     };
 }

package/complete/index.d.ts

@@ -96,7 +96,7 @@ export declare function nodeFetch(input: string, init?: {
   inflateGzip?: boolean;
 }): Promise<any>;
 
-/** openPikeletFile's options.blobCache: bytes keyed by their sha256 (lowercase hex). */
+/** openPikeletFile's options.blobCache: bytes keyed by their sha256 (lowercase hex, 32-64 digits). */
 export interface BlobCache {
   get(key: string): Promise<Uint8Array | null> | Uint8Array | null;
   put(key: string, bytes: Uint8Array): Promise<void> | void;
@@ -104,17 +104,23 @@ export interface BlobCache {
 }
 
 /**
- * Node only (the 'node' export condition): a BlobCache on disk, one file per
- * key under `dir` (default defaultBlobCacheDir()), evicting least recently
- * used entries past maxBytes (default 2 GiB). flush() settles pending writes.
+ * Node only (the 'node' export condition): a BlobCache on disk under `dir`
+ * (default defaultBlobCacheDir()). Entries of 64 KiB or more get a file each,
+ * evicting least recently used past maxBytes (default 2 GiB). Smaller ones
+ * (per-query rows, records, digest pages) go to append-only log generations
+ * capped at smallMaxBytes in total (default 64 MiB; 0 disables them), the
+ * oldest generation dropped whole. flush() settles pending writes; close()
+ * releases the log's file handles.
  */
-export declare function diskBlobCache(dir?: string, options?: { maxBytes?: number }): BlobCache & {
+export declare function diskBlobCache(dir?: string, options?: { maxBytes?: number; smallMaxBytes?: number }): BlobCache & {
   dir: string;
   flush(): Promise<void>;
+  close(): Promise<void>;
 };
 /** $PIKELET_CACHE_DIR/blobs, else $XDG_CACHE_HOME/pikelet/blobs, else ~/.cache/pikelet/blobs. */
 export declare function defaultBlobCacheDir(env?: Record<string, string | undefined>): string;
 export declare const DEFAULT_BLOB_CACHE_BYTES: number;
+export declare const DEFAULT_SMALL_CACHE_BYTES: number;
 
 export interface CompleteQueryResult {
   /**
@@ -308,14 +314,18 @@ export declare function openPikeletFile(
      */
     prefetchEncoder?: boolean;
     /**
-     * A store for the regions a pack commits to by sha256 that dominate a
-     * cold URL mount: the sketch's resident prefix and the query-interp
-     * segment holding the inline encoder. Keys are those digests (lowercase
-     * hex), so entries never go stale. Every hit is hashed against its key
-     * before use and a mismatch is treated as a miss (and deleted), so the
-     * store need not be trusted; get/put/delete faults never fail an open.
-     * diskBlobCache() (Node) is a ready implementation. Ignored by an open
-     * whose resident verification is off.
+     * A store for bytes a pack commits to by sha256. At mount: the sketch's
+     * resident prefix and the query-interp segment holding the inline
+     * encoder, which dominate a cold URL mount. Per query: rerank vector
+     * rows (format 2), corpus records, and the digest pages that verify
+     * both. Keys are those digests in lowercase hex (a row's digest may be
+     * truncated to 16-31 bytes), so entries never go stale. Every hit is
+     * hashed and must match its key (a prefix, for a truncated key); a
+     * mismatch is treated as a miss and deleted, so the store need not be
+     * trusted, and get/put/delete faults never fail a read. Only reads that
+     * are verified on their own use it: nothing when verification is off,
+     * and never the lazy lexical index. diskBlobCache() (Node) is a ready
+     * implementation.
      */
     blobCache?: BlobCache;
     /**

package/complete/index.mjs

@@ -97,9 +97,9 @@ const COVERAGE_BATCH_COUNT = 5;
 // Only BM25 hits within this fraction of the top lexical score join fusion.
 // Common query terms give every matching document a near-tied score (idf
 // collapses them flat); a wide cutoff let that tied mass ride into fusion on
-// a technicality. Kept identical to pikelet/src/calibrate.mjs's
-// LEXICAL_CUTOFF so calibration is fit against the same fusion the reader
-// actually serves.
+// a technicality. Calibration (pikelet/src/calibrate.mjs) gathers its
+// evidence through this reader's own retrieval, so it is fit against the
+// same cutoff and fusion the reader serves.
 export const LEXICAL_CUTOFF = 1.5;
 // Query-interpretation segments above this size open lazily (kind 3's
 // inline encoder is ~25 MiB); smaller ones keep the one-read eager path.
@@ -1200,12 +1200,16 @@ export async function openPikeletFile(input, options = {}) {
             const pending = (async () => {
                 const from = Math.min(recordCount, page * pageRecords);
                 const to = Math.min(recordCount, (page + 1) * pageRecords);
+                const key = toHex(pageTable.subarray(32 * page, 32 * page + 32));
+                const cached = await blobCacheGet(options.blobCache, key, 32 * (to - from));
+                if (cached) return cached;
                 const bytes = await readChecked(source, corpus.offset + recordDigestsAt + 32 * from, 32 * (to - from),
                     `corpus digest page ${page}`, maxReadBytes, fileBytes);
-                if (await sha256hex(bytes) !== toHex(pageTable.subarray(32 * page, 32 * page + 32))) {
+                if (await sha256hex(bytes) !== key) {
                     digestPages.delete(page);
                     throw new Error(`.pikelet corpus digest page ${page} failed hash verification`);
                 }
+                blobCachePut(options.blobCache, key, bytes);
                 return bytes;
             })();
             digestPages.set(page, pending);
@@ -1234,14 +1238,29 @@ export async function openPikeletFile(input, options = {}) {
             }
             const start = recordOffsets[id];
             const end = recordOffsets[id + 1];
-            const bytes = await readChecked(source, corpus.offset + start, end - start, `corpus record ${id}`, maxRecordBytes, fileBytes);
-            if (perRecord && verifyRecords) {
+            // With options.blobCache a verified record is keyed by its own
+            // digest, so the digest comes first; another process may already
+            // have fetched the record. Only verified reads use the cache.
+            const cacheable = perRecord && verifyRecords && options.blobCache;
+            let digest = null;
+            if (cacheable) {
                 const page = Math.floor(id / pageRecords);
-                const digests = await pageDigests(page);
                 const at = 32 * (id - page * pageRecords);
-                if (await sha256hex(bytes) !== toHex(digests.subarray(at, at + 32))) {
+                digest = toHex((await pageDigests(page)).subarray(at, at + 32));
+                const cached = await blobCacheGet(options.blobCache, digest, end - start);
+                if (cached) return cached;
+            }
+            const bytes = await readChecked(source, corpus.offset + start, end - start, `corpus record ${id}`, maxRecordBytes, fileBytes);
+            if (perRecord && verifyRecords) {
+                if (!digest) {
+                    const page = Math.floor(id / pageRecords);
+                    const at = 32 * (id - page * pageRecords);
+                    digest = toHex((await pageDigests(page)).subarray(at, at + 32));
+                }
+                if (await sha256hex(bytes) !== digest) {
                     throw new Error(`.pikelet corpus record ${id} failed integrity verification`);
                 }
+                blobCachePut(options.blobCache, digest, bytes);
             }
             return bytes;
         };

package/complete/index.node.mjs

@@ -9,7 +9,7 @@ import { nodeFetch } from './node-fetch.mjs';
 
 export * from './index.mjs';
 export { nodeFetch };
-export { diskBlobCache, defaultBlobCacheDir, DEFAULT_BLOB_CACHE_BYTES } from './blob-cache.node.mjs';
+export { diskBlobCache, defaultBlobCacheDir, DEFAULT_BLOB_CACHE_BYTES, DEFAULT_SMALL_CACHE_BYTES } from './blob-cache.node.mjs';
 
 export function httpRangeSource(url, options = {}) {
     return portableHttpRangeSource(url, { fetch: nodeFetch, ...options });

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pikelet-wasm",
-  "version": "0.13.0",
+  "version": "0.14.0",
   "description": "Vector search engine and .pikelet artifact readers/builders, in WebAssembly. Compile a corpus into one self-contained, hash-verified search file, or use the underlying HNSW engine directly. Runs in Node.js, browsers, and Cloudflare Workers with no native dependencies.",
   "main": "src/pikelet.js",
   "types": "src/pikelet.d.ts",

package/src/artifact/pikelet-artifact-common.js

@@ -462,25 +462,31 @@ async function sha256BytesAsync(bytes) {
     }
 }
 
-// Read-through cache for whole regions that carry a sha256 commitment (the
-// sketch's resident prefix, a pack's query-interp segment), keyed by that
-// digest. A hit is hashed exactly as a network read would be, so the cache
-// can only ever hand back the committed bytes: anything else it holds (a
-// partial write, disk corruption, a bug elsewhere) reads as a miss and the
-// entry is dropped. Because the key is the content, entries never go stale
-// and need no invalidation. Cache faults never fail a read.
+// Read-through cache for bytes a pack commits to by sha256: the whole
+// regions a mount reads (the sketch's resident prefix, a pack's query-interp
+// segment) and the pieces a query reads (rerank rows, corpus records, the
+// digest pages that verify them), keyed by that digest. A key may be a
+// truncated digest of at least 128 bits (sketch row digests are 16-32
+// bytes), and a hit must hash to a digest that starts with it. A hit is
+// hashed exactly as a network read would be, so the cache can only ever hand
+// back the committed bytes: anything else it holds (a partial write, disk
+// corruption, a bug elsewhere) reads as a miss and the entry is dropped.
+// Because the key is the content, entries never go stale and need no
+// invalidation. Cache faults never fail a read. `length`, when given, must
+// match too.
 async function blobCacheGet(cache, digestHex, length) {
     if (!cache || typeof cache.get !== 'function') return null;
+    if (typeof digestHex !== 'string' || digestHex.length < 32 || digestHex.length > 64) return null;
     let hit;
     try {
         hit = await cache.get(digestHex);
         if (!hit) return null;
         hit = asUint8Array(hit);
     } catch { return null; }
-    const digest = hit.byteLength === length ? await sha256BytesAsync(hit) : null;
+    const digest = length === undefined || hit.byteLength === length ? await sha256BytesAsync(hit) : null;
     let hex = '';
     if (digest) for (const b of digest) hex += b.toString(16).padStart(2, '0');
-    if (hex === digestHex) return hit;
+    if (hex && hex.startsWith(digestHex)) return hit;
     try { await cache.delete?.(digestHex); } catch { /* a miss either way */ }
     return null;
 }

package/src/artifact/pikelet-artifact-sketch.js

@@ -471,6 +471,9 @@ class PikeletSketchArtifact {
         // Per-row verification on format 2 follows the same switch as the
         // resident hash: on by default, verify:false opts the whole open out.
         artifact.verifyRows = version >= 2 && verify;
+        // Per-query pieces (digest pages and rows) use the same cache as the
+        // resident prefix, but only where each read is verified on its own.
+        artifact.blobCache = artifact.verifyRows && options.blobCache ? options.blobCache : null;
         const staged = options.staged === true && artifact.microDims > 0;
         artifact.tier = 'full';
         artifact.fullyResident = Promise.resolve(artifact);
@@ -781,8 +784,37 @@ class PikeletSketchArtifact {
             }
         }
         if (!missing.length) return rows;
-        missing.sort((a, b) => a - b);
         const { dim, rowsPerBlock, rowDigestBytes, pageBytes, blockBytes, vectorsOffset } = this;
+        // options.blobCache at open: a block's digest page is keyed by its
+        // page-table entry and each row by its digest in that page, so a row
+        // another process already fetched and verified is read from disk.
+        // Every hit is hash-checked against its key (blobCacheGet).
+        const cache = this.blobCache;
+        if (cache) {
+            const byBlock = new Map();
+            for (const id of missing) {
+                const b = Math.floor(id / rowsPerBlock);
+                if (!byBlock.has(b)) byBlock.set(b, []);
+                byBlock.get(b).push(id);
+            }
+            const still = [];
+            await mapLimit([...byBlock], 16, async ([b, ids]) => {
+                const page = await blobCacheGet(cache, hexOf(this.pageTable.subarray(b * 32, b * 32 + 32)), pageBytes);
+                if (!page) { still.push(...ids); return; }
+                for (const id of ids) {
+                    const idx = id - b * rowsPerBlock;
+                    const hit = await blobCacheGet(cache, hexOf(page.subarray(idx * rowDigestBytes, (idx + 1) * rowDigestBytes)), dim);
+                    if (!hit) { still.push(id); continue; }
+                    const row = new Uint8Array(hit);
+                    rows.set(id, row);
+                    this.cacheRow(id, row);
+                }
+            });
+            if (!still.length) return rows;
+            missing.length = 0;
+            missing.push(...still);
+        }
+        missing.sort((a, b) => a - b);
         const blocks = new Map();
         for (const id of missing) {
             const b = Math.floor(id / rowsPerBlock);
@@ -836,6 +868,7 @@ class PikeletSketchArtifact {
                             throw pikeletError(PIKELET_ERROR_CODES.SNAPSHOT_INVALID, 'Sketch artifact row digest page failed hash verification', { block: b });
                         }
                     }
+                    if (cache) blobCachePut(cache, hexOf(expected), page.slice());
                 }
                 for (const id of blocks.get(b).ids) {
                     const idx = id - b * rowsPerBlock;
@@ -848,6 +881,7 @@ class PikeletSketchArtifact {
                                 throw pikeletError(PIKELET_ERROR_CODES.SNAPSHOT_INVALID, 'Sketch artifact row failed digest verification', { id });
                             }
                         }
+                        if (cache) blobCachePut(cache, hexOf(slot), row);
                     }
                     if (rows.has(id)) rows.set(id, row);
                     this.cacheRow(id, row);