pikelet-wasm 0.12.0 → 0.14.0

package/CHANGELOG.md

@@ -8,6 +8,160 @@ Through 0.6.0 these were published as `pancake-wasm` and
 packages are pre-1.0: a minor bump may carry breaking changes, and each entry lists them
 first.
 
+## pikelet-wasm 0.14.0 / pikelet 0.14.0 — 2026-11-04
+
+Asking a URL pack a question an earlier process already asked is now
+mostly local too: `pikelet mcp` keeps the verified rows, records and digest
+pages a query reads in its disk cache, so on the Wikipedia pack over R2 a
+repeated question transferred 178 KiB instead of 1.3 MiB and answered in
+0.65 s instead of 1.45 s. The cache is now private to its owner (0.13.0
+left it readable by other local users under the usual umask), and two
+processes sharing it no longer mis-index each other's writes. Upgrade if
+you mount packs by URL, and especially if other people have accounts on
+the machine. Packs compiled before this release still verify and serve,
+and compiling the same inputs gives the same identities as 0.13.0. Both
+packages move to 0.14.0 together, and `pikelet` requires `pikelet-wasm`
+^0.14.0.
+
+### Compatibility
+
+- **The cache also holds query reads.** On top of the mount entries (up
+  to 2 GiB), URL packs now store what queries read in append-only logs
+  under `~/.cache/pikelet/blobs/small`, up to 64 MiB. `--no-cache` still
+  turns all of it off, and local packs still never use it.
+- **The cache root becomes 0700.** The first write tightens the store's
+  directory to owner-only, including a `$PIKELET_CACHE_DIR/blobs` or a
+  directory passed to `diskBlobCache(dir)` that was opened up on purpose.
+  Nothing above that directory is touched.
+- **Library callers: close a `diskBlobCache()` you are done with.** A store
+  that has served a query entry holds read handles on its log until
+  `close()`. Node 26 treats a file handle left to garbage collection as a
+  fatal error, so a store dropped without `close()` can end the process
+  there. The CLI keeps its store for the life of the process and is not
+  affected.
+
+### Added
+
+- **Queries use the disk cache too.** With `blobCache`, the verified pieces
+  a query reads are stored and served by digest: rerank vector rows (by
+  their row digests; format 2), corpus records (by their record digests),
+  and the digest pages that verify both (by their page-table hashes). Every
+  hit is hash-checked against its key, a truncated row digest of at least
+  128 bits included, and only reads verified on their own use the cache:
+  the lazy lexical index never does. `diskBlobCache()` keeps entries under
+  64 KiB in append-only log generations (64 MiB in total by default,
+  `smallMaxBytes`; the oldest generation dropped whole, hits rewritten to
+  the newest) instead of one file each, and gains `close()`. On the
+  Wikipedia pack over R2, asking a question an earlier process already
+  asked transferred 178 KiB in 11 requests instead of 1.3 MiB in 29, and
+  answered in 0.65 s instead of 1.45 s; a new question costs what it did.
+
+### Fixed
+
+- **Generated configs point at a schema that exists.** The configs the
+  CLI generates carried a `$schema` URL from before the repository moved
+  to `packages/`, which returned 404; it and the schema's own `$id` now name
+  `packages/pikelet/schemas/v1/pikelet.config.schema.json`. Configs
+  generated earlier keep the old URL until it is edited.
+- **Two processes sharing a disk cache no longer mis-index each other's
+  appends.** A store computed a log batch's offset as the file size after
+  its append, so another process appending in between left its index
+  pointing at the wrong record: about a third of each writer's own
+  lookups across three processes. Every hit is hash-checked, so this cost
+  refetches, never a wrong byte, and a fresh process re-indexed the log
+  correctly. A batch is now read back at its offset before it is indexed,
+  and if the end moved, the generation is rescanned to find where it
+  landed.
+
+### Security
+
+- **The disk cache is private to its owner.** `diskBlobCache()` stored
+  entries with the process umask, so under the usual 022 its directories
+  were 0755 and its files 0644: readable by other local users wherever
+  the parent directories allowed it. Entries hold record text, which for a
+  pack of a private repository, a journal, or conversation history is the
+  source itself. Directories are now created 0700 and files 0600, and the
+  first write tightens the store's root to 0700, which also closes off
+  entries a 0.13.0 cache already wrote.
+
+## pikelet-wasm 0.13.0 / pikelet 0.13.0 — 2026-11-01
+
+Mounting a pack over HTTP is much cheaper the second time: `pikelet mcp`
+now keeps the two largest mount reads of a URL pack on disk, so a fresh
+process mounting the 649 MiB Wikipedia pack from R2 transfers 9.1 MiB
+instead of 80.1 MiB. A slow remote mount no longer leaves an MCP client
+with no tools. `compile` and `append` now name what they drop instead of
+dropping it silently, and a layered pack keeps its name over MCP. Upgrade
+if you mount packs by URL or update them with `append`. Packs compiled
+before this release still verify and serve, and compiling the same inputs
+gives the same identities as 0.12.0. Both packages move to 0.13.0 together,
+and `pikelet` requires `pikelet-wasm` ^0.13.0.
+
+### Compatibility
+
+- **`pikelet mcp` writes a cache for URL packs.** The resident index prefix
+  and the query encoder of each URL pack mounted (about 71 MiB for the
+  Wikipedia pack) are stored in `~/.cache/pikelet/blobs`, up to 2 GiB,
+  evicting least recently used entries first. `$PIKELET_CACHE_DIR` or
+  `$XDG_CACHE_HOME` moves it, and `--no-cache` turns it off. Local packs
+  never use it.
+- **A layered pack mounts under its pack's name.** A chain is named like a
+  single pack: from the shelf entry, else the base's manifest name, else
+  the file name with the `.0002` layer number dropped. A chain mounted from
+  `docs.0002.pikelet` was served as `docs.0002`; it is now `docs`, the
+  name its base alone mounted under. Tool calls that name the old pack
+  need updating. A single pack keeps a four-digit suffix (`wiki.2025`).
+- **A missing calibration questions file is an error.** A
+  `--calibration-queries` file (or `runtime.calibrationQueries`) that does
+  not exist used to calibrate without the questions and say nothing; it
+  now fails, naming the path.
+- **`tools/list` can answer before the packs have mounted.** After 2
+  seconds it answers with the same tools described without pack names,
+  uncached, and sends `notifications/tools/list_changed` once the mounts
+  settle; `initialize` and `server/discover` advertise
+  `tools.listChanged`. Tool calls still wait for the mounts.
+
+### Added
+
+- **On-disk cache for URL packs (`pikelet-wasm`).** `openPikeletFile`
+  takes `blobCache`, a store keyed by sha256. It holds the sketch's
+  resident prefix and the query-interp segment that carries the encoder.
+  Every hit is hashed against its key, so a corrupt entry is a miss that
+  is deleted and refetched, and cache faults never fail an open.
+  `diskBlobCache()` and `defaultBlobCacheDir()` (Node, from
+  `pikelet-wasm/complete`) provide one on disk, and `pikelet mcp` uses it
+  for URL packs. Measured against the Wikipedia pack on R2, a fresh
+  process went from 80.1 MiB and 11.5 s to its first answer down to
+  9.1 MiB and 6.9 s. Per-query reads are not cached.
+- **A chain reports its pack's name.** `openPikeletChain(...).info().name`
+  is the base's name.
+
+### Fixed
+
+- **A slow remote mount left an MCP client with no tools.** `tools/list`
+  waited for every mount; when a URL mount stalled (6.3 MiB in 37 s from
+  R2), Claude Code gave up on the list and started the session with no
+  tools at all. See Compatibility for how it answers now.
+- **A calibration question naming no record was dropped silently.** Each
+  is now logged by name with the reason, and with the title it probably
+  meant when Markdown formatting is the difference (titles are matched
+  after formatting is stripped). An `expectId` outside the corpus is named
+  the same way.
+- **`append` dropped short pages without saying which.** Documents under
+  25 whitespace-separated tokens are skipped whole; `compile` and `append`
+  now log them with their token counts, and `append`'s refusal when
+  nothing is left names them.
+- **`append`'s closing hint was not a usable next step.** It suggested
+  `openPikeletChain([...])`, which takes a query vector, not text; it now
+  prints `pikelet mcp --pack <layer>#<identity>`.
+
+### Documentation
+
+- **Deleting is not redacting.** `append --remove` and `--supersede` take a
+  record out of search, but earlier files keep its bytes and `get_record`
+  serves it by id, marked `tombstoned`, so old citations resolve.
+  `compact` writes a base without it. Both READMEs say so.
+
 ## pikelet-wasm 0.12.0 / pikelet 0.12.0 — 2026-10-30
 
 Query embedding is about 1.65x faster, because the inline encoder now

package/README.md

@@ -88,7 +88,7 @@ npx pikelet append --parent docs.0001.pikelet --remove 412 --out docs.0002.pikel
 npx pikelet mcp --pack docs.0002.pikelet
 ```
 
-On a 20-record test pack with a 24.6 MiB base, deleting a record took a 1.2 KiB layer and replacing one took 6.9 KiB. Mounting the newest layer finds and verifies the rest of the chain, locally or over HTTP, and `#<sha256>` on it pins the whole history. Layers inherit the base's calibration until the chain drifts past 20% of the base's records; then it serves `unscored` until `compact` folds it into a fresh base. The format is specified in [`LAYERED_PROFILE.md`](LAYERED_PROFILE.md).
+On a 20-record test pack with a 24.6 MiB base, deleting a record took a 1.2 KiB layer and replacing one took 6.9 KiB. Deleting hides a record from search but does not erase it: the older files still hold its bytes, and `get_record` still serves it by id, marked `tombstoned`, so old citations resolve. `compact` writes a new base without it. Mounting the newest layer finds and verifies the rest of the chain, locally or over HTTP, and `#<sha256>` on it pins the whole history. Layers inherit the base's calibration until the chain drifts past 20% of the base's records; then it serves `unscored` until `compact` folds it into a fresh base. The format is specified in [`LAYERED_PROFILE.md`](LAYERED_PROFILE.md).
 
 ## Match quality
 

package/complete/blob-cache.node.mjs

@@ -0,0 +1,337 @@
+// A disk store for openPikeletFile's options.blobCache: bytes a pack commits
+// to by sha256, filed under that digest. Two kinds of entry reach it:
+//
+//   - whole regions read at mount (the sketch's resident prefix, the
+//     query-interp segment holding the encoder): most of what a URL mount
+//     downloads, about 71 of 77 MiB on the 649 MiB Simple English Wikipedia
+//     pack. One file each.
+//   - pieces read per query (rerank vector rows, corpus records, and the
+//     digest pages that verify them): a few hundred bytes to a few KiB, and
+//     hundreds per query. One file each would be a poor fit, so entries under
+//     SMALL_ENTRY_BYTES go into append-only log files (generations) with an
+//     in-memory index, and whole generations rotate out under their own
+//     budget.
+//
+// The store is untrusted by design: the reader hashes every hit against the
+// digest it asked for and treats a mismatch as a miss (blobCacheGet in
+// pikelet-artifact-common.js), so a torn write, a corrupted file, or two
+// processes appending to one log at once costs a refetch, never a wrong byte.
+// Entries are immutable and never go stale; the only policy is size.
+
+import fs from 'node:fs/promises';
+import os from 'node:os';
+import path from 'node:path';
+
+// A full sha256, or a truncated one of at least 128 bits (sketch row digests
+// may be 16-32 bytes).
+const KEY = /^(?:[0-9a-f]{2}){16,32}$/;
+export const DEFAULT_BLOB_CACHE_BYTES = 2 * 1024 * 1024 * 1024;
+export const DEFAULT_SMALL_CACHE_BYTES = 64 * 1024 * 1024;
+// Entries below this go to the log; at or above it, a file of their own.
+export const SMALL_ENTRY_BYTES = 64 * 1024;
+// The log keeps this many generations; the oldest is deleted whole when the
+// newest fills (smallMaxBytes / GENERATIONS each).
+const GENERATIONS = 4;
+// Log record: magic 'PKB1' (u32 LE), key length (u8), key bytes, data length
+// (u32 LE), data. A scan stops at the first record that does not parse.
+const RECORD_MAGIC = 0x3142_4b50;
+const GEN_FILE = /^gen-(\d{8})\.log$/;
+// Entries are pack content, record text included, and a pack can be a
+// private repository or a conversation history: the store is private to its
+// owner whatever the umask.
+const DIR_MODE = 0o700;
+const FILE_MODE = 0o600;
+
+// $PIKELET_CACHE_DIR, else $XDG_CACHE_HOME/pikelet/blobs, else
+// ~/.cache/pikelet/blobs: the same root the CLI already caches encoder
+// weights under.
+export function defaultBlobCacheDir(env = process.env) {
+    if (env.PIKELET_CACHE_DIR) return path.resolve(env.PIKELET_CACHE_DIR, 'blobs');
+    const base = env.XDG_CACHE_HOME ? path.resolve(env.XDG_CACHE_HOME) : path.join(os.homedir(), '.cache');
+    return path.join(base, 'pikelet', 'blobs');
+}
+
+export function diskBlobCache(dir = defaultBlobCacheDir(), { maxBytes = DEFAULT_BLOB_CACHE_BYTES, smallMaxBytes = DEFAULT_SMALL_CACHE_BYTES } = {}) {
+    for (const [label, value] of [['maxBytes', maxBytes], ['smallMaxBytes', smallMaxBytes]]) {
+        if (!Number.isSafeInteger(value) || value < 0) {
+            throw new Error(`diskBlobCache ${label} must be a non-negative integer`);
+        }
+    }
+    const root = path.resolve(dir);
+    const checkKey = (key) => {
+        if (typeof key !== 'string' || !KEY.test(key)) {
+            throw new Error(`blob cache key must be a lowercase sha256 hex digest (32-64 hex digits), got ${JSON.stringify(key)}`);
+        }
+        return key;
+    };
+    const fileOf = (key) => path.join(root, key.slice(0, 2), checkKey(key));
+    // Once per instance, before the first write: a root made by an older
+    // store (or by mkdir -p under umask 022) is tightened, which also closes
+    // off whatever it already holds. Best effort; on Windows modes are moot.
+    let privateRoot = null;
+    const makeRootPrivate = () => (privateRoot ??= (async () => {
+        await fs.mkdir(root, { recursive: true, mode: DIR_MODE });
+        await fs.chmod(root, DIR_MODE).catch(() => {});
+    })());
+    const pending = new Set();
+    const track = (promise) => {
+        pending.add(promise);
+        promise.finally(() => pending.delete(promise)).catch(() => {});
+        return promise;
+    };
+
+    const evict = async (keep) => {
+        const entries = [];
+        let total = 0;
+        for (const shard of await fs.readdir(root).catch(() => [])) {
+            if (shard === 'small') continue;
+            const shardDir = path.join(root, shard);
+            for (const name of await fs.readdir(shardDir).catch(() => [])) {
+                if (!KEY.test(name)) continue;
+                const file = path.join(shardDir, name);
+                const st = await fs.stat(file).catch(() => null);
+                if (!st) continue;
+                total += st.size;
+                entries.push({ file, size: st.size, mtime: st.mtimeMs });
+            }
+        }
+        entries.sort((a, b) => a.mtime - b.mtime);
+        for (const entry of entries) {
+            if (total <= maxBytes) break;
+            if (entry.file === keep) continue;
+            await fs.rm(entry.file, { force: true });
+            total -= entry.size;
+        }
+    };
+
+    // --- the small-entry log -------------------------------------------------
+    const smallDir = path.join(root, 'small');
+    const genBytes = Math.max(1, Math.floor(smallMaxBytes / GENERATIONS));
+    const genPath = (gen) => path.join(smallDir, `gen-${String(gen).padStart(8, '0')}.log`);
+    const index = new Map(); // key -> { gen, offset, length }
+    const handles = new Map(); // gen -> Promise<FileHandle> (read)
+    let current = 0;
+    // Set when the newest generation ends in bytes that do not parse (a torn
+    // write): a scan stops there, so anything appended after it would never
+    // be indexed again. Writing resumes in a fresh generation instead.
+    let sealed = false;
+    let queue = [];
+    let flushing = null;
+
+    const listGens = async () => (await fs.readdir(smallDir).catch(() => []))
+        .map((name) => GEN_FILE.exec(name)).filter(Boolean).map((m) => Number(m[1])).sort((a, b) => a - b);
+
+    const forgetGen = async (gen) => {
+        for (const [key, at] of index) if (at.gen === gen) index.delete(key);
+        const handle = handles.get(gen);
+        handles.delete(gen);
+        if (handle) await handle.then((h) => h.close(), () => {});
+    };
+
+    const scan = (gen, bytes) => {
+        const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+        let at = 0;
+        while (at + 9 <= bytes.length && view.getUint32(at, true) === RECORD_MAGIC) {
+            const keyLen = bytes[at + 4];
+            if (keyLen < 16 || keyLen > 32 || at + 5 + keyLen + 4 > bytes.length) break;
+            const key = Buffer.from(bytes.subarray(at + 5, at + 5 + keyLen)).toString('hex');
+            const length = view.getUint32(at + 5 + keyLen, true);
+            const offset = at + 9 + keyLen;
+            if (offset + length > bytes.length) break;
+            index.set(key, { gen, offset, length });
+            at = offset + length;
+        }
+        return at === bytes.length;
+    };
+
+    // Built once, on first use: every generation still on disk is read and
+    // indexed (at most smallMaxBytes), later generations overriding earlier.
+    let ready = null;
+    const smallReady = () => (ready ??= (async () => {
+        if (smallMaxBytes === 0) return;
+        const gens = await listGens();
+        for (const gen of gens.slice(0, Math.max(0, gens.length - GENERATIONS))) {
+            await fs.rm(genPath(gen), { force: true });
+        }
+        for (const gen of gens.slice(-GENERATIONS)) {
+            const bytes = await fs.readFile(genPath(gen)).catch(() => null);
+            const whole = bytes ? scan(gen, bytes) : true;
+            if (gen === gens[gens.length - 1]) sealed = !whole;
+        }
+        current = gens.length ? gens[gens.length - 1] : 0;
+    })());
+
+    const readSmall = async (key) => {
+        const at = index.get(key);
+        if (!at) return null;
+        let handle = handles.get(at.gen);
+        if (!handle) {
+            handle = fs.open(genPath(at.gen), 'r');
+            handles.set(at.gen, handle);
+            handle.catch(() => { if (handles.get(at.gen) === handle) handles.delete(at.gen); });
+        }
+        let fh;
+        try { fh = await handle; } catch (err) {
+            // Another process rotated this generation away.
+            if (err?.code === 'ENOENT') { await forgetGen(at.gen); return null; }
+            throw err;
+        }
+        const out = new Uint8Array(at.length);
+        const { bytesRead } = await fh.read(out, 0, at.length, at.offset);
+        if (bytesRead !== at.length) { index.delete(key); return null; }
+        // A hit from an older generation is written again to the newest, so
+        // rotation drops what has not been read lately: LRU by generation.
+        if (at.gen !== current) enqueue(key, out);
+        return out;
+    };
+
+    const rotate = async () => {
+        await makeRootPrivate();
+        await fs.mkdir(smallDir, { recursive: true, mode: DIR_MODE });
+        const gens = await listGens();
+        current = Math.max(current, gens.length ? gens[gens.length - 1] : 0) + 1;
+        // Exclusive create; another process that rotated first owns the
+        // name, and appending after its records is fine.
+        await fs.writeFile(genPath(current), new Uint8Array(0), { flag: 'wx', mode: FILE_MODE }).catch((err) => {
+            if (err?.code !== 'EEXIST') throw err;
+        });
+        for (const gen of (await listGens()).filter((g) => g <= current - GENERATIONS)) {
+            await forgetGen(gen);
+            await fs.rm(genPath(gen), { force: true });
+        }
+    };
+
+    const flushSmall = async () => {
+        while (queue.length) {
+            const batch = queue;
+            queue = [];
+            const total = batch.reduce((n, r) => n + r.record.length, 0);
+            if (!current || sealed) {
+                await rotate();
+                sealed = false;
+            } else {
+                const st = await fs.stat(genPath(current)).catch(() => null);
+                if (!st || st.size + total > genBytes) await rotate();
+            }
+            const buf = Buffer.concat(batch.map((r) => r.record));
+            const fh = await fs.open(genPath(current), 'a+', FILE_MODE);
+            let base;
+            try {
+                await fh.write(buf);
+                // O_APPEND puts the batch at the end; its offset is the size
+                // now minus its length, unless another process appended
+                // between the write and the stat. So the batch is read back
+                // there before it is indexed: records are content-addressed,
+                // so wherever these exact bytes sit is a right offset.
+                base = (await fh.stat()).size - buf.length;
+                if (base >= 0) {
+                    const back = Buffer.alloc(buf.length);
+                    const { bytesRead } = await fh.read(back, 0, buf.length, base);
+                    if (bytesRead !== buf.length || !back.equals(buf)) base = -1;
+                }
+            } finally {
+                await fh.close();
+            }
+            if (base < 0) {
+                // The end moved: find where the batch landed by scanning the
+                // generation, which indexes the other writer's records too.
+                // A record still being written stops the scan; whatever lies
+                // past it is a miss, not a wrong offset.
+                const bytes = await fs.readFile(genPath(current)).catch(() => null);
+                if (bytes) scan(current, bytes);
+                continue;
+            }
+            let at = base;
+            for (const r of batch) {
+                index.set(r.key, { gen: current, offset: at + r.headerBytes, length: r.length });
+                at += r.record.length;
+            }
+        }
+    };
+
+    const enqueue = (key, bytes) => {
+        if (index.has(key) && index.get(key).gen === current) return;
+        const raw = Buffer.from(key, 'hex');
+        const headerBytes = 9 + raw.length;
+        const record = Buffer.alloc(headerBytes + bytes.byteLength);
+        record.writeUInt32LE(RECORD_MAGIC, 0);
+        record[4] = raw.length;
+        raw.copy(record, 5);
+        record.writeUInt32LE(bytes.byteLength, 5 + raw.length);
+        record.set(bytes, headerBytes);
+        queue.push({ key, record, headerBytes, length: bytes.byteLength });
+        // One append per burst: a query stores hundreds of rows at once.
+        flushing ??= track(new Promise((resolve) => setImmediate(resolve))
+            .then(() => flushSmall())
+            .finally(() => { flushing = null; }));
+        return flushing;
+    };
+
+    return {
+        dir: root,
+        async get(key) {
+            checkKey(key);
+            if (smallMaxBytes > 0) {
+                await smallReady();
+                const small = await readSmall(key);
+                if (small) return small;
+            }
+            const file = fileOf(key);
+            let bytes;
+            try { bytes = await fs.readFile(file); } catch (err) {
+                if (err?.code === 'ENOENT') return null;
+                throw err;
+            }
+            // Touch for LRU; best effort.
+            const now = new Date();
+            fs.utimes(file, now, now).catch(() => {});
+            return new Uint8Array(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+        },
+        put(key, bytes) {
+            checkKey(key);
+            if (bytes.byteLength < SMALL_ENTRY_BYTES) {
+                if (smallMaxBytes === 0 || bytes.byteLength > genBytes) return Promise.resolve();
+                return track(smallReady().then(() => enqueue(key, bytes)));
+            }
+            const file = fileOf(key);
+            return track((async () => {
+                if (bytes.byteLength > maxBytes) return;
+                await makeRootPrivate();
+                await fs.mkdir(path.dirname(file), { recursive: true, mode: DIR_MODE });
+                // Write-then-rename, so a reader in another process sees
+                // either no entry or a whole one.
+                const tmp = `${file}.${process.pid}.${Math.random().toString(36).slice(2)}.tmp`;
+                try {
+                    await fs.writeFile(tmp, bytes, { mode: FILE_MODE });
+                    await fs.rename(tmp, file);
+                } catch (err) {
+                    await fs.rm(tmp, { force: true });
+                    throw err;
+                }
+                await evict(file);
+            })());
+        },
+        async delete(key) {
+            checkKey(key);
+            // A log entry cannot be removed in place; forgetting it is enough,
+            // and rotation reclaims the bytes.
+            index.delete(key);
+            await fs.rm(fileOf(key), { force: true });
+        },
+        // Settles every write started so far: for tests, and for a process
+        // that wants its cache populated before it exits.
+        async flush() {
+            while (pending.size) await Promise.allSettled([...pending]);
+        },
+        // Closes the log's read handles (they otherwise live as long as the
+        // process). The store stays usable; handles reopen on demand.
+        async close() {
+            await this.flush();
+            for (const gen of [...handles.keys()]) {
+                const handle = handles.get(gen);
+                handles.delete(gen);
+                await handle.then((h) => h.close(), () => {});
+            }
+        },
+    };
+}

package/complete/index.d.ts

@@ -96,6 +96,32 @@ export declare function nodeFetch(input: string, init?: {
   inflateGzip?: boolean;
 }): Promise<any>;
 
+/** openPikeletFile's options.blobCache: bytes keyed by their sha256 (lowercase hex, 32-64 digits). */
+export interface BlobCache {
+  get(key: string): Promise<Uint8Array | null> | Uint8Array | null;
+  put(key: string, bytes: Uint8Array): Promise<void> | void;
+  delete?(key: string): Promise<void> | void;
+}
+
+/**
+ * Node only (the 'node' export condition): a BlobCache on disk under `dir`
+ * (default defaultBlobCacheDir()). Entries of 64 KiB or more get a file each,
+ * evicting least recently used past maxBytes (default 2 GiB). Smaller ones
+ * (per-query rows, records, digest pages) go to append-only log generations
+ * capped at smallMaxBytes in total (default 64 MiB; 0 disables them), the
+ * oldest generation dropped whole. flush() settles pending writes; close()
+ * releases the log's file handles.
+ */
+export declare function diskBlobCache(dir?: string, options?: { maxBytes?: number; smallMaxBytes?: number }): BlobCache & {
+  dir: string;
+  flush(): Promise<void>;
+  close(): Promise<void>;
+};
+/** $PIKELET_CACHE_DIR/blobs, else $XDG_CACHE_HOME/pikelet/blobs, else ~/.cache/pikelet/blobs. */
+export declare function defaultBlobCacheDir(env?: Record<string, string | undefined>): string;
+export declare const DEFAULT_BLOB_CACHE_BYTES: number;
+export declare const DEFAULT_SMALL_CACHE_BYTES: number;
+
 export interface CompleteQueryResult {
   /**
    * The pack's calibrated verdict on this retrieval. On packs calibrated
@@ -287,6 +313,21 @@ export declare function openPikeletFile(
      * true.
      */
     prefetchEncoder?: boolean;
+    /**
+     * A store for bytes a pack commits to by sha256. At mount: the sketch's
+     * resident prefix and the query-interp segment holding the inline
+     * encoder, which dominate a cold URL mount. Per query: rerank vector
+     * rows (format 2), corpus records, and the digest pages that verify
+     * both. Keys are those digests in lowercase hex (a row's digest may be
+     * truncated to 16-31 bytes), so entries never go stale. Every hit is
+     * hashed and must match its key (a prefix, for a truncated key); a
+     * mismatch is treated as a miss and deleted, so the store need not be
+     * trusted, and get/put/delete faults never fail a read. Only reads that
+     * are verified on their own use it: nothing when verification is off,
+     * and never the lazy lexical index. diskBlobCache() (Node) is a ready
+     * implementation.
+     */
+    blobCache?: BlobCache;
     /**
      * Fully verify the index's lazy vector rows at open (one streamed pass
      * against the identity-anchored vectorsSha256). Default false: without

package/complete/index.mjs

@@ -27,6 +27,7 @@ import { createAbstentionScorer } from './retrieval-abstention.mjs';
 import { openLexicalIndex, openLexicalIndexLazy } from './lexical.mjs';
 import { parseRowMapSegment } from './rowmap.mjs';
 import { PikeletSketchArtifact } from '../src/artifact/pikelet-artifact.js';
+import { blobCacheGet, blobCachePut } from '../src/artifact/pikelet-artifact-common.js';
 import { MAGIC, HEADER_BYTES, TABLE_ENTRY_BYTES, KINDS, KIND_NAMES } from './format.mjs';
 
 import {
@@ -96,9 +97,9 @@ const COVERAGE_BATCH_COUNT = 5;
 // Only BM25 hits within this fraction of the top lexical score join fusion.
 // Common query terms give every matching document a near-tied score (idf
 // collapses them flat); a wide cutoff let that tied mass ride into fusion on
-// a technicality. Kept identical to pikelet/src/calibrate.mjs's
-// LEXICAL_CUTOFF so calibration is fit against the same fusion the reader
-// actually serves.
+// a technicality. Calibration (pikelet/src/calibrate.mjs) gathers its
+// evidence through this reader's own retrieval, so it is fit against the
+// same cutoff and fusion the reader serves.
 export const LEXICAL_CUTOFF = 1.5;
 // Query-interpretation segments above this size open lazily (kind 3's
 // inline encoder is ~25 MiB); smaller ones keep the one-read eager path.
@@ -785,6 +786,7 @@ export async function openPikeletFile(input, options = {}) {
             // the same bytes the sketch parses (see expectedHeaderSha256).
             PikeletSketchArtifact.open(windowSource(source, idx.offset, idx.length), {
                 maxReadBytes,
+                ...(options.blobCache ? { blobCache: options.blobCache } : {}),
                 ...(perRecord ? { expectedHeaderSha256: manifest.index.headerSha256 } : {}),
             }),
             readChecked(source, corpus.offset, tablesBytes, 'corpus tables', maxReadBytes, fileBytes),
@@ -1004,9 +1006,16 @@ export async function openPikeletFile(input, options = {}) {
             const loadEmbedder = async () => {
                 let encBytes = encoderBytes;
                 if (qiDeferred) {
-                    const full = await readChecked(source, qi.offset, qi.length, 'query-interp segment', maxReadBytes, fileBytes);
-                    if (await sha256hex(full) !== qi.sha256) {
-                        throw new Error('.pikelet query-interp segment failed hash verification');
+                    // The ~25 MiB encoder is the other bulk of a cold mount;
+                    // options.blobCache serves it by the manifest's segment
+                    // digest (hash-checked on the way out, as a fetch is).
+                    const cached = await blobCacheGet(options.blobCache, qi.sha256, qi.length);
+                    const full = cached || await readChecked(source, qi.offset, qi.length, 'query-interp segment', maxReadBytes, fileBytes);
+                    if (!cached) {
+                        if (await sha256hex(full) !== qi.sha256) {
+                            throw new Error('.pikelet query-interp segment failed hash verification');
+                        }
+                        blobCachePut(options.blobCache, qi.sha256, full);
                     }
                     // The header and calibration slices used at open were
                     // unverified; the digest-verified segment must contain
@@ -1191,12 +1200,16 @@ export async function openPikeletFile(input, options = {}) {
             const pending = (async () => {
                 const from = Math.min(recordCount, page * pageRecords);
                 const to = Math.min(recordCount, (page + 1) * pageRecords);
+                const key = toHex(pageTable.subarray(32 * page, 32 * page + 32));
+                const cached = await blobCacheGet(options.blobCache, key, 32 * (to - from));
+                if (cached) return cached;
                 const bytes = await readChecked(source, corpus.offset + recordDigestsAt + 32 * from, 32 * (to - from),
                     `corpus digest page ${page}`, maxReadBytes, fileBytes);
-                if (await sha256hex(bytes) !== toHex(pageTable.subarray(32 * page, 32 * page + 32))) {
+                if (await sha256hex(bytes) !== key) {
                     digestPages.delete(page);
                     throw new Error(`.pikelet corpus digest page ${page} failed hash verification`);
                 }
+                blobCachePut(options.blobCache, key, bytes);
                 return bytes;
             })();
             digestPages.set(page, pending);
@@ -1225,14 +1238,29 @@ export async function openPikeletFile(input, options = {}) {
             }
             const start = recordOffsets[id];
             const end = recordOffsets[id + 1];
-            const bytes = await readChecked(source, corpus.offset + start, end - start, `corpus record ${id}`, maxRecordBytes, fileBytes);
-            if (perRecord && verifyRecords) {
+            // With options.blobCache a verified record is keyed by its own
+            // digest, so the digest comes first; another process may already
+            // have fetched the record. Only verified reads use the cache.
+            const cacheable = perRecord && verifyRecords && options.blobCache;
+            let digest = null;
+            if (cacheable) {
                 const page = Math.floor(id / pageRecords);
-                const digests = await pageDigests(page);
                 const at = 32 * (id - page * pageRecords);
-                if (await sha256hex(bytes) !== toHex(digests.subarray(at, at + 32))) {
+                digest = toHex((await pageDigests(page)).subarray(at, at + 32));
+                const cached = await blobCacheGet(options.blobCache, digest, end - start);
+                if (cached) return cached;
+            }
+            const bytes = await readChecked(source, corpus.offset + start, end - start, `corpus record ${id}`, maxRecordBytes, fileBytes);
+            if (perRecord && verifyRecords) {
+                if (!digest) {
+                    const page = Math.floor(id / pageRecords);
+                    const at = 32 * (id - page * pageRecords);
+                    digest = toHex((await pageDigests(page)).subarray(at, at + 32));
+                }
+                if (await sha256hex(bytes) !== digest) {
                     throw new Error(`.pikelet corpus record ${id} failed integrity verification`);
                 }
+                blobCachePut(options.blobCache, digest, bytes);
             }
             return bytes;
         };

package/complete/index.node.mjs

@@ -9,6 +9,7 @@ import { nodeFetch } from './node-fetch.mjs';
 
 export * from './index.mjs';
 export { nodeFetch };
+export { diskBlobCache, defaultBlobCacheDir, DEFAULT_BLOB_CACHE_BYTES, DEFAULT_SMALL_CACHE_BYTES } from './blob-cache.node.mjs';
 
 export function httpRangeSource(url, options = {}) {
     return portableHttpRangeSource(url, { fetch: nodeFetch, ...options });

package/complete/layer-reader.mjs

@@ -369,6 +369,10 @@ export async function openPikeletChain(members, options = {}) {
                     profile: LAYER_PROFILE,
                     identity: chainMembers[chainMembers.length - 1].identity,
                     baseIdentity: base.identity,
+                    // The pack's name is the base's: layers add records, not a
+                    // new name. Without it a host fell back to naming the
+                    // chain after its newest layer's file.
+                    name: opened[0].info().name ?? null,
                     layers: chainMembers.length,
                     searchTiers: searchTiers.length,
                     records: table.rowTotal,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pikelet-wasm",
-  "version": "0.12.0",
+  "version": "0.14.0",
   "description": "Vector search engine and .pikelet artifact readers/builders, in WebAssembly. Compile a corpus into one self-contained, hash-verified search file, or use the underlying HNSW engine directly. Runs in Node.js, browsers, and Cloudflare Workers with no native dependencies.",
   "main": "src/pikelet.js",
   "types": "src/pikelet.d.ts",

package/src/artifact/pikelet-artifact-common.js

@@ -462,6 +462,42 @@ async function sha256BytesAsync(bytes) {
     }
 }
 
+// Read-through cache for bytes a pack commits to by sha256: the whole
+// regions a mount reads (the sketch's resident prefix, a pack's query-interp
+// segment) and the pieces a query reads (rerank rows, corpus records, the
+// digest pages that verify them), keyed by that digest. A key may be a
+// truncated digest of at least 128 bits (sketch row digests are 16-32
+// bytes), and a hit must hash to a digest that starts with it. A hit is
+// hashed exactly as a network read would be, so the cache can only ever hand
+// back the committed bytes: anything else it holds (a partial write, disk
+// corruption, a bug elsewhere) reads as a miss and the entry is dropped.
+// Because the key is the content, entries never go stale and need no
+// invalidation. Cache faults never fail a read. `length`, when given, must
+// match too.
+async function blobCacheGet(cache, digestHex, length) {
+    if (!cache || typeof cache.get !== 'function') return null;
+    if (typeof digestHex !== 'string' || digestHex.length < 32 || digestHex.length > 64) return null;
+    let hit;
+    try {
+        hit = await cache.get(digestHex);
+        if (!hit) return null;
+        hit = asUint8Array(hit);
+    } catch { return null; }
+    const digest = length === undefined || hit.byteLength === length ? await sha256BytesAsync(hit) : null;
+    let hex = '';
+    if (digest) for (const b of digest) hex += b.toString(16).padStart(2, '0');
+    if (hex && hex.startsWith(digestHex)) return hit;
+    try { await cache.delete?.(digestHex); } catch { /* a miss either way */ }
+    return null;
+}
+
+// Store verified bytes. Fire-and-forget: a slow or failing cache must not
+// hold up the open that just fetched them.
+function blobCachePut(cache, digestHex, bytes) {
+    if (!cache || typeof cache.put !== 'function') return;
+    Promise.resolve().then(() => cache.put(digestHex, bytes)).catch(() => {});
+}
+
 // Verify bytes against an expected 32-byte SHA-256 digest. Verification was
 // requested by the caller, so a missing crypto backend fails closed rather
 // than admitting unverified bytes.
@@ -556,6 +592,8 @@ module.exports = {
     parseUint8Snapshot,
     sha256Bytes,
     sha256BytesAsync,
+    blobCacheGet,
+    blobCachePut,
     verifySha256,
     verifySegmentSha256,
 };

package/src/artifact/pikelet-artifact-sketch.js

@@ -17,6 +17,8 @@ const {
     parseUint8Snapshot,
     sha256Bytes,
     sha256BytesAsync,
+    blobCacheGet,
+    blobCachePut,
     verifySegmentSha256,
 } = require('./pikelet-artifact-common.js');
 const {
@@ -28,6 +30,12 @@ const {
 // ============================================================================
 // Sketch artifact profile (.pikelet-sketch) — spec/SKETCH_PROFILE.md
 // ============================================================================
+
+function hexOf(bytes) {
+    let hex = '';
+    for (const b of bytes) hex += b.toString(16).padStart(2, '0');
+    return hex;
+}
 //
 // Layout: 256-byte header | scales f32[count] | offsets f32[count] |
 // packed pooled sketches | zero padding to 16-byte alignment | raw u8 rows.
@@ -463,12 +471,23 @@ class PikeletSketchArtifact {
         // Per-row verification on format 2 follows the same switch as the
         // resident hash: on by default, verify:false opts the whole open out.
         artifact.verifyRows = version >= 2 && verify;
+        // Per-query pieces (digest pages and rows) use the same cache as the
+        // resident prefix, but only where each read is verified on its own.
+        artifact.blobCache = artifact.verifyRows && options.blobCache ? options.blobCache : null;
         const staged = options.staged === true && artifact.microDims > 0;
         artifact.tier = 'full';
         artifact.fullyResident = Promise.resolve(artifact);
 
-        if (!staged) {
-            const resident = await readChecked(source, SKETCH_HEADER_BYTES, vectorsOffset - SKETCH_HEADER_BYTES, 'resident prefix', artifact.maxReadBytes);
+        // options.blobCache: the resident prefix is the bulk of a mount (46 MiB
+        // of a 649 MiB wiki pack) and is committed by residentSha256, so a
+        // cache keyed by that digest serves it to later processes. A hit is
+        // hash-checked inside blobCacheGet; only verified opens consult it,
+        // since verify:false has nothing to check a hit against. A cache hit
+        // also skips staging: the full tier is already local.
+        const cacheKey = verify && options.blobCache ? hexOf(header.subarray(56, 88)) : null;
+        const cached = cacheKey ? await blobCacheGet(options.blobCache, cacheKey, vectorsOffset - SKETCH_HEADER_BYTES) : null;
+        if (!staged || cached) {
+            const resident = cached || await readChecked(source, SKETCH_HEADER_BYTES, vectorsOffset - SKETCH_HEADER_BYTES, 'resident prefix', artifact.maxReadBytes);
             if (resident.byteLength !== vectorsOffset - SKETCH_HEADER_BYTES) {
                 throw pikeletError(PIKELET_ERROR_CODES.SNAPSHOT_INVALID, 'Sketch artifact resident prefix is truncated');
             }
@@ -477,7 +496,9 @@ class PikeletSketchArtifact {
             const residentCopy = new Uint8Array(resident.byteLength);
             residentCopy.set(resident);
             artifact._adoptResident(residentCopy, count, sketchRowBytes, microRowBytes, microOffset);
-            if (verify) await artifact._verifyFullResident(residentCopy, header);
+            if (cached) artifact.residentVerified = true;
+            else if (verify) await artifact._verifyFullResident(residentCopy, header);
+            if (cacheKey && !cached) blobCachePut(options.blobCache, cacheKey, residentCopy);
             return artifact;
         }
 
@@ -763,8 +784,37 @@ class PikeletSketchArtifact {
             }
         }
         if (!missing.length) return rows;
-        missing.sort((a, b) => a - b);
         const { dim, rowsPerBlock, rowDigestBytes, pageBytes, blockBytes, vectorsOffset } = this;
+        // options.blobCache at open: a block's digest page is keyed by its
+        // page-table entry and each row by its digest in that page, so a row
+        // another process already fetched and verified is read from disk.
+        // Every hit is hash-checked against its key (blobCacheGet).
+        const cache = this.blobCache;
+        if (cache) {
+            const byBlock = new Map();
+            for (const id of missing) {
+                const b = Math.floor(id / rowsPerBlock);
+                if (!byBlock.has(b)) byBlock.set(b, []);
+                byBlock.get(b).push(id);
+            }
+            const still = [];
+            await mapLimit([...byBlock], 16, async ([b, ids]) => {
+                const page = await blobCacheGet(cache, hexOf(this.pageTable.subarray(b * 32, b * 32 + 32)), pageBytes);
+                if (!page) { still.push(...ids); return; }
+                for (const id of ids) {
+                    const idx = id - b * rowsPerBlock;
+                    const hit = await blobCacheGet(cache, hexOf(page.subarray(idx * rowDigestBytes, (idx + 1) * rowDigestBytes)), dim);
+                    if (!hit) { still.push(id); continue; }
+                    const row = new Uint8Array(hit);
+                    rows.set(id, row);
+                    this.cacheRow(id, row);
+                }
+            });
+            if (!still.length) return rows;
+            missing.length = 0;
+            missing.push(...still);
+        }
+        missing.sort((a, b) => a - b);
         const blocks = new Map();
         for (const id of missing) {
             const b = Math.floor(id / rowsPerBlock);
@@ -818,6 +868,7 @@ class PikeletSketchArtifact {
                             throw pikeletError(PIKELET_ERROR_CODES.SNAPSHOT_INVALID, 'Sketch artifact row digest page failed hash verification', { block: b });
                         }
                     }
+                    if (cache) blobCachePut(cache, hexOf(expected), page.slice());
                 }
                 for (const id of blocks.get(b).ids) {
                     const idx = id - b * rowsPerBlock;
@@ -830,6 +881,7 @@ class PikeletSketchArtifact {
                                 throw pikeletError(PIKELET_ERROR_CODES.SNAPSHOT_INVALID, 'Sketch artifact row failed digest verification', { id });
                             }
                         }
+                        if (cache) blobCachePut(cache, hexOf(slot), row);
                     }
                     if (rows.has(id)) rows.set(id, row);
                     this.cacheRow(id, row);