pikelet-wasm 0.8.0 → 0.9.0

package/README.md

@@ -8,12 +8,28 @@ A model can interrogate a 456,153-record knowledge base whose backend is a stati
 
 A `.pikelet` can carry the source text, semantic index, keyword index, query encoder, integrity commitments, retrieval calibration, and evaluation fixtures needed to interrogate that corpus. Put the file on disk, S3, R2, a CDN, or any static HTTP host. A reader can mount it locally or over HTTP Range and search it without a vector database, embedding API, or retrieval server.
 
-Requires Node 20+.
+The `pikelet` CLI requires Node 20+; the `pikelet-wasm` library runs on Node 18+ (CI tests 18, 20, 22), browsers, and Cloudflare Workers.
+
+> **Upgrade to `pikelet-wasm@0.8.1` if you mount packs you did not compile.**
+> 0.8.0 and earlier sized the inline encoder's WASM buffers from the
+> artifact's own `maxTokens` with an upper bound only, so a pack declaring an
+> out-of-range value could overflow them — at mount, before any query, and
+> with every integrity check passing, because the pack's author is the one who
+> signed it. 0.8.1 validates the bound. Packs the toolchain produces were
+> never affected. `pikelet@0.8.1` picks the fix up through its dependency
+> range; see the CHANGELOG for the detail.
 
 ```bash
 npx pikelet compile --source ./docs --out docs.pikelet
 ```
 
+Installing rather than using `npx`? Use `npm install -g pikelet
+--omit=optional`. The optional dependency is `@xenova/transformers`, which
+only the `create` scaffold path uses. Measured on a clean project, omitting it
+takes the install from **82 packages / 257 MB / "6 vulnerabilities (5 high, 1
+critical)"** to **2 packages / 1.8 MB / "found 0 vulnerabilities"** — the
+advisories are all inside a dependency `compile` and `mcp` never load.
+
 ```text
 Ingested 3 docs -> 3 chunks
 Embedded 3/3 chunks with inline transformer
@@ -47,7 +63,7 @@ console.log(out.matchQuality, out.results[0]?.title);
 // 'strong' 'Snapshot restore'
 ```
 
-That's the whole loop. `compile` also takes a live URL (`--source https://docs.example.com`) instead of a directory. If you want a deployed search app — a Worker + UI, not a file — use `npx pikelet create` instead; see [`pikelet/README.md`](pikelet/README.md) for the full CLI reference and the tradeoffs between the two. `compile` never needs the scaffold path's `@xenova/transformers` dependency; skip its ~140 MB install with `npm install -g pikelet --omit=optional` if you only need `compile`/`mcp`.
+That's the whole loop. `compile` also takes a live URL (`--source https://docs.example.com`) instead of a directory. If you want a deployed search app — a Worker + UI, not a file — use `npx pikelet create` instead; see [`packages/pikelet/README.md`](packages/pikelet/README.md) for the full CLI reference and the tradeoffs between the two. `create` is the one path that needs the optional `@xenova/transformers` dependency, which is why the install above omits it by default.
 
 Under the hood, the file is one container for everything a reader needs:
 
@@ -82,6 +98,22 @@ Search is the interface. **The file is the knowledge deployment unit.**
 
 A real Pikelet pack contains **456,153 Simple English Wikipedia records** in a **648.5 MiB** artifact.
 
+Try the live browser demo:
+
+```text
+https://pikelet-wiki-playground.pages.dev/
+```
+
+The page is static HTML/JS on Cloudflare Pages. It mounts the pack from R2:
+
+```text
+https://pub-6da2384a3bca4a44b2b2fa29a94cc811.r2.dev/wikipedia.pikelet
+```
+
+There is no `/search`, `/embed`, `/query`, or `/api` backend behind the demo.
+The browser opens the `.pikelet` file over HTTP Range, verifies the artifact,
+loads the embedded query encoder, and shows the byte ranges it reads.
+
 That file was served by a deliberately dumb HTTP server whose only relevant capability was:
 
 ```text
@@ -131,7 +163,7 @@ That is the product in one demonstration:
 
 > **A model is interrogating a 456,153-record knowledge base whose backend is a static file.**
 
-The model may already contain some of these facts in its pretrained parameters. This test demonstrates the retrieval, synthesis, citation, and deployment path; the Veyra ablation below tests whether support changes when evidence is removed from the pack.
+The model may already contain some of these facts in its pretrained parameters. This test demonstrates the retrieval, synthesis, citation, and deployment path; the [Veyra ablation](docs/veyra-ablation.md) tests whether support changes when evidence is removed from the pack.
 
 **Network cost.** In a single persistent session (one mount, five queries, one repeat):
 
@@ -154,97 +186,6 @@ The headless-Claude test above used a fresh process per question, so each invoca
 
 ---
 
-## What happens if the answer is changed or removed from the file?
-
-This is different from asking whether search returns sensible documents. To test whether Pikelet's retrieval-quality signal could form a useful evidence boundary, a synthetic corpus called **Station Veyra Registry** was built. One version contained the fact:
-
-```text
-The Tovash project is housed in Chamber 17.
-```
-
-A second pack was byte-for-byte identical except that the record containing that fact was removed.
-
-```text
-full pack:     matchQuality: strong    confidence: 0.915   → Chamber 17
-ablated pack:  matchQuality: none      confidence: 0.136   → unsupported
-```
-
-After the model had already seen the answer, it was queried against the ablated pack with prompts like "Confirm Tovash is in Chamber 17" and "Tovash project Chamber 17 location." The retrieval result stayed unsupported and the model declined to confirm the location from the pack. A separate, informal session then probed a fresh isolated agent with neutral prompts, leading prompts, authority pressure, invitations to use general knowledge, cross-record distractors, and repeated pressure — six adversarial framings — and it continued distinguishing supported facts from the removed one in every case; that session wasn't captured as a script, so treat it as a described observation rather than a reproducible result.
-
-This does **not** mean Pikelet can prevent an LLM from hallucinating. It means the artifact can expose an explicit evidence boundary that a consuming model can choose to respect — and removing evidence from the artifact changed what that model was able to support from the mounted source.
-
-A second intervention changed only the Tovash location source record from **Chamber 17** to **Chamber 43**, rebuilt the pack, and repeated the same prompt in a fresh session:
-
-```text
-chamber43 pack: matchQuality: strong    confidence: 0.916   → Chamber 43
-```
-
-The model answered **Chamber 43** and cited the same logical source record. With Veyra not mounted at all, the same prompt produced no chamber number and the model declined to guess. Change the evidence and the grounded answer changes with it; remove the evidence source and the answer disappears.
-
-The test is synthetic and intentionally narrow. All three packs and the `matchQuality`/`confidence` numbers above are reproducible: `node examples/05-one-file-search/web/public/reproduce-ablation.mjs`. The retrieval side is scripted; the paired LLM-session claims (declining to confirm the removed fact, answering Chamber 43, declining without the pack mounted) were run separately and aren't reproduced by that script.
-
----
-
-## What does the compression cost?
-
-The artifact architecture is only useful if the compact representation does not quietly destroy retrieval. The retrieval path was decomposed on BEIR rather than evaluated only end-to-end, across four configurations that isolate each transformation:
-
-```text
-A  upstream all-MiniLM-L6-v2, float32 corpus, exhaustive search
-B  Pikelet embedded encoder,  float32 corpus, exhaustive search
-C  Pikelet embedded encoder,  affine-u8 corpus, exhaustive search
-D  Pikelet embedded encoder,  affine-u8 corpus, Pikelet HNSW
-```
-
-Corpus mapping and evaluation config were frozen before the runs. Official BEIR qrels are used; no LLM judges relevance.
-
-| Dataset  | Config                                     |    nDCG@10 |  Recall@10 | Recall@100 | median ms/q | p95 ms/q |
-| -------- | ------------------------------------------ | ---------: | ---------: | ---------: | ----------: | -------: |
-| SciFact  | A — upstream float exhaustive              |     0.6451 |     0.7833 |     0.9250 |         3.6 |      4.0 |
-| SciFact  | B — Pikelet encoder / float exhaustive     |     0.6512 |     0.7942 |     0.9417 |         3.8 |      4.4 |
-| SciFact  | C — Pikelet encoder / affine-u8 exhaustive |     0.6500 |     0.7942 |     0.9417 |         6.9 |      8.7 |
-| SciFact  | D — Pikelet encoder / affine-u8 HNSW       | **0.6500** | **0.7942** | **0.9417** |    **0.15** | **0.24** |
-| NFCorpus | A — upstream float exhaustive              |     0.3159 |     0.1550 |     0.3115 |         2.6 |      3.1 |
-| NFCorpus | B — Pikelet encoder / float exhaustive     |     0.3154 |     0.1511 |     0.3044 |         2.8 |      3.3 |
-| NFCorpus | C — Pikelet encoder / affine-u8 exhaustive |     0.3154 |     0.1511 |     0.3036 |         2.9 |      3.6 |
-| NFCorpus | D — Pikelet encoder / affine-u8 HNSW       | **0.3135** | **0.1482** | **0.3061** |    **0.16** | **0.25** |
-| ArguAna  | A — upstream float exhaustive              |     0.3698 |     0.7653 |     0.9772 |         7.8 |     10.8 |
-| ArguAna  | B — Pikelet encoder / float exhaustive     |     0.3506 |     0.7397 |     0.9801 |         6.1 |      7.3 |
-| ArguAna  | C — Pikelet encoder / affine-u8 exhaustive |     0.3496 |     0.7368 |     0.9801 |         6.4 |      7.5 |
-| ArguAna  | D — Pikelet encoder / affine-u8 HNSW       | **0.3496** | **0.7368** | **0.9808** |    **0.12** | **0.20** |
-
-These results are more useful because they are not uniformly flattering.
-
-**Affine-u8 storage is not the main quality cost** (B→C nDCG@10: SciFact 0.6512→0.6500, NFCorpus 0.3154→0.3154, ArguAna 0.3506→0.3496). **HNSW is similarly close to exhaustive search** at these settings (C→D: SciFact and ArguAna unchanged at reported precision; NFCorpus exposes a small approximation loss, 0.3154→0.3135).
-
-**The largest observed loss is the embedded query encoder on ArguAna**: A→B moves nDCG@10 from 0.3698 to 0.3506, about a 5% relative reduction. Recall@100 actually improves slightly (0.9772→0.9801) — the relevant document is generally still in the candidate set; the degradation is in fine ordering near the top. That limitation isn't hidden: MiniLM is small *because* the goal is to fit the query encoder inside the artifact. It is not state of the art, and the compact implementation is not behaviorally identical to an upstream sentence-transformers runtime on every task.
-
-The benchmark harness, quantization-conformance test, frozen configuration, and raw runs live under `bench/beir/`.
-
----
-
-## Why make knowledge a file?
-
-The conventional retrieval deployment looks something like:
-
-```text
-documents → chunking → embedding service → vector database
-                                          → keyword database
-                                          → retrieval service → application API → agent
-```
-
-That is the correct architecture for many workloads. But it is a lot of machinery when the corpus is fundamentally a release artifact: product documentation, a manual, a source tree, a legal code, a standards corpus, a research collection, a book, a knowledge snapshot, an offline reference set. These datasets often change daily, weekly, monthly, or with releases — not hundreds of times per second.
-
-For those workloads, Pikelet asks a different question:
-
-> **What if retrieval could be compiled ahead of time and distributed with the corpus?**
-
-Then deployment becomes: copy the file, cache it, pin it, put it behind a CDN, mount it by URL. The storage layer does not need to know that the file contains vectors.
-
-It is not a replacement for a mutable vector database. It is an attempt to make **static and slowly changing knowledge deploy like any other artifact**.
-
----
-
 ## Attach a pack to an LLM
 
 `pikelet mcp` exposes one or more packs through the Model Context Protocol:
@@ -261,55 +202,29 @@ A pack mounted with a content hash has a stable identity — `https://example.co
 
 **A mounted pack's content reaches the model as tool output.** `verify_pack` proves the bytes are intact and match their pinned identity; it does not prove the corpus itself is trustworthy. Mounting a pack from a source you don't control is the same trust decision as giving an agent any other untrusted-content tool — treat pack text the way you'd treat search results or fetched web pages, not as instructions.
 
-**Try it with your own questions.** The three Veyra packs from the ablation test above (`examples/05-one-file-search/web/public/veyra*.pikelet`) are committed to this repo, so there's a working MCP mount you can query with anything, not just the fixed prompts above:
+**Try it with your own questions.** The three Veyra packs from the [ablation](docs/veyra-ablation.md) are published as pinned GitHub release assets. Fetch and verify them first:
+
+```bash
+npm run demo:veyra
+```
+
+Then mount all three through Pikelet's MCP server:
 
 ```bash
 claude -p "your question here" \
-  --mcp-config examples/05-one-file-search/web/public/veyra.mcp.json \
+  --mcp-config examples/one-file-search/web/public/veyra.mcp.json \
   --strict-mcp-config \
   --allowedTools "mcp__veyra-demo__search,mcp__veyra-demo__list_packs,mcp__veyra-demo__get_record"
 ```
 
-or drop the flags and just run `claude` from the repo root — Claude Code picks up project-level `.mcp.json` configs automatically, so pointing one at this file (or copying its `veyra-demo` server into your own) gets the same three packs into an interactive session. All three packs share the same 94-ish-record synthetic corpus, so the interesting thing to ask about is Tovash's chamber — the full and Chamber 43 packs will disagree with each other, and the ablated pack won't answer at all.
+For an interactive session, use the same config without `-p`:
 
----
-
-## How a remote query runs
-
-Over a network, what makes search slow is not bytes; it's sequential round trips. Graph traversal is a chain of dependent reads — fetch node, inspect neighbors, fetch the next node, repeat — and across object storage those dependent round trips dominate. Measured against SIFT1M over real network storage, a resident scan plus one batched candidate-fetch phase beat graph traversal by about 5× at equal recall. So the remote artifact path carries no graph. The HNSW graph still exists — it's the in-memory engine, used when the whole index is local and round trips are free; that's the path BEIR Config D measures.
-
-For the remote path:
-
-1. **Resident tier** — a pooled sketch of every semantic row loads once when the artifact opens and stays in memory. For the 456,153-record Wikipedia pack: 648.5 MiB artifact, 51.7 MiB mount fetch. A SIMD kernel scans all of it in milliseconds.
-2. **Candidate selection** — the scan picks the top `C` candidates against the query vector; the lexical BM25 index can contribute additional candidates into the same set. `C` is the only real knob: the compiler measures recall against brute force on held-out queries at build time and writes the operating point into the file, so readers don't have to guess it.
-3. **Batched byte-range fetch** — full affine-u8 rows for the candidate set are fetched in one parallel round, nearby ranges coalesced, rather than a graph-dependent network walk.
-4. **Full rerank** — fetched rows are scored against the float query and verified against their digest.
-5. **Corpus hydration** — only the records needed for returned results are fetched, each carrying record ID, title, section, source, and artifact identity. The reader returns evidence, not anonymous vector IDs.
-
-That architecture is why a 648.5 MiB pack can answer a fresh query while fetching about 0.5–1.1 MiB after warmup.
-
----
-
-## Integrity is part of the read path
-
-A range-readable artifact cannot hash the entire file on every open without defeating the point of range reads, so integrity is layered: the resident structural portion is verified during open; lazily fetched index rows and corpus records carry independent commitments and are checked when read. Bytes a query never touches don't have to cross the network merely to prove the bytes it *did* use were correct.
-
-One exception: a lexical (BM25) segment above 8 MiB opens lazily and is covered only by the manifest's whole-segment digest, not per-read like index rows and corpus records — the same transitional stance format-1 sketch rows carry. A pack large enough for this to apply can have its lexical candidates altered between open and a full verification pass without failing a query.
-
-The failure-mode suite (`bench/range-proof/failure-modes.mjs`) exercises the important cases:
-
-```text
-correct Range server         → mounts, queries successfully
-server ignores Range         → small files fall back to a bounded download;
-                                large files refuse ("host ignores Range and the
-                                file is 680029254 bytes; refusing full download" —
-                                the cap is currently 64 MiB)
-bytes change under a pinned identity → refused; tampered data is not silently served
-truncated Range response     → fetch fails; no partial result is silently
-                                interpreted as valid data
+```bash
+claude \
+  --mcp-config examples/one-file-search/web/public/veyra.mcp.json
 ```
 
-The point is not that static HTTP is magically reliable. The point is that a static artifact can fail in explicit, testable ways.
+All three packs derive from the same small synthetic Station Veyra corpus, with controlled differences in the Tovash evidence. Ask where the Tovash project is housed: the full pack supports Chamber 17, the modified pack supports Chamber 43, and the ablated pack should abstain.
 
 ---
 
@@ -321,94 +236,12 @@ A nearest neighbour is not automatically evidence that a corpus answers a questi
 
 ---
 
-## What's inside a `.pikelet`
-
-```text
-manifest        format versions, segment offsets, artifact identity
-semantic index  resident sketch rows, full affine-u8 rows, row commitments
-corpus          source records, offsets, per-record commitments
-query encoder   WordPiece vocabulary, quantized MiniLM weights, encoder declaration
-lexical index   BM25 postings
-calibration     supported / unsupported retrieval model
-evaluation      golden queries / expected behavior
-```
-
-The artifact is immutable. Publish a new corpus by publishing a new artifact; old artifacts retain their identity.
-
-**Encoder profiles.** Self-containment has a cost — the default profile carries ~25 MiB of MiniLM data regardless of corpus size. Three arrangements: **inline** (tokenizer + weights + runtime in the artifact — largest file, no external dependency, used by the Wikipedia demo above), **distilled/compact** (smaller corpus-specific representation, lower footprint, potentially lower quality), **host-supplied** (the artifact declares expected encoder behavior and verifies the host implementation against embedded test vectors — smallest artifact, no longer fully self-contained). The format treats the encoder as a capability, not Pikelet's identity — MiniLM is the current choice, not a permanent requirement.
-
----
-
-## One decision
-
-Most of the implementation follows from how a vector is stored. Each row is 8-bit integers plus two floats:
-
-```text
-x[d] ≈ offset + scale · q[d]        q[d] ∈ 0..255
-```
-
-That's a per-row affine map. It costs 4× less memory than float32, which is the ordinary reason to do it. The reason it runs through the whole project is that every operation search needs — dot products, sums, averages — is linear, and affine terms factor out of linear operations.
-
-**Query against a stored row.** The query stays float32 and is never quantized.
-
-```text
-y·x = offset·Σy + scale·(y·q)
-```
-
-The inner loop multiplies floats by bytes. The two constants come in once at the end. No decompressed copy of the row ever exists.
-
-**Stored row against stored row.** This is what graph construction needs, thousands of times per insert.
-
-```text
-x_i·x_j = D·o_i·o_j + o_i·s_j·Σq_j + o_j·s_i·Σq_i + s_i·s_j·(q_i·q_j)
-```
-
-`D` is the vector's dimensionality (384 for the bundled encoder). `q_i·q_j` is an integer dot over two byte arrays, which maps efficiently onto SIMD. `Σq` is stored per row. The HNSW graph is built and repaired entirely on compressed data.
-
-**Pooling.** Average adjacent groups of `p` bytes and you get a shorter row. Because the mean of `offset + scale·q` over a group equals `offset + scale·mean(q)`, the shorter row keeps the *same two constants*. That's the resident sketch tier; a micro tier is the same thing done twice.
-
-**Weights.** The bundled query encoder is a 6-layer MiniLM whose matrices are stored the same way, one (scale, offset) per 64-column block. Activations stay float32, weights stay bytes, dequantization happens inside the matmul — the same widen-and-multiply trick, a separate kernel, that scores vectors.
-
-The consequence for the file format: a row decodes from its own bytes and its own two floats and nothing else. No codebook, no global statistics. Every row is a fixed-size byte range at a computable offset — locatable by arithmetic, fetchable on its own, hashable on its own, verifiable on the read that fetches it.
-
-**Why not product quantization.** Product quantization would compress harder. It would also need a codebook, a training pass, table lookups in the distance kernel, and rows that mean nothing without the codebook. Four-to-one with no shared state was the better trade for a file meant to be read in pieces.
-
----
-
-## Build once, publish anywhere
+## Deeper dives
 
-```bash
-npx pikelet compile --source ./docs --out docs.pikelet
-```
-
-```js
-import { openPikeletFile } from 'pikelet-wasm/complete';
-
-const pack = await openPikeletFile('https://example.com/docs.pikelet#<sha256>');
-const result = await pack.query('how do workers restore snapshots', { k: 5 });
-
-console.log(result.matchQuality);
-for (const hit of result.results) console.log(hit.title, hit.section);
-
-await pack.close();
-```
-
-The same artifact can also be opened from a local path or any custom source implementing `{ size, read(offset, length) }`.
-
-**Or use the search engine directly**, if you already have vectors and don't need the artifact layer:
-
-```js
-import Pikelet from 'pikelet-wasm';
-
-const index = await Pikelet.create({ dim: 384, maxElements: 100000, metric: 'cosine', quantized: true });
-index.add(vector);
-const results = index.search(query, 10);
-const snapshot = index.export();
-```
-
-float32 HNSW, affine-u8 HNSW, insert/delete, compaction, import/export, deterministic snapshots, WASM with no native addon dependency. Pikelet is usable as a vector engine — that's just no longer the main reason the project exists.
-
----
+- [How a `.pikelet` query runs](docs/how-a-query-runs.md) — what is inside the file, what a query costs on the wire, remote execution, and the integrity checks on the read path.
+- [The Veyra ablation](docs/veyra-ablation.md) — what happens to answers when the supporting record is edited or removed from the pack, with the reproduction commands.
+- [Why make knowledge a file](docs/why-a-file.md) — the reasoning behind the single-artifact design, and what it commits to.
+- [Architecture](docs/architecture.md) — the engine, the C ABI and the JavaScript wrapper; the artifact formats are specified in [`spec/`](spec/).
 
 ## What this is not
 
@@ -422,21 +255,6 @@ float32 HNSW, affine-u8 HNSW, insert/delete, compaction, import/export, determin
 
 ---
 
-## Why the file format matters
-
-A retrieval service is identified by an endpoint. A Pikelet is identified by its contents.
-
-- **Reproducibility** — pack hash + record ID identifies the exact evidence a model retrieved.
-- **Versioning** — different releases are different artifacts (`docs-v1.pikelet`, `docs-v2.pikelet`); no silent mutation.
-- **Distribution** — a pack can be mirrored by infrastructure that knows nothing about semantic search.
-- **Offline use** — once local, no network or embedding service needed.
-- **Customer-controlled knowledge** — hand someone the artifact instead of granting access to internal retrieval infrastructure.
-- **Agent knowledge environments** — run a task against a frozen snapshot, reproduce it later against the same bytes.
-
-> **Can knowledge become a first-class software artifact rather than something that always has to live behind a service?**
-
----
-
 ## When Pikelet is a good fit — and when it isn't
 
 **Use it when:**
@@ -468,23 +286,37 @@ The resident remote scan is linear in row count. The current architecture target
 ## Repository map
 
 ```text
-src/                              C++/WASM vector engine: HNSW, float32 and
+packages/pikelet-wasm/            The published pikelet-wasm package:
+  engine/                          C++ vector engine (HNSW, float32 and
                                    affine-u8 backends, mutation, compaction,
-                                   snapshot import/export
-complete/, pikelet-artifact.js    Readers and builders for the complete
-                                   range-readable artifact
-pikelet/                          CLI, compiler, MCP server, encoder
-                                   integration, higher-level tooling
+                                   snapshot import/export), compiled to WASM
+                                   in dist/. Used at compile time to build and
+                                   quantize the index; query time does not load
+                                   it (see complete/index.mjs's header comment)
+                                   — .pikelet reads run a pure-JS sketch scan,
+                                   optionally SIMD-accelerated.
+  src/                             Engine wrapper and artifact readers/builders
+                                   (the entrypoints, core/, artifact/, errors/)
+  complete/                        Reader and builder for the complete
+                                   range-readable .pikelet artifact
+  native/                          N-API build of the same engine (benchmarks)
+packages/pikelet/                 The published pikelet CLI: compiler, MCP
+                                   server, encoder integration, Docusaurus plugin
+docs/                             Deeper dives (how a query runs, the Veyra
+                                   ablation, why a file), architecture notes,
+                                   rename history, measurement reports
 spec/                             Byte-level artifact contracts
-bench/beir/                       Frozen BEIR ablation harness (encoder,
+benchmarks/beir/                       Frozen BEIR ablation harness (encoder,
                                    quantization, HNSW quality)
-bench/range-proof/                The deliberately boring static-HTTP proof:
+benchmarks/range-proof/                The deliberately boring static-HTTP proof:
                                    dumb-server.mjs, proof.mjs, mcp-proof.mjs,
                                    llm-proof.mjs, failure-modes.mjs
-examples/05-one-file-search/      Large single-artifact search and embedded
+examples/one-file-search/      Large single-artifact search and embedded
                                    encoder work
-examples/06-mcp-knowledge-pack/   Compile, mount, search, and hydrate
+examples/mcp-knowledge-pack/   Compile, mount, search, and hydrate
                                    records through MCP
+local-packs/                      Prebuilt example .pikelet artifacts used
+                                   by the demo commands above
 packs/                            Pack hosting and distribution examples
 ```
 
@@ -492,9 +324,9 @@ packs/                            Pack hosting and distribution examples
 
 ## Reproduce the proofs
 
-**Range proof** (`bench/range-proof/`) is built around a server that does not understand Pikelet. It checks: static HTTP + Range + Pikelet client = remote semantic retrieval, without a full download. The main script reports artifact size, records, mount bytes/requests, per-query bytes/requests, cache behavior. `failure-modes.mjs` independently exercises Range supported / ignored / tampered bytes / truncated response. `mcp-proof.mjs` and `llm-proof.mjs` use a real MCP client and a real headless Claude process rather than a mocked model response.
+**Range proof** (`benchmarks/range-proof/`) is built around a server that does not understand Pikelet. It checks: static HTTP + Range + Pikelet client = remote semantic retrieval, without a full download. The main script reports artifact size, records, mount bytes/requests, per-query bytes/requests, cache behavior. `failure-modes.mjs` independently exercises Range supported / ignored / tampered bytes / truncated response. `mcp-proof.mjs` and `llm-proof.mjs` use a real MCP client and a real headless Claude process rather than a mocked model response.
 
-**BEIR evaluation** (`bench/beir/`) keeps stages separate so a quality change can be attributed to the component that caused it, plus a standalone conformance fixture checking the JS benchmark path against the actual C++ affine-u8 representation. Don't read the latency columns as a universal native-performance comparison — they measure the benchmark paths under the stated harness. The quality deltas are the important part.
+**BEIR evaluation** (`benchmarks/beir/`) keeps stages separate so a quality change can be attributed to the component that caused it, plus a standalone conformance fixture checking the JS benchmark path against the actual C++ affine-u8 representation. Don't read the latency columns as a universal native-performance comparison — they measure the benchmark paths under the stated harness. The quality deltas are the important part.
 
 ---
 
@@ -504,7 +336,7 @@ Pikelet is early. The implementation is real; the format is not frozen. One prim
 
 - prebuilt WASM included
 - local and HTTP readers
-- affine-u8 and float HNSW backends
+- affine-u8 and float HNSW backends for compile-time indexing/quantization
 - embedded query encoder
 - BM25 hybrid retrieval
 - content identity and lazy-read integrity verification
@@ -516,18 +348,10 @@ Pikelet is early. The implementation is real; the format is not frozen. One prim
 
 `npm test` exercises the engine, artifact profiles, MCP, ingestion, format hardening, and related conformance suites.
 
-The project was renamed from **Pancake** to **Pikelet** in September 2026; some old fixture filenames and compatibility identifiers still use the previous name. `pancake-wasm` and `create-pancake-search` are deprecated compatibility pointers to the Pikelet packages. Pikelet is unrelated to the pre-existing Pikelet programming language.
+Previously known as **Pancake** (renamed September 2026); artifacts, profile strings and files from before the rename remain readable — see [`docs/history.md`](docs/history.md) for what kept the old name and why. Pikelet is unrelated to the pre-existing Pikelet programming language.
 
 ---
 
-## The experiment
-
-> **For static and slowly changing corpora, useful semantic retrieval can be compiled into the artifact being distributed instead of operated as a separate service.**
-
-The implementation currently demonstrates: one file, 456,153 records, 648.5 MiB, embedded query encoder, semantic + lexical retrieval, content identity, per-record integrity, MCP mounting, static HTTP hosting, ~0.5–1.1 MiB fresh-query range traffic after warmup, real multi-record LLM synthesis, no retrieval backend.
-
-There are many reasons a database remains the right answer. Pikelet exists for the cases where the knowledge itself should be something you can **build, hash, copy, cache, publish, mount, query, and keep.**
-
 ## License
 
 Apache-2.0. See [LICENSE](LICENSE).

package/complete/builder.mjs

@@ -81,8 +81,25 @@ export function buildQueryInterpSegment(kind, encoderBytes, calibrationBytes) {
 // and skips a small function-word stopword list — readers need no
 // coordination with the list, because an unindexed query token simply finds
 // no postings.
+//
+// Interrogative words (how/does/what/who/why/where/when/which/do/did) are
+// deliberately in this list, not just the base one: without them, a
+// natural-language question's own scaffolding becomes real, indexable BM25
+// content — a passage containing literal dialogue like "How does Georgiana
+// get on, Darcy?" then wins a high lexical score against the query "How
+// does Elizabeth initially react to Mr. Darcy?" purely from "how does" and
+// "darcy" co-occurring, with zero topical relevance. Confirmed live: this
+// pushed a genuinely correct chunk (ranked #2 by the vector search alone)
+// completely out of the fused top-10, replaced by RRF-boosted noise. Not
+// borrowed wholesale from calibrate.mjs's COVERAGE_STOPWORDS, which also
+// excludes domain words ("tell", "explain", "overview", "history",
+// "definition") that are legitimate lexical CONTENT on many real corpora
+// (a docs site literally titled "History") — this list only adds the
+// unambiguous interrogative set, since the lexical index (unlike coverage)
+// also indexes passage text, not just the query.
 const LEXICAL_LAYOUT = 'bm25-v1';
-const LEXICAL_STOPWORDS = new Set(['the', 'a', 'an', 'and', 'or', 'but', 'of', 'to', 'in', 'on', 'at', 'by', 'for', 'with', 'from', 'into', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'it', 'its', 'this', 'that', 'these', 'those', 'as', 'their', 'they', 'them', 'his', 'her', 'has', 'have', 'had', 'not', 'can', 'will', 'if', 'we', 'you', 'your', 'i']);
+const LEXICAL_STOPWORDS = new Set(['the', 'a', 'an', 'and', 'or', 'but', 'of', 'to', 'in', 'on', 'at', 'by', 'for', 'with', 'from', 'into', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'it', 'its', 'this', 'that', 'these', 'those', 'as', 'their', 'they', 'them', 'his', 'her', 'has', 'have', 'had', 'not', 'can', 'will', 'if', 'we', 'you', 'your', 'i',
+  'how', 'does', 'did', 'do', 'what', 'who', 'why', 'where', 'when', 'which']);
 const lexTokenize = (text) => (String(text).toLowerCase().match(/[a-z0-9']+/g) || [])
   .filter((w) => w.length >= 2 && w.length <= 32 && !LEXICAL_STOPWORDS.has(w));
 
@@ -439,16 +456,29 @@ function selfQueriesFromSnapshot(graph, maxQueries) {
 // explicitly instead of misreading the corpus tables.
 export const PROFILE_V1 = 'pikelet-complete-v1';
 export const PROFILE_V2 = 'pikelet-complete-v2';
-export const FORMAT_VERSIONS = { [PROFILE_V1]: 1, [PROFILE_V2]: 2 };
+// A layer is a complete-profile container under formatVersion 2 with its own
+// manifest profile string (LAYERED_PROFILE.md 4.1). A reader that predates the
+// layered profile sees the string under formatVersion 2 and refuses with its
+// existing "unsupported profile" error, which is the intended behavior: a layer
+// opened alone would silently serve a fraction of a corpus.
+export const PROFILE_LAYER = 'pikelet-layer-v1';
+export const FORMAT_VERSIONS = { [PROFILE_V1]: 1, [PROFILE_V2]: 2, [PROFILE_LAYER]: 2 };
 
 export function assemblePikeletFile(manifestFields, segments, outPath) {
   const formatVersion = FORMAT_VERSIONS[manifestFields.profile];
   if (!formatVersion) {
     throw new Error(`manifest.profile must be ${PROFILE_V1} or ${PROFILE_V2}, got ${manifestFields.profile}`);
   }
+  const isLayer = manifestFields.profile === PROFILE_LAYER;
   const layout = manifestFields.corpus?.layout;
-  if (formatVersion === 2 && layout !== CORPUS_LAYOUT_V2) {
-    throw new Error(`${PROFILE_V2} requires manifest.corpus from buildCorpusSegment() (layout ${CORPUS_LAYOUT_V2})`);
+  // A tombstone-only layer (4.1) carries a corpus object with records 0 and
+  // MUST omit the layout fields, because it has no corpus segment to describe.
+  const tombstoneOnlyLayer = isLayer && manifestFields.corpus?.records === 0;
+  if (formatVersion === 2 && !tombstoneOnlyLayer && layout !== CORPUS_LAYOUT_V2) {
+    throw new Error(`${manifestFields.profile} requires manifest.corpus from buildCorpusSegment() (layout ${CORPUS_LAYOUT_V2})`);
+  }
+  if (tombstoneOnlyLayer && layout !== undefined) {
+    throw new Error('a tombstone-only layer must omit corpus.layout');
   }
   if (formatVersion === 1 && layout !== undefined) {
     throw new Error(`${PROFILE_V1} carries corpus layout v1 (buildCorpusSegmentFromBuffers); got layout ${layout}`);
@@ -462,7 +492,7 @@ export function assemblePikeletFile(manifestFields, segments, outPath) {
   // sketch's own integrity chain to the complete identity without forcing a
   // whole-segment read at open (the index segment stays lazy).
   let indexCommitment;
-  if (formatVersion >= 2) {
+  if (formatVersion >= 2 && !tombstoneOnlyLayer) {
     const index = segments.find((s) => s.kind === 'index');
     if (!index || index.bytes.length < 256) {
       throw new Error('index segment must carry a 256-byte sketch header');

package/complete/format.mjs

@@ -4,5 +4,10 @@
 export const MAGIC = 0x31465350; // "PSF1"
 export const HEADER_BYTES = 64;
 export const TABLE_ENTRY_BYTES = 48;
-export const KINDS = { index: 1, corpus: 2, 'query-interp': 3, evaluation: 4, lexical: 5 };
+// Kinds 6 and 7 belong to the layered profile (LAYERED_PROFILE.md 4.2): a
+// layer carries exactly one tombstones segment, and a base compacted from a
+// chain carries at most one lineage segment. They are registered here so the
+// shared table has one definition; a complete-profile artifact never has
+// either, and the required-segment tables stay per-profile.
+export const KINDS = { index: 1, corpus: 2, 'query-interp': 3, evaluation: 4, lexical: 5, tombstones: 6, lineage: 7 };
 export const KIND_NAMES = Object.fromEntries(Object.entries(KINDS).map(([name, kind]) => [kind, name]));

package/complete/fusion.mjs

@@ -0,0 +1,72 @@
+// Vector/lexical rank fusion for hybrid retrieval — the one definition the
+// reader (index.mjs), the calibrator (pikelet/src/calibrate.mjs, which fits
+// coverage on the passage fusion ranks first) and the BEIR ladder
+// (benchmarks/beir/query-E.mjs) all call, so a fit-time or benchmark
+// ranking is the ranking the reader serves.
+//
+// Reciprocal-rank fusion over the exact-reranked vector order and the BM25
+// order: score = 1/(K + vRank) + lexicalWeight/(K + lexRank), lexical term
+// present only for records BM25 returned. Every lexical candidate joins the
+// vector pool (the reader reranks them exactly), so it always has a vRank.
+//
+// The vector-margin guard: when the vector top-1 leads the top-2 by at
+// least guardMargin of its own distance, it keeps rank 1 regardless of
+// fusion. Without it, with K=60 any record that appears in both lists
+// outranks a vector-only rank 1 (1/(60+v) + w/(60+l) > 1/60 for any finite
+// v when w=1), so a query whose terms match many chunks — a character name
+// in a novel — fills the top ranks with term matches and buries the
+// passage the encoder placed first. Measured on three sets with
+// scripts/fusion-sim.mjs before any default changed.
+
+export const FUSION_DEFAULTS = Object.freeze({
+    rrfK: 60,
+    lexicalWeight: 0.5,
+    guardMargin: 0.05,
+});
+
+/**
+ * @param {Array<{id: number, distance: number}>} vectorOrder hits sorted by
+ *   exact distance ascending (the rerank output), including every lexical
+ *   candidate.
+ * @param {number[]} lexicalIds record ids in BM25 order (after the cutoff).
+ * @param {{rrfK?: number, lexicalWeight?: number, guardMargin?: number}} [options]
+ * @returns the same hit objects in fused order.
+ */
+export function fuseCandidates(vectorOrder, lexicalIds, options = {}) {
+    const rrfK = options.rrfK ?? FUSION_DEFAULTS.rrfK;
+    const lexicalWeight = options.lexicalWeight ?? FUSION_DEFAULTS.lexicalWeight;
+    const guardMargin = options.guardMargin ?? FUSION_DEFAULTS.guardMargin;
+    if (!(rrfK > 0) || !(lexicalWeight >= 0) || !(guardMargin >= 0)) {
+        throw new Error('fuseCandidates: rrfK must be > 0, lexicalWeight and guardMargin >= 0');
+    }
+    if (!lexicalIds.length) return vectorOrder.slice();
+    const lexRank = new Map();
+    lexicalIds.forEach((id, i) => { if (!lexRank.has(id)) lexRank.set(id, i); });
+    let fused = vectorOrder
+        .map((hit, vRank) => ({
+            hit,
+            score: 1 / (rrfK + vRank) + (lexRank.has(hit.id) ? lexicalWeight / (rrfK + lexRank.get(hit.id)) : 0),
+        }))
+        .sort((a, b) => (b.score - a.score) || (a.hit.distance - b.hit.distance))
+        .map((entry) => entry.hit);
+    if (guardMargin > 0 && vectorOrder.length > 1) {
+        const [first, second] = vectorOrder;
+        // The margin is relative to the top-1's own distance, which needs a
+        // positive denominator. first.distance === 0 is not a degenerate case
+        // to skip but the STRONGEST possible top-1 -- cosine distance is
+        // 1 - dot, so a query matching passage text verbatim scores exactly 0
+        // -- and gating on `first.distance > 0` disabled the guard for exactly
+        // the hit it exists to protect (a worse top-1 at 0.001 was guarded, a
+        // perfect one at 0 was not). Any second distance strictly greater than
+        // a zero first is an infinite relative lead, so the guard fires.
+        // Distances are non-negative here (cosine/L2 as returned by the
+        // rerank); a metric admitting negatives would need an absolute margin.
+        const lead = first.distance > 0
+            ? (second.distance - first.distance) / first.distance
+            : (second.distance > 0 ? Infinity : 0);
+        if (lead >= guardMargin && fused[0] !== first) {
+            fused = [first, ...fused.filter((hit) => hit !== first)];
+        }
+    }
+    return fused;
+}

package/complete/index.d.ts

@@ -245,6 +245,29 @@ export declare function openPikeletFile(
 
 /** Header formatVersion -> manifest profile accepted by this reader. */
 export declare const SUPPORTED_PROFILES: Readonly<Record<number, string>>;
+
+/** Rank-fusion parameters shared by the reader, the calibrator and the BEIR ladder. */
+export interface FusionOptions {
+  /** Reciprocal-rank-fusion constant (default 60). */
+  rrfK?: number;
+  /** Weight of the lexical (BM25) rank term relative to the vector term (default 0.5). */
+  lexicalWeight?: number;
+  /**
+   * Relative distance margin (top-2 minus top-1, over top-1) at or above which
+   * the vector top-1 keeps rank 1 (default 0.05; 0 disables the guard).
+   */
+  guardMargin?: number;
+}
+export declare const FUSION_DEFAULTS: Readonly<Required<FusionOptions>>;
+/**
+ * Fuse the exact-reranked vector order (every lexical candidate included) with the BM25 order.
+ * Returns the same hit objects, reordered.
+ */
+export declare function fuseCandidates<T extends { id: number; distance: number }>(
+  vectorOrder: T[],
+  lexicalIds: number[],
+  options?: FusionOptions,
+): T[];
 export declare const CORPUS_LAYOUT_V2: 'records-v2';
 /**
  * Check a host encoder against a kind-2 declaration's verification vectors