pikelet-wasm 0.9.1 → 0.9.3

package/CHANGELOG.md

@@ -8,6 +8,121 @@ Through 0.6.0 these were published as `pancake-wasm` and
 packages are pre-1.0: a minor bump may carry breaking changes, and each entry lists them
 first.
 
+## pikelet-wasm 0.9.3 / pikelet 0.9.3 — 2026-10-06
+
+Hardening for readers that mount packs from other publishers or hosts:
+the index header commitment now covers the bytes actually parsed, and a
+pack can no longer steer lazy reads, demand an exhaustive rerank, or turn
+a malformed abstention asset into confident answers. Upgrade if you mount
+packs you did not build, including over `pikelet mcp`. Both packages move
+to 0.9.3 together, and `pikelet` requires `pikelet-wasm` ^0.9.3 so that
+`pikelet mcp` serves through the hardened reader.
+
+### Compatibility
+
+- **A pack that declares a rerank depth over 4096 is served at 4096 by
+  default**, unless the caller passes `rerank`. No published pack is
+  affected.
+- **A pack whose abstention asset has malformed `thresholds` or
+  `coverage.topK` now reports `unscored`** instead of answering. Packs
+  built by `compile` are not affected.
+- **`compile` declares at most 4096** for a pack over 4096 records whose
+  sweep does not meet its recall target, and warns. Other packs compile
+  as under 0.9.2.
+
+### Fixed
+
+- **The index header commitment binds the bytes the reader parses.** The
+  complete reader checked `manifest.index.headerSha256` on its own read of
+  the sketch header, while the sketch parsed a separate read of the same
+  range. A host answering the two reads differently could pass the check
+  with the genuine header and have a forged one parsed, together with the
+  integrity anchors it carries, replacing the index under a pinned
+  identity. `PikeletSketchArtifact.open()` now takes
+  `expectedHeaderSha256` and checks the header it parses.
+- **Lazily read keyword segments cannot steer reads or inflate them.**
+  Lexical segments over 8 MiB are read on demand without per-read
+  verification. Reads are now confined to the segment's window, and each
+  term entry is checked before its postings are fetched: inside the
+  postings region, `df` no larger than the corpus, and a length a postings
+  list of that `df` can occupy. `info().lexical.integrity` reports
+  `'unverified-lazy-reads'` for such segments and `'segment-sha256'` for
+  eagerly verified ones.
+- **A pack's declared rerank depth is capped at 4096 by default.** The
+  sketch header's `recommendedRerank` is a producer hint; a value near the
+  row count made every query fetch, hash and rerank the whole vectors
+  region. Readers default to at most 4096, keep the declared value as
+  `declaredRecommendedRerank`, and still honor an explicit `rerank`. The
+  published wiki, astro-docs and rust-book packs declare 200, 40 and 30.
+- **`compile` and `append` no longer declare a rerank depth readers will not
+  use.** `compile`'s rerank sweep fell back to the full row count when no
+  depth up to 640 met its recall target; it now stops at the same 4096
+  ceiling, records the recall measured there, and logs a warning when the
+  target was not met. `append` caps a layer's exhaustive hint the same way.
+  `measureRecommendedRerank` returns `targetReached` and `rerankCeiling`,
+  and `pikelet-wasm/artifact` exports `MAX_RECOMMENDED_RERANK`.
+- **A malformed abstention asset serves `unscored`.** Missing or
+  non-numeric `thresholds` compared false against every score, so the pack
+  answered every query as if calibrated; out-of-order or out-of-range ones
+  gave verdicts the fit never defined. A `coverage.topK` outside 1–10 is
+  refused too, since each counted passage is a record read on every query.
+  The scorer now rejects such an asset and the pack reports `unscored`.
+  Assets written by `compile` are unaffected.
+
+## pikelet-wasm 0.9.2 / pikelet 0.9.2 — 2026-10-01
+
+Makes the layered update path work as documented: `append` on a freshly
+compiled pack needs no flag, `pikelet mcp --pack` takes the newest layer,
+and `verify_pack` reports honestly on a chain. Both packages move to 0.9.2
+together, and `pikelet` requires `pikelet-wasm` ^0.9.2, whose chain reader
+supplies the integrity fields `verify_pack` reports.
+
+### Compatibility
+
+- **A pack compiled from the same source gets a different identity than
+  under 0.9.1**, because its manifest gains `corpus.ingest`. Pins and
+  citations to packs already published are unaffected; only new compiles
+  change.
+
+### Changed
+
+- **`compile` records how a pack's records were made, so `append` works on
+  it with no flag.** New packs carry `corpus.ingest` (the chunker and its
+  parameters). Packs compiled before it still need `--assert-ingest`, and
+  `append`'s error now prints the exact JSON to assert. `append` refuses a
+  chain whose declaration names a chunker this `pikelet` does not
+  implement. The new manifest field means a pack compiled from the same
+  source gets a different identity than under 0.9.1.
+
+### Fixed
+
+- **`verify_pack` on a chain reports what it actually checked.** It returned
+  "no tests" with no encoder, integrity or calibration figures, and a note
+  blaming an "older build", so an agent concluded no verdict could be
+  trusted. It now runs the base's golden queries and probes against the
+  chain, labelled as the base's, skips a golden whose expected record a
+  layer removed, and reports the base's calibration figures, the chain's
+  calibration status and drift, and each member's integrity. A chain's
+  `info()` gains `encoderVerified`, `corpusIntegrity` and
+  `indexRowIntegrity`, and per-member integrity under `members`.
+- **`pikelet mcp --pack` accepts the newest layer of an updated pack.** A
+  layer passed to `--pack`, or named by a shelf entry without a `lineage`,
+  failed to mount with "unsupported profile pikelet-layer-v1". It now mounts
+  the chain by walking each layer's parent locator, as LAYERED_PROFILE.md 7
+  specifies, locally or over HTTP; `#<sha256>` pins the head. Shelf entries
+  with a `lineage` mount as before.
+
+### Documentation
+
+- **Both READMEs describe layered updates** (`append`, `compact`, `rebase`,
+  mounting the newest layer, pins, the drift limit) and citations that
+  survive updates. The root README's opening states that a pack fixes how
+  its corpus is queried; the long Wikipedia demonstration moved to
+  `docs/wikipedia-demonstration.md`, and the abstention signals to
+  `docs/how-a-query-runs.md`. Stale claims were corrected: a "retrieval
+  agreement" signal that does not exist, MCP returning zero results under
+  `none`, and the old `PIKELET_SEARCH_EMBED_WORKERS` name.
+
 ## pikelet-wasm 0.9.1 / pikelet 0.9.1 — 2026-09-30
 
 Fixes to the layered profile shipped in 0.9.0, headlined by a chain that

package/README.md

@@ -2,34 +2,18 @@
 
 **Knowledge that ships as a file.**
 
-Pikelet compiles a corpus into one self-contained, queryable artifact.
+Pikelet compiles a corpus into one self-contained, queryable artifact. A model can interrogate a 456,153-record knowledge base whose backend is a static file.
 
-A model can interrogate a 456,153-record knowledge base whose backend is a static file.
+A `.pikelet` carries the source text, semantic and keyword indexes, the query encoder, retrieval calibration, evaluation fixtures, and integrity commitments over all of them. So it is more than stored search data: the file fixes how its corpus is queried — the model that turns a question into a vector, how candidates are ranked, and when the honest answer is "this corpus doesn't say" — and its identity is a hash that commits to all of it. **Pikelet makes retrieval semantics part of the versioned knowledge artifact.**
 
-A `.pikelet` can carry the source text, semantic index, keyword index, query encoder, integrity commitments, retrieval calibration, and evaluation fixtures needed to interrogate that corpus. Put the file on disk, S3, R2, a CDN, or any static HTTP host. A reader can mount it locally or over HTTP Range and search it without a vector database, embedding API, or retrieval server.
+Put the file on disk, S3, R2, a CDN, or any static HTTP host. A reader can mount it locally or over HTTP Range and search it without a vector database, embedding API, or retrieval server. Updates ship as small immutable layers, and an answer can name the exact version of the knowledge it came from.
 
 The `pikelet` CLI requires Node 20+; the `pikelet-wasm` library runs on Node 18+ (CI tests 18, 20, 22), browsers, and Cloudflare Workers.
 
-> **Upgrade to `pikelet-wasm@0.8.1` if you mount packs you did not compile.**
-> 0.8.0 and earlier sized the inline encoder's WASM buffers from the
-> artifact's own `maxTokens` with an upper bound only, so a pack declaring an
-> out-of-range value could overflow them — at mount, before any query, and
-> with every integrity check passing, because the pack's author is the one who
-> signed it. 0.8.1 validates the bound. Packs the toolchain produces were
-> never affected. `pikelet@0.8.1` picks the fix up through its dependency
-> range; see the CHANGELOG for the detail.
-
 ```bash
 npx pikelet compile --source ./docs --out docs.pikelet
 ```
 
-Installing rather than using `npx`? Use `npm install -g pikelet
---omit=optional`. The optional dependency is `@xenova/transformers`, which
-only the `create` scaffold path uses. Measured on a clean project, omitting it
-takes the install from **82 packages / 257 MB / "6 vulnerabilities (5 high, 1
-critical)"** to **2 packages / 1.8 MB / "found 0 vulnerabilities"** — the
-advisories are all inside a dependency `compile` and `mcp` never load.
-
 ```text
 Ingested 3 docs -> 3 chunks
 Embedded 3/3 chunks with inline transformer
@@ -63,7 +47,7 @@ console.log(out.matchQuality, out.results[0]?.title);
 // 'strong' 'Snapshot restore'
 ```
 
-That's the whole loop. `compile` also takes a live URL (`--source https://docs.example.com`) instead of a directory. If you want a deployed search app — a Worker + UI, not a file — use `npx pikelet create` instead; see [`packages/pikelet/README.md`](packages/pikelet/README.md) for the full CLI reference and the tradeoffs between the two. `create` is the one path that needs the optional `@xenova/transformers` dependency, which is why the install above omits it by default.
+That's the whole loop. `compile` also takes a live URL (`--source https://docs.example.com`) instead of a directory. If you want a deployed search app — a Worker + UI, not a file — use `npx pikelet create` instead; see [`packages/pikelet/README.md`](packages/pikelet/README.md) for the full CLI reference and the tradeoffs between the two. To install rather than use `npx`, run `npm install -g pikelet --omit=optional`: the optional dependency is only for `create` ([why it matters](packages/pikelet/README.md#installing)).
 
 Under the hood, the file is one container for everything a reader needs:
 
@@ -96,93 +80,18 @@ Search is the interface. **The file is the knowledge deployment unit.**
 
 ## The shortest demonstration
 
-A real Pikelet pack contains **456,153 Simple English Wikipedia records** in a **648.5 MiB** artifact.
-
-Try the live browser demo:
-
-```text
-https://pikelet-wiki-playground.pages.dev/
-```
-
-The page is static HTML/JS on Cloudflare Pages. It mounts the pack from R2:
-
-```text
-https://pub-6da2384a3bca4a44b2b2fa29a94cc811.r2.dev/wikipedia.pikelet
-```
-
-There is no `/search`, `/embed`, `/query`, or `/api` backend behind the demo.
-The browser opens the `.pikelet` file over HTTP Range, verifies the artifact,
-loads the embedded query encoder, and shows the byte ranges it reads.
-
-That file was served by a deliberately dumb HTTP server whose only relevant capability was:
-
-```text
-fs.createReadStream + HTTP Range
-```
-
-There was no vector database, no search service, no embedding service, no application server that understood the pack, no index process on the server.
-
-The pack was mounted by URL and exposed to a real headless Claude process through MCP. Each invocation was restricted with `--strict-mcp-config` to only:
-
-```text
-mcp__wiki-pack__search
-mcp__wiki-pack__list_packs
-mcp__wiki-pack__get_record
-```
-
-No web tool. No filesystem tool. No alternate retrieval tool.
-
-One of the actual responses:
-
-> **Q: What causes earthquakes?**
->
-> Earthquakes are caused by the movement of tectonic plates in the Earth's crust. Stress builds up along fault boundaries where plates grind against or ride over one another; when that stress exceeds the friction holding the locked fault in place, it suddenly releases as shock waves (seismic waves) that shake the ground [record 429637].
->
-> There are three main fault types that can produce quakes — normal, reverse (thrust), and strike-slip — and some quakes have less clear-cut causes, like deep shifts in ancient rock or mine-collapse "tremors" [record 429638].
->
-> Two related mechanisms compound this:
->
-> * Subduction, where one plate is forced under another into the mantle, produces the most powerful earthquakes and can trigger tsunamis [record 132833].
-> * More broadly, the slow convection-driven movement of crustal plates over the hot mantle is the underlying driver of quakes, volcanoes, and mountain formation [record 132267].
->
-> Volcanic eruptions can also cause earthquakes, though these are localized to areas near active volcanoes [record 429637].
-
-The model did not just repeat the first search hit. It retrieved multiple records and synthesized them into a sourced answer.
-
-Other runs produced:
-
-```text
-Who was the first person on the moon?          → Neil Armstrong          (records 408686, 408684)
-How do volcanoes form?                          → plates + subduction + hotspots
-                                                                          (records 132770, 132765, 132852, 132853)
-How does photosynthesis work?                   → two-phase reaction, cited (5 records: 114219, 114220, 114222, 114223, 269279)
-What is the capital of France?                  → Paris                  (records 454641, 302684)
-```
-
-That is the product in one demonstration:
-
-> **A model is interrogating a 456,153-record knowledge base whose backend is a static file.**
+A real pack holds **456,153 Simple English Wikipedia records** in a **648.5 MiB** file. The [live browser demo](https://pikelet-wiki-playground.pages.dev/) is static HTML on Cloudflare Pages that mounts it straight from an R2 bucket — there is no `/search`, `/embed`, `/query` or `/api` behind it. The browser range-reads the file, verifies it, loads the query encoder from inside it, and shows every byte range it fetches.
 
-The model may already contain some of these facts in its pretrained parameters. This test demonstrates the retrieval, synthesis, citation, and deployment path; the [Veyra ablation](docs/veyra-ablation.md) tests whether support changes when evidence is removed from the pack.
+For the agent test, the same file was served by a server whose only relevant capability was `fs.createReadStream` + HTTP Range and mounted by URL into a headless Claude restricted to the pack's three MCP tools (no web, no filesystem). The model answered questions such as "What causes earthquakes?" by searching, reading several records, and citing them.
 
-**Network cost.** In a single persistent session (one mount, five queries, one repeat):
+| In one persistent session                | Transferred                 |
+| ---------------------------------------- | --------------------------: |
+| Mount (once)                             | 51.7 MiB in 11 requests     |
+| Query encoder (once, with the first query) | ~25 MiB                   |
+| Each fresh query after that              | ~0.5–1.1 MiB                |
+| Whole session: 5 queries and a repeat    | 80.8 MiB, 12.5% of the file |
 
-| Operation                       | Bytes transferred | Requests |
-| ------------------------------- | ----------------: | -------: |
-| Initial mount                   |          51.7 MiB |       11 |
-| Query 1 + one-time encoder load |          25.9 MiB |       77 |
-| Query 2                         |         504.4 KiB |       37 |
-| Query 3                         |         499.4 KiB |       46 |
-| Query 4                         |         968.9 KiB |       75 |
-| Query 5                         |           1.1 MiB |      116 |
-| Repeated query 1                |         238.4 KiB |        4 |
-| **Total**                       |      **80.8 MiB** |  **366** |
-
-Roughly **12.5% of the 648.5 MiB artifact** crossed the wire across that whole session. Excluding the one-time ~25 MiB encoder load, fresh-query traffic ran **~0.5–1.1 MiB per query**. The complete artifact was never downloaded.
-
-The headless-Claude test above used a fresh process per question, so each invocation repaid the ~52 MiB mount and ~25 MiB encoder cost — about 78 MiB per cold query. That's a real operational distinction: **persistent sessions amortize mount and encoder cost; independent cold processes do not.**
-
-*(The large benchmark fixture still carries its historical `.pancake` filename from before the Pikelet rename; current artifacts use `.pikelet`.)*
+The complete artifact is never downloaded. The full transcript, the per-query table, and what a cold process per question costs instead are in [the Wikipedia demonstration](docs/wikipedia-demonstration.md); [`benchmarks/range-proof/`](benchmarks/range-proof/) reproduces it.
 
 ---
 
@@ -198,7 +107,7 @@ npx pikelet mcp install \
 
 The agent gets `search`, `get_record`, `list_packs`, `verify_pack`. A search result carries the identity of the pack and the location of the source record, so an agent can work against `product-docs.pikelet`, `rust-reference.pikelet`, `policy-2026-09.pikelet`, `customer-manual-v4.pikelet` without each publisher operating a retrieval API. The artifact can be local, private, public, behind authenticated object storage, or distributed like any other static asset.
 
-A pack mounted with a content hash has a stable identity — `https://example.com/docs.pikelet#8d731...` — so "what body of knowledge did this agent query?" has a reproducible answer.
+A pack mounted with a content hash has a stable identity — `https://example.com/docs.pikelet#8d731...` — so "what body of knowledge did this agent query?" has a reproducible answer. A citation is `(pack identity, record id)`, and it stays meaningful as the pack changes: a chain of layers resolves citations made against any of its earlier versions, and a pack compacted from a chain translates them forward, reporting what an old record became or that it was deleted (the old record's bytes stay in the old files). Citation translation is a library call today, `citation(identity, id)`; the MCP tools do not expose it yet.
 
 **A mounted pack's content reaches the model as tool output.** `verify_pack` proves the bytes are intact and match their pinned identity; it does not prove the corpus itself is trustworthy. Mounting a pack from a source you don't control is the same trust decision as giving an agent any other untrusted-content tool — treat pack text the way you'd treat search results or fetched web pages, not as instructions.
 
@@ -228,9 +137,23 @@ All three packs derive from the same small synthetic Station Veyra corpus, with
 
 ---
 
+## Update a pack without rebuilding it
+
+A pack doesn't have to be recompiled when a few documents change. `append` never modifies its parent: it writes a new, small `.pikelet` — a **layer** — holding new records, deletions, or replacements, and the base plus its layers (a chain) is searched as one pack. Every published file stays byte-for-byte immutable; the newest layer defines the current state, and its identity pins the whole history.
+
+```bash
+npx pikelet append --parent docs.pikelet --source ./changed-docs --out docs.0001.pikelet
+npx pikelet append --parent docs.0001.pikelet --remove 412 --out docs.0002.pikelet
+npx pikelet mcp --pack docs.0002.pikelet
+```
+
+Layers are small. On a 20-record test pack with a 24.6 MiB base, deleting a record took a 1.2 KiB layer and replacing one took 6.9 KiB; a layer that adds records also carries a vocabulary filter sized to its base's (about 43 KiB on a 3,255-record docs pack).
+
+A layer records its parent's identity, and its location when the parent sits in the same directory, so mounting the newest layer finds the rest of the chain — locally or over HTTP — and verifies every link. `#<sha256>` on the newest layer pins the whole chain. Layers always inherit the base's encoder. They inherit its calibration only while the chain stays within the drift limit (by default, records added plus deleted up to 20% of the base's); past it, the chain serves `unscored` until `compact` folds it into one fresh base. `rebase` moves a layer onto a different parent. The format is specified in [`LAYERED_PROFILE.md`](LAYERED_PROFILE.md); `npx pikelet --help` lists every flag.
+
 ## Match quality and abstention
 
-A nearest neighbour is not automatically evidence that a corpus answers a question. Pikelet can calibrate retrieval signals at build time (best semantic distance, distance margin, lexical coverage, retrieval agreement). When the corpus supports a reliable classifier, results carry `matchQuality: strong | weak | none`. When calibration can't separate supported from unsupported reliably — a single novel may be semantically homogeneous enough that the fit isn't trustworthy — Pikelet reports `matchQuality: unscored` and records why calibration was skipped, rather than manufacturing confidence. A `none` verdict withholds `results` by default; pass `query(text, { showAbstained: true })` to see the raw retrieval anyway — `matchQuality` and `confidence` are unaffected either way.
+A nearest neighbour is not automatically evidence that a corpus answers a question. At build time Pikelet can fit a classifier over retrieval signals — distances, and how much of the question the top passages actually contain ([details](docs/how-a-query-runs.md#how-abstention-is-decided)). When the corpus supports a reliable classifier, results carry `matchQuality: strong | weak | none`. When calibration can't separate supported from unsupported reliably — a single novel may be semantically homogeneous enough that the fit isn't trustworthy — Pikelet reports `matchQuality: unscored` and records why calibration was skipped, rather than manufacturing confidence. In the library, a `none` verdict withholds `results` by default; pass `query(text, { showAbstained: true })` to see the raw retrieval anyway — `matchQuality` and `confidence` are unaffected either way. MCP `search` does the opposite: it returns results under `none` with a note telling the model the support is indirect, because the calibrator can misjudge a paraphrase; `showAbstained: false` withholds them.
 
 `matchQuality` is evidence about retrieval support. It is **not** a guarantee that an LLM will never hallucinate.
 
@@ -238,14 +161,15 @@ A nearest neighbour is not automatically evidence that a corpus answers a questi
 
 ## Deeper dives
 
-- [How a `.pikelet` query runs](docs/how-a-query-runs.md) — what is inside the file, what a query costs on the wire, remote execution, and the integrity checks on the read path.
+- [The Wikipedia demonstration](docs/wikipedia-demonstration.md) — the full agent transcript, the per-query network table, and what cold processes cost.
+- [How a `.pikelet` query runs](docs/how-a-query-runs.md) — what is inside the file, what a query costs on the wire, remote execution, the integrity checks on the read path, and how abstention is decided.
 - [The Veyra ablation](docs/veyra-ablation.md) — what happens to answers when the supporting record is edited or removed from the pack, with the reproduction commands.
 - [Why make knowledge a file](docs/why-a-file.md) — the reasoning behind the single-artifact design, and what it commits to.
 - [Architecture](docs/architecture.md) — the engine, the C ABI and the JavaScript wrapper; the artifact formats are specified in [`spec/`](spec/).
 
 ## What this is not
 
-**Not a claim that vector databases are obsolete.** If your corpus changes continuously, needs transactional updates, serves many tenants, or already lives comfortably in a database, use a database. Pikelet targets `build → publish → query many times → replace on release`.
+**Not a claim that vector databases are obsolete.** If your corpus changes continuously, needs transactional updates, serves many tenants, or already lives comfortably in a database, use a database. Pikelet targets `build → publish → query many times → replace on release`, with small layered updates in between.
 
 **Not state-of-the-art embedding research.** The bundled model is MiniLM-L6, chosen for being small enough to live inside the artifact. ArguAna demonstrates a real quality cost from the compact encoder.
 
@@ -298,7 +222,8 @@ packages/pikelet-wasm/            The published pikelet-wasm package:
   src/                             Engine wrapper and artifact readers/builders
                                    (the entrypoints, core/, artifact/, errors/)
   complete/                        Reader and builder for the complete
-                                   range-readable .pikelet artifact
+                                   range-readable .pikelet artifact, and the
+                                   layered-chain reader and producers
   native/                          N-API build of the same engine (benchmarks)
 packages/pikelet/                 The published pikelet CLI: compiler, MCP
                                    server, encoder integration, Docusaurus plugin
@@ -306,6 +231,8 @@ docs/                             Deeper dives (how a query runs, the Veyra
                                    ablation, why a file), architecture notes,
                                    rename history, measurement reports
 spec/                             Byte-level artifact contracts
+LAYERED_PROFILE.md                The layered profile: append, compact,
+                                   rebase, and how chains are read
 benchmarks/beir/                       Frozen BEIR ablation harness (encoder,
                                    quantization, HNSW quality)
 benchmarks/range-proof/                The deliberately boring static-HTTP proof:
@@ -342,6 +269,7 @@ Pikelet is early. The implementation is real; the format is not frozen. One prim
 - content identity and lazy-read integrity verification
 - calibration/abstention support
 - MCP mounting
+- layered updates: append, compact, rebase
 - BEIR retrieval evaluation
 - range-read and failure-mode tests
 - a large 456k-record example artifact

package/complete/builder.d.ts

@@ -71,6 +71,10 @@ export declare function measureRecommendedRerank(input: {
 }): Promise<{
   recommendedRerank: number;
   recall: number;
+  /** False when no rung up to rerankCeiling reached targetRecall; recommendedRerank is then the ceiling. */
+  targetReached: boolean;
+  /** min(row count, MAX_RECOMMENDED_RERANK): the deepest rerank the sweep measures. */
+  rerankCeiling: number;
   k: number;
   targetRecall: number;
   queries: number;

package/complete/builder.mjs

@@ -351,6 +351,12 @@ export function buildInlineTransformerEncoderSegment({ declaration, vocabBytes,
 // targetRecall — plus the whole curve for the evaluation segment. Queries
 // are the corpus's own float embeddings when the caller has them, else rows
 // dequantized out of the snapshot.
+//
+// The sweep stops at the reader's default ceiling (MAX_RECOMMENDED_RERANK):
+// it used to fall back to the full row count, a depth readers no longer
+// honor by default, so a pack could record a recall it would not be served
+// at. The last rung is the ceiling, measured like any other, and
+// targetReached says whether the target was met below it.
 export async function measureRecommendedRerank({
   artifactModule,
   sketchBytes,
@@ -387,9 +393,10 @@ export async function measureRecommendedRerank({
     for (const q of queries) {
       truth.push(new Set((await sketch.search(q, topK, { rerank: count })).results.map((r) => r.id)));
     }
+    const rerankCeiling = Math.min(count, artifactModule.MAX_RECOMMENDED_RERANK ?? count);
     const ladder = (sweep || [10, 15, 20, 30, 40, 60, 80, 120, 160, 240, 320, 480, 640])
-      .filter((c) => c >= topK && c < count);
-    ladder.push(count);
+      .filter((c) => c >= topK && c < rerankCeiling);
+    ladder.push(rerankCeiling);
     const curve = [];
     let recommendedRerank = null;
     let recommendedRecall = null;
@@ -409,13 +416,16 @@ export async function measureRecommendedRerank({
         recommendedRecall = recall;
       }
     }
-    if (recommendedRerank === null) {
-      recommendedRerank = count;
+    const targetReached = recommendedRerank !== null;
+    if (!targetReached) {
+      recommendedRerank = rerankCeiling;
       recommendedRecall = curve[curve.length - 1].recall;
     }
     return {
       recommendedRerank,
       recall: recommendedRecall,
+      targetReached,
+      rerankCeiling,
       k: topK,
       targetRecall,
       queries: queries.length,

package/complete/index.d.ts

@@ -134,7 +134,13 @@ export interface CompleteSearch {
      * order fuses distance and BM25 rankings by reciprocal rank). Null on
      * vector-only artifacts.
      */
-    lexical: { terms: number; docCount: number; lazy?: true } | null;
+    lexical: {
+        terms: number;
+        docCount: number;
+        /** 'segment-sha256' when read eagerly and verified whole; 'unverified-lazy-reads' for lazily opened segments. */
+        integrity: 'segment-sha256' | 'unverified-lazy-reads';
+        lazy?: true;
+    } | null;
     /**
      * 'engine' once a WASM scan kernel serves the resident scan
      * (auto-staged or injected via options.sketchScanner); 'js' before

package/complete/index.mjs

@@ -780,7 +780,12 @@ export async function openPikeletFile(input, options = {}) {
                 }
                 return bytes;
             })(),
-            PikeletSketchArtifact.open(windowSource(source, idx.offset, idx.length), { maxReadBytes }),
+            // Format 2's header commitment is checked inside open(), against
+            // the same bytes the sketch parses (see expectedHeaderSha256).
+            PikeletSketchArtifact.open(windowSource(source, idx.offset, idx.length), {
+                maxReadBytes,
+                ...(perRecord ? { expectedHeaderSha256: manifest.index.headerSha256 } : {}),
+            }),
             readChecked(source, corpus.offset, tablesBytes, 'corpus tables', maxReadBytes, fileBytes),
             // Lexical index (kind 5, OPTIONAL). Small segments: eager
             // whole-segment read, digest-verified like the query-interp
@@ -791,7 +796,15 @@ export async function openPikeletFile(input, options = {}) {
             segments.has('lexical') ? (async () => {
                 const seg = segments.get('lexical');
                 if (seg.length > LAZY_LEXICAL_BYTES) {
-                    const lexRead = (off, len) => readChecked(source, seg.offset + off, len, 'lexical segment', maxReadBytes, fileBytes);
+                    // Confined to the segment's own window: lazy reads are
+                    // not individually verified, so no field they carry may
+                    // steer a read into another segment.
+                    const lexRead = (off, len) => {
+                        if (!Number.isSafeInteger(off) || !Number.isSafeInteger(len) || off < 0 || len < 0 || off + len > seg.length) {
+                            throw new Error('.pikelet lexical segment read is outside the segment window');
+                        }
+                        return readChecked(source, seg.offset + off, len, 'lexical segment', maxReadBytes, fileBytes);
+                    };
                     return { lazyIndex: await openLexicalIndexLazy(lexRead, seg.length) };
                 }
                 const bytes = await readChecked(source, seg.offset, seg.length, 'lexical segment', maxReadBytes, fileBytes);
@@ -800,12 +813,6 @@ export async function openPikeletFile(input, options = {}) {
                 }
                 return { bytes };
             })() : null,
-            perRecord ? (async () => {
-                const headerBytes = await readChecked(source, idx.offset, SKETCH_HEADER_BYTES, 'index header', maxReadBytes, fileBytes);
-                if (await sha256hex(headerBytes) !== manifest.index.headerSha256) {
-                    throw new Error('.pikelet index header failed hash verification against the manifest');
-                }
-            })() : null,
         ]);
 
         const lexicalIndex = lexicalBytes
@@ -1289,6 +1296,9 @@ export async function openPikeletFile(input, options = {}) {
                         ? {
                             terms: lexicalIndex.termCount,
                             docCount: lexicalIndex.docCount,
+                            // Lazy reads are bounded to the segment but not
+                            // individually verified (LAZY_LEXICAL_BYTES).
+                            integrity: lexicalIndex.lazy ? 'unverified-lazy-reads' : 'segment-sha256',
                             ...(lexicalIndex.lazy ? { lazy: true } : {}),
                         }
                         : null,

package/complete/layer-reader.mjs

@@ -121,6 +121,15 @@ async function walkToBase(head, options) {
     throw new Error(`walking to the base exceeded the depth limit ${maxDepth} (2.7): a layer.parent cycle or an over-deep chain`);
 }
 
+function memberIntegritySummary(opened) {
+    const infos = opened.filter((r) => typeof r?.info === 'function').map((r) => r.info());
+    const one = (key) => {
+        const values = [...new Set(infos.map((i) => i[key]))];
+        return values.length === 1 ? values[0] : 'mixed';
+    };
+    return { corpusIntegrity: one('corpusIntegrity'), indexRowIntegrity: one('indexRowIntegrity') };
+}
+
 const isHttpUrl = (loc) => /^https?:\/\//i.test(loc);
 const describeLocation = (loc) => (typeof loc === 'string' ? loc : (loc?.url ?? loc?.location ?? '<source>'));
 
@@ -370,7 +379,24 @@ export async function openPikeletChain(members, options = {}) {
                     // to that member. A chain surfaces the base's, which are
                     // the ones fit against the corpus the encoder saw.
                     sampleQueries: Array.isArray(base.manifest.sampleQueries) ? base.manifest.sampleQueries : [],
-                    members: chainMembers.map((m) => ({ depth: m.depth, identity: m.identity, records: m.records })),
+                    // Each member's own integrity, from its reader. A
+                    // tombstone-only layer is a verified shell (manifest plus
+                    // tombstones), with no corpus or index to report on.
+                    members: chainMembers.map((m, i) => {
+                        const ri = typeof opened[i]?.info === 'function' ? opened[i].info() : null;
+                        return {
+                            depth: m.depth, identity: m.identity, records: m.records,
+                            ...(ri ? { corpusIntegrity: ri.corpusIntegrity, indexRowIntegrity: ri.indexRowIntegrity }
+                                : { tombstoneOnly: true }),
+                        };
+                    }),
+                    // The chain encodes with the base's encoder (4.5), so the
+                    // base's verification is the chain's. Chain-level integrity
+                    // is the members' when they agree, else 'mixed'. Without
+                    // these, verify_pack on a chain reported no encoder or
+                    // integrity state at all.
+                    encoderVerified: opened[0].info().encoderVerified,
+                    ...memberIntegritySummary(opened),
                 };
             },
 

package/complete/lexical.mjs

@@ -181,6 +181,20 @@ export async function openLexicalIndexLazy(read, segLength) {
                 return e && { ...e, term: t };
             }))).filter(Boolean);
             if (!found.length) return [];
+            // Term entries arrive by lazy read, unverified (see the stance
+            // note above), so each one is held to the segment before its
+            // postings are fetched: inside the postings region (spec 3.8, as
+            // the eager reader enforces), a df no larger than the corpus, and
+            // a length a df-entry list can actually occupy (a doc-gap and a
+            // tf varint, each at most 6 bytes). Without these, one forged
+            // entry could steer a read anywhere in the file or ask for
+            // hundreds of MiB per query term.
+            for (const e of found) {
+                if (!Number.isSafeInteger(e.postingsRel) || e.postingsRel + e.postingsLen > postingsBytes
+                    || e.df > docCount || e.postingsLen > 12 * e.df) {
+                    throw new Error('.pikelet lexical postings out of bounds');
+                }
+            }
             const postings = await Promise.all(found.map((e) => read(postingsOffset + e.postingsRel, e.postingsLen)));
             const scores = new Map();
             found.forEach((e, i) => {

package/complete/retrieval-abstention.mjs

@@ -34,11 +34,25 @@
 // one — there is no scenario where re-adding this via a config flag was
 // going to end differently a third time.
 
+// Upper bound on asset.coverage.topK. calibrate.mjs ships 5; every passage
+// counted here is a record hydrated on every query, so a pack must not be
+// able to turn one query into a hydration of its whole rerank window.
+export const MAX_COVERAGE_TOP_K = 10;
+
 export function createAbstentionScorer(asset, bloomBytes) {
     if (!asset || !Array.isArray(asset.weights) || !asset.thresholds) return null;
+    // Thresholds that are missing, non-numeric or out of order compare false
+    // against every p and would answer everything; a malformed asset must
+    // degrade to unscored instead (callers map a null scorer to it).
+    const { hard, weak } = asset.thresholds;
+    if (!Number.isFinite(hard) || !Number.isFinite(weak) || hard < 0 || hard > weak || weak > 1) return null;
     const coverageCfg = asset.coverage && typeof asset.coverage.weight === 'number'
         && Number.isFinite(asset.coverage.mean) && Number.isFinite(asset.coverage.std)
         ? asset.coverage : null;
+    if (coverageCfg && coverageCfg.topK !== undefined
+        && !(Number.isInteger(coverageCfg.topK) && coverageCfg.topK >= 1 && coverageCfg.topK <= MAX_COVERAGE_TOP_K)) {
+        return null;
+    }
     const coverageStopwords = coverageCfg ? new Set(coverageCfg.stopwords || []) : null;
     const coverageMinLen = coverageCfg ? (coverageCfg.minWordLen || 3) : 3;
     // Words the corpus uses everywhere ground any query that mentions them,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pikelet-wasm",
-  "version": "0.9.1",
+  "version": "0.9.3",
   "description": "Vector search engine and .pikelet artifact readers/builders, in WebAssembly. Compile a corpus into one self-contained, hash-verified search file, or use the underlying HNSW engine directly. Runs in Node.js, browsers, and Cloudflare Workers with no native dependencies.",
   "main": "src/pikelet.js",
   "types": "src/pikelet.d.ts",

package/src/artifact/pikelet-artifact-sketch.js

@@ -37,6 +37,12 @@ const {
 const SKETCH_MAGIC = 0x31415350; // PSA1
 const SKETCH_HEADER_BYTES = 256;
 const SKETCH_KIND_U8 = 1;
+// Ceiling on the header's recommendedRerank as a reader default. The field
+// is a producer hint, and a hostile (or exhaustive-fallback) value up to the
+// row count would make every query fetch, hash and rerank the whole vectors
+// region — at wiki scale, minutes per query with every check passing. An
+// explicit options.rerank is the caller's own choice and is not capped here.
+const MAX_RECOMMENDED_RERANK = 4096;
 
 // The affine table feeds every rerank distance; a NaN/Infinity entry makes
 // that row's distance non-finite and lets it sit in the result set
@@ -331,6 +337,28 @@ class PikeletSketchArtifact {
         if (header.byteLength !== SKETCH_HEADER_BYTES) {
             throw pikeletError(PIKELET_ERROR_CODES.SNAPSHOT_INVALID, 'Sketch artifact header is truncated');
         }
+        // options.expectedHeaderSha256 (hex): a container that commits to this
+        // header (the complete profile's manifest.index.headerSha256) checks
+        // it HERE, against the very bytes parsed below. Checking a separate
+        // read instead would let a host serve the genuine header to that read
+        // and a rewritten one (with a self-consistent residentSha256 and page
+        // table) to this one, replacing the index under a pinned identity.
+        if (options.expectedHeaderSha256 !== undefined) {
+            const expected = options.expectedHeaderSha256;
+            if (typeof expected !== 'string' || !/^[0-9a-f]{64}$/.test(expected)) {
+                throw pikeletError(PIKELET_ERROR_CODES.INVALID_ARGUMENT, 'SketchArtifact.open() expectedHeaderSha256 must be 64 lowercase hex characters');
+            }
+            const digest = await sha256BytesAsync(header);
+            if (!digest) {
+                throw pikeletError(PIKELET_ERROR_CODES.SNAPSHOT_INVALID,
+                    'Sketch artifact header verification requested but no crypto backend is available');
+            }
+            let hex = '';
+            for (const b of digest) hex += b.toString(16).padStart(2, '0');
+            if (hex !== expected) {
+                throw pikeletError(PIKELET_ERROR_CODES.SNAPSHOT_INVALID, '.pikelet index header failed hash verification against the manifest');
+            }
+        }
         const view = new DataView(header.buffer, header.byteOffset, header.byteLength);
         if (view.getUint32(0, true) !== SKETCH_MAGIC) {
             throw pikeletError(PIKELET_ERROR_CODES.SNAPSHOT_INVALID, 'Not a Pikelet sketch artifact (bad magic)');
@@ -353,7 +381,8 @@ class PikeletSketchArtifact {
         const sketchesOffset = view.getUint32(40, true);
         artifact.vectorsOffset = view.getUint32(44, true);
         const fileBytes = Number(view.getBigUint64(48, true));
-        artifact.recommendedRerank = view.getUint32(120, true);
+        artifact.declaredRecommendedRerank = view.getUint32(120, true);
+        artifact.recommendedRerank = Math.min(artifact.declaredRecommendedRerank, MAX_RECOMMENDED_RERANK);
         // Retained so the lazy tier stays verifiable after open — see
         // verifyVectors(). Copied out of the header read buffer.
         artifact.vectorsSha256 = new Uint8Array(header.subarray(88, 120));
@@ -1131,6 +1160,7 @@ async function createSketchScanner(loadEngine, artifact, options = {}) {
 
 module.exports = {
     PikeletSketchArtifact,
+    MAX_RECOMMENDED_RERANK,
     createSketchScanner,
     buildSketchArtifact,
     buildSketchArtifactBytes,

package/src/artifact/pikelet-artifact.d.ts

@@ -105,5 +105,12 @@ export declare function exportSketchArtifact(
   options?: SketchArtifactBuildOptions
 ): SketchArtifactBuildManifest;
 
+/**
+ * Ceiling on the header's recommendedRerank that readers use as their default
+ * rerank depth. Producers should not declare more: a reader will not honor it
+ * unless the caller passes an explicit rerank.
+ */
+export declare const MAX_RECOMMENDED_RERANK: number;
+
 /** Parse and validate an untrusted uint8 engine snapshot (fails closed). */
 export declare function parseUint8Snapshot(bytes: Uint8Array | ArrayBufferLike): Uint8SnapshotGraph;

package/src/artifact/pikelet-artifact.js

@@ -12,13 +12,14 @@
 const { NodeFileRangeSource, parseUint8Snapshot } = require('./pikelet-artifact-common.js');
 const { PikeletRangeArtifact, buildRangeArtifact, buildRangeArtifactFile } = require('./pikelet-artifact-range.js');
 const {
-    PikeletSketchArtifact, createSketchScanner,
+    PikeletSketchArtifact, MAX_RECOMMENDED_RERANK, createSketchScanner,
     buildSketchArtifact, buildSketchArtifactBytes, buildSketchArtifactFile, exportSketchArtifact,
 } = require('./pikelet-artifact-sketch.js');
 
 module.exports = {
     PikeletRangeArtifact,
     PikeletSketchArtifact,
+    MAX_RECOMMENDED_RERANK,
     createSketchScanner,
     NodeFileRangeSource,
     buildRangeArtifact,