pikelet-wasm 0.8.1 → 0.9.0

package/CHANGELOG.md

@@ -8,43 +8,22 @@ Through 0.6.0 these were published as `pancake-wasm` and
 packages are pre-1.0: a minor bump may carry breaking changes, and each entry lists them
 first.
 
-## 0.8.1 — `pikelet-wasm`
-
-Security release, cut from `main` ahead of the rest of the unreleased
-backlog below. `pikelet` is unchanged at 0.8.1 and picks this up through its
-`^0.8.0` dependency range.
-
-### Security
+## pikelet-wasm 0.9.0 / pikelet 0.9.0 — 2026-09-29
 
-- **A `.pikelet` can no longer overflow the encoder's WASM heap through an
-  out-of-range `maxTokens`.** `complete/inline-transformer.mjs` sized the
-  kernel's ids and hidden-state buffers from the artifact's own encoder
-  declaration, clamped with `Math.min(declaration.maxTokens || 512, 512)` —
-  an upper bound only. A pack declaring `maxTokens: 1` made the windowing
-  arithmetic (`windowLen = maxSeq - 2`) negative, so `interior.slice(w *
-  windowLen, (w + 1) * windowLen)` returned nearly the whole token sequence
-  and the forward pass wrote past both allocations: a 60-token query wrote
-  about 92 KB into a 1,536-byte buffer. `maxTokens` is now validated as an
-  integer in `[8, 512]` at parse, before anything is allocated, `maxSeq` is
-  clamped on both sides, and both call sites assert the window fits.
-
-  This fired at **mount**, before any query: the reader awaits the encoder
-  for a non-deferred query-interp segment (and prefetches it otherwise), and
-  the declaration's own `testVectors` are verified after the buffers exist,
-  so a long test vector was itself a trigger. Every integrity check passed,
-  because the attacker authored the file and its digests bind their bytes.
-
-  Affects anyone who mounts a `.pikelet` they did not compile — the format's
-  intended use, since packs are distributed and range-read from URLs. The
-  write is confined to the encoder's WASM linear memory and does not reach
-  host JavaScript objects, which is why it is rated high rather than
-  critical. Upgrade to get the bound; there is no workaround short of not
-  mounting third-party packs.
-
-## Unreleased
+Everything merged to `main` since 0.8.0 that the 0.8.1 and 0.8.2 point
+releases below did not carry, headlined by the layered profile. Both
+packages move to 0.9.0 together. Pre-1.0, this minor carries breaking
+changes; they are listed first.
 
 ### Breaking / compatibility
 
+- **`pikelet` 0.9.0 requires `pikelet-wasm` ^0.9.0.** Its new `append`,
+  `compact` and `rebase` commands and its chain mounts import
+  `pikelet-wasm/complete/layer-*.mjs` and `complete/format.mjs`, which no
+  0.8.x exports; upgrade both together. A layer's profile string,
+  `pikelet-layer-v1`, is refused by every earlier reader by design: a layer
+  opened alone would serve a fraction of a corpus.
+
 - **The `PANCAKE_ENCODER_WEIGHTS_URL`, `PANCAKE_SEARCH_EMBED_WORKERS`,
   `PANCAKE_SEARCH_STUB_EMBEDDINGS`, and `PANCAKE_SEARCH_PYTHON`
   environment variable fallbacks are removed.** 0.7.0 introduced
@@ -59,6 +38,42 @@ backlog below. `pikelet` is unchanged at 0.8.1 and picks this up through its
   have; CI tests 18, 20 and 22. The `pikelet` CLI is unchanged at
   `>=20`.
 
+### Added
+
+- **Layered search artifacts** (`LAYERED_PROFILE.md`, Draft 4). A pack can
+  be updated by appending a small `pikelet-layer-v1` file instead of
+  recompiling: a layer carries only its new records plus a cumulative
+  tombstone mask and supersession edges, and commits to its parent's
+  identity, so a chain's head identity pins every byte a query can touch.
+  `openPikeletChain` (`pikelet-wasm/complete/layer-reader.mjs`) mounts base
+  and layers, verifies every parent commitment and spec rule at mount, and
+  queries them as one corpus: masked candidate generation per tier,
+  chain-global BM25 statistics, one fusion, and the base's abstention fit
+  inherited until calibration drift passes the reader's limit (0.20 by
+  default), after which results ship `unscored`. Records carry the layer
+  that owns them; `record()` reports tombstoned records and their
+  successors; `citation()` resolves ids against any head on the mounted
+  history. A chain of local files mounts from its head alone through parent
+  locators; URL chains mount from a shelf `lineage`.
+- **`pikelet append`** writes a layer over a base or chain: new documents
+  from `--source` (repeatable), deletions with `--remove`, and replacements
+  with `--supersede <id>=<path>`, reusing the base's own encoder, chunking
+  and vocabulary-bloom geometry. It refuses a layer that would push
+  calibration drift past the limit unless `--allow-drift` is given.
+- **`pikelet compact`** collapses a chain into a fresh depth-0 base: live
+  records and quantized rows copied byte for byte, ids renumbered densely,
+  exact lexical statistics, and a refit abstention calibration (`--no-refit`
+  ships unscored). A lineage segment lets the compacted base translate
+  citations made against the chain.
+- **`pikelet rebase`** moves a layer onto a different head (for instance a
+  fork sibling), replaying only what the layer itself did, with
+  `--on-conflict refuse|skip|keep-both` and `--on-foreign refuse|drop`.
+- **MCP serves chains.** A shelf entry with a `lineage` mounts a chain,
+  pinned by its head identity; `list_packs` reports `chain`, `layers`,
+  `tombstones` and `calibrationStatus`, and counts live records; `get_record`
+  and search results carry `tombstoned`, `layer`, `supersededBy` and
+  `currentSuccessor`.
+
 ### Changed
 
 - **The repository is an npm workspace with the two packages under
@@ -249,6 +264,68 @@ backlog below. `pikelet` is unchanged at 0.8.1 and picks this up through its
 - Browser verification without a WebCrypto backend fails with a clear
   error instead of an obscure one.
 
+## 0.8.2 — `pikelet`
+
+### Fixed
+
+- **`pikelet mcp` answers `initialize` without waiting for its packs to
+  mount.** The server mounted every pack before replying to the MCP
+  handshake, and a URL mount fetches the pack's resident prefix first: about
+  5.6 s for the rust-book pack and 9.2 s for the 649 MiB Simple English
+  Wikipedia one. The published three-pack shelf took **10.2 s** to answer
+  `initialize`, past the point most MCP clients give up — so the documented
+  `mcp --shelf` command timed out for a new user.
+
+  Mounting now runs concurrently with the protocol loop. `initialize` returns
+  in about **0.6 s**; `tools/call` and `tools/list` await the mounts, so a
+  client gets a fast handshake and pays the mount cost on its first real call.
+  `tools/list` waits deliberately: it advertises the mounted pack names, and
+  answering early would name fewer packs than the server goes on to serve.
+
+### Documentation
+
+- The install instructions lead with `npm install -g pikelet --omit=optional`.
+  Measured on a clean project, that is 2 packages / 1.8 MB / "found 0
+  vulnerabilities" against 82 packages / 257 MB / "6 vulnerabilities (5 high,
+  1 critical)" — every advisory is inside `@xenova/transformers`, which only
+  the `create` scaffold path loads. The previous README put the flag in a
+  mid-paragraph aside and estimated its cost at "~140 MB".
+- README carries an upgrade note for the `pikelet-wasm@0.8.1` heap-overflow
+  fix, which shipped without one.
+
+## 0.8.1 — `pikelet-wasm`
+
+Security release, cut from `main` ahead of the rest of the backlog, which
+shipped in 0.9.0. `pikelet` is unchanged at 0.8.1 and picks this up through its
+`^0.8.0` dependency range.
+
+### Security
+
+- **A `.pikelet` can no longer overflow the encoder's WASM heap through an
+  out-of-range `maxTokens`.** `complete/inline-transformer.mjs` sized the
+  kernel's ids and hidden-state buffers from the artifact's own encoder
+  declaration, clamped with `Math.min(declaration.maxTokens || 512, 512)` —
+  an upper bound only. A pack declaring `maxTokens: 1` made the windowing
+  arithmetic (`windowLen = maxSeq - 2`) negative, so `interior.slice(w *
+  windowLen, (w + 1) * windowLen)` returned nearly the whole token sequence
+  and the forward pass wrote past both allocations: a 60-token query wrote
+  about 92 KB into a 1,536-byte buffer. `maxTokens` is now validated as an
+  integer in `[8, 512]` at parse, before anything is allocated, `maxSeq` is
+  clamped on both sides, and both call sites assert the window fits.
+
+  This fired at **mount**, before any query: the reader awaits the encoder
+  for a non-deferred query-interp segment (and prefetches it otherwise), and
+  the declaration's own `testVectors` are verified after the buffers exist,
+  so a long test vector was itself a trigger. Every integrity check passed,
+  because the attacker authored the file and its digests bind their bytes.
+
+  Affects anyone who mounts a `.pikelet` they did not compile — the format's
+  intended use, since packs are distributed and range-read from URLs. The
+  write is confined to the encoder's WASM linear memory and does not reach
+  host JavaScript objects, which is why it is rated high rather than
+  critical. Upgrade to get the bound; there is no workaround short of not
+  mounting third-party packs.
+
 ## pikelet 0.8.1 — 2026-09-18
 
 - `pikelet` records `mcpName: "io.github.mcn92/pikelet"` and ships

package/README.md

@@ -10,10 +10,26 @@ A `.pikelet` can carry the source text, semantic index, keyword index, query enc
 
 The `pikelet` CLI requires Node 20+; the `pikelet-wasm` library runs on Node 18+ (CI tests 18, 20, 22), browsers, and Cloudflare Workers.
 
+> **Upgrade to `pikelet-wasm@0.8.1` if you mount packs you did not compile.**
+> 0.8.0 and earlier sized the inline encoder's WASM buffers from the
+> artifact's own `maxTokens` with an upper bound only, so a pack declaring an
+> out-of-range value could overflow them — at mount, before any query, and
+> with every integrity check passing, because the pack's author is the one who
+> signed it. 0.8.1 validates the bound. Packs the toolchain produces were
+> never affected. `pikelet@0.8.1` picks the fix up through its dependency
+> range; see the CHANGELOG for the detail.
+
 ```bash
 npx pikelet compile --source ./docs --out docs.pikelet
 ```
 
+Installing rather than using `npx`? Use `npm install -g pikelet
+--omit=optional`. The optional dependency is `@xenova/transformers`, which
+only the `create` scaffold path uses. Measured on a clean project, omitting it
+takes the install from **82 packages / 257 MB / "6 vulnerabilities (5 high, 1
+critical)"** to **2 packages / 1.8 MB / "found 0 vulnerabilities"** — the
+advisories are all inside a dependency `compile` and `mcp` never load.
+
 ```text
 Ingested 3 docs -> 3 chunks
 Embedded 3/3 chunks with inline transformer
@@ -47,7 +63,7 @@ console.log(out.matchQuality, out.results[0]?.title);
 // 'strong' 'Snapshot restore'
 ```
 
-That's the whole loop. `compile` also takes a live URL (`--source https://docs.example.com`) instead of a directory. If you want a deployed search app — a Worker + UI, not a file — use `npx pikelet create` instead; see [`packages/pikelet/README.md`](packages/pikelet/README.md) for the full CLI reference and the tradeoffs between the two. `compile` never needs the scaffold path's `@xenova/transformers` dependency; skip its ~140 MB install with `npm install -g pikelet --omit=optional` if you only need `compile`/`mcp`.
+That's the whole loop. `compile` also takes a live URL (`--source https://docs.example.com`) instead of a directory. If you want a deployed search app — a Worker + UI, not a file — use `npx pikelet create` instead; see [`packages/pikelet/README.md`](packages/pikelet/README.md) for the full CLI reference and the tradeoffs between the two. `create` is the one path that needs the optional `@xenova/transformers` dependency, which is why the install above omits it by default.
 
 Under the hood, the file is one container for everything a reader needs:
 

package/complete/builder.mjs

@@ -456,16 +456,29 @@ function selfQueriesFromSnapshot(graph, maxQueries) {
 // explicitly instead of misreading the corpus tables.
 export const PROFILE_V1 = 'pikelet-complete-v1';
 export const PROFILE_V2 = 'pikelet-complete-v2';
-export const FORMAT_VERSIONS = { [PROFILE_V1]: 1, [PROFILE_V2]: 2 };
+// A layer is a complete-profile container under formatVersion 2 with its own
+// manifest profile string (LAYERED_PROFILE.md 4.1). A reader that predates the
+// layered profile sees the string under formatVersion 2 and refuses with its
+// existing "unsupported profile" error, which is the intended behavior: a layer
+// opened alone would silently serve a fraction of a corpus.
+export const PROFILE_LAYER = 'pikelet-layer-v1';
+export const FORMAT_VERSIONS = { [PROFILE_V1]: 1, [PROFILE_V2]: 2, [PROFILE_LAYER]: 2 };
 
 export function assemblePikeletFile(manifestFields, segments, outPath) {
   const formatVersion = FORMAT_VERSIONS[manifestFields.profile];
   if (!formatVersion) {
     throw new Error(`manifest.profile must be ${PROFILE_V1} or ${PROFILE_V2}, got ${manifestFields.profile}`);
   }
+  const isLayer = manifestFields.profile === PROFILE_LAYER;
   const layout = manifestFields.corpus?.layout;
-  if (formatVersion === 2 && layout !== CORPUS_LAYOUT_V2) {
-    throw new Error(`${PROFILE_V2} requires manifest.corpus from buildCorpusSegment() (layout ${CORPUS_LAYOUT_V2})`);
+  // A tombstone-only layer (4.1) carries a corpus object with records 0 and
+  // MUST omit the layout fields, because it has no corpus segment to describe.
+  const tombstoneOnlyLayer = isLayer && manifestFields.corpus?.records === 0;
+  if (formatVersion === 2 && !tombstoneOnlyLayer && layout !== CORPUS_LAYOUT_V2) {
+    throw new Error(`${manifestFields.profile} requires manifest.corpus from buildCorpusSegment() (layout ${CORPUS_LAYOUT_V2})`);
+  }
+  if (tombstoneOnlyLayer && layout !== undefined) {
+    throw new Error('a tombstone-only layer must omit corpus.layout');
   }
   if (formatVersion === 1 && layout !== undefined) {
     throw new Error(`${PROFILE_V1} carries corpus layout v1 (buildCorpusSegmentFromBuffers); got layout ${layout}`);
@@ -479,7 +492,7 @@ export function assemblePikeletFile(manifestFields, segments, outPath) {
   // sketch's own integrity chain to the complete identity without forcing a
   // whole-segment read at open (the index segment stays lazy).
   let indexCommitment;
-  if (formatVersion >= 2) {
+  if (formatVersion >= 2 && !tombstoneOnlyLayer) {
     const index = segments.find((s) => s.kind === 'index');
     if (!index || index.bytes.length < 256) {
       throw new Error('index segment must carry a 256-byte sketch header');

package/complete/format.mjs

@@ -4,5 +4,10 @@
 export const MAGIC = 0x31465350; // "PSF1"
 export const HEADER_BYTES = 64;
 export const TABLE_ENTRY_BYTES = 48;
-export const KINDS = { index: 1, corpus: 2, 'query-interp': 3, evaluation: 4, lexical: 5 };
+// Kinds 6 and 7 belong to the layered profile (LAYERED_PROFILE.md 4.2): a
+// layer carries exactly one tombstones segment, and a base compacted from a
+// chain carries at most one lineage segment. They are registered here so the
+// shared table has one definition; a complete-profile artifact never has
+// either, and the required-segment tables stay per-profile.
+export const KINDS = { index: 1, corpus: 2, 'query-interp': 3, evaluation: 4, lexical: 5, tombstones: 6, lineage: 7 };
 export const KIND_NAMES = Object.fromEntries(Object.entries(KINDS).map(([name, kind]) => [kind, name]));