pikelet-wasm 0.8.1 → 0.9.1

package/CHANGELOG.md

@@ -8,43 +8,81 @@ Through 0.6.0 these were published as `pancake-wasm` and
 packages are pre-1.0: a minor bump may carry breaking changes, and each entry lists them
 first.
 
-## 0.8.1 — `pikelet-wasm`
+## pikelet-wasm 0.9.1 / pikelet 0.9.1 — 2026-09-30
 
-Security release, cut from `main` ahead of the rest of the unreleased
-backlog below. `pikelet` is unchanged at 0.8.1 and picks this up through its
-`^0.8.0` dependency range.
+Fixes to the layered profile shipped in 0.9.0, headlined by a chain that
+lost its abstention verdicts as soon as a layer deleted anything. Both
+packages move to 0.9.1 together, and `pikelet` requires `pikelet-wasm`
+^0.9.1 so that `pikelet mcp` serves chains with the reader fixes.
 
-### Security
+### Compatibility
 
-- **A `.pikelet` can no longer overflow the encoder's WASM heap through an
-  out-of-range `maxTokens`.** `complete/inline-transformer.mjs` sized the
-  kernel's ids and hidden-state buffers from the artifact's own encoder
-  declaration, clamped with `Math.min(declaration.maxTokens || 512, 512)` —
-  an upper bound only. A pack declaring `maxTokens: 1` made the windowing
-  arithmetic (`windowLen = maxSeq - 2`) negative, so `interior.slice(w *
-  windowLen, (w + 1) * windowLen)` returned nearly the whole token sequence
-  and the forward pass wrote past both allocations: a 60-token query wrote
-  about 92 KB into a 1,536-byte buffer. `maxTokens` is now validated as an
-  integer in `[8, 512]` at parse, before anything is allocated, `maxSeq` is
-  clamped on both sides, and both call sites assert the window fits.
+- **Layers from `append` and `rebase` now record a parent locator** when
+  the parent is a local file under the output's directory, so the same
+  inputs produce a layer with a different identity than under 0.9.0.
+- **`--remove` and `--supersede` refuse ids they used to truncate**
+  (`1,2,3`, `12abc`, `1.5`, `1e3`); give one id per flag.
+- **A chain-member location ending in `#` plus hex is read as a pin.** A
+  full 64-hex suffix is verified against the member's identity, and a
+  shorter one is refused; it is no longer treated as part of a filename.
 
-  This fired at **mount**, before any query: the reader awaits the encoder
-  for a non-deferred query-interp segment (and prefetches it otherwise), and
-  the declaration's own `testVectors` are verified after the buffers exist,
-  so a long test vector was itself a trigger. Every integrity check passed,
-  because the attacker authored the file and its digests bind their bytes.
-
-  Affects anyone who mounts a `.pikelet` they did not compile — the format's
-  intended use, since packs are distributed and range-read from URLs. The
-  write is confined to the encoder's WASM linear memory and does not reach
-  host JavaScript objects, which is why it is rated high rather than
-  critical. Upgrade to get the bound; there is no workaround short of not
-  mounting third-party packs.
+### Fixed
 
-## Unreleased
+- **`pikelet-wasm` now exports `./package.json`.** `require.resolve('pikelet-wasm/package.json')`
+  and JSON imports of it threw `ERR_PACKAGE_PATH_NOT_EXPORTED`, which
+  tooling that reads the installed version relies on.
+- **`pikelet append` parses `--remove` and `--supersede` ids strictly.**
+  `--remove 1,2,3` used to tombstone record 1 alone, and `--remove 12abc`
+  record 12, without an error; each flag now takes exactly one decimal id.
+- **`#<sha256>` pins on chain members are enforced.** Every layered
+  producer's usage line advertised `--parent <file|url>[#identity]`, but
+  nothing parsed the pin: a local path failed as a missing file, and a URL
+  dropped the fragment and built on whatever the server returned. `--parent`,
+  `--head`, `--onto`, `--old-parent` and `--layer` now verify each pinned
+  member's manifest identity before opening anything, and refuse a
+  truncated pin.
+- **`append` and `rebase` record a parent locator by default.** They wrote
+  one only when given `--parent-locator`, so a rebased layer could not be
+  mounted from its head alone and its next rebase needed `--old-parent`.
+  When the parent is a local file under the output's directory, its relative
+  path is now recorded. An explicit `--parent-locator` is validated up front
+  instead of being written as-is and refused by every reader.
+- **A URL chain mounts from its head alone, and locator redirects are
+  confined.** Walking up from a URL head failed at the first hop ("a locator
+  needs the child's own location") because a range source carried no URL,
+  and a parent's URL would then have been read as a file path. The walk now
+  resolves each parent over HTTP. A parent named by a publisher's locator is
+  confined to the child's origin and directory at every redirect hop, as
+  5.1.1 requires; lineage and host-resolver locations are the operator's
+  choice and are not confined. `httpRangeSource` gains `url` and a
+  `redirectGuard` option.
+- **A tombstone-only layer no longer unscores its chain.** A layer that only
+  deletes records carries no vocabulary bloom, and the reader counted that
+  as a layer hiding its vocabulary, marking the whole chain
+  `drift-exceeded`: on the astro pack, deleting 3 of 3,255 records (drift
+  0.0009 against a 0.20 limit) turned every query's verdict to `unscored`.
+  The rule now applies only to layers with records, and LAYERED_PROFILE.md
+  4.5 and 5.5 say so. Tombstone-only layers already written with 0.9.0
+  score again with no rebuild.
+- **`pikelet --help` documents `append`, `compact` and `rebase`**, including
+  `--supersede`, `--allow-drift`, `--max-depth` and pins.
+
+## pikelet-wasm 0.9.0 / pikelet 0.9.0 — 2026-09-29
+
+Everything merged to `main` since 0.8.0 that the 0.8.1 and 0.8.2 point
+releases below did not carry, headlined by the layered profile. Both
+packages move to 0.9.0 together. Pre-1.0, this minor carries breaking
+changes; they are listed first.
 
 ### Breaking / compatibility
 
+- **`pikelet` 0.9.0 requires `pikelet-wasm` ^0.9.0.** Its new `append`,
+  `compact` and `rebase` commands and its chain mounts import
+  `pikelet-wasm/complete/layer-*.mjs` and `complete/format.mjs`, which no
+  0.8.x exports; upgrade both together. A layer's profile string,
+  `pikelet-layer-v1`, is refused by every earlier reader by design: a layer
+  opened alone would serve a fraction of a corpus.
+
 - **The `PANCAKE_ENCODER_WEIGHTS_URL`, `PANCAKE_SEARCH_EMBED_WORKERS`,
   `PANCAKE_SEARCH_STUB_EMBEDDINGS`, and `PANCAKE_SEARCH_PYTHON`
   environment variable fallbacks are removed.** 0.7.0 introduced
@@ -59,6 +97,42 @@ backlog below. `pikelet` is unchanged at 0.8.1 and picks this up through its
   have; CI tests 18, 20 and 22. The `pikelet` CLI is unchanged at
   `>=20`.
 
+### Added
+
+- **Layered search artifacts** (`LAYERED_PROFILE.md`, Draft 4). A pack can
+  be updated by appending a small `pikelet-layer-v1` file instead of
+  recompiling: a layer carries only its new records plus a cumulative
+  tombstone mask and supersession edges, and commits to its parent's
+  identity, so a chain's head identity pins every byte a query can touch.
+  `openPikeletChain` (`pikelet-wasm/complete/layer-reader.mjs`) mounts base
+  and layers, verifies every parent commitment and spec rule at mount, and
+  queries them as one corpus: masked candidate generation per tier,
+  chain-global BM25 statistics, one fusion, and the base's abstention fit
+  inherited until calibration drift passes the reader's limit (0.20 by
+  default), after which results ship `unscored`. Records carry the layer
+  that owns them; `record()` reports tombstoned records and their
+  successors; `citation()` resolves ids against any head on the mounted
+  history. A chain of local files mounts from its head alone through parent
+  locators; URL chains mount from a shelf `lineage`.
+- **`pikelet append`** writes a layer over a base or chain: new documents
+  from `--source` (repeatable), deletions with `--remove`, and replacements
+  with `--supersede <id>=<path>`, reusing the base's own encoder, chunking
+  and vocabulary-bloom geometry. It refuses a layer that would push
+  calibration drift past the limit unless `--allow-drift` is given.
+- **`pikelet compact`** collapses a chain into a fresh depth-0 base: live
+  records and quantized rows copied byte for byte, ids renumbered densely,
+  exact lexical statistics, and a refit abstention calibration (`--no-refit`
+  ships unscored). A lineage segment lets the compacted base translate
+  citations made against the chain.
+- **`pikelet rebase`** moves a layer onto a different head (for instance a
+  fork sibling), replaying only what the layer itself did, with
+  `--on-conflict refuse|skip|keep-both` and `--on-foreign refuse|drop`.
+- **MCP serves chains.** A shelf entry with a `lineage` mounts a chain,
+  pinned by its head identity; `list_packs` reports `chain`, `layers`,
+  `tombstones` and `calibrationStatus`, and counts live records; `get_record`
+  and search results carry `tombstoned`, `layer`, `supersededBy` and
+  `currentSuccessor`.
+
 ### Changed
 
 - **The repository is an npm workspace with the two packages under
@@ -249,6 +323,68 @@ backlog below. `pikelet` is unchanged at 0.8.1 and picks this up through its
 - Browser verification without a WebCrypto backend fails with a clear
   error instead of an obscure one.
 
+## 0.8.2 — `pikelet`
+
+### Fixed
+
+- **`pikelet mcp` answers `initialize` without waiting for its packs to
+  mount.** The server mounted every pack before replying to the MCP
+  handshake, and a URL mount fetches the pack's resident prefix first: about
+  5.6 s for the rust-book pack and 9.2 s for the 649 MiB Simple English
+  Wikipedia one. The published three-pack shelf took **10.2 s** to answer
+  `initialize`, past the point most MCP clients give up — so the documented
+  `mcp --shelf` command timed out for a new user.
+
+  Mounting now runs concurrently with the protocol loop. `initialize` returns
+  in about **0.6 s**; `tools/call` and `tools/list` await the mounts, so a
+  client gets a fast handshake and pays the mount cost on its first real call.
+  `tools/list` waits deliberately: it advertises the mounted pack names, and
+  answering early would name fewer packs than the server goes on to serve.
+
+### Documentation
+
+- The install instructions lead with `npm install -g pikelet --omit=optional`.
+  Measured on a clean project, that is 2 packages / 1.8 MB / "found 0
+  vulnerabilities" against 82 packages / 257 MB / "6 vulnerabilities (5 high,
+  1 critical)" — every advisory is inside `@xenova/transformers`, which only
+  the `create` scaffold path loads. The previous README put the flag in a
+  mid-paragraph aside and estimated its cost at "~140 MB".
+- README carries an upgrade note for the `pikelet-wasm@0.8.1` heap-overflow
+  fix, which shipped without one.
+
+## 0.8.1 — `pikelet-wasm`
+
+Security release, cut from `main` ahead of the rest of the backlog, which
+shipped in 0.9.0. `pikelet` is unchanged at 0.8.1 and picks this up through its
+`^0.8.0` dependency range.
+
+### Security
+
+- **A `.pikelet` can no longer overflow the encoder's WASM heap through an
+  out-of-range `maxTokens`.** `complete/inline-transformer.mjs` sized the
+  kernel's ids and hidden-state buffers from the artifact's own encoder
+  declaration, clamped with `Math.min(declaration.maxTokens || 512, 512)` —
+  an upper bound only. A pack declaring `maxTokens: 1` made the windowing
+  arithmetic (`windowLen = maxSeq - 2`) negative, so `interior.slice(w *
+  windowLen, (w + 1) * windowLen)` returned nearly the whole token sequence
+  and the forward pass wrote past both allocations: a 60-token query wrote
+  about 92 KB into a 1,536-byte buffer. `maxTokens` is now validated as an
+  integer in `[8, 512]` at parse, before anything is allocated, `maxSeq` is
+  clamped on both sides, and both call sites assert the window fits.
+
+  This fired at **mount**, before any query: the reader awaits the encoder
+  for a non-deferred query-interp segment (and prefetches it otherwise), and
+  the declaration's own `testVectors` are verified after the buffers exist,
+  so a long test vector was itself a trigger. Every integrity check passed,
+  because the attacker authored the file and its digests bind their bytes.
+
+  Affects anyone who mounts a `.pikelet` they did not compile — the format's
+  intended use, since packs are distributed and range-read from URLs. The
+  write is confined to the encoder's WASM linear memory and does not reach
+  host JavaScript objects, which is why it is rated high rather than
+  critical. Upgrade to get the bound; there is no workaround short of not
+  mounting third-party packs.
+
 ## pikelet 0.8.1 — 2026-09-18
 
 - `pikelet` records `mcpName: "io.github.mcn92/pikelet"` and ships

package/README.md

@@ -10,10 +10,26 @@ A `.pikelet` can carry the source text, semantic index, keyword index, query enc
 
 The `pikelet` CLI requires Node 20+; the `pikelet-wasm` library runs on Node 18+ (CI tests 18, 20, 22), browsers, and Cloudflare Workers.
 
+> **Upgrade to `pikelet-wasm@0.8.1` if you mount packs you did not compile.**
+> 0.8.0 and earlier sized the inline encoder's WASM buffers from the
+> artifact's own `maxTokens` with an upper bound only, so a pack declaring an
+> out-of-range value could overflow them — at mount, before any query, and
+> with every integrity check passing, because the pack's author is the one who
+> signed it. 0.8.1 validates the bound. Packs the toolchain produces were
+> never affected. `pikelet@0.8.1` picks the fix up through its dependency
+> range; see the CHANGELOG for the detail.
+
 ```bash
 npx pikelet compile --source ./docs --out docs.pikelet
 ```
 
+Installing rather than using `npx`? Use `npm install -g pikelet
+--omit=optional`. The optional dependency is `@xenova/transformers`, which
+only the `create` scaffold path uses. Measured on a clean project, omitting it
+takes the install from **82 packages / 257 MB / "6 vulnerabilities (5 high, 1
+critical)"** to **2 packages / 1.8 MB / "found 0 vulnerabilities"** — the
+advisories are all inside a dependency `compile` and `mcp` never load.
+
 ```text
 Ingested 3 docs -> 3 chunks
 Embedded 3/3 chunks with inline transformer
@@ -47,7 +63,7 @@ console.log(out.matchQuality, out.results[0]?.title);
 // 'strong' 'Snapshot restore'
 ```
 
-That's the whole loop. `compile` also takes a live URL (`--source https://docs.example.com`) instead of a directory. If you want a deployed search app — a Worker + UI, not a file — use `npx pikelet create` instead; see [`packages/pikelet/README.md`](packages/pikelet/README.md) for the full CLI reference and the tradeoffs between the two. `compile` never needs the scaffold path's `@xenova/transformers` dependency; skip its ~140 MB install with `npm install -g pikelet --omit=optional` if you only need `compile`/`mcp`.
+That's the whole loop. `compile` also takes a live URL (`--source https://docs.example.com`) instead of a directory. If you want a deployed search app — a Worker + UI, not a file — use `npx pikelet create` instead; see [`packages/pikelet/README.md`](packages/pikelet/README.md) for the full CLI reference and the tradeoffs between the two. `create` is the one path that needs the optional `@xenova/transformers` dependency, which is why the install above omits it by default.
 
 Under the hood, the file is one container for everything a reader needs:
 

package/complete/builder.mjs

@@ -456,16 +456,29 @@ function selfQueriesFromSnapshot(graph, maxQueries) {
 // explicitly instead of misreading the corpus tables.
 export const PROFILE_V1 = 'pikelet-complete-v1';
 export const PROFILE_V2 = 'pikelet-complete-v2';
-export const FORMAT_VERSIONS = { [PROFILE_V1]: 1, [PROFILE_V2]: 2 };
+// A layer is a complete-profile container under formatVersion 2 with its own
+// manifest profile string (LAYERED_PROFILE.md 4.1). A reader that predates the
+// layered profile sees the string under formatVersion 2 and refuses with its
+// existing "unsupported profile" error, which is the intended behavior: a layer
+// opened alone would silently serve a fraction of a corpus.
+export const PROFILE_LAYER = 'pikelet-layer-v1';
+export const FORMAT_VERSIONS = { [PROFILE_V1]: 1, [PROFILE_V2]: 2, [PROFILE_LAYER]: 2 };
 
 export function assemblePikeletFile(manifestFields, segments, outPath) {
   const formatVersion = FORMAT_VERSIONS[manifestFields.profile];
   if (!formatVersion) {
     throw new Error(`manifest.profile must be ${PROFILE_V1} or ${PROFILE_V2}, got ${manifestFields.profile}`);
   }
+  const isLayer = manifestFields.profile === PROFILE_LAYER;
   const layout = manifestFields.corpus?.layout;
-  if (formatVersion === 2 && layout !== CORPUS_LAYOUT_V2) {
-    throw new Error(`${PROFILE_V2} requires manifest.corpus from buildCorpusSegment() (layout ${CORPUS_LAYOUT_V2})`);
+  // A tombstone-only layer (4.1) carries a corpus object with records 0 and
+  // MUST omit the layout fields, because it has no corpus segment to describe.
+  const tombstoneOnlyLayer = isLayer && manifestFields.corpus?.records === 0;
+  if (formatVersion === 2 && !tombstoneOnlyLayer && layout !== CORPUS_LAYOUT_V2) {
+    throw new Error(`${manifestFields.profile} requires manifest.corpus from buildCorpusSegment() (layout ${CORPUS_LAYOUT_V2})`);
+  }
+  if (tombstoneOnlyLayer && layout !== undefined) {
+    throw new Error('a tombstone-only layer must omit corpus.layout');
   }
   if (formatVersion === 1 && layout !== undefined) {
     throw new Error(`${PROFILE_V1} carries corpus layout v1 (buildCorpusSegmentFromBuffers); got layout ${layout}`);
@@ -479,7 +492,7 @@ export function assemblePikeletFile(manifestFields, segments, outPath) {
   // sketch's own integrity chain to the complete identity without forcing a
   // whole-segment read at open (the index segment stays lazy).
   let indexCommitment;
-  if (formatVersion >= 2) {
+  if (formatVersion >= 2 && !tombstoneOnlyLayer) {
     const index = segments.find((s) => s.kind === 'index');
     if (!index || index.bytes.length < 256) {
       throw new Error('index segment must carry a 256-byte sketch header');

package/complete/format.mjs

@@ -4,5 +4,10 @@
 export const MAGIC = 0x31465350; // "PSF1"
 export const HEADER_BYTES = 64;
 export const TABLE_ENTRY_BYTES = 48;
-export const KINDS = { index: 1, corpus: 2, 'query-interp': 3, evaluation: 4, lexical: 5 };
+// Kinds 6 and 7 belong to the layered profile (LAYERED_PROFILE.md 4.2): a
+// layer carries exactly one tombstones segment, and a base compacted from a
+// chain carries at most one lineage segment. They are registered here so the
+// shared table has one definition; a complete-profile artifact never has
+// either, and the required-segment tables stay per-profile.
+export const KINDS = { index: 1, corpus: 2, 'query-interp': 3, evaluation: 4, lexical: 5, tombstones: 6, lineage: 7 };
 export const KIND_NAMES = Object.fromEntries(Object.entries(KINDS).map(([name, kind]) => [kind, name]));

package/complete/index.d.ts

@@ -34,6 +34,8 @@ export interface HttpRangeSourceStats {
 
 export interface HttpRangeSource extends CompleteRangeSource {
   stats: HttpRangeSourceStats;
+  /** The URL as given, before any redirect; a layer's parent locator resolves against it. */
+  url: string;
   init(): Promise<void>;
 }
 
@@ -55,6 +57,13 @@ export declare function httpRangeSource(url: string, options?: {
    * disables. CDNs rate-limit the parallel range bursts a query issues.
    */
   maxRetries?: number;
+  /**
+   * Called with each redirect target before it is followed; throw to refuse
+   * the hop. When set, range reads never auto-follow redirects. A chain walk
+   * sets it for a parent named by a layer's locator, confining every hop to
+   * the child's origin and directory (LAYERED_PROFILE.md 5.1.1).
+   */
+  redirectGuard?: (target: string) => void;
 }): HttpRangeSource;
 
 export interface CompleteQueryResult {