pikelet-wasm 0.7.0 → 0.8.1

package/CHANGELOG.md

@@ -1,15 +1,350 @@
 # Changelog
 
 Release notes for the two published packages in this repository,
-`pikelet-wasm` (engine + artifact readers/builders) and
-`create-pancake-search` (scaffolder, `doctor`, Docusaurus plugin). Both are
-pre-1.0: a minor bump may carry breaking changes, and each entry lists them
+`pikelet-wasm` (engine + artifact readers/builders) and `pikelet` (the
+CLI: compile, mcp, doctor, scaffold; plus the Docusaurus plugin).
+Through 0.6.0 these were published as `pancake-wasm` and
+`create-pancake-search`; both old names are deprecated pointers. Both
+packages are pre-1.0: a minor bump may carry breaking changes, and each entry lists them
 first.
 
+## 0.8.1 — `pikelet-wasm`
+
+Security release, cut from `main` ahead of the rest of the unreleased
+backlog below. `pikelet` is unchanged at 0.8.1 and picks this up through its
+`^0.8.0` dependency range.
+
+### Security
+
+- **A `.pikelet` can no longer overflow the encoder's WASM heap through an
+  out-of-range `maxTokens`.** `complete/inline-transformer.mjs` sized the
+  kernel's ids and hidden-state buffers from the artifact's own encoder
+  declaration, clamped with `Math.min(declaration.maxTokens || 512, 512)` —
+  an upper bound only. A pack declaring `maxTokens: 1` made the windowing
+  arithmetic (`windowLen = maxSeq - 2`) negative, so `interior.slice(w *
+  windowLen, (w + 1) * windowLen)` returned nearly the whole token sequence
+  and the forward pass wrote past both allocations: a 60-token query wrote
+  about 92 KB into a 1,536-byte buffer. `maxTokens` is now validated as an
+  integer in `[8, 512]` at parse, before anything is allocated, `maxSeq` is
+  clamped on both sides, and both call sites assert the window fits.
+
+  This fired at **mount**, before any query: the reader awaits the encoder
+  for a non-deferred query-interp segment (and prefetches it otherwise), and
+  the declaration's own `testVectors` are verified after the buffers exist,
+  so a long test vector was itself a trigger. Every integrity check passed,
+  because the attacker authored the file and its digests bind their bytes.
+
+  Affects anyone who mounts a `.pikelet` they did not compile — the format's
+  intended use, since packs are distributed and range-read from URLs. The
+  write is confined to the encoder's WASM linear memory and does not reach
+  host JavaScript objects, which is why it is rated high rather than
+  critical. Upgrade to get the bound; there is no workaround short of not
+  mounting third-party packs.
+
 ## Unreleased
 
 ### Breaking / compatibility
 
+- **The `PANCAKE_ENCODER_WEIGHTS_URL`, `PANCAKE_SEARCH_EMBED_WORKERS`,
+  `PANCAKE_SEARCH_STUB_EMBEDDINGS`, and `PANCAKE_SEARCH_PYTHON`
+  environment variable fallbacks are removed.** 0.7.0 introduced
+  `PIKELET_ENCODER_WEIGHTS_URL` / `PIKELET_EMBED_WORKERS` while still
+  honoring the old names; those old names no longer work. Use
+  `PIKELET_ENCODER_WEIGHTS_URL`, `PIKELET_EMBED_WORKERS`,
+  `PIKELET_SEARCH_EMBED_WORKERS`, `PIKELET_SEARCH_STUB_EMBEDDINGS`, and
+  `PIKELET_SEARCH_PYTHON`. Pre-1.0; scripts/CI referencing the old
+  names need updating.
+- **`pikelet-wasm` declares `engines.node >=18`** (was `>=16`).
+  `complete/sources.mjs` uses the global `fetch`, which Node 16 does not
+  have; CI tests 18, 20 and 22. The `pikelet` CLI is unchanged at
+  `>=20`.
+
+### Changed
+
+- **The repository is an npm workspace with the two packages under
+  `packages/`.** `packages/pikelet-wasm/` holds the published library —
+  entrypoints and readers under `src/` (`src/core`, `src/artifact`,
+  `src/errors`, `src/loader`), the C++ engine under `engine/`
+  (formerly the root `src/`), `complete/`, `native/` and the four tracked
+  `dist/` build outputs; `packages/pikelet/` holds the CLI (formerly
+  `pikelet/`). The root `package.json` is a private workspace root
+  carrying the test ladder and dev tooling; `npm install` links
+  `node_modules/pikelet-wasm` and `node_modules/pikelet` to the checkouts,
+  so tests, benchmarks and examples import `pikelet-wasm` exactly as
+  consumers do, and the CLI's and Docusaurus plugin's "repo root is
+  pikelet-wasm" resolution fallbacks are gone. The published tarball's
+  file list is unchanged: `main`/`types`/`exports` point into `src/`, and
+  `CHANGELOG.md`, `LICENSE`, `NOTICE` and `README.md` are staged into the
+  package by an npm `prepack` script instead of being duplicated in the
+  tree. `run_tests.js` lives at `test/run_tests.js`; the engine benchmark
+  scripts, whose relative requires had been left pointing one directory
+  too high by the benchmarks move, resolve the package again.
+- **Examples renamed, not renumbered**: `examples/04-static-wiki-pack` →
+  `examples/static-wiki-pack`, `examples/05-one-file-search` →
+  `examples/one-file-search`, `examples/06-mcp-knowledge-pack` →
+  `examples/mcp-knowledge-pack`; `npm run demo:veyra` and the committed
+  `veyra.mcp.json` follow. The raw browser demo page moves from `dist/`
+  to `examples/technical-demo.html` (loads `../dist/engine.js`), and
+  `docs/SYSTEM_DESIGN.md` is `docs/architecture.md`. The wiki playground's
+  built bundle and deployment are unchanged.
+- **README trimmed to the pitch, quickstart, live demo, numbers and
+  links** (577 → ~340 lines). Its long-form sections moved verbatim to
+  `docs/how-a-query-runs.md` (what is inside the file, compression cost,
+  remote execution, integrity), `docs/veyra-ablation.md` and
+  `docs/why-a-file.md`, linked from a new "Deeper dives" section.
+- **Benchmarks consolidated under `benchmarks/`**: `bench/beir` →
+  `benchmarks/beir`, `bench/range-proof` → `benchmarks/range-proof`, the
+  engine suite (formerly `benchmarks/`) → `benchmarks/engine`, and the
+  committed release runs (formerly `benchmark_results/release`) →
+  `benchmarks/results`, with raw runs under the ignored
+  `benchmarks/results/raw`. `npm run bench` runs `benchmarks/engine/run.js`.
+  Nothing published changes.
+- **Abstention calibration is fit against a different negative pool.**
+  Hard negatives are now ablation pairs (a verified positive's own text,
+  scored with its source excluded), entity swaps (a proper noun replaced
+  by a corpus entity that never co-occurs with it), and held-out-document
+  questions (title questions scored with that document excluded). A fit
+  whose `d0` weight comes out non-negative ships `unscored` instead. The
+  calibration summary (`fitAuc`, `cvAuc`, `cvAucHard`, per-class counts)
+  is persisted in the evaluation segment and reported by `verify_pack`;
+  golden queries are sampled from retrieval-verified positives. Packs
+  rebuilt with this calibrator get a different fit and therefore a
+  different identity; existing packs are unaffected.
+- **`query()` returns `grounding`** — `{ recordId, coverage, covered,
+  uncovered }`: which of the query's content words the best passage
+  restates and which no passage contains — and each result carries
+  `vectorRank`, `lexicalRank` and `fusedRank`. A verbatim match of three
+  or more consecutive query words is pinned to rank 1 ahead of semantic
+  near-misses. Phrase pinning and the coverage term now hydrate their
+  candidates in one batch.
+- **MCP `search` ships results under a `"none"` verdict by default**,
+  with a note, instead of withholding them (`showAbstained: false`
+  restores withholding); packs without a fitted calibrator get an
+  `unscoredWarning`. Each section carries `grounding`. The tool
+  description and `server/discover` instructions describe `matchQuality`
+  as how directly the text restates the question, not whether the answer
+  is present, and tell the caller to treat a queried-for term in
+  `grounding.uncovered` as unestablished and to search again for the
+  second hop of a chained question.
+- **The reader no longer reads `coverage.useMaxSim`**; the word-level
+  semantic grounding path is removed rather than left dormant (a pack
+  with the flag set was measured at 1.5–4.6 s per query).
+- `scripts/calibration-harness.mjs` reports false-abstention and
+  false-answer rates on labeled question sets; the sets live under
+  `test/relevance/`.
+- Public browser playground for the Wikipedia pack, with an ablation
+  tab.
+
+- **`pikelet-wasm/complete` exports `fuseCandidates()` and
+  `FUSION_DEFAULTS`.** The hybrid rank-fusion rule (reciprocal-rank fusion
+  of the exact-reranked vector order with the BM25 order) is one function
+  the reader, the calibrator (`calibrate.mjs`, which fits coverage on the
+  fused top passage) and the BEIR ladder (`query-E.mjs`) all call, instead
+  of three copies of the arithmetic. It takes `{ rrfK, lexicalWeight,
+  guardMargin }`.
+- **Hybrid fusion weights the lexical rank at 0.5 and keeps a confident
+  vector top-1 in place.** `FUSION_DEFAULTS` is now `{ rrfK: 60,
+  lexicalWeight: 0.5, guardMargin: 0.05 }`: the BM25 rank term counts half
+  as much as the vector rank term, and when the vector top-1 leads the
+  top-2 by at least 5% of its own distance it keeps rank 1 regardless of
+  fusion. Under equal weights any record present in both lists outranked a
+  vector-only rank 1 (`1/(60+v) + 1/(60+l) > 1/60` for every finite `v`),
+  so a query whose terms match many chunks — a character's name in a
+  novel — filled the top ranks with term matches and buried the passage
+  the encoder placed first. Measured with the evidence-rank report and the
+  fusion simulator (`test/relevance/README.md`): evidence in the top 3 on
+  Pride & Prejudice 36 → 42 of 157 direct questions, internal-docs
+  paraphrases 19 → 21 of 24, Veyra direct 19 → 20 of 20 and multihop
+  17 → 18 of 20; BEIR E by fused rank scifact nDCG@10 0.694 → 0.699
+  (recall@10 0.827 → 0.853), nfcorpus 0.3345 → 0.3359, arguana unchanged.
+  Verdicts on packs calibrated under the old rule move the same way (false
+  answers: internal-docs 10 → 9, Pride & Prejudice 74 → 66, Veyra 13 → 13);
+  a Veyra pack recalibrated under the new rule fits at cvAUC 0.890 (was
+  0.870). Packs need no rebuild to get the new ranking.
+### Fixed
+
+- **Engine constructors validate their configuration.** `pikelet_init`
+  (WASM and the native addon) passed caller-supplied `dims`, `M` and
+  `max_elements` straight into the index constructors. `M = 1` made the
+  level multiplier `1/log(1) = +inf`, whose conversion to an integer on
+  the first insert is undefined behaviour, and `max_elements * dims` or
+  `max_elements * M0` could wrap a 32-bit `size_t` on wasm32 into an
+  undersized arena. Both backends now reject `dims` outside 1..65536, `M`
+  outside 2..128 (0 still selects the default), `max_elements = 0`, and
+  products that do not fit `size_t`; WASM returns `INVALID_HANDLE`, the
+  native addon throws a `RangeError` (a C++ exception used to escape the
+  N-API boundary and abort the process). `Pikelet.create()` mirrors the
+  `dim` ceiling with an `INVALID_ARGUMENT` error, and the native
+  `pikelet_import` rejects a non-buffer argument with a `TypeError`
+  instead of continuing with a null view while a JS exception is pending.
+- **Artifacts with a NaN or infinite scale/offset are rejected.** A
+  producer signs whatever bytes it writes, so a non-finite affine pair in
+  a `.pikelet-range` record or a sketch artifact's affine table passed
+  every digest check and then made that row's distance non-finite, which
+  the result heap never evicts. The range reader rejects the record at
+  decode and the sketch reader rejects the table wherever it adopts it,
+  both with `SNAPSHOT_INVALID`.
+- **`stripMarkdown` no longer backtracks quadratically on backtick runs.**
+  The fenced-block and inline-code regexes took seconds to minutes on a
+  line of 40k backticks (each candidate fence length re-scanned the rest
+  of the document; each suffix of an inline run was retried). Both are
+  replaced by linear scanners with CommonMark's rules: a fence closes on
+  the first later fence line of the same character at least as long, an
+  unclosed fence runs to the end of the document, and an inline span of N
+  backticks closes on the next run of exactly N on the same line, may
+  contain runs of other lengths (`` ``a ` b`` ``), and drops one padding
+  space from each end. Under the old regexes a shorter closing fence or
+  an unclosed one left the block as prose.
+- **Folder ingestion reports what it skips and caps file size.** The walk
+  skipped symlinked files and directories silently (Dirents carry lstat
+  semantics); it now logs `warn: skipped symlink <path>` for each. Files
+  over 16 MiB are skipped with a warning naming the path and the limit
+  instead of being read whole into the build, matching the crawl path's
+  body cap.
+- **`pikelet rebuild --verbose` parses.** `--verbose` was read as a
+  boolean (`verbose: !!flags.verbose`) but was not registered as one, so
+  as the last argument it failed with "Missing value for --verbose" and
+  before another flag it swallowed that flag as its value.
+- **`verify_pack` surfaces a corrupt evaluation segment.** The MCP tool
+  turned any `evaluation()` failure into "no evaluation", reporting 0
+  golden queries and 0 probes, all passed, for a pack whose evaluation
+  segment failed its hash or did not parse. Those errors now propagate
+  as tool errors; a pack with no evaluation segment still reports zero.
+- **Generated Workers no longer echo internal error messages on 5xx.**
+  The `pikelet` scaffold's `worker.js` / `worker.artifact.js` templates
+  returned `error.message` for every failure, including unexpected ones,
+  which can name bundled asset paths and load state. A 5xx now carries
+  `{ error: "Internal server error", code }` and the original error goes
+  to the Worker log (`wrangler tail`); 4xx and 503 (`EMBED_UNAVAILABLE`)
+  bodies still carry the message. The legacy reference Worker's
+  "Invalid JSON body" error no longer appends the parser's message.
+  Projects generated by earlier versions keep their old template.
+- **References the 0.8.0 rename left broken are repaired.** The engine
+  benchmark scripts called `native.pancake_*` (the N-API addon exports
+  `pikelet_*`); the ANN-Benchmarks adapter imported `pancake_py` (the
+  pybind module is `pikelet_py`) and, after the benchmarks move, resolved
+  the repo root one level short; `dist/technical-demo.html` and
+  `test/ios-smoke.html` called `_pancake_*` WASM exports (the iOS page also
+  inlined a pre-rename engine build and used the pre-`ef_search` query
+  ABI — regenerated from the current `dist/`); the Docusaurus plugin's
+  webpack rule matched `pancake-(artifact|errors).js`, which no longer
+  exists, so it never applied; READMEs showed `openPancakeFile`,
+  `PancakeRangeArtifact`, `PancakeSketchArtifact` and `PancakeIndex`. The
+  scaffold template's UI title and the wiki example page's wordmark and
+  GitHub links now say Pikelet. `docs/history.md` records the rename and
+  lists what intentionally keeps the old name.
+- **`compact()`'s rebuild path no longer leaves the index half-cleared**
+  when a reinsertion fails: the old graph is released only after the
+  rebuilt one is complete, so a failed rebuild returns with the index
+  unchanged. Previously the next `search`/`insert` read out of bounds.
+- Coverage word rules: possessives (`darcy's`) match plain mentions,
+  eleven quantifier/hedge words are stopwords, and numerals are exempt
+  from the minimum word length (`Chamber 4` vs `Chamber 5`).
+- Ingest: `**/dir/**` globs no longer substring-match sibling names,
+  minified HTML with unquoted attributes chunks correctly, a leading BOM
+  no longer breaks frontmatter, numeric entities decode, and
+  `stripMarkdown` no longer deletes hyphens, underscores and minus
+  signs; merging a section with a child heading keeps its provenance;
+  ingestion conformance runs on Windows.
+- Browser verification without a WebCrypto backend fails with a clear
+  error instead of an obscure one.
+
+## pikelet 0.8.1 — 2026-09-18
+
+- `pikelet` records `mcpName: "io.github.mcn92/pikelet"` and ships
+  `server.json`, the manifest for the official MCP registry entry
+  (`pikelet mcp <pack>`; one required positional argument, any
+  `.pikelet` file). No code change.
+
+## pikelet-wasm 0.8.0 / pikelet 0.8.0 — 2026-09-14
+
+### Breaking / compatibility
+
+- **Manifest profile strings are renamed** `pancake-complete-v1/v2` →
+  `pikelet-complete-v1/v2`. New builds write the `pikelet-` string;
+  readers accept either for the same format version, so every pack
+  published before this release keeps opening with its identity
+  unchanged. Because the profile string is part of the manifest that
+  the identity hashes, **a 0.8.0 compile of the same inputs produces a
+  different identity than 0.7.0 did**. This reverses the 0.7.0 note that
+  the wire format would keep its pancake-era names.
+- **Published API names are renamed** to their Pikelet forms:
+  `PancakeError`/`PANCAKE_ERROR_CODES` → `PikeletError`/
+  `PIKELET_ERROR_CODES`, `PancakeIndex` → `PikeletIndex`,
+  `PancakeRangeArtifact`/`PancakeSketchArtifact` → `PikeletRangeArtifact`/
+  `PikeletSketchArtifact`, `PancakeApi`/`NodePancakeApi` → `PikeletApi`/
+  `NodePikeletApi`, `openPancakeFile`/`assemblePancakeFile` →
+  `openPikeletFile`/`assemblePikeletFile`. The compatibility aliases are
+  dropped. The C ABI's `pancake_*` symbols are `pikelet_*`; the native
+  addon source is `native/pikelet_napi.cpp`. The `.pancake-range` and
+  `.pancake-sketch` extension conventions are `.pikelet-range` and
+  `.pikelet-sketch` (readers dispatch on magic bytes, not extension).
+- **`export()` refuses while deleted vectors are resident.** The snapshot
+  format has no deleted flag, so a snapshot taken with ghosts present
+  resurrected them on import. C++ `serialize()` throws, the WASM
+  `pikelet_export` returns null, the native binding throws; call
+  `compact()` first. The JS wrapper already enforced this; the native
+  path did not.
+- **`@xenova/transformers` is an optional dependency of `pikelet`.** It
+  is needed only by the scaffold (`create`) path; `npm install -g pikelet
+  --omit=optional` skips its ~140 MB for `compile`/`mcp`-only use.
+- `pikelet` depends on `pikelet-wasm ^0.8.0`.
+
+### Added
+
+- **Swappable inline encoder.** `compile --encoder-model
+  --encoder-weights --encoder-vocab --encoder-pooling mean|cls
+  --encoder-query-prefix --encoder-passage-prefix` records a different
+  BERT-shaped model (same hidden size, layers, heads and vocab size as
+  MiniLM-L6, e.g. Snowflake `arctic-embed-xs`) as artifact data; CLS
+  pooling is honored end to end, and `export_encoder_blob.py` checks the
+  architecture before exporting.
+- **The inline-transformer kernel's window is 512 tokens** (was 128).
+  Chunks longer than 128 tokens are no longer silently truncated at
+  embed time; long chunks cost proportionally more to embed. Existing
+  artifacts declaring `maxTokens: 128` open and verify unchanged.
+- **`query({ showAbstained: true })`** returns the retrieval under a
+  `"none"` verdict alongside the unchanged `matchQuality`/`confidence`.
+- `examples/06-mcp-knowledge-pack`: compile a corpus, mount it over MCP,
+  answer with citations. Engine-reference examples move to
+  `examples/legacy/`.
+- `bench/beir/` retrieval-quality ladder (dense, hybrid RRF, lexical-only,
+  Arctic-XS configs) and `bench/range-proof/` (dumb static server, MCP
+  and headless-LLM proofs of the static-hosting claim).
+- A committed MCP config makes the Veyra demo packs queryable
+  interactively; the packs themselves are GitHub release assets fetched
+  by `web/veyra-corpus/fetch-veyra.mjs` rather than tracked files.
+- Encoder-conformance fixtures live under `test/fixtures/` instead of
+  inside an example.
+
+### Fixed
+
+- **The abstention verdict no longer depends on the caller's `k`.** The
+  scorer windows its inputs to the fixed top-10 the calibrator was fit
+  on; at `k=3`, `margin`, `mean10` and coverage were computed over
+  fewer candidates than at fit time and flipped verdicts between
+  `strong` and `weak` for identical retrieval.
+- **Coverage is scored against the fused (vector + BM25) ranking**
+  in both the calibrator and the reader, with the lexical cutoff shared
+  between them. Calibration was fit on vector-only signals while the
+  reader served hybrid results; on a 2,841-record Wikipedia sample every
+  measured false abstention had the correct record at hybrid rank 1–6.
+- `httpRangeSource` treats a `200` whose ETag differs from the pinned
+  one as `ARTIFACT_CHANGED` instead of a Range-unsupported host; the
+  sketch row-digest floor is 16 bytes (readers still accept 8–32);
+  `deserialize()` validates every header field before assigning it; the
+  L2 insert/search path rejects non-finite vectors (Cosine already did).
+- `chunkDocs` no longer emits title-only stub records from leading
+  undersized sections or attributes a merged chunk to the wrong section.
+- Calibration probes: phrasing diversity across question frames, a
+  saturation rail on the hard threshold, and a minimum weak band.
+- `spec/COMPLETE_PROFILE.md` Draft 2 resolves the corpus-compression,
+  `sampleQueries` placement and browser-packaging questions.
+
+## pikelet-wasm 0.7.0 / pikelet 0.7.0 — 2026-09-02
+
+### Breaking / compatibility
+
 - **The project is renamed to Pikelet.** Packages: `pancake-wasm` →
   `pikelet-wasm`, `create-pancake-search` → `pikelet` (one CLI:
   `npx pikelet create|compile|doctor|mcp`; the old `npm create
@@ -392,7 +727,8 @@ first.
 - **The v1 spec no longer lists hybrid search as a non-goal**:
   docs/CREATE_PANCAKE_SEARCH_SPEC.md's non-goals entry for "Hybrid BM25 +
   vector scoring" is struck through with a pointer to the shipped lexical
-  segment (spec §3.8) and the `query({ retrieval })` modes.
+  segment (spec §3.8) and the `query({ retrieval })` modes. (That planning
+  document was moved out of the repository on 2026-09-05.)
 - **Section-aware ingestion and chunking.** Markdown/MDX documents are
   parsed into their heading structure before chunking — heading lines are
   recognized only outside code fences, explicit `{#custom-id}` heading ids

package/NOTICE

@@ -0,0 +1,5 @@
+Pikelet
+Copyright 2026 Matthew Noonan
+
+This product includes software and file-format specifications developed as
+part of the Pikelet project (https://github.com/mcn92/pikelet).