pikelet-wasm 0.7.0 → 0.8.1

package/README.md

@@ -1,589 +1,340 @@
 # Pikelet
 
-Pikelet is a search engine you host like a file.
+**Knowledge that ships as a file.**
 
-> Renamed from **Pancake** (2026-09). Same project, same wire format —
-> existing `.pancake` artifacts and their identities remain valid, and the
-> old package names are deprecated pointers to `pikelet-wasm` and
-> `pikelet`.
+Pikelet compiles a corpus into one self-contained, queryable artifact.
 
-At the bottom there's a small WebAssembly HNSW vector engine that runs
-anywhere JavaScript does — Node, browsers, Cloudflare Workers — with no
-native dependencies. On top of it sit immutable search artifact formats:
-files that package an index, corpus records, query interpretation,
-calibration, and evaluation data, and that can be served from any storage
-that answers byte-range requests.
+A model can interrogate a 456,153-record knowledge base whose backend is a static file.
 
-Where this has been heading is the **complete search artifact**: one
-`.pikelet` file that answers natural-language queries with hydrated,
-confidence-scored results, using nothing but the file, range reads, and a
-reader.
+A `.pikelet` can carry the source text, semantic index, keyword index, query encoder, integrity commitments, retrieval calibration, and evaluation fixtures needed to interrogate that corpus. Put the file on disk, S3, R2, a CDN, or any static HTTP host. A reader can mount it locally or over HTTP Range and search it without a vector database, embedding API, or retrieval server.
 
-```text
-query text
-  -> artifact-carried query interpreter
-  -> vector search over compressed sketch/index data
-     + BM25 over the embedded lexical index, rank-fused
-  -> lazy range reads for records
-  -> calibrated hydrated results
-```
-
-The `pikelet-wasm` package carries both the vector engine and the artifact
-runtime.
-
-## Knowledge Packs
-
-The same file doubles as a **knowledge pack** for LLMs. A `.pikelet` is a
-portable, immutable corpus that contains its own query encoder, semantic
-and lexical indexes, calibrated abstention, provenance, and integrity
-metadata. Publish it as a static file; an MCP client attaches and queries
-it directly over HTTP range requests — no index to rebuild, no vector
-database to run, no embedding service to call:
+The `pikelet` CLI requires Node 20+; the `pikelet-wasm` library runs on Node 18+ (CI tests 18, 20, 22), browsers, and Cloudflare Workers.
 
 ```bash
-npx pikelet mcp install --client claude-code \
-  --pack https://github.com/mcn92/pancake/releases/download/artifact-wiki-inline-v4/pancake-wiki-inline.pancake#1b180adf4c6cebb2dcd5615256df6a25dac5fda8738dbbc11d60af86046f97f3
+npx pikelet compile --source ./docs --out docs.pikelet
 ```
 
-Then ask questions. That line pins 456,153 passages of Simple English
-Wikipedia by content hash: the mount transfers a ~52 MiB resident slice
-of the 649 MiB file, each question costs ~127 range reads, results carry
-their provenance (pack identity, title, section, source), and a pack that
-cannot answer says so instead of guessing. Details in the
-pikelet README ("Attaching packs to an LLM") and `packs/`.
-
-## What's in This Repo
-
-Three layers, bottom to top:
-
-1. **WASM vector engine**
-   - In-memory HNSW index with float32 and row-wise affine uint8 backends.
-   - JavaScript API: `Pikelet.create()`, `add()`, `search()`, `export()`,
-     `restore()`.
-   - Source in `src/`, package entrypoints at `pancake.js`,
-     `pancake.node.mjs`, `pancake.web.mjs`.
-
-2. **Search artifacts**
-   - Immutable, integrity-checked files built from indexes and corpus data,
-     range-readable from local files, R2, S3, CDNs — anything that serves
-     byte ranges.
-   - One product format, one index layer beneath it:
-     - `.pikelet`: the complete one-file search artifact — the product.
-     - `.pancake-sketch`: its index segment (resident sketch plus lazy
-       rerank rows), also usable standalone; format 2 (the builder's
-       default) interleaves per-row digests with the rows, so every row a
-       query fetches is verified on that read.
-   - Two adjacent formats that are not search artifacts in the contract
-     sense:
-     - `.pnck`: the engine's snapshot envelope (`export()`/`restore()`
-       serialization), consumed by the compilers and by layer-1 users.
-     - `.pancake-range`: **deprecated** range-readable index profile. The
-       sketch geometry replaced it (5x faster over real networks at a
-       third of the size); readers stay supported for existing artifacts,
-       but no new ones should be built.
-   - Specs: [spec/SEARCH_ARTIFACT_CONTRACT.md](spec/SEARCH_ARTIFACT_CONTRACT.md),
-     [spec/SKETCH_PROFILE.md](spec/SKETCH_PROFILE.md), and
-     [spec/COMPLETE_PROFILE.md](spec/COMPLETE_PROFILE.md).
-
-3. **Products and compilers**
-   - `pikelet/`: the fast path. One command compiles a docs
-     folder or a live site into a complete `.pikelet` file:
-     ```bash
-     npx pikelet compile --source ./docs --out search.pikelet
-     ```
-     or scaffolds a deployable search app:
-     ```bash
-     npx pikelet create --name my-docs-search --source ./docs --no-deploy --yes
-     ```
-   - `examples/05-one-file-search/`: the flagship path — the complete
-     `.pikelet` container, its compilers, and acceptance tests.
-   - `examples/04-static-wiki-pack/`: wiki-scale corpus preparation and the
-     sketch artifact pipeline.
-   - `examples/03-edge-docs-search/`: a Worker search app with a bundled
-     snapshot, distilled query encoder, and calibrated abstention.
-
-## Why Pikelet Exists
-
-Most vector search libraries assume a server process, native binaries, or a
-database service. Pikelet is built for a different deployment shape:
-
-- static files instead of always-on index servers;
-- byte-range reads instead of whole-index downloads;
-- readers for browsers, Workers, and Node;
-- explicit artifact identity and digest verification;
-- query interpretation and evaluation data carried inside the artifact;
-- compact corpus-side vector representations that keep memory and transfer
-  costs sane.
-
-The result is search you can host like an asset instead of operating like a
-database.
-
-## The Flagship: One-File Search
-
-The complete profile (`.pikelet`) packs the five components of a search
-application into one content-addressed file:
-
 ```text
-index       embedded .pancake-sketch artifact
-corpus      JSON records behind an offsets table
-encoder     query interpreter declaration or inline model data
-calibration abstention / match-quality model
-evaluation  golden queries and expected behavior
-lexical     BM25 inverted index for hybrid retrieval (optional; readers
-            that predate it serve vector-only from the same file)
+Ingested 3 docs -> 3 chunks
+Embedded 3/3 chunks with inline transformer
+Built complete .pikelet artifact with 24.5 MB
+Compiled docs.pikelet
+  24.5 MB, 3 records, identity 8d731a...
 ```
 
-The reader verifies the manifest and eager segments, opens the sketch index
-against a range source, and hydrates only the records the final results
-need.
-
-Query interpretation comes in three kinds. Each picks two of
-**self-contained**, **small**, and **teacher-quality**, and gives up the
-third — that trade-off, not history, is why all three exist:
-
-- **kind 3: `inline-transformer-v1`** — self-contained + teacher quality.
-  The artifact carries the WordPiece vocab and quantized MiniLM teacher
-  weights as verified data; the reader ships the execution kernels. Costs
-  ~25 MB of encoder no matter how small the corpus. This is the default:
-  `pikelet compile` builds kind 3.
-- **kind 1: `student-inline-v1`** — self-contained + small. A
-  corpus-distilled student encoder (~1 MiB), pure-JS execution. The only
-  way a small docs corpus compiles to a file measured in single-digit MB;
-  quality is bounded by the distillation.
-- **kind 2: `external-transformers-v1`** — small + teacher quality, but
-  not self-contained: the artifact pins a host-supplied encoder (model id,
-  pooling, normalization, max tokens) and verifies it against embedded
-  test vectors before serving. The escape hatch for encoders too big to
-  inline and hosts that already run one.
-
-Kind 3 is the flagship: the file carries the corpus, index, tokenizer,
-encoder weights, calibration, and evaluation data — the reader supplies
-code, the artifact supplies everything else. Ask it about volcanoes and no
-service, model host, or network dependency is involved beyond range reads
-of the file itself.
+First run fetches the ~25 MiB query encoder from a GitHub release and caches it; every `.pikelet` file is at least that size regardless of corpus, because the encoder ships inside it — a 5-file folder and a 500-file folder both start around 25 MiB.
+
+Then query it from an LLM — Claude Code, Claude Desktop, or any MCP client:
 
 ```bash
-cd examples/05-one-file-search
-node compile-wiki.mjs --inline-encoder
-node compile.mjs --inspect pancake-wiki-inline.pancake
-node test-inline.mjs
+npx pikelet mcp install --client claude-code --pack ./docs.pikelet
+# or --client claude-desktop; any other MCP client can run
+# `npx pikelet mcp --pack ./docs.pikelet` directly, no install step
 ```
 
-Where the inline wiki artifact currently stands:
-
 ```text
-456,153 records
-~649 MiB complete artifact (680,029,254 bytes)
-manifest identity: 1b180adf4c6cebb2dcd5615256df6a25dac5fda8738dbbc11d60af86046f97f3
-recall@10: 95.2% at the recommended rerank (97.0% at C=600), against exact
-  fp32 brute-force ground truth on the 200-query pre-registered eval —
-  98.5% on hand-written natural-language questions, 94.8% on single-title
-  lookups (augmented retrieval: hybrid candidates, distance order)
-serves over HTTP range reads: opens on ~52 MiB of the file, ~127 range
-  requests per query (cluster-ordered row layout + manifest fetch hints)
-natural-language query with no host encoder option, ~110 ms/query end to
-  end locally (the resident scan auto-stages the engine's SIMD kernel at
-  this scale; ~540 ms on the pure-JS scan it falls back to)
+Wrote MCP server "knowledge-packs" to ./.mcp.json
+Claude Code picks it up on the next session in this project.
 ```
 
-Release asset (v4 — recovers the pack's canonical build: k-means
-cluster-ordered rows and the 192-dim 2:1-pooled sketch, where v1–v3
-embedded a 96-dim 4:1 sketch in unpermuted order and measured 82.4%; adds
-the hybrid BM25 lexical segment and a format-2 sketch with per-row read
-verification; earlier versions stay published for older checkouts):
-`https://github.com/mcn92/pancake/releases/download/artifact-wiki-inline-v4/pancake-wiki-inline.pancake`
+Claude now has a `search` tool over your docs. Or query it directly from code:
 
-`node test-inline.mjs` downloads that file automatically when it is missing
-locally, verifies the manifest identity, and then runs the acceptance
-checks.
-
-See [examples/05-one-file-search/README.md](examples/05-one-file-search/README.md)
-for the full one-file walkthrough.
+```js
+import { openPikeletFile } from 'pikelet-wasm/complete';
+const pack = await openPikeletFile('docs.pikelet');
+const out = await pack.query('how do workers restore snapshots', { k: 5 });
+console.log(out.matchQuality, out.results[0]?.title);
+// 'strong' 'Snapshot restore'
+```
 
-## The Design Choice Everything Hangs On
+That's the whole loop. `compile` also takes a live URL (`--source https://docs.example.com`) instead of a directory. If you want a deployed search app — a Worker + UI, not a file — use `npx pikelet create` instead; see [`packages/pikelet/README.md`](packages/pikelet/README.md) for the full CLI reference and the tradeoffs between the two. `compile` never needs the scaffold path's `@xenova/transformers` dependency; skip its ~140 MB install with `npm install -g pikelet --omit=optional` if you only need `compile`/`mcp`.
 
-The engine's quantized backend stores corpus vectors as row-wise affine
-uint8:
+Under the hood, the file is one container for everything a reader needs:
 
 ```text
-value ~= offset[row] + scale[row] * byte
+documents
+    │
+    ▼
+┌──────────────────┐
+│   docs.pikelet   │
+│                  │
+│ corpus           │
+│ semantic index   │
+│ lexical index    │
+│ query encoder    │
+│ integrity        │
+│ calibration      │
+│ evaluation       │
+└──────────────────┘
+    │
+    ├── local file
+    └── static HTTP / object storage
+              │
+              ▼
+        browser / Node / edge / agent
 ```
 
-That one constraint propagates through the whole system:
+Search is the interface. **The file is the knowledge deployment unit.**
 
-- HNSW search keeps the query as float32 and scores it against compressed
-  corpus rows.
-- Distance kernels fuse dequantization with dot/L2 accumulation.
-- SIMD paths load bytes, widen lanes, apply scale/offset, and accumulate.
-- Graph maintenance uses symmetric node-to-node distances derived from the
-  compressed rows and row statistics.
-- Snapshots serialize compact scale/offset/qdata arrays.
-- Sketch artifacts scan compressed resident rows and fetch only rerank
-  candidates.
-- Range-backed readers keep the large corpus side compressed while queries
-  stay precise and transient.
+---
 
-This is why the engine work and the artifact work belong in one project:
-the same corpus-side representation serves in-memory search, serialized
-snapshots, resident sketches, range reads, and reranking.
+## The shortest demonstration
 
-## Install
+A real Pikelet pack contains **456,153 Simple English Wikipedia records** in a **648.5 MiB** artifact.
 
-From npm:
+Try the live browser demo:
 
-```bash
-npm install pancake-wasm
+```text
+https://pikelet-wiki-playground.pages.dev/
 ```
 
-From a checkout:
+The page is static HTML/JS on Cloudflare Pages. It mounts the pack from R2:
 
-```bash
-git clone https://github.com/mcn92/pancake.git
-cd pancake
-npm install
+```text
+https://pub-6da2384a3bca4a44b2b2fa29a94cc811.r2.dev/wikipedia.pikelet
 ```
 
-The checkout includes prebuilt WASM in `dist/`, so you don't need to
-rebuild the engine unless you edit `src/`.
+There is no `/search`, `/embed`, `/query`, or `/api` backend behind the demo.
+The browser opens the `.pikelet` file over HTTP Range, verifies the artifact,
+loads the embedded query encoder, and shows the byte ranges it reads.
 
-## Runtime Entry Points
+That file was served by a deliberately dumb HTTP server whose only relevant capability was:
 
-Published package:
-
-```js
-// Node.js CJS
-const Pikelet = require('pikelet-wasm');
-
-// Node.js ESM
-import Pikelet from 'pikelet-wasm';
-
-// Explicit Node/browser entries
-import Pikelet from 'pikelet-wasm/node';
-import Pikelet from 'pikelet-wasm/web';
-
-// Search Artifact layer (sketch readers/builders; the deprecated
-// .pancake-range reader stays exported for existing artifacts)
-import { PancakeSketchArtifact, buildSketchArtifact } from 'pikelet-wasm/artifact';
-
-// Complete one-file profile: reader (any runtime) and builder (Node)
-import { openPikeletFile } from 'pikelet-wasm/complete';
-import { assemblePancakeFile } from 'pikelet-wasm/complete/builder';
-```
-
-Repository checkout:
-
-```js
-const Pikelet = require('./pancake.js');
-import Pikelet from './pancake.node.mjs';
-import Pikelet from './pancake.web.mjs';
+```text
+fs.createReadStream + HTTP Range
 ```
 
-The browser entry expects the runtime or bundler to resolve
-`dist/engine.wasm`.
+There was no vector database, no search service, no embedding service, no application server that understood the pack, no index process on the server.
 
-## Engine Quick Start
+The pack was mounted by URL and exposed to a real headless Claude process through MCP. Each invocation was restricted with `--strict-mcp-config` to only:
 
-Use the core engine when you already have vectors and want an in-memory ANN
-index.
-
-```js
-import Pikelet from 'pikelet-wasm';
-
-const index = await Pikelet.create({
-  dim: 384,
-  maxElements: 100000,
-  metric: 'cosine',
-  quantized: true,
-});
+```text
+mcp__wiki-pack__search
+mcp__wiki-pack__list_packs
+mcp__wiki-pack__get_record
+```
 
-const docVector = new Float32Array(384).fill(0.1);   // your embedding here
-const queryVector = new Float32Array(384).fill(0.1); // your query embedding here
+No web tool. No filesystem tool. No alternate retrieval tool.
 
-const id = index.add(docVector);
+One of the actual responses:
 
-const results = index.search(queryVector, 10);
-// [{ id: 0, distance: 0.12 }, ...]
+> **Q: What causes earthquakes?**
+>
+> Earthquakes are caused by the movement of tectonic plates in the Earth's crust. Stress builds up along fault boundaries where plates grind against or ride over one another; when that stress exceeds the friction holding the locked fault in place, it suddenly releases as shock waves (seismic waves) that shake the ground [record 429637].
+>
+> There are three main fault types that can produce quakes — normal, reverse (thrust), and strike-slip — and some quakes have less clear-cut causes, like deep shifts in ancient rock or mine-collapse "tremors" [record 429638].
+>
+> Two related mechanisms compound this:
+>
+> * Subduction, where one plate is forced under another into the mantle, produces the most powerful earthquakes and can trigger tsunamis [record 132833].
+> * More broadly, the slow convection-driven movement of crustal plates over the hot mantle is the underlying driver of quakes, volcanoes, and mountain formation [record 132267].
+>
+> Volcanic eruptions can also cause earthquakes, though these are localized to areas near active volcanoes [record 429637].
 
-index.delete(id);
-index.compact();
+The model did not just repeat the first search hit. It retrieved multiple records and synthesized them into a sourced answer.
 
-const snapshot = index.export();
-const restored = await Pikelet.restore(snapshot, { maxElements: 100000 });
+Other runs produced:
 
-index.dispose();
-restored.dispose();
+```text
+Who was the first person on the moon?          → Neil Armstrong          (records 408686, 408684)
+How do volcanoes form?                          → plates + subduction + hotspots
+                                                                          (records 132770, 132765, 132852, 132853)
+How does photosynthesis work?                   → two-phase reaction, cited (5 records: 114219, 114220, 114222, 114223, 269279)
+What is the capital of France?                  → Paris                  (records 454641, 302684)
 ```
 
-If you have records with vectors:
+That is the product in one demonstration:
 
-```js
-const rows = [
-  { id: 'doc-1', vector: embedding1 },
-  { id: 'doc-2', vector: embedding2 },
-];
-
-const { index, idMap } = await Pikelet.fromVectors(rows, {
-  metric: 'cosine',
-  quantized: true,
-});
-
-const hits = index.search(queryEmbedding, 5);
-console.log(idMap.get(hits[0].id));
-```
+> **A model is interrogating a 456,153-record knowledge base whose backend is a static file.**
 
-On Node entrypoints, file helpers are available:
+The model may already contain some of these facts in its pretrained parameters. This test demonstrates the retrieval, synthesis, citation, and deployment path; the [Veyra ablation](docs/veyra-ablation.md) tests whether support changes when evidence is removed from the pack.
 
-```js
-const { index, idMap } = await Pikelet.loadJsonFile('vectors.jsonl', {
-  metric: 'cosine',
-  vectorKey: 'embedding',
-  idKey: 'docId',
-  maxFileBytes: 64 * 1024 * 1024,
-});
-
-const restored = await Pikelet.loadSnapshotFile('index.pnck', {
-  dim: 384,
-  maxElements: 100000,
-  metric: 'cosine',
-  quantized: true,
-});
-```
+**Network cost.** In a single persistent session (one mount, five queries, one repeat):
 
-## Complete Artifact Quick Start
+| Operation                       | Bytes transferred | Requests |
+| ------------------------------- | ----------------: | -------: |
+| Initial mount                   |          51.7 MiB |       11 |
+| Query 1 + one-time encoder load |          25.9 MiB |       77 |
+| Query 2                         |         504.4 KiB |       37 |
+| Query 3                         |         499.4 KiB |       46 |
+| Query 4                         |         968.9 KiB |       75 |
+| Query 5                         |           1.1 MiB |      116 |
+| Repeated query 1                |         238.4 KiB |        4 |
+| **Total**                       |      **80.8 MiB** |  **366** |
 
-Use the complete reader when you want a file that is itself the search
-application data plane. To build one from your own docs, see the compile
-command above (`npx pikelet compile`).
+Roughly **12.5% of the 648.5 MiB artifact** crossed the wire across that whole session. Excluding the one-time ~25 MiB encoder load, fresh-query traffic ran **~0.5–1.1 MiB per query**. The complete artifact was never downloaded.
 
-```js
-import { openPikeletFile } from 'pikelet-wasm/complete'; // or './complete/index.mjs' in a checkout
+The headless-Claude test above used a fresh process per question, so each invocation repaid the ~52 MiB mount and ~25 MiB encoder cost — about 78 MiB per cold query. That's a real operational distinction: **persistent sessions amortize mount and encoder cost; independent cold processes do not.**
+
+*(The large benchmark fixture still carries its historical `.pancake` filename from before the Pikelet rename; current artifacts use `.pikelet`.)*
 
-const search = await openPikeletFile('examples/05-one-file-search/pancake-docs.pancake');
+---
 
-const out = await search.query('how do workers restore snapshots', { k: 5 });
+## Attach a pack to an LLM
 
-console.log(out.matchQuality, out.confidence);
-console.log(out.results[0].title);
+`pikelet mcp` exposes one or more packs through the Model Context Protocol:
 
-await search.close();
+```bash
+npx pikelet mcp install \
+  --client claude-code \
+  --pack https://example.com/docs.pikelet#<sha256>
 ```
 
-For kind-3 artifacts, no host encoder is needed:
+The agent gets `search`, `get_record`, `list_packs`, `verify_pack`. A search result carries the identity of the pack and the location of the source record, so an agent can work against `product-docs.pikelet`, `rust-reference.pikelet`, `policy-2026-09.pikelet`, `customer-manual-v4.pikelet` without each publisher operating a retrieval API. The artifact can be local, private, public, behind authenticated object storage, or distributed like any other static asset.
 
-```js
-const search = await openPikeletFile('pancake-wiki-inline.pancake');
-const out = await search.query('how do volcanoes form', { k: 5 });
-```
+A pack mounted with a content hash has a stable identity — `https://example.com/docs.pikelet#8d731...` — so "what body of knowledge did this agent query?" has a reproducible answer.
 
-For kind-2 artifacts, pass `options.encodeQuery`; the reader runs it
-against the declaration's verification vectors before serving and refuses
-the open if they disagree (`info().encoderVerified` reports the outcome).
+**A mounted pack's content reaches the model as tool output.** `verify_pack` proves the bytes are intact and match their pinned identity; it does not prove the corpus itself is trustworthy. Mounting a pack from a source you don't control is the same trust decision as giving an agent any other untrusted-content tool — treat pack text the way you'd treat search results or fetched web pages, not as instructions.
 
-The reader also accepts a range source, which is how a `.pikelet` on R2 or
-a CDN gets queried without downloading it:
+**Try it with your own questions.** The three Veyra packs from the [ablation](docs/veyra-ablation.md) are published as pinned GitHub release assets. Fetch and verify them first:
 
-```js
-const search = await openPikeletFile({
-  size,
-  async read(offset, length) {
-    // return bytes for [offset, offset + length)
-  },
-});
+```bash
+npm run demo:veyra
 ```
 
-On integrity: every artifact-derived read is bounds-checked and budgeted
-(`maxReadBytes`, `maxRecordBytes`). Format-2 files (`pancake-complete-v2`,
-the builder's default) verify each hydrated record against a per-record
-digest on its own range read (`info().corpusIntegrity`), and an embedded
-format-2 sketch verifies each lazily fetched index row the same way
-(`info().indexRowIntegrity`) — so a tampered transport fails the query
-instead of skewing its results. Older files with a format-1 sketch commit
-the lazy index rows to the identity but don't verify them per read; for
-those, pass `verifyIndexVectors: true` (or call `verifyVectors()`) to
-authenticate them in one full pass before trusting results from an
-untrusted transport (spec/COMPLETE_PROFILE.md section 6).
-
-### Attaching packs to an LLM
-
-A complete artifact doubles as a portable knowledge pack: everything an
-LLM needs to query a body of knowledge — corpus, indexes, encoder,
-calibrated abstention, integrity commitments — in one immutable,
-identity-pinned file. `pikelet mcp` serves packs over the
-Model Context Protocol so Claude Code, Claude Desktop, or any agent
-framework can attach them as a retrieval tool with one config line — no
-vector database, no embedding service, no retrieval backend. Results
-carry full provenance (pack, manifest identity, title, heading path,
-source), and a pack that cannot answer says so instead of returning
-plausible noise. See the pikelet README's
-"Attaching packs to an LLM (MCP)" section.
-
-## Demos
-
-Start with [DEMOS.md](DEMOS.md) for the maintained demo path.
-
-Useful commands:
+Then mount all three through Pikelet's MCP server:
 
 ```bash
-npm run demo                  # synthetic engine validation
-npm run demo:artifact         # range artifact search
-npm run demo:sketch           # sketch artifact search/rerank path
-npm run test:browser          # browser smoke test
-node examples/05-one-file-search/test-file.mjs
-node examples/05-one-file-search/test-wiki.mjs
-node examples/05-one-file-search/test-inline.mjs
+claude -p "your question here" \
+  --mcp-config examples/one-file-search/web/public/veyra.mcp.json \
+  --strict-mcp-config \
+  --allowedTools "mcp__veyra-demo__search,mcp__veyra-demo__list_packs,mcp__veyra-demo__get_record"
 ```
 
-The one-file example is the closest thing to the project's current center
-of gravity:
+For an interactive session, use the same config without `-p`:
 
 ```bash
-cd examples/05-one-file-search
-node compile.mjs
-node test-file.mjs
-node serve.mjs
+claude \
+  --mcp-config examples/one-file-search/web/public/veyra.mcp.json
 ```
 
-## API Snapshot
+All three packs derive from the same small synthetic Station Veyra corpus, with controlled differences in the Tovash evidence. Ask where the Tovash project is housed: the full pack supports Chamber 17, the modified pack supports Chamber 43, and the ablated pack should abstain.
 
-Core index:
+---
 
-```js
-const index = await Pikelet.create(options);
-index.add(vector);
-index.addBatch(vectors);
-index.search(query, k, options?);
-index.searchFiltered(query, k, allowedIds, options?); // allowedIds: Set of ids
-index.delete(id);
-index.compact();
-index.export();
-index.dispose();
-```
+## Match quality and abstention
 
-Common options:
+A nearest neighbour is not automatically evidence that a corpus answers a question. Pikelet can calibrate retrieval signals at build time (best semantic distance, distance margin, lexical coverage, retrieval agreement). When the corpus supports a reliable classifier, results carry `matchQuality: strong | weak | none`. When calibration can't separate supported from unsupported reliably — a single novel may be semantically homogeneous enough that the fit isn't trustworthy — Pikelet reports `matchQuality: unscored` and records why calibration was skipped, rather than manufacturing confidence. A `none` verdict withholds `results` by default; pass `query(text, { showAbstained: true })` to see the raw retrieval anyway — `matchQuality` and `confidence` are unaffected either way.
 
-```js
-{
-  dim: 384,
-  maxElements: 100000,
-  metric: 'cosine',       // 'cosine' or 'l2'
-  quantized: true,
-  M: 12,
-  efConstruction: 75,
-  efSearch: 100,
-  seed: 108
-}
-```
+`matchQuality` is evidence about retrieval support. It is **not** a guarantee that an LLM will never hallucinate.
 
-Snapshot helpers:
+---
 
-```js
-await Pikelet.restore(bytes, options);
-await Pikelet.loadSnapshotFile(path, options);
-await Pikelet.loadJsonFile(path, options);
-await Pikelet.fromVectors(rows, options);
-```
+## Deeper dives
 
-Range and sketch artifact APIs ship at `pikelet-wasm/artifact`; the
-complete-profile reader and builder ship at `pikelet-wasm/complete` and
-`pikelet-wasm/complete/builder`. The wiki-scale compilers and acceptance
-tests live in `examples/05-one-file-search/`.
+- [How a `.pikelet` query runs](docs/how-a-query-runs.md) — what is inside the file, what a query costs on the wire, remote execution, and the integrity checks on the read path.
+- [The Veyra ablation](docs/veyra-ablation.md) — what happens to answers when the supporting record is edited or removed from the pack, with the reproduction commands.
+- [Why make knowledge a file](docs/why-a-file.md) — the reasoning behind the single-artifact design, and what it commits to.
+- [Architecture](docs/architecture.md) — the engine, the C ABI and the JavaScript wrapper; the artifact formats are specified in [`spec/`](spec/).
 
-## Building From Source
+## What this is not
 
-You only need this if you change the C++ engine under `src/`.
+**Not a claim that vector databases are obsolete.** If your corpus changes continuously, needs transactional updates, serves many tenants, or already lives comfortably in a database, use a database. Pikelet targets `build → publish → query many times → replace on release`.
 
-Requirements:
+**Not state-of-the-art embedding research.** The bundled model is MiniLM-L6, chosen for being small enough to live inside the artifact. ArguAna demonstrates a real quality cost from the compact encoder.
 
-- Node.js
-- Emscripten SDK
+**Not "LLMs can no longer hallucinate."** The artifact exposes retrieved evidence, provenance, integrity, and an explicit support signal. Whether an agent obeys those signals is an agent behavior question.
 
-Build:
+**Not a new HNSW algorithm.** HNSW is HNSW. HTTP Range is HTTP Range. BM25 is BM25. MiniLM is MiniLM. Affine quantization is not new either. The project is about what becomes possible when those pieces are arranged around one constraint: **the knowledge base itself must be distributable as a file and remain useful without dedicated retrieval infrastructure.**
 
-```bash
-npm run build
-```
+---
 
-Build scalar and SIMD variants:
+## When Pikelet is a good fit — and when it isn't
 
-```bash
-npm run build:all
-```
+**Use it when:**
 
-The build emits `dist/engine.js`, `dist/engine.wasm`, and scalar variants.
+- your corpus is static or changes on a release cycle
+- you want semantic search without operating search infrastructure
+- you want search inside a browser, Node process, edge worker, Electron app, or offline tool
+- you want to distribute a searchable corpus to someone else
+- you want an LLM agent to query a frozen, versioned body of knowledge
+- static/object storage is easier to deploy than a database
+- provenance and reproducibility matter
+- a content-addressed knowledge snapshot is useful
 
-## Tests
+Examples: documentation, SDK/API references, technical manuals, research corpora, standards, legal texts, books, product knowledge, code/documentation snapshots, evaluation corpora, agent reference packs.
 
-Core (`npm test` also runs the artifact and complete-profile conformance
-suites):
+**Don't use it if:**
 
-```bash
-npm test
-npm run test:fuzz
-npm run test:simd
-npm run test:browser
-npm run test:scaffold
-```
+- your corpus has heavy online writes
+- you need transactional mutation
+- you need a multi-tenant authoritative search service
+- you need exact nearest-neighbour search at very large scale
+- your corpus is so large that a linear resident sketch scan is inappropriate
+- you already operate a vector database happily and portability buys you nothing
 
-Complete profile, end to end:
+The resident remote scan is linear in row count. The current architecture targets corpora through the low millions of records, not arbitrary web scale.
 
-```bash
-node examples/05-one-file-search/test-file.mjs
-node examples/05-one-file-search/test-wiki.mjs
-node examples/05-one-file-search/test-inline.mjs
-```
+---
 
-The inline wiki acceptance test downloads the large release artifact on
-demand when `examples/05-one-file-search/pancake-wiki-inline.pancake` is
-missing.
+## Repository map
 
-## Performance Notes
+```text
+packages/pikelet-wasm/            The published pikelet-wasm package:
+  engine/                          C++ vector engine (HNSW, float32 and
+                                   affine-u8 backends, mutation, compaction,
+                                   snapshot import/export), compiled to WASM
+                                   in dist/. Used at compile time to build and
+                                   quantize the index; query time does not load
+                                   it (see complete/index.mjs's header comment)
+                                   — .pikelet reads run a pure-JS sketch scan,
+                                   optionally SIMD-accelerated.
+  src/                             Engine wrapper and artifact readers/builders
+                                   (the entrypoints, core/, artifact/, errors/)
+  complete/                        Reader and builder for the complete
+                                   range-readable .pikelet artifact
+  native/                          N-API build of the same engine (benchmarks)
+packages/pikelet/                 The published pikelet CLI: compiler, MCP
+                                   server, encoder integration, Docusaurus plugin
+docs/                             Deeper dives (how a query runs, the Veyra
+                                   ablation, why a file), architecture notes,
+                                   rename history, measurement reports
+spec/                             Byte-level artifact contracts
+benchmarks/beir/                       Frozen BEIR ablation harness (encoder,
+                                   quantization, HNSW quality)
+benchmarks/range-proof/                The deliberately boring static-HTTP proof:
+                                   dumb-server.mjs, proof.mjs, mcp-proof.mjs,
+                                   llm-proof.mjs, failure-modes.mjs
+examples/one-file-search/      Large single-artifact search and embedded
+                                   encoder work
+examples/mcp-knowledge-pack/   Compile, mount, search, and hydrate
+                                   records through MCP
+local-packs/                      Prebuilt example .pikelet artifacts used
+                                   by the demo commands above
+packs/                            Pack hosting and distribution examples
+```
 
-Performance is workload-dependent, but these are the knobs that matter:
+---
 
-- `quantized: true` for much smaller corpus-side vector storage.
-- `efSearch` for the recall/latency tradeoff.
-- sketch artifacts for depth-1 candidate selection before lazy rerank
-  reads.
-- corpus layout and clustering for fewer remote read rounds.
-- kind-3 inline encoders when deployment shouldn't depend on an external
-  ML runtime.
+## Reproduce the proofs
 
-The older engine-level benchmarks are still useful for HNSW behavior. The
-newer artifact-level benchmarks are more representative of what the project
-is actually for: search from static files and range-readable storage.
+**Range proof** (`benchmarks/range-proof/`) is built around a server that does not understand Pikelet. It checks: static HTTP + Range + Pikelet client = remote semantic retrieval, without a full download. The main script reports artifact size, records, mount bytes/requests, per-query bytes/requests, cache behavior. `failure-modes.mjs` independently exercises Range supported / ignored / tampered bytes / truncated response. `mcp-proof.mjs` and `llm-proof.mjs` use a real MCP client and a real headless Claude process rather than a mocked model response.
 
-The native baselines the engine benchmarks compare against (faiss-node,
-hnswlib-node, usearch) are not dependencies of `pikelet-wasm`; install
-them on demand with `cd benchmarks && npm install` before running those
-scripts. See `benchmarks/`, `benchmark_results/`, and the example READMEs
-for current numbers.
+**BEIR evaluation** (`benchmarks/beir/`) keeps stages separate so a quality change can be attributed to the component that caused it, plus a standalone conformance fixture checking the JS benchmark path against the actual C++ affine-u8 representation. Don't read the latency columns as a universal native-performance comparison — they measure the benchmark paths under the stated harness. The quality deltas are the important part.
 
-## Tradeoffs
+---
 
-Pikelet is a good fit when:
+## Status
 
-- you want search in JavaScript runtimes without native addons;
-- you want static-file or edge deployment;
-- you need reproducible, inspectable search artifacts;
-- your corpus can be built offline;
-- byte-range reads are cheaper than operating a dedicated search service.
+Pikelet is early. The implementation is real; the format is not frozen. One primary author. Draft 2 artifact format.
 
-It's the wrong tool when:
+- prebuilt WASM included
+- local and HTTP readers
+- affine-u8 and float HNSW backends for compile-time indexing/quantization
+- embedded query encoder
+- BM25 hybrid retrieval
+- content identity and lazy-read integrity verification
+- calibration/abstention support
+- MCP mounting
+- BEIR retrieval evaluation
+- range-read and failure-mode tests
+- a large 456k-record example artifact
 
-- you need high-rate online mutation;
-- you need a multi-tenant database server;
-- you require exact nearest-neighbor search at large scale;
-- your deployment can already use a mature native ANN service with no
-  portability constraints.
+`npm test` exercises the engine, artifact profiles, MCP, ingestion, format hardening, and related conformance suites.
 
-## Status
+Previously known as **Pancake** (renamed September 2026); artifacts, profile strings and files from before the rename remain readable — see [`docs/history.md`](docs/history.md) for what kept the old name and why. Pikelet is unrelated to the pre-existing Pikelet programming language.
 
-The engine and the artifact readers/builders are published and usable
-today (`pikelet-wasm`, with `pikelet` as the app scaffold).
-The complete `.pikelet` profile's spec is still marked draft — Draft 2,
-under review, not frozen — but its reader and builder ship in the package,
-and it is the clearest expression of where the project is going: a search
-engine packaged as one verified, range-readable file.
+---
 
 ## License