pikelet-wasm 0.7.0 → 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (32) hide show
  1. package/CHANGELOG.md +19 -3
  2. package/NOTICE +5 -0
  3. package/README.md +397 -454
  4. package/complete/builder.d.ts +4 -4
  5. package/complete/builder.mjs +8 -8
  6. package/complete/encoder-kernels/encoder.node.wasm +0 -0
  7. package/complete/encoder-kernels/encoder.wasm +0 -0
  8. package/complete/index.d.ts +11 -7
  9. package/complete/index.mjs +161 -112
  10. package/complete/inline-transformer.mjs +22 -9
  11. package/complete/lexical.mjs +11 -11
  12. package/complete/retrieval-abstention.mjs +18 -6
  13. package/complete/sources.mjs +22 -1
  14. package/dist/engine.js +1 -1
  15. package/dist/engine.scalar.js +1 -1
  16. package/dist/engine.scalar.wasm +0 -0
  17. package/dist/engine.wasm +0 -0
  18. package/package.json +58 -51
  19. package/{pancake-artifact-common.js → pikelet-artifact-common.js} +42 -42
  20. package/{pancake-artifact-range.js → pikelet-artifact-range.js} +36 -36
  21. package/{pancake-artifact-sketch.js → pikelet-artifact-sketch.js} +76 -66
  22. package/{pancake-artifact.d.ts → pikelet-artifact.d.ts} +11 -11
  23. package/{pancake-artifact.js → pikelet-artifact.js} +10 -10
  24. package/{pancake-core.js → pikelet-core.js} +136 -136
  25. package/{pancake-errors.js → pikelet-errors.js} +9 -8
  26. package/{pancake.d.ts → pikelet.d.ts} +55 -55
  27. package/{pancake.js → pikelet.js} +44 -44
  28. package/{pancake.node.mjs → pikelet.node.mjs} +46 -46
  29. package/{pancake.web.d.ts → pikelet.web.d.ts} +13 -13
  30. package/{pancake.web.mjs → pikelet.web.mjs} +20 -20
  31. package/{pancake.workerd.mjs → pikelet.workerd.mjs} +18 -18
  32. /package/{pancake-loader.js → pikelet-loader.js} +0 -0
package/README.md CHANGED
@@ -1,589 +1,532 @@
1
1
  # Pikelet
2
2
 
3
- Pikelet is a search engine you host like a file.
3
+ **Knowledge that ships as a file.**
4
4
 
5
- > Renamed from **Pancake** (2026-09). Same project, same wire format —
6
- > existing `.pancake` artifacts and their identities remain valid, and the
7
- > old package names are deprecated pointers to `pikelet-wasm` and
8
- > `pikelet`.
5
+ Pikelet compiles a corpus into one self-contained, queryable artifact.
9
6
 
10
- At the bottom there's a small WebAssembly HNSW vector engine that runs
11
- anywhere JavaScript does — Node, browsers, Cloudflare Workers — with no
12
- native dependencies. On top of it sit immutable search artifact formats:
13
- files that package an index, corpus records, query interpretation,
14
- calibration, and evaluation data, and that can be served from any storage
15
- that answers byte-range requests.
7
+ A model can interrogate a 456,153-record knowledge base whose backend is a static file.
16
8
 
17
- Where this has been heading is the **complete search artifact**: one
18
- `.pikelet` file that answers natural-language queries with hydrated,
19
- confidence-scored results, using nothing but the file, range reads, and a
20
- reader.
9
+ A `.pikelet` can carry the source text, semantic index, keyword index, query encoder, integrity commitments, retrieval calibration, and evaluation fixtures needed to interrogate that corpus. Put the file on disk, S3, R2, a CDN, or any static HTTP host. A reader can mount it locally or over HTTP Range and search it without a vector database, embedding API, or retrieval server.
21
10
 
22
- ```text
23
- query text
24
- -> artifact-carried query interpreter
25
- -> vector search over compressed sketch/index data
26
- + BM25 over the embedded lexical index, rank-fused
27
- -> lazy range reads for records
28
- -> calibrated hydrated results
11
+ Requires Node 20+.
12
+
13
+ ```bash
14
+ npx pikelet compile --source ./docs --out docs.pikelet
29
15
  ```
30
16
 
31
- The `pikelet-wasm` package carries both the vector engine and the artifact
32
- runtime.
17
+ ```text
18
+ Ingested 3 docs -> 3 chunks
19
+ Embedded 3/3 chunks with inline transformer
20
+ Built complete .pikelet artifact with 24.5 MB
21
+ Compiled docs.pikelet
22
+ 24.5 MB, 3 records, identity 8d731a...
23
+ ```
33
24
 
34
- ## Knowledge Packs
25
+ First run fetches the ~25 MiB query encoder from a GitHub release and caches it; every `.pikelet` file is at least that size regardless of corpus, because the encoder ships inside it — a 5-file folder and a 500-file folder both start around 25 MiB.
35
26
 
36
- The same file doubles as a **knowledge pack** for LLMs. A `.pikelet` is a
37
- portable, immutable corpus that contains its own query encoder, semantic
38
- and lexical indexes, calibrated abstention, provenance, and integrity
39
- metadata. Publish it as a static file; an MCP client attaches and queries
40
- it directly over HTTP range requests — no index to rebuild, no vector
41
- database to run, no embedding service to call:
27
+ Then query it from an LLM — Claude Code, Claude Desktop, or any MCP client:
42
28
 
43
29
  ```bash
44
- npx pikelet mcp install --client claude-code \
45
- --pack https://github.com/mcn92/pancake/releases/download/artifact-wiki-inline-v4/pancake-wiki-inline.pancake#1b180adf4c6cebb2dcd5615256df6a25dac5fda8738dbbc11d60af86046f97f3
30
+ npx pikelet mcp install --client claude-code --pack ./docs.pikelet
31
+ # or --client claude-desktop; any other MCP client can run
32
+ # `npx pikelet mcp --pack ./docs.pikelet` directly, no install step
33
+ ```
34
+
35
+ ```text
36
+ Wrote MCP server "knowledge-packs" to ./.mcp.json
37
+ Claude Code picks it up on the next session in this project.
46
38
  ```
47
39
 
48
- Then ask questions. That line pins 456,153 passages of Simple English
49
- Wikipedia by content hash: the mount transfers a ~52 MiB resident slice
50
- of the 649 MiB file, each question costs ~127 range reads, results carry
51
- their provenance (pack identity, title, section, source), and a pack that
52
- cannot answer says so instead of guessing. Details in the
53
- pikelet README ("Attaching packs to an LLM") and `packs/`.
54
-
55
- ## What's in This Repo
56
-
57
- Three layers, bottom to top:
58
-
59
- 1. **WASM vector engine**
60
- - In-memory HNSW index with float32 and row-wise affine uint8 backends.
61
- - JavaScript API: `Pikelet.create()`, `add()`, `search()`, `export()`,
62
- `restore()`.
63
- - Source in `src/`, package entrypoints at `pancake.js`,
64
- `pancake.node.mjs`, `pancake.web.mjs`.
65
-
66
- 2. **Search artifacts**
67
- - Immutable, integrity-checked files built from indexes and corpus data,
68
- range-readable from local files, R2, S3, CDNs — anything that serves
69
- byte ranges.
70
- - One product format, one index layer beneath it:
71
- - `.pikelet`: the complete one-file search artifact — the product.
72
- - `.pancake-sketch`: its index segment (resident sketch plus lazy
73
- rerank rows), also usable standalone; format 2 (the builder's
74
- default) interleaves per-row digests with the rows, so every row a
75
- query fetches is verified on that read.
76
- - Two adjacent formats that are not search artifacts in the contract
77
- sense:
78
- - `.pnck`: the engine's snapshot envelope (`export()`/`restore()`
79
- serialization), consumed by the compilers and by layer-1 users.
80
- - `.pancake-range`: **deprecated** range-readable index profile. The
81
- sketch geometry replaced it (5x faster over real networks at a
82
- third of the size); readers stay supported for existing artifacts,
83
- but no new ones should be built.
84
- - Specs: [spec/SEARCH_ARTIFACT_CONTRACT.md](spec/SEARCH_ARTIFACT_CONTRACT.md),
85
- [spec/SKETCH_PROFILE.md](spec/SKETCH_PROFILE.md), and
86
- [spec/COMPLETE_PROFILE.md](spec/COMPLETE_PROFILE.md).
87
-
88
- 3. **Products and compilers**
89
- - `pikelet/`: the fast path. One command compiles a docs
90
- folder or a live site into a complete `.pikelet` file:
91
- ```bash
92
- npx pikelet compile --source ./docs --out search.pikelet
93
- ```
94
- or scaffolds a deployable search app:
95
- ```bash
96
- npx pikelet create --name my-docs-search --source ./docs --no-deploy --yes
97
- ```
98
- - `examples/05-one-file-search/`: the flagship path — the complete
99
- `.pikelet` container, its compilers, and acceptance tests.
100
- - `examples/04-static-wiki-pack/`: wiki-scale corpus preparation and the
101
- sketch artifact pipeline.
102
- - `examples/03-edge-docs-search/`: a Worker search app with a bundled
103
- snapshot, distilled query encoder, and calibrated abstention.
104
-
105
- ## Why Pikelet Exists
106
-
107
- Most vector search libraries assume a server process, native binaries, or a
108
- database service. Pikelet is built for a different deployment shape:
109
-
110
- - static files instead of always-on index servers;
111
- - byte-range reads instead of whole-index downloads;
112
- - readers for browsers, Workers, and Node;
113
- - explicit artifact identity and digest verification;
114
- - query interpretation and evaluation data carried inside the artifact;
115
- - compact corpus-side vector representations that keep memory and transfer
116
- costs sane.
117
-
118
- The result is search you can host like an asset instead of operating like a
119
- database.
120
-
121
- ## The Flagship: One-File Search
122
-
123
- The complete profile (`.pikelet`) packs the five components of a search
124
- application into one content-addressed file:
40
+ Claude now has a `search` tool over your docs. Or query it directly from code:
41
+
42
+ ```js
43
+ import { openPikeletFile } from 'pikelet-wasm/complete';
44
+ const pack = await openPikeletFile('docs.pikelet');
45
+ const out = await pack.query('how do workers restore snapshots', { k: 5 });
46
+ console.log(out.matchQuality, out.results[0]?.title);
47
+ // 'strong' 'Snapshot restore'
48
+ ```
49
+
50
+ That's the whole loop. `compile` also takes a live URL (`--source https://docs.example.com`) instead of a directory. If you want a deployed search app — a Worker + UI, not a file — use `npx pikelet create` instead; see [`pikelet/README.md`](pikelet/README.md) for the full CLI reference and the tradeoffs between the two. `compile` never needs the scaffold path's `@xenova/transformers` dependency; skip its ~140 MB install with `npm install -g pikelet --omit=optional` if you only need `compile`/`mcp`.
51
+
52
+ Under the hood, the file is one container for everything a reader needs:
125
53
 
126
54
  ```text
127
- index embedded .pancake-sketch artifact
128
- corpus JSON records behind an offsets table
129
- encoder query interpreter declaration or inline model data
130
- calibration abstention / match-quality model
131
- evaluation golden queries and expected behavior
132
- lexical BM25 inverted index for hybrid retrieval (optional; readers
133
- that predate it serve vector-only from the same file)
55
+ documents
56
+ │
57
+ ▼
58
+ ┌──────────────────┐
59
+ │ docs.pikelet │
60
+ │ │
61
+ │ corpus │
62
+ │ semantic index │
63
+ │ lexical index │
64
+ │ query encoder │
65
+ │ integrity │
66
+ │ calibration │
67
+ │ evaluation │
68
+ └──────────────────┘
69
+ │
70
+ ├── local file
71
+ └── static HTTP / object storage
72
+ │
73
+ ▼
74
+ browser / Node / edge / agent
134
75
  ```
135
76
 
136
- The reader verifies the manifest and eager segments, opens the sketch index
137
- against a range source, and hydrates only the records the final results
138
- need.
139
-
140
- Query interpretation comes in three kinds. Each picks two of
141
- **self-contained**, **small**, and **teacher-quality**, and gives up the
142
- third — that trade-off, not history, is why all three exist:
143
-
144
- - **kind 3: `inline-transformer-v1`** — self-contained + teacher quality.
145
- The artifact carries the WordPiece vocab and quantized MiniLM teacher
146
- weights as verified data; the reader ships the execution kernels. Costs
147
- ~25 MB of encoder no matter how small the corpus. This is the default:
148
- `pikelet compile` builds kind 3.
149
- - **kind 1: `student-inline-v1`** — self-contained + small. A
150
- corpus-distilled student encoder (~1 MiB), pure-JS execution. The only
151
- way a small docs corpus compiles to a file measured in single-digit MB;
152
- quality is bounded by the distillation.
153
- - **kind 2: `external-transformers-v1`** — small + teacher quality, but
154
- not self-contained: the artifact pins a host-supplied encoder (model id,
155
- pooling, normalization, max tokens) and verifies it against embedded
156
- test vectors before serving. The escape hatch for encoders too big to
157
- inline and hosts that already run one.
158
-
159
- Kind 3 is the flagship: the file carries the corpus, index, tokenizer,
160
- encoder weights, calibration, and evaluation data — the reader supplies
161
- code, the artifact supplies everything else. Ask it about volcanoes and no
162
- service, model host, or network dependency is involved beyond range reads
163
- of the file itself.
77
+ Search is the interface. **The file is the knowledge deployment unit.**
164
78
 
165
- ```bash
166
- cd examples/05-one-file-search
167
- node compile-wiki.mjs --inline-encoder
168
- node compile.mjs --inspect pancake-wiki-inline.pancake
169
- node test-inline.mjs
79
+ ---
80
+
81
+ ## The shortest demonstration
82
+
83
+ A real Pikelet pack contains **456,153 Simple English Wikipedia records** in a **648.5 MiB** artifact.
84
+
85
+ That file was served by a deliberately dumb HTTP server whose only relevant capability was:
86
+
87
+ ```text
88
+ fs.createReadStream + HTTP Range
170
89
  ```
171
90
 
172
- Where the inline wiki artifact currently stands:
91
+ There was no vector database, no search service, no embedding service, no application server that understood the pack, no index process on the server.
92
+
93
+ The pack was mounted by URL and exposed to a real headless Claude process through MCP. Each invocation was restricted with `--strict-mcp-config` to only:
173
94
 
174
95
  ```text
175
- 456,153 records
176
- ~649 MiB complete artifact (680,029,254 bytes)
177
- manifest identity: 1b180adf4c6cebb2dcd5615256df6a25dac5fda8738dbbc11d60af86046f97f3
178
- recall@10: 95.2% at the recommended rerank (97.0% at C=600), against exact
179
- fp32 brute-force ground truth on the 200-query pre-registered eval —
180
- 98.5% on hand-written natural-language questions, 94.8% on single-title
181
- lookups (augmented retrieval: hybrid candidates, distance order)
182
- serves over HTTP range reads: opens on ~52 MiB of the file, ~127 range
183
- requests per query (cluster-ordered row layout + manifest fetch hints)
184
- natural-language query with no host encoder option, ~110 ms/query end to
185
- end locally (the resident scan auto-stages the engine's SIMD kernel at
186
- this scale; ~540 ms on the pure-JS scan it falls back to)
96
+ mcp__wiki-pack__search
97
+ mcp__wiki-pack__list_packs
98
+ mcp__wiki-pack__get_record
187
99
  ```
188
100
 
189
- Release asset (v4 — recovers the pack's canonical build: k-means
190
- cluster-ordered rows and the 192-dim 2:1-pooled sketch, where v1–v3
191
- embedded a 96-dim 4:1 sketch in unpermuted order and measured 82.4%; adds
192
- the hybrid BM25 lexical segment and a format-2 sketch with per-row read
193
- verification; earlier versions stay published for older checkouts):
194
- `https://github.com/mcn92/pancake/releases/download/artifact-wiki-inline-v4/pancake-wiki-inline.pancake`
101
+ No web tool. No filesystem tool. No alternate retrieval tool.
195
102
 
196
- `node test-inline.mjs` downloads that file automatically when it is missing
197
- locally, verifies the manifest identity, and then runs the acceptance
198
- checks.
103
+ One of the actual responses:
199
104
 
200
- See [examples/05-one-file-search/README.md](examples/05-one-file-search/README.md)
201
- for the full one-file walkthrough.
105
+ > **Q: What causes earthquakes?**
106
+ >
107
+ > Earthquakes are caused by the movement of tectonic plates in the Earth's crust. Stress builds up along fault boundaries where plates grind against or ride over one another; when that stress exceeds the friction holding the locked fault in place, it suddenly releases as shock waves (seismic waves) that shake the ground [record 429637].
108
+ >
109
+ > There are three main fault types that can produce quakes — normal, reverse (thrust), and strike-slip — and some quakes have less clear-cut causes, like deep shifts in ancient rock or mine-collapse "tremors" [record 429638].
110
+ >
111
+ > Two related mechanisms compound this:
112
+ >
113
+ > * Subduction, where one plate is forced under another into the mantle, produces the most powerful earthquakes and can trigger tsunamis [record 132833].
114
+ > * More broadly, the slow convection-driven movement of crustal plates over the hot mantle is the underlying driver of quakes, volcanoes, and mountain formation [record 132267].
115
+ >
116
+ > Volcanic eruptions can also cause earthquakes, though these are localized to areas near active volcanoes [record 429637].
202
117
 
203
- ## The Design Choice Everything Hangs On
118
+ The model did not just repeat the first search hit. It retrieved multiple records and synthesized them into a sourced answer.
204
119
 
205
- The engine's quantized backend stores corpus vectors as row-wise affine
206
- uint8:
120
+ Other runs produced:
207
121
 
208
122
  ```text
209
- value ~= offset[row] + scale[row] * byte
123
+ Who was the first person on the moon? → Neil Armstrong (records 408686, 408684)
124
+ How do volcanoes form? → plates + subduction + hotspots
125
+ (records 132770, 132765, 132852, 132853)
126
+ How does photosynthesis work? → two-phase reaction, cited (5 records: 114219, 114220, 114222, 114223, 269279)
127
+ What is the capital of France? → Paris (records 454641, 302684)
210
128
  ```
211
129
 
212
- That one constraint propagates through the whole system:
130
+ That is the product in one demonstration:
213
131
 
214
- - HNSW search keeps the query as float32 and scores it against compressed
215
- corpus rows.
216
- - Distance kernels fuse dequantization with dot/L2 accumulation.
217
- - SIMD paths load bytes, widen lanes, apply scale/offset, and accumulate.
218
- - Graph maintenance uses symmetric node-to-node distances derived from the
219
- compressed rows and row statistics.
220
- - Snapshots serialize compact scale/offset/qdata arrays.
221
- - Sketch artifacts scan compressed resident rows and fetch only rerank
222
- candidates.
223
- - Range-backed readers keep the large corpus side compressed while queries
224
- stay precise and transient.
132
+ > **A model is interrogating a 456,153-record knowledge base whose backend is a static file.**
225
133
 
226
- This is why the engine work and the artifact work belong in one project:
227
- the same corpus-side representation serves in-memory search, serialized
228
- snapshots, resident sketches, range reads, and reranking.
134
+ The model may already contain some of these facts in its pretrained parameters. This test demonstrates the retrieval, synthesis, citation, and deployment path; the Veyra ablation below tests whether support changes when evidence is removed from the pack.
229
135
 
230
- ## Install
136
+ **Network cost.** In a single persistent session (one mount, five queries, one repeat):
231
137
 
232
- From npm:
138
+ | Operation | Bytes transferred | Requests |
139
+ | ------------------------------- | ----------------: | -------: |
140
+ | Initial mount | 51.7 MiB | 11 |
141
+ | Query 1 + one-time encoder load | 25.9 MiB | 77 |
142
+ | Query 2 | 504.4 KiB | 37 |
143
+ | Query 3 | 499.4 KiB | 46 |
144
+ | Query 4 | 968.9 KiB | 75 |
145
+ | Query 5 | 1.1 MiB | 116 |
146
+ | Repeated query 1 | 238.4 KiB | 4 |
147
+ | **Total** | **80.8 MiB** | **366** |
233
148
 
234
- ```bash
235
- npm install pancake-wasm
149
+ Roughly **12.5% of the 648.5 MiB artifact** crossed the wire across that whole session. Excluding the one-time ~25 MiB encoder load, fresh-query traffic ran **~0.5–1.1 MiB per query**. The complete artifact was never downloaded.
150
+
151
+ The headless-Claude test above used a fresh process per question, so each invocation repaid the ~52 MiB mount and ~25 MiB encoder cost — about 78 MiB per cold query. That's a real operational distinction: **persistent sessions amortize mount and encoder cost; independent cold processes do not.**
152
+
153
+ *(The large benchmark fixture still carries its historical `.pancake` filename from before the Pikelet rename; current artifacts use `.pikelet`.)*
154
+
155
+ ---
156
+
157
+ ## What happens if the answer is changed or removed from the file?
158
+
159
+ This is different from asking whether search returns sensible documents. To test whether Pikelet's retrieval-quality signal could form a useful evidence boundary, a synthetic corpus called **Station Veyra Registry** was built. One version contained the fact:
160
+
161
+ ```text
162
+ The Tovash project is housed in Chamber 17.
236
163
  ```
237
164
 
238
- From a checkout:
165
+ A second pack was byte-for-byte identical except that the record containing that fact was removed.
239
166
 
240
- ```bash
241
- git clone https://github.com/mcn92/pancake.git
242
- cd pancake
243
- npm install
167
+ ```text
168
+ full pack: matchQuality: strong confidence: 0.915 → Chamber 17
169
+ ablated pack: matchQuality: none confidence: 0.136 → unsupported
244
170
  ```
245
171
 
246
- The checkout includes prebuilt WASM in `dist/`, so you don't need to
247
- rebuild the engine unless you edit `src/`.
172
+ After the model had already seen the answer, it was queried against the ablated pack with prompts like "Confirm Tovash is in Chamber 17" and "Tovash project Chamber 17 location." The retrieval result stayed unsupported and the model declined to confirm the location from the pack. A separate, informal session then probed a fresh isolated agent with neutral prompts, leading prompts, authority pressure, invitations to use general knowledge, cross-record distractors, and repeated pressure — six adversarial framings — and it continued distinguishing supported facts from the removed one in every case; that session wasn't captured as a script, so treat it as a described observation rather than a reproducible result.
248
173
 
249
- ## Runtime Entry Points
174
+ This does **not** mean Pikelet can prevent an LLM from hallucinating. It means the artifact can expose an explicit evidence boundary that a consuming model can choose to respect — and removing evidence from the artifact changed what that model was able to support from the mounted source.
250
175
 
251
- Published package:
176
+ A second intervention changed only the Tovash location source record from **Chamber 17** to **Chamber 43**, rebuilt the pack, and repeated the same prompt in a fresh session:
252
177
 
253
- ```js
254
- // Node.js CJS
255
- const Pikelet = require('pikelet-wasm');
178
+ ```text
179
+ chamber43 pack: matchQuality: strong confidence: 0.916 → Chamber 43
180
+ ```
256
181
 
257
- // Node.js ESM
258
- import Pikelet from 'pikelet-wasm';
182
+ The model answered **Chamber 43** and cited the same logical source record. With Veyra not mounted at all, the same prompt produced no chamber number and the model declined to guess. Change the evidence and the grounded answer changes with it; remove the evidence source and the answer disappears.
259
183
 
260
- // Explicit Node/browser entries
261
- import Pikelet from 'pikelet-wasm/node';
262
- import Pikelet from 'pikelet-wasm/web';
184
+ The test is synthetic and intentionally narrow. All three packs and the `matchQuality`/`confidence` numbers above are reproducible: `node examples/05-one-file-search/web/public/reproduce-ablation.mjs`. The retrieval side is scripted; the paired LLM-session claims (declining to confirm the removed fact, answering Chamber 43, declining without the pack mounted) were run separately and aren't reproduced by that script.
263
185
 
264
- // Search Artifact layer (sketch readers/builders; the deprecated
265
- // .pancake-range reader stays exported for existing artifacts)
266
- import { PancakeSketchArtifact, buildSketchArtifact } from 'pikelet-wasm/artifact';
186
+ ---
267
187
 
268
- // Complete one-file profile: reader (any runtime) and builder (Node)
269
- import { openPikeletFile } from 'pikelet-wasm/complete';
270
- import { assemblePancakeFile } from 'pikelet-wasm/complete/builder';
271
- ```
188
+ ## What does the compression cost?
272
189
 
273
- Repository checkout:
190
+ The artifact architecture is only useful if the compact representation does not quietly destroy retrieval. The retrieval path was decomposed on BEIR rather than evaluated only end-to-end, across four configurations that isolate each transformation:
274
191
 
275
- ```js
276
- const Pikelet = require('./pancake.js');
277
- import Pikelet from './pancake.node.mjs';
278
- import Pikelet from './pancake.web.mjs';
192
+ ```text
193
+ A upstream all-MiniLM-L6-v2, float32 corpus, exhaustive search
194
+ B Pikelet embedded encoder, float32 corpus, exhaustive search
195
+ C Pikelet embedded encoder, affine-u8 corpus, exhaustive search
196
+ D Pikelet embedded encoder, affine-u8 corpus, Pikelet HNSW
279
197
  ```
280
198
 
281
- The browser entry expects the runtime or bundler to resolve
282
- `dist/engine.wasm`.
199
+ Corpus mapping and evaluation config were frozen before the runs. Official BEIR qrels are used; no LLM judges relevance.
283
200
 
284
- ## Engine Quick Start
201
+ | Dataset | Config | nDCG@10 | Recall@10 | Recall@100 | median ms/q | p95 ms/q |
202
+ | -------- | ------------------------------------------ | ---------: | ---------: | ---------: | ----------: | -------: |
203
+ | SciFact | A — upstream float exhaustive | 0.6451 | 0.7833 | 0.9250 | 3.6 | 4.0 |
204
+ | SciFact | B — Pikelet encoder / float exhaustive | 0.6512 | 0.7942 | 0.9417 | 3.8 | 4.4 |
205
+ | SciFact | C — Pikelet encoder / affine-u8 exhaustive | 0.6500 | 0.7942 | 0.9417 | 6.9 | 8.7 |
206
+ | SciFact | D — Pikelet encoder / affine-u8 HNSW | **0.6500** | **0.7942** | **0.9417** | **0.15** | **0.24** |
207
+ | NFCorpus | A — upstream float exhaustive | 0.3159 | 0.1550 | 0.3115 | 2.6 | 3.1 |
208
+ | NFCorpus | B — Pikelet encoder / float exhaustive | 0.3154 | 0.1511 | 0.3044 | 2.8 | 3.3 |
209
+ | NFCorpus | C — Pikelet encoder / affine-u8 exhaustive | 0.3154 | 0.1511 | 0.3036 | 2.9 | 3.6 |
210
+ | NFCorpus | D — Pikelet encoder / affine-u8 HNSW | **0.3135** | **0.1482** | **0.3061** | **0.16** | **0.25** |
211
+ | ArguAna | A — upstream float exhaustive | 0.3698 | 0.7653 | 0.9772 | 7.8 | 10.8 |
212
+ | ArguAna | B — Pikelet encoder / float exhaustive | 0.3506 | 0.7397 | 0.9801 | 6.1 | 7.3 |
213
+ | ArguAna | C — Pikelet encoder / affine-u8 exhaustive | 0.3496 | 0.7368 | 0.9801 | 6.4 | 7.5 |
214
+ | ArguAna | D — Pikelet encoder / affine-u8 HNSW | **0.3496** | **0.7368** | **0.9808** | **0.12** | **0.20** |
285
215
 
286
- Use the core engine when you already have vectors and want an in-memory ANN
287
- index.
216
+ These results are more useful because they are not uniformly flattering.
288
217
 
289
- ```js
290
- import Pikelet from 'pikelet-wasm';
218
+ **Affine-u8 storage is not the main quality cost** (B→C nDCG@10: SciFact 0.6512→0.6500, NFCorpus 0.3154→0.3154, ArguAna 0.3506→0.3496). **HNSW is similarly close to exhaustive search** at these settings (C→D: SciFact and ArguAna unchanged at reported precision; NFCorpus exposes a small approximation loss, 0.3154→0.3135).
291
219
 
292
- const index = await Pikelet.create({
293
- dim: 384,
294
- maxElements: 100000,
295
- metric: 'cosine',
296
- quantized: true,
297
- });
220
+ **The largest observed loss is the embedded query encoder on ArguAna**: A→B moves nDCG@10 from 0.3698 to 0.3506, about a 5% relative reduction. Recall@100 actually improves slightly (0.9772→0.9801) — the relevant document is generally still in the candidate set; the degradation is in fine ordering near the top. That limitation isn't hidden: MiniLM is small *because* the goal is to fit the query encoder inside the artifact. It is not state of the art, and the compact implementation is not behaviorally identical to an upstream sentence-transformers runtime on every task.
298
221
 
299
- const docVector = new Float32Array(384).fill(0.1); // your embedding here
300
- const queryVector = new Float32Array(384).fill(0.1); // your query embedding here
222
+ The benchmark harness, quantization-conformance test, frozen configuration, and raw runs live under `bench/beir/`.
301
223
 
302
- const id = index.add(docVector);
224
+ ---
303
225
 
304
- const results = index.search(queryVector, 10);
305
- // [{ id: 0, distance: 0.12 }, ...]
226
+ ## Why make knowledge a file?
306
227
 
307
- index.delete(id);
308
- index.compact();
228
+ The conventional retrieval deployment looks something like:
309
229
 
310
- const snapshot = index.export();
311
- const restored = await Pikelet.restore(snapshot, { maxElements: 100000 });
230
+ ```text
231
+ documents → chunking → embedding service → vector database
232
+ → keyword database
233
+ → retrieval service → application API → agent
234
+ ```
235
+
236
+ That is the correct architecture for many workloads. But it is a lot of machinery when the corpus is fundamentally a release artifact: product documentation, a manual, a source tree, a legal code, a standards corpus, a research collection, a book, a knowledge snapshot, an offline reference set. These datasets often change daily, weekly, monthly, or with releases — not hundreds of times per second.
237
+
238
+ For those workloads, Pikelet asks a different question:
312
239
 
313
- index.dispose();
314
- restored.dispose();
240
+ > **What if retrieval could be compiled ahead of time and distributed with the corpus?**
241
+
242
+ Then deployment becomes: copy the file, cache it, pin it, put it behind a CDN, mount it by URL. The storage layer does not need to know that the file contains vectors.
243
+
244
+ It is not a replacement for a mutable vector database. It is an attempt to make **static and slowly changing knowledge deploy like any other artifact**.
245
+
246
+ ---
247
+
248
+ ## Attach a pack to an LLM
249
+
250
+ `pikelet mcp` exposes one or more packs through the Model Context Protocol:
251
+
252
+ ```bash
253
+ npx pikelet mcp install \
254
+ --client claude-code \
255
+ --pack https://example.com/docs.pikelet#<sha256>
315
256
  ```
316
257
 
317
- If you have records with vectors:
258
+ The agent gets `search`, `get_record`, `list_packs`, `verify_pack`. A search result carries the identity of the pack and the location of the source record, so an agent can work against `product-docs.pikelet`, `rust-reference.pikelet`, `policy-2026-09.pikelet`, `customer-manual-v4.pikelet` without each publisher operating a retrieval API. The artifact can be local, private, public, behind authenticated object storage, or distributed like any other static asset.
318
259
 
319
- ```js
320
- const rows = [
321
- { id: 'doc-1', vector: embedding1 },
322
- { id: 'doc-2', vector: embedding2 },
323
- ];
324
-
325
- const { index, idMap } = await Pikelet.fromVectors(rows, {
326
- metric: 'cosine',
327
- quantized: true,
328
- });
329
-
330
- const hits = index.search(queryEmbedding, 5);
331
- console.log(idMap.get(hits[0].id));
260
+ A pack mounted with a content hash has a stable identity — `https://example.com/docs.pikelet#8d731...` — so "what body of knowledge did this agent query?" has a reproducible answer.
261
+
262
+ **A mounted pack's content reaches the model as tool output.** `verify_pack` proves the bytes are intact and match their pinned identity; it does not prove the corpus itself is trustworthy. Mounting a pack from a source you don't control is the same trust decision as giving an agent any other untrusted-content tool — treat pack text the way you'd treat search results or fetched web pages, not as instructions.
263
+
264
+ **Try it with your own questions.** The three Veyra packs from the ablation test above (`examples/05-one-file-search/web/public/veyra*.pikelet`) are committed to this repo, so there's a working MCP mount you can query with anything, not just the fixed prompts above:
265
+
266
+ ```bash
267
+ claude -p "your question here" \
268
+ --mcp-config examples/05-one-file-search/web/public/veyra.mcp.json \
269
+ --strict-mcp-config \
270
+ --allowedTools "mcp__veyra-demo__search,mcp__veyra-demo__list_packs,mcp__veyra-demo__get_record"
332
271
  ```
333
272
 
334
- On Node entrypoints, file helpers are available:
273
+ or drop the flags and just run `claude` from the repo root — Claude Code picks up project-level `.mcp.json` configs automatically, so pointing one at this file (or copying its `veyra-demo` server into your own) gets the same three packs into an interactive session. All three packs share the same 94-ish-record synthetic corpus, so the interesting thing to ask about is Tovash's chamber — the full and Chamber 43 packs will disagree with each other, and the ablated pack won't answer at all.
335
274
 
336
- ```js
337
- const { index, idMap } = await Pikelet.loadJsonFile('vectors.jsonl', {
338
- metric: 'cosine',
339
- vectorKey: 'embedding',
340
- idKey: 'docId',
341
- maxFileBytes: 64 * 1024 * 1024,
342
- });
343
-
344
- const restored = await Pikelet.loadSnapshotFile('index.pnck', {
345
- dim: 384,
346
- maxElements: 100000,
347
- metric: 'cosine',
348
- quantized: true,
349
- });
275
+ ---
276
+
277
+ ## How a remote query runs
278
+
279
+ Over a network, what makes search slow is not bytes; it's sequential round trips. Graph traversal is a chain of dependent reads — fetch node, inspect neighbors, fetch the next node, repeat — and across object storage those dependent round trips dominate. Measured against SIFT1M over real network storage, a resident scan plus one batched candidate-fetch phase beat graph traversal by about 5× at equal recall. So the remote artifact path carries no graph. The HNSW graph still exists — it's the in-memory engine, used when the whole index is local and round trips are free; that's the path BEIR Config D measures.
280
+
281
+ For the remote path:
282
+
283
+ 1. **Resident tier** — a pooled sketch of every semantic row loads once when the artifact opens and stays in memory. For the 456,153-record Wikipedia pack: 648.5 MiB artifact, 51.7 MiB mount fetch. A SIMD kernel scans all of it in milliseconds.
284
+ 2. **Candidate selection** — the scan picks the top `C` candidates against the query vector; the lexical BM25 index can contribute additional candidates into the same set. `C` is the only real knob: the compiler measures recall against brute force on held-out queries at build time and writes the operating point into the file, so readers don't have to guess it.
285
+ 3. **Batched byte-range fetch** — full affine-u8 rows for the candidate set are fetched in one parallel round, nearby ranges coalesced, rather than a graph-dependent network walk.
286
+ 4. **Full rerank** — fetched rows are scored against the float query and verified against their digest.
287
+ 5. **Corpus hydration** — only the records needed for returned results are fetched, each carrying record ID, title, section, source, and artifact identity. The reader returns evidence, not anonymous vector IDs.
288
+
289
+ That architecture is why a 648.5 MiB pack can answer a fresh query while fetching about 0.5–1.1 MiB after warmup.
290
+
291
+ ---
292
+
293
+ ## Integrity is part of the read path
294
+
295
+ A range-readable artifact cannot hash the entire file on every open without defeating the point of range reads, so integrity is layered: the resident structural portion is verified during open; lazily fetched index rows and corpus records carry independent commitments and are checked when read. Bytes a query never touches don't have to cross the network merely to prove the bytes it *did* use were correct.
296
+
297
+ One exception: a lexical (BM25) segment above 8 MiB opens lazily and is covered only by the manifest's whole-segment digest, not per-read like index rows and corpus records — the same transitional stance format-1 sketch rows carry. A pack large enough for this to apply can have its lexical candidates altered between open and a full verification pass without failing a query.
298
+
299
+ The failure-mode suite (`bench/range-proof/failure-modes.mjs`) exercises the important cases:
300
+
301
+ ```text
302
+ correct Range server → mounts, queries successfully
303
+ server ignores Range → small files fall back to a bounded download;
304
+ large files refuse ("host ignores Range and the
305
+ file is 680029254 bytes; refusing full download" —
306
+ the cap is currently 64 MiB)
307
+ bytes change under a pinned identity → refused; tampered data is not silently served
308
+ truncated Range response → fetch fails; no partial result is silently
309
+ interpreted as valid data
350
310
  ```
351
311
 
352
- ## Complete Artifact Quick Start
312
+ The point is not that static HTTP is magically reliable. The point is that a static artifact can fail in explicit, testable ways.
353
313
 
354
- Use the complete reader when you want a file that is itself the search
355
- application data plane. To build one from your own docs, see the compile
356
- command above (`npx pikelet compile`).
314
+ ---
357
315
 
358
- ```js
359
- import { openPikeletFile } from 'pikelet-wasm/complete'; // or './complete/index.mjs' in a checkout
316
+ ## Match quality and abstention
317
+
318
+ A nearest neighbour is not automatically evidence that a corpus answers a question. Pikelet can calibrate retrieval signals at build time (best semantic distance, distance margin, lexical coverage, retrieval agreement). When the corpus supports a reliable classifier, results carry `matchQuality: strong | weak | none`. When calibration can't separate supported from unsupported reliably — a single novel may be semantically homogeneous enough that the fit isn't trustworthy — Pikelet reports `matchQuality: unscored` and records why calibration was skipped, rather than manufacturing confidence. A `none` verdict withholds `results` by default; pass `query(text, { showAbstained: true })` to see the raw retrieval anyway — `matchQuality` and `confidence` are unaffected either way.
360
319
 
361
- const search = await openPikeletFile('examples/05-one-file-search/pancake-docs.pancake');
320
+ `matchQuality` is evidence about retrieval support. It is **not** a guarantee that an LLM will never hallucinate.
362
321
 
363
- const out = await search.query('how do workers restore snapshots', { k: 5 });
322
+ ---
364
323
 
365
- console.log(out.matchQuality, out.confidence);
366
- console.log(out.results[0].title);
324
+ ## What's inside a `.pikelet`
367
325
 
368
- await search.close();
326
+ ```text
327
+ manifest format versions, segment offsets, artifact identity
328
+ semantic index resident sketch rows, full affine-u8 rows, row commitments
329
+ corpus source records, offsets, per-record commitments
330
+ query encoder WordPiece vocabulary, quantized MiniLM weights, encoder declaration
331
+ lexical index BM25 postings
332
+ calibration supported / unsupported retrieval model
333
+ evaluation golden queries / expected behavior
369
334
  ```
370
335
 
371
- For kind-3 artifacts, no host encoder is needed:
336
+ The artifact is immutable. Publish a new corpus by publishing a new artifact; old artifacts retain their identity.
372
337
 
373
- ```js
374
- const search = await openPikeletFile('pancake-wiki-inline.pancake');
375
- const out = await search.query('how do volcanoes form', { k: 5 });
338
+ **Encoder profiles.** Self-containment has a cost — the default profile carries ~25 MiB of MiniLM data regardless of corpus size. Three arrangements: **inline** (tokenizer + weights + runtime in the artifact — largest file, no external dependency, used by the Wikipedia demo above), **distilled/compact** (smaller corpus-specific representation, lower footprint, potentially lower quality), **host-supplied** (the artifact declares expected encoder behavior and verifies the host implementation against embedded test vectors — smallest artifact, no longer fully self-contained). The format treats the encoder as a capability, not Pikelet's identity — MiniLM is the current choice, not a permanent requirement.
339
+
340
+ ---
341
+
342
+ ## One decision
343
+
344
+ Most of the implementation follows from how a vector is stored. Each row is 8-bit integers plus two floats:
345
+
346
+ ```text
347
+ x[d] ≈ offset + scale · q[d] q[d] ∈ 0..255
376
348
  ```
377
349
 
378
- For kind-2 artifacts, pass `options.encodeQuery`; the reader runs it
379
- against the declaration's verification vectors before serving and refuses
380
- the open if they disagree (`info().encoderVerified` reports the outcome).
350
+ That's a per-row affine map. It costs 4× less memory than float32, which is the ordinary reason to do it. The reason it runs through the whole project is that every operation search needs — dot products, sums, averages — is linear, and affine terms factor out of linear operations.
381
351
 
382
- The reader also accepts a range source, which is how a `.pikelet` on R2 or
383
- a CDN gets queried without downloading it:
352
+ **Query against a stored row.** The query stays float32 and is never quantized.
384
353
 
385
- ```js
386
- const search = await openPikeletFile({
387
- size,
388
- async read(offset, length) {
389
- // return bytes for [offset, offset + length)
390
- },
391
- });
354
+ ```text
355
+ y·x = offset·Σy + scale·(y·q)
392
356
  ```
393
357
 
394
- On integrity: every artifact-derived read is bounds-checked and budgeted
395
- (`maxReadBytes`, `maxRecordBytes`). Format-2 files (`pancake-complete-v2`,
396
- the builder's default) verify each hydrated record against a per-record
397
- digest on its own range read (`info().corpusIntegrity`), and an embedded
398
- format-2 sketch verifies each lazily fetched index row the same way
399
- (`info().indexRowIntegrity`) — so a tampered transport fails the query
400
- instead of skewing its results. Older files with a format-1 sketch commit
401
- the lazy index rows to the identity but don't verify them per read; for
402
- those, pass `verifyIndexVectors: true` (or call `verifyVectors()`) to
403
- authenticate them in one full pass before trusting results from an
404
- untrusted transport (spec/COMPLETE_PROFILE.md section 6).
405
-
406
- ### Attaching packs to an LLM
407
-
408
- A complete artifact doubles as a portable knowledge pack: everything an
409
- LLM needs to query a body of knowledge — corpus, indexes, encoder,
410
- calibrated abstention, integrity commitments — in one immutable,
411
- identity-pinned file. `pikelet mcp` serves packs over the
412
- Model Context Protocol so Claude Code, Claude Desktop, or any agent
413
- framework can attach them as a retrieval tool with one config line — no
414
- vector database, no embedding service, no retrieval backend. Results
415
- carry full provenance (pack, manifest identity, title, heading path,
416
- source), and a pack that cannot answer says so instead of returning
417
- plausible noise. See the pikelet README's
418
- "Attaching packs to an LLM (MCP)" section.
419
-
420
- ## Demos
421
-
422
- Start with [DEMOS.md](DEMOS.md) for the maintained demo path.
423
-
424
- Useful commands:
358
+ The inner loop multiplies floats by bytes. The two constants come in once at the end. No decompressed copy of the row ever exists.
425
359
 
426
- ```bash
427
- npm run demo # synthetic engine validation
428
- npm run demo:artifact # range artifact search
429
- npm run demo:sketch # sketch artifact search/rerank path
430
- npm run test:browser # browser smoke test
431
- node examples/05-one-file-search/test-file.mjs
432
- node examples/05-one-file-search/test-wiki.mjs
433
- node examples/05-one-file-search/test-inline.mjs
360
+ **Stored row against stored row.** This is what graph construction needs, thousands of times per insert.
361
+
362
+ ```text
363
+ x_i·x_j = D·o_i·o_j + o_i·s_j·Σq_j + o_j·s_i·Σq_i + s_i·s_j·(q_i·q_j)
434
364
  ```
435
365
 
436
- The one-file example is the closest thing to the project's current center
437
- of gravity:
366
+ `D` is the vector's dimensionality (384 for the bundled encoder). `q_i·q_j` is an integer dot over two byte arrays, which maps efficiently onto SIMD. `Σq` is stored per row. The HNSW graph is built and repaired entirely on compressed data.
367
+
368
+ **Pooling.** Average adjacent groups of `p` bytes and you get a shorter row. Because the mean of `offset + scale·q` over a group equals `offset + scale·mean(q)`, the shorter row keeps the *same two constants*. That's the resident sketch tier; a micro tier is the same thing done twice.
369
+
370
+ **Weights.** The bundled query encoder is a 6-layer MiniLM whose matrices are stored the same way, one (scale, offset) per 64-column block. Activations stay float32, weights stay bytes, dequantization happens inside the matmul — the same widen-and-multiply trick, a separate kernel, that scores vectors.
371
+
372
+ The consequence for the file format: a row decodes from its own bytes and its own two floats and nothing else. No codebook, no global statistics. Every row is a fixed-size byte range at a computable offset — locatable by arithmetic, fetchable on its own, hashable on its own, verifiable on the read that fetches it.
373
+
374
+ **Why not product quantization.** Product quantization would compress harder. It would also need a codebook, a training pass, table lookups in the distance kernel, and rows that mean nothing without the codebook. Four-to-one with no shared state was the better trade for a file meant to be read in pieces.
375
+
376
+ ---
377
+
378
+ ## Build once, publish anywhere
438
379
 
439
380
  ```bash
440
- cd examples/05-one-file-search
441
- node compile.mjs
442
- node test-file.mjs
443
- node serve.mjs
381
+ npx pikelet compile --source ./docs --out docs.pikelet
444
382
  ```
445
383
 
446
- ## API Snapshot
384
+ ```js
385
+ import { openPikeletFile } from 'pikelet-wasm/complete';
447
386
 
448
- Core index:
387
+ const pack = await openPikeletFile('https://example.com/docs.pikelet#<sha256>');
388
+ const result = await pack.query('how do workers restore snapshots', { k: 5 });
449
389
 
450
- ```js
451
- const index = await Pikelet.create(options);
452
- index.add(vector);
453
- index.addBatch(vectors);
454
- index.search(query, k, options?);
455
- index.searchFiltered(query, k, allowedIds, options?); // allowedIds: Set of ids
456
- index.delete(id);
457
- index.compact();
458
- index.export();
459
- index.dispose();
390
+ console.log(result.matchQuality);
391
+ for (const hit of result.results) console.log(hit.title, hit.section);
392
+
393
+ await pack.close();
460
394
  ```
461
395
 
462
- Common options:
396
+ The same artifact can also be opened from a local path or any custom source implementing `{ size, read(offset, length) }`.
397
+
398
+ **Or use the search engine directly**, if you already have vectors and don't need the artifact layer:
463
399
 
464
400
  ```js
465
- {
466
- dim: 384,
467
- maxElements: 100000,
468
- metric: 'cosine', // 'cosine' or 'l2'
469
- quantized: true,
470
- M: 12,
471
- efConstruction: 75,
472
- efSearch: 100,
473
- seed: 108
474
- }
401
+ import Pikelet from 'pikelet-wasm';
402
+
403
+ const index = await Pikelet.create({ dim: 384, maxElements: 100000, metric: 'cosine', quantized: true });
404
+ index.add(vector);
405
+ const results = index.search(query, 10);
406
+ const snapshot = index.export();
475
407
  ```
476
408
 
477
- Snapshot helpers:
409
+ float32 HNSW, affine-u8 HNSW, insert/delete, compaction, import/export, deterministic snapshots, WASM with no native addon dependency. Pikelet is usable as a vector engine — that's just no longer the main reason the project exists.
478
410
 
479
- ```js
480
- await Pikelet.restore(bytes, options);
481
- await Pikelet.loadSnapshotFile(path, options);
482
- await Pikelet.loadJsonFile(path, options);
483
- await Pikelet.fromVectors(rows, options);
484
- ```
411
+ ---
485
412
 
486
- Range and sketch artifact APIs ship at `pikelet-wasm/artifact`; the
487
- complete-profile reader and builder ship at `pikelet-wasm/complete` and
488
- `pikelet-wasm/complete/builder`. The wiki-scale compilers and acceptance
489
- tests live in `examples/05-one-file-search/`.
413
+ ## What this is not
490
414
 
491
- ## Building From Source
415
+ **Not a claim that vector databases are obsolete.** If your corpus changes continuously, needs transactional updates, serves many tenants, or already lives comfortably in a database, use a database. Pikelet targets `build → publish → query many times → replace on release`.
492
416
 
493
- You only need this if you change the C++ engine under `src/`.
417
+ **Not state-of-the-art embedding research.** The bundled model is MiniLM-L6, chosen for being small enough to live inside the artifact. ArguAna demonstrates a real quality cost from the compact encoder.
494
418
 
495
- Requirements:
419
+ **Not "LLMs can no longer hallucinate."** The artifact exposes retrieved evidence, provenance, integrity, and an explicit support signal. Whether an agent obeys those signals is an agent behavior question.
496
420
 
497
- - Node.js
498
- - Emscripten SDK
421
+ **Not a new HNSW algorithm.** HNSW is HNSW. HTTP Range is HTTP Range. BM25 is BM25. MiniLM is MiniLM. Affine quantization is not new either. The project is about what becomes possible when those pieces are arranged around one constraint: **the knowledge base itself must be distributable as a file and remain useful without dedicated retrieval infrastructure.**
499
422
 
500
- Build:
423
+ ---
501
424
 
502
- ```bash
503
- npm run build
504
- ```
425
+ ## Why the file format matters
505
426
 
506
- Build scalar and SIMD variants:
427
+ A retrieval service is identified by an endpoint. A Pikelet is identified by its contents.
507
428
 
508
- ```bash
509
- npm run build:all
510
- ```
429
+ - **Reproducibility** — pack hash + record ID identifies the exact evidence a model retrieved.
430
+ - **Versioning** — different releases are different artifacts (`docs-v1.pikelet`, `docs-v2.pikelet`); no silent mutation.
431
+ - **Distribution** — a pack can be mirrored by infrastructure that knows nothing about semantic search.
432
+ - **Offline use** — once local, no network or embedding service needed.
433
+ - **Customer-controlled knowledge** — hand someone the artifact instead of granting access to internal retrieval infrastructure.
434
+ - **Agent knowledge environments** — run a task against a frozen snapshot, reproduce it later against the same bytes.
511
435
 
512
- The build emits `dist/engine.js`, `dist/engine.wasm`, and scalar variants.
436
+ > **Can knowledge become a first-class software artifact rather than something that always has to live behind a service?**
513
437
 
514
- ## Tests
438
+ ---
515
439
 
516
- Core (`npm test` also runs the artifact and complete-profile conformance
517
- suites):
440
+ ## When Pikelet is a good fit — and when it isn't
518
441
 
519
- ```bash
520
- npm test
521
- npm run test:fuzz
522
- npm run test:simd
523
- npm run test:browser
524
- npm run test:scaffold
525
- ```
442
+ **Use it when:**
526
443
 
527
- Complete profile, end to end:
444
+ - your corpus is static or changes on a release cycle
445
+ - you want semantic search without operating search infrastructure
446
+ - you want search inside a browser, Node process, edge worker, Electron app, or offline tool
447
+ - you want to distribute a searchable corpus to someone else
448
+ - you want an LLM agent to query a frozen, versioned body of knowledge
449
+ - static/object storage is easier to deploy than a database
450
+ - provenance and reproducibility matter
451
+ - a content-addressed knowledge snapshot is useful
528
452
 
529
- ```bash
530
- node examples/05-one-file-search/test-file.mjs
531
- node examples/05-one-file-search/test-wiki.mjs
532
- node examples/05-one-file-search/test-inline.mjs
533
- ```
453
+ Examples: documentation, SDK/API references, technical manuals, research corpora, standards, legal texts, books, product knowledge, code/documentation snapshots, evaluation corpora, agent reference packs.
534
454
 
535
- The inline wiki acceptance test downloads the large release artifact on
536
- demand when `examples/05-one-file-search/pancake-wiki-inline.pancake` is
537
- missing.
455
+ **Don't use it if:**
538
456
 
539
- ## Performance Notes
457
+ - your corpus has heavy online writes
458
+ - you need transactional mutation
459
+ - you need a multi-tenant authoritative search service
460
+ - you need exact nearest-neighbour search at very large scale
461
+ - your corpus is so large that a linear resident sketch scan is inappropriate
462
+ - you already operate a vector database happily and portability buys you nothing
540
463
 
541
- Performance is workload-dependent, but these are the knobs that matter:
464
+ The resident remote scan is linear in row count. The current architecture targets corpora through the low millions of records, not arbitrary web scale.
542
465
 
543
- - `quantized: true` for much smaller corpus-side vector storage.
544
- - `efSearch` for the recall/latency tradeoff.
545
- - sketch artifacts for depth-1 candidate selection before lazy rerank
546
- reads.
547
- - corpus layout and clustering for fewer remote read rounds.
548
- - kind-3 inline encoders when deployment shouldn't depend on an external
549
- ML runtime.
466
+ ---
550
467
 
551
- The older engine-level benchmarks are still useful for HNSW behavior. The
552
- newer artifact-level benchmarks are more representative of what the project
553
- is actually for: search from static files and range-readable storage.
468
+ ## Repository map
554
469
 
555
- The native baselines the engine benchmarks compare against (faiss-node,
556
- hnswlib-node, usearch) are not dependencies of `pikelet-wasm`; install
557
- them on demand with `cd benchmarks && npm install` before running those
558
- scripts. See `benchmarks/`, `benchmark_results/`, and the example READMEs
559
- for current numbers.
470
+ ```text
471
+ src/ C++/WASM vector engine: HNSW, float32 and
472
+ affine-u8 backends, mutation, compaction,
473
+ snapshot import/export
474
+ complete/, pikelet-artifact.js Readers and builders for the complete
475
+ range-readable artifact
476
+ pikelet/ CLI, compiler, MCP server, encoder
477
+ integration, higher-level tooling
478
+ spec/ Byte-level artifact contracts
479
+ bench/beir/ Frozen BEIR ablation harness (encoder,
480
+ quantization, HNSW quality)
481
+ bench/range-proof/ The deliberately boring static-HTTP proof:
482
+ dumb-server.mjs, proof.mjs, mcp-proof.mjs,
483
+ llm-proof.mjs, failure-modes.mjs
484
+ examples/05-one-file-search/ Large single-artifact search and embedded
485
+ encoder work
486
+ examples/06-mcp-knowledge-pack/ Compile, mount, search, and hydrate
487
+ records through MCP
488
+ packs/ Pack hosting and distribution examples
489
+ ```
560
490
 
561
- ## Tradeoffs
491
+ ---
562
492
 
563
- Pikelet is a good fit when:
493
+ ## Reproduce the proofs
564
494
 
565
- - you want search in JavaScript runtimes without native addons;
566
- - you want static-file or edge deployment;
567
- - you need reproducible, inspectable search artifacts;
568
- - your corpus can be built offline;
569
- - byte-range reads are cheaper than operating a dedicated search service.
495
+ **Range proof** (`bench/range-proof/`) is built around a server that does not understand Pikelet. It checks: static HTTP + Range + Pikelet client = remote semantic retrieval, without a full download. The main script reports artifact size, records, mount bytes/requests, per-query bytes/requests, cache behavior. `failure-modes.mjs` independently exercises Range supported / ignored / tampered bytes / truncated response. `mcp-proof.mjs` and `llm-proof.mjs` use a real MCP client and a real headless Claude process rather than a mocked model response.
570
496
 
571
- It's the wrong tool when:
497
+ **BEIR evaluation** (`bench/beir/`) keeps stages separate so a quality change can be attributed to the component that caused it, plus a standalone conformance fixture checking the JS benchmark path against the actual C++ affine-u8 representation. Don't read the latency columns as a universal native-performance comparison — they measure the benchmark paths under the stated harness. The quality deltas are the important part.
572
498
 
573
- - you need high-rate online mutation;
574
- - you need a multi-tenant database server;
575
- - you require exact nearest-neighbor search at large scale;
576
- - your deployment can already use a mature native ANN service with no
577
- portability constraints.
499
+ ---
578
500
 
579
501
  ## Status
580
502
 
581
- The engine and the artifact readers/builders are published and usable
582
- today (`pikelet-wasm`, with `pikelet` as the app scaffold).
583
- The complete `.pikelet` profile's spec is still marked draft — Draft 2,
584
- under review, not frozen — but its reader and builder ship in the package,
585
- and it is the clearest expression of where the project is going: a search
586
- engine packaged as one verified, range-readable file.
503
+ Pikelet is early. The implementation is real; the format is not frozen. One primary author. Draft 2 artifact format.
504
+
505
+ - prebuilt WASM included
506
+ - local and HTTP readers
507
+ - affine-u8 and float HNSW backends
508
+ - embedded query encoder
509
+ - BM25 hybrid retrieval
510
+ - content identity and lazy-read integrity verification
511
+ - calibration/abstention support
512
+ - MCP mounting
513
+ - BEIR retrieval evaluation
514
+ - range-read and failure-mode tests
515
+ - a large 456k-record example artifact
516
+
517
+ `npm test` exercises the engine, artifact profiles, MCP, ingestion, format hardening, and related conformance suites.
518
+
519
+ The project was renamed from **Pancake** to **Pikelet** in September 2026; some old fixture filenames and compatibility identifiers still use the previous name. `pancake-wasm` and `create-pancake-search` are deprecated compatibility pointers to the Pikelet packages. Pikelet is unrelated to the pre-existing Pikelet programming language.
520
+
521
+ ---
522
+
523
+ ## The experiment
524
+
525
+ > **For static and slowly changing corpora, useful semantic retrieval can be compiled into the artifact being distributed instead of operated as a separate service.**
526
+
527
+ The implementation currently demonstrates: one file, 456,153 records, 648.5 MiB, embedded query encoder, semantic + lexical retrieval, content identity, per-record integrity, MCP mounting, static HTTP hosting, ~0.5–1.1 MiB fresh-query range traffic after warmup, real multi-record LLM synthesis, no retrieval backend.
528
+
529
+ There are many reasons a database remains the right answer. Pikelet exists for the cases where the knowledge itself should be something you can **build, hash, copy, cache, publish, mount, query, and keep.**
587
530
 
588
531
  ## License
589
532