pikelet-wasm 0.10.1 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +146 -0
- package/README.md +58 -200
- package/complete/builder.mjs +1 -0
- package/complete/format.mjs +4 -2
- package/complete/index.d.ts +38 -0
- package/complete/index.mjs +235 -116
- package/complete/index.node.mjs +15 -0
- package/complete/layer-reader.mjs +16 -1
- package/complete/node-fetch.mjs +113 -0
- package/complete/rowmap.mjs +62 -0
- package/complete/sources.mjs +49 -6
- package/dist/engine.scalar.wasm +0 -0
- package/dist/engine.wasm +0 -0
- package/package.json +7 -1
package/CHANGELOG.md
CHANGED
|
@@ -8,6 +8,152 @@ Through 0.6.0 these were published as `pancake-wasm` and
|
|
|
8
8
|
packages are pre-1.0: a minor bump may carry breaking changes, and each entry lists them
|
|
9
9
|
first.
|
|
10
10
|
|
|
11
|
+
## pikelet-wasm 0.11.0 / pikelet 0.11.0 — 2026-10-24
|
|
12
|
+
|
|
13
|
+
Pack reads on Node no longer go through the runtime's fetch, whose
|
|
14
|
+
response decoder is vulnerable on unpatched Node releases, and the CLI now
|
|
15
|
+
requires a patched Node 22, 24 or 26. Upgrade if you mount packs over HTTP
|
|
16
|
+
from Node. This release also fixes an out-of-bounds read on crafted engine
|
|
17
|
+
snapshots, a signed-URL refresh that could mix bytes from two artifacts,
|
|
18
|
+
and an MCP server that stopped serving every pack when one failed to
|
|
19
|
+
mount. It adds a writable agent journal, packs compiled from git history
|
|
20
|
+
and past conversations, and sentence windows. Both packages move to 0.11.0
|
|
21
|
+
together, and `pikelet` requires `pikelet-wasm` ^0.11.0.
|
|
22
|
+
|
|
23
|
+
### Compatibility
|
|
24
|
+
|
|
25
|
+
- **`pikelet` requires Node 22.23.4+, 24.22.1+ or 26.12.1+** (`engines`
|
|
26
|
+
was `>=20`). Earlier releases ship a fetch whose response decoder runs
|
|
27
|
+
on any body a server labels with a `Content-Encoding`; 18.x, 20.x, 23.x
|
|
28
|
+
and 25.x are end-of-life and will not get the fix. npm only warns on an
|
|
29
|
+
engines mismatch and an existing install never re-checks it, so the CLI
|
|
30
|
+
now prints a warning to stderr on every start under an unsupported
|
|
31
|
+
version (`PIKELET_NO_NODE_WARNING=1` silences it). CI tests Node 22, 24
|
|
32
|
+
and 26 instead of 18, 20 and 22. `pikelet-wasm`'s own range is unchanged.
|
|
33
|
+
- **Under Node, `pikelet-wasm/complete` resolves to a Node entry** whose
|
|
34
|
+
`httpRangeSource` reads over `nodeFetch` (see Fixed) instead of
|
|
35
|
+
`globalThis.fetch`. Pass `options.fetch` to choose otherwise. Browsers
|
|
36
|
+
and Workers still get the portable module.
|
|
37
|
+
- **The CLI refuses unknown flags** with `Unknown flag --x`. An
|
|
38
|
+
unregistered flag used to be read as taking a value and swallowed the
|
|
39
|
+
next argument (`pikelet mcp --no-resolve-parents pack.pikelet` mounted
|
|
40
|
+
nothing), and a typo was ignored. A script passing a flag pikelet never
|
|
41
|
+
read now exits with an error.
|
|
42
|
+
- **Calibration is fit on the reader's own retrieval.** `compile` and
|
|
43
|
+
`compact` now assemble the pack unscored, open it with the reader, and
|
|
44
|
+
run every calibration probe through the same retrieval `query()` serves
|
|
45
|
+
from (sketch scan, BM25 candidates, exact rerank, fusion, phrase pin),
|
|
46
|
+
then write the fitted pack. Before, calibration searched a separate
|
|
47
|
+
in-memory HNSW index and approximated fusion with 5 BM25 candidates
|
|
48
|
+
instead of 24 and a placeholder distance for lexical hits outside its
|
|
49
|
+
pool. Measured on a 1,608-record docs pack, the passages coverage was fit
|
|
50
|
+
on matched the ones the reader scores for only 27% of calibration
|
|
51
|
+
queries; they now match for all of them. Packs are unaffected until
|
|
52
|
+
rebuilt; a rebuilt pack ranks exactly as before, and on that pack 4 of
|
|
53
|
+
331 calibration queries changed `matchQuality` band, in both directions.
|
|
54
|
+
- **`compact` no longer re-embeds every record to refit calibration**: the
|
|
55
|
+
refit runs on the compacted pack's own rows.
|
|
56
|
+
|
|
57
|
+
### Fixed
|
|
58
|
+
|
|
59
|
+
- **Pack bytes on Node no longer pass through the runtime's response
|
|
60
|
+
decoder.** `pikelet-wasm` ships `nodeFetch`
|
|
61
|
+
(`pikelet-wasm/complete/node-fetch.mjs`), a fetch over
|
|
62
|
+
`node:http`/`node:https` that never decompresses, sends
|
|
63
|
+
`Accept-Encoding: identity`, and refuses any other `Content-Encoding`
|
|
64
|
+
(parsed as a list, any case). It refuses an `https` -> `http` redirect,
|
|
65
|
+
bounds itself to 32 keep-alive sockets per host (the range source's
|
|
66
|
+
default parallelism), and rejects `arrayBuffer()`/`text()` when a body is
|
|
67
|
+
cut off. Under Node, `pikelet-wasm/complete` now resolves to a Node entry
|
|
68
|
+
(export condition `node`; `workerd` and `browser` still get the portable
|
|
69
|
+
module) whose `httpRangeSource` defaults to it and which exports it.
|
|
70
|
+
`httpRangeSource` takes `options.fetch` to choose another transport, and
|
|
71
|
+
exposes it as `source.fetch`; a chain walk opens every parent it reaches
|
|
72
|
+
over HTTP with the head's transport (or `openPikeletChain`'s `fetch`
|
|
73
|
+
option). Every network read the CLI makes uses it: URL packs and chain
|
|
74
|
+
parents under `mcp`, `append` and `compact`, `--shelf`, `doctor`, the
|
|
75
|
+
encoder-weights download, and the URL crawler. `pikelet` therefore needs
|
|
76
|
+
`pikelet-wasm` 0.11.0 or later.
|
|
77
|
+
- **`httpRangeSource` refuses an encoded response on any transport.** A
|
|
78
|
+
200 or 206 with a `Content-Encoding` other than `identity` fails closed,
|
|
79
|
+
next to the existing `Content-Range` checks: a pack is content-addressed
|
|
80
|
+
bytes, and an encoded range cannot be sliced. In browsers the header is
|
|
81
|
+
only visible on same-origin responses or when the host exposes it. Its
|
|
82
|
+
redirect walk also refuses `https` -> `http`.
|
|
83
|
+
- **The URL crawler still reads sites that compress regardless.** A page
|
|
84
|
+
served with `gzip` or `deflate` despite `Accept-Encoding: identity` is
|
|
85
|
+
inflated with `node:zlib` (crawled HTML only; packs stay identity-only).
|
|
86
|
+
A page in any other coding is skipped with a warning naming the URL and
|
|
87
|
+
the header.
|
|
88
|
+
- **Engine snapshots with inconsistent graph levels are refused.** An
|
|
89
|
+
entry point below the snapshot's max level, or a level-l edge into a node
|
|
90
|
+
whose top level is below l, made traversal read past a node's edge lists
|
|
91
|
+
(a segfault in the native build, an out-of-bounds read in WASM). Such a
|
|
92
|
+
snapshot now fails `import()` like any other malformed one. Both `.wasm`
|
|
93
|
+
builds are rebuilt.
|
|
94
|
+
- **A signed URL re-resolved onto a replaced artifact fails.** When a
|
|
95
|
+
signed URL expired and was resolved again, the reader adopted the new
|
|
96
|
+
HEAD's ETag, so a pack re-uploaded under the same name was compared
|
|
97
|
+
against itself and its bytes served alongside the old ones. The first
|
|
98
|
+
ETag and size now stay pinned and a mismatch throws `ARTIFACT_CHANGED`.
|
|
99
|
+
`If-Range` is sent only with a strong ETag.
|
|
100
|
+
- **One pack failing to mount no longer takes down `pikelet mcp`.** Each
|
|
101
|
+
pack mounts on its own; the rest keep serving, `list_packs` lists the
|
|
102
|
+
failures under `unavailable`, and the exit status is still non-zero.
|
|
103
|
+
Before, the first failure made every tool call, `tools/list` included,
|
|
104
|
+
return that error.
|
|
105
|
+
- **`create --force` replaces a project only after the new one builds**,
|
|
106
|
+
and works when the temp directory is on another filesystem (WSL's `/tmp`
|
|
107
|
+
vs `/mnt/c` used to fail with `EXDEV`). It refuses to replace the
|
|
108
|
+
current directory, a parent of it, or a directory overlapping
|
|
109
|
+
`--source`.
|
|
110
|
+
|
|
111
|
+
### Added
|
|
112
|
+
|
|
113
|
+
- **An agent journal** (`pikelet journal init|list|compact`, `mcp
|
|
114
|
+
--journal`): a private pack an agent writes to over MCP with
|
|
115
|
+
`write_note` and reads back with `latest`, `search` and `get_record`.
|
|
116
|
+
Each note is one immutable layer; the chain compacts itself at the depth
|
|
117
|
+
limit, writes are serialized across processes, and several sessions can
|
|
118
|
+
share one journal. Notes are labelled agent-written, and their author is
|
|
119
|
+
the MCP client, never the model's arguments.
|
|
120
|
+
- **Packs from git history and past conversations.** `compile --git
|
|
121
|
+
<repo> [--git-rev <rev|range>]` makes one record per non-merge commit,
|
|
122
|
+
cited by commit URL or sha, with git run without a shell and with the
|
|
123
|
+
repository's program-running config neutralized. `compile
|
|
124
|
+
--conversations <file|folder>` makes one record per exchange from Claude
|
|
125
|
+
Code transcripts, Claude.ai exports and ChatGPT exports, with tool output,
|
|
126
|
+
credential shapes and email addresses redacted (best effort: keep such a
|
|
127
|
+
pack private).
|
|
128
|
+
- **`info().layout` and `recordRange(id)`** report where each part of a
|
|
129
|
+
`.pikelet` file and each record's bytes live, without a read. The
|
|
130
|
+
playground draws its range reads on a map of the file with them.
|
|
131
|
+
- **`appendLayer` and `compactChain` run programmatically**, with a `log`
|
|
132
|
+
option, ready-made `chunks` for `append`, and the written file's
|
|
133
|
+
identity and records returned.
|
|
134
|
+
- **`pikelet` warns on a plain `http://` pack or shelf URL** (loopback
|
|
135
|
+
excepted). The digests still refuse altered bytes, but anyone on the
|
|
136
|
+
path can see what is read and stall or block it.
|
|
137
|
+
- **Sentence windows (`compile --sentence-windows <n>`, opt-in).** The
|
|
138
|
+
index also holds a vector for every run of `n` consecutive sentences of
|
|
139
|
+
each chunk, mapped back to the chunk by a new row-map segment (kind 8,
|
|
140
|
+
`COMPLETE_PROFILE.md` 3.9); results, citations and the corpus stay whole
|
|
141
|
+
chunks. A chunk's single vector averages everything it says, so a
|
|
142
|
+
one-sentence answer could lose to a chunk merely about the same topic.
|
|
143
|
+
On a 3,742-chunk docs corpus with 51 judged questions, `n = 2` raised
|
|
144
|
+
vector-only nDCG@10 from 0.551 to 0.595 (hybrid 0.592 to 0.600), at
|
|
145
|
+
9.4 rows per chunk: file 31.9 to 47.2 MiB, HTTP query 49 to 111 ms.
|
|
146
|
+
Readers without kind-8 support refuse such a pack (row-count check)
|
|
147
|
+
rather than misread it; `append`, `compact` and chain mounts refuse it
|
|
148
|
+
until the layered profile defines rows. `info()` gains `indexRows` and
|
|
149
|
+
`rowMap`.
|
|
150
|
+
- **Reproducible builds.** `compile` honours `SOURCE_DATE_EPOCH` for the
|
|
151
|
+
build time it records in the evaluation segment (and `create`/`rebuild`
|
|
152
|
+
in the project manifest). The build time was the only nondeterministic
|
|
153
|
+
input, and it is part of a pack's identity, so every rebuild of the same
|
|
154
|
+
inputs used to get a new identity; with the variable set, the same
|
|
155
|
+
inputs give byte-identical packs.
|
|
156
|
+
|
|
11
157
|
## pikelet-wasm 0.10.1 / pikelet 0.10.1 — 2026-10-11
|
|
12
158
|
|
|
13
159
|
Layered packs now serve the same grounding and fetch tuning as single
|
package/README.md
CHANGED
|
@@ -2,13 +2,15 @@
|
|
|
2
2
|
|
|
3
3
|
**Knowledge that ships as a file.**
|
|
4
4
|
|
|
5
|
-
Pikelet compiles a corpus into one self-contained,
|
|
5
|
+
Pikelet compiles a corpus into one self-contained `.pikelet` file that can be searched without a vector database, an embedding API, or a retrieval server. Put it on disk, S3, R2, a CDN, or any static HTTP host; a browser, Node process, edge worker, or LLM agent mounts it locally or over HTTP Range and reads only the parts a query needs.
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
The file carries the source text, semantic and keyword indexes, the query encoder, retrieval calibration, evaluation fixtures, and integrity commitments over all of them, and its identity is a hash that commits to all of it. So the file fixes not just what the corpus says but how it is queried: the model that turns a question into a vector, how candidates are ranked, and when the honest answer is "this corpus doesn't say." **Pikelet makes retrieval semantics part of the versioned knowledge artifact.**
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
It is built for corpora that are static or change on a release cycle: documentation, API references, manuals, standards, legal and policy texts, research corpora, and reference packs for agents.
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
## Quick start
|
|
12
|
+
|
|
13
|
+
The `pikelet` CLI requires Node 22.23.4+, 24.22.1+ or 26.12.1+ (earlier releases' fetch decompresses whatever a server labels as encoded; 18.x and 20.x are end-of-life and cannot be fixed); the `pikelet-wasm` library runs on Node 18+, browsers, and Cloudflare Workers.
|
|
12
14
|
|
|
13
15
|
```bash
|
|
14
16
|
npx pikelet compile --source ./docs --out docs.pikelet
|
|
@@ -22,22 +24,17 @@ Compiled docs.pikelet
|
|
|
22
24
|
24.5 MB, 3 records, identity 8d731a...
|
|
23
25
|
```
|
|
24
26
|
|
|
25
|
-
|
|
27
|
+
Every `.pikelet` is at least ~25 MiB, because the query encoder ships inside it: a 5-file folder and a 500-file folder both start around that size. The first run downloads the encoder from a GitHub release and caches it.
|
|
26
28
|
|
|
27
|
-
|
|
29
|
+
Give it to an LLM — Claude Code, Claude Desktop, or any MCP client:
|
|
28
30
|
|
|
29
31
|
```bash
|
|
30
32
|
npx pikelet mcp install --client claude-code --pack ./docs.pikelet
|
|
31
33
|
# or --client claude-desktop; any other MCP client can run
|
|
32
|
-
# `npx pikelet mcp --pack ./docs.pikelet` directly
|
|
33
|
-
```
|
|
34
|
-
|
|
35
|
-
```text
|
|
36
|
-
Wrote MCP server "knowledge-packs" to ./.mcp.json
|
|
37
|
-
Claude Code picks it up on the next session in this project.
|
|
34
|
+
# `npx pikelet mcp --pack ./docs.pikelet` directly
|
|
38
35
|
```
|
|
39
36
|
|
|
40
|
-
|
|
37
|
+
Or query it from code:
|
|
41
38
|
|
|
42
39
|
```js
|
|
43
40
|
import { openPikeletFile } from 'pikelet-wasm/complete';
|
|
@@ -47,99 +44,43 @@ console.log(out.matchQuality, out.results[0]?.title);
|
|
|
47
44
|
// 'strong' 'Snapshot restore'
|
|
48
45
|
```
|
|
49
46
|
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
Under the hood, the file is one container for everything a reader needs:
|
|
53
|
-
|
|
54
|
-
```text
|
|
55
|
-
documents
|
|
56
|
-
│
|
|
57
|
-
▼
|
|
58
|
-
┌──────────────────┐
|
|
59
|
-
│ docs.pikelet │
|
|
60
|
-
│ │
|
|
61
|
-
│ corpus │
|
|
62
|
-
│ semantic index │
|
|
63
|
-
│ lexical index │
|
|
64
|
-
│ query encoder │
|
|
65
|
-
│ integrity │
|
|
66
|
-
│ calibration │
|
|
67
|
-
│ evaluation │
|
|
68
|
-
└──────────────────┘
|
|
69
|
-
│
|
|
70
|
-
├── local file
|
|
71
|
-
└── static HTTP / object storage
|
|
72
|
-
│
|
|
73
|
-
▼
|
|
74
|
-
browser / Node / edge / agent
|
|
75
|
-
```
|
|
76
|
-
|
|
77
|
-
Search is the interface. **The file is the knowledge deployment unit.**
|
|
78
|
-
|
|
79
|
-
---
|
|
47
|
+
`compile` also takes a live URL (`--source https://docs.example.com`), a repository's commit history (`--git ./repo`), or exported chats (`--conversations <file|folder>`). To install instead of using `npx`, run `npm install -g pikelet --omit=optional`; the optional dependency is only for `create`, which builds a deployed search app (a Worker and UI) rather than a file ([why it matters](packages/pikelet/README.md#installing)). [QUICKSTART.md](QUICKSTART.md) walks the whole lifecycle; [`packages/pikelet/README.md`](packages/pikelet/README.md) is the full CLI reference.
|
|
80
48
|
|
|
81
49
|
## The shortest demonstration
|
|
82
50
|
|
|
83
|
-
A real pack holds **456,153 Simple English Wikipedia records** in a **648.5 MiB** file. The [live browser demo](https://pikelet-wiki-playground.pages.dev/) is static HTML on Cloudflare Pages that mounts it straight from an R2 bucket — there is no `/search`, `/embed`,
|
|
84
|
-
|
|
85
|
-
For the agent test, the same file was served by a server whose only relevant capability was `fs.createReadStream` + HTTP Range and mounted by URL into a headless Claude restricted to the pack's three MCP tools (no web, no filesystem). The model answered questions such as "What causes earthquakes?" by searching, reading several records, and citing them.
|
|
51
|
+
A real pack holds **456,153 Simple English Wikipedia records** in a **648.5 MiB** file. The [live browser demo](https://pikelet-wiki-playground.pages.dev/) is static HTML on Cloudflare Pages that mounts it straight from an R2 bucket — there is no `/search`, `/embed`, or `/api` behind it. The browser range-reads the file, verifies it, loads the query encoder from inside it, and shows every byte range it fetches.
|
|
86
52
|
|
|
87
|
-
|
|
88
|
-
| ---------------------------------------- | --------------------------: |
|
|
89
|
-
| Mount (once) | 51.7 MiB in 11 requests |
|
|
90
|
-
| Query encoder (once, with the first query) | ~25 MiB |
|
|
91
|
-
| Each fresh query after that | ~0.5–1.1 MiB |
|
|
92
|
-
| Whole session: 5 queries and a repeat | 80.8 MiB, 12.5% of the file |
|
|
53
|
+
For the agent test, the same file was served by a server whose only relevant capability was `fs.createReadStream` plus HTTP Range, and mounted by URL into a headless Claude restricted to the pack's MCP tools (no web, no filesystem). The model answered questions such as "What causes earthquakes?" by searching, reading several records, and citing them.
|
|
93
54
|
|
|
94
|
-
|
|
55
|
+
| In one persistent session | Transferred |
|
|
56
|
+
| ------------------------------------------ | --------------------------: |
|
|
57
|
+
| Mount (once) | 51.7 MiB in 11 requests |
|
|
58
|
+
| Query encoder (once, with the first query) | ~25 MiB |
|
|
59
|
+
| Each fresh query after that | ~0.5–1.1 MiB |
|
|
60
|
+
| Whole session: 5 queries and a repeat | 80.8 MiB, 12.5% of the file |
|
|
95
61
|
|
|
96
|
-
|
|
62
|
+
The complete artifact is never downloaded. The transcript and per-query table are in [the Wikipedia demonstration](docs/wikipedia-demonstration.md); [`benchmarks/range-proof/`](benchmarks/range-proof/) reproduces it.
|
|
97
63
|
|
|
98
|
-
##
|
|
64
|
+
## Packs for agents
|
|
99
65
|
|
|
100
|
-
`pikelet mcp`
|
|
66
|
+
`pikelet mcp` serves one or more packs over the Model Context Protocol, with four tools: `search`, `get_record`, `list_packs`, and `verify_pack`, which replays the tests stored inside the pack. If one pack fails to mount (an unreachable host, a refused pin), the others still serve and `list_packs` names the missing one and why.
|
|
101
67
|
|
|
102
|
-
|
|
103
|
-
npx pikelet mcp install \
|
|
104
|
-
--client claude-code \
|
|
105
|
-
--pack https://example.com/docs.pikelet#<sha256>
|
|
106
|
-
```
|
|
107
|
-
|
|
108
|
-
The agent gets `search`, `get_record`, `list_packs`, `verify_pack`. A search result carries the identity of the pack and the location of the source record, so an agent can work against `product-docs.pikelet`, `rust-reference.pikelet`, `policy-2026-09.pikelet`, `customer-manual-v4.pikelet` without each publisher operating a retrieval API. The artifact can be local, private, public, behind authenticated object storage, or distributed like any other static asset.
|
|
68
|
+
A pack mounted with a content hash — `https://example.com/docs.pikelet#<sha256>` — has a stable identity, so "what body of knowledge did this agent query?" has a reproducible answer. A citation is `(pack identity, record id)`, and it survives updates: a chain of layers resolves citations made against its earlier versions, and a compacted pack translates them forward or reports that the record was deleted. Citation translation is a library call today (`citation(identity, id)`); the MCP tools do not expose it yet.
|
|
109
69
|
|
|
110
|
-
A
|
|
70
|
+
**A mounted pack's content reaches the model as tool output.** `verify_pack` proves the bytes are intact and match their pinned identity; it does not prove the corpus is trustworthy. Mounting a pack from a source you don't control is the same trust decision as giving an agent any other untrusted-content tool — treat pack text like search results or fetched web pages, not as instructions.
|
|
111
71
|
|
|
112
|
-
**A
|
|
113
|
-
|
|
114
|
-
**Try it with your own questions.** The three Veyra packs from the [ablation](docs/veyra-ablation.md) are published as pinned GitHub release assets. Fetch and verify them first:
|
|
115
|
-
|
|
116
|
-
```bash
|
|
117
|
-
npm run demo:veyra
|
|
118
|
-
```
|
|
119
|
-
|
|
120
|
-
Then mount all three through Pikelet's MCP server:
|
|
121
|
-
|
|
122
|
-
```bash
|
|
123
|
-
claude -p "your question here" \
|
|
124
|
-
--mcp-config examples/one-file-search/web/public/veyra.mcp.json \
|
|
125
|
-
--strict-mcp-config \
|
|
126
|
-
--allowedTools "mcp__veyra-demo__search,mcp__veyra-demo__list_packs,mcp__veyra-demo__get_record"
|
|
127
|
-
```
|
|
72
|
+
**A journal the agent can write (experimental).** Packs are read-only to the model, with one opt-in exception: `--journal <name>` mounts a private pack the agent appends notes to — handoffs, decisions, dead ends, measurements — through `write_note` and `latest`, so a later session, from any vendor, can pick up where the work stands. Notes come back labelled agent-written and unverified. See [the CLI README](packages/pikelet/README.md#a-journal-the-agent-can-write-experimental) and [`docs/agent-journal.md`](docs/agent-journal.md).
|
|
128
73
|
|
|
129
|
-
|
|
74
|
+
**Try it.** The three packs from the [Veyra ablation](docs/veyra-ablation.md) are pinned GitHub release assets. They derive from one small synthetic corpus with controlled differences in a single fact: ask where the Tovash project is housed, and the full pack supports Chamber 17, the modified pack Chamber 43, and the ablated pack no chamber at all.
|
|
130
75
|
|
|
131
76
|
```bash
|
|
132
|
-
|
|
133
|
-
|
|
77
|
+
npm run demo:veyra # fetch and verify the packs
|
|
78
|
+
claude --mcp-config examples/one-file-search/web/public/veyra.mcp.json
|
|
134
79
|
```
|
|
135
80
|
|
|
136
|
-
|
|
81
|
+
## Updating a pack
|
|
137
82
|
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
## Update a pack without rebuilding it
|
|
141
|
-
|
|
142
|
-
A pack doesn't have to be recompiled when a few documents change. `append` never modifies its parent: it writes a new, small `.pikelet` — a **layer** — holding new records, deletions, or replacements, and the base plus its layers (a chain) is searched as one pack. Every published file stays byte-for-byte immutable; the newest layer defines the current state, and its identity pins the whole history.
|
|
83
|
+
A pack doesn't need recompiling when a few documents change. `append` writes a small new `.pikelet` — a **layer** — holding new records, deletions, or replacements, and never touches its parent. The base plus its layers is searched as one pack, and every published file stays byte-for-byte immutable.
|
|
143
84
|
|
|
144
85
|
```bash
|
|
145
86
|
npx pikelet append --parent docs.pikelet --source ./changed-docs --out docs.0001.pikelet
|
|
@@ -147,138 +88,55 @@ npx pikelet append --parent docs.0001.pikelet --remove 412 --out docs.0002.pikel
|
|
|
147
88
|
npx pikelet mcp --pack docs.0002.pikelet
|
|
148
89
|
```
|
|
149
90
|
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
A layer records its parent's identity, and its location when the parent sits in the same directory, so mounting the newest layer finds the rest of the chain — locally or over HTTP — and verifies every link. `#<sha256>` on the newest layer pins the whole chain. Layers always inherit the base's encoder. They inherit its calibration only while the chain stays within the drift limit (by default, records added plus deleted up to 20% of the base's); past it, the chain serves `unscored` until `compact` folds it into one fresh base. `rebase` moves a layer onto a different parent. The format is specified in [`LAYERED_PROFILE.md`](LAYERED_PROFILE.md); `npx pikelet --help` lists every flag.
|
|
91
|
+
On a 20-record test pack with a 24.6 MiB base, deleting a record took a 1.2 KiB layer and replacing one took 6.9 KiB. Mounting the newest layer finds and verifies the rest of the chain, locally or over HTTP, and `#<sha256>` on it pins the whole history. Layers inherit the base's calibration until the chain drifts past 20% of the base's records; then it serves `unscored` until `compact` folds it into a fresh base. The format is specified in [`LAYERED_PROFILE.md`](LAYERED_PROFILE.md).
|
|
153
92
|
|
|
154
|
-
## Match quality
|
|
93
|
+
## Match quality
|
|
155
94
|
|
|
156
|
-
A nearest neighbour is not automatically evidence that a corpus covers a question. At build time Pikelet fits a classifier over retrieval signals
|
|
95
|
+
A nearest neighbour is not automatically evidence that a corpus covers a question. At build time Pikelet fits a classifier over retrieval signals against queries the corpus does not cover, and every result carries `matchQuality: strong | weak | none`. It measures topic relevance: `strong` means the pack covers the question's topic, not that a passage states the fact asked for. For that, each result lists the question's words no top passage contains (`grounding.uncovered`). When calibration can't separate on-topic from off-topic queries, the pack reports `unscored` rather than manufacturing confidence. [How abstention is decided](docs/how-a-query-runs.md#how-abstention-is-decided) covers the details, including how the library and MCP differ in what they return under `none`.
|
|
157
96
|
|
|
158
97
|
`matchQuality` is evidence about retrieval support. It is **not** a guarantee that an LLM will never hallucinate.
|
|
159
98
|
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
## Deeper dives
|
|
163
|
-
|
|
164
|
-
- [The Wikipedia demonstration](docs/wikipedia-demonstration.md) — the full agent transcript, the per-query network table, and what cold processes cost.
|
|
165
|
-
- [How a `.pikelet` query runs](docs/how-a-query-runs.md) — what is inside the file, what a query costs on the wire, remote execution, the integrity checks on the read path, and how abstention is decided.
|
|
166
|
-
- [The Veyra ablation](docs/veyra-ablation.md) — what happens to answers when the supporting record is edited or removed from the pack, with the reproduction commands.
|
|
167
|
-
- [Why make knowledge a file](docs/why-a-file.md) — the reasoning behind the single-artifact design, and what it commits to.
|
|
168
|
-
- [Architecture](docs/architecture.md) — the engine, the C ABI and the JavaScript wrapper; the artifact formats are specified in [`spec/`](spec/).
|
|
169
|
-
|
|
170
|
-
## What this is not
|
|
171
|
-
|
|
172
|
-
**Not a claim that vector databases are obsolete.** If your corpus changes continuously, needs transactional updates, serves many tenants, or already lives comfortably in a database, use a database. Pikelet targets `build → publish → query many times → replace on release`, with small layered updates in between.
|
|
173
|
-
|
|
174
|
-
**Not state-of-the-art embedding research.** The bundled model is MiniLM-L6, chosen for being small enough to live inside the artifact. ArguAna demonstrates a real quality cost from the compact encoder.
|
|
175
|
-
|
|
176
|
-
**Not "LLMs can no longer hallucinate."** The artifact exposes retrieved evidence, provenance, integrity, and an explicit support signal. Whether an agent obeys those signals is an agent behavior question.
|
|
177
|
-
|
|
178
|
-
**Not a new HNSW algorithm.** HNSW is HNSW. HTTP Range is HTTP Range. BM25 is BM25. MiniLM is MiniLM. Affine quantization is not new either. The project is about what becomes possible when those pieces are arranged around one constraint: **the knowledge base itself must be distributable as a file and remain useful without dedicated retrieval infrastructure.**
|
|
99
|
+
## When it fits, and when it doesn't
|
|
179
100
|
|
|
180
|
-
|
|
101
|
+
Pikelet targets `build → publish → query many times → replace on release`, with small layered updates in between. Use it when you want search without operating search infrastructure, search inside a browser, edge worker, or offline tool, a corpus you can hand to someone else, or an agent working against a frozen, versioned body of knowledge.
|
|
181
102
|
|
|
182
|
-
|
|
103
|
+
Use a database instead if your corpus has heavy online writes, needs transactional mutation, serves many tenants, or already lives comfortably in one. The remote scan is linear in row count, so the design targets corpora through the low millions of records, not web scale.
|
|
183
104
|
|
|
184
|
-
**
|
|
105
|
+
It is also not new retrieval research. HNSW is HNSW, BM25 is BM25, HTTP Range is HTTP Range, and the bundled MiniLM-L6 encoder was chosen for being small enough to live inside the file — ArguAna shows a real quality cost from that. The project is about arranging those pieces around one constraint: **the knowledge base itself must be distributable as a file and stay useful without retrieval infrastructure.**
|
|
185
106
|
|
|
186
|
-
|
|
187
|
-
- you want semantic search without operating search infrastructure
|
|
188
|
-
- you want search inside a browser, Node process, edge worker, Electron app, or offline tool
|
|
189
|
-
- you want to distribute a searchable corpus to someone else
|
|
190
|
-
- you want an LLM agent to query a frozen, versioned body of knowledge
|
|
191
|
-
- static/object storage is easier to deploy than a database
|
|
192
|
-
- provenance and reproducibility matter
|
|
193
|
-
- a content-addressed knowledge snapshot is useful
|
|
194
|
-
|
|
195
|
-
Examples: documentation, SDK/API references, technical manuals, research corpora, standards, legal texts, books, product knowledge, code/documentation snapshots, evaluation corpora, agent reference packs.
|
|
196
|
-
|
|
197
|
-
**Don't use it if:**
|
|
198
|
-
|
|
199
|
-
- your corpus has heavy online writes
|
|
200
|
-
- you need transactional mutation
|
|
201
|
-
- you need a multi-tenant authoritative search service
|
|
202
|
-
- you need exact nearest-neighbour search at very large scale
|
|
203
|
-
- your corpus is so large that a linear resident sketch scan is inappropriate
|
|
204
|
-
- you already operate a vector database happily and portability buys you nothing
|
|
205
|
-
|
|
206
|
-
The resident remote scan is linear in row count. The current architecture targets corpora through the low millions of records, not arbitrary web scale.
|
|
107
|
+
## Deeper dives
|
|
207
108
|
|
|
208
|
-
|
|
109
|
+
- [How a `.pikelet` query runs](docs/how-a-query-runs.md) — what is inside the file, what a query costs on the wire, integrity checks on the read path, and abstention.
|
|
110
|
+
- [The Wikipedia demonstration](docs/wikipedia-demonstration.md) — the agent transcript, the network table, and what cold processes cost.
|
|
111
|
+
- [The Veyra ablation](docs/veyra-ablation.md) — what happens to answers when the supporting record is edited or removed.
|
|
112
|
+
- [Why make knowledge a file](docs/why-a-file.md) — the reasoning behind the single-artifact design.
|
|
113
|
+
- [Architecture](docs/architecture.md) — the engine, the C ABI, and the JavaScript wrapper; byte-level formats are in [`spec/`](spec/).
|
|
209
114
|
|
|
210
115
|
## Repository map
|
|
211
116
|
|
|
212
117
|
```text
|
|
213
|
-
packages/pikelet-wasm/
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
complete/ Reader and builder for the complete
|
|
225
|
-
range-readable .pikelet artifact, and the
|
|
226
|
-
layered-chain reader and producers
|
|
227
|
-
native/ N-API build of the same engine (benchmarks)
|
|
228
|
-
packages/pikelet/ The published pikelet CLI: compiler, MCP
|
|
229
|
-
server, encoder integration, Docusaurus plugin
|
|
230
|
-
docs/ Deeper dives (how a query runs, the Veyra
|
|
231
|
-
ablation, why a file), architecture notes,
|
|
232
|
-
rename history, measurement reports
|
|
233
|
-
spec/ Byte-level artifact contracts
|
|
234
|
-
LAYERED_PROFILE.md The layered profile: append, compact,
|
|
235
|
-
rebase, and how chains are read
|
|
236
|
-
benchmarks/beir/ Frozen BEIR ablation harness (encoder,
|
|
237
|
-
quantization, HNSW quality)
|
|
238
|
-
benchmarks/range-proof/ The deliberately boring static-HTTP proof:
|
|
239
|
-
dumb-server.mjs, proof.mjs, mcp-proof.mjs,
|
|
240
|
-
llm-proof.mjs, failure-modes.mjs
|
|
241
|
-
examples/one-file-search/ Large single-artifact search and embedded
|
|
242
|
-
encoder work
|
|
243
|
-
examples/mcp-knowledge-pack/ Compile, mount, search, and hydrate
|
|
244
|
-
records through MCP
|
|
245
|
-
local-packs/ Prebuilt example .pikelet artifacts used
|
|
246
|
-
by the demo commands above
|
|
247
|
-
packs/ Pack hosting and distribution examples
|
|
118
|
+
packages/pikelet-wasm/ The pikelet-wasm library: C++ HNSW engine compiled to WASM
|
|
119
|
+
(used at compile time), readers and builders for .pikelet
|
|
120
|
+
packages/pikelet/ The pikelet CLI: compiler, MCP server, Docusaurus plugin
|
|
121
|
+
docs/ Deeper dives, design notes, measurement reports
|
|
122
|
+
spec/ Byte-level artifact contracts
|
|
123
|
+
LAYERED_PROFILE.md The layered profile: append, compact, rebase
|
|
124
|
+
benchmarks/beir/ BEIR ablation harness (encoder, quantization, HNSW quality)
|
|
125
|
+
benchmarks/range-proof/ Static-HTTP proof against a server that knows nothing of Pikelet
|
|
126
|
+
examples/ Runnable examples: MCP packs, one-file search, static wiki pack
|
|
127
|
+
packs/ Pack hosting and distribution examples
|
|
128
|
+
test/ Conformance suites (`npm test`)
|
|
248
129
|
```
|
|
249
130
|
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
## Reproduce the proofs
|
|
131
|
+
## Reproducing the proofs
|
|
253
132
|
|
|
254
|
-
**
|
|
255
|
-
|
|
256
|
-
**BEIR evaluation** (`benchmarks/beir/`) keeps stages separate so a quality change can be attributed to the component that caused it, plus a standalone conformance fixture checking the JS benchmark path against the actual C++ affine-u8 representation. Don't read the latency columns as a universal native-performance comparison — they measure the benchmark paths under the stated harness. The quality deltas are the important part.
|
|
257
|
-
|
|
258
|
-
---
|
|
133
|
+
The **range proof** uses a server that does not understand Pikelet and reports mount and per-query bytes and requests; `failure-modes.mjs` exercises Range ignored, tampered bytes, and truncated responses, and `mcp-proof.mjs` and `llm-proof.mjs` use a real MCP client and a real headless Claude. The **BEIR evaluation** keeps encoder, quantization, and index stages separate so a quality change can be attributed to the component that caused it. Read its quality deltas, not its latency columns, which measure the benchmark paths under that harness.
|
|
259
134
|
|
|
260
135
|
## Status
|
|
261
136
|
|
|
262
|
-
Pikelet is early
|
|
263
|
-
|
|
264
|
-
- prebuilt WASM included
|
|
265
|
-
- local and HTTP readers
|
|
266
|
-
- affine-u8 and float HNSW backends for compile-time indexing/quantization
|
|
267
|
-
- embedded query encoder
|
|
268
|
-
- BM25 hybrid retrieval
|
|
269
|
-
- content identity and lazy-read integrity verification
|
|
270
|
-
- calibration/abstention support
|
|
271
|
-
- MCP mounting
|
|
272
|
-
- layered updates: append, compact, rebase
|
|
273
|
-
- BEIR retrieval evaluation
|
|
274
|
-
- range-read and failure-mode tests
|
|
275
|
-
- a large 456k-record example artifact
|
|
276
|
-
|
|
277
|
-
`npm test` exercises the engine, artifact profiles, MCP, ingestion, format hardening, and related conformance suites.
|
|
278
|
-
|
|
279
|
-
Previously known as **Pancake** (renamed September 2026); artifacts, profile strings and files from before the rename remain readable — see [`docs/history.md`](docs/history.md) for what kept the old name and why. Pikelet is unrelated to the pre-existing Pikelet programming language.
|
|
137
|
+
Pikelet is early: the implementation is real, the format is not frozen (Draft 2), and there is one primary author. Prebuilt WASM, local and HTTP readers, the embedded encoder, BM25 hybrid retrieval, content identity with lazy-read verification, calibrated abstention, MCP mounting, layered updates, and the experimental journal all ship today. `npm test` runs the engine, artifact-profile, MCP, ingestion, and format-hardening conformance suites.
|
|
280
138
|
|
|
281
|
-
|
|
139
|
+
Previously known as **Pancake** (renamed September 2026); files from before the rename remain readable — see [`docs/history.md`](docs/history.md). Pikelet is unrelated to the Pikelet programming language.
|
|
282
140
|
|
|
283
141
|
## License
|
|
284
142
|
|
package/complete/builder.mjs
CHANGED
|
@@ -2,6 +2,7 @@ import crypto from 'node:crypto';
|
|
|
2
2
|
import fs from 'node:fs';
|
|
3
3
|
import { expectedBlobBytes } from './inline-transformer.mjs';
|
|
4
4
|
import { MAGIC, HEADER_BYTES, TABLE_ENTRY_BYTES, KINDS, KIND_NAMES } from './format.mjs';
|
|
5
|
+
export { buildRowMapSegment } from './rowmap.mjs';
|
|
5
6
|
|
|
6
7
|
export { MAGIC, HEADER_BYTES, TABLE_ENTRY_BYTES, KINDS, KIND_NAMES };
|
|
7
8
|
|
package/complete/format.mjs
CHANGED
|
@@ -8,6 +8,8 @@ export const TABLE_ENTRY_BYTES = 48;
|
|
|
8
8
|
// layer carries exactly one tombstones segment, and a base compacted from a
|
|
9
9
|
// chain carries at most one lineage segment. They are registered here so the
|
|
10
10
|
// shared table has one definition; a complete-profile artifact never has
|
|
11
|
-
// either, and the required-segment tables stay per-profile.
|
|
12
|
-
|
|
11
|
+
// either, and the required-segment tables stay per-profile. Kind 8, the row
|
|
12
|
+
// map, is OPTIONAL in the complete profile (COMPLETE_PROFILE.md 3.9): present
|
|
13
|
+
// when the index holds several vectors per record.
|
|
14
|
+
export const KINDS = { index: 1, corpus: 2, 'query-interp': 3, evaluation: 4, lexical: 5, tombstones: 6, lineage: 7, rowmap: 8 };
|
|
13
15
|
export const KIND_NAMES = Object.fromEntries(Object.entries(KINDS).map(([name, kind]) => [kind, name]));
|
package/complete/index.d.ts
CHANGED
|
@@ -69,8 +69,33 @@ export declare function httpRangeSource(url: string, options?: {
|
|
|
69
69
|
* the child's origin and directory (LAYERED_PROFILE.md 5.1.1).
|
|
70
70
|
*/
|
|
71
71
|
redirectGuard?: (target: string) => void;
|
|
72
|
+
/**
|
|
73
|
+
* The fetch used for every request. Default nodeFetch under Node
|
|
74
|
+
* (the package's node export condition), else globalThis.fetch. A runtime
|
|
75
|
+
* whose fetch transparently decodes response bodies can pass one that does
|
|
76
|
+
* not.
|
|
77
|
+
*/
|
|
78
|
+
fetch?: (input: string, init?: {
|
|
79
|
+
method?: string;
|
|
80
|
+
headers?: Record<string, string>;
|
|
81
|
+
redirect?: 'follow' | 'manual' | 'error';
|
|
82
|
+
}) => Promise<any>;
|
|
72
83
|
}): HttpRangeSource;
|
|
73
84
|
|
|
85
|
+
/**
|
|
86
|
+
* Node only (the 'node' export condition): a fetch over node:http/node:https
|
|
87
|
+
* that never decompresses, sends Accept-Encoding: identity and refuses any
|
|
88
|
+
* other Content-Encoding. httpRangeSource defaults to it under Node.
|
|
89
|
+
*/
|
|
90
|
+
export declare function nodeFetch(input: string, init?: {
|
|
91
|
+
method?: 'GET' | 'HEAD';
|
|
92
|
+
headers?: Record<string, string>;
|
|
93
|
+
redirect?: 'follow' | 'manual' | 'error';
|
|
94
|
+
signal?: unknown;
|
|
95
|
+
/** Inflate a gzip/deflate body a server sent despite identity (crawled HTML); Brotli/zstd still refused. */
|
|
96
|
+
inflateGzip?: boolean;
|
|
97
|
+
}): Promise<any>;
|
|
98
|
+
|
|
74
99
|
export interface CompleteQueryResult {
|
|
75
100
|
/**
|
|
76
101
|
* The pack's calibrated verdict on this retrieval. On packs calibrated
|
|
@@ -126,6 +151,10 @@ export interface CompleteSearch {
|
|
|
126
151
|
*/
|
|
127
152
|
license: string | null;
|
|
128
153
|
records: number;
|
|
154
|
+
/** Rows in the index: equal to records unless a row map groups several rows per record. */
|
|
155
|
+
indexRows: number;
|
|
156
|
+
/** The policy behind a multi-row index (COMPLETE_PROFILE.md 3.9), e.g. sentence windows; null otherwise. */
|
|
157
|
+
rowMap: { kind: string; sentences: number; rows: number } | null;
|
|
129
158
|
dim: number;
|
|
130
159
|
metric: string | number;
|
|
131
160
|
encoder: Record<string, unknown>;
|
|
@@ -177,6 +206,13 @@ export interface CompleteSearch {
|
|
|
177
206
|
*/
|
|
178
207
|
residentScan: 'engine' | 'js';
|
|
179
208
|
sampleQueries: string[];
|
|
209
|
+
/**
|
|
210
|
+
* The file's parts in file order: 'header' (fixed header, manifest and
|
|
211
|
+
* segment table) followed by each segment of a known kind ('index',
|
|
212
|
+
* 'corpus', 'lexical', 'query-interp', 'evaluation', ...). Absolute
|
|
213
|
+
* byte offsets.
|
|
214
|
+
*/
|
|
215
|
+
layout: Array<{ kind: string; offset: number; length: number }>;
|
|
180
216
|
};
|
|
181
217
|
/**
|
|
182
218
|
* Verify the index's lazy vector rows against the identity-anchored
|
|
@@ -212,6 +248,8 @@ export interface CompleteSearch {
|
|
|
212
248
|
}): Promise<CompleteQueryResult>;
|
|
213
249
|
/** Hydrate one corpus record by id (verified per record on format 2). */
|
|
214
250
|
record(id: number): Promise<Record<string, unknown>>;
|
|
251
|
+
/** The absolute byte range one corpus record's stored bytes occupy in the file. */
|
|
252
|
+
recordRange(id: number): { offset: number; length: number };
|
|
215
253
|
/** The evaluation segment's JSON object, or null when the artifact carries none. */
|
|
216
254
|
evaluation(): Promise<Record<string, unknown> | null>;
|
|
217
255
|
close(): Promise<void>;
|