static-site-search-eval 0.1.0__tar.gz

static_site_search_eval-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,18 @@
+name: ci
+
+on:
+  push:
+    branches: ["main"]
+  pull_request:
+    branches: ["main"]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uv sync
+      - run: uv run pytest
+      - run: uv run ruff check .
+      - run: uv run black --check .

static_site_search_eval-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,19 @@
+name: publish
+
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write        # required for trusted publishing
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uv sync
+      - run: uv run pytest
+      - run: uv build
+      - uses: pypa/gh-action-pypi-publish@release/v1

static_site_search_eval-0.1.0/.gitignore

@@ -0,0 +1,8 @@
+.venv/
+__pycache__/
+.embed-cache/
+build/
+dist/
+node_modules/
+*.egg-info/
+.pytest_cache/

static_site_search_eval-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Bart de Goede
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

static_site_search_eval-0.1.0/PKG-INFO

@@ -0,0 +1,260 @@
+Metadata-Version: 2.4
+Name: static-site-search-eval
+Version: 0.1.0
+Summary: Chunk, embed, and evaluate semantic search over a static site's markdown
+Project-URL: Homepage, https://github.com/bartdegoede/static-site-search-eval
+Project-URL: Issues, https://github.com/bartdegoede/static-site-search-eval/issues
+Author: Bart de Goede
+License-Expression: MIT
+License-File: LICENSE
+Keywords: embeddings,evaluation,model2vec,search,static-site
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Text Processing :: Indexing
+Requires-Python: >=3.12
+Requires-Dist: model2vec>=0.6.0
+Requires-Dist: numpy>=2.1
+Requires-Dist: pyyaml>=6.0
+Description-Content-Type: text/markdown
+
+# static-site-search-eval
+
+`sss-eval` chunks a static site's markdown, embeds the chunks with a static embedding
+model (model2vec/potion), writes browser-readable index artifacts, and evaluates
+retrieval quality against a query set. The point of the package is the claim in its
+name: the whole thing was built to find out whether a static site can ship real
+semantic search without shipping a neural network to the browser. It can.
+
+## Why the model fits in a few megabytes
+
+A model2vec / potion model is not a neural network. It is a lookup table: one
+row of floats per vocabulary token. There is no attention, no matrix multiply
+chain, no ONNX runtime, no WASM binary. Inference is four steps:
+
+```python
+ids = tokenizer.encode(text)          # WordPiece token ids, no [CLS]/[SEP]
+rows = embedding[ids]                 # gather: one row per token id
+pooled = rows.mean(axis=0)            # mean-pool the *unnormalized* rows
+vector = pooled / np.linalg.norm(pooled)  # L2-normalize the pooled result
+```
+
+That's the forward pass. Training (distilling a real sentence-transformer down to
+a static table via PCA) is expensive; using the result is a gather and a mean.
+Because inference has no learned computation graph, the entire "model" a browser
+needs is the token table itself — an `int8` matrix plus a small vocabulary list —
+which is why the winning configuration below downloads 4.21 MB instead of the
+23+ MB an ONNX sentence-transformer needs for the same job.
+
+## Install
+
+```bash
+pip install static-site-search-eval
+# or
+uv add static-site-search-eval
+```
+
+Requires Python 3.12+.
+
+## Building an index
+
+The winning configuration found by the eval in this repository (see
+[Results](#results) below):
+
+```bash
+sss-eval build \
+  --corpus content/post \
+  --outdir static/search \
+  --model minishlab/potion-base-8M \
+  --dims 128 \
+  --chunk-size 600 \
+  --chunk-overlap 120
+```
+
+Title-prefixing (each chunk's embedded text is prefixed with its post title) is
+the default; pass `--no-title-prefix` to disable it. `--cache-root` (default
+`.embed-cache`) controls where content-hash-keyed embeddings are cached between
+runs, so re-running `build` after editing one post only re-embeds that post's
+chunks.
+
+This writes six files to `--outdir`:
+
+- `manifest.json`
+- `chunks.<hash>.json`
+- `docs.<hash>.bin`
+- `tokens.<hash>.bin`
+- `scales.<hash>.bin`
+- `vocab.<hash>.json`
+
+and prints a one-line summary:
+
+```
+313 chunks, 128d, vocab 29528 -> static/search
+```
+
+To confirm the index only points at anchors that actually exist in the rendered
+site (heading anchors are indexed today so that section-level deep links can be
+turned on later without re-embedding, but nothing renders them yet — data that
+nothing reads rots quietly unless something checks it):
+
+```bash
+hugo
+sss-eval verify-anchors --search-dir static/search --public public
+```
+
+## The artifact format
+
+A browser only ever fetches `manifest.json` by a fixed URL; everything else it
+names is content-hashed (`<stem>.<sha256-prefix-12>.<ext>`) and can be cached
+forever. Bump the corpus or the model and the hash changes, so there is no
+versioning scheme to get wrong and no stale-cache class of bug — `manifest.json`
+is the only response that must be served with `no-cache`.
+
+| file | contents |
+|---|---|
+| `manifest.json` | model id, dims, chunk size/overlap, `doc_scale` (127.0), and the filenames of the other five artifacts. The only URL that must never be cached. |
+| `chunks.<hash>.json` | JSON array, one record per row of `docs.bin`, same order: `{post, title, href, snippet, heading, anchor}`. |
+| `docs.<hash>.bin` | raw `int8`, shape `n_chunks × dims`, C order (row-major, row `i` is chunk `i`'s vector). Quantized with a single **global** scale of 127 because document vectors are already L2-normalized. |
+| `tokens.<hash>.bin` | raw `int8`, shape `vocab_size × dims`, C order. This *is* the model — the entire embedding table, quantized. |
+| `scales.<hash>.bin` | `float32`, little-endian, length `vocab_size` — one scale **per row** of `tokens.bin`. Token rows carry the model's zipf/SIF-style downweighting in their magnitude (that's how it downweights "the" without a stopword list), so a single global scale would destroy that signal; each row needs its own. |
+| `vocab.<hash>.json` | JSON array of token strings, indexed by token id — the id `tokens.bin` row `i` corresponds to is the position of that id's string in this array. |
+
+Every binary artifact is a raw `ArrayBuffer`, never base64. Base64 costs +33%
+on the largest file shipped (`tokens.bin`); for a payload whose entire point is
+its download size, that is not a rounding error.
+
+## The three tokenizer traps
+
+Reimplementing potion's tokenizer in JavaScript (rather than shipping a WASM
+tokenizer, which would defeat the purpose) means reproducing three
+non-obvious behaviors of HuggingFace `tokenizers`, each verified against
+`model2vec`'s own `StaticModel.tokenize()`:
+
+1. **No `[CLS]` / `[SEP]`.** `tokenize()` calls
+   `tokenizer.encode_batch_fast(sentences, add_special_tokens=False)`. The
+   `TemplateProcessing` post-processor recorded in `tokenizer.json` describes
+   how special tokens *would* be inserted — it is a decoy left over from the
+   tokenizer's BERT ancestry. `add_special_tokens=False` means it never runs.
+   A browser implementation that adds `[CLS]`/`[SEP]` ids will embed a vector
+   for a token sequence the Python pipeline never produces.
+
+2. **`[UNK]` ids are deleted, not embedded.** After tokenizing, model2vec
+   filters `unk_token_id` out of the id list entirely:
+   `[t for t in token_ids if t != unk_token_id]`. It does not gather the
+   `[UNK]` row and mean-pool it in. A query that is entirely out-of-vocabulary
+   (e.g. a string of emoji, or a language the tokenizer's vocab doesn't cover)
+   tokenizes to an **empty** id list, and mean-pooling zero rows must yield a
+   zero vector rather than throwing or returning the `[UNK]` embedding.
+
+3. **Accents are stripped even though the config says they aren't.**
+   `tokenizer.json`'s normalizer sets `"strip_accents": null`. That reads like
+   "leave accents alone." It doesn't: HuggingFace `tokenizers`' `BertNormalizer`
+   treats a `null` `strip_accents` as "inherit from `lowercase`," and
+   `lowercase` is `true`. So accents ARE stripped — `café` and `cafe` tokenize
+   identically. A browser reimplementation that takes the JSON at face value
+   and skips accent-stripping will silently diverge from the Python tokenizer
+   on any query containing a diacritic.
+
+## Two facts a browser implementer needs
+
+Both measured against this package's own quantization code
+(`src/sss_eval/quantize.py`), not assumed:
+
+1. **The browser never has to dequantize `docs.bin`.** Cosine similarity is
+   invariant to a positive per-row scale, and every row of `docs.bin` was
+   quantized with the *same* global scale (127, applied to unit vectors), so
+   that scale is a constant factor across every document's score — it cannot
+   change the ranking. A `Float32Array` query vector dotted directly against
+   the raw `int8` document rows produces the same ranking as dotting against
+   the dequantized `float32` rows, to within `1e-6`. Query-side, the query
+   vector is built once per search from the float32 token table math above,
+   dequantizing only what's needed (the tokens actually present in the
+   query), never the 313-row-or-larger document matrix.
+
+2. **`int8 @ int8` wraps around silently.** A 300-dimension dot product of two
+   int8 vectors whose true mathematical value is 3,000,000 does not throw, does
+   not produce `NaN`, and does not saturate — it wraps modulo 256 per
+   multiply-accumulate step in a naively-typed accumulator and can return
+   something like `-64`. There is no exception to catch. The fix is
+   structural, not defensive: accumulate the running dot product into a
+   `Float32Array` (or a plain JS number, which is already a float64), never
+   into an `Int8Array` or `Int32Array` sized to the inputs.
+
+## Reproducing the eval
+
+The eval this package shipped is pinned against a frozen corpus snapshot; see
+[`examples/degoe-de/SNAPSHOT.md`](examples/degoe-de/SNAPSHOT.md) for exactly
+which commit of which site it was taken from and why it is not meant to be
+refreshed — the numbers in the [Results](#results) section below and in the
+accompanying blog post are only meaningful against that exact snapshot.
+
+```bash
+uv sync
+pnpm install
+
+# Dump chunk texts (and query texts) to JSON so the Node arms embed exactly
+# what Python did -- same chunker, same text, different runtime.
+uv run python -m sss_eval.dump_chunks \
+  --corpus examples/degoe-de/corpus \
+  --out build/chunks.json \
+  --queries examples/degoe-de/queries.yaml
+
+# Embed those chunks and queries with the non-potion arms, each with the
+# real runtime a browser would use: transformers.js (ONNX q8) for MiniLM,
+# the ternlight WASM engine for its two arms. build_arms (below) looks for
+# these three files next to --outdir, i.e. in --outdir's parent.
+node node/build_minilm.mjs build/chunks.json build/minilm.json
+node node/build_ternlight.mjs build/chunks.json build/ternlight-base.json @ternlight/base
+node node/build_ternlight.mjs build/chunks.json build/ternlight-mini.json @ternlight/mini
+
+# Rank the same 30 queries with the site's real Fuse.js config, for the
+# keyword baseline and the RRF-fused hybrid arms.
+node node/rank_fuse.mjs examples/degoe-de/index.json examples/degoe-de/queries.yaml build/fuse-ranks.json
+
+# Assemble every arm (potion swept over dims/chunking/title-prefix, plus the
+# Node arms above) into the format evaluate.py reads.
+uv run python -m sss_eval.build_arms \
+  --corpus examples/degoe-de/corpus \
+  --queries examples/degoe-de/queries.yaml \
+  --manifest build/arms.json
+
+# Score every arm, semantic-only and RRF-fused with the keyword baseline,
+# against the 30-query eval set, and apply the pre-registered ship rule
+# (smallest download within 0.03 recall@3 of the best arm; MRR breaks ties).
+uv run python -m sss_eval.evaluate \
+  --queries examples/degoe-de/queries.yaml \
+  --arms build/arms.json \
+  --keyword-ranks build/fuse-ranks.json \
+  --json build/results.json
+```
+
+Node is only needed to reproduce the MiniLM/ternlight/keyword arms of the
+eval; it is not a dependency of `sss-eval build` or of using the artifacts in
+a browser.
+
+## Results
+
+Winner: **`potion-base-8M`, PCA-truncated to 128 dims, 600-char chunks with 120
+overlap, title-prefixed**, RRF-fused with a keyword ranking (Fuse.js, already
+shipped with the site).
+
+| | download | recall@1 | recall@3 | MRR@10 |
+|---|---|---|---|---|
+| potion-base-8M (128d, 600/120, title-prefixed) + RRF | 4.21 MB | 0.717 | 0.967 | 0.944 |
+| MiniLM-L6-v2, ONNX q8 | 23.10 MB | 0.700 | — | — |
+
+First-query download of 4,205,974 bytes (4.21 MB) is the `int8` token table
+plus its `float32` per-row scales plus the JSON vocabulary — the three files a
+browser needs before it can embed a single query. The document index itself,
+for the 13-post corpus in `examples/degoe-de/`, is 313 chunks × 128 `int8`
+dims = 40 KB, fetched once and cached alongside the rest of the site's static
+assets.
+
+<!-- TODO: link the blog post once published -->
+
+## License
+
+The code in this repository is MIT licensed (see [`LICENSE`](LICENSE)).
+`examples/degoe-de/corpus/` contains Bart de Goede's blog posts, included as
+evaluation data for the eval above — that content is copyright Bart de Goede
+and is not covered by the MIT license.

static_site_search_eval-0.1.0/README.md

@@ -0,0 +1,240 @@
+# static-site-search-eval
+
+`sss-eval` chunks a static site's markdown, embeds the chunks with a static embedding
+model (model2vec/potion), writes browser-readable index artifacts, and evaluates
+retrieval quality against a query set. The point of the package is the claim in its
+name: the whole thing was built to find out whether a static site can ship real
+semantic search without shipping a neural network to the browser. It can.
+
+## Why the model fits in a few megabytes
+
+A model2vec / potion model is not a neural network. It is a lookup table: one
+row of floats per vocabulary token. There is no attention, no matrix multiply
+chain, no ONNX runtime, no WASM binary. Inference is four steps:
+
+```python
+ids = tokenizer.encode(text)          # WordPiece token ids, no [CLS]/[SEP]
+rows = embedding[ids]                 # gather: one row per token id
+pooled = rows.mean(axis=0)            # mean-pool the *unnormalized* rows
+vector = pooled / np.linalg.norm(pooled)  # L2-normalize the pooled result
+```
+
+That's the forward pass. Training (distilling a real sentence-transformer down to
+a static table via PCA) is expensive; using the result is a gather and a mean.
+Because inference has no learned computation graph, the entire "model" a browser
+needs is the token table itself — an `int8` matrix plus a small vocabulary list —
+which is why the winning configuration below downloads 4.21 MB instead of the
+23+ MB an ONNX sentence-transformer needs for the same job.
+
+## Install
+
+```bash
+pip install static-site-search-eval
+# or
+uv add static-site-search-eval
+```
+
+Requires Python 3.12+.
+
+## Building an index
+
+The winning configuration found by the eval in this repository (see
+[Results](#results) below):
+
+```bash
+sss-eval build \
+  --corpus content/post \
+  --outdir static/search \
+  --model minishlab/potion-base-8M \
+  --dims 128 \
+  --chunk-size 600 \
+  --chunk-overlap 120
+```
+
+Title-prefixing (each chunk's embedded text is prefixed with its post title) is
+the default; pass `--no-title-prefix` to disable it. `--cache-root` (default
+`.embed-cache`) controls where content-hash-keyed embeddings are cached between
+runs, so re-running `build` after editing one post only re-embeds that post's
+chunks.
+
+This writes six files to `--outdir`:
+
+- `manifest.json`
+- `chunks.<hash>.json`
+- `docs.<hash>.bin`
+- `tokens.<hash>.bin`
+- `scales.<hash>.bin`
+- `vocab.<hash>.json`
+
+and prints a one-line summary:
+
+```
+313 chunks, 128d, vocab 29528 -> static/search
+```
+
+To confirm the index only points at anchors that actually exist in the rendered
+site (heading anchors are indexed today so that section-level deep links can be
+turned on later without re-embedding, but nothing renders them yet — data that
+nothing reads rots quietly unless something checks it):
+
+```bash
+hugo
+sss-eval verify-anchors --search-dir static/search --public public
+```
+
+## The artifact format
+
+A browser only ever fetches `manifest.json` by a fixed URL; everything else it
+names is content-hashed (`<stem>.<sha256-prefix-12>.<ext>`) and can be cached
+forever. Bump the corpus or the model and the hash changes, so there is no
+versioning scheme to get wrong and no stale-cache class of bug — `manifest.json`
+is the only response that must be served with `no-cache`.
+
+| file | contents |
+|---|---|
+| `manifest.json` | model id, dims, chunk size/overlap, `doc_scale` (127.0), and the filenames of the other five artifacts. The only URL that must never be cached. |
+| `chunks.<hash>.json` | JSON array, one record per row of `docs.bin`, same order: `{post, title, href, snippet, heading, anchor}`. |
+| `docs.<hash>.bin` | raw `int8`, shape `n_chunks × dims`, C order (row-major, row `i` is chunk `i`'s vector). Quantized with a single **global** scale of 127 because document vectors are already L2-normalized. |
+| `tokens.<hash>.bin` | raw `int8`, shape `vocab_size × dims`, C order. This *is* the model — the entire embedding table, quantized. |
+| `scales.<hash>.bin` | `float32`, little-endian, length `vocab_size` — one scale **per row** of `tokens.bin`. Token rows carry the model's zipf/SIF-style downweighting in their magnitude (that's how it downweights "the" without a stopword list), so a single global scale would destroy that signal; each row needs its own. |
+| `vocab.<hash>.json` | JSON array of token strings, indexed by token id — the id `tokens.bin` row `i` corresponds to is the position of that id's string in this array. |
+
+Every binary artifact is a raw `ArrayBuffer`, never base64. Base64 costs +33%
+on the largest file shipped (`tokens.bin`); for a payload whose entire point is
+its download size, that is not a rounding error.
+
+## The three tokenizer traps
+
+Reimplementing potion's tokenizer in JavaScript (rather than shipping a WASM
+tokenizer, which would defeat the purpose) means reproducing three
+non-obvious behaviors of HuggingFace `tokenizers`, each verified against
+`model2vec`'s own `StaticModel.tokenize()`:
+
+1. **No `[CLS]` / `[SEP]`.** `tokenize()` calls
+   `tokenizer.encode_batch_fast(sentences, add_special_tokens=False)`. The
+   `TemplateProcessing` post-processor recorded in `tokenizer.json` describes
+   how special tokens *would* be inserted — it is a decoy left over from the
+   tokenizer's BERT ancestry. `add_special_tokens=False` means it never runs.
+   A browser implementation that adds `[CLS]`/`[SEP]` ids will embed a vector
+   for a token sequence the Python pipeline never produces.
+
+2. **`[UNK]` ids are deleted, not embedded.** After tokenizing, model2vec
+   filters `unk_token_id` out of the id list entirely:
+   `[t for t in token_ids if t != unk_token_id]`. It does not gather the
+   `[UNK]` row and mean-pool it in. A query that is entirely out-of-vocabulary
+   (e.g. a string of emoji, or a language the tokenizer's vocab doesn't cover)
+   tokenizes to an **empty** id list, and mean-pooling zero rows must yield a
+   zero vector rather than throwing or returning the `[UNK]` embedding.
+
+3. **Accents are stripped even though the config says they aren't.**
+   `tokenizer.json`'s normalizer sets `"strip_accents": null`. That reads like
+   "leave accents alone." It doesn't: HuggingFace `tokenizers`' `BertNormalizer`
+   treats a `null` `strip_accents` as "inherit from `lowercase`," and
+   `lowercase` is `true`. So accents ARE stripped — `café` and `cafe` tokenize
+   identically. A browser reimplementation that takes the JSON at face value
+   and skips accent-stripping will silently diverge from the Python tokenizer
+   on any query containing a diacritic.
+
+## Two facts a browser implementer needs
+
+Both measured against this package's own quantization code
+(`src/sss_eval/quantize.py`), not assumed:
+
+1. **The browser never has to dequantize `docs.bin`.** Cosine similarity is
+   invariant to a positive per-row scale, and every row of `docs.bin` was
+   quantized with the *same* global scale (127, applied to unit vectors), so
+   that scale is a constant factor across every document's score — it cannot
+   change the ranking. A `Float32Array` query vector dotted directly against
+   the raw `int8` document rows produces the same ranking as dotting against
+   the dequantized `float32` rows, to within `1e-6`. Query-side, the query
+   vector is built once per search from the float32 token table math above,
+   dequantizing only what's needed (the tokens actually present in the
+   query), never the 313-row-or-larger document matrix.
+
+2. **`int8 @ int8` wraps around silently.** A 300-dimension dot product of two
+   int8 vectors whose true mathematical value is 3,000,000 does not throw, does
+   not produce `NaN`, and does not saturate — it wraps modulo 256 per
+   multiply-accumulate step in a naively-typed accumulator and can return
+   something like `-64`. There is no exception to catch. The fix is
+   structural, not defensive: accumulate the running dot product into a
+   `Float32Array` (or a plain JS number, which is already a float64), never
+   into an `Int8Array` or `Int32Array` sized to the inputs.
+
+## Reproducing the eval
+
+The eval this package shipped is pinned against a frozen corpus snapshot; see
+[`examples/degoe-de/SNAPSHOT.md`](examples/degoe-de/SNAPSHOT.md) for exactly
+which commit of which site it was taken from and why it is not meant to be
+refreshed — the numbers in the [Results](#results) section below and in the
+accompanying blog post are only meaningful against that exact snapshot.
+
+```bash
+uv sync
+pnpm install
+
+# Dump chunk texts (and query texts) to JSON so the Node arms embed exactly
+# what Python did -- same chunker, same text, different runtime.
+uv run python -m sss_eval.dump_chunks \
+  --corpus examples/degoe-de/corpus \
+  --out build/chunks.json \
+  --queries examples/degoe-de/queries.yaml
+
+# Embed those chunks and queries with the non-potion arms, each with the
+# real runtime a browser would use: transformers.js (ONNX q8) for MiniLM,
+# the ternlight WASM engine for its two arms. build_arms (below) looks for
+# these three files next to --outdir, i.e. in --outdir's parent.
+node node/build_minilm.mjs build/chunks.json build/minilm.json
+node node/build_ternlight.mjs build/chunks.json build/ternlight-base.json @ternlight/base
+node node/build_ternlight.mjs build/chunks.json build/ternlight-mini.json @ternlight/mini
+
+# Rank the same 30 queries with the site's real Fuse.js config, for the
+# keyword baseline and the RRF-fused hybrid arms.
+node node/rank_fuse.mjs examples/degoe-de/index.json examples/degoe-de/queries.yaml build/fuse-ranks.json
+
+# Assemble every arm (potion swept over dims/chunking/title-prefix, plus the
+# Node arms above) into the format evaluate.py reads.
+uv run python -m sss_eval.build_arms \
+  --corpus examples/degoe-de/corpus \
+  --queries examples/degoe-de/queries.yaml \
+  --manifest build/arms.json
+
+# Score every arm, semantic-only and RRF-fused with the keyword baseline,
+# against the 30-query eval set, and apply the pre-registered ship rule
+# (smallest download within 0.03 recall@3 of the best arm; MRR breaks ties).
+uv run python -m sss_eval.evaluate \
+  --queries examples/degoe-de/queries.yaml \
+  --arms build/arms.json \
+  --keyword-ranks build/fuse-ranks.json \
+  --json build/results.json
+```
+
+Node is only needed to reproduce the MiniLM/ternlight/keyword arms of the
+eval; it is not a dependency of `sss-eval build` or of using the artifacts in
+a browser.
+
+## Results
+
+Winner: **`potion-base-8M`, PCA-truncated to 128 dims, 600-char chunks with 120
+overlap, title-prefixed**, RRF-fused with a keyword ranking (Fuse.js, already
+shipped with the site).
+
+| | download | recall@1 | recall@3 | MRR@10 |
+|---|---|---|---|---|
+| potion-base-8M (128d, 600/120, title-prefixed) + RRF | 4.21 MB | 0.717 | 0.967 | 0.944 |
+| MiniLM-L6-v2, ONNX q8 | 23.10 MB | 0.700 | — | — |
+
+First-query download of 4,205,974 bytes (4.21 MB) is the `int8` token table
+plus its `float32` per-row scales plus the JSON vocabulary — the three files a
+browser needs before it can embed a single query. The document index itself,
+for the 13-post corpus in `examples/degoe-de/`, is 313 chunks × 128 `int8`
+dims = 40 KB, fetched once and cached alongside the rest of the site's static
+assets.
+
+<!-- TODO: link the blog post once published -->
+
+## License
+
+The code in this repository is MIT licensed (see [`LICENSE`](LICENSE)).
+`examples/degoe-de/corpus/` contains Bart de Goede's blog posts, included as
+evaluation data for the eval above — that content is copyright Bart de Goede
+and is not covered by the MIT license.

static_site_search_eval-0.1.0/examples/degoe-de/SNAPSHOT.md

@@ -0,0 +1,15 @@
+# Corpus snapshot: bart.degoe.de
+
+- **Source:** https://github.com/bartdegoede/blog — `content/post/`
+- **Commit:** dc70bff724340d88caf2838c39646431099aaf85
+- **Taken:** 2026-07-09
+- **Posts:** 13
+- **Size:** 196 KB
+
+Frozen deliberately. The benchmark in the accompanying blog post is measured against exactly
+these files, and re-running the eval on a newer corpus would produce different numbers than the
+post reports. Do not refresh this snapshot to "keep it current" — that is the opposite of what
+it is for.
+
+The posts are copyright Bart de Goede and are included here as evaluation data. The code in
+this repository is MIT licensed; the posts are not.