sweet-search 2.5.5 → 2.5.7

package/README.md

@@ -2,7 +2,7 @@
 
 <img src="assets/sweet-search-banner-pixelated.svg" alt="sweet-search" width="100%" />
 
-### *Maybe grep isn't all you need…*
+### *Maybe grep isn't all you need…* 🍬
 
 **A local-first hybrid code-search engine built for AI coding agents.**
 Semantic + lexical + structural search over your working tree, GPU-accelerated local inference,
@@ -82,10 +82,7 @@ updates itself as you type.
 <sub>reconcile daemon tracks your working tree</sub>
 
 [🦀 The Native Engine Room](#-the-native-engine-room)<br>
-<sub>four Rust crates + TurboQuant compression</sub>
-
-[🎯 The Ranking Stack](#-the-ranking-stack)<br>
-<sub>route → retrieve → fuse → rerank → expand</sub>
+<sub>four Rust crates + INT4 LI compression</sub>
 
 </td>
 <td width="24%" valign="top">
@@ -93,7 +90,7 @@ updates itself as you type.
 **THE RECEIPTS**
 
 [📊 Benchmarks](#-benchmarks)<br>
-<sub>full-corpus MRR, no distractor shortcuts</sub>
+<sub>agent cost savings · engine speed · full-corpus MRR</sub>
 
 [🙏 Prior Art & Acknowledgements](#-prior-art--acknowledgements)<br>
 <sub>the shoulders we stand on</sub>
@@ -143,6 +140,99 @@ sweet-search uninstall              # clean removal: models, caches, config —
 
 ## 📊 Benchmarks
 
+We measure sweet-search four ways — from how much it helps a real agent down to raw engine throughput:
+
+<table>
+<tr>
+<td width="50%" valign="top">
+
+🤖 **[① Code-retrieval](#bench-code-retrieval)** *(agent-in-the-loop)*<br>
+<sub>Does it make a real coding agent **cheaper and more useful** when it searches your repo? Paired against each model's own grep-and-read loop.</sub>
+
+</td>
+<td width="50%" valign="top">
+
+🚧 **[② Task-completion](#bench-task-completion)** *(coming soon)*<br>
+<sub>Does cheaper, denser context **compound** into a higher resolve-rate on multi-step engineering tasks? Harness in progress.</sub>
+
+</td>
+</tr>
+<tr>
+<td width="50%" valign="top">
+
+📄 **[③ Paper-type IR](#bench-paper-type)** *(academic)*<br>
+<sub>The standard NL→code retrieval suites (GCSN, M2CRB, CoSQA…), full-corpus MRR@10.</sub>
+
+</td>
+<td width="50%" valign="top">
+
+⚡ **[④ Engine speed](#bench-engine-speed)**<br>
+<sub>Raw systems numbers — grep throughput, query latency, rerank kernels, HNSW.</sub>
+
+</td>
+</tr>
+</table>
+
+---
+
+<a id="bench-code-retrieval"></a>
+### 🤖 1. Code-retrieval benchmarks — *the agent-in-the-loop test*
+
+We install the evolved agent prompt (the [GEPA-evolved search discipline](#-an-agent-prompt-that-was-evolved-not-written)), point a coding agent at a real repo, and pair it **probe-for-probe against the same model running its own native grep-and-read loop**. Same model, same tasks, same judge — the only difference is whether sweet-search is wired in.
+
+<div align="center">
+
+<img src="assets/code-retrieval-stats.svg" alt="up to 34% lower cost on Codex · up to 56% fewer tool calls · 1.5–2× more useful context per response · +3pp accuracy on weak models" width="100%" />
+
+<sub>top-of-range figures · full per-harness ranges in the dropdown · 11 model×harness cells, paired, multiplicity-controlled</sub>
+
+</div>
+
+**The headline, in four claims:**
+
+- 💰 **Cheaper where the agent thrashes** — up to **−34%** realized cost on Codex; **−18 to −32%** across the GPT-5.5 / opencode / bare-API harnesses.
+- 🔧 **Fewer round-trips** — up to **−56%** tool calls, significant on **9 of 11** cells.
+- ✨ **More useful per response** — **+0.18 to +0.31** on a 5-dimension usefulness score, and *still* denser when length-matched (significant on **8 of 11** cells).
+- 🎯 **Accuracy held — and lifted on the weak** — a statistical tie on flagship models (saturated at 0.94–0.99), and **+3 pp** (up to **+8 pp** out-of-distribution) on weaker models like GLM-5.1 and DeepSeek.
+
+<details>
+<summary><b>📋 Full per-harness results & how it's measured</b></summary>
+
+The win is **harness-adaptive**: where the native loop is disciplined (Claude Code) it shows up as *denser, more useful context per token*; where it thrashes (Codex floods 30k+ tokens of its own grep output into context) it shows up as a *large cost and tool-call cut*. Either way, **final-answer accuracy never significantly regresses**.
+
+| 🧰 Native agent harness | 💰 Realized cost | 🔧 Tool calls | ✨ Useful content / response | 🎯 Final accuracy |
+|---|---:|---:|---:|:--|
+| 🤖 **Codex** (GPT-5.5) | **−30 to −34%** | **−44 to −56%** | +0.06 → +0.17 ↑ | tie *(saturated)* |
+| 🐚 **opencode** (GPT-5.5 / GLM-5.1) | **−18 to −22%** | −15 to −49% | **+0.23 to +0.31** ↑ | tie |
+| 🔌 **bare API** (GPT-5.5 / GLM / DeepSeek) | −15 to −32% ᵃ | −15 to −33% | +0.08 to +0.24 ↑ | tie · **+3 pp on weak models** |
+| 🟣 **Claude Code** (Sonnet / Opus) | −10% to +14% ᵇ | −5 to −33% | +0.18 to +0.29 ↑ | tie |
+
+<sub>↑ "Useful content / response" is the per-response delta on a 5-dimension usefulness score (answer-grounding · workable-code · navigability · edit-locality · sufficiency), 0–1 scale. "tie" = final-answer correctness statistically indistinguishable (saturated in the 0.94–0.99 band on flagships).<br>ᵃ the two cheapest bare models cost fractions of a cent either way (GLM +27% of $0.008; DeepSeek −15% of $0.004). ᵇ Opus −5/−10%; Sonnet +8–14%, which is ≈1¢ on a flat-rate subscription for a richer answer.</sub>
+
+**Denser, not just longer.** The usefulness lift survives **length-matching** — comparing sweet-search and native responses of *equal token length*, sweet-search's content is significantly higher on **8 of 11** cells. The validated single-number usefulness composite (grounding × content × density) is significant on **all 11** sealed cells.
+
+- **What's being compared:** the installed `sweet-search` agent prompt + tools vs. the *same model* using only its built-in file-reading and shell-grep tools. Not a different model — the same model, with and without sweet-search.
+- **Design:** 11 model×harness cells. **Sealed vault** (n=60/arm, the pre-registered primary) opened once; plus **held-out** (n=30) and **out-of-distribution** (n=40) sets for generalization. Stratified, fixed-seed splits.
+- **Judging:** 3-judge panel (DeepSeek-V4-flash + Gemini-3.1-flash-lite + MiniMax-M2.7), paired by probe, 20k-sample bootstrap CIs, **Benjamini–Hochberg FDR** multiplicity correction across each metric family. We report family-level survival counts, never a single cherry-picked cell.
+- **What survives FDR (vault):** useful-content **10/11**, density-composite **11/11**, length-matched content **8/11**, fewer-tool-calls **9/11**. Generalization (held-out + OOD): content **17–18/20**, fewer calls **14/20**.
+- **The token fact that drives everything:** sweet-search's footprint is nearly constant (~1.3k–3.3k tokens) because the tool responses are capped; native's footprint is whatever the model decides to grep — up to **37k tokens** on Codex. That single fact is what drives the cost and tool-call gaps.
+- **Honest caveats we keep attached:** (1) accuracy **ties** on flagship models — it is *not* an accuracy win there, it's saturated; the accuracy gains are real only on weaker models. (2) The two weakest cells for *length-matched* density (Codex-low, DeepSeek) are correct-sign but underpowered — Codex's responses are so token-divergent that too few equal-length pairs exist to reach significance, and DeepSeek is simply under-powered. Those are honest non-victories, not wins.
+- Full methodology and per-cell tables: [`docs/PHASE7.md`](docs/PHASE7.md).
+
+</details>
+
+---
+
+<a id="bench-task-completion"></a>
+### 🚧 2. Task-completion benchmarks — *coming soon*
+
+> Retrieval quality is necessary but not sufficient. Cheaper, denser context only matters if it **compounds across a real, multi-step engineering task** — finding the code, understanding it, changing it, and not breaking anything. The next suite measures exactly that: **resolve-rate on SWE-bench-style multi-file tasks**, sweet-search-wired vs. native, on the same paired, multiplicity-controlled bar as above. Harness and pilot are in progress — numbers land here when they clear that bar, and not before.
+
+---
+
+<a id="bench-paper-type"></a>
+### 📄 3. Paper-type retrieval benchmarks — *academic NL→code IR*
+
 > [!WARNING]
 > ⚠️ **THESE NUMBERS ARE STALE — TREAT THEM AS A FLOOR, NOT THE CURRENT SCORE.** ⚠️
 > Several results below were measured on builds that predate major accuracy work
@@ -153,15 +243,15 @@ sweet-search uninstall              # clean removal: models, caches, config —
 Every number below is the **`ss-search` pipeline end-to-end** — the same binary you install, querying
 against the **full corpus** (no 99-distractor shortcuts), measured at 26–41 ms p50 on an M3 Max.
 
-| Benchmark | What it tests | Queries | MRR@10 |
+| 📚 Benchmark | 🔍 What it tests | # Queries | 🎯 MRR@10 |
 |-----------|---------------|--------:|-------:|
-| **GenCodeSearchNet** | NL→code, 6 languages | 6,000 | **86.6** |
-| **M2CRB** | multilingual NL→code (ES/PT/DE/FR → Py/Java/JS) | 2,814 | **60.2** |
-| CoSQA (test split) | web queries → Python | 500 | 97.0 |
-| CoSQA+ | web queries → Python, multi-match | 20,604 | 72.1 |
-| CLARC | NL→C/C++ (systems code) | 1,245 | 67.4 |
-| AdvTest † | adversarially renamed Python | 1,000 | 91.5 |
-| CoIR † | 10 datasets, 14 languages | 4,500 | 57.3 |
+| 🌐 **GenCodeSearchNet** | NL→code, 6 languages | 6,000 | **86.6** |
+| 🗺️ **M2CRB** | multilingual NL→code (ES/PT/DE/FR → Py/Java/JS) | 2,814 | **60.2** |
+| 🐍 CoSQA (test split) | web queries → Python | 500 | 97.0 |
+| 🐍 CoSQA+ | web queries → Python, multi-match | 20,604 | 72.1 |
+| ⚙️ CLARC | NL→C/C++ (systems code) | 1,245 | 67.4 |
+| 🛡️ AdvTest † | adversarially renamed Python | 1,000 | 91.5 |
+| 🌍 CoIR † | 10 datasets, 14 languages | 4,500 | 57.3 |
 
 **GenCodeSearchNet: the strongest result published anywhere, as far as we can tell.** The benchmark's
 own paper tops out at MRR ≤ 0.42 for its fine-tuned baselines (and ≤ 0.10 on the cross-lingual subsets),
@@ -174,7 +264,7 @@ pools). sweet-search reaches **60.2 full-corpus MRR@10 out of the box**, on Span
 and French queries.
 
 <details>
-<summary><b>Methodology, staleness flags & systems numbers</b></summary>
+<summary><b>Methodology & staleness flags</b></summary>
 
 - **Reproduction:** result artifacts live in `eval/results/`; rerun via `eval/run_all.js`.
 - **Protocol note:** published baselines for GCSN and CoSQA-style benchmarks typically rank the gold snippet against 99 sampled distractors. All sweet-search numbers rank against the full benchmark corpus — strictly harder.
@@ -182,18 +272,27 @@ and French queries.
 - **Honesty corner:** CrossCodeEval — cross-file *completion context* retrieval, a different task than NL search — sits at 0.12. We don't optimize for it and report it anyway.
 - Dates and per-language breakdowns: [`docs/BENCHMARKS_EXPLAINED.md`](docs/BENCHMARKS_EXPLAINED.md).
 
-Systems performance, measured in-repo:
+</details>
 
-| What | Result | Source |
-|------|--------|--------|
-| Indexed grep vs ripgrep | **10.2× faster** at the median (8.5–17.7× across 5 repos, 353 realistic queries, 1 ms p50 — identical match counts on every query) | [`docs/GREP_INDEXING_STRATEGY.md`](docs/GREP_INDEXING_STRATEGY.md) |
-| Warm query latency (native CLI) | **2.9 ms** warm · 108 ms cold | [`docs/INIT_STRATEGY.md`](docs/INIT_STRATEGY.md) |
-| MaxSim rerank kernels | **1.26 s → 27 ms** for a 231-candidate pass (47× native Rust; 16× WASM SIMD) | [`docs/MAXSIM_OPTIMIZATION.md`](docs/MAXSIM_OPTIMIZATION.md) |
-| HNSW tuning for code | **−33%** search p50, **+5.9 pp** recall@200 | [`docs/HNSW_APPROACH.md`](docs/HNSW_APPROACH.md) |
-| Indexing memory | peak JS heap **785 MB → 213 MB** | [`docs/DISK_FLUSHING_STRATEGY.md`](docs/DISK_FLUSHING_STRATEGY.md) |
-| CoreML cascade (M3 Max) | **18% faster** full indexing vs the Metal baseline | [`docs/INIT_STRATEGY.md`](docs/INIT_STRATEGY.md) |
+---
 
-</details>
+<a id="bench-engine-speed"></a>
+### ⚡ 4. Engine speed — *systems benchmarks, measured in-repo*
+
+<div align="center">
+
+**10.2×** ripgrep's median grep &nbsp;·&nbsp; **2.9 ms** warm queries &nbsp;·&nbsp; **47×** MaxSim kernels &nbsp;·&nbsp; **−33%** HNSW search p50
+
+</div>
+
+| ⚙️ What | 📈 Result | 📄 Source |
+|------|--------|--------|
+| ⚡ Indexed grep vs ripgrep | **10.2× faster** at the median (8.5–17.7× across 5 repos, 353 realistic queries, 1 ms p50 — identical match counts on every query) | [`docs/GREP_INDEXING_STRATEGY.md`](docs/GREP_INDEXING_STRATEGY.md) |
+| ⏱️ Warm query latency (native CLI) | **2.9 ms** warm · 108 ms cold | [`docs/INIT_STRATEGY.md`](docs/INIT_STRATEGY.md) |
+| 🧮 MaxSim rerank kernels | **1.26 s → 27 ms** for a 231-candidate pass (47× native Rust; 16× WASM SIMD) | [`docs/MAXSIM_OPTIMIZATION.md`](docs/MAXSIM_OPTIMIZATION.md) |
+| 🧠 HNSW tuning for code | **−33%** search p50, **+5.9 pp** recall@200 | [`docs/HNSW_APPROACH.md`](docs/HNSW_APPROACH.md) |
+| 💾 Indexing memory | peak JS heap **785 MB → 213 MB** | [`docs/DISK_FLUSHING_STRATEGY.md`](docs/DISK_FLUSHING_STRATEGY.md) |
+| 🍏 CoreML cascade (M3 Max) | **18% faster** full indexing vs the Metal baseline | [`docs/INIT_STRATEGY.md`](docs/INIT_STRATEGY.md) |
 
 ## 🧰 The Six Tools
 
@@ -202,43 +301,115 @@ to be *consumed by an agent* — a useful answer, not a wall of matches to scrol
 
 | Tool | What you give it | What you get back |
 |------|------------------|-------------------|
-| `ss-search` | a natural-language query | ranked, **self-contained code blocks** |
-| `ss-grep` | an exact regex/literal | `file:line` hits, **ranked** |
-| `ss-find` | a regex **+** a query | regex matches, **semantically re-ranked, as code blocks** |
-| `ss-semantic` | a file **+** a question | just the **relevant spans** of that file |
-| `ss-trace` | a symbol | **callers + callees + impact**, in one call |
-| `ss-read` | a file (± line range) | exact bytes **+ symbol metadata** |
+| 1. [`ss-search`](#tool-ss-search) | a natural-language query | ranked, **self-contained code blocks** |
+| 2. [`ss-grep`](#tool-ss-grep) | an exact regex/literal | every `file:line` hit, **ripgrep-identical** |
+| 3. [`ss-find`](#tool-ss-find) | a regex **+** a query | regex matches, **semantically re-ranked, as code blocks** |
+| 4. [`ss-semantic`](#tool-ss-semantic) | a file **+** a question | just the **relevant spans** of that file |
+| 5. [`ss-trace`](#tool-ss-trace) | a symbol | **callers + callees + impact**, in one call |
+| 6. [`ss-read`](#tool-ss-read) | a file (± line range) | exact bytes **+ symbol metadata** |
 
-### `ss-search` — the full retrieval stack in one call
+---
 
-```bash
-ss-search "how are websocket reconnects handled?" -k 5
+<a id="tool-ss-search"></a>
+### 1. 🔍 `ss-search` — hybrid search powerhouse
+
+A hybrid search pipeline with late interaction reranking that returns actual code blocks.
+
+SOTA in several published [`benchmarks`](#-benchmarks).
+
+```mermaid
+flowchart TD
+    Q(["🔍  natural-language query"]) --> ROUTE{{"🧭 WASM CatBoost router · lexical / hybrid"}}
+
+    ROUTE --> BM["📑 <b>BM25F</b><br/>field-weighted FTS5"]
+    ROUTE --> ANN
+
+    subgraph ANN ["🧬 three-stage ANN cascade"]
+        direction LR
+        BIN["binary <b>HNSW</b><br/>Hamming · ~100µs"] --> INT["INT8<br/>rescore"] --> FL["float32<br/>mmap sidecar"]
+    end
+
+    BM  --> FUSE
+    ANN --> FUSE
+    FUSE["🔀 <b>CCFusion</b><br/>convex combo · RRF fallback"] --> ROW1
+
+    subgraph ROW1 [" "]
+        direction LR
+        IAR["⚓ <b>IAR</b><br/>exact-symbol injection"] --> INTENT["🎯 intent rerank<br/>demote docs · tests · config"]
+    end
+
+    ROW1 --> ROW2
+
+    subgraph ROW2 [" "]
+        direction LR
+        GRAPH["🕸️ graph expansion<br/>typed edges · 1–2 hops · <b>PathRAG</b>"] --> MAXSIM["🧮 <b>Late-Interaction Rerank</b><br/>⚡ native Rust MaxSim kernel"] --> OUT(["🏁 <b>self-contained code blocks</b><br/>whole functions · 3k/8k/12k budget"])
+    end
+
+    classDef io    fill:#fde68a,stroke:#f59e0b,color:#000;
+    classDef out   fill:#bbf7d0,stroke:#15803d,color:#000,stroke-width:3px;
+    classDef route fill:#e0e7ff,stroke:#818cf8,color:#000;
+    classDef lex   fill:#dbeafe,stroke:#60a5fa,color:#000;
+    classDef fuse  fill:#f3e8ff,stroke:#c084fc,color:#000;
+    classDef rank  fill:#ffe4e6,stroke:#fb7185,color:#000;
+
+    class Q io;
+    class OUT out;
+    class ROUTE route;
+    class BM,BIN,INT,FL lex;
+    class FUSE,IAR fuse;
+    class INTENT,GRAPH,MAXSIM rank;
+
+    style ANN  fill:#eff6ff,stroke:#93c5fd,color:#000;
+    style ROW1 fill:none,stroke:none;
+    style ROW2 fill:none,stroke:none;
 ```
 
-One query fires the whole pipeline:
+<sub>↑ The diagram traces the **hybrid** route. A pure-lexical query — or a literal file path — short-circuits at the router straight to BM25F, skipping the vector cascade and fusion.</sub>
 
-1. **CatBoost query router** — a 498-tree gradient-boosted classifier compiled to WASM decides lexical vs hybrid from 50 single-pass features (camelCase/snake_case decomposition, CJK density, path shape…) in microseconds, with a low-confidence reject option that falls back to max-recall hybrid. Real file paths short-circuit straight to lexical.
-2. **Dual retrieval** — **BM25F** over field-weighted FTS5 (a hit on a function's *name* outweighs one buried in its body 10:1) runs in parallel with a **three-stage ANN cascade**: binary HNSW (Hamming distance over 64-byte binarized vectors, candidates in ~100 µs) → INT8 rescoring → full-precision float32 rescoring from a memory-mapped sidecar.
-3. **Convex-combination fusion** with route-specific weights and quantile normalization — and an automatic **RRF** fallback when score distributions degenerate.
-4. **Identifier-Anchored Retrieval (IAR)** — if your English mentions a real symbol, an exact-name lookup against the code graph injects that entity into the pool, even when the encoder ranked something tangential higher.
-5. **Intent-aware reranking** — docs/tests/config demoted when you want implementation; log-scaled call-site reference boosts surface the function everyone actually calls.
-6. **Adaptive graph expansion** — typed-edge walks (imports / extends / calls / uses) 1–2 hops out along the AST-derived knowledge graph, with intent-selected edge types, PathRAG-style flow-threshold pruning, and degree normalization so hub entities can't dominate.
-7. **Late-interaction rerank** — ColBERT-style per-token MaxSim over the quantized token index, on kernels that took a 231-candidate scoring pass from **1.26 s to 27 ms**.
-8. **Answer packaging** — near-duplicate siblings collapse to the best-matching member, MMR balances diversity, and entity-aware expansion emits *self-contained* blocks (whole functions with imports, docstrings, decorators) under an auto-selected **3k / 8k / 12k token budget** driven by post-ranking signals like top-1 dominance.
+| Stage | What it actually does |
+|-------|-----------------------|
+| 🧭 **Route** | **WASM-exported CatBoost** · lexical / hybrid · **~10 µs** routing · low-confidence → max-recall hybrid |
+| 🧬 **Retrieve** | • **Lexical** — **BM25F** over field-weighted FTS5 (name 10× · signature 5× · alias 4× · doc 1×)<br/>• **Embed** — query vectorized by the local **CodeRankEmbed** model (swappable for Voyage / Jina / Codestral)<br/>• **Vector cascade** — binary **HNSW** (Hamming, 64-byte, ~100 µs) → INT8 rescore → exact float32 from a memory-mapped sidecar |
+| 🔀 **Fuse** | • **CCFusion** — convex-combine both rankings · per-route weights · quantile-normalized<br/>• **MMR** (λ=0.9) diversity pass over the fused list<br/>• auto **RRF** (k=60) fallback on degenerate score distributions |
+| ⚓ **Anchor** | • **IAR** (Identifier Anchor Retrieval) — a real symbol in the query fires an exact-name code-graph lookup that injects that entity, even when the encoder ranked it too low |
+| 🎯 **Intent Rerank** | • demote docs / tests / config when you want implementation<br/>• log-scaled call-site boosts surface the most-referenced function |
+| 🕸️ **Graph Expansion** | • typed-edge walks (`imports`/`extends`/`calls`/`uses`) · adaptive 2-hop on the AST graph · edges picked by intent<br/>• **PathRAG** flow pruning + degree normalization → hubs can't dominate |
+| 🧮 **Late interaction Rerank** | • Query embedded per-token by **LateOn-Code** (149M; a 17M **edge** variant auto-selected on low-RAM hosts)<br/>• **MaxSim** against the pre-indexed quantized token vectors<br/>• native Rust+Rayon MaxSim kernel ⚡ · WASM-SIMD fallback (1.26 s → 27 ms on a 231-candidate rerank) |
+| 📦 **Package** | • entity-aware expansion → whole functions (imports, docstrings, decorators)<br/>• same-file overlap demotion → diverse, non-overlapping spans<br/>• auto-selected **3k / 8k / 12k** token budget |
 
 <details>
-<summary><b>More</b></summary>
+<summary><b>🌶️ Extra spice — the bits that didn't fit the diagram</b></summary>
 
-- The expensive 8k/12k tiers are tuned to fire on roughly 1–5% of queries — the default case stays cheap. Force a tier with `--full` / `--xl`, or a mode with `--mode lexical|semantic|hybrid|pattern`.
-- Also available as `sweet-search "<query>"` on the CLI and the `search` MCP tool.
+**🧠 The HNSW, in full** ([full writeup](docs/HNSW_APPROACH.md)). Stage 1 is a from-scratch binary HNSW, and every "advanced" trick ships **on by default**:
+- **Heuristic neighbor selection** (HNSW Algorithm 4) + **M0 = 2M** on layer 0 — a real graph backbone, not naïve closest-M
+- **Shuffled insertion order** — no filesystem-ordering bias baked into the highway structure
+- **Discovery-rate adaptive early termination** + **adaptive ef** — easy queries stop early, hard ones keep their budget
+- A **denser graph than most vendors ship** (M=64 · efC=800 · efS=400) — which broke an 80.6 % → 86.5 % recall@200 plateau and cut p50 latency ~33 %
+- **Zero-GC search**: typed-array heaps + generation-stamped visited lists — no per-query allocation
+- 64-byte sign-bit vectors (Hamming) → INT8 → exact float32 from a memory-mapped sidecar
+
+**⚡ Why it's quick.** A native Rust + Rayon **MaxSim kernel** (47× over scalar; 16× WASM-SIMD fallback) · int4-quantized, binary-packed token vectors (plain INT4 is the shipped path — the full [TurboQuant](docs/LI_QUANTIZATION_STRATEGY.md) algorithm is researched but deferred; binary packing alone cut the LI index ~3.4×, 1.34 GiB → ~396 MiB) · a memory-mapped float32 sidecar that skips SQL on the rescore hot path · **score-spread adaptive pooling** (decisive queries shrink the rescore pool, ambiguous ones widen it) · and a warm daemon that answers in a single NAPI call — no process is ever forked.
+
+**🎛️ Priors & structure.**
+- **Quality priors:** every chunk carries a 0–1 prior from test proximity, git recency, symbol centrality (PageRank), comment density, and complexity — production code surfaces, stale fixtures sink.
+- **Community structure:** a canonical **Leiden** pass detects code communities on the entity graph at index time, feeding vocabulary prewarming and structural signals — it understands your modules, not just your directories.
+- **Multilingual:** 14 languages get full tree-sitter AST treatment; a 39-config registry covers 70+ extensions beyond that. Router features handle camelCase/snake_case, CJK density, and German compounds.
+- **Format-gated signals:** structure-aware boosts and demotions (symbol-exact, path-token, mega-entity) fire only in agent mode — they help agent-shaped queries and would hurt plain NL, so they stay gated by default.
+
+**🛟 Rescues & honest trade-offs.**
+- **Long-query rescue:** wordy NL queries that FTS5 would tokenize into an unsatisfiable `AND` fall back to multi-query BM25F + RRF — one query per content keyword, fused.
+- **Near-duplicate dedup:** a SimHash + MinHash-LSH pass (Jaccard τ=0.9) clusters copy-paste and vendored code at index time; aliases reuse their exemplar's vectors and skip *both* the bi-encoder and late-interaction encoding.
+- **A negative result we ship anyway:** we built a full cross-encoder rerank cascade behind an adaptive confidence gate, measured it on our eval sets — and it didn't beat MaxSim at 3× the latency. So it ships **disabled** (`SWEET_SEARCH_CASCADE_ENABLED=true` to try it). We'd rather ship the faster path than a fancier diagram.
+- **Budget tiers:** the expensive 8k/12k tiers fire on ~1–5 % of queries — the default stays cheap. Force one with `--full` / `--xl`, or pick a mode with `--mode lexical|semantic|hybrid|pattern`.
+
+Also available as `sweet-search "<query>"` on the CLI and the `search` MCP tool.
 
 </details>
 
-### `ss-grep` — grep, minus every wasted millisecond
+---
 
-```bash
-ss-grep "parseRetryAfter" -k 10
-```
+<a id="tool-ss-grep"></a>
+### 2. ⚡ `ss-grep` — grep, minus every wasted millisecond
 
 **10.2× faster than ripgrep end-to-end at the median** — measured across **353 realistic queries on 5 real repos**
 (range 8.5–17.7× per repo, 1 ms p50), with **identical match counts on every single query**. Three things buy that:
@@ -247,7 +418,7 @@ ss-grep "parseRetryAfter" -k 10
 - **Regex-AST literal extraction + SIMD intersection**: required substrings are pulled from the pattern's syntax tree, posting lists are intersected with NEON/SSE2 block merges (galloping search for skewed sizes), and only the files that *can* match — typically 0.1–5% of the corpus — see the real regex.
 - **Fully in-process**: verification runs on Rust's regex crate with Rayon across all cores, inside the warm daemon, in a single NAPI call. No child process is ever spawned — zero fork/exec, zero pipe I/O, zero JSON re-parsing.
 
-Hits come back **ranked and scored**, so an agent can trust the top one and stop.
+Every match comes back in stable `file:line` order — ripgrep-identical counts, optional context lines — with no relevance guessing, no subprocess, in one warm call.
 
 <details>
 <summary><b>More</b></summary>
@@ -257,7 +428,10 @@ Hits come back **ranked and scored**, so an agent can trust the top one and stop
 
 </details>
 
-### `ss-find` — ColGrep, on a faster engine
+---
+
+<a id="tool-ss-find"></a>
+### 3. `ss-find` — ColGrep, on a faster engine
 
 ```bash
 ss-find "token refresh logic" --regex "refresh.*[Tt]oken"
@@ -280,15 +454,18 @@ semantically ranked — but rebuilt on our own substrate:
 
 </details>
 
-### `ss-semantic` — hybrid retrieval, scoped to one file
+---
+
+<a id="tool-ss-semantic"></a>
+### 4. `ss-semantic` — hybrid retrieval, scoped to one file
 
 ```bash
 ss-semantic src/auth/session.ts "where does the cookie get its expiry?"
 ```
 
 You know the file; this finds the lines. Every indexed chunk of the file is scored by **three independent
-signals** — BM25-style lexical term match, exact symbol-name match (weighted 1.5×), and ColBERT-style
-MaxSim over late-interaction token embeddings — fused with **Reciprocal Rank Fusion** (k=60), with
+signals** — BM25-style lexical term match, exact symbol-name match (weighted 1.5×), and per-token
+**MaxSim** late interaction over the **LateOn-Code** embeddings — fused with **Reciprocal Rank Fusion** (k=60), with
 symbol-less fragment chunks demoted 0.85× so real definitions win ties. The top spans are then
 **re-read from disk** (±2 context lines, overlapping spans merged), so the answer is filesystem ground
 truth even mid-edit; if the file is newer than its index entry you get an explicit staleness warning.
@@ -303,7 +480,10 @@ The useful answer: just the relevant spans with line numbers — not the whole f
 
 </details>
 
-### `ss-trace` — graph algorithms, not grep guesswork
+---
+
+<a id="tool-ss-trace"></a>
+### 5. `ss-trace` — graph algorithms, not grep guesswork
 
 ```bash
 ss-trace processOrder --in src/orders/service.py
@@ -330,7 +510,10 @@ bounds impact traversal (1–4).
 
 </details>
 
-### `ss-read` — exact bytes, with the index's knowledge attached
+---
+
+<a id="tool-ss-read"></a>
+### 6. `ss-read` — exact bytes, with the index's knowledge attached
 
 ```bash
 ss-read src/db/pool.js 120 180
@@ -353,6 +536,8 @@ without another search.
 > capability is equally available as `sweet-search` CLI subcommands and as MCP tools — see
 > [Works With Your Agent](#-works-with-your-agent).
 
+---
+
 ## 🧠 An Agent Prompt That Was Evolved, Not Written
 
 Giving an agent six tools is easy. Getting it to *stop grepping in circles* is not.
@@ -385,36 +570,86 @@ What it teaches:
 
 ## ⚡ GPU-Accelerated Indexing, Fully Local
 
-All inference is on-device, in Rust, via [candle](https://github.com/huggingface/candle) — with the
-attention path swapped for **fused kernels tuned per backend**, and an honest CPU story for machines
-with no accelerator at all.
+> **Chunk → enrich → embed → quantize** — every step on-device and in Rust. Batches are sized to *your CPU's actual cache*, two open code-models do the encoding, and two separate quantizations make the index both **faster to build** and **small enough to live in RAM**. Zero API keys; nothing ever leaves the machine.
+
+<table>
+<tr>
+<td width="50%" valign="top">
+
+① 🧩 **[Structure-aware chunk](#idx-chunk)**<br>
+<sub>cAST over tree-sitter ASTs — whole functions, never sliced mid-body</sub>
+
+</td>
+<td width="50%" valign="top">
+
+② 🏷️ **[Enrich from structure](#idx-enrich)**<br>
+<sub>deterministic preamble from the code graph — **no LLM call**</sub>
+
+</td>
+</tr>
+<tr>
+<td width="50%" valign="top">
+
+③ 🤖 **[Embed — two models](#idx-embed)**<br>
+<sub>dense **CodeRankEmbed** + per-token **LateOn-Code**</sub>
+
+</td>
+<td width="50%" valign="top">
+
+④ 🗜️ **[Quantize + persist](#idx-quantize)**<br>
+<sub>INT8 weights → **2× faster build** · INT4 vectors → **fits in RAM**</sub>
+
+</td>
+</tr>
+</table>
+
+**The inference engine, picked for your silicon:**
 
 | Your hardware | What runs |
-|---------------|-----------|
-| Apple Silicon (M1+) | candle **Metal**, BF16, fused SDPA attention |
-| Apple Silicon (M3+) | … plus a **CoreML Neural Engine cascade** (~18% faster full indexing, measured on M3 Max) |
-| NVIDIA GPU (SM 7.0+) | candle **CUDA**; **flash-attention** on Ampere+ |
-| Anything else | **ONNX Runtime INT8** — optimized CPU path, ~139 MB embedding model, no GPU weights downloaded |
-
-Before a single token is embedded, files are chunked by **[cAST](https://arxiv.org/abs/2506.15655)** —
-structure-aware chunking over real **tree-sitter ASTs**. A recursive split-then-merge greedily packs
-adjacent sibling AST nodes into a chunk until the size cap, and recurses *into* nodes too big to fit —
-so every chunk is whole code: a function, a class, a contiguous run of declarations. Never a function
-sliced mid-body, never a string split mid-literal. 14 languages get true AST grammars (JS/TS/TSX,
-Python, Go, Rust, Java, C, C++, Ruby, PHP, Kotlin, Swift, C#); a 39-config regex registry extends
-structure-aware chunking to 70+ file extensions beyond those. Each chunk carries its symbol name,
-entity type, signature, and line span — the metadata that feeds the code graph, `ss-read`'s
-annotations, and the self-contained answers everywhere else.
+|--|--|
+| 🍏 Apple Silicon (M1+) | candle **Metal**, BF16, fused SDPA attention |
+| 🍏 Apple Silicon (M3+) | …​ plus a **CoreML Neural Engine cascade** — ~18% faster full index (measured, M3 Max) |
+| 🟩 NVIDIA GPU (SM 7.0+) | candle **CUDA**; **flash-attention** on Ampere+ |
+| 💻 No accelerator | **ONNX Runtime INT8** — tuned CPU path, 132 MB model, **zero GPU weights downloaded** |
+
+<a id="idx-chunk"></a>
+### 🧩 Chunking — every chunk is whole code, never a fixed window
+- **[cAST](https://arxiv.org/abs/2506.15655)** structure-aware chunking over real **tree-sitter** ASTs: a recursive *split-then-merge* greedily packs sibling AST nodes up to the size cap and recurses *into* nodes too big to fit. So a chunk is always a **function, a class, or a contiguous run of declarations** — never a body cut in half, never a string split mid-literal.
+- **14 languages** get true AST grammars — `JS · TS · TSX · Python · Go · Rust · Java · C · C++ · Ruby · PHP · Kotlin · Swift · C#` — and a **39-config regex registry** carries structure-aware chunking to **70+ more extensions**.
+
+<a id="idx-enrich"></a>
+### 🏷️ Metadata — context the encoder can actually see
+- Every chunk ships its **symbol name · entity type · signature · line span** — the metadata that powers the code graph, `ss-read` annotations, and the self-contained answers everywhere else.
+- **Contextual enrichment:** before embedding, each chunk is prefixed with a structured preamble assembled from the AST + code graph — *file path · enclosing-scope breadcrumb · name & type · merged siblings · the imports it actually uses*. **Both** encoders see it, so a bare `getId()` still retrieves on the class and module around it.
+- Our nod to **[Anthropic's Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval)** — except they prepend an *LLM-generated* summary (one model call per chunk); we derive the context **deterministically from structure**: no LLM, no per-chunk inference, regenerated for free on every reindex. **Tuned per language** from GenCodeSearchNet ablations — Python stays minimal, the Java family keeps a slug-stripped path, JS/Ruby/Go/C/C++/Rust get the full preamble where closures and imports earn their keep.
+
+### 🧠 Cache-aware batching — we read your CPU before we batch it
+- We **detect your last-level cache at runtime** — `hw.perflevel0.l2cachesize` (the 16 MB P-cluster on Apple Silicon, *not* the smaller E-cluster), Intel L3, or `/sys/.../cache` on Linux — then size every embedding batch so **one transformer layer's weights *plus* the batch's activations stay resident in cache**. No spilling to main memory mid-layer; on a long-sequence tail that's the difference between B=1 and a measured **2.1× per-chunk slowdown**.
+- **Uses every core the hardware really has** — full count on ARM/Apple Silicon; x86 SMT siblings discounted because they don't scale inference linearly.
+- **ORT drives the CPU path** (ONNX Runtime); GPU hosts swap in fused kernels (below). Either way inference runs off the event loop as a napi `AsyncTask`, so tokenization and SQLite writes overlap compute instead of stalling behind it.
+
+<a id="idx-quantize"></a>
+### 🗜️ Two quantizations — one buys speed, one buys size
+| | **Model weights** · INT8 ORT | **Index vectors** · INT4 binary |
+|:--|:--|:--|
+| **Job** | build the index faster on CPU | keep the on-disk index tiny |
+| **Win** | **~2× faster** indexing · 4× smaller model (**132 MB**) | LI index **1.34 GiB → ~396 MiB** · INT4 nibble-packing halves it again |
+| **Fidelity** | **≥ 0.96 cosine** vs FP32 | **no measurable retrieval loss** (A/B-tested vs INT8) |
+
+<a id="idx-embed"></a>
+### 🤖 Two models — both open, both local, both code-specialized
+- **[CodeRankEmbed](https://huggingface.co/nomic-ai/CodeRankEmbed)** — 768-d dense bi-encoder (137M, Apache-2.0) for first-stage recall.
+- **[LateOn-Code](https://huggingface.co/lightonai/LateOn-Code)** — ModernBERT per-token **late interaction** (149M) for the rerank.
+- **Edge fallback for leaner machines:** a **17M `edge` LateOn-Code** (~9× smaller FP32 backbone) auto-selects on low-RAM hosts, and the whole CPU path runs INT8 with **no GPU weights ever downloaded** — full local search on a laptop with no accelerator.
 
 <details>
-<summary><b>What's actually custom here</b></summary>
+<summary><b>What's actually custom here — the kernels we hand-wrote</b></summary>
 
-- **Surgical attention swap:** we vendor the upstream model implementations (NomicBERT for embeddings, ModernBERT for late interaction) and replace only the attention forward pass — an MLX-ported fused SDPA kernel on Metal, `candle-flash-attn` with varlen packing on CUDA Ampere+, and byte-for-byte upstream math on CPU so the fallback is provably identical.
+- **Surgical attention swap:** we vendor the upstream model implementations (NomicBERT for embeddings, ModernBERT for late interaction) and replace **only the attention forward pass** — an MLX-ported fused SDPA kernel on Metal, `candle-flash-attn` with varlen packing on CUDA Ampere+, and byte-for-byte upstream math on CPU so the fallback is provably identical.
 - **A silent-NaN bug, found and fixed:** Apple's Metal SDPA kernel downcasts attention masks to F16, which saturates the standard `f32::MIN` mask to `-Inf` and quietly produces NaN on padded rows — collapsing retrieval quality. We clamp the mask and serialize Metal command-buffer submissions (concurrent submission corrupts outputs on shared queues). Details in [`crates/sweet-search-native/src/inference/`](crates/sweet-search-native/src/inference/).
 - **CoreML cascade:** 18 pre-traced `.mlpackage` variants (bucketed by sequence length) dispatched to the Apple Neural Engine through an Objective-C shim; oversized batches fall through to Metal. Gated to M3+ because on M1/M2 the ANE doesn't beat its own compile overhead — we measured, so it's off there.
-- **GPU off the event loop:** inference runs as napi `AsyncTask` on libuv worker threads, so tokenization and SQLite writes overlap GPU compute instead of stalling behind it.
-- **Pipelined indexing:** while batch *N+1* embeds, batch *N*'s vectors stream into SQLite through zero-copy buffer views; full rebuilds write to a temp file and atomically swap, so a crash never leaves you serving half an index.
-- **Models:** CodeRankEmbed (768-d, code-specialized) for embeddings; LateOn-Code (ModernBERT) for per-token late interaction, in a full-fidelity `standard` and a compact `edge` variant (~9× smaller FP32 backbone; ~2× smaller on the INT8 CPU path).
+- **Structure-routed enrichment:** the preamble (path · scope chain · symbol · siblings · imports) is assembled at index time from a code-graph line-range overlap query — never an LLM call — then routed per language family (full enriched text for JS/Ruby/Go/C-family/Rust, a slimmer path policy for Python and the Java family), every decision settled by per-language ablation rather than a global default.
+- **Pipelined, crash-safe indexing:** while batch *N+1* embeds, batch *N*'s vectors stream into SQLite through zero-copy buffer views; full rebuilds write to a temp file and atomically swap, so a crash never leaves you serving half an index.
 
 </details>
 
@@ -464,16 +699,16 @@ Four Rust crates do the heavy lifting, each with a graceful fallback so the engi
 
 </details>
 
-### 🗜️ TurboQuant: an index that fits in RAM
+### 🗜️ INT4 binary segments: the on-disk format behind the RAM-sized index
 
-A 17k-document codebase's late-interaction index weighed **1.34 GiB** as JSON-encoded INT8. The binary
-segment format cut the same index to **~396 MiB** (3.4× of pure ASCII bloat, gone) — and the INT4
-default packs token vectors at half a byte each on top of that. Laptop-sized, fully in RAM.
+The quantization headline lives [up in indexing](#-gpu-accelerated-indexing-fully-local) — `1.34 GiB → ~396 MiB`,
+INT4-halved again. Here's the **SSLX** segment format that delivers it: crash-safe by construction, and
+the three-stage retrieval it feeds at query time.
 
 <details>
 <summary><b>Deep dive</b></summary>
 
-- **INT4 by default:** per-token min/scale quantization with nibble packing (two values per byte), A/B-tested against the INT8 baseline with no meaningful retrieval regression before becoming the default.
+- **INT4 by default:** per-token min/scale quantization with nibble packing (two values per byte), A/B-tested against the INT8 baseline with no meaningful retrieval regression before becoming the default. We borrowed the *rotation insight* from Google's [TurboQuant](docs/LI_QUANTIZATION_STRATEGY.md), but ship plain INT4 — the full TurboQuant algorithm (WHT + PolarQuant + QJL) is researched and deferred, not in the product path.
 - **SSLX binary segments:** the index persists as ~10k-document binary segment files with structured headers and CRC32 footers — a crash costs you at most one segment, not the index.
 - **Three-stage retrieval:** a binary HNSW (Hamming distance over 64-byte binarized vectors, ~32× smaller than float HNSW) produces candidates in ~100 µs, INT8 rescoring narrows them, and a float32 sidecar rescores the final pool — speed without giving up top-result quality.
 - **Memory-mapped HNSW:** the float graph index loads via `mmap` (USearch `view()`), contributing **0 MB** to the V8 heap at search time; the OS reclaims pages under pressure.
@@ -482,30 +717,6 @@ default packs token vectors at half a byte each on top of that. Laptop-sized, fu
 
 </details>
 
-## 🎯 The Ranking Stack
-
-Retrieval quality comes from *layers*, each one cheap, each one earning its place:
-
-1. **Route** — CatBoost classifies the query (lexical / semantic / hybrid) and sets fusion weights; real file paths short-circuit straight to lexical
-2. **Retrieve** — BM25F field-weighted lexical (a match on a function's *name* outranks one buried in a body) in parallel with the three-stage vector pipeline
-3. **Fuse** — convex combination with per-route weights and quantile normalization, falling back to Reciprocal Rank Fusion on degenerate score distributions
-4. **Anchor** — name a real symbol in your query and identifier-anchored retrieval injects the exact-name entity, even when the encoder ranked something tangential higher
-5. **Rerank** — ColBERT-style MaxSim late interaction over the quantized token index
-6. **Expand** — typed-edge graph walks (1–2 hops, intent-adaptive, PathRAG-style flow pruning) pull in the related code a single chunk can't show
-7. **Polish** — intent-aware demotion of docs/tests/config when you want implementation, call-site reference boosts, MMR diversity, near-duplicate sibling re-ranking
-
-<details>
-<summary><b>Deep dive & design honesty</b></summary>
-
-- **Intent awareness:** a lightweight classifier distinguishes "fix this crash" from "how do I use this API" and tunes graph-edge selection, result limits, and chunk-type preferences per intent.
-- **Quality priors:** each chunk carries a 0–1 prior from test proximity, git recency, symbol centrality (PageRank), comment density, and complexity — production code surfaces, stale fixtures sink.
-- **Community structure:** a canonical Leiden algorithm detects code communities on the entity graph at index time, feeding vocabulary prewarming and structural signals — the engine understands your modules, not just your directories.
-- **Multilingual:** 14 languages get full tree-sitter AST treatment; a 39-config registry covers 70+ extensions beyond that; router features handle camelCase/snake_case decomposition, CJK density, and German compounds.
-- **Long-query rescue:** wordy natural-language queries that FTS5 would tokenize into an unsatisfiable AND get a multi-query BM25F + RRF fallback — one query per content keyword, fused.
-- **A negative result we ship anyway:** we built a full cross-encoder rerank cascade behind an adaptive confidence gate, measured it on our evaluation sets — and it didn't beat MaxSim at 3× the latency. So it ships **disabled** (`SWEET_SEARCH_CASCADE_ENABLED=true` if you want to try). We'd rather ship the faster path than a fancier diagram.
-
-</details>
-
 ## 🔌 Works With Your Agent
 
 sweet-search meets your agent wherever it is — shell tools, MCP, or injected instructions:
@@ -565,6 +776,7 @@ sweet-search stands on a lot of shoulders, and we'd rather name them than preten
 - **[CatBoost](https://catboost.ai/)** — the query router model; **Traag et al.** for the [Leiden algorithm](https://arxiv.org/abs/1810.08473); **Cormack et al.** for RRF; **[PathRAG](https://arxiv.org/abs/2502.14902)** for flow-pruned graph expansion; **[cAST](https://arxiv.org/abs/2506.15655)** for structure-aware chunking
 - **[GEPA](https://arxiv.org/abs/2507.19457)** — the reflective evolutionary prompt-optimization paradigm behind our agent prompt
 - **[nomic-ai](https://huggingface.co/nomic-ai)** — the CodeRankEmbed embedding model
+- **[Anthropic](https://www.anthropic.com/news/contextual-retrieval)** — the Contextual Retrieval idea behind our chunk enrichment, here derived from code structure instead of an LLM summary
 
 ## 📄 License
 

package/assets/banner/banner-manifest.json

@@ -0,0 +1,10 @@
+{
+  "format": "grid-webp",
+  "file": "banner-frames.webp",
+  "gridCols": 8,
+  "cellW": 1200,
+  "cellH": 400,
+  "count": 96,
+  "fps": 12,
+  "frameMs": 83
+}