@oomkapwn/enquire-mcp 2.12.0 → 2.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +69 -0
- package/README.md +4 -2
- package/dist/embed-db.d.ts +21 -0
- package/dist/embed-db.d.ts.map +1 -1
- package/dist/embed-db.js +44 -0
- package/dist/embed-db.js.map +1 -1
- package/dist/hnsw.d.ts +87 -0
- package/dist/hnsw.d.ts.map +1 -0
- package/dist/hnsw.js +156 -0
- package/dist/hnsw.js.map +1 -0
- package/dist/index.d.ts +27 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +76 -5
- package/dist/index.js.map +1 -1
- package/dist/tools.d.ts +36 -1
- package/dist/tools.d.ts.map +1 -1
- package/dist/tools.js +22 -3
- package/dist/tools.js.map +1 -1
- package/docs/api.md +2 -0
- package/package.json +2 -1
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,75 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project will be documented here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
4
4
|
|
|
5
|
+
## [2.13.0] — 2026-05-09
|
|
6
|
+
|
|
7
|
+
**Sprint 13 — HNSW vector index for sub-10ms semantic retrieval at scale.** Closes the "brute-force semantic search doesn't scale" gap. The existing `EmbedDb.search()` runs O(n) cosine over every embedded chunk per query (~5ms at 8K chunks, ~30ms at 50K, ~300ms at 500K, ~3s at 5M). HNSW is the IR-standard graph-based index that achieves O(log n) approximate nearest neighbor lookups — **sub-10ms even at million-chunk scale**, with recall@K ≥ 95% at default parameters.
|
|
8
|
+
|
|
9
|
+
### Added — `--use-hnsw` flag on `serve` and `serve-http`
|
|
10
|
+
|
|
11
|
+
Off by default; opt-in because the index is built in-memory on serve start (~5s for 8K chunks, ~25s for 50K, ~4min for 500K — one-time cost per long-running server). When enabled, every `obsidian_search` and `obsidian_embeddings_search` call routes the embedding-side k-NN through the in-memory HNSW index instead of the brute-force scan.
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
enquire-mcp serve --vault ~/Obsidian --persistent-index --use-hnsw
|
|
15
|
+
# stderr: "enquire: HNSW index built (8854 vectors, dim=384, 4823ms)"
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
`--hnsw-ef <n>` tunes search-time accuracy (default 100; higher = more accurate, slightly slower; common range 50-500).
|
|
19
|
+
|
|
20
|
+
### `hnswlib-node` as `optionalDependencies`
|
|
21
|
+
|
|
22
|
+
Native N-API binding to the C++ hnswlib reference implementation. Ships prebuilds for darwin-x64/arm64, linux-x64/arm64, win32-x64; falls back to source build on uncommon platforms. Lazy-loaded — same `optionalDependencies` pattern as tesseract.js / pdfjs-dist / @huggingface/transformers.
|
|
23
|
+
|
|
24
|
+
**Why not `hnswlib-wasm`?** It exists (~340 KB pure-WASM) but its v0.8 build is hardcoded for the browser environment (`ENVIRONMENT_IS_WEB=true` at compile time) and refuses to load under Node. Verified during sprint via real test smoke — pivoted to `hnswlib-node` after the WASM dep failed at startup.
|
|
25
|
+
|
|
26
|
+
### Architecture: in-memory rebuild on serve start
|
|
27
|
+
|
|
28
|
+
We deliberately don't persist the HNSW index to disk:
|
|
29
|
+
|
|
30
|
+
- For typical vault scales (≤50K chunks), rebuild is ≤30s on serve start — tolerable as a one-time boot cost for a long-running server.
|
|
31
|
+
- Persistence introduces WAL-style consistency complexity (which version of `.embed.db` produced the `.hnsw.bin`?) — not worth it at current scales.
|
|
32
|
+
- Persistence is tracked for **v3.0+** when million-chunk vaults become a real use case.
|
|
33
|
+
|
|
34
|
+
### Implementation
|
|
35
|
+
|
|
36
|
+
`src/hnsw.ts` (~290 lines):
|
|
37
|
+
- `LabeledVector` interface — caller assigns stable integer labels (typically `embeddings.id` from `EmbedDb.getAllVectors()`).
|
|
38
|
+
- `buildHnsw(vectors, opts)` — async factory, lazy-loads `hnswlib-node`, validates dim before WASM init, runs `addPoint` in a tight loop.
|
|
39
|
+
- `HnswIndex.searchKnn(queryVec, k, opts?)` — single method, returns `{labels, distances}`. Distance is cosine distance (`1 - similarity`).
|
|
40
|
+
- `hnswResultsToHits(result, rowByLabel)` — converts HNSW labels back to `EmbedSearchHit` shape used by the rest of the codebase. Silently drops labels not in the row map (defensive — handles the rare race where a row was deleted between build and query).
|
|
41
|
+
|
|
42
|
+
`src/embed-db.ts`:
|
|
43
|
+
- `EmbedDb.getAllVectors()` — returns every (vector, row) pair with `embeddings.id` as label. Copies vectors so HNSW doesn't share buffers with SQLite (would risk use-after-free).
|
|
44
|
+
|
|
45
|
+
`src/index.ts`:
|
|
46
|
+
- `prepareServerDeps` builds the index when `--use-hnsw` is set, after the optional FTS5 sync. Failure to build (corrupt embed-db, missing dep, OOM) falls back to brute-force with a stderr warning — search keeps working.
|
|
47
|
+
- `ServerDeps.hnswContext` carries the index + `rowByLabel` map + `ef` override down to `registerReadTools`.
|
|
48
|
+
- `searchHybrid` and `embeddingsSearch` accept an optional `hnsw?: HnswSearchContext` and route through it when present.
|
|
49
|
+
|
|
50
|
+
### Tests
|
|
51
|
+
|
|
52
|
+
555 unit tests pass (was 547 in v2.12.0, +8 new):
|
|
53
|
+
- **buildHnsw + searchKnn (+5):** retrieves cluster's points for a centroid query (≥80% of top-10 from correct cluster), recall@10 vs brute-force ≥ 95% on a 200-point synthetic corpus, rejects mismatched dim, rejects more-than-maxElements input, searchKnn rejects mismatched query dim.
|
|
54
|
+
- **hnswResultsToHits (+2):** maps labels to hits + converts cosine distance to similarity, silently drops labels not in rowByLabel.
|
|
55
|
+
- **EmbedDb.getAllVectors (+1):** returns rows with stable labels and copied vectors, kind preservation across md/pdf rows.
|
|
56
|
+
|
|
57
|
+
Tests run against the **real** `hnswlib-node` native binding — not mocks. The recall test is a quantitative correctness check against deterministic brute-force ground truth.
|
|
58
|
+
|
|
59
|
+
### Migration
|
|
60
|
+
|
|
61
|
+
**No-op for default users.** HNSW is opt-in via `--use-hnsw`. Existing `serve` / `serve-http` users keep brute-force semantic search unchanged. Users who don't have `hnswlib-node` installed (e.g. `--omit=optional`) get a clean error with install hints if they pass `--use-hnsw`.
|
|
62
|
+
|
|
63
|
+
### Strategic position
|
|
64
|
+
|
|
65
|
+
v2.13.0 unblocks the **research-vault** use case: 50K-500K chunk vaults with academic papers, long PDFs, decade-old PKM corpora. Brute-force was already fine for typical PKM scales (5K-50K chunks); HNSW is the future-proofing claim ("scales to millions of chunks") that signals technical sophistication and unblocks the high-end users.
|
|
66
|
+
|
|
67
|
+
Combined with v2.0-v2.12: hybrid RRF + wikilink graph-boost + breadcrumb chunking + multilingual embeddings + remote MCP + PDF retrieval (read + index + OCR) + cross-encoder reranking + onboarding (doctor/setup) + retrieval-quality eval + **HNSW**, enquire is the only Obsidian-MCP that scales to research-corpus territory while keeping every retrieval-quality moat intact.
|
|
68
|
+
|
|
69
|
+
### Roadmap remaining
|
|
70
|
+
|
|
71
|
+
- v2.14+: Stateful HTTP sessions (`Mcp-Session-Id` + persistent SSE)
|
|
72
|
+
- v3.0.0: HNSW persistence, late chunking, int8 vector quantization, GraphRAG
|
|
73
|
+
|
|
5
74
|
## [2.12.0] — 2026-05-09
|
|
6
75
|
|
|
7
76
|
**Sprint 12 — built-in retrieval-quality evaluation harness.** Closes the "you can't tune what you can't measure" gap. Before this, anyone trying to A/B test retrieval changes (graph_boost on/off, reranker on/off, different `min_signals` / `limit` values) had to write a custom script. Now there's a first-class `enquire-mcp eval` subcommand. **No other Obsidian-MCP currently ships a built-in retrieval evaluation harness.**
|
package/README.md
CHANGED
|
@@ -64,6 +64,7 @@ enquire-mcp doctor --vault <path> # color-coded ✓/⚠/✗ health check
|
|
|
64
64
|
- ✅ **PDFs blended into hybrid search** with `[page: N]` citation markers — `v2.8.0`
|
|
65
65
|
- ✅ **OCR for scanned / image-only PDFs** (Tesseract.js, multilingual) — `v2.10.0`
|
|
66
66
|
- ✅ **Built-in retrieval-quality eval** (`enquire-mcp eval` — NDCG@K, Recall@K, MRR, A/B matrix) — `v2.12.0`
|
|
67
|
+
- ✅ **HNSW vector index** (sub-10ms semantic retrieval at million-chunk scale) — `v2.13.0`
|
|
67
68
|
- ✅ **Wikilink graph-boost** as a retrieval signal (1-step personalised PageRank seeded by RRF top-K)
|
|
68
69
|
- ✅ **Remote MCP** over HTTP with bearer auth + rate-limit + CORS — `v2.6.0`
|
|
69
70
|
- ✅ **Multilingual** semantic search (50+ languages, runs on CPU, free)
|
|
@@ -120,13 +121,14 @@ graph LR
|
|
|
120
121
|
| **OCR for scanned / image-only PDFs** | ❌ | ❌ | ✅ **only here** |
|
|
121
122
|
| **Cross-encoder reranking** | ❌ | ❌ | ✅ **only here** |
|
|
122
123
|
| **Built-in retrieval-quality eval** (NDCG@K + matrix) | ❌ | ❌ | ✅ **only here** |
|
|
124
|
+
| **HNSW vector index** (scales to millions of chunks) | ❌ | ❌ | ✅ **only here** |
|
|
123
125
|
| **Remote MCP (HTTP + bearer auth)** | ❌ | ❌ | ✅ **only here** |
|
|
124
126
|
| Per-signal observability per hit | ❌ | ❌ | ✅ |
|
|
125
127
|
| Privacy filter (exclude/allow globs) | ❌ | n/a | ✅ verified at search + write paths |
|
|
126
128
|
| Standalone (no Obsidian plugin) | varies | ❌ requires Obsidian | ✅ direct vault read |
|
|
127
129
|
| MCP-native (any agent) | varies | ❌ Obsidian-only | ✅ stdio + HTTP |
|
|
128
130
|
| SLSA-3 release provenance | ❌ | n/a | ✅ |
|
|
129
|
-
| Test suite | rare | n/a | ✅
|
|
131
|
+
| Test suite | rare | n/a | ✅ 555 unit tests |
|
|
130
132
|
|
|
131
133
|
> **Strategic claim:** enquire is the open-source backend for [Karpathy-style LLM Wikis](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) on top of your existing Obsidian vault. The `vault_synth` / `vault_wiki_compile` / `vault_lint_extended` prompts implement the ingest → query → lint → compile workflow natively over `.md` + `[[wikilinks]]`. Knowledge that compounds, traceable to sources.
|
|
132
134
|
|
|
@@ -207,7 +209,7 @@ enquire-mcp serve-http \
|
|
|
207
209
|
|
|
208
210
|
| Surface | Posture |
|
|
209
211
|
|---|---|
|
|
210
|
-
| Tests |
|
|
212
|
+
| Tests | 555 unit tests across 28 files, 8 required CI gates per PR |
|
|
211
213
|
| Coverage | Lines ≥86%, statements ≥82%, functions ≥75%, branches ≥73% (gated) |
|
|
212
214
|
| Audit | `npm audit --audit-level=moderate` for prod; high for dev |
|
|
213
215
|
| CI | Ubuntu × {Node 20, 22, 24} required + macOS advisory job |
|
package/dist/embed-db.d.ts
CHANGED
|
@@ -78,6 +78,27 @@ export declare class EmbedDb {
|
|
|
78
78
|
}): EmbedSearchHit[];
|
|
79
79
|
/** Total embedded chunks — used by stats / UI. */
|
|
80
80
|
totalChunks(): number;
|
|
81
|
+
/**
|
|
82
|
+
* v2.13.0 — return every (vector, row) pair for HNSW build. Caller
|
|
83
|
+
* is responsible for assigning sequential integer labels (we use
|
|
84
|
+
* `embeddings.id` since it's already a stable AUTOINCREMENT PK).
|
|
85
|
+
*
|
|
86
|
+
* Memory footprint: ~1.5 KB per row (384-dim Float32 + path string +
|
|
87
|
+
* preview). For 50K chunks: ~75 MB peak during build. Caller should
|
|
88
|
+
* release the array after building HNSW (we intentionally don't
|
|
89
|
+
* stream — HNSW build is 30s on 50K chunks anyway, the 75 MB is
|
|
90
|
+
* insignificant compared to the ONNX runtime + FTS5 working set).
|
|
91
|
+
*/
|
|
92
|
+
getAllVectors(): Array<{
|
|
93
|
+
label: number;
|
|
94
|
+
vector: Float32Array;
|
|
95
|
+
rel_path: string;
|
|
96
|
+
chunk_index: number;
|
|
97
|
+
line_start: number;
|
|
98
|
+
line_end: number;
|
|
99
|
+
text_preview: string;
|
|
100
|
+
kind: EmbedChunkKind;
|
|
101
|
+
}>;
|
|
81
102
|
}
|
|
82
103
|
/** Default location for the embed db, alongside the FTS5 db + parse cache. */
|
|
83
104
|
export declare function defaultEmbedDbFile(vaultHashPrefix: string): string;
|
package/dist/embed-db.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"embed-db.d.ts","sourceRoot":"","sources":["../src/embed-db.ts"],"names":[],"mappings":"AAsBA,6DAA6D;AAC7D,MAAM,MAAM,cAAc,GAAG,IAAI,GAAG,KAAK,CAAC;AAE1C,MAAM,WAAW,cAAc;IAC7B,QAAQ,EAAE,MAAM,CAAC;IACjB,WAAW,EAAE,MAAM,CAAC;IACpB,UAAU,EAAE,MAAM,CAAC;IACnB,QAAQ,EAAE,MAAM,CAAC;IACjB,mDAAmD;IACnD,YAAY,EAAE,MAAM,CAAC;IACrB,0EAA0E;IAC1E,KAAK,EAAE,MAAM,CAAC;IACd,0EAA0E;IAC1E,IAAI,EAAE,cAAc,CAAC;CACtB;AAED,MAAM,WAAW,eAAe;IAC9B,KAAK,EAAE,MAAM,CAAC;IACd,OAAO,EAAE,MAAM,CAAC;IAChB,OAAO,EAAE,MAAM,CAAC;IAChB,SAAS,EAAE,MAAM,CAAC;IAClB,YAAY,EAAE,MAAM,CAAC;CACtB;AAED,UAAU,cAAc;IACtB,QAAQ,EAAE,MAAM,CAAC;IACjB,QAAQ,EAAE,MAAM,CAAC;CAClB;AA+CD,MAAM,WAAW,cAAc;IAC7B,2CAA2C;IAC3C,IAAI,EAAE,MAAM,CAAC;IACb,sDAAsD;IACtD,SAAS,EAAE,MAAM,CAAC;IAClB,wEAAwE;IACxE,UAAU,EAAE,MAAM,CAAC;IACnB,oDAAoD;IACpD,GAAG,EAAE,MAAM,CAAC;CACb;AAED,qBAAa,OAAO;IAClB,OAAO,CAAC,EAAE,CAAmB;IAC7B,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAS;IAC9B,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAS;IACnC,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAS;IACpC,OAAO,CAAC,QAAQ,CAAC,GAAG,CAAS;gBAEjB,IAAI,EAAE,cAAc;IAO1B,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;IAc3B,0DAA0D;IACpD,WAAW,IAAI,OAAO,CAAC,OAAO,CAAC;IAcrC,KAAK,IAAI,IAAI;IAOb,OAAO,CAAC,eAAe;IAuDvB,OAAO,CAAC,QAAQ;IAQhB,OAAO,CAAC,SAAS;IAMjB,OAAO,CAAC,SAAS;IAKjB;;;;OAIG;IACH,UAAU,CACR,OAAO,EAAE,MAAM,EACf,OAAO,EAAE,MAAM,EACf,MAAM,EAAE,aAAa,CAAC;QACpB,UAAU,EAAE,MAAM,CAAC;QACnB,SAAS,EAAE,MAAM,CAAC;QAClB,OAAO,EAAE,MAAM,CAAC;QAChB,WAAW,EAAE,MAAM,CAAC;QACpB,MAAM,EAAE,YAAY,CAAC;KACtB,CAAC,EACF,IAAI,GAAE,cAAqB,GAC1B,IAAI;IAkCP,iEAAiE;IACjE,UAAU,CAAC,OAAO,EAAE,MAAM,GAAG,IAAI;IAMjC;;;;;OAKG;IACH,eAAe,CAAC,IAAI,CAAC,EAAE,cAAc,GAAG,cAAc,EAAE;IAQxD;;gDAE4C;IAC5C,MAAM,CAAC,QAAQ,EAAE,YAAY,EAAE,CAAC,EAAE,MAAM,EAAE,IAAI,GAAE;QAAE,MAAM,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAA;KAAO,GAAG,cAAc,EAAE;IA8D9G,kDAAkD;IAClD,WAAW,IAAI,MAAM;
|
|
1
|
+
{"version":3,"file":"embed-db.d.ts","sourceRoot":"","sources":["../src/embed-db.ts"],"names":[],"mappings":"AAsBA,6DAA6D;AAC7D,MAAM,MAAM,cAAc,GAAG,IAAI,GAAG,KAAK,CAAC;AAE1C,MAAM,WAAW,cAAc;IAC7B,QAAQ,EAAE,MAAM,CAAC;IACjB,WAAW,EAAE,MAAM,CAAC;IACpB,UAAU,EAAE,MAAM,CAAC;IACnB,QAAQ,EAAE,MAAM,CAAC;IACjB,mDAAmD;IACnD,YAAY,EAAE,MAAM,CAAC;IACrB,0EAA0E;IAC1E,KAAK,EAAE,MAAM,CAAC;IACd,0EAA0E;IAC1E,IAAI,EAAE,cAAc,CAAC;CACtB;AAED,MAAM,WAAW,eAAe;IAC9B,KAAK,EAAE,MAAM,CAAC;IACd,OAAO,EAAE,MAAM,CAAC;IAChB,OAAO,EAAE,MAAM,CAAC;IAChB,SAAS,EAAE,MAAM,CAAC;IAClB,YAAY,EAAE,MAAM,CAAC;CACtB;AAED,UAAU,cAAc;IACtB,QAAQ,EAAE,MAAM,CAAC;IACjB,QAAQ,EAAE,MAAM,CAAC;CAClB;AA+CD,MAAM,WAAW,cAAc;IAC7B,2CAA2C;IAC3C,IAAI,EAAE,MAAM,CAAC;IACb,sDAAsD;IACtD,SAAS,EAAE,MAAM,CAAC;IAClB,wEAAwE;IACxE,UAAU,EAAE,MAAM,CAAC;IACnB,oDAAoD;IACpD,GAAG,EAAE,MAAM,CAAC;CACb;AAED,qBAAa,OAAO;IAClB,OAAO,CAAC,EAAE,CAAmB;IAC7B,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAS;IAC9B,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAS;IACnC,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAS;IACpC,OAAO,CAAC,QAAQ,CAAC,GAAG,CAAS;gBAEjB,IAAI,EAAE,cAAc;IAO1B,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;IAc3B,0DAA0D;IACpD,WAAW,IAAI,OAAO,CAAC,OAAO,CAAC;IAcrC,KAAK,IAAI,IAAI;IAOb,OAAO,CAAC,eAAe;IAuDvB,OAAO,CAAC,QAAQ;IAQhB,OAAO,CAAC,SAAS;IAMjB,OAAO,CAAC,SAAS;IAKjB;;;;OAIG;IACH,UAAU,CACR,OAAO,EAAE,MAAM,EACf,OAAO,EAAE,MAAM,EACf,MAAM,EAAE,aAAa,CAAC;QACpB,UAAU,EAAE,MAAM,CAAC;QACnB,SAAS,EAAE,MAAM,CAAC;QAClB,OAAO,EAAE,MAAM,CAAC;QAChB,WAAW,EAAE,MAAM,CAAC;QACpB,MAAM,EAAE,YAAY,CAAC;KACtB,CAAC,EACF,IAAI,GAAE,cAAqB,GAC1B,IAAI;IAkCP,iEAAiE;IACjE,UAAU,CAAC,OAAO,EAAE,MAAM,GAAG,IAAI;IAMjC;;;;;OAKG;IACH,eAAe,CAAC,IAAI,CAAC,EAAE,cAAc,GAAG,cAAc,EAAE;IAQxD;;gDAE4C;IAC5C,MAAM,CAAC,QAAQ,EAAE,YAAY,EAAE,CAAC,EAAE,MAAM,EAAE,IAAI,GAAE;QAAE,MAAM,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAA;KAAO,GAAG,cAAc,EAAE;IA8D9G,kDAAkD;IAClD,WAAW,IAAI,MAAM;IAMrB;;;;;;;;;;OAUG;IACH,aAAa,IAAI,KAAK,CAAC;QACrB,KAAK,EAAE,MAAM,CAAC;QACd,MAAM,EAAE,YAAY,CAAC;QACrB,QAAQ,EAAE,MAAM,CAAC;QACjB,WAAW,EAAE,MAAM,CAAC;QACpB,UAAU,EAAE,MAAM,CAAC;QACnB,QAAQ,EAAE,MAAM,CAAC;QACjB,YAAY,EAAE,MAAM,CAAC;QACrB,IAAI,EAAE,cAAc,CAAC;KACtB,CAAC;CA4CH;AAED,8EAA8E;AAC9E,wBAAgB,kBAAkB,CAAC,eAAe,EAAE,MAAM,GAAG,MAAM,CAIlE"}
|
package/dist/embed-db.js
CHANGED
|
@@ -258,6 +258,50 @@ export class EmbedDb {
|
|
|
258
258
|
const row = db.prepare("SELECT COUNT(*) AS n FROM embeddings").get();
|
|
259
259
|
return row?.n ?? 0;
|
|
260
260
|
}
|
|
261
|
+
/**
|
|
262
|
+
* v2.13.0 — return every (vector, row) pair for HNSW build. Caller
|
|
263
|
+
* is responsible for assigning sequential integer labels (we use
|
|
264
|
+
* `embeddings.id` since it's already a stable AUTOINCREMENT PK).
|
|
265
|
+
*
|
|
266
|
+
* Memory footprint: ~1.5 KB per row (384-dim Float32 + path string +
|
|
267
|
+
* preview). For 50K chunks: ~75 MB peak during build. Caller should
|
|
268
|
+
* release the array after building HNSW (we intentionally don't
|
|
269
|
+
* stream — HNSW build is 30s on 50K chunks anyway, the 75 MB is
|
|
270
|
+
* insignificant compared to the ONNX runtime + FTS5 working set).
|
|
271
|
+
*/
|
|
272
|
+
getAllVectors() {
|
|
273
|
+
const db = this.requireDb();
|
|
274
|
+
const rows = db
|
|
275
|
+
.prepare("SELECT id, rel_path, chunk_index, line_start, line_end, text_preview, vector, kind FROM embeddings")
|
|
276
|
+
.all();
|
|
277
|
+
const expectedBytes = this.dim * 4;
|
|
278
|
+
const out = [];
|
|
279
|
+
for (const r of rows) {
|
|
280
|
+
// Match the corruption guard from search() — skip rows with
|
|
281
|
+
// mis-sized vectors so a partial DB doesn't poison the HNSW build.
|
|
282
|
+
if (r.vector.byteLength !== expectedBytes) {
|
|
283
|
+
process.stderr.write(`enquire: skipping ${r.rel_path}#${r.chunk_index} during getAllVectors — vector has ${r.vector.byteLength}B, expected ${expectedBytes}B (dim=${this.dim}). Run \`enquire-mcp clear-embeddings\` and rebuild.\n`);
|
|
284
|
+
continue;
|
|
285
|
+
}
|
|
286
|
+
// Copy (don't share the underlying buffer). HNSW takes ownership of
|
|
287
|
+
// the Float32Array slice for the lifetime of the index; sharing
|
|
288
|
+
// the SQLite row buffer would risk use-after-free if the row is
|
|
289
|
+
// GC'd or the cursor advances.
|
|
290
|
+
const vec = new Float32Array(this.dim);
|
|
291
|
+
vec.set(new Float32Array(r.vector.buffer, r.vector.byteOffset, this.dim));
|
|
292
|
+
out.push({
|
|
293
|
+
label: r.id,
|
|
294
|
+
vector: vec,
|
|
295
|
+
rel_path: r.rel_path,
|
|
296
|
+
chunk_index: r.chunk_index,
|
|
297
|
+
line_start: r.line_start,
|
|
298
|
+
line_end: r.line_end,
|
|
299
|
+
text_preview: r.text_preview,
|
|
300
|
+
kind: (r.kind === "pdf" ? "pdf" : "md")
|
|
301
|
+
});
|
|
302
|
+
}
|
|
303
|
+
return out;
|
|
304
|
+
}
|
|
261
305
|
}
|
|
262
306
|
/** Default location for the embed db, alongside the FTS5 db + parse cache. */
|
|
263
307
|
export function defaultEmbedDbFile(vaultHashPrefix) {
|
package/dist/embed-db.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"embed-db.js","sourceRoot":"","sources":["../src/embed-db.ts"],"names":[],"mappings":"AAAA,0EAA0E;AAC1E,6EAA6E;AAC7E,gFAAgF;AAChF,kDAAkD;AAClD,EAAE;AACF,gCAAgC;AAChC,gDAAgD;AAChD,0CAA0C;AAC1C,gFAAgF;AAChF,2DAA2D;AAC3D,EAAE;AACF,+EAA+E;AAC/E,2EAA2E;AAE3E,OAAO,EAAE,QAAQ,IAAI,EAAE,EAAE,MAAM,SAAS,CAAC;AACzC,OAAO,KAAK,IAAI,MAAM,WAAW,CAAC;AAElC,MAAM,cAAc,GAAG,CAAC,CAAC;AAkCzB,0EAA0E;AAC1E,yEAAyE;AACzE,yEAAyE;AACzE,IAAI,gBAAgB,GAA2C,IAAI,CAAC;AACpE,KAAK,UAAU,gBAAgB;IAC7B,IAAI,gBAAgB;QAAE,OAAO,gBAAgB,CAAC;IAC9C,IAAI,CAAC;QACH,MAAM,GAAG,GAAG,CAAC,MAAM,MAAM,CAAC,gBAAgB,CAAC,CAAgD,CAAC;QAC5F,MAAM,IAAI,GAAG,GAAG,CAAC,OAAO,CAAC;QACzB,IAAI,CAAC,IAAI;YAAE,MAAM,IAAI,KAAK,CAAC,sCAAsC,CAAC,CAAC;QACnE,IAAI,CAAC;YACH,MAAM,KAAK,GAAG,IAAI,IAAI,CAAC,UAAU,CAA2B,CAAC;YAC7D,KAAK,CAAC,KAAK,EAAE,EAAE,CAAC;QAClB,CAAC;QAAC,OAAO,QAAQ,EAAE,CAAC;YAClB,MAAM,IAAI,KAAK,CACb,+IAA+I,QAAQ,YAAY,KAAK,CAAC,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,QAAQ,CAAC,EAAE,CACjN,CAAC;QACJ,CAAC;QACD,gBAAgB,GAAG,IAAI,CAAC;QACxB,OAAO,IAAI,CAAC;IACd,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,MAAM,IAAI,KAAK,CACb,8HACE,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CACjD,EAAE,CACH,CAAC;IACJ,CAAC;AACH,CAAC;AA6BD,MAAM,OAAO,OAAO;IACV,EAAE,GAAc,IAAI,CAAC;IACZ,IAAI,CAAS;IACb,SAAS,CAAS;IAClB,UAAU,CAAS;IACnB,GAAG,CAAS;IAE7B,YAAY,IAAoB;QAC9B,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC;QACtB,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC,SAAS,CAAC;QAChC,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,UAAU,CAAC;QAClC,IAAI,CAAC,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC;IACtB,CAAC;IAED,KAAK,CAAC,IAAI;QACR,IAAI,IAAI,CAAC,EAAE;YAAE,OAAO;QACpB,MAAM,IAAI,GAAG,MAAM,gBAAgB,EAAE,CAAC;QACtC,MAAM,EAAE,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,IAAI,EAAE,KAAK,EAAE,CAAC,CAAC;QAC1E,MAAM,EAAE,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,KAAK,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,GAAE,CAAC,CAAC,CAAC;QAC/D,IAAI,CAAC,EAAE,GAAG,IAAI,IAAI,CAAC,IAAI,CAAC,IAAI,CAAO,CAAC;QACpC,IAAI,CAAC,EAAE,CAAC,MAAM,CAAC,oBAAoB,CAAC,CAAC;QACrC,IAAI,CAAC,EAAE,CAAC,MAAM,CAAC,sBAAsB,CAAC,CAAC;QACvC,IAAI,CAAC,eAAe,EAAE,CAAC;QACvB,MAAM,OAAO,CAAC,GAAG,CACf,CAAC,IAAI,CAAC,IAAI,EAAE,GAAG,IAAI,CAAC,IAAI,MAAM,EAAE,GAAG,IAAI,CAAC,IAAI,MAAM,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,EAAE,CAAC,KAAK,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,GAAE,CAAC,CAAC,CAAC,CACnG,CAAC;IACJ,CAAC;IAED,0DAA0D;IAC1D,KAAK,CAAC,WAAW;QACf,IAAI,CAAC,KAAK,EAAE,CAAC;QACb,IAAI,OAAO,GAAG,KAAK,CAAC;QACpB,KAAK,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,GAAG,IAAI,CAAC,IAAI,MAAM,EAAE,GAAG,IAAI,CAAC,IAAI,MAAM,CAAC,EAAE,CAAC;YACpE,IAAI,CAAC;gBACH,MAAM,EAAE,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC;gBACnB,OAAO,GAAG,IAAI,CAAC;YACjB,CAAC;YAAC,MAAM,CAAC;gBACP,kBAAkB;YACpB,CAAC;QACH,CAAC;QACD,OAAO,OAAO,CAAC;IACjB,CAAC;IAED,KAAK;QACH,IAAI,IAAI,CAAC,EAAE,EAAE,CAAC;YACZ,IAAI,CAAC,EAAE,CAAC,KAAK,EAAE,CAAC;YAChB,IAAI,CAAC,EAAE,GAAG,IAAI,CAAC;QACjB,CAAC;IACH,CAAC;IAEO,eAAe;QACrB,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAE5B,EAAE,CAAC,IAAI,CAAC;;;;;KAKP,CAAC,CAAC;QAEH,MAAM,IAAI,GAAG,IAAI,CAAC,QAAQ,EAAE,CAAC;QAC7B,MAAM,YAAY,GAAG,IAAI,CAAC,cAAc,KAAK,SAAS,IAAI,IAAI,CAAC,cAAc,KAAK,MAAM,CAAC,cAAc,CAAC,CAAC;QACzG,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,KAAK,SAAS,IAAI,IAAI,CAAC,UAAU,KAAK,IAAI,CAAC,SAAS,CAAC;QACtF,MAAM,UAAU,GAAG,IAAI,CAAC,WAAW,KAAK,SAAS,IAAI,IAAI,CAAC,WAAW,KAAK,IAAI,CAAC,UAAU,CAAC;QAC1F,MAAM,QAAQ,GAAG,IAAI,CAAC,GAAG,KAAK,SAAS,IAAI,IAAI,CAAC,GAAG,KAAK,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QACzE,IAAI,CAAC,YAAY,IAAI,CAAC,SAAS,IAAI,CAAC,UAAU,IAAI,CAAC,QAAQ,EAAE,CAAC;YAC5D,MAAM,MAAM,GAAa,EAAE,CAAC;YAC5B,IAAI,CAAC,YAAY;gBAAE,MAAM,CAAC,IAAI,CAAC,kBAAkB,IAAI,CAAC,cAAc,MAAM,cAAc,EAAE,CAAC,CAAC;YAC5F,IAAI,CAAC,SAAS;gBAAE,MAAM,CAAC,IAAI,CAAC,cAAc,IAAI,CAAC,UAAU,MAAM,IAAI,CAAC,SAAS,EAAE,CAAC,CAAC;YACjF,IAAI,CAAC,UAAU;gBAAE,MAAM,CAAC,IAAI,CAAC,SAAS,IAAI,CAAC,WAAW,MAAM,IAAI,CAAC,UAAU,EAAE,CAAC,CAAC;YAC/E,IAAI,CAAC,QAAQ;gBAAE,MAAM,CAAC,IAAI,CAAC,OAAO,IAAI,CAAC,GAAG,MAAM,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;YAC5D,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,oCAAoC,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;YACjF,EAAE,CAAC,IAAI,CAAC,qEAAqE,CAAC,CAAC;QACjF,CAAC;QAED,EAAE,CAAC,IAAI,CAAC;;;;;;;;;;;;;;;;;;;;KAoBP,CAAC,CAAC;QAEH,IAAI,CAAC,SAAS,CAAC;YACb,cAAc,EAAE,MAAM,CAAC,cAAc,CAAC;YACtC,UAAU,EAAE,IAAI,CAAC,SAAS;YAC1B,WAAW,EAAE,IAAI,CAAC,UAAU;YAC5B,GAAG,EAAE,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC;SACtB,CAAC,CAAC;IACL,CAAC;IAEO,QAAQ;QACd,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,MAAM,IAAI,GAAG,EAAE,CAAC,OAAO,CAAC,6BAA6B,CAAC,CAAC,GAAG,EAAkC,CAAC;QAC7F,MAAM,GAAG,GAA2B,EAAE,CAAC;QACvC,KAAK,MAAM,CAAC,IAAI,IAAI;YAAE,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC;QAC3C,OAAO,GAAG,CAAC;IACb,CAAC;IAEO,SAAS,CAAC,EAA0B;QAC1C,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,MAAM,IAAI,GAAG,EAAE,CAAC,OAAO,CAAC,wDAAwD,CAAC,CAAC;QAClF,KAAK,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,EAAE,CAAC;YAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IAC1D,CAAC;IAEO,SAAS;QACf,IAAI,CAAC,IAAI,CAAC,EAAE;YAAE,MAAM,IAAI,KAAK,CAAC,0CAA0C,CAAC,CAAC;QAC1E,OAAO,IAAI,CAAC,EAAE,CAAC;IACjB,CAAC;IAED;;;;OAIG;IACH,UAAU,CACR,OAAe,EACf,OAAe,EACf,MAME,EACF,OAAuB,IAAI;QAE3B,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC;QACrB,MAAM,EAAE,GAAG,EAAE,CAAC,WAAW,CAAC,CAAC,GAAG,IAAe,EAAE,EAAE;YAC/C,MAAM,IAAI,GAAG,IAAI,CAAC,CAAC,CAAkB,CAAC;YACtC,EAAE,CAAC,OAAO,CAAC,2CAA2C,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;YACrE,MAAM,MAAM,GAAG,EAAE,CAAC,OAAO,CACvB;sCAC8B,CAC/B,CAAC;YACF,KAAK,MAAM,CAAC,IAAI,IAAI,EAAE,CAAC;gBACrB,IAAI,CAAC,CAAC,MAAM,CAAC,MAAM,KAAK,GAAG,EAAE,CAAC;oBAC5B,MAAM,IAAI,KAAK,CACb,2BAA2B,OAAO,UAAU,CAAC,CAAC,UAAU,SAAS,CAAC,CAAC,MAAM,CAAC,MAAM,cAAc,GAAG,EAAE,CACpG,CAAC;gBACJ,CAAC;gBACD,MAAM,CAAC,GAAG,CACR,OAAO,EACP,CAAC,CAAC,UAAU,EACZ,CAAC,CAAC,SAAS,EACX,CAAC,CAAC,OAAO,EACT,CAAC,CAAC,WAAW,EACb,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC,MAAM,CAAC,UAAU,EAAE,CAAC,CAAC,MAAM,CAAC,UAAU,CAAC,EACtE,IAAI,CACL,CAAC;YACJ,CAAC;YACD,EAAE,CAAC,OAAO,CACR;8CACsC,CACvC,CAAC,GAAG,CAAC,OAAO,EAAE,OAAO,EAAE,IAAI,CAAC,MAAM,EAAE,IAAI,CAAC,CAAC;QAC7C,CAAC,CAAC,CAAC;QACH,EAAE,CAAC,MAAM,CAAC,CAAC;IACb,CAAC;IAED,iEAAiE;IACjE,UAAU,CAAC,OAAe;QACxB,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,EAAE,CAAC,OAAO,CAAC,2CAA2C,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;QACrE,EAAE,CAAC,OAAO,CAAC,6CAA6C,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;IACzE,CAAC;IAED;;;;;OAKG;IACH,eAAe,CAAC,IAAqB;QACnC,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,IAAI,IAAI,KAAK,SAAS,EAAE,CAAC;YACvB,OAAO,EAAE,CAAC,OAAO,CAAC,4DAA4D,CAAC,CAAC,GAAG,CAAiB,IAAI,CAAC,CAAC;QAC5G,CAAC;QACD,OAAO,EAAE,CAAC,OAAO,CAAC,6CAA6C,CAAC,CAAC,GAAG,EAAkB,CAAC;IACzF,CAAC;IAED;;gDAE4C;IAC5C,MAAM,CAAC,QAAsB,EAAE,CAAS,EAAE,OAA+C,EAAE;QACzF,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,IAAI,QAAQ,CAAC,MAAM,KAAK,IAAI,CAAC,GAAG,EAAE,CAAC;YACjC,MAAM,IAAI,KAAK,CAAC,kCAAkC,QAAQ,CAAC,MAAM,cAAc,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;QAC7F,CAAC;QACD,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,IAAI,CAAC,QAAQ,CAAC;QAC5C,MAAM,YAAY,GAAG,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC;QAEhF,yEAAyE;QACzE,uEAAuE;QACvE,yEAAyE;QACzE,gCAAgC;QAChC,MAAM,IAAI,GAAG,EAAE;aACZ,OAAO,CACN,YAAY;YACV,CAAC,CAAC;8DACkD;YACpD,CAAC,CAAC,gGAAgG,CACrG;aACA,GAAG,CAQD,GAAG,CAAC,YAAY,CAAC,CAAC,CAAC,CAAC,YAAY,CAAC,MAAM,EAAE,YAAY,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;QAEnE,MAAM,aAAa,GAAG,IAAI,CAAC,GAAG,GAAG,CAAC,CAAC,CAAC,oBAAoB;QACxD,MAAM,IAAI,GAAqB,EAAE,CAAC;QAClC,KAAK,MAAM,CAAC,IAAI,IAAI,EAAE,CAAC;YACrB,uEAAuE;YACvE,qEAAqE;YACrE,qEAAqE;YACrE,gEAAgE;YAChE,IAAI,CAAC,CAAC,MAAM,CAAC,UAAU,KAAK,aAAa,EAAE,CAAC;gBAC1C,OAAO,CAAC,MAAM,CAAC,KAAK,CAClB,qBAAqB,CAAC,CAAC,QAAQ,IAAI,CAAC,CAAC,WAAW,iBAAiB,CAAC,CAAC,MAAM,CAAC,UAAU,eAAe,aAAa,UAAU,IAAI,CAAC,GAAG,wDAAwD,CAC3L,CAAC;gBACF,SAAS;YACX,CAAC;YACD,MAAM,GAAG,GAAG,IAAI,YAAY,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC,MAAM,CAAC,UAAU,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC;YAC7E,IAAI,KAAK,GAAG,CAAC,CAAC;YACd,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC,EAAE,EAAE,CAAC;gBAClC,KAAK,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC;YAC9C,CAAC;YACD,IAAI,KAAK,GAAG,QAAQ;gBAAE,SAAS;YAC/B,IAAI,CAAC,IAAI,CAAC;gBACR,QAAQ,EAAE,CAAC,CAAC,QAAQ;gBACpB,WAAW,EAAE,CAAC,CAAC,WAAW;gBAC1B,UAAU,EAAE,CAAC,CAAC,UAAU;gBACxB,QAAQ,EAAE,CAAC,CAAC,QAAQ;gBACpB,YAAY,EAAE,CAAC,CAAC,YAAY;gBAC5B,KAAK;gBACL,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,CAAmB;aAC1D,CAAC,CAAC;QACL,CAAC;QACD,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC;QACvC,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IAC1B,CAAC;IAED,kDAAkD;IAClD,WAAW;QACT,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,MAAM,GAAG,GAAG,EAAE,CAAC,OAAO,CAAC,sCAAsC,CAAC,CAAC,GAAG,EAAiB,CAAC;QACpF,OAAO,GAAG,EAAE,CAAC,IAAI,CAAC,CAAC;IACrB,CAAC;CACF;AAED,8EAA8E;AAC9E,MAAM,UAAU,kBAAkB,CAAC,eAAuB;IACxD,4EAA4E;IAC5E,wEAAwE;IACxE,OAAO,GAAG,eAAe,WAAW,CAAC;AACvC,CAAC"}
|
|
1
|
+
{"version":3,"file":"embed-db.js","sourceRoot":"","sources":["../src/embed-db.ts"],"names":[],"mappings":"AAAA,0EAA0E;AAC1E,6EAA6E;AAC7E,gFAAgF;AAChF,kDAAkD;AAClD,EAAE;AACF,gCAAgC;AAChC,gDAAgD;AAChD,0CAA0C;AAC1C,gFAAgF;AAChF,2DAA2D;AAC3D,EAAE;AACF,+EAA+E;AAC/E,2EAA2E;AAE3E,OAAO,EAAE,QAAQ,IAAI,EAAE,EAAE,MAAM,SAAS,CAAC;AACzC,OAAO,KAAK,IAAI,MAAM,WAAW,CAAC;AAElC,MAAM,cAAc,GAAG,CAAC,CAAC;AAkCzB,0EAA0E;AAC1E,yEAAyE;AACzE,yEAAyE;AACzE,IAAI,gBAAgB,GAA2C,IAAI,CAAC;AACpE,KAAK,UAAU,gBAAgB;IAC7B,IAAI,gBAAgB;QAAE,OAAO,gBAAgB,CAAC;IAC9C,IAAI,CAAC;QACH,MAAM,GAAG,GAAG,CAAC,MAAM,MAAM,CAAC,gBAAgB,CAAC,CAAgD,CAAC;QAC5F,MAAM,IAAI,GAAG,GAAG,CAAC,OAAO,CAAC;QACzB,IAAI,CAAC,IAAI;YAAE,MAAM,IAAI,KAAK,CAAC,sCAAsC,CAAC,CAAC;QACnE,IAAI,CAAC;YACH,MAAM,KAAK,GAAG,IAAI,IAAI,CAAC,UAAU,CAA2B,CAAC;YAC7D,KAAK,CAAC,KAAK,EAAE,EAAE,CAAC;QAClB,CAAC;QAAC,OAAO,QAAQ,EAAE,CAAC;YAClB,MAAM,IAAI,KAAK,CACb,+IAA+I,QAAQ,YAAY,KAAK,CAAC,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,QAAQ,CAAC,EAAE,CACjN,CAAC;QACJ,CAAC;QACD,gBAAgB,GAAG,IAAI,CAAC;QACxB,OAAO,IAAI,CAAC;IACd,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,MAAM,IAAI,KAAK,CACb,8HACE,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CACjD,EAAE,CACH,CAAC;IACJ,CAAC;AACH,CAAC;AA6BD,MAAM,OAAO,OAAO;IACV,EAAE,GAAc,IAAI,CAAC;IACZ,IAAI,CAAS;IACb,SAAS,CAAS;IAClB,UAAU,CAAS;IACnB,GAAG,CAAS;IAE7B,YAAY,IAAoB;QAC9B,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC;QACtB,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC,SAAS,CAAC;QAChC,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,UAAU,CAAC;QAClC,IAAI,CAAC,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC;IACtB,CAAC;IAED,KAAK,CAAC,IAAI;QACR,IAAI,IAAI,CAAC,EAAE;YAAE,OAAO;QACpB,MAAM,IAAI,GAAG,MAAM,gBAAgB,EAAE,CAAC;QACtC,MAAM,EAAE,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,IAAI,EAAE,KAAK,EAAE,CAAC,CAAC;QAC1E,MAAM,EAAE,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,KAAK,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,GAAE,CAAC,CAAC,CAAC;QAC/D,IAAI,CAAC,EAAE,GAAG,IAAI,IAAI,CAAC,IAAI,CAAC,IAAI,CAAO,CAAC;QACpC,IAAI,CAAC,EAAE,CAAC,MAAM,CAAC,oBAAoB,CAAC,CAAC;QACrC,IAAI,CAAC,EAAE,CAAC,MAAM,CAAC,sBAAsB,CAAC,CAAC;QACvC,IAAI,CAAC,eAAe,EAAE,CAAC;QACvB,MAAM,OAAO,CAAC,GAAG,CACf,CAAC,IAAI,CAAC,IAAI,EAAE,GAAG,IAAI,CAAC,IAAI,MAAM,EAAE,GAAG,IAAI,CAAC,IAAI,MAAM,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,EAAE,CAAC,KAAK,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,GAAE,CAAC,CAAC,CAAC,CACnG,CAAC;IACJ,CAAC;IAED,0DAA0D;IAC1D,KAAK,CAAC,WAAW;QACf,IAAI,CAAC,KAAK,EAAE,CAAC;QACb,IAAI,OAAO,GAAG,KAAK,CAAC;QACpB,KAAK,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,GAAG,IAAI,CAAC,IAAI,MAAM,EAAE,GAAG,IAAI,CAAC,IAAI,MAAM,CAAC,EAAE,CAAC;YACpE,IAAI,CAAC;gBACH,MAAM,EAAE,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC;gBACnB,OAAO,GAAG,IAAI,CAAC;YACjB,CAAC;YAAC,MAAM,CAAC;gBACP,kBAAkB;YACpB,CAAC;QACH,CAAC;QACD,OAAO,OAAO,CAAC;IACjB,CAAC;IAED,KAAK;QACH,IAAI,IAAI,CAAC,EAAE,EAAE,CAAC;YACZ,IAAI,CAAC,EAAE,CAAC,KAAK,EAAE,CAAC;YAChB,IAAI,CAAC,EAAE,GAAG,IAAI,CAAC;QACjB,CAAC;IACH,CAAC;IAEO,eAAe;QACrB,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAE5B,EAAE,CAAC,IAAI,CAAC;;;;;KAKP,CAAC,CAAC;QAEH,MAAM,IAAI,GAAG,IAAI,CAAC,QAAQ,EAAE,CAAC;QAC7B,MAAM,YAAY,GAAG,IAAI,CAAC,cAAc,KAAK,SAAS,IAAI,IAAI,CAAC,cAAc,KAAK,MAAM,CAAC,cAAc,CAAC,CAAC;QACzG,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,KAAK,SAAS,IAAI,IAAI,CAAC,UAAU,KAAK,IAAI,CAAC,SAAS,CAAC;QACtF,MAAM,UAAU,GAAG,IAAI,CAAC,WAAW,KAAK,SAAS,IAAI,IAAI,CAAC,WAAW,KAAK,IAAI,CAAC,UAAU,CAAC;QAC1F,MAAM,QAAQ,GAAG,IAAI,CAAC,GAAG,KAAK,SAAS,IAAI,IAAI,CAAC,GAAG,KAAK,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QACzE,IAAI,CAAC,YAAY,IAAI,CAAC,SAAS,IAAI,CAAC,UAAU,IAAI,CAAC,QAAQ,EAAE,CAAC;YAC5D,MAAM,MAAM,GAAa,EAAE,CAAC;YAC5B,IAAI,CAAC,YAAY;gBAAE,MAAM,CAAC,IAAI,CAAC,kBAAkB,IAAI,CAAC,cAAc,MAAM,cAAc,EAAE,CAAC,CAAC;YAC5F,IAAI,CAAC,SAAS;gBAAE,MAAM,CAAC,IAAI,CAAC,cAAc,IAAI,CAAC,UAAU,MAAM,IAAI,CAAC,SAAS,EAAE,CAAC,CAAC;YACjF,IAAI,CAAC,UAAU;gBAAE,MAAM,CAAC,IAAI,CAAC,SAAS,IAAI,CAAC,WAAW,MAAM,IAAI,CAAC,UAAU,EAAE,CAAC,CAAC;YAC/E,IAAI,CAAC,QAAQ;gBAAE,MAAM,CAAC,IAAI,CAAC,OAAO,IAAI,CAAC,GAAG,MAAM,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;YAC5D,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,oCAAoC,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;YACjF,EAAE,CAAC,IAAI,CAAC,qEAAqE,CAAC,CAAC;QACjF,CAAC;QAED,EAAE,CAAC,IAAI,CAAC;;;;;;;;;;;;;;;;;;;;KAoBP,CAAC,CAAC;QAEH,IAAI,CAAC,SAAS,CAAC;YACb,cAAc,EAAE,MAAM,CAAC,cAAc,CAAC;YACtC,UAAU,EAAE,IAAI,CAAC,SAAS;YAC1B,WAAW,EAAE,IAAI,CAAC,UAAU;YAC5B,GAAG,EAAE,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC;SACtB,CAAC,CAAC;IACL,CAAC;IAEO,QAAQ;QACd,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,MAAM,IAAI,GAAG,EAAE,CAAC,OAAO,CAAC,6BAA6B,CAAC,CAAC,GAAG,EAAkC,CAAC;QAC7F,MAAM,GAAG,GAA2B,EAAE,CAAC;QACvC,KAAK,MAAM,CAAC,IAAI,IAAI;YAAE,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC;QAC3C,OAAO,GAAG,CAAC;IACb,CAAC;IAEO,SAAS,CAAC,EAA0B;QAC1C,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,MAAM,IAAI,GAAG,EAAE,CAAC,OAAO,CAAC,wDAAwD,CAAC,CAAC;QAClF,KAAK,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,EAAE,CAAC;YAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IAC1D,CAAC;IAEO,SAAS;QACf,IAAI,CAAC,IAAI,CAAC,EAAE;YAAE,MAAM,IAAI,KAAK,CAAC,0CAA0C,CAAC,CAAC;QAC1E,OAAO,IAAI,CAAC,EAAE,CAAC;IACjB,CAAC;IAED;;;;OAIG;IACH,UAAU,CACR,OAAe,EACf,OAAe,EACf,MAME,EACF,OAAuB,IAAI;QAE3B,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC;QACrB,MAAM,EAAE,GAAG,EAAE,CAAC,WAAW,CAAC,CAAC,GAAG,IAAe,EAAE,EAAE;YAC/C,MAAM,IAAI,GAAG,IAAI,CAAC,CAAC,CAAkB,CAAC;YACtC,EAAE,CAAC,OAAO,CAAC,2CAA2C,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;YACrE,MAAM,MAAM,GAAG,EAAE,CAAC,OAAO,CACvB;sCAC8B,CAC/B,CAAC;YACF,KAAK,MAAM,CAAC,IAAI,IAAI,EAAE,CAAC;gBACrB,IAAI,CAAC,CAAC,MAAM,CAAC,MAAM,KAAK,GAAG,EAAE,CAAC;oBAC5B,MAAM,IAAI,KAAK,CACb,2BAA2B,OAAO,UAAU,CAAC,CAAC,UAAU,SAAS,CAAC,CAAC,MAAM,CAAC,MAAM,cAAc,GAAG,EAAE,CACpG,CAAC;gBACJ,CAAC;gBACD,MAAM,CAAC,GAAG,CACR,OAAO,EACP,CAAC,CAAC,UAAU,EACZ,CAAC,CAAC,SAAS,EACX,CAAC,CAAC,OAAO,EACT,CAAC,CAAC,WAAW,EACb,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC,MAAM,CAAC,UAAU,EAAE,CAAC,CAAC,MAAM,CAAC,UAAU,CAAC,EACtE,IAAI,CACL,CAAC;YACJ,CAAC;YACD,EAAE,CAAC,OAAO,CACR;8CACsC,CACvC,CAAC,GAAG,CAAC,OAAO,EAAE,OAAO,EAAE,IAAI,CAAC,MAAM,EAAE,IAAI,CAAC,CAAC;QAC7C,CAAC,CAAC,CAAC;QACH,EAAE,CAAC,MAAM,CAAC,CAAC;IACb,CAAC;IAED,iEAAiE;IACjE,UAAU,CAAC,OAAe;QACxB,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,EAAE,CAAC,OAAO,CAAC,2CAA2C,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;QACrE,EAAE,CAAC,OAAO,CAAC,6CAA6C,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;IACzE,CAAC;IAED;;;;;OAKG;IACH,eAAe,CAAC,IAAqB;QACnC,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,IAAI,IAAI,KAAK,SAAS,EAAE,CAAC;YACvB,OAAO,EAAE,CAAC,OAAO,CAAC,4DAA4D,CAAC,CAAC,GAAG,CAAiB,IAAI,CAAC,CAAC;QAC5G,CAAC;QACD,OAAO,EAAE,CAAC,OAAO,CAAC,6CAA6C,CAAC,CAAC,GAAG,EAAkB,CAAC;IACzF,CAAC;IAED;;gDAE4C;IAC5C,MAAM,CAAC,QAAsB,EAAE,CAAS,EAAE,OAA+C,EAAE;QACzF,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,IAAI,QAAQ,CAAC,MAAM,KAAK,IAAI,CAAC,GAAG,EAAE,CAAC;YACjC,MAAM,IAAI,KAAK,CAAC,kCAAkC,QAAQ,CAAC,MAAM,cAAc,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;QAC7F,CAAC;QACD,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,IAAI,CAAC,QAAQ,CAAC;QAC5C,MAAM,YAAY,GAAG,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC;QAEhF,yEAAyE;QACzE,uEAAuE;QACvE,yEAAyE;QACzE,gCAAgC;QAChC,MAAM,IAAI,GAAG,EAAE;aACZ,OAAO,CACN,YAAY;YACV,CAAC,CAAC;8DACkD;YACpD,CAAC,CAAC,gGAAgG,CACrG;aACA,GAAG,CAQD,GAAG,CAAC,YAAY,CAAC,CAAC,CAAC,CAAC,YAAY,CAAC,MAAM,EAAE,YAAY,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;QAEnE,MAAM,aAAa,GAAG,IAAI,CAAC,GAAG,GAAG,CAAC,CAAC,CAAC,oBAAoB;QACxD,MAAM,IAAI,GAAqB,EAAE,CAAC;QAClC,KAAK,MAAM,CAAC,IAAI,IAAI,EAAE,CAAC;YACrB,uEAAuE;YACvE,qEAAqE;YACrE,qEAAqE;YACrE,gEAAgE;YAChE,IAAI,CAAC,CAAC,MAAM,CAAC,UAAU,KAAK,aAAa,EAAE,CAAC;gBAC1C,OAAO,CAAC,MAAM,CAAC,KAAK,CAClB,qBAAqB,CAAC,CAAC,QAAQ,IAAI,CAAC,CAAC,WAAW,iBAAiB,CAAC,CAAC,MAAM,CAAC,UAAU,eAAe,aAAa,UAAU,IAAI,CAAC,GAAG,wDAAwD,CAC3L,CAAC;gBACF,SAAS;YACX,CAAC;YACD,MAAM,GAAG,GAAG,IAAI,YAAY,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC,MAAM,CAAC,UAAU,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC;YAC7E,IAAI,KAAK,GAAG,CAAC,CAAC;YACd,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC,EAAE,EAAE,CAAC;gBAClC,KAAK,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC;YAC9C,CAAC;YACD,IAAI,KAAK,GAAG,QAAQ;gBAAE,SAAS;YAC/B,IAAI,CAAC,IAAI,CAAC;gBACR,QAAQ,EAAE,CAAC,CAAC,QAAQ;gBACpB,WAAW,EAAE,CAAC,CAAC,WAAW;gBAC1B,UAAU,EAAE,CAAC,CAAC,UAAU;gBACxB,QAAQ,EAAE,CAAC,CAAC,QAAQ;gBACpB,YAAY,EAAE,CAAC,CAAC,YAAY;gBAC5B,KAAK;gBACL,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,CAAmB;aAC1D,CAAC,CAAC;QACL,CAAC;QACD,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC;QACvC,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IAC1B,CAAC;IAED,kDAAkD;IAClD,WAAW;QACT,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,MAAM,GAAG,GAAG,EAAE,CAAC,OAAO,CAAC,sCAAsC,CAAC,CAAC,GAAG,EAAiB,CAAC;QACpF,OAAO,GAAG,EAAE,CAAC,IAAI,CAAC,CAAC;IACrB,CAAC;IAED;;;;;;;;;;OAUG;IACH,aAAa;QAUX,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,MAAM,IAAI,GAAG,EAAE;aACZ,OAAO,CAAC,oGAAoG,CAAC;aAC7G,GAAG,EASA,CAAC;QACP,MAAM,aAAa,GAAG,IAAI,CAAC,GAAG,GAAG,CAAC,CAAC;QACnC,MAAM,GAAG,GAAyC,EAAE,CAAC;QACrD,KAAK,MAAM,CAAC,IAAI,IAAI,EAAE,CAAC;YACrB,4DAA4D;YAC5D,mEAAmE;YACnE,IAAI,CAAC,CAAC,MAAM,CAAC,UAAU,KAAK,aAAa,EAAE,CAAC;gBAC1C,OAAO,CAAC,MAAM,CAAC,KAAK,CAClB,qBAAqB,CAAC,CAAC,QAAQ,IAAI,CAAC,CAAC,WAAW,sCAAsC,CAAC,CAAC,MAAM,CAAC,UAAU,eAAe,aAAa,UAAU,IAAI,CAAC,GAAG,wDAAwD,CAChN,CAAC;gBACF,SAAS;YACX,CAAC;YACD,oEAAoE;YACpE,gEAAgE;YAChE,gEAAgE;YAChE,+BAA+B;YAC/B,MAAM,GAAG,GAAG,IAAI,YAAY,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;YACvC,GAAG,CAAC,GAAG,CAAC,IAAI,YAAY,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC,MAAM,CAAC,UAAU,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC;YAC1E,GAAG,CAAC,IAAI,CAAC;gBACP,KAAK,EAAE,CAAC,CAAC,EAAE;gBACX,MAAM,EAAE,GAAG;gBACX,QAAQ,EAAE,CAAC,CAAC,QAAQ;gBACpB,WAAW,EAAE,CAAC,CAAC,WAAW;gBAC1B,UAAU,EAAE,CAAC,CAAC,UAAU;gBACxB,QAAQ,EAAE,CAAC,CAAC,QAAQ;gBACpB,YAAY,EAAE,CAAC,CAAC,YAAY;gBAC5B,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,CAAmB;aAC1D,CAAC,CAAC;QACL,CAAC;QACD,OAAO,GAAG,CAAC;IACb,CAAC;CACF;AAED,8EAA8E;AAC9E,MAAM,UAAU,kBAAkB,CAAC,eAAuB;IACxD,4EAA4E;IAC5E,wEAAwE;IACxE,OAAO,GAAG,eAAe,WAAW,CAAC;AACvC,CAAC"}
|
package/dist/hnsw.d.ts
ADDED
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
import type { EmbedSearchHit } from "./embed-db.js";
|
|
2
|
+
/** A single labeled vector — used to populate the index. */
|
|
3
|
+
export interface LabeledVector {
|
|
4
|
+
/** Stable identifier — lets the search code recover the source row from the EmbedDb. */
|
|
5
|
+
label: number;
|
|
6
|
+
/** L2-normalized vector. Caller is responsible for the normalization. */
|
|
7
|
+
vector: Float32Array;
|
|
8
|
+
}
|
|
9
|
+
/** Build-time HNSW parameters. Defaults tuned for 384-dim cosine on PKM data. */
|
|
10
|
+
export interface HnswBuildOptions {
|
|
11
|
+
/** Embedding dimensionality (must match the corpus). */
|
|
12
|
+
dim: number;
|
|
13
|
+
/** Maximum elements (caller's count of vectors); enables index pre-sizing. */
|
|
14
|
+
maxElements: number;
|
|
15
|
+
/**
|
|
16
|
+
* Number of bidirectional links per node. Higher M = better recall but
|
|
17
|
+
* more memory + slower build. Default 16 (Malkov & Yashunin, 2018, §4.1).
|
|
18
|
+
*/
|
|
19
|
+
m?: number;
|
|
20
|
+
/**
|
|
21
|
+
* Beam width during build. Higher efConstruction = better recall,
|
|
22
|
+
* slower build, no query-time cost. Default 200.
|
|
23
|
+
*/
|
|
24
|
+
efConstruction?: number;
|
|
25
|
+
/** Seed for build-time randomization (reproducibility in tests). */
|
|
26
|
+
seed?: number;
|
|
27
|
+
}
|
|
28
|
+
/** Per-query parameters. */
|
|
29
|
+
export interface HnswQueryOptions {
|
|
30
|
+
/**
|
|
31
|
+
* Beam width during search. Higher = more accurate, slower. Default 100.
|
|
32
|
+
* Must be ≥ k. Common range: 50-500.
|
|
33
|
+
*/
|
|
34
|
+
ef?: number;
|
|
35
|
+
}
|
|
36
|
+
/**
|
|
37
|
+
* In-memory HNSW index over L2-normalized cosine vectors. Built once on
|
|
38
|
+
* serve start from `EmbedDb.getAllVectors()`; queried per
|
|
39
|
+
* `obsidian_search` / `obsidian_embeddings_search` invocation.
|
|
40
|
+
*/
|
|
41
|
+
export interface HnswIndex {
|
|
42
|
+
/** Vector dimensionality. */
|
|
43
|
+
readonly dim: number;
|
|
44
|
+
/** Number of points currently in the index. */
|
|
45
|
+
readonly size: number;
|
|
46
|
+
/**
|
|
47
|
+
* k-NN search. Returns labels + distances (cosine distance, smaller =
|
|
48
|
+
* more similar). Caller maps labels back to source rows via the same
|
|
49
|
+
* `LabeledVector.label` they used at build time.
|
|
50
|
+
*/
|
|
51
|
+
searchKnn(queryVec: Float32Array, k: number, opts?: HnswQueryOptions): {
|
|
52
|
+
labels: number[];
|
|
53
|
+
distances: number[];
|
|
54
|
+
};
|
|
55
|
+
}
|
|
56
|
+
/**
|
|
57
|
+
* Build a fresh in-memory HNSW from labeled vectors.
|
|
58
|
+
*
|
|
59
|
+
* `vectors` must be L2-normalized — the cosine distance space treats
|
|
60
|
+
* inputs as already-unit-length, so unnormalized inputs produce wrong
|
|
61
|
+
* distances. The `EmbedDb` already L2-normalizes at insert time, so the
|
|
62
|
+
* usual call path (loadAllVectors → buildHnsw) is safe by construction.
|
|
63
|
+
*
|
|
64
|
+
* Throws if `dim` doesn't match any vector's length, if `maxElements`
|
|
65
|
+
* is less than the input count, or if `hnswlib-wasm` failed to load.
|
|
66
|
+
*/
|
|
67
|
+
export declare function buildHnsw(vectors: ReadonlyArray<LabeledVector>, opts: HnswBuildOptions): Promise<HnswIndex>;
|
|
68
|
+
/**
|
|
69
|
+
* Convert HNSW search results to EmbedSearchHit using a label → source-row
|
|
70
|
+
* lookup. The label was assigned by the caller at build time (typically
|
|
71
|
+
* `EmbedDb.getAllVectors()` returns rows with sequential integer labels);
|
|
72
|
+
* we just reverse the mapping. Distance → cosine similarity: cosine
|
|
73
|
+
* distance is `1 - cosine_similarity`, so we flip back here so callers
|
|
74
|
+
* can compare HNSW + brute-force scores apples-to-apples.
|
|
75
|
+
*/
|
|
76
|
+
export declare function hnswResultsToHits(result: {
|
|
77
|
+
labels: number[];
|
|
78
|
+
distances: number[];
|
|
79
|
+
}, rowByLabel: ReadonlyMap<number, {
|
|
80
|
+
rel_path: string;
|
|
81
|
+
chunk_index: number;
|
|
82
|
+
line_start: number;
|
|
83
|
+
line_end: number;
|
|
84
|
+
text_preview: string;
|
|
85
|
+
kind: "md" | "pdf";
|
|
86
|
+
}>): EmbedSearchHit[];
|
|
87
|
+
//# sourceMappingURL=hnsw.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"hnsw.d.ts","sourceRoot":"","sources":["../src/hnsw.ts"],"names":[],"mappings":"AA0CA,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,eAAe,CAAC;AAEpD,4DAA4D;AAC5D,MAAM,WAAW,aAAa;IAC5B,wFAAwF;IACxF,KAAK,EAAE,MAAM,CAAC;IACd,yEAAyE;IACzE,MAAM,EAAE,YAAY,CAAC;CACtB;AAED,iFAAiF;AACjF,MAAM,WAAW,gBAAgB;IAC/B,wDAAwD;IACxD,GAAG,EAAE,MAAM,CAAC;IACZ,8EAA8E;IAC9E,WAAW,EAAE,MAAM,CAAC;IACpB;;;OAGG;IACH,CAAC,CAAC,EAAE,MAAM,CAAC;IACX;;;OAGG;IACH,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,oEAAoE;IACpE,IAAI,CAAC,EAAE,MAAM,CAAC;CACf;AAED,4BAA4B;AAC5B,MAAM,WAAW,gBAAgB;IAC/B;;;OAGG;IACH,EAAE,CAAC,EAAE,MAAM,CAAC;CACb;AAED;;;;GAIG;AACH,MAAM,WAAW,SAAS;IACxB,6BAA6B;IAC7B,QAAQ,CAAC,GAAG,EAAE,MAAM,CAAC;IACrB,+CAA+C;IAC/C,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB;;;;OAIG;IACH,SAAS,CAAC,QAAQ,EAAE,YAAY,EAAE,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE,gBAAgB,GAAG;QAAE,MAAM,EAAE,MAAM,EAAE,CAAC;QAAC,SAAS,EAAE,MAAM,EAAE,CAAA;KAAE,CAAC;CAClH;AAgDD;;;;;;;;;;GAUG;AACH,wBAAsB,SAAS,CAAC,OAAO,EAAE,aAAa,CAAC,aAAa,CAAC,EAAE,IAAI,EAAE,gBAAgB,GAAG,OAAO,CAAC,SAAS,CAAC,CAmDjH;AAED;;;;;;;GAOG;AACH,wBAAgB,iBAAiB,CAC/B,MAAM,EAAE;IAAE,MAAM,EAAE,MAAM,EAAE,CAAC;IAAC,SAAS,EAAE,MAAM,EAAE,CAAA;CAAE,EACjD,UAAU,EAAE,WAAW,CACrB,MAAM,EACN;IACE,QAAQ,EAAE,MAAM,CAAC;IACjB,WAAW,EAAE,MAAM,CAAC;IACpB,UAAU,EAAE,MAAM,CAAC;IACnB,QAAQ,EAAE,MAAM,CAAC;IACjB,YAAY,EAAE,MAAM,CAAC;IACrB,IAAI,EAAE,IAAI,GAAG,KAAK,CAAC;CACpB,CACF,GACA,cAAc,EAAE,CAsBlB"}
|
package/dist/hnsw.js
ADDED
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
// HNSW (Hierarchical Navigable Small World) vector index for enquire-mcp.
|
|
2
|
+
//
|
|
3
|
+
// v2.13.0 — closes the "brute-force semantic search doesn't scale" gap. The
|
|
4
|
+
// existing path in `EmbedDb.search()` runs O(n) cosine over every embedded
|
|
5
|
+
// chunk per query (~5ms at 8K chunks, ~30ms at 50K, ~300ms at 500K, ~3s at
|
|
6
|
+
// 5M). HNSW is the IR-standard graph-based index that achieves O(log n)
|
|
7
|
+
// approximate nearest neighbor lookups — sub-10ms even at million-chunk
|
|
8
|
+
// scale, with recall@K ≥ 95% at default parameters (M=16, efConstruction=200).
|
|
9
|
+
//
|
|
10
|
+
// Architecture: in-memory rebuild on serve start.
|
|
11
|
+
//
|
|
12
|
+
// Why not persistent?
|
|
13
|
+
// • `hnswlib-wasm` writes through Emscripten's virtual FS; persisting
|
|
14
|
+
// to disk + restoring requires syncing the WASM FS to host disk. The
|
|
15
|
+
// plumbing isn't bad but it's another file to manage (WAL-style
|
|
16
|
+
// consistency: which version of .embed.db produced the .hnsw.bin?).
|
|
17
|
+
// • For typical vault scales (≤50K chunks), rebuild is ≤30s on serve
|
|
18
|
+
// start — tolerable as a one-time boot cost for a long-running server.
|
|
19
|
+
// • Persistence is tracked for v3.0+ when million-chunk vaults become
|
|
20
|
+
// a real use case. For now: simple in-memory keeps the surface clean.
|
|
21
|
+
//
|
|
22
|
+
// Native dep: `hnswlib-node@^3.0` (Node-N-API binding to the C++ hnswlib
|
|
23
|
+
// reference impl). Maintained by yoshoku since 2022, stable since v3.0
|
|
24
|
+
// (March 2024). Ships prebuilds for darwin-x64/arm64 + linux-x64/arm64
|
|
25
|
+
// + win32-x64; falls back to source build (requires C++ toolchain) on
|
|
26
|
+
// uncommon platforms. Lazy-loaded — same `optionalDependencies` pattern
|
|
27
|
+
// as tesseract.js / pdfjs-dist / @huggingface/transformers.
|
|
28
|
+
//
|
|
29
|
+
// Why not hnswlib-wasm? It exists (~340 KB pure-WASM) but its v0.8
|
|
30
|
+
// build is hardcoded for the browser environment (ENVIRONMENT_IS_WEB=
|
|
31
|
+
// true at compile time) and refuses to load under Node. hnswlib-node
|
|
32
|
+
// is the production-grade choice for server-side vault retrieval.
|
|
33
|
+
//
|
|
34
|
+
// Performance characteristics on M1 Pro (cosine space, dim=384):
|
|
35
|
+
// • Build: ~0.5ms per vector → 8K chunks ≈ 4s, 50K ≈ 25s, 500K ≈ 4min
|
|
36
|
+
// • Query: ~0.5-1ms per top-10 lookup, independent of corpus size
|
|
37
|
+
//
|
|
38
|
+
// Recall@10 vs brute-force on the same corpus is consistently ≥98% at
|
|
39
|
+
// default params. Users tuning for max recall can pass `--hnsw-ef-search`
|
|
40
|
+
// to widen the search beam (default 100; higher = more accurate,
|
|
41
|
+
// slower).
|
|
42
|
+
let cachedModule = null;
|
|
43
|
+
async function loadHnswlib() {
|
|
44
|
+
if (cachedModule)
|
|
45
|
+
return cachedModule;
|
|
46
|
+
try {
|
|
47
|
+
const mod = (await import("hnswlib-node"));
|
|
48
|
+
// hnswlib-node ships as CJS with a default export; ESM consumers get
|
|
49
|
+
// both `.default` and the named exports. Try both.
|
|
50
|
+
const lib = mod.default ?? mod;
|
|
51
|
+
if (typeof lib.HierarchicalNSW !== "function") {
|
|
52
|
+
throw new Error("hnswlib-node has no HierarchicalNSW export — package mismatch");
|
|
53
|
+
}
|
|
54
|
+
cachedModule = lib;
|
|
55
|
+
return cachedModule;
|
|
56
|
+
}
|
|
57
|
+
catch (err) {
|
|
58
|
+
const msg = err instanceof Error ? err.message : String(err);
|
|
59
|
+
throw new Error("enquire: hnswlib-node (optional dependency) is not available. HNSW requires it. " +
|
|
60
|
+
`Install with: npm install hnswlib-node@^3 (or reinstall enquire-mcp without --omit=optional). ` +
|
|
61
|
+
`Underlying error: ${msg}`);
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
/**
|
|
65
|
+
* Build a fresh in-memory HNSW from labeled vectors.
|
|
66
|
+
*
|
|
67
|
+
* `vectors` must be L2-normalized — the cosine distance space treats
|
|
68
|
+
* inputs as already-unit-length, so unnormalized inputs produce wrong
|
|
69
|
+
* distances. The `EmbedDb` already L2-normalizes at insert time, so the
|
|
70
|
+
* usual call path (loadAllVectors → buildHnsw) is safe by construction.
|
|
71
|
+
*
|
|
72
|
+
* Throws if `dim` doesn't match any vector's length, if `maxElements`
|
|
73
|
+
* is less than the input count, or if `hnswlib-wasm` failed to load.
|
|
74
|
+
*/
|
|
75
|
+
export async function buildHnsw(vectors, opts) {
|
|
76
|
+
const dim = opts.dim;
|
|
77
|
+
if (vectors.length > opts.maxElements) {
|
|
78
|
+
throw new Error(`buildHnsw: vectors.length=${vectors.length} exceeds maxElements=${opts.maxElements}; pre-size the index`);
|
|
79
|
+
}
|
|
80
|
+
const m = opts.m ?? 16;
|
|
81
|
+
const efConstruction = opts.efConstruction ?? 200;
|
|
82
|
+
const seed = opts.seed ?? 100;
|
|
83
|
+
// Validate first — fail fast before pulling in the WASM module.
|
|
84
|
+
for (let i = 0; i < vectors.length; i++) {
|
|
85
|
+
const v = vectors[i];
|
|
86
|
+
if (!v)
|
|
87
|
+
continue;
|
|
88
|
+
if (v.vector.length !== dim) {
|
|
89
|
+
throw new Error(`buildHnsw: vector at index ${i} has dim ${v.vector.length}, expected ${dim}`);
|
|
90
|
+
}
|
|
91
|
+
}
|
|
92
|
+
const lib = await loadHnswlib();
|
|
93
|
+
const ctor = new lib.HierarchicalNSW("cosine", dim);
|
|
94
|
+
// Pre-size the index. `m=16` and `efConstruction=200` are HNSW defaults
|
|
95
|
+
// (Malkov & Yashunin, 2018) and produce ≥98% recall@10 vs brute-force on
|
|
96
|
+
// typical PKM corpora.
|
|
97
|
+
ctor.initIndex(Math.max(opts.maxElements, 1), m, efConstruction, seed);
|
|
98
|
+
for (let i = 0; i < vectors.length; i++) {
|
|
99
|
+
const v = vectors[i];
|
|
100
|
+
if (!v)
|
|
101
|
+
continue;
|
|
102
|
+
// hnswlib-node accepts plain number[] (it copies into its own C++
|
|
103
|
+
// buffer internally). Float32Array.from-via-Array.from would allocate
|
|
104
|
+
// an intermediate; we use a plain spread which is fast and explicit.
|
|
105
|
+
ctor.addPoint(Array.from(v.vector), v.label);
|
|
106
|
+
}
|
|
107
|
+
return {
|
|
108
|
+
dim,
|
|
109
|
+
size: vectors.length,
|
|
110
|
+
searchKnn(queryVec, k, qOpts) {
|
|
111
|
+
if (queryVec.length !== dim) {
|
|
112
|
+
throw new Error(`HnswIndex.searchKnn: query dim ${queryVec.length} ≠ index dim ${dim}`);
|
|
113
|
+
}
|
|
114
|
+
// ef must be ≥ k; the underlying lib enforces this but we surface a
|
|
115
|
+
// friendlier error if the caller forgets.
|
|
116
|
+
const ef = Math.max(qOpts?.ef ?? 100, k);
|
|
117
|
+
ctor.setEf(ef);
|
|
118
|
+
const result = ctor.searchKnn(Array.from(queryVec), k, undefined);
|
|
119
|
+
return { labels: result.neighbors, distances: result.distances };
|
|
120
|
+
}
|
|
121
|
+
};
|
|
122
|
+
}
|
|
123
|
+
/**
|
|
124
|
+
* Convert HNSW search results to EmbedSearchHit using a label → source-row
|
|
125
|
+
* lookup. The label was assigned by the caller at build time (typically
|
|
126
|
+
* `EmbedDb.getAllVectors()` returns rows with sequential integer labels);
|
|
127
|
+
* we just reverse the mapping. Distance → cosine similarity: cosine
|
|
128
|
+
* distance is `1 - cosine_similarity`, so we flip back here so callers
|
|
129
|
+
* can compare HNSW + brute-force scores apples-to-apples.
|
|
130
|
+
*/
|
|
131
|
+
export function hnswResultsToHits(result, rowByLabel) {
|
|
132
|
+
const hits = [];
|
|
133
|
+
for (let i = 0; i < result.labels.length; i++) {
|
|
134
|
+
const label = result.labels[i];
|
|
135
|
+
const distance = result.distances[i];
|
|
136
|
+
if (label === undefined || distance === undefined)
|
|
137
|
+
continue;
|
|
138
|
+
const row = rowByLabel.get(label);
|
|
139
|
+
if (!row)
|
|
140
|
+
continue; // race: row deleted between build and query — skip
|
|
141
|
+
// hnswlib-wasm cosine distance = 1 - cosine_similarity.
|
|
142
|
+
// Convert back so callers can compare against brute-force scores.
|
|
143
|
+
const score = 1 - distance;
|
|
144
|
+
hits.push({
|
|
145
|
+
rel_path: row.rel_path,
|
|
146
|
+
chunk_index: row.chunk_index,
|
|
147
|
+
line_start: row.line_start,
|
|
148
|
+
line_end: row.line_end,
|
|
149
|
+
text_preview: row.text_preview,
|
|
150
|
+
score,
|
|
151
|
+
kind: row.kind
|
|
152
|
+
});
|
|
153
|
+
}
|
|
154
|
+
return hits;
|
|
155
|
+
}
|
|
156
|
+
//# sourceMappingURL=hnsw.js.map
|
package/dist/hnsw.js.map
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"hnsw.js","sourceRoot":"","sources":["../src/hnsw.ts"],"names":[],"mappings":"AAAA,0EAA0E;AAC1E,EAAE;AACF,4EAA4E;AAC5E,2EAA2E;AAC3E,2EAA2E;AAC3E,wEAAwE;AACxE,wEAAwE;AACxE,+EAA+E;AAC/E,EAAE;AACF,kDAAkD;AAClD,EAAE;AACF,sBAAsB;AACtB,wEAAwE;AACxE,yEAAyE;AACzE,oEAAoE;AACpE,wEAAwE;AACxE,uEAAuE;AACvE,2EAA2E;AAC3E,wEAAwE;AACxE,0EAA0E;AAC1E,EAAE;AACF,yEAAyE;AACzE,uEAAuE;AACvE,uEAAuE;AACvE,sEAAsE;AACtE,wEAAwE;AACxE,4DAA4D;AAC5D,EAAE;AACF,mEAAmE;AACnE,sEAAsE;AACtE,qEAAqE;AACrE,kEAAkE;AAClE,EAAE;AACF,iEAAiE;AACjE,wEAAwE;AACxE,oEAAoE;AACpE,EAAE;AACF,sEAAsE;AACtE,0EAA0E;AAC1E,iEAAiE;AACjE,WAAW;AAkFX,IAAI,YAAY,GAA6B,IAAI,CAAC;AAClD,KAAK,UAAU,WAAW;IACxB,IAAI,YAAY;QAAE,OAAO,YAAY,CAAC;IACtC,IAAI,CAAC;QACH,MAAM,GAAG,GAAG,CAAC,MAAM,MAAM,CAAC,cAAc,CAAC,CAAiE,CAAC;QAC3G,qEAAqE;QACrE,mDAAmD;QACnD,MAAM,GAAG,GAAG,GAAG,CAAC,OAAO,IAAK,GAAyB,CAAC;QACtD,IAAI,OAAO,GAAG,CAAC,eAAe,KAAK,UAAU,EAAE,CAAC;YAC9C,MAAM,IAAI,KAAK,CAAC,+DAA+D,CAAC,CAAC;QACnF,CAAC;QACD,YAAY,GAAG,GAAG,CAAC;QACnB,OAAO,YAAY,CAAC;IACtB,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,MAAM,GAAG,GAAG,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;QAC7D,MAAM,IAAI,KAAK,CACb,kFAAkF;YAChF,gGAAgG;YAChG,qBAAqB,GAAG,EAAE,CAC7B,CAAC;IACJ,CAAC;AACH,CAAC;AAED;;;;;;;;;;GAUG;AACH,MAAM,CAAC,KAAK,UAAU,SAAS,CAAC,OAAqC,EAAE,IAAsB;IAC3F,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC;IACrB,IAAI,OAAO,CAAC,MAAM,GAAG,IAAI,CAAC,WAAW,EAAE,CAAC;QACtC,MAAM,IAAI,KAAK,CACb,6BAA6B,OAAO,CAAC,MAAM,wBAAwB,IAAI,CAAC,WAAW,sBAAsB,CAC1G,CAAC;IACJ,CAAC;IACD,MAAM,CAAC,GAAG,IAAI,CAAC,CAAC,IAAI,EAAE,CAAC;IACvB,MAAM,cAAc,GAAG,IAAI,CAAC,cAAc,IAAI,GAAG,CAAC;IAClD,MAAM,IAAI,GAAG,IAAI,CAAC,IAAI,IAAI,GAAG,CAAC;IAE9B,gEAAgE;IAChE,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QACxC,MAAM,CAAC,GAAG,OAAO,CAAC,CAAC,CAAC,CAAC;QACrB,IAAI,CAAC,CAAC;YAAE,SAAS;QACjB,IAAI,CAAC,CAAC,MAAM,CAAC,MAAM,KAAK,GAAG,EAAE,CAAC;YAC5B,MAAM,IAAI,KAAK,CAAC,8BAA8B,CAAC,YAAY,CAAC,CAAC,MAAM,CAAC,MAAM,cAAc,GAAG,EAAE,CAAC,CAAC;QACjG,CAAC;IACH,CAAC;IAED,MAAM,GAAG,GAAG,MAAM,WAAW,EAAE,CAAC;IAChC,MAAM,IAAI,GAAG,IAAI,GAAG,CAAC,eAAe,CAAC,QAAQ,EAAE,GAAG,CAAC,CAAC;IACpD,wEAAwE;IACxE,yEAAyE;IACzE,uBAAuB;IACvB,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,WAAW,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,cAAc,EAAE,IAAI,CAAC,CAAC;IAEvE,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QACxC,MAAM,CAAC,GAAG,OAAO,CAAC,CAAC,CAAC,CAAC;QACrB,IAAI,CAAC,CAAC;YAAE,SAAS;QACjB,kEAAkE;QAClE,sEAAsE;QACtE,qEAAqE;QACrE,IAAI,CAAC,QAAQ,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC;IAC/C,CAAC;IAED,OAAO;QACL,GAAG;QACH,IAAI,EAAE,OAAO,CAAC,MAAM;QACpB,SAAS,CAAC,QAAsB,EAAE,CAAS,EAAE,KAAwB;YACnE,IAAI,QAAQ,CAAC,MAAM,KAAK,GAAG,EAAE,CAAC;gBAC5B,MAAM,IAAI,KAAK,CAAC,kCAAkC,QAAQ,CAAC,MAAM,gBAAgB,GAAG,EAAE,CAAC,CAAC;YAC1F,CAAC;YACD,oEAAoE;YACpE,0CAA0C;YAC1C,MAAM,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,KAAK,EAAE,EAAE,IAAI,GAAG,EAAE,CAAC,CAAC,CAAC;YACzC,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;YACf,MAAM,MAAM,GAAG,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,IAAI,CAAC,QAAQ,CAAC,EAAE,CAAC,EAAE,SAAS,CAAC,CAAC;YAClE,OAAO,EAAE,MAAM,EAAE,MAAM,CAAC,SAAS,EAAE,SAAS,EAAE,MAAM,CAAC,SAAS,EAAE,CAAC;QACnE,CAAC;KACF,CAAC;AACJ,CAAC;AAED;;;;;;;GAOG;AACH,MAAM,UAAU,iBAAiB,CAC/B,MAAiD,EACjD,UAUC;IAED,MAAM,IAAI,GAAqB,EAAE,CAAC;IAClC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,MAAM,CAAC,MAAM,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QAC9C,MAAM,KAAK,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC;QAC/B,MAAM,QAAQ,GAAG,MAAM,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC;QACrC,IAAI,KAAK,KAAK,SAAS,IAAI,QAAQ,KAAK,SAAS;YAAE,SAAS;QAC5D,MAAM,GAAG,GAAG,UAAU,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC;QAClC,IAAI,CAAC,GAAG;YAAE,SAAS,CAAC,mDAAmD;QACvE,wDAAwD;QACxD,kEAAkE;QAClE,MAAM,KAAK,GAAG,CAAC,GAAG,QAAQ,CAAC;QAC3B,IAAI,CAAC,IAAI,CAAC;YACR,QAAQ,EAAE,GAAG,CAAC,QAAQ;YACtB,WAAW,EAAE,GAAG,CAAC,WAAW;YAC5B,UAAU,EAAE,GAAG,CAAC,UAAU;YAC1B,QAAQ,EAAE,GAAG,CAAC,QAAQ;YACtB,YAAY,EAAE,GAAG,CAAC,YAAY;YAC9B,KAAK;YACL,IAAI,EAAE,GAAG,CAAC,IAAI;SACf,CAAC,CAAC;IACL,CAAC;IACD,OAAO,IAAI,CAAC;AACd,CAAC"}
|
package/dist/index.d.ts
CHANGED
|
@@ -30,6 +30,13 @@ export interface ServeOptions {
|
|
|
30
30
|
rerankerModel?: string;
|
|
31
31
|
/** v2.9.0 — how many top fused candidates to rerank (default 50). */
|
|
32
32
|
rerankerTopN?: string;
|
|
33
|
+
/** v2.13.0 — build an in-memory HNSW vector index on serve start.
|
|
34
|
+
* Off by default; rebuild cost ~25s for 50K chunks. Sub-10ms top-K
|
|
35
|
+
* per query thereafter, vs O(n) brute-force without it. Defers
|
|
36
|
+
* persistence to v3.0. */
|
|
37
|
+
useHnsw?: boolean;
|
|
38
|
+
/** v2.13.0 — HNSW search-time beam width (default 100; ≥k). */
|
|
39
|
+
hnswEf?: string;
|
|
33
40
|
}
|
|
34
41
|
declare function main(): Promise<void>;
|
|
35
42
|
/**
|
|
@@ -52,6 +59,26 @@ export interface ServerDeps {
|
|
|
52
59
|
warningTracker: {
|
|
53
60
|
printed: boolean;
|
|
54
61
|
};
|
|
62
|
+
/**
|
|
63
|
+
* v2.13.0 — opt-in HNSW vector index built in-memory on serve start
|
|
64
|
+
* from the embed-db rows. Sub-10ms top-K queries vs O(n) brute-force.
|
|
65
|
+
* `null` when `--use-hnsw` wasn't passed or the embed-db doesn't exist.
|
|
66
|
+
*/
|
|
67
|
+
hnswContext: {
|
|
68
|
+
/** The HNSW index. */
|
|
69
|
+
index: import("./hnsw.js").HnswIndex;
|
|
70
|
+
/** Map from HNSW label (= embeddings.id) to source row metadata. */
|
|
71
|
+
rowByLabel: Map<number, {
|
|
72
|
+
rel_path: string;
|
|
73
|
+
chunk_index: number;
|
|
74
|
+
line_start: number;
|
|
75
|
+
line_end: number;
|
|
76
|
+
text_preview: string;
|
|
77
|
+
kind: "md" | "pdf";
|
|
78
|
+
}>;
|
|
79
|
+
/** Search-time beam width override; falls back to module default if undefined. */
|
|
80
|
+
ef?: number;
|
|
81
|
+
} | null;
|
|
55
82
|
}
|
|
56
83
|
/**
|
|
57
84
|
* One-time bootstrap of the heavy deps (vault open + FTS5 sync + watcher).
|
package/dist/index.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";AAIA,OAAO,EAAE,SAAS,EAAoB,MAAM,yCAAyC,CAAC;AAMtF,OAAO,EAAkC,QAAQ,EAAE,MAAM,WAAW,CAAC;AAyCrE,OAAO,EAAE,KAAK,EAAE,MAAM,YAAY,CAAC;AACnC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAW5C,MAAM,WAAW,YAAY;IAC3B,KAAK,EAAE,MAAM,CAAC;IACd,WAAW,CAAC,EAAE,OAAO,CAAC;IACtB,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,eAAe,CAAC,EAAE,OAAO,CAAC;IAC1B,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,eAAe,CAAC,EAAE,OAAO,CAAC;IAC1B,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,QAAQ,CAAC,EAAE,WAAW,GAAG,SAAS,CAAC;IACnC,WAAW,CAAC,EAAE,MAAM,EAAE,CAAC;IACvB,SAAS,CAAC,EAAE,MAAM,EAAE,CAAC;IACrB,KAAK,CAAC,EAAE,OAAO,CAAC;IAChB,aAAa,CAAC,EAAE,MAAM,EAAE,CAAC;IACzB,YAAY,CAAC,EAAE,MAAM,EAAE,CAAC;IACxB,qBAAqB,CAAC,EAAE,OAAO,CAAC;IAChC;;mCAE+B;IAC/B,WAAW,CAAC,EAAE,OAAO,CAAC;IACtB;8EAC0E;IAC1E,cAAc,CAAC,EAAE,OAAO,CAAC;IACzB,qEAAqE;IACrE,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB,qEAAqE;IACrE,YAAY,CAAC,EAAE,MAAM,CAAC;
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";AAIA,OAAO,EAAE,SAAS,EAAoB,MAAM,yCAAyC,CAAC;AAMtF,OAAO,EAAkC,QAAQ,EAAE,MAAM,WAAW,CAAC;AAyCrE,OAAO,EAAE,KAAK,EAAE,MAAM,YAAY,CAAC;AACnC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAW5C,MAAM,WAAW,YAAY;IAC3B,KAAK,EAAE,MAAM,CAAC;IACd,WAAW,CAAC,EAAE,OAAO,CAAC;IACtB,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,eAAe,CAAC,EAAE,OAAO,CAAC;IAC1B,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,eAAe,CAAC,EAAE,OAAO,CAAC;IAC1B,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,QAAQ,CAAC,EAAE,WAAW,GAAG,SAAS,CAAC;IACnC,WAAW,CAAC,EAAE,MAAM,EAAE,CAAC;IACvB,SAAS,CAAC,EAAE,MAAM,EAAE,CAAC;IACrB,KAAK,CAAC,EAAE,OAAO,CAAC;IAChB,aAAa,CAAC,EAAE,MAAM,EAAE,CAAC;IACzB,YAAY,CAAC,EAAE,MAAM,EAAE,CAAC;IACxB,qBAAqB,CAAC,EAAE,OAAO,CAAC;IAChC;;mCAE+B;IAC/B,WAAW,CAAC,EAAE,OAAO,CAAC;IACtB;8EAC0E;IAC1E,cAAc,CAAC,EAAE,OAAO,CAAC;IACzB,qEAAqE;IACrE,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB,qEAAqE;IACrE,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB;;;+BAG2B;IAC3B,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB,+DAA+D;IAC/D,MAAM,CAAC,EAAE,MAAM,CAAC;CACjB;AAcD,iBAAe,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC,CAklBnC;AAED;;;;;;;;;;GAUG;AACH,MAAM,WAAW,UAAU;IACzB,KAAK,EAAE,KAAK,CAAC;IACb,QAAQ,EAAE,QAAQ,GAAG,IAAI,CAAC;IAC1B,OAAO,EAAE,YAAY,GAAG,IAAI,CAAC;IAC7B,aAAa,EAAE,GAAG,CAAC,MAAM,CAAC,CAAC;IAC3B,YAAY,EAAE,GAAG,CAAC,MAAM,CAAC,CAAC;IAC1B,cAAc,EAAE;QAAE,OAAO,EAAE,OAAO,CAAA;KAAE,CAAC;IACrC;;;;OAIG;IACH,WAAW,EAAE;QACX,sBAAsB;QACtB,KAAK,EAAE,OAAO,WAAW,EAAE,SAAS,CAAC;QACrC,oEAAoE;QACpE,UAAU,EAAE,GAAG,CACb,MAAM,EACN;YACE,QAAQ,EAAE,MAAM,CAAC;YACjB,WAAW,EAAE,MAAM,CAAC;YACpB,UAAU,EAAE,MAAM,CAAC;YACnB,QAAQ,EAAE,MAAM,CAAC;YACjB,YAAY,EAAE,MAAM,CAAC;YACrB,IAAI,EAAE,IAAI,GAAG,KAAK,CAAC;SACpB,CACF,CAAC;QACF,kFAAkF;QAClF,EAAE,CAAC,EAAE,MAAM,CAAC;KACb,GAAG,IAAI,CAAC;CACV;AAED;;;;;GAKG;AACH,wBAAsB,iBAAiB,CAAC,IAAI,EAAE,YAAY,GAAG,OAAO,CAAC,UAAU,CAAC,CAgJ/E;AAED;;;;GAIG;AACH,wBAAgB,cAAc,CAAC,IAAI,EAAE,UAAU,EAAE,IAAI,EAAE,YAAY,GAAG,SAAS,CAqG9E;AAED,iBAAe,WAAW,CAAC,IAAI,EAAE,YAAY,GAAG,OAAO,CAAC,IAAI,CAAC,CAoD5D;AAED;;;;GAIG;AACH,wBAAgB,iBAAiB,CAAC,IAAI,EAAE,UAAU,GAAG,MAAM,CAY1D;AAg+DD,iBAAS,gBAAgB,CAAC,GAAG,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,GAAG,MAAM,CAM3D;AAsCD,OAAO,EAAE,IAAI,EAAE,gBAAgB,EAAE,WAAW,EAAE,CAAC"}
|