@oomkapwn/enquire-mcp 2.6.0 → 2.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +94 -0
- package/README.md +8 -5
- package/SECURITY.md +34 -0
- package/dist/embed-db.d.ts +17 -5
- package/dist/embed-db.d.ts.map +1 -1
- package/dist/embed-db.js +28 -14
- package/dist/embed-db.js.map +1 -1
- package/dist/fts5.d.ts +29 -2
- package/dist/fts5.d.ts.map +1 -1
- package/dist/fts5.js +51 -6
- package/dist/fts5.js.map +1 -1
- package/dist/index.d.ts +4 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +225 -7
- package/dist/index.js.map +1 -1
- package/dist/pdf.d.ts +51 -0
- package/dist/pdf.d.ts.map +1 -0
- package/dist/pdf.js +162 -0
- package/dist/pdf.js.map +1 -0
- package/dist/tools.d.ts +60 -0
- package/dist/tools.d.ts.map +1 -1
- package/dist/tools.js +111 -8
- package/dist/tools.js.map +1 -1
- package/docs/api.md +1 -1
- package/package.json +3 -2
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,100 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project will be documented here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
4
4
|
|
|
5
|
+
## [2.8.0] — 2026-05-08
|
|
6
|
+
|
|
7
|
+
**Sprint 8 — PDF retrieval integration.** v2.7.0 added PDF text-extraction tools (`obsidian_list_pdfs` / `obsidian_read_pdf`); v2.8.0 makes PDFs **first-class citizens of `obsidian_search`**. Index PDFs into the same FTS5 + embedding stores as markdown, blend them in hybrid retrieval (BM25 + TF-IDF + embeddings → RRF fusion), and surface a `kind: "md" | "pdf"` flag on every hit so agents can distinguish content sources at a glance.
|
|
8
|
+
|
|
9
|
+
### Added — `--include-pdfs` flag on `serve`, `index`, `build-embeddings`
|
|
10
|
+
|
|
11
|
+
Off by default — opt-in because PDF extraction is ~10-30× slower per file than markdown chunking. When enabled:
|
|
12
|
+
|
|
13
|
+
- `enquire-mcp serve --vault <path> --persistent-index --include-pdfs` → boots and incrementally syncs PDFs into the FTS5 index alongside markdown.
|
|
14
|
+
- `enquire-mcp index --vault <path> --include-pdfs` → cold-build / refresh the FTS5 index for both markdown and PDFs.
|
|
15
|
+
- `enquire-mcp build-embeddings --vault <path> --include-pdfs` → embed PDF chunks too.
|
|
16
|
+
|
|
17
|
+
Bad PDFs (encrypted without password / corrupt / image-only / scanned) are caught per-file and surfaced via stderr without taking down the markdown index path. Image-only / scanned PDFs are skipped with a clear log line — OCR is tracked for v2.9+ (Tesseract.js).
|
|
18
|
+
|
|
19
|
+
### Schema migration — FTS5 v3 → v4, embed-db v1 → v2
|
|
20
|
+
|
|
21
|
+
Both indexes added a `kind` column (`'md' | 'pdf'`, default `'md'`). Schema bump auto-rebuilds the index on first open after upgrade — same pattern as the `tokenize_mode` / `vault_root` cross-config-change guards. Existing markdown indexes are preserved (they re-sync from the markdown source as kind=md).
|
|
22
|
+
|
|
23
|
+
### `obsidian_search` returns `kind` on every hit
|
|
24
|
+
|
|
25
|
+
Both `note` and `block` granularity propagate the kind flag. PDF hits use the filename without the `.pdf` extension as the title (so titles read naturally in agent output). The tool description was updated to flag the v2.8.0 capability so MCP clients introspecting `tools/list` see it immediately.
|
|
26
|
+
|
|
27
|
+
### Page-citation markers in PDF chunks
|
|
28
|
+
|
|
29
|
+
When indexing PDFs, page boundaries are preserved as `[page: N]\n` markers in the joined text before chunking. The chunker may split a page across chunks or merge short pages, but the markers travel with the text — so search snippets carry page citations the agent can extract. Same `chunkContent` pipeline as markdown, so chunk identity matches across BM25 / TF-IDF / embeddings (RRF requires stable IDs).
|
|
30
|
+
|
|
31
|
+
### Independent sync paths via kind-aware diff()
|
|
32
|
+
|
|
33
|
+
`FtsIndex.diff(live, kind?)` and `EmbedDb.getSourceStates(kind?)` now accept an optional kind filter. Lets the markdown-sync path run independently from the PDF-sync path against the same DB without one's "missing files" being mistakenly deleted by the other. Backward-compat: omitting the kind arg returns all rows (legacy behavior).
|
|
34
|
+
|
|
35
|
+
### Tests
|
|
36
|
+
|
|
37
|
+
493 unit tests pass (was 481 in v2.7.0, +12 new):
|
|
38
|
+
- **FTS5 PDF (+6):** indexes PDF chunks with kind='pdf' alongside markdown, page markers travel through chunks for snippets, kind-scoped diff() doesn't see other-kind rows, kind-undefined diff() shows both, reindexPdfFile is atomically idempotent, schema bump v3→v4 auto-rebuilds.
|
|
39
|
+
- **Embed-db PDF (+3):** upserts with kind='pdf' and search returns kind='pdf', getSourceStates(kind=…) doesn't overlap, schema bump v1→v2 idempotent on matching schema.
|
|
40
|
+
- **searchHybrid kind (+3):** blended hits with both kind='md' and kind='pdf', PDF hits use .pdf-stripped titles, kind defaults to 'md' on TF-IDF-only matches.
|
|
41
|
+
|
|
42
|
+
### Surface delta vs v2.7.0
|
|
43
|
+
|
|
44
|
+
- **No new tools.** The 38 from v2.7.0 stay. PDF retrieval is a property of `obsidian_search` (and the diagnostic single-ranker tools), not a new tool surface.
|
|
45
|
+
- **+1 CLI flag** (`--include-pdfs`) wired on three subcommands (`serve`, `index`, `build-embeddings`).
|
|
46
|
+
- **Schema bumps** auto-rebuild legacy indexes on first open.
|
|
47
|
+
|
|
48
|
+
### Migration
|
|
49
|
+
|
|
50
|
+
**No-op for default users.** PDF indexing is opt-in via `--include-pdfs`. Existing `serve` / `serve-http` / `index` / `build-embeddings` users keep working unchanged. Once you opt in, the FTS5 + embed-db files auto-rebuild on first open (same one-time cost as `tokenize_mode` change in earlier versions).
|
|
51
|
+
|
|
52
|
+
### Strategic position
|
|
53
|
+
|
|
54
|
+
v2.7.0 added the foundation (PDF extraction tools); v2.8.0 makes them retrievable. Combined with v2.0-v2.6's hybrid RRF + wikilink graph-boost + breadcrumb chunking + multilingual embeddings + remote MCP transport, **enquire-mcp is the only Obsidian-MCP that searches markdown and PDFs in a unified hybrid retrieval surface**. Smart Connections (paid) doesn't index PDFs. Khoj indexes PDFs but doesn't run on Obsidian's substrate (separate app, separate vault). The intersection is uniquely ours.
|
|
55
|
+
|
|
56
|
+
## [2.7.0] — 2026-05-08
|
|
57
|
+
|
|
58
|
+
**Sprint 7 — PDF as a first-class indexable content type.** PDFs are the #1 non-markdown content kind in real research vaults (papers, scanned notes, downloaded references). **No other Obsidian-MCP currently indexes them.** v2.7.0 adds two new read tools that work identically over stdio + `serve-http`, gated behind `pdfjs-dist` as an `optionalDependency` so the markdown-only path stays zero-cost.
|
|
59
|
+
|
|
60
|
+
### Added — `obsidian_list_pdfs`
|
|
61
|
+
|
|
62
|
+
Lists `.pdf` files in the vault with size + last-modified timestamp. Sorted by mtime descending. Honors `--exclude-glob` and `--read-paths`. Discovery entry point — call this before `obsidian_read_pdf` to find what's available.
|
|
63
|
+
|
|
64
|
+
### Added — `obsidian_read_pdf`
|
|
65
|
+
|
|
66
|
+
Extracts plain text from one PDF, returning per-page text + a `full_text` join + doc-level metadata (title / author / subject / keywords / creator / producer / creation date / mod date). Optional `pages` slice (1-indexed inclusive range, e.g. `[2, 5]`) for partial reads of long documents — `total_page_count` is preserved so consumers know how much they didn't read. Image-only / scanned PDFs surface `has_text: false` so agents can detect-and-recommend OCR (deferred to v2.8+).
|
|
67
|
+
|
|
68
|
+
Per-page extraction speed: ~50-200ms cold, ~10-30ms warm on M1. No rendering, no canvas. Same path-safety + privacy filter (`--exclude-glob` / `--read-paths`) as `obsidian_read_note` — there are no PDF-specific shortcuts.
|
|
69
|
+
|
|
70
|
+
### Added — `pdfjs-dist` as `optionalDependencies`
|
|
71
|
+
|
|
72
|
+
Mozilla's [PDF.js](https://mozilla.github.io/pdf.js/) parser. Pure JS (no native deps), Apache-2.0, SLSA-3 published, Node 20+ compatible (pinned `pdfjs-dist@^4.10.38`). The PDF tools surface a clean install-hint error on missing optional dep, never a cryptic module-not-found stack trace. Server-side hardening: `isEvalSupported: false`, `useSystemFonts: false`, `verbosity: 0`. No outbound HTTP, no eval, no font fetches.
|
|
73
|
+
|
|
74
|
+
### Surface delta vs v2.6.0
|
|
75
|
+
|
|
76
|
+
- **+2 read tools** — `obsidian_list_pdfs`, `obsidian_read_pdf`
|
|
77
|
+
- **Total surface:** 38 tools (27 always-on read + 1 opt-in via `--persistent-index` + 3 opt-in diagnostic + 7 opt-in write) + 17 prompts
|
|
78
|
+
|
|
79
|
+
### Tests
|
|
80
|
+
|
|
81
|
+
481 unit tests pass (was 459 in v2.6.0, +22 PDF tests). Synthetic PDF builder in `tests/helpers/make-pdf.ts` produces minimal valid PDF 1.4 byte sequences for tests — no committed binary fixtures, no PDF-writer dev-dependency. Coverage:
|
|
82
|
+
|
|
83
|
+
- `extractPdfText`: single-page, multi-page in-order, Title/Author metadata round-trip, char_count correctness, escape-paren-and-backslash safety.
|
|
84
|
+
- `listPdfs`: recursive walk, mtime-desc sort, folder filter, `--exclude-glob` privacy filter parity with markdown listing, `--read-paths` allowlist parity, limit honored.
|
|
85
|
+
- `readPdf`: round-trip, optional `.pdf` extension, page-range slicing (with original `total_page_count` preserved), `include_metadata` flag, missing-path error, excluded-by-privacy-filter error, page numbers preserved through slicing, empty-path error.
|
|
86
|
+
|
|
87
|
+
### Smoke
|
|
88
|
+
|
|
89
|
+
`scripts/smoke.mjs` updated: tool count goes from 28/29 → 30/31 (with/without `--persistent-index`), `obsidian_list_pdfs` + `obsidian_read_pdf` added to baseTools.
|
|
90
|
+
|
|
91
|
+
### Migration
|
|
92
|
+
|
|
93
|
+
**No-op.** All additions are new tools. Existing tool calls behave identically. Users who skipped `pdfjs-dist` (`npm install --omit=optional`) keep the full markdown surface; PDF tools register but throw a clean install-hint when called.
|
|
94
|
+
|
|
95
|
+
### Strategic position
|
|
96
|
+
|
|
97
|
+
The retrieval moats from v2.0-v2.6 (hybrid RRF, wikilink graph-boost, breadcrumb chunking, multilingual embeddings, remote MCP) extend cleanly to PDFs once you've extracted text. The next logical step is integrating PDF chunks into the FTS5 + embedding indexes so `obsidian_search` returns blended markdown + PDF hits with a `kind` flag — tracked for v2.8+. v2.7.0 ships the foundation.
|
|
98
|
+
|
|
5
99
|
## [2.6.0] — 2026-05-08
|
|
6
100
|
|
|
7
101
|
**Sprint 6 — remote-MCP HTTP transport.** New `serve-http` subcommand running the same server (same tools, same vault, same hybrid retrieval) over [Streamable HTTP](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#streamable-http) — the protocol Claude.ai web, ChatGPT, Cursor's HTTP mode, and most mobile MCP clients use to talk to a remote server. **No other Obsidian-MCP currently ships a remote-HTTP transport.**
|
package/README.md
CHANGED
|
@@ -38,13 +38,14 @@ That's it. Your AI now has structured access to wikilinks, backlinks, frontmatte
|
|
|
38
38
|
| TF-IDF semantic search | ❌ | ❌ | ✅ |
|
|
39
39
|
| **ML embeddings (multilingual)** | ❌ | ✅ paid | ✅ **free** |
|
|
40
40
|
| **Hybrid (BM25+TF-IDF+embeddings, RRF)** | ❌ | ❌ | ✅ **only here** |
|
|
41
|
+
| **PDFs blended into hybrid search** | ❌ | ❌ | ✅ **only here** (v2.8.0) |
|
|
41
42
|
| Per-signal observability on each hit | ❌ | ❌ | ✅ |
|
|
42
43
|
| Privacy filter (`--exclude-glob` / `--read-paths`) | ❌ | n/a | ✅ verified at search + write paths |
|
|
43
44
|
| Standalone (no Obsidian plugin) | varies | ❌ requires Obsidian | ✅ direct vault read |
|
|
44
45
|
| MCP-native (any agent) | varies | ❌ Obsidian-only | ✅ stdio JSON-RPC |
|
|
45
46
|
| **Remote MCP (HTTP transport, bearer auth)** | ❌ | ❌ | ✅ **only here** (v2.6.0) |
|
|
46
47
|
| SLSA-3 provenance | ❌ | n/a | ✅ |
|
|
47
|
-
| Test suite | rare | n/a | ✅
|
|
48
|
+
| Test suite | rare | n/a | ✅ 493 unit tests |
|
|
48
49
|
|
|
49
50
|
---
|
|
50
51
|
|
|
@@ -125,13 +126,13 @@ No other Obsidian-MCP currently ships a remote-HTTP transport. Same vault, same
|
|
|
125
126
|
|
|
126
127
|
---
|
|
127
128
|
|
|
128
|
-
## Tools (
|
|
129
|
+
## Tools (38 total)
|
|
129
130
|
|
|
130
|
-
###
|
|
131
|
+
### 27 always-on read tools
|
|
131
132
|
|
|
132
133
|
| Tool | What it does |
|
|
133
134
|
|---|---|
|
|
134
|
-
| `obsidian_search` | **Hybrid retrieval** — fuses BM25 + TF-IDF + ML embeddings via RRF. The default search tool. Auto-detects available signals. v2.2.0: `granularity: "block"` arg returns chunks instead of notes. |
|
|
135
|
+
| `obsidian_search` | **Hybrid retrieval** — fuses BM25 + TF-IDF + ML embeddings via RRF. The default search tool. Auto-detects available signals. v2.2.0: `granularity: "block"` arg returns chunks instead of notes. **v2.8.0:** with `--include-pdfs`, PDF chunks blend in alongside markdown — every hit carries `kind: "md" \| "pdf"` and PDF snippets include `[page: N]` markers for citation. |
|
|
135
136
|
| `obsidian_context_pack` | **v2.2.0.** Token-budgeted context bundling: takes a question, runs hybrid search, gathers note bodies + backlinks + optionally recent dailies, returns one ready-to-paste markdown bundle. Saves ~5 tool calls. |
|
|
136
137
|
| `obsidian_chat_thread_read` | **v2.2.0.** Parse a note's `## Chat: <title>` block into structured messages (role/timestamp/content/line-range). Pair with `_append` (write) for note-tethered AI conversations. |
|
|
137
138
|
| `obsidian_frontmatter_get` | **v2.3.0.** Read parsed YAML frontmatter for a note. With `key`, returns just that field. |
|
|
@@ -155,6 +156,8 @@ No other Obsidian-MCP currently ships a remote-HTTP transport. Same vault, same
|
|
|
155
156
|
| `obsidian_validate_note_proposal` | Lint a draft note before writing (closes the #1 LLM-write pain). |
|
|
156
157
|
| `obsidian_list_canvases` | List `.canvas` files with node + edge counts. |
|
|
157
158
|
| `obsidian_read_canvas` | Parse `.canvas` into typed nodes (text/file/link/group) + edges. |
|
|
159
|
+
| `obsidian_list_pdfs` | **v2.7.0.** List `.pdf` files with size + mtime. PDFs are the #1 non-markdown content kind in real vaults; no other Obsidian-MCP indexes them. |
|
|
160
|
+
| `obsidian_read_pdf` | **v2.7.0.** Extract page-by-page text + doc metadata (title/author/etc) from a PDF. Optional 1-indexed page-range slice. Image-only / scanned PDFs surface `has_text: false`. Powered by Mozilla PDF.js (Apache-2.0, pinned to `optionalDependencies` so the markdown-only path stays zero-cost). |
|
|
158
161
|
| `obsidian_open_in_ui` | Emit `obsidian://open?vault=...` URI. |
|
|
159
162
|
|
|
160
163
|
### 4 opt-in read tools
|
|
@@ -228,7 +231,7 @@ Full posture: [SECURITY.md](./SECURITY.md). Report vulnerabilities to `oomkapwn@
|
|
|
228
231
|
|---|---|
|
|
229
232
|
| Language | TypeScript strict + `noUncheckedIndexedAccess` |
|
|
230
233
|
| Lint | Biome 2 (zero-warning policy) |
|
|
231
|
-
| Tests |
|
|
234
|
+
| Tests | 493 unit tests across 23 files |
|
|
232
235
|
| CI | ubuntu × {Node 20, 22, 24} required + macOS advisory job |
|
|
233
236
|
| Coverage | Lines ≥86%, statements ≥82%, functions ≥75%, branches ≥73% (gated) |
|
|
234
237
|
| Audit | `npm audit --audit-level=moderate` for prod; high for dev |
|
package/SECURITY.md
CHANGED
|
@@ -167,3 +167,37 @@ When `--persistent-index` is enabled, the search-index file at `<vault-hash>.fts
|
|
|
167
167
|
- Cross-vault contamination guard: a `meta` table stores `vault_root` and `tokenize_mode`; if either changes between runs, the index is dropped and rebuilt with a stderr warning.
|
|
168
168
|
- Manual purge: `enquire-mcp clear-index --vault <path>` removes the `.fts5.db`, `.fts5.db-wal`, and `.fts5.db-shm` files.
|
|
169
169
|
- **Caveat:** SQLite WAL mode keeps the most-recent uncommitted writes in `<file>-wal`. If you delete only `<file>` manually (not via `clear-index`), some recently-indexed chunks may persist in the sidecar. Always use `clear-index` for full removal.
|
|
170
|
+
|
|
171
|
+
## HTTP transport (v2.6.0): bearer auth + remote-MCP posture
|
|
172
|
+
|
|
173
|
+
The `serve-http` subcommand (added v2.6.0) exposes the same MCP server over [Streamable HTTP](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#streamable-http) so claude.ai web, ChatGPT, Cursor HTTP mode, and mobile MCP clients can reach a remote vault. It introduces a network-exposed endpoint that the default stdio path doesn't have. The threat model + deployment recipes are documented at length in [`docs/http-transport.md`](docs/http-transport.md); this section is the canonical security posture.
|
|
174
|
+
|
|
175
|
+
### What we protect against
|
|
176
|
+
|
|
177
|
+
- **Unauthenticated read.** Wrong/missing token → 401, fail-closed at the auth middleware before any tool dispatch. The startup itself refuses to bind without `--bearer-token` (or `--bearer-token-env`) of length ≥16 chars.
|
|
178
|
+
- **Token timing leaks.** Bearer compare hashes both presented and expected token with SHA-256 first, then `crypto.timingSafeEqual` on the equal-length 32-byte digests. No length oracle; equal-length compare is constant-time.
|
|
179
|
+
- **Token logging.** Stderr / log output uses the SHA-256 prefix as the rate-limit key — the raw bearer token never appears in logs, error messages, or rate-limit state.
|
|
180
|
+
- **Rate-limit abuse.** Per-token sliding 60-second window, default 120 requests/minute. Tunable via `--rate-limit` (`0` disables for trusted private LANs). 429 + `Retry-After: 60` on overflow.
|
|
181
|
+
- **CORS-based credential leakage.** `--cors-origin` is a strict allowlist. Default empty (no `Access-Control-Allow-Origin` sent — same-origin works regardless). With explicit origins, we send `Access-Control-Allow-Credentials: true` so cookies + bearer requests work cross-origin. With the `*` wildcard, we deliberately OMIT `Allow-Credentials` (browsers reject the combo anyway, and reflecting credentialed CORS to attacker-controlled origins is the [CodeQL `js/cors-misconfiguration-for-credentials` class of bug](https://codeql.github.com/codeql-query-help/javascript/js-cors-misconfiguration-for-credentials/)). The response value is sourced from the allowlist itself, not from `req.headers.origin`, so static-analysis taint flows from a server-controlled root.
|
|
182
|
+
- **Body bombs.** Per-request body size cap (4 MB), enforced as we accumulate chunks. Bigger requests get `400 Parse error` before we attempt JSON.parse.
|
|
183
|
+
- **Privacy filter (`--exclude-glob` / `--read-paths`)** applies identically to HTTP and stdio paths. The same audit-tested filter runs at every search/read/walker boundary; there are no HTTP-specific shortcuts. v2.0.0-beta.2 P0 fixes for FTS5 + embed-index search apply here.
|
|
184
|
+
- **Tag-deletion / write-tool gating.** All write tools (chat-thread-append, frontmatter-set, create/append/rename/replace/archive) remain gated by `--enable-write` regardless of transport. HTTP doesn't lower the bar.
|
|
185
|
+
|
|
186
|
+
### What we do NOT protect against (deliberate non-goals)
|
|
187
|
+
|
|
188
|
+
- **TLS termination.** We bind plain HTTP and assume a tunnel (Tailscale Funnel / Cloudflare Tunnel / nginx + Let's Encrypt) handles HTTPS. We deliberately default `--host 127.0.0.1` so direct internet exposure is opt-in via `0.0.0.0`. Recipes in `docs/http-transport.md` walk through the tunneled deployments.
|
|
189
|
+
- **Compromised client.** A user who pastes their bearer token into a malicious chat or who lets it leak via `ps aux` / shell history is owned. Hence `--bearer-token-env <name>` (read from env) and `enquire-mcp gen-token` (32-byte base64url generator). Treat the token like a password.
|
|
190
|
+
- **DoS from a single token.** A malicious client can fire rate-limit-budget worth of requests indefinitely; we just answer 429 once over budget. Use the tunnel's WAF for upstream DoS protection. Single-process — for shared limits use a reverse proxy with its own rate-limit module.
|
|
191
|
+
- **Multi-tenant cross-token attacks.** This is a single-tenant tool. A small team should run **one process per user** (e.g. systemd template unit) and not share tokens. We don't do tenant isolation in-process beyond the per-token rate-limit.
|
|
192
|
+
- **OAuth.** No OAuth flow, no token minting, no refresh logic. Static long-lived bearer is by design — generated with `enquire-mcp gen-token`, rotated manually. OAuth is tracked for v2.7+ if a user explicitly needs it.
|
|
193
|
+
|
|
194
|
+
### Stateful sessions
|
|
195
|
+
|
|
196
|
+
v2.6.0 ships **stateless** mode only (`sessionIdGenerator: undefined`) — fresh `McpServer` per request over the SHARED vault + FTS5 + embedding handles. No long-lived session map; no SSE persistent streams. This is the right default for our short-running tools (search, read, frontmatter ops) and avoids the persistence-aware shutdown complexity. Stateful sessions with `Mcp-Session-Id` + persistent SSE streams are tracked for v2.7+ if there's demand.
|
|
197
|
+
|
|
198
|
+
### Observability
|
|
199
|
+
|
|
200
|
+
- Ready banner on stderr: `enquire <version> ready (read-only|WRITE-ENABLED, vault=…) (transport=http, bound=…)`.
|
|
201
|
+
- Transport errors written to stderr with no token / no credential leakage.
|
|
202
|
+
- `/health` endpoint (`GET /health → 200 ok`) is **unauthenticated** and exists specifically for tunnel/uptime monitors. It returns the literal string `ok` — no version info, no vault path, no operational metadata. Health probes can't be used to fingerprint the deployment.
|
|
203
|
+
- `OPTIONS` preflight requests are unauthenticated (per CORS spec) but only emit CORS headers when the request's `Origin` is in the allowlist.
|
package/dist/embed-db.d.ts
CHANGED
|
@@ -1,3 +1,5 @@
|
|
|
1
|
+
/** Content-source kind. Mirrors ChunkKind in src/fts5.ts. */
|
|
2
|
+
export type EmbedChunkKind = "md" | "pdf";
|
|
1
3
|
export interface EmbedSearchHit {
|
|
2
4
|
rel_path: string;
|
|
3
5
|
chunk_index: number;
|
|
@@ -7,6 +9,8 @@ export interface EmbedSearchHit {
|
|
|
7
9
|
text_preview: string;
|
|
8
10
|
/** Cosine similarity (since vectors are L2-normalized at insert time). */
|
|
9
11
|
score: number;
|
|
12
|
+
/** v2.8.0 — content-source kind. Defaults to "md" for backward compat. */
|
|
13
|
+
kind: EmbedChunkKind;
|
|
10
14
|
}
|
|
11
15
|
export interface EmbedSyncReport {
|
|
12
16
|
added: number;
|
|
@@ -44,19 +48,27 @@ export declare class EmbedDb {
|
|
|
44
48
|
private readMeta;
|
|
45
49
|
private writeMeta;
|
|
46
50
|
private requireDb;
|
|
47
|
-
/**
|
|
51
|
+
/**
|
|
52
|
+
* Replace all embeddings for a single note. Caller computes vectors.
|
|
53
|
+
* v2.8.0: optional `kind` parameter ("md" | "pdf"); defaults to "md" so
|
|
54
|
+
* existing callers (markdown indexing path) need no changes.
|
|
55
|
+
*/
|
|
48
56
|
upsertNote(relPath: string, mtimeMs: number, chunks: ReadonlyArray<{
|
|
49
57
|
chunkIndex: number;
|
|
50
58
|
lineStart: number;
|
|
51
59
|
lineEnd: number;
|
|
52
60
|
textPreview: string;
|
|
53
61
|
vector: Float32Array;
|
|
54
|
-
}
|
|
62
|
+
}>, kind?: EmbedChunkKind): void;
|
|
55
63
|
/** Drop a note's embeddings entirely (used on file deletion). */
|
|
56
64
|
deleteNote(relPath: string): void;
|
|
57
|
-
/**
|
|
58
|
-
*
|
|
59
|
-
|
|
65
|
+
/**
|
|
66
|
+
* Read the source-state table — caller compares mtimes to decide what to
|
|
67
|
+
* re-embed. v2.8.0: optional `kind` filter — when set, only rows of that
|
|
68
|
+
* kind are returned. Lets the markdown-sync and PDF-sync paths run
|
|
69
|
+
* independently without one's "missing files" being deleted by the other.
|
|
70
|
+
*/
|
|
71
|
+
getSourceStates(kind?: EmbedChunkKind): SourceStateRow[];
|
|
60
72
|
/** Brute-force cosine top-K. Vectors are L2-normalized at insert time so
|
|
61
73
|
* cosine == dot product. Acceptable up to ~50K chunks; v2.1 will swap to
|
|
62
74
|
* HNSW if real vaults hit that ceiling. */
|
package/dist/embed-db.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"embed-db.d.ts","sourceRoot":"","sources":["../src/embed-db.ts"],"names":[],"mappings":"
|
|
1
|
+
{"version":3,"file":"embed-db.d.ts","sourceRoot":"","sources":["../src/embed-db.ts"],"names":[],"mappings":"AAsBA,6DAA6D;AAC7D,MAAM,MAAM,cAAc,GAAG,IAAI,GAAG,KAAK,CAAC;AAE1C,MAAM,WAAW,cAAc;IAC7B,QAAQ,EAAE,MAAM,CAAC;IACjB,WAAW,EAAE,MAAM,CAAC;IACpB,UAAU,EAAE,MAAM,CAAC;IACnB,QAAQ,EAAE,MAAM,CAAC;IACjB,mDAAmD;IACnD,YAAY,EAAE,MAAM,CAAC;IACrB,0EAA0E;IAC1E,KAAK,EAAE,MAAM,CAAC;IACd,0EAA0E;IAC1E,IAAI,EAAE,cAAc,CAAC;CACtB;AAED,MAAM,WAAW,eAAe;IAC9B,KAAK,EAAE,MAAM,CAAC;IACd,OAAO,EAAE,MAAM,CAAC;IAChB,OAAO,EAAE,MAAM,CAAC;IAChB,SAAS,EAAE,MAAM,CAAC;IAClB,YAAY,EAAE,MAAM,CAAC;CACtB;AAED,UAAU,cAAc;IACtB,QAAQ,EAAE,MAAM,CAAC;IACjB,QAAQ,EAAE,MAAM,CAAC;CAClB;AA+CD,MAAM,WAAW,cAAc;IAC7B,2CAA2C;IAC3C,IAAI,EAAE,MAAM,CAAC;IACb,sDAAsD;IACtD,SAAS,EAAE,MAAM,CAAC;IAClB,wEAAwE;IACxE,UAAU,EAAE,MAAM,CAAC;IACnB,oDAAoD;IACpD,GAAG,EAAE,MAAM,CAAC;CACb;AAED,qBAAa,OAAO;IAClB,OAAO,CAAC,EAAE,CAAmB;IAC7B,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAS;IAC9B,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAS;IACnC,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAS;IACpC,OAAO,CAAC,QAAQ,CAAC,GAAG,CAAS;gBAEjB,IAAI,EAAE,cAAc;IAO1B,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;IAc3B,0DAA0D;IACpD,WAAW,IAAI,OAAO,CAAC,OAAO,CAAC;IAcrC,KAAK,IAAI,IAAI;IAOb,OAAO,CAAC,eAAe;IAuDvB,OAAO,CAAC,QAAQ;IAQhB,OAAO,CAAC,SAAS;IAMjB,OAAO,CAAC,SAAS;IAKjB;;;;OAIG;IACH,UAAU,CACR,OAAO,EAAE,MAAM,EACf,OAAO,EAAE,MAAM,EACf,MAAM,EAAE,aAAa,CAAC;QACpB,UAAU,EAAE,MAAM,CAAC;QACnB,SAAS,EAAE,MAAM,CAAC;QAClB,OAAO,EAAE,MAAM,CAAC;QAChB,WAAW,EAAE,MAAM,CAAC;QACpB,MAAM,EAAE,YAAY,CAAC;KACtB,CAAC,EACF,IAAI,GAAE,cAAqB,GAC1B,IAAI;IAkCP,iEAAiE;IACjE,UAAU,CAAC,OAAO,EAAE,MAAM,GAAG,IAAI;IAMjC;;;;;OAKG;IACH,eAAe,CAAC,IAAI,CAAC,EAAE,cAAc,GAAG,cAAc,EAAE;IAQxD;;gDAE4C;IAC5C,MAAM,CAAC,QAAQ,EAAE,YAAY,EAAE,CAAC,EAAE,MAAM,EAAE,IAAI,GAAE;QAAE,MAAM,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAA;KAAO,GAAG,cAAc,EAAE;IA8D9G,kDAAkD;IAClD,WAAW,IAAI,MAAM;CAKtB;AAED,8EAA8E;AAC9E,wBAAgB,kBAAkB,CAAC,eAAe,EAAE,MAAM,GAAG,MAAM,CAIlE"}
|
package/dist/embed-db.js
CHANGED
|
@@ -13,7 +13,7 @@
|
|
|
13
13
|
// on 50K × 384 floats). HNSW comes in v2.1 if real users hit that ceiling.
|
|
14
14
|
import { promises as fs } from "node:fs";
|
|
15
15
|
import * as path from "node:path";
|
|
16
|
-
const SCHEMA_VERSION =
|
|
16
|
+
const SCHEMA_VERSION = 2;
|
|
17
17
|
// v2.0.0-beta.1 P2 fix: probe the native binding via :memory: open so the
|
|
18
18
|
// "JS package present but *.node binary missing" failure mode produces a
|
|
19
19
|
// clean error pointing at `npm rebuild`, not a raw bindings stack trace.
|
|
@@ -120,6 +120,7 @@ export class EmbedDb {
|
|
|
120
120
|
line_end INTEGER NOT NULL,
|
|
121
121
|
text_preview TEXT NOT NULL,
|
|
122
122
|
vector BLOB NOT NULL,
|
|
123
|
+
kind TEXT NOT NULL DEFAULT 'md',
|
|
123
124
|
UNIQUE(rel_path, chunk_index)
|
|
124
125
|
);
|
|
125
126
|
CREATE INDEX IF NOT EXISTS embeddings_rel_path ON embeddings(rel_path);
|
|
@@ -127,6 +128,7 @@ export class EmbedDb {
|
|
|
127
128
|
rel_path TEXT PRIMARY KEY,
|
|
128
129
|
mtime_ms INTEGER NOT NULL,
|
|
129
130
|
n_chunks INTEGER NOT NULL,
|
|
131
|
+
kind TEXT NOT NULL DEFAULT 'md',
|
|
130
132
|
indexed_at TEXT NOT NULL
|
|
131
133
|
);
|
|
132
134
|
`);
|
|
@@ -156,23 +158,27 @@ export class EmbedDb {
|
|
|
156
158
|
throw new Error("EmbedDb is not open — call .open() first");
|
|
157
159
|
return this.db;
|
|
158
160
|
}
|
|
159
|
-
/**
|
|
160
|
-
|
|
161
|
+
/**
|
|
162
|
+
* Replace all embeddings for a single note. Caller computes vectors.
|
|
163
|
+
* v2.8.0: optional `kind` parameter ("md" | "pdf"); defaults to "md" so
|
|
164
|
+
* existing callers (markdown indexing path) need no changes.
|
|
165
|
+
*/
|
|
166
|
+
upsertNote(relPath, mtimeMs, chunks, kind = "md") {
|
|
161
167
|
const db = this.requireDb();
|
|
162
168
|
const dim = this.dim;
|
|
163
169
|
const tx = db.transaction((...args) => {
|
|
164
170
|
const rows = args[0];
|
|
165
171
|
db.prepare("DELETE FROM embeddings WHERE rel_path = ?").run(relPath);
|
|
166
|
-
const insert = db.prepare(`INSERT INTO embeddings (rel_path, chunk_index, line_start, line_end, text_preview, vector)
|
|
167
|
-
VALUES (?, ?, ?, ?, ?, ?)`);
|
|
172
|
+
const insert = db.prepare(`INSERT INTO embeddings (rel_path, chunk_index, line_start, line_end, text_preview, vector, kind)
|
|
173
|
+
VALUES (?, ?, ?, ?, ?, ?, ?)`);
|
|
168
174
|
for (const c of rows) {
|
|
169
175
|
if (c.vector.length !== dim) {
|
|
170
176
|
throw new Error(`vector dim mismatch for ${relPath} chunk ${c.chunkIndex}: got ${c.vector.length}, expected ${dim}`);
|
|
171
177
|
}
|
|
172
|
-
insert.run(relPath, c.chunkIndex, c.lineStart, c.lineEnd, c.textPreview, Buffer.from(c.vector.buffer, c.vector.byteOffset, c.vector.byteLength));
|
|
178
|
+
insert.run(relPath, c.chunkIndex, c.lineStart, c.lineEnd, c.textPreview, Buffer.from(c.vector.buffer, c.vector.byteOffset, c.vector.byteLength), kind);
|
|
173
179
|
}
|
|
174
|
-
db.prepare(`INSERT OR REPLACE INTO source_state (rel_path, mtime_ms, n_chunks, indexed_at)
|
|
175
|
-
VALUES (?, ?, ?, datetime('now'))`).run(relPath, mtimeMs, rows.length);
|
|
180
|
+
db.prepare(`INSERT OR REPLACE INTO source_state (rel_path, mtime_ms, n_chunks, kind, indexed_at)
|
|
181
|
+
VALUES (?, ?, ?, ?, datetime('now'))`).run(relPath, mtimeMs, rows.length, kind);
|
|
176
182
|
});
|
|
177
183
|
tx(chunks);
|
|
178
184
|
}
|
|
@@ -182,10 +188,17 @@ export class EmbedDb {
|
|
|
182
188
|
db.prepare("DELETE FROM embeddings WHERE rel_path = ?").run(relPath);
|
|
183
189
|
db.prepare("DELETE FROM source_state WHERE rel_path = ?").run(relPath);
|
|
184
190
|
}
|
|
185
|
-
/**
|
|
186
|
-
*
|
|
187
|
-
|
|
191
|
+
/**
|
|
192
|
+
* Read the source-state table — caller compares mtimes to decide what to
|
|
193
|
+
* re-embed. v2.8.0: optional `kind` filter — when set, only rows of that
|
|
194
|
+
* kind are returned. Lets the markdown-sync and PDF-sync paths run
|
|
195
|
+
* independently without one's "missing files" being deleted by the other.
|
|
196
|
+
*/
|
|
197
|
+
getSourceStates(kind) {
|
|
188
198
|
const db = this.requireDb();
|
|
199
|
+
if (kind !== undefined) {
|
|
200
|
+
return db.prepare("SELECT rel_path, mtime_ms FROM source_state WHERE kind = ?").all(kind);
|
|
201
|
+
}
|
|
189
202
|
return db.prepare("SELECT rel_path, mtime_ms FROM source_state").all();
|
|
190
203
|
}
|
|
191
204
|
/** Brute-force cosine top-K. Vectors are L2-normalized at insert time so
|
|
@@ -204,9 +217,9 @@ export class EmbedDb {
|
|
|
204
217
|
// FtsIndex.search() in fts5.ts.
|
|
205
218
|
const rows = db
|
|
206
219
|
.prepare(folderPrefix
|
|
207
|
-
? `SELECT rel_path, chunk_index, line_start, line_end, text_preview, vector
|
|
220
|
+
? `SELECT rel_path, chunk_index, line_start, line_end, text_preview, vector, kind
|
|
208
221
|
FROM embeddings WHERE substr(rel_path, 1, ?) = ?`
|
|
209
|
-
: `SELECT rel_path, chunk_index, line_start, line_end, text_preview, vector FROM embeddings`)
|
|
222
|
+
: `SELECT rel_path, chunk_index, line_start, line_end, text_preview, vector, kind FROM embeddings`)
|
|
210
223
|
.all(...(folderPrefix ? [folderPrefix.length, folderPrefix] : []));
|
|
211
224
|
const expectedBytes = this.dim * 4; // Float32 = 4 bytes
|
|
212
225
|
const heap = [];
|
|
@@ -232,7 +245,8 @@ export class EmbedDb {
|
|
|
232
245
|
line_start: r.line_start,
|
|
233
246
|
line_end: r.line_end,
|
|
234
247
|
text_preview: r.text_preview,
|
|
235
|
-
score
|
|
248
|
+
score,
|
|
249
|
+
kind: (r.kind === "pdf" ? "pdf" : "md")
|
|
236
250
|
});
|
|
237
251
|
}
|
|
238
252
|
heap.sort((a, b) => b.score - a.score);
|
package/dist/embed-db.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"embed-db.js","sourceRoot":"","sources":["../src/embed-db.ts"],"names":[],"mappings":"AAAA,0EAA0E;AAC1E,6EAA6E;AAC7E,gFAAgF;AAChF,kDAAkD;AAClD,EAAE;AACF,gCAAgC;AAChC,gDAAgD;AAChD,0CAA0C;AAC1C,gFAAgF;AAChF,2DAA2D;AAC3D,EAAE;AACF,+EAA+E;AAC/E,2EAA2E;AAE3E,OAAO,EAAE,QAAQ,IAAI,EAAE,EAAE,MAAM,SAAS,CAAC;AACzC,OAAO,KAAK,IAAI,MAAM,WAAW,CAAC;AAElC,MAAM,cAAc,GAAG,CAAC,CAAC;
|
|
1
|
+
{"version":3,"file":"embed-db.js","sourceRoot":"","sources":["../src/embed-db.ts"],"names":[],"mappings":"AAAA,0EAA0E;AAC1E,6EAA6E;AAC7E,gFAAgF;AAChF,kDAAkD;AAClD,EAAE;AACF,gCAAgC;AAChC,gDAAgD;AAChD,0CAA0C;AAC1C,gFAAgF;AAChF,2DAA2D;AAC3D,EAAE;AACF,+EAA+E;AAC/E,2EAA2E;AAE3E,OAAO,EAAE,QAAQ,IAAI,EAAE,EAAE,MAAM,SAAS,CAAC;AACzC,OAAO,KAAK,IAAI,MAAM,WAAW,CAAC;AAElC,MAAM,cAAc,GAAG,CAAC,CAAC;AAkCzB,0EAA0E;AAC1E,yEAAyE;AACzE,yEAAyE;AACzE,IAAI,gBAAgB,GAA2C,IAAI,CAAC;AACpE,KAAK,UAAU,gBAAgB;IAC7B,IAAI,gBAAgB;QAAE,OAAO,gBAAgB,CAAC;IAC9C,IAAI,CAAC;QACH,MAAM,GAAG,GAAG,CAAC,MAAM,MAAM,CAAC,gBAAgB,CAAC,CAAgD,CAAC;QAC5F,MAAM,IAAI,GAAG,GAAG,CAAC,OAAO,CAAC;QACzB,IAAI,CAAC,IAAI;YAAE,MAAM,IAAI,KAAK,CAAC,sCAAsC,CAAC,CAAC;QACnE,IAAI,CAAC;YACH,MAAM,KAAK,GAAG,IAAI,IAAI,CAAC,UAAU,CAA2B,CAAC;YAC7D,KAAK,CAAC,KAAK,EAAE,EAAE,CAAC;QAClB,CAAC;QAAC,OAAO,QAAQ,EAAE,CAAC;YAClB,MAAM,IAAI,KAAK,CACb,+IAA+I,QAAQ,YAAY,KAAK,CAAC,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,QAAQ,CAAC,EAAE,CACjN,CAAC;QACJ,CAAC;QACD,gBAAgB,GAAG,IAAI,CAAC;QACxB,OAAO,IAAI,CAAC;IACd,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,MAAM,IAAI,KAAK,CACb,8HACE,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CACjD,EAAE,CACH,CAAC;IACJ,CAAC;AACH,CAAC;AA6BD,MAAM,OAAO,OAAO;IACV,EAAE,GAAc,IAAI,CAAC;IACZ,IAAI,CAAS;IACb,SAAS,CAAS;IAClB,UAAU,CAAS;IACnB,GAAG,CAAS;IAE7B,YAAY,IAAoB;QAC9B,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC;QACtB,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC,SAAS,CAAC;QAChC,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,UAAU,CAAC;QAClC,IAAI,CAAC,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC;IACtB,CAAC;IAED,KAAK,CAAC,IAAI;QACR,IAAI,IAAI,CAAC,EAAE;YAAE,OAAO;QACpB,MAAM,IAAI,GAAG,MAAM,gBAAgB,EAAE,CAAC;QACtC,MAAM,EAAE,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,IAAI,EAAE,KAAK,EAAE,CAAC,CAAC;QAC1E,MAAM,EAAE,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,KAAK,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,GAAE,CAAC,CAAC,CAAC;QAC/D,IAAI,CAAC,EAAE,GAAG,IAAI,IAAI,CAAC,IAAI,CAAC,IAAI,CAAO,CAAC;QACpC,IAAI,CAAC,EAAE,CAAC,MAAM,CAAC,oBAAoB,CAAC,CAAC;QACrC,IAAI,CAAC,EAAE,CAAC,MAAM,CAAC,sBAAsB,CAAC,CAAC;QACvC,IAAI,CAAC,eAAe,EAAE,CAAC;QACvB,MAAM,OAAO,CAAC,GAAG,CACf,CAAC,IAAI,CAAC,IAAI,EAAE,GAAG,IAAI,CAAC,IAAI,MAAM,EAAE,GAAG,IAAI,CAAC,IAAI,MAAM,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,EAAE,CAAC,KAAK,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,GAAE,CAAC,CAAC,CAAC,CACnG,CAAC;IACJ,CAAC;IAED,0DAA0D;IAC1D,KAAK,CAAC,WAAW;QACf,IAAI,CAAC,KAAK,EAAE,CAAC;QACb,IAAI,OAAO,GAAG,KAAK,CAAC;QACpB,KAAK,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,GAAG,IAAI,CAAC,IAAI,MAAM,EAAE,GAAG,IAAI,CAAC,IAAI,MAAM,CAAC,EAAE,CAAC;YACpE,IAAI,CAAC;gBACH,MAAM,EAAE,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC;gBACnB,OAAO,GAAG,IAAI,CAAC;YACjB,CAAC;YAAC,MAAM,CAAC;gBACP,kBAAkB;YACpB,CAAC;QACH,CAAC;QACD,OAAO,OAAO,CAAC;IACjB,CAAC;IAED,KAAK;QACH,IAAI,IAAI,CAAC,EAAE,EAAE,CAAC;YACZ,IAAI,CAAC,EAAE,CAAC,KAAK,EAAE,CAAC;YAChB,IAAI,CAAC,EAAE,GAAG,IAAI,CAAC;QACjB,CAAC;IACH,CAAC;IAEO,eAAe;QACrB,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAE5B,EAAE,CAAC,IAAI,CAAC;;;;;KAKP,CAAC,CAAC;QAEH,MAAM,IAAI,GAAG,IAAI,CAAC,QAAQ,EAAE,CAAC;QAC7B,MAAM,YAAY,GAAG,IAAI,CAAC,cAAc,KAAK,SAAS,IAAI,IAAI,CAAC,cAAc,KAAK,MAAM,CAAC,cAAc,CAAC,CAAC;QACzG,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,KAAK,SAAS,IAAI,IAAI,CAAC,UAAU,KAAK,IAAI,CAAC,SAAS,CAAC;QACtF,MAAM,UAAU,GAAG,IAAI,CAAC,WAAW,KAAK,SAAS,IAAI,IAAI,CAAC,WAAW,KAAK,IAAI,CAAC,UAAU,CAAC;QAC1F,MAAM,QAAQ,GAAG,IAAI,CAAC,GAAG,KAAK,SAAS,IAAI,IAAI,CAAC,GAAG,KAAK,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QACzE,IAAI,CAAC,YAAY,IAAI,CAAC,SAAS,IAAI,CAAC,UAAU,IAAI,CAAC,QAAQ,EAAE,CAAC;YAC5D,MAAM,MAAM,GAAa,EAAE,CAAC;YAC5B,IAAI,CAAC,YAAY;gBAAE,MAAM,CAAC,IAAI,CAAC,kBAAkB,IAAI,CAAC,cAAc,MAAM,cAAc,EAAE,CAAC,CAAC;YAC5F,IAAI,CAAC,SAAS;gBAAE,MAAM,CAAC,IAAI,CAAC,cAAc,IAAI,CAAC,UAAU,MAAM,IAAI,CAAC,SAAS,EAAE,CAAC,CAAC;YACjF,IAAI,CAAC,UAAU;gBAAE,MAAM,CAAC,IAAI,CAAC,SAAS,IAAI,CAAC,WAAW,MAAM,IAAI,CAAC,UAAU,EAAE,CAAC,CAAC;YAC/E,IAAI,CAAC,QAAQ;gBAAE,MAAM,CAAC,IAAI,CAAC,OAAO,IAAI,CAAC,GAAG,MAAM,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;YAC5D,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,oCAAoC,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;YACjF,EAAE,CAAC,IAAI,CAAC,qEAAqE,CAAC,CAAC;QACjF,CAAC;QAED,EAAE,CAAC,IAAI,CAAC;;;;;;;;;;;;;;;;;;;;KAoBP,CAAC,CAAC;QAEH,IAAI,CAAC,SAAS,CAAC;YACb,cAAc,EAAE,MAAM,CAAC,cAAc,CAAC;YACtC,UAAU,EAAE,IAAI,CAAC,SAAS;YAC1B,WAAW,EAAE,IAAI,CAAC,UAAU;YAC5B,GAAG,EAAE,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC;SACtB,CAAC,CAAC;IACL,CAAC;IAEO,QAAQ;QACd,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,MAAM,IAAI,GAAG,EAAE,CAAC,OAAO,CAAC,6BAA6B,CAAC,CAAC,GAAG,EAAkC,CAAC;QAC7F,MAAM,GAAG,GAA2B,EAAE,CAAC;QACvC,KAAK,MAAM,CAAC,IAAI,IAAI;YAAE,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC;QAC3C,OAAO,GAAG,CAAC;IACb,CAAC;IAEO,SAAS,CAAC,EAA0B;QAC1C,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,MAAM,IAAI,GAAG,EAAE,CAAC,OAAO,CAAC,wDAAwD,CAAC,CAAC;QAClF,KAAK,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,EAAE,CAAC;YAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IAC1D,CAAC;IAEO,SAAS;QACf,IAAI,CAAC,IAAI,CAAC,EAAE;YAAE,MAAM,IAAI,KAAK,CAAC,0CAA0C,CAAC,CAAC;QAC1E,OAAO,IAAI,CAAC,EAAE,CAAC;IACjB,CAAC;IAED;;;;OAIG;IACH,UAAU,CACR,OAAe,EACf,OAAe,EACf,MAME,EACF,OAAuB,IAAI;QAE3B,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC;QACrB,MAAM,EAAE,GAAG,EAAE,CAAC,WAAW,CAAC,CAAC,GAAG,IAAe,EAAE,EAAE;YAC/C,MAAM,IAAI,GAAG,IAAI,CAAC,CAAC,CAAkB,CAAC;YACtC,EAAE,CAAC,OAAO,CAAC,2CAA2C,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;YACrE,MAAM,MAAM,GAAG,EAAE,CAAC,OAAO,CACvB;sCAC8B,CAC/B,CAAC;YACF,KAAK,MAAM,CAAC,IAAI,IAAI,EAAE,CAAC;gBACrB,IAAI,CAAC,CAAC,MAAM,CAAC,MAAM,KAAK,GAAG,EAAE,CAAC;oBAC5B,MAAM,IAAI,KAAK,CACb,2BAA2B,OAAO,UAAU,CAAC,CAAC,UAAU,SAAS,CAAC,CAAC,MAAM,CAAC,MAAM,cAAc,GAAG,EAAE,CACpG,CAAC;gBACJ,CAAC;gBACD,MAAM,CAAC,GAAG,CACR,OAAO,EACP,CAAC,CAAC,UAAU,EACZ,CAAC,CAAC,SAAS,EACX,CAAC,CAAC,OAAO,EACT,CAAC,CAAC,WAAW,EACb,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC,MAAM,CAAC,UAAU,EAAE,CAAC,CAAC,MAAM,CAAC,UAAU,CAAC,EACtE,IAAI,CACL,CAAC;YACJ,CAAC;YACD,EAAE,CAAC,OAAO,CACR;8CACsC,CACvC,CAAC,GAAG,CAAC,OAAO,EAAE,OAAO,EAAE,IAAI,CAAC,MAAM,EAAE,IAAI,CAAC,CAAC;QAC7C,CAAC,CAAC,CAAC;QACH,EAAE,CAAC,MAAM,CAAC,CAAC;IACb,CAAC;IAED,iEAAiE;IACjE,UAAU,CAAC,OAAe;QACxB,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,EAAE,CAAC,OAAO,CAAC,2CAA2C,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;QACrE,EAAE,CAAC,OAAO,CAAC,6CAA6C,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;IACzE,CAAC;IAED;;;;;OAKG;IACH,eAAe,CAAC,IAAqB;QACnC,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,IAAI,IAAI,KAAK,SAAS,EAAE,CAAC;YACvB,OAAO,EAAE,CAAC,OAAO,CAAC,4DAA4D,CAAC,CAAC,GAAG,CAAiB,IAAI,CAAC,CAAC;QAC5G,CAAC;QACD,OAAO,EAAE,CAAC,OAAO,CAAC,6CAA6C,CAAC,CAAC,GAAG,EAAkB,CAAC;IACzF,CAAC;IAED;;gDAE4C;IAC5C,MAAM,CAAC,QAAsB,EAAE,CAAS,EAAE,OAA+C,EAAE;QACzF,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,IAAI,QAAQ,CAAC,MAAM,KAAK,IAAI,CAAC,GAAG,EAAE,CAAC;YACjC,MAAM,IAAI,KAAK,CAAC,kCAAkC,QAAQ,CAAC,MAAM,cAAc,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;QAC7F,CAAC;QACD,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,IAAI,CAAC,QAAQ,CAAC;QAC5C,MAAM,YAAY,GAAG,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC;QAEhF,yEAAyE;QACzE,uEAAuE;QACvE,yEAAyE;QACzE,gCAAgC;QAChC,MAAM,IAAI,GAAG,EAAE;aACZ,OAAO,CACN,YAAY;YACV,CAAC,CAAC;8DACkD;YACpD,CAAC,CAAC,gGAAgG,CACrG;aACA,GAAG,CAQD,GAAG,CAAC,YAAY,CAAC,CAAC,CAAC,CAAC,YAAY,CAAC,MAAM,EAAE,YAAY,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;QAEnE,MAAM,aAAa,GAAG,IAAI,CAAC,GAAG,GAAG,CAAC,CAAC,CAAC,oBAAoB;QACxD,MAAM,IAAI,GAAqB,EAAE,CAAC;QAClC,KAAK,MAAM,CAAC,IAAI,IAAI,EAAE,CAAC;YACrB,uEAAuE;YACvE,qEAAqE;YACrE,qEAAqE;YACrE,gEAAgE;YAChE,IAAI,CAAC,CAAC,MAAM,CAAC,UAAU,KAAK,aAAa,EAAE,CAAC;gBAC1C,OAAO,CAAC,MAAM,CAAC,KAAK,CAClB,qBAAqB,CAAC,CAAC,QAAQ,IAAI,CAAC,CAAC,WAAW,iBAAiB,CAAC,CAAC,MAAM,CAAC,UAAU,eAAe,aAAa,UAAU,IAAI,CAAC,GAAG,wDAAwD,CAC3L,CAAC;gBACF,SAAS;YACX,CAAC;YACD,MAAM,GAAG,GAAG,IAAI,YAAY,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC,MAAM,CAAC,UAAU,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC;YAC7E,IAAI,KAAK,GAAG,CAAC,CAAC;YACd,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC,EAAE,EAAE,CAAC;gBAClC,KAAK,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC;YAC9C,CAAC;YACD,IAAI,KAAK,GAAG,QAAQ;gBAAE,SAAS;YAC/B,IAAI,CAAC,IAAI,CAAC;gBACR,QAAQ,EAAE,CAAC,CAAC,QAAQ;gBACpB,WAAW,EAAE,CAAC,CAAC,WAAW;gBAC1B,UAAU,EAAE,CAAC,CAAC,UAAU;gBACxB,QAAQ,EAAE,CAAC,CAAC,QAAQ;gBACpB,YAAY,EAAE,CAAC,CAAC,YAAY;gBAC5B,KAAK;gBACL,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,CAAmB;aAC1D,CAAC,CAAC;QACL,CAAC;QACD,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC;QACvC,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IAC1B,CAAC;IAED,kDAAkD;IAClD,WAAW;QACT,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,EAAE,CAAC;QAC5B,MAAM,GAAG,GAAG,EAAE,CAAC,OAAO,CAAC,sCAAsC,CAAC,CAAC,GAAG,EAAiB,CAAC;QACpF,OAAO,GAAG,EAAE,CAAC,IAAI,CAAC,CAAC;IACrB,CAAC;CACF;AAED,8EAA8E;AAC9E,MAAM,UAAU,kBAAkB,CAAC,eAAuB;IACxD,4EAA4E;IAC5E,wEAAwE;IACxE,OAAO,GAAG,eAAe,WAAW,CAAC;AACvC,CAAC"}
|
package/dist/fts5.d.ts
CHANGED
|
@@ -1,4 +1,6 @@
|
|
|
1
1
|
export type TokenizeMode = "unicode61" | "trigram";
|
|
2
|
+
/** Content-source kind. v2.7.0 added `pdf`; v2.8.0 indexes them. */
|
|
3
|
+
export type ChunkKind = "md" | "pdf";
|
|
2
4
|
export interface FtsSearchHit {
|
|
3
5
|
rel_path: string;
|
|
4
6
|
chunk_index: number;
|
|
@@ -6,6 +8,8 @@ export interface FtsSearchHit {
|
|
|
6
8
|
line_end: number;
|
|
7
9
|
snippet: string;
|
|
8
10
|
score: number;
|
|
11
|
+
/** v2.8.0 — content-source kind. Defaults to "md" for backward compat. */
|
|
12
|
+
kind: ChunkKind;
|
|
9
13
|
}
|
|
10
14
|
export interface FtsSyncReport {
|
|
11
15
|
added: number;
|
|
@@ -36,11 +40,18 @@ export declare class FtsIndex {
|
|
|
36
40
|
* Diff the on-disk source_state against the live vault snapshot. Returns
|
|
37
41
|
* categorized lists; caller is expected to feed `added` + `updated` paths
|
|
38
42
|
* back into reindexFile() and pass `deleted` to dropFile().
|
|
43
|
+
*
|
|
44
|
+
* v2.8.0: optional `kind` filter — when set, the diff only considers
|
|
45
|
+
* source_state rows of that kind. Lets the markdown-sync and PDF-sync
|
|
46
|
+
* paths run independently against the same DB without one's "missing
|
|
47
|
+
* files" being mistakenly deleted by the other. Default `undefined`
|
|
48
|
+
* means "all kinds" (used by older callers + diff queries that want
|
|
49
|
+
* a global view).
|
|
39
50
|
*/
|
|
40
51
|
diff(liveEntries: Array<{
|
|
41
52
|
relPath: string;
|
|
42
53
|
mtimeMs: number;
|
|
43
|
-
}
|
|
54
|
+
}>, kind?: ChunkKind): {
|
|
44
55
|
added: string[];
|
|
45
56
|
updated: string[];
|
|
46
57
|
deleted: string[];
|
|
@@ -48,8 +59,24 @@ export declare class FtsIndex {
|
|
|
48
59
|
};
|
|
49
60
|
/** Drop a file's chunks + state row. Idempotent. */
|
|
50
61
|
dropFile(relPath: string): void;
|
|
51
|
-
/** Re-chunk a single file, replacing its existing chunks atomically. */
|
|
62
|
+
/** Re-chunk a single markdown file, replacing its existing chunks atomically. */
|
|
52
63
|
reindexFile(relPath: string, mtimeMs: number, content: string, wikilinkTargets?: string[], tags?: string[]): number;
|
|
64
|
+
/**
|
|
65
|
+
* v2.8.0 — re-chunk a single PDF, replacing its existing chunks atomically.
|
|
66
|
+
* Caller pre-extracts page text via `extractPdfText` (src/pdf.ts) so this
|
|
67
|
+
* method stays decoupled from pdfjs-dist (which is an optionalDependency).
|
|
68
|
+
*
|
|
69
|
+
* Page boundaries are preserved as `[page: N]` markers in the joined text
|
|
70
|
+
* before chunking — the chunker may split a page across chunks or merge
|
|
71
|
+
* short pages, but the markers travel with the text so search snippets
|
|
72
|
+
* carry page citations. Same `chunkContent` pipeline as markdown so chunk
|
|
73
|
+
* IDs match across the BM25 / TF-IDF / embeddings rankers (RRF requires
|
|
74
|
+
* stable IDs).
|
|
75
|
+
*/
|
|
76
|
+
reindexPdfFile(relPath: string, mtimeMs: number, pages: ReadonlyArray<{
|
|
77
|
+
pageNumber: number;
|
|
78
|
+
text: string;
|
|
79
|
+
}>): number;
|
|
53
80
|
search(rawQuery: string, opts?: {
|
|
54
81
|
limit?: number;
|
|
55
82
|
folder?: string;
|
package/dist/fts5.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"fts5.d.ts","sourceRoot":"","sources":["../src/fts5.ts"],"names":[],"mappings":"
|
|
1
|
+
{"version":3,"file":"fts5.d.ts","sourceRoot":"","sources":["../src/fts5.ts"],"names":[],"mappings":"AA2BA,MAAM,MAAM,YAAY,GAAG,WAAW,GAAG,SAAS,CAAC;AAEnD,oEAAoE;AACpE,MAAM,MAAM,SAAS,GAAG,IAAI,GAAG,KAAK,CAAC;AAErC,MAAM,WAAW,YAAY;IAC3B,QAAQ,EAAE,MAAM,CAAC;IACjB,WAAW,EAAE,MAAM,CAAC;IACpB,UAAU,EAAE,MAAM,CAAC;IACnB,QAAQ,EAAE,MAAM,CAAC;IACjB,OAAO,EAAE,MAAM,CAAC;IAChB,KAAK,EAAE,MAAM,CAAC;IACd,0EAA0E;IAC1E,IAAI,EAAE,SAAS,CAAC;CACjB;AAED,MAAM,WAAW,aAAa;IAC5B,KAAK,EAAE,MAAM,CAAC;IACd,OAAO,EAAE,MAAM,CAAC;IAChB,OAAO,EAAE,MAAM,CAAC;IAChB,SAAS,EAAE,MAAM,CAAC;IAClB,YAAY,EAAE,MAAM,CAAC;CACtB;AA4DD,qBAAa,QAAQ;IACnB,OAAO,CAAC,EAAE,CAAmB;IAC7B,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAS;IAC9B,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAe;IACxC,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAS;gBAEvB,IAAI,EAAE;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,YAAY,CAAA;KAAE;IAMxE,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;IAiB3B,iEAAiE;IAC3D,WAAW,IAAI,OAAO,CAAC,OAAO,CAAC;IAcrC,KAAK,IAAI,IAAI;IAOb,OAAO,CAAC,eAAe;IAuDvB,OAAO,CAAC,QAAQ;IAQhB,OAAO,CAAC,SAAS;IAMjB,OAAO,CAAC,SAAS;IAKjB;;;;;;;;;;;OAWG;IACH,IAAI,CACF,WAAW,EAAE,KAAK,CAAC;QAAE,OAAO,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAA;KAAE,CAAC,EACxD,IAAI,CAAC,EAAE,SAAS,GACf;QACD,KAAK,EAAE,MAAM,EAAE,CAAC;QAChB,OAAO,EAAE,MAAM,EAAE,CAAC;QAClB,OAAO,EAAE,MAAM,EAAE,CAAC;QAClB,SAAS,EAAE,MAAM,EAAE,CAAC;KACrB;IA0BD,oDAAoD;IACpD,QAAQ,CAAC,OAAO,EAAE,MAAM,GAAG,IAAI;IAM/B,iFAAiF;IACjF,WAAW,CACT,OAAO,EAAE,MAAM,EACf,OAAO,EAAE,MAAM,EACf,OAAO,EAAE,MAAM,EACf,eAAe,GAAE,MAAM,EAAO,EAC9B,IAAI,GAAE,MAAM,EAAO,GAClB,MAAM;IA8BT;;;;;;;;;;;OAWG;IACH,cAAc,CAAC,OAAO,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,KAAK,EAAE,aAAa,CAAC;QAAE,UAAU,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,MAAM,CAAA;KAAE,CAAC,GAAG,MAAM;IAsBpH,MAAM,CACJ,QAAQ,EAAE,MAAM,EAChB,IAAI,GAAE;QAAE,KAAK,CAAC,EAAE,MAAM,CAAC;QAAC,MAAM,CAAC,EAAE,MAAM,CAAC;QAAC,GAAG,CAAC,EAAE,MAAM,CAAC;QAAC,YAAY,CAAC,EAAE,MAAM,CAAA;KAAO,GAClF,YAAY,EAAE;IA+DjB;;;;;;;OAOG;IACH,QAAQ,CAAC,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,GAAG;QAAE,OAAO,EAAE,MAAM,CAAC;QAAC,UAAU,EAAE,MAAM,CAAC;QAAC,QAAQ,EAAE,MAAM,CAAA;KAAE,GAAG,IAAI;IAQ/G,WAAW,IAAI,MAAM;IAMrB,UAAU,IAAI,MAAM;CAKrB;AAMD,wBAAgB,aAAa,CAAC,CAAC,EAAE,MAAM,GAAG,MAAM,CAe/C;AAED,UAAU,YAAY;IACpB,IAAI,EAAE,MAAM,CAAC;IACb,SAAS,EAAE,MAAM,CAAC;IAClB,OAAO,EAAE,MAAM,CAAC;IAChB;;;;+EAI2E;IAC3E,UAAU,EAAE,MAAM,CAAC;CACpB;AAID;;;;;;;;GAQG;AACH,wBAAgB,YAAY,CAAC,OAAO,EAAE,MAAM,EAAE,QAAQ,SAAkB,GAAG,YAAY,EAAE,CAwDxF;AAuED,wBAAgB,gBAAgB,CAAC,SAAS,EAAE,MAAM,GAAG,MAAM,CAM1D"}
|
package/dist/fts5.js
CHANGED
|
@@ -14,7 +14,7 @@ import { createHash } from "node:crypto";
|
|
|
14
14
|
import { promises as fs } from "node:fs";
|
|
15
15
|
import * as os from "node:os";
|
|
16
16
|
import * as path from "node:path";
|
|
17
|
-
const SCHEMA_VERSION =
|
|
17
|
+
const SCHEMA_VERSION = 4;
|
|
18
18
|
// Lazy-loaded better-sqlite3 binding so missing native module surfaces only
|
|
19
19
|
// when --persistent-index is actually used.
|
|
20
20
|
//
|
|
@@ -133,12 +133,14 @@ export class FtsIndex {
|
|
|
133
133
|
line_end UNINDEXED,
|
|
134
134
|
tags UNINDEXED,
|
|
135
135
|
raw_content UNINDEXED,
|
|
136
|
+
kind UNINDEXED,
|
|
136
137
|
tokenize='${tokenizeArg}'
|
|
137
138
|
);
|
|
138
139
|
CREATE TABLE IF NOT EXISTS source_state (
|
|
139
140
|
rel_path TEXT PRIMARY KEY,
|
|
140
141
|
mtime_ms INTEGER NOT NULL,
|
|
141
142
|
n_chunks INTEGER NOT NULL,
|
|
143
|
+
kind TEXT NOT NULL DEFAULT 'md',
|
|
142
144
|
indexed_at TEXT NOT NULL
|
|
143
145
|
);
|
|
144
146
|
`);
|
|
@@ -171,10 +173,19 @@ export class FtsIndex {
|
|
|
171
173
|
* Diff the on-disk source_state against the live vault snapshot. Returns
|
|
172
174
|
* categorized lists; caller is expected to feed `added` + `updated` paths
|
|
173
175
|
* back into reindexFile() and pass `deleted` to dropFile().
|
|
176
|
+
*
|
|
177
|
+
* v2.8.0: optional `kind` filter — when set, the diff only considers
|
|
178
|
+
* source_state rows of that kind. Lets the markdown-sync and PDF-sync
|
|
179
|
+
* paths run independently against the same DB without one's "missing
|
|
180
|
+
* files" being mistakenly deleted by the other. Default `undefined`
|
|
181
|
+
* means "all kinds" (used by older callers + diff queries that want
|
|
182
|
+
* a global view).
|
|
174
183
|
*/
|
|
175
|
-
diff(liveEntries) {
|
|
184
|
+
diff(liveEntries, kind) {
|
|
176
185
|
const db = this.requireDb();
|
|
177
|
-
const stored =
|
|
186
|
+
const stored = kind !== undefined
|
|
187
|
+
? db.prepare("SELECT rel_path, mtime_ms FROM source_state WHERE kind = ?").all(kind)
|
|
188
|
+
: db.prepare("SELECT rel_path, mtime_ms FROM source_state").all();
|
|
178
189
|
const storedMap = new Map();
|
|
179
190
|
for (const r of stored)
|
|
180
191
|
storedMap.set(r.rel_path, r.mtime_ms);
|
|
@@ -205,12 +216,12 @@ export class FtsIndex {
|
|
|
205
216
|
db.prepare("DELETE FROM chunks WHERE rel_path = ?").run(relPath);
|
|
206
217
|
db.prepare("DELETE FROM source_state WHERE rel_path = ?").run(relPath);
|
|
207
218
|
}
|
|
208
|
-
/** Re-chunk a single file, replacing its existing chunks atomically. */
|
|
219
|
+
/** Re-chunk a single markdown file, replacing its existing chunks atomically. */
|
|
209
220
|
reindexFile(relPath, mtimeMs, content, wikilinkTargets = [], tags = []) {
|
|
210
221
|
const db = this.requireDb();
|
|
211
222
|
const chunks = chunkContent(content);
|
|
212
223
|
db.prepare("DELETE FROM chunks WHERE rel_path = ?").run(relPath);
|
|
213
|
-
const insert = db.prepare("INSERT INTO chunks (content, rel_path, chunk_index, line_start, line_end, tags, raw_content) VALUES (?, ?, ?, ?, ?, ?,
|
|
224
|
+
const insert = db.prepare("INSERT INTO chunks (content, rel_path, chunk_index, line_start, line_end, tags, raw_content, kind) VALUES (?, ?, ?, ?, ?, ?, ?, 'md')");
|
|
214
225
|
// `tags` is a comma-delimited list so the filter LIKE pattern can wrap it
|
|
215
226
|
// with leading/trailing commas for exact-tag matching at query time.
|
|
216
227
|
const tagsSerialized = tags.length ? tags.join(",") : "";
|
|
@@ -228,7 +239,36 @@ export class FtsIndex {
|
|
|
228
239
|
const enriched = `${breadcrumbPrefix}${c.text}${linksSuffix}`;
|
|
229
240
|
insert.run(enriched, relPath, i, c.lineStart, c.lineEnd, tagsSerialized, c.text);
|
|
230
241
|
});
|
|
231
|
-
db.prepare("INSERT OR REPLACE INTO source_state (rel_path, mtime_ms, n_chunks, indexed_at) VALUES (?, ?, ?, ?)").run(relPath, mtimeMs, chunks.length, new Date().toISOString());
|
|
242
|
+
db.prepare("INSERT OR REPLACE INTO source_state (rel_path, mtime_ms, n_chunks, kind, indexed_at) VALUES (?, ?, ?, 'md', ?)").run(relPath, mtimeMs, chunks.length, new Date().toISOString());
|
|
243
|
+
return chunks.length;
|
|
244
|
+
}
|
|
245
|
+
/**
|
|
246
|
+
* v2.8.0 — re-chunk a single PDF, replacing its existing chunks atomically.
|
|
247
|
+
* Caller pre-extracts page text via `extractPdfText` (src/pdf.ts) so this
|
|
248
|
+
* method stays decoupled from pdfjs-dist (which is an optionalDependency).
|
|
249
|
+
*
|
|
250
|
+
* Page boundaries are preserved as `[page: N]` markers in the joined text
|
|
251
|
+
* before chunking — the chunker may split a page across chunks or merge
|
|
252
|
+
* short pages, but the markers travel with the text so search snippets
|
|
253
|
+
* carry page citations. Same `chunkContent` pipeline as markdown so chunk
|
|
254
|
+
* IDs match across the BM25 / TF-IDF / embeddings rankers (RRF requires
|
|
255
|
+
* stable IDs).
|
|
256
|
+
*/
|
|
257
|
+
reindexPdfFile(relPath, mtimeMs, pages) {
|
|
258
|
+
const db = this.requireDb();
|
|
259
|
+
// Join pages with explicit `[page: N]` markers so the chunker can carry
|
|
260
|
+
// page provenance through. Empty pages (image-only / scanned) still get
|
|
261
|
+
// a marker so chunks downstream of them can still cite the right page.
|
|
262
|
+
const joined = pages.map((p) => `[page: ${p.pageNumber}]\n${p.text}`).join("\n\n");
|
|
263
|
+
const chunks = chunkContent(joined);
|
|
264
|
+
db.prepare("DELETE FROM chunks WHERE rel_path = ?").run(relPath);
|
|
265
|
+
const insert = db.prepare("INSERT INTO chunks (content, rel_path, chunk_index, line_start, line_end, tags, raw_content, kind) VALUES (?, ?, ?, ?, ?, '', ?, 'pdf')");
|
|
266
|
+
chunks.forEach((c, i) => {
|
|
267
|
+
// No wikilink/tag enrichment for PDFs (they don't have either). The
|
|
268
|
+
// page marker is already in c.text so it shows up in snippets.
|
|
269
|
+
insert.run(c.text, relPath, i, c.lineStart, c.lineEnd, c.text);
|
|
270
|
+
});
|
|
271
|
+
db.prepare("INSERT OR REPLACE INTO source_state (rel_path, mtime_ms, n_chunks, kind, indexed_at) VALUES (?, ?, ?, 'pdf', ?)").run(relPath, mtimeMs, chunks.length, new Date().toISOString());
|
|
232
272
|
return chunks.length;
|
|
233
273
|
}
|
|
234
274
|
search(rawQuery, opts = {}) {
|
|
@@ -262,6 +302,7 @@ export class FtsIndex {
|
|
|
262
302
|
const sql = `
|
|
263
303
|
SELECT chunks.rel_path AS rel_path, chunks.chunk_index AS chunk_index,
|
|
264
304
|
chunks.line_start AS line_start, chunks.line_end AS line_end,
|
|
305
|
+
chunks.kind AS kind,
|
|
265
306
|
snippet(chunks, 0, '«', '»', '…', 25) AS snippet,
|
|
266
307
|
bm25(chunks) AS score
|
|
267
308
|
FROM chunks
|
|
@@ -277,6 +318,10 @@ export class FtsIndex {
|
|
|
277
318
|
chunk_index: r.chunk_index,
|
|
278
319
|
line_start: r.line_start,
|
|
279
320
|
line_end: r.line_end,
|
|
321
|
+
// v2.8.0: kind defaults to "md" for chunks indexed before the schema
|
|
322
|
+
// bump (legacy DBs auto-rebuild via SCHEMA_VERSION mismatch, but the
|
|
323
|
+
// null fallback is defense-in-depth).
|
|
324
|
+
kind: (r.kind === "pdf" ? "pdf" : "md"),
|
|
280
325
|
snippet: r.snippet,
|
|
281
326
|
score: -r.score // BM25 is negative; flip so higher = better for callers
|
|
282
327
|
}));
|