@andespindola/brainlink 1.0.6 → 1.1.0

package/CHANGELOG.md

@@ -1,5 +1,27 @@
 # Changelog
 
+## Unreleased
+
+- Added an optional **binary storage backend** (`storageBackend: "binary"`) that stores chunk embeddings as dense little-endian Float32 (`vectors.f32`) and document/chunk/link metadata as Brotli-compressed JSON, cutting the on-disk index ~75% on a typical vault with much cheaper loads than parsing decimal-text vectors. The graph/search/link reads now run through a shared query core so the JSON and binary backends are behavior-identical for the same candidates. Switching to `binary` converts an existing `index.json` in place on first use (no re-embedding) and falls back to it for reads until the binary store is materialized. The canonical Markdown is unaffected and the index stays fully regenerable.
+- Introduced a `KnowledgeStore` port and a shared, backend-agnostic ranker, so the index storage layout can evolve without touching the application layer and every backend produces an identical ranking.
+- Added **vault versioning** for the canonical Markdown. Git: `vault-init-git`, `vault-commit`, `vault-status`, `vault-history`, `vault-push`, `vault-pull`, `vault-restore`, `vault-clone` (driven through the system `git` binary; the derived `.brainlink/` is git-ignored). Portable snapshots: `vault-export`/`vault-import` produce and restore a self-contained gzip `.blvault` with per-file SHA-256. Every operation that changes Markdown reindexes automatically so the vault is immediately usable, merge conflicts abort cleanly instead of leaving markers in Markdown, conflicting local edits are preserved as `*.conflict-<ts>.md`, and remote URLs with inline credentials are rejected.
+- Pinned the transitive `fast-uri` dependency to `>=4.1.0` to clear a high-severity Snyk advisory affecting the entire 3.x line.
+
+- Added `brainlink_similar_notes` MCP tool: find the notes most similar to a given note (selected by title or path), using the note's own content as the retrieval query and excluding the note itself.
+- Added batch write tools `brainlink_add_notes` and `brainlink_delete_notes` that write or delete up to 100 notes in a single call and reindex once for the whole batch; batch deletion reports per-note failures without aborting the batch.
+
+- Made chunking heading-aware: a Markdown heading starts a new chunk once the current chunk already holds substantial content, so large notes split on topical (heading) boundaries instead of mid-section, while small notes and trailing metadata sections (e.g. `## Context Links`) stay in a single chunk. The new boundaries take effect on the next full reindex (`index --full`); existing chunks are reused until then.
+- Added maximal-marginal-relevance ordering to context assembly so the token budget is not spent on several near-duplicate notes: each selected document balances its own relevance against how similar it is to documents already chosen. The strongest document still leads.
+
+- Hardened index reliability: a corrupt `index.json` is now detected, logged and quarantined to `index.json.corrupt` (so the next `index --full` rebuilds it) instead of being silently read as an empty index. Search treats corruption as a signal to fall back to the independent compressed search packs rather than returning no results.
+
+- Added pluggable external embedding providers (`ollama`, `openai`) alongside the default deterministic `local` provider, configured through a new optional `embedding` block (`url`, `model`, `apiKeyEnv`, `timeoutMs`, `batchSize`). External vectors are L2-normalized to match the local cosine/bucket contract, and the API key is referenced by environment variable name, never stored in config.
+- Switching the embedding provider or external model now rebuilds vectors automatically (the embedding model is folded into the index reindex signature). If an external provider is unreachable, query-time search degrades to lexical-only while index-time surfaces a clear error instead of mixing embedding spaces.
+
+- Replaced the naive additive lexical score with field-weighted BM25 ranking (IDF, term-frequency saturation and chunk-length normalization) so rare query terms outweigh common ones and long chunks no longer dominate on raw hit counts.
+- Reworked hybrid search to combine lexical and semantic components on a shared min-max-normalized [0,1] scale instead of the fixed `text + semantic * 8` blend, keeping neither the unbounded lexical magnitude nor the bounded cosine value able to swamp the other. Relevance filtering now runs on raw component scores before normalization so a legitimate weakest-in-mode match is never scaled to zero and dropped.
+- Context token budgeting now reuses the per-chunk token count cached at index time and shares a single, more accurate token estimator (`max(words * 1.3, chars / 4)`), removing the duplicated character-ratio heuristic while staying conservative so rendered packages remain within budget.
+
 ## 1.0.0
 
 - Promoted Brainlink to the first stable functional release.

package/README.md

@@ -540,14 +540,17 @@ Available tools:
 - `brainlink_context_packs`: list or clear persisted CAG context packs.
 - `brainlink_search`: search indexed notes.
 - `brainlink_explain`: explain why indexed notes matched a query.
+- `brainlink_similar_notes`: find the notes most similar to a given note (by title or path), excluding the note itself.
 - `brainlink_dedupe`: detect duplicate candidates using exact hash + semantic similarity scores.
 - `brainlink_resolve_duplicate`: resolve duplicate pairs (`merge`, `link`, `ignore`) with connectivity-safe fallback edges.
 - `brainlink_add_note`: write durable Markdown memory and reindex.
+- `brainlink_add_notes`: write several notes in one batch, reindexing once.
 - `brainlink_remember`: capture durable memory with inferred title, tags and Context Links; supports dry-run.
 - `brainlink_inbox_add`: capture a quick untriaged memory item.
 - `brainlink_inbox_list`: list untriaged inbox memory items.
 - `brainlink_inbox_process`: suggest titles, tags and links for inbox items.
 - `brainlink_delete_note`: delete a durable Markdown note by title or path after explicit confirmation and reindex.
+- `brainlink_delete_notes`: delete several notes in one confirmed batch, reindexing once; per-note failures are reported.
 - `brainlink_add_file`: ingest a local file as a note and reindex.
 - `brainlink_canonicalize_context_links`: ensure existing notes link to inferred context hubs.
 - `brainlink_volatile_add`: write temporary agent-decided memory with TTL; volatile sections are included in context and never create durable graph edges.
@@ -756,6 +759,17 @@ When the configured default vault is changed manually in config files, Brainlink
 Use `--global` to write to `$BRAINLINK_HOME/brainlink.config.json`, `--no-migrate` to skip migration, and `--no-index` to skip post-migration indexing.
 `config doctor` is dry-run by default; use `--fix` to apply safe config normalization and allowlist fixes.
 
+#### Storage backend
+
+`storageBackend` selects how the derived index is stored (the canonical Markdown is unaffected):
+
+- `json` (default) — the original single `index.json`.
+- `binary` — Float32 vectors plus Brotli-compressed metadata; a smaller, faster-to-load index (~75% smaller on a typical vault). Switching is seamless: an existing `index.json` is converted in place on first use (no re-embedding) and reads fall back to it until then.
+
+```json
+{ "storageBackend": "binary" }
+```
+
 ### `vaults`
 
 ```bash
@@ -909,6 +923,30 @@ blink pack-backup --vault ./vault --json
 Creates an offline backup artifact of encrypted search packs with a second compression pass.
 This is intentionally outside the online retrieval path (`index`, `search`, `context`).
 
+### Vault Versioning
+
+Version the canonical Markdown with git, or move a vault as a single portable file.
+The derived `.brainlink/` directory is never versioned; any operation that changes
+Markdown reindexes automatically so the vault stays immediately usable.
+
+```bash
+# Git: history, remotes, restore
+blink vault-init-git --remote git@github.com:you/your-vault.git
+blink vault-commit -m "snapshot notes"
+blink vault-push
+blink vault-pull                 # pulls and reindexes changed notes
+blink vault-history --limit 20
+blink vault-restore <commit>     # restore Markdown from a ref and reindex
+blink vault-clone git@github.com:you/your-vault.git ./vault   # clone + build index
+blink vault-status
+
+# Portable snapshot (self-contained, index excluded; import rebuilds it)
+blink vault-export -o vault.blvault
+blink vault-import vault.blvault  # conflicting local edits preserved as *.conflict-<ts>.md
+```
+
+Inline credentials in a remote URL are rejected — use SSH or a git credential helper.
+
 ### `agents`
 
 ```bash
@@ -1214,6 +1252,41 @@ spec:
 
 Use `"embeddingProvider": "none"` when you want FTS-only indexing.
 
+### Embedding providers
+
+`embeddingProvider` selects how chunk and query embeddings are produced:
+
+- `local` (default): deterministic, dependency-free local vectors. No network.
+- `none`: FTS-only; no semantic vectors.
+- `ollama`: embeddings from a local [Ollama](https://ollama.com) server.
+- `openai`: embeddings from the OpenAI embeddings API.
+
+External providers are configured through the optional `embedding` block. The API key is referenced by environment variable name and never stored in the config file:
+
+```json
+{
+  "embeddingProvider": "ollama",
+  "embedding": {
+    "url": "http://127.0.0.1:11434",
+    "model": "nomic-embed-text",
+    "timeoutMs": 10000,
+    "batchSize": 64
+  }
+}
+```
+
+```json
+{
+  "embeddingProvider": "openai",
+  "embedding": {
+    "model": "text-embedding-3-small",
+    "apiKeyEnv": "OPENAI_API_KEY"
+  }
+}
+```
+
+Switching the provider — or the external `model` — changes the embedding space, so the next index run rebuilds vectors automatically. If an external provider is unreachable at query time, search degrades to lexical-only instead of failing; at index time it surfaces a clear error rather than mixing embedding spaces.
+
 For local security checks, set your Snyk token in the environment:
 
 ```bash

package/dist/application/find-similar-notes.js

@@ -0,0 +1,75 @@
+import { resolve } from 'node:path';
+import { parseMarkdownDocument } from '../domain/markdown.js';
+import { ensureVault, readMarkdownFiles } from '../infrastructure/file-system-vault.js';
+import { searchKnowledge } from './search-knowledge.js';
+const defaultLimit = 5;
+// Pull extra candidates before deduping by document and dropping the note
+// itself, so a note with many chunks does not crowd out distinct results.
+const candidateOverscan = 8;
+const maxQueryCharacters = 2000;
+const normalizeTitleKey = (title) => title.trim().replace(/\.md$/i, '').toLowerCase();
+const resolveTarget = async (vaultPath, input) => {
+    const absoluteVaultPath = await ensureVault(vaultPath);
+    const files = await readMarkdownFiles(vaultPath);
+    const documents = files.map((file) => ({
+        file,
+        document: parseMarkdownDocument({
+            absolutePath: file.absolutePath,
+            vaultPath: absoluteVaultPath,
+            content: file.content,
+            createdAt: file.createdAt,
+            updatedAt: file.updatedAt
+        })
+    }));
+    if (input.path != null && input.path.trim().length > 0) {
+        const expected = resolve(absoluteVaultPath, input.path);
+        const match = documents.find((entry) => resolve(entry.file.absolutePath) === expected);
+        if (!match) {
+            throw new Error(`No note found at path: ${input.path}`);
+        }
+        return { title: match.document.title, path: match.document.path, content: match.file.content };
+    }
+    const expectedTitle = normalizeTitleKey(input.title ?? '');
+    const matches = documents.filter((entry) => normalizeTitleKey(entry.document.title) === expectedTitle &&
+        (input.agentId == null || entry.document.agentId === input.agentId));
+    if (matches.length === 0) {
+        throw new Error(`No note found with title: ${input.title}`);
+    }
+    if (matches.length > 1) {
+        throw new Error(`Multiple notes match title "${input.title}". Use path instead: ${matches.map((entry) => entry.document.path).join(', ')}`);
+    }
+    return { title: matches[0].document.title, path: matches[0].document.path, content: matches[0].file.content };
+};
+const buildQuery = (target) => `${target.title}\n${target.content}`.slice(0, maxQueryCharacters);
+// Find notes most similar to a given note by using the note's own title and
+// content as the retrieval query, then excluding the note itself.
+export const findSimilarNotes = async (vaultPath, input) => {
+    const hasTitle = input.title != null && input.title.trim().length > 0;
+    const hasPath = input.path != null && input.path.trim().length > 0;
+    if (hasTitle === hasPath) {
+        throw new Error('Use exactly one selector: title or path.');
+    }
+    const target = await resolveTarget(vaultPath, input);
+    const limit = Math.max(1, input.limit ?? defaultLimit);
+    const results = await searchKnowledge(vaultPath, buildQuery(target), limit + candidateOverscan, input.agentId, input.mode ?? 'hybrid');
+    const bestByPath = new Map();
+    for (const result of results) {
+        if (result.path === target.path) {
+            continue;
+        }
+        const existing = bestByPath.get(result.path);
+        if (!existing || result.score > existing.score) {
+            bestByPath.set(result.path, {
+                title: result.title,
+                path: result.path,
+                score: result.score,
+                searchMode: result.searchMode,
+                tags: result.tags
+            });
+        }
+    }
+    const similar = Array.from(bestByPath.values())
+        .sort((left, right) => right.score - left.score || left.title.localeCompare(right.title))
+        .slice(0, limit);
+    return { target: { title: target.title, path: target.path }, similar };
+};

package/dist/application/get-graph-node.js

@@ -1,8 +1,8 @@
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { openFileIndex } from '../infrastructure/file-index.js';
+import { openKnowledgeStore } from '../infrastructure/knowledge-store/index.js';
 export const getGraphNode = async (vaultPath, id, agentId) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openFileIndex(absoluteVaultPath);
+    const index = openKnowledgeStore(absoluteVaultPath);
     try {
         return await index.getGraphNode(id, agentId);
     }

package/dist/application/get-graph-summary.js

@@ -1,8 +1,8 @@
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { openFileIndex } from '../infrastructure/file-index.js';
+import { openKnowledgeStore } from '../infrastructure/knowledge-store/index.js';
 export const getGraphSummary = async (vaultPath, agentId) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openFileIndex(absoluteVaultPath);
+    const index = openKnowledgeStore(absoluteVaultPath);
     try {
         return await index.getGraphSummary(agentId);
     }

package/dist/application/get-graph.js

@@ -1,8 +1,8 @@
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { openFileIndex } from '../infrastructure/file-index.js';
+import { openKnowledgeStore } from '../infrastructure/knowledge-store/index.js';
 export const getGraph = async (vaultPath, agentId) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openFileIndex(absoluteVaultPath);
+    const index = openKnowledgeStore(absoluteVaultPath);
     try {
         return await index.getGraph(agentId);
     }

package/dist/application/index-vault-phases.js

@@ -1,6 +1,7 @@
 import { createIndexedDocument, graphLinkModelVersion } from '../domain/markdown.js';
 import { sharedAgentId } from '../domain/agents.js';
 import { createEmbeddingBuckets, createEmbeddingProvider } from '../domain/embeddings.js';
+import { embeddingSignature } from '../infrastructure/config.js';
 import { buildSearchPacks, toSearchPackBuildOptions } from '../infrastructure/search-packs.js';
 const toTitleKey = (title) => title.toLowerCase();
 const appendTitleEntry = (map, document) => {
@@ -30,11 +31,11 @@ export const createTitleMaps = (documents) => [...documents]
 const createScopedTitleResolver = (document, titleMaps) => ({
     get: (title) => titleMaps.byAgent.get(document.agentId)?.get(title)?.id ?? titleMaps.shared.get(title)?.id
 });
-const embedIndexedDocuments = async (documents, providerName) => {
+const embedIndexedDocuments = async (documents, providerName, embeddingConfig) => {
     if (documents.length === 0) {
         return documents;
     }
-    const provider = createEmbeddingProvider(providerName);
+    const provider = createEmbeddingProvider(providerName, embeddingConfig);
     const chunks = documents.flatMap((document) => document.chunks);
     const embeddings = await provider.embed(chunks.map((chunk) => chunk.content));
     const embeddingByChunkId = new Map(chunks.map((chunk, index) => [chunk.id, embeddings[index] ?? []]));
@@ -70,7 +71,7 @@ export const detectChanges = (params) => {
     const fullSourceReindex = fullReindex || graphLinkModelChanged;
     const settingsChanged = previousState == null ||
         previousState.chunkSize !== config.chunkSize ||
-        previousState.embeddingProvider !== config.embeddingProvider ||
+        previousState.embeddingProvider !== embeddingSignature(config) ||
         graphLinkModelChanged;
     const packSettingsChanged = previousState == null ||
         previousState.searchPackRowChunkSize !== config.searchPack.rowChunkSize ||
@@ -135,7 +136,7 @@ export const embedChangedDocuments = async (params) => {
         changedDocuments: changedDocumentsByPath.size
     });
     const changedIndexedDocuments = changedDocumentsByPath.size > 0
-        ? await embedIndexedDocuments(Array.from(changedDocumentsByPath.values()).map((document) => createIndexedDocument(document, createScopedTitleResolver(document, titleMaps), config.chunkSize)), config.embeddingProvider)
+        ? await embedIndexedDocuments(Array.from(changedDocumentsByPath.values()).map((document) => createIndexedDocument(document, createScopedTitleResolver(document, titleMaps), config.chunkSize)), config.embeddingProvider, config.embedding)
         : [];
     emit('embed', changedDocumentsByPath.size > 0 ? 'finish' : 'skip', changedDocumentsByPath.size > 0 ? 'Embedding complete' : 'Embedding skipped', {
         changedIndexedDocuments: changedIndexedDocuments.length

package/dist/application/index-vault.js

@@ -1,10 +1,10 @@
 import { readFile } from 'node:fs/promises';
 import { graphLinkModelVersion, parseMarkdownDocument } from '../domain/markdown.js';
-import { loadBrainlinkConfig } from '../infrastructure/config.js';
+import { embeddingSignature, loadBrainlinkConfig } from '../infrastructure/config.js';
 import { ensureVault, readMarkdownFileSummaries } from '../infrastructure/file-system-vault.js';
 import { readIndexState, writeIndexState } from '../infrastructure/index-state.js';
 import { ensureSearchPackManifest } from '../infrastructure/search-packs.js';
-import { openFileIndex } from '../infrastructure/file-index.js';
+import { openKnowledgeStore } from '../infrastructure/knowledge-store/index.js';
 import { assembleIndexedDocuments, createTitleMaps, decidePackRebuild, detectChanges, embedChangedDocuments, rebuildSearchPacksIfNeeded } from './index-vault-phases.js';
 const toIndexResult = (documents) => ({
     documentCount: documents.length,
@@ -56,7 +56,7 @@ export const indexVaultWithOptions = async (vaultPath, options) => {
         hasPreviousState: previousState != null
     });
     const fullReindex = options.full === true;
-    const index = openFileIndex(absoluteVaultPath);
+    const index = openKnowledgeStore(absoluteVaultPath);
     try {
         const existingIndexedDocuments = await index.getIndexedDocuments();
         const existingByPath = new Map(existingIndexedDocuments.map((document) => [document.document.path, document]));
@@ -161,7 +161,7 @@ export const indexVaultWithOptions = async (vaultPath, options) => {
         const packResultReason = shouldRebuildPacks && !packsRebuilt ? `${packReason} (failed)` : packReason;
         await writeIndexState(absoluteVaultPath, {
             chunkSize: config.chunkSize,
-            embeddingProvider: config.embeddingProvider,
+            embeddingProvider: embeddingSignature(config),
             graphLinkModelVersion,
             searchPackRowChunkSize: config.searchPack.rowChunkSize,
             searchPackCompressionLevel: config.searchPack.compressionLevel,

package/dist/application/list-agents.js

@@ -1,8 +1,8 @@
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { openFileIndex } from '../infrastructure/file-index.js';
+import { openKnowledgeStore } from '../infrastructure/knowledge-store/index.js';
 export const listAgents = async (vaultPath) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openFileIndex(absoluteVaultPath);
+    const index = openKnowledgeStore(absoluteVaultPath);
     try {
         return await index.listAgents();
     }

package/dist/application/list-links.js

@@ -1,8 +1,8 @@
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { openFileIndex } from '../infrastructure/file-index.js';
+import { openKnowledgeStore } from '../infrastructure/knowledge-store/index.js';
 export const listLinks = async (vaultPath, agentId) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openFileIndex(absoluteVaultPath);
+    const index = openKnowledgeStore(absoluteVaultPath);
     try {
         return await index.listLinks(agentId);
     }
@@ -12,7 +12,7 @@ export const listLinks = async (vaultPath, agentId) => {
 };
 export const listBacklinks = async (vaultPath, title, agentId) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openFileIndex(absoluteVaultPath);
+    const index = openKnowledgeStore(absoluteVaultPath);
     try {
         return await index.listBacklinks(title, agentId);
     }

package/dist/application/memory-suggestions.js

@@ -127,7 +127,10 @@ export const suggestContextLinks = async (vaultPath, content, agentId, limit) =>
     for (const result of searchResults) {
         const key = result.title.toLowerCase();
         const current = byTitle.get(key);
-        const score = Number(Math.min(1, Math.max(result.score / 20, result.semanticScore)).toFixed(4));
+        // Hybrid search already returns a min-max-normalized [0,1] score, so it is
+        // compared directly against the cosine semanticScore (also [0,1]) instead of
+        // being rescaled from the previous unbounded blend.
+        const score = Number(Math.min(1, Math.max(result.score, result.semanticScore)).toFixed(4));
         if (!current || score > current.score) {
             byTitle.set(key, {
                 title: result.title,

package/dist/application/ports/knowledge-store.js

@@ -0,0 +1 @@
+export {};

package/dist/application/ranking/run-search.js

@@ -0,0 +1,177 @@
+// Backend-agnostic ranking. Given a candidate set (chunks already selected by a
+// storage backend), this module owns the deterministic scoring math: field
+// weighting, BM25 lexical relevance, cosine semantic relevance, hybrid fusion,
+// and stable ordering. Keeping it here means every KnowledgeStore backend
+// (JSON, binary, LMDB) produces an identical ranking for the same candidates.
+import { dotProduct } from '../../domain/embeddings.js';
+import { bm25TermScore, minMaxNormalize } from '../../domain/scoring.js';
+const queryTokenPattern = /[\p{L}\p{N}_-]+/gu;
+export const normalizeToken = (value) => value
+    .normalize('NFKD')
+    .replace(/\p{Diacritic}/gu, '')
+    .toLowerCase();
+export const tokenize = (query) => query
+    .match(queryTokenPattern)
+    ?.map(normalizeToken)
+    .filter((token) => token.length > 1) ?? [];
+const countOccurrences = (text, token) => {
+    let hits = 0;
+    let cursor = 0;
+    while (cursor < text.length) {
+        const index = text.indexOf(token, cursor);
+        if (index < 0) {
+            break;
+        }
+        hits += 1;
+        cursor = index + token.length;
+    }
+    return hits;
+};
+const titleFieldWeight = 5;
+const tagFieldWeight = 4;
+const pathFieldWeight = 2;
+const contentHitCap = 6;
+const hybridTextWeight = 0.6;
+const hybridSemanticWeight = 0.4;
+const normalizeFields = (row) => ({
+    title: normalizeToken(row.title),
+    path: normalizeToken(row.path),
+    content: normalizeToken(row.content),
+    tags: normalizeToken(row.tags.join(' '))
+});
+// Field-weighted term frequency: title/tag/path matches count more than body
+// matches, and body hits saturate at a cap so a term repeated many times in one
+// chunk cannot dominate. This weighted frequency feeds BM25 below.
+const weightedTermFrequency = (fields, token) => {
+    const titleHits = countOccurrences(fields.title, token);
+    const tagHits = countOccurrences(fields.tags, token);
+    const pathHits = countOccurrences(fields.path, token);
+    const contentHits = countOccurrences(fields.content, token);
+    return (titleHits * titleFieldWeight +
+        tagHits * tagFieldWeight +
+        pathHits * pathFieldWeight +
+        Math.min(contentHits, contentHitCap));
+};
+// Naive additive field score retained for graph-node title/path lookup, where
+// the candidate set is documents (not chunks) and corpus-wide BM25 statistics
+// add no signal over a direct weighted hit count.
+export const lexicalFieldScore = (row, tokens) => {
+    if (tokens.length === 0) {
+        return 0;
+    }
+    const fields = normalizeFields(row);
+    return tokens.reduce((score, token) => score + weightedTermFrequency(fields, token), 0);
+};
+// A single chunk with a non-finite cached tokenCount (legacy index, partial
+// write, manual edit) would otherwise poison the shared averageLength and zero
+// out lexical scoring for the whole corpus, so coerce to a safe positive length.
+const safeDocumentLength = (tokenCount) => Number.isFinite(tokenCount) && tokenCount > 0 ? tokenCount : 1;
+const buildCorpusStatistics = (prepared, tokens) => {
+    const documentCount = prepared.length;
+    const totalLength = prepared.reduce((sum, entry) => sum + safeDocumentLength(entry.row.tokenCount), 0);
+    const documentFrequencyByTerm = new Map();
+    for (const term of tokens) {
+        let documentFrequency = 0;
+        for (const entry of prepared) {
+            if ((entry.weightedTermFrequencies.get(term) ?? 0) > 0) {
+                documentFrequency += 1;
+            }
+        }
+        documentFrequencyByTerm.set(term, documentFrequency);
+    }
+    return {
+        documentCount,
+        averageLength: documentCount > 0 ? totalLength / documentCount : 0,
+        documentFrequencyByTerm
+    };
+};
+const bm25TextScore = (entry, tokens, stats) => tokens.reduce((score, term) => score +
+    bm25TermScore({
+        termFrequency: entry.weightedTermFrequencies.get(term) ?? 0,
+        documentLength: safeDocumentLength(entry.row.tokenCount),
+        documentCount: stats.documentCount,
+        documentFrequency: stats.documentFrequencyByTerm.get(term) ?? 0,
+        averageLength: stats.averageLength
+    }), 0);
+const semanticScore = (row, queryEmbedding) => queryEmbedding.length > 0 && row.embedding.length > 0 ? dotProduct(queryEmbedding, row.embedding) : 0;
+const rangeOf = (values) => values.reduce((range, value) => ({ min: Math.min(range.min, value), max: Math.max(range.max, value) }), { min: Infinity, max: -Infinity });
+const toResult = (row, mode, score, text, semantic) => ({
+    documentId: row.documentId,
+    agentId: row.agentId,
+    title: row.title,
+    path: row.path,
+    chunkId: row.chunkId,
+    chunkOrdinal: row.chunkOrdinal,
+    content: row.content,
+    score,
+    textScore: text,
+    semanticScore: semantic,
+    tokenCount: row.tokenCount,
+    searchMode: mode,
+    tags: row.tags
+});
+// Rank an already-selected candidate set. Backends own candidate selection
+// (full scan, bucket prefilter, ANN); this function owns scoring and ordering so
+// the result is identical regardless of how the candidates were gathered.
+export const runSearch = (candidates, query, limit, mode, queryEmbedding = []) => {
+    const tokens = tokenize(query);
+    // Pass 1: field-weighted term frequencies per candidate, then corpus-wide
+    // statistics (document frequency, average length) needed for BM25 IDF and
+    // length normalization.
+    const prepared = candidates.map((row) => {
+        const fields = normalizeFields(row);
+        const weightedTermFrequencies = new Map();
+        for (const term of tokens) {
+            const frequency = weightedTermFrequency(fields, term);
+            if (frequency > 0) {
+                weightedTermFrequencies.set(term, frequency);
+            }
+        }
+        return { row, weightedTermFrequencies };
+    });
+    const stats = buildCorpusStatistics(prepared, tokens);
+    // Pass 2: raw lexical (BM25) and semantic (cosine) component scores.
+    const componentScored = prepared.map((entry) => ({
+        row: entry.row,
+        text: tokens.length > 0 ? bm25TextScore(entry, tokens, stats) : 0,
+        semantic: semanticScore(entry.row, queryEmbedding)
+    }));
+    // Filter on raw relevance before normalization so a legitimate match that
+    // happens to be the weakest in its mode is never min-max scaled to zero
+    // and dropped. fts stays lexical-only and semantic stays vector-only.
+    const relevant = componentScored.filter((entry) => {
+        if (tokens.length === 0) {
+            return true;
+        }
+        if (mode === 'fts') {
+            return entry.text > 0;
+        }
+        if (mode === 'semantic') {
+            return entry.semantic > 0;
+        }
+        return entry.text > 0 || entry.semantic > 0;
+    });
+    if (relevant.length === 0) {
+        return [];
+    }
+    const textRange = rangeOf(relevant.map((entry) => entry.text));
+    const semanticRange = rangeOf(relevant.map((entry) => entry.semantic));
+    const finalScore = (entry) => {
+        if (mode === 'fts') {
+            return entry.text;
+        }
+        if (mode === 'semantic') {
+            return entry.semantic;
+        }
+        // Hybrid mixes the two components on a shared [0, 1] scale so neither the
+        // unbounded BM25 magnitude nor the bounded cosine value can swamp the
+        // other, replacing the previous fixed `text + semantic * 8` blend.
+        const textNorm = minMaxNormalize(entry.text, textRange.min, textRange.max);
+        const semanticNorm = minMaxNormalize(entry.semantic, semanticRange.min, semanticRange.max);
+        return textNorm * hybridTextWeight + semanticNorm * hybridSemanticWeight;
+    };
+    return relevant
+        .map((entry) => toResult(entry.row, mode, finalScore(entry), entry.text, entry.semantic))
+        .sort((left, right) => right.score - left.score || left.title.localeCompare(right.title))
+        .slice(0, Math.max(0, limit));
+};

package/dist/application/search-graph-node-ids.js

@@ -1,6 +1,7 @@
 import { stat } from 'node:fs/promises';
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { indexStoragePath, openFileIndex } from '../infrastructure/file-index.js';
+import { indexStoragePath } from '../infrastructure/file-index.js';
+import { openKnowledgeStore } from '../infrastructure/knowledge-store/index.js';
 import { getGraphLayout } from './get-graph-layout.js';
 const graphSearchCacheTtlMs = 20_000;
 const graphSearchCacheMaxEntries = 120;
@@ -52,7 +53,7 @@ export const searchGraphNodeIds = async (vaultPath, query, limit, agentId, conte
     const contextNodeIds = context
         ? new Set((await getGraphLayout(absoluteVaultPath, { agentId, context })).layout.nodes.map((node) => node.id))
         : new Set();
-    const index = openFileIndex(absoluteVaultPath);
+    const index = openKnowledgeStore(absoluteVaultPath);
     try {
         const searchLimit = context ? Math.max(limit, 5000) : limit;
         const foundNodeIds = await index.searchGraphNodeIds(query, searchLimit, agentId);

package/dist/application/search-knowledge.js

@@ -1,9 +1,20 @@
 import { stat } from 'node:fs/promises';
 import { ensureVault } from '../infrastructure/file-system-vault.js';
 import { ensurePrivatePacksFromLegacyIndex, searchInPacks } from '../infrastructure/search-packs.js';
-import { indexStoragePath, openFileIndex } from '../infrastructure/file-index.js';
+import { indexStoragePath } from '../infrastructure/file-index.js';
+import { openKnowledgeStore } from '../infrastructure/knowledge-store/index.js';
 import { createEmbeddingProvider } from '../domain/embeddings.js';
 import { loadBrainlinkConfig, sanitizeSearchMode } from '../infrastructure/config.js';
+const embedQuerySafely = async (provider, query) => {
+    try {
+        return (await provider.embed([query]))[0] ?? [];
+    }
+    catch (error) {
+        const reason = error instanceof Error ? error.message : String(error);
+        process.stderr.write(`brainlink: query embedding via ${provider.name} failed, using lexical search (${reason})\n`);
+        return [];
+    }
+};
 const hybridCacheTtlMs = 30_000;
 const hybridCacheMaxEntries = 200;
 const hybridSearchCache = new Map();
@@ -53,13 +64,19 @@ export const searchKnowledge = async (vaultPath, query, limit, agentId, mode) =>
     if (cached) {
         return cached;
     }
-    const provider = createEmbeddingProvider(config.embeddingProvider);
+    const provider = createEmbeddingProvider(config.embeddingProvider, config.embedding);
     const shouldEmbedQuery = searchMode !== 'fts' && provider.name !== 'none';
-    const queryEmbedding = shouldEmbedQuery ? (await provider.embed([query]))[0] ?? [] : [];
+    // An external provider can be unreachable at query time. Degrade to
+    // lexical-only retrieval rather than substituting a local-space vector that
+    // would not match the stored embeddings, keeping search available.
+    const queryEmbedding = shouldEmbedQuery ? await embedQuerySafely(provider, query) : [];
+    // Pure semantic mode with no usable embedding would return nothing; fall back
+    // to lexical (fts) so a transient provider outage still yields results.
+    const effectiveMode = searchMode === 'semantic' && shouldEmbedQuery && queryEmbedding.length === 0 ? 'fts' : searchMode;
     try {
-        const index = openFileIndex(absoluteVaultPath);
+        const index = openKnowledgeStore(absoluteVaultPath);
         try {
-            const results = await index.search(query, limit, agentId, searchMode, queryEmbedding);
+            const results = await index.search(query, limit, agentId, effectiveMode, queryEmbedding);
             if (cacheKey) {
                 cacheSet({
                     key: cacheKey,