@andespindola/brainlink 1.0.6 → 1.0.7

package/CHANGELOG.md

@@ -1,5 +1,22 @@
 # Changelog
 
+## Unreleased
+
+- Added `brainlink_similar_notes` MCP tool: find the notes most similar to a given note (selected by title or path), using the note's own content as the retrieval query and excluding the note itself.
+- Added batch write tools `brainlink_add_notes` and `brainlink_delete_notes` that write or delete up to 100 notes in a single call and reindex once for the whole batch; batch deletion reports per-note failures without aborting the batch.
+
+- Made chunking heading-aware: a Markdown heading starts a new chunk once the current chunk already holds substantial content, so large notes split on topical (heading) boundaries instead of mid-section, while small notes and trailing metadata sections (e.g. `## Context Links`) stay in a single chunk. The new boundaries take effect on the next full reindex (`index --full`); existing chunks are reused until then.
+- Added maximal-marginal-relevance ordering to context assembly so the token budget is not spent on several near-duplicate notes: each selected document balances its own relevance against how similar it is to documents already chosen. The strongest document still leads.
+
+- Hardened index reliability: a corrupt `index.json` is now detected, logged and quarantined to `index.json.corrupt` (so the next `index --full` rebuilds it) instead of being silently read as an empty index. Search treats corruption as a signal to fall back to the independent compressed search packs rather than returning no results.
+
+- Added pluggable external embedding providers (`ollama`, `openai`) alongside the default deterministic `local` provider, configured through a new optional `embedding` block (`url`, `model`, `apiKeyEnv`, `timeoutMs`, `batchSize`). External vectors are L2-normalized to match the local cosine/bucket contract, and the API key is referenced by environment variable name, never stored in config.
+- Switching the embedding provider or external model now rebuilds vectors automatically (the embedding model is folded into the index reindex signature). If an external provider is unreachable, query-time search degrades to lexical-only while index-time surfaces a clear error instead of mixing embedding spaces.
+
+- Replaced the naive additive lexical score with field-weighted BM25 ranking (IDF, term-frequency saturation and chunk-length normalization) so rare query terms outweigh common ones and long chunks no longer dominate on raw hit counts.
+- Reworked hybrid search to combine lexical and semantic components on a shared min-max-normalized [0,1] scale instead of the fixed `text + semantic * 8` blend, keeping neither the unbounded lexical magnitude nor the bounded cosine value able to swamp the other. Relevance filtering now runs on raw component scores before normalization so a legitimate weakest-in-mode match is never scaled to zero and dropped.
+- Context token budgeting now reuses the per-chunk token count cached at index time and shares a single, more accurate token estimator (`max(words * 1.3, chars / 4)`), removing the duplicated character-ratio heuristic while staying conservative so rendered packages remain within budget.
+
 ## 1.0.0
 
 - Promoted Brainlink to the first stable functional release.

package/README.md

@@ -540,14 +540,17 @@ Available tools:
 - `brainlink_context_packs`: list or clear persisted CAG context packs.
 - `brainlink_search`: search indexed notes.
 - `brainlink_explain`: explain why indexed notes matched a query.
+- `brainlink_similar_notes`: find the notes most similar to a given note (by title or path), excluding the note itself.
 - `brainlink_dedupe`: detect duplicate candidates using exact hash + semantic similarity scores.
 - `brainlink_resolve_duplicate`: resolve duplicate pairs (`merge`, `link`, `ignore`) with connectivity-safe fallback edges.
 - `brainlink_add_note`: write durable Markdown memory and reindex.
+- `brainlink_add_notes`: write several notes in one batch, reindexing once.
 - `brainlink_remember`: capture durable memory with inferred title, tags and Context Links; supports dry-run.
 - `brainlink_inbox_add`: capture a quick untriaged memory item.
 - `brainlink_inbox_list`: list untriaged inbox memory items.
 - `brainlink_inbox_process`: suggest titles, tags and links for inbox items.
 - `brainlink_delete_note`: delete a durable Markdown note by title or path after explicit confirmation and reindex.
+- `brainlink_delete_notes`: delete several notes in one confirmed batch, reindexing once; per-note failures are reported.
 - `brainlink_add_file`: ingest a local file as a note and reindex.
 - `brainlink_canonicalize_context_links`: ensure existing notes link to inferred context hubs.
 - `brainlink_volatile_add`: write temporary agent-decided memory with TTL; volatile sections are included in context and never create durable graph edges.
@@ -1214,6 +1217,41 @@ spec:
 
 Use `"embeddingProvider": "none"` when you want FTS-only indexing.
 
+### Embedding providers
+
+`embeddingProvider` selects how chunk and query embeddings are produced:
+
+- `local` (default): deterministic, dependency-free local vectors. No network.
+- `none`: FTS-only; no semantic vectors.
+- `ollama`: embeddings from a local [Ollama](https://ollama.com) server.
+- `openai`: embeddings from the OpenAI embeddings API.
+
+External providers are configured through the optional `embedding` block. The API key is referenced by environment variable name and never stored in the config file:
+
+```json
+{
+  "embeddingProvider": "ollama",
+  "embedding": {
+    "url": "http://127.0.0.1:11434",
+    "model": "nomic-embed-text",
+    "timeoutMs": 10000,
+    "batchSize": 64
+  }
+}
+```
+
+```json
+{
+  "embeddingProvider": "openai",
+  "embedding": {
+    "model": "text-embedding-3-small",
+    "apiKeyEnv": "OPENAI_API_KEY"
+  }
+}
+```
+
+Switching the provider — or the external `model` — changes the embedding space, so the next index run rebuilds vectors automatically. If an external provider is unreachable at query time, search degrades to lexical-only instead of failing; at index time it surfaces a clear error rather than mixing embedding spaces.
+
 For local security checks, set your Snyk token in the environment:
 
 ```bash

package/dist/application/find-similar-notes.js

@@ -0,0 +1,75 @@
+import { resolve } from 'node:path';
+import { parseMarkdownDocument } from '../domain/markdown.js';
+import { ensureVault, readMarkdownFiles } from '../infrastructure/file-system-vault.js';
+import { searchKnowledge } from './search-knowledge.js';
+const defaultLimit = 5;
+// Pull extra candidates before deduping by document and dropping the note
+// itself, so a note with many chunks does not crowd out distinct results.
+const candidateOverscan = 8;
+const maxQueryCharacters = 2000;
+const normalizeTitleKey = (title) => title.trim().replace(/\.md$/i, '').toLowerCase();
+const resolveTarget = async (vaultPath, input) => {
+    const absoluteVaultPath = await ensureVault(vaultPath);
+    const files = await readMarkdownFiles(vaultPath);
+    const documents = files.map((file) => ({
+        file,
+        document: parseMarkdownDocument({
+            absolutePath: file.absolutePath,
+            vaultPath: absoluteVaultPath,
+            content: file.content,
+            createdAt: file.createdAt,
+            updatedAt: file.updatedAt
+        })
+    }));
+    if (input.path != null && input.path.trim().length > 0) {
+        const expected = resolve(absoluteVaultPath, input.path);
+        const match = documents.find((entry) => resolve(entry.file.absolutePath) === expected);
+        if (!match) {
+            throw new Error(`No note found at path: ${input.path}`);
+        }
+        return { title: match.document.title, path: match.document.path, content: match.file.content };
+    }
+    const expectedTitle = normalizeTitleKey(input.title ?? '');
+    const matches = documents.filter((entry) => normalizeTitleKey(entry.document.title) === expectedTitle &&
+        (input.agentId == null || entry.document.agentId === input.agentId));
+    if (matches.length === 0) {
+        throw new Error(`No note found with title: ${input.title}`);
+    }
+    if (matches.length > 1) {
+        throw new Error(`Multiple notes match title "${input.title}". Use path instead: ${matches.map((entry) => entry.document.path).join(', ')}`);
+    }
+    return { title: matches[0].document.title, path: matches[0].document.path, content: matches[0].file.content };
+};
+const buildQuery = (target) => `${target.title}\n${target.content}`.slice(0, maxQueryCharacters);
+// Find notes most similar to a given note by using the note's own title and
+// content as the retrieval query, then excluding the note itself.
+export const findSimilarNotes = async (vaultPath, input) => {
+    const hasTitle = input.title != null && input.title.trim().length > 0;
+    const hasPath = input.path != null && input.path.trim().length > 0;
+    if (hasTitle === hasPath) {
+        throw new Error('Use exactly one selector: title or path.');
+    }
+    const target = await resolveTarget(vaultPath, input);
+    const limit = Math.max(1, input.limit ?? defaultLimit);
+    const results = await searchKnowledge(vaultPath, buildQuery(target), limit + candidateOverscan, input.agentId, input.mode ?? 'hybrid');
+    const bestByPath = new Map();
+    for (const result of results) {
+        if (result.path === target.path) {
+            continue;
+        }
+        const existing = bestByPath.get(result.path);
+        if (!existing || result.score > existing.score) {
+            bestByPath.set(result.path, {
+                title: result.title,
+                path: result.path,
+                score: result.score,
+                searchMode: result.searchMode,
+                tags: result.tags
+            });
+        }
+    }
+    const similar = Array.from(bestByPath.values())
+        .sort((left, right) => right.score - left.score || left.title.localeCompare(right.title))
+        .slice(0, limit);
+    return { target: { title: target.title, path: target.path }, similar };
+};

package/dist/application/index-vault-phases.js

@@ -1,6 +1,7 @@
 import { createIndexedDocument, graphLinkModelVersion } from '../domain/markdown.js';
 import { sharedAgentId } from '../domain/agents.js';
 import { createEmbeddingBuckets, createEmbeddingProvider } from '../domain/embeddings.js';
+import { embeddingSignature } from '../infrastructure/config.js';
 import { buildSearchPacks, toSearchPackBuildOptions } from '../infrastructure/search-packs.js';
 const toTitleKey = (title) => title.toLowerCase();
 const appendTitleEntry = (map, document) => {
@@ -30,11 +31,11 @@ export const createTitleMaps = (documents) => [...documents]
 const createScopedTitleResolver = (document, titleMaps) => ({
     get: (title) => titleMaps.byAgent.get(document.agentId)?.get(title)?.id ?? titleMaps.shared.get(title)?.id
 });
-const embedIndexedDocuments = async (documents, providerName) => {
+const embedIndexedDocuments = async (documents, providerName, embeddingConfig) => {
     if (documents.length === 0) {
         return documents;
     }
-    const provider = createEmbeddingProvider(providerName);
+    const provider = createEmbeddingProvider(providerName, embeddingConfig);
     const chunks = documents.flatMap((document) => document.chunks);
     const embeddings = await provider.embed(chunks.map((chunk) => chunk.content));
     const embeddingByChunkId = new Map(chunks.map((chunk, index) => [chunk.id, embeddings[index] ?? []]));
@@ -70,7 +71,7 @@ export const detectChanges = (params) => {
     const fullSourceReindex = fullReindex || graphLinkModelChanged;
     const settingsChanged = previousState == null ||
         previousState.chunkSize !== config.chunkSize ||
-        previousState.embeddingProvider !== config.embeddingProvider ||
+        previousState.embeddingProvider !== embeddingSignature(config) ||
         graphLinkModelChanged;
     const packSettingsChanged = previousState == null ||
         previousState.searchPackRowChunkSize !== config.searchPack.rowChunkSize ||
@@ -135,7 +136,7 @@ export const embedChangedDocuments = async (params) => {
         changedDocuments: changedDocumentsByPath.size
     });
     const changedIndexedDocuments = changedDocumentsByPath.size > 0
-        ? await embedIndexedDocuments(Array.from(changedDocumentsByPath.values()).map((document) => createIndexedDocument(document, createScopedTitleResolver(document, titleMaps), config.chunkSize)), config.embeddingProvider)
+        ? await embedIndexedDocuments(Array.from(changedDocumentsByPath.values()).map((document) => createIndexedDocument(document, createScopedTitleResolver(document, titleMaps), config.chunkSize)), config.embeddingProvider, config.embedding)
         : [];
     emit('embed', changedDocumentsByPath.size > 0 ? 'finish' : 'skip', changedDocumentsByPath.size > 0 ? 'Embedding complete' : 'Embedding skipped', {
         changedIndexedDocuments: changedIndexedDocuments.length

package/dist/application/index-vault.js

@@ -1,6 +1,6 @@
 import { readFile } from 'node:fs/promises';
 import { graphLinkModelVersion, parseMarkdownDocument } from '../domain/markdown.js';
-import { loadBrainlinkConfig } from '../infrastructure/config.js';
+import { embeddingSignature, loadBrainlinkConfig } from '../infrastructure/config.js';
 import { ensureVault, readMarkdownFileSummaries } from '../infrastructure/file-system-vault.js';
 import { readIndexState, writeIndexState } from '../infrastructure/index-state.js';
 import { ensureSearchPackManifest } from '../infrastructure/search-packs.js';
@@ -161,7 +161,7 @@ export const indexVaultWithOptions = async (vaultPath, options) => {
         const packResultReason = shouldRebuildPacks && !packsRebuilt ? `${packReason} (failed)` : packReason;
         await writeIndexState(absoluteVaultPath, {
             chunkSize: config.chunkSize,
-            embeddingProvider: config.embeddingProvider,
+            embeddingProvider: embeddingSignature(config),
             graphLinkModelVersion,
             searchPackRowChunkSize: config.searchPack.rowChunkSize,
             searchPackCompressionLevel: config.searchPack.compressionLevel,

package/dist/application/memory-suggestions.js

@@ -127,7 +127,10 @@ export const suggestContextLinks = async (vaultPath, content, agentId, limit) =>
     for (const result of searchResults) {
         const key = result.title.toLowerCase();
         const current = byTitle.get(key);
-        const score = Number(Math.min(1, Math.max(result.score / 20, result.semanticScore)).toFixed(4));
+        // Hybrid search already returns a min-max-normalized [0,1] score, so it is
+        // compared directly against the cosine semanticScore (also [0,1]) instead of
+        // being rescaled from the previous unbounded blend.
+        const score = Number(Math.min(1, Math.max(result.score, result.semanticScore)).toFixed(4));
         if (!current || score > current.score) {
             byTitle.set(key, {
                 title: result.title,

package/dist/application/search-knowledge.js

@@ -4,6 +4,16 @@ import { ensurePrivatePacksFromLegacyIndex, searchInPacks } from '../infrastruct
 import { indexStoragePath, openFileIndex } from '../infrastructure/file-index.js';
 import { createEmbeddingProvider } from '../domain/embeddings.js';
 import { loadBrainlinkConfig, sanitizeSearchMode } from '../infrastructure/config.js';
+const embedQuerySafely = async (provider, query) => {
+    try {
+        return (await provider.embed([query]))[0] ?? [];
+    }
+    catch (error) {
+        const reason = error instanceof Error ? error.message : String(error);
+        process.stderr.write(`brainlink: query embedding via ${provider.name} failed, using lexical search (${reason})\n`);
+        return [];
+    }
+};
 const hybridCacheTtlMs = 30_000;
 const hybridCacheMaxEntries = 200;
 const hybridSearchCache = new Map();
@@ -53,13 +63,19 @@ export const searchKnowledge = async (vaultPath, query, limit, agentId, mode) =>
     if (cached) {
         return cached;
     }
-    const provider = createEmbeddingProvider(config.embeddingProvider);
+    const provider = createEmbeddingProvider(config.embeddingProvider, config.embedding);
     const shouldEmbedQuery = searchMode !== 'fts' && provider.name !== 'none';
-    const queryEmbedding = shouldEmbedQuery ? (await provider.embed([query]))[0] ?? [] : [];
+    // An external provider can be unreachable at query time. Degrade to
+    // lexical-only retrieval rather than substituting a local-space vector that
+    // would not match the stored embeddings, keeping search available.
+    const queryEmbedding = shouldEmbedQuery ? await embedQuerySafely(provider, query) : [];
+    // Pure semantic mode with no usable embedding would return nothing; fall back
+    // to lexical (fts) so a transient provider outage still yields results.
+    const effectiveMode = searchMode === 'semantic' && shouldEmbedQuery && queryEmbedding.length === 0 ? 'fts' : searchMode;
     try {
         const index = openFileIndex(absoluteVaultPath);
         try {
-            const results = await index.search(query, limit, agentId, searchMode, queryEmbedding);
+            const results = await index.search(query, limit, agentId, effectiveMode, queryEmbedding);
             if (cacheKey) {
                 cacheSet({
                     key: cacheKey,

package/dist/domain/context.js

@@ -1,21 +1,32 @@
 import { middleOutIndices } from './middle-out.js';
+import { estimateTokenCount } from './tokens.js';
+import { orderByMaximalMarginalRelevance } from './diversity.js';
+const contentTokenPattern = /[\p{L}\p{N}_-]+/gu;
+const tokenizeForDiversity = (text) => new Set((text.toLowerCase().match(contentTokenPattern) ?? []).filter((token) => token.length > 1));
 const maxSectionsPerDocument = 3;
-const charsPerToken = 4;
+// Character budget used only for truncating an overflowing chunk back down to a
+// token target; the conservative per-token character ratio keeps the rendered
+// slice within budget even though estimateTokenCount is the source of truth for
+// counting.
+const truncationCharsPerToken = 4;
 const sectionSeparatorTokens = 1;
 const headerReserveTokens = 8;
-const estimateTokens = (text) => Math.ceil(text.length / charsPerToken);
 const truncateToTokens = (text, maxTokens) => {
-    const maxChars = Math.max(0, maxTokens) * charsPerToken;
+    const maxChars = Math.max(0, maxTokens) * truncationCharsPerToken;
     return text.length <= maxChars ? text : text.slice(0, maxChars);
 };
+// Prefer the token count cached on the chunk at index time; fall back to a live
+// estimate when the result did not carry one (legacy indexes, pack fallback).
+const contentTokens = (result) => result.tokenCount ?? estimateTokenCount(result.content);
 // Mirror the per-section framing that formatContextPackage renders (heading,
 // Source/Tags/Score/Mode lines and the trailing separator) so the budget
 // reflects the package the agent actually receives, not just chunk content.
-const estimateSectionTokens = (result) => {
+const estimateFramingTokens = (result) => {
     const tagsLine = result.tags.length > 0 ? `Tags: ${result.tags.map((tag) => `#${tag}`).join(' ')}\n` : '';
     const framing = `## . ${result.title}\nSource: ${result.path}\n${tagsLine}Score: 0.000\nMode: ${result.searchMode}\n\n`;
-    return estimateTokens(result.content) + estimateTokens(framing) + sectionSeparatorTokens;
+    return estimateTokenCount(framing) + sectionSeparatorTokens;
 };
+const estimateSectionTokens = (result) => contentTokens(result) + estimateFramingTokens(result);
 const byScore = (left, right) => right.score - left.score || left.title.localeCompare(right.title);
 const byOrdinal = (left, right) => (left.chunkOrdinal ?? Number.MAX_SAFE_INTEGER) - (right.chunkOrdinal ?? Number.MAX_SAFE_INTEGER);
 const middleOutDocumentResults = (results) => {
@@ -36,14 +47,16 @@ export const selectContextSections = (results, maxTokens) => {
         state.set(result.documentId, [...current, result]);
         return state;
     }, new Map());
-    const documentOrder = Array.from(results.reduce((state, result) => {
-        if (!state.has(result.documentId)) {
-            state.set(result.documentId, result.score);
-        }
-        return state;
-    }, new Map()).entries())
-        .sort((left, right) => right[1] - left[1] || left[0].localeCompare(right[0]))
-        .map(([documentId]) => documentId);
+    // Order documents by maximal marginal relevance so the budget is not spent on
+    // several near-duplicate notes: each pick trades its own relevance against how
+    // similar it is to documents already chosen. The strongest document still
+    // leads (no redundancy penalty on an empty selection).
+    const documentCandidates = Array.from(grouped.entries()).map(([documentId, group]) => ({
+        id: documentId,
+        relevance: group.reduce((max, result) => Math.max(max, result.score), Number.NEGATIVE_INFINITY),
+        tokens: tokenizeForDiversity(group.map((result) => result.content).join(' '))
+    }));
+    const documentOrder = orderByMaximalMarginalRelevance(documentCandidates);
     const selected = documentOrder.reduce((state, documentId) => {
         const ordered = middleOutDocumentResults(grouped.get(documentId) ?? []);
         let usedTokens = state.usedTokens;
@@ -91,7 +104,7 @@ export const selectContextSections = (results, maxTokens) => {
     // brainlink_context consistent with brainlink_search instead of reporting
     // "No relevant context found." whenever the strongest chunk is large.
     const topResult = [...results].sort(byScore)[0];
-    const framingTokens = estimateSectionTokens({ ...topResult, content: '' });
+    const framingTokens = estimateFramingTokens(topResult);
     const contentTokenBudget = maxTokens - headerReserveTokens - framingTokens;
     return [
         {

package/dist/domain/diversity.js

@@ -0,0 +1,64 @@
+// Maximal Marginal Relevance ordering. Greedily orders candidates so each pick
+// balances its own relevance against how redundant it is versus everything
+// already chosen, preventing a context window from filling with near-duplicate
+// material. Pure and deterministic: ties resolve by relevance then id.
+const jaccardSimilarity = (left, right) => {
+    if (left.size === 0 || right.size === 0) {
+        return 0;
+    }
+    let intersection = 0;
+    const [small, large] = left.size <= right.size ? [left, right] : [right, left];
+    for (const token of small) {
+        if (large.has(token)) {
+            intersection += 1;
+        }
+    }
+    const union = left.size + right.size - intersection;
+    return union === 0 ? 0 : intersection / union;
+};
+const normalizeRelevance = (candidates) => {
+    // Reduce instead of Math.min(...values): the candidate list can be as large as
+    // the caller's result limit, and spreading a very large array overflows the
+    // call stack.
+    const bounds = candidates.reduce((range, candidate) => ({
+        min: Math.min(range.min, candidate.relevance),
+        max: Math.max(range.max, candidate.relevance)
+    }), { min: Infinity, max: -Infinity });
+    const span = bounds.max - bounds.min;
+    return new Map(candidates.map((candidate) => [candidate.id, span === 0 ? 1 : (candidate.relevance - bounds.min) / span]));
+};
+// lambda in [0,1]: 1 ignores redundancy (pure relevance order), lower values
+// push diverse picks higher. The default balances relevance and diversity; the
+// strongest candidate still always leads (no redundancy penalty on an empty
+// selection).
+export const orderByMaximalMarginalRelevance = (candidates, lambda = 0.5) => {
+    if (candidates.length <= 1) {
+        return candidates.map((candidate) => candidate.id);
+    }
+    const relevanceById = normalizeRelevance(candidates);
+    const remaining = new Map(candidates.map((candidate) => [candidate.id, candidate]));
+    const selected = [];
+    const order = [];
+    while (remaining.size > 0) {
+        let best = null;
+        for (const candidate of remaining.values()) {
+            const relevance = relevanceById.get(candidate.id) ?? 0;
+            const redundancy = selected.reduce((max, picked) => Math.max(max, jaccardSimilarity(candidate.tokens, picked.tokens)), 0);
+            const score = lambda * relevance - (1 - lambda) * redundancy;
+            if (best === null ||
+                score > best.score ||
+                (score === best.score &&
+                    (relevance > best.relevance ||
+                        (relevance === best.relevance && candidate.id.localeCompare(best.candidate.id) < 0)))) {
+                best = { candidate, score, relevance };
+            }
+        }
+        if (best === null) {
+            break;
+        }
+        selected.push(best.candidate);
+        order.push(best.candidate.id);
+        remaining.delete(best.candidate.id);
+    }
+    return order;
+};

package/dist/domain/embeddings.js

@@ -141,4 +141,120 @@ export const createLocalEmbeddingProvider = () => ({
     name: 'local',
     embed: async (input) => input.map(createLocalEmbedding)
 });
-export const createEmbeddingProvider = (name) => name === 'local' ? createLocalEmbeddingProvider() : createDisabledEmbeddingProvider();
+const defaultExternalTimeoutMs = 10_000;
+const defaultExternalBatchSize = 64;
+const ollamaDefaultUrl = 'http://127.0.0.1:11434';
+const ollamaDefaultModel = 'nomic-embed-text';
+const openAiDefaultUrl = 'https://api.openai.com/v1';
+const openAiDefaultModel = 'text-embedding-3-small';
+const trimTrailingSlash = (value) => value.replace(/\/+$/, '');
+const chunkBatches = (items, size) => {
+    const batchSize = Number.isFinite(size) && size > 0 ? Math.floor(size) : defaultExternalBatchSize;
+    const batches = [];
+    for (let index = 0; index < items.length; index += batchSize) {
+        batches.push(items.slice(index, index + batchSize));
+    }
+    return batches;
+};
+const isNumberArray = (value) => Array.isArray(value) && value.every((entry) => typeof entry === 'number' && Number.isFinite(entry));
+const fetchJson = async (url, init, timeoutMs, providerName) => {
+    const controller = new AbortController();
+    const timeout = setTimeout(() => controller.abort(), Math.max(1, timeoutMs));
+    try {
+        const response = await fetch(url, { ...init, signal: controller.signal });
+        if (!response.ok) {
+            throw new Error(`${providerName} embedding request failed with HTTP ${response.status}`);
+        }
+        return (await response.json());
+    }
+    catch (error) {
+        const reason = error instanceof Error ? error.message : String(error);
+        throw new Error(`${providerName} embedding request to ${url} failed: ${reason}`);
+    }
+    finally {
+        clearTimeout(timeout);
+    }
+};
+// External providers return vectors in their own space; L2-normalize so dot
+// product equals cosine similarity, matching the contract local embeddings rely
+// on across the retrieval and bucket-prefilter paths.
+const createOllamaEmbeddingProvider = (config) => {
+    const baseUrl = trimTrailingSlash(config.url?.trim() || ollamaDefaultUrl);
+    const model = config.model?.trim() || ollamaDefaultModel;
+    const timeoutMs = config.timeoutMs ?? defaultExternalTimeoutMs;
+    return {
+        name: 'ollama',
+        embed: async (input) => {
+            const vectors = [];
+            for (const batch of chunkBatches(input, config.batchSize ?? defaultExternalBatchSize)) {
+                const payload = await fetchJson(`${baseUrl}/api/embed`, {
+                    method: 'POST',
+                    headers: { 'content-type': 'application/json' },
+                    body: JSON.stringify({ model, input: batch })
+                }, timeoutMs, 'ollama');
+                const embeddings = payload.embeddings;
+                if (!Array.isArray(embeddings) || embeddings.length !== batch.length) {
+                    throw new Error('ollama embedding response shape was invalid');
+                }
+                for (const embedding of embeddings) {
+                    if (!isNumberArray(embedding) || embedding.length === 0) {
+                        throw new Error('ollama embedding response contained an invalid vector');
+                    }
+                    vectors.push(normalizeVector(embedding));
+                }
+            }
+            return vectors;
+        }
+    };
+};
+const createOpenAiEmbeddingProvider = (config) => {
+    const baseUrl = trimTrailingSlash(config.url?.trim() || openAiDefaultUrl);
+    const model = config.model?.trim() || openAiDefaultModel;
+    const timeoutMs = config.timeoutMs ?? defaultExternalTimeoutMs;
+    const apiKeyEnv = config.apiKeyEnv?.trim() || 'OPENAI_API_KEY';
+    return {
+        name: 'openai',
+        embed: async (input) => {
+            const apiKey = process.env[apiKeyEnv]?.trim();
+            if (!apiKey) {
+                throw new Error(`openai embedding requires the ${apiKeyEnv} environment variable to be set`);
+            }
+            const vectors = [];
+            for (const batch of chunkBatches(input, config.batchSize ?? defaultExternalBatchSize)) {
+                const payload = await fetchJson(`${baseUrl}/embeddings`, {
+                    method: 'POST',
+                    headers: {
+                        'content-type': 'application/json',
+                        authorization: `Bearer ${apiKey}`
+                    },
+                    body: JSON.stringify({ model, input: batch })
+                }, timeoutMs, 'openai');
+                const data = payload.data;
+                if (!Array.isArray(data) || data.length !== batch.length) {
+                    throw new Error('openai embedding response shape was invalid');
+                }
+                const ordered = [...data].sort((left, right) => (left.index ?? 0) - (right.index ?? 0));
+                for (const entry of ordered) {
+                    const embedding = entry.embedding;
+                    if (!isNumberArray(embedding) || embedding.length === 0) {
+                        throw new Error('openai embedding response contained an invalid vector');
+                    }
+                    vectors.push(normalizeVector(embedding));
+                }
+            }
+            return vectors;
+        }
+    };
+};
+export const createEmbeddingProvider = (name, config = {}) => {
+    switch (name) {
+        case 'local':
+            return createLocalEmbeddingProvider();
+        case 'ollama':
+            return createOllamaEmbeddingProvider(config);
+        case 'openai':
+            return createOpenAiEmbeddingProvider(config);
+        default:
+            return createDisabledEmbeddingProvider();
+    }
+};

package/dist/domain/markdown.js

@@ -188,7 +188,15 @@ const splitLongParagraph = (paragraph, maxCharacters) => {
         return [...state, sentence];
     }, []);
 };
+const headingParagraphPattern = /^#{1,6}\s/;
+const startsWithHeading = (paragraph) => headingParagraphPattern.test(paragraph);
 export const splitIntoChunks = (documentId, content, maxCharacters = 1200) => {
+    // A heading should begin a new chunk so a section's body is not glued onto the
+    // tail of the previous section — but only once the current chunk already holds
+    // substantial content. This keeps large notes split on topical (heading)
+    // boundaries while small notes (and trailing metadata sections like
+    // "## Context Links") stay in a single chunk instead of fragmenting.
+    const minChunkCharacters = Math.floor(maxCharacters / 2);
     const paragraphs = normalizeChunkContent(stripFrontmatter(content))
         .split(/\n{2,}/)
         .filter(Boolean)
@@ -198,8 +206,9 @@ export const splitIntoChunks = (documentId, content, maxCharacters = 1200) => {
         if (!lastChunk) {
             return [paragraph];
         }
+        const startsNewSection = startsWithHeading(paragraph) && lastChunk.length >= minChunkCharacters;
         const merged = `${lastChunk}\n\n${paragraph}`;
-        if (merged.length <= maxCharacters) {
+        if (!startsNewSection && merged.length <= maxCharacters) {
             return [...state.slice(0, -1), merged];
         }
         return [...state, paragraph];

package/dist/domain/scoring.js

@@ -0,0 +1,42 @@
+// Pure retrieval scoring primitives: BM25 lexical relevance and min-max
+// normalization. Kept dependency-free and deterministic so the hot retrieval
+// loop in the infrastructure layer stays testable and the domain owns the
+// ranking math.
+export const defaultBm25Parameters = {
+    k1: 1.2,
+    b: 0.75
+};
+// Robertson/Sparck-Jones probabilistic IDF with the +0.5 smoothing that keeps
+// the value finite for terms present in every chunk. Floored at 0 so a term
+// shared by the whole candidate set can never push a score negative.
+export const inverseDocumentFrequency = (documentCount, documentFrequency) => {
+    if (documentCount <= 0) {
+        return 0;
+    }
+    const raw = Math.log(1 + (documentCount - documentFrequency + 0.5) / (documentFrequency + 0.5));
+    return raw > 0 ? raw : 0;
+};
+// BM25 contribution for a single query term given its (field-weighted)
+// frequency in a chunk. `documentLength` and `averageLength` are in the same
+// weighted unit so length normalization stays consistent with how the caller
+// counts terms.
+export const bm25TermScore = (params) => {
+    if (params.termFrequency <= 0) {
+        return 0;
+    }
+    const { k1, b } = params.parameters ?? defaultBm25Parameters;
+    const idf = inverseDocumentFrequency(params.documentCount, params.documentFrequency);
+    const normalizedLength = params.averageLength > 0 ? params.documentLength / params.averageLength : 1;
+    const denominator = params.termFrequency + k1 * (1 - b + b * normalizedLength);
+    return denominator > 0 ? (idf * (params.termFrequency * (k1 + 1))) / denominator : 0;
+};
+// Scale a value into [0, 1] given the observed range. When every candidate
+// shares the same score (range collapses) a positive score maps to 1 and a
+// zero score to 0, so a single result is never silently nulled out.
+export const minMaxNormalize = (value, minimum, maximum) => {
+    if (maximum <= minimum) {
+        return value > 0 ? 1 : 0;
+    }
+    const scaled = (value - minimum) / (maximum - minimum);
+    return scaled < 0 ? 0 : scaled > 1 ? 1 : scaled;
+};

package/dist/domain/tokens.js

@@ -1 +1,17 @@
-export const estimateTokenCount = (content) => Math.ceil(content.trim().split(/\s+/).filter(Boolean).join(' ').length / 4);
+// Subword tokenizers emit roughly 1.3 tokens per whitespace word for natural
+// language. Pure prose tracks that ratio; punctuation- or symbol-dense content
+// (code, URLs, long unbroken strings) tokenizes far denser, which the word
+// count alone underestimates. Taking the max with the character heuristic keeps
+// the estimate conservative — never below chars/4 — so callers that budget
+// against it can rely on the rendered output staying within the token limit.
+const tokensPerWord = 1.3;
+const charactersPerToken = 4;
+export const estimateTokenCount = (content) => {
+    const words = content.trim().split(/\s+/).filter(Boolean);
+    if (words.length === 0) {
+        return 0;
+    }
+    const characterEstimate = Math.ceil(words.join(' ').length / charactersPerToken);
+    const wordEstimate = Math.ceil(words.length * tokensPerWord);
+    return Math.max(characterEstimate, wordEstimate);
+};