@andespindola/brainlink 1.0.5 → 1.0.7

package/CHANGELOG.md

@@ -1,5 +1,22 @@
 # Changelog
 
+## Unreleased
+
+- Added `brainlink_similar_notes` MCP tool: find the notes most similar to a given note (selected by title or path), using the note's own content as the retrieval query and excluding the note itself.
+- Added batch write tools `brainlink_add_notes` and `brainlink_delete_notes` that write or delete up to 100 notes in a single call and reindex once for the whole batch; batch deletion reports per-note failures without aborting the batch.
+
+- Made chunking heading-aware: a Markdown heading starts a new chunk once the current chunk already holds substantial content, so large notes split on topical (heading) boundaries instead of mid-section, while small notes and trailing metadata sections (e.g. `## Context Links`) stay in a single chunk. The new boundaries take effect on the next full reindex (`index --full`); existing chunks are reused until then.
+- Added maximal-marginal-relevance ordering to context assembly so the token budget is not spent on several near-duplicate notes: each selected document balances its own relevance against how similar it is to documents already chosen. The strongest document still leads.
+
+- Hardened index reliability: a corrupt `index.json` is now detected, logged and quarantined to `index.json.corrupt` (so the next `index --full` rebuilds it) instead of being silently read as an empty index. Search treats corruption as a signal to fall back to the independent compressed search packs rather than returning no results.
+
+- Added pluggable external embedding providers (`ollama`, `openai`) alongside the default deterministic `local` provider, configured through a new optional `embedding` block (`url`, `model`, `apiKeyEnv`, `timeoutMs`, `batchSize`). External vectors are L2-normalized to match the local cosine/bucket contract, and the API key is referenced by environment variable name, never stored in config.
+- Switching the embedding provider or external model now rebuilds vectors automatically (the embedding model is folded into the index reindex signature). If an external provider is unreachable, query-time search degrades to lexical-only while index-time surfaces a clear error instead of mixing embedding spaces.
+
+- Replaced the naive additive lexical score with field-weighted BM25 ranking (IDF, term-frequency saturation and chunk-length normalization) so rare query terms outweigh common ones and long chunks no longer dominate on raw hit counts.
+- Reworked hybrid search to combine lexical and semantic components on a shared min-max-normalized [0,1] scale instead of the fixed `text + semantic * 8` blend, keeping neither the unbounded lexical magnitude nor the bounded cosine value able to swamp the other. Relevance filtering now runs on raw component scores before normalization so a legitimate weakest-in-mode match is never scaled to zero and dropped.
+- Context token budgeting now reuses the per-chunk token count cached at index time and shares a single, more accurate token estimator (`max(words * 1.3, chars / 4)`), removing the duplicated character-ratio heuristic while staying conservative so rendered packages remain within budget.
+
 ## 1.0.0
 
 - Promoted Brainlink to the first stable functional release.

package/README.md

@@ -115,6 +115,11 @@ blink --help
 
 Use `brainlink` when clarity matters. Use `blink` for faster daily terminal usage.
 
+Brainlink checks npm for a newer package version at most once per day and prints
+an update notice to `stderr` when one is available. It does not update itself
+silently. Disable this with `BRAINLINK_NO_UPDATE_CHECK=1` or with
+`"autoUpdateCheck": false` in config.
+
 The npm package page may show `npm i @andespindola/brainlink`. That installs
 Brainlink as a project dependency. Use `-g` when you want the terminal commands
 available globally, or run it without a global install:
@@ -535,14 +540,17 @@ Available tools:
 - `brainlink_context_packs`: list or clear persisted CAG context packs.
 - `brainlink_search`: search indexed notes.
 - `brainlink_explain`: explain why indexed notes matched a query.
+- `brainlink_similar_notes`: find the notes most similar to a given note (by title or path), excluding the note itself.
 - `brainlink_dedupe`: detect duplicate candidates using exact hash + semantic similarity scores.
 - `brainlink_resolve_duplicate`: resolve duplicate pairs (`merge`, `link`, `ignore`) with connectivity-safe fallback edges.
 - `brainlink_add_note`: write durable Markdown memory and reindex.
+- `brainlink_add_notes`: write several notes in one batch, reindexing once.
 - `brainlink_remember`: capture durable memory with inferred title, tags and Context Links; supports dry-run.
 - `brainlink_inbox_add`: capture a quick untriaged memory item.
 - `brainlink_inbox_list`: list untriaged inbox memory items.
 - `brainlink_inbox_process`: suggest titles, tags and links for inbox items.
 - `brainlink_delete_note`: delete a durable Markdown note by title or path after explicit confirmation and reindex.
+- `brainlink_delete_notes`: delete several notes in one confirmed batch, reindexing once; per-note failures are reported.
 - `brainlink_add_file`: ingest a local file as a note and reindex.
 - `brainlink_canonicalize_context_links`: ensure existing notes link to inferred context hubs.
 - `brainlink_volatile_add`: write temporary agent-decided memory with TTL; volatile sections are included in context and never create durable graph edges.
@@ -1088,6 +1096,8 @@ If no `vault` is configured and no `--vault` flag is passed, Brainlink uses `$HO
   "defaultAgent": "shared",
   "autoIndexOnWrite": true,
   "autoCanonicalContextLinks": true,
+  "autoUpdateCheck": true,
+  "updateCheckIntervalMs": 86400000,
   "defaultSearchLimit": 8,
   "defaultContextTokens": 1500,
   "defaultContextStrategy": "auto",
@@ -1120,6 +1130,7 @@ If no `vault` is configured and no `--vault` flag is passed, Brainlink uses `$HO
 
 `autoIndexOnWrite` is optional and defaults to `true`. Set it to `false` to defer indexing after writes.
 `autoCanonicalContextLinks` is optional and defaults to `true`. When enabled, `blink add`, `brainlink_add_note` and `brainlink_add_file` add a canonical `## Context Links` entry to the inferred context hub, creating that hub when needed.
+`autoUpdateCheck` is optional and defaults to `true`. Brainlink checks the npm registry no more often than `updateCheckIntervalMs` and prints an update notice to `stderr`; it never installs updates automatically.
 
 ## Remote MCP Server
 
@@ -1206,6 +1217,41 @@ spec:
 
 Use `"embeddingProvider": "none"` when you want FTS-only indexing.
 
+### Embedding providers
+
+`embeddingProvider` selects how chunk and query embeddings are produced:
+
+- `local` (default): deterministic, dependency-free local vectors. No network.
+- `none`: FTS-only; no semantic vectors.
+- `ollama`: embeddings from a local [Ollama](https://ollama.com) server.
+- `openai`: embeddings from the OpenAI embeddings API.
+
+External providers are configured through the optional `embedding` block. The API key is referenced by environment variable name and never stored in the config file:
+
+```json
+{
+  "embeddingProvider": "ollama",
+  "embedding": {
+    "url": "http://127.0.0.1:11434",
+    "model": "nomic-embed-text",
+    "timeoutMs": 10000,
+    "batchSize": 64
+  }
+}
+```
+
+```json
+{
+  "embeddingProvider": "openai",
+  "embedding": {
+    "model": "text-embedding-3-small",
+    "apiKeyEnv": "OPENAI_API_KEY"
+  }
+}
+```
+
+Switching the provider — or the external `model` — changes the embedding space, so the next index run rebuilds vectors automatically. If an external provider is unreachable at query time, search degrades to lexical-only instead of failing; at index time it surfaces a clear error rather than mixing embedding spaces.
+
 For local security checks, set your Snyk token in the environment:
 
 ```bash

package/dist/application/add-note.js

@@ -2,7 +2,7 @@ import { writeMarkdownFile } from '../infrastructure/file-system-vault.js';
 import { sanitizeAgentId, sharedAgentId } from '../domain/agents.js';
 import { validateNoteInput } from '../domain/note-safety.js';
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { addCanonicalContextLinkToContent, ensureCanonicalContextHub } from './canonical-context-links.js';
+import { addCanonicalContextLinkToContent, ensureCanonicalContextHub, loadVisualContextRules } from './canonical-context-links.js';
 const slugify = (title) => title
     .normalize('NFKD')
     .replace(/[\u0300-\u036f]/g, '')
@@ -31,7 +31,7 @@ export const addNoteWithMetadata = async (vaultPath, title, content, agentId = s
     await ensureVault(vaultPath);
     const canonical = options.autoContextLinks === false
         ? null
-        : addCanonicalContextLinkToContent(title, content.trim());
+        : addCanonicalContextLinkToContent(title, content.trim(), await loadVisualContextRules(vaultPath), filename);
     const hub = canonical?.changed
         ? await ensureCanonicalContextHub(vaultPath, canonical.context, sanitizedAgentId)
         : null;

package/dist/application/build-context.js

@@ -74,7 +74,7 @@ export const buildContextPackage = async (vaultPath, query, limit, maxTokens, ag
     const cacheKey = toCacheKey(vaultPath, query, limit, maxTokens, agentId, mode, strategy);
     const dataSignature = await readContextDataSignature(vaultPath);
     const cached = contextCacheGet(cacheKey, dataSignature, contextCacheTtlMs);
-    if (cached) {
+    if (cached && cached.sections.length > 0) {
         return cached;
     }
     const shouldUseContextPack = strategy === 'cag' || strategy === 'auto';
@@ -83,7 +83,7 @@ export const buildContextPackage = async (vaultPath, query, limit, maxTokens, ag
         const packReadStart = performance.now();
         const pack = await readContextPack(vaultPath, { query, limit, maxTokens, agentId, mode }, dataSignature);
         packReadMs = elapsedMs(packReadStart);
-        if (pack.status === 'hit') {
+        if (pack.status === 'hit' && pack.context.sections.length > 0) {
             const contextFromPack = {
                 ...pack.context,
                 strategy: 'cag',
@@ -133,8 +133,12 @@ export const buildContextPackage = async (vaultPath, query, limit, maxTokens, ag
                     : 'RAG was requested; Brainlink assembled context directly from current retrieval results.'
         }
     };
+    // Never persist an empty assembly: a transient empty context (e.g. an index
+    // that was momentarily unavailable) must not be cached as a fresh pack and
+    // then served on every later read until the data signature changes.
+    const hasSections = sections.length > 0;
     const packWriteStart = performance.now();
-    const packPath = shouldUseContextPack
+    const packPath = shouldUseContextPack && hasSections
         ? await writeContextPack(vaultPath, { query, limit, maxTokens, agentId, mode }, dataSignature, context)
         : undefined;
     const packWriteMs = packPath ? elapsedMs(packWriteStart) : 0;
@@ -154,13 +158,15 @@ export const buildContextPackage = async (vaultPath, query, limit, maxTokens, ag
             packWriteMs
         })
     };
-    contextCacheSet({
-        key: cacheKey,
-        createdAt: Date.now(),
-        dataSignature,
-        strategy,
-        context: contextWithMetrics
-    });
+    if (hasSections) {
+        contextCacheSet({
+            key: cacheKey,
+            createdAt: Date.now(),
+            dataSignature,
+            strategy,
+            context: contextWithMetrics
+        });
+    }
     return contextWithMetrics;
 };
 export const buildContext = async (vaultPath, query, limit, maxTokens, agentId, mode, strategy, contextCacheTtlMs = 120_000) => {

package/dist/application/canonical-context-links.js

@@ -1,6 +1,6 @@
 import { readFile } from 'node:fs/promises';
 import { join } from 'node:path';
-import { inferVisualGraphContext } from '../domain/graph-contexts.js';
+import { deriveVisualContextRules, inferVisualGraphContext } from '../domain/graph-contexts.js';
 import { sanitizeAgentId, sharedAgentId } from '../domain/agents.js';
 import { extractContextLinkWeights, parseMarkdownDocument } from '../domain/markdown.js';
 import { ensureVault, readMarkdownFileSummaries, writeMarkdownFile } from '../infrastructure/file-system-vault.js';
@@ -60,6 +60,42 @@ const buildHubContent = (hubTitle, contextTitle, agentId) => [
     `Canonical hub for the ${contextTitle} context. #memory #hub`,
     ''
 ].join('\n');
+const isHubNotePath = (relativePath) => /(^|\/)[^/]*-hub\.md$/i.test(relativePath);
+// Hub notes change rarely, but loadVisualContextRules runs on every auto-linked
+// note write. Cache the derived rules per vault for a short window so a burst of
+// writes reads the hub notes once. Creating a hub clears the cache so a new
+// context is picked up immediately.
+const visualContextRulesCacheTtlMs = 15_000;
+const visualContextRulesCache = new Map();
+const invalidateVisualContextRulesCache = () => {
+    visualContextRulesCache.clear();
+};
+// Build visual-context rules from the vault's hub notes only, so a single-note
+// write path can classify into vault-specific contexts without reading the
+// whole vault.
+export const loadVisualContextRules = async (vaultPath) => {
+    const absoluteVaultPath = await ensureVault(vaultPath);
+    const cached = visualContextRulesCache.get(absoluteVaultPath);
+    if (cached && Date.now() - cached.createdAt <= visualContextRulesCacheTtlMs) {
+        return cached.rules;
+    }
+    const summaries = await readMarkdownFileSummaries(absoluteVaultPath);
+    const documents = await Promise.all(summaries
+        .filter((summary) => isHubNotePath(summary.relativePath))
+        .map(async (summary) => {
+        const content = await readFile(summary.absolutePath, 'utf8');
+        return parseMarkdownDocument({
+            absolutePath: summary.absolutePath,
+            vaultPath: absoluteVaultPath,
+            content,
+            createdAt: summary.createdAt,
+            updatedAt: summary.updatedAt
+        });
+    }));
+    const rules = deriveVisualContextRules(documents);
+    visualContextRulesCache.set(absoluteVaultPath, { createdAt: Date.now(), rules });
+    return rules;
+};
 const readNotes = async (vaultPath) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
     const summaries = await readMarkdownFileSummaries(absoluteVaultPath);
@@ -92,6 +128,7 @@ export const ensureCanonicalContextHub = async (vaultPath, contextTitle, agentId
         };
     }
     const path = await writeMarkdownFile(vaultPath, hubPath, buildHubContent(hubTitle, contextTitle, agentId));
+    invalidateVisualContextRulesCache();
     return {
         created: true,
         title: hubTitle,
@@ -102,6 +139,7 @@ export const canonicalizeContextLinks = async (vaultPath, options = {}) => {
     const agentId = options.agentId ? sanitizeAgentId(options.agentId) : undefined;
     const createMissingHubs = options.createMissingHubs !== false;
     const notes = await readNotes(vaultPath);
+    const rules = deriveVisualContextRules(notes.map((note) => note.document));
     const scopedNotes = agentId ? notes.filter((note) => note.document.agentId === agentId) : notes;
     const knownTitles = new Set(notes.map((note) => normalizeTitle(note.document.title)));
     const entries = [];
@@ -124,6 +162,7 @@ export const canonicalizeContextLinks = async (vaultPath, options = {}) => {
         knownTitles.add(normalizeTitle(hubTitle));
         if (!options.dryRun) {
             await writeMarkdownFile(vaultPath, path, buildHubContent(hubTitle, contextTitle, targetAgentId));
+            invalidateVisualContextRulesCache();
         }
         entries.push({
             path,
@@ -136,7 +175,7 @@ export const canonicalizeContextLinks = async (vaultPath, options = {}) => {
         return true;
     };
     for (const note of scopedNotes) {
-        const context = inferVisualGraphContext(note.document);
+        const context = inferVisualGraphContext(note.document, rules);
         const hubTitle = hubTitleForContext(context.title);
         const isHub = normalizeTitle(note.document.title) === normalizeTitle(hubTitle);
         if (isHub) {
@@ -189,15 +228,15 @@ export const canonicalizeContextLinks = async (vaultPath, options = {}) => {
         entries
     };
 };
-export const addCanonicalContextLinkToContent = (title, content) => {
+export const addCanonicalContextLinkToContent = (title, content, rules = [], notePath = '') => {
     const context = inferVisualGraphContext({
         id: '',
         agentId: sharedAgentId,
         title,
-        path: '',
+        path: notePath,
         content,
         tags: [],
-    });
+    }, rules);
     const hubTitle = hubTitleForContext(context.title);
     const nextContent = normalizeTitle(title) === normalizeTitle(hubTitle) ? content : upsertCanonicalContextLink(content, hubTitle);
     return {

package/dist/application/check-package-update.js

@@ -0,0 +1,105 @@
+import { mkdir, readFile, writeFile } from 'node:fs/promises';
+import { dirname, join } from 'node:path';
+import { getBrainlinkHomePath } from '../infrastructure/paths.js';
+const defaultTimeoutMs = 800;
+const registryBaseUrl = 'https://registry.npmjs.org';
+const isRecord = (value) => typeof value === 'object' && value !== null && !Array.isArray(value);
+const parseCache = (value) => isRecord(value) ? { checkedAt: typeof value.checkedAt === 'string' ? value.checkedAt : undefined, latestVersion: typeof value.latestVersion === 'string' ? value.latestVersion : undefined } : {};
+const readCache = async (path) => {
+    try {
+        return parseCache(JSON.parse(await readFile(path, 'utf8')));
+    }
+    catch (error) {
+        if (error instanceof Error && 'code' in error && error.code === 'ENOENT') {
+            return {};
+        }
+        return {};
+    }
+};
+const writeCache = async (path, cache) => {
+    await mkdir(dirname(path), { recursive: true, mode: 0o700 });
+    await writeFile(path, `${JSON.stringify(cache, null, 2)}\n`, { encoding: 'utf8', mode: 0o600 });
+};
+const parseVersionParts = (version) => {
+    const [core] = version.split('-');
+    const parts = core?.split('.').map((part) => Number.parseInt(part, 10)) ?? [];
+    return parts.length > 0 && parts.every((part) => Number.isFinite(part) && part >= 0) ? parts : null;
+};
+export const comparePackageVersions = (left, right) => {
+    const leftParts = parseVersionParts(left);
+    const rightParts = parseVersionParts(right);
+    if (!leftParts || !rightParts) {
+        return left.localeCompare(right);
+    }
+    const length = Math.max(leftParts.length, rightParts.length);
+    for (let index = 0; index < length; index += 1) {
+        const leftPart = leftParts[index] ?? 0;
+        const rightPart = rightParts[index] ?? 0;
+        if (leftPart !== rightPart) {
+            return leftPart > rightPart ? 1 : -1;
+        }
+    }
+    return 0;
+};
+const isCacheFresh = (cache, now, intervalMs) => {
+    if (!cache.checkedAt) {
+        return false;
+    }
+    const checkedAt = new Date(cache.checkedAt).getTime();
+    return Number.isFinite(checkedAt) && now.getTime() - checkedAt < intervalMs;
+};
+const createSkippedStatus = (input, now, reason, latestVersion = null) => ({
+    currentVersion: input.currentVersion,
+    latestVersion,
+    updateAvailable: latestVersion ? comparePackageVersions(latestVersion, input.currentVersion) > 0 : false,
+    checkedAt: now.toISOString(),
+    installCommand: `npm install -g ${input.packageName}@latest`,
+    skipped: true,
+    reason
+});
+const readLatestVersion = async (packageName, fetchImpl, timeoutMs) => {
+    const controller = new AbortController();
+    const timeout = setTimeout(() => controller.abort(), timeoutMs);
+    try {
+        const response = await fetchImpl(`${registryBaseUrl}/${encodeURIComponent(packageName)}/latest`, { signal: controller.signal });
+        if (!response.ok) {
+            return null;
+        }
+        const payload = await response.json();
+        return isRecord(payload) && typeof payload.version === 'string' ? payload.version : null;
+    }
+    finally {
+        clearTimeout(timeout);
+    }
+};
+export const checkPackageUpdate = async (input) => {
+    const now = input.now ?? new Date();
+    const cachePath = input.cachePath ?? join(getBrainlinkHomePath(), 'update-check.json');
+    const cache = await readCache(cachePath);
+    if (!input.enabled) {
+        return createSkippedStatus(input, now, 'disabled');
+    }
+    if (isCacheFresh(cache, now, input.intervalMs)) {
+        return createSkippedStatus(input, now, 'cache-fresh', cache.latestVersion ?? null);
+    }
+    const fetchImpl = input.fetch ?? globalThis.fetch;
+    if (!fetchImpl) {
+        return createSkippedStatus(input, now, 'fetch-unavailable');
+    }
+    const latestVersion = await readLatestVersion(input.packageName, fetchImpl, input.timeoutMs ?? defaultTimeoutMs);
+    if (!latestVersion) {
+        return createSkippedStatus(input, now, 'latest-unavailable');
+    }
+    await writeCache(cachePath, {
+        checkedAt: now.toISOString(),
+        latestVersion
+    });
+    return {
+        currentVersion: input.currentVersion,
+        latestVersion,
+        updateAvailable: comparePackageVersions(latestVersion, input.currentVersion) > 0,
+        checkedAt: now.toISOString(),
+        installCommand: `npm install -g ${input.packageName}@latest`,
+        skipped: false
+    };
+};

package/dist/application/find-similar-notes.js

@@ -0,0 +1,75 @@
+import { resolve } from 'node:path';
+import { parseMarkdownDocument } from '../domain/markdown.js';
+import { ensureVault, readMarkdownFiles } from '../infrastructure/file-system-vault.js';
+import { searchKnowledge } from './search-knowledge.js';
+const defaultLimit = 5;
+// Pull extra candidates before deduping by document and dropping the note
+// itself, so a note with many chunks does not crowd out distinct results.
+const candidateOverscan = 8;
+const maxQueryCharacters = 2000;
+const normalizeTitleKey = (title) => title.trim().replace(/\.md$/i, '').toLowerCase();
+const resolveTarget = async (vaultPath, input) => {
+    const absoluteVaultPath = await ensureVault(vaultPath);
+    const files = await readMarkdownFiles(vaultPath);
+    const documents = files.map((file) => ({
+        file,
+        document: parseMarkdownDocument({
+            absolutePath: file.absolutePath,
+            vaultPath: absoluteVaultPath,
+            content: file.content,
+            createdAt: file.createdAt,
+            updatedAt: file.updatedAt
+        })
+    }));
+    if (input.path != null && input.path.trim().length > 0) {
+        const expected = resolve(absoluteVaultPath, input.path);
+        const match = documents.find((entry) => resolve(entry.file.absolutePath) === expected);
+        if (!match) {
+            throw new Error(`No note found at path: ${input.path}`);
+        }
+        return { title: match.document.title, path: match.document.path, content: match.file.content };
+    }
+    const expectedTitle = normalizeTitleKey(input.title ?? '');
+    const matches = documents.filter((entry) => normalizeTitleKey(entry.document.title) === expectedTitle &&
+        (input.agentId == null || entry.document.agentId === input.agentId));
+    if (matches.length === 0) {
+        throw new Error(`No note found with title: ${input.title}`);
+    }
+    if (matches.length > 1) {
+        throw new Error(`Multiple notes match title "${input.title}". Use path instead: ${matches.map((entry) => entry.document.path).join(', ')}`);
+    }
+    return { title: matches[0].document.title, path: matches[0].document.path, content: matches[0].file.content };
+};
+const buildQuery = (target) => `${target.title}\n${target.content}`.slice(0, maxQueryCharacters);
+// Find notes most similar to a given note by using the note's own title and
+// content as the retrieval query, then excluding the note itself.
+export const findSimilarNotes = async (vaultPath, input) => {
+    const hasTitle = input.title != null && input.title.trim().length > 0;
+    const hasPath = input.path != null && input.path.trim().length > 0;
+    if (hasTitle === hasPath) {
+        throw new Error('Use exactly one selector: title or path.');
+    }
+    const target = await resolveTarget(vaultPath, input);
+    const limit = Math.max(1, input.limit ?? defaultLimit);
+    const results = await searchKnowledge(vaultPath, buildQuery(target), limit + candidateOverscan, input.agentId, input.mode ?? 'hybrid');
+    const bestByPath = new Map();
+    for (const result of results) {
+        if (result.path === target.path) {
+            continue;
+        }
+        const existing = bestByPath.get(result.path);
+        if (!existing || result.score > existing.score) {
+            bestByPath.set(result.path, {
+                title: result.title,
+                path: result.path,
+                score: result.score,
+                searchMode: result.searchMode,
+                tags: result.tags
+            });
+        }
+    }
+    const similar = Array.from(bestByPath.values())
+        .sort((left, right) => right.score - left.score || left.title.localeCompare(right.title))
+        .slice(0, limit);
+    return { target: { title: target.title, path: target.path }, similar };
+};