@andespindola/brainlink 1.0.7 → 1.2.0

package/CHANGELOG.md

@@ -1,6 +1,21 @@
 # Changelog
 
-## Unreleased
+## 1.2.0
+
+- **Reciprocal Rank Fusion for hybrid search.** Hybrid mode now fuses the lexical (BM25) and semantic rankings by rank position (`Σ 1/(k+rank)`, k=60) instead of a fixed `0.6/0.4` min-max blend, which is robust to the two components' very different score scales. Gated by a new `hybridFusion` config (`rrf` default | `linear`); `fts`/`semantic` modes and stored data are unchanged.
+- **Per-document CAG invalidation.** Persisted context packs now record the documents they were assembled from and are fresh only while each cited document is unchanged, so editing one note refreshes only the packs that cite it instead of the whole cache. A new optional `cagPackTtlMs` bounds pack lifetime as a safety net for newly added notes. Freshness is evaluated with a few `stat`s and no index load.
+- **int8 ANN with partial exact-vector reads (binary backend).** Search loads a compact int8 scan index and reads the exact Float32 vectors by offset only for the shortlist it reranks (embedding-bucket coarse prune → int8 scan → exact Float32 rerank), so the exact vectors never become fully resident on a large vault. Small vaults rerank every candidate exactly, staying bit-identical to a full scan; `fts` reads no vectors.
+- **The binary storage backend is now the default** (`storageBackend: "binary"`). Existing `json` vaults auto-migrate on first use (no re-embedding), and switching backends forces a one-time full reindex so the index is rewritten in the active format. The in-memory search/context caches key on a backend-agnostic commit signature (fixing a stale-read window on the binary backend). The `json` backend remains available.
+- **Vault versioning over MCP.** `brainlink_vault_status` / `_init` / `_commit` / `_history` / `_pull` / `_restore` / `_export` / `_import` expose the git and `.blvault` versioning to agents (previously CLI-only); pull/restore/import reindex automatically.
+- **Self-managed versioning repository.** `brainlink_vault_provision` / `blink vault-provision` create a private GitHub repository for the vault with the authenticated `gh` CLI, wire it as origin, push the first snapshot, and enable `autoVersion` — after which a successful index that changed Markdown commits and pushes automatically (best-effort; a git/network failure never breaks indexing).
+- **Encrypted vault versioning.** With `vaultEncryption` (enabled by `vault-provision` by default), the git repository holds AES-256-GCM ciphertext (`<note>.md.enc`) and git-ignores the plaintext Markdown, so the repository is opaque to anyone without the per-vault key (in `$BRAINLINK_HOME/keys`, never committed, or via `BRAINLINK_VAULT_KEY`). Encryption is deterministic per note (no git churn); pull/restore/clone decrypt back to Markdown before reindexing.
+
+## 1.1.0
+
+- Added an optional **binary storage backend** (`storageBackend: "binary"`) that stores chunk embeddings as dense little-endian Float32 (`vectors.f32`) and document/chunk/link metadata as Brotli-compressed JSON, cutting the on-disk index ~75% on a typical vault with much cheaper loads than parsing decimal-text vectors. The graph/search/link reads now run through a shared query core so the JSON and binary backends are behavior-identical for the same candidates. Switching to `binary` converts an existing `index.json` in place on first use (no re-embedding) and falls back to it for reads until the binary store is materialized. The canonical Markdown is unaffected and the index stays fully regenerable.
+- Introduced a `KnowledgeStore` port and a shared, backend-agnostic ranker, so the index storage layout can evolve without touching the application layer and every backend produces an identical ranking.
+- Added **vault versioning** for the canonical Markdown. Git: `vault-init-git`, `vault-commit`, `vault-status`, `vault-history`, `vault-push`, `vault-pull`, `vault-restore`, `vault-clone` (driven through the system `git` binary; the derived `.brainlink/` is git-ignored). Portable snapshots: `vault-export`/`vault-import` produce and restore a self-contained gzip `.blvault` with per-file SHA-256. Every operation that changes Markdown reindexes automatically so the vault is immediately usable, merge conflicts abort cleanly instead of leaving markers in Markdown, conflicting local edits are preserved as `*.conflict-<ts>.md`, and remote URLs with inline credentials are rejected.
+- Pinned the transitive `fast-uri` dependency to `>=4.1.0` to clear a high-severity Snyk advisory affecting the entire 3.x line.
 
 - Added `brainlink_similar_notes` MCP tool: find the notes most similar to a given note (selected by title or path), using the note's own content as the retrieval query and excluding the note itself.
 - Added batch write tools `brainlink_add_notes` and `brainlink_delete_notes` that write or delete up to 100 notes in a single call and reindex once for the whole batch; batch deletion reports per-note failures without aborting the batch.

package/README.md

@@ -71,7 +71,7 @@ Legacy `.jsonl.gz` packs are upgraded to `.blpk` automatically on first search/c
 - Full-text, semantic and hybrid retrieval on a local file index.
 - Middle-out context assembly around the strongest chunk per document.
 - In-process index and context caching with automatic invalidation on index updates.
-- Optional CAG context packs at `.brainlink/context-packs/*.json`, derived from the current index signature and reusable across repeated context calls.
+- Optional CAG context packs at `.brainlink/context-packs/*.json`, invalidated per cited document (editing one note only refreshes the packs that cite it) and reusable across repeated context calls.
 - HTTP graph server caches generated frontend assets and graph-layout JSON payloads by signature, and skips layout serialization when ETag returns `304`.
 - Compressed-space prefiltering for `.blpk` packs before decryption and scan.
 - Incremental indexing that reprocesses only changed markdown files and reuses existing chunks/embeddings for unchanged notes.
@@ -560,6 +560,9 @@ Available tools:
 - `brainlink_doctor_actions`: return vault checks plus prioritized executable next actions.
 - `brainlink_validate`: validate broken links and orphan notes.
 - `brainlink_sync`: run index, stats, validation, broken-link and orphan checks in one call.
+- `brainlink_vault_provision`: create a private GitHub repository for the vault with the authenticated `gh` CLI, wire it as `origin`, push the first snapshot, and enable `autoVersion` so later indexed changes are committed and pushed automatically. By default the repository is **encrypted** (`vaultEncryption`): it holds AES-256-GCM ciphertext, decryptable only by a Brainlink instance with the local key.
+- `brainlink_vault_status` / `brainlink_vault_init` / `brainlink_vault_commit` / `brainlink_vault_history` / `brainlink_vault_pull` / `brainlink_vault_restore`: version the vault's canonical Markdown with git (the derived index is git-ignored). Pull and restore reindex changed files automatically so the vault stays immediately searchable.
+- `brainlink_vault_export` / `brainlink_vault_import`: write and read a portable `.blvault` snapshot (gzip envelope of the Markdown with per-file SHA-256). Import verifies checksums, preserves divergent local files as `*.conflict-<ts>.md`, and reindexes.
 - `brainlink_graph`: read indexed graph nodes and weighted links.
 - `brainlink_graph_contexts`: list the visual graph contexts used by the local server.
 - `brainlink_broken_links`: list unresolved wiki links.
@@ -759,6 +762,30 @@ When the configured default vault is changed manually in config files, Brainlink
 Use `--global` to write to `$BRAINLINK_HOME/brainlink.config.json`, `--no-migrate` to skip migration, and `--no-index` to skip post-migration indexing.
 `config doctor` is dry-run by default; use `--fix` to apply safe config normalization and allowlist fixes.
 
+#### Storage backend
+
+`storageBackend` selects how the derived index is stored (the canonical Markdown is unaffected):
+
+- `binary` (default) — Float32 vectors plus Brotli-compressed metadata; a smaller, faster-to-load index (~75% smaller on a typical vault). Switching is seamless: an existing `index.json` is converted in place on first use (no re-embedding), and changing the backend forces a one-time full reindex so the index is rewritten in the active format. On large vaults, semantic/hybrid search uses an approximate-nearest-neighbour path — a compact int8 vector scan (with embedding-bucket coarse pruning) shortlists candidates, and only that shortlist's exact Float32 vectors are read from disk and reranked, so the exact vectors never become fully resident. Small vaults rerank every candidate exactly, staying bit-identical to a full scan.
+- `json` — the original single `index.json`. Larger and loaded whole, but a simple, human-readable artifact.
+
+```json
+{ "storageBackend": "json" }
+```
+
+#### Hybrid fusion
+
+`hybridFusion` selects how the lexical (BM25) and semantic (cosine) rankings are combined in `hybrid` search mode:
+
+- `rrf` (default) — Reciprocal Rank Fusion. Combines the two rankings by rank position (`Σ 1/(k + rank)`), so a document both retrievers agree on ranks highly regardless of the very different score scales of BM25 and cosine. No weight tuning required.
+- `linear` — the legacy min-max normalized weighted blend (`0.6·lexical + 0.4·semantic`).
+
+`fts` and `semantic` modes are unaffected. The change is additive: existing vaults need no reindex.
+
+```json
+{ "hybridFusion": "linear" }
+```
+
 ### `vaults`
 
 ```bash
@@ -912,6 +939,36 @@ blink pack-backup --vault ./vault --json
 Creates an offline backup artifact of encrypted search packs with a second compression pass.
 This is intentionally outside the online retrieval path (`index`, `search`, `context`).
 
+### Vault Versioning
+
+Version the canonical Markdown with git, or move a vault as a single portable file.
+The derived `.brainlink/` directory is never versioned; any operation that changes
+Markdown reindexes automatically so the vault stays immediately usable.
+
+```bash
+# One-shot: create an ENCRYPTED private GitHub repo (via authenticated gh),
+# push, and enable autoVersion so later indexed changes commit and push
+# automatically. The repo holds AES-256-GCM ciphertext; only a Brainlink
+# instance with the local key can decrypt. Use --no-encrypt for plaintext.
+blink vault-provision --name brainlink-vault
+
+# Git: history, remotes, restore
+blink vault-init-git --remote git@github.com:you/your-vault.git
+blink vault-commit -m "snapshot notes"
+blink vault-push
+blink vault-pull                 # pulls and reindexes changed notes
+blink vault-history --limit 20
+blink vault-restore <commit>     # restore Markdown from a ref and reindex
+blink vault-clone git@github.com:you/your-vault.git ./vault   # clone + build index
+blink vault-status
+
+# Portable snapshot (self-contained, index excluded; import rebuilds it)
+blink vault-export -o vault.blvault
+blink vault-import vault.blvault  # conflicting local edits preserved as *.conflict-<ts>.md
+```
+
+Inline credentials in a remote URL are rejected — use SSH or a git credential helper.
+
 ### `agents`
 
 ```bash
@@ -955,7 +1012,9 @@ blink context-packs --vault ./vault --stale --clear
 
 Builds a compact context package for an agent.
 Repeated calls with the same vault, agent, query, mode and token/limit settings are served from a short in-memory cache while the index is unchanged.
-The default strategy is configured by `defaultContextStrategy` and starts as `auto`, which uses CAG when a fresh pack exists and RAG otherwise, refreshing a pack for future calls. `--strategy cag` enables cache-augmented context generation by reading or refreshing a persisted context pack under `.brainlink/context-packs`; `--strategy rag` forces fresh retrieval assembly from the current index. Context responses include `cache`, `metrics`, `requestedStrategy` and `recommendedStrategy` metadata. Packs are derived artifacts and become stale when the index or volatile memory signature changes.
+The default strategy is configured by `defaultContextStrategy` and starts as `auto`, which uses CAG when a fresh pack exists and RAG otherwise, refreshing a pack for future calls. `--strategy cag` enables cache-augmented context generation by reading or refreshing a persisted context pack under `.brainlink/context-packs`; `--strategy rag` forces fresh retrieval assembly from the current index. Context responses include `cache`, `metrics`, `requestedStrategy` and `recommendedStrategy` metadata.
+
+Packs are derived artifacts invalidated **per cited document**: a pack becomes stale only when one of the documents it was assembled from (or volatile memory) changes, so editing a single note no longer invalidates every pack. `cagPackTtlMs` (default `0`, disabled) sets an optional upper bound in milliseconds on how long a pack survives even when all its cited documents are unchanged — a safety net for a newly added note that would now rank into the result but is not among the pack's existing dependencies.
 
 ### `links`
 
@@ -1129,6 +1188,8 @@ If no `vault` is configured and no `--vault` flag is passed, Brainlink uses `$HO
 `agentProfiles` is optional. When present, CLI and MCP resolve `mode`, `limit`, `tokens` and context `strategy` per agent automatically, then fallback to global defaults.
 
 `autoIndexOnWrite` is optional and defaults to `true`. Set it to `false` to defer indexing after writes.
+`autoVersion` is optional and defaults to `false`. When enabled (e.g. by `blink vault-provision` / `brainlink_vault_provision`), a successful index that changed Markdown commits and pushes the vault to its git remote automatically, best-effort — a git or network failure is logged and never breaks indexing. It is a no-op unless the vault is a git repo with an `origin` remote.
+`vaultEncryption` is optional and defaults to `false`. When enabled, vault versioning commits AES-256-GCM ciphertext (`<note>.md.enc`) and git-ignores the plaintext Markdown, so the repository is opaque to anyone without the key — only a Brainlink instance holding the per-vault key can decrypt it. The key lives at `$BRAINLINK_HOME/keys/vault-versioning-<hash>.key` (outside the vault, never committed) or can be supplied via `BRAINLINK_VAULT_KEY` to restore an encrypted vault on another machine. Pull, restore and clone decrypt back to Markdown before reindexing; encryption is deterministic per note, so unchanged notes add no git churn. `blink vault-provision` enables this by default.
 `autoCanonicalContextLinks` is optional and defaults to `true`. When enabled, `blink add`, `brainlink_add_note` and `brainlink_add_file` add a canonical `## Context Links` entry to the inferred context hub, creating that hub when needed.
 `autoUpdateCheck` is optional and defaults to `true`. Brainlink checks the npm registry no more often than `updateCheckIntervalMs` and prints an update notice to `stderr`; it never installs updates automatically.
 

package/dist/application/analyze-vault.js

@@ -50,7 +50,7 @@ export const getExtendedStats = async (vaultPath, agentId) => {
     await searchKnowledge(absoluteVaultPath, probeQuery, Math.min(defaults.defaultSearchLimit, 8), agentId, 'hybrid');
     const searchLatency = performance.now() - searchStart;
     const contextStart = performance.now();
-    await buildContextPackage(absoluteVaultPath, probeQuery, Math.min(defaults.defaultSearchLimit, 8), defaults.defaultContextTokens, agentId, 'hybrid', undefined, defaults.defaultContextCacheTtlMs);
+    await buildContextPackage(absoluteVaultPath, probeQuery, Math.min(defaults.defaultSearchLimit, 8), defaults.defaultContextTokens, agentId, 'hybrid', undefined, defaults.defaultContextCacheTtlMs, defaults.cagPackTtlMs);
     const contextLatency = performance.now() - contextStart;
     return {
         stats,

package/dist/application/build-context.js

@@ -2,7 +2,7 @@ import { stat } from 'node:fs/promises';
 import { performance } from 'node:perf_hooks';
 import { formatContextPackage, selectContextSections } from '../domain/context.js';
 import { readContextPack, writeContextPack } from '../infrastructure/context-packs.js';
-import { indexStoragePath } from '../infrastructure/file-index.js';
+import { indexCommitSignature } from '../infrastructure/index-signature.js';
 import { searchVolatileMemory, volatileMemoryStoragePath } from '../infrastructure/volatile-memory.js';
 import { searchKnowledge } from './search-knowledge.js';
 const contextCacheMaxEntries = 200;
@@ -16,7 +16,12 @@ const readFileSignature = async (path) => {
         return '0:0';
     }
 };
-export const readContextDataSignature = async (vaultPath) => `${await readFileSignature(indexStoragePath(vaultPath))}|${await readFileSignature(volatileMemoryStoragePath(vaultPath))}`;
+export const readContextDataSignature = async (vaultPath) => `${await indexCommitSignature(vaultPath)}|${await readFileSignature(volatileMemoryStoragePath(vaultPath))}`;
+// Signature of just the volatile-memory file. Disk context packs no longer gate
+// on the whole-vault index signature (that invalidated every pack on any edit);
+// they gate per cited document plus this volatile signature, so turn memory
+// changes still refresh a pack while unrelated document edits do not.
+export const readVolatileSignature = (vaultPath) => readFileSignature(volatileMemoryStoragePath(vaultPath));
 const toCacheKey = (vaultPath, query, limit, maxTokens, agentId, mode, strategy) => JSON.stringify({
     vaultPath,
     query: query.trim().toLowerCase(),
@@ -69,7 +74,7 @@ const emptyMetrics = (context, totalMs, overrides = {}) => ({
     estimatedTokens: estimateSectionTokens(context),
     ...overrides
 });
-export const buildContextPackage = async (vaultPath, query, limit, maxTokens, agentId, mode, strategy = 'rag', contextCacheTtlMs = 120_000) => {
+export const buildContextPackage = async (vaultPath, query, limit, maxTokens, agentId, mode, strategy = 'rag', contextCacheTtlMs = 120_000, cagPackTtlMs = 0) => {
     const totalStart = performance.now();
     const cacheKey = toCacheKey(vaultPath, query, limit, maxTokens, agentId, mode, strategy);
     const dataSignature = await readContextDataSignature(vaultPath);
@@ -78,10 +83,12 @@ export const buildContextPackage = async (vaultPath, query, limit, maxTokens, ag
         return cached;
     }
     const shouldUseContextPack = strategy === 'cag' || strategy === 'auto';
+    const volatileSignature = await readVolatileSignature(vaultPath);
+    const packFreshness = { volatileSignature, ttlMs: cagPackTtlMs };
     let packReadMs = 0;
     if (shouldUseContextPack) {
         const packReadStart = performance.now();
-        const pack = await readContextPack(vaultPath, { query, limit, maxTokens, agentId, mode }, dataSignature);
+        const pack = await readContextPack(vaultPath, { query, limit, maxTokens, agentId, mode }, packFreshness);
         packReadMs = elapsedMs(packReadStart);
         if (pack.status === 'hit' && pack.context.sections.length > 0) {
             const contextFromPack = {
@@ -139,7 +146,7 @@ export const buildContextPackage = async (vaultPath, query, limit, maxTokens, ag
     const hasSections = sections.length > 0;
     const packWriteStart = performance.now();
     const packPath = shouldUseContextPack && hasSections
-        ? await writeContextPack(vaultPath, { query, limit, maxTokens, agentId, mode }, dataSignature, context)
+        ? await writeContextPack(vaultPath, { query, limit, maxTokens, agentId, mode }, context, { volatileSignature })
         : undefined;
     const packWriteMs = packPath ? elapsedMs(packWriteStart) : 0;
     const contextWithMetadata = withCacheMetadata(context, {
@@ -169,7 +176,7 @@ export const buildContextPackage = async (vaultPath, query, limit, maxTokens, ag
     }
     return contextWithMetrics;
 };
-export const buildContext = async (vaultPath, query, limit, maxTokens, agentId, mode, strategy, contextCacheTtlMs = 120_000) => {
-    const contextPackage = await buildContextPackage(vaultPath, query, limit, maxTokens, agentId, mode, strategy, contextCacheTtlMs);
+export const buildContext = async (vaultPath, query, limit, maxTokens, agentId, mode, strategy, contextCacheTtlMs = 120_000, cagPackTtlMs = 0) => {
+    const contextPackage = await buildContextPackage(vaultPath, query, limit, maxTokens, agentId, mode, strategy, contextCacheTtlMs, cagPackTtlMs);
     return contextPackage.content;
 };

package/dist/application/get-graph-node.js

@@ -1,8 +1,8 @@
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { openFileIndex } from '../infrastructure/file-index.js';
+import { openKnowledgeStore } from '../infrastructure/knowledge-store/index.js';
 export const getGraphNode = async (vaultPath, id, agentId) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openFileIndex(absoluteVaultPath);
+    const index = openKnowledgeStore(absoluteVaultPath);
     try {
         return await index.getGraphNode(id, agentId);
     }

package/dist/application/get-graph-summary.js

@@ -1,8 +1,8 @@
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { openFileIndex } from '../infrastructure/file-index.js';
+import { openKnowledgeStore } from '../infrastructure/knowledge-store/index.js';
 export const getGraphSummary = async (vaultPath, agentId) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openFileIndex(absoluteVaultPath);
+    const index = openKnowledgeStore(absoluteVaultPath);
     try {
         return await index.getGraphSummary(agentId);
     }

package/dist/application/get-graph.js

@@ -1,8 +1,8 @@
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { openFileIndex } from '../infrastructure/file-index.js';
+import { openKnowledgeStore } from '../infrastructure/knowledge-store/index.js';
 export const getGraph = async (vaultPath, agentId) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openFileIndex(absoluteVaultPath);
+    const index = openKnowledgeStore(absoluteVaultPath);
     try {
         return await index.getGraph(agentId);
     }

package/dist/application/index-vault-phases.js

@@ -72,6 +72,7 @@ export const detectChanges = (params) => {
     const settingsChanged = previousState == null ||
         previousState.chunkSize !== config.chunkSize ||
         previousState.embeddingProvider !== embeddingSignature(config) ||
+        previousState.storageBackend !== config.storageBackend ||
         graphLinkModelChanged;
     const packSettingsChanged = previousState == null ||
         previousState.searchPackRowChunkSize !== config.searchPack.rowChunkSize ||

package/dist/application/index-vault.js

@@ -4,7 +4,8 @@ import { embeddingSignature, loadBrainlinkConfig } from '../infrastructure/confi
 import { ensureVault, readMarkdownFileSummaries } from '../infrastructure/file-system-vault.js';
 import { readIndexState, writeIndexState } from '../infrastructure/index-state.js';
 import { ensureSearchPackManifest } from '../infrastructure/search-packs.js';
-import { openFileIndex } from '../infrastructure/file-index.js';
+import { openKnowledgeStore } from '../infrastructure/knowledge-store/index.js';
+import { autoVersionVault } from './vault-git-core.js';
 import { assembleIndexedDocuments, createTitleMaps, decidePackRebuild, detectChanges, embedChangedDocuments, rebuildSearchPacksIfNeeded } from './index-vault-phases.js';
 const toIndexResult = (documents) => ({
     documentCount: documents.length,
@@ -56,7 +57,7 @@ export const indexVaultWithOptions = async (vaultPath, options) => {
         hasPreviousState: previousState != null
     });
     const fullReindex = options.full === true;
-    const index = openFileIndex(absoluteVaultPath);
+    const index = openKnowledgeStore(absoluteVaultPath);
     try {
         const existingIndexedDocuments = await index.getIndexedDocuments();
         const existingByPath = new Map(existingIndexedDocuments.map((document) => [document.document.path, document]));
@@ -162,6 +163,7 @@ export const indexVaultWithOptions = async (vaultPath, options) => {
         await writeIndexState(absoluteVaultPath, {
             chunkSize: config.chunkSize,
             embeddingProvider: embeddingSignature(config),
+            storageBackend: config.storageBackend,
             graphLinkModelVersion,
             searchPackRowChunkSize: config.searchPack.rowChunkSize,
             searchPackCompressionLevel: config.searchPack.compressionLevel,
@@ -169,6 +171,18 @@ export const indexVaultWithOptions = async (vaultPath, options) => {
             files: currentSnapshot,
             pendingPackChanges: packsRebuilt ? 0 : pendingPackChanges
         });
+        // Auto-version the vault when enabled and Markdown actually changed. Runs
+        // after the index is committed, is best-effort (a git/network failure never
+        // breaks indexing), and is a no-op unless the vault is a git repo with an
+        // origin remote — commitVault itself skips when nothing is staged.
+        if (config.autoVersion && changedDocumentsByPath.size > 0) {
+            try {
+                await autoVersionVault(absoluteVaultPath, `chore(vault): update ${new Date().toISOString()}`);
+            }
+            catch (error) {
+                process.stderr.write(`brainlink: auto-version skipped (${error instanceof Error ? error.message : String(error)})\n`);
+            }
+        }
         const result = {
             ...toIndexResult(indexedDocuments),
             elapsedMs: elapsedMs(),

package/dist/application/list-agents.js

@@ -1,8 +1,8 @@
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { openFileIndex } from '../infrastructure/file-index.js';
+import { openKnowledgeStore } from '../infrastructure/knowledge-store/index.js';
 export const listAgents = async (vaultPath) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openFileIndex(absoluteVaultPath);
+    const index = openKnowledgeStore(absoluteVaultPath);
     try {
         return await index.listAgents();
     }

package/dist/application/list-links.js

@@ -1,8 +1,8 @@
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { openFileIndex } from '../infrastructure/file-index.js';
+import { openKnowledgeStore } from '../infrastructure/knowledge-store/index.js';
 export const listLinks = async (vaultPath, agentId) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openFileIndex(absoluteVaultPath);
+    const index = openKnowledgeStore(absoluteVaultPath);
     try {
         return await index.listLinks(agentId);
     }
@@ -12,7 +12,7 @@ export const listLinks = async (vaultPath, agentId) => {
 };
 export const listBacklinks = async (vaultPath, title, agentId) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openFileIndex(absoluteVaultPath);
+    const index = openKnowledgeStore(absoluteVaultPath);
     try {
         return await index.listBacklinks(title, agentId);
     }

package/dist/application/ports/knowledge-store.js

@@ -0,0 +1 @@
+export {};

package/dist/application/provision-vault.js

@@ -0,0 +1,74 @@
+// Provision a private GitHub repository for the vault using the already
+// authenticated `gh` CLI, wire it as origin, push the first snapshot, and enable
+// auto-versioning so later changes are committed and pushed automatically. Only
+// the canonical Markdown is versioned; the derived index stays git-ignored.
+import { execFile } from 'node:child_process';
+import { promisify } from 'node:util';
+import { loadRawConfig, writeRawConfig } from '../infrastructure/config.js';
+import { commitVault, hasRemote, initVaultGit, pushVault } from './vault-git-core.js';
+const execFileAsync = promisify(execFile);
+export class GhUnavailableError extends Error {
+    constructor() {
+        super('gh executable not found. Install and authenticate the GitHub CLI (gh auth login) to provision a vault repo.');
+        this.name = 'GhUnavailableError';
+    }
+}
+export class GhCommandError extends Error {
+    constructor(args, stderr) {
+        super(`gh ${args.join(' ')} failed: ${stderr.trim() || 'unknown error'}`);
+        this.name = 'GhCommandError';
+    }
+}
+const isMissingBinary = (error) => error instanceof Error && 'code' in error && error.code === 'ENOENT';
+const runGh = async (cwd, args) => {
+    try {
+        const { stdout } = await execFileAsync('gh', [...args], { cwd, maxBuffer: 16 * 1024 * 1024 });
+        return stdout;
+    }
+    catch (error) {
+        if (isMissingBinary(error)) {
+            throw new GhUnavailableError();
+        }
+        const stderr = error.stderr ?? (error instanceof Error ? error.message : String(error));
+        throw new GhCommandError(args, stderr);
+    }
+};
+// Fail early with an actionable message if gh is not authenticated, rather than
+// letting `repo create` fail deep in the flow.
+const assertGhAuthenticated = async (cwd) => {
+    await runGh(cwd, ['auth', 'status']);
+};
+// Persist config toggles to the global config. Global scope matches the common
+// case of a single default vault; a project-local vault can set these in its own
+// config. Written BEFORE the first commit so the initial snapshot is encrypted
+// and the gitignore is correct from the start.
+const mergeGlobalConfig = async (patch) => {
+    const current = await loadRawConfig('global');
+    await writeRawConfig('global', { ...current, ...patch });
+};
+export const provisionVaultRepo = async (vaultPath, options = {}) => {
+    const name = options.name ?? 'brainlink-vault';
+    const enableAutoVersion = options.enableAutoVersion !== false;
+    const encrypt = options.encrypt !== false;
+    // Toggle config first so initVaultGit/commitVault encrypt from the very first
+    // snapshot and git-ignore the plaintext Markdown.
+    await mergeGlobalConfig({
+        ...(encrypt ? { vaultEncryption: true } : {}),
+        ...(enableAutoVersion ? { autoVersion: true } : {})
+    });
+    const { root } = await initVaultGit(vaultPath);
+    const committed = await commitVault(vaultPath, options.message ?? 'chore(vault): initial snapshot');
+    // A pre-existing origin means the repo is already provisioned: just push the
+    // current state. Only creating a new repository needs gh, so the reuse path
+    // works without it.
+    const created = !(await hasRemote(root));
+    if (created) {
+        await assertGhAuthenticated(root);
+        const visibility = options.private === false ? '--public' : '--private';
+        await runGh(root, ['repo', 'create', name, visibility, '--source', root, '--remote', 'origin', '--push']);
+    }
+    else {
+        await pushVault(vaultPath);
+    }
+    return { repo: name, created, committed, pushed: true, autoVersionEnabled: enableAutoVersion, encrypted: encrypt };
+};

package/dist/application/ranking/run-search.js

@@ -0,0 +1,212 @@
+// Backend-agnostic ranking. Given a candidate set (chunks already selected by a
+// storage backend), this module owns the deterministic scoring math: field
+// weighting, BM25 lexical relevance, cosine semantic relevance, hybrid fusion,
+// and stable ordering. Keeping it here means every KnowledgeStore backend
+// (JSON, binary, LMDB) produces an identical ranking for the same candidates.
+import { dotProduct } from '../../domain/embeddings.js';
+import { bm25TermScore, minMaxNormalize } from '../../domain/scoring.js';
+const queryTokenPattern = /[\p{L}\p{N}_-]+/gu;
+export const normalizeToken = (value) => value
+    .normalize('NFKD')
+    .replace(/\p{Diacritic}/gu, '')
+    .toLowerCase();
+export const tokenize = (query) => query
+    .match(queryTokenPattern)
+    ?.map(normalizeToken)
+    .filter((token) => token.length > 1) ?? [];
+const countOccurrences = (text, token) => {
+    let hits = 0;
+    let cursor = 0;
+    while (cursor < text.length) {
+        const index = text.indexOf(token, cursor);
+        if (index < 0) {
+            break;
+        }
+        hits += 1;
+        cursor = index + token.length;
+    }
+    return hits;
+};
+const titleFieldWeight = 5;
+const tagFieldWeight = 4;
+const pathFieldWeight = 2;
+const contentHitCap = 6;
+const hybridTextWeight = 0.6;
+const hybridSemanticWeight = 0.4;
+// Rank-fusion damping constant (Cormack et al.). 60 is the field-standard value:
+// large enough that the top ranks are not wildly separated, small enough that a
+// document ranked highly by one retriever still clearly outweighs low ranks.
+const rrfConstant = 60;
+const normalizeFields = (row) => ({
+    title: normalizeToken(row.title),
+    path: normalizeToken(row.path),
+    content: normalizeToken(row.content),
+    tags: normalizeToken(row.tags.join(' '))
+});
+// Field-weighted term frequency: title/tag/path matches count more than body
+// matches, and body hits saturate at a cap so a term repeated many times in one
+// chunk cannot dominate. This weighted frequency feeds BM25 below.
+const weightedTermFrequency = (fields, token) => {
+    const titleHits = countOccurrences(fields.title, token);
+    const tagHits = countOccurrences(fields.tags, token);
+    const pathHits = countOccurrences(fields.path, token);
+    const contentHits = countOccurrences(fields.content, token);
+    return (titleHits * titleFieldWeight +
+        tagHits * tagFieldWeight +
+        pathHits * pathFieldWeight +
+        Math.min(contentHits, contentHitCap));
+};
+// Naive additive field score retained for graph-node title/path lookup, where
+// the candidate set is documents (not chunks) and corpus-wide BM25 statistics
+// add no signal over a direct weighted hit count.
+export const lexicalFieldScore = (row, tokens) => {
+    if (tokens.length === 0) {
+        return 0;
+    }
+    const fields = normalizeFields(row);
+    return tokens.reduce((score, token) => score + weightedTermFrequency(fields, token), 0);
+};
+// A single chunk with a non-finite cached tokenCount (legacy index, partial
+// write, manual edit) would otherwise poison the shared averageLength and zero
+// out lexical scoring for the whole corpus, so coerce to a safe positive length.
+const safeDocumentLength = (tokenCount) => Number.isFinite(tokenCount) && tokenCount > 0 ? tokenCount : 1;
+const buildCorpusStatistics = (prepared, tokens) => {
+    const documentCount = prepared.length;
+    const totalLength = prepared.reduce((sum, entry) => sum + safeDocumentLength(entry.row.tokenCount), 0);
+    const documentFrequencyByTerm = new Map();
+    for (const term of tokens) {
+        let documentFrequency = 0;
+        for (const entry of prepared) {
+            if ((entry.weightedTermFrequencies.get(term) ?? 0) > 0) {
+                documentFrequency += 1;
+            }
+        }
+        documentFrequencyByTerm.set(term, documentFrequency);
+    }
+    return {
+        documentCount,
+        averageLength: documentCount > 0 ? totalLength / documentCount : 0,
+        documentFrequencyByTerm
+    };
+};
+const bm25TextScore = (entry, tokens, stats) => tokens.reduce((score, term) => score +
+    bm25TermScore({
+        termFrequency: entry.weightedTermFrequencies.get(term) ?? 0,
+        documentLength: safeDocumentLength(entry.row.tokenCount),
+        documentCount: stats.documentCount,
+        documentFrequency: stats.documentFrequencyByTerm.get(term) ?? 0,
+        averageLength: stats.averageLength
+    }), 0);
+const semanticScore = (row, queryEmbedding) => queryEmbedding.length > 0 && row.embedding.length > 0 ? dotProduct(queryEmbedding, row.embedding) : 0;
+const rangeOf = (values) => values.reduce((range, value) => ({ min: Math.min(range.min, value), max: Math.max(range.max, value) }), { min: Infinity, max: -Infinity });
+const toResult = (row, mode, score, text, semantic) => ({
+    documentId: row.documentId,
+    agentId: row.agentId,
+    title: row.title,
+    path: row.path,
+    chunkId: row.chunkId,
+    chunkOrdinal: row.chunkOrdinal,
+    content: row.content,
+    score,
+    textScore: text,
+    semanticScore: semantic,
+    tokenCount: row.tokenCount,
+    searchMode: mode,
+    tags: row.tags
+});
+// Reciprocal Rank Fusion: rank the candidates independently by each component
+// (lexical, semantic) and fuse by rank position, `Σ 1/(k + rank)`. Because it
+// consumes ranks rather than raw magnitudes it is immune to the scale mismatch
+// between unbounded BM25 and bounded cosine, so a strong lexical hit and a
+// strong semantic hit contribute comparably without hand-tuned weights. A
+// candidate absent from a component's ranking (zero score there) simply
+// contributes nothing for that component. Ties break by title to stay
+// deterministic, matching the final ordering.
+const reciprocalRankFusion = (entries) => {
+    const ranksByField = (component) => {
+        const ranked = entries
+            .filter((entry) => component(entry) > 0)
+            .slice()
+            .sort((left, right) => component(right) - component(left) || left.row.title.localeCompare(right.row.title));
+        return new Map(ranked.map((entry, index) => [entry, index + 1]));
+    };
+    const textRanks = ranksByField((entry) => entry.text);
+    const semanticRanks = ranksByField((entry) => entry.semantic);
+    return new Map(entries.map((entry) => {
+        const textRank = textRanks.get(entry);
+        const semanticRank = semanticRanks.get(entry);
+        const score = (textRank ? 1 / (rrfConstant + textRank) : 0) + (semanticRank ? 1 / (rrfConstant + semanticRank) : 0);
+        return [entry, score];
+    }));
+};
+// Rank an already-selected candidate set. Backends own candidate selection
+// (full scan, bucket prefilter, ANN); this function owns scoring and ordering so
+// the result is identical regardless of how the candidates were gathered.
+export const runSearch = (candidates, query, limit, mode, queryEmbedding = [], fusion = 'rrf') => {
+    const tokens = tokenize(query);
+    // Pass 1: field-weighted term frequencies per candidate, then corpus-wide
+    // statistics (document frequency, average length) needed for BM25 IDF and
+    // length normalization.
+    const prepared = candidates.map((row) => {
+        const fields = normalizeFields(row);
+        const weightedTermFrequencies = new Map();
+        for (const term of tokens) {
+            const frequency = weightedTermFrequency(fields, term);
+            if (frequency > 0) {
+                weightedTermFrequencies.set(term, frequency);
+            }
+        }
+        return { row, weightedTermFrequencies };
+    });
+    const stats = buildCorpusStatistics(prepared, tokens);
+    // Pass 2: raw lexical (BM25) and semantic (cosine) component scores.
+    const componentScored = prepared.map((entry) => ({
+        row: entry.row,
+        text: tokens.length > 0 ? bm25TextScore(entry, tokens, stats) : 0,
+        semantic: semanticScore(entry.row, queryEmbedding)
+    }));
+    // Filter on raw relevance before normalization so a legitimate match that
+    // happens to be the weakest in its mode is never min-max scaled to zero
+    // and dropped. fts stays lexical-only and semantic stays vector-only.
+    const relevant = componentScored.filter((entry) => {
+        if (tokens.length === 0) {
+            return true;
+        }
+        if (mode === 'fts') {
+            return entry.text > 0;
+        }
+        if (mode === 'semantic') {
+            return entry.semantic > 0;
+        }
+        return entry.text > 0 || entry.semantic > 0;
+    });
+    if (relevant.length === 0) {
+        return [];
+    }
+    const textRange = rangeOf(relevant.map((entry) => entry.text));
+    const semanticRange = rangeOf(relevant.map((entry) => entry.semantic));
+    // Rank-fused hybrid scores are precomputed once over the whole candidate set
+    // (RRF needs each component's global ranking); other modes score per entry.
+    const rrfScores = mode === 'hybrid' && fusion === 'rrf' ? reciprocalRankFusion(relevant) : null;
+    const finalScore = (entry) => {
+        if (mode === 'fts') {
+            return entry.text;
+        }
+        if (mode === 'semantic') {
+            return entry.semantic;
+        }
+        if (rrfScores) {
+            return rrfScores.get(entry) ?? 0;
+        }
+        // Linear hybrid mixes the two components on a shared [0, 1] scale so neither
+        // the unbounded BM25 magnitude nor the bounded cosine value can swamp the
+        // other, replacing the previous fixed `text + semantic * 8` blend.
+        const textNorm = minMaxNormalize(entry.text, textRange.min, textRange.max);
+        const semanticNorm = minMaxNormalize(entry.semantic, semanticRange.min, semanticRange.max);
+        return textNorm * hybridTextWeight + semanticNorm * hybridSemanticWeight;
+    };
+    return relevant
+        .map((entry) => toResult(entry.row, mode, finalScore(entry), entry.text, entry.semantic))
+        .sort((left, right) => right.score - left.score || left.title.localeCompare(right.title))
+        .slice(0, Math.max(0, limit));
+};