@andespindola/brainlink 0.1.0-beta.13 → 0.1.0-beta.15

package/AGENTS.md

@@ -6,19 +6,19 @@ This file tells coding agents and AI assistants how to use this repository.
 
 Brainlink is a local-first knowledge memory for agents.
 
-It reads a Markdown vault, extracts `[[wiki links]]` and `#tags`, builds a local SQLite full-text index, and returns compact context packages that agents can inject into prompts.
+It reads a Markdown vault, extracts `[[wiki links]]` and `#tags`, builds a local file index at `.brainlink/index.json`, and returns compact context packages that agents can inject into prompts.
 
 ## Source Of Truth
 
 Markdown files are the source of truth.
 
-The SQLite database at `.brainlink/brainlink.db` is a derived index. It can be deleted and rebuilt with:
+The JSON index at `.brainlink/index.json` is derived. It can be deleted and rebuilt with:
 
 ```bash
 npm run dev -- index --vault ./vault
 ```
 
-Do not store permanent knowledge only in SQLite.
+Do not store permanent knowledge only in index artifacts.
 
 By default, the installed Brainlink CLI uses `$HOME/.brainlink/vault` as its vault. Passing `--vault` or setting `vault` in `brainlink.config.json` intentionally selects a custom vault such as `./vault`.
 
@@ -107,10 +107,10 @@ npm run dev -- doctor --vault ./vault
 
 - Keep domain rules in `src/domain`.
 - Keep use cases in `src/application`.
-- Keep filesystem and SQLite details in `src/infrastructure`.
+- Keep filesystem and index details in `src/infrastructure`.
 - Keep CLI concerns in `src/cli`.
 - Prefer pure functions for parsing, ranking, formatting, and transformation.
-- Do not make SQLite the canonical storage layer.
+- Do not make index artifacts the canonical storage layer.
 - Do not add comments with emojis.
 - Keep JSON output backwards compatible where possible.
 

package/CHANGELOG.md

@@ -49,8 +49,8 @@
 ## 0.1.0-alpha.0
 
 - Added local-first Markdown vault indexing.
-- Added SQLite FTS, local semantic retrieval, wiki links, backlinks and graph retrieval.
-- Added SQLite semantic bucket indexing to narrow vector candidates for larger vaults.
+- Added local full-text indexing, local semantic retrieval, wiki links, backlinks and graph retrieval.
+- Added semantic candidate bucket indexing to narrow vector candidates for larger vaults.
 - Optimized title/link resolution with precomputed agent-scoped title maps.
 - Added CLI, JSON output, HTTP API and graph UI.
 - Added vault diagnostics: stats, broken links, orphans, validation and doctor.

package/CONTRIBUTING.md

@@ -22,7 +22,7 @@ npm run pack:smoke
 ## Design Rules
 
 - Markdown files are the source of truth.
-- SQLite is a derived index and must remain rebuildable.
+- Local index artifacts are derived and must remain rebuildable.
 - Domain parsing, graph analysis and layout should stay pure and testable.
-- CLI, HTTP, filesystem and SQLite code are adapters around application use cases.
+- CLI, HTTP, filesystem and index code are adapters around application use cases.
 - MCP integration should live outside this package by wrapping the CLI with `--json`.

package/README.md

@@ -52,14 +52,14 @@ LLMs do not have infinite context. Brainlink gives agents an external memory lay
 1. Durable knowledge is written as Markdown.
 2. Notes are connected with `[[wiki links]]`.
 3. Concepts are classified with `#tags`.
-4. Brainlink builds a local SQLite index with FTS records and local embeddings.
+4. Brainlink builds a local JSON index (`.brainlink/index.json`) and private encrypted search packs.
 5. Agents query the index before responding.
 6. Brainlink returns compact, source-backed context.
 
-Markdown is the source of truth. `.brainlink/brainlink.db` is only a rebuildable index.
-Brainlink now keeps an automatic rollback snapshot at `.brainlink/brainlink.db.backup` plus rotating snapshots in `.brainlink/brainlink.db.backup.snapshots/`. If the main SQLite file is corrupted, Brainlink automatically restores the newest valid snapshot (or recreates a clean index when no snapshot exists).
-After each index run, Brainlink also writes private encrypted search packs at `.brainlink/search-packs/*.blpk`. If SQLite is unavailable, search falls back to these packs automatically.
+Markdown is the source of truth. `.brainlink/index.json` is a rebuildable index artifact.
+After each index run, Brainlink also writes private encrypted search packs at `.brainlink/search-packs/*.blpk` to preserve fast retrieval and portable recovery.
 Pack decryption uses a Brainlink key from `$BRAINLINK_HOME/keys` or from `BRAINLINK_SEARCH_PACK_KEY` when explicitly configured.
+Legacy `.jsonl.gz` packs are upgraded to `.blpk` automatically on first search/context access.
 
 ## Features
 
@@ -68,7 +68,7 @@ Pack decryption uses a Brainlink key from `$BRAINLINK_HOME/keys` or from `BRAINL
 - Weighted graph edges so agents can rank relationship importance and priority.
 - Backlinks, broken-link reports, orphan detection and validation.
 - Full-text, semantic and hybrid retrieval modes.
-- SQLite-backed semantic candidate buckets for larger vaults.
+- Full-text, semantic and hybrid retrieval on a local file index.
 - Agent namespaces under `agents/<agent-id>/`.
 - S3-compatible bucket vaults through `s3://bucket/prefix` URIs.
 - CLI with machine-readable `--json` output.
@@ -285,7 +285,7 @@ export BRAINLINK_S3_FORCE_PATH_STYLE=1
 
 Bucket vaults mirror Markdown into a local cache under
 `$BRAINLINK_HOME/bucket-cache`. The bucket remains canonical; the local
-`.brainlink/brainlink.db` stays a disposable index. Run `index` after remote
+`.brainlink/index.json` stays a disposable index artifact. Run `index` after remote
 bucket changes before relying on `search`, `context`, graph or validation
 commands. Watch mode is only supported for local filesystem vaults.
 
@@ -301,7 +301,7 @@ vault/
     research-agent/
       source-review-policy.md
   .brainlink/
-    brainlink.db
+    index.json
 ```
 
 Permanent data:
@@ -311,7 +311,7 @@ Permanent data:
 
 Rebuildable data:
 
-- `.brainlink/brainlink.db`
+- `.brainlink/index.json`
 - full-text records
 - local embedding vectors
 - local embedding buckets
@@ -723,8 +723,8 @@ If `--mode` or `--limit` is omitted, Brainlink resolves values from the current
 
 Modes:
 
-- `hybrid`: default; combines SQLite FTS with local embedding similarity.
-- `fts`: exact lexical retrieval through SQLite FTS.
+- `hybrid`: default; combines lexical matching with local embedding similarity.
+- `fts`: exact lexical retrieval from the file index.
 - `semantic`: local deterministic embedding similarity only.
 
 Hybrid results are cached in-memory for a short TTL and invalidated automatically when the local index file changes.
@@ -976,7 +976,7 @@ src/
   application/      use cases
   cli/              command-line adapter
   domain/           pure knowledge rules
-  infrastructure/   filesystem and SQLite adapters
+  infrastructure/   filesystem and index adapters
 ```
 
 Detailed notes:
@@ -988,7 +988,6 @@ Detailed notes:
 ## Current Limits
 
 - Semantic search uses deterministic local embeddings, not a remote model provider.
-- Semantic search uses SQLite embedding buckets to narrow candidates before cosine scoring.
 - `embeddingProvider` currently supports `local` and `none`.
 - Link resolution is title-based inside each agent namespace, with `shared` as fallback.
 - HTTP API is local and unauthenticated.
@@ -999,7 +998,7 @@ Detailed notes:
 The `0.1.0-beta` line is intended to stabilize the local-first memory loop:
 
 - Markdown as durable memory.
-- SQLite FTS plus local embeddings and semantic buckets as rebuildable retrieval index.
+- Rebuildable file index plus local embeddings and encrypted pack exports.
 - CLI as the primary agent interface.
 - HTTP graph API and frontend as inspection tools.
 - Agent namespaces to avoid context mixing.
@@ -1015,7 +1014,7 @@ Brainlink is local-first by default.
 - Brainlink HTTP is localhost-only and refuses non-loopback hosts.
 - Brainlink blocks common secret patterns by default when adding notes. Use `--allow-sensitive` only for intentional, protected vaults.
 - Do not store secrets, credentials, API keys or regulated personal data unless the vault is protected by your own storage controls.
-- Treat `.brainlink/brainlink.db` as disposable derived data.
+- Treat `.brainlink/index.json` and `.brainlink/search-packs/` as disposable derived artifacts.
 
 See [SECURITY.md](SECURITY.md).
 

package/SECURITY.md

@@ -7,7 +7,7 @@ Brainlink is local-first.
 - The HTTP server binds to `127.0.0.1` by default.
 - The HTTP server always refuses non-loopback hosts.
 - The HTTP server is read-only and does not expose note creation, indexing or update routes.
-- The SQLite database is a derived local index.
+- Local index artifacts (`.brainlink/index.json` and `.brainlink/search-packs/`) are derived data.
 - Markdown files are user-owned source data.
 - Brainlink-created Markdown files use `0600` permissions.
 - Brainlink-created directories and `.brainlink` use `0700` permissions.

package/dist/application/analyze-vault.js

@@ -1,7 +1,5 @@
 import { stat } from 'node:fs/promises';
-import { existsSync, readdirSync } from 'node:fs';
 import { performance } from 'node:perf_hooks';
-import { join } from 'node:path';
 import { validateGraph, getBrokenLinks, getOrphanNodes, getVaultStats } from '../domain/graph-analysis.js';
 import { ensureVault, listVaultFiles, readMarkdownFiles } from '../infrastructure/file-system-vault.js';
 import { resolveAgentRuntimeDefaults } from '../infrastructure/config.js';
@@ -96,23 +94,11 @@ export const doctorVault = async (vaultPath) => {
     const files = await readMarkdownFiles(absoluteVaultPath);
     const graph = await getGraphSummary(absoluteVaultPath);
     const validation = validateGraph(graph);
-    const backupPath = join(absoluteVaultPath, '.brainlink', 'brainlink.db.backup');
-    const snapshotDirectory = join(absoluteVaultPath, '.brainlink', 'brainlink.db.backup.snapshots');
-    const hasBackup = existsSync(backupPath);
-    const snapshotCount = existsSync(snapshotDirectory)
-        ? readdirSync(snapshotDirectory).filter((name) => name.endsWith('.db')).length
-        : 0;
-    const backupReady = graph.nodes.length === 0 || hasBackup;
     const checks = [
         createCheck('vault', true, `Vault ready at ${absoluteVaultPath}`),
         createCheck('markdown-files', files.length > 0, `${files.length} markdown files found`),
         createCheck('index', graph.nodes.length > 0, `${graph.nodes.length} indexed documents found`),
-        createCheck('broken-links', validation.brokenLinks.length === 0, `${validation.brokenLinks.length} broken links found`),
-        createCheck('index-backup', backupReady, backupReady
-            ? (hasBackup
-                ? `SQLite recovery snapshot is available (${snapshotCount} rotating snapshots)`
-                : 'No index yet. Snapshot will be created after first indexing run')
-            : 'Recovery snapshot missing. Run blink index to create a rollback snapshot')
+        createCheck('broken-links', validation.brokenLinks.length === 0, `${validation.brokenLinks.length} broken links found`)
     ];
     const recommendations = files.length === 0 && graph.nodes.length === 0
         ? [

package/dist/application/get-graph-layout.js

@@ -1,12 +1,12 @@
 import { createHash } from 'node:crypto';
 import { stat } from 'node:fs/promises';
-import { join } from 'node:path';
 import { createCauliflowerGraphLayout } from '../domain/graph-layout.js';
+import { indexStoragePath } from '../infrastructure/file-index.js';
 import { getGraphSummary } from './get-graph-summary.js';
 const graphLayoutCache = new Map();
 const readDatabaseSignature = async (vaultPath) => {
     try {
-        const info = await stat(join(vaultPath, '.brainlink', 'brainlink.db'));
+        const info = await stat(indexStoragePath(vaultPath));
         return `${Math.floor(info.mtimeMs)}:${info.size}`;
     }
     catch {

package/dist/application/get-graph-node.js

@@ -1,10 +1,10 @@
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { openSqliteIndex } from '../infrastructure/sqlite-index.js';
+import { openFileIndex } from '../infrastructure/file-index.js';
 export const getGraphNode = async (vaultPath, id, agentId) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openSqliteIndex(absoluteVaultPath);
+    const index = openFileIndex(absoluteVaultPath);
     try {
-        return index.getGraphNode(id, agentId);
+        return await index.getGraphNode(id, agentId);
     }
     finally {
         index.close();

package/dist/application/get-graph-summary.js

@@ -1,10 +1,10 @@
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { openSqliteIndex } from '../infrastructure/sqlite-index.js';
+import { openFileIndex } from '../infrastructure/file-index.js';
 export const getGraphSummary = async (vaultPath, agentId) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openSqliteIndex(absoluteVaultPath);
+    const index = openFileIndex(absoluteVaultPath);
     try {
-        return index.getGraphSummary(agentId);
+        return await index.getGraphSummary(agentId);
     }
     finally {
         index.close();

package/dist/application/get-graph.js

@@ -1,10 +1,10 @@
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { openSqliteIndex } from '../infrastructure/sqlite-index.js';
+import { openFileIndex } from '../infrastructure/file-index.js';
 export const getGraph = async (vaultPath, agentId) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openSqliteIndex(absoluteVaultPath);
+    const index = openFileIndex(absoluteVaultPath);
     try {
-        return index.getGraph(agentId);
+        return await index.getGraph(agentId);
     }
     finally {
         index.close();

package/dist/application/index-vault.js

@@ -4,7 +4,7 @@ import { createEmbeddingProvider } from '../domain/embeddings.js';
 import { loadBrainlinkConfig } from '../infrastructure/config.js';
 import { ensureVault, readMarkdownFiles } from '../infrastructure/file-system-vault.js';
 import { buildSearchPacks } from '../infrastructure/search-packs.js';
-import { openSqliteIndex } from '../infrastructure/sqlite-index.js';
+import { openFileIndex } from '../infrastructure/file-index.js';
 const toTitleKey = (title) => title.toLowerCase();
 const appendTitleEntry = (map, document) => {
     const key = toTitleKey(document.title);
@@ -60,15 +60,15 @@ export const indexVault = async (vaultPath) => {
     }));
     const titleMaps = createTitleMaps(documents);
     const indexedDocuments = await embedIndexedDocuments(documents.map((document) => createIndexedDocument(document, createScopedTitleResolver(document, titleMaps), config.chunkSize)), config.embeddingProvider);
-    const index = openSqliteIndex(absoluteVaultPath);
+    const index = openFileIndex(absoluteVaultPath);
     try {
-        index.reset();
-        index.saveDocuments(indexedDocuments);
+        await index.reset();
+        await index.saveDocuments(indexedDocuments);
         try {
             await buildSearchPacks(absoluteVaultPath, indexedDocuments);
         }
         catch {
-            // Pack generation is best-effort. SQLite index remains the primary path.
+            // Pack generation is best-effort. The JSON index remains the primary path.
         }
         return {
             documentCount: indexedDocuments.length,

package/dist/application/list-agents.js

@@ -1,10 +1,10 @@
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { openSqliteIndex } from '../infrastructure/sqlite-index.js';
+import { openFileIndex } from '../infrastructure/file-index.js';
 export const listAgents = async (vaultPath) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openSqliteIndex(absoluteVaultPath);
+    const index = openFileIndex(absoluteVaultPath);
     try {
-        return index.listAgents();
+        return await index.listAgents();
     }
     finally {
         index.close();

package/dist/application/list-links.js

@@ -1,10 +1,10 @@
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { openSqliteIndex } from '../infrastructure/sqlite-index.js';
+import { openFileIndex } from '../infrastructure/file-index.js';
 export const listLinks = async (vaultPath, agentId) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openSqliteIndex(absoluteVaultPath);
+    const index = openFileIndex(absoluteVaultPath);
     try {
-        return index.listLinks(agentId);
+        return await index.listLinks(agentId);
     }
     finally {
         index.close();
@@ -12,9 +12,9 @@ export const listLinks = async (vaultPath, agentId) => {
 };
 export const listBacklinks = async (vaultPath, title, agentId) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openSqliteIndex(absoluteVaultPath);
+    const index = openFileIndex(absoluteVaultPath);
     try {
-        return index.listBacklinks(title, agentId);
+        return await index.listBacklinks(title, agentId);
     }
     finally {
         index.close();

package/dist/application/search-graph-node-ids.js

@@ -1,10 +1,10 @@
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { openSqliteIndex } from '../infrastructure/sqlite-index.js';
+import { openFileIndex } from '../infrastructure/file-index.js';
 export const searchGraphNodeIds = async (vaultPath, query, limit, agentId) => {
     const absoluteVaultPath = await ensureVault(vaultPath);
-    const index = openSqliteIndex(absoluteVaultPath);
+    const index = openFileIndex(absoluteVaultPath);
     try {
-        return index.searchGraphNodeIds(query, limit, agentId);
+        return await index.searchGraphNodeIds(query, limit, agentId);
     }
     finally {
         index.close();

package/dist/application/search-knowledge.js

@@ -1,8 +1,7 @@
 import { stat } from 'node:fs/promises';
-import { join } from 'node:path';
 import { ensureVault } from '../infrastructure/file-system-vault.js';
-import { searchInPacks } from '../infrastructure/search-packs.js';
-import { openSqliteIndex } from '../infrastructure/sqlite-index.js';
+import { ensurePrivatePacksFromLegacyIndex, searchInPacks } from '../infrastructure/search-packs.js';
+import { indexStoragePath, openFileIndex } from '../infrastructure/file-index.js';
 import { createEmbeddingProvider } from '../domain/embeddings.js';
 import { loadBrainlinkConfig, sanitizeSearchMode } from '../infrastructure/config.js';
 const hybridCacheTtlMs = 30_000;
@@ -10,7 +9,7 @@ const hybridCacheMaxEntries = 200;
 const hybridSearchCache = new Map();
 const readIndexMtimeMs = async (vaultPath) => {
     try {
-        return (await stat(join(vaultPath, '.brainlink', 'brainlink.db'))).mtimeMs;
+        return (await stat(indexStoragePath(vaultPath))).mtimeMs;
     }
     catch {
         return 0;
@@ -47,6 +46,7 @@ export const searchKnowledge = async (vaultPath, query, limit, agentId, mode) =>
     const absoluteVaultPath = await ensureVault(vaultPath);
     const config = await loadBrainlinkConfig();
     const searchMode = sanitizeSearchMode(mode, config.defaultSearchMode);
+    await ensurePrivatePacksFromLegacyIndex(absoluteVaultPath);
     const cacheKey = searchMode === 'hybrid' ? toCacheKey(absoluteVaultPath, query, limit, agentId) : undefined;
     const indexMtimeMs = cacheKey ? await readIndexMtimeMs(absoluteVaultPath) : 0;
     const cached = cacheKey ? cacheGet(cacheKey, indexMtimeMs) : undefined;
@@ -57,9 +57,9 @@ export const searchKnowledge = async (vaultPath, query, limit, agentId, mode) =>
     const shouldEmbedQuery = searchMode !== 'fts' && provider.name !== 'none';
     const queryEmbedding = shouldEmbedQuery ? (await provider.embed([query]))[0] ?? [] : [];
     try {
-        const index = openSqliteIndex(absoluteVaultPath);
+        const index = openFileIndex(absoluteVaultPath);
         try {
-            const results = index.search(query, limit, agentId, searchMode, queryEmbedding);
+            const results = await index.search(query, limit, agentId, searchMode, queryEmbedding);
             if (cacheKey) {
                 cacheSet({
                     key: cacheKey,

package/dist/benchmarks/large-vault.js

@@ -21,7 +21,7 @@ const readOptions = (args) => ({
 });
 const topics = [
     'authentication jwt token refresh policy',
-    'sqlite graph backlinks markdown vault indexing',
+    'graph backlinks markdown vault indexing',
     'frontend canvas layout graph interaction',
     'agent memory context retrieval summarization',
     'security local server vault path allowlist',