@andespindola/brainlink 0.1.0-beta.12 → 0.1.0-beta.120

package/docs/AGENT_USAGE.md

@@ -18,7 +18,7 @@ The correct dependency direction is:
 agent -> Brainlink CLI -> Markdown vault + derived index
 ```
 
-Agents should never depend on the internal SQLite schema as a public API.
+Agents should never depend on internal index persistence files as a public API.
 
 The installed CLI exposes two equivalent binaries:
 
@@ -52,6 +52,8 @@ Use `blink config where` and `blink config doctor` to inspect active paths and e
 
 You can also set `defaultAgent` in `brainlink.config.json` / `.brainlink.json` (for example `"defaultAgent": "coding-agent"`). When set, CLI commands and MCP calls reuse it when `--agent`/`agent` is not passed.
 You can set `agentProfiles` to define per-agent defaults for `defaultSearchMode`, `defaultSearchLimit` and `defaultContextTokens`.
+You can tune search-pack compression with `searchPack.rowChunkSize`, `searchPack.compressionLevel` and `searchPack.useDictionary`.
+Guardrails for benchmark acceptance are configured with `searchPack.guardrailMinSavingsPercent` and `searchPack.guardrailMaxLatencyRegressionPercent`.
 
 `autoIndexOnWrite` (default: `true`) controls whether `add` and MCP write tools index right after writing.
 
@@ -180,16 +182,16 @@ Required write behavior:
 Good linked note:
 
 ```bash
-blink add "SQLite Index Rebuild" \
+blink add "Index Rebuild" \
   --agent coding-agent \
-  --content "Legacy derived indexes without agent columns are rebuilt because SQLite is disposable. Related: [[Architecture]], [[Agent Namespaces]]. #sqlite #architecture #decision"
+  --content "Derived index artifacts are rebuildable and disposable. Related: [[Architecture]], [[Agent Namespaces]]. #index #architecture #decision"
 blink validate --agent coding-agent
 ```
 
 Poor disconnected note:
 
 ```bash
-blink add "SQLite Index Rebuild" \
+blink add "Index Rebuild" \
   --agent coding-agent \
   --content "We rebuild old indexes now."
 ```
@@ -377,6 +379,18 @@ blink migrate-vault --from ~/.brainlink/vault --to ./team-vault --report ./migra
 
 Use `--dry-run` to preview `copied`, `conflicted`, `unchanged` before writing files.
 
+### Import Legacy SQLite DB
+
+```bash
+blink db-import --vault ./team-vault
+blink db-import --vault ./team-vault --db ./legacy/brainlink.db
+blink db-import --vault ./team-vault --db ./legacy/brainlink.db --table legacy_notes --dry-run
+```
+
+`db-import` migrates rows from legacy SQLite memory into Markdown notes in the current vault and indexes the result by default.
+Without `--db`, Brainlink auto-detects common legacy database paths.
+Use `--agent` to force namespace, `--limit` for staged migration, `--dry-run` to preview writes, and `--no-index` to postpone indexing.
+
 ### Install Agent Integration
 
 ```bash
@@ -390,6 +404,7 @@ blink agent status
 ```
 
 `agent install` configures Brainlink MCP in `~/.codex/config.toml` so compatible agents can use Brainlink by default.
+`agent install` and `agent upgrade` automatically apply the `fully-auto` MCP bootstrap policy (`enforceBootstrap=true`, `enforceContextFirst=true`, `autoBootstrapOnRead=true`, `autoBootstrapOnStartup=true`) so all plug-and-play Brainlink features start enabled.
 Use `agent upgrade` on legacy installations to reapply the latest defaults and run self-test diagnostics.
 Use `agent policy --preset fully-auto` to keep startup/read auto-bootstrap enabled, or `agent policy --preset strict` to force explicit bootstrap calls.
 
@@ -417,6 +432,25 @@ This creates a slugged Markdown file with frontmatter and a heading.
 
 The CLI blocks common secret patterns by default. Do not use `--allow-sensitive` unless the vault is intentionally protected.
 Brainlink also auto-connects notes that have no `[[wiki links]]` by adding a fallback edge to an agent hub note, so new memory does not stay disconnected.
+`add` also returns `possibleDuplicates` (exact hash + semantic candidates) so agents can decide duplicate resolution immediately.
+
+### Detect Duplicate Notes
+
+```bash
+blink dedupe --vault ./vault --json
+blink dedupe --vault ./vault --agent coding-agent --limit 20 --min-score 0.92 --json
+blink dedupe --vault ./vault --no-semantic --json
+```
+
+### Resolve Duplicate Notes
+
+```bash
+blink dedupe-resolve --vault ./vault --left agents/shared/a.md --right agents/shared/b.md --action merge --json
+blink dedupe-resolve --vault ./vault --left agents/shared/a.md --right agents/shared/b.md --action link --json
+blink dedupe-resolve --vault ./vault --left agents/shared/a.md --right agents/shared/b.md --action ignore --json
+```
+
+`dedupe-resolve` keeps connectivity: non-merge actions still create a low-priority related edge (`#related-to`).
 
 For agent-private memory:
 
@@ -446,6 +480,37 @@ This scans Markdown files and rebuilds:
 - links
 - full-text search records
 
+### Benchmark Indexing Realtime
+
+```bash
+blink bench --vault ./vault
+blink bench --vault ./vault --watch
+blink bench --vault ./vault --watch --debounce 500
+blink bench --vault ./vault --json
+```
+
+`bench` runs indexing with realtime phase events and prints a run summary with:
+
+- indexed totals (documents, chunks, links)
+- elapsed time and changed document count
+- pack rebuild status and reason
+- pack compression metrics (`inputBytes`, `outputBytes`, ratio/saved percentage)
+- objective guardrails (`guardrailMinSavingsPercent`, `guardrailMaxLatencyRegressionPercent`)
+
+Use `--watch` for continuous benchmark runs while editing notes. Watch mode is supported only for local filesystem vaults.
+If pack manifest metadata is missing but encrypted `.blpk` files are present, Brainlink repairs manifest metadata before deciding rebuild policy to avoid unnecessary full repacks on small updates.
+
+### Create Offline Pack Backup
+
+```bash
+blink pack-backup --vault ./vault
+blink pack-backup --vault ./vault --output ./vault/.brainlink/backups/custom.blpkbak.gz
+blink pack-backup --vault ./vault --json
+```
+
+`pack-backup` creates an offline artifact with second-stage compression on top of encrypted `.blpk` packs.
+This is outside the online retrieval path (`index`, `search`, `context`), which keeps a single compression stage.
+
 ### Search Knowledge
 
 ```bash
@@ -460,11 +525,12 @@ If `--mode`/`--limit` are omitted, Brainlink resolves those values from the acti
 
 Search modes:
 
-- `hybrid`: default; combines SQLite FTS and local embedding similarity.
-- `fts`: lexical SQLite full-text search only.
-- `semantic`: local deterministic embedding similarity with SQLite bucket candidate narrowing.
+- `hybrid`: default; combines lexical matching and local embedding similarity.
+- `fts`: lexical full-text matching only.
+- `semantic`: local deterministic embedding similarity.
 
-Hybrid results are cached in-memory for a short TTL and invalidated when `.brainlink/brainlink.db` changes.
+Hybrid results are cached in-memory for a short TTL and invalidated when `.brainlink/index.json` changes.
+Context assembly uses middle-out ordering inside each note: the highest-scoring chunk is selected first, then nearby chunks are expanded while token budget allows.
 
 ### Build Agent Context
 
@@ -523,15 +589,26 @@ shared: 30 documents
 ```bash
 blink server --host 127.0.0.1 --port 4321
 blink server --vault ./vault --host 127.0.0.1 --port 4321
+blink server --vault ./vault --host 127.0.0.1 --port 4321 --no-open
 ```
 
 This starts a local frontend for inspecting the knowledge graph.
+By default it tries to open the graph in a native desktop GUI window:
+- macOS: Swift + WebKit
+- Windows: PowerShell WinForms WebBrowser
+- Linux: optional Python GTK + WebKit2 (requires `python3` + `gi` + `WebKit2`)
+
+On Linux, native GUI is disabled by default for better startup performance. Enable it with `BRAINLINK_LINUX_NATIVE_GUI=1`.
+If native GUI launch is unavailable, it falls back to dedicated app-window mode and then to the default browser.
+Use `--no-open` to keep the server headless.
+When native GUI is active, the GUI window closes automatically when the `blink server` process stops.
 
 Without `--vault`, the graph UI serves `$HOME/.brainlink/vault`.
 
-The frontend includes an agent selector. Selecting an agent calls the same read APIs with `agent=<agent-id>` and renders that namespace instead of merging every agent into one graph.
+The frontend includes an agent selector that shows only the agent id. Selecting an agent calls the same read APIs with `agent=<agent-id>` and renders that namespace instead of merging every agent into one graph.
 
-Graph navigation controls include zoom in, zoom out, fit visible nodes and reset-to-fit-all nodes. Mouse wheel zoom is anchored to the cursor. Totals for notes, links and tags stay visible as floating metrics under the Brainlink title, and node details open on click in a modal (tags, outgoing links, backlinks and Markdown content).
+Graph navigation controls include zoom in, zoom out, fit visible nodes and reset-to-fit-all nodes. Mouse wheel zoom (including `cmd+scroll` and `ctrl+scroll`) is anchored to the cursor. Keyboard shortcuts are `+` (zoom in), `-` (zoom out) and `0` (reset fit). Double-click on canvas zooms in at cursor position. Totals for notes, links and tags stay visible as floating metrics under the Brainlink title, and node details open on click in a modal (tags, outgoing links, backlinks and Markdown content). Vaults above 1000 notes also expose stable hierarchy groups of up to 1000 direct nodes, with recursive parent groups when a level exceeds 1000 groups; zoom-out renders those groups as normal graph nodes, and zoom-in expands the focused group into its child graph without drawing a background glow layer.
+During graph filtering, Brainlink keeps hub context nodes visible (`Memory Hub`/`MOC`/high-degree fallback) so filtered views still show relationship anchors.
 
 The command reindexes by default, then serves:
 
@@ -589,6 +666,8 @@ Available MCP tools:
 - `brainlink_recommendations`
 - `brainlink_context`
 - `brainlink_search`
+- `brainlink_dedupe`
+- `brainlink_resolve_duplicate`
 - `brainlink_add_note`
 - `brainlink_add_file`
 - `brainlink_index`
@@ -634,8 +713,8 @@ GET  /api/validate
 
 The HTTP API is read-only. Use the CLI for writes and indexing.
 
-Brainlink maintains an automatic SQLite rollback snapshot at `.brainlink/brainlink.db.backup`. When `.brainlink/brainlink.db` is corrupted, Brainlink restores from snapshot automatically or recreates a clean index if no snapshot exists yet.
-Indexing also writes private encrypted search packs at `.brainlink/search-packs/*.blpk`; when SQLite cannot be opened, Brainlink falls back to pack-based search automatically.
+Indexing writes private encrypted search packs at `.brainlink/search-packs/*.blpk` for resilient retrieval and portability.
+Pack search now uses compressed-space prefiltering (token bloom index per pack) before decrypting/reading pack payloads.
 Pack decryption keys are resolved from `$BRAINLINK_HOME/keys` (or `BRAINLINK_SEARCH_PACK_KEY` when explicitly set).
 
 ## Agent Integration Contract
@@ -669,9 +748,9 @@ Non-goals:
 ## Operational Rules
 
 - Re-run `index` after modifying notes.
-- Treat `.brainlink/brainlink.db` as disposable.
-- Commit Markdown notes, not local database files.
-- Do not manually edit the database.
+- Treat `.brainlink/index.json` and `.brainlink/search-packs/` as disposable.
+- Commit Markdown notes, not local index files.
+- Do not manually edit generated index artifacts.
 - Keep generated context short enough for the target model.
 - Prefer specific queries over broad queries.
 - Write explicit `[[wiki links]]` when durable memory should be connected.
@@ -701,9 +780,9 @@ Weak retrieval usually means:
 
 ## Current Limits
 
-- Search supports FTS, local semantic embeddings, SQLite semantic buckets and hybrid ranking.
+- Search supports FTS, local semantic embeddings and hybrid ranking.
 - Local embeddings are deterministic and provider-free; remote embedding providers are not implemented yet.
 - MCP integration is available through the `brainlink-mcp` stdio server.
 - HTTP API is local and unauthenticated.
-- Bucket vaults support S3-compatible `s3://bucket/prefix` URIs and use a local cache for SQLite indexes.
+- Bucket vaults support S3-compatible `s3://bucket/prefix` URIs and use local cache/index artifacts.
 - Watch mode depends on platform filesystem watcher behavior and is only supported for local filesystem vaults.

package/docs/ARCHITECTURE.md

@@ -8,7 +8,7 @@ CLI -> application use cases -> domain functions -> infrastructure adapters
 
 The core rule is simple:
 
-Domain code must not know about the CLI, filesystem, or SQLite.
+Domain code must not know about the CLI, filesystem, or index persistence format.
 
 ## Modules
 
@@ -53,14 +53,11 @@ src/
     types.ts
 
   infrastructure/
-    sqlite/
-      document-writer.ts
-      graph-reader.ts
-      schema.ts
-      search-reader.ts
+    file-index.ts
     file-system-vault.ts
+    private-pack-codec.ts
+    search-packs.ts
     session-state.ts
-    sqlite-index.ts
 
   mcp/
     main.ts
@@ -80,7 +77,6 @@ The domain layer contains pure knowledge rules:
 - extract `#tags`
 - split documents into chunks
 - create deterministic local embeddings
-- create deterministic embedding buckets for semantic candidate retrieval
 - calculate cosine similarity
 - estimate token counts
 - select context sections
@@ -116,12 +112,11 @@ The infrastructure layer handles side effects:
 - mirroring S3-compatible bucket Markdown into a local cache
 - writing Markdown notes
 - creating `.brainlink`
-- writing and querying SQLite
-- running FTS, semantic and hybrid retrieval
-- narrowing semantic candidates through SQLite embedding buckets before cosine scoring
+- writing and querying file-based indexes
+- running lexical, semantic and hybrid retrieval
 
-SQLite is an index, not the canonical storage model. For bucket vaults, Markdown
-objects in the bucket remain canonical and SQLite is still local derived data.
+
+Index artifacts are rebuildable and are not canonical storage. For bucket vaults, Markdown objects in the bucket remain canonical and local index files are derived data.
 
 ## Indexing Flow
 
@@ -132,11 +127,9 @@ read markdown files
   -> resolve links
   -> split chunks
   -> create chunk embeddings
-  -> reset SQLite index
+  -> reset file index
   -> persist documents, chunks and links
-  -> populate FTS records
-  -> persist embedding vectors
-  -> persist embedding buckets
+  -> persist chunks, links and embeddings in file index
 ```
 
 ## Retrieval Flow
@@ -145,8 +138,10 @@ read markdown files
 question
   -> selected mode: fts | semantic | hybrid
   -> optional query embedding
-  -> FTS query and/or embedding bucket candidate lookup
+  -> optional compressed pack prefilter (token bloom)
+  -> lexical scoring and/or semantic cosine scoring
   -> cosine similarity over candidate chunks
+  -> middle-out context expansion around strongest chunk
   -> ranked chunks with textScore and semanticScore
   -> token-budget selection
   -> Markdown context package
@@ -163,7 +158,7 @@ server command
   -> browser renders graph canvas
 ```
 
-The graph UI is intentionally read-only. Markdown remains the write interface and SQLite remains a derived index.
+The graph UI is intentionally read-only. Markdown remains the write interface and index artifacts remain derived data.
 
 ## HTTP API Flow
 
@@ -171,7 +166,7 @@ The graph UI is intentionally read-only. Markdown remains the write interface an
 HTTP request
   -> route handler
   -> application use case
-  -> filesystem and SQLite adapters
+  -> filesystem and index adapters
   -> JSON response
 ```
 
@@ -282,11 +277,10 @@ vault/agents/<agent-id>/**/*.md
 
 Rebuildable:
 
-- `.brainlink/brainlink.db`
+- `.brainlink/index.json`
+- `.brainlink/search-packs/*.blpk`
 - `$BRAINLINK_HOME/bucket-cache`
-- FTS records
 - local embedding vectors
-- local embedding bucket index
 - chunks
 - resolved links
 
@@ -296,13 +290,14 @@ Rebuildable:
 
 Markdown keeps the system portable, inspectable, Git-friendly, and compatible with Obsidian-like workflows.
 
-### SQLite As Local Index
+### File Index As Local Index
 
-SQLite gives fast local search, local vector storage and rebuildable retrieval without forcing users to run external infrastructure.
+Brainlink uses a local JSON index plus encrypted pack exports for fast rebuildable retrieval without external infrastructure.
 Hybrid retrieval also uses a short-lived in-memory cache keyed by vault/query/agent and invalidated by index file mtime to reduce repeated query latency.
-Brainlink also writes a local rollback snapshot (`.brainlink/brainlink.db.backup`) after successful indexing. On corruption detection (`quick_check`/SQLite malformed errors), Brainlink restores from snapshot automatically before reopening the index.
-Indexing additionally exports private encrypted pack files (`.brainlink/search-packs/*.blpk`) from indexed chunks. Search falls back to these packs when SQLite is unavailable, preserving retrieval continuity in degraded mode.
+Indexing exports private encrypted pack files (`.brainlink/search-packs/*.blpk`) from indexed chunks for fast retrieval and recovery continuity.
+Pack manifests include compressed-space token bloom metadata so retrieval can skip unrelated packs before decryption.
 Pack encryption keys are resolved from `$BRAINLINK_HOME/keys` or from `BRAINLINK_SEARCH_PACK_KEY` when configured.
+Legacy `.jsonl.gz` search packs are auto-upgraded to `.blpk` on first retrieval flow.
 
 ### CLI First
 

package/docs/QUICKSTART.md

@@ -102,3 +102,10 @@ S3 target:
 ```bash
 blink migrate-vault --from ~/.brainlink/vault --to "s3://my-memory-bucket/brainlink" --dry-run
 ```
+
+Legacy SQLite import:
+
+```bash
+blink db-import --vault ./team-vault
+blink db-import --vault ./team-vault --db ./legacy/brainlink.db --dry-run
+```

package/package.json

@@ -1,10 +1,10 @@
 {
   "name": "@andespindola/brainlink",
-  "version": "0.1.0-beta.12",
+  "version": "0.1.0-beta.120",
   "description": "Local-first knowledge memory for agents with Markdown, backlinks, indexing and context retrieval.",
   "type": "module",
   "license": "MIT",
-  "author": "Anderson Espindola",
+  "author": "Substructa",
   "homepage": "https://github.com/andersonflima/brainlink#readme",
   "repository": {
     "type": "git",
@@ -32,6 +32,7 @@
     "dist",
     "assets",
     "README.md",
+    "COPYRIGHT.md",
     "LICENSE",
     "CHANGELOG.md",
     "CONTRIBUTING.md",
@@ -58,12 +59,13 @@
   "dependencies": {
     "@aws-sdk/client-s3": "^3.1038.0",
     "@modelcontextprotocol/sdk": "^1.29.0",
-    "better-sqlite3": "^12.9.0",
     "commander": "^14.0.2",
     "zod": "^4.3.6"
   },
+  "overrides": {
+    "qs": "6.15.2"
+  },
   "devDependencies": {
-    "@types/better-sqlite3": "^7.6.13",
     "@types/node": "^24.9.2",
     "tsx": "^4.21.0",
     "typescript": "^5.9.3",

package/dist/infrastructure/sqlite/document-writer.js

@@ -1,51 +0,0 @@
-import { createEmbeddingBuckets } from '../../domain/embeddings.js';
-const toTitleKey = (title) => title.toLowerCase();
-export const createIndexWriter = (database) => ({
-    reset: () => {
-        database.exec(`
-      DELETE FROM embedding_buckets;
-      DELETE FROM chunks_fts;
-      DELETE FROM links;
-      DELETE FROM chunks;
-      DELETE FROM documents;
-    `);
-    },
-    saveDocuments: (documents) => {
-        const insertDocument = database.prepare(`
-      INSERT INTO documents (id, agent_id, title, path, content, tags_json, frontmatter_json, created_at, updated_at)
-      VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
-    `);
-        const insertChunk = database.prepare(`
-      INSERT INTO chunks (id, document_id, ordinal, content, token_count, embedding_provider, embedding_json)
-      VALUES (?, ?, ?, ?, ?, ?, ?)
-    `);
-        const insertChunkFts = database.prepare(`
-      INSERT INTO chunks_fts (chunk_id, document_id, agent_id, title, content)
-      VALUES (?, ?, ?, ?, ?)
-    `);
-        const insertEmbeddingBucket = database.prepare(`
-      INSERT OR IGNORE INTO embedding_buckets (bucket, chunk_id)
-      VALUES (?, ?)
-    `);
-        const insertLink = database.prepare(`
-      INSERT INTO links (from_document_id, to_title, to_title_key, to_document_id, weight, priority)
-      VALUES (?, ?, ?, ?, ?, ?)
-    `);
-        const transaction = database.transaction(() => {
-            documents.forEach(({ document, chunks, links }) => {
-                insertDocument.run(document.id, document.agentId, document.title, document.path, document.content, JSON.stringify(document.tags), JSON.stringify(document.frontmatter), document.createdAt, document.updatedAt);
-                chunks.forEach((chunk) => {
-                    insertChunk.run(chunk.id, chunk.documentId, chunk.ordinal, chunk.content, chunk.tokenCount, chunk.embeddingProvider, JSON.stringify(chunk.embedding));
-                    insertChunkFts.run(chunk.id, chunk.documentId, document.agentId, document.title, chunk.content);
-                    createEmbeddingBuckets(chunk.embedding).forEach((bucket) => insertEmbeddingBucket.run(bucket, chunk.id));
-                });
-            });
-            documents.forEach(({ links }) => {
-                links.forEach((link) => {
-                    insertLink.run(link.fromDocumentId, link.toTitle, toTitleKey(link.toTitle), link.toDocumentId, link.weight, link.priority);
-                });
-            });
-        });
-        transaction();
-    }
-});