@andespindola/brainlink 0.1.0-beta.43 → 0.1.0-beta.44

package/AGENTS.md

@@ -85,6 +85,7 @@ npm run dev -- server --vault ./vault --watch
 npm run dev -- watch --vault ./vault
 npm run dev -- bench --vault ./vault
 npm run dev -- bench --vault ./vault --watch
+npm run dev -- pack-backup --vault ./vault
 ```
 
 Start MCP over stdio:

package/CHANGELOG.md

@@ -39,6 +39,11 @@
 - Improved massive-graph UI responsiveness with stricter render budgets, adaptive heavy-graph frame throttling, reduced interaction hit-test frequency, and URL-first agent selection on initial graph load.
 - Improved 50k+ graph LOD behavior so zoomed-out views render lightweight cluster overviews and progressively reveal nodes/edges only as zoom increases.
 - Added `blink bench` with realtime index phase telemetry and per-run compressed-pack analysis (input/output bytes, ratio, saved space, rebuild reason and duration), including continuous watch mode.
+- Added tunable single-stage search-pack compression settings (`searchPack.rowChunkSize`, `searchPack.compressionLevel`, `searchPack.useDictionary`).
+- Added benchmark guardrails for compression savings and latency regression (`searchPack.guardrailMinSavingsPercent`, `searchPack.guardrailMaxLatencyRegressionPercent`), reported in `blink bench`.
+- Added `blink pack-backup` for offline second-stage compression backups of encrypted `.blpk` packs, outside the online query path.
+- Hardened Linux browser launch flags for Ubuntu 26 Chromium/Wayland compatibility (`--disable-vulkan`, `--use-gl=swiftshader`, `--ozone-platform-hint=x11`).
+- Improved pack resilience by auto-repairing missing search-pack manifests from existing `.blpk` files, avoiding unnecessary full repacks on small incremental updates.
 
 ## 0.1.0-beta.3
 

package/README.md

@@ -58,6 +58,7 @@ LLMs do not have infinite context. Brainlink gives agents an external memory lay
 
 Markdown is the source of truth. `.brainlink/index.json` is a rebuildable index artifact.
 After each index run, Brainlink also writes private encrypted search packs at `.brainlink/search-packs/*.blpk` to preserve fast retrieval and portable recovery.
+Online retrieval always uses a single compression stage per pack; optional second-stage compression is reserved for offline backup artifacts only.
 Pack decryption uses a Brainlink key from `$BRAINLINK_HOME/keys` or from `BRAINLINK_SEARCH_PACK_KEY` when explicitly configured.
 Legacy `.jsonl.gz` packs are upgraded to `.blpk` automatically on first search/context access.
 
@@ -570,7 +571,7 @@ By default, `blink server` tries to open the graph in a native desktop GUI windo
 
 On Linux, native GUI is disabled by default for better startup performance. Enable it with `BRAINLINK_LINUX_NATIVE_GUI=1`.
 If native GUI launch is unavailable on your system, it falls back to dedicated app-window mode and then to the default browser.
-For Chromium-family browsers on Linux (`chromium`, `chromium-browser`, `google-chrome`, `microsoft-edge`, `brave-browser`), Brainlink now auto-applies compatibility flags during launch (`--ozone-platform=x11`, `--disable-gpu`, `--disable-features=Vulkan,VaapiVideoDecoder`, `--disable-background-networking`) to avoid common Wayland/Vulkan/VAAPI startup issues.
+For Chromium-family browsers on Linux (`chromium`, `chromium-browser`, `google-chrome`, `microsoft-edge`, `brave-browser`), Brainlink now auto-applies compatibility flags during launch (`--ozone-platform=x11`, `--ozone-platform-hint=x11`, `--disable-gpu`, `--disable-vulkan`, `--use-gl=swiftshader`, `--disable-features=Vulkan,VaapiVideoDecoder`, `--disable-background-networking`) to avoid common Wayland/Vulkan/VAAPI startup issues.
 Use `--no-open` to keep it headless.
 When native GUI is used, the GUI window automatically closes when the `blink server` process stops.
 
@@ -776,8 +777,21 @@ Summary includes compression behavior for `.blpk` packs when rebuild happens:
 - pack count and pack build duration
 - uncompressed input bytes vs compressed output bytes
 - saved percentage
+- objective guardrails (minimum savings and maximum latency regression thresholds)
 
 Use `--watch` to keep benchmarking incremental reindex runs after Markdown changes (local filesystem vaults only).
+When `.brainlink/search-packs/manifest.json` is missing but `.blpk` files exist, Brainlink repairs the manifest first and avoids unnecessary full pack rebuild on small edits.
+
+### `pack-backup`
+
+```bash
+blink pack-backup --vault ./vault
+blink pack-backup --vault ./vault --output ./vault/.brainlink/backups/custom.blpkbak.gz
+blink pack-backup --vault ./vault --json
+```
+
+Creates an offline backup artifact of encrypted search packs with a second compression pass.
+This is intentionally outside the online retrieval path (`index`, `search`, `context`).
 
 ### `agents`
 
@@ -951,6 +965,13 @@ If no `vault` is configured and no `--vault` flag is passed, Brainlink uses `$HO
   "embeddingProvider": "local",
   "defaultSearchMode": "hybrid",
   "chunkSize": 1200,
+  "searchPack": {
+    "rowChunkSize": 5000,
+    "compressionLevel": 5,
+    "useDictionary": true,
+    "guardrailMinSavingsPercent": 8,
+    "guardrailMaxLatencyRegressionPercent": 5
+  },
   "agentProfiles": {
     "coding-agent": {
       "defaultSearchMode": "semantic",

package/dist/application/index-vault.js

@@ -1,12 +1,11 @@
-import { readFile, stat } from 'node:fs/promises';
-import { join } from 'node:path';
+import { readFile } from 'node:fs/promises';
 import { createIndexedDocument, parseMarkdownDocument } from '../domain/markdown.js';
 import { sharedAgentId } from '../domain/agents.js';
 import { createEmbeddingProvider } from '../domain/embeddings.js';
 import { loadBrainlinkConfig } from '../infrastructure/config.js';
 import { ensureVault, readMarkdownFileSummaries } from '../infrastructure/file-system-vault.js';
 import { readIndexState, writeIndexState } from '../infrastructure/index-state.js';
-import { buildSearchPacks } from '../infrastructure/search-packs.js';
+import { buildSearchPacks, ensureSearchPackManifest, toSearchPackBuildOptions } from '../infrastructure/search-packs.js';
 import { openFileIndex } from '../infrastructure/file-index.js';
 const toTitleKey = (title) => title.toLowerCase();
 const appendTitleEntry = (map, document) => {
@@ -76,16 +75,6 @@ const toSnapshot = (summaries) => summaries.map((summary) => ({
     size: summary.size
 }));
 const createSnapshotMap = (snapshot) => new Map(snapshot.map((entry) => [entry.path, entry]));
-const packManifestPath = (vaultPath) => join(vaultPath, '.brainlink', 'search-packs', 'manifest.json');
-const hasPackManifest = async (vaultPath) => {
-    try {
-        await stat(packManifestPath(vaultPath));
-        return true;
-    }
-    catch {
-        return false;
-    }
-};
 const readChangedDocuments = async (absoluteVaultPath, changedSummaries) => {
     const parsed = await Promise.all(changedSummaries.map(async (summary) => parseMarkdownDocument({
         absolutePath: summary.absolutePath,
@@ -134,6 +123,10 @@ export const indexVaultWithOptions = async (vaultPath, options) => {
         const settingsChanged = previousState == null ||
             previousState.chunkSize !== config.chunkSize ||
             previousState.embeddingProvider !== config.embeddingProvider;
+        const packSettingsChanged = previousState == null ||
+            previousState.searchPackRowChunkSize !== config.searchPack.rowChunkSize ||
+            previousState.searchPackCompressionLevel !== config.searchPack.compressionLevel ||
+            previousState.searchPackUseDictionary !== config.searchPack.useDictionary;
         const changedPaths = new Set();
         for (let index = 0; index < summaries.length; index += 1) {
             const summary = summaries[index];
@@ -150,6 +143,7 @@ export const indexVaultWithOptions = async (vaultPath, options) => {
         const hasDeletes = previousState
             ? previousState.files.some((entry) => !currentSnapshotMap.has(entry.path))
             : false;
+        const manifestRecovery = await ensureSearchPackManifest(absoluteVaultPath);
         if (changedPaths.size === 0 &&
             !hasDeletes &&
             existingIndexedDocuments.length === summaries.length &&
@@ -160,11 +154,13 @@ export const indexVaultWithOptions = async (vaultPath, options) => {
                 changedDocumentCount: 0,
                 packs: {
                     rebuilt: false,
-                    reason: 'No changes detected'
+                    reason: manifestRecovery.repaired ? 'No changes detected; pack manifest repaired' : 'No changes detected'
                 }
             };
             emit('complete', 'skip', 'Index skipped: no changes detected', {
-                elapsedMs: result.elapsedMs
+                elapsedMs: result.elapsedMs,
+                manifestRepaired: manifestRecovery.repaired,
+                manifestRecoverySource: manifestRecovery.source
             });
             return result;
         }
@@ -213,7 +209,7 @@ export const indexVaultWithOptions = async (vaultPath, options) => {
         emit('persist', 'finish', 'Index persisted', {
             indexedDocuments: indexedDocuments.length
         });
-        const existingPackManifest = await hasPackManifest(absoluteVaultPath);
+        const existingPackManifest = manifestRecovery.repaired || manifestRecovery.source === 'not-needed';
         const changedCount = changedPaths.size;
         const documentCount = Math.max(indexedDocuments.length, 1);
         const changeRatio = changedCount / documentCount;
@@ -221,6 +217,7 @@ export const indexVaultWithOptions = async (vaultPath, options) => {
         const pendingPackChanges = previousPendingPackChanges + changedCount;
         const shouldRebuildPacks = !existingPackManifest ||
             settingsChanged ||
+            packSettingsChanged ||
             hasDeletes ||
             changedCount >= 400 ||
             changeRatio >= 0.04 ||
@@ -228,23 +225,27 @@ export const indexVaultWithOptions = async (vaultPath, options) => {
         let packResult;
         const packReason = !existingPackManifest
             ? 'Missing pack manifest'
-            : settingsChanged
-                ? 'Index settings changed'
-                : hasDeletes
-                    ? 'Document deletions detected'
-                    : changedCount >= 400
-                        ? 'Changed file count threshold reached'
-                        : changeRatio >= 0.04
-                            ? 'Change ratio threshold reached'
-                            : pendingPackChanges >= 1200
-                                ? 'Pending pack changes threshold reached'
-                                : 'Pack rebuild skipped';
+            : manifestRecovery.repaired
+                ? 'Pack manifest repaired from existing packs'
+                : settingsChanged
+                    ? 'Index settings changed'
+                    : packSettingsChanged
+                        ? 'Search pack settings changed'
+                        : hasDeletes
+                            ? 'Document deletions detected'
+                            : changedCount >= 400
+                                ? 'Changed file count threshold reached'
+                                : changeRatio >= 0.04
+                                    ? 'Change ratio threshold reached'
+                                    : pendingPackChanges >= 1200
+                                        ? 'Pending pack changes threshold reached'
+                                        : 'Pack rebuild skipped';
         if (shouldRebuildPacks) {
             emit('packs', 'start', 'Rebuilding compressed search packs', {
                 reason: packReason
             });
             try {
-                packResult = await buildSearchPacks(absoluteVaultPath, indexedDocuments);
+                packResult = await buildSearchPacks(absoluteVaultPath, indexedDocuments, toSearchPackBuildOptions(config));
                 emit('packs', 'finish', 'Compressed packs rebuilt', {
                     reason: packReason,
                     packCount: packResult.packCount,
@@ -270,6 +271,9 @@ export const indexVaultWithOptions = async (vaultPath, options) => {
         await writeIndexState(absoluteVaultPath, {
             chunkSize: config.chunkSize,
             embeddingProvider: config.embeddingProvider,
+            searchPackRowChunkSize: config.searchPack.rowChunkSize,
+            searchPackCompressionLevel: config.searchPack.compressionLevel,
+            searchPackUseDictionary: config.searchPack.useDictionary,
             files: currentSnapshot,
             pendingPackChanges: packsRebuilt ? 0 : pendingPackChanges
         });

package/dist/application/offline-pack-backup.js

@@ -0,0 +1,44 @@
+import { mkdir, readdir, readFile, writeFile } from 'node:fs/promises';
+import { dirname, join } from 'node:path';
+import { gzipSync } from 'node:zlib';
+import { ensureVault } from '../infrastructure/file-system-vault.js';
+const packsDirectory = (vaultPath) => join(vaultPath, '.brainlink', 'search-packs');
+const toSortedBackupFiles = async (vaultPath) => {
+    const directory = packsDirectory(vaultPath);
+    const names = await readdir(directory);
+    return names
+        .filter((name) => name.endsWith('.blpk') || name === 'manifest.json')
+        .sort((left, right) => left.localeCompare(right));
+};
+export const createOfflinePackBackup = async (input) => {
+    const vaultPath = await ensureVault(input.vaultPath);
+    const fileNames = await toSortedBackupFiles(vaultPath);
+    const files = [];
+    let inputBytes = 0;
+    for (const name of fileNames) {
+        const content = await readFile(join(packsDirectory(vaultPath), name));
+        inputBytes += content.byteLength;
+        files.push({
+            name,
+            contentB64: content.toString('base64')
+        });
+    }
+    const envelope = {
+        version: 1,
+        createdAt: new Date().toISOString(),
+        files
+    };
+    const serialized = Buffer.from(JSON.stringify(envelope), 'utf8');
+    const compressed = gzipSync(serialized, { level: 9 });
+    await mkdir(dirname(input.outputPath), { recursive: true });
+    await writeFile(input.outputPath, compressed);
+    const safeInput = Math.max(inputBytes, 1);
+    return {
+        outputPath: input.outputPath,
+        fileCount: files.length,
+        inputBytes,
+        outputBytes: compressed.byteLength,
+        ratio: compressed.byteLength / safeInput,
+        savedBytes: Math.max(inputBytes - compressed.byteLength, 0)
+    };
+};

package/dist/cli/commands/write-commands.js

@@ -1,5 +1,5 @@
 import { existsSync, readFileSync, writeFileSync } from 'node:fs';
-import { mkdir, writeFile } from 'node:fs/promises';
+import { mkdir, readFile, writeFile } from 'node:fs/promises';
 import { dirname, join, relative, resolve } from 'node:path';
 import { platform, tmpdir } from 'node:os';
 import { spawn, spawnSync } from 'node:child_process';
@@ -9,6 +9,7 @@ import { resolveDuplicateNotes, scanDuplicateNotes } from '../../application/ded
 import { importLegacySqliteDatabase } from '../../application/import-legacy-sqlite.js';
 import { indexVault, indexVaultWithOptions } from '../../application/index-vault.js';
 import { migrateVaultContent, planVaultMigration, previewVaultMigration, shouldMigrateDefaultVault } from '../../application/migrate-vault.js';
+import { createOfflinePackBackup } from '../../application/offline-pack-backup.js';
 import { startServer } from '../../application/start-server.js';
 import { startVaultWatcher } from '../../application/watch-vault.js';
 import { doctorVault, getStats, validateVault } from '../../application/analyze-vault.js';
@@ -83,6 +84,71 @@ const printBenchSummary = (json, trigger, vault, result) => {
         ].join('\n');
     });
 };
+const benchHistoryPath = (vaultPath) => join(vaultPath, '.brainlink', 'benchmarks', 'latest.json');
+const readBenchHistory = async (vaultPath) => {
+    try {
+        const parsed = JSON.parse(await readFile(benchHistoryPath(vaultPath), 'utf8'));
+        if (typeof parsed.elapsedMs !== 'number' || typeof parsed.timestamp !== 'string') {
+            return null;
+        }
+        return {
+            elapsedMs: parsed.elapsedMs,
+            timestamp: parsed.timestamp,
+            ...(typeof parsed.compressionRatio === 'number' ? { compressionRatio: parsed.compressionRatio } : {})
+        };
+    }
+    catch {
+        return null;
+    }
+};
+const writeBenchHistory = async (vaultPath, result) => {
+    await mkdir(dirname(benchHistoryPath(vaultPath)), { recursive: true });
+    const payload = {
+        elapsedMs: result.elapsedMs ?? 0,
+        timestamp: new Date().toISOString(),
+        ...(typeof result.packs?.compression?.ratio === 'number' ? { compressionRatio: result.packs.compression.ratio } : {})
+    };
+    await writeFile(benchHistoryPath(vaultPath), `${JSON.stringify(payload, null, 2)}\n`, 'utf8');
+};
+const evaluateBenchGuardrails = (config, result, baseline) => {
+    const compressionRatio = result.packs?.compression?.ratio;
+    const compressionSavingsPercent = typeof compressionRatio === 'number' ? Math.max(0, (1 - compressionRatio) * 100) : undefined;
+    const compressionPass = compressionSavingsPercent != null
+        ? compressionSavingsPercent >= config.searchPack.guardrailMinSavingsPercent
+        : undefined;
+    const latencyRegressionPercent = baseline && baseline.elapsedMs > 0 && typeof result.elapsedMs === 'number'
+        ? ((result.elapsedMs - baseline.elapsedMs) / baseline.elapsedMs) * 100
+        : undefined;
+    const latencyPass = latencyRegressionPercent != null
+        ? latencyRegressionPercent <= config.searchPack.guardrailMaxLatencyRegressionPercent
+        : undefined;
+    return {
+        ...(compressionSavingsPercent != null ? { compressionSavingsPercent } : {}),
+        ...(compressionPass != null ? { compressionPass } : {}),
+        ...(latencyRegressionPercent != null ? { latencyRegressionPercent } : {}),
+        ...(latencyPass != null ? { latencyPass } : {})
+    };
+};
+const printBenchGuardrails = (json, vault, config, guardrails) => {
+    print(json, {
+        event: 'bench-guardrails',
+        vault,
+        thresholds: {
+            minSavingsPercent: config.searchPack.guardrailMinSavingsPercent,
+            maxLatencyRegressionPercent: config.searchPack.guardrailMaxLatencyRegressionPercent
+        },
+        guardrails
+    }, () => {
+        const savings = guardrails.compressionSavingsPercent;
+        const latency = guardrails.latencyRegressionPercent;
+        return [
+            '[bench] guardrails',
+            `minSavings=${config.searchPack.guardrailMinSavingsPercent.toFixed(1)}% maxLatencyRegression=${config.searchPack.guardrailMaxLatencyRegressionPercent.toFixed(1)}%`,
+            `compressionSavings=${savings != null ? `${savings.toFixed(2)}%` : 'n/a'} pass=${guardrails.compressionPass != null ? (guardrails.compressionPass ? 'yes' : 'no') : 'n/a'}`,
+            `latencyRegression=${latency != null ? `${latency.toFixed(2)}%` : 'n/a'} pass=${guardrails.latencyPass != null ? (guardrails.latencyPass ? 'yes' : 'no') : 'n/a'}`
+        ].join('\n');
+    });
+};
 const spawnDetached = (command, args, envOverrides) => {
     try {
         const child = spawn(command, args, {
@@ -371,7 +437,10 @@ const openGraphInAppWindow = (url) => {
     const appArgument = `--app=${url}`;
     const linuxChromiumStableFlags = [
         '--ozone-platform=x11',
+        '--ozone-platform-hint=x11',
         '--disable-gpu',
+        '--disable-vulkan',
+        '--use-gl=swiftshader',
         '--disable-features=Vulkan,VaapiVideoDecoder',
         '--disable-background-networking'
     ];
@@ -406,7 +475,10 @@ const openGraphInDetectedBrowser = (url) => {
     }
     const linuxChromiumStableFlags = [
         '--ozone-platform=x11',
+        '--ozone-platform-hint=x11',
         '--disable-gpu',
+        '--disable-vulkan',
+        '--use-gl=swiftshader',
         '--disable-features=Vulkan,VaapiVideoDecoder',
         '--disable-background-networking'
     ];
@@ -689,6 +761,7 @@ export const registerWriteCommands = (program) => {
         .description('benchmark indexing in realtime, including compressed pack behavior')
         .action(async (options) => {
         const resolved = await resolveOptions(options);
+        const config = await loadBrainlinkConfig();
         const emitProgress = (event) => {
             printBenchRealtimeEvent(options.json, event);
         };
@@ -697,10 +770,14 @@ export const registerWriteCommands = (program) => {
             print(options.json, { event: 'bench-error', message }, () => `[bench] error ${message}`);
         };
         const runAndPrint = async (trigger) => {
+            const baseline = await readBenchHistory(resolved.vault);
             const result = await indexVaultWithOptions(resolved.vault, {
                 onProgress: emitProgress
             });
             printBenchSummary(options.json, trigger, resolved.vault, result);
+            const guardrails = evaluateBenchGuardrails(config, result, baseline);
+            printBenchGuardrails(options.json, resolved.vault, config, guardrails);
+            await writeBenchHistory(resolved.vault, result);
             return result;
         };
         if (!options.watch) {
@@ -749,6 +826,30 @@ export const registerWriteCommands = (program) => {
         });
         process.exitCode = report.ok ? 0 : 1;
     });
+    program
+        .command('pack-backup')
+        .option('-v, --vault <vault>', 'vault directory')
+        .option('-o, --output <path>', 'output file path (.blpkbak.gz)')
+        .option('--json', 'print machine-readable JSON')
+        .description('create offline backup with second-stage compression for encrypted search packs')
+        .action(async (options) => {
+        const resolved = await resolveOptions(options);
+        const outputPath = options.output?.trim().length
+            ? resolve(options.output)
+            : join(resolved.vault, '.brainlink', 'backups', `search-packs-${new Date().toISOString().replace(/[:.]/g, '-')}.blpkbak.gz`);
+        const backup = await createOfflinePackBackup({
+            vaultPath: resolved.vault,
+            outputPath
+        });
+        print(options.json, {
+            vault: resolved.vault,
+            backup
+        }, () => [
+            `Offline backup created: ${backup.outputPath}`,
+            `files=${backup.fileCount}`,
+            `input=${formatBytes(backup.inputBytes)} output=${formatBytes(backup.outputBytes)} saved=${((1 - backup.ratio) * 100).toFixed(2)}%`
+        ].join('\n'));
+    });
     program
         .command('watch')
         .option('-v, --vault <vault>', 'vault directory')

package/dist/infrastructure/config.js

@@ -15,6 +15,13 @@ export const defaultBrainlinkConfig = {
     embeddingProvider: 'local',
     defaultSearchMode: 'hybrid',
     chunkSize: 1200,
+    searchPack: {
+        rowChunkSize: 5_000,
+        compressionLevel: 5,
+        useDictionary: true,
+        guardrailMinSavingsPercent: 8,
+        guardrailMaxLatencyRegressionPercent: 5
+    },
     agentProfiles: {}
 };
 const configFilenames = ['brainlink.config.json', '.brainlink.json'];
@@ -37,6 +44,36 @@ const sanitizeEmbeddingProvider = (value) => typeof value === 'string' && embedd
 export const sanitizeSearchMode = (value, fallback = defaultBrainlinkConfig.defaultSearchMode) => typeof value === 'string' && searchModes.has(value) ? value : fallback;
 const sanitizeAllowedVaults = (value) => Array.isArray(value) ? value.filter((item) => typeof item === 'string' && item.trim().length > 0) : [];
 const sanitizePositiveNumber = (value) => typeof value === 'number' && Number.isFinite(value) && value > 0 ? value : undefined;
+const sanitizeIntegerInRange = (value, fallback, minimum, maximum) => {
+    if (typeof value !== 'number' || !Number.isFinite(value)) {
+        return fallback;
+    }
+    const rounded = Math.round(value);
+    if (rounded < minimum) {
+        return minimum;
+    }
+    if (rounded > maximum) {
+        return maximum;
+    }
+    return rounded;
+};
+const sanitizeSearchPackConfig = (value) => {
+    const fallback = defaultBrainlinkConfig.searchPack;
+    if (!isRecord(value)) {
+        return fallback;
+    }
+    return {
+        rowChunkSize: sanitizeIntegerInRange(value.rowChunkSize, fallback.rowChunkSize, 100, 100_000),
+        compressionLevel: sanitizeIntegerInRange(value.compressionLevel, fallback.compressionLevel, 0, 11),
+        useDictionary: typeof value.useDictionary === 'boolean' ? value.useDictionary : fallback.useDictionary,
+        guardrailMinSavingsPercent: typeof value.guardrailMinSavingsPercent === 'number' && Number.isFinite(value.guardrailMinSavingsPercent)
+            ? Math.max(0, Math.min(95, value.guardrailMinSavingsPercent))
+            : fallback.guardrailMinSavingsPercent,
+        guardrailMaxLatencyRegressionPercent: typeof value.guardrailMaxLatencyRegressionPercent === 'number' && Number.isFinite(value.guardrailMaxLatencyRegressionPercent)
+            ? Math.max(0, Math.min(300, value.guardrailMaxLatencyRegressionPercent))
+            : fallback.guardrailMaxLatencyRegressionPercent
+    };
+};
 const sanitizeAgentProfile = (value) => {
     if (!isRecord(value)) {
         return null;
@@ -130,6 +167,7 @@ const sanitizeConfig = (value) => ({
         : defaultBrainlinkConfig.defaultContextTokens,
     allowedVaults: [...sanitizeAllowedVaults(value.allowedVaults), ...readAllowedVaultsFromEnv()],
     chunkSize: typeof value.chunkSize === 'number' && value.chunkSize > 0 ? value.chunkSize : defaultBrainlinkConfig.chunkSize,
+    searchPack: sanitizeSearchPackConfig(value.searchPack),
     embeddingProvider: sanitizeEmbeddingProvider(value.embeddingProvider),
     defaultSearchMode: sanitizeSearchMode(value.defaultSearchMode),
     agentProfiles: sanitizeAgentProfiles(value.agentProfiles)

package/dist/infrastructure/index-state.js

@@ -29,6 +29,9 @@ export const readIndexState = async (vaultPath) => {
             updatedAt: typeof parsed.updatedAt === 'string' ? parsed.updatedAt : new Date().toISOString(),
             chunkSize: typeof parsed.chunkSize === 'number' ? parsed.chunkSize : 1200,
             embeddingProvider: typeof parsed.embeddingProvider === 'string' ? parsed.embeddingProvider : 'none',
+            searchPackRowChunkSize: typeof parsed.searchPackRowChunkSize === 'number' ? parsed.searchPackRowChunkSize : 5_000,
+            searchPackCompressionLevel: typeof parsed.searchPackCompressionLevel === 'number' ? parsed.searchPackCompressionLevel : 5,
+            searchPackUseDictionary: typeof parsed.searchPackUseDictionary === 'boolean' ? parsed.searchPackUseDictionary : true,
             files,
             pendingPackChanges: typeof parsed.pendingPackChanges === 'number' && parsed.pendingPackChanges >= 0 ? parsed.pendingPackChanges : 0
         };
@@ -43,6 +46,9 @@ export const writeIndexState = async (vaultPath, state) => {
         updatedAt: new Date().toISOString(),
         chunkSize: state.chunkSize,
         embeddingProvider: state.embeddingProvider,
+        searchPackRowChunkSize: state.searchPackRowChunkSize,
+        searchPackCompressionLevel: state.searchPackCompressionLevel,
+        searchPackUseDictionary: state.searchPackUseDictionary,
         files: [...state.files].sort((left, right) => left.path.localeCompare(right.path)),
         pendingPackChanges: Math.max(0, Math.floor(state.pendingPackChanges))
     };

package/dist/infrastructure/private-pack-codec.js

@@ -1,13 +1,25 @@
 import { createCipheriv, createDecipheriv, createHash, randomBytes } from 'node:crypto';
-import { brotliCompressSync, brotliDecompressSync } from 'node:zlib';
+import { brotliCompressSync, brotliDecompressSync, constants as zlibConstants } from 'node:zlib';
 import { mkdir, readFile, writeFile } from 'node:fs/promises';
 import { dirname, join } from 'node:path';
 import { getBrainlinkHomePath } from './paths.js';
 const magic = Buffer.from('BLPK2', 'ascii');
-const version = 1;
+const legacyVersion = 1;
+const currentVersion = 2;
 const nonceLength = 12;
 const authTagLength = 16;
 const algorithm = 'aes-256-gcm';
+const compressionLevelMask = 0x0f;
+const compressionDictionaryMask = 0x10;
+const defaultCompressionLevel = 5;
+const builtinDictionary = Buffer.from([
+    '"documentId","agentId","title","path","chunkId","chunkOrdinal","content","tags"',
+    '"searchMode","textScore","semanticScore","weight","priority","shared"',
+    'agents/shared memory-hub architecture context index search graph markdown tags links',
+    '#memory #architecture #context #graph #search #index [[Memory Hub]] [[Architecture]]',
+    'The quick brown fox jumps over the lazy dog. Brainlink context package metadata.',
+    'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-:/.#[]{}(), '
+].join('\n'), 'utf8');
 const keyFilePath = (vaultPath) => {
     const vaultHash = createHash('sha256').update(vaultPath).digest('hex').slice(0, 24);
     return join(getBrainlinkHomePath(), 'keys', `search-pack-${vaultHash}.key`);
@@ -40,34 +52,83 @@ const parseHeader = (payload) => {
         throw new Error('Invalid private pack payload: too short.');
     }
     const payloadMagic = payload.subarray(0, magic.length);
-    const payloadVersion = payload[magic.length];
-    if (!payloadMagic.equals(magic) || payloadVersion !== version) {
+    const payloadVersion = payload[magic.length] ?? 0;
+    if (!payloadMagic.equals(magic) || (payloadVersion !== legacyVersion && payloadVersion !== currentVersion)) {
         throw new Error('Invalid private pack payload: unsupported format.');
     }
-    const nonceStart = magic.length + 1;
+    const hasCompressionSettings = payloadVersion >= 2;
+    const settingsByte = hasCompressionSettings ? payload[magic.length + 1] ?? 0 : null;
+    const nonceStart = magic.length + 1 + (hasCompressionSettings ? 1 : 0);
     const authTagStart = nonceStart + nonceLength;
     const dataStart = authTagStart + authTagLength;
     return {
+        compression: settingsByte != null
+            ? {
+                compressionLevel: settingsByte & compressionLevelMask,
+                useDictionary: (settingsByte & compressionDictionaryMask) !== 0
+            }
+            : {
+                compressionLevel: defaultCompressionLevel,
+                useDictionary: false
+            },
         nonce: payload.subarray(nonceStart, authTagStart),
         authTag: payload.subarray(authTagStart, dataStart),
         ciphertext: payload.subarray(dataStart)
     };
 };
-export const encodePrivatePack = async (vaultPath, content) => {
+const toCompressionLevel = (value) => {
+    if (typeof value !== 'number' || !Number.isFinite(value)) {
+        return defaultCompressionLevel;
+    }
+    const normalized = Math.round(value);
+    if (normalized < 0) {
+        return 0;
+    }
+    if (normalized > 11) {
+        return 11;
+    }
+    return normalized;
+};
+const encodeCompressionSettings = (settings) => (settings.compressionLevel & compressionLevelMask) | (settings.useDictionary ? compressionDictionaryMask : 0);
+const brotliEncode = (content, settings) => {
+    const options = {
+        params: {
+            [zlibConstants.BROTLI_PARAM_MODE]: zlibConstants.BROTLI_MODE_TEXT,
+            [zlibConstants.BROTLI_PARAM_QUALITY]: settings.compressionLevel
+        }
+    };
+    if (settings.useDictionary) {
+        options.dictionary = builtinDictionary;
+    }
+    return brotliCompressSync(content, options);
+};
+const brotliDecode = (content, settings) => {
+    const options = {};
+    if (settings.useDictionary) {
+        options.dictionary = builtinDictionary;
+    }
+    return brotliDecompressSync(content, options);
+};
+export const encodePrivatePack = async (vaultPath, content, settings) => {
     const key = await readOrCreateKey(vaultPath);
     const nonce = randomBytes(nonceLength);
-    const compressed = brotliCompressSync(content);
+    const normalizedSettings = {
+        compressionLevel: toCompressionLevel(settings?.compressionLevel),
+        useDictionary: settings?.useDictionary ?? true
+    };
+    const compressed = brotliEncode(content, normalizedSettings);
     const cipher = createCipheriv(algorithm, key, nonce);
     const ciphertext = Buffer.concat([cipher.update(compressed), cipher.final()]);
     const authTag = cipher.getAuthTag();
-    return Buffer.concat([magic, Buffer.from([version]), nonce, authTag, ciphertext]);
+    const settingsByte = Buffer.from([encodeCompressionSettings(normalizedSettings)]);
+    return Buffer.concat([magic, Buffer.from([currentVersion]), settingsByte, nonce, authTag, ciphertext]);
 };
 export const decodePrivatePack = async (vaultPath, payload) => {
     const key = await readOrCreateKey(vaultPath);
-    const { nonce, authTag, ciphertext } = parseHeader(payload);
+    const { nonce, authTag, ciphertext, compression } = parseHeader(payload);
     const decipher = createDecipheriv(algorithm, key, nonce);
     decipher.setAuthTag(authTag);
     const compressed = Buffer.concat([decipher.update(ciphertext), decipher.final()]);
-    return brotliDecompressSync(compressed);
+    return brotliDecode(compressed, compression);
 };
 export const isPrivatePackPayload = (payload) => payload.length >= magic.length + 1 && payload.subarray(0, magic.length).equals(magic);

package/dist/infrastructure/search-packs.js

@@ -5,7 +5,11 @@ import { middleOutIndices } from '../domain/middle-out.js';
 import { decodePrivatePack, encodePrivatePack, isPrivatePackPayload } from './private-pack-codec.js';
 const packsDirectoryName = 'search-packs';
 const manifestFileName = 'manifest.json';
-const rowChunkSize = 5_000;
+const defaultBuildOptions = {
+    rowChunkSize: 5_000,
+    compressionLevel: 5,
+    useDictionary: true
+};
 const queryTokenPattern = /[\p{L}\p{N}_-]+/gu;
 const bloomBytes = 256;
 const bloomBitSize = bloomBytes * 8;
@@ -94,7 +98,37 @@ const readManifest = async (vaultPath) => {
                 packCount: typeof parsed.packCount === 'number' ? parsed.packCount : packIndex.length,
                 recordCount: typeof parsed.recordCount === 'number' ? parsed.recordCount : 0,
                 format: 'private-v2',
-                packIndex
+                packIndex,
+                ...(parsed.packConfig && typeof parsed.packConfig === 'object'
+                    ? {
+                        packConfig: {
+                            rowChunkSize: typeof parsed.packConfig.rowChunkSize === 'number'
+                                ? parsed.packConfig.rowChunkSize
+                                : defaultBuildOptions.rowChunkSize,
+                            compressionLevel: typeof parsed.packConfig.compressionLevel === 'number'
+                                ? parsed.packConfig.compressionLevel
+                                : defaultBuildOptions.compressionLevel,
+                            useDictionary: typeof parsed.packConfig.useDictionary === 'boolean'
+                                ? parsed.packConfig.useDictionary
+                                : defaultBuildOptions.useDictionary
+                        }
+                    }
+                    : {}),
+                ...(parsed.compression &&
+                    typeof parsed.compression === 'object' &&
+                    typeof parsed.compression.inputBytes === 'number' &&
+                    typeof parsed.compression.outputBytes === 'number' &&
+                    typeof parsed.compression.ratio === 'number' &&
+                    typeof parsed.compression.savedBytes === 'number'
+                    ? {
+                        compression: {
+                            inputBytes: parsed.compression.inputBytes,
+                            outputBytes: parsed.compression.outputBytes,
+                            ratio: parsed.compression.ratio,
+                            savedBytes: parsed.compression.savedBytes
+                        }
+                    }
+                    : {})
             };
         }
         return null;
@@ -103,6 +137,37 @@ const readManifest = async (vaultPath) => {
         return null;
     }
 };
+export const ensureSearchPackManifest = async (vaultPath) => {
+    const manifest = await readManifest(vaultPath);
+    if (manifest) {
+        return {
+            repaired: false,
+            source: 'not-needed',
+            packCount: manifest.packCount
+        };
+    }
+    const files = await sortedPackFiles(vaultPath);
+    const packFiles = files.filter((file) => file.endsWith('.blpk'));
+    if (packFiles.length === 0) {
+        return {
+            repaired: false,
+            source: 'no-packs',
+            packCount: 0
+        };
+    }
+    await writeManifest(vaultPath, {
+        version: 2,
+        createdAt: new Date().toISOString(),
+        packCount: packFiles.length,
+        recordCount: 0,
+        format: 'private-v2'
+    });
+    return {
+        repaired: true,
+        source: 'existing-packs',
+        packCount: packFiles.length
+    };
+};
 const chunkRows = (rows, size) => {
     const chunks = [];
     for (let index = 0; index < rows.length; index += size) {
@@ -220,7 +285,7 @@ const sortedPackFiles = async (vaultPath) => {
         throw error;
     }
 };
-const writeRowsAsPrivatePacks = async (vaultPath, rows, clearExisting) => {
+const writeRowsAsPrivatePacks = async (vaultPath, rows, clearExisting, options) => {
     const startedAt = process.hrtime.bigint();
     const directory = toPackDirectory(vaultPath);
     await mkdir(directory, { recursive: true });
@@ -230,7 +295,7 @@ const writeRowsAsPrivatePacks = async (vaultPath, rows, clearExisting) => {
             .filter((name) => name.endsWith('.blpk') || name.endsWith('.jsonl.gz') || name === manifestFileName)
             .map((name) => rm(join(directory, name), { force: true })));
     }
-    const chunks = chunkRows(rows, rowChunkSize);
+    const chunks = chunkRows(rows, options.rowChunkSize);
     const packIndex = [];
     let inputBytes = 0;
     let outputBytes = 0;
@@ -238,7 +303,10 @@ const writeRowsAsPrivatePacks = async (vaultPath, rows, clearExisting) => {
         const chunk = chunks[index];
         const fileName = `pack-${String(index + 1).padStart(4, '0')}.blpk`;
         const serialized = `${chunk.map((row) => JSON.stringify(row)).join('\n')}\n`;
-        const compressed = await encodePrivatePack(vaultPath, Buffer.from(serialized, 'utf8'));
+        const compressed = await encodePrivatePack(vaultPath, Buffer.from(serialized, 'utf8'), {
+            compressionLevel: options.compressionLevel,
+            useDictionary: options.useDictionary
+        });
         const tokenBloomB64 = bloomToBase64(bloomFromRows(chunk));
         await writeFile(join(directory, fileName), compressed);
         inputBytes += Buffer.byteLength(serialized, 'utf8');
@@ -256,7 +324,18 @@ const writeRowsAsPrivatePacks = async (vaultPath, rows, clearExisting) => {
         packCount: chunks.length,
         recordCount: rows.length,
         format: 'private-v2',
-        packIndex
+        packIndex,
+        packConfig: {
+            rowChunkSize: options.rowChunkSize,
+            compressionLevel: options.compressionLevel,
+            useDictionary: options.useDictionary
+        },
+        compression: {
+            inputBytes,
+            outputBytes,
+            ratio: outputBytes / Math.max(inputBytes, 1),
+            savedBytes: Math.max(inputBytes - outputBytes, 0)
+        }
     });
     const durationMs = Number(process.hrtime.bigint() - startedAt) / 1_000_000;
     const safeInput = Math.max(inputBytes, 1);
@@ -305,8 +384,13 @@ const selectCandidatePackFiles = async (vaultPath, tokens, agentId) => {
     }
     return byAgent.length > 0 ? byAgent.map((entry) => entry.fileName) : allFiles;
 };
-export const buildSearchPacks = async (vaultPath, documents) => {
-    return writeRowsAsPrivatePacks(vaultPath, toRows(documents), true);
+export const buildSearchPacks = async (vaultPath, documents, options) => {
+    const resolvedOptions = {
+        rowChunkSize: options?.rowChunkSize ?? defaultBuildOptions.rowChunkSize,
+        compressionLevel: options?.compressionLevel ?? defaultBuildOptions.compressionLevel,
+        useDictionary: options?.useDictionary ?? defaultBuildOptions.useDictionary
+    };
+    return writeRowsAsPrivatePacks(vaultPath, toRows(documents), true, resolvedOptions);
 };
 export const ensurePrivatePacksFromLegacyIndex = async (vaultPath) => {
     const files = await sortedPackFiles(vaultPath);
@@ -320,7 +404,7 @@ export const ensurePrivatePacksFromLegacyIndex = async (vaultPath) => {
             const parsed = await parseRowsFromPack(vaultPath, await readFile(join(toPackDirectory(vaultPath), file)));
             rows.push(...parsed);
         }
-        const report = await writeRowsAsPrivatePacks(vaultPath, rows, true);
+        const report = await writeRowsAsPrivatePacks(vaultPath, rows, true, defaultBuildOptions);
         return {
             imported: true,
             source: 'legacy-packs',
@@ -329,6 +413,11 @@ export const ensurePrivatePacksFromLegacyIndex = async (vaultPath) => {
     }
     return { imported: false };
 };
+export const toSearchPackBuildOptions = (config) => ({
+    rowChunkSize: config.searchPack.rowChunkSize,
+    compressionLevel: config.searchPack.compressionLevel,
+    useDictionary: config.searchPack.useDictionary
+});
 export const searchInPacks = async (vaultPath, query, limit, agentId) => {
     const normalizedAgent = agentId?.trim();
     const tokens = tokenize(query);

package/docs/AGENT_USAGE.md

@@ -52,6 +52,8 @@ Use `blink config where` and `blink config doctor` to inspect active paths and e
 
 You can also set `defaultAgent` in `brainlink.config.json` / `.brainlink.json` (for example `"defaultAgent": "coding-agent"`). When set, CLI commands and MCP calls reuse it when `--agent`/`agent` is not passed.
 You can set `agentProfiles` to define per-agent defaults for `defaultSearchMode`, `defaultSearchLimit` and `defaultContextTokens`.
+You can tune search-pack compression with `searchPack.rowChunkSize`, `searchPack.compressionLevel` and `searchPack.useDictionary`.
+Guardrails for benchmark acceptance are configured with `searchPack.guardrailMinSavingsPercent` and `searchPack.guardrailMaxLatencyRegressionPercent`.
 
 `autoIndexOnWrite` (default: `true`) controls whether `add` and MCP write tools index right after writing.
 
@@ -492,8 +494,21 @@ blink bench --vault ./vault --json
 - elapsed time and changed document count
 - pack rebuild status and reason
 - pack compression metrics (`inputBytes`, `outputBytes`, ratio/saved percentage)
+- objective guardrails (`guardrailMinSavingsPercent`, `guardrailMaxLatencyRegressionPercent`)
 
 Use `--watch` for continuous benchmark runs while editing notes. Watch mode is supported only for local filesystem vaults.
+If pack manifest metadata is missing but encrypted `.blpk` files are present, Brainlink repairs manifest metadata before deciding rebuild policy to avoid unnecessary full repacks on small updates.
+
+### Create Offline Pack Backup
+
+```bash
+blink pack-backup --vault ./vault
+blink pack-backup --vault ./vault --output ./vault/.brainlink/backups/custom.blpkbak.gz
+blink pack-backup --vault ./vault --json
+```
+
+`pack-backup` creates an offline artifact with second-stage compression on top of encrypted `.blpk` packs.
+This is outside the online retrieval path (`index`, `search`, `context`), which keeps a single compression stage.
 
 ### Search Knowledge
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@andespindola/brainlink",
-  "version": "0.1.0-beta.43",
+  "version": "0.1.0-beta.44",
   "description": "Local-first knowledge memory for agents with Markdown, backlinks, indexing and context retrieval.",
   "type": "module",
   "license": "MIT",