@bod.ee/db 0.12.2 → 0.12.6

package/.claude/skills/developing-bod-db.md

@@ -86,7 +86,7 @@ Push paths are append-only logs. `StreamEngine` adds consumer group offsets (`_s
 `MQEngine` owns all MQ SQL via `storage.db.prepare()` — same pattern as StreamEngine. Columns: `mq_status` (pending/inflight), `mq_inflight_until` (Unix ms), `mq_delivery_count`. `fetch()` uses SQLite transaction with TOCTOU guard (`changes > 0`). Ack = DELETE. Sweep reclaims expired inflight; exhausted messages move to DLQ at `<queue>/_dlq/<key>`. Per-queue options via longest prefix match on `queues` config.
 
 ### Replication
-`ReplicationEngine` — primary/replica + multi-source feed subscriptions via `_repl` stream. Primary: `onWrite` hooks emit events to `_repl` stream (updates flattened to per-path sets). Replica: bootstraps via `streamMaterialize`, subscribes for ongoing events, proxies writes to primary. Guards: `_replaying` prevents re-emission, `_emitting` prevents recursion from `db.push('_repl')`. Sweep deletes are replicated. Transport checks `isReplica` and forwards write ops.
+`ReplicationEngine` — primary/replica + multi-source feed subscriptions via `_repl` stream. Primary: `onWrite` hooks emit events to `_repl` stream (updates flattened to per-path sets). Auto-compact on write threshold (`autoCompactThreshold`, default 500) + on startup keeps `_repl` bounded. Replica: bootstraps via cursor-based `streamMaterialize` pagination (`batchSize: 200`), subscribes for ongoing events, proxies writes to primary. `bootstrapFromStream()` helper handles all 3 bootstrap sites (replica, router-based, sources). Guards: `_replaying` prevents re-emission, `_emitting` prevents recursion from `db.push('_repl')`. Sweep deletes are replicated. Transport checks `isReplica` and forwards write ops.
 
 **Sources** (`ReplicationSource[]`): independent of role. Each source creates a `BodClient`, bootstraps filtered `_repl` snapshot, subscribes for ongoing events. `matchesSourcePaths()` filters by path prefix. `remapPath()` prepends `localPrefix`. Events applied with `_replaying=true`. Sources connect via `Promise.allSettled` — individual failures logged, others continue. Deterministic `groupId` default: `source_${url}_${paths.join('+')}`.
 

package/.claude/skills/using-bod-db.md

@@ -27,6 +27,12 @@ import { BodDB, increment, serverTimestamp, arrayUnion, arrayRemove, ref } from
 const db = new BodDB({
   path: './data.db',       // SQLite file (default: ':memory:')
   port: 4400,              // optional — only needed if calling db.serve()
+  log: {                   // optional structured logging
+    enabled: true,
+    level: 'info',         // 'debug' | 'info' | 'warn' | 'error'
+    components: '*',       // '*' or ['storage', 'transport', 'subs', 'replication', 'keyauth']
+    logDir: './logs',      // writes bod-YYYYMMDD-HHmmss.log per run; omit for console-only
+  },
   rules: {                 // inline rules, or path to .json/.ts file
     'users/$uid': {
       read: true,
@@ -301,6 +307,11 @@ ws.send(JSON.stringify({ id: '20', op: 'batch-sub', subscriptions: [
 // Stream extended ops
 ws.send(JSON.stringify({ id: '21', op: 'stream-snapshot', path: 'events/orders' }));
 ws.send(JSON.stringify({ id: '21', op: 'stream-materialize', path: 'events/orders', keepKey: 'orderId' }));
+// Cursor-based pagination (for large streams):
+ws.send(JSON.stringify({ id: '21b', op: 'stream-materialize', path: 'events/orders', keepKey: 'orderId', batchSize: 200 }));
+// → { id: '21b', ok: true, data: { data: {...}, nextCursor: 'abc123' } }
+// Follow-up page:
+ws.send(JSON.stringify({ id: '21c', op: 'stream-materialize', path: 'events/orders', keepKey: 'orderId', batchSize: 200, cursor: 'abc123' }));
 ws.send(JSON.stringify({ id: '22', op: 'stream-compact', path: 'events/orders', maxAge: 86400 }));
 ws.send(JSON.stringify({ id: '23', op: 'stream-reset', path: 'events/orders' }));
 ```
@@ -450,6 +461,13 @@ const similar = await client.vectorSearch({ query: [0.1, 0.2, 0.3], path: 'docs'
 // Stream snapshot, materialize, compact, reset
 const snap = await client.streamSnapshot('events/orders');
 const view = await client.streamMaterialize('events/orders', { keepKey: 'orderId' });
+// Cursor-based materialize for large streams (avoids huge single response):
+let cursor: string | undefined;
+do {
+  const page = await client.streamMaterialize('events/orders', { keepKey: 'orderId', batchSize: 200, cursor });
+  // page.data contains this batch, page.nextCursor is undefined when done
+  cursor = page.nextCursor;
+} while (cursor);
 await client.streamCompact('events/orders', { maxAge: 86400 });
 await client.streamReset('events/orders');
 ```
@@ -485,7 +503,7 @@ await replica.replication!.start();
 - Reads served locally on replica, writes forwarded to primary
 - Replicas bootstrap full state on first connect, then consume live events
 - Push keys preserved across replicas (deterministic)
-- Auto-compaction on `_repl` stream (keepKey: 'path', maxCount: 10000)
+- Auto-compaction on `_repl` stream (keepKey: 'path', maxCount: 500, auto-compact every 500 writes)
 - Excluded prefixes: `_repl`, `_streams`, `_mq`, `_auth` (internal data not replicated)
 
 ### Per-Path Topology

package/CLAUDE.md

@@ -71,7 +71,7 @@ config.ts                  — demo instance config (open rules, indexes, fts, v
 - **BodClientCached**: two-tier cache wrapper around BodClient. Memory (Map, LRU eviction) + IndexedDB persistence. Stale-while-revalidate: subscribed paths always fresh, unsubscribed return stale + background refetch. Writes (`set/update/delete`) invalidate path + ancestors. `init()` opens IDB + sweeps expired. `warmup(paths[])` bulk-loads from IDB. Passthrough for `push/batch/query/search/mq/stream/vfs` via `cachedClient.client`.
 - **MCP**: `MCPAdapter` wraps a `BodClient` as a JSON-RPC MCP server (stdio + HTTP). Connects to a running BodDB instance over WebSocket — no embedded DB. Entry point: `mcp.ts`. Tools: CRUD (6), FTS (2), vectors (2), streams (4), MQ (7) = 21 tools. Use `--stdio` for Claude Code/Desktop, `--http` for remote agents.
 - **VFS (Virtual File System)**: `VFSEngine` — files stored outside SQLite via pluggable `VFSBackend` interface. `LocalBackend` stores at `<storageRoot>/<fileId>` using `Bun.file`/`Bun.write`. Metadata at `_vfs/<virtualPath>/` (size, mime, mtime, fileId, isDir) — gets subs/rules/replication for free. `fileId = pushId` so move/rename is metadata-only. REST: `POST/GET/DELETE /files/<path>`, `?stat=1`, `?list=1`, `?mkdir=1`, `PUT ?move=<dst>`. WS chunked fallback: base64-encoded `vfs-upload-init/chunk/done`, `vfs-download-init` → `vfs-download-chunk` push messages. Client: `VFSClient` via `client.vfs()` — `upload/download` (REST) + `uploadWS/downloadWS` (WS) + `stat/list/mkdir/delete/move`.
-- **Replication**: `ReplicationEngine` — single primary + N read replicas + multi-source feed subscriptions. Star topology. Primary emits write events to `_repl` stream via `onWrite` hooks. Replicas bootstrap via `streamMaterialize('_repl', { keepKey: 'path' })`, then subscribe for ongoing events. Write proxy: replica forwards writes to primary via BodClient, primary applies + emits, replica consumes. `_replaying` flag prevents re-emission loops. `_emitting` guard prevents recursion from `db.push('_repl')`. Updates flattened to per-path set events for correct compaction keying. Sweep delete events replicated. Excluded prefixes: `_repl`, `_streams`, `_mq`, `_auth`. **Sources**: `ReplicationSource[]` — subscribe to specific paths from multiple remote DBs. Each source is an independent BodClient that filters `_repl` events by path prefix, with optional `localPrefix` remapping (e.g. remote `users/u1` → local `db-a/users/u1`). Sources connect in parallel; individual failures don't block others. Sources are independent of role — a DB can be primary AND consume sources. **Per-path topology**: `PathTopologyRouter` — when `paths` config is set, each path prefix gets an independent mode: `primary` (local authoritative, emits), `replica` (remote authoritative, proxies writes), `sync` (bidirectional, both emit+apply), `readonly` (pull-only, rejects writes), `writeonly` (push-only, ignores remote). Longest-prefix match resolves mode. `writeProxy: 'proxy'|'reject'` overrides replica write behavior. Bootstrap skips sync paths (ongoing stream only). Auth/rules checked before proxy in all handlers. `shouldProxyPath(path)`/`shouldRejectPath(path)` replace `isReplica` checks. `emitsToRepl`/`pullsFromPrimary` getters for compact/bootstrap decisions. Stable `replicaId` from config hash. Falls back to `role` when `paths` absent (backward compat).
+- **Replication**: `ReplicationEngine` — single primary + N read replicas + multi-source feed subscriptions. Star topology. Primary emits write events to `_repl` stream via `onWrite` hooks. Replicas bootstrap via cursor-based `streamMaterialize('_repl', { keepKey: 'path', batchSize: 200 })` pagination (avoids huge single WS frame), then subscribe for ongoing events. Auto-compact on write threshold (`autoCompactThreshold`, default 500) + on startup keeps `_repl` bounded. Write proxy: replica forwards writes to primary via BodClient, primary applies + emits, replica consumes. `_replaying` flag prevents re-emission loops. `_emitting` guard prevents recursion from `db.push('_repl')`. Updates flattened to per-path set events for correct compaction keying. Sweep delete events replicated. Excluded prefixes: `_repl`, `_streams`, `_mq`, `_admin`, `_auth`. **Sources**: `ReplicationSource[]` — subscribe to specific paths from multiple remote DBs. Each source is an independent BodClient that filters `_repl` events by path prefix, with optional `localPrefix` remapping (e.g. remote `users/u1` → local `db-a/users/u1`). Sources connect in parallel; individual failures don't block others. Sources are independent of role — a DB can be primary AND consume sources. **Per-path topology**: `PathTopologyRouter` — when `paths` config is set, each path prefix gets an independent mode: `primary` (local authoritative, emits), `replica` (remote authoritative, proxies writes), `sync` (bidirectional, both emit+apply), `readonly` (pull-only, rejects writes), `writeonly` (push-only, ignores remote). Longest-prefix match resolves mode. `writeProxy: 'proxy'|'reject'` overrides replica write behavior. Bootstrap skips sync paths (ongoing stream only). Auth/rules checked before proxy in all handlers. `shouldProxyPath(path)`/`shouldRejectPath(path)` replace `isReplica` checks. `emitsToRepl`/`pullsFromPrimary` getters for compact/bootstrap decisions. Stable `replicaId` from config hash. Falls back to `role` when `paths` absent (backward compat).
 - **KeyAuth integration guide**: `docs/keyauth-integration.md` — flows for signup, signin, new device, autoAuth, IAM roles, common mistakes.
 - **Para-chat integration guide**: `docs/para-chat-integration.md` — how para-chat uses BodDB: per-path topology, VFS, KeyAuth, caching, file sync.
 - **KeyAuth**: `KeyAuthEngine` — portable Ed25519 identity & IAM. Identity hierarchy: Root (server-level, key on filesystem), Account (portable, password-encrypted private key in DB or device-generated), Device (delegate, linked via password unlock). Challenge-response auth: server sends nonce → client signs with Ed25519 → server verifies + creates session. Self-signed tokens (no JWT lib): `base64url(payload).base64url(Ed25519_sign)`. Data model at `_auth/` prefix (protected from external writes). Device reverse-index at `_auth/deviceIndex/{dfp}` for O(1) lookup. Password change is atomic (single `db.update()`). IAM: roles with path-based permissions, account role assignment. `_auth/` excluded from replication. Transport guards: `auth-link-device` and `auth-change-password` require authenticated session; non-root users can only change own password. **Device registration**: `registerDevice(publicKey)` — client-generated keypair, no password, idempotent; `allowOpenRegistration: false` requires authenticated session. **Browser crypto**: `keyAuth.browser.ts` uses `@noble/ed25519` with DER↔raw key bridge for server compatibility. **BodClient autoAuth**: `autoAuth: true` auto-generates keypair (localStorage), registers, authenticates — zero-config device identity. `client.auth.*` convenience methods for all auth ops. **IAM transport ops**: `auth-create-role`, `auth-delete-role`, `auth-update-roles` (root only), `auth-list-accounts`, `auth-list-roles`. Device accounts (no encrypted key) safely reject `linkDevice`/`changePassword`.

package/admin/ui.html

@@ -8,7 +8,7 @@
   body { font-family: monospace; font-size: 13px; background: #0d0d0d; color: #d4d4d4; display: flex; flex-direction: column; height: 100vh; overflow: hidden; }
 
   /* Metrics bar */
-  #metrics-bar { display: flex; background: #0a0a0a; border-bottom: 1px solid #2a2a2a; flex-shrink: 0; overflow-x: auto; align-items: stretch; }
+  #metrics-bar { display: flex; background: #0a0a0a; border-bottom: 1px solid #2a2a2a; flex-shrink: 0; align-items: stretch; width: 100%; }
   .metric-card { display: flex; flex-direction: column; padding: 5px 10px 4px; border-right: 1px solid #181818; min-width: 140px; flex-shrink: 0; gap: 1px; overflow: hidden; }
   .metric-card:last-child { border-right: none; width: auto; }
   .metric-right { margin-left: auto; }
@@ -127,15 +127,15 @@
     <div class="metric-top"><span class="metric-label">Ping</span><span class="metric-value" id="s-ping">—</span></div>
     <canvas class="metric-canvas" id="g-ping" width="100" height="28"></canvas>
   </div>
-  <div class="metric-card metric-right" id="repl-card" style="border-left:1px solid #282828;display:none;width:180px">
+  <div class="metric-card" id="repl-card" style="border-left:1px solid #282828;display:none;width:180px">
     <div class="metric-top"><span class="metric-label">Replication</span><span class="metric-value dim" id="s-repl-role">—</span></div>
     <div style="margin-top:4px;font-size:10px" id="s-repl-sources"></div>
   </div>
-  <div class="metric-card" style="border-left:1px solid #282828">
+  <div class="metric-card metric-right" style="border-left:1px solid #282828;justify-content:space-between">
     <div class="metric-top"><span class="metric-label">Uptime</span><span class="metric-value dim" id="s-uptime">—</span></div>
-    <div style="margin-top:4px;font-size:10px;color:#555" id="s-ts">—</div>
-    <div style="margin-top:4px"><span class="metric-label">WS<span id="ws-dot"></span></span> <span style="font-size:10px;color:#555"><span id="s-clients">0</span> clients · <span id="s-subs">0</span> subs</span></div>
-    <div style="margin-top:6px"><button id="stats-toggle" class="sm" onclick="toggleStats()" title="Toggle server stats collection">Stats: ON</button></div>
+    <div style="font-size:10px;color:#555;display:flex;justify-content:space-between"><span id="s-ts">—</span><span>v<span id="s-version">—</span></span></div>
+    <div><span class="metric-label">WS<span id="ws-dot"></span></span> <span style="font-size:10px;color:#555"><span id="s-clients">0</span> clients · <span id="s-subs">0</span> subs</span></div>
+    <div><button id="stats-toggle" class="sm" onclick="toggleStats()" title="Toggle server stats collection">Stats: ON</button></div>
   </div>
 </div>
 
@@ -1257,6 +1257,7 @@ db.on('_admin/stats', (snap) => {
   document.getElementById('s-subs').textContent = s.subs ?? 0;
   document.getElementById('s-uptime').textContent = fmtUptime(s.process.uptimeSec);
   document.getElementById('s-ts').textContent = new Date(s.ts).toLocaleTimeString();
+  if (s.version) document.getElementById('s-version').textContent = s.version;
 
   // Replication stats
   if (s.repl) {

package/docs/para-chat-integration.md

@@ -68,12 +68,13 @@ const db = new BodDB({
     role: 'primary',           // fallback for unconfigured paths
     primaryUrl: repl.remoteUrl, // wss://bod.ee/db
     paths: [
-      { path: '_vfs', mode: 'primary' },      // local files are authoritative
-      { path: '_auth', mode: 'replica' },      // bod.ee is auth authority
-      { path: 'config', mode: 'sync' },        // bidirectional app config
-      { path: 'storage', mode: 'primary' },    // local collections (notifications, etc.)
+      { path: '_vfs', mode: 'primary' },          // local files are authoritative
+      { path: '_auth', mode: 'replica' },          // bod.ee is auth authority
+      { path: '_auth/sessions', mode: 'primary' }, // sessions are local (longest-prefix wins)
+      { path: '_auth/server', mode: 'primary' },   // server keypair is local
+      { path: 'config', mode: 'sync' },            // bidirectional app config
+      { path: 'storage', mode: 'primary' },        // local collections (notifications, etc.)
     ],
-    excludePrefixes: ['_repl', '_streams', '_mq'],
   },
 });
 ```
@@ -93,7 +94,7 @@ const db = new BodDB({
 1. **`_auth` writes are proxied** — `createAccount`, `linkDevice` go through bod.ee automatically. No separate HTTP call needed.
 2. **`_vfs` emits to remote** — file metadata changes push to bod.ee via `_repl` stream. No manual upload sync.
 3. **Bootstrap is selective** — only `_auth` (replica) pulls from remote on connect. `_vfs` (primary) and `storage` (primary) keep local state.
-4. **`_auth/sessions` and `_auth/server`** — automatically excluded from replication (`_auth` prefix excluded by default; replica mode pulls from remote but these internal paths are local-only).
+4. **`_auth/sessions` and `_auth/server`** — kept local via explicit sub-path overrides (`mode: 'primary'`). Longest-prefix match ensures `_auth/sessions/x` resolves to `primary` even though `_auth` is `replica`.
 
 ---
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bod.ee/db",
-  "version": "0.12.2",
+  "version": "0.12.6",
   "module": "index.ts",
   "type": "module",
   "exports": {

package/src/client/BodClient.ts

@@ -203,9 +203,11 @@ export class BodClient {
           this.scheduleReconnect();
         }
 
-        if (!this.connectPromise) return;
-        this.connectPromise = null;
-        // Only reject if this was the initial connect
+        // Reject initial connect promise so callers don't hang forever
+        if (this.connectPromise) {
+          this.connectPromise = null;
+          reject(new Error(`WebSocket connection failed (${this.options.url})`));
+        }
       };
 
       ws.onerror = () => {
@@ -456,8 +458,10 @@ export class BodClient {
     return this.send('stream-snapshot', { path });
   }
 
-  async streamMaterialize(path: string, opts?: { keepKey?: string }): Promise<Record<string, unknown>> {
-    return this.send('stream-materialize', { path, keepKey: opts?.keepKey }) as Promise<Record<string, unknown>>;
+  async streamMaterialize(path: string, opts?: { keepKey?: string }): Promise<Record<string, unknown>>;
+  async streamMaterialize(path: string, opts: { keepKey?: string; batchSize: number; cursor?: string }): Promise<{ data: Record<string, unknown>; nextCursor?: string }>;
+  async streamMaterialize(path: string, opts?: { keepKey?: string; batchSize?: number; cursor?: string }): Promise<Record<string, unknown> | { data: Record<string, unknown>; nextCursor?: string }> {
+    return this.send('stream-materialize', { path, ...opts }) as any;
   }
 
   async streamCompact(path: string, opts?: { maxAge?: number; maxCount?: number; keepKey?: string }): Promise<unknown> {

package/src/server/BodDB.ts

@@ -12,6 +12,8 @@ import { VFSEngine, type VFSEngineOptions } from './VFSEngine.ts';
 import { KeyAuthEngine, type KeyAuthEngineOptions } from './KeyAuthEngine.ts';
 import { validatePath } from '../shared/pathUtils.ts';
 import { Logger, type LogConfig } from '../shared/logger.ts';
+import pkg from '../../package.json' with { type: 'json' };
+const PKG_VERSION: string = pkg.version ?? 'unknown';
 
 export interface TransactionProxy {
   get(path: string): unknown;
@@ -79,7 +81,8 @@ export class BodDB {
     this.options = { ...new BodDBOptions(), ...options };
     this.log = new Logger(this.options.log);
     const _log = this.log.forComponent('db');
-    _log.info(`Initializing BodDB (path: ${this.options.path})`);
+    console.log(`[BodDB] v${PKG_VERSION} (path: ${this.options.path})`);
+    _log.info(`Initializing BodDB v${PKG_VERSION} (path: ${this.options.path})`);
     this.storage = new StorageEngine({ path: this.options.path });
     this.subs = new SubscriptionEngine();
     this.stream = new StreamEngine(this.storage, this.subs, { compact: this.options.compact });
@@ -409,6 +412,7 @@ export class BodDB {
 
     // Reuse a single stats object to minimize allocations
     const statsData: Record<string, unknown> = {
+      version: PKG_VERSION,
       process: {}, db: {}, system: {},
       subs: 0, clients: 0, repl: null, ts: 0,
     };
@@ -537,6 +541,7 @@ export class BodDB {
     this.stop();
     this.subs.clear();
     this.storage.close();
+    this.log.close();
   }
 
   private snapshotExisting(path: string): Set<string> {

package/src/server/ReplicationEngine.ts

@@ -128,11 +128,15 @@ export class ReplicationOptions {
   /** Bootstrap replica from primary's full state before applying _repl stream */
   fullBootstrap: boolean = true;
   compact?: CompactOptions;
+  /** Auto-compact _repl after this many emitted writes (0 = disabled, default 500) */
+  autoCompactThreshold: number = 500;
   sources?: ReplicationSource[];
   /** Per-path topology: strings default to 'sync', objects specify mode. When absent, role governs all paths. */
   paths?: Array<string | PathTopology>;
 }
 
+const BOOTSTRAP_BATCH_SIZE = 200;
+
 type ProxyableMessage = Extract<ClientMessage, { op: 'set' | 'delete' | 'update' | 'push' | 'batch' }>;
 
 export class ReplicationEngine {
@@ -145,6 +149,7 @@ export class ReplicationEngine {
   private _started = false;
   private _seq = 0;
   private _emitting = false;
+  private _emitCount = 0;
   private _pendingReplEvents: WriteEvent[] | null = null;
   private log: ComponentLogger;
 
@@ -251,7 +256,7 @@ export class ReplicationEngine {
   /** Stop replication */
   stop(): void {
     this._started = false;
-    if (this._compactTimer) { clearInterval(this._compactTimer); this._compactTimer = null; }
+    this._emitCount = 0;
     this.unsubWrite?.();
     this.unsubWrite = null;
     this.unsubStream?.();
@@ -263,6 +268,7 @@ export class ReplicationEngine {
       sc.client.disconnect();
     }
     this.sourceConns = [];
+    this._pendingReplEvents = null;
   }
 
   /** Proxy a write operation to the primary (replica mode) */
@@ -289,22 +295,18 @@ export class ReplicationEngine {
 
   // --- Primary mode ---
 
-  private _compactTimer: ReturnType<typeof setInterval> | null = null;
-
   private startPrimary(): void {
     this.unsubWrite = this.db.onWrite((ev: WriteEvent) => {
-      this.emit(ev);
+      // Defer emit — onWrite fires inside the SQLite write transaction,
+      // and db.push('_repl') inside that transaction gets silently dropped.
+      const evCopy = { ...ev };
+      setTimeout(() => { if (this._started) this.emit(evCopy); }, 0);
     });
 
-    // Auto-compact _repl stream to prevent unbounded growth
+    // Compact on startup
     const compact = this.options.compact ?? { maxCount: 500, keepKey: 'path' };
     if (compact.maxCount || compact.maxAge) {
-      // Compact on startup
       try { this.db.stream.compact('_repl', compact); } catch {}
-      // Then periodically (every 5 minutes)
-      this._compactTimer = setInterval(() => {
-        try { this.db.stream.compact('_repl', compact); } catch {}
-      }, 5 * 60_000);
     }
   }
 
@@ -322,22 +324,29 @@ export class ReplicationEngine {
 
   /** Buffer replication events during transactions, emit immediately otherwise */
   private emit(ev: WriteEvent): void {
-    if (this._emitting) return;
+    if (this._emitting) {
+      this.log.debug('emit: skipped (re-entrant)', { path: ev.path });
+      return;
+    }
     if (this.router) {
-      // Single resolve: check exclusion override + shouldEmit together
       const resolved = this.router.resolve(ev.path);
       const isConfigured = resolved.path !== '';
-      // If in excludePrefixes but not explicitly configured, skip
-      if (!isConfigured && this.options.excludePrefixes.some(p => ev.path.startsWith(p))) return;
+      if (!isConfigured && this.options.excludePrefixes.some(p => ev.path.startsWith(p))) {
+        this.log.debug('emit: skipped (excluded, not configured)', { path: ev.path });
+        return;
+      }
       const mode = resolved.mode;
-      if (mode !== 'primary' && mode !== 'sync' && mode !== 'writeonly') return;
+      if (mode !== 'primary' && mode !== 'sync' && mode !== 'writeonly') {
+        this.log.debug('emit: skipped (mode)', { path: ev.path, mode });
+        return;
+      }
     } else {
       if (this.isExcluded(ev.path)) return;
     }
 
-    // If buffering (transaction in progress), collect events
     if (this._pendingReplEvents) {
       this._pendingReplEvents.push(ev);
+      this.log.debug('emit: buffered (batch)', { path: ev.path });
       return;
     }
 
@@ -351,9 +360,20 @@ export class ReplicationEngine {
       const seq = this._seq++;
       const idempotencyKey = `${replEvent.ts}:${seq}:${ev.path}`;
       this.db.push('_repl', replEvent, { idempotencyKey });
+      this.log.info('_repl emit', { seq, op: ev.op, path: ev.path });
+    } catch (e: any) {
+      this.log.error('_repl emit failed', { path: ev.path, error: e.message });
     } finally {
       this._emitting = false;
     }
+
+    this._emitCount++;
+    const threshold = this.options.autoCompactThreshold;
+    if (threshold > 0 && this._emitCount >= threshold) {
+      this._emitCount = 0;
+      const compact = this.options.compact ?? { maxCount: 500, keepKey: 'path' };
+      try { this.db.stream.compact('_repl', compact); } catch {}
+    }
   }
 
   /** Start buffering replication events (call before transaction) */
@@ -394,30 +414,22 @@ export class ReplicationEngine {
       await this.bootstrapFullState(bootstrapPaths);
     }
 
-    // Stream bootstrap filtered
-    const snapshot = await this.client!.streamMaterialize('_repl', { keepKey: 'path' });
-    if (snapshot) {
-      this.db.setReplaying(true);
-      try {
-        for (const [, event] of Object.entries(snapshot)) {
-          const ev = event as ReplEvent;
-          if (this.matchesPathPrefixes(ev.path, pathPrefixes)) {
-            this.applyEvent(ev);
-          }
-        }
-      } finally {
-        this.db.setReplaying(false);
-      }
-    }
+    // Stream bootstrap filtered (cursor-based to avoid huge single response)
+    const applied = await this.bootstrapFromStream(this.client!, { filter: ev => this.matchesPathPrefixes(ev.path, pathPrefixes) });
+    this.log.info(`Stream bootstrap (paths): ${applied} events applied`);
 
     // Subscribe to ongoing events, filter by paths
     const groupId = this.options.replicaId!;
+    this.log.info('Subscribing to _repl stream', { groupId, pathPrefixes });
     this.unsubStream = this.client.stream('_repl', groupId).on((events) => {
+      this.log.info('_repl events received', { count: events.length });
       this.db.setReplaying(true);
       try {
         for (const e of events) {
           const ev = e.val() as ReplEvent;
-          if (this.matchesPathPrefixes(ev.path, pathPrefixes)) {
+          const matched = this.matchesPathPrefixes(ev.path, pathPrefixes);
+          this.log.info('_repl event', { op: ev.op, path: ev.path, matched });
+          if (matched) {
             this.applyEvent(ev);
           }
           this.client!.stream('_repl', groupId).ack(e.key).catch(() => {});
@@ -449,20 +461,9 @@ export class ReplicationEngine {
       await this.bootstrapFullState();
     }
 
-    // Stream bootstrap: apply _repl events on top (catches recent writes, deduped by idempotent set)
-    const snapshot = await this.client!.streamMaterialize('_repl', { keepKey: 'path' });
-    this.log.info(`Stream bootstrap: ${snapshot ? Object.keys(snapshot).length + ' events' : 'no events'}`);
-    if (snapshot) {
-      this.db.setReplaying(true);
-      try {
-        for (const [, event] of Object.entries(snapshot)) {
-          const ev = event as ReplEvent;
-          this.applyEvent(ev);
-        }
-      } finally {
-        this.db.setReplaying(false);
-      }
-    }
+    // Stream bootstrap: cursor-based to avoid huge single response
+    const applied = await this.bootstrapFromStream(this.client!);
+    this.log.info(`Stream bootstrap: ${applied} events applied`);
 
     // Subscribe to ongoing events
     const groupId = this.options.replicaId!;
@@ -526,21 +527,11 @@ export class ReplicationEngine {
     const client = new BodClient({ url: source.url, auth: source.auth });
     await client.connect();
 
-    // Bootstrap: materialize _repl, filter by source paths
-    const snapshot = await client.streamMaterialize('_repl', { keepKey: 'path' });
-    if (snapshot) {
-      this.db.setReplaying(true);
-      try {
-        for (const [, event] of Object.entries(snapshot)) {
-          const ev = event as ReplEvent;
-          if (this.matchesSourcePaths(ev.path, source)) {
-            this.applyEvent(ev, source);
-          }
-        }
-      } finally {
-        this.db.setReplaying(false);
-      }
-    }
+    // Bootstrap: cursor-based materialize _repl, filter by source paths
+    await this.bootstrapFromStream(client, {
+      filter: ev => this.matchesSourcePaths(ev.path, source),
+      source,
+    });
 
     // Subscribe to ongoing events
     const groupId = source.id || `source_${source.url}_${source.paths.sort().join('+')}`;
@@ -577,10 +568,37 @@ export class ReplicationEngine {
     return source.localPrefix ? `${source.localPrefix}/${path}` : path;
   }
 
+  /** Cursor-based stream bootstrap: pages through _repl materialize to avoid huge single responses */
+  private async bootstrapFromStream(client: BodClient, opts?: { filter?: (ev: ReplEvent) => boolean; source?: ReplicationSource }): Promise<number> {
+    let cursor: string | undefined;
+    let applied = 0;
+    const filter = opts?.filter;
+    const source = opts?.source;
+    this.db.setReplaying(true);
+    try {
+      do {
+        const page = await client.streamMaterialize('_repl', { keepKey: 'path', batchSize: BOOTSTRAP_BATCH_SIZE, cursor });
+        if (page.data) {
+          for (const [, event] of Object.entries(page.data)) {
+            const ev = event as ReplEvent;
+            if (!filter || filter(ev)) {
+              this.applyEvent(ev, source);
+              applied++;
+            }
+          }
+        }
+        cursor = page.nextCursor;
+      } while (cursor);
+    } finally {
+      this.db.setReplaying(false);
+    }
+    return applied;
+  }
+
   private applyEvent(ev: ReplEvent, source?: ReplicationSource): void {
     const path = source ? this.remapPath(ev.path, source) : ev.path;
     // Defense-in-depth: skip events for paths we shouldn't apply (primary/writeonly)
-    if (!source && this.router && !this.router.shouldApply(path)) return;
+    if (this.router && !this.router.shouldApply(path)) return;
     switch (ev.op) {
       case 'set':
         this.db.set(path, ev.value, ev.ttl ? { ttl: ev.ttl } : undefined);

package/src/server/Transport.ts

@@ -515,10 +515,11 @@ export class Transport {
             }
 
             case 'batch': {
-              // Upfront rules check before proxy (defense-in-depth)
+              // Upfront auth prefix + rules check before proxy (defense-in-depth)
               for (const batchOp of msg.operations) {
                 const opPaths = batchOp.op === 'update' ? Object.keys(batchOp.updates) : [batchOp.path];
                 for (const p of opPaths) {
+                  if (guardAuthPrefix(p)) return;
                   if (self.rules && !self.rules.check('write', p, ws.data.auth)) {
                     return error(`Permission denied for ${p}`, Errors.PERMISSION_DENIED);
                   }
@@ -706,7 +707,11 @@ export class Transport {
               if (self.rules && !self.rules.check('read', msg.path, ws.data.auth)) {
                 return error('Permission denied', Errors.PERMISSION_DENIED);
               }
-              return reply(self.db.stream.materialize(msg.path, msg.keepKey ? { keepKey: msg.keepKey } : undefined));
+              const matOpts: { keepKey?: string; batchSize?: number; cursor?: string } = {};
+              if (msg.keepKey) matOpts.keepKey = msg.keepKey;
+              if (msg.batchSize) matOpts.batchSize = msg.batchSize;
+              if (msg.cursor) matOpts.cursor = msg.cursor;
+              return reply(self.db.stream.materialize(msg.path, Object.keys(matOpts).length ? matOpts : undefined));
             }
             case 'stream-compact': {
               if (self.rules && !self.rules.check('write', msg.path, ws.data.auth)) {

package/src/shared/logger.ts

@@ -3,6 +3,9 @@
  * Disabled by default. Enable via `log` option in BodDBOptions.
  */
 
+import { createWriteStream, mkdirSync } from 'fs';
+import type { WriteStream } from 'fs';
+
 export type LogLevel = 'debug' | 'info' | 'warn' | 'error';
 
 export interface LogConfig {
@@ -12,6 +15,12 @@ export interface LogConfig {
   level?: LogLevel;
   /** Enable specific components: ['storage', 'transport', 'subs', 'replication', 'stats', 'keyauth'] or '*' for all */
   components?: string[] | '*';
+  /**
+   * Directory to write log files into. Each run creates a new file named
+   * `bod-YYYYMMDD-HHmmss.log`. If omitted, logs go to console only.
+   * Only active when `enabled: true`.
+   */
+  logDir?: string;
 }
 
 const LEVELS: Record<LogLevel, number> = { debug: 0, info: 1, warn: 2, error: 3 };
@@ -21,11 +30,31 @@ export class Logger {
   private minLevel: number;
   private components: Set<string> | '*';
   private _cache = new Map<string, ComponentLogger>();
+  private _stream: WriteStream | null = null;
+  private _streamDead = false;
 
   constructor(config?: LogConfig) {
     this.enabled = config?.enabled ?? false;
     this.minLevel = LEVELS[config?.level ?? 'info'];
     this.components = config?.components === '*' ? '*' : new Set(config?.components ?? []);
+    if (this.enabled && config?.logDir) {
+      try {
+        mkdirSync(config.logDir, { recursive: true });
+        const d = new Date();
+        const ts = `${d.getFullYear()}${pad(d.getMonth() + 1)}${pad(d.getDate())}-${pad(d.getHours())}${pad(d.getMinutes())}${pad(d.getSeconds())}`;
+        const file = `${config.logDir}/bod-${ts}.log`;
+        this._stream = createWriteStream(file, { flags: 'a' });
+        this._stream.on('error', (err) => {
+          if (!this._streamDead) {
+            console.error(`[BodDB logger] File write error, disabling disk logging: ${err.message}`);
+            this._streamDead = true;
+            this._stream = null;
+          }
+        });
+      } catch (err: any) {
+        console.error(`[BodDB logger] Failed to open log file in "${config.logDir}": ${err.message}`);
+      }
+    }
   }
 
   forComponent(name: string): ComponentLogger {
@@ -50,9 +79,20 @@ export class Logger {
     } else {
       console[level === 'debug' ? 'log' : level](prefix, msg);
     }
+    if (this._stream && !this._streamDead) {
+      const dataPart = data !== undefined ? ' ' + JSON.stringify(data) : '';
+      this._stream.write(`${prefix} ${msg}${dataPart}\n`);
+    }
+  }
+
+  close(): void {
+    this._stream?.end();
+    this._stream = null;
   }
 }
 
+function pad(n: number): string { return n.toString().padStart(2, '0'); }
+
 export class ComponentLogger {
   readonly isDebug: boolean;
   constructor(private logger: Logger, private component: string) {

package/src/shared/protocol.ts

@@ -26,7 +26,7 @@ export type ClientMessage =
   | { id: string; op: 'vector-search'; query: number[]; path?: string; limit?: number; threshold?: number }
   | { id: string; op: 'vector-store'; path: string; embedding: number[] }
   | { id: string; op: 'stream-snapshot'; path: string }
-  | { id: string; op: 'stream-materialize'; path: string; keepKey?: string }
+  | { id: string; op: 'stream-materialize'; path: string; keepKey?: string; batchSize?: number; cursor?: string }
   | { id: string; op: 'stream-compact'; path: string; maxAge?: number; maxCount?: number; keepKey?: string }
   | { id: string; op: 'stream-reset'; path: string }
   // VFS ops

package/tests/optimization.test.ts

@@ -325,7 +325,7 @@ describe('B5: Cursor-Based Stream Materialize', () => {
 });
 
 describe('B6: Batched Replication Events', () => {
-  test('transaction batches repl events', () => {
+  test('transaction batches repl events', async () => {
     const db = new BodDB({ path: ':memory:', replication: { role: 'primary' } });
     db.replication!['_started'] = true;
     db.replication!['startPrimary']();
@@ -336,6 +336,7 @@ describe('B6: Batched Replication Events', () => {
       tx.set('b', 2);
       tx.set('c', 3);
     });
+    await new Promise(r => setTimeout(r, 50));
 
     // All 3 should have been emitted to _repl
     const replEvents = db.storage.query('_repl');
@@ -344,12 +345,13 @@ describe('B6: Batched Replication Events', () => {
     db.close();
   });
 
-  test('non-transactional writes emit immediately', () => {
+  test('non-transactional writes emit immediately', async () => {
     const db = new BodDB({ path: ':memory:', replication: { role: 'primary' } });
     db.replication!['_started'] = true;
     db.replication!['startPrimary']();
 
     db.set('x', 'val');
+    await new Promise(r => setTimeout(r, 50));
     const replEvents = db.storage.query('_repl');
     expect(replEvents.length).toBe(1);