@bod.ee/db 0.12.1 → 0.12.2

package/.claude/skills/config-file.md

@@ -47,6 +47,12 @@ export default {
   log: { enabled: true, level: 'info', components: ['transport', 'stats', 'storage'] }, // or components: '*' for all
   replication: {
     role: 'primary',
+    primaryUrl: 'ws://remote:4400', // needed for replica/sync paths
+    paths: [
+      { path: '_vfs', mode: 'primary' },
+      { path: '_auth', mode: 'replica' },
+      { path: 'config', mode: 'sync' },
+    ],
     sources: [
       { url: 'ws://other-db:4400', paths: ['catalog'], localPrefix: 'ext', id: 'my-source' },
     ],

package/.claude/skills/deploying-bod-db.md

@@ -44,6 +44,27 @@ export default {
 ```
 Primary config just needs `replication: { role: 'primary' }`. Replicas auto-bootstrap on startup.
 
+## Per-Path Topology
+Mixed ownership — each path prefix gets independent replication mode:
+```typescript
+// deploy/prod-edge.config.ts
+export default {
+  path: './data-edge.db',
+  port: 4401,
+  replication: {
+    role: 'primary',
+    primaryUrl: 'ws://central:4400',
+    paths: [
+      { path: '_vfs', mode: 'primary' },   // local files → push to central
+      { path: '_auth', mode: 'replica' },   // auth from central → pull
+      { path: 'config', mode: 'sync' },     // bidirectional
+      { path: 'telemetry', mode: 'writeonly' }, // push metrics, ignore remote
+    ],
+  },
+};
+```
+Modes: `primary` (emit), `replica` (proxy writes), `sync` (both), `readonly` (pull-only), `writeonly` (push-only). Unconfigured paths fall back to `role`.
+
 ## Multi-Source Feed Subscriptions
 Pull specific paths from multiple remote DBs:
 ```typescript

package/.claude/skills/using-bod-db.md

@@ -488,6 +488,36 @@ await replica.replication!.start();
 - Auto-compaction on `_repl` stream (keepKey: 'path', maxCount: 10000)
 - Excluded prefixes: `_repl`, `_streams`, `_mq`, `_auth` (internal data not replicated)
 
+### Per-Path Topology
+
+Mixed ownership per path — some paths local-authoritative, others remote-authoritative.
+
+```typescript
+const db = new BodDB({
+  path: './data.db',
+  replication: {
+    role: 'primary',  // fallback for unconfigured paths
+    primaryUrl: 'ws://remote:4400',
+    paths: [
+      { path: '_vfs', mode: 'primary' },      // local authoritative, emits to remote
+      { path: '_auth', mode: 'replica' },      // remote authoritative, proxies writes
+      { path: 'config', mode: 'sync' },        // bidirectional
+      { path: 'feeds', mode: 'readonly' },     // pull-only, rejects writes
+      { path: 'telemetry', mode: 'writeonly' }, // push-only, ignores remote
+    ],
+  },
+});
+await db.replication!.start();
+```
+
+- 5 modes: `primary`, `replica`, `sync`, `readonly`, `writeonly`
+- Longest-prefix match: `_auth/nonces` (primary) overrides `_auth` (replica)
+- String paths default to `sync`: `paths: ['_vfs', '_auth']` = both sync
+- `writeProxy: 'reject'` on replica paths blocks writes instead of proxying
+- Bootstrap pulls only `replica`/`readonly` paths; `sync` gets ongoing stream only
+- Unconfigured paths fall back to `role`
+- Backward compatible: no `paths` = role-based (existing behavior)
+
 ### Multi-Source Feed Subscriptions
 
 Subscribe to specific paths from multiple remote BodDB instances — like RSS feeds for database paths.

package/CLAUDE.md

@@ -71,8 +71,9 @@ config.ts                  — demo instance config (open rules, indexes, fts, v
 - **BodClientCached**: two-tier cache wrapper around BodClient. Memory (Map, LRU eviction) + IndexedDB persistence. Stale-while-revalidate: subscribed paths always fresh, unsubscribed return stale + background refetch. Writes (`set/update/delete`) invalidate path + ancestors. `init()` opens IDB + sweeps expired. `warmup(paths[])` bulk-loads from IDB. Passthrough for `push/batch/query/search/mq/stream/vfs` via `cachedClient.client`.
 - **MCP**: `MCPAdapter` wraps a `BodClient` as a JSON-RPC MCP server (stdio + HTTP). Connects to a running BodDB instance over WebSocket — no embedded DB. Entry point: `mcp.ts`. Tools: CRUD (6), FTS (2), vectors (2), streams (4), MQ (7) = 21 tools. Use `--stdio` for Claude Code/Desktop, `--http` for remote agents.
 - **VFS (Virtual File System)**: `VFSEngine` — files stored outside SQLite via pluggable `VFSBackend` interface. `LocalBackend` stores at `<storageRoot>/<fileId>` using `Bun.file`/`Bun.write`. Metadata at `_vfs/<virtualPath>/` (size, mime, mtime, fileId, isDir) — gets subs/rules/replication for free. `fileId = pushId` so move/rename is metadata-only. REST: `POST/GET/DELETE /files/<path>`, `?stat=1`, `?list=1`, `?mkdir=1`, `PUT ?move=<dst>`. WS chunked fallback: base64-encoded `vfs-upload-init/chunk/done`, `vfs-download-init` → `vfs-download-chunk` push messages. Client: `VFSClient` via `client.vfs()` — `upload/download` (REST) + `uploadWS/downloadWS` (WS) + `stat/list/mkdir/delete/move`.
-- **Replication**: `ReplicationEngine` — single primary + N read replicas + multi-source feed subscriptions. Star topology. Primary emits write events to `_repl` stream via `onWrite` hooks. Replicas bootstrap via `streamMaterialize('_repl', { keepKey: 'path' })`, then subscribe for ongoing events. Write proxy: replica forwards writes to primary via BodClient, primary applies + emits, replica consumes. `_replaying` flag prevents re-emission loops. `_emitting` guard prevents recursion from `db.push('_repl')`. Updates flattened to per-path set events for correct compaction keying. Sweep delete events replicated. Excluded prefixes: `_repl`, `_streams`, `_mq`, `_auth`. **Sources**: `ReplicationSource[]` — subscribe to specific paths from multiple remote DBs. Each source is an independent BodClient that filters `_repl` events by path prefix, with optional `localPrefix` remapping (e.g. remote `users/u1` → local `db-a/users/u1`). Sources connect in parallel; individual failures don't block others. Sources are independent of role — a DB can be primary AND consume sources.
+- **Replication**: `ReplicationEngine` — single primary + N read replicas + multi-source feed subscriptions. Star topology. Primary emits write events to `_repl` stream via `onWrite` hooks. Replicas bootstrap via `streamMaterialize('_repl', { keepKey: 'path' })`, then subscribe for ongoing events. Write proxy: replica forwards writes to primary via BodClient, primary applies + emits, replica consumes. `_replaying` flag prevents re-emission loops. `_emitting` guard prevents recursion from `db.push('_repl')`. Updates flattened to per-path set events for correct compaction keying. Sweep delete events replicated. Excluded prefixes: `_repl`, `_streams`, `_mq`, `_auth`. **Sources**: `ReplicationSource[]` — subscribe to specific paths from multiple remote DBs. Each source is an independent BodClient that filters `_repl` events by path prefix, with optional `localPrefix` remapping (e.g. remote `users/u1` → local `db-a/users/u1`). Sources connect in parallel; individual failures don't block others. Sources are independent of role — a DB can be primary AND consume sources. **Per-path topology**: `PathTopologyRouter` — when `paths` config is set, each path prefix gets an independent mode: `primary` (local authoritative, emits), `replica` (remote authoritative, proxies writes), `sync` (bidirectional, both emit+apply), `readonly` (pull-only, rejects writes), `writeonly` (push-only, ignores remote). Longest-prefix match resolves mode. `writeProxy: 'proxy'|'reject'` overrides replica write behavior. Bootstrap skips sync paths (ongoing stream only). Auth/rules checked before proxy in all handlers. `shouldProxyPath(path)`/`shouldRejectPath(path)` replace `isReplica` checks. `emitsToRepl`/`pullsFromPrimary` getters for compact/bootstrap decisions. Stable `replicaId` from config hash. Falls back to `role` when `paths` absent (backward compat).
 - **KeyAuth integration guide**: `docs/keyauth-integration.md` — flows for signup, signin, new device, autoAuth, IAM roles, common mistakes.
+- **Para-chat integration guide**: `docs/para-chat-integration.md` — how para-chat uses BodDB: per-path topology, VFS, KeyAuth, caching, file sync.
 - **KeyAuth**: `KeyAuthEngine` — portable Ed25519 identity & IAM. Identity hierarchy: Root (server-level, key on filesystem), Account (portable, password-encrypted private key in DB or device-generated), Device (delegate, linked via password unlock). Challenge-response auth: server sends nonce → client signs with Ed25519 → server verifies + creates session. Self-signed tokens (no JWT lib): `base64url(payload).base64url(Ed25519_sign)`. Data model at `_auth/` prefix (protected from external writes). Device reverse-index at `_auth/deviceIndex/{dfp}` for O(1) lookup. Password change is atomic (single `db.update()`). IAM: roles with path-based permissions, account role assignment. `_auth/` excluded from replication. Transport guards: `auth-link-device` and `auth-change-password` require authenticated session; non-root users can only change own password. **Device registration**: `registerDevice(publicKey)` — client-generated keypair, no password, idempotent; `allowOpenRegistration: false` requires authenticated session. **Browser crypto**: `keyAuth.browser.ts` uses `@noble/ed25519` with DER↔raw key bridge for server compatibility. **BodClient autoAuth**: `autoAuth: true` auto-generates keypair (localStorage), registers, authenticates — zero-config device identity. `client.auth.*` convenience methods for all auth ops. **IAM transport ops**: `auth-create-role`, `auth-delete-role`, `auth-update-roles` (root only), `auth-list-accounts`, `auth-list-roles`. Device accounts (no encrypted key) safely reject `linkDevice`/`changePassword`.
 
 ## MCP Server

package/README.md

@@ -214,6 +214,20 @@ const aggregator = new BodDB({
 });
 await aggregator.replication!.start();
 // Remote catalog/item → local a/catalog/item
+
+// Per-path topology — mixed ownership per path prefix
+const edge = new BodDB({
+  replication: {
+    role: 'primary',
+    primaryUrl: 'ws://central:4400',
+    paths: [
+      { path: '_vfs', mode: 'primary' },      // local authoritative
+      { path: '_auth', mode: 'replica' },      // remote authoritative
+      { path: 'config', mode: 'sync' },        // bidirectional
+      { path: 'telemetry', mode: 'writeonly' }, // push-only
+    ],
+  },
+});
 ```
 
 ## Rules

package/docs/para-chat-integration.md

@@ -0,0 +1,198 @@
+# Para-Chat — BodDB Integration Guide
+
+Para-chat is an AI coder companion that uses BodDB for real-time data, auth, file system, and cross-device sync.
+
+---
+
+## Architecture
+
+```
+para-chat instance
+├── Bun.serve (shared port)
+│   ├── /api/*          — REST API
+│   ├── /db (WS)        — BodDB transport
+│   ├── /db (REST)      — BodDB REST endpoints
+│   └── /files/*        — VFS file upload/download
+├── BodDB (.internal/boddb.sqlite)
+│   ├── StorageEngine   — collections (notifications, terminal-sessions, etc.)
+│   ├── VFSEngine       — file metadata at _vfs/, files on disk via DiskBackend
+│   ├── KeyAuthEngine   — Ed25519 device identity, sessions, IAM
+│   └── ReplicationEngine — sync with bod.ee (sources-based)
+└── Frontend (BodClientCached)
+    ├── BodClient (WS)  — CRUD, subscriptions, auth
+    └── Memory + IndexedDB cache (300 entries, 24h TTL)
+```
+
+---
+
+## Current Replication Setup
+
+Para-chat uses **sources** to pull `_vfs` and `_auth` from a remote bod.ee instance:
+
+```typescript
+// api/lib/boddb.ts (current)
+replEngine = new ReplicationEngine(instance, {
+  role: 'primary',
+  sources: [{
+    url: repl.remoteUrl,         // wss://bod.ee/db
+    auth: () => token,
+    paths: repl.paths || ['_vfs', '_auth'],
+    excludePrefixes: ['_auth/sessions', '_auth/server'],
+  }],
+});
+```
+
+Config file (`.internal/boddb.config.json`):
+```json
+{
+  "keyAuth": { "allowOpenRegistration": true, "sessionTtl": 604800 },
+  "replication": { "remoteUrl": "wss://bod.ee/db", "paths": ["_vfs", "_auth"] }
+}
+```
+
+**Limitation**: Sources are one-way pull. Local writes to `_vfs` don't automatically push to bod.ee — that's handled by a separate HTTP upload flow with dedup guards.
+
+---
+
+## Recommended: Per-Path Topology
+
+With per-path topology, each prefix gets explicit ownership semantics:
+
+```typescript
+// api/lib/boddb.ts (recommended)
+const db = new BodDB({
+  path: dbPath,
+  vfs: { backend: new DiskBackend(vfsRoot), pathAsFileId: true },
+  keyAuth: { allowOpenRegistration: true, sessionTtl: 604800 },
+  replication: {
+    role: 'primary',           // fallback for unconfigured paths
+    primaryUrl: repl.remoteUrl, // wss://bod.ee/db
+    paths: [
+      { path: '_vfs', mode: 'primary' },      // local files are authoritative
+      { path: '_auth', mode: 'replica' },      // bod.ee is auth authority
+      { path: 'config', mode: 'sync' },        // bidirectional app config
+      { path: 'storage', mode: 'primary' },    // local collections (notifications, etc.)
+    ],
+    excludePrefixes: ['_repl', '_streams', '_mq'],
+  },
+});
+```
+
+### Mode mapping for para-chat
+
+| Path | Mode | Rationale |
+|------|------|-----------|
+| `_vfs` | `primary` | Local files on disk are source of truth. Emits metadata to remote. |
+| `_auth` | `replica` | bod.ee is the identity provider. Writes proxied. Sessions/server excluded automatically. |
+| `config/app` | `sync` | App config editable from either side. |
+| `storage/*` | `primary` | Notifications, terminal-sessions are local. |
+| `telemetry` | `writeonly` | Push metrics to bod.ee, never pull. |
+
+### What changes
+
+1. **`_auth` writes are proxied** — `createAccount`, `linkDevice` go through bod.ee automatically. No separate HTTP call needed.
+2. **`_vfs` emits to remote** — file metadata changes push to bod.ee via `_repl` stream. No manual upload sync.
+3. **Bootstrap is selective** — only `_auth` (replica) pulls from remote on connect. `_vfs` (primary) and `storage` (primary) keep local state.
+4. **`_auth/sessions` and `_auth/server`** — automatically excluded from replication (`_auth` prefix excluded by default; replica mode pulls from remote but these internal paths are local-only).
+
+---
+
+## Data Paths
+
+| Path | Type | Description |
+|------|------|-------------|
+| `storage/notifications` | Collection | User notifications (id, type, title, body, timestamp) |
+| `storage/terminal-sessions` | Collection | PTY session persistence |
+| `storage/{collection}/{id}` | Collection | Generic app collections |
+| `config/app` | Object | App configuration |
+| `auth/tokens/{name}` | TokenEntry | Workspace auth tokens |
+| `_vfs/{path}` | VFS metadata | File size, mime, mtime, fileId, isDir |
+| `_auth/accounts/{fp}` | KeyAuth | User accounts (replicated from bod.ee) |
+| `_auth/devices/{fp}` | KeyAuth | Device registrations |
+| `_auth/sessions/{token}` | KeyAuth | Session tokens (local only) |
+| `_auth/server` | KeyAuth | Server keypair (local only) |
+
+---
+
+## Client Setup
+
+```typescript
+// src/lib/boddbClient.ts
+import { BodClient, BodClientCached } from '@bod.ee/db/client';
+
+const client = new BodClient({
+  url: `${wsProtocol}//${host}/db`,
+  auth: () => sessionToken,
+  reconnect: true,
+  reconnectInterval: 1000,
+  maxReconnectInterval: 30000,
+});
+
+const cached = new BodClientCached(client, {
+  maxMemoryEntries: 300,
+  maxAge: 24 * 60 * 60 * 1000,
+});
+await cached.init();
+await cached.warmup(['config/app', 'storage/notifications']);
+```
+
+### Real-time subscriptions
+
+```typescript
+// Notifications — real-time via WS
+cached.onChild('storage/notifications', (event) => {
+  dispatch({ type: 'notification', payload: event.val() });
+});
+
+// Fallback to SSE when WS unavailable
+const sse = new EventSource('/api/v1/notifications/stream');
+```
+
+### VFS (file operations)
+
+```typescript
+const vfs = cached.vfs();
+await vfs.upload('docs/readme.md', file);
+const data = await vfs.download('docs/readme.md');
+const files = await vfs.list('docs');
+await vfs.move('docs/old.md', 'archive/old.md');
+```
+
+---
+
+## Auth Flow
+
+Para-chat uses **device-based identity** (KeyAuth):
+
+1. Client generates Ed25519 keypair → stored in `localStorage`
+2. Device registered with bod.ee via `client.auth.authenticate(pubKey, signFn)`
+3. Challenge-response: server sends nonce → client signs → session token issued
+4. Session token auto-injected on all subsequent requests
+
+With `_auth` as `replica`, auth operations proxy to bod.ee automatically — accounts, devices, and roles managed centrally.
+
+---
+
+## File Sync
+
+Current flow uses HTTP-based file sync with dedup:
+1. VFS metadata replicated via `_repl` stream
+2. On metadata change → check if file exists locally
+3. If missing → HTTP GET from remote `/files/{path}`
+4. Dedup guard: 5s TTL on recently downloaded files
+
+With per-path topology (`_vfs: primary`):
+- Local file changes emit metadata to remote via `_repl`
+- Remote instances pull metadata + download files via HTTP
+- No manual sync code needed for the metadata layer
+
+---
+
+## Common Mistakes
+
+| Mistake | Problem | Fix |
+|---------|---------|-----|
+| Using `sources` + manual HTTP sync | Duplicated sync logic, race conditions | Use per-path topology with `primaryUrl` |
+| Not excluding `_auth/sessions` | Remote sessions overwrite local on bootstrap | `replica` mode + default `excludePrefixes` handles this |
+| `autoAuth: true` for user accounts | Identity lost on localStorage clear | Use `createAccount` + `linkDevice` for persistent identity |
+| Starting replication before `serve()` | Transport not ready for proxy writes | Call `serve()` first, then `replication.start()` |

package/docs/repl-stream-bloat-spec.md

@@ -0,0 +1,48 @@
+# _repl Stream Bloat: Root Cause & Fixes
+
+## Problem
+`streamMaterialize('_repl', { keepKey: 'path' })` hangs when `_repl` has >2000 entries. This blocks `ReplicationEngine.start()` on replicas indefinitely — the replica never connects and no sync happens. No timeout, no error, just a hung promise.
+
+## Root Cause
+1. **No auto-compact on _repl**: Every `db.set()` on a primary-mode path pushes to `_repl`. With 400+ VFS files and frequent saves, `_repl` grows unbounded.
+2. **`streamMaterialize` is O(n) over WS**: Server reads all entries, deduplicates by `keepKey`, serializes as one JSON message. At ~2300 entries the WS message is too large / slow for BodClient to handle within the 30s `requestTimeout`.
+3. **No backpressure or pagination**: The entire materialized result is sent as a single WS frame.
+
+## Current Workaround (applied)
+- Auto-compact in `ReplicationEngine.startPrimary()`: compacts `_repl` on startup and every 5min with `{ maxCount: 500, keepKey: 'path' }`.
+- Manual compact via `db.stream.compact('_repl', { maxCount: 50 })` to unblock stuck replicas.
+
+## Proper Fixes (prioritized)
+
+### P0: Timeout + retry in replica bootstrap
+**File:** `ReplicationEngine.ts` → `startReplicaForPaths()` / `startReplica()`
+- Wrap `streamMaterialize` call in a timeout (e.g. 15s)
+- On timeout: log warning, skip bootstrap, proceed to stream subscription
+- Replica will catch up via ongoing `_repl` events (eventually consistent)
+- This ensures a hung materialize never blocks the entire replication lifecycle
+
+### P1: Paginated streamMaterialize
+**File:** `StreamEngine.ts`, `Transport.ts`, `BodClient.ts`
+- Add `limit` + `cursor` params to `stream-materialize` op
+- Server returns pages of N entries + a cursor for the next page
+- Client iterates pages until done
+- Avoids single massive WS frame; works for any stream size
+
+### P2: Compact-on-materialize (server-side)
+**File:** `StreamEngine.ts`
+- When `streamMaterialize` is called with `keepKey`, the server already deduplicates
+- After dedup: auto-delete the redundant entries that were collapsed
+- Result: materialize is self-healing — each call prunes the stream
+- Trade-off: adds write I/O during a read op; should be opt-in via flag
+
+### P3: Delta-based bootstrap
+**File:** `ReplicationEngine.ts`
+- Replica persists its last-seen `_repl` key/timestamp locally
+- On reconnect: request only entries after the last-seen key (`startAt` param)
+- Eliminates the need to materialize the full stream on every connect
+- Requires `stream-read` with `startAfter` support (already exists via `startAt`)
+
+## Metrics to Add
+- Log `_repl` entry count at ReplicationEngine start
+- Log `streamMaterialize` duration
+- Alert/warn when `_repl` exceeds 1000 entries before compact

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bod.ee/db",
-  "version": "0.12.1",
+  "version": "0.12.2",
   "module": "index.ts",
   "type": "module",
   "exports": {

package/src/server/BodDB.ts

@@ -145,12 +145,12 @@ export class BodDB {
     // Init replication
     if (this.options.replication) {
       this.replication = new ReplicationEngine(this, this.options.replication);
-      // Auto-add _repl compaction for primary
-      if (this.replication.isPrimary) {
+      // Auto-add _repl compaction for any node that emits to _repl
+      if (this.replication.emitsToRepl) {
         const compactOpts = this.options.replication.compact ?? { keepKey: 'path', maxCount: 10000 };
         this.stream.options.compact = { ...this.stream.options.compact, _repl: compactOpts };
       }
-      _log.info(`Replication enabled (role: ${this.replication.isPrimary ? 'primary' : 'replica'})`);
+      _log.info(`Replication enabled (role: ${this.options.replication.role}${this.replication.router ? ', per-path topology' : ''})`);
     }
 
     _log.info('BodDB ready');