@bod.ee/db 0.7.0

package/CLAUDE.md

@@ -0,0 +1,110 @@
+# BodDB
+
+Embedded reactive database — SQLite-backed (bun:sqlite), zero dependencies.
+
+## Architecture
+
+```
+src/shared/pathUtils.ts    — path normalize/validate, flatten/reconstruct, ancestors, prefixEnd
+src/shared/protocol.ts     — wire message types (client↔server), batch ops
+src/shared/errors.ts       — error codes
+src/shared/transforms.ts   — sentinel classes (increment, serverTimestamp, arrayUnion, arrayRemove, ref)
+src/server/StorageEngine.ts — SQLite CRUD, leaf flattening, prefix queries, transforms, TTL, push IDs
+src/server/SubscriptionEngine.ts — value + child subscriptions, ancestor walk, dedup, added/changed/removed
+src/server/QueryEngine.ts  — fluent query builder → StorageEngine.query
+src/server/BodDB.ts        — main facade: storage + subs + rules + transport + FTS + vectors + TTL sweep
+src/server/RulesEngine.ts  — path-based permission checks with $wildcard capture (functions + expressions)
+src/server/ExpressionRules.ts — safe AST-based expression parser (tokenizer → parser → evaluator, no eval)
+src/server/Transport.ts    — Bun.serve WebSocket + REST + SSE endpoints, batch/push ops, sub lifecycle
+src/server/FTSEngine.ts    — SQLite FTS5 full-text search: index, search, auto-index
+src/server/VectorEngine.ts — vector similarity search: brute-force cosine/euclidean, Float32Array storage
+src/server/StreamEngine.ts — Kafka-like event streaming: consumer groups, offset tracking, replay, durable subs
+src/server/MQEngine.ts     — SQS-style message queue: push/fetch/ack/nack, visibility timeout, DLQ, exactly-once claim
+src/server/FileAdapter.ts  — file system sync: scan, watch, metadata, content read/write-through
+src/server/MCPAdapter.ts   — MCP (Model Context Protocol) server: JSON-RPC dispatch, tool registry, stdio + HTTP transports
+src/client/BodClient.ts    — WS client: connect, auth, CRUD, batch, push, subscriptions, auto-reconnect
+src/react/hooks.ts         — useValue, useChildren, useQuery, useMutation
+client.ts                  — client entry point (import from 'bod-db/client')
+react.ts                   — React hooks entry point (import from 'bod-db/react')
+cli.ts                     — CLI entry point: bod-db [config] [--port/--path/--memory/--init]
+mcp.ts                     — MCP server entry point: bod-db-mcp [--stdio/--http] [--url/--port]
+config.ts                  — demo instance config (open rules, indexes, fts, vectors, mq)
+```
+
+## Key patterns
+
+- **Leaf flattening**: nested objects → flat rows keyed by full path. Subtrees reconstructed via prefix scan.
+- **Push rows**: `db.push()` stores as single JSON row (NOT flattened) with time-sortable key (8-char timestamp + 4-char random).
+- **Prefix queries**: `WHERE path >= 'users/u1/' AND path < 'users/u1/\uffff'`
+- **Subscription notify**: writes produce changedPaths → notify exact + all ancestors (deduped). Child events track added/changed/removed via pre-write existence snapshot.
+- **Path validation**: all public APIs use `validatePath()` — rejects empty paths, normalizes slashes.
+- **Options pattern**: `new XOptions()` defaults merged with partial user options.
+- **Rules**: path patterns with `$wildcard` capture, most-specific match wins. Context: `auth`, `params`, `data`, `newData`. Supports functions, booleans, expression strings, and JSON/TS config files.
+- **Expression rules**: safe tokenizer → recursive descent parser → AST evaluator. Supports `auth.x`, `$wildcard`, `data`/`newData`, comparisons, logical ops, negation, parens. No eval.
+- **Transforms**: sentinel values (`increment`, `serverTimestamp`, `arrayUnion`, `arrayRemove`, `ref`) resolved against current data before write.
+- **Refs**: `ref(path)` stores `{ _ref: path }`. Resolved at read time via `storage.get(path, { resolve: true })`.
+- **Transactions**: `db.transaction(fn)` wraps operations in SQLite transaction. Notifications fire after commit.
+- **Batch protocol**: `{ op: 'batch', operations: [...] }` — executes all ops in single SQLite transaction.
+- **TTL**: `db.set(path, val, { ttl: seconds })` → `expires_at` column. Background sweep (configurable interval) deletes expired + notifies subscribers.
+- **FTS5**: `db.search({ text, path?, limit? })` using SQLite FTS5. Index via `db.index(path, content)` or `db.index(path, fields[])`.
+- **Vectors**: `db.vectorSearch({ query, path?, limit?, threshold? })` — brute-force cosine similarity. Store via `db.vectors.store(path, embedding)`.
+- **Streams**: Kafka-like consumer groups over push paths. `db.stream.read(topic, groupId)` / `.ack()` / `.subscribe()`. Offsets stored in `_streams/<topic>/groups/<groupId>/offset`. Subscribe-then-backfill prevents race conditions. `queryAfterKey()` for SQL-level replay (avoids loading entire topic). Idempotent push via `idempotency_key` column + unique index. **Compaction** folds events into a snapshot (`_streams/<topic>/snapshot`), then deletes them. Snapshot is the base state; live events layer on top. `compact(topic, { maxAge, maxCount, keepKey })`. `materialize(topic, { keepKey })` returns merged snapshot+events view. Auto-compact on sweep via `compact` config. Safety: never folds events beyond minimum consumer group offset.
+- **MQ (Message Queue)**: SQS-style work queue. `db.mq.push(queue, value)` / `.fetch(queue, count)` / `.ack(queue, key)` / `.nack(queue, key)`. Each message claimed by exactly one worker (visibility timeout). Columns: `mq_status` (pending/inflight/dead), `mq_inflight_until`, `mq_delivery_count`. `sweep()` reclaims expired inflight; exhausted messages (>maxDeliveries) move to DLQ at `<queue>/_dlq/<key>` with `mq_status='dead'`. Ack = DELETE (keeps queue lean). `purge(queue)` deletes pending only; `purge(queue, { all: true })` deletes all (pending + inflight + DLQ). Per-queue options via `queues` config (longest prefix match). Client: `MQReader` / `MQMessageSnapshot`.
+- **FileAdapter**: scans directory, syncs metadata to DB, optional fs.watch, content read/write-through.
+- **SSE fallback**: `GET /sse/<path>?event=value|child` returns `text/event-stream`. Initial `: ok` comment flushes the stream connection.
+- **Perf**: `snapshotExisting` and `notify` are skipped when no subscriptions are active. `exists()` uses `LIMIT 1`.
+- **Transport**: WS messages follow `protocol.ts` types. REST at `/db/<path>`. Auth via `op:'auth'` message. Subs cleaned up on disconnect.
+- **BodClient**: id-correlated request/response over WS. Auto-reconnect with exponential backoff. Re-subscribes all active subs on reconnect. `ValueSnapshot` with `.val()`, `.key`, `.path`, `.exists()`.
+- **MCP**: `MCPAdapter` wraps a `BodClient` as a JSON-RPC MCP server (stdio + HTTP). Connects to a running BodDB instance over WebSocket — no embedded DB. Entry point: `mcp.ts`. Tools: CRUD (6), FTS (2), vectors (2), streams (4), MQ (7) = 21 tools. Use `--stdio` for Claude Code/Desktop, `--http` for remote agents.
+
+## MCP Server
+
+The MCP server is a **client** that connects to a running BodDB server via WebSocket.
+
+```bash
+# Start BodDB server first
+bun run start                       # ws://localhost:4400
+
+# Then start MCP (separate process)
+bun run mcp.ts --stdio              # stdio transport (default, for Claude Code)
+bun run mcp.ts --stdio --port 4400  # explicit port
+bun run mcp.ts --http --http-port 4401  # HTTP transport for remote agents
+```
+
+### Claude Code integration (.mcp.json)
+```json
+{
+  "mcpServers": {
+    "bod-db": {
+      "type": "stdio",
+      "command": "bun",
+      "args": ["run", "/path/to/zuzdb/mcp.ts", "--stdio"]
+    }
+  }
+}
+```
+
+### HTTP integration
+```bash
+curl -X POST http://localhost:4401 \
+  -H 'Content-Type: application/json' \
+  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
+```
+
+## Testing
+
+```bash
+bun test              # run all tests
+bun test tests/storage.test.ts  # single file
+```
+
+## Implementation status
+
+- [x] Phase 1: Core (storage, subscriptions, queries) — server-only, no network
+- [x] Phase 2: Transport (WebSocket + REST) + RulesEngine (function-based V1)
+- [x] Phase 3: Client SDK (BodClient — CRUD, queries, subscriptions, auto-reconnect)
+- [x] Phase 4: Expression rules V2, SSE fallback, React hooks, benchmarks, examples
+- [x] Phase 5: Rules config files, transforms/sentinels, refs, transactions, batch, push, TTL, FileAdapter, FTS5, VectorEngine
+- [x] Phase 6: Event streaming (consumer groups, offset tracking, replay, idempotent push, StreamEngine)
+- [x] Phase 7: Message queue (MQEngine — push/fetch/ack/nack, visibility timeout, DLQ, exactly-once claim)
+- [x] Phase 8: MCP server (MCPAdapter — stdio + HTTP, BodClient-based, 21 tools)

package/README.md

@@ -0,0 +1,252 @@
+# BodDB
+
+
+----
+187 tests passing · ~200k ops/sec · Zero dependencies · Full docs + examples
+----
+
+
+Embedded reactive database — SQLite-backed, zero dependencies, Firebase-like API.
+
+- **In-process**: no cloud, no external services
+- **SQLite WAL**: crash-safe, instant startup
+- **Real-time**: value + child subscriptions with ancestor propagation
+- **Network**: WebSocket + REST + SSE transport with auth and permission rules
+- **Transforms**: increment, serverTimestamp, arrayUnion, arrayRemove, refs
+- **Streaming**: Kafka-like consumer groups with offset tracking, replay, compaction
+- **Message Queue**: SQS-style work queue — exactly-once delivery, visibility timeout, DLQ
+- **Advanced**: transactions, batch ops, push IDs, TTL, FTS5, vector search, file sync
+- **TypeScript-first**: full type safety, Bun-native
+
+## Install
+
+```bash
+bun add bod-db
+```
+
+## Server
+
+```typescript
+import { BodDB, increment, serverTimestamp, ref } from 'bod-db';
+
+const db = new BodDB({
+  path: './data.db',
+  rules: {
+    'users/$uid': { read: true, write: "auth.uid === $uid" },
+  },
+  indexes: { 'users': ['role', 'createdAt'] },
+  sweepInterval: 60000,  // TTL sweep every 60s
+  fts: {},               // enable full-text search
+  vectors: { dimensions: 384 }, // enable vector search
+});
+
+// CRUD
+db.set('users/u1', { name: 'Alice', role: 'admin' });
+db.get('users/u1');              // { name: 'Alice', role: 'admin' }
+db.update({ 'users/u1/name': 'Bob', 'counters/visits': 42 });
+db.delete('users/u1');
+
+// Transforms
+db.set('counters/visits', increment(1));        // atomic increment
+db.set('posts/p1/updatedAt', serverTimestamp()); // server timestamp
+db.set('posts/p1/author', ref('users/u1'));      // reference
+
+// Push (append-only with time-sortable key)
+const key = db.push('logs', { level: 'info', msg: 'started' });
+
+// TTL
+db.set('sessions/temp', { token: 'abc' }, { ttl: 3600 }); // expires in 1h
+
+// Transaction
+db.transaction((tx) => {
+  const user = tx.get('users/u1');
+  tx.set('users/u1/lastLogin', Date.now());
+  tx.update({ 'stats/logins': { total: 1 } });
+});
+
+// Query
+db.query('users')
+  .where('role', '==', 'admin')
+  .order('name')
+  .limit(10)
+  .get();
+
+// Full-text search
+db.index('posts/p1', 'Hello world tutorial');
+db.search({ text: 'hello', path: 'posts', limit: 10 });
+
+// Vector search
+db.vectors!.store('docs/d1', embedding);
+db.vectorSearch({ query: embedding, path: 'docs', limit: 5, threshold: 0.7 });
+
+// Subscribe
+const off = db.on('users/u1', (snap) => console.log(snap.val()));
+db.onChild('users', (e) => console.log(e.type, e.key, e.val()));
+
+// Serve over network
+db.serve({ port: 4400 });
+```
+
+## Client
+
+```typescript
+import { BodClient } from 'bod-db/client';
+
+const client = new BodClient({
+  url: 'ws://localhost:4400',
+  auth: () => 'my-token',
+  reconnect: true,
+});
+
+await client.connect();
+
+// CRUD
+await client.set('users/u1', { name: 'Alice' });
+const user = await client.get('users/u1');
+
+// Batch (atomic multi-op)
+await client.batch([
+  { op: 'set', path: 'users/u1/name', value: 'Bob' },
+  { op: 'delete', path: 'users/u2' },
+]);
+
+// Push
+const key = await client.push('logs', { msg: 'hello' });
+
+// Subscribe
+const off = client.on('users/u1', (snap) => console.log(snap.val()));
+client.onChild('users', (e) => console.log(e.type, e.key));
+
+client.disconnect();
+```
+
+## Streams (Kafka-like)
+
+```typescript
+// Push events to a topic
+db.push('events/orders', { orderId: 'o1', amount: 250 });
+
+// Consumer groups — each group tracks its own offset (fan-out)
+const events = db.stream.read('events/orders', 'billing', 50);
+db.stream.ack('events/orders', 'billing', events.at(-1)!.key);
+
+// Compact old events into a snapshot
+db.stream.compact('events/orders', { maxCount: 1000, keepKey: 'orderId' });
+const view = db.stream.materialize('events/orders', { keepKey: 'orderId' });
+
+// Client
+const reader = client.stream('events/orders', 'billing');
+const unsub = reader.on((events) => { /* live delivery */ });
+```
+
+## Message Queue (SQS-like)
+
+```typescript
+// Server
+const db = new BodDB({ mq: { visibilityTimeout: 30, maxDeliveries: 5 } });
+
+db.mq.push('queues/jobs', { type: 'email', to: 'alice@example.com' });
+const jobs = db.mq.fetch('queues/jobs', 5);  // claim up to 5
+db.mq.ack('queues/jobs', jobs[0].key);        // done — delete
+db.mq.nack('queues/jobs', jobs[0].key);       // failed — back to pending
+db.mq.peek('queues/jobs');                     // view without claiming
+db.mq.dlq('queues/jobs');                      // dead letter queue
+db.mq.purge('queues/jobs');                    // delete pending
+db.mq.purge('queues/jobs', { all: true });     // delete all (pending + inflight + DLQ)
+
+// Client
+const q = client.mq('queues/jobs');
+await q.push({ type: 'email' });
+const msgs = await q.fetch(5);
+await q.ack(msgs[0].key);
+```
+
+## Rules
+
+Function-based, expression strings, or JSON config files:
+
+```typescript
+// Inline
+const db = new BodDB({
+  rules: {
+    'users/$uid': {
+      read: true,
+      write: "auth.uid === $uid",
+    },
+    'admin': {
+      read: "auth.role == 'admin'",
+      write: (ctx) => ctx.auth?.role === 'admin',
+    },
+  },
+});
+
+// From JSON file
+const db2 = new BodDB({ rules: './rules.json' });
+// rules.json: { "rules": { "users/$uid": { "read": true, "write": "auth.uid === $uid" } } }
+
+// From TS file (async)
+const db3 = await BodDB.create({ rules: './rules.ts' });
+```
+
+Expression rules support: `auth.*`, `$wildcard` params, `data`/`newData`, comparisons (`==`, `!=`, `>=`, `<`, etc.), logical ops (`&&`, `||`, `!`), parens, `null` checks. Safe AST evaluation — no `eval`.
+
+## REST API
+
+```
+GET    /db/users/u1     → { ok: true, data: { name: 'Alice' } }
+PUT    /db/users/u1     → body: JSON → { ok: true }
+DELETE /db/users/u1     → { ok: true }
+```
+
+## SSE
+
+```
+GET /sse/users/u1              → text/event-stream (value events)
+GET /sse/users?event=child     → text/event-stream (child events)
+```
+
+## React Hooks
+
+```typescript
+import { useValue, useChildren, useQuery, useMutation } from 'bod-db/react';
+
+const { data, loading } = useValue(client, 'users/u1');
+const { children, loading } = useChildren(client, 'users');
+const { data, loading } = useQuery(client, 'users', { filters: [...], limit: 10 });
+const { set, update, del, loading } = useMutation(client);
+```
+
+## File Adapter
+
+```typescript
+import { FileAdapter } from 'bod-db';
+
+const adapter = new FileAdapter(db, {
+  root: './uploads',
+  basePath: 'files',
+  watch: true,
+  metadata: true,
+});
+await adapter.start();
+// Files synced to db as: files/<relPath> → { size, mtime, mime }
+```
+
+## Benchmark
+
+```bash
+bun run tests/bench.ts
+```
+
+| Workload | ops/sec | p50 | p99 |
+|---|---|---|---|
+| Read-only | ~200K | 0.00ms | 0.03ms |
+| Read-heavy (95/5) | ~170K | 0.00ms | 0.04ms |
+| Update-heavy (50/50) | ~94K | 0.01ms | 0.07ms |
+| Insert-heavy | ~89K | 0.01ms | 0.05ms |
+| Subscription (100 subs) | ~3.5K | 0.03ms | 4ms |
+
+## Test
+
+```bash
+bun test         # 187 tests
+```

package/admin/rules.ts

@@ -0,0 +1,12 @@
+import type { PathRule } from '../src/server/RulesEngine.ts';
+
+/**
+ * Edit this file to configure access rules for the admin server.
+ * Pattern syntax: exact segments or $wildcard (e.g. users/$uid)
+ * Context: { auth, path, params, data, newData }
+ */
+export const rules: Record<string, PathRule> = {
+  '_admin/$any':   { read: false, write: false },
+  'users/$uid':    { write: ({ auth, params }) => auth?.role === 'admin' || ['alice', 'bob'].includes(params.uid) },
+  'settings/$key': { write: ({ auth }) => !!auth },
+};