event-sourced-collection 0.0.1 → 0.0.2

package/ARCHITECTURE.md

@@ -0,0 +1,479 @@
+# Architecture & Design Decisions
+
+Technical reference for contributors and anyone evaluating this library's tradeoffs.
+
+---
+
+## Core Philosophy
+
+This library answers one question: **How do you sync local-first data without downloading entire tables?**
+
+The answer: log every mutation as an event. Sync the event log. Replay events on both sides.
+
+This is event sourcing applied to client-side state, scoped to a single user's data.
+
+---
+
+## System Overview
+
+```
+┌──────────────────────────────────────────────────────────────────────┐
+│  Client (Browser / React Native / Expo / Node)                        │
+│                                                                        │
+│  ┌───────────────────────────────┐  ┌───────────────────────────────┐ │
+│  │  outbox (persisted collection)│  │  inbox (persisted collection) │ │
+│  │  local mutations to push      │  │  server events pulled down    │ │
+│  │                               │  │                               │ │
+│  │  eventId | type | key | sync  │  │  eventId | globalSeq | sync   │ │
+│  │  ──────────────────────────── │  │  ──────────────────────────── │ │
+│  │   e1     | ins  | t1  | false │  │   e9     |   50      | true   │ │
+│  │   e2     | upd  | t1  | true  │  │   e10    |   51      | false  │ │
+│  └───────────────────────────────┘  └───────────────────────────────┘ │
+│           ▲                                            │               │
+│    onInsert/onUpdate/onDelete                   acceptMutations        │
+│    hooks append to outbox                       replays inbox into     │
+│           │                                     state collections      │
+│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐                │
+│  │  users       │  │  todos       │  │  settings    │                │
+│  │  (persisted) │  │  (persisted) │  │  (persisted) │                │
+│  │  collection  │  │  collection  │  │  collection  │                │
+│  └──────────────┘  └──────────────┘  └──────────────┘                │
+│                                                                        │
+│  Pull cursor = max(synced globalSeq) across inbox (derived)           │
+└──────────────────────────────────────────────────────────────────────┘
+         │  push (outbox.sync = false)    │  pull (since derived cursor)
+         ▼                                ▼
+┌──────────────────────────────────────────────────────────────────────┐
+│  Server                                                                │
+│                                                                        │
+│  events table (BIGSERIAL global_seq, event_id UNIQUE, payload JSONB)  │
+│                                                                        │
+│  POST /api/events → assigns global_seq, deduplicates by event_id     │
+│  GET  /api/events?since=N → returns events after N, sorted ASC       │
+└──────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Design Decisions
+
+### 1. Separate Outbox and Inbox (Not One Mixed Table)
+
+**Decision:** Outgoing local mutations live in an `outbox` collection; pulled server events live in an `inbox` collection. Both are ordinary persisted TanStack DB collections, exposed as `db.collections.outbox` and `db.collections.inbox`. There is no separate raw event table and no separate cursor table.
+
+**Why:**
+
+| Factor               | One Mixed Table                                          | Outbox + Inbox                                              |
+| -------------------- | -------------------------------------------------------- | ---------------------------------------------------------- |
+| Mental model         | A single row could be "ours, pending" or "theirs, synced" | Direction is explicit: outbox = ours, inbox = theirs        |
+| Per-row `sync` flag  | Overloaded — means "pushed" or "applied" depending on row | Unambiguous — outbox `sync` = pushed, inbox `sync` = replayed |
+| Visualization        | Needs raw SQL access to inspect                          | Just `useLiveQuery(db.collections.outbox)`                  |
+| Cursor               | Dedicated `esdb_cursor` row to keep in step              | Derived from `max(synced globalSeq)` across inbox           |
+| Storage layer        | Hand-rolled SQLite table + indexes                       | Reuses TanStack DB persistence (reactive, multi-tab aware)  |
+
+**Per-row sync semantics:**
+
+- **outbox** — `sync: false` means the mutation has not been pushed yet. `sync: true` means the server accepted it (no errors) and assigned a `globalSeq`.
+- **outbox `syncStatus`** — `pending` means retryable, `synced` means confirmed, and `failed` means the server or custom push handler explicitly rejected that event. Failed rows also store `attemptCount`, `lastAttemptAt`, `lastError`, `lastErrorCode`, and `retryable`.
+- **inbox** — `sync: false` means the event was pulled but not yet replayed. `sync: true` means it has been applied on top of the local data, or it originated locally and was already applied before push.
+
+**FK ordering** is still preserved: the inbox replays server events sorted by `globalSeq`, so parents always precede children.
+
+**Tradeoff:** The pull cursor is derived by scanning synced inbox state rather than read from a dedicated row. For the target scope (single-user, modest event counts) this is negligible.
+
+**Constraint:** Because both are persisted collections, durability and multi-tab coordination are inherited from TanStack DB's persistence layer rather than implemented here.
+
+---
+
+### 2. Server Assigns Order (Not Client)
+
+**Decision:** The server assigns `global_seq` via `BIGSERIAL`. The client never determines canonical order.
+
+**Why:**
+
+- Foreign keys require causal ordering (parent before child)
+- The network delivers events in arbitrary order
+- Multiple devices produce events independently
+- Only a central authority can guarantee a consistent total order
+
+**Tradeoff:** Requires a server. This library does not support peer-to-peer sync.
+
+**Constraint:** If you need P2P, you need CRDTs or vector clocks — a fundamentally different architecture that adds significant complexity for marginal benefit in single-user scenarios.
+
+---
+
+### 3. Hooks Intercept Normal Collection API (No Special Dispatch)
+
+**Decision:** Users call `collection.insert()` / `update()` / `delete()` normally. Injected `onInsert` / `onUpdate` / `onDelete` hooks log events transparently.
+
+**Why:**
+
+- Zero learning curve — same API as TanStack DB without this library
+- `useLiveQuery` works unchanged
+- No "dual write" where users must remember to call both `collection.insert()` and `events.log()`
+- Impossible to forget to log an event
+
+**Tradeoff:** You cannot log custom domain events (e.g., `"user:promoted"` as a distinct event type). Every event is either `insert`, `update`, or `delete`.
+
+**Constraint:** If you need rich domain events, you'd extend this library's hook to derive a domain event type from the mutation data, or add a `dispatch()` API alongside the collection API.
+
+---
+
+### 4. Replay Uses `acceptMutations` (No Re-Logging Guard Needed)
+
+**Decision:** Server events are replayed into state collections via the persistence layer's `acceptMutations` util, which bypasses the `onInsert` / `onUpdate` / `onDelete` write handlers entirely.
+
+**Why:**
+
+The mutation hooks (which append to the outbox) only fire on user-initiated writes. `acceptMutations` applies and persists rows without invoking those handlers, so replaying a pulled event never re-appends it to the outbox. This removes the need for the old global `isSyncing` boolean and its associated race window — there is no longer a guard that could drop a concurrent user mutation.
+
+**Idempotent recovery:** Replay is keyed by the row's primary key, so re-applying the same event is a no-op upsert. On startup, any inbox row still marked `sync: false` (e.g. the process crashed between writing the inbox row and applying it) is simply replayed again. The inbox `sync` flag is the durable record of "applied or not".
+
+**Tradeoff:** A user mutation made during an in-flight pull is appended to the outbox immediately and will be pushed on the next sync — there is no longer any window where it is silently skipped.
+
+---
+
+### 5. State Collections Are Source of Truth Locally
+
+**Decision:** The local persisted collections (state) are the source of truth for the UI. The event log is an audit trail + sync mechanism. We don't rebuild state from events on startup.
+
+**Why:**
+
+- Instant hydration — state is already materialized in SQLite
+- No replay cost on app start (could be seconds for large event logs)
+- TanStack DB's persistence layer already handles this correctly
+- The event log is only used for sync, not for state reconstruction
+
+**Tradeoff:** If the event log and state diverge (bug, partial failure), the state wins locally. The server's event log is the ultimate arbiter — next sync will correct any drift.
+
+**Constraint:** This means you cannot "replay from event 0" to rebuild state client-side. If you need that (time-travel debugging, undo/redo), extend the library to add a `rebuildFromEvents()` method.
+
+---
+
+### 6. Platform Dependencies Are Injected (Not Imported)
+
+**Decision:** The library never imports `@tanstack/browser-db-sqlite-persistence` or `@tanstack/react-native-db-sqlite-persistence` directly. Users pass them in via a `load()` callback on the platform helpers, or as explicit parameters to the low-level `createEventSourcedDB` API.
+
+**Why:**
+
+- One npm package works on all platforms
+- No conditional imports or build-time platform detection
+- No React Native metro bundler issues with browser-only code
+- The library has zero bundled platform or framework dependencies
+- Users choose `@tanstack/react-db` vs `@tanstack/db` vs `@tanstack/react-native-db` in their own `load()` — the package never decides
+
+**Recommended setup (browser):**
+
+```typescript
+import { createBrowserEventSourcedDB } from "event-sourced-collection/browser";
+
+const { ensureDb, db } = createBrowserEventSourcedDB({
+  databaseName: "my-app.sqlite",
+  collections: { todos: { getKey: (t) => t.id } },
+  sync: { pushEvents, pullEvents },
+  load: async () => {
+    const { createCollection } = await import("@tanstack/react-db");
+    const platform = await import("@tanstack/browser-db-sqlite-persistence");
+    return { ...platform, createCollection };
+  },
+});
+```
+
+`createBrowserEventSourcedDB` / `createReactNativeEventSourcedDB` orchestrate platform setup, call `createEventSourcedDB`, and return `{ ensureDb, db, close }` backed by `createLazySingleton` (deduped init + proxy). The `load` callback keeps dynamic imports in userland so SSR bundles stay clean.
+
+**Low-level alternative:** Pass `createCollection`, `persistedCollectionOptions`, and `persistence` directly to `createEventSourcedDB` + `createBrowserPlatform` when you need full control.
+
+**Tradeoff:** Users write a small `load()` function. That is intentional — it is the price of zero coupling.
+
+**Constraint:** If even `load()` feels verbose, the helpers already collapse ~80 lines of manual singleton/proxy/platform wiring into one call.
+
+---
+
+### 7. Handler-First Sync With HTTP Fallback
+
+**Decision:** Sync accepts typed `pushEvents` and `pullEvents` functions first, plus `pushUrl` and `pullUrl` as HTTP convenience fallbacks. The legacy `SyncTransport` shape is still accepted.
+
+**Why:**
+
+- HTTP is universally available (browser, RN, Node)
+- RESTful endpoints are the easiest to implement server-side
+- Type-safe RPC/server-function clients should not lose type safety by routing through raw URLs
+- Function handlers let users call existing clients, queues, workers, or local persistence without the library knowing the transport
+
+**Tradeoff:** HTTP is request/response — no real-time push from server. The client must poll or call `sync()` explicitly unless user-provided handlers integrate a realtime trigger.
+
+**Future improvement:** Add a `subscribe()` option that uses SSE or WebSocket for real-time server push, falling back to polling.
+
+---
+
+### 8. No CRDT, No Vector Clocks
+
+**Decision:** Conflicts are resolved by server ordering. The server's `global_seq` is law.
+
+**Why:**
+
+- Target scope: single user, possibly multi-device
+- Cross-user conflicts are extremely rare (data is scoped to userId)
+- Same-user conflicts on multi-device are resolved by "last writer wins" (server decides sequence)
+- CRDTs add: metadata overhead, merge complexity, tombstone management, convergence testing
+- None of that is needed when one server is the authority
+
+**Tradeoff:** Two simultaneous offline edits to the same field → server picks one order. No automatic merge. The "losing" edit is still in the event log (audit trail), but the state reflects the server's chosen order.
+
+**Constraint:** If you need real-time collaborative editing (Google Docs style), this is the wrong architecture. Use Yjs, Automerge, or similar.
+
+---
+
+### 9. `BIGSERIAL` for Server Ordering (Not UUIDv7)
+
+**Decision:** Server uses PostgreSQL's `BIGSERIAL` for `global_seq`.
+
+**Why:**
+
+| Factor              | BIGSERIAL                 | UUIDv7                            |
+| ------------------- | ------------------------- | --------------------------------- |
+| Storage per event   | 8 bytes                   | 16 bytes                          |
+| Index performance   | Faster integer comparison | Slower string comparison          |
+| Ordering guarantee  | Database-level monotonic  | Timestamp-based (clock skew risk) |
+| Readability         | `seq: 42`                 | `019abc12-...`                    |
+| Distributed writers | No (single server)        | Yes                               |
+
+Since there's one server (not distributed), `BIGSERIAL` is strictly superior.
+
+**Constraint:** If you ever need multiple independent write servers, you'd need to switch to UUIDv7 or a distributed sequence generator. For single-server (which covers 99% of apps), this is irrelevant.
+
+---
+
+### 10. Delete Events Store the Full Object (Not Just the Key)
+
+**Decision:** When a row is deleted, the event payload contains the entire deleted object, not just the key.
+
+**Why:**
+
+- Server can process the delete without looking up what was deleted
+- Enables "undo delete" on the server side
+- Audit trail shows what was deleted, not just that something was deleted
+- Replay can restore the object if the delete is later reversed
+
+**Tradeoff:** Slightly larger event payloads for deletes.
+
+---
+
+## Pros
+
+- Zero learning curve for TanStack DB users
+- Offline-first by default — works without network
+- Partial sync — never downloads entire tables
+- Foreign key safety — server controls insertion order
+- Audit trail — complete history of every mutation
+- Multi-device — events from all devices converge
+- Platform agnostic — browser, React Native, Expo, Node
+- Type-safe — full inference from collection definitions
+- Small — ~4KB, minimal dependencies
+- No build plugins or code generation
+
+## Cons
+
+- Requires a server (not P2P)
+- No real-time push (polling or explicit sync)
+- Event log grows unbounded (need pruning strategy for very long-lived apps)
+- No automatic conflict resolution beyond "server decides order"
+- Cannot replay from event 0 to rebuild state (state is materialized separately)
+- Update events store full modified object (not a minimal diff)
+- Pull cursor is derived by scanning synced inbox state rather than stored as a single authoritative value
+
+---
+
+## When NOT to Use This
+
+| Scenario                                      | Better Alternative             |
+| --------------------------------------------- | ------------------------------ |
+| Real-time multiplayer / collaborative editing | CRDTs (Yjs, Automerge)         |
+| Server-authoritative with no offline          | Direct API calls + React Query |
+| Simple cache layer                            | TanStack Query                 |
+| Multi-tenant with cross-user data             | Electric SQL, PowerSync        |
+| Streaming large datasets                      | TanStack DB with Electric sync |
+
+---
+
+## When to Use This
+
+- Single-user app with multi-device sync
+- Offline-first mobile app (React Native)
+- Todo apps, note-taking, personal finance, habit trackers
+- Any app where the user owns their data and edits offline
+- Apps that need an audit trail of changes
+- Apps where you control the backend and want simple sync
+
+---
+
+## Adjacent TanStack Libraries & How They Compose
+
+TanStack DB 0.6 shipped several features that directly improve this library's capabilities. Here's how each one fits.
+
+### `createEffect` — Reactive Sync Trigger
+
+**What it is:** A database-trigger-like API that fires `onEnter` / `onUpdate` / `onExit` callbacks when rows enter/leave a live query result. Runs incrementally on deltas, not full result sets.
+
+**How it helps us:**
+
+| Without createEffect                      | With createEffect                                          |
+| ----------------------------------------- | ---------------------------------------------------------- |
+| User must call `db.sync()` manually       | Auto-syncs when pending events exist                       |
+| Polling interval for connectivity         | Reactive — fires the instant an event is logged            |
+| No way to react to server-applied changes | `onEnter` fires when a new server event materializes a row |
+
+**Integration pattern:**
+
+```typescript
+createEffect({
+  query: (q) => q.from({ e: db.collections.outbox }).where(({ e }) => eq(e.sync, false)),
+  skipInitial: false,
+  onEnter: async (_event, ctx) => {
+    await syncWithRetry({ signal: ctx.signal });
+  },
+});
+```
+
+This replaces `setInterval(() => db.sync(), 30000)` with a reactive approach that fires immediately when data is dirty.
+
+**Trade-off:** Adds a dependency on `createEffect` from `@tanstack/db`. Currently optional — users who prefer manual sync can ignore it.
+
+---
+
+### `@tanstack/offline-transactions` — Production Sync Reliability
+
+**What it is:** An `OfflineExecutor` that orchestrates:
+
+- Persistent outbox (IndexedDB/localStorage)
+- Leader election (WebLocks/BroadcastChannel)
+- Retry with exponential backoff
+- Connectivity detection
+- Idempotency keys for at-least-once delivery
+- Graceful degradation when storage is unavailable
+
+**How it helps us:**
+
+Our library's `sync()` function is a simple push+pull. In production, you need:
+
+- Only one tab performing sync (leader election)
+- Automatic retry when push fails (network blip)
+- Backoff so you don't DDoS your own server
+- Connectivity awareness (don't try to sync when offline)
+
+`@tanstack/offline-transactions` provides all of this.
+
+**Integration pattern:**
+
+```typescript
+import { createOfflineTransaction } from "@tanstack/offline-transactions";
+
+const offlineSync = createOfflineTransaction({
+  mutationFn: async () => {
+    const result = await db.sync();
+    if (result.errors.length > 0) throw result.errors[0];
+  },
+  retryConfig: { maxRetries: 5, backoffMs: 1000 },
+});
+```
+
+**Trade-off:** Optional dependency. Our library works without it (manual sync), but production apps should use it.
+
+---
+
+### Virtual Props (`$synced`, `$origin`) — Sync State Visibility
+
+**What they are:**
+
+- `$synced: boolean` — whether the row is confirmed by sync or still optimistic/local
+- `$origin: 'local' | 'remote'` — whether the last confirmed change came from this client or upstream
+
+**How they help us:**
+
+| Problem                                 | Solution with Virtual Props                              |
+| --------------------------------------- | -------------------------------------------------------- |
+| "Which todos haven't synced yet?"       | `where(({ todo }) => eq(todo.$synced, false))`           |
+| "Show a spinner on unsynced items"      | Render based on `todo.$synced` in the UI                 |
+| "Skip re-logging server-applied events" | Already handled — `acceptMutations` bypasses the write hooks |
+
+**Note:** This library no longer needs a re-logging guard. Server replay goes through `acceptMutations`, which never invokes the outbox-logging hooks (see Design Decision #4). Virtual props remain useful for surfacing optimistic vs confirmed state in the UI.
+
+---
+
+### `includes` — Hierarchical Queries
+
+**What it is:** Nested subqueries that project normalized data into the hierarchical shape of your UI, without N+1 queries. Each included field is a child collection with independent reactivity.
+
+**How it helps us:**
+
+Event-sourced state collections are flat (users, todos, settings). But UIs are hierarchical (a project has issues, each issue has comments). With `includes`, users query across their event-sourced collections with full hierarchical projection:
+
+```typescript
+const { data: users } = useLiveQuery((q) =>
+  q.from({ u: db.collections.users }).select(({ u }) => ({
+    id: u.id,
+    name: u.name,
+    todos: q
+      .from({ t: db.collections.todos })
+      .where(({ t }) => eq(t.userId, u.id))
+      .select(({ t }) => ({ id: t.id, title: t.title })),
+  })),
+);
+```
+
+This would be extremely painful to build from scratch (incremental nested query evaluation). TanStack DB handles it in one query graph.
+
+---
+
+### `queryOnce` — One-Shot Reads
+
+**What it is:** Execute a query once and get the result as a promise, without subscribing to updates.
+
+**How it helps us:**
+
+Useful internally for sync operations:
+
+- "Get all pending events" — one-shot, no subscription needed
+- "Get current cursor state" — read once at sync start
+- Useful for migration scripts, exports, debugging
+
+```typescript
+const pending = await queryOnce((q) =>
+  q
+    .from({ e: db.collections.outbox })
+    .where(({ e }) => eq(e.sync, false))
+    .orderBy(({ e }) => e.localSeq, "asc"),
+);
+```
+
+---
+
+### Summary: What We Get for Free vs What We Build
+
+| Layer                                         | Source                         | Lines of Code |
+| --------------------------------------------- | ------------------------------ | ------------- |
+| SQLite persistence (8 platforms)              | TanStack DB                    | ~2,800        |
+| Reactive query engine (differential dataflow) | TanStack DB                    | ~8,000        |
+| Multi-tab coordination                        | TanStack DB                    | ~1,200        |
+| Optimistic mutations + rollback               | TanStack DB                    | ~2,000        |
+| Hierarchical includes                         | TanStack DB                    | ~1,500        |
+| Reactive effects (createEffect)               | TanStack DB                    | ~600          |
+| Offline retry + leader election               | @tanstack/offline-transactions | ~800          |
+| **Event log + sync protocol + hook wiring**   | **This library**               | **~300**      |
+
+We build 300 lines. We get 17,000+ lines of battle-tested infrastructure for free.
+
+---
+
+## Future Directions
+
+1. **Event pruning** — compact old synced events after N days
+2. **Real-time subscribe** — SSE/WebSocket for instant server push
+3. **Selective sync** — sync only specific collections or subsets
+4. **Batched writes** — group rapid mutations into single events
+5. **Undo/redo** — leverage event log for time-travel
+6. **Schema migrations** — event upcasters for evolving payload shapes
+7. **Compression** — delta encoding for update events (store diff, not full object)
+8. **Retry queue** — exponential backoff for failed pushes
+9. **Multi-user awareness** — optional user_id scoping on the event table
+10. **Rebuild from events** — optional cold-start replay for disaster recovery