@firtoz/drizzle-indexeddb 0.2.0 → 0.4.0

package/CHANGELOG.md

@@ -1,5 +1,65 @@
 # @firtoz/drizzle-indexeddb
 
+## 0.4.0
+
+### Minor Changes
+
+- [`46059a2`](https://github.com/firtoz/fullstack-toolkit/commit/46059a28bd0135414b9ed022ffe162a2292adae3) Thanks [@firtoz](https://github.com/firtoz)! - Add IDB Proxy system for multi-client IndexedDB sync over messaging layers:
+
+  **New Proxy Module** (`@firtoz/drizzle-indexeddb/proxy`):
+
+  - **`IDBProxyServer`** - Server that manages database lifecycle, migrations, and broadcasts mutations to connected clients
+  - **`IDBProxyClient`** - Client implementing `IDBDatabaseLike`, routing operations through a transport layer
+  - **`createMultiClientTransport()`** - In-memory transport for testing N clients connected to one server
+  - **`createProxyDbCreator()`** - Factory to create `dbCreator` for `DrizzleIndexedDBProvider`
+  - **`createCollectionSyncHandler()`** - Adapter connecting proxy sync messages to collection's external sync
+
+  **Real-time Multi-Client Sync**:
+
+  - Server broadcasts `sync:add`, `sync:put`, `sync:delete`, `sync:clear` messages to all clients (excluding initiator)
+  - All mutations automatically sync across connected clients
+
+  **Provider Enhancements**:
+
+  - New `onSyncReady` prop for wiring up external sync handlers
+  - `handleProxySync` method routes sync messages to the appropriate collection
+
+  **Collection Truncate**:
+
+  - `collection.utils.truncate()` clears all data and syncs to other clients
+  - `handleTruncate` implemented in IndexedDB backend
+
+  **Bug Fixes**:
+
+  - Server handles concurrent database initialization requests (race condition fix)
+
+### Patch Changes
+
+- Updated dependencies [[`46059a2`](https://github.com/firtoz/fullstack-toolkit/commit/46059a28bd0135414b9ed022ffe162a2292adae3)]:
+  - @firtoz/drizzle-utils@0.3.0
+
+## 0.3.0
+
+### Minor Changes
+
+- [`5e854a6`](https://github.com/firtoz/fullstack-toolkit/commit/5e854a62236a811918a47037a59df23329856614) Thanks [@firtoz](https://github.com/firtoz)! - ### Breaking Changes
+
+  - Removed `migrateIndexedDB` and `IndexedDBMigrationConfig` exports - use `migrateIndexedDBWithFunctions` instead
+  - Removed snapshot-based migration system in favor of function-based migrations
+
+  ### New Features
+
+  - Added `drizzle-indexeddb-generate` CLI tool to generate IndexedDB migration functions from Drizzle snapshots
+  - Added `generateIndexedDBMigrations` export for programmatic migration generation
+  - Added `./generate` export path
+
+  ### Migration Guide
+
+  Instead of importing snapshots directly and using `migrateIndexedDB`, you now:
+
+  1. Run `bun drizzle-indexeddb-generate` (or `npx drizzle-indexeddb-generate`) after `drizzle-kit generate`
+  2. Import the generated migrations and use `migrateIndexedDBWithFunctions`
+
 ## 0.2.0
 
 ### Minor Changes
@@ -41,18 +101,6 @@
   - Sync configuration for real-time updates
   - Works seamlessly with React hooks
 
-  ### Snapshot-Based Migration
-
-  **`migrateIndexedDB(dbName, config, debug?)`** - Automatically migrates IndexedDB databases using Drizzle snapshot files:
-
-  - Reads Drizzle journal and snapshot files
-  - Tracks applied migrations in `__drizzle_migrations` store
-  - Creates/updates object stores and indexes based on schema changes
-  - Handles table deletion, index changes, and schema evolution
-  - Incremental migrations - only applies pending changes
-  - Validates primary key structure changes (requires manual migration if keys change)
-  - Performance logging when debug mode is enabled
-
   ### Function-Based Migration
 
   **`migrateIndexedDBWithFunctions(dbName, migrations, debug?)`** - Run migrations using custom migration functions:
@@ -87,19 +135,15 @@
   ## Example
 
   ```typescript
-  import { migrateIndexedDB } from "@firtoz/drizzle-indexeddb";
-  import journal from "./drizzle/journal.json";
-  import * as snapshots from "./drizzle/snapshots";
+  import { migrateIndexedDBWithFunctions } from "@firtoz/drizzle-indexeddb";
+  import migrations from "./drizzle/indexeddb-migrations";
 
-  // Migrate database using Drizzle snapshots
-  const db = await migrateIndexedDB(
+  // Migrate database using function-based migrations
+  const db = await migrateIndexedDBWithFunctions(
     "my-app-db",
-    {
-      journal,
-      snapshots,
-    },
-    true
-  ); // debug mode
+    migrations,
+    true // debug mode
+  );
 
   // Use with TanStack DB
   import { createCollection } from "@tanstack/db";
@@ -136,9 +180,9 @@
 
   ## Migration Workflow
 
-  1. Generate Drizzle snapshots: `drizzle-kit generate`
-  2. Import journal and snapshots
-  3. Call `migrateIndexedDB()` on app startup
+  1. Generate Drizzle migrations: `drizzle-kit generate`
+  2. Generate IndexedDB migrations: `bun drizzle-indexeddb-generate`
+  3. Import migrations and call `migrateIndexedDBWithFunctions()` on app startup
   4. Database automatically updates to latest schema
 
   ## Dependencies

package/README.md

@@ -1,10 +1,10 @@
 # @firtoz/drizzle-indexeddb
 
-TanStack DB collections backed by IndexedDB with automatic migrations powered by Drizzle ORM snapshots. Build reactive, type-safe IndexedDB applications with the power of Drizzle's schema management.
+TanStack DB collections backed by IndexedDB with automatic migrations powered by Drizzle ORM. Build reactive, type-safe IndexedDB applications with the power of Drizzle's schema management.
 
 > **⚠️ Early WIP Notice:** This package is in very early development and is **not production-ready**. It is TypeScript-only and may have breaking changes. While I (the maintainer) have limited time, I'm open to PRs for features, bug fixes, or additional support (like JS builds). Please feel free to try it out and contribute! See [CONTRIBUTING.md](../../CONTRIBUTING.md) for details.
 
-> **Note:** This package currently builds on top of Drizzle's SQLite integration (using `drizzle-orm/sqlite-core` types and snapshots) until Drizzle adds native IndexedDB support. The migration system reads Drizzle's SQLite snapshots and translates them into IndexedDB object stores and indexes.
+> **Note:** This package currently builds on top of Drizzle's SQLite integration (using `drizzle-orm/sqlite-core` types) until Drizzle adds native IndexedDB support. The migration system uses function-based migrations generated from Drizzle's SQLite migrations to create IndexedDB object stores and indexes.
 
 ## Installation
 
@@ -19,8 +19,8 @@ npm install @firtoz/drizzle-indexeddb @firtoz/drizzle-utils drizzle-orm @tanstac
 - 🔍 **Query optimization** - Leverage IndexedDB indexes for fast queries
 - 📦 **Soft deletes** - Built-in support for `deletedAt` column
 - ⚛️ **React hooks** - Provider and hooks for easy React integration
-- 🔄 **Snapshot-based migrations** - Use Drizzle's generated snapshots to migrate IndexedDB
-- 📝 **Function-based migrations** - Write custom migration functions for complex changes
+- 📝 **Function-based migrations** - Generated migration functions from Drizzle schema changes
+- 🔄 **Multi-client sync** - IDB Proxy system for real-time sync across multiple clients (Chrome extensions, etc.)
 
 ## Quick Start
 
@@ -40,22 +40,25 @@ export const todoTable = syncableTable("todos", {
 ### 2. Generate Migrations
 
 ```bash
-# Generate Drizzle snapshots
+# Generate Drizzle migrations
 drizzle-kit generate
+
+# Generate IndexedDB migration functions
+bun drizzle-indexeddb-generate
 ```
 
 ### 3. Migrate IndexedDB
 
 ```typescript
 // db.ts
-import { migrateIndexedDB } from "@firtoz/drizzle-indexeddb";
-import journal from "./drizzle/meta/_journal.json";
-import * as snapshots from "./drizzle/snapshots";
-
-export const db = await migrateIndexedDB("my-app", {
-  journal,
-  snapshots,
-}, true); // Enable debug logging
+import { migrateIndexedDBWithFunctions } from "@firtoz/drizzle-indexeddb";
+import migrations from "./drizzle/indexeddb-migrations";
+
+export const db = await migrateIndexedDBWithFunctions(
+  "my-app",
+  migrations,
+  true // Enable debug logging
+);
 ```
 
 ### 4. Use with React
@@ -181,19 +184,19 @@ collection.find({
 
 ## Migration Methods
 
-### Snapshot-Based Migration
+### Function-Based Migration
 
-Use Drizzle's snapshot files to automatically migrate your IndexedDB schema:
+Use generated migration functions to migrate your IndexedDB schema:
 
 ```typescript
-import { migrateIndexedDB } from "@firtoz/drizzle-indexeddb";
-import journal from "./drizzle/meta/_journal.json";
-import * as snapshots from "./drizzle/snapshots";
+import { migrateIndexedDBWithFunctions } from "@firtoz/drizzle-indexeddb";
+import migrations from "./drizzle/indexeddb-migrations";
 
-const db = await migrateIndexedDB("my-app-db", {
-  journal,
-  snapshots,
-}, true); // debug flag
+const db = await migrateIndexedDBWithFunctions(
+  "my-app-db",
+  migrations,
+  true // debug flag
+);
 
 console.log("Database migrated successfully!");
 ```
@@ -219,40 +222,49 @@ interface MigrationRecord {
 }
 ```
 
-### Function-Based Migration
+### Custom Migration Functions
 
-For complex migrations that require custom logic:
+For complex migrations that require custom logic, you can write migration functions directly:
 
 ```typescript
 import { migrateIndexedDBWithFunctions } from "@firtoz/drizzle-indexeddb";
 
 const migrations = [
   // Migration 0: Initial schema
-  async (db: IDBDatabase, transaction: IDBTransaction) => {
-    const store = db.createObjectStore("todos", { keyPath: "id" });
-    store.createIndex("title", "title", { unique: false });
+  {
+    tag: "0000_initial",
+    migrate: async (db: IDBDatabase, transaction: IDBTransaction) => {
+      const store = db.createObjectStore("todos", { keyPath: "id" });
+      store.createIndex("title", "title", { unique: false });
+    },
   },
   
   // Migration 1: Add completed index
-  async (db: IDBDatabase, transaction: IDBTransaction) => {
-    const store = transaction.objectStore("todos");
-    store.createIndex("completed", "completed", { unique: false });
+  {
+    tag: "0001_add_completed",
+    migrate: async (db: IDBDatabase, transaction: IDBTransaction) => {
+      const store = transaction.objectStore("todos");
+      store.createIndex("completed", "completed", { unique: false });
+    },
   },
   
   // Migration 2: Transform data
-  async (db: IDBDatabase, transaction: IDBTransaction) => {
-    const store = transaction.objectStore("todos");
-    const todos = await new Promise<any[]>((resolve, reject) => {
-      const req = store.getAll();
-      req.onsuccess = () => resolve(req.result);
-      req.onerror = () => reject(req.error);
-    });
-    
-    // Transform data
-    for (const todo of todos) {
-      todo.priority = todo.priority || "medium";
-      store.put(todo);
-    }
+  {
+    tag: "0002_add_priority",
+    migrate: async (db: IDBDatabase, transaction: IDBTransaction) => {
+      const store = transaction.objectStore("todos");
+      const todos = await new Promise<any[]>((resolve, reject) => {
+        const req = store.getAll();
+        req.onsuccess = () => resolve(req.result);
+        req.onerror = () => reject(req.error);
+      });
+      
+      // Transform data
+      for (const todo of todos) {
+        todo.priority = todo.priority || "medium";
+        store.put(todo);
+      }
+    },
   },
 ];
 
@@ -338,6 +350,56 @@ Useful for:
 - Clearing user data on logout
 - Testing scenarios
 
+### generateIndexedDBMigrations
+
+Generate IndexedDB migration files from Drizzle snapshots programmatically:
+
+```typescript
+import { generateIndexedDBMigrations } from "@firtoz/drizzle-indexeddb";
+
+generateIndexedDBMigrations({
+  drizzleDir: "./drizzle",           // Path to Drizzle directory (default: ./drizzle)
+  outputDir: "./drizzle/indexeddb-migrations",  // Output directory (default: ./drizzle/indexeddb-migrations)
+});
+```
+
+## CLI
+
+The package includes a CLI tool to generate IndexedDB migrations from Drizzle schema snapshots.
+
+### Usage
+
+```bash
+# Generate migrations (run after drizzle-kit generate)
+bun drizzle-indexeddb-generate
+
+# With custom paths
+bun drizzle-indexeddb-generate --drizzle-dir ./db/drizzle
+bun drizzle-indexeddb-generate --output-dir ./src/migrations
+
+# Show help
+bun drizzle-indexeddb-generate --help
+```
+
+### Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `--drizzle-dir <path>` | Path to Drizzle directory | `./drizzle` |
+| `--output-dir <path>` | Path to output directory | `./drizzle/indexeddb-migrations` |
+
+### npm scripts
+
+Add to your `package.json`:
+
+```json
+{
+  "scripts": {
+    "db:generate": "bun drizzle-kit generate && bun drizzle-indexeddb-generate"
+  }
+}
+```
+
 ## Advanced Usage
 
 ### Custom Sync Configuration
@@ -355,17 +417,178 @@ const collection = createCollection(
 );
 ```
 
+### Collection Truncate
+
+Clear all data from a collection:
+
+```typescript
+// Clear all todos
+await todoCollection.utils.truncate();
+```
+
+This clears the IndexedDB store and updates the local reactive store.
+
+## IDB Proxy System
+
+For scenarios where IndexedDB needs to be accessed over a messaging layer (e.g., Chrome extensions, WebSockets), the proxy system enables multi-client sync:
+
+### Overview
+
+```
+┌─────────┐     ┌─────────┐     ┌─────────┐
+│ Client 1│     │ Client 2│     │ Client N│
+└────┬────┘     └────┬────┘     └────┬────┘
+     │               │               │
+     └───────────────┼───────────────┘
+                     │
+              ┌──────▼──────┐
+              │   Server    │
+              │  (manages   │
+              │  IndexedDB) │
+              └─────────────┘
+```
+
+- **Server** manages database lifecycle, migrations, and broadcasts mutations
+- **Clients** connect via a transport layer and receive real-time sync updates
+- All insert/update/delete/truncate operations sync to all connected clients
+
+### Basic Setup
+
+```typescript
+import {
+  createMultiClientTransport,
+  createProxyServer,
+  createProxyDbCreator,
+  migrateIndexedDBWithFunctions,
+  DrizzleIndexedDBProvider,
+} from "@firtoz/drizzle-indexeddb";
+
+// Create transport (in-memory for testing, or custom for production)
+const { createClientTransport, serverTransport } = createMultiClientTransport();
+
+// Create server with migrations
+const server = createProxyServer({
+  transport: serverTransport,
+  dbCreator: async (dbName) => {
+    return await migrateIndexedDBWithFunctions(dbName, migrations);
+  },
+});
+
+// Create client
+const clientTransport = createClientTransport();
+const dbCreator = createProxyDbCreator(clientTransport);
+
+// Use with React provider
+function App() {
+  const handleSyncReady = useCallback((handleSync) => {
+    clientTransport.onSync(handleSync);
+  }, []);
+
+  return (
+    <DrizzleIndexedDBProvider
+      dbName="my-app.db"
+      schema={schema}
+      dbCreator={dbCreator}
+      onSyncReady={handleSyncReady}
+    >
+      <YourApp />
+    </DrizzleIndexedDBProvider>
+  );
+}
+```
+
+### Multiple Clients
+
+```typescript
+// Server setup (once)
+const { createClientTransport, serverTransport } = createMultiClientTransport();
+const server = createProxyServer({ transport: serverTransport, ... });
+
+// Each client gets its own transport
+const client1Transport = createClientTransport();
+const client2Transport = createClientTransport();
+const client3Transport = createClientTransport();
+
+// All clients share the same data and receive real-time sync
+```
+
+### Sync Operations
+
+All standard collection operations automatically sync:
+
+```typescript
+// Client 1 inserts
+await todoCollection.insert({ title: "Buy milk", completed: false });
+// → Client 2, 3, N receive the new todo instantly
+
+// Client 2 updates
+await todoCollection.update(todoId, (draft) => {
+  draft.completed = true;
+});
+// → Client 1, 3, N see the update instantly
+
+// Client 3 deletes
+await todoCollection.delete(todoId);
+// → Client 1, 2, N see the deletion instantly
+
+// Client N truncates
+await todoCollection.utils.truncate();
+// → All clients are cleared instantly
+```
+
+### Custom Transport
+
+For production use (Chrome extension, WebSocket, etc.), implement the transport interface:
+
+```typescript
+import type { IDBProxyClientTransport, IDBProxyServerTransport } from "@firtoz/drizzle-indexeddb";
+
+// Client transport (e.g., in content script)
+const clientTransport: IDBProxyClientTransport = {
+  sendRequest: async (request) => {
+    // Send to background script and wait for response
+    return await chrome.runtime.sendMessage(request);
+  },
+  onSync: (handler) => {
+    // Listen for sync broadcasts
+    chrome.runtime.onMessage.addListener((msg) => {
+      if (msg.type?.startsWith("sync:")) handler(msg);
+    });
+  },
+};
+
+// Server transport (e.g., in background script)
+const serverTransport: IDBProxyServerTransport = {
+  onRequest: (handler) => {
+    chrome.runtime.onMessage.addListener(async (msg, sender, sendResponse) => {
+      const response = await handler(msg);
+      sendResponse(response);
+    });
+  },
+  broadcast: (message, excludeClientId) => {
+    // Broadcast to all connected tabs except sender
+    chrome.tabs.query({}, (tabs) => {
+      for (const tab of tabs) {
+        if (tab.id !== excludeClientId) {
+          chrome.tabs.sendMessage(tab.id, message);
+        }
+      }
+    });
+  },
+};
+```
+
 ### Handling Migration Errors
 
 ```typescript
 try {
-  const db = await migrateIndexedDB("my-app", config, true);
+  const db = await migrateIndexedDBWithFunctions("my-app", migrations, true);
 } catch (error) {
   console.error("Migration failed:", error);
   
   // Option 1: Delete and start fresh
   await deleteIndexedDB("my-app");
-  const db = await migrateIndexedDB("my-app", config, true);
+  const db = await migrateIndexedDBWithFunctions("my-app", migrations, true);
   
   // Option 2: Handle specific errors
   if (error.message.includes("Primary key structure changed")) {
@@ -378,10 +601,10 @@ try {
 
 ```typescript
 // Enable debug mode to see performance metrics
-const db = await migrateIndexedDB("my-app", config, true);
+const db = await migrateIndexedDBWithFunctions("my-app", migrations, true);
 
 // Output shows:
-// [PERF] IndexedDB snapshot migrator start for my-app
+// [PERF] IndexedDB function migrator start for my-app
 // [PERF] Latest applied migration index: 5 (checked 5 migrations)
 // [PERF] Found 2 pending migrations to apply: ["add_priority", "add_category"]
 // [PERF] Upgrade started: v5 → v7
@@ -432,9 +655,9 @@ const todoTable = syncableTable("todos", {
 
 ### Renaming a Column
 
-Drizzle snapshots don't track renames directly, but you can:
+Drizzle migrations don't track renames directly, but you can:
 
-1. Use function-based migrations to handle data transformation
+1. Modify the generated migration function to handle data transformation
 2. Or: Add new column, copy data, delete old column (3 separate migrations)
 
 ### Deleting a Table
@@ -455,8 +678,8 @@ This happens when you change the primary key of a table. IndexedDB doesn't suppo
 
 ### Migrations Not Applying
 
-- Check that journal and snapshots are correctly imported
-- Verify the snapshot files exist in `drizzle/snapshots/`
+- Check that migrations are correctly imported from `drizzle/indexeddb-migrations/`
+- Verify the migration files exist - run `bun drizzle-indexeddb-generate` to regenerate
 - Enable debug mode to see what's happening
 - Check browser DevTools → Application → IndexedDB
 

package/package.json

@@ -1,17 +1,25 @@
 {
 	"name": "@firtoz/drizzle-indexeddb",
-	"version": "0.2.0",
-	"description": "IndexedDB migrations powered by Drizzle ORM snapshots",
+	"version": "0.4.0",
+	"description": "IndexedDB migrations powered by Drizzle ORM",
 	"main": "./src/index.ts",
 	"module": "./src/index.ts",
 	"types": "./src/index.ts",
 	"type": "module",
+	"bin": {
+		"drizzle-indexeddb-generate": "./src/bin/generate-migrations.ts"
+	},
 	"exports": {
 		".": {
 			"types": "./src/index.ts",
 			"import": "./src/index.ts",
 			"require": "./src/index.ts"
 		},
+		"./generate": {
+			"types": "./src/bin/generate-migrations.ts",
+			"import": "./src/bin/generate-migrations.ts",
+			"require": "./src/bin/generate-migrations.ts"
+		},
 		"./*": {
 			"types": "./src/*.ts",
 			"import": "./src/*.ts",
@@ -56,16 +64,16 @@
 		"access": "public"
 	},
 	"dependencies": {
-		"@firtoz/drizzle-utils": "^0.2.0",
-		"@tanstack/db": "^0.5.0",
+		"@firtoz/drizzle-utils": "^0.3.0",
+		"@tanstack/db": "^0.5.10",
 		"drizzle-orm": "^0.44.7",
 		"drizzle-valibot": "^0.4.2",
-		"valibot": "^1.0.0"
+		"valibot": "^1.2.0"
 	},
 	"peerDependencies": {
 		"react": "^19.2.0"
 	},
 	"devDependencies": {
-		"@types/react": "^19.2.5"
+		"@types/react": "^19.2.7"
 	}
 }