or3-provider-sqlite 0.0.4 → 0.0.7

package/README.md

@@ -7,6 +7,9 @@ SQLite sync and workspace store provider for OR3 Chat. Provides a lightweight, s
 - **AuthWorkspaceStore** (`sqlite`) — user identity mapping, workspace CRUD, role resolution
 - **SyncGatewayAdapter** (`sqlite`) — push/pull sync, consistent materialized snapshot pages, LWW conflict resolution, and cursor tracking
 - **ConnectStore** (`sqlite`) — durable, atomic device enrollment and connected-computer records for OR3 Connect
+- **WebhookStore** (`sqlite`) — durable webhook registrations and delivery logs (local, Bun, and Turso runtimes)
+- **Admin stores** (`sqlite`) — workspace access/lifecycle, workspace settings, user search, and deployment-admin grants (local, Bun, and Turso runtimes)
+- **RateLimitProvider** (`sqlite`) — durable rate-limit counters
 
 ## Install
 
@@ -20,19 +23,20 @@ Add to your provider module list (e.g. `or3.providers.generated.ts`):
 export default ['or3-provider-sqlite/nuxt'];
 ```
 
-## Configuration
+## Native runtimes and configuration
 
-### Environment Variables
+`better-sqlite3` remains the default, so existing local-file installations do
+not need to change. Set `OR3_SQLITE_DRIVER` only when selecting another native
+runtime.
 
-| Variable | Required | Default | Description |
+| Runtime | `OR3_SQLITE_DRIVER` | Required configuration | Extra install |
 |---|---|---|---|
-| `OR3_SQLITE_DB_PATH` | Yes (non-test) | None | Path to SQLite database file |
-| `OR3_SQLITE_PRAGMA_JOURNAL_MODE` | No | `WAL` | SQLite journal mode |
-| `OR3_SQLITE_PRAGMA_SYNCHRONOUS` | No | `NORMAL` | SQLite synchronous pragma |
-| `OR3_SQLITE_ALLOW_IN_MEMORY` | No | `false` | Allow `:memory:` in non-test environments (ephemeral data) |
-| `OR3_SQLITE_STRICT` | No | `false` | Fail startup if `:memory:` is used |
+| Local Node (default) | `better-sqlite3` | `OR3_SQLITE_DB_PATH` | `better-sqlite3` |
+| Bun | `bun` | `OR3_SQLITE_DB_PATH` | None; uses built-in `bun:sqlite` |
+| Turso/libSQL | `turso` | `OR3_SQLITE_TURSO_URL`, `OR3_SQLITE_TURSO_AUTH_TOKEN` | `libsql` |
+| Cloudflare D1 | `d1` | `OR3_SQLITE_D1_BINDING` (defaults to `DB`) | None; uses the Worker binding |
 
-### Production example
+### Local Node (existing default)
 
 ```bash
 OR3_SQLITE_DB_PATH=/data/or3-sync.db
@@ -40,15 +44,73 @@ OR3_SQLITE_PRAGMA_JOURNAL_MODE=WAL
 OR3_SQLITE_PRAGMA_SYNCHRONOUS=NORMAL
 ```
 
-OR3 Connect uses this same database by default:
+### Bun
+
+```bash
+OR3_SQLITE_DRIVER=bun
+OR3_SQLITE_DB_PATH=/data/or3-sync.db
+```
+
+### Turso/libSQL
+
+```bash
+bun add libsql
+```
+
+```bash
+OR3_SQLITE_DRIVER=turso
+OR3_SQLITE_TURSO_URL=libsql://your-database.turso.io
+OR3_SQLITE_TURSO_AUTH_TOKEN=your-server-only-token
+```
+
+### Cloudflare D1
+
+Run the application in a Cloudflare Worker that already has a D1 binding, then
+set its binding name:
+
+```toml
+[[d1_databases]]
+binding = "DB"
+database_name = "or3"
+database_id = "your-database-id"
+```
+
+```bash
+OR3_SQLITE_DRIVER=d1
+OR3_SQLITE_D1_BINDING=DB
+```
+
+The provider initializes D1 and applies pending migrations on the first Worker
+request, which keeps D1 I/O inside Cloudflare's request context.
+
+For D1, use Workers-compatible auth and storage providers. The local Basic
+Auth and filesystem providers depend on Node-native local storage and are not
+Workers-compatible. OR3 Connect with D1 throws at startup, persistent webhooks
+are not registered (a warning is logged), and server-side admin stores are
+registered with all capabilities disabled; the install wizard reports these
+boundaries explicitly.
+
+`OR3_SQLITE_PRAGMA_*`, `OR3_SQLITE_ALLOW_IN_MEMORY`, and
+`OR3_SQLITE_STRICT` apply only to local-file runtimes. The Turso auth token is
+server-only and must not be exposed through public runtime configuration.
+
+| Variable | Default | Notes |
+|---|---|---|
+| `OR3_SQLITE_PRAGMA_JOURNAL_MODE` | `WAL` | better-sqlite3 and Bun only |
+| `OR3_SQLITE_PRAGMA_SYNCHRONOUS` | `NORMAL` | better-sqlite3 and Bun only |
+| `OR3_SQLITE_ALLOW_IN_MEMORY` | unset | required to run on `:memory:` outside tests; data is lost on restart |
+| `OR3_SQLITE_STRICT` | unset | forbids `:memory:`; requires `OR3_SQLITE_DB_PATH` |
+
+`OR3_SQLITE_DB_PATH` is required in non-test environments unless
+`OR3_SQLITE_ALLOW_IN_MEMORY=true`; otherwise startup fails with a clear error.
+
+OR3 Connect uses the same database with local, Bun, or Turso runtimes:
 
 ```bash
 OR3_CONNECT_ENABLED=true
 OR3_CONNECT_PROVIDER=sqlite
 ```
 
-No Convex deployment or second database is required.
-
 ## How it works
 
 ### Registration
@@ -60,8 +122,12 @@ On server startup, the Nitro plugin:
 3. Registers `AuthWorkspaceStore` with ID `sqlite`
 4. Registers `SyncGatewayAdapter` when SQLite sync is selected
 5. Registers `ConnectStore` when SQLite Connect persistence is selected
+6. Registers `WebhookStore` and the admin stores (workspace access, workspace
+   settings, user search) — all runtimes except D1
+7. Registers `RateLimitProvider` with ID `sqlite` and the sync admin adapter
 
-Registration is skipped when `auth.enabled` is `false` (local-only mode).
+Registration is skipped when `auth.enabled` is `false`, or when neither SQLite
+sync nor SQLite Connect is selected (local-only mode).
 
 ### Schema
 
@@ -73,19 +139,23 @@ Ordered migrations create and evolve all tables:
 - **006_sync_snapshots**: winning operation IDs plus immutable snapshot headers/items
 - **009_or3_connect**: single-use device authorizations and connected computers
 
+Additional migrations (007–008, 010–016) evolve device-cursor ownership,
+upload intents, and Connect credential/lifecycle hardening and rate limits.
+
 All tables use snake_case aligned with the sync wire format.
 
 ### Sync semantics
 
-- **Push**: validates ops → checks `op_id` idempotency → allocates contiguous `server_version` block → applies LWW to materialized tables → writes change_log → upserts tombstones for deletes
+- **Push**: validates ops → checks `op_id` idempotency → allocates contiguous `server_version` block → writes change_log → applies LWW to materialized tables → upserts tombstones for deletes
 - **Pull**: returns ordered changes for `server_version > cursor` with limit/pagination and optional table filtering
-- **Snapshot**: captures canonical live rows and current tombstones with one `highWatermark` under `BEGIN IMMEDIATE`, then serves immutable, keyset-paginated pages ordered by `(tableName, pk, kind)`
+- **Snapshot**: captures canonical live rows and current tombstones at one `highWatermark`, then serves immutable, keyset-paginated pages ordered by `(tableName, pk, kind)`
 - **Cursor**: forward-only per-device cursor tracking
 - **Retention safety**: tombstone and `change_log` GC is enabled only under the explicit `snapshot-v1` capability and deletes old revisions acknowledged by every registered device
 
 LWW conflict resolution: incoming wins when `clock` is higher, or when clocks are equal and `hlc` is lexicographically greater.
 
-Push uses `BEGIN IMMEDIATE` transactions to prevent concurrent server_version races.
+Local, Bun, and Turso runtimes use `BEGIN IMMEDIATE` transactions. D1 uses its
+native atomic batch API for grouped writes.
 
 ### Workspace store
 
@@ -95,13 +165,16 @@ Push uses `BEGIN IMMEDIATE` transactions to prevent concurrent server_version ra
 
 ## Backup
 
-Since everything lives in a single SQLite file:
+For local-file and Bun runtimes, everything lives in a single SQLite file:
 
 ```bash
 # While the app is running (WAL mode supports this)
 sqlite3 /data/or3-sync.db ".backup /backup/or3-sync-$(date +%s).db"
 ```
 
+Use Turso or Cloudflare's own backup/export facilities for their managed
+databases.
+
 ## Development
 
 ```bash
@@ -115,11 +188,31 @@ bun run build       # build for distribution
 
 - Works with multiple auth providers (`basic-auth`, `clerk`, or custom)
 - Replaces `or3-provider-convex` for sync + workspace store functionality
-- Provides OR3 Connect persistence without Convex
+- Provides OR3 Connect persistence without Convex for local, Bun, and Turso runtimes
 - Does NOT provide storage — pair with `or3-provider-fs` for file storage
 
 ### Known differences vs Convex
 
 - Single-process SQLite vs distributed Convex backend
 - No real-time subscriptions (gateway polling only)
-- Migrations run on boot; schema changes require restart
+- Migrations run on boot, or on the first Worker request for D1; schema changes require restart
+
+## Troubleshooting
+
+- **`OR3_SQLITE_DB_PATH is required in non-test environments`** — set a file
+  path, or pass `OR3_SQLITE_ALLOW_IN_MEMORY=true` only for ephemeral storage.
+- **`OR3_SQLITE_STRICT=true forbids in-memory SQLite`** — the two settings
+  cannot be combined; set `OR3_SQLITE_DB_PATH`.
+- **`Unsupported OR3_SQLITE_DRIVER value`** — use `better-sqlite3`, `bun`,
+  `turso`, or `d1`.
+- **`Unable to load better-sqlite3`** — install `better-sqlite3`, or select
+  another driver via `OR3_SQLITE_DRIVER`.
+- **`OR3_SQLITE_TURSO_URL` / `OR3_SQLITE_TURSO_AUTH_TOKEN` required** — both
+  must be set when `OR3_SQLITE_DRIVER=turso`, and `libsql` must be installed.
+- **`Cloudflare D1 binding "…" was not found`** — set `OR3_SQLITE_D1_BINDING`
+  to the binding name declared in your Worker's `wrangler.jsonc`.
+- **`Cloudflare D1 supports Auth and Sync, but OR3 Connect still requires a
+  synchronous SQLite runtime`** — Connect is unavailable with D1; set
+  `OR3_CONNECT_ENABLED=false` or switch to better-sqlite3, Bun, or Turso.
+- **`OR3_SQLITE_DRIVER=bun requires Bun`** — the Bun driver only runs under
+  the Bun runtime with its built-in `bun:sqlite`.

package/dist/module.json

@@ -1,7 +1,7 @@
 {
   "name": "or3-provider-sqlite",
   "configKey": "or3-provider-sqlite",
-  "version": "0.0.4",
+  "version": "0.0.7",
   "builder": {
     "@nuxt/module-builder": "1.0.2",
     "unbuild": "3.6.1"

package/dist/runtime/server/admin/adapters/sync-sqlite.js

@@ -1,38 +1,85 @@
 import { createError } from "h3";
-import { SYNC_HISTORY_GC_POLICY } from "../../sync/history-gc-policy.js";
+import { SYNC_HISTORY_GC_POLICY, canRunSyncHistoryGc } from "../../sync/history-gc-policy.js";
+import {
+  beginSyncMaintenanceRun,
+  completeSyncMaintenanceRun,
+  computeSyncMaintenanceBacklog,
+  failSyncMaintenanceRun,
+  getSyncMaintenanceState,
+  SYNC_MAINTENANCE_RETENTION_SECONDS
+} from "../../sync/maintenance-state.js";
+import { createSqliteSyncGatewayAdapter } from "../../sync/sqlite-sync-gateway-adapter.js";
+import { getSqliteDriver } from "../../db/kysely.js";
 const SQLITE_PROVIDER_ID = "sqlite";
+let gatewayInstance;
+function getGateway() {
+  gatewayInstance ??= createSqliteSyncGatewayAdapter();
+  return gatewayInstance;
+}
 export const sqliteSyncAdminAdapter = {
   id: SQLITE_PROVIDER_ID,
   kind: "sync",
   async getStatus(_event, _ctx) {
+    const driver = getSqliteDriver();
+    const isLocalDriver = driver === "better-sqlite3" || driver === "bun";
     const dbPath = process.env.OR3_SQLITE_DB_PATH;
     const warnings = [];
-    if (!dbPath) {
+    if (isLocalDriver && !dbPath) {
       warnings.push({
         level: "warning",
         message: "OR3_SQLITE_DB_PATH is not set. SQLite may run in ephemeral mode."
       });
-    } else if (dbPath === ":memory:") {
+    } else if (isLocalDriver && dbPath === ":memory:") {
       warnings.push({
         level: "warning",
         message: "SQLite is configured with :memory:. Data will not persist across restarts."
       });
     }
-    warnings.push({
-      level: "warning",
-      message: SYNC_HISTORY_GC_POLICY.reason
-    });
+    if (driver === "d1") {
+      warnings.push({
+        level: "warning",
+        message: "Cloudflare D1 does not provide server-side admin stores or persistent webhooks."
+      });
+    }
+    const maintenance = getSyncMaintenanceState();
+    if (maintenance.state === "failed") {
+      warnings.push({
+        level: "warning",
+        message: `Sync history maintenance failed: ${maintenance.lastError ?? "unknown error"}`
+      });
+    }
+    const backlog = computeSyncMaintenanceBacklog(SYNC_MAINTENANCE_RETENTION_SECONDS);
     return {
       details: {
-        dbPath: dbPath ?? ":memory:",
-        journalMode: process.env.OR3_SQLITE_PRAGMA_JOURNAL_MODE ?? "WAL",
-        synchronous: process.env.OR3_SQLITE_PRAGMA_SYNCHRONOUS ?? "NORMAL"
+        driver,
+        dbPath: isLocalDriver ? dbPath ?? ":memory:" : void 0,
+        tursoUrl: driver === "turso" ? process.env.OR3_SQLITE_TURSO_URL : void 0,
+        d1Binding: driver === "d1" ? process.env.OR3_SQLITE_D1_BINDING ?? "DB" : void 0,
+        journalMode: isLocalDriver ? process.env.OR3_SQLITE_PRAGMA_JOURNAL_MODE ?? "WAL" : void 0,
+        synchronous: isLocalDriver ? process.env.OR3_SQLITE_PRAGMA_SYNCHRONOUS ?? "NORMAL" : void 0,
+        maintenance: {
+          ...maintenance,
+          backlog
+        }
       },
       warnings,
-      actions: []
+      actions: [
+        {
+          id: "sync.gc-change-log",
+          label: "GC change log",
+          description: `Prune change-log entries older than ${SYNC_MAINTENANCE_RETENTION_SECONDS / 86400} days and behind every device cursor.`,
+          danger: true
+        },
+        {
+          id: "sync.gc-tombstones",
+          label: "GC tombstones",
+          description: `Prune tombstones older than ${SYNC_MAINTENANCE_RETENTION_SECONDS / 86400} days and behind every device cursor.`,
+          danger: true
+        }
+      ]
     };
   },
-  async runAction(_event, actionId, _payload, ctx) {
+  async runAction(event, actionId, _payload, ctx) {
     if (!ctx.session.workspace?.id) {
       throw createError({
         statusCode: 400,
@@ -40,10 +87,37 @@ export const sqliteSyncAdminAdapter = {
       });
     }
     if (actionId === "sync.gc-change-log" || actionId === "sync.gc-tombstones") {
-      throw createError({
-        statusCode: 503,
-        statusMessage: SYNC_HISTORY_GC_POLICY.reason
-      });
+      if (!canRunSyncHistoryGc()) {
+        throw createError({
+          statusCode: 503,
+          statusMessage: SYNC_HISTORY_GC_POLICY.reason
+        });
+      }
+      const workspaceId = ctx.session.workspace.id;
+      const retentionSeconds = SYNC_MAINTENANCE_RETENTION_SECONDS;
+      const gateway = getGateway();
+      beginSyncMaintenanceRun();
+      try {
+        if (actionId === "sync.gc-change-log") {
+          await gateway.gcChangeLog(event, { scope: { workspaceId }, retentionSeconds });
+        } else {
+          await gateway.gcTombstones(event, { scope: { workspaceId }, retentionSeconds });
+        }
+        const backlog = computeSyncMaintenanceBacklog(retentionSeconds, true);
+        const lastRun = (/* @__PURE__ */ new Date()).toISOString();
+        completeSyncMaintenanceRun({ lastRun, backlog });
+        return {
+          ok: true,
+          actionId,
+          workspaceId,
+          retentionSeconds,
+          lastRun,
+          backlog
+        };
+      } catch (error) {
+        failSyncMaintenanceRun(error instanceof Error ? error.message : String(error));
+        throw error;
+      }
     }
     throw createError({ statusCode: 400, statusMessage: "Unknown action" });
   }

package/dist/runtime/server/admin/stores/sqlite-store.js

@@ -1,4 +1,3 @@
-import { randomUUID } from "node:crypto";
 import { getSqliteDb } from "../../db/kysely.js";
 function nowEpoch() {
   return Math.floor(Date.now() / 1e3);
@@ -40,7 +39,7 @@ class SqliteWorkspaceAccessStore {
       if (existing) {
         userId = existing.id;
       } else {
-        userId = randomUUID();
+        userId = globalThis.crypto.randomUUID();
         await this.db.insertInto("users").values({
           id: userId,
           email: normalized,
@@ -60,7 +59,7 @@ class SqliteWorkspaceAccessStore {
         }
       }
       if (!userId) {
-        userId = randomUUID();
+        userId = globalThis.crypto.randomUUID();
         await this.db.insertInto("users").values({
           id: userId,
           email: null,
@@ -69,7 +68,7 @@ class SqliteWorkspaceAccessStore {
           created_at: now
         }).execute();
         await this.db.insertInto("auth_accounts").values({
-          id: randomUUID(),
+          id: globalThis.crypto.randomUUID(),
           user_id: userId,
           provider: provider || "custom",
           provider_user_id: lookup,
@@ -89,7 +88,7 @@ class SqliteWorkspaceAccessStore {
       return;
     }
     await this.db.insertInto("workspace_members").values({
-      id: randomUUID(),
+      id: globalThis.crypto.randomUUID(),
       workspace_id: input.workspaceId,
       user_id: userId,
       role: input.role,
@@ -249,7 +248,7 @@ class SqliteWorkspaceAccessStore {
     if (!owner) {
       throw new Error("Owner user not found");
     }
-    const workspaceId = randomUUID();
+    const workspaceId = globalThis.crypto.randomUUID();
     const now = nowEpoch();
     await this.db.transaction().execute(async (tx) => {
       await tx.insertInto("workspaces").values({
@@ -262,7 +261,7 @@ class SqliteWorkspaceAccessStore {
         deleted_at: null
       }).execute();
       await tx.insertInto("workspace_members").values({
-        id: randomUUID(),
+        id: globalThis.crypto.randomUUID(),
         workspace_id: workspaceId,
         user_id: input.ownerUserId,
         role: "owner",
@@ -312,7 +311,7 @@ class SqliteWorkspaceSettingsStore {
   }
   async set(workspaceId, key, value) {
     await this.db.insertInto("admin_workspace_settings").values({
-      id: randomUUID(),
+      id: globalThis.crypto.randomUUID(),
       workspace_id: workspaceId,
       key,
       value,

package/dist/runtime/server/auth/sqlite-auth-workspace-store.d.ts

@@ -122,5 +122,10 @@ export declare class SqliteAuthWorkspaceStore implements AuthWorkspaceStore {
         ok: false;
         reason: 'not_found' | 'expired' | 'revoked' | 'already_used' | 'token_mismatch';
     }>;
+    private getOrCreateUserInD1;
+    private validateInviteInD1;
+    private acceptInviteAndProvisionUserInD1;
+    private consumeInviteInD1;
+    private inviteProvisionFailureInD1;
 }
 export declare function createSqliteAuthWorkspaceStore(): AuthWorkspaceStore;