loro-repo 0.16.11 → 0.17.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -56,6 +56,133 @@ await repo.unloadDoc("note:welcome");
56
56
  - `WebSocketTransportAdapter` (`src/transport/websocket.ts`)
57
57
  - `IndexedDBStorageAdaptor` (`src/storage/indexeddb.ts`)
58
58
  - `FileSystemStorageAdaptor` (`src/storage/filesystem.ts`)
59
+ - `SqliteRepoStore` (`src/storage/sqlite.ts`, Node.js only)
60
+
61
+ ## SQLite storage (Node.js / Electron)
62
+
63
+ `SqliteRepoStore` packs everything a repo needs into a single SQLite database
64
+ file and is the recommended Node-side storage when the streams transport is in
65
+ use. Compared to `FileSystemStorageAdaptor`, it avoids creating many small
66
+ files for incremental updates, and it lets remote-cursor advances become
67
+ single-row `UPDATE`s instead of rewriting a large JSON blob.
68
+
69
+ A single `SqliteRepoStore` instance exposes two adapters that share one
70
+ database connection:
71
+
72
+ - `store.storage` — a `StorageAdapter` for doc snapshots, doc updates, and
73
+ metadata (Flock) snapshots/updates.
74
+ - `store.cursorStore` — a `RemoteCursorStore` for the streams transport's
75
+ `remoteCursorStore` option.
76
+
77
+ ```ts
78
+ import { LoroRepo } from "loro-repo";
79
+ import { SqliteRepoStore } from "loro-repo/storage/sqlite";
80
+ import { StreamsTransportAdapter } from "loro-repo/transport/streams";
81
+
82
+ const store = new SqliteRepoStore({ path: "./data/repo.db" });
83
+
84
+ const repo = await LoroRepo.create({
85
+ storageAdapter: store.storage,
86
+ transportAdapter: new StreamsTransportAdapter({
87
+ bucketId: "your-bucket",
88
+ auth: yourAuthProvider,
89
+ remoteCursorStore: store.cursorStore,
90
+ // Required when `remoteCursorStore` persists across restarts. Both hooks
91
+ // run inside `beforeRemoteCursorSave`, so the local doc / Flock state is
92
+ // durable *before* the cursor advances. Skipping them risks a crash that
93
+ // leaves the cursor pointing past data that was never persisted locally.
94
+ onPersistDoc: async (docId, doc) => {
95
+ await store.storage.save({
96
+ type: "doc-snapshot",
97
+ docId,
98
+ snapshot: doc.export({ mode: "snapshot" }),
99
+ });
100
+ },
101
+ onPersistMeta: async (flock) => {
102
+ await store.storage.save({
103
+ type: "meta-snapshot",
104
+ snapshot: flock.exportFile(),
105
+ });
106
+ },
107
+ }),
108
+ });
109
+
110
+ // Shutdown
111
+ await repo.destroy();
112
+ store.close();
113
+ ```
114
+
115
+ > ⚠️ `onPersistDoc` and `onPersistMeta` are not optional when you wire a
116
+ > durable `remoteCursorStore`. The streams transport advances cursors only
117
+ > after `beforeRemoteCursorSave` resolves; if local persistence does not run
118
+ > there, a crash can leave SQLite with stale doc/meta state while the cursor
119
+ > resumes after data that was never applied locally. The `InMemoryRemoteCursorStore`
120
+ > default is the only configuration where omitting the hooks is safe, since
121
+ > the cursor itself is lost on restart.
122
+
123
+ `SqliteRepoStoreOptions`:
124
+
125
+ | Option | Default | Description |
126
+ | --- | --- | --- |
127
+ | `path` | `":memory:"` | SQLite file path. Use `":memory:"` for an ephemeral store. |
128
+ | `database` | — | Pass a pre-constructed `better-sqlite3` `Database` to share a connection; the caller owns its lifetime. |
129
+ | `tablePrefix` | `""` | Optional prefix for all table names, useful when multiple repos share one database. |
130
+ | `wal` | `true` | Enables `journal_mode=WAL` + `synchronous=NORMAL` for fast, durable writes. |
131
+
132
+ The schema (created on first use) is:
133
+
134
+ ```
135
+ docs (doc_id PK, snapshot, updated_at, snapshot_digest)
136
+ doc_updates (id, doc_id, update_data, created_at)
137
+ meta_snapshot (id=1, snapshot, updated_at)
138
+ meta_updates (id, update_data, created_at)
139
+ remote_cursors (stream_url PK, next_offset, server_lower_bound_version, updated_at_ms)
140
+ ```
141
+
142
+ `loadDoc` / `loadMeta` opportunistically compact replayed updates into a fresh
143
+ snapshot inside a single transaction, mirroring the filesystem adapter's
144
+ behaviour but without any file churn.
145
+
146
+ ### Backups in WAL mode
147
+
148
+ With the default `wal: true`, SQLite writes to two sidecar files alongside
149
+ the main DB: `<path>-wal` and `<path>-shm`. Recent writes may live only in
150
+ the WAL until a checkpoint runs. If you copy the database file out for
151
+ backup, also copy both sidecars **or** force a checkpoint first — pass a
152
+ pre-constructed `Database` via the `database` option so your code keeps
153
+ access to the connection and can call `pragma("wal_checkpoint(TRUNCATE)")`
154
+ before snapshotting. Otherwise the `.db`-only copy can miss recent writes.
155
+
156
+ ### Installation
157
+
158
+ `better-sqlite3` is declared as an **optional** peer dependency. Install it
159
+ explicitly in projects that want SQLite storage, along with `@types/better-sqlite3`
160
+ for TypeScript users (the runtime package ships no `.d.ts` of its own and the
161
+ published `loro-repo/storage/sqlite` types reference it):
162
+
163
+ ```bash
164
+ pnpm add better-sqlite3
165
+ pnpm add -D @types/better-sqlite3
166
+ # or: npm install better-sqlite3 && npm install -D @types/better-sqlite3
167
+ ```
168
+
169
+ #### Node version compatibility
170
+
171
+ `loro-repo` itself supports Node ≥ 18. `better-sqlite3` versions matter:
172
+
173
+ | Node version | Use `better-sqlite3` |
174
+ | --- | --- |
175
+ | 18.x | `^11` (v11 is the last line that supports Node 18) |
176
+ | ≥ 20 | `^11` or `^12` |
177
+
178
+ If you're on Node 18, install `better-sqlite3@^11` explicitly — picking up v12
179
+ will fail at runtime with an unsupported-Node error. The `loro-repo`
180
+ implementation only uses APIs present in both lines, so either works.
181
+
182
+ Browsers should keep using `IndexedDBStorageAdaptor` (see the previous
183
+ section). The `loro-repo/storage/sqlite` entry is the only place that imports
184
+ `better-sqlite3`, so importing the main package or any other adapter never
185
+ pulls native bindings into a browser bundle.
59
186
 
60
187
  ## Core API surface
61
188
 
package/dist/index.cjs CHANGED
@@ -85,18 +85,29 @@ var RepoEventBus = class {
85
85
  function yieldToMacroTask() {
86
86
  return new Promise((resolve) => setTimeout(resolve, 0));
87
87
  }
88
+ const YIELD_BUDGET_MS = 5;
89
+ const nowMs = typeof globalThis.performance?.now === "function" ? globalThis.performance.now.bind(globalThis.performance) : Date.now;
88
90
  var DocManager = class {
89
91
  storage;
90
92
  ensureDocExists;
91
93
  eventBus;
94
+ persistDebounceMs;
92
95
  docs = /* @__PURE__ */ new Map();
93
96
  docSubscriptions = /* @__PURE__ */ new Map();
94
97
  docPersistedVersions = /* @__PURE__ */ new Map();
95
98
  pendingDocLoads = /* @__PURE__ */ new Map();
99
+ dirtyDocs = /* @__PURE__ */ new Set();
100
+ existenceMaterialized = /* @__PURE__ */ new Set();
101
+ debounceTimer;
102
+ flushScheduled = false;
103
+ operationPromise = Promise.resolve();
104
+ destroyed = false;
96
105
  constructor(options) {
97
106
  this.storage = options.storage;
98
107
  this.ensureDocExists = options.ensureDocExists;
99
108
  this.eventBus = options.eventBus;
109
+ const configured = options.persistDebounceMs;
110
+ this.persistDebounceMs = typeof configured === "number" && Number.isFinite(configured) && configured >= 0 ? configured : 0;
100
111
  }
101
112
  async openPersistedDoc(docId) {
102
113
  return this.ensureDoc(docId);
@@ -125,9 +136,11 @@ var DocManager = class {
125
136
  const previousVersion = this.docPersistedVersions.get(docId);
126
137
  const snapshot = doc.export({ mode: "snapshot" });
127
138
  const nextVersion = doc.version();
139
+ const oplogAtCapture = doc.oplogVersion();
128
140
  if (!this.storage) {
129
141
  this.docPersistedVersions.set(docId, nextVersion);
130
142
  await this.materializeExistence(docId, doc);
143
+ this.clearDirtyIfStable(docId, oplogAtCapture, doc);
131
144
  return;
132
145
  }
133
146
  this.docPersistedVersions.set(docId, nextVersion);
@@ -138,6 +151,7 @@ var DocManager = class {
138
151
  snapshot
139
152
  });
140
153
  await this.materializeExistence(docId, doc);
154
+ this.clearDirtyIfStable(docId, oplogAtCapture, doc);
141
155
  } catch (error) {
142
156
  if (previousVersion) this.docPersistedVersions.set(docId, previousVersion);
143
157
  else this.docPersistedVersions.delete(docId);
@@ -145,8 +159,10 @@ var DocManager = class {
145
159
  }
146
160
  }
147
161
  async materializeExistence(docId, doc) {
162
+ if (this.existenceMaterialized.has(docId)) return;
148
163
  if (doc.oplogFrontiers().length === 0) return;
149
164
  await this.ensureDocExists(docId);
165
+ this.existenceMaterialized.add(docId);
150
166
  }
151
167
  async unloadDoc(docId) {
152
168
  const doc = this.docs.get(docId);
@@ -156,29 +172,42 @@ var DocManager = class {
156
172
  this.docSubscriptions.delete(docId);
157
173
  this.docs.delete(docId);
158
174
  this.docPersistedVersions.delete(docId);
175
+ this.existenceMaterialized.delete(docId);
159
176
  }
160
177
  async dropDoc(docId) {
161
178
  this.docSubscriptions.get(docId)?.();
162
179
  this.docSubscriptions.delete(docId);
163
180
  this.docs.delete(docId);
164
181
  this.docPersistedVersions.delete(docId);
182
+ this.dirtyDocs.delete(docId);
183
+ this.existenceMaterialized.delete(docId);
165
184
  if (this.storage?.deleteDoc) await this.storage.deleteDoc(docId);
166
185
  }
167
186
  async flush() {
168
- for (const [docId, doc] of this.docs) {
169
- await this.persistDocUpdate(docId, doc);
170
- await yieldToMacroTask();
187
+ this.cancelDebounceTimer();
188
+ this.flushScheduled = false;
189
+ while (true) {
190
+ await this.enqueueOperation(() => this.drainDirtyDocs());
191
+ if (this.dirtyDocs.size === 0 && !this.flushScheduled) break;
192
+ this.flushScheduled = false;
171
193
  }
172
194
  }
173
195
  async close() {
174
- await this.flush();
175
- for (const unsubscribe of this.docSubscriptions.values()) try {
176
- unsubscribe();
177
- } catch {}
178
- this.docSubscriptions.clear();
179
- this.docs.clear();
180
- this.docPersistedVersions.clear();
181
- this.pendingDocLoads.clear();
196
+ this.destroyed = true;
197
+ this.cancelDebounceTimer();
198
+ try {
199
+ await this.flush();
200
+ } finally {
201
+ for (const unsubscribe of this.docSubscriptions.values()) try {
202
+ unsubscribe();
203
+ } catch {}
204
+ this.docSubscriptions.clear();
205
+ this.docs.clear();
206
+ this.docPersistedVersions.clear();
207
+ this.pendingDocLoads.clear();
208
+ this.dirtyDocs.clear();
209
+ this.existenceMaterialized.clear();
210
+ }
182
211
  }
183
212
  async loadOrCreateDoc(docId) {
184
213
  const cached = this.docs.get(docId);
@@ -229,14 +258,19 @@ var DocManager = class {
229
258
  if (!this.storage) {
230
259
  this.docPersistedVersions.set(docId, nextVersion);
231
260
  await this.materializeExistence(docId, doc);
261
+ this.clearDirtyIfStable(docId, nextVersion, doc);
232
262
  return;
233
263
  }
234
264
  if (!previousVersion) {
235
265
  await this.persistDoc(docId, doc);
236
266
  this.docPersistedVersions.set(docId, nextVersion);
267
+ this.clearDirtyIfStable(docId, nextVersion, doc);
268
+ return;
269
+ }
270
+ if (previousVersion.compare(nextVersion) === 0) {
271
+ this.dirtyDocs.delete(docId);
237
272
  return;
238
273
  }
239
- if (previousVersion.compare(nextVersion) === 0) return;
240
274
  const update = doc.export({
241
275
  mode: "update",
242
276
  from: previousVersion
@@ -249,13 +283,78 @@ var DocManager = class {
249
283
  update
250
284
  });
251
285
  await this.materializeExistence(docId, doc);
286
+ this.clearDirtyIfStable(docId, nextVersion, doc);
252
287
  } catch (error) {
253
288
  this.docPersistedVersions.set(docId, previousVersion);
254
289
  throw error;
255
290
  }
256
291
  }
292
+ clearDirtyIfStable(docId, capturedOplog, doc) {
293
+ if (doc.oplogVersion().compare(capturedOplog) === 0) this.dirtyDocs.delete(docId);
294
+ }
257
295
  onDocEvent(docId, doc, _batch, _by) {
258
- this.persistDocUpdate(docId, doc).catch(require_logging.logAsyncError(`doc ${docId} event processing`));
296
+ if (!this.docs.has(docId)) return;
297
+ this.dirtyDocs.add(docId);
298
+ this.scheduleDirtyFlush();
299
+ }
300
+ scheduleDirtyFlush() {
301
+ if (this.destroyed) return;
302
+ if (this.persistDebounceMs > 0) {
303
+ if (this.debounceTimer) clearTimeout(this.debounceTimer);
304
+ this.flushScheduled = true;
305
+ this.debounceTimer = setTimeout(() => {
306
+ this.debounceTimer = void 0;
307
+ this.flushScheduled = false;
308
+ this.enqueueOperation(() => this.drainDirtyDocs()).catch(require_logging.logAsyncError("doc dirty flush"));
309
+ }, this.persistDebounceMs);
310
+ return;
311
+ }
312
+ if (this.flushScheduled) return;
313
+ this.flushScheduled = true;
314
+ queueMicrotask(() => {
315
+ this.flushScheduled = false;
316
+ this.enqueueOperation(() => this.drainDirtyDocs()).catch(require_logging.logAsyncError("doc dirty flush"));
317
+ });
318
+ }
319
+ cancelDebounceTimer() {
320
+ if (this.debounceTimer) {
321
+ clearTimeout(this.debounceTimer);
322
+ this.debounceTimer = void 0;
323
+ }
324
+ }
325
+ enqueueOperation(operation) {
326
+ const run = this.operationPromise.catch(() => {}).then(operation);
327
+ this.operationPromise = run.then(() => {}, () => {});
328
+ return run;
329
+ }
330
+ async drainDirtyDocs() {
331
+ let accumulatedMs = 0;
332
+ let firstError = void 0;
333
+ const requeue = [];
334
+ while (this.dirtyDocs.size > 0 && firstError === void 0) {
335
+ const batch = [...this.dirtyDocs];
336
+ this.dirtyDocs.clear();
337
+ for (let i = 0; i < batch.length; i += 1) {
338
+ const docId = batch[i];
339
+ const doc = this.docs.get(docId);
340
+ if (!doc) continue;
341
+ const start = nowMs();
342
+ try {
343
+ await this.persistDocUpdate(docId, doc);
344
+ } catch (error) {
345
+ requeue.push(docId);
346
+ if (firstError === void 0) firstError = error;
347
+ }
348
+ accumulatedMs += nowMs() - start;
349
+ const moreInBatch = i < batch.length - 1;
350
+ if (accumulatedMs > YIELD_BUDGET_MS && (moreInBatch || this.dirtyDocs.size > 0)) {
351
+ await yieldToMacroTask();
352
+ accumulatedMs = 0;
353
+ }
354
+ }
355
+ }
356
+ for (const docId of requeue) this.dirtyDocs.add(docId);
357
+ if (firstError !== void 0) throw firstError;
259
358
  }
260
359
  };
261
360
 
@@ -995,11 +1094,20 @@ var SyncRunner = class {
995
1094
  this.suppressMetaLiveMonitor = true;
996
1095
  this.leaveRooms();
997
1096
  await this.metaHydrationQueue.catch(() => {});
998
- await this.docManager.close();
1097
+ let firstError;
1098
+ const safe = async (op) => {
1099
+ try {
1100
+ await op();
1101
+ } catch (error) {
1102
+ if (firstError === void 0) firstError = error;
1103
+ }
1104
+ };
1105
+ await safe(() => this.docManager.close());
999
1106
  this.eventBus.clear();
1000
1107
  this.metadataManager.clear();
1001
1108
  this.readyPromise = void 0;
1002
- await this.transport?.close();
1109
+ await safe(() => this.transport?.close() ?? Promise.resolve());
1110
+ if (firstError !== void 0) throw firstError;
1003
1111
  }
1004
1112
  async ensureMetaLiveMonitor() {
1005
1113
  if (this.unsubscribeMetaFlock) return;
@@ -1158,7 +1266,7 @@ var MetaPersister = class {
1158
1266
  this.queueMetaCompaction();
1159
1267
  return;
1160
1268
  }
1161
- if (this.compactionTimer) clearTimeout(this.compactionTimer);
1269
+ if (this.compactionTimer) return;
1162
1270
  this.compactionTimer = setTimeout(() => {
1163
1271
  this.compactionTimer = void 0;
1164
1272
  this.queueMetaCompaction();
@@ -1237,7 +1345,8 @@ var LoroRepo = class LoroRepo {
1237
1345
  this.docManager = new DocManager({
1238
1346
  storage: this.storage,
1239
1347
  ensureDocExists: (docId) => this.metadataManager.ensureExists(docId),
1240
- eventBus: this.eventBus
1348
+ eventBus: this.eventBus,
1349
+ persistDebounceMs: options.docPersistDebounceMs
1241
1350
  });
1242
1351
  if (this.storage) this.metaPersister = new MetaPersister({
1243
1352
  getMetaFlock,
@@ -1406,6 +1515,13 @@ var LoroRepo = class LoroRepo {
1406
1515
  getRawMeta() {
1407
1516
  return this.rawMetaFlock;
1408
1517
  }
1518
+ /**
1519
+ * @deprecated Holdover from an earlier `flock-sqlite`-based design that was
1520
+ * never shipped; always returns `false`. The current SQLite persistence path
1521
+ * routes everything through the regular `StorageAdapter` interface, so there
1522
+ * is no separate "SQLite mode" on the repo itself. Will be removed in the
1523
+ * next major release.
1524
+ */
1409
1525
  isSqliteMode() {
1410
1526
  return false;
1411
1527
  }
@@ -1420,6 +1536,7 @@ var LoroRepo = class LoroRepo {
1420
1536
  }
1421
1537
  async flush() {
1422
1538
  await this.docManager.flush();
1539
+ this.metaFlock.commit();
1423
1540
  await this.metaPersister?.flushNow();
1424
1541
  }
1425
1542
  get destroyed() {
@@ -1428,10 +1545,20 @@ var LoroRepo = class LoroRepo {
1428
1545
  async destroy() {
1429
1546
  if (this._destroyed) return;
1430
1547
  this._destroyed = true;
1431
- await this.metaPersister?.destroy();
1432
- await this.syncRunner.destroy();
1433
- this.storage?.close?.();
1434
- await this.transport?.close();
1548
+ this.metaFlock.disableAutoDebounceCommit();
1549
+ let firstError;
1550
+ const safe = async (op) => {
1551
+ try {
1552
+ await op();
1553
+ } catch (error) {
1554
+ if (firstError === void 0) firstError = error;
1555
+ }
1556
+ };
1557
+ await safe(() => this.metaPersister?.destroy());
1558
+ await safe(() => this.syncRunner.destroy());
1559
+ await safe(() => this.storage?.close?.());
1560
+ await safe(() => this.transport?.close());
1561
+ if (firstError !== void 0) throw firstError;
1435
1562
  }
1436
1563
  };
1437
1564