loro-repo 0.16.10 → 0.17.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -56,6 +56,133 @@ await repo.unloadDoc("note:welcome");
56
56
  - `WebSocketTransportAdapter` (`src/transport/websocket.ts`)
57
57
  - `IndexedDBStorageAdaptor` (`src/storage/indexeddb.ts`)
58
58
  - `FileSystemStorageAdaptor` (`src/storage/filesystem.ts`)
59
+ - `SqliteRepoStore` (`src/storage/sqlite.ts`, Node.js only)
60
+
61
+ ## SQLite storage (Node.js / Electron)
62
+
63
+ `SqliteRepoStore` packs everything a repo needs into a single SQLite database
64
+ file and is the recommended Node-side storage when the streams transport is in
65
+ use. Compared to `FileSystemStorageAdaptor`, it avoids creating many small
66
+ files for incremental updates, and it lets remote-cursor advances become
67
+ single-row `UPDATE`s instead of rewriting a large JSON blob.
68
+
69
+ A single `SqliteRepoStore` instance exposes two adapters that share one
70
+ database connection:
71
+
72
+ - `store.storage` — a `StorageAdapter` for doc snapshots, doc updates, and
73
+ metadata (Flock) snapshots/updates.
74
+ - `store.cursorStore` — a `RemoteCursorStore` for the streams transport's
75
+ `remoteCursorStore` option.
76
+
77
+ ```ts
78
+ import { LoroRepo } from "loro-repo";
79
+ import { SqliteRepoStore } from "loro-repo/storage/sqlite";
80
+ import { StreamsTransportAdapter } from "loro-repo/transport/streams";
81
+
82
+ const store = new SqliteRepoStore({ path: "./data/repo.db" });
83
+
84
+ const repo = await LoroRepo.create({
85
+ storageAdapter: store.storage,
86
+ transportAdapter: new StreamsTransportAdapter({
87
+ bucketId: "your-bucket",
88
+ auth: yourAuthProvider,
89
+ remoteCursorStore: store.cursorStore,
90
+ // Required when `remoteCursorStore` persists across restarts. Both hooks
91
+ // run inside `beforeRemoteCursorSave`, so the local doc / Flock state is
92
+ // durable *before* the cursor advances. Skipping them risks a crash that
93
+ // leaves the cursor pointing past data that was never persisted locally.
94
+ onPersistDoc: async (docId, doc) => {
95
+ await store.storage.save({
96
+ type: "doc-snapshot",
97
+ docId,
98
+ snapshot: doc.export({ mode: "snapshot" }),
99
+ });
100
+ },
101
+ onPersistMeta: async (flock) => {
102
+ await store.storage.save({
103
+ type: "meta-snapshot",
104
+ snapshot: flock.exportFile(),
105
+ });
106
+ },
107
+ }),
108
+ });
109
+
110
+ // Shutdown
111
+ await repo.destroy();
112
+ store.close();
113
+ ```
114
+
115
+ > ⚠️ `onPersistDoc` and `onPersistMeta` are not optional when you wire a
116
+ > durable `remoteCursorStore`. The streams transport advances cursors only
117
+ > after `beforeRemoteCursorSave` resolves; if local persistence does not run
118
+ > there, a crash can leave SQLite with stale doc/meta state while the cursor
119
+ > resumes after data that was never applied locally. The `InMemoryRemoteCursorStore`
120
+ > default is the only configuration where omitting the hooks is safe, since
121
+ > the cursor itself is lost on restart.
122
+
123
+ `SqliteRepoStoreOptions`:
124
+
125
+ | Option | Default | Description |
126
+ | --- | --- | --- |
127
+ | `path` | `":memory:"` | SQLite file path. Use `":memory:"` for an ephemeral store. |
128
+ | `database` | — | Pass a pre-constructed `better-sqlite3` `Database` to share a connection; the caller owns its lifetime. |
129
+ | `tablePrefix` | `""` | Optional prefix for all table names, useful when multiple repos share one database. |
130
+ | `wal` | `true` | Enables `journal_mode=WAL` + `synchronous=NORMAL` for fast, durable writes. |
131
+
132
+ The schema (created on first use) is:
133
+
134
+ ```
135
+ docs (doc_id PK, snapshot, updated_at, snapshot_digest)
136
+ doc_updates (id, doc_id, update_data, created_at)
137
+ meta_snapshot (id=1, snapshot, updated_at)
138
+ meta_updates (id, update_data, created_at)
139
+ remote_cursors (stream_url PK, next_offset, server_lower_bound_version, updated_at_ms)
140
+ ```
141
+
142
+ `loadDoc` / `loadMeta` opportunistically compact replayed updates into a fresh
143
+ snapshot inside a single transaction, mirroring the filesystem adapter's
144
+ behaviour but without any file churn.
145
+
146
+ ### Backups in WAL mode
147
+
148
+ With the default `wal: true`, SQLite writes to two sidecar files alongside
149
+ the main DB: `<path>-wal` and `<path>-shm`. Recent writes may live only in
150
+ the WAL until a checkpoint runs. If you copy the database file out for
151
+ backup, also copy both sidecars **or** force a checkpoint first — pass a
152
+ pre-constructed `Database` via the `database` option so your code keeps
153
+ access to the connection and can call `pragma("wal_checkpoint(TRUNCATE)")`
154
+ before snapshotting. Otherwise the `.db`-only copy can miss recent writes.
155
+
156
+ ### Installation
157
+
158
+ `better-sqlite3` is declared as an **optional** peer dependency. Install it
159
+ explicitly in projects that want SQLite storage, along with `@types/better-sqlite3`
160
+ for TypeScript users (the runtime package ships no `.d.ts` of its own and the
161
+ published `loro-repo/storage/sqlite` types reference it):
162
+
163
+ ```bash
164
+ pnpm add better-sqlite3
165
+ pnpm add -D @types/better-sqlite3
166
+ # or: npm install better-sqlite3 && npm install -D @types/better-sqlite3
167
+ ```
168
+
169
+ #### Node version compatibility
170
+
171
+ `loro-repo` itself supports Node ≥ 18. `better-sqlite3` versions matter:
172
+
173
+ | Node version | Use `better-sqlite3` |
174
+ | --- | --- |
175
+ | 18.x | `^11` (v11 is the last line that supports Node 18) |
176
+ | ≥ 20 | `^11` or `^12` |
177
+
178
+ If you're on Node 18, install `better-sqlite3@^11` explicitly — picking up v12
179
+ will fail at runtime with an unsupported-Node error. The `loro-repo`
180
+ implementation only uses APIs present in both lines, so either works.
181
+
182
+ Browsers should keep using `IndexedDBStorageAdaptor` (see the previous
183
+ section). The `loro-repo/storage/sqlite` entry is the only place that imports
184
+ `better-sqlite3`, so importing the main package or any other adapter never
185
+ pulls native bindings into a browser bundle.
59
186
 
60
187
  ## Core API surface
61
188
 
package/dist/index.cjs CHANGED
@@ -75,18 +75,39 @@ var RepoEventBus = class {
75
75
 
76
76
  //#endregion
77
77
  //#region src/internal/doc-manager.ts
78
+ /**
79
+ * Yield control back to the event loop's macro task queue, letting any pending
80
+ * timers, I/O callbacks and other tasks run before continuing. A microtask
81
+ * (`Promise.resolve()`/`queueMicrotask`) is not enough here: microtasks drain
82
+ * before the event loop processes macro tasks, so they would not unblock the
83
+ * loop between CPU-heavy doc exports.
84
+ */
85
+ function yieldToMacroTask() {
86
+ return new Promise((resolve) => setTimeout(resolve, 0));
87
+ }
88
+ const YIELD_BUDGET_MS = 5;
89
+ const nowMs = typeof globalThis.performance?.now === "function" ? globalThis.performance.now.bind(globalThis.performance) : Date.now;
78
90
  var DocManager = class {
79
91
  storage;
80
92
  ensureDocExists;
81
93
  eventBus;
94
+ persistDebounceMs;
82
95
  docs = /* @__PURE__ */ new Map();
83
96
  docSubscriptions = /* @__PURE__ */ new Map();
84
97
  docPersistedVersions = /* @__PURE__ */ new Map();
85
98
  pendingDocLoads = /* @__PURE__ */ new Map();
99
+ dirtyDocs = /* @__PURE__ */ new Set();
100
+ existenceMaterialized = /* @__PURE__ */ new Set();
101
+ debounceTimer;
102
+ flushScheduled = false;
103
+ operationPromise = Promise.resolve();
104
+ destroyed = false;
86
105
  constructor(options) {
87
106
  this.storage = options.storage;
88
107
  this.ensureDocExists = options.ensureDocExists;
89
108
  this.eventBus = options.eventBus;
109
+ const configured = options.persistDebounceMs;
110
+ this.persistDebounceMs = typeof configured === "number" && Number.isFinite(configured) && configured >= 0 ? configured : 0;
90
111
  }
91
112
  async openPersistedDoc(docId) {
92
113
  return this.ensureDoc(docId);
@@ -115,9 +136,11 @@ var DocManager = class {
115
136
  const previousVersion = this.docPersistedVersions.get(docId);
116
137
  const snapshot = doc.export({ mode: "snapshot" });
117
138
  const nextVersion = doc.version();
139
+ const oplogAtCapture = doc.oplogVersion();
118
140
  if (!this.storage) {
119
141
  this.docPersistedVersions.set(docId, nextVersion);
120
142
  await this.materializeExistence(docId, doc);
143
+ this.clearDirtyIfStable(docId, oplogAtCapture, doc);
121
144
  return;
122
145
  }
123
146
  this.docPersistedVersions.set(docId, nextVersion);
@@ -128,6 +151,7 @@ var DocManager = class {
128
151
  snapshot
129
152
  });
130
153
  await this.materializeExistence(docId, doc);
154
+ this.clearDirtyIfStable(docId, oplogAtCapture, doc);
131
155
  } catch (error) {
132
156
  if (previousVersion) this.docPersistedVersions.set(docId, previousVersion);
133
157
  else this.docPersistedVersions.delete(docId);
@@ -135,8 +159,10 @@ var DocManager = class {
135
159
  }
136
160
  }
137
161
  async materializeExistence(docId, doc) {
162
+ if (this.existenceMaterialized.has(docId)) return;
138
163
  if (doc.oplogFrontiers().length === 0) return;
139
164
  await this.ensureDocExists(docId);
165
+ this.existenceMaterialized.add(docId);
140
166
  }
141
167
  async unloadDoc(docId) {
142
168
  const doc = this.docs.get(docId);
@@ -146,28 +172,42 @@ var DocManager = class {
146
172
  this.docSubscriptions.delete(docId);
147
173
  this.docs.delete(docId);
148
174
  this.docPersistedVersions.delete(docId);
175
+ this.existenceMaterialized.delete(docId);
149
176
  }
150
177
  async dropDoc(docId) {
151
178
  this.docSubscriptions.get(docId)?.();
152
179
  this.docSubscriptions.delete(docId);
153
180
  this.docs.delete(docId);
154
181
  this.docPersistedVersions.delete(docId);
182
+ this.dirtyDocs.delete(docId);
183
+ this.existenceMaterialized.delete(docId);
155
184
  if (this.storage?.deleteDoc) await this.storage.deleteDoc(docId);
156
185
  }
157
186
  async flush() {
158
- const promises = [];
159
- for (const [docId, doc] of this.docs) promises.push(this.persistDocUpdate(docId, doc));
160
- await Promise.all(promises);
187
+ this.cancelDebounceTimer();
188
+ this.flushScheduled = false;
189
+ while (true) {
190
+ await this.enqueueOperation(() => this.drainDirtyDocs());
191
+ if (this.dirtyDocs.size === 0 && !this.flushScheduled) break;
192
+ this.flushScheduled = false;
193
+ }
161
194
  }
162
195
  async close() {
163
- await this.flush();
164
- for (const unsubscribe of this.docSubscriptions.values()) try {
165
- unsubscribe();
166
- } catch {}
167
- this.docSubscriptions.clear();
168
- this.docs.clear();
169
- this.docPersistedVersions.clear();
170
- this.pendingDocLoads.clear();
196
+ this.destroyed = true;
197
+ this.cancelDebounceTimer();
198
+ try {
199
+ await this.flush();
200
+ } finally {
201
+ for (const unsubscribe of this.docSubscriptions.values()) try {
202
+ unsubscribe();
203
+ } catch {}
204
+ this.docSubscriptions.clear();
205
+ this.docs.clear();
206
+ this.docPersistedVersions.clear();
207
+ this.pendingDocLoads.clear();
208
+ this.dirtyDocs.clear();
209
+ this.existenceMaterialized.clear();
210
+ }
171
211
  }
172
212
  async loadOrCreateDoc(docId) {
173
213
  const cached = this.docs.get(docId);
@@ -218,14 +258,19 @@ var DocManager = class {
218
258
  if (!this.storage) {
219
259
  this.docPersistedVersions.set(docId, nextVersion);
220
260
  await this.materializeExistence(docId, doc);
261
+ this.clearDirtyIfStable(docId, nextVersion, doc);
221
262
  return;
222
263
  }
223
264
  if (!previousVersion) {
224
265
  await this.persistDoc(docId, doc);
225
266
  this.docPersistedVersions.set(docId, nextVersion);
267
+ this.clearDirtyIfStable(docId, nextVersion, doc);
268
+ return;
269
+ }
270
+ if (previousVersion.compare(nextVersion) === 0) {
271
+ this.dirtyDocs.delete(docId);
226
272
  return;
227
273
  }
228
- if (previousVersion.compare(nextVersion) === 0) return;
229
274
  const update = doc.export({
230
275
  mode: "update",
231
276
  from: previousVersion
@@ -238,13 +283,78 @@ var DocManager = class {
238
283
  update
239
284
  });
240
285
  await this.materializeExistence(docId, doc);
286
+ this.clearDirtyIfStable(docId, nextVersion, doc);
241
287
  } catch (error) {
242
288
  this.docPersistedVersions.set(docId, previousVersion);
243
289
  throw error;
244
290
  }
245
291
  }
292
+ clearDirtyIfStable(docId, capturedOplog, doc) {
293
+ if (doc.oplogVersion().compare(capturedOplog) === 0) this.dirtyDocs.delete(docId);
294
+ }
246
295
  onDocEvent(docId, doc, _batch, _by) {
247
- this.persistDocUpdate(docId, doc).catch(require_logging.logAsyncError(`doc ${docId} event processing`));
296
+ if (!this.docs.has(docId)) return;
297
+ this.dirtyDocs.add(docId);
298
+ this.scheduleDirtyFlush();
299
+ }
300
+ scheduleDirtyFlush() {
301
+ if (this.destroyed) return;
302
+ if (this.persistDebounceMs > 0) {
303
+ if (this.debounceTimer) clearTimeout(this.debounceTimer);
304
+ this.flushScheduled = true;
305
+ this.debounceTimer = setTimeout(() => {
306
+ this.debounceTimer = void 0;
307
+ this.flushScheduled = false;
308
+ this.enqueueOperation(() => this.drainDirtyDocs()).catch(require_logging.logAsyncError("doc dirty flush"));
309
+ }, this.persistDebounceMs);
310
+ return;
311
+ }
312
+ if (this.flushScheduled) return;
313
+ this.flushScheduled = true;
314
+ queueMicrotask(() => {
315
+ this.flushScheduled = false;
316
+ this.enqueueOperation(() => this.drainDirtyDocs()).catch(require_logging.logAsyncError("doc dirty flush"));
317
+ });
318
+ }
319
+ cancelDebounceTimer() {
320
+ if (this.debounceTimer) {
321
+ clearTimeout(this.debounceTimer);
322
+ this.debounceTimer = void 0;
323
+ }
324
+ }
325
+ enqueueOperation(operation) {
326
+ const run = this.operationPromise.catch(() => {}).then(operation);
327
+ this.operationPromise = run.then(() => {}, () => {});
328
+ return run;
329
+ }
330
+ async drainDirtyDocs() {
331
+ let accumulatedMs = 0;
332
+ let firstError = void 0;
333
+ const requeue = [];
334
+ while (this.dirtyDocs.size > 0 && firstError === void 0) {
335
+ const batch = [...this.dirtyDocs];
336
+ this.dirtyDocs.clear();
337
+ for (let i = 0; i < batch.length; i += 1) {
338
+ const docId = batch[i];
339
+ const doc = this.docs.get(docId);
340
+ if (!doc) continue;
341
+ const start = nowMs();
342
+ try {
343
+ await this.persistDocUpdate(docId, doc);
344
+ } catch (error) {
345
+ requeue.push(docId);
346
+ if (firstError === void 0) firstError = error;
347
+ }
348
+ accumulatedMs += nowMs() - start;
349
+ const moreInBatch = i < batch.length - 1;
350
+ if (accumulatedMs > YIELD_BUDGET_MS && (moreInBatch || this.dirtyDocs.size > 0)) {
351
+ await yieldToMacroTask();
352
+ accumulatedMs = 0;
353
+ }
354
+ }
355
+ }
356
+ for (const docId of requeue) this.dirtyDocs.add(docId);
357
+ if (firstError !== void 0) throw firstError;
248
358
  }
249
359
  };
250
360
 
@@ -984,11 +1094,20 @@ var SyncRunner = class {
984
1094
  this.suppressMetaLiveMonitor = true;
985
1095
  this.leaveRooms();
986
1096
  await this.metaHydrationQueue.catch(() => {});
987
- await this.docManager.close();
1097
+ let firstError;
1098
+ const safe = async (op) => {
1099
+ try {
1100
+ await op();
1101
+ } catch (error) {
1102
+ if (firstError === void 0) firstError = error;
1103
+ }
1104
+ };
1105
+ await safe(() => this.docManager.close());
988
1106
  this.eventBus.clear();
989
1107
  this.metadataManager.clear();
990
1108
  this.readyPromise = void 0;
991
- await this.transport?.close();
1109
+ await safe(() => this.transport?.close() ?? Promise.resolve());
1110
+ if (firstError !== void 0) throw firstError;
992
1111
  }
993
1112
  async ensureMetaLiveMonitor() {
994
1113
  if (this.unsubscribeMetaFlock) return;
@@ -1147,7 +1266,7 @@ var MetaPersister = class {
1147
1266
  this.queueMetaCompaction();
1148
1267
  return;
1149
1268
  }
1150
- if (this.compactionTimer) clearTimeout(this.compactionTimer);
1269
+ if (this.compactionTimer) return;
1151
1270
  this.compactionTimer = setTimeout(() => {
1152
1271
  this.compactionTimer = void 0;
1153
1272
  this.queueMetaCompaction();
@@ -1226,7 +1345,8 @@ var LoroRepo = class LoroRepo {
1226
1345
  this.docManager = new DocManager({
1227
1346
  storage: this.storage,
1228
1347
  ensureDocExists: (docId) => this.metadataManager.ensureExists(docId),
1229
- eventBus: this.eventBus
1348
+ eventBus: this.eventBus,
1349
+ persistDebounceMs: options.docPersistDebounceMs
1230
1350
  });
1231
1351
  if (this.storage) this.metaPersister = new MetaPersister({
1232
1352
  getMetaFlock,
@@ -1395,6 +1515,13 @@ var LoroRepo = class LoroRepo {
1395
1515
  getRawMeta() {
1396
1516
  return this.rawMetaFlock;
1397
1517
  }
1518
+ /**
1519
+ * @deprecated Holdover from an earlier `flock-sqlite`-based design that was
1520
+ * never shipped; always returns `false`. The current SQLite persistence path
1521
+ * routes everything through the regular `StorageAdapter` interface, so there
1522
+ * is no separate "SQLite mode" on the repo itself. Will be removed in the
1523
+ * next major release.
1524
+ */
1398
1525
  isSqliteMode() {
1399
1526
  return false;
1400
1527
  }
@@ -1409,6 +1536,7 @@ var LoroRepo = class LoroRepo {
1409
1536
  }
1410
1537
  async flush() {
1411
1538
  await this.docManager.flush();
1539
+ this.metaFlock.commit();
1412
1540
  await this.metaPersister?.flushNow();
1413
1541
  }
1414
1542
  get destroyed() {
@@ -1417,10 +1545,20 @@ var LoroRepo = class LoroRepo {
1417
1545
  async destroy() {
1418
1546
  if (this._destroyed) return;
1419
1547
  this._destroyed = true;
1420
- await this.metaPersister?.destroy();
1421
- await this.syncRunner.destroy();
1422
- this.storage?.close?.();
1423
- await this.transport?.close();
1548
+ this.metaFlock.disableAutoDebounceCommit();
1549
+ let firstError;
1550
+ const safe = async (op) => {
1551
+ try {
1552
+ await op();
1553
+ } catch (error) {
1554
+ if (firstError === void 0) firstError = error;
1555
+ }
1556
+ };
1557
+ await safe(() => this.metaPersister?.destroy());
1558
+ await safe(() => this.syncRunner.destroy());
1559
+ await safe(() => this.storage?.close?.());
1560
+ await safe(() => this.transport?.close());
1561
+ if (firstError !== void 0) throw firstError;
1424
1562
  }
1425
1563
  };
1426
1564