@loradb/lora-node 0.35.1 → 0.36.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (57) hide show
  1. package/README.md +5 -0
  2. package/dist/types.d.ts +2 -1
  3. package/dist/types.d.ts.map +1 -1
  4. package/dist/types.js +1 -0
  5. package/dist/types.js.map +1 -1
  6. package/docs/README.md +86 -0
  7. package/docs/concepts/graph-model.md +270 -0
  8. package/docs/concepts/nodes.md +285 -0
  9. package/docs/concepts/properties.md +348 -0
  10. package/docs/concepts/relationships.md +325 -0
  11. package/docs/concepts/result-formats.md +183 -0
  12. package/docs/concepts/schema-free.md +183 -0
  13. package/docs/cookbook.md +1081 -0
  14. package/docs/data-types/lists-and-maps.md +406 -0
  15. package/docs/data-types/overview.md +210 -0
  16. package/docs/data-types/scalars.md +376 -0
  17. package/docs/data-types/spatial.md +291 -0
  18. package/docs/data-types/temporal.md +349 -0
  19. package/docs/data-types/vectors.md +563 -0
  20. package/docs/errors.md +183 -0
  21. package/docs/functions/aggregation.md +372 -0
  22. package/docs/functions/list.md +522 -0
  23. package/docs/functions/map.md +207 -0
  24. package/docs/functions/math.md +421 -0
  25. package/docs/functions/number.md +197 -0
  26. package/docs/functions/overview.md +545 -0
  27. package/docs/functions/spatial.md +384 -0
  28. package/docs/functions/string.md +502 -0
  29. package/docs/functions/temporal.md +590 -0
  30. package/docs/functions/utility.md +137 -0
  31. package/docs/functions/vectors.md +772 -0
  32. package/docs/getting-started/installation.md +204 -0
  33. package/docs/getting-started/node.md +582 -0
  34. package/docs/getting-started/tutorial.md +334 -0
  35. package/docs/index.md +203 -0
  36. package/docs/limitations.md +291 -0
  37. package/docs/performance.md +219 -0
  38. package/docs/queries/aggregation.md +520 -0
  39. package/docs/queries/cheat-sheet.md +302 -0
  40. package/docs/queries/constraints.md +239 -0
  41. package/docs/queries/create.md +331 -0
  42. package/docs/queries/examples.md +453 -0
  43. package/docs/queries/index.md +169 -0
  44. package/docs/queries/indexes.md +538 -0
  45. package/docs/queries/match.md +429 -0
  46. package/docs/queries/ordering.md +359 -0
  47. package/docs/queries/parameters.md +328 -0
  48. package/docs/queries/paths.md +462 -0
  49. package/docs/queries/return-with.md +500 -0
  50. package/docs/queries/set-delete.md +369 -0
  51. package/docs/queries/unwind-merge.md +358 -0
  52. package/docs/queries/where.md +416 -0
  53. package/docs/snapshot.md +545 -0
  54. package/docs/troubleshooting.md +1158 -0
  55. package/docs/wal.md +570 -0
  56. package/docs/why.md +273 -0
  57. package/package.json +13 -14
package/docs/wal.md ADDED
@@ -0,0 +1,570 @@
1
+ ---
2
+ title: "WAL and Checkpoints"
3
+ description: "Continuous local durability in LoraDB via the write-ahead log. Enable WAL on Rust, Node, Python, Go, Ruby, or lora-server; recover committed writes; checkpoint to snapshots; inspect and truncate the log."
4
+ online: "https://loradb.com/docs/wal"
5
+ package: "@loradb/lora-node@0.36.1"
6
+ ---
7
+
8
+ # WAL and Checkpoints
9
+
10
+ LoraDB is still an in-memory engine: the live graph is held in RAM and
11
+ one process owns it. The write-ahead log adds a local durability layer
12
+ for surfaces that own a filesystem and a process lifecycle. Mutating
13
+ queries are recorded as WAL transactions, then a later boot can reopen
14
+ the same directory and replay only committed writes.
15
+
16
+ Snapshots still matter. A snapshot is the portable file you can copy,
17
+ archive, seed another process from, or use as a checkpoint fence. The
18
+ WAL covers the gap between snapshots; checkpoints fold the two
19
+ together.
20
+
21
+ ## Where WAL exists today
22
+
23
+ | Surface | How to enable it | What you get |
24
+ |---|---|---|
25
+ | Rust (`lora-database`) | `Database::open_with_wal(...)`, `Database::recover(...)`, `Database::open_named(...)` | Full WAL config, recovery, checkpoints, status, truncation; `snapshot_to_dir` online backups of a named database |
26
+ | Node (`@loradb/lora-node`) | `createDatabase("app", { databaseDir })` or `openWalDatabase({ walDir, snapshotDir })` | Archive-backed `.loradb` files, or explicit WAL directories with managed commit-count snapshots and sync-mode control; `db.snapshot()` online backups and `db.checkpoint()` |
27
+ | Python (`lora_python`) | `Database.create("app", {"database_dir": ...})` or `Database.open_wal(wal_dir, options)`; async mirrors both | Archive-backed `.loradb` files, or explicit WAL directories with managed commit-count snapshots |
28
+ | Go (`lora-go`) | `lora.New("app", lora.Options{DatabaseDir: ...})` or `lora.OpenWal(lora.WalOptions{...})` | Archive-backed `.loradb` files, or explicit WAL directories with managed commit-count snapshots |
29
+ | Ruby (`lora-ruby`) | `LoraRuby::Database.create("app", { database_dir: ... })` or `LoraRuby::Database.open_wal(wal_dir, options)` | Archive-backed `.loradb` files, or explicit WAL directories with managed commit-count snapshots |
30
+ | HTTP server (`lora-server`) | `--wal-dir`, `--wal-sync-mode`, `--restore-from` | Recovery, sync-mode control, `/admin/checkpoint`, `/admin/wal/status`, `/admin/wal/truncate` |
31
+ | WASM (`@loradb/lora-wasm`) | Not exposed | Snapshot-only today |
32
+
33
+ Raw WAL helpers default to `SyncMode::GroupSync` and an 8 MiB segment
34
+ target. Archive-backed named databases default to GroupSync with a 1s
35
+ interval. Rust can override the low-level WAL config directly;
36
+ Node exposes `syncMode` on both container-backed and explicit WAL opens;
37
+ `lora-server` exposes it through `--wal-sync-mode`.
38
+
39
+ ## Pick a persistence shape
40
+
41
+ | Shape | How to start | Recovery behavior |
42
+ |---|---|---|
43
+ | In-memory only | `createDatabase()`, `Database::in_memory()`, or `lora-server` with no durability flags | Restart starts from an empty graph |
44
+ | Snapshot only | Save with `save_snapshot` / `POST /admin/snapshot/save`, restore with `load_snapshot` / `--restore-from` | Restart returns to the last snapshot only |
45
+ | WAL only | Open the same WAL directory or `.loradb` archive again | Replays committed writes from the WAL into an empty graph |
46
+ | Snapshot + WAL | Restore a snapshot and open the same WAL directory or `.loradb` archive | Loads the snapshot, reads its `walLsn` fence, then replays committed WAL records newer than that fence |
47
+
48
+ Use WAL-only when the log is small enough to replay from scratch. Add
49
+ checkpoints when replay time or log size starts to matter.
50
+
51
+ ## Quick start
52
+
53
+ ### Node.js
54
+
55
+ ```ts
56
+ import { createDatabase, openWalDatabase } from '@loradb/lora-node';
57
+
58
+ const scratch = await createDatabase(); // in-memory
59
+ const db = await createDatabase('app', { databaseDir: './data' }); // ./data/app.loradb
60
+
61
+ const walDb = await openWalDatabase({
62
+ walDir: './data/app.wal',
63
+ snapshotDir: './data/app.snapshots',
64
+ snapshotEveryCommits: 1000,
65
+ snapshotOptions: { compression: { format: 'gzip', level: 1 } },
66
+ syncMode: 'groupSync',
67
+ });
68
+ ```
69
+
70
+ The name is validated and resolved under `databaseDir` as a `.loradb`
71
+ archive. Relative paths resolve from the current working directory.
72
+ Reopening the same name and directory replays committed WAL records before
73
+ the handle is returned. The open runs on the libuv threadpool, so the event
74
+ loop keeps serving timers and requests while a large database loads.
75
+
76
+ Use `openWalDatabase` when you want a raw WAL directory instead of a
77
+ portable `.loradb` archive. Pair it with `snapshotDir` to let the
78
+ database write managed checkpoints every `snapshotEveryCommits`
79
+ committed transactions, and in any case each time the WAL has grown by
80
+ 64 MiB or by a quarter of the snapshot's size, whichever is larger. Without `snapshotDir` nothing is checkpointed automatically. Node does not expose WAL status or truncate
81
+ helpers yet; use Rust or `lora-server` for those operator controls.
82
+
83
+ ### Python
84
+
85
+ ```python
86
+ from lora_python import Database, AsyncDatabase
87
+
88
+ scratch = Database.create() # in-memory
89
+ db = Database.create("app", {"database_dir": "./data"}) # container-backed
90
+ also_db = Database("app", {"database_dir": "./data"}) # equivalent
91
+
92
+ wal_db = Database.open_wal(
93
+ "./data/app.wal",
94
+ {
95
+ "snapshot_dir": "./data/app.snapshots",
96
+ "snapshot_every_commits": 1000,
97
+ "snapshot_options": {"compression": {"format": "gzip", "level": 1}},
98
+ },
99
+ )
100
+
101
+ async_db = await AsyncDatabase.create("app", {"database_dir": "./data"})
102
+ async_wal = await AsyncDatabase.open_wal(
103
+ "./data/async.wal",
104
+ {"snapshot_dir": "./data/async.snapshots", "snapshot_every_commits": 1000},
105
+ )
106
+ ```
107
+
108
+ ### Go
109
+
110
+ ```go
111
+ scratch, err := lora.New() // in-memory
112
+ db, err := lora.New("app", lora.Options{DatabaseDir: "./data"}) // container-backed
113
+ walDb, err := lora.OpenWal(lora.WalOptions{
114
+ WalDir: "./data/app.wal",
115
+ SnapshotDir: "./data/app.snapshots",
116
+ SnapshotEveryCommits: 1000,
117
+ })
118
+ ```
119
+
120
+ ### Ruby
121
+
122
+ ```ruby
123
+ scratch = LoraRuby::Database.create # in-memory
124
+ db = LoraRuby::Database.create("app", {"database_dir": "./data"}) # container-backed
125
+ wal_db = LoraRuby::Database.open_wal(
126
+ "./data/app.wal",
127
+ snapshot_dir: "./data/app.snapshots",
128
+ snapshot_every_commits: 1000,
129
+ )
130
+ ```
131
+
132
+ ### Rust
133
+
134
+ ```rust
135
+ use lora_database::{Database, WalConfig};
136
+
137
+ let db = Database::open_with_wal(WalConfig::enabled("./app"))?;
138
+ db.execute("CREATE (:Person {name: 'Ada'})", None)?;
139
+
140
+ // Later: restore a snapshot, then replay WAL above its fence.
141
+ let recovered = Database::recover("graph.bin", WalConfig::enabled("./app"))?;
142
+ ```
143
+
144
+ `WalConfig::enabled(path)` uses the current defaults:
145
+
146
+ | Setting | Value |
147
+ |---|---|
148
+ | Sync mode | `SyncMode::GroupSync { interval_ms: 50 }` |
149
+ | Segment target | 8 MiB |
150
+
151
+ Use the explicit enum variant when you need different knobs:
152
+
153
+ ```rust
154
+ use lora_database::{Database, SyncMode, WalConfig};
155
+
156
+ let db = Database::open_with_wal(WalConfig::Enabled {
157
+ dir: "./app".into(),
158
+ sync_mode: SyncMode::GroupSync { interval_ms: 50 },
159
+ segment_target_bytes: 16 * 1024 * 1024,
160
+ })?;
161
+ ```
162
+
163
+ ### HTTP server
164
+
165
+ ```bash
166
+ # Fresh boot with a WAL.
167
+ lora-server --wal-dir /var/lib/lora/wal
168
+
169
+ # Snapshot + WAL recovery, with checkpoint default path.
170
+ lora-server \
171
+ --wal-dir /var/lib/lora/wal \
172
+ --snapshot-path /var/lib/lora/graph.bin \
173
+ --restore-from /var/lib/lora/graph.bin
174
+ ```
175
+
176
+ When `--restore-from` and `--wal-dir` are both set, the server loads
177
+ the snapshot first and then replays committed WAL records newer than
178
+ the snapshot's `walLsn`. If the snapshot file is missing, recovery
179
+ falls back to WAL-only and starts from an empty graph before replay.
180
+
181
+ `--snapshot-path` is not required for WAL recovery. It only enables the
182
+ snapshot save/load admin routes and gives `/admin/checkpoint` a default
183
+ target path.
184
+
185
+ ## What gets logged
186
+
187
+ The database arms the WAL once a query has parsed and compiled. The WAL
188
+ does not allocate a transaction until the first primitive mutation
189
+ fires.
190
+
191
+ | Query outcome | WAL behavior |
192
+ |---|---|
193
+ | Read-only query | Writes no records and does not fsync |
194
+ | Successful mutating query | Usually writes one committed `MutationBatch`; transaction-style scopes may write `TxBegin`, `Mutation` records, then `TxCommit` |
195
+ | Failed mutating query | Does not publish a committed batch; transaction-style scopes that opened a WAL transaction may write `TxAbort` |
196
+
197
+ Replay is query-atomic even when the log records individual primitive
198
+ mutations. A crashed process can leave an uncommitted tail; replay drops it. A
199
+ torn record at the end of the active segment is truncated back to the last
200
+ valid record before new appends resume, and the cut bytes are kept beside the
201
+ segment as `<segment>.wal.torn-<offset>`.
202
+
203
+ Damage anywhere else fails the open with `LORA_WAL_CORRUPTION` instead of
204
+ being cut off: a bad record followed by valid records, a bad record in an
205
+ older or sealed segment, or a record whose checksum is valid but whose
206
+ contents do not decode. So does a WAL that does not reach back to the replay
207
+ fence, for example a log whose early segments a checkpoint removed, opened
208
+ without that checkpoint's snapshot, or a log missing a segment in the middle.
209
+
210
+ Opening a corrupt WAL needs an explicit decision from you; see
211
+ [Corrupt WAL: incident procedure](#corrupt-wal-incident-procedure).
212
+
213
+ A transaction too large for one WAL record is split across several
214
+ `MutationBatch` records inside one `TxBegin`/`TxCommit` pair. Property values
215
+ nested more than 128 lists or maps deep are rejected when they are written.
216
+
217
+ ## Recovery and checkpoints
218
+
219
+ Recovery follows the same steps in Rust and `lora-server`:
220
+
221
+ 1. Load the snapshot if one was supplied. Its `walLsn` becomes the
222
+ replay fence. A snapshot saved without a WAL has `walLsn: null`,
223
+ which is treated as fence `0`.
224
+ 2. Open the WAL directory or `.loradb` archive and replay every committed transaction above
225
+ the fence.
226
+ 3. Install the live WAL recorder, then accept new queries.
227
+
228
+ Any snapshot saved from a WAL-backed database carries a fence, whether
229
+ it came from `saveSnapshot`, `POST /admin/snapshot/save` or a checkpoint.
230
+ The save fsyncs the WAL and reads its durable LSN while holding the
231
+ writer lock, so the snapshot and the fence describe the same graph and
232
+ recovery replays exactly the writes made after the save. Writers pause
233
+ for that fsync, not for the snapshot encode. A plain save does not
234
+ write a `Checkpoint` marker or drop WAL segments.
235
+
236
+ A snapshot with `walLsn: null` (saved without a WAL) is a seed: start
237
+ it over an empty WAL directory with
238
+ `lora-server --restore-from seed.bin --wal-dir <dir>`, and keep passing
239
+ the same seed on later boots. Over a WAL whose early segments a
240
+ checkpoint removed, recovery fails with `LORA_WAL_CORRUPTION` instead of
241
+ replaying a partial log onto it. If the WAL is poisoned, a save writes
242
+ `walLsn: null` too and logs a warning; use that file as a seed for a
243
+ new WAL directory.
244
+
245
+ Checkpointing creates a new fence:
246
+
247
+ 1. Drain the WAL according to the configured sync mode.
248
+ 2. Read the WAL's current `durableLsn`.
249
+ 3. Save a snapshot stamped with that LSN in its `walLsn` header field.
250
+ 4. Rename the snapshot into place.
251
+ 5. Append a WAL `Checkpoint` marker.
252
+ 6. Best-effort truncate sealed WAL segments that are safe to drop.
253
+
254
+ The checkpoint marker is written after the snapshot rename succeeds, so
255
+ a marker implies the snapshot existed at the time of the checkpoint.
256
+ If recovery sees a newer checkpoint marker than the snapshot you
257
+ supplied, it prints a warning and still replays from the snapshot's own
258
+ fence. That is safe, but it may do more replay work than necessary.
259
+
260
+ ## Online backups of a named database
261
+
262
+ A named database (`createDatabase("app", { databaseDir })`,
263
+ `Database::open_named`) belongs to the process that opened it. Copying
264
+ its directory while that process writes does not give a database: the
265
+ WAL can be cut inside a record and the `.loradb` container lags the
266
+ WAL. A second process cannot open the directory to copy it either,
267
+ because the directory lock refuses it with `LORA_LOCKED`.
268
+
269
+ Take the copy from the owning process instead. `db.snapshot(targetDir)`
270
+ writes a self-contained `<targetDir>/<name>.loradb` while reads and
271
+ writes keep running:
272
+
273
+ ```ts
274
+ import { createDatabase } from '@loradb/lora-node';
275
+
276
+ const db = await createDatabase('festimap', { databaseDir: './data' });
277
+ await db.execute('UNWIND range(1, 100000) AS i CREATE (:N {i: i})');
278
+
279
+ const snapshot = await db.snapshot('./backups/2026-10-26');
280
+ console.log(snapshot);
281
+ // {
282
+ // bytes: 69283,
283
+ // lsn: 3,
284
+ // nodeCount: 100000,
285
+ // path: './backups/2026-10-26/festimap.loradb',
286
+ // relationshipCount: 0
287
+ // }
288
+
289
+ // The copy opens like any database directory.
290
+ const copy = await createDatabase('festimap', { databaseDir: './backups/2026-10-26' });
291
+ await copy.execute('MATCH (n:N) RETURN count(n) AS nodes'); // [{ nodes: 100000 }]
292
+ ```
293
+
294
+ What the copy holds and what it costs:
295
+
296
+ - **One committed transaction.** The snapshot holds every transaction
297
+ committed at or below `lsn` and none above it, never part of one. The
298
+ change feed batch with that `lsn` is the last write in it.
299
+ - **Writers do not wait for the copy.** The call takes the writer lock
300
+ only to fsync the WAL and take the graph; the graph's storage is
301
+ shared copy-on-write, so later writes copy what they change and the
302
+ snapshot keeps reading the version it took. It runs off the JS thread
303
+ and streams, so memory stays bounded by its buffers plus whatever the
304
+ concurrent writes change.
305
+ - **Complete or absent.** The container is written under a temporary
306
+ name, fsynced, and renamed into place. If the process dies mid-copy
307
+ (even `kill -9`), the target holds no `.loradb`, only a temporary
308
+ `<name>.loradb.<pid>.<time>.<n>.tmp` that the next open of that
309
+ directory removes. A copy whose promise resolved is durable.
310
+ - **The target must be missing or empty** (`LORA_VALIDATION`
311
+ otherwise) and may be on another filesystem: only the rename stays
312
+ inside it. Ship the directory off-host with any tool once the call
313
+ resolves.
314
+ - **Named databases only.** An in-memory database, or one opened with
315
+ `openWalDatabase`, rejects with `LORA_CONFIG`: save those with
316
+ `saveSnapshot`.
317
+
318
+ `db.checkpoint()` checkpoints now and reports the snapshot it wrote. On
319
+ a named database it does what `sync()` does (the container becomes a
320
+ snapshot of the graph plus the WAL since); with `walDir` and
321
+ `snapshotDir` it writes a managed snapshot:
322
+
323
+ ```ts
324
+ await db.checkpoint();
325
+ // { formatVersion: 2, nodeCount: 100000, relationshipCount: 0, walLsn: 3 }
326
+ ```
327
+
328
+ In Rust the same calls are `Database::snapshot_to_dir`, which returns a
329
+ `DirectorySnapshot { path, bytes, lsn, node_count, relationship_count }`,
330
+ and `Database::checkpoint`, which returns a `SnapshotMeta`:
331
+
332
+ ```rust
333
+ use lora_database::{Database, DatabaseOpenOptions};
334
+
335
+ let db = Database::open_named("festimap", DatabaseOpenOptions::default().with_database_dir("./data"))?;
336
+ let snapshot = db.snapshot_to_dir("./backups/2026-10-26")?;
337
+ println!("{} bytes up to LSN {}", snapshot.bytes, snapshot.lsn);
338
+ ```
339
+
340
+ A GraphQL server built on `@loradb/lora-graphql` exposes the same copy
341
+ as `lora.snapshot(targetDir)`; see [Serving](https://loradb.com/docs/graphql/serving#backups).
342
+
343
+ ### A closed directory: `lora snapshot`
344
+
345
+ For a database no process has open, `@loradb/lora-node` ships a `lora`
346
+ command:
347
+
348
+ ```bash
349
+ npx lora snapshot ./data ./backups/offline
350
+ # ./backups/offline/festimap.loradb: 69283 bytes, 100000 nodes, 0 relationships, at LSN 3
351
+ ```
352
+
353
+ It opens the database, which takes the directory lock, then writes the
354
+ same snapshot. A directory another process owns is refused rather than
355
+ copied mid-write:
356
+
357
+ ```text
358
+ lora: database `festimap` in `./data` is open in another process; snapshot it from that process with `await db.snapshot(targetDir)`
359
+ ```
360
+
361
+ When the directory holds several databases, pick one with
362
+ `--name <name>`. Opening also recovers a directory a crash left behind
363
+ before the copy is taken.
364
+
365
+ ## Corrupt WAL: incident procedure
366
+
367
+ When the WAL is damaged in the middle (a flipped bit on disk, a segment
368
+ overwritten by another tool), the database refuses to open with
369
+ `LORA_WAL_CORRUPTION`. The error names the segment file, the byte
370
+ offset, and what was wrong there, for example:
371
+
372
+ ```text
373
+ WAL segment `/var/lib/lora/wal/0000000003.wal` is corrupt at byte 52113: CRC mismatch ...; a valid record follows at byte 52190, so this is not a torn tail
374
+ ```
375
+
376
+ Nothing is changed on disk. Work through these steps:
377
+
378
+ 1. **Stop and copy.** Keep the server stopped and copy the whole WAL
379
+ directory (and your snapshots) somewhere safe before trying anything.
380
+ 2. **Prefer a better copy.** If you have a backup, a replica, or a
381
+ recent snapshot plus an intact WAL elsewhere, restore that instead.
382
+ 3. **Decide whether to cut.** If the WAL in place is the only copy,
383
+ you can start from everything before the damage. Writes recorded
384
+ after the corrupt point are not recovered. If the damage is in an
385
+ older segment, that includes every later segment.
386
+ 4. **Start once with the cut enabled**, then restart without it:
387
+
388
+ ```bash
389
+ lora-server --wal-dir /var/lib/lora/wal --wal-truncate-corrupt-tail
390
+ # or LORA_SERVER_WAL_TRUNCATE_CORRUPT_TAIL=true
391
+ ```
392
+
393
+ In Rust, set `truncate_corrupt_tail: true` on `WalConfig::Enabled`
394
+ for `Database::open_with_wal`, `open_with_wal_snapshots` or
395
+ `recover`.
396
+
397
+ With the option set, the open cuts the log at the first corrupt record
398
+ and starts with the transactions committed before it. Nothing is
399
+ deleted:
400
+
401
+ | What | Where it goes |
402
+ |---|---|
403
+ | Bytes of the corrupt segment from the bad record to its end | `<segment>.wal.corrupt-<offset>` beside it |
404
+ | Every later segment | Renamed to `<segment>.wal.after-corrupt-<corrupt segment>-<offset>` |
405
+ | The corrupt segment | Truncated at the bad record; if it was sealed, a new segment starts after it |
406
+
407
+ `lora-server` prints each of these on stderr at startup, and the
408
+ embedded library logs them as a `tracing` warning. In Rust,
409
+ `Wal::corrupt_cut()` (through `db.wal()`) reports them too. The open
410
+ only acts on `LORA_WAL_CORRUPTION` from a damaged record; a missing
411
+ segment (a gap) or an unreadable segment header still fails.
412
+
413
+ Once the server is up, take a snapshot or checkpoint, and do not keep
414
+ the option on: a later open should fail loudly if the disk damages the
415
+ log again. The option is not wired into named databases
416
+ (`openDatabase(name)` and the `.loradb` container) or the language
417
+ bindings' WAL options.
418
+
419
+ ## Sync modes
420
+
421
+ `lora-server --wal-sync-mode` and the Rust API expose GroupSync
422
+ durability:
423
+
424
+ | Mode | `fsync` cadence | Crash window | When to use |
425
+ |---|---|---|---|
426
+ | `group-sync` | Background fsync, explicit sync, checkpoint, or clean drop | Up to the group interval | Higher write rates with explicit sync points when needed |
427
+
428
+ In `group-sync` mode, append and commit records are written before the query
429
+ returns, but the `fsync` happens on a background cadence. If that
430
+ background fsync fails, the failure is latched: future WAL operations
431
+ return a poisoned error and `/admin/wal/status` reports `bgFailure`.
432
+ Restart from the last consistent snapshot + WAL after fixing the
433
+ underlying disk issue.
434
+
435
+ `group-sync` is the only mode; `lora-server` runs its background fsync
436
+ every 50 ms, so a power loss or kernel crash can lose up to that much
437
+ acknowledged work. A process crash alone loses nothing that was already
438
+ written to the OS.
439
+
440
+ A change feed on a WAL-backed database only delivers a commit once it is
441
+ fsynced, because a commit lost in a crash would have its LSN reused by the
442
+ next write after the restart. The feed asks for the fsync straight away, so
443
+ an occasional write reaches the feed after one fsync; under a steady stream
444
+ of writes batches trail the commit by up to the group interval.
445
+
446
+ For checkpointed deployments, call `sync()` when you need an immediate
447
+ durability point. On a named `.loradb` database `sync()` is also the
448
+ checkpoint: it writes the graph as the archive's base snapshot, streamed so it
449
+ holds no second copy of the graph, and drops the WAL segments that snapshot
450
+ covers. A change feed can then resume only from the last `sync()` on. An open
451
+ that replays more WAL than a quarter of its snapshot, and at least 64 MiB of it,
452
+ takes the same checkpoint before it returns, so the next open loads the
453
+ snapshot instead of replaying the log again. A named database also takes
454
+ that checkpoint on its own, on a background thread, each time its WAL has grown
455
+ by 64 MiB since the last one, or by a quarter of its snapshot's size when
456
+ that is larger (`DatabaseOpenOptions::checkpoint_wal_bytes`, 0 turns it off), so a
457
+ process that never calls `sync()` does not keep every segment and a bulk load
458
+ into a large graph does not rewrite the snapshot over and over. A checkpoint snapshot is stamped with the WAL's current
459
+ `durableLsn`; in `group-sync` mode that fence
460
+ can trail the newest writes until the background fsync catches up.
461
+
462
+ ## What's in the WAL directory
463
+
464
+ ```text
465
+ <wal-dir>/
466
+ .lora-wal.lock held while a handle has the directory open
467
+ 0000000001.wal sealed, oldest
468
+ 0000000002.wal sealed
469
+ 0000000003.wal active (highest id)
470
+ ```
471
+
472
+ Segments rotate at the segment target (8 MiB by default), always at a
473
+ transaction boundary. There is no `CURRENT` pointer file: the active
474
+ segment is the one with the highest id. Files that don't match the
475
+ segment name pattern (such as a leftover `.tmp`) are ignored at boot.
476
+ Treat the directory as one unit: back it up, move it, or delete it
477
+ whole, never segment by segment.
478
+
479
+ ## HTTP admin routes
480
+
481
+ When `lora-server` starts with `--wal-dir`, these routes are mounted:
482
+
483
+ | Method | Path | Purpose |
484
+ |---|---|---|
485
+ | `POST` | `/admin/wal/status` | Inspect durable LSN, next LSN, segment ids, and any latched background fsync failure |
486
+ | `POST` | `/admin/wal/truncate` | Drop sealed WAL segments up to a fence LSN |
487
+ | `POST` | `/admin/checkpoint` | Write a checkpoint snapshot and truncate safe WAL history |
488
+
489
+ Examples:
490
+
491
+ ```bash
492
+ curl -sX POST http://127.0.0.1:4747/admin/wal/status
493
+
494
+ curl -sX POST http://127.0.0.1:4747/admin/wal/truncate \
495
+ -H 'content-type: application/json' \
496
+ -d '{"fenceLsn": 4815}'
497
+
498
+ curl -sX POST http://127.0.0.1:4747/admin/checkpoint \
499
+ -H 'content-type: application/json' \
500
+ -d '{"path": "/var/lib/lora/checkpoint.bin"}'
501
+ ```
502
+
503
+ `/admin/wal/status` returns:
504
+
505
+ ```json
506
+ {
507
+ "durableLsn": 4815,
508
+ "nextLsn": 4820,
509
+ "activeSegmentId": 3,
510
+ "oldestSegmentId": 2,
511
+ "bgFailure": null
512
+ }
513
+ ```
514
+
515
+ `/admin/checkpoint` can omit the body only when `--snapshot-path` is
516
+ configured. Without a configured default, the request body must include
517
+ `path` or the route returns `400 Bad Request`. A checkpoint blocks writes
518
+ while it runs; reads continue.
519
+
520
+ `/admin/wal/truncate` can omit the body; in that case it truncates up
521
+ to the WAL's current `durableLsn`. It drops only sealed segments whose
522
+ whole LSN range is at or below the fence. The active segment and the
523
+ one just before it are always kept.
524
+
525
+ ## Boundaries
526
+
527
+ - **Checkpoint automation is by WAL size and commit count, not by time.**
528
+ Named databases and managed snapshots checkpoint on a background thread
529
+ each time the WAL has grown by 64 MiB, or by the size of the current
530
+ snapshot when that is larger (`checkpoint_wal_bytes` in Rust; the
531
+ bindings use the default). Explicit WAL helpers in Node, Python, Go, and
532
+ Ruby can also write managed snapshots after N committed transactions. A
533
+ plain WAL directory with no snapshot directory, which is what `lora-server
534
+ --wal-dir` opens, has nowhere to put a snapshot: it keeps every segment
535
+ until `POST /admin/checkpoint` or a checkpoint call, and logs one warning
536
+ once it passes 64 MiB. There is no wall-clock scheduler built into the
537
+ engine.
538
+ - **No auth on the admin surface.** Snapshot and WAL admin routes are
539
+ off by default but unauthenticated when enabled. Put them behind
540
+ authenticated ingress only.
541
+ - **No shared WAL/container root.** One live handle owns one WAL directory or
542
+ `.loradb` archive, enforced by an advisory lock (`.lora-wal.lock` inside a
543
+ WAL directory; archives take their own lock).
544
+ Opening the same root from another process, or from a second live handle in
545
+ the same process, fails with `LORA_LOCKED` until the first handle is closed.
546
+ See [Troubleshooting](./troubleshooting.md#walcontainer-root-is-already-open).
547
+ - **Binding support is asymmetric.** The filesystem-backed bindings can
548
+ open WAL-backed databases. Rust and `lora-server` expose full
549
+ checkpoint, truncate, status, and sync-mode controls. Node also
550
+ exposes sync-mode control. Python, Go, and Ruby expose simple raw WAL
551
+ opens plus managed snapshot options. WASM stays snapshot-only.
552
+ - **No cross-version WAL compatibility guarantee yet.** Snapshots are
553
+ the portable backup and migration artifact. Treat WAL directories as
554
+ local runtime state for the same deployment line unless a release
555
+ explicitly says otherwise.
556
+
557
+ ## See also
558
+
559
+ - [**Snapshots**](./snapshot.md) - save / load the graph as a single file.
560
+ - [**HTTP server guide**](https://loradb.com/docs/getting-started/server) - run
561
+ `lora-server`, configure flags, and probe it with `curl`.
562
+ - [**HTTP API**](https://loradb.com/docs/api/http) - route-by-route request / response
563
+ reference.
564
+ - [**Troubleshooting**](./troubleshooting.md#wal-and-checkpoints) - common
565
+ WAL setup and recovery errors.
566
+ - [**Node guide**](./getting-started/node.md#persisting-your-graph) -
567
+ `createDatabase("app", { databaseDir: "./data" })` and
568
+ `openWalDatabase({ walDir })` in context.
569
+ - [**Rust guide**](https://loradb.com/docs/getting-started/rust#persisting-your-graph) -
570
+ embedding the engine directly.