@loradb/lora-node 0.35.1 → 0.36.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +5 -0
- package/dist/types.d.ts +2 -1
- package/dist/types.d.ts.map +1 -1
- package/dist/types.js +1 -0
- package/dist/types.js.map +1 -1
- package/docs/README.md +86 -0
- package/docs/concepts/graph-model.md +270 -0
- package/docs/concepts/nodes.md +285 -0
- package/docs/concepts/properties.md +348 -0
- package/docs/concepts/relationships.md +325 -0
- package/docs/concepts/result-formats.md +183 -0
- package/docs/concepts/schema-free.md +183 -0
- package/docs/cookbook.md +1081 -0
- package/docs/data-types/lists-and-maps.md +406 -0
- package/docs/data-types/overview.md +210 -0
- package/docs/data-types/scalars.md +376 -0
- package/docs/data-types/spatial.md +291 -0
- package/docs/data-types/temporal.md +349 -0
- package/docs/data-types/vectors.md +563 -0
- package/docs/errors.md +183 -0
- package/docs/functions/aggregation.md +372 -0
- package/docs/functions/list.md +522 -0
- package/docs/functions/map.md +207 -0
- package/docs/functions/math.md +421 -0
- package/docs/functions/number.md +197 -0
- package/docs/functions/overview.md +545 -0
- package/docs/functions/spatial.md +384 -0
- package/docs/functions/string.md +502 -0
- package/docs/functions/temporal.md +590 -0
- package/docs/functions/utility.md +137 -0
- package/docs/functions/vectors.md +772 -0
- package/docs/getting-started/installation.md +204 -0
- package/docs/getting-started/node.md +582 -0
- package/docs/getting-started/tutorial.md +334 -0
- package/docs/index.md +203 -0
- package/docs/limitations.md +291 -0
- package/docs/performance.md +219 -0
- package/docs/queries/aggregation.md +520 -0
- package/docs/queries/cheat-sheet.md +302 -0
- package/docs/queries/constraints.md +239 -0
- package/docs/queries/create.md +331 -0
- package/docs/queries/examples.md +453 -0
- package/docs/queries/index.md +169 -0
- package/docs/queries/indexes.md +538 -0
- package/docs/queries/match.md +429 -0
- package/docs/queries/ordering.md +359 -0
- package/docs/queries/parameters.md +328 -0
- package/docs/queries/paths.md +462 -0
- package/docs/queries/return-with.md +500 -0
- package/docs/queries/set-delete.md +369 -0
- package/docs/queries/unwind-merge.md +358 -0
- package/docs/queries/where.md +416 -0
- package/docs/snapshot.md +545 -0
- package/docs/troubleshooting.md +1158 -0
- package/docs/wal.md +570 -0
- package/docs/why.md +273 -0
- package/package.json +13 -14
package/docs/wal.md
ADDED
|
@@ -0,0 +1,570 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "WAL and Checkpoints"
|
|
3
|
+
description: "Continuous local durability in LoraDB via the write-ahead log. Enable WAL on Rust, Node, Python, Go, Ruby, or lora-server; recover committed writes; checkpoint to snapshots; inspect and truncate the log."
|
|
4
|
+
online: "https://loradb.com/docs/wal"
|
|
5
|
+
package: "@loradb/lora-node@0.36.1"
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# WAL and Checkpoints
|
|
9
|
+
|
|
10
|
+
LoraDB is still an in-memory engine: the live graph is held in RAM and
|
|
11
|
+
one process owns it. The write-ahead log adds a local durability layer
|
|
12
|
+
for surfaces that own a filesystem and a process lifecycle. Mutating
|
|
13
|
+
queries are recorded as WAL transactions, then a later boot can reopen
|
|
14
|
+
the same directory and replay only committed writes.
|
|
15
|
+
|
|
16
|
+
Snapshots still matter. A snapshot is the portable file you can copy,
|
|
17
|
+
archive, seed another process from, or use as a checkpoint fence. The
|
|
18
|
+
WAL covers the gap between snapshots; checkpoints fold the two
|
|
19
|
+
together.
|
|
20
|
+
|
|
21
|
+
## Where WAL exists today
|
|
22
|
+
|
|
23
|
+
| Surface | How to enable it | What you get |
|
|
24
|
+
|---|---|---|
|
|
25
|
+
| Rust (`lora-database`) | `Database::open_with_wal(...)`, `Database::recover(...)`, `Database::open_named(...)` | Full WAL config, recovery, checkpoints, status, truncation; `snapshot_to_dir` online backups of a named database |
|
|
26
|
+
| Node (`@loradb/lora-node`) | `createDatabase("app", { databaseDir })` or `openWalDatabase({ walDir, snapshotDir })` | Archive-backed `.loradb` files, or explicit WAL directories with managed commit-count snapshots and sync-mode control; `db.snapshot()` online backups and `db.checkpoint()` |
|
|
27
|
+
| Python (`lora_python`) | `Database.create("app", {"database_dir": ...})` or `Database.open_wal(wal_dir, options)`; async mirrors both | Archive-backed `.loradb` files, or explicit WAL directories with managed commit-count snapshots |
|
|
28
|
+
| Go (`lora-go`) | `lora.New("app", lora.Options{DatabaseDir: ...})` or `lora.OpenWal(lora.WalOptions{...})` | Archive-backed `.loradb` files, or explicit WAL directories with managed commit-count snapshots |
|
|
29
|
+
| Ruby (`lora-ruby`) | `LoraRuby::Database.create("app", { database_dir: ... })` or `LoraRuby::Database.open_wal(wal_dir, options)` | Archive-backed `.loradb` files, or explicit WAL directories with managed commit-count snapshots |
|
|
30
|
+
| HTTP server (`lora-server`) | `--wal-dir`, `--wal-sync-mode`, `--restore-from` | Recovery, sync-mode control, `/admin/checkpoint`, `/admin/wal/status`, `/admin/wal/truncate` |
|
|
31
|
+
| WASM (`@loradb/lora-wasm`) | Not exposed | Snapshot-only today |
|
|
32
|
+
|
|
33
|
+
Raw WAL helpers default to `SyncMode::GroupSync` and an 8 MiB segment
|
|
34
|
+
target. Archive-backed named databases default to GroupSync with a 1s
|
|
35
|
+
interval. Rust can override the low-level WAL config directly;
|
|
36
|
+
Node exposes `syncMode` on both container-backed and explicit WAL opens;
|
|
37
|
+
`lora-server` exposes it through `--wal-sync-mode`.
|
|
38
|
+
|
|
39
|
+
## Pick a persistence shape
|
|
40
|
+
|
|
41
|
+
| Shape | How to start | Recovery behavior |
|
|
42
|
+
|---|---|---|
|
|
43
|
+
| In-memory only | `createDatabase()`, `Database::in_memory()`, or `lora-server` with no durability flags | Restart starts from an empty graph |
|
|
44
|
+
| Snapshot only | Save with `save_snapshot` / `POST /admin/snapshot/save`, restore with `load_snapshot` / `--restore-from` | Restart returns to the last snapshot only |
|
|
45
|
+
| WAL only | Open the same WAL directory or `.loradb` archive again | Replays committed writes from the WAL into an empty graph |
|
|
46
|
+
| Snapshot + WAL | Restore a snapshot and open the same WAL directory or `.loradb` archive | Loads the snapshot, reads its `walLsn` fence, then replays committed WAL records newer than that fence |
|
|
47
|
+
|
|
48
|
+
Use WAL-only when the log is small enough to replay from scratch. Add
|
|
49
|
+
checkpoints when replay time or log size starts to matter.
|
|
50
|
+
|
|
51
|
+
## Quick start
|
|
52
|
+
|
|
53
|
+
### Node.js
|
|
54
|
+
|
|
55
|
+
```ts
|
|
56
|
+
import { createDatabase, openWalDatabase } from '@loradb/lora-node';
|
|
57
|
+
|
|
58
|
+
const scratch = await createDatabase(); // in-memory
|
|
59
|
+
const db = await createDatabase('app', { databaseDir: './data' }); // ./data/app.loradb
|
|
60
|
+
|
|
61
|
+
const walDb = await openWalDatabase({
|
|
62
|
+
walDir: './data/app.wal',
|
|
63
|
+
snapshotDir: './data/app.snapshots',
|
|
64
|
+
snapshotEveryCommits: 1000,
|
|
65
|
+
snapshotOptions: { compression: { format: 'gzip', level: 1 } },
|
|
66
|
+
syncMode: 'groupSync',
|
|
67
|
+
});
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
The name is validated and resolved under `databaseDir` as a `.loradb`
|
|
71
|
+
archive. Relative paths resolve from the current working directory.
|
|
72
|
+
Reopening the same name and directory replays committed WAL records before
|
|
73
|
+
the handle is returned. The open runs on the libuv threadpool, so the event
|
|
74
|
+
loop keeps serving timers and requests while a large database loads.
|
|
75
|
+
|
|
76
|
+
Use `openWalDatabase` when you want a raw WAL directory instead of a
|
|
77
|
+
portable `.loradb` archive. Pair it with `snapshotDir` to let the
|
|
78
|
+
database write managed checkpoints every `snapshotEveryCommits`
|
|
79
|
+
committed transactions, and in any case each time the WAL has grown by
|
|
80
|
+
64 MiB or by a quarter of the snapshot's size, whichever is larger. Without `snapshotDir` nothing is checkpointed automatically. Node does not expose WAL status or truncate
|
|
81
|
+
helpers yet; use Rust or `lora-server` for those operator controls.
|
|
82
|
+
|
|
83
|
+
### Python
|
|
84
|
+
|
|
85
|
+
```python
|
|
86
|
+
from lora_python import Database, AsyncDatabase
|
|
87
|
+
|
|
88
|
+
scratch = Database.create() # in-memory
|
|
89
|
+
db = Database.create("app", {"database_dir": "./data"}) # container-backed
|
|
90
|
+
also_db = Database("app", {"database_dir": "./data"}) # equivalent
|
|
91
|
+
|
|
92
|
+
wal_db = Database.open_wal(
|
|
93
|
+
"./data/app.wal",
|
|
94
|
+
{
|
|
95
|
+
"snapshot_dir": "./data/app.snapshots",
|
|
96
|
+
"snapshot_every_commits": 1000,
|
|
97
|
+
"snapshot_options": {"compression": {"format": "gzip", "level": 1}},
|
|
98
|
+
},
|
|
99
|
+
)
|
|
100
|
+
|
|
101
|
+
async_db = await AsyncDatabase.create("app", {"database_dir": "./data"})
|
|
102
|
+
async_wal = await AsyncDatabase.open_wal(
|
|
103
|
+
"./data/async.wal",
|
|
104
|
+
{"snapshot_dir": "./data/async.snapshots", "snapshot_every_commits": 1000},
|
|
105
|
+
)
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
### Go
|
|
109
|
+
|
|
110
|
+
```go
|
|
111
|
+
scratch, err := lora.New() // in-memory
|
|
112
|
+
db, err := lora.New("app", lora.Options{DatabaseDir: "./data"}) // container-backed
|
|
113
|
+
walDb, err := lora.OpenWal(lora.WalOptions{
|
|
114
|
+
WalDir: "./data/app.wal",
|
|
115
|
+
SnapshotDir: "./data/app.snapshots",
|
|
116
|
+
SnapshotEveryCommits: 1000,
|
|
117
|
+
})
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
### Ruby
|
|
121
|
+
|
|
122
|
+
```ruby
|
|
123
|
+
scratch = LoraRuby::Database.create # in-memory
|
|
124
|
+
db = LoraRuby::Database.create("app", {"database_dir": "./data"}) # container-backed
|
|
125
|
+
wal_db = LoraRuby::Database.open_wal(
|
|
126
|
+
"./data/app.wal",
|
|
127
|
+
snapshot_dir: "./data/app.snapshots",
|
|
128
|
+
snapshot_every_commits: 1000,
|
|
129
|
+
)
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
### Rust
|
|
133
|
+
|
|
134
|
+
```rust
|
|
135
|
+
use lora_database::{Database, WalConfig};
|
|
136
|
+
|
|
137
|
+
let db = Database::open_with_wal(WalConfig::enabled("./app"))?;
|
|
138
|
+
db.execute("CREATE (:Person {name: 'Ada'})", None)?;
|
|
139
|
+
|
|
140
|
+
// Later: restore a snapshot, then replay WAL above its fence.
|
|
141
|
+
let recovered = Database::recover("graph.bin", WalConfig::enabled("./app"))?;
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
`WalConfig::enabled(path)` uses the current defaults:
|
|
145
|
+
|
|
146
|
+
| Setting | Value |
|
|
147
|
+
|---|---|
|
|
148
|
+
| Sync mode | `SyncMode::GroupSync { interval_ms: 50 }` |
|
|
149
|
+
| Segment target | 8 MiB |
|
|
150
|
+
|
|
151
|
+
Use the explicit enum variant when you need different knobs:
|
|
152
|
+
|
|
153
|
+
```rust
|
|
154
|
+
use lora_database::{Database, SyncMode, WalConfig};
|
|
155
|
+
|
|
156
|
+
let db = Database::open_with_wal(WalConfig::Enabled {
|
|
157
|
+
dir: "./app".into(),
|
|
158
|
+
sync_mode: SyncMode::GroupSync { interval_ms: 50 },
|
|
159
|
+
segment_target_bytes: 16 * 1024 * 1024,
|
|
160
|
+
})?;
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
### HTTP server
|
|
164
|
+
|
|
165
|
+
```bash
|
|
166
|
+
# Fresh boot with a WAL.
|
|
167
|
+
lora-server --wal-dir /var/lib/lora/wal
|
|
168
|
+
|
|
169
|
+
# Snapshot + WAL recovery, with checkpoint default path.
|
|
170
|
+
lora-server \
|
|
171
|
+
--wal-dir /var/lib/lora/wal \
|
|
172
|
+
--snapshot-path /var/lib/lora/graph.bin \
|
|
173
|
+
--restore-from /var/lib/lora/graph.bin
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
When `--restore-from` and `--wal-dir` are both set, the server loads
|
|
177
|
+
the snapshot first and then replays committed WAL records newer than
|
|
178
|
+
the snapshot's `walLsn`. If the snapshot file is missing, recovery
|
|
179
|
+
falls back to WAL-only and starts from an empty graph before replay.
|
|
180
|
+
|
|
181
|
+
`--snapshot-path` is not required for WAL recovery. It only enables the
|
|
182
|
+
snapshot save/load admin routes and gives `/admin/checkpoint` a default
|
|
183
|
+
target path.
|
|
184
|
+
|
|
185
|
+
## What gets logged
|
|
186
|
+
|
|
187
|
+
The database arms the WAL once a query has parsed and compiled. The WAL
|
|
188
|
+
does not allocate a transaction until the first primitive mutation
|
|
189
|
+
fires.
|
|
190
|
+
|
|
191
|
+
| Query outcome | WAL behavior |
|
|
192
|
+
|---|---|
|
|
193
|
+
| Read-only query | Writes no records and does not fsync |
|
|
194
|
+
| Successful mutating query | Usually writes one committed `MutationBatch`; transaction-style scopes may write `TxBegin`, `Mutation` records, then `TxCommit` |
|
|
195
|
+
| Failed mutating query | Does not publish a committed batch; transaction-style scopes that opened a WAL transaction may write `TxAbort` |
|
|
196
|
+
|
|
197
|
+
Replay is query-atomic even when the log records individual primitive
|
|
198
|
+
mutations. A crashed process can leave an uncommitted tail; replay drops it. A
|
|
199
|
+
torn record at the end of the active segment is truncated back to the last
|
|
200
|
+
valid record before new appends resume, and the cut bytes are kept beside the
|
|
201
|
+
segment as `<segment>.wal.torn-<offset>`.
|
|
202
|
+
|
|
203
|
+
Damage anywhere else fails the open with `LORA_WAL_CORRUPTION` instead of
|
|
204
|
+
being cut off: a bad record followed by valid records, a bad record in an
|
|
205
|
+
older or sealed segment, or a record whose checksum is valid but whose
|
|
206
|
+
contents do not decode. So does a WAL that does not reach back to the replay
|
|
207
|
+
fence, for example a log whose early segments a checkpoint removed, opened
|
|
208
|
+
without that checkpoint's snapshot, or a log missing a segment in the middle.
|
|
209
|
+
|
|
210
|
+
Opening a corrupt WAL needs an explicit decision from you; see
|
|
211
|
+
[Corrupt WAL: incident procedure](#corrupt-wal-incident-procedure).
|
|
212
|
+
|
|
213
|
+
A transaction too large for one WAL record is split across several
|
|
214
|
+
`MutationBatch` records inside one `TxBegin`/`TxCommit` pair. Property values
|
|
215
|
+
nested more than 128 lists or maps deep are rejected when they are written.
|
|
216
|
+
|
|
217
|
+
## Recovery and checkpoints
|
|
218
|
+
|
|
219
|
+
Recovery follows the same steps in Rust and `lora-server`:
|
|
220
|
+
|
|
221
|
+
1. Load the snapshot if one was supplied. Its `walLsn` becomes the
|
|
222
|
+
replay fence. A snapshot saved without a WAL has `walLsn: null`,
|
|
223
|
+
which is treated as fence `0`.
|
|
224
|
+
2. Open the WAL directory or `.loradb` archive and replay every committed transaction above
|
|
225
|
+
the fence.
|
|
226
|
+
3. Install the live WAL recorder, then accept new queries.
|
|
227
|
+
|
|
228
|
+
Any snapshot saved from a WAL-backed database carries a fence, whether
|
|
229
|
+
it came from `saveSnapshot`, `POST /admin/snapshot/save` or a checkpoint.
|
|
230
|
+
The save fsyncs the WAL and reads its durable LSN while holding the
|
|
231
|
+
writer lock, so the snapshot and the fence describe the same graph and
|
|
232
|
+
recovery replays exactly the writes made after the save. Writers pause
|
|
233
|
+
for that fsync, not for the snapshot encode. A plain save does not
|
|
234
|
+
write a `Checkpoint` marker or drop WAL segments.
|
|
235
|
+
|
|
236
|
+
A snapshot with `walLsn: null` (saved without a WAL) is a seed: start
|
|
237
|
+
it over an empty WAL directory with
|
|
238
|
+
`lora-server --restore-from seed.bin --wal-dir <dir>`, and keep passing
|
|
239
|
+
the same seed on later boots. Over a WAL whose early segments a
|
|
240
|
+
checkpoint removed, recovery fails with `LORA_WAL_CORRUPTION` instead of
|
|
241
|
+
replaying a partial log onto it. If the WAL is poisoned, a save writes
|
|
242
|
+
`walLsn: null` too and logs a warning; use that file as a seed for a
|
|
243
|
+
new WAL directory.
|
|
244
|
+
|
|
245
|
+
Checkpointing creates a new fence:
|
|
246
|
+
|
|
247
|
+
1. Drain the WAL according to the configured sync mode.
|
|
248
|
+
2. Read the WAL's current `durableLsn`.
|
|
249
|
+
3. Save a snapshot stamped with that LSN in its `walLsn` header field.
|
|
250
|
+
4. Rename the snapshot into place.
|
|
251
|
+
5. Append a WAL `Checkpoint` marker.
|
|
252
|
+
6. Best-effort truncate sealed WAL segments that are safe to drop.
|
|
253
|
+
|
|
254
|
+
The checkpoint marker is written after the snapshot rename succeeds, so
|
|
255
|
+
a marker implies the snapshot existed at the time of the checkpoint.
|
|
256
|
+
If recovery sees a newer checkpoint marker than the snapshot you
|
|
257
|
+
supplied, it prints a warning and still replays from the snapshot's own
|
|
258
|
+
fence. That is safe, but it may do more replay work than necessary.
|
|
259
|
+
|
|
260
|
+
## Online backups of a named database
|
|
261
|
+
|
|
262
|
+
A named database (`createDatabase("app", { databaseDir })`,
|
|
263
|
+
`Database::open_named`) belongs to the process that opened it. Copying
|
|
264
|
+
its directory while that process writes does not give a database: the
|
|
265
|
+
WAL can be cut inside a record and the `.loradb` container lags the
|
|
266
|
+
WAL. A second process cannot open the directory to copy it either,
|
|
267
|
+
because the directory lock refuses it with `LORA_LOCKED`.
|
|
268
|
+
|
|
269
|
+
Take the copy from the owning process instead. `db.snapshot(targetDir)`
|
|
270
|
+
writes a self-contained `<targetDir>/<name>.loradb` while reads and
|
|
271
|
+
writes keep running:
|
|
272
|
+
|
|
273
|
+
```ts
|
|
274
|
+
import { createDatabase } from '@loradb/lora-node';
|
|
275
|
+
|
|
276
|
+
const db = await createDatabase('festimap', { databaseDir: './data' });
|
|
277
|
+
await db.execute('UNWIND range(1, 100000) AS i CREATE (:N {i: i})');
|
|
278
|
+
|
|
279
|
+
const snapshot = await db.snapshot('./backups/2026-10-26');
|
|
280
|
+
console.log(snapshot);
|
|
281
|
+
// {
|
|
282
|
+
// bytes: 69283,
|
|
283
|
+
// lsn: 3,
|
|
284
|
+
// nodeCount: 100000,
|
|
285
|
+
// path: './backups/2026-10-26/festimap.loradb',
|
|
286
|
+
// relationshipCount: 0
|
|
287
|
+
// }
|
|
288
|
+
|
|
289
|
+
// The copy opens like any database directory.
|
|
290
|
+
const copy = await createDatabase('festimap', { databaseDir: './backups/2026-10-26' });
|
|
291
|
+
await copy.execute('MATCH (n:N) RETURN count(n) AS nodes'); // [{ nodes: 100000 }]
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
What the copy holds and what it costs:
|
|
295
|
+
|
|
296
|
+
- **One committed transaction.** The snapshot holds every transaction
|
|
297
|
+
committed at or below `lsn` and none above it, never part of one. The
|
|
298
|
+
change feed batch with that `lsn` is the last write in it.
|
|
299
|
+
- **Writers do not wait for the copy.** The call takes the writer lock
|
|
300
|
+
only to fsync the WAL and take the graph; the graph's storage is
|
|
301
|
+
shared copy-on-write, so later writes copy what they change and the
|
|
302
|
+
snapshot keeps reading the version it took. It runs off the JS thread
|
|
303
|
+
and streams, so memory stays bounded by its buffers plus whatever the
|
|
304
|
+
concurrent writes change.
|
|
305
|
+
- **Complete or absent.** The container is written under a temporary
|
|
306
|
+
name, fsynced, and renamed into place. If the process dies mid-copy
|
|
307
|
+
(even `kill -9`), the target holds no `.loradb`, only a temporary
|
|
308
|
+
`<name>.loradb.<pid>.<time>.<n>.tmp` that the next open of that
|
|
309
|
+
directory removes. A copy whose promise resolved is durable.
|
|
310
|
+
- **The target must be missing or empty** (`LORA_VALIDATION`
|
|
311
|
+
otherwise) and may be on another filesystem: only the rename stays
|
|
312
|
+
inside it. Ship the directory off-host with any tool once the call
|
|
313
|
+
resolves.
|
|
314
|
+
- **Named databases only.** An in-memory database, or one opened with
|
|
315
|
+
`openWalDatabase`, rejects with `LORA_CONFIG`: save those with
|
|
316
|
+
`saveSnapshot`.
|
|
317
|
+
|
|
318
|
+
`db.checkpoint()` checkpoints now and reports the snapshot it wrote. On
|
|
319
|
+
a named database it does what `sync()` does (the container becomes a
|
|
320
|
+
snapshot of the graph plus the WAL since); with `walDir` and
|
|
321
|
+
`snapshotDir` it writes a managed snapshot:
|
|
322
|
+
|
|
323
|
+
```ts
|
|
324
|
+
await db.checkpoint();
|
|
325
|
+
// { formatVersion: 2, nodeCount: 100000, relationshipCount: 0, walLsn: 3 }
|
|
326
|
+
```
|
|
327
|
+
|
|
328
|
+
In Rust the same calls are `Database::snapshot_to_dir`, which returns a
|
|
329
|
+
`DirectorySnapshot { path, bytes, lsn, node_count, relationship_count }`,
|
|
330
|
+
and `Database::checkpoint`, which returns a `SnapshotMeta`:
|
|
331
|
+
|
|
332
|
+
```rust
|
|
333
|
+
use lora_database::{Database, DatabaseOpenOptions};
|
|
334
|
+
|
|
335
|
+
let db = Database::open_named("festimap", DatabaseOpenOptions::default().with_database_dir("./data"))?;
|
|
336
|
+
let snapshot = db.snapshot_to_dir("./backups/2026-10-26")?;
|
|
337
|
+
println!("{} bytes up to LSN {}", snapshot.bytes, snapshot.lsn);
|
|
338
|
+
```
|
|
339
|
+
|
|
340
|
+
A GraphQL server built on `@loradb/lora-graphql` exposes the same copy
|
|
341
|
+
as `lora.snapshot(targetDir)`; see [Serving](https://loradb.com/docs/graphql/serving#backups).
|
|
342
|
+
|
|
343
|
+
### A closed directory: `lora snapshot`
|
|
344
|
+
|
|
345
|
+
For a database no process has open, `@loradb/lora-node` ships a `lora`
|
|
346
|
+
command:
|
|
347
|
+
|
|
348
|
+
```bash
|
|
349
|
+
npx lora snapshot ./data ./backups/offline
|
|
350
|
+
# ./backups/offline/festimap.loradb: 69283 bytes, 100000 nodes, 0 relationships, at LSN 3
|
|
351
|
+
```
|
|
352
|
+
|
|
353
|
+
It opens the database, which takes the directory lock, then writes the
|
|
354
|
+
same snapshot. A directory another process owns is refused rather than
|
|
355
|
+
copied mid-write:
|
|
356
|
+
|
|
357
|
+
```text
|
|
358
|
+
lora: database `festimap` in `./data` is open in another process; snapshot it from that process with `await db.snapshot(targetDir)`
|
|
359
|
+
```
|
|
360
|
+
|
|
361
|
+
When the directory holds several databases, pick one with
|
|
362
|
+
`--name <name>`. Opening also recovers a directory a crash left behind
|
|
363
|
+
before the copy is taken.
|
|
364
|
+
|
|
365
|
+
## Corrupt WAL: incident procedure
|
|
366
|
+
|
|
367
|
+
When the WAL is damaged in the middle (a flipped bit on disk, a segment
|
|
368
|
+
overwritten by another tool), the database refuses to open with
|
|
369
|
+
`LORA_WAL_CORRUPTION`. The error names the segment file, the byte
|
|
370
|
+
offset, and what was wrong there, for example:
|
|
371
|
+
|
|
372
|
+
```text
|
|
373
|
+
WAL segment `/var/lib/lora/wal/0000000003.wal` is corrupt at byte 52113: CRC mismatch ...; a valid record follows at byte 52190, so this is not a torn tail
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
Nothing is changed on disk. Work through these steps:
|
|
377
|
+
|
|
378
|
+
1. **Stop and copy.** Keep the server stopped and copy the whole WAL
|
|
379
|
+
directory (and your snapshots) somewhere safe before trying anything.
|
|
380
|
+
2. **Prefer a better copy.** If you have a backup, a replica, or a
|
|
381
|
+
recent snapshot plus an intact WAL elsewhere, restore that instead.
|
|
382
|
+
3. **Decide whether to cut.** If the WAL in place is the only copy,
|
|
383
|
+
you can start from everything before the damage. Writes recorded
|
|
384
|
+
after the corrupt point are not recovered. If the damage is in an
|
|
385
|
+
older segment, that includes every later segment.
|
|
386
|
+
4. **Start once with the cut enabled**, then restart without it:
|
|
387
|
+
|
|
388
|
+
```bash
|
|
389
|
+
lora-server --wal-dir /var/lib/lora/wal --wal-truncate-corrupt-tail
|
|
390
|
+
# or LORA_SERVER_WAL_TRUNCATE_CORRUPT_TAIL=true
|
|
391
|
+
```
|
|
392
|
+
|
|
393
|
+
In Rust, set `truncate_corrupt_tail: true` on `WalConfig::Enabled`
|
|
394
|
+
for `Database::open_with_wal`, `open_with_wal_snapshots` or
|
|
395
|
+
`recover`.
|
|
396
|
+
|
|
397
|
+
With the option set, the open cuts the log at the first corrupt record
|
|
398
|
+
and starts with the transactions committed before it. Nothing is
|
|
399
|
+
deleted:
|
|
400
|
+
|
|
401
|
+
| What | Where it goes |
|
|
402
|
+
|---|---|
|
|
403
|
+
| Bytes of the corrupt segment from the bad record to its end | `<segment>.wal.corrupt-<offset>` beside it |
|
|
404
|
+
| Every later segment | Renamed to `<segment>.wal.after-corrupt-<corrupt segment>-<offset>` |
|
|
405
|
+
| The corrupt segment | Truncated at the bad record; if it was sealed, a new segment starts after it |
|
|
406
|
+
|
|
407
|
+
`lora-server` prints each of these on stderr at startup, and the
|
|
408
|
+
embedded library logs them as a `tracing` warning. In Rust,
|
|
409
|
+
`Wal::corrupt_cut()` (through `db.wal()`) reports them too. The open
|
|
410
|
+
only acts on `LORA_WAL_CORRUPTION` from a damaged record; a missing
|
|
411
|
+
segment (a gap) or an unreadable segment header still fails.
|
|
412
|
+
|
|
413
|
+
Once the server is up, take a snapshot or checkpoint, and do not keep
|
|
414
|
+
the option on: a later open should fail loudly if the disk damages the
|
|
415
|
+
log again. The option is not wired into named databases
|
|
416
|
+
(`openDatabase(name)` and the `.loradb` container) or the language
|
|
417
|
+
bindings' WAL options.
|
|
418
|
+
|
|
419
|
+
## Sync modes
|
|
420
|
+
|
|
421
|
+
`lora-server --wal-sync-mode` and the Rust API expose GroupSync
|
|
422
|
+
durability:
|
|
423
|
+
|
|
424
|
+
| Mode | `fsync` cadence | Crash window | When to use |
|
|
425
|
+
|---|---|---|---|
|
|
426
|
+
| `group-sync` | Background fsync, explicit sync, checkpoint, or clean drop | Up to the group interval | Higher write rates with explicit sync points when needed |
|
|
427
|
+
|
|
428
|
+
In `group-sync` mode, append and commit records are written before the query
|
|
429
|
+
returns, but the `fsync` happens on a background cadence. If that
|
|
430
|
+
background fsync fails, the failure is latched: future WAL operations
|
|
431
|
+
return a poisoned error and `/admin/wal/status` reports `bgFailure`.
|
|
432
|
+
Restart from the last consistent snapshot + WAL after fixing the
|
|
433
|
+
underlying disk issue.
|
|
434
|
+
|
|
435
|
+
`group-sync` is the only mode; `lora-server` runs its background fsync
|
|
436
|
+
every 50 ms, so a power loss or kernel crash can lose up to that much
|
|
437
|
+
acknowledged work. A process crash alone loses nothing that was already
|
|
438
|
+
written to the OS.
|
|
439
|
+
|
|
440
|
+
A change feed on a WAL-backed database only delivers a commit once it is
|
|
441
|
+
fsynced, because a commit lost in a crash would have its LSN reused by the
|
|
442
|
+
next write after the restart. The feed asks for the fsync straight away, so
|
|
443
|
+
an occasional write reaches the feed after one fsync; under a steady stream
|
|
444
|
+
of writes batches trail the commit by up to the group interval.
|
|
445
|
+
|
|
446
|
+
For checkpointed deployments, call `sync()` when you need an immediate
|
|
447
|
+
durability point. On a named `.loradb` database `sync()` is also the
|
|
448
|
+
checkpoint: it writes the graph as the archive's base snapshot, streamed so it
|
|
449
|
+
holds no second copy of the graph, and drops the WAL segments that snapshot
|
|
450
|
+
covers. A change feed can then resume only from the last `sync()` on. An open
|
|
451
|
+
that replays more WAL than a quarter of its snapshot, and at least 64 MiB of it,
|
|
452
|
+
takes the same checkpoint before it returns, so the next open loads the
|
|
453
|
+
snapshot instead of replaying the log again. A named database also takes
|
|
454
|
+
that checkpoint on its own, on a background thread, each time its WAL has grown
|
|
455
|
+
by 64 MiB since the last one, or by a quarter of its snapshot's size when
|
|
456
|
+
that is larger (`DatabaseOpenOptions::checkpoint_wal_bytes`, 0 turns it off), so a
|
|
457
|
+
process that never calls `sync()` does not keep every segment and a bulk load
|
|
458
|
+
into a large graph does not rewrite the snapshot over and over. A checkpoint snapshot is stamped with the WAL's current
|
|
459
|
+
`durableLsn`; in `group-sync` mode that fence
|
|
460
|
+
can trail the newest writes until the background fsync catches up.
|
|
461
|
+
|
|
462
|
+
## What's in the WAL directory
|
|
463
|
+
|
|
464
|
+
```text
|
|
465
|
+
<wal-dir>/
|
|
466
|
+
.lora-wal.lock held while a handle has the directory open
|
|
467
|
+
0000000001.wal sealed, oldest
|
|
468
|
+
0000000002.wal sealed
|
|
469
|
+
0000000003.wal active (highest id)
|
|
470
|
+
```
|
|
471
|
+
|
|
472
|
+
Segments rotate at the segment target (8 MiB by default), always at a
|
|
473
|
+
transaction boundary. There is no `CURRENT` pointer file: the active
|
|
474
|
+
segment is the one with the highest id. Files that don't match the
|
|
475
|
+
segment name pattern (such as a leftover `.tmp`) are ignored at boot.
|
|
476
|
+
Treat the directory as one unit: back it up, move it, or delete it
|
|
477
|
+
whole, never segment by segment.
|
|
478
|
+
|
|
479
|
+
## HTTP admin routes
|
|
480
|
+
|
|
481
|
+
When `lora-server` starts with `--wal-dir`, these routes are mounted:
|
|
482
|
+
|
|
483
|
+
| Method | Path | Purpose |
|
|
484
|
+
|---|---|---|
|
|
485
|
+
| `POST` | `/admin/wal/status` | Inspect durable LSN, next LSN, segment ids, and any latched background fsync failure |
|
|
486
|
+
| `POST` | `/admin/wal/truncate` | Drop sealed WAL segments up to a fence LSN |
|
|
487
|
+
| `POST` | `/admin/checkpoint` | Write a checkpoint snapshot and truncate safe WAL history |
|
|
488
|
+
|
|
489
|
+
Examples:
|
|
490
|
+
|
|
491
|
+
```bash
|
|
492
|
+
curl -sX POST http://127.0.0.1:4747/admin/wal/status
|
|
493
|
+
|
|
494
|
+
curl -sX POST http://127.0.0.1:4747/admin/wal/truncate \
|
|
495
|
+
-H 'content-type: application/json' \
|
|
496
|
+
-d '{"fenceLsn": 4815}'
|
|
497
|
+
|
|
498
|
+
curl -sX POST http://127.0.0.1:4747/admin/checkpoint \
|
|
499
|
+
-H 'content-type: application/json' \
|
|
500
|
+
-d '{"path": "/var/lib/lora/checkpoint.bin"}'
|
|
501
|
+
```
|
|
502
|
+
|
|
503
|
+
`/admin/wal/status` returns:
|
|
504
|
+
|
|
505
|
+
```json
|
|
506
|
+
{
|
|
507
|
+
"durableLsn": 4815,
|
|
508
|
+
"nextLsn": 4820,
|
|
509
|
+
"activeSegmentId": 3,
|
|
510
|
+
"oldestSegmentId": 2,
|
|
511
|
+
"bgFailure": null
|
|
512
|
+
}
|
|
513
|
+
```
|
|
514
|
+
|
|
515
|
+
`/admin/checkpoint` can omit the body only when `--snapshot-path` is
|
|
516
|
+
configured. Without a configured default, the request body must include
|
|
517
|
+
`path` or the route returns `400 Bad Request`. A checkpoint blocks writes
|
|
518
|
+
while it runs; reads continue.
|
|
519
|
+
|
|
520
|
+
`/admin/wal/truncate` can omit the body; in that case it truncates up
|
|
521
|
+
to the WAL's current `durableLsn`. It drops only sealed segments whose
|
|
522
|
+
whole LSN range is at or below the fence. The active segment and the
|
|
523
|
+
one just before it are always kept.
|
|
524
|
+
|
|
525
|
+
## Boundaries
|
|
526
|
+
|
|
527
|
+
- **Checkpoint automation is by WAL size and commit count, not by time.**
|
|
528
|
+
Named databases and managed snapshots checkpoint on a background thread
|
|
529
|
+
each time the WAL has grown by 64 MiB, or by the size of the current
|
|
530
|
+
snapshot when that is larger (`checkpoint_wal_bytes` in Rust; the
|
|
531
|
+
bindings use the default). Explicit WAL helpers in Node, Python, Go, and
|
|
532
|
+
Ruby can also write managed snapshots after N committed transactions. A
|
|
533
|
+
plain WAL directory with no snapshot directory, which is what `lora-server
|
|
534
|
+
--wal-dir` opens, has nowhere to put a snapshot: it keeps every segment
|
|
535
|
+
until `POST /admin/checkpoint` or a checkpoint call, and logs one warning
|
|
536
|
+
once it passes 64 MiB. There is no wall-clock scheduler built into the
|
|
537
|
+
engine.
|
|
538
|
+
- **No auth on the admin surface.** Snapshot and WAL admin routes are
|
|
539
|
+
off by default but unauthenticated when enabled. Put them behind
|
|
540
|
+
authenticated ingress only.
|
|
541
|
+
- **No shared WAL/container root.** One live handle owns one WAL directory or
|
|
542
|
+
`.loradb` archive, enforced by an advisory lock (`.lora-wal.lock` inside a
|
|
543
|
+
WAL directory; archives take their own lock).
|
|
544
|
+
Opening the same root from another process, or from a second live handle in
|
|
545
|
+
the same process, fails with `LORA_LOCKED` until the first handle is closed.
|
|
546
|
+
See [Troubleshooting](./troubleshooting.md#walcontainer-root-is-already-open).
|
|
547
|
+
- **Binding support is asymmetric.** The filesystem-backed bindings can
|
|
548
|
+
open WAL-backed databases. Rust and `lora-server` expose full
|
|
549
|
+
checkpoint, truncate, status, and sync-mode controls. Node also
|
|
550
|
+
exposes sync-mode control. Python, Go, and Ruby expose simple raw WAL
|
|
551
|
+
opens plus managed snapshot options. WASM stays snapshot-only.
|
|
552
|
+
- **No cross-version WAL compatibility guarantee yet.** Snapshots are
|
|
553
|
+
the portable backup and migration artifact. Treat WAL directories as
|
|
554
|
+
local runtime state for the same deployment line unless a release
|
|
555
|
+
explicitly says otherwise.
|
|
556
|
+
|
|
557
|
+
## See also
|
|
558
|
+
|
|
559
|
+
- [**Snapshots**](./snapshot.md) - save / load the graph as a single file.
|
|
560
|
+
- [**HTTP server guide**](https://loradb.com/docs/getting-started/server) - run
|
|
561
|
+
`lora-server`, configure flags, and probe it with `curl`.
|
|
562
|
+
- [**HTTP API**](https://loradb.com/docs/api/http) - route-by-route request / response
|
|
563
|
+
reference.
|
|
564
|
+
- [**Troubleshooting**](./troubleshooting.md#wal-and-checkpoints) - common
|
|
565
|
+
WAL setup and recovery errors.
|
|
566
|
+
- [**Node guide**](./getting-started/node.md#persisting-your-graph) -
|
|
567
|
+
`createDatabase("app", { databaseDir: "./data" })` and
|
|
568
|
+
`openWalDatabase({ walDir })` in context.
|
|
569
|
+
- [**Rust guide**](https://loradb.com/docs/getting-started/rust#persisting-your-graph) -
|
|
570
|
+
embedding the engine directly.
|