@jcbuisson/express-x-plugins 5.0.7 → 5.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +180 -1
- package/docs/electric-fsm.md +244 -0
- package/docs/sync-fsm.md +117 -0
- package/package.json +34 -2
- package/src/electric-offline-model.mjs +14 -2
- package/src/electric-server-plugin.mjs +21 -5
- package/src/electric-sql-client-plugin.mjs +52 -0
- package/src/electric-sql-server-plugin.mjs +69 -0
- package/src/electric-sync-relations.mjs +99 -0
- package/src/sync-algorithm.mjs +146 -0
- package/src/sync-client-plugin.mjs +1023 -0
- package/src/sync-server-plugin.mjs +541 -0
- package/test/electric-relations-postgres.test.mjs +113 -0
- package/test/electric-relations.test.mjs +220 -0
- package/test/electric-server-plugin.test.mjs +3 -2
- package/test/electric-sql-plugin.test.mjs +74 -0
- package/test/sync-algorithm.test.mjs +229 -0
- package/test/sync-offline.test.mjs +4643 -0
package/README.md
CHANGED
|
@@ -4,6 +4,7 @@
|
|
|
4
4
|
Currently includes:
|
|
5
5
|
|
|
6
6
|
- a plugin which preserves room membership and socket data across page reloads
|
|
7
|
+
|
|
7
8
|
- a plugin integrating ElectricSQL sync engine into express-x, which greatly simplifies relational database
|
|
8
9
|
access and provides powerful local-first features
|
|
9
10
|
|
|
@@ -17,7 +18,10 @@ npm install @jcbuisson/express-x-plugins
|
|
|
17
18
|
|
|
18
19
|
## Electric plugin: local-first Postgres
|
|
19
20
|
|
|
20
|
-
|
|
21
|
+
Without managed sync (sync: false) or a supplied localDb, the client streams Shapes directly and sends CRUD calls directly;
|
|
22
|
+
it has no persistent local mutation queue or owner election.
|
|
23
|
+
|
|
24
|
+
Wth managed sync (sync: true) Express-X handles authorized PostgreSQL mutations; Electric streams committed rows to clients.
|
|
21
25
|
|
|
22
26
|
The electric plugin has two separate elections:
|
|
23
27
|
|
|
@@ -28,6 +32,11 @@ Each PGlite instance has its own in-memory state: caches, transaction state, and
|
|
|
28
32
|
They prevent duplicate database engines and duplicate synchronization across tabs. The database leader and sync owner can be different tabs.
|
|
29
33
|
|
|
30
34
|
|
|
35
|
+
Electric’s does incremental replication: after the initial snapshot, it streams database changes instead of repeatedly comparing client/server metadata.
|
|
36
|
+
Its Shapes can represent filtered subsets of tables
|
|
37
|
+
|
|
38
|
+
|
|
39
|
+
|
|
31
40
|
### Settings
|
|
32
41
|
```bash
|
|
33
42
|
# set lc_messages=C for the PostgreSQL role so Electric receives recognizable English errors
|
|
@@ -130,6 +139,48 @@ Import `prepareElectricSyncSchema` alongside `electricServerPlugin`. It adds
|
|
|
130
139
|
table. The primary key must be UUID in sync mode. All writes to synced tables must
|
|
131
140
|
advance `version`; hard deletes bypass tombstone confirmation.
|
|
132
141
|
|
|
142
|
+
#### Relationships and offline deletes
|
|
143
|
+
|
|
144
|
+
In sync mode, the server discovers PostgreSQL `ON DELETE CASCADE` foreign keys
|
|
145
|
+
between configured models. Deleting a parent writes tombstones for its current
|
|
146
|
+
children and descendants in the same transaction, including rows the deleting
|
|
147
|
+
offline client never saw. A delayed create or update referencing a deleted parent
|
|
148
|
+
is resolved as a child tombstone, so deletion wins in either arrival order.
|
|
149
|
+
The result includes `syncRejection: { code: 'sync-parent-deleted', message }`.
|
|
150
|
+
Retries replay that result through the normal mutation cursor.
|
|
151
|
+
|
|
152
|
+
The browser removes a rejected optimistic row immediately, acknowledges its
|
|
153
|
+
tombstone version, and reports an error through `onError` with `code` and `rowId`.
|
|
154
|
+
It does not resend the rejected mutation. Electric's tombstone snapshot completes
|
|
155
|
+
queue cleanup. An edit to an already deleted row is handled the same way, with
|
|
156
|
+
`code: 'sync-row-deleted'`. If the parent is missing rather than deleted, the
|
|
157
|
+
write remains retriable: its parent's offline create may still be queued.
|
|
158
|
+
|
|
159
|
+
Pass the same complete model list, including `tombstoneData`, to
|
|
160
|
+
`prepareElectricSyncSchema` and `electricServerPlugin`. Startup preparation also
|
|
161
|
+
repairs existing active children of soft-deleted parents and their descendants.
|
|
162
|
+
Clients only need to queue the parent deletion; explicit child deletions remain
|
|
163
|
+
compatible. Cascades are part of the authorized parent deletion and do not invoke
|
|
164
|
+
separate child authorization callbacks.
|
|
165
|
+
|
|
166
|
+
These rules apply to single-column `ON DELETE CASCADE` foreign keys referencing
|
|
167
|
+
the configured parent's primary key. Other foreign-key delete policies retain
|
|
168
|
+
their existing behavior. Unsupported CASCADE keys between configured models
|
|
169
|
+
fail discovery rather than silently losing integrity. Configure every model in
|
|
170
|
+
the relationship graph. Restart the plugin after changing foreign keys: discovery
|
|
171
|
+
is cached after the first sync mutation.
|
|
172
|
+
|
|
173
|
+
Transactions affecting these relationships acquire one shared PostgreSQL advisory
|
|
174
|
+
lock before cursor or row locks. This deliberately serializes relational writes
|
|
175
|
+
across plugin instances, including cyclic graphs; models without discovered
|
|
176
|
+
relationships remain parallel. All participating instances must use these rules.
|
|
177
|
+
Direct SQL writes and non-sync services bypass this coordination.
|
|
178
|
+
|
|
179
|
+
Regression tests run with `npm test`. To also exercise actual concurrent PostgreSQL
|
|
180
|
+
transactions, set `ELECTRIC_TEST_DATABASE_URL` to a test database and run `npm test`.
|
|
181
|
+
The PostgreSQL test creates and removes its own isolated schema and requires
|
|
182
|
+
permission to create schemas; it does not use application tables.
|
|
183
|
+
|
|
133
184
|
Without `sync: true`, the existing direct service methods remain available and
|
|
134
185
|
return created, updated, or deleted rows directly.
|
|
135
186
|
|
|
@@ -375,3 +426,131 @@ With `sync: true`, the sync owner's Electric stream stays active when its tab is
|
|
|
375
426
|
hidden because it serves other tabs too. To customize this behavior, pass
|
|
376
427
|
`streamOptions.runtimeVisibility` to `createElectricModel`. Other stream options
|
|
377
428
|
are preserved. Independent streams retain Electric's normal visibility behavior.
|
|
429
|
+
|
|
430
|
+
## Custom metadata sync plugin
|
|
431
|
+
|
|
432
|
+
The original timestamp/metadata synchronization engine is available separately
|
|
433
|
+
from Electric:
|
|
434
|
+
|
|
435
|
+
```sh
|
|
436
|
+
npm install dexie uuid @vueuse/core rxjs vue drizzle-orm
|
|
437
|
+
```
|
|
438
|
+
|
|
439
|
+
On the server, register Drizzle tables with a `uid` primary key and a shared
|
|
440
|
+
metadata table (`uid` primary key, `created_at`, `updated_at`, `deleted_at`
|
|
441
|
+
timestamps):
|
|
442
|
+
|
|
443
|
+
```js
|
|
444
|
+
import { syncServerPlugin } from '@jcbuisson/express-x-plugins/sync-server'
|
|
445
|
+
|
|
446
|
+
app.configure(syncServerPlugin, db, metadata, [todos], {
|
|
447
|
+
authorize: async (context, { modelName, action, args }) => {
|
|
448
|
+
return Boolean(context.socket?.data?.user)
|
|
449
|
+
},
|
|
450
|
+
})
|
|
451
|
+
```
|
|
452
|
+
|
|
453
|
+
The plugin registers each model's `findUnique`, `findMany`, `createWithMeta`,
|
|
454
|
+
`updateWithMeta`, and `deleteWithMeta` methods, plus `sync.go`. Configure each
|
|
455
|
+
model service's Express-X publishing policy to send mutation events to its
|
|
456
|
+
authorized subscribers. Authorization is required; `maxSyncRecords` defaults
|
|
457
|
+
to 10,000. Use separate applications for Electric and custom sync services
|
|
458
|
+
when their model names overlap.
|
|
459
|
+
|
|
460
|
+
On the client:
|
|
461
|
+
|
|
462
|
+
```js
|
|
463
|
+
import { syncClientPlugin } from '@jcbuisson/express-x-plugins/sync-client'
|
|
464
|
+
|
|
465
|
+
app.configure(syncClientPlugin, { cacheNamespace: 'my-app:current-user' })
|
|
466
|
+
const todo = app.createOfflineModel('todos', ['label'])
|
|
467
|
+
const subscription = todo.getObservable({}).subscribe(rows => console.log(rows))
|
|
468
|
+
await todo.create({ label: 'Available offline' })
|
|
469
|
+
await todo.update(uid, { label: 'Edited offline' })
|
|
470
|
+
await todo.remove(uid)
|
|
471
|
+
subscription.unsubscribe()
|
|
472
|
+
```
|
|
473
|
+
|
|
474
|
+
The client persists rows and dirty mutations in Dexie/IndexedDB and reconciles
|
|
475
|
+
subscribed filters on reconnect. `cacheNamespace` must be non-empty and should
|
|
476
|
+
isolate each backend/user. Models support `findByUID`, `reset`, and
|
|
477
|
+
`synchronizeAll` alongside observable reads and mutations. Vue scope disposal
|
|
478
|
+
cleans up model listeners and filter subscriptions.
|
|
479
|
+
|
|
480
|
+
The original names `offlinePlugin` and `drizzleOfflinePlugin` remain exported.
|
|
481
|
+
The pure algorithm is exported as `computeSyncResult` from `/sync-algorithm`
|
|
482
|
+
and `/sync-server`. It compares metadata timestamps and returns additions,
|
|
483
|
+
updates, and deletions for each side. Writes outside the plugin must maintain
|
|
484
|
+
the metadata table too. Tombstones persist; safe pruning requires knowing all
|
|
485
|
+
clients have observed a deletion.
|
|
486
|
+
|
|
487
|
+
Restored from the last committed custom implementations, dated July 8, 2026:
|
|
488
|
+
`express-x` **edb1db3** (algorithm and regression tests), `express-x-client`
|
|
489
|
+
**944c398** (client), and `express-x-drizzle` **ddf6804** (server). The algorithm
|
|
490
|
+
was removed from Express-X in **47c2293** on July 19. Only module imports and
|
|
491
|
+
plugin export aliases were adapted; the synchronization logic is preserved.
|
|
492
|
+
|
|
493
|
+
## Minimal SQL-oriented Electric plugin
|
|
494
|
+
|
|
495
|
+
A separate plugin provides two client methods: `sql.watch` for live table rows
|
|
496
|
+
and `sql.query` for named, parameterized PostgreSQL statements. It needs no models,
|
|
497
|
+
ORM, local database, Vue, RxJS, schema changes, or sync metadata.
|
|
498
|
+
|
|
499
|
+
```js
|
|
500
|
+
// Server (pg Pool or Client)
|
|
501
|
+
import { electricSqlServerPlugin } from '@jcbuisson/express-x-plugins/electric-sql-server'
|
|
502
|
+
|
|
503
|
+
app.configure(electricSqlServerPlugin, pool, {
|
|
504
|
+
tables: ['todos'],
|
|
505
|
+
statements: {
|
|
506
|
+
add: 'INSERT INTO todos (label) VALUES ($1) RETURNING *',
|
|
507
|
+
complete: 'UPDATE todos SET done = $1 WHERE id = $2 RETURNING *',
|
|
508
|
+
remove: 'DELETE FROM todos WHERE id = $1 RETURNING *',
|
|
509
|
+
},
|
|
510
|
+
electricUrl: 'http://localhost:3000/v1/shape',
|
|
511
|
+
authorize(context, operation) {
|
|
512
|
+
// Supply your application's authorization policy. Check statement name and
|
|
513
|
+
// params for query calls; table and SQL filter params for shape calls.
|
|
514
|
+
return Boolean(context.request?.user)
|
|
515
|
+
},
|
|
516
|
+
})
|
|
517
|
+
```
|
|
518
|
+
|
|
519
|
+
```js
|
|
520
|
+
// Browser (install @electric-sql/client)
|
|
521
|
+
import { electricSqlClientPlugin } from '@jcbuisson/express-x-plugins/electric-sql-client'
|
|
522
|
+
|
|
523
|
+
app.configure(electricSqlClientPlugin)
|
|
524
|
+
const stop = app.sql.watch('todos', {
|
|
525
|
+
where: 'done = $1',
|
|
526
|
+
params: [false],
|
|
527
|
+
}, rows => console.log(rows))
|
|
528
|
+
|
|
529
|
+
const [todo] = await app.sql.query('add', ['Buy milk'])
|
|
530
|
+
await app.sql.query('complete', [true, todo.id])
|
|
531
|
+
await app.sql.query('remove', [todo.id])
|
|
532
|
+
stop() // unsubscribe and abort this stream
|
|
533
|
+
app.sql.dispose() // stop every active watch
|
|
534
|
+
```
|
|
535
|
+
|
|
536
|
+
`watch(table, callback)` watches the whole table. Filters use Electric-supported
|
|
537
|
+
PostgreSQL WHERE expressions with `$1`, `$2`, etc.; null comparisons use `IS NULL`.
|
|
538
|
+
Callbacks receive the initial synchronized rows and later committed changes.
|
|
539
|
+
Electric Shapes are table subscriptions, not arbitrary SELECTs or joins. Named
|
|
540
|
+
statements can use full PostgreSQL SQL, including joins and `RETURNING`; `query`
|
|
541
|
+
returns their rows. SQL text stays on the server, and values are bound through pg.
|
|
542
|
+
|
|
543
|
+
Authorization is required for both statements and Shapes. A table allowlist alone
|
|
544
|
+
is not row authorization: enforce tenant restrictions in server SQL and validate
|
|
545
|
+
Shape filters in your policy. The proxy ignores client-supplied source credentials.
|
|
546
|
+
Server options include `sourceId`, `sourceSecret`, `serviceName` (default `sql`),
|
|
547
|
+
and `shapePath` (default `/electric-sql/v1/shape/:table`). Match a custom service
|
|
548
|
+
name and Shape base path on the client. Client `headers` and `onError` are passed
|
|
549
|
+
to Electric for authentication and stream error handling.
|
|
550
|
+
|
|
551
|
+
This plugin performs online writes with server-generated keys and physical deletes.
|
|
552
|
+
A successful write resolves before its Electric echo necessarily arrives. It does
|
|
553
|
+
not queue offline writes or apply optimistic updates. Electric reconnects and
|
|
554
|
+
resumes live reads; the existing managed Electric plugin remains available for
|
|
555
|
+
persistent offline storage and writes. Use separate service names and paths when
|
|
556
|
+
installing both plugins in one application.
|
|
@@ -0,0 +1,244 @@
|
|
|
1
|
+
# Electric sync state machines
|
|
2
|
+
|
|
3
|
+
These diagrams model `electric-managed-sync.mjs`, `electric-offline-model.mjs`,
|
|
4
|
+
and `electric-server-plugin.mjs`. They describe the plugin's own control flow,
|
|
5
|
+
not Electric's internal replication protocol. State names are analytical labels
|
|
6
|
+
unless explicitly identified as persisted fields. Timestamps, revisions, rows,
|
|
7
|
+
and versions are data guards, so the precise model is an extended FSM.
|
|
8
|
+
|
|
9
|
+
The main diagrams assume `sync: true` on both client and server. Without managed
|
|
10
|
+
sync or a supplied `localDb`, the client streams Shapes directly and sends CRUD
|
|
11
|
+
calls directly; it has no persistent local mutation queue or owner election.
|
|
12
|
+
|
|
13
|
+
## One managed model in one tab
|
|
14
|
+
|
|
15
|
+
```mermaid
|
|
16
|
+
stateDiagram-v2
|
|
17
|
+
[*] --> preparing
|
|
18
|
+
preparing --> active: database and schema ready
|
|
19
|
+
preparing --> initFailed: initialization rejected
|
|
20
|
+
preparing --> stopped: stop before preparation completes
|
|
21
|
+
state active {
|
|
22
|
+
[*] --> networkCheck
|
|
23
|
+
state networkCheck <<choice>>
|
|
24
|
+
networkCheck --> offline: browser reports offline
|
|
25
|
+
networkCheck --> waiting: browser permits network
|
|
26
|
+
offline --> waiting: online event
|
|
27
|
+
waiting --> owner: Web Lock acquired
|
|
28
|
+
waiting --> offline: offline event cancels request
|
|
29
|
+
owner --> offline: offline event releases ownership
|
|
30
|
+
}
|
|
31
|
+
active --> stopped: stop
|
|
32
|
+
stopped --> active: start after readiness
|
|
33
|
+
stopped --> initFailed: outstanding preparation rejects
|
|
34
|
+
preparing --> disposed: dispose
|
|
35
|
+
active --> disposed: dispose
|
|
36
|
+
stopped --> disposed: dispose
|
|
37
|
+
initFailed --> disposed: dispose cleanup
|
|
38
|
+
disposed --> [*]
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
- `preparing` creates or obtains the PGlite database, prepares local tables, and
|
|
42
|
+
installs the BroadcastChannel. Preparation is serialized across this plugin
|
|
43
|
+
instance's models. Reads and writes wait for readiness; failure rejects them.
|
|
44
|
+
- `waiting` is a follower with a pending Web Lock request. It can read and write
|
|
45
|
+
shared local rows and enqueue mutations; only the owner streams and flushes.
|
|
46
|
+
- The lock key is database/model-specific. One tab can own one model and follow
|
|
47
|
+
another. PGlite's database leader is a separate role and need not be the sync
|
|
48
|
+
owner. Closing an owner lets another waiting tab acquire the lock.
|
|
49
|
+
- `offline` means the browser reports offline, so ownership is released. This
|
|
50
|
+
differs from a stream error while the browser still reports online: a stream
|
|
51
|
+
error does not by itself release ownership.
|
|
52
|
+
- Release invalidates old stream callbacks, cancels the pending lock request,
|
|
53
|
+
and stops owner work. The lock callback drains background work before exiting.
|
|
54
|
+
Snapshot reset is ordered after previously queued snapshot tasks.
|
|
55
|
+
- `stop()` retains local rows and the mutation queue. Reads and local mutations
|
|
56
|
+
remain available. `start()` requests ownership; its promise does not wait for
|
|
57
|
+
ownership, a snapshot, or complete synchronization.
|
|
58
|
+
- `dispose()` removes network listeners, stops the model, waits for outstanding
|
|
59
|
+
work, closes its channel, and removes it from the model cache. The old API
|
|
60
|
+
cannot restart; a new `createElectricModel` call can create a fresh instance
|
|
61
|
+
while the plugin itself remains open. Disposing the plugin closes its owned
|
|
62
|
+
database after disposing all models; a supplied `localDb` is caller-owned.
|
|
63
|
+
- `initFailed` has no built-in initialization retry for that model instance.
|
|
64
|
+
Disposal still performs cleanup, although its promise can reject when
|
|
65
|
+
readiness failed. A non-abort Web Lock request failure is reported and does
|
|
66
|
+
not automatically enqueue a replacement request.
|
|
67
|
+
|
|
68
|
+
## The owner's stream connection
|
|
69
|
+
|
|
70
|
+
```mermaid
|
|
71
|
+
stateDiagram-v2
|
|
72
|
+
[*] --> connecting
|
|
73
|
+
connecting --> connected: Shape rows or connected stream message
|
|
74
|
+
connecting --> disconnected: stream error or disconnected message
|
|
75
|
+
connected --> disconnected: stream error or disconnected message
|
|
76
|
+
disconnected --> connected: later connected message or Shape rows
|
|
77
|
+
connected --> connected: apply Shape snapshot
|
|
78
|
+
connected --> connected: must-refetch invalidates cached snapshot
|
|
79
|
+
disconnected --> disconnected: must-refetch invalidates cached snapshot
|
|
80
|
+
connecting --> closed: ownership released or model stopped
|
|
81
|
+
connected --> closed: ownership released or model stopped
|
|
82
|
+
disconnected --> closed: ownership released or model stopped
|
|
83
|
+
closed --> [*]
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Actual reconnection/retry mechanics belong to `ShapeStream`. This wrapper marks
|
|
87
|
+
its connection false on a stream error and reports the error; it does not create
|
|
88
|
+
a fresh stream solely because an error occurred. Generation guards ignore late
|
|
89
|
+
callbacks from stopped streams. A `must-refetch` message resets the cached
|
|
90
|
+
snapshot; the connection flag is still independently taken from the stream.
|
|
91
|
+
|
|
92
|
+
Queue uploads are gated by ownership and browser network state, **not** by the
|
|
93
|
+
stream's `connected` flag. Writes can reach the server while the Shape stream
|
|
94
|
+
is disconnected. Followers receive the owner's connection status through
|
|
95
|
+
BroadcastChannel, so `online` is an observation, not a convergence guarantee.
|
|
96
|
+
|
|
97
|
+
## One local mutation queue entry
|
|
98
|
+
|
|
99
|
+
There is at most one entry per `(model_name, row_id)`. Local CRUD and queue
|
|
100
|
+
updates commit together in a PGlite transaction. A later local edit replaces
|
|
101
|
+
the entry's payload and allocates a new monotonically increasing revision.
|
|
102
|
+
|
|
103
|
+
```mermaid
|
|
104
|
+
stateDiagram-v2
|
|
105
|
+
[*] --> unqueued
|
|
106
|
+
unqueued --> pending: local create, update, or remove
|
|
107
|
+
pending --> sending: owner selects next revision
|
|
108
|
+
sending --> pending: transient error
|
|
109
|
+
sending --> failed: permanent error for current revision
|
|
110
|
+
sending --> awaitingEcho: response version recorded for current revision
|
|
111
|
+
sending --> pending: newer local revision supersedes request
|
|
112
|
+
awaitingEcho --> unqueued: snapshot version reaches acknowledged version
|
|
113
|
+
pending --> pending: local edit replaces revision
|
|
114
|
+
awaitingEcho --> pending: local edit resets acknowledgement
|
|
115
|
+
failed --> pending: local edit resets failure
|
|
116
|
+
failed --> failed: ordinary flush skips failed entry
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
| Diagram state | Persisted queue representation |
|
|
120
|
+
| --- | --- |
|
|
121
|
+
| Unqueued | No queue entry |
|
|
122
|
+
| Pending | `status = 'pending'`, `acknowledged_version IS NULL` |
|
|
123
|
+
| Sending | Same persisted state as Pending; selected request is in flight |
|
|
124
|
+
| Awaiting echo | `status = 'pending'`, `acknowledged_version` is set |
|
|
125
|
+
| Failed | `status = 'failed'`, with `failure_reason` |
|
|
126
|
+
|
|
127
|
+
The critical acknowledgement condition is:
|
|
128
|
+
|
|
129
|
+
```text
|
|
130
|
+
incoming row ID = queued row ID
|
|
131
|
+
AND incoming.version >= queued.acknowledged_version
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
The snapshot transaction removes acknowledged queue entries first, then adopts
|
|
135
|
+
unprotected live rows and removes unprotected rows missing from the live
|
|
136
|
+
snapshot. Tombstones carry versions too, allowing deletes to be acknowledged.
|
|
137
|
+
A mutation response alone does not clear the queue; if a sufficient snapshot
|
|
138
|
+
was already cached, reconciliation can clear it immediately after the response.
|
|
139
|
+
|
|
140
|
+
Additional transition guards and actions:
|
|
141
|
+
|
|
142
|
+
- Only the owner flushes, with one flush loop at a time per local model instance.
|
|
143
|
+
It selects unacknowledged Pending entries in revision order. A timer, local
|
|
144
|
+
write, owner startup, online event, owner BroadcastChannel notification, or
|
|
145
|
+
explicit `flushQueue()` can trigger a flush.
|
|
146
|
+
- A transient failure leaves the entry Pending and ends the current flush loop.
|
|
147
|
+
A later trigger retries it. A permanent failure marks the matching revision
|
|
148
|
+
Failed and permits the loop to continue with other entries.
|
|
149
|
+
- Permanent means `forbidden`, a `TypeError`, or HTTP 4xx except 408, 425, and
|
|
150
|
+
429. A response without a version is an error and remains retryable.
|
|
151
|
+
- Responses and failure writes are revision-guarded. An old response must not
|
|
152
|
+
acknowledge a newer local edit. If a create was edited while its request was
|
|
153
|
+
in flight, its successful response converts the queued create into an update
|
|
154
|
+
with another revision. If the old request failed, the newer create stays
|
|
155
|
+
queued and the server handles its newer revision idempotently.
|
|
156
|
+
- Any queue entry, including Failed and Awaiting echo, protects the optimistic
|
|
157
|
+
local row from snapshot replacement. Failed entries are not automatically
|
|
158
|
+
retried or rolled back. A subsequent local operation resets their status.
|
|
159
|
+
- Local remove hides the row immediately and queues delete. Update requires a
|
|
160
|
+
local row. Updates to an unacknowledged queued create retain action `create`;
|
|
161
|
+
updates after its acknowledgement use action `update`.
|
|
162
|
+
|
|
163
|
+
## One server mutation request
|
|
164
|
+
|
|
165
|
+
```mermaid
|
|
166
|
+
stateDiagram-v2
|
|
167
|
+
[*] --> validating
|
|
168
|
+
validating --> rejected: forbidden or invalid mutation
|
|
169
|
+
validating --> transaction: authorized and valid
|
|
170
|
+
state transaction {
|
|
171
|
+
[*] --> cursorLocked
|
|
172
|
+
state revisionCheck <<choice>>
|
|
173
|
+
cursorLocked --> revisionCheck: cursor row locked
|
|
174
|
+
revisionCheck --> replayed: revision at or below stored cursor
|
|
175
|
+
revisionCheck --> applied: revision above stored cursor
|
|
176
|
+
applied --> recorded: save revision and result
|
|
177
|
+
replayed --> [*]
|
|
178
|
+
recorded --> [*]
|
|
179
|
+
}
|
|
180
|
+
transaction --> committed: commit and return result
|
|
181
|
+
transaction --> rolledBack: transactional failure
|
|
182
|
+
rejected --> [*]
|
|
183
|
+
committed --> [*]
|
|
184
|
+
rolledBack --> [*]
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
The cursor key is `(table_name, client_id, row_id)`. Equal or older revisions
|
|
188
|
+
return that cursor's stored result without applying another mutation. A newer
|
|
189
|
+
revision applies its operation and saves its result atomically with the row
|
|
190
|
+
change. Losing the response after commit is therefore safe to retry. An older
|
|
191
|
+
revision returns the latest stored cursor result, not necessarily its own
|
|
192
|
+
original result.
|
|
193
|
+
|
|
194
|
+
The cursor serializes requests for the same client/row. It does not impose a
|
|
195
|
+
revision ordering across different clients; PostgreSQL row conflicts and
|
|
196
|
+
transaction execution determine their resulting writes. Server versions come
|
|
197
|
+
from `electric_sync_version_seq`, not client timestamps.
|
|
198
|
+
|
|
199
|
+
## One server record in sync mode
|
|
200
|
+
|
|
201
|
+
```mermaid
|
|
202
|
+
stateDiagram-v2
|
|
203
|
+
[*] --> absent
|
|
204
|
+
absent --> live: first create inserts row
|
|
205
|
+
absent --> tombstone: update or delete
|
|
206
|
+
live --> live: accepted update or edited create advances version
|
|
207
|
+
live --> live: initial create conflicts and preserves row
|
|
208
|
+
live --> tombstone: accepted delete advances version
|
|
209
|
+
tombstone --> tombstone: update or delete advances tombstone version
|
|
210
|
+
tombstone --> tombstone: create preserves tombstone
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
These transitions describe accepted newer cursor revisions. Replayed revisions
|
|
214
|
+
do not change the record. An initial create conflicts by preserving the existing
|
|
215
|
+
row. A later create from a cursor that already has an accepted revision is
|
|
216
|
+
treated as an update when it contains non-key data. Updates affect only live
|
|
217
|
+
rows; updating an absent or deleted row writes a tombstone instead. This sync
|
|
218
|
+
API has no tombstone-to-live transition or automatic tombstone pruning.
|
|
219
|
+
|
|
220
|
+
| State | PostgreSQL representation | Local snapshot result without a queued mutation |
|
|
221
|
+
| --- | --- | --- |
|
|
222
|
+
| Absent | No row | No local row |
|
|
223
|
+
| Live | Row with `deleted = false` and `version` | Local row mirrors snapshot |
|
|
224
|
+
| Tombstone | Row with `deleted = true` and `version` | Local row removed |
|
|
225
|
+
|
|
226
|
+
`tombstoneData` supplies values needed to insert a missing-row tombstone and
|
|
227
|
+
can clear selected fields on an existing row. It does not necessarily erase
|
|
228
|
+
every other stored column.
|
|
229
|
+
|
|
230
|
+
## What status means
|
|
231
|
+
|
|
232
|
+
`getStatus()` is a combination of independent observations, not a single FSM
|
|
233
|
+
state:
|
|
234
|
+
|
|
235
|
+
- `pending` counts both unsent mutations and acknowledged mutations awaiting
|
|
236
|
+
their Electric echo.
|
|
237
|
+
- `failed` counts permanently failed queue entries.
|
|
238
|
+
- `online` combines the browser network flag and observed stream connection.
|
|
239
|
+
- Managed `error` is the latest reported error message. It is not explicitly
|
|
240
|
+
cleared on recovery, so its presence alone does not establish current failure.
|
|
241
|
+
|
|
242
|
+
`pending = 0` does not establish initial snapshot readiness, and the plugin
|
|
243
|
+
exposes no explicit fully-caught-up state. The main completion boundary for a
|
|
244
|
+
local mutation is its queue entry being removed after version reconciliation.
|
package/docs/sync-fsm.md
ADDED
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
# Drizzle sync state machines
|
|
2
|
+
|
|
3
|
+
This models the restored implementation in `src/sync-server-plugin.mjs` and
|
|
4
|
+
`src/sync-algorithm.mjs`. The state names are analytical labels, not persisted
|
|
5
|
+
plugin fields. Because timestamps, UIDs, filters, and rows are unbounded data,
|
|
6
|
+
the precise model is an extended FSM: finite control states plus data guards.
|
|
7
|
+
|
|
8
|
+
## One server sync request
|
|
9
|
+
|
|
10
|
+
Each `sync.go(modelName, where, clientMetadataDict)` invocation has its own
|
|
11
|
+
machine. There is no single global synchronization state.
|
|
12
|
+
|
|
13
|
+
```mermaid
|
|
14
|
+
stateDiagram-v2
|
|
15
|
+
[*] --> validating
|
|
16
|
+
validating --> waiting: configured, authorized, valid envelope
|
|
17
|
+
validating --> rejected: validation or authorization error
|
|
18
|
+
waiting --> locked: scope lock granted
|
|
19
|
+
waiting --> rejected: lock acquisition error
|
|
20
|
+
state locked {
|
|
21
|
+
[*] --> reading
|
|
22
|
+
reading --> scoping: bounded rows loaded
|
|
23
|
+
scoping --> metadata: exclude existing rows outside filter
|
|
24
|
+
metadata --> planning: load metadata and repair eligible gaps
|
|
25
|
+
planning --> deleting: computeSyncResult
|
|
26
|
+
deleting --> ready: server deletions completed
|
|
27
|
+
ready --> [*]
|
|
28
|
+
}
|
|
29
|
+
locked --> succeeded: release lock and return plan
|
|
30
|
+
locked --> failed: exception and release lock
|
|
31
|
+
rejected --> [*]
|
|
32
|
+
failed --> [*]
|
|
33
|
+
succeeded --> [*]
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
- `validating` checks model configuration, authorization, metadata envelope,
|
|
37
|
+
and client record limit. Detailed metadata timestamp validation happens in
|
|
38
|
+
`planning` with the default algorithm; filter validation happens during reads.
|
|
39
|
+
- The lock queues overlapping filters for the same model. Independent scopes
|
|
40
|
+
can proceed concurrently. This lock is local to the plugin instance.
|
|
41
|
+
- `metadata` considers server row UIDs and effective client UIDs, so deleted
|
|
42
|
+
server records can still be found through their metadata tombstones.
|
|
43
|
+
- `deleting` calls authorized `deleteWithMeta` operations. Each mutation has its
|
|
44
|
+
own transaction and metadata row lock. The entire sync request is **not one
|
|
45
|
+
transaction**: failure can leave earlier repairs or deletions committed.
|
|
46
|
+
- Success returns a reconciliation plan. It does **not** mean the client has
|
|
47
|
+
applied the plan or uploaded its mutations. Retry is a new request initiated
|
|
48
|
+
by the client, not an automatic transition inside the server request.
|
|
49
|
+
|
|
50
|
+
## One server record
|
|
51
|
+
|
|
52
|
+
For a fixed `(modelName, uid)`, the normal persistent states are:
|
|
53
|
+
|
|
54
|
+
| State | Model row | Metadata |
|
|
55
|
+
| --- | --- | --- |
|
|
56
|
+
| Absent | Missing | Missing |
|
|
57
|
+
| Live | Present | No deletion timestamp |
|
|
58
|
+
| Tombstone | Missing | Deletion timestamp retained |
|
|
59
|
+
|
|
60
|
+
`t` is the incoming operation timestamp. `v` is the stored version selected
|
|
61
|
+
by the handler: `deleted_at`, otherwise `updated_at`, otherwise `created_at`.
|
|
62
|
+
The transitions below assume valid timestamps, authorization, successful
|
|
63
|
+
transactions, and normal metadata integrity.
|
|
64
|
+
|
|
65
|
+
```mermaid
|
|
66
|
+
stateDiagram-v2
|
|
67
|
+
[*] --> absent
|
|
68
|
+
absent --> live: create or timestamped update
|
|
69
|
+
absent --> tombstone: delete
|
|
70
|
+
live --> live: create or update with t greater than v
|
|
71
|
+
live --> live: stale create or update ignored
|
|
72
|
+
live --> live: delete with t less than v ignored
|
|
73
|
+
live --> tombstone: delete with t greater than or equal to v
|
|
74
|
+
tombstone --> live: create or update with t greater than v
|
|
75
|
+
tombstone --> tombstone: create or update with t less than or equal to v
|
|
76
|
+
tombstone --> tombstone: delete retains or advances tombstone
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
A deletion wins a tie with a live version. Equal live timestamps produce no
|
|
80
|
+
reconciliation action; the algorithm does not compare row contents. Newer
|
|
81
|
+
creates or updates can resurrect a tombstone. No automatic tombstone pruning
|
|
82
|
+
or transition back to Absent exists.
|
|
83
|
+
|
|
84
|
+
Integrity gaps add exceptional states: a row without metadata, metadata without
|
|
85
|
+
a row, or a row coexisting with a tombstone. The implementation repairs selected
|
|
86
|
+
gaps during sync and mutation handling. These repairs depend on the incoming
|
|
87
|
+
metadata and timestamps; they are not unconditional background transitions.
|
|
88
|
+
`updateWithMeta` also permits a missing timestamp, which can create
|
|
89
|
+
timestamp-free metadata when no stored version blocks it; the normal-state
|
|
90
|
+
diagram intentionally assumes timestamped operations.
|
|
91
|
+
|
|
92
|
+
## Reconciliation outputs
|
|
93
|
+
|
|
94
|
+
The pure algorithm is a guarded decision function over a snapshot. Its six
|
|
95
|
+
outputs are actions, not persistent record states.
|
|
96
|
+
|
|
97
|
+
| Output | Meaning | Executor |
|
|
98
|
+
| --- | --- | --- |
|
|
99
|
+
| `addClient` | Copy server row and metadata into local cache | Client |
|
|
100
|
+
| `updateClient` | Replace local copy with server version | Client |
|
|
101
|
+
| `deleteClient` | Remove local copy using the deletion version | Client |
|
|
102
|
+
| `addDatabase` | Upload a locally held row | Client calls `createWithMeta` |
|
|
103
|
+
| `updateDatabase` | Upload the newer local version | Client calls `updateWithMeta` |
|
|
104
|
+
| `deleteDatabase` | Delete a server row and retain metadata | Server during `sync.go` |
|
|
105
|
+
|
|
106
|
+
For live versions on both sides, the newer timestamp wins. A deletion wins
|
|
107
|
+
against an equal or older live version. When the server row is absent, retained
|
|
108
|
+
server metadata still participates: a sufficiently new server version can
|
|
109
|
+
remove the client copy, while a newer live client version can recreate the row.
|
|
110
|
+
Server records outside `where` are excluded before these comparisons.
|
|
111
|
+
|
|
112
|
+
The client applies downloads before uploads. Failed sync uploads preserve local
|
|
113
|
+
dirty mutations for a later reconnect or explicit `synchronizeAll()`. Individual
|
|
114
|
+
upload failures can be caught while the rest of the synchronization continues,
|
|
115
|
+
so a resolved synchronization call does not prove every dirty row was accepted.
|
|
116
|
+
Timestamp guards on acknowledgements prevent an older response from clearing
|
|
117
|
+
a newer local mutation.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@jcbuisson/express-x-plugins",
|
|
3
|
-
"version": "5.0
|
|
3
|
+
"version": "5.1.0",
|
|
4
4
|
"description": "Plugins for express-x",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./src/electric-server-plugin.mjs",
|
|
@@ -9,7 +9,12 @@
|
|
|
9
9
|
"./electric-client": "./src/electric-client-plugin.mjs",
|
|
10
10
|
"./reload-server": "./src/reload-server-plugin.mjs",
|
|
11
11
|
"./reload-client": "./src/reload-client-plugin.mjs",
|
|
12
|
-
"./electric-worker": "./src/electric-pglite-worker.mjs"
|
|
12
|
+
"./electric-worker": "./src/electric-pglite-worker.mjs",
|
|
13
|
+
"./sync-client": "./src/sync-client-plugin.mjs",
|
|
14
|
+
"./sync-server": "./src/sync-server-plugin.mjs",
|
|
15
|
+
"./sync-algorithm": "./src/sync-algorithm.mjs",
|
|
16
|
+
"./electric-sql-server": "./src/electric-sql-server-plugin.mjs",
|
|
17
|
+
"./electric-sql-client": "./src/electric-sql-client-plugin.mjs"
|
|
13
18
|
},
|
|
14
19
|
"private": false,
|
|
15
20
|
"publishConfig": {
|
|
@@ -37,7 +42,11 @@
|
|
|
37
42
|
"peerDependencies": {
|
|
38
43
|
"@electric-sql/client": "^1.5.24",
|
|
39
44
|
"@electric-sql/pglite": "^0.5.8",
|
|
45
|
+
"@vueuse/core": "^14.3.0",
|
|
46
|
+
"dexie": "^4.4.4",
|
|
47
|
+
"drizzle-orm": "^0.45.2",
|
|
40
48
|
"rxjs": "^7.8.2",
|
|
49
|
+
"uuid": "^14.0.1",
|
|
41
50
|
"vue": "^3.5.0"
|
|
42
51
|
},
|
|
43
52
|
"peerDependenciesMeta": {
|
|
@@ -52,12 +61,35 @@
|
|
|
52
61
|
},
|
|
53
62
|
"vue": {
|
|
54
63
|
"optional": true
|
|
64
|
+
},
|
|
65
|
+
"dexie": {
|
|
66
|
+
"optional": true
|
|
67
|
+
},
|
|
68
|
+
"uuid": {
|
|
69
|
+
"optional": true
|
|
70
|
+
},
|
|
71
|
+
"@vueuse/core": {
|
|
72
|
+
"optional": true
|
|
73
|
+
},
|
|
74
|
+
"drizzle-orm": {
|
|
75
|
+
"optional": true
|
|
55
76
|
}
|
|
56
77
|
},
|
|
57
78
|
"devDependencies": {
|
|
58
79
|
"@electric-sql/client": "^1.5.24",
|
|
59
80
|
"@electric-sql/pglite": "^0.5.8",
|
|
81
|
+
"@jcbuisson/express-x": "^4.0.5",
|
|
82
|
+
"@vueuse/core": "^14.3.0",
|
|
83
|
+
"bcryptjs": "^2.4.3",
|
|
84
|
+
"dexie": "^4.4.4",
|
|
85
|
+
"drizzle-orm": "^0.45.2",
|
|
86
|
+
"express": "^5.3.0",
|
|
87
|
+
"fake-indexeddb": "^6.2.4",
|
|
88
|
+
"pg": "^8.19.0",
|
|
60
89
|
"rxjs": "^7.8.2",
|
|
90
|
+
"socket.io": "^4.8.4",
|
|
91
|
+
"socket.io-client": "^4.8.3",
|
|
92
|
+
"uuid": "^14.0.1",
|
|
61
93
|
"vue": "^3.5.0"
|
|
62
94
|
}
|
|
63
95
|
}
|