matrix-nodejs-homeserver 0.3.2 → 0.3.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +34 -7
- package/dist/api/slidingsync.js +2 -2
- package/dist/api/sync.js +157 -70
- package/dist/core/push.js +26 -20
- package/dist/core/pushrules.js +24 -5
- package/dist/homeserver.js +2 -1
- package/dist/storage/batch.js +72 -1
- package/dist/storage/schema.js +7 -2
- package/dist/storage/sql.d.ts +22 -3
- package/dist/storage/sql.js +126 -28
- package/dist/storage/types.d.ts +15 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
# matrix-nodejs-homeserver
|
|
2
2
|
|
|
3
|
+
[](https://github.com/k18k/matrix-nodejs-homeserver/actions/workflows/ci.yml)
|
|
3
4
|
[](https://github.com/k18k/matrix-nodejs-homeserver/actions/workflows/loadtest.yml)
|
|
4
5
|
[](https://github.com/k18k/matrix-nodejs-homeserver/actions/workflows/loadtest.yml)
|
|
5
6
|
[](https://github.com/k18k/matrix-nodejs-homeserver/actions/workflows/loadtest.yml)
|
|
@@ -65,12 +66,21 @@ mysqlAdapter(mysql.createPool(uri)); // mysql2/promise
|
|
|
65
66
|
memoryAdapter(); // tests / demos
|
|
66
67
|
```
|
|
67
68
|
|
|
69
|
+
SQLite runs in WAL mode with `synchronous = NORMAL`: a commit survives a crash of the process,
|
|
70
|
+
but a power loss can roll back the last few. Pass `sqliteAdapter(db, { synchronous: "full" })`
|
|
71
|
+
to wait for the disk on every commit, at the cost of blocking the event loop each time.
|
|
72
|
+
|
|
68
73
|
Tables are created on startup. Writes for each event are atomic: SQLite uses
|
|
69
74
|
`BEGIN IMMEDIATE`, Postgres and MySQL use a dedicated pooled connection, and the
|
|
70
75
|
memory adapter keeps an undo log.
|
|
71
76
|
|
|
77
|
+
The Postgres adapter prepares its statements on each connection and reuses them. Behind a
|
|
78
|
+
pooler that does not support prepared statements (PgBouncer before 1.21, or in transaction
|
|
79
|
+
mode without `max_prepared_statements`), pass `postgresAdapter(pool, { preparedStatements: false })`.
|
|
80
|
+
|
|
72
81
|
To use another store, implement `DatabaseAdapter` (`create`, `upsert`, `findOne`,
|
|
73
|
-
`findMany`, `update`, `delete`, `max`, and optionally `migrate
|
|
82
|
+
`findMany`, `update`, `delete`, `max`, and optionally `migrate`, `transaction`,
|
|
83
|
+
`findManyBatch` and `upsertMany`).
|
|
74
84
|
The exported `schema` describes every model. If your driver can run parameterised
|
|
75
85
|
SQL, `sqlAdapter({ dialect, executor, transaction })` handles the SQL generation for you.
|
|
76
86
|
|
|
@@ -459,6 +469,7 @@ Live status, updated after every Complement run (full or targeted) with
|
|
|
459
469
|
|
|
460
470
|
| Run | Top-level | Subtests | Note |
|
|
461
471
|
|---|---|---|---|
|
|
472
|
+
| 10 | 214 / 214 | 851 / 851 | load-test fixes #25–#35: batched /sync reads, sync concurrency limit, Postgres lock leases, group commit (248ac8d) |
|
|
462
473
|
| 9 | 214 / 214 | 851 / 851 | zod, procedures, M/UNSTABLE, SSO confirmation, outbound address policy, admin guard (ec5aa79) |
|
|
463
474
|
| 8 | 214 / 214 | 851 / 851 | Complement runs in a Go container (no submodule, no local Go); account locking/suspension, policy servers, 3PIDs, legacy sync endpoints |
|
|
464
475
|
| 7 | 214 / 214 | 851 / 851 | All Complement tests pass (previous full run: 211/214). Fallback keys stored apart from one-time keys; user directory public-room rules; MSC4311 create event required from v12 |
|
|
@@ -513,18 +524,34 @@ Latest results: 1000 users online in 25 rooms (40 per room), 50 messages/s, 30 s
|
|
|
513
524
|
|
|
514
525
|
| Setup | req/s | send | /sync | join | delivery | messages delivered | failed |
|
|
515
526
|
|---|---|---|---|---|---|---|---|
|
|
516
|
-
| SQLite, 1 process
|
|
517
|
-
| Postgres, no cluster |
|
|
518
|
-
| Postgres, cluster, 1 node |
|
|
519
|
-
| Postgres, cluster, 5 nodes |
|
|
527
|
+
| SQLite, 1 process | 662 | 97 ms / 0.16 s | 0.87 s / 1.6 s | 10 ms / 91 ms | 1.64 s / 2.70 s | 53,935 | 0% |
|
|
528
|
+
| Postgres, no cluster | 670 | 15 ms / 19 ms | 1.1 s / 1.8 s | 14 ms / 23 ms | 1.76 s / 2.31 s | 58,667 | 0% |
|
|
529
|
+
| Postgres, cluster, 1 node | 593 | 27 ms / 38 ms | 1.3 s / 2.0 s | 36 ms / 51 ms | 1.96 s / 2.57 s | 58,493 | 0% |
|
|
530
|
+
| Postgres, cluster, 5 nodes | 871 | 60 ms / 88 ms | 0.50 s / 1.1 s | 62 ms / 119 ms | 0.70 s / 1.09 s | 59,298 | 0% |
|
|
520
531
|
|
|
521
532
|
What the load tests found and fixed so far: presence written on every `/sync` (#26), one commit
|
|
522
533
|
per push notification (#27), a missing index (#28), Postgres cluster locks that could deadlock
|
|
523
534
|
the pool (#29), stream writes serialized per instance (#30), `/sync` reading every change on the
|
|
524
535
|
server (#31) and every joined room (#33), lock handover by polling (#32). Then `/sync` reads were batched
|
|
525
536
|
(#34) and a process builds at most four sync responses at once (#35), so sends no longer queue
|
|
526
|
-
behind the syncs a message wakes.
|
|
527
|
-
|
|
537
|
+
behind the syncs a message wakes.
|
|
538
|
+
|
|
539
|
+
Since 0.3.2 (#38–#45), Postgres delivery time fell by about 55% on one node and 50% on five:
|
|
540
|
+
- changed rooms are built concurrently, with fewer reads each (#38);
|
|
541
|
+
- push notifications of a message are written in one statement (#39);
|
|
542
|
+
- Postgres statements are prepared once per connection (#40);
|
|
543
|
+
- push rules are no longer copied per member (#41);
|
|
544
|
+
- to-device acknowledgements run only when needed (#42);
|
|
545
|
+
- reads issued at the same moment by concurrent requests share one statement (#43);
|
|
546
|
+
- presence and device-list changes are found through a stream index (#44);
|
|
547
|
+
- batched reads take half as long to plan (#45).
|
|
548
|
+
|
|
549
|
+
On SQLite, commits no longer wait for an fsync (#46). Database calls also yield to the event
|
|
550
|
+
loop (#47): before that, a wave of syncs could hold it for seconds, sends took 0.74 s and the
|
|
551
|
+
server took in only about 40% of the messages the scenario sends.
|
|
552
|
+
|
|
553
|
+
With every user receiving about two messages a second, delivery time is how long it takes a
|
|
554
|
+
process to rebuild every user's `/sync` in turn: about 750 per second per Postgres node here.
|
|
528
555
|
<!-- loadtest-results:end -->
|
|
529
556
|
|
|
530
557
|
## License
|
package/dist/api/slidingsync.js
CHANGED
|
@@ -421,10 +421,10 @@ async function buildExtensions(hs, s, ext, since, to, roomIds, next) {
|
|
|
421
421
|
else if (roomIds.includes(r.room_id) || since !== null) (rooms[r.room_id] ??= []).push(ev);
|
|
422
422
|
}
|
|
423
423
|
if (since === null && !global.some((e) => e.type === M.PUSH_RULES)) {
|
|
424
|
-
const {
|
|
424
|
+
const { readOnlyRules } = await import("../core/pushrules.js");
|
|
425
425
|
global.push({
|
|
426
426
|
type: M.PUSH_RULES,
|
|
427
|
-
content: { global: await
|
|
427
|
+
content: { global: await readOnlyRules(hs, userId) }
|
|
428
428
|
});
|
|
429
429
|
}
|
|
430
430
|
out.account_data = {
|
package/dist/api/sync.js
CHANGED
|
@@ -2,26 +2,27 @@ import { M, UNSTABLE } from "../core/m.js";
|
|
|
2
2
|
import { eventIdOf, splitStateKey, stateKey, toClientEvent } from "../core/events.js";
|
|
3
3
|
import { errors } from "../core/errors.js";
|
|
4
4
|
import { getState } from "../core/authrules.js";
|
|
5
|
-
import {
|
|
5
|
+
import { readOnlyRules } from "../core/pushrules.js";
|
|
6
6
|
import { inviteDecision } from "../core/invitefilter.js";
|
|
7
7
|
import { deviceListChangesQuery } from "../core/keys.js";
|
|
8
8
|
import { body } from "../http/input.js";
|
|
9
9
|
import { procedure } from "../http/procedure.js";
|
|
10
|
-
import { matchesFilter, roomAllowed, validateSyncFilter } from "./filters.js";
|
|
11
10
|
import { readBatch } from "../storage/batch.js";
|
|
11
|
+
import { matchesFilter, roomAllowed, validateSyncFilter } from "./filters.js";
|
|
12
12
|
import { Semaphore } from "../core/semaphore.js";
|
|
13
13
|
import { fullProfile } from "../core/profile.js";
|
|
14
14
|
//#region src/api/sync.ts
|
|
15
15
|
function parseSyncToken(raw) {
|
|
16
16
|
if (!raw) return null;
|
|
17
|
-
const m = /^s(\d+)(?:_(\d+))?$/.exec(raw);
|
|
17
|
+
const m = /^s(\d+)(?:_(\d+))?(_a)?$/.exec(raw);
|
|
18
18
|
if (!m) throw errors.invalidParam("Invalid since token");
|
|
19
19
|
return {
|
|
20
20
|
pos: Number(m[1]),
|
|
21
|
-
toDevice: Number(m[2] ?? m[1])
|
|
21
|
+
toDevice: Number(m[2] ?? m[1]),
|
|
22
|
+
ack: !!m[3]
|
|
22
23
|
};
|
|
23
24
|
}
|
|
24
|
-
const formatToken = (t) => `s${t.pos}_${t.toDevice}`;
|
|
25
|
+
const formatToken = (t) => `s${t.pos}_${t.toDevice}${t.ack ? "_a" : ""}`;
|
|
25
26
|
async function loadFilter(hs, userId, raw) {
|
|
26
27
|
if (!raw) return {};
|
|
27
28
|
if (raw.trim().startsWith("{")) {
|
|
@@ -88,6 +89,8 @@ async function roomTimeline(hs, p, roomId, since, upTo) {
|
|
|
88
89
|
operator: "gt",
|
|
89
90
|
value: since
|
|
90
91
|
});
|
|
92
|
+
const sinceIdsRead = since !== null ? snapshotAt(hs, roomId, since).then((id) => hs.rooms.snapshotIds(id)) : Promise.resolve({});
|
|
93
|
+
const scanned = [];
|
|
91
94
|
const events = [];
|
|
92
95
|
let limited = false;
|
|
93
96
|
let cursor = upTo;
|
|
@@ -106,6 +109,7 @@ async function roomTimeline(hs, p, roomId, since, upTo) {
|
|
|
106
109
|
limit: batchSize
|
|
107
110
|
});
|
|
108
111
|
if (rows.length === 0) break;
|
|
112
|
+
scanned.push(...rows);
|
|
109
113
|
const stored = rows.map((r) => hs.rooms.rowToStored(r));
|
|
110
114
|
const visible = await hs.rooms.filterVisible(p.session.userId, roomId, stored);
|
|
111
115
|
const ok = new Map(visible.map((v) => [v.pdu.event_id, v]));
|
|
@@ -143,25 +147,50 @@ async function roomTimeline(hs, p, roomId, since, upTo) {
|
|
|
143
147
|
events.splice(0, lastGap);
|
|
144
148
|
limited = true;
|
|
145
149
|
}
|
|
146
|
-
if (!limited && events.length) limited = (await hs.db.findMany("events", {
|
|
147
|
-
where: [...where.filter((c) => c.operator !== "lte"), {
|
|
148
|
-
field: "stream",
|
|
149
|
-
operator: "lt",
|
|
150
|
-
value: events[0].stream
|
|
151
|
-
}],
|
|
152
|
-
limit: 1,
|
|
153
|
-
fields: ["stream"]
|
|
154
|
-
})).length > 0;
|
|
155
150
|
const startPos = events.length ? Math.min(...events.map((e) => e.stream)) - 1 : upTo;
|
|
156
|
-
const
|
|
151
|
+
const newestAtOrBefore = async (pos) => {
|
|
152
|
+
const row = scanned.find((r) => r.stream <= pos);
|
|
153
|
+
if (row) return row;
|
|
154
|
+
return (await hs.db.findMany("events", {
|
|
155
|
+
where: [{
|
|
156
|
+
field: "room_id",
|
|
157
|
+
value: roomId
|
|
158
|
+
}, {
|
|
159
|
+
field: "stream",
|
|
160
|
+
operator: "lte",
|
|
161
|
+
value: pos
|
|
162
|
+
}],
|
|
163
|
+
sortBy: {
|
|
164
|
+
field: "stream",
|
|
165
|
+
direction: "desc"
|
|
166
|
+
},
|
|
167
|
+
limit: 1,
|
|
168
|
+
fields: ["stream", "state_snapshot"]
|
|
169
|
+
}))[0] ?? null;
|
|
170
|
+
};
|
|
171
|
+
const [atStart, atEnd] = await Promise.all([newestAtOrBefore(startPos), startPos === upTo ? null : newestAtOrBefore(upTo)]);
|
|
172
|
+
const startSnapshot = atStart?.state_snapshot ?? 0;
|
|
173
|
+
const endSnapshot = startPos === upTo ? startSnapshot : atEnd?.state_snapshot ?? 0;
|
|
174
|
+
if (!limited && events.length) {
|
|
175
|
+
if (events[0].stream === startPos + 1) limited = atStart !== null && (since === null || atStart.stream > since);
|
|
176
|
+
else limited = (await hs.db.findMany("events", {
|
|
177
|
+
where: [...where.filter((c) => c.operator !== "lte"), {
|
|
178
|
+
field: "stream",
|
|
179
|
+
operator: "lt",
|
|
180
|
+
value: events[0].stream
|
|
181
|
+
}],
|
|
182
|
+
limit: 1,
|
|
183
|
+
fields: ["stream"]
|
|
184
|
+
})).length > 0;
|
|
185
|
+
}
|
|
186
|
+
const [startState, endState] = await Promise.all([hs.rooms.stateAtSnapshot(startSnapshot), hs.rooms.stateAtSnapshot(endSnapshot)]);
|
|
157
187
|
const timelineIds = new Set(events.map((e) => e.pdu.event_id));
|
|
158
|
-
const endState = await hs.rooms.stateAtSnapshot(await snapshotAt(hs, roomId, upTo));
|
|
159
188
|
for (const [key, ev] of endState) {
|
|
160
189
|
if (timelineIds.has(ev.event_id) || startState.get(key)?.event_id === ev.event_id) continue;
|
|
161
190
|
if (events.some((e) => e.pdu.state_key !== void 0 && stateKey(e.pdu.type, e.pdu.state_key) === key)) continue;
|
|
162
191
|
startState.set(key, ev);
|
|
163
192
|
}
|
|
164
|
-
const sinceIds =
|
|
193
|
+
const sinceIds = await sinceIdsRead;
|
|
165
194
|
const sf = p.filter.room?.state;
|
|
166
195
|
const lazy = !!sf?.lazy_load_members;
|
|
167
196
|
const senders = new Set(events.map((e) => e.pdu.sender));
|
|
@@ -181,15 +210,12 @@ async function roomTimeline(hs, p, roomId, since, upTo) {
|
|
|
181
210
|
if (member && matchesFilter(sf, member)) state.push(member);
|
|
182
211
|
}
|
|
183
212
|
const stateAfter = [];
|
|
184
|
-
if (p.useStateAfter) {
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
if (lazy && type === M.ROOM.MEMBER && !senders.has(sk)) continue;
|
|
191
|
-
stateAfter.push(ev);
|
|
192
|
-
}
|
|
213
|
+
if (p.useStateAfter) for (const [key, ev] of endState) {
|
|
214
|
+
if (sinceIds[key] === ev.event_id) continue;
|
|
215
|
+
const [type, sk] = splitStateKey(key);
|
|
216
|
+
if (!matchesFilter(sf, ev)) continue;
|
|
217
|
+
if (lazy && type === M.ROOM.MEMBER && !senders.has(sk)) continue;
|
|
218
|
+
stateAfter.push(ev);
|
|
193
219
|
}
|
|
194
220
|
return {
|
|
195
221
|
stateAfter,
|
|
@@ -369,6 +395,23 @@ async function changesSince(hs, p, memberships, since, accountRows) {
|
|
|
369
395
|
memberChanges
|
|
370
396
|
};
|
|
371
397
|
}
|
|
398
|
+
/** Presence or device-list changes after a token read without a user filter, at most (#44). */
|
|
399
|
+
const RECENT_CHANGES = 500;
|
|
400
|
+
/** How many of a sync's joined rooms are built at once (#38). */
|
|
401
|
+
const JOINED_ROOMS_AT_ONCE = 8;
|
|
402
|
+
/** `fn` over `items` with at most `limit` running at once; results in input order. */
|
|
403
|
+
async function mapConcurrently(items, limit, fn) {
|
|
404
|
+
const out = [];
|
|
405
|
+
let next = 0;
|
|
406
|
+
const worker = async () => {
|
|
407
|
+
while (next < items.length) {
|
|
408
|
+
const i = next++;
|
|
409
|
+
out[i] = await fn(items[i]);
|
|
410
|
+
}
|
|
411
|
+
};
|
|
412
|
+
await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker));
|
|
413
|
+
return out;
|
|
414
|
+
}
|
|
372
415
|
async function buildSync(hs, p) {
|
|
373
416
|
const userId = p.session.userId;
|
|
374
417
|
const since = p.since?.pos ?? null;
|
|
@@ -380,7 +423,7 @@ async function buildSync(hs, p) {
|
|
|
380
423
|
field: "device_id",
|
|
381
424
|
value: p.session.deviceId
|
|
382
425
|
}];
|
|
383
|
-
const acknowledged = p.since ? hs.db.delete("to_device", [...deviceWhere, {
|
|
426
|
+
const acknowledged = p.since?.ack ? hs.db.delete("to_device", [...deviceWhere, {
|
|
384
427
|
field: "stream",
|
|
385
428
|
operator: "lte",
|
|
386
429
|
value: p.since.toDevice
|
|
@@ -469,39 +512,14 @@ async function buildSync(hs, p) {
|
|
|
469
512
|
const leave = {};
|
|
470
513
|
const knock = {};
|
|
471
514
|
let empty = true;
|
|
515
|
+
const joinedBuilds = [];
|
|
472
516
|
for (const m of memberships) {
|
|
473
517
|
if (m.stream > p.to || !roomAllowed(p.filter, m.room_id)) continue;
|
|
474
518
|
const roomId = m.room_id;
|
|
475
519
|
const changed = since === null || m.stream > since;
|
|
476
520
|
if (m.membership === "join") {
|
|
477
521
|
if (changes && !changed && !changes.touched.has(roomId) && hs.typing.typingSince(roomId, since) === null) continue;
|
|
478
|
-
|
|
479
|
-
const completedAt = await hs.partialState.completedAt(roomId);
|
|
480
|
-
const initial = since === null || p.fullState || m.stream > since || completedAt !== null && completedAt > since && completedAt <= p.to;
|
|
481
|
-
const { timeline, state, stateAfter, timelineEvents } = await roomTimeline(hs, p, roomId, initial ? null : since, p.to);
|
|
482
|
-
const ephemeral = await roomEphemeral(hs, p, roomId, changes ? changes.receipts.get(roomId) ?? [] : void 0);
|
|
483
|
-
const roomAccountData = await accountData(hs, p, roomId, changes ? changes.accountData.get(roomId) ?? [] : void 0);
|
|
484
|
-
if (!initial && timelineEvents.length === 0 && state.length === 0 && stateAfter.length === 0 && ephemeral.length === 0 && roomAccountData.length === 0) continue;
|
|
485
|
-
join[roomId] = {
|
|
486
|
-
timeline,
|
|
487
|
-
...p.useStateAfter ? { [p.stateAfterKey ?? "state_after"]: { events: stateAfter.map((e) => toClientEvent(e, { includeRoomId: false })) } } : { state: { events: state.map((e) => toClientEvent(e, { includeRoomId: false })) } },
|
|
488
|
-
ephemeral: { events: ephemeral },
|
|
489
|
-
account_data: { events: roomAccountData },
|
|
490
|
-
summary: await roomSummary(hs, roomId, userId)
|
|
491
|
-
};
|
|
492
|
-
const counts = await hs.push.unreadCounts(userId, roomId);
|
|
493
|
-
if (p.filter.room?.timeline?.unread_thread_notifications) {
|
|
494
|
-
join[roomId].unread_notifications = counts.room;
|
|
495
|
-
if (Object.keys(counts.threads).length) join[roomId].unread_thread_notifications = counts.threads;
|
|
496
|
-
} else {
|
|
497
|
-
const total = { ...counts.room };
|
|
498
|
-
for (const t of Object.values(counts.threads)) {
|
|
499
|
-
total.notification_count += t.notification_count;
|
|
500
|
-
total.highlight_count += t.highlight_count;
|
|
501
|
-
}
|
|
502
|
-
join[roomId].unread_notifications = total;
|
|
503
|
-
}
|
|
504
|
-
empty = false;
|
|
522
|
+
joinedBuilds.push(m);
|
|
505
523
|
} else if (m.membership === "invite") {
|
|
506
524
|
if (!changed && !p.fullState) continue;
|
|
507
525
|
const pdu = await hs.rooms.getPdu(m.event_id);
|
|
@@ -533,10 +551,49 @@ async function buildSync(hs, p) {
|
|
|
533
551
|
empty = false;
|
|
534
552
|
}
|
|
535
553
|
}
|
|
554
|
+
const joinedRoom = async (m) => {
|
|
555
|
+
const roomId = m.room_id;
|
|
556
|
+
if (!p.filter.room?.state?.lazy_load_members && await hs.partialState.isPartial(roomId)) return null;
|
|
557
|
+
const completedAt = await hs.partialState.completedAt(roomId);
|
|
558
|
+
const initial = since === null || p.fullState || m.stream > since || completedAt !== null && completedAt > since && completedAt <= p.to;
|
|
559
|
+
const [{ timeline, state, stateAfter, timelineEvents }, ephemeral, roomAccountData, summary, counts] = await Promise.all([
|
|
560
|
+
roomTimeline(hs, p, roomId, initial ? null : since, p.to),
|
|
561
|
+
roomEphemeral(hs, p, roomId, changes ? changes.receipts.get(roomId) ?? [] : void 0),
|
|
562
|
+
accountData(hs, p, roomId, changes ? changes.accountData.get(roomId) ?? [] : void 0),
|
|
563
|
+
roomSummary(hs, roomId, userId),
|
|
564
|
+
hs.push.unreadCounts(userId, roomId)
|
|
565
|
+
]);
|
|
566
|
+
if (!initial && timelineEvents.length === 0 && state.length === 0 && stateAfter.length === 0 && ephemeral.length === 0 && roomAccountData.length === 0) return null;
|
|
567
|
+
const room = {
|
|
568
|
+
timeline,
|
|
569
|
+
...p.useStateAfter ? { [p.stateAfterKey ?? "state_after"]: { events: stateAfter.map((e) => toClientEvent(e, { includeRoomId: false })) } } : { state: { events: state.map((e) => toClientEvent(e, { includeRoomId: false })) } },
|
|
570
|
+
ephemeral: { events: ephemeral },
|
|
571
|
+
account_data: { events: roomAccountData },
|
|
572
|
+
summary
|
|
573
|
+
};
|
|
574
|
+
if (p.filter.room?.timeline?.unread_thread_notifications) {
|
|
575
|
+
room.unread_notifications = counts.room;
|
|
576
|
+
if (Object.keys(counts.threads).length) room.unread_thread_notifications = counts.threads;
|
|
577
|
+
} else {
|
|
578
|
+
const total = { ...counts.room };
|
|
579
|
+
for (const t of Object.values(counts.threads)) {
|
|
580
|
+
total.notification_count += t.notification_count;
|
|
581
|
+
total.highlight_count += t.highlight_count;
|
|
582
|
+
}
|
|
583
|
+
room.unread_notifications = total;
|
|
584
|
+
}
|
|
585
|
+
return room;
|
|
586
|
+
};
|
|
587
|
+
const built = await mapConcurrently(joinedBuilds, JOINED_ROOMS_AT_ONCE, joinedRoom);
|
|
588
|
+
joinedBuilds.forEach((m, i) => {
|
|
589
|
+
if (!built[i]) return;
|
|
590
|
+
join[m.room_id] = built[i];
|
|
591
|
+
empty = false;
|
|
592
|
+
});
|
|
536
593
|
const globalAccountData = await accountData(hs, p, "", changes ? changes.accountData.get("") ?? [] : void 0);
|
|
537
594
|
if (since === null && !globalAccountData.some((e) => e.type === M.PUSH_RULES)) globalAccountData.push({
|
|
538
595
|
type: M.PUSH_RULES,
|
|
539
|
-
content: { global: await
|
|
596
|
+
content: { global: await readOnlyRules(hs, userId) }
|
|
540
597
|
});
|
|
541
598
|
if (globalAccountData.length) empty = false;
|
|
542
599
|
await acknowledged;
|
|
@@ -557,19 +614,38 @@ async function buildSync(hs, p) {
|
|
|
557
614
|
}
|
|
558
615
|
}
|
|
559
616
|
const sharingList = [...sharing];
|
|
560
|
-
const
|
|
561
|
-
|
|
562
|
-
|
|
563
|
-
|
|
564
|
-
field: "user_id",
|
|
565
|
-
operator: "in",
|
|
566
|
-
value: wantPresence ? sharingList : []
|
|
567
|
-
}, ...since !== null ? [{
|
|
617
|
+
const recent = (model) => ({
|
|
618
|
+
model,
|
|
619
|
+
opts: {
|
|
620
|
+
where: [{
|
|
568
621
|
field: "stream",
|
|
569
622
|
operator: "gt",
|
|
570
623
|
value: since
|
|
571
|
-
}]
|
|
572
|
-
|
|
624
|
+
}],
|
|
625
|
+
sortBy: {
|
|
626
|
+
field: "stream",
|
|
627
|
+
direction: "asc"
|
|
628
|
+
},
|
|
629
|
+
limit: 501
|
|
630
|
+
}
|
|
631
|
+
});
|
|
632
|
+
const presenceQuery = since === null ? {
|
|
633
|
+
model: "presence",
|
|
634
|
+
opts: { where: [{
|
|
635
|
+
field: "user_id",
|
|
636
|
+
operator: "in",
|
|
637
|
+
value: wantPresence ? sharingList : []
|
|
638
|
+
}] }
|
|
639
|
+
} : wantPresence ? recent("presence") : {
|
|
640
|
+
model: "presence",
|
|
641
|
+
opts: { where: [{
|
|
642
|
+
field: "user_id",
|
|
643
|
+
operator: "in",
|
|
644
|
+
value: []
|
|
645
|
+
}] }
|
|
646
|
+
};
|
|
647
|
+
let [presenceRows = [], newlySharedPresence = [], deviceRows] = await readBatch(hs.db, [
|
|
648
|
+
presenceQuery,
|
|
573
649
|
{
|
|
574
650
|
model: "presence",
|
|
575
651
|
opts: { where: [{
|
|
@@ -578,11 +654,21 @@ async function buildSync(hs, p) {
|
|
|
578
654
|
value: [...newlyShared]
|
|
579
655
|
}] }
|
|
580
656
|
},
|
|
581
|
-
...since !== null ? [
|
|
582
|
-
model: "device_list_changes",
|
|
583
|
-
opts: deviceListChangesQuery(sharing, since)
|
|
584
|
-
}] : []
|
|
657
|
+
...since !== null ? [recent("device_list_changes")] : []
|
|
585
658
|
]);
|
|
659
|
+
if (since !== null) {
|
|
660
|
+
if (presenceRows.length > RECENT_CHANGES) presenceRows = await hs.db.findMany("presence", { where: [{
|
|
661
|
+
field: "user_id",
|
|
662
|
+
operator: "in",
|
|
663
|
+
value: sharingList
|
|
664
|
+
}, {
|
|
665
|
+
field: "stream",
|
|
666
|
+
operator: "gt",
|
|
667
|
+
value: since
|
|
668
|
+
}] });
|
|
669
|
+
else presenceRows = presenceRows.filter((r) => sharing.has(r.user_id));
|
|
670
|
+
if (deviceRows.length > RECENT_CHANGES) deviceRows = await hs.db.findMany("device_list_changes", deviceListChangesQuery(sharing, since));
|
|
671
|
+
}
|
|
586
672
|
if (wantPresence) {
|
|
587
673
|
const seen = /* @__PURE__ */ new Set();
|
|
588
674
|
for (const row of [...presenceRows, ...newlySharedPresence]) {
|
|
@@ -609,7 +695,8 @@ async function buildSync(hs, p) {
|
|
|
609
695
|
const body = {
|
|
610
696
|
next_batch: formatToken({
|
|
611
697
|
pos: p.to,
|
|
612
|
-
toDevice: nextToDevice
|
|
698
|
+
toDevice: nextToDevice,
|
|
699
|
+
ack: toDevice.length > 0
|
|
613
700
|
}),
|
|
614
701
|
rooms: Object.fromEntries(Object.entries({
|
|
615
702
|
join,
|
package/dist/core/push.js
CHANGED
|
@@ -3,8 +3,9 @@ import { M, UNSTABLE } from "./m.js";
|
|
|
3
3
|
import { splitStateKey, toClientEvent } from "./events.js";
|
|
4
4
|
import { getState, powerLevels, userLevel } from "./authrules.js";
|
|
5
5
|
import { globMatch, globSearchWords } from "./glob.js";
|
|
6
|
-
import { KINDS,
|
|
6
|
+
import { KINDS, readOnlyRules } from "./pushrules.js";
|
|
7
7
|
import { isSubscribed } from "../api/threadsubs.js";
|
|
8
|
+
import { upsertMany } from "../storage/batch.js";
|
|
8
9
|
//#region src/core/push.ts
|
|
9
10
|
/** Look up a dotted path; `\.` escapes a literal dot in a key (e.g. `content.m\.relates_to`). */
|
|
10
11
|
function valueAtPath(obj, path) {
|
|
@@ -109,7 +110,7 @@ var PushService = class {
|
|
|
109
110
|
const notified = [];
|
|
110
111
|
for (const userId of members) {
|
|
111
112
|
if (userId === event.sender || !this.hs.isLocalUser(userId)) continue;
|
|
112
|
-
const actions = evaluateRules(await
|
|
113
|
+
const actions = evaluateRules(await readOnlyRules(this.hs, userId), {
|
|
113
114
|
event,
|
|
114
115
|
userId,
|
|
115
116
|
displayName: getState(state, M.ROOM.MEMBER, userId)?.content?.displayname ?? null,
|
|
@@ -136,9 +137,7 @@ var PushService = class {
|
|
|
136
137
|
});
|
|
137
138
|
}
|
|
138
139
|
if (notified.length === 0) return;
|
|
139
|
-
await this.hs.transaction(
|
|
140
|
-
for (const n of notified) await db.upsert("notifications", n.row);
|
|
141
|
-
});
|
|
140
|
+
await this.hs.transaction((db) => upsertMany(db, "notifications", notified.map((n) => n.row)));
|
|
142
141
|
const pushers = await this.hs.db.findMany("pushers", { where: [{
|
|
143
142
|
field: "user_id",
|
|
144
143
|
operator: "in",
|
|
@@ -179,21 +178,28 @@ var PushService = class {
|
|
|
179
178
|
async unreadCounts(userId, roomId) {
|
|
180
179
|
const read = await this.readPositions(userId, roomId);
|
|
181
180
|
const unthreaded = read.unthreaded;
|
|
182
|
-
const rows = await this.hs.db.findMany("notifications", {
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
181
|
+
const rows = await this.hs.db.findMany("notifications", {
|
|
182
|
+
where: [
|
|
183
|
+
{
|
|
184
|
+
field: "user_id",
|
|
185
|
+
value: userId
|
|
186
|
+
},
|
|
187
|
+
{
|
|
188
|
+
field: "room_id",
|
|
189
|
+
value: roomId
|
|
190
|
+
},
|
|
191
|
+
{
|
|
192
|
+
field: "stream",
|
|
193
|
+
operator: "gt",
|
|
194
|
+
value: unthreaded
|
|
195
|
+
}
|
|
196
|
+
],
|
|
197
|
+
fields: [
|
|
198
|
+
"stream",
|
|
199
|
+
"thread_id",
|
|
200
|
+
"highlight"
|
|
201
|
+
]
|
|
202
|
+
});
|
|
197
203
|
const room = {
|
|
198
204
|
notification_count: 0,
|
|
199
205
|
highlight_count: 0
|
package/dist/core/pushrules.js
CHANGED
|
@@ -131,14 +131,33 @@ function cacheFor(hs) {
|
|
|
131
131
|
}
|
|
132
132
|
return c;
|
|
133
133
|
}
|
|
134
|
+
/** A user's push rules, as a copy the caller may change. */
|
|
134
135
|
async function loadRules(hs, userId) {
|
|
136
|
+
return structuredClone(await readOnlyRules(hs, userId));
|
|
137
|
+
}
|
|
138
|
+
/**
|
|
139
|
+
* A user's push rules, shared and frozen: for callers that only read them.
|
|
140
|
+
*
|
|
141
|
+
* https://github.com/k18k/matrix-nodejs-homeserver/issues/41: push evaluation copied every
|
|
142
|
+
* member's rules for every message, about 6% of the CPU under load, though it only reads them.
|
|
143
|
+
* The cached rulesets are deep-frozen, so a caller that tries to change one fails loudly instead
|
|
144
|
+
* of changing everyone's.
|
|
145
|
+
*/
|
|
146
|
+
async function readOnlyRules(hs, userId) {
|
|
135
147
|
const cached = cacheFor(hs).get(userId);
|
|
136
|
-
if (cached) return
|
|
137
|
-
const rules = await readRules(hs, userId);
|
|
148
|
+
if (cached) return cached;
|
|
149
|
+
const rules = deepFreeze(await readRules(hs, userId));
|
|
138
150
|
const cache = cacheFor(hs);
|
|
139
151
|
if (cache.size > 5e3) cache.clear();
|
|
140
152
|
cache.set(userId, rules);
|
|
141
|
-
return
|
|
153
|
+
return rules;
|
|
154
|
+
}
|
|
155
|
+
function deepFreeze(value) {
|
|
156
|
+
if (value && typeof value === "object" && !Object.isFrozen(value)) {
|
|
157
|
+
Object.freeze(value);
|
|
158
|
+
for (const v of Object.values(value)) deepFreeze(v);
|
|
159
|
+
}
|
|
160
|
+
return value;
|
|
142
161
|
}
|
|
143
162
|
async function readRules(hs, userId) {
|
|
144
163
|
const row = await hs.db.findOne("account_data", [
|
|
@@ -162,7 +181,7 @@ async function readRules(hs, userId) {
|
|
|
162
181
|
return rules;
|
|
163
182
|
}
|
|
164
183
|
async function saveRules(hs, userId, rules) {
|
|
165
|
-
cacheFor(hs).set(userId, structuredClone(rules));
|
|
184
|
+
cacheFor(hs).set(userId, deepFreeze(structuredClone(rules)));
|
|
166
185
|
hs.cluster.publish("pushrules", {
|
|
167
186
|
from: hs.instanceId,
|
|
168
187
|
userId
|
|
@@ -226,4 +245,4 @@ async function copyRules(hs, userId, oldRoomId, newRoomId) {
|
|
|
226
245
|
if (changed) await saveRules(hs, userId, rules);
|
|
227
246
|
}
|
|
228
247
|
//#endregion
|
|
229
|
-
export { KINDS, copyRoomPushRules, defaultRules, loadRules, saveRules, withRulesLock };
|
|
248
|
+
export { KINDS, copyRoomPushRules, defaultRules, loadRules, readOnlyRules, saveRules, withRulesLock };
|
package/dist/homeserver.js
CHANGED
|
@@ -13,6 +13,7 @@ import { KeyService } from "./core/keys.js";
|
|
|
13
13
|
import { Notifier } from "./core/notifier.js";
|
|
14
14
|
import { StreamManager, localCluster, newInstanceId } from "./core/cluster.js";
|
|
15
15
|
import { TypingService } from "./core/typing.js";
|
|
16
|
+
import { coalesceReads } from "./storage/batch.js";
|
|
16
17
|
import { PushService } from "./core/push.js";
|
|
17
18
|
import { SsoService } from "./core/sso.js";
|
|
18
19
|
import { AdminService } from "./core/admin.js";
|
|
@@ -125,7 +126,7 @@ var Homeserver = class extends EventEmitter {
|
|
|
125
126
|
}
|
|
126
127
|
};
|
|
127
128
|
const db = options.databaseAdapter;
|
|
128
|
-
if (db && typeof db.findOne === "function") this.db = db;
|
|
129
|
+
if (db && typeof db.findOne === "function") this.db = coalesceReads(db);
|
|
129
130
|
else {
|
|
130
131
|
this.logger.warn("No database adapter given; using in-memory storage (data is lost on restart)");
|
|
131
132
|
this.db = memoryAdapter();
|
package/dist/storage/batch.js
CHANGED
|
@@ -7,5 +7,76 @@ function readBatch(db, queries) {
|
|
|
7
7
|
if (db.findManyBatch) return db.findManyBatch(queries);
|
|
8
8
|
return Promise.all(queries.map((query) => db.findMany(query.model, query.opts)));
|
|
9
9
|
}
|
|
10
|
+
/** Upsert several rows: in a few statements when the adapter can (`upsertMany`), otherwise one by one. */
|
|
11
|
+
async function upsertMany(db, model, rows) {
|
|
12
|
+
if (rows.length === 0) return;
|
|
13
|
+
if (db.upsertMany) return db.upsertMany(model, rows);
|
|
14
|
+
for (const row of rows) await db.upsert(model, row);
|
|
15
|
+
}
|
|
16
|
+
/** At most this many reads go into one coalesced statement. */
|
|
17
|
+
const MAX_COALESCED = 32;
|
|
18
|
+
/**
|
|
19
|
+
* An adapter whose reads (`findMany`, `findOne`) issued within one turn of the event loop are
|
|
20
|
+
* sent together, through `findManyBatch`. Writes and transactions go straight to `db`. Adapters
|
|
21
|
+
* without `findManyBatch` are returned as they are.
|
|
22
|
+
*
|
|
23
|
+
* https://github.com/k18k/matrix-nodejs-homeserver/issues/43: under load, most of the homeserver's
|
|
24
|
+
* and the database's CPU went to the per-statement cost (a socket write, a round trip, parsing)
|
|
25
|
+
* of many small reads issued at the same moment by concurrent requests. Nothing is kept between
|
|
26
|
+
* turns: this merges requests, it does not cache results. A read waits for at most the rest of
|
|
27
|
+
* the current turn. If a coalesced statement fails, its reads are run one by one, so a bad read
|
|
28
|
+
* fails alone.
|
|
29
|
+
*/
|
|
30
|
+
function coalesceReads(db) {
|
|
31
|
+
if (!db.findManyBatch) return db;
|
|
32
|
+
let queue = [];
|
|
33
|
+
const runAlone = (read) => db.findMany(read.query.model, read.query.opts).then(read.resolve, read.reject);
|
|
34
|
+
const run = async (reads) => {
|
|
35
|
+
if (reads.length === 1) return runAlone(reads[0]);
|
|
36
|
+
let results;
|
|
37
|
+
try {
|
|
38
|
+
results = await db.findManyBatch(reads.map((r) => r.query), { prepare: false });
|
|
39
|
+
} catch {
|
|
40
|
+
for (const read of reads) runAlone(read);
|
|
41
|
+
return;
|
|
42
|
+
}
|
|
43
|
+
reads.forEach((read, i) => read.resolve(results[i]));
|
|
44
|
+
};
|
|
45
|
+
const flush = () => {
|
|
46
|
+
const reads = queue;
|
|
47
|
+
queue = [];
|
|
48
|
+
for (let i = 0; i < reads.length; i += MAX_COALESCED) run(reads.slice(i, i + MAX_COALESCED));
|
|
49
|
+
};
|
|
50
|
+
const enqueue = (query) => new Promise((resolve, reject) => {
|
|
51
|
+
if (queue.length === 0) setImmediate(flush);
|
|
52
|
+
queue.push({
|
|
53
|
+
query,
|
|
54
|
+
resolve,
|
|
55
|
+
reject
|
|
56
|
+
});
|
|
57
|
+
});
|
|
58
|
+
return {
|
|
59
|
+
findMany: (model, opts) => enqueue({
|
|
60
|
+
model,
|
|
61
|
+
opts
|
|
62
|
+
}),
|
|
63
|
+
findOne: async (model, where) => (await enqueue({
|
|
64
|
+
model,
|
|
65
|
+
opts: {
|
|
66
|
+
where,
|
|
67
|
+
limit: 1
|
|
68
|
+
}
|
|
69
|
+
}))[0] ?? null,
|
|
70
|
+
findManyBatch: (queries, options) => db.findManyBatch(queries, options),
|
|
71
|
+
create: (model, data) => db.create(model, data),
|
|
72
|
+
upsert: (model, data) => db.upsert(model, data),
|
|
73
|
+
...db.upsertMany ? { upsertMany: (model, rows) => db.upsertMany(model, rows) } : {},
|
|
74
|
+
update: (model, where, data) => db.update(model, where, data),
|
|
75
|
+
delete: (model, where) => db.delete(model, where),
|
|
76
|
+
max: (model, field) => db.max(model, field),
|
|
77
|
+
...db.migrate ? { migrate: (schema) => db.migrate(schema) } : {},
|
|
78
|
+
...db.transaction ? { transaction: (fn) => db.transaction(fn) } : {}
|
|
79
|
+
};
|
|
80
|
+
}
|
|
10
81
|
//#endregion
|
|
11
|
-
export { readBatch };
|
|
82
|
+
export { coalesceReads, readBatch, upsertMany };
|
package/dist/storage/schema.js
CHANGED
|
@@ -192,7 +192,11 @@ const schema = {
|
|
|
192
192
|
ts: n
|
|
193
193
|
},
|
|
194
194
|
primaryKey: ["user_id", "event_id"],
|
|
195
|
-
indexes: [[
|
|
195
|
+
indexes: [[
|
|
196
|
+
"user_id",
|
|
197
|
+
"room_id",
|
|
198
|
+
"stream"
|
|
199
|
+
], ["user_id", "stream"]]
|
|
196
200
|
},
|
|
197
201
|
sso_identities: {
|
|
198
202
|
fields: {
|
|
@@ -397,7 +401,8 @@ const schema = {
|
|
|
397
401
|
last_active_ts: n,
|
|
398
402
|
stream: n
|
|
399
403
|
},
|
|
400
|
-
primaryKey: ["user_id"]
|
|
404
|
+
primaryKey: ["user_id"],
|
|
405
|
+
indexes: [["stream"]]
|
|
401
406
|
},
|
|
402
407
|
media: {
|
|
403
408
|
fields: {
|
package/dist/storage/sql.d.ts
CHANGED
|
@@ -2,8 +2,10 @@ import { DatabaseAdapter, Row } from "./types.js";
|
|
|
2
2
|
//#region src/storage/sql.d.ts
|
|
3
3
|
export type SqlDialect = "sqlite" | "postgres" | "mysql";
|
|
4
4
|
export interface SqlExecutor {
|
|
5
|
-
/** Run a query that returns rows. */
|
|
6
|
-
query(sql: string, params: unknown[]
|
|
5
|
+
/** Run a query that returns rows; `prepare: false` when the statement is not worth preparing for reuse. */
|
|
6
|
+
query(sql: string, params: unknown[], options?: {
|
|
7
|
+
prepare?: boolean;
|
|
8
|
+
}): Promise<Row[]>;
|
|
7
9
|
/** Run a statement; resolve with the number of affected rows. */
|
|
8
10
|
execute(sql: string, params: unknown[]): Promise<number>;
|
|
9
11
|
}
|
|
@@ -36,9 +38,20 @@ export interface SqliteDatabaseLike {
|
|
|
36
38
|
/** SQLite adapter for `node:sqlite` (`new DatabaseSync(path)`) or `better-sqlite3`. */
|
|
37
39
|
export declare function sqliteAdapter(db: SqliteDatabaseLike, options?: {
|
|
38
40
|
tablePrefix?: string;
|
|
41
|
+
/**
|
|
42
|
+
* When commits reach the disk (SQLite's `synchronous`). `"normal"` (the default): at
|
|
43
|
+
* checkpoints; a commit survives a crash of the process, and a power loss or operating
|
|
44
|
+
* system crash can roll back the most recent ones, without corrupting the database.
|
|
45
|
+
* `"full"`: on every commit, which blocks the event loop for an fsync each time.
|
|
46
|
+
*/
|
|
47
|
+
synchronous?: "normal" | "full";
|
|
39
48
|
}): DatabaseAdapter;
|
|
40
49
|
interface PgQueryable {
|
|
41
|
-
query(
|
|
50
|
+
query(config: {
|
|
51
|
+
text: string;
|
|
52
|
+
values?: unknown[];
|
|
53
|
+
name?: string;
|
|
54
|
+
}): Promise<{
|
|
42
55
|
rows: Row[];
|
|
43
56
|
rowCount: number | null;
|
|
44
57
|
}>;
|
|
@@ -53,6 +66,12 @@ export interface PgPoolLike extends PgQueryable {
|
|
|
53
66
|
/** PostgreSQL adapter for `pg` (node-postgres). Pass a `Pool` to get transactions. */
|
|
54
67
|
export declare function postgresAdapter(pool: PgPoolLike, options?: {
|
|
55
68
|
tablePrefix?: string;
|
|
69
|
+
/**
|
|
70
|
+
* Prepare statements on each connection and reuse them (default true). Turn off behind a
|
|
71
|
+
* connection pooler that does not support prepared statements, such as PgBouncer before 1.21
|
|
72
|
+
* or in transaction mode without `max_prepared_statements`.
|
|
73
|
+
*/
|
|
74
|
+
preparedStatements?: boolean;
|
|
56
75
|
}): DatabaseAdapter;
|
|
57
76
|
interface MysqlQueryable {
|
|
58
77
|
query(sql: string, values?: unknown[]): Promise<[any, unknown]>;
|
package/dist/storage/sql.js
CHANGED
|
@@ -1,5 +1,8 @@
|
|
|
1
|
+
import { createHash } from "node:crypto";
|
|
1
2
|
import { AsyncLocalStorage } from "node:async_hooks";
|
|
2
3
|
//#region src/storage/sql.ts
|
|
4
|
+
/** Parameters per statement for multi-row writes: within SQLite's (32766), Postgres' and MySQL's (65535) limits. */
|
|
5
|
+
const MAX_PARAMS = 4e3;
|
|
3
6
|
const OPS = {
|
|
4
7
|
eq: "=",
|
|
5
8
|
ne: "<>",
|
|
@@ -60,6 +63,12 @@ function sqlAdapter({ dialect, executor: rootExecutor, tablePrefix = "", transac
|
|
|
60
63
|
return out;
|
|
61
64
|
};
|
|
62
65
|
const placeholder = (i) => dialect === "postgres" ? `$${i}` : "?";
|
|
66
|
+
const limitClause = (limit, params) => {
|
|
67
|
+
const n = Math.max(0, Math.floor(limit));
|
|
68
|
+
if (dialect === "mysql") return ` LIMIT ${n}`;
|
|
69
|
+
params.push(n);
|
|
70
|
+
return ` LIMIT ${placeholder(params.length)}`;
|
|
71
|
+
};
|
|
63
72
|
const buildWhere = (model, where = [], params) => {
|
|
64
73
|
if (where.length === 0) return "";
|
|
65
74
|
return ` WHERE ${where.map((w) => {
|
|
@@ -67,6 +76,10 @@ function sqlAdapter({ dialect, executor: rootExecutor, tablePrefix = "", transac
|
|
|
67
76
|
if (op === "in") {
|
|
68
77
|
const values = w.value;
|
|
69
78
|
if (values.length === 0) return "1=0";
|
|
79
|
+
if (dialect === "postgres") {
|
|
80
|
+
params.push(values.map((v) => toDb(model, w.field, v)));
|
|
81
|
+
return `${q(w.field)} = ANY(${placeholder(params.length)})`;
|
|
82
|
+
}
|
|
70
83
|
const ph = values.map((v) => {
|
|
71
84
|
params.push(toDb(model, w.field, v));
|
|
72
85
|
return placeholder(params.length);
|
|
@@ -83,11 +96,15 @@ function sqlAdapter({ dialect, executor: rootExecutor, tablePrefix = "", transac
|
|
|
83
96
|
};
|
|
84
97
|
const build = (executor, inTx) => {
|
|
85
98
|
const insert = async (model, data, mode) => {
|
|
86
|
-
const
|
|
87
|
-
const
|
|
99
|
+
const rows = Array.isArray(data) ? data : [data];
|
|
100
|
+
const fields = Object.keys(rows[0]).filter((k) => rows[0][k] !== void 0);
|
|
101
|
+
const params = [];
|
|
102
|
+
const values = rows.map((row) => `(${fields.map((f) => {
|
|
103
|
+
params.push(toDb(model, f, row[f]));
|
|
104
|
+
return placeholder(params.length);
|
|
105
|
+
}).join(", ")})`);
|
|
88
106
|
const cols = fields.map(q).join(", ");
|
|
89
|
-
|
|
90
|
-
let sql = `INSERT INTO ${t(model)} (${cols}) VALUES (${ph})`;
|
|
107
|
+
let sql = `INSERT INTO ${t(model)} (${cols}) VALUES ${values.join(", ")}`;
|
|
91
108
|
if (mode === "upsert") {
|
|
92
109
|
const pk = schema[model]?.primaryKey ?? [];
|
|
93
110
|
const rest = fields.filter((f) => !pk.includes(f));
|
|
@@ -123,6 +140,20 @@ function sqlAdapter({ dialect, executor: rootExecutor, tablePrefix = "", transac
|
|
|
123
140
|
},
|
|
124
141
|
create: (model, data) => insert(model, data, "insert"),
|
|
125
142
|
upsert: (model, data) => insert(model, data, "upsert"),
|
|
143
|
+
async upsertMany(model, rows) {
|
|
144
|
+
const pk = schema[model]?.primaryKey ?? [];
|
|
145
|
+
const byKey = /* @__PURE__ */ new Map();
|
|
146
|
+
for (const row of rows) byKey.set(JSON.stringify(pk.map((f) => row[f])), row);
|
|
147
|
+
const groups = /* @__PURE__ */ new Map();
|
|
148
|
+
for (const row of byKey.values()) {
|
|
149
|
+
const shape = Object.keys(row).filter((k) => row[k] !== void 0).join("\0");
|
|
150
|
+
groups.set(shape, [...groups.get(shape) ?? [], row]);
|
|
151
|
+
}
|
|
152
|
+
for (const group of groups.values()) {
|
|
153
|
+
const perStatement = Math.max(1, Math.floor(MAX_PARAMS / Object.keys(group[0]).length));
|
|
154
|
+
for (let i = 0; i < group.length; i += perStatement) await insert(model, group.slice(i, i + perStatement), "upsert");
|
|
155
|
+
}
|
|
156
|
+
},
|
|
126
157
|
async findOne(model, where) {
|
|
127
158
|
const params = [];
|
|
128
159
|
const rows = await executor.query(`SELECT * FROM ${t(model)}${buildWhere(model, where, params)} LIMIT 1`, params);
|
|
@@ -132,25 +163,26 @@ function sqlAdapter({ dialect, executor: rootExecutor, tablePrefix = "", transac
|
|
|
132
163
|
const params = [];
|
|
133
164
|
let sql = `SELECT ${opts.fields?.length ? opts.fields.map(q).join(", ") : "*"} FROM ${t(model)}${buildWhere(model, opts.where, params)}`;
|
|
134
165
|
if (opts.sortBy) sql += ` ORDER BY ${q(opts.sortBy.field)} ${opts.sortBy.direction === "asc" ? "ASC" : "DESC"}`;
|
|
135
|
-
if (opts.limit !== void 0) sql +=
|
|
166
|
+
if (opts.limit !== void 0) sql += limitClause(opts.limit, params);
|
|
136
167
|
return (await executor.query(sql, params)).map((r) => fromDb(model, r));
|
|
137
168
|
},
|
|
138
|
-
...dialect === "postgres" ? { async findManyBatch(queries) {
|
|
169
|
+
...dialect === "postgres" ? { async findManyBatch(queries, options) {
|
|
139
170
|
if (queries.length === 0) return [];
|
|
140
171
|
const params = [];
|
|
141
172
|
const parts = queries.map(({ model, opts = {} }, i) => {
|
|
142
173
|
const fields = opts.fields?.length && opts.sortBy && !opts.fields.includes(opts.sortBy.field) ? [...opts.fields, opts.sortBy.field] : opts.fields;
|
|
143
174
|
let inner = `SELECT ${fields?.length ? fields.map(q).join(", ") : "*"} FROM ${t(model)}${buildWhere(model, opts.where, params)}`;
|
|
144
|
-
const
|
|
145
|
-
if (
|
|
146
|
-
if (opts.limit !== void 0) inner +=
|
|
147
|
-
return `SELECT ${i} AS i,
|
|
175
|
+
const direction = opts.sortBy?.direction === "asc" ? "ASC" : "DESC";
|
|
176
|
+
if (opts.sortBy) inner += ` ORDER BY ${q(opts.sortBy.field)} ${direction}`;
|
|
177
|
+
if (opts.limit !== void 0) inner += limitClause(opts.limit, params);
|
|
178
|
+
return `SELECT ${i} AS i, (SELECT json_agg(b${opts.sortBy ? ` ORDER BY b.${q(opts.sortBy.field)} ${direction}` : ""}) FROM (${inner}) b) AS r`;
|
|
148
179
|
});
|
|
149
|
-
const rows = await executor.query(
|
|
180
|
+
const rows = await executor.query(parts.join(" UNION ALL "), params, options);
|
|
150
181
|
const out = queries.map(() => []);
|
|
151
182
|
for (const row of rows) {
|
|
152
|
-
const
|
|
153
|
-
|
|
183
|
+
const list = typeof row.r === "string" ? JSON.parse(row.r) : row.r;
|
|
184
|
+
const model = queries[Number(row.i)].model;
|
|
185
|
+
out[Number(row.i)] = (list ?? []).map((r) => fromDb(model, r));
|
|
154
186
|
}
|
|
155
187
|
return out;
|
|
156
188
|
} } : {},
|
|
@@ -175,21 +207,36 @@ function sqlAdapter({ dialect, executor: rootExecutor, tablePrefix = "", transac
|
|
|
175
207
|
};
|
|
176
208
|
return build(rootExecutor, false);
|
|
177
209
|
}
|
|
210
|
+
/** How long SQLite statements may run back to back before yielding to the event loop (#47). */
|
|
211
|
+
const YIELD_AFTER_MS = 10;
|
|
212
|
+
/** Prepared statements an SQLite adapter keeps (#40). */
|
|
213
|
+
const STATEMENT_CACHE = 500;
|
|
178
214
|
/** SQLite adapter for `node:sqlite` (`new DatabaseSync(path)`) or `better-sqlite3`. */
|
|
179
215
|
function sqliteAdapter(db, options = {}) {
|
|
180
216
|
try {
|
|
181
217
|
db.exec("PRAGMA journal_mode = WAL");
|
|
218
|
+
db.exec(`PRAGMA synchronous = ${options.synchronous === "full" ? "FULL" : "NORMAL"}`);
|
|
182
219
|
} catch {}
|
|
183
220
|
const cache = /* @__PURE__ */ new Map();
|
|
184
221
|
const stmt = (sql) => {
|
|
185
222
|
let s = cache.get(sql);
|
|
186
|
-
if (
|
|
223
|
+
if (s) cache.delete(sql);
|
|
224
|
+
else s = db.prepare(sql);
|
|
225
|
+
cache.set(sql, s);
|
|
226
|
+
if (cache.size > STATEMENT_CACHE) cache.delete(cache.keys().next().value);
|
|
187
227
|
return s;
|
|
188
228
|
};
|
|
189
229
|
const inTransaction = new AsyncLocalStorage();
|
|
190
230
|
let gate = Promise.resolve();
|
|
231
|
+
let lastYield = performance.now();
|
|
191
232
|
const waitGate = async () => {
|
|
192
|
-
if (
|
|
233
|
+
if (inTransaction.getStore()) return;
|
|
234
|
+
await gate;
|
|
235
|
+
if (performance.now() - lastYield > YIELD_AFTER_MS) {
|
|
236
|
+
await new Promise((resolve) => setImmediate(resolve));
|
|
237
|
+
lastYield = performance.now();
|
|
238
|
+
await gate;
|
|
239
|
+
}
|
|
193
240
|
};
|
|
194
241
|
const executor = {
|
|
195
242
|
async query(sql, params) {
|
|
@@ -229,29 +276,80 @@ function sqliteAdapter(db, options = {}) {
|
|
|
229
276
|
}
|
|
230
277
|
});
|
|
231
278
|
}
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
279
|
+
/**
|
|
280
|
+
* https://github.com/k18k/matrix-nodejs-homeserver/issues/40: unnamed statements are parsed and
|
|
281
|
+
* planned by Postgres on every call, which took most of its CPU under load. Named ones are prepared
|
|
282
|
+
* once per connection and reused. Statement texts depend only on the code (lists and limits are
|
|
283
|
+
* parameters), so there are a few hundred of them; past `MAX_PREPARED` distinct texts, further
|
|
284
|
+
* ones run unnamed, which bounds the memory both node-postgres and Postgres keep per connection.
|
|
285
|
+
*/
|
|
286
|
+
const MAX_PREPARED = 2e3;
|
|
287
|
+
function statementNamer(enabled) {
|
|
288
|
+
let names = /* @__PURE__ */ new Map();
|
|
289
|
+
let generation = 0;
|
|
290
|
+
return {
|
|
291
|
+
name(sql) {
|
|
292
|
+
if (!enabled) return void 0;
|
|
293
|
+
let name = names.get(sql);
|
|
294
|
+
if (!name && names.size < MAX_PREPARED) names.set(sql, name = `m${generation}_${createHash("sha1").update(sql).digest("base64url")}`);
|
|
295
|
+
return name;
|
|
296
|
+
},
|
|
297
|
+
reset() {
|
|
298
|
+
names = /* @__PURE__ */ new Map();
|
|
299
|
+
generation++;
|
|
300
|
+
}
|
|
301
|
+
};
|
|
302
|
+
}
|
|
303
|
+
/**
|
|
304
|
+
* A prepared `SELECT *` fails once its table has gained a column, for example when another node of
|
|
305
|
+
* a cluster migrated the schema during a rolling update. Then the statements are prepared again
|
|
306
|
+
* under new names and the query is retried once.
|
|
307
|
+
*/
|
|
308
|
+
const isStalePlan = (err) => err?.code === "0A000" && /cached plan/.test(String(err.message));
|
|
309
|
+
const pgExecutor = (conn, namer) => {
|
|
310
|
+
const run = async (sql, params, prepare = true) => {
|
|
311
|
+
const name = () => prepare ? namer.name(sql) : void 0;
|
|
312
|
+
try {
|
|
313
|
+
return await conn.query({
|
|
314
|
+
text: sql,
|
|
315
|
+
values: params,
|
|
316
|
+
name: name()
|
|
317
|
+
});
|
|
318
|
+
} catch (err) {
|
|
319
|
+
if (!isStalePlan(err)) throw err;
|
|
320
|
+
namer.reset();
|
|
321
|
+
return conn.query({
|
|
322
|
+
text: sql,
|
|
323
|
+
values: params,
|
|
324
|
+
name: name()
|
|
325
|
+
});
|
|
326
|
+
}
|
|
327
|
+
};
|
|
328
|
+
return {
|
|
329
|
+
async query(sql, params, options) {
|
|
330
|
+
return (await run(sql, params, options?.prepare)).rows;
|
|
331
|
+
},
|
|
332
|
+
async execute(sql, params) {
|
|
333
|
+
return (await run(sql, params)).rowCount ?? 0;
|
|
334
|
+
}
|
|
335
|
+
};
|
|
336
|
+
};
|
|
240
337
|
/** PostgreSQL adapter for `pg` (node-postgres). Pass a `Pool` to get transactions. */
|
|
241
338
|
function postgresAdapter(pool, options = {}) {
|
|
339
|
+
const namer = statementNamer(options.preparedStatements ?? true);
|
|
242
340
|
return sqlAdapter({
|
|
243
341
|
dialect: "postgres",
|
|
244
342
|
tablePrefix: options.tablePrefix,
|
|
245
|
-
executor: pgExecutor(pool),
|
|
343
|
+
executor: pgExecutor(pool, namer),
|
|
246
344
|
transaction: pool.connect ? async (fn) => {
|
|
247
345
|
const client = await pool.connect();
|
|
248
346
|
try {
|
|
249
|
-
await client.query("BEGIN");
|
|
250
|
-
const result = await fn(pgExecutor(client));
|
|
251
|
-
await client.query("COMMIT");
|
|
347
|
+
await client.query({ text: "BEGIN" });
|
|
348
|
+
const result = await fn(pgExecutor(client, namer));
|
|
349
|
+
await client.query({ text: "COMMIT" });
|
|
252
350
|
return result;
|
|
253
351
|
} catch (err) {
|
|
254
|
-
await client.query("ROLLBACK").catch(() => {});
|
|
352
|
+
await client.query({ text: "ROLLBACK" }).catch(() => {});
|
|
255
353
|
throw err;
|
|
256
354
|
} finally {
|
|
257
355
|
client.release();
|
package/dist/storage/types.d.ts
CHANGED
|
@@ -21,6 +21,14 @@ export interface WhereClause {
|
|
|
21
21
|
operator?: Operator;
|
|
22
22
|
}
|
|
23
23
|
export type Row = Record<string, any>;
|
|
24
|
+
/** How a `findManyBatch` is run. */
|
|
25
|
+
export interface BatchOptions {
|
|
26
|
+
/**
|
|
27
|
+
* The combination of reads is a one-off (as when unrelated reads are coalesced), so preparing
|
|
28
|
+
* the statement for reuse would only fill the database's statement cache. Default true.
|
|
29
|
+
*/
|
|
30
|
+
prepare?: boolean;
|
|
31
|
+
}
|
|
24
32
|
/** One read of a `findManyBatch`. */
|
|
25
33
|
export interface BatchQuery {
|
|
26
34
|
model: string;
|
|
@@ -50,12 +58,18 @@ export interface DatabaseAdapter {
|
|
|
50
58
|
* an adapter that talks to a database server can send them in one round trip. Without it,
|
|
51
59
|
* callers run the reads separately (see `readBatch`).
|
|
52
60
|
*/
|
|
53
|
-
findManyBatch?(queries: BatchQuery[]): Promise<Row[][]>;
|
|
61
|
+
findManyBatch?(queries: BatchQuery[], options?: BatchOptions): Promise<Row[][]>;
|
|
54
62
|
/** Create tables / collections for the given schema. Must be idempotent. */
|
|
55
63
|
migrate?(schema: Schema): Promise<void>;
|
|
56
64
|
create(model: string, data: Row): Promise<void>;
|
|
57
65
|
/** Insert or replace by primary key. */
|
|
58
66
|
upsert(model: string, data: Row): Promise<void>;
|
|
67
|
+
/**
|
|
68
|
+
* `upsert` for several rows at once (the last row wins when two share a primary key). Optional;
|
|
69
|
+
* an adapter that talks to a database server can write them in a few statements. Without it,
|
|
70
|
+
* callers upsert the rows one by one (see `upsertMany`).
|
|
71
|
+
*/
|
|
72
|
+
upsertMany?(model: string, rows: Row[]): Promise<void>;
|
|
59
73
|
findOne(model: string, where: WhereClause[]): Promise<Row | null>;
|
|
60
74
|
findMany(model: string, options?: FindManyOptions): Promise<Row[]>;
|
|
61
75
|
update(model: string, where: WhereClause[], data: Row): Promise<number>;
|