matrix-nodejs-homeserver 0.3.2 → 0.3.4

package/README.md

@@ -1,9 +1,10 @@
 # matrix-nodejs-homeserver
 
+[![Complement](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fk18k%2Fmatrix-nodejs-homeserver%2Fcomplement-results%2Fbadges%2Fcomplement.json)](https://github.com/k18k/matrix-nodejs-homeserver/actions/workflows/ci.yml)
 [![Load test: SQLite](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fk18k%2Fmatrix-nodejs-homeserver%2Floadtest-results%2Fbadges%2Fsqlite.json)](https://github.com/k18k/matrix-nodejs-homeserver/actions/workflows/loadtest.yml)
 [![Load test: Postgres](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fk18k%2Fmatrix-nodejs-homeserver%2Floadtest-results%2Fbadges%2Fpg-nocluster.json)](https://github.com/k18k/matrix-nodejs-homeserver/actions/workflows/loadtest.yml)
 [![Load test: Postgres cluster, 1 node](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fk18k%2Fmatrix-nodejs-homeserver%2Floadtest-results%2Fbadges%2Fpg-cluster-1.json)](https://github.com/k18k/matrix-nodejs-homeserver/actions/workflows/loadtest.yml)
-[![Load test: Postgres cluster, 5 nodes](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fk18k%2Fmatrix-nodejs-homeserver%2Floadtest-results%2Fbadges%2Fpg-cluster-5.json)](https://github.com/k18k/matrix-nodejs-homeserver/actions/workflows/loadtest.yml)
+[![Load test: Postgres cluster, 2 nodes](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fk18k%2Fmatrix-nodejs-homeserver%2Floadtest-results%2Fbadges%2Fpg-cluster-2.json)](https://github.com/k18k/matrix-nodejs-homeserver/actions/workflows/loadtest.yml)
 
 A [Matrix](https://matrix.org) homeserver as an importable Node.js library. It isn't
 tied to any HTTP framework, database, auth system or deployment shape. You can use
@@ -65,12 +66,21 @@ mysqlAdapter(mysql.createPool(uri));                 // mysql2/promise
 memoryAdapter();                                     // tests / demos
 ```
 
+SQLite runs in WAL mode with `synchronous = NORMAL`: a commit survives a crash of the process,
+but a power loss can roll back the last few. Pass `sqliteAdapter(db, { synchronous: "full" })`
+to wait for the disk on every commit, at the cost of blocking the event loop each time.
+
 Tables are created on startup. Writes for each event are atomic: SQLite uses
 `BEGIN IMMEDIATE`, Postgres and MySQL use a dedicated pooled connection, and the
 memory adapter keeps an undo log.
 
+The Postgres adapter prepares its statements on each connection and reuses them. Behind a
+pooler that does not support prepared statements (PgBouncer before 1.21, or in transaction
+mode without `max_prepared_statements`), pass `postgresAdapter(pool, { preparedStatements: false })`.
+
 To use another store, implement `DatabaseAdapter` (`create`, `upsert`, `findOne`,
-`findMany`, `update`, `delete`, `max`, and optionally `migrate` and `transaction`).
+`findMany`, `update`, `delete`, `max`, and optionally `migrate`, `transaction`,
+`findManyBatch` and `upsertMany`).
 The exported `schema` describes every model. If your driver can run parameterised
 SQL, `sqlAdapter({ dialect, executor, transaction })` handles the SQL generation for you.
 
@@ -459,6 +469,7 @@ Live status, updated after every Complement run (full or targeted) with
 
 | Run | Top-level | Subtests | Note |
 |---|---|---|---|
+| 10 | 214 / 214 | 851 / 851 | load-test fixes #25–#35: batched /sync reads, sync concurrency limit, Postgres lock leases, group commit (248ac8d) |
 | 9 | 214 / 214 | 851 / 851 | zod, procedures, M/UNSTABLE, SSO confirmation, outbound address policy, admin guard (ec5aa79) |
 | 8 | 214 / 214 | 851 / 851 | Complement runs in a Go container (no submodule, no local Go); account locking/suspension, policy servers, 3PIDs, legacy sync endpoints |
 | 7 | 214 / 214 | 851 / 851 | All Complement tests pass (previous full run: 211/214). Fallback keys stored apart from one-time keys; user directory public-room rules; MSC4311 create event required from v12 |
@@ -505,26 +516,53 @@ BASE_URL=http://localhost:8008 ONLINE_USERS=1000 MESSAGES_PER_SECOND=50 ROOMS=25
 
 <!-- loadtest-results:start -->
 The badges at the top come from the [Load test workflow](https://github.com/k18k/matrix-nodejs-homeserver/actions/workflows/loadtest.yml) on main: the same
-scenario on one GitHub runner (4 CPUs), with k6 on the same machine. Each run's job summary has
+scenario on one GitHub runner (4 CPUs), with k6 on the same machine, and clusters of two nodes
+(five nodes would compete with Postgres and k6 for four CPUs). Each run's job summary has
 the full table.
 
-Latest results: 1000 users online in 25 rooms (40 per room), 50 messages/s, 30 s ramp-up then
-30 s at full load, on one developer machine (WSL2, Docker Desktop). Times are p50 / p95.
-
-| Setup | req/s | send | /sync | join | delivery | messages delivered | failed |
-|---|---|---|---|---|---|---|---|
-| SQLite, 1 process (before #28–#35) | ~800 | 46 ms / 0.69 s | 0.50 s / 2.4 s | 61 ms / 0.15 s | 0.58 s / 0.78 s | 5,816 | 0% |
-| Postgres, no cluster | 404 | 49 ms / 71 ms | 2.6 s / 3.6 s | 18 ms / 27 ms | 4.6 s / 6.5 s | 56,297 | 0% |
-| Postgres, cluster, 1 node | 314 | 81 ms / 124 ms | 2.8 s / 4.4 s | 50 ms / 70 ms | 5.2 s / 7.5 s | 52,071 | 0% |
-| Postgres, cluster, 5 nodes | 840 | 118 ms / 197 ms | 0.90 s / 1.4 s | 86 ms / 151 ms | 1.6 s / 2.3 s | 58,298 | 0% |
-
-What the load tests found and fixed so far: presence written on every `/sync` (#26), one commit
-per push notification (#27), a missing index (#28), Postgres cluster locks that could deadlock
-the pool (#29), stream writes serialized per instance (#30), `/sync` reading every change on the
-server (#31) and every joined room (#33), lock handover by polling (#32). Then `/sync` reads were batched
-(#34) and a process builds at most four sync responses at once (#35), so sends no longer queue
-behind the syncs a message wakes. A Postgres node now builds ~330 syncs/s here; delivery time is
-syncs waiting for their turn, and more nodes shorten it.
+Latest results, on one developer machine (WSL2, Docker Desktop): 1000 users online in 25 rooms
+(40 per room), 30 s ramp-up, then 30 s of messages. Times are p50 / p95; "sent" is the share of
+the messages the scenario asks for that the server accepted (a server that falls behind takes
+fewer, and then looks faster than it is).
+
+At 10 messages/s (every user receiving one message every 2.5 s):
+
+| Setup | send | delivery | sent | failed |
+|---|---|---|---|---|
+| SQLite, 1 process | 5 ms / 15 ms | 14 ms / 0.18 s | 100% | 0% |
+| Postgres, no cluster | 9 ms / 30 ms | 29 ms / 0.60 s | 100% | 0% |
+
+At 50 messages/s (every user receiving two messages a second, 59,000 deliveries in 30 s):
+
+| Setup | req/s | send | /sync | delivery | sent | failed |
+|---|---|---|---|---|---|---|
+| SQLite, 1 process | 722 | 11 ms / 25 ms | 0.77 s / 1.3 s | 1.2 s / 1.6 s | 100% | 0% |
+| Postgres, no cluster | 687 | 35 ms / 49 ms | 0.79 s / 1.2 s | 1.2 s / 1.6 s | 100% | 0% |
+| Postgres, cluster, 1 node | 766 | 72 ms / 98 ms | 1.1 s / 1.5 s | 1.7 s / 2.2 s | 100% | 0% |
+| Postgres, cluster, 5 nodes | 910 | 146 ms / 233 ms | 0.42 s / 1.6 s | 0.37 s / 1.5 s | 100% | 0% |
+
+At 50 messages/s, delivery time is how long a process takes to rebuild the `/sync` of every
+user in turn, since every user has something new in nearly every rebuild.
+
+What the load tests found and fixed:
+- presence written on every `/sync` (#26), one commit per push notification (#27), a missing
+  index (#28), Postgres cluster locks that could deadlock the pool (#29), stream writes
+  serialized per instance (#30), `/sync` reading every change on the server (#31) and every
+  joined room (#33), lock handover by polling (#32), unbatched `/sync` reads (#34), syncs
+  crowding out sends (#35);
+- in 0.3.3 (#38–#47): rooms built one after another (#38), a statement per push notification
+  (#39), Postgres re-planning every statement (#40), push rules copied per member (#41),
+  needless to-device deletes (#42), concurrent reads as separate statements (#43), presence and
+  device lists looked up per user (#44), costly batched reads (#45), an fsync per SQLite commit
+  (#46), a blocked event loop (#47);
+- since 0.3.3: waiting `/sync` requests rebuilt whenever anything on the server changed, and
+  about two thirds of all rebuilds found nothing new (#50); a single instance now keeps track of
+  where data last changed, so `/sync` skips reads that would find nothing, like Synapse's stream
+  change caches (#51); queued syncs held the event loop, and yielding per statement made sends
+  slow (#52).
+
+The load test itself was fixed too: its senders registered during the measurement, so a slow
+server was sent fewer messages, and nginx could start after k6 (a red badge with "100% failed").
 <!-- loadtest-results:end -->
 
 ## License

package/dist/api/longpoll.js

@@ -0,0 +1,61 @@
+//#region src/api/longpoll.ts
+/**
+* The long-poll loop of `/sync` and sliding sync: build a response up to the current stream
+* position, and while it has nothing new, wait for a change for `userId` and build again.
+*
+* https://github.com/k18k/matrix-nodejs-homeserver/issues/50: after a build with nothing new, the
+* loop used to build again whenever the server-wide position had moved, which under load is
+* always, so waiting requests rebuilt back to back. Now it registers for the user's wake-ups
+* before reading the position, so none is missed during a build, and otherwise sleeps. A wake-up
+* says which position its change has; if the next build cannot see it yet (another write in
+* flight holds the position back), the loop waits for the position to get there.
+*
+* `build(to)` returns whether the response is final (`done`); returns the last built value, the
+* position it was built at, and whether the request was aborted while waiting.
+*/
+async function longPoll(hs, userId, timeoutMs, signal, build) {
+	const deadline = Date.now() + timeoutMs;
+	let pending = 0;
+	for (;;) {
+		const listener = hs.notifier.listen(userId);
+		pending = Math.max(pending, listener.recent);
+		let to;
+		let built;
+		try {
+			to = await hs.streams.position();
+			if (pending > to) to = await hs.streams.position(true);
+			built = await build(to);
+		} catch (err) {
+			listener.cancel();
+			throw err;
+		}
+		if (pending <= to) pending = 0;
+		const remaining = deadline - Date.now();
+		if (built.done || remaining <= 0 || signal?.aborted) {
+			listener.cancel();
+			return {
+				value: built.value,
+				to,
+				aborted: !!signal?.aborted
+			};
+		}
+		if (pending > to) {
+			listener.cancel();
+			if (await hs.streams.waitFor(pending, remaining, signal)) continue;
+			return {
+				value: built.value,
+				to,
+				aborted: !!signal?.aborted
+			};
+		}
+		const woke = await listener.wait(remaining, signal);
+		if (woke === null) return {
+			value: built.value,
+			to,
+			aborted: !!signal?.aborted
+		};
+		pending = Math.max(pending, woke);
+	}
+}
+//#endregion
+export { longPoll };

package/dist/api/slidingsync.js

@@ -6,6 +6,7 @@ import { getState } from "../core/authrules.js";
 import { record, string, unknown } from "../vendor/zod/v4/classic/schemas.js";
 import { body, lenient } from "../http/input.js";
 import { procedure } from "../http/procedure.js";
+import { longPoll } from "./longpoll.js";
 import { roomEphemeral, roomTimeline, toStripped } from "./sync.js";
 //#region src/api/slidingsync.ts
 const slidingSyncBody = body({
@@ -104,37 +105,34 @@ async function slidingSync(ctx) {
 			rooms: {}
 		};
 	}
-	const deadline = Date.now() + timeout;
-	for (;;) {
-		const to = await hs.streams.position();
-		const { response, empty, next } = await buildResponse(hs, s, body, conn, since, to);
-		const remaining = deadline - Date.now();
-		if (since === null || !empty || remaining <= 0) {
-			await hs.db.upsert("sliding_sync_connections", {
-				user_id: s.userId,
-				device_id: s.deviceId,
-				conn_id: connId,
-				pos: to,
-				state: {
-					...next,
-					prevPos: since
-				},
-				updated_ts: Date.now()
-			});
-			return {
-				pos: String(to),
-				...response
-			};
-		}
-		if (await hs.streams.position() > to) continue;
-		await hs.notifier.wait(s.userId, remaining, ctx.request.signal);
-		if (ctx.request.signal?.aborted) return {
-			pos: String(since),
-			lists: {},
-			rooms: {},
-			extensions: {}
+	const { value, to, aborted } = await longPoll(hs, s.userId, timeout, ctx.request.signal, async (to) => {
+		const built = await buildResponse(hs, s, body, conn, since, to);
+		return {
+			value: built,
+			done: since === null || !built.empty
 		};
-	}
+	});
+	if (aborted && value.empty) return {
+		pos: String(since),
+		lists: {},
+		rooms: {},
+		extensions: {}
+	};
+	await hs.db.upsert("sliding_sync_connections", {
+		user_id: s.userId,
+		device_id: s.deviceId,
+		conn_id: connId,
+		pos: to,
+		state: {
+			...value.next,
+			prevPos: since
+		},
+		updated_ts: Date.now()
+	});
+	return {
+		pos: String(to),
+		...value.response
+	};
 }
 async function buildResponse(hs, s, body, conn, since, to) {
 	const userId = s.userId;
@@ -421,10 +419,10 @@ async function buildExtensions(hs, s, ext, since, to, roomIds, next) {
 			else if (roomIds.includes(r.room_id) || since !== null) (rooms[r.room_id] ??= []).push(ev);
 		}
 		if (since === null && !global.some((e) => e.type === M.PUSH_RULES)) {
-			const { loadRules } = await import("../core/pushrules.js");
+			const { readOnlyRules } = await import("../core/pushrules.js");
 			global.push({
 				type: M.PUSH_RULES,
-				content: { global: await loadRules(hs, userId) }
+				content: { global: await readOnlyRules(hs, userId) }
 			});
 		}
 		out.account_data = {