matrix-nodejs-homeserver 0.3.1 → 0.3.3

package/README.md

@@ -1,11 +1,17 @@
 # matrix-nodejs-homeserver
 
+[![Complement](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fk18k%2Fmatrix-nodejs-homeserver%2Fcomplement-results%2Fbadges%2Fcomplement.json)](https://github.com/k18k/matrix-nodejs-homeserver/actions/workflows/ci.yml)
+[![Load test: SQLite](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fk18k%2Fmatrix-nodejs-homeserver%2Floadtest-results%2Fbadges%2Fsqlite.json)](https://github.com/k18k/matrix-nodejs-homeserver/actions/workflows/loadtest.yml)
+[![Load test: Postgres](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fk18k%2Fmatrix-nodejs-homeserver%2Floadtest-results%2Fbadges%2Fpg-nocluster.json)](https://github.com/k18k/matrix-nodejs-homeserver/actions/workflows/loadtest.yml)
+[![Load test: Postgres cluster, 1 node](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fk18k%2Fmatrix-nodejs-homeserver%2Floadtest-results%2Fbadges%2Fpg-cluster-1.json)](https://github.com/k18k/matrix-nodejs-homeserver/actions/workflows/loadtest.yml)
+[![Load test: Postgres cluster, 5 nodes](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fk18k%2Fmatrix-nodejs-homeserver%2Floadtest-results%2Fbadges%2Fpg-cluster-5.json)](https://github.com/k18k/matrix-nodejs-homeserver/actions/workflows/loadtest.yml)
+
 A [Matrix](https://matrix.org) homeserver as an importable Node.js library. It isn't
 tied to any HTTP framework, database, auth system or deployment shape. You can use
 it for a demo on SQLite, embed it in a monolith next to your existing users, or run
 it as a horizontally scaled chat backend that replaces Slack or Teams.
 
-- Node.js **26+**. TypeScript with only erasable syntax (`erasableSyntaxOnly`), and no runtime dependencies ([zod](https://zod.dev), used for request validation, is bundled).
+- Node.js **22+** (tested on 22, 24 and 26; development uses 26). TypeScript with only erasable syntax (`erasableSyntaxOnly`), and no runtime dependencies ([zod](https://zod.dev), used for request validation, is bundled).
 - Client-Server API, Server-Server API (federation), end-to-end encryption, sliding sync, VoIP/MatrixRTC and push
 - Runs on `node:http`, Express, Fastify, Hono or any fetch runtime
 - Stores data in SQLite, PostgreSQL, MySQL, memory or your own adapter
@@ -60,12 +66,21 @@ mysqlAdapter(mysql.createPool(uri));                 // mysql2/promise
 memoryAdapter();                                     // tests / demos
 ```
 
+SQLite runs in WAL mode with `synchronous = NORMAL`: a commit survives a crash of the process,
+but a power loss can roll back the last few. Pass `sqliteAdapter(db, { synchronous: "full" })`
+to wait for the disk on every commit, at the cost of blocking the event loop each time.
+
 Tables are created on startup. Writes for each event are atomic: SQLite uses
 `BEGIN IMMEDIATE`, Postgres and MySQL use a dedicated pooled connection, and the
 memory adapter keeps an undo log.
 
+The Postgres adapter prepares its statements on each connection and reuses them. Behind a
+pooler that does not support prepared statements (PgBouncer before 1.21, or in transaction
+mode without `max_prepared_statements`), pass `postgresAdapter(pool, { preparedStatements: false })`.
+
 To use another store, implement `DatabaseAdapter` (`create`, `upsert`, `findOne`,
-`findMany`, `update`, `delete`, `max`, and optionally `migrate` and `transaction`).
+`findMany`, `update`, `delete`, `max`, and optionally `migrate`, `transaction`,
+`findManyBatch` and `upsertMany`).
 The exported `schema` describes every model. If your driver can run parameterised
 SQL, `sqlAdapter({ dialect, executor, transaction })` handles the SQL generation for you.
 
@@ -75,12 +90,12 @@ By default a homeserver assumes it is the only process using its database. To ru
 several instances behind a load balancer, give them a shared **cluster adapter**:
 
 ```ts
-import { postgresCluster, redisCluster } from "matrix-nodejs-homeserver";
+import { postgresCluster } from "matrix-nodejs-homeserver";
 
 createHomeserver({
   homeserver: "example.com",
   databaseAdapter: postgresAdapter(pool),
-  cluster: postgresCluster(pool),          // or: redisCluster({ client: new Redis(), subscriber: new Redis() })
+  cluster: postgresCluster(pool),
 });
 ```
 
@@ -95,8 +110,7 @@ The cluster adapter provides:
 - **Shared ephemeral state:** UIA sessions, SSO logins and transaction deduplication.
 
 [examples/multi-node.ts](examples/multi-node.ts) is meant to run unchanged on several
-machines behind a load balancer, all sharing one Postgres database (and optionally Redis
-for coordination). It also shows a media adapter backed by that database, since local
+machines behind a load balancer, all sharing one Postgres database, which also coordinates them. It also shows a media adapter backed by that database, since local
 disks differ between machines.
 
 The database adapter has to support concurrent access from several processes, so
@@ -130,6 +144,9 @@ createHomeserver({
     clientId, clientSecret, mapUser: (claims) => ({ localpart: claims.email.split("@")[0] }) }] },
   publicBaseUrl: "https://matrix.example.com",
   ```
+  After signing in, users confirm the client the login token goes to, on a page
+  that names it. Clients you trust skip that page:
+  `sso: { providers, clientAllowlist: ["https://app.element.io/"] }`.
 - **Delegated OAuth 2.0** (the Matrix OAuth API, e.g. with
   matrix-authentication-service, Keycloak or Authentik). The provider owns the
   accounts, and the homeserver validates its access tokens by introspection:
@@ -328,6 +345,7 @@ paths (server admins only), so existing tools keep working.
 | `publicBaseUrl` | Advertised via `/.well-known/matrix/client`. Required for SSO. |
 | `oauth`, `sso` | Delegated OAuth or upstream OIDC SSO. |
 | `federation` | `{ enabled, verifyTls, allowedServers, blockedServers, fasterJoins }`. Off unless `enabled: true`. |
+| `allowPrivateAddresses` | Lets requests to addresses that users or remote servers choose (federation, push gateways, identity servers, URL previews, remote media) reach private addresses: `true`, or ranges such as `["127.0.0.1/32"]`. Blocked by default. |
 | `rtc` | `{ turn, transports }` |
 | `delayedEvents` | `{ maxDelayMs, maxPerUser }` |
 | `rateLimits` | `false`, or `{ limiter, rules }`. In-memory token buckets by default. Plug in a shared limiter when running a cluster. |
@@ -451,6 +469,8 @@ Live status, updated after every Complement run (full or targeted) with
 
 | Run | Top-level | Subtests | Note |
 |---|---|---|---|
+| 10 | 214 / 214 | 851 / 851 | load-test fixes #25–#35: batched /sync reads, sync concurrency limit, Postgres lock leases, group commit (248ac8d) |
+| 9 | 214 / 214 | 851 / 851 | zod, procedures, M/UNSTABLE, SSO confirmation, outbound address policy, admin guard (ec5aa79) |
 | 8 | 214 / 214 | 851 / 851 | Complement runs in a Go container (no submodule, no local Go); account locking/suspension, policy servers, 3PIDs, legacy sync endpoints |
 | 7 | 214 / 214 | 851 / 851 | All Complement tests pass (previous full run: 211/214). Fallback keys stored apart from one-time keys; user directory public-room rules; MSC4311 create event required from v12 |
 | 6 | 189 / 214 | 803 / 851 | Partial-state joins (MSC3902): all subtests pass |
@@ -471,6 +491,69 @@ Live status, updated after every Complement run (full or targeted) with
 CI runs the whole suite with the same script. `complement/passing.txt` lists the tests
 (and subtests) that are known to pass, and CI fails if any of them regress.
 
+### Load testing
+
+Tuning and investigation runs are local; a [workflow](.github/workflows/loadtest.yml) runs the same scenarios on main for the badges. [loadtest/](loadtest/) holds a k6 scenario and a
+homeserver that reports, every 10 s, requests per second, memory, event-loop delay and
+(with Postgres) how many queries wait for a pool connection and how long locks are held.
+
+The scenario: `ONLINE_USERS` each hold a long-polling `/sync` open in one of `ROOMS` rooms,
+while `MESSAGES_PER_SECOND` messages are sent in total. Every user registers and joins
+through the client API. It measures each request type and the time from sending a message to
+its arrival in other users' `/sync` (`message_delivery`).
+
+```sh
+# Postgres with N homeserver nodes behind nginx (round-robin, no sticky sessions)
+ONLINE_USERS=1000 MESSAGES_PER_SECOND=50 ROOMS=25 RAMP=30s DURATION=30s \
+  docker compose -p matrix-loadtest -f loadtest/docker-compose.yml up --build \
+  --scale homeserver=5 --abort-on-container-exit --exit-code-from k6 k6
+# CLUSTER=0: one process on Postgres without a cluster adapter
+
+# SQLite, one process, no Docker (needs k6 installed)
+DATA_DIR=/tmp/lt node loadtest/server.ts &
+BASE_URL=http://localhost:8008 ONLINE_USERS=1000 MESSAGES_PER_SECOND=50 ROOMS=25 k6 run loadtest/k6/matrix.js
+```
+
+<!-- loadtest-results:start -->
+The badges at the top come from the [Load test workflow](https://github.com/k18k/matrix-nodejs-homeserver/actions/workflows/loadtest.yml) on main: the same
+scenario on one GitHub runner (4 CPUs), with k6 on the same machine. Each run's job summary has
+the full table.
+
+Latest results: 1000 users online in 25 rooms (40 per room), 50 messages/s, 30 s ramp-up then
+30 s at full load, on one developer machine (WSL2, Docker Desktop). Times are p50 / p95.
+
+| Setup | req/s | send | /sync | join | delivery | messages delivered | failed |
+|---|---|---|---|---|---|---|---|
+| SQLite, 1 process | 662 | 97 ms / 0.16 s | 0.87 s / 1.6 s | 10 ms / 91 ms | 1.64 s / 2.70 s | 53,935 | 0% |
+| Postgres, no cluster | 670 | 15 ms / 19 ms | 1.1 s / 1.8 s | 14 ms / 23 ms | 1.76 s / 2.31 s | 58,667 | 0% |
+| Postgres, cluster, 1 node | 593 | 27 ms / 38 ms | 1.3 s / 2.0 s | 36 ms / 51 ms | 1.96 s / 2.57 s | 58,493 | 0% |
+| Postgres, cluster, 5 nodes | 871 | 60 ms / 88 ms | 0.50 s / 1.1 s | 62 ms / 119 ms | 0.70 s / 1.09 s | 59,298 | 0% |
+
+What the load tests found and fixed so far: presence written on every `/sync` (#26), one commit
+per push notification (#27), a missing index (#28), Postgres cluster locks that could deadlock
+the pool (#29), stream writes serialized per instance (#30), `/sync` reading every change on the
+server (#31) and every joined room (#33), lock handover by polling (#32). Then `/sync` reads were batched
+(#34) and a process builds at most four sync responses at once (#35), so sends no longer queue
+behind the syncs a message wakes.
+
+Since 0.3.2 (#38–#45), Postgres delivery time fell by about 55% on one node and 50% on five:
+- changed rooms are built concurrently, with fewer reads each (#38);
+- push notifications of a message are written in one statement (#39);
+- Postgres statements are prepared once per connection (#40);
+- push rules are no longer copied per member (#41);
+- to-device acknowledgements run only when needed (#42);
+- reads issued at the same moment by concurrent requests share one statement (#43);
+- presence and device-list changes are found through a stream index (#44);
+- batched reads take half as long to plan (#45).
+
+On SQLite, commits no longer wait for an fsync (#46). Database calls also yield to the event
+loop (#47): before that, a wave of syncs could hold it for seconds, sends took 0.74 s and the
+server took in only about 40% of the messages the scenario sends.
+
+With every user receiving about two messages a second, delivery time is how long it takes a
+process to rebuild every user's `/sync` in turn: about 750 per second per Postgres node here.
+<!-- loadtest-results:end -->
+
 ## License
 
 [PolyForm Noncommercial 1.0.0](LICENSE). Free to use, modify and share for any noncommercial

package/dist/adapters/cluster.d.ts

@@ -19,37 +19,12 @@ export interface PgClusterPoolLike {
   connect(): Promise<PgClientLike>;
 }
 /**
- * Cluster adapter backed by PostgreSQL (`pg` Pool): sequences, advisory locks,
+ * Cluster adapter backed by PostgreSQL (`pg` Pool): sequences, leased locks,
  * LISTEN/NOTIFY pub/sub and an expiring key-value table. Lets several homeserver
  * instances share one database.
  */
 export declare function postgresCluster(pool: PgClusterPoolLike, options?: {
   tablePrefix?: string;
 }): ClusterAdapter;
-/** Minimal ioredis-compatible client. */
-export interface RedisLike {
-  incr(key: string): Promise<number>;
-  get(key: string): Promise<string | null>;
-  set(key: string, value: string, ...args: (string | number)[]): Promise<string | null>;
-  del(...keys: string[]): Promise<number>;
-  eval(script: string, numKeys: number, ...args: (string | number)[]): Promise<unknown>;
-  publish(channel: string, message: string): Promise<number>;
-  scan(cursor: string | number, ...args: (string | number)[]): Promise<[string, string[]]>;
-  mget(...keys: string[]): Promise<(string | null)[]>;
-}
-/** A Redis connection in subscriber mode (a separate connection, as Redis requires). */
-export interface RedisSubscriberLike {
-  subscribe(...channels: string[]): Promise<unknown>;
-  on(event: "message", listener: (channel: string, message: string) => void): unknown;
-}
-/**
- * Cluster adapter backed by Redis (ioredis API). Pass two connections: one for
- * commands and one dedicated to subscriptions.
- */
-export declare function redisCluster(opts: {
-  client: RedisLike;
-  subscriber: RedisSubscriberLike;
-  prefix?: string;
-}): ClusterAdapter;
 //#endregion
 export { type ClusterAdapter, localCluster };

package/dist/adapters/cluster.js

@@ -16,9 +16,13 @@ function localQueue() {
 	};
 }
 const PG_CHANNEL = "matrix_cluster";
+/** Internal pub/sub channel announcing released locks. */
+const RELEASED = "__lock_released";
+/** How long a lock lease lasts without renewal (it is renewed every third of that while held). */
+const LOCK_TTL_MS = 3e4;
 const MAX_NOTIFY_PAYLOAD = 7e3;
 /**
-* Cluster adapter backed by PostgreSQL (`pg` Pool): sequences, advisory locks,
+* Cluster adapter backed by PostgreSQL (`pg` Pool): sequences, leased locks,
 * LISTEN/NOTIFY pub/sub and an expiring key-value table. Lets several homeserver
 * instances share one database.
 */
@@ -27,6 +31,20 @@ function postgresCluster(pool, options = {}) {
 	const kv = `"${options.tablePrefix ?? ""}cluster_kv"`;
 	const handlers = /* @__PURE__ */ new Map();
 	const queue = localQueue();
+	const waiting = /* @__PURE__ */ new Map();
+	const releasedOrAfter = (lockKey, ms) => new Promise((resolve) => {
+		let set = waiting.get(lockKey);
+		if (!set) waiting.set(lockKey, set = /* @__PURE__ */ new Set());
+		const wake = () => {
+			clearTimeout(timer);
+			set.delete(wake);
+			if (set.size === 0 && waiting.get(lockKey) === set) waiting.delete(lockKey);
+			resolve();
+		};
+		const timer = setTimeout(wake, ms);
+		set.add(wake);
+	});
+	handlers.set(RELEASED, /* @__PURE__ */ new Set([(lockKey) => [...waiting.get(lockKey) ?? []].forEach((wake) => wake())]));
 	let listener = null;
 	let cleanup = null;
 	const ready = (async () => {
@@ -80,16 +98,38 @@ function postgresCluster(pool, options = {}) {
 		lock(key, fn) {
 			return queue(key, async () => {
 				await ready;
-				const client = await pool.connect();
+				const lockKey = `lock:${key}`;
+				const token = randomUUID();
+				const acquire = async () => {
+					const now = Date.now();
+					return ((await pool.query(`INSERT INTO ${kv} (key, value, expires_at) VALUES ($1, $2, $3)
+             ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value, expires_at = EXCLUDED.expires_at
+             WHERE ${kv}.expires_at IS NOT NULL AND ${kv}.expires_at <= $4 RETURNING key`, [
+						lockKey,
+						token,
+						now + LOCK_TTL_MS,
+						now
+					])).rowCount ?? 0) > 0;
+				};
+				for (let delay = 5; !await acquire(); delay = Math.min(delay * 2, 200)) await releasedOrAfter(lockKey, delay);
+				const renew = setInterval(() => void pool.query(`UPDATE ${kv} SET expires_at = $3 WHERE key = $1 AND value = $2`, [
+					lockKey,
+					token,
+					Date.now() + LOCK_TTL_MS
+				]).catch(() => {}), LOCK_TTL_MS / 3);
 				try {
-					await client.query("SELECT pg_advisory_lock(hashtextextended($1, 0))", [key]);
-					try {
-						return await fn();
-					} finally {
-						await client.query("SELECT pg_advisory_unlock(hashtextextended($1, 0))", [key]);
-					}
+					return await fn();
 				} finally {
-					client.release();
+					clearInterval(renew);
+					await pool.query(`WITH released AS (DELETE FROM ${kv} WHERE key = $1 AND value = $2 RETURNING key) SELECT pg_notify($3, $4) FROM released`, [
+						lockKey,
+						token,
+						PG_CHANNEL,
+						JSON.stringify({
+							channel: RELEASED,
+							message: lockKey
+						})
+					]).catch(() => {});
 				}
 			});
 		},
@@ -152,98 +192,5 @@ function postgresCluster(pool, options = {}) {
 		}
 	};
 }
-const UNLOCK = `if redis.call("get", KEYS[1]) == ARGV[1] then return redis.call("del", KEYS[1]) else return 0 end`;
-const EXTEND = `if redis.call("get", KEYS[1]) == ARGV[1] then return redis.call("pexpire", KEYS[1], ARGV[2]) else return 0 end`;
-/**
-* Cluster adapter backed by Redis (ioredis API). Pass two connections: one for
-* commands and one dedicated to subscriptions.
-*/
-function redisCluster(opts) {
-	const { client, subscriber } = opts;
-	const prefix = opts.prefix ?? "matrix:";
-	const channel = `${prefix}cluster`;
-	const handlers = /* @__PURE__ */ new Map();
-	const queue = localQueue();
-	const ready = subscriber.subscribe(channel);
-	subscriber.on("message", (ch, raw) => {
-		if (ch !== channel) return;
-		try {
-			const { channel: name, message } = JSON.parse(raw);
-			for (const h of handlers.get(name) ?? []) h(message);
-		} catch {}
-	});
-	const k = (key) => `${prefix}${key}`;
-	return {
-		async nextId(name) {
-			return Number(await client.incr(k(`seq:${name}`)));
-		},
-		async currentId(name) {
-			return Number(await client.get(k(`seq:${name}`)) ?? 0);
-		},
-		async advance(name, value) {
-			await client.eval(`local v = tonumber(redis.call("get", KEYS[1]) or "0") if v < tonumber(ARGV[1]) then redis.call("set", KEYS[1], ARGV[1]) end return 1`, 1, k(`seq:${name}`), value);
-		},
-		lock(key, fn) {
-			return queue(key, async () => {
-				const lockKey = k(`lock:${key}`);
-				const token = randomUUID();
-				const ttl = 3e4;
-				for (let delay = 5;; delay = Math.min(delay * 2, 200)) {
-					if (await client.set(lockKey, token, "PX", ttl, "NX") === "OK") break;
-					await new Promise((r) => setTimeout(r, delay));
-				}
-				const renew = setInterval(() => void client.eval(EXTEND, 1, lockKey, token, ttl).catch(() => {}), ttl / 3);
-				try {
-					return await fn();
-				} finally {
-					clearInterval(renew);
-					await client.eval(UNLOCK, 1, lockKey, token).catch(() => {});
-				}
-			});
-		},
-		async publish(name, message) {
-			await ready;
-			await client.publish(channel, JSON.stringify({
-				channel: name,
-				message
-			}));
-		},
-		subscribe(name, handler) {
-			let set = handlers.get(name);
-			if (!set) handlers.set(name, set = /* @__PURE__ */ new Set());
-			set.add(handler);
-		},
-		async get(key) {
-			const v = await client.get(k(`kv:${key}`));
-			return v === null ? null : JSON.parse(v);
-		},
-		async set(key, value, ttlMs) {
-			if (ttlMs) await client.set(k(`kv:${key}`), JSON.stringify(value), "PX", ttlMs);
-			else await client.set(k(`kv:${key}`), JSON.stringify(value));
-		},
-		async setIfAbsent(key, value, ttlMs) {
-			return (ttlMs ? await client.set(k(`kv:${key}`), JSON.stringify(value), "PX", ttlMs, "NX") : await client.set(k(`kv:${key}`), JSON.stringify(value), "NX")) === "OK";
-		},
-		async delete(key) {
-			await client.del(k(`kv:${key}`));
-		},
-		async entries(keyPrefix) {
-			const out = [];
-			let cursor = "0";
-			const pattern = `${k(`kv:${keyPrefix}`).replace(/[*?[\]]/g, (c) => `\\${c}`)}*`;
-			do {
-				const [next, keys] = await client.scan(cursor, "MATCH", pattern, "COUNT", 200);
-				cursor = next;
-				if (keys.length) {
-					const values = await client.mget(...keys);
-					keys.forEach((key, i) => {
-						if (values[i] !== null) out.push([key.slice(k("kv:").length), JSON.parse(values[i])]);
-					});
-				}
-			} while (cursor !== "0");
-			return out;
-		}
-	};
-}
 //#endregion
-export { localCluster, postgresCluster, redisCluster };
+export { localCluster, postgresCluster };

package/dist/api/account.js

@@ -4,11 +4,13 @@ import { M, UNSTABLE } from "../core/m.js";
 import { MatrixError, errors } from "../core/errors.js";
 import { LOCALPART_RE } from "../core/ids.js";
 import { toMsisdn } from "../core/phone.js";
+import { escapeHtml } from "../core/html.js";
 import { canonicalAddress, isMedium } from "../core/threepid.js";
 import { boolean, looseObject, string } from "../vendor/zod/v4/classic/schemas.js";
 import { body, fail, lenient } from "../http/input.js";
 import { procedure } from "../http/procedure.js";
 import { requireUia } from "./uia.js";
+import { checkRedirectUrl } from "../core/sso.js";
 import { createHmac, randomInt, timingSafeEqual } from "node:crypto";
 //#region src/api/account.ts
 const deviceFields = {
@@ -146,10 +148,26 @@ function registerAccountRoutes(r) {
 		if (!ctx.hs.oauth) throw errors.unrecognized("OAuth is not enabled");
 		return { issuer: ctx.hs.oauth.options.issuer };
 	} }), ["v1"]);
+	const ssoConfirmPage = (serverName, result) => {
+		const client = escapeHtml(result.client);
+		const html = `<!doctype html><meta charset="utf-8"><meta name="viewport" content="width=device-width"><title>Continue to ${client}</title>
+<style>body{font:16px system-ui,sans-serif;max-width:32rem;margin:3rem auto;padding:0 1rem}a.button{display:inline-block;padding:.6rem 1.2rem;background:#0b7a55;color:#fff;border-radius:6px;text-decoration:none}</style>
+<h1>Continue to ${client}?</h1>
+<p>You signed in to <b>${escapeHtml(serverName)}</b>. <b>${client}</b> will get access to your account.</p>
+<p>If you did not just sign in with this app, close this page.</p>
+<p><a class="button" href="${escapeHtml(result.url)}">Continue to ${client}</a></p>`;
+		return new Response(html, { headers: {
+			"content-type": "text/html; charset=utf-8",
+			"cache-control": "no-store",
+			"referrer-policy": "no-referrer",
+			"x-frame-options": "DENY",
+			"content-security-policy": "default-src 'none'; style-src 'unsafe-inline'; frame-ancestors 'none'"
+		} });
+	};
 	const ssoRedirect = async (ctx) => {
 		if (!ctx.hs.sso) throw errors.unrecognized("SSO is not enabled");
-		const redirectUrl = ctx.query.get("redirectUrl");
-		if (!redirectUrl || !/^https?:\/\/|^[a-z][a-z0-9+.-]*:/i.test(redirectUrl)) throw errors.missingParam("Missing or invalid redirectUrl");
+		const redirectUrl = checkRedirectUrl(ctx.query.get("redirectUrl"));
+		if (!redirectUrl) throw errors.missingParam("Missing or invalid redirectUrl");
 		return Response.redirect(await ctx.hs.sso.startLogin(ctx.params.idpId ?? null, redirectUrl), 302);
 	};
 	r.client("POST", "/login/get_token", procedure({
@@ -170,7 +188,8 @@ function registerAccountRoutes(r) {
 	r.client("GET", "/login/sso/redirect/:idpId", procedure({ handler: ssoRedirect }));
 	r.client("GET", "/login/sso/callback/:idpId", procedure({ handler: async (ctx) => {
 		if (!ctx.hs.sso) throw errors.unrecognized("SSO is not enabled");
-		return Response.redirect(await ctx.hs.sso.finishLogin(ctx.params.idpId, ctx.query), 302);
+		const result = await ctx.hs.sso.finishLogin(ctx.params.idpId, ctx.query);
+		return result.confirm ? ssoConfirmPage(ctx.hs.serverName, result) : Response.redirect(result.url, 302);
 	} }));
 	r.client("POST", "/login", procedure({
 		rateLimit: "login",

package/dist/api/admin.js

@@ -30,10 +30,13 @@ async function requireAdmin(ctx) {
 	});
 	return s;
 }
+const adminGuard = async (ctx) => {
+	await requireAdmin(ctx);
+	return {};
+};
 /** Admin HTTP API using Synapse-compatible paths, so existing tooling keeps working. */
 function registerAdminRoutes(r) {
 	const notice = async (ctx) => {
-		await requireAdmin(ctx);
 		const body = ctx.input;
 		return { event_id: await ctx.hs.admin.sendServerNotice(body.user_id, body.content, {
 			type: body.type,
@@ -71,10 +74,12 @@ function registerAdminRoutes(r) {
 	r.get("/_matrix/client/v1/admin/suspend/:userId", procedure({ handler: suspend.get }));
 	r.put("/_matrix/client/v1/admin/suspend/:userId", procedure({ handler: suspend.put }));
 	r.post("/_synapse/admin/v1/send_server_notice", procedure({
+		guard: adminGuard,
 		input: noticeBody,
 		handler: notice
 	}));
 	r.put("/_synapse/admin/v1/send_server_notice/:txnId", procedure({
+		guard: adminGuard,
 		input: noticeBody,
 		handler: notice
 	}));
@@ -128,9 +133,9 @@ function registerAdminRoutes(r) {
 		return ctx.hs.admin.getUser(userId);
 	} }));
 	r.post("/_synapse/admin/v1/deactivate/:userId", procedure({
+		guard: adminGuard,
 		input: body({ erase: lenient(boolean()) }),
 		handler: async (ctx) => {
-			await requireAdmin(ctx);
 			const body = ctx.input;
 			await ctx.hs.admin.deactivateUser(ctx.params.userId, { erase: body.erase === true });
 			return { id_server_unbind_result: "no-support" };
@@ -141,9 +146,9 @@ function registerAdminRoutes(r) {
 		logout_devices: lenient(boolean())
 	});
 	r.post("/_synapse/admin/v1/reset_password/:userId", procedure({
+		guard: adminGuard,
 		input: resetPasswordBody,
 		handler: async (ctx) => {
-			await requireAdmin(ctx);
 			const body = ctx.input;
 			await ctx.hs.admin.resetPassword(ctx.params.userId, body.new_password, body.logout_devices !== false);
 			return {};
@@ -154,9 +159,9 @@ function registerAdminRoutes(r) {
 		return { registration_tokens: await ctx.hs.admin.listRegistrationTokens() };
 	} }));
 	r.post("/_synapse/admin/v1/registration_tokens/new", procedure({
+		guard: adminGuard,
 		input: registrationTokenBody,
 		handler: async (ctx) => {
-			await requireAdmin(ctx);
 			const body = ctx.input;
 			return ctx.hs.admin.createRegistrationToken({
 				token: body.token,

package/dist/api/media.js

@@ -53,7 +53,7 @@ async function remoteMedia(ctx) {
 	}
 	let fetched;
 	try {
-		fetched = await hs.federation.fetchMedia(serverName, mediaId);
+		fetched = await hs.federation.fetchMedia(serverName, mediaId, maxUpload(hs));
 	} catch (err) {
 		if (err instanceof MatrixError) throw err;
 		throw new MatrixError(502, "M_UNKNOWN", "Could not fetch remote media");

package/dist/api/preview.js

@@ -1,40 +1,9 @@
 import { randomString } from "../core/crypto.js";
-import { errors } from "../core/errors.js";
+import { MatrixError, errors } from "../core/errors.js";
+import { publicGet } from "../core/outbound.js";
 import { procedure } from "../http/procedure.js";
-import { lookup } from "node:dns/promises";
-import { BlockList, isIP } from "node:net";
 //#region src/api/preview.ts
 const MAX_BYTES = 1048576;
-const privateRanges = new BlockList();
-for (const [net, prefix] of [
-	["0.0.0.0", 8],
-	["10.0.0.0", 8],
-	["100.64.0.0", 10],
-	["127.0.0.0", 8],
-	["169.254.0.0", 16],
-	["172.16.0.0", 12],
-	["192.0.0.0", 24],
-	["192.168.0.0", 16],
-	["198.18.0.0", 15],
-	["224.0.0.0", 4],
-	["240.0.0.0", 4]
-]) privateRanges.addSubnet(net, prefix, "ipv4");
-for (const [net, prefix] of [
-	["::", 128],
-	["::1", 128],
-	["fc00::", 7],
-	["fe80::", 10],
-	["ff00::", 8],
-	["::ffff:0:0", 96]
-]) privateRanges.addSubnet(net, prefix, "ipv6");
-async function assertPublicHost(hostname) {
-	const host = hostname.replace(/^\[|\]$/g, "");
-	const addresses = isIP(host) ? [{
-		address: host,
-		family: isIP(host)
-	}] : await lookup(host, { all: true });
-	for (const { address, family } of addresses) if (privateRanges.check(address, family === 6 ? "ipv6" : "ipv4")) throw errors.forbidden("URL resolves to a private address");
-}
 function decodeEntities(s) {
 	return s.replace(/&(amp|lt|gt|quot|#39|#x27);/g, (m) => ({
 		"&": "&",
@@ -72,42 +41,37 @@ async function preview(hs, rawUrl) {
 		throw errors.invalidParam("Invalid URL");
 	}
 	if (url.protocol !== "http:" && url.protocol !== "https:") throw errors.invalidParam("Only http(s) URLs can be previewed");
-	for (let redirects = 0; redirects < 5; redirects++) {
-		if (!opts.allowPrivateAddresses) await assertPublicHost(url.hostname);
-		const res = await fetch(url, {
-			redirect: "manual",
-			signal: AbortSignal.timeout(1e4),
-			headers: { "user-agent": "matrix-nodejs-homeserver" }
-		});
-		if (res.status >= 300 && res.status < 400 && res.headers.get("location")) {
-			url = new URL(res.headers.get("location"), url);
-			continue;
-		}
-		if (!res.ok || !res.body) throw errors.notFound("Could not fetch URL");
-		const type = res.headers.get("content-type") ?? "";
-		if (type.startsWith("image/")) return {
-			...await storeImage(hs, res, type),
-			"og:url": url.href
+	const get = opts.allowPrivateAddresses ? hs.fetch : hs.publicFetch;
+	const headers = { "user-agent": "matrix-nodejs-homeserver" };
+	const res = await publicGet(get, url, {
+		maxBytes: MAX_BYTES * 10,
+		truncate: true,
+		timeoutMs: 1e4,
+		headers
+	}).catch((err) => {
+		throw err instanceof MatrixError ? err : errors.notFound("Could not fetch URL");
+	});
+	if (res.status !== 200) throw errors.notFound("Could not fetch URL");
+	const type = res.headers.get("content-type") ?? "";
+	if (type.startsWith("image/")) {
+		if (res.body.length >= MAX_BYTES * 10) return {};
+		return {
+			...await storeImage(hs, res.body, type),
+			"og:url": res.url.href
 		};
-		if (!type.includes("html")) return {};
-		let size = 0;
-		const chunks = [];
-		for await (const chunk of res.body) {
-			size += chunk.length;
-			if (size > MAX_BYTES) break;
-			chunks.push(chunk);
-		}
-		const og = parseOpenGraph(Buffer.concat(chunks).toString("utf8"));
-		if (og["og:image"]) try {
-			const imageUrl = new URL(og["og:image"], url);
-			if (!opts.allowPrivateAddresses) await assertPublicHost(imageUrl.hostname);
-			const img = await fetch(imageUrl, { signal: AbortSignal.timeout(1e4) });
-			const imgType = img.headers.get("content-type") ?? "";
-			if (img.ok && imgType.startsWith("image/")) Object.assign(og, await storeImage(hs, img, imgType));
-		} catch {}
-		return og;
 	}
-	throw errors.notFound("Too many redirects");
+	if (!type.includes("html")) return {};
+	const og = parseOpenGraph(res.body.subarray(0, MAX_BYTES).toString("utf8"));
+	if (og["og:image"]) try {
+		const img = await publicGet(get, new URL(og["og:image"], res.url), {
+			maxBytes: MAX_BYTES * 10,
+			timeoutMs: 1e4,
+			headers
+		});
+		const imgType = img.headers.get("content-type") ?? "";
+		if (img.status === 200 && imgType.startsWith("image/")) Object.assign(og, await storeImage(hs, img.body, imgType));
+	} catch {}
+	return og;
 }
 /** Read width/height from PNG, GIF, JPEG or WebP headers. */
 function imageSize(data) {
@@ -157,9 +121,8 @@ function imageSize(data) {
 	return null;
 }
 /** Store a fetched image in the media repository and describe it for a preview. */
-async function storeImage(hs, res, type) {
-	const data = new Uint8Array(await res.arrayBuffer());
-	if (data.length > MAX_BYTES * 10) return {};
+async function storeImage(hs, body, type) {
+	const data = new Uint8Array(body);
 	const mediaId = randomString(24);
 	await hs.media.put(mediaId, data, {
 		contentType: type,

package/dist/api/slidingsync.js

@@ -421,10 +421,10 @@ async function buildExtensions(hs, s, ext, since, to, roomIds, next) {
 			else if (roomIds.includes(r.room_id) || since !== null) (rooms[r.room_id] ??= []).push(ev);
 		}
 		if (since === null && !global.some((e) => e.type === M.PUSH_RULES)) {
-			const { loadRules } = await import("../core/pushrules.js");
+			const { readOnlyRules } = await import("../core/pushrules.js");
 			global.push({
 				type: M.PUSH_RULES,
-				content: { global: await loadRules(hs, userId) }
+				content: { global: await readOnlyRules(hs, userId) }
 			});
 		}
 		out.account_data = {