@harperfast/harper 5.3.2 → 5.3.3

package/server/DESIGN.md

@@ -358,6 +358,7 @@ A persisted QoS 1/2 entry is `{ qos, topic, startTime, databaseGeneration? }`, a
 - **Before CONNACK,** a collection entry bound to its database's current generation is checked against the floor (a record entry's own walk decides after CONNACK); a failure deletes the record and reports the session as not present. The connection then writes its incarnation (`takeOver`), and every later write first checks the stored incarnation: a connection that finds another's stops writing and closes (0x8E), and one resumed from a record never creates one. Any newer authenticated connection for the client id on the same thread, a clean start included, closes the older durable connection the same way before it reads or deletes the record, once that connection has saved its positions, and waits for that save to settle; a connection that has already closed counts until its last save settles. A save that failed leaves the record at the last one that landed, so the newer connection resumes from there and may redeliver, never skip. Simultaneous CONNECTs for one client id take their turn through a per-thread claim, so each sees the one before it; one that gives up waiting keeps its place until the one before it finishes. The CONNECT's pending session is assigned before it waits, so packets sent right behind the CONNECT wait with it rather than being refused; the wait is bounded (10 s, then CONNACK 0x88, or 0x03 on v3.1.1), and a CONNECT whose client closes while it waits takes nothing over. A session that has ended, by a disconnect or by yielding to a takeover, saves nothing after its final save, so a SUBSCRIBE still in flight cannot write over the record a newer connection claimed.
 - **Subscription changes are serial.** Packets are handled concurrently, so a durable session applies SUBSCRIBE, UNSUBSCRIBE and `resume()` one at a time: a topic's state belongs to one subscription. A SUBSCRIBE of any QoS is saved before SUBACK. A failed SUBSCRIBE leaves its topic unsubscribed and saves that. It ends the subscription it started, and drops the state of the one it replaced, which replacing ended. An UNSUBSCRIBE whose save fails is reported failed again on a retry until the removal is saved, and any failed save starts the checkpoint timer, so a session with only QoS 0 topics retries it too.
 - **Wills are kept per connection,** keyed `[clientId, incarnation]`: a connection closing after a takeover publishes and deletes only its own will (MQTT 5 §3.1.4 publishes a taken-over client's will), never the will of the connection that replaced it.
+- **Only a will's publish is authorized** (`settleWill`, harper#1874). The `hdb_session_will` row is the session's bookkeeping: it is read and deleted on contexts of its own, not the session's `authorize: true` context, which spends its one-shot check on the first call (roles are not granted system tables, so a non-super user's will never published and its row leaked). The publish then runs on a fresh session context, as the connection's user or a scoped will's embedded role, and the row is deleted after any outcome, a refused or failed publish included, so a will that fails for a transient reason is lost: the boot sweep, the only retry path, would replay a kept row only at the next restart, with no liveness check (harper#3034). The publish commits before the delete, so a crash between them leaves the row for that sweep and the will may be published twice (by design; untested). `unitTests/server/sessionWill.test.js` and `integrationTests/mqtt/will-authorization.test.ts` enforce that only the publish is authorized and that the row is deleted after a publish, a refusal and a clean DISCONNECT.
 - **One terminal path** (`subscriptionFailed`): a refused resume subscribe, a final `Error` value (observed through the subscription's `close` as well as the consumer loop, which may be parked on socket back-pressure), or an iterator failure. Only a 410 (pruned history) discards the session — writers fenced, siblings ended, a v5 DISCONNECT (0x83, the client-error message when problem information is allowed) and a socket close that never waits on storage, then a deletion only if the record still carries this incarnation. Anything else, a 409 from a restored or copied database included, saves positions and closes the connection so the client resumes.
 - **Only the current generation is checked.** A position bound to another generation, another node's (`hdb_durable_session` replicates, and each node mints its own generations) or this database's before a restore or copy, resumes best-effort: it is not checked before CONNACK, replays unchecked from where it is, and binds again at the first certified position. Until a live write gives it a watermark, that replay saves no position, so a reconnect on a quiet database replays it again.
 - **Not covered:** concurrent connections for one client id on different threads, together with crash replay of a failed conditional write (harper#2940); late commits below a saved position, including a late transaction still queued behind the message being sent, which the session cannot see (harper#2928); a replay from another generation, which is unchecked.
@@ -647,6 +648,7 @@ The SQL and job paths are additive rather than exclusive: `verifyPermsAST` valid
 - **Headers commit exactly once, through `this.writeHead`.** `write()`, `end()`, `flushHeaders()` and `_implicitHeader()` all reach `this.writeHead(this.statusCode)` by property lookup, so a `writeHead` that `on-headers` replaced on the instance runs its listeners (the ones that set `Content-Encoding` and remove `Content-Length`) before the promise resolves. After commit, `setHeader`/`appendHeader`/`removeHeader` and a second `writeHead` throw `ERR_HTTP_HEADERS_SENT` as Node's do; `_header` (which `compression` ≤ 1.7 tests instead of `headersSent`) and `finished` (which `on-finished` tests) derive from that state.
 - **The adapter owns the `'error'` listener.** A `destroy(err)` right after `writeHead()` emits before the awaiting caller can attach one; the error stays in the stream's `errored` state for `pipeline()`, `finished()` or async iteration. Client disconnect (`Request.signal`) destroys the response without an error after headers (a plain premature close, which `pipeBodyToResponse` treats as routine) and rejects the promise with the abort reason before them; a handler that throws or rejects before ending the response destroys it, and one that fails after `end()` is logged at warn.
 - **Header names are case-insensitive on removal too.** `Headers.delete` lowercases like `set`/`get`/`has`; the inherited `Map.delete` silently left `send`'s `Content-Length` on a gzip body (truncated transfers).
+- **Every piped chunk reaches `write()` by property lookup.** Node 26.11's `Readable.pipe` fast path (nodejs/node#66182) calls `_write()` directly when pipe's listener is a byte-mode source's only `'data'` listener, so `compression`'s instance patch of `res.write` was skipped for most chunks of `send(...).pipe(res)`: raw bytes inside the gzip stream, or a stall once `compression` had routed pipe's `'drain'` to its idle zlib stream. On runtimes that have the fast path (`'writeKnownBuffer' in Writable`, a static it added) the response adds an inert `'data'` listener to each byte-mode core `Readable` on `'pipe'`, which fails the fast path's single-listener check, and removes it on `'unpipe'` through a listener prepended in the constructor: pipe's own cleanup resumes a source awaiting `'drain'` while any `'data'` listener remains, which would leave an unpiped source flowing into nothing. Object-mode and decoded sources, and legacy `Stream.prototype.pipe` (which never emits `'unpipe'`), are skipped. Remove it once every supported Node 26 release dispatches pipe through an overridden `write()`.
 - **Express is not a target.** `express`'s `app.handle()` replaces the response's prototype with one rooted at `http.ServerResponse.prototype`, which no Writable-derived response survives, and would do the same to the request `Proxy`'s target, Harper's real `IncomingMessage`. Middleware that duck-types the response (Next.js, `compression`, `send`, `serve-static`, `finalhandler`, h3, fastify) is the supported surface.
 
 ## `manageThreads` has two different `workerCount`s (`server/threads/manageThreads.js`)

package/server/DurableSubscriptionsSession.ts

@@ -484,22 +484,7 @@ class SubscriptionsSession {
 		const willId = this.willId;
 		this.willId = undefined;
 		if (willId) {
-			const context = this.createContext();
-			transaction(context, async () => {
-				try {
-					if (!clientTerminated) {
-						const will = await getLastWill().get(willId);
-						if (will && !isWillFromExpiredScopedToken(will)) {
-							// A scoped will authorizes under its own embedded role, never the disconnecting
-							// session's user (which may be a later same-clientId reconnect).
-							const willContext = will.user?._scopedToken ? { ...context, user: will.user } : context;
-							await publishMessage(will, will.data, willContext);
-						}
-					}
-				} finally {
-					await getLastWill().delete(willId);
-				}
-			}).catch((error) => {
+			this.settleWill(willId, !clientTerminated).catch((error) => {
 				warn(`Error publishing MQTT will for ${this.sessionId}`, error);
 			});
 		}
@@ -509,6 +494,23 @@ class SubscriptionsSession {
 		}
 		this.subscriptions = [];
 	}
+	/** The will row is session bookkeeping on contexts of its own; only the publish runs as the session (server/DESIGN.md). */
+	async settleWill(willId: [string, string], publish: boolean) {
+		try {
+			if (publish) {
+				const will = await getLastWill().get(willId, {});
+				if (will && !isWillFromExpiredScopedToken(will)) {
+					const context = this.createContext();
+					// A scoped will authorizes under its own embedded role, never the disconnecting
+					// session's user (which may be a later same-clientId reconnect).
+					if (will.user?._scopedToken) context.user = will.user;
+					await publishMessage(will, will.data, context);
+				}
+			}
+		} finally {
+			await getLastWill().delete(willId, {});
+		}
+	}
 	receivedPacket() {
 		if (this.keepalive) {
 			clearTimeout(this.keepaliveTimer);

package/server/mqtt.ts

@@ -206,6 +206,19 @@ const SERVER_SHUTTING_DOWN = 0x8b; // MQTT v5 DISCONNECT reason code
 const REASON_STRING_LIMIT = 256; // bytes
 /** Fixed header, message id and reason code, plus the property identifier and its length prefix. */
 const ACK_PACKET_OVERHEAD = 16;
+/** `AccessViolation` is 401 for a request with no user and 403 for a user lacking permission; an MQTT client sees both as "Not authorized". */
+const isNotAuthorized = (error: any) => error?.statusCode === 401 || error?.statusCode === 403;
+/** Only failures this layer maps to a code of their own are safe to describe: any other carries an internal message. */
+const hasClientSafeMessage = (error: any) => isNotAuthorized(error) || error?.statusCode === 404;
+/** REASON_STRING_LIMIT counts encoded bytes, and a character must not be cut in half. */
+function boundedReasonString(message: unknown): string {
+	const text = String(message);
+	if (Buffer.byteLength(text) <= REASON_STRING_LIMIT) return text;
+	const bytes = Buffer.from(text, 'utf8');
+	let end = REASON_STRING_LIMIT;
+	while (end > 0 && (bytes[end] & 0xc0) === 0x80) end--;
+	return bytes.subarray(0, end).toString('utf8');
+}
 
 /**
  * Every live MQTT connection on this worker, so a shutdown can announce itself: a bare socket close
@@ -327,17 +340,14 @@ function onSocket(socket, send, request, user, mqttSettings) {
 	function closeConnection(error?: Error) {
 		try {
 			if (mqttOptions.protocolVersion >= 5) {
-				const packet: any = { cmd: 'disconnect', reasonCode: error ? 0x83 : 0x8e };
+				const packet: any = { cmd: 'disconnect', reasonCode: error ? (isNotAuthorized(error) ? 0x87 : 0x83) : 0x8e };
 				// as for PUBACK: only failures this layer describes itself are safe to hand a client
 				const describable =
 					error instanceof DatabaseGenerationChangedError ||
 					error instanceof ResumeHistoryUnavailableError ||
-					(error as any)?.statusCode === 403 ||
-					(error as any)?.statusCode === 404;
+					hasClientSafeMessage(error);
 				const reasonString =
-					describable && error.message && sendProblemInformation
-						? Buffer.from(String(error.message), 'utf8').subarray(0, REASON_STRING_LIMIT).toString('utf8')
-						: undefined;
+					describable && error.message && sendProblemInformation ? boundedReasonString(error.message) : undefined;
 				if (
 					reasonString &&
 					(!maximumPacketSize || maximumPacketSize >= Buffer.byteLength(reasonString) + ACK_PACKET_OVERHEAD)
@@ -547,7 +557,11 @@ function onSocket(socket, send, request, user, mqttSettings) {
 					});
 					const listener = async (topic, message, messageId, subscription, version) => {
 						try {
-							if (disconnected) throw new Error('Session disconnected while trying to send message to', topic);
+							if (disconnected) {
+								// a delivery already past the subscription when the client left; the disconnect has ended the session
+								mqttLog.debug?.(`Session ${session?.sessionId} disconnected while trying to send message to ${topic}`);
+								return false;
+							}
 							const slashIndex = topic.indexOf('/', 1);
 							const generalTopic = slashIndex > 0 ? topic.slice(0, slashIndex) : topic;
 							const qos = subscription.qos || 0;
@@ -606,6 +620,8 @@ function onSocket(socket, send, request, user, mqttSettings) {
 					break;
 				case 'subscribe':
 					const granted = [];
+					// a SUBACK carries one reason string for the whole packet, so the first refusal's is the one sent
+					let refusalMessage: string | undefined;
 					for (const subscription of packet.subscriptions) {
 						let grantedQos;
 						try {
@@ -624,21 +640,27 @@ function onSocket(socket, send, request, user, mqttSettings) {
 							grantedQos =
 								mqttOptions.protocolVersion < 5
 									? 0x80 // the only error code in v3.1.1
-									: error.statusCode === 403
+									: isNotAuthorized(error)
 										? 0x87 // unauthorized
 										: error.statusCode === 404
 											? 0x8f // invalid topic
 											: 0x80; // generic failure
+							if (mqttOptions.protocolVersion >= 5 && error.message && hasClientSafeMessage(error))
+								refusalMessage ??= error.message;
 						}
 						granted.push(grantedQos);
 					}
 					await session.committed;
-					generateAndSendPacket({
-						// Send a subscription acknowledgment
-						cmd: 'suback',
-						granted,
-						messageId: packet.messageId,
-					});
+					const subackPacket: any = { cmd: 'suback', granted, messageId: packet.messageId };
+					if (refusalMessage !== undefined && sendProblemInformation)
+						subackPacket.properties = { reasonString: boundedReasonString(refusalMessage) };
+					let suback = generate(subackPacket, mqttOptions);
+					// the result vector grows with the filter count, so only the encoded packet says whether the string fits
+					if (subackPacket.properties && maximumPacketSize && suback.length > maximumPacketSize) {
+						delete subackPacket.properties;
+						suback = generate(subackPacket, mqttOptions);
+					}
+					sendPacket(suback, 'suback');
 					break;
 				case 'unsubscribe': {
 					const granted = [];
@@ -691,23 +713,18 @@ function onSocket(socket, send, request, user, mqttSettings) {
 								reasonCode:
 									mqttOptions.protocolVersion < 5
 										? 0x80 // the only error code in v3.1.1
-										: error?.statusCode === 403
+										: isNotAuthorized(error)
 											? 0x87 // not authorized
 											: error?.statusCode === 404
 												? 0x90 // topic name invalid
 												: 0x80, // unspecified error
 							};
-							// Only errors this layer maps to a code of their own are safe to describe: any other
-							// failure carries an internal message that a client must not be handed. Both limits
-							// are in encoded bytes, so measure and trim the string in bytes too.
 							const describable =
 								mqttOptions.protocolVersion >= 5 &&
 								sendProblemInformation &&
 								error?.message &&
-								(error.statusCode === 403 || error.statusCode === 404);
-							const reasonString = describable
-								? Buffer.from(String(error.message), 'utf8').subarray(0, REASON_STRING_LIMIT).toString('utf8')
-								: undefined;
+								hasClientSafeMessage(error);
+							const reasonString = describable ? boundedReasonString(error.message) : undefined;
 							if (
 								reasonString &&
 								(!maximumPacketSize || maximumPacketSize >= Buffer.byteLength(reasonString) + ACK_PACKET_OVERHEAD)

package/server/serverHelpers/NodeAdapterResponse.ts

@@ -6,7 +6,7 @@ import type {
 	ServerResponse as NodeServerResponse,
 } from 'node:http';
 import type { Socket } from 'node:net';
-import { PassThrough } from 'node:stream';
+import { PassThrough, Readable, Writable } from 'node:stream';
 import { Headers as ResponseHeaders, applyWriteHeadHeaders } from './Headers.ts';
 
 export interface AdaptedResponse {
@@ -29,6 +29,20 @@ class UnsupportedResponseMethodError extends Error {
 }
 
 const ignoreError = () => {};
+const ignoreChunk = () => {};
+
+// Node 26.11's pipe fast path (nodejs/node#66182) skips write() overrides such as compression's while pipe's listener
+// is a byte-mode source's only 'data' listener; see server/DESIGN.md
+const pipeSkipsWrite = 'writeKnownBuffer' in Writable;
+function guardPipeSource(source: Readable) {
+	// object mode, a decoder, and legacy or userland pipe() never take the fast path; legacy pipe() never emits 'unpipe'
+	if (source instanceof Readable && !source.readableObjectMode && !source.readableEncoding)
+		source.on('data', ignoreChunk);
+}
+// must run before pipe's own unpipe cleanup, which resumes a source awaiting 'drain' while any 'data' listener remains
+function releasePipeSource(source: unknown) {
+	if (source instanceof Readable) source.removeListener('data', ignoreChunk);
+}
 
 export class NodeAdapterResponse extends PassThrough implements NodeServerResponse {
 	statusCode = 200;
@@ -60,6 +74,10 @@ export class NodeAdapterResponse extends PassThrough implements NodeServerRespon
 		this.#resolve = resolve;
 		this.#reject = reject;
 		this.on('error', ignoreError);
+		if (pipeSkipsWrite) {
+			this.on('pipe', guardPipeSource);
+			this.prependListener('unpipe', releasePipeSource);
+		}
 		if (typeof nodeResponse?.on === 'function') {
 			const forward = (...args: any[]) => this.emit('timeout', ...args);
 			let forwarding = false;