harper 5.3.3 → 5.3.4

package/resources/DESIGN.md

@@ -153,7 +153,7 @@ Consequences worth knowing:
   branches and, when a RocksDB record version differs from its log key, the record's own audit head.
   Under stage 2 the stored reference field can be renamed; until then, "fixing" it to the record
   version silently unaddresses the entry it points at.
-- **Retried duplicates use committed identity before audit I/O ([#3361](https://github.com/HarperFast/harper/issues/3361)).** `Table._writeUpdate` recognizes an implicit head only when origin, version and received log key match and no explicit refs exist; `replicatedRetryDuplicate.test.js` covers real conflicts, including multiple writes to the same key.
+- **Replicated duplicates use committed identity before audit I/O ([#3361](https://github.com/HarperFast/harper/issues/3361), [#3467](https://github.com/HarperFast/harper/issues/3467)).** `Table._writeUpdate` recognizes an implicit head only when origin, version and received log key match and no explicit refs exist, on first delivery and on retry alike. Identity comes from the commit base unless another write to the key in the transaction has saved without being skipped in the current retry round (a retry re-saves writes in order, so a later sibling's flags are stale until it re-stages); then it is read from the committed record, outside the transaction, because a publish or relocate saves when it is added and restamps the base, before or after this write, which would make a write that never applied look like its own head. The out-of-order ref check reads its refs from the same record, and a source-applied relocate that ties with the committed head is skipped too: re-applied, it would restamp the head, which can be its own transaction's put. The head's own log entry is not required: a base-copied record has none, and looking for it costs an end-of-log scan and, on a miss, re-appends the duplicate for every peer to re-receive. On retry, standalone keyed checks are shielded and walk skips require known committed-origin provenance; an ambiguous chain can double-apply. Fresh-delivery orphan handling remains out of scope pending native conflict-before-log validation (#3253). A retry's pinned log watermark can still hide a duplicate flushed afterward. Retained-read-snapshot replay clears audit flags for writes staged on a never-submitted handle. `replicatedRetryDuplicate.test.js` covers real conflicts, including multiple writes to the same key, same-key restamps, terminal-abort orphans, committed controls, a retried same-key batch whose duplicate survives only in audit history, and a retry whose own entry is on another handle; `relayedHeadTieDuplicate.test.js` covers a relayed duplicate of a base-copied record, a put before or after a same-key publish or relocate, a competing delivery committing while a publish is staged, a log key that differs from the version, and a re-delivered non-resident relocate.
 - **Crash replay uses both.** `replayLogs` replays each write at its stored `version`; stamping a
   replayed record at its log key would move its version forward and make a later legitimate write
   look stale. Its transactions follow native commits: one ends at a commit's last-entry `endTxn`
@@ -387,7 +387,7 @@ Opt-in deflate compression for file-backed blobs (harper#2443) has three load-be
 
 **Extending the budget for one known-long write:** `DatabaseTransaction.timeoutBudget` is a per-transaction RocksDB floor applied whenever the transaction is re-armed (initial reads, writes, and active multi-store-chain propagation); the effective timeout is `Math.max(txnExpiration, timeoutBudget)`. This makes the budget sticky across a write's pre-commit existing-entry read and later writes, while never shortening a larger global `STORAGE_MAXTRANSACTIONOPENTIME`; RocksDB links added for another store inherit the same floor. Reads after a pending write do not re-arm the transaction: that preserves the idle-limit invariant for orphaned write-holding requests. Also, `resources/transaction.ts`'s `transaction(callback)` (no explicit context) joins whatever transaction is already open on the ambient AsyncLocalStorage context rather than guaranteeing a fresh one. `components/deploymentRecorder.ts`'s `withIsolatedTransaction` builds a new context from only the ambient audit/session fields — not the request `signal`, so a deploy client that stops waiting cannot cancel the tracking writes and strand an `hdb_deployment` row mid-phase (see the disconnect note below) — and every recorder write commits independently without inheriting transaction controls. It uses the sticky budget to give `ingestPayload`'s blob-gated writes a size-appropriate limit instead of the generic default, while coalesced progress flushes are drained and suppressed until ingest settles to avoid same-row transaction conflicts. The ingest helper deliberately floors the shared `deployment_timeout` at ten minutes because `0` means “poll once” for peer waits; consequently an ingest can pin its system-database snapshot for that minimum. Known gap (harper#2057): the extension only reaches RocksDB transactions — on `HARPER_STORAGE_ENGINE=lmdb`, `Table.txnForContext()` chains a separate `LMDBTransaction` (`txn.next`) with its own independently-reset timeout that the LMDB engine's monitor tracks instead. **A commit's conflict retries have their own deadline, separate from the open-transaction limit** (issue #2450). rocksdb-js ≥2.8 wakes a commit parked on another transaction's write intent after `ROCKSDB_JS_PARK_TIMEOUT_MS` (5s) and returns `RETRY_NOW_VALUE` even when the holder never releases, so a wedged intent presents as a stream of transient conflicts rather than one hung commit — and the `MAX_RETRIES` cap alone then keeps the request pending for ~40 park timeouts, minutes past the configured queue limit. `DatabaseTransaction.commitStartedAt` is stamped on the **chain root** (or on the `ImmediateTransaction` instance while a single handle is active, migrating onto the per-write native `RocksTransaction` handle when concurrent commits overlap or a late immediate commit runs on a closed request scope) at its first native submission and read at both retry decisions (the coordinated `RETRY_NOW_VALUE` resolve path and the `ERR_BUSY`/`ERR_TRY_AGAIN` rejection path); past `Math.max(STORAGE_MAXTRANSACTIONQUEUETIME, timeoutBudget)` an unlogged commit takes the existing `abortChainAfterRetries()` cleanup and throws a 503 `TransactionCommitConflictTimeoutError`. One clock per _logical_ commit, deliberately not per attempt: the per-attempt clock is `trackOutstandingCommit()`'s, which measures native liveness and drives `checkOverloaded()`'s thread-wide shedding, so back-dating it would let one uncapped `sourceApply` retry shed every unrelated request on the thread. The clock is released through the promise `commit()` returns (which settles only after the chained stores' commits), so a reused transaction's next batch starts fresh. `retryable` is true only on a chain root that has rotated through no mid-scope commit — anywhere else an earlier store already wrote durable audit entries and ran its hooks that a replayed request would repeat. `sourceApply` is exempt, as it is from the attempt cap, for the harper-pro#348 divergence reason above.
 
-- **Once a transaction has a durable transaction-log entry, it is never aborted on retry** (harper#3236). In rocksdb-js, `Transaction::Commit` flushes staged `TransactionLog` batches (`executeLogWork`, assigning `committedPosition`) _before_ `txn->Commit()` checks optimistic conflicts; on conflict the handle resets onto a fresh snapshot while keeping `committedPosition > 0` so the log is write-once (rocksdb-js#668), and aborting it throws `ERR_TRANSACTION_ABANDONED` and orphans a durable log entry that live reads miss while subscriptions and `replayLogs()` apply.
+- **Once a transaction has a durable transaction-log entry, it is never aborted on retry** (harper#3236). In rocksdb-js, `Transaction::Commit` flushes staged `TransactionLog` batches (`executeLogWork`, assigning `committedPosition`) _before_ `txn->Commit()` checks optimistic conflicts; on conflict the handle resets onto a fresh snapshot while keeping `committedPosition > 0` so the log is write-once (rocksdb-js#668). Aborting then throws `ERR_TRANSACTION_ABANDONED` and publishes a durable audit entry even though the primary record never changed; keyed audit reads, subscriptions and `replayLogs()` can observe that orphan, so audit replay may diverge from the primary store. Fresh-delivery handling of this orphan remains out of scope pending native conflict-before-log validation (harper#3253).
   - **Per-handle invariant:** once a handle's first native commit returns a conflict (`hasDurableLog(transaction)` = `isRetry && hasStagedLog`, evaluated per native handle so an unlogged link in a multi-store chain still respects `suspendDatabaseCommits` and retry bounds), `DatabaseTransaction` treats it like `sourceApply`: the caller stays awaiting an uncapped retry (so the #2450 deadline and 40-attempt cap apply only before the first log flush or on `audit: false` tables), record-lock lease-expiry aborts are skipped while post-expiry/release native unlock is deferred up to the conflict budget, `deferForCommitInFlight` reclaims separate read snapshots without poisoning or logging the unknown-outcome restart error, and `ImmediateTransaction` stages only writes whose `save()` has been invoked while isolating per-commit state on the native handle when concurrent commits overlap.
   - **Operational cost:** while a logged retry waits on a conflicting write intent (which `abortDueToTimeout` clears after ~2 ticks of `STORAGE_MAXTRANSACTIONOPENTIME` if held by an unsubmitted open transaction), its position in `uncommittedPositions` pins the database's `'local'` transaction-log read watermark for live subscriptions and replication (reported by `noteOverdueLoggedCommit`), the caller awaits convergence without a 503-pending or detached background commit, and `waitForDrain()` holds across retry backoff delays up to its 120s deadline.
   - **Native follow-up (harper#3253):** validating conflicts before `executeLogWork` so a conflicting commit never flushes a log entry or pins the read watermark, exposing `committedPosition`, and reconciling non-conflict or post-drain aborts live in rocksdb-js.
@@ -405,6 +405,10 @@ Opt-in deflate compression for file-backed blobs (harper#2443) has three load-be
 
 RocksDB 2.9.0 binds ranges to their supplied transaction and invalidates them when that transaction ends. An abandoned range is still bounded by the idle monitor. Resuming after its snapshot has been released throws an `ReadSnapshotExpiredError` (503) before accessing the native iterator; retry only the read, without replaying previously committed writes. A poisoned write-bearing transaction retains its existing 422 error and rollback behavior. Early iterator return remains a cleanup operation, including after expiration.
 
+## A read reference belongs to the handle acquisition it was taken on (`DatabaseTransaction.useReadTxn`/`doneReadTxn`, harper#3308)
+
+`readTxnsUsed` is never positive without a read handle, and `doneReadTxn(generation)` ignores a return from an earlier acquisition (`readTxnGeneration`, bumped per acquisition, because lmdb-js renews the same `Txn` object for a new snapshot). Enforced by `unitTests/resources/detachedReadRefs.test.js`.
+
 ## Replicated apply failure listeners (`resources/replicatedApplyFailure.ts`)
 
 `registerReplicatedApplyFailureListener(database, listener)` and

package/resources/DatabaseTransaction.ts

@@ -606,6 +606,7 @@ export type TransactionWrite = {
 	stagedIn?: DatabaseTransaction;
 	// the preceding write to the same store and key in this transaction, if any (linked in addWrite)
 	priorWrite?: TransactionWrite;
+	keyId?: unknown;
 	// set only by a write that BOTH reads priorStagedWrite() and publishes stagedEntry; addWrite orders
 	// those against earlier same-key writes. A write that does one or neither would be ordered against a
 	// basis it cannot consume.
@@ -646,6 +647,10 @@ export type TransactionWrite = {
 	// save() must reload that base through the committing transaction's snapshot
 	reloadCommitBase?: boolean;
 	stagedOn?: RocksTransaction;
+	// the handle and retry round this write last ran its commit handler in: a retry re-saves writes in order,
+	// so a sibling's saved/skipped flags describe the current snapshot only once it has staged in this round
+	committedOn?: RocksTransaction;
+	committedRound?: number;
 	baseReadTxn?: any;
 	writeGeneration?: WriteGeneration;
 	instanceClosed?: boolean;
@@ -764,6 +769,10 @@ export class DatabaseTransaction implements Transaction {
 	readTxn: ReadTransaction;
 	readTxnRefCount: number;
 	readTxnsUsed: number;
+	// Bumped each time this instance acquires a read handle, so a reference can be returned to the
+	// acquisition it came from (doneReadTxn()). Not the handle's identity: lmdb-js hands back the same
+	// read Txn object for a new snapshot.
+	readTxnGeneration = 0;
 	timeout: number;
 	// Write recency, tracked separately from `timeout`: set only by addWrite, never by a read. `timeout`
 	// is re-armed by reads too (on a link with no pending writes of its own, see the fast path in
@@ -1006,6 +1015,7 @@ export class DatabaseTransaction implements Transaction {
 		readTransactionOwners.set(transaction, this);
 		this.rangeReadActive = false;
 		this.readTxnsUsed = 1;
+		this.readTxnGeneration++;
 		this.baseReadRefConsumed = false;
 		this.handleOpenedAt = performance.now();
 	}
@@ -1106,12 +1116,12 @@ export class DatabaseTransaction implements Transaction {
 		// the array it never got threw (issue #2222). Capturing an Error per read that nothing can dump
 		// is not worth doing either, so those reads stay untraced.
 		if (DEBUG_LONG_TXNS && this.stackTraces) this.stackTraces.push(new StartedTransaction());
-		this.readTxnsUsed++;
+		if (readTxn) this.readTxnsUsed++;
 		return readTxn;
 	}
 
-	doneReadTxn() {
-		if (!this.transaction) return;
+	doneReadTxn(generation = this.readTxnGeneration) {
+		if (!this.transaction || generation !== this.readTxnGeneration) return;
 		if (--this.readTxnsUsed === 0) {
 			// The native handle was only being held open for the read iterators: any writes staged
 			// through it were already committed at commit() time by replaying them onto a fresh
@@ -1191,7 +1201,7 @@ export class DatabaseTransaction implements Transaction {
 		if (operation.key === undefined) return;
 		let writesForStore = (this.writesByKey ??= new Map()).get(operation.store);
 		if (!writesForStore) this.writesByKey.set(operation.store, (writesForStore = new Map()));
-		const keyId = writeKeyId(operation.key);
+		const keyId = (operation.keyId = writeKeyId(operation.key));
 		const priorWrite = writesForStore.get(keyId);
 		if (priorWrite) operation.priorWrite = priorWrite;
 		writesForStore.set(keyId, operation);
@@ -2093,6 +2103,8 @@ export class DatabaseTransaction implements Transaction {
 			if (shouldStampStagedOn) operation.stagedOn = transaction;
 			if (lockHandle || this.recordLocks) operation.trackRecordVersion = true;
 			if (operation.trackRecordVersion) operation.recordVersionApplied = false;
+			operation.committedOn = transaction;
+			operation.committedRound = txnRetries;
 			try {
 				completion = operation.commit(
 					writeVersion,
@@ -2561,10 +2573,18 @@ export class DatabaseTransaction implements Transaction {
 							if (transaction) this.releaseActiveCommitHandle(transaction);
 							this.trackActiveCommitHandle(replayTransaction);
 							if (prevStagedTxnTime) this.setHandleStagedTxnTime(replayTransaction, prevStagedTxnTime);
+							const abandoned = transaction;
+							const abandonedLogged = hasDurableLog(abandoned);
 							transaction = replayTransaction;
 							try {
 								for (const operation of this.writes) {
 									if (!operation?.saved) continue;
+									// A non-durable abandoned handle left no readable audit entry; a durable one may
+									// have published its log and must retain the retry shield.
+									if (operation.appendedTo === abandoned && !abandonedLogged) {
+										operation.appendedAuditEntry = false;
+										operation.appendedTo = undefined;
+									}
 									this.save(operation, replayTransaction, true, options);
 								}
 							} catch (error) {
@@ -3218,7 +3238,7 @@ export class DatabaseTransaction implements Transaction {
 						this.baseReadRefConsumed = true;
 					}
 				} else {
-					while (this.readTxnsUsed > 0) this.doneReadTxn(); // release the read snapshot when we abort, we assume we don't need it
+					while (this.transaction && this.readTxnsUsed > 0) this.doneReadTxn(); // release the read snapshot when we abort, we assume we don't need it
 				}
 				// A write-only transaction never took a read reference (getReadTxn was never called), so the
 				// loop above releases nothing even though save() created a native handle. Keep an iterator's

package/resources/LMDBTransaction.ts

@@ -70,6 +70,7 @@ export class LMDBTransaction extends DatabaseTransaction {
 		this.readTxn = (this.db as any).useReadTransaction();
 
 		this.readTxnsUsed = 1;
+		this.readTxnGeneration++;
 		this.baseReadRefConsumed = false;
 		if ((this.readTxn as any).openTimer) (this.readTxn as any).openTimer = 0;
 		trackedTxns.add(this as any);
@@ -89,8 +90,8 @@ export class LMDBTransaction extends DatabaseTransaction {
 		return this.readTxn;
 	}
 
-	doneReadTxn() {
-		if (!this.readTxn) return;
+	doneReadTxn(generation = this.readTxnGeneration) {
+		if (!this.readTxn || generation !== this.readTxnGeneration) return;
 		if (this.readTxn instanceof RocksTransaction) {
 			// TODO: Implement this for RocksDB
 		} else {
@@ -434,7 +435,7 @@ export class LMDBTransaction extends DatabaseTransaction {
 					this.baseReadRefConsumed = true;
 				}
 			} else {
-				while (this.readTxnsUsed > 0) this.doneReadTxn(); // release the read snapshot when we abort, we assume we don't need it
+				while (this.readTxn && this.readTxnsUsed > 0) this.doneReadTxn(); // release the read snapshot when we abort, we assume we don't need it
 			}
 			this.open = TRANSACTION_STATE.CLOSED;
 			this.drainCompletions();

package/resources/Table.ts

@@ -261,6 +261,8 @@ const MAX_CONCURRENT_LMDB_HISTORY_REMOVALS = 1000;
 // Each evict otherwise pays a full transaction commit, so batching amortizes that cost. LMDB already
 // coalesces async writes per event turn (eventTurnBatching), so it keeps the per-record path.
 const EVICTION_BATCH_SIZE = 100;
+const COMMITTED_READ = { uncachedRead: true };
+const UNRESOLVED = Symbol('unresolved');
 // Cap on eviction-batch commits in flight at once, so commit I/O overlaps scan/staging without
 // letting an unbounded number of open transactions (and their snapshots) accumulate.
 const MAX_INFLIGHT_EVICTION_BATCHES = 4;
@@ -3796,10 +3798,23 @@ export function makeTable(options): TableResourceClass {
 					(this.constructor as any).source?.relocate && !(context as any)?.source
 						? (this.constructor as any).source.relocate.bind((this.constructor as any).source, id, undefined, context)
 						: undefined,
-				commit: (txnTime, existingEntry, _retry, transaction: any) => {
+				commit: (txnTime, existingEntry, _retry, transaction: any, committedBy?: DatabaseTransaction) => {
 					const txnLogKey =
 						isRocksDB && options?.version != null ? (transaction?.getTimestamp?.() ?? txnTime) : txnTime;
-					if (precedesExistingVersion(txnTime, existingEntry, options?.nodeId) < 0) return;
+					write.skipped = false;
+					const precedesExisting = precedesExistingVersion(txnTime, existingEntry, options?.nodeId);
+					if (precedesExisting < 0) return;
+					if (
+						// a re-delivered relocate would restamp the committed head, which can be another write of its transaction
+						precedesExisting === 0 &&
+						committedBy?.sourceApply &&
+						isRocksDB &&
+						txnLogKey === txnTime &&
+						isHeadIdentity(identityBase(write, existingEntry, committedBy), txnTime, options?.nodeId)
+					) {
+						write.skipped = true;
+						return;
+					}
 					const residency = TableResource.getResidencyRecord(options.residencyId);
 					let metadata = 0;
 					let newRecord = null;
@@ -4648,15 +4663,13 @@ export function makeTable(options): TableResourceClass {
 					transaction: any,
 					committedBy?: DatabaseTransaction
 				) => {
-					// Whether a prior attempt of THIS write appended its own audit entry (sticky, set in
-					// save(); log entries are not part of the aborted rocks transaction, so they survive).
-					// Only such a write can find its own orphaned entry in the dedup lookups below and must
-					// not treat it as "already applied". Per-write on purpose: the transaction-wide retry
-					// flag would also suppress dedup for a genuine re-delivered duplicate co-batched with
-					// the conflicting write (double-applying it) and for fresh writes staged through a
-					// reused transaction whose retries counter is stale. Sticky on purpose: a proxy read
-					// from the last attempt's skipped state launders when a recommit round self-skips
-					// (walk identity tie against its own staged record) before a fresh-transaction replay.
+					// Whether a prior attempt of THIS write appended its audit entry (sticky, set in
+					// save()). A retry cannot infer that its record mutation committed from an exact log
+					// match alone, so keyed probes stay shielded and walk skips require committed provenance.
+					// Per-write: a transaction-wide retry flag would also suppress dedup for unrelated
+					// deliveries co-batched with the conflict or fresh writes in a reused transaction.
+					// Sticky: a proxy read from the last attempt's skipped state launders when a recommit
+					// round self-skips (walk identity tie against its own staged record) before a replay.
 					const stagedOwnAuditEntry = retry && write.appendedAuditEntry === true;
 					write.skipped = false; // reset on each retry; cleanup happens after commit if still true
 					write.stagedEntry = undefined; // likewise: only set once this round actually stores a record
@@ -4738,15 +4751,18 @@ export function makeTable(options): TableResourceClass {
 						let belowAuditFloor = false;
 						let dedupVersionCouldBeRetained: (version: number) => boolean;
 						if (audit) {
+							let identityRecord: Partial<Entry> | undefined | typeof UNRESOLVED = UNRESOLVED;
 							if (
-								stagedOwnAuditEntry &&
 								committedBy?.sourceApply &&
 								isRocksDB &&
 								!replaying &&
 								precedesExisting === 0 &&
 								txnLogKey === txnTime &&
-								(existingEntry.nodeId ?? 0) === (options?.nodeId ?? getThisNodeId(auditStore)) &&
-								!existingEntry.additionalAuditRefs?.length
+								isHeadIdentity(
+									(identityRecord = identityBase(write, existingEntry, committedBy)),
+									txnTime,
+									options?.nodeId
+								)
 							) {
 								write.skipped = true;
 								return;
@@ -4764,7 +4780,10 @@ export function makeTable(options): TableResourceClass {
 							// === 0 is the identity tie: same version AND same node (the local node is id 0, so an undefined
 							// options?.nodeId resolves to the same 0 the ref stored).
 							if (
-								existingEntry.additionalAuditRefs?.some(
+								(identityRecord === UNRESOLVED
+									? identityBase(write, existingEntry, committedBy)
+									: identityRecord
+								)?.additionalAuditRefs?.some(
 									(ref) =>
 										ref.version === txnLogKey &&
 										precedesExistingVersion(
@@ -4822,11 +4841,7 @@ export function makeTable(options): TableResourceClass {
 							// RocksDB-only: LMDB audit entries are keyed by local audit time, so this lookup doesn't apply
 							// there (LMDB keeps the exact unbounded walk). A miss (the keyed lookup can lag a back-to-back re-delivery — #1137)
 							// simply falls through to the walk, so this never changes correctness; the additionalAuditRefs
-							// check above remains the read-your-writes guard. Never when this write staged in a prior
-							// failed attempt: that attempt already appended this write's own audit entry, so the lookup
-							// would find it and skip the write as "already applied" when the record was never committed.
-							// A recommit of the same transaction survived that skip only because the old write batch
-							// still carried the put; a fresh-transaction replay (ERR_TRY_AGAIN) would drop the write.
+							// check above remains the read-your-writes guard. Not on a same-write retry (see stagedOwnAuditEntry).
 							if (isRocksDB && !replaying && !stagedOwnAuditEntry && dedupVersionCouldBeRetained(txnLogKey)) {
 								const priorAudit = auditStore.get(txnLogKey, tableId, id, options?.nodeId);
 								if (
@@ -4878,10 +4893,21 @@ export function makeTable(options): TableResourceClass {
 							// `previousNodeId` is never encoded, so absent there means the log is unknown and only the
 							// aggregate lookup can resolve a cross-origin predecessor.
 							let nodeId = initialAuditHead.nodeId ?? 0;
+							// A retry can load a base restamped by an earlier saved publish/relocate from this
+							// transaction. Those writes do not set stagedEntry, so priorStaged alone cannot
+							// establish that the base is committed.
+							const retryBaseIsCommitted =
+								!priorStaged && !(stagedOwnAuditEntry && hasSavedSameKeySibling(write, committedBy));
+							let walkOriginKnown = retryBaseIsCommitted;
 							const succeedingUpdates = []; // record the "future" updates, as we need to apply the updates in reverse order
-							const auditRefsToVisit: Array<{ localTime: number; nodeId: number }> = existingEntry.additionalAuditRefs
-								? existingEntry.additionalAuditRefs.map((ref) => ({ localTime: ref.version, nodeId: ref.nodeId }))
-								: [];
+							const auditRefsToVisit: Array<{ localTime: number; nodeId: number; originKnown: boolean }> =
+								existingEntry.additionalAuditRefs
+									? existingEntry.additionalAuditRefs.map((ref) => ({
+											localTime: ref.version,
+											nodeId: ref.nodeId,
+											originKnown: retryBaseIsCommitted,
+										}))
+									: [];
 
 							// Out-of-order merges retain every existing branch head; per-origin log keys are not globally ordered.
 							if (existingEntry.additionalAuditRefs) {
@@ -4890,13 +4916,13 @@ export function makeTable(options): TableResourceClass {
 								}
 							}
 							let addedAuditRef = false;
-							let nextRef: { localTime: number; nodeId: number };
+							let nextRef: { localTime: number; nodeId: number; originKnown: boolean };
 							const visitedAuditRefs = new Set<string>();
-							const queuePreviousAuditRefs = (auditRecord) => {
+							const queuePreviousAuditRefs = (auditRecord, originKnown: boolean) => {
 								const previousRefs = auditRecord.previousAdditionalAuditRefs;
 								if (previousRefs) {
 									for (const ref of previousRefs) {
-										auditRefsToVisit.push({ localTime: ref.version, nodeId: ref.nodeId });
+										auditRefsToVisit.push({ localTime: ref.version, nodeId: ref.nodeId, originKnown });
 										logger.debug?.('Adding audit ref from audit record to visit queue', {
 											version: ref.version,
 											nodeId: ref.nodeId,
@@ -4912,6 +4938,7 @@ export function makeTable(options): TableResourceClass {
 										: { txnLogKey: auditRecord.previousVersion, nodeId: auditRecord.previousNodeId };
 								localTime = previousHead.txnLogKey;
 								nodeId = previousHead.nodeId;
+								walkOriginKnown = walkOriginKnown && previousHead.nodeId != null;
 							};
 							let walkSteps = 0;
 							let auditWalkCapped = false;
@@ -4927,9 +4954,7 @@ export function makeTable(options): TableResourceClass {
 							// A re-delivered write whose exact (version, nodeId) is already in the audit log was already
 							// applied; drop it rather than re-applying it (double-applying commutative ops) or writing a
 							// duplicate audit-only record. Used by the early-out and the depth-cap block below.
-							// Never a duplicate for a write that staged in a prior failed attempt: that attempt already
-							// appended this write's own audit entry, so the lookup would match it while the record was
-							// never committed (see the up-front keyed dedup above).
+							// Not for a same-write retry (see stagedOwnAuditEntry).
 							const isReDeliveredDuplicate = () => {
 								if (replaying || stagedOwnAuditEntry) return false;
 								if (!dedupVersionCouldBeRetained(txnLogKey)) return false; // pre-retention log key — skip the end-of-log scan (best-effort; see above)
@@ -4959,17 +4984,15 @@ export function makeTable(options): TableResourceClass {
 									}
 									const auditRecord = auditStore.get(localTime, tableId, id, nodeId);
 									if (!auditRecord) break;
-									queuePreviousAuditRefs(auditRecord);
+									queuePreviousAuditRefs(auditRecord, walkOriginKnown);
 									if (
 										isRocksDB &&
 										!replaying &&
-										!stagedOwnAuditEntry &&
+										// on a retry only a link from committed state proves the write committed: an
+										// orphaned log entry can match the identity but is never linked from a committed record
+										(!stagedOwnAuditEntry || walkOriginKnown) &&
 										localTime === txnLogKey &&
-										precedesExistingVersion(
-											txnTime,
-											{ version: txnTime, localTime: txnLogKey, key: id, nodeId: auditRecord.nodeId },
-											options?.nodeId
-										) === 0
+										(auditRecord.nodeId ?? 0) === (options?.nodeId ?? getThisNodeId(auditStore))
 									) {
 										write.skipped = true;
 										return;
@@ -4987,9 +5010,8 @@ export function makeTable(options): TableResourceClass {
 													// Same origin and record version, but a distinct write. Its per-origin log key
 													// orders the otherwise non-unique record clock without comparing keys across origins.
 													precedesExisting = txnLogKey > localTime ? 1 : -1;
-												} else if (replaying || stagedOwnAuditEntry) {
-													// The log entry being replayed (or staged by this write's failed attempt) is
-													// the write itself, not proof that its primary-store mutation committed.
+												} else if (replaying || (stagedOwnAuditEntry && !walkOriginKnown)) {
+													// replayed, or reached on a retry without committed provenance: not proof it committed
 													precedesExisting = 1;
 												} else {
 													logger.debug?.(
@@ -5075,6 +5097,7 @@ export function makeTable(options): TableResourceClass {
 								if (nextRef) {
 									localTime = auditedVersion = nextRef.localTime;
 									nodeId = nextRef.nodeId;
+									walkOriginKnown = nextRef.originKnown;
 									logger.debug?.('Following additional audit ref to continue scanning', { localTime, nodeId });
 								}
 							} while (nextRef);
@@ -5968,6 +5991,7 @@ export function makeTable(options): TableResourceClass {
 			// scans), the read transaction reads against the latest committed data without pinning a
 			// consistent snapshot, so the scan doesn't hold a snapshot that blocks compaction.
 			const readTxn = txn.useReadTxn(target.snapshot === false);
+			const readTxnGeneration = txn.readTxnGeneration;
 			// The explicit row filter participates in query execution: it is pushed into HNSW
 			// traversal for vector sorts, applied after conditions otherwise, and checked again after
 			// cache/source materialization below. A policy error aborts the query instead of silently
@@ -6110,7 +6134,7 @@ export function makeTable(options): TableResourceClass {
 						} finally {
 							// We own the iteration here (no results.onDone consumer), so release the read
 							// transaction unconditionally — including when the drain throws — or the snapshot leaks.
-							txn.doneReadTxn();
+							if (readTxn) txn.doneReadTxn(readTxnGeneration);
 						}
 						let total: number | null;
 						if (wantExact) {
@@ -6150,7 +6174,7 @@ export function makeTable(options): TableResourceClass {
 				results.onDone = () => {
 					results.onDone = null; // ensure that it isn't called twice
 					txn.unregisterReadIterator(results);
-					txn.doneReadTxn();
+					if (readTxn) txn.doneReadTxn(readTxnGeneration);
 				};
 				// Recorded ownership: if the request dies before anything consumes these results, the
 				// transaction closes them itself rather than leaving its read snapshot pinned.
@@ -6159,7 +6183,7 @@ export function makeTable(options): TableResourceClass {
 				results.getColumns = getColumns;
 				return results;
 			} catch (error) {
-				txn.doneReadTxn();
+				if (readTxn) txn.doneReadTxn(readTxnGeneration);
 				throw error;
 			}
 		}
@@ -9124,6 +9148,39 @@ export function makeTable(options): TableResourceClass {
 		if (!(existingEntry?.metadataFlags & INVALIDATED)) return existingEntry;
 	}
 
+	// The commit base is the committed record unless another write to the key already saved without being
+	// skipped this round: a publish or relocate saves when it is added and restamps the base without staging it.
+	function identityBase(write: any, existingEntry: Entry, transaction: DatabaseTransaction | undefined) {
+		if (hasSavedSameKeySibling(write, transaction)) return committedEntry(write.key);
+		return existingEntry;
+	}
+
+	function hasSavedSameKeySibling(write: any, transaction: DatabaseTransaction | undefined): boolean {
+		for (let other = transaction?.writesByKey?.get(write.store)?.get(write.keyId); other; other = other.priorWrite) {
+			if (
+				other !== write &&
+				other.saved &&
+				!other.skipped &&
+				other.committedOn === write.committedOn &&
+				other.committedRound === write.committedRound
+			)
+				return true;
+		}
+		return false;
+	}
+
+	function committedEntry(id: Id): Entry | undefined {
+		return primaryStore.getEntry(id, COMMITTED_READ);
+	}
+
+	function isHeadIdentity(entry: Partial<Entry> | undefined, version: number, nodeId?: number): boolean {
+		return (
+			entry?.version === version &&
+			(entry.nodeId ?? 0) === (nodeId ?? getThisNodeId(auditStore)) &&
+			!entry.additionalAuditRefs?.length
+		);
+	}
+
 	function precedesExistingVersion(txnTime: number, existingEntry: Partial<Entry>, nodeId?: number): number {
 		if (nodeId === undefined) {
 			nodeId = getThisNodeId(auditStore);

package/server/DESIGN.md

@@ -73,6 +73,11 @@ A request entering `http.ts` does **not** go through Fastify unless no Harper ha
 > all. Only mutating an object _without_ changing its version can serve stale bytes, and that is
 > indistinguishable from re-sending the same message.
 
+> **Pass-through payloads are `Buffer | string`.** A message that carries its own `{ contentType, data }`
+> is delivered as its `data`, but binary data leaves `serializeMessage` as a `Buffer`: on a worker other
+> than the publisher's, the record is decoded from the transaction log and `data` is a plain `Uint8Array`,
+> which mqtt-packet cannot encode. Pinned by `integrationTests/mqtt/opaque-payload-across-workers.test.ts`.
+
 ### Threads
 
 | File                       | Purpose                                                                                                                                                                           |
@@ -353,10 +358,11 @@ The `@table(cacheControl:)` value is persisted on the primary-key attribute (lik
 
 A persisted QoS 1/2 entry is `{ qos, topic, startTime, databaseGeneration? }`, and the record carries the `incarnation` of the connection that owns it (harper#2448). A QoS 0 entry is `{ qos, topic }`: it is session state with no position, resumed live with retain handling 2 and no acknowledgement tracking.
 
-- **Positions are certified, never clocks.** A new subscription starts at its registration watermark (`registeredThrough`), bound to the database generation when one exists, and otherwise at `getNextMonotonicTime()` unbound until the first certified position replaces it. A SUBSCRIBE that replaces a QoS 1/2 topic's subscription continues that topic's position through the checked replay, since replacing must lose nothing; a pruned-history refusal there resets the session as it would at reconnect. A topic's position is `min(subscription.progress(), previousKey of its oldest unacknowledged delivery)`; a transaction's messages share a key, so one unacked message holds the position before its whole transaction, and while a message is still queued the position stays before the last delivered transaction, which that message may share. Live events arrive in commit order, and a RocksDB transaction takes its key before it commits, so a transaction can arrive after a higher key. Its previousKey is then the position just below its own key, and while such a late delivery is unacknowledged the position takes the lowest previousKey of all of them rather than the oldest's (harper#3101). Scan deliveries (`fromScan`) stay out of this ordering: they are retained state, which a resumed session does not resend.
+- **Positions are certified, never clocks.** A new subscription starts at its registration watermark (`registeredThrough`), bound to the database generation when one exists, and otherwise at `getNextMonotonicTime()` unbound until the first certified position replaces it. A SUBSCRIBE that replaces a QoS 1/2 topic's subscription continues that topic's position through the checked replay, since replacing must lose nothing; a pruned-history refusal there resets the session as it would at reconnect. A topic's position is `min(subscription.progress(), previousKey of its oldest unacknowledged delivery)`; a transaction's messages share a key, so one unacked message holds the position before its whole transaction, and while a message is still queued the position stays before the last delivered transaction, which that message may share. Live events arrive in commit order, and a RocksDB transaction takes its key before it commits, so a transaction can arrive after a higher key. Its previousKey is then the position just below its own key, and while such a late delivery is unacknowledged the position takes the lowest previousKey of all of them rather than the oldest's (harper#3101). Scan deliveries (`fromScan`) stay out of this ordering: they are retained state, which a resumed session does not resend: `resumeTopics` subscribes with retain handling 2 (`addSubscription` omits current values for that or an explicit `omitCurrent`), since a record topic whose saved position covers its current version would otherwise send that version again.
 - **Checkpoints:** coalesced after acknowledgements, on an unref'd timer (`min(auditRetention / 10, 1h)`), and at disconnect, where positions are taken before subscriptions end because `close()` clears a queue and would read as idle. Nothing past a resumed position is saved until that subscription's `resumeVerified` resolves true. Out-of-order acks are not persisted: a partly acked transaction is redelivered. Legacy `acks` lists are ignored.
 - **Before CONNACK,** a collection entry bound to its database's current generation is checked against the floor (a record entry's own walk decides after CONNACK); a failure deletes the record and reports the session as not present. The connection then writes its incarnation (`takeOver`), and every later write first checks the stored incarnation: a connection that finds another's stops writing and closes (0x8E), and one resumed from a record never creates one. Any newer authenticated connection for the client id on the same thread, a clean start included, closes the older durable connection the same way before it reads or deletes the record, once that connection has saved its positions, and waits for that save to settle; a connection that has already closed counts until its last save settles. A save that failed leaves the record at the last one that landed, so the newer connection resumes from there and may redeliver, never skip. Simultaneous CONNECTs for one client id take their turn through a per-thread claim, so each sees the one before it; one that gives up waiting keeps its place until the one before it finishes. The CONNECT's pending session is assigned before it waits, so packets sent right behind the CONNECT wait with it rather than being refused; the wait is bounded (10 s, then CONNACK 0x88, or 0x03 on v3.1.1), and a CONNECT whose client closes while it waits takes nothing over. A session that has ended, by a disconnect or by yielding to a takeover, saves nothing after its final save, so a SUBSCRIBE still in flight cannot write over the record a newer connection claimed.
 - **Subscription changes are serial.** Packets are handled concurrently, so a durable session applies SUBSCRIBE, UNSUBSCRIBE and `resume()` one at a time: a topic's state belongs to one subscription. A SUBSCRIBE of any QoS is saved before SUBACK. A failed SUBSCRIBE leaves its topic unsubscribed and saves that. It ends the subscription it started, and drops the state of the one it replaced, which replacing ended. An UNSUBSCRIBE whose save fails is reported failed again on a retry until the removal is saved, and any failed save starts the checkpoint timer, so a session with only QoS 0 topics retries it too.
+- **A delivery that cannot be sent ends the connection, not just the session.** The listener's `catch` in `mqtt.ts` aborts the transport (`destroy`/`terminate`, as for a parse error) and `onClose` ends the session, so the client sees the drop and reconnects. Ending only the session, as before, left the client connected and receiving nothing. An abort rather than a graceful end, because teardown on a graceful TCP end waits for the peer to close its side. A delivery that fails every time therefore repeats on each reconnect (a durable QoS≥1 session replays the unacknowledged message).
 - **Wills are kept per connection,** keyed `[clientId, incarnation]`: a connection closing after a takeover publishes and deletes only its own will (MQTT 5 §3.1.4 publishes a taken-over client's will), never the will of the connection that replaced it.
 - **Only a will's publish is authorized** (`settleWill`, harper#1874). The `hdb_session_will` row is the session's bookkeeping: it is read and deleted on contexts of its own, not the session's `authorize: true` context, which spends its one-shot check on the first call (roles are not granted system tables, so a non-super user's will never published and its row leaked). The publish then runs on a fresh session context, as the connection's user or a scoped will's embedded role, and the row is deleted after any outcome, a refused or failed publish included, so a will that fails for a transient reason is lost: the boot sweep, the only retry path, would replay a kept row only at the next restart, with no liveness check (harper#3034). The publish commits before the delete, so a crash between them leaves the row for that sweep and the will may be published twice (by design; untested). `unitTests/server/sessionWill.test.js` and `integrationTests/mqtt/will-authorization.test.ts` enforce that only the publish is authorized and that the row is deleted after a publish, a refusal and a clean DISCONNECT.
 - **One terminal path** (`subscriptionFailed`): a refused resume subscribe, a final `Error` value (observed through the subscription's `close` as well as the consumer loop, which may be parked on socket back-pressure), or an iterator failure. Only a 410 (pruned history) discards the session — writers fenced, siblings ended, a v5 DISCONNECT (0x83, the client-error message when problem information is allowed) and a socket close that never waits on storage, then a deletion only if the record still carries this incarnation. Anything else, a 409 from a restored or copied database included, saves positions and closes the connection so the client resumes.

package/server/DurableSubscriptionsSession.ts

@@ -254,13 +254,13 @@ class SubscriptionsSession {
 		if (path.indexOf('.') > -1) throw new Error('Dots are not allowed in topic names');
 		// might be faster to somehow modify existing subscription and re-get the retained record, but this should work for now
 		const existingSubscription = this.subscriptions.find((subscription) => subscription.topic === topic);
-		let omitCurrent;
+		let omitCurrent = subscriptionRequest.omitCurrent === true;
 		if (existingSubscription) {
-			omitCurrent = retainHandling > 0;
+			omitCurrent ||= retainHandling > 0;
 			existingSubscription.end();
 			this.subscriptions.splice(this.subscriptions.indexOf(existingSubscription), 1);
 		} else {
-			omitCurrent = retainHandling === 2;
+			omitCurrent ||= retainHandling === 2;
 		}
 		if (startTime) trace('Resuming subscription from', topic, 'from', startTime);
 		const entry = resources.getMatch(path, 'mqtt');
@@ -672,11 +672,8 @@ export class DurableSubscriptionsSession extends SubscriptionsSession {
 			const { qos, topic, startTime, databaseGeneration } = state.entry;
 			const durable = qos > 0;
 			try {
-				// retain handling, not omitCurrent, decides whether a subscription starts with current values
 				await this.resumeSubscription(
-					durable
-						? { omitCurrent: true, topic, qos, startTime, databaseGeneration, reportProgress: true }
-						: { topic, qos, rh: 2 },
+					durable ? { topic, qos, rh: 2, startTime, databaseGeneration, reportProgress: true } : { topic, qos, rh: 2 },
 					durable
 				);
 			} catch (error) {

package/server/mqtt.ts

@@ -329,6 +329,8 @@ function onSocket(socket, send, request, user, mqttSettings) {
 		}
 	}
 	let disconnected;
+	// set by a failed delivery, so sibling subscriptions stop writing before the aborted socket's 'close' arrives
+	let aborted = false;
 	numberOfConnections++;
 	const connection = {
 		protocolVersion: () => mqttOptions.protocolVersion,
@@ -557,9 +559,12 @@ function onSocket(socket, send, request, user, mqttSettings) {
 					});
 					const listener = async (topic, message, messageId, subscription, version) => {
 						try {
-							if (disconnected) {
-								// a delivery already past the subscription when the client left; the disconnect has ended the session
-								mqttLog.debug?.(`Session ${session?.sessionId} disconnected while trying to send message to ${topic}`);
+							if (disconnected || aborted) {
+								// a delivery already past the subscription when the client left or a failed delivery aborted
+								// the connection; onClose ends the session
+								mqttLog.debug?.(
+									`Session ${session?.sessionId} ${aborted ? 'aborted' : 'disconnected'} while trying to send message to ${topic}`
+								);
 								return false;
 							}
 							const slashIndex = topic.indexOf('/', 1);
@@ -569,11 +574,12 @@ function onSocket(socket, send, request, user, mqttSettings) {
 							// payload is encoded once per (message, content type) and reused across all of them.
 							const encoding = getSharedMessageEncoding(message, request, version);
 							const encoded = encoding.payload;
+							let payload: Buffer | string;
 							// only pay for a microtask when the serialization is genuinely still pending
-							const payload =
-								typeof (encoded as any)?.then === 'function'
-									? await resolveSharedPayload(encoding, message, request, version)
-									: (encoded as Buffer | string);
+							if (typeof (encoded as any)?.then === 'function') {
+								payload = await resolveSharedPayload(encoding, message, request, version);
+								if (disconnected || aborted) return false;
+							} else payload = encoded as Buffer | string;
 							if (qos > 0) {
 								// mqtt-packet requires a numeric message identifier once qos is non-zero
 								const packetData: any = {
@@ -609,9 +615,11 @@ function onSocket(socket, send, request, user, mqttSettings) {
 							return !rawSocket.closed;
 						} catch (error) {
 							mqttLog.error?.(error);
-							session?.disconnect(false);
-							mqttSettings.sessions.delete(session);
-							releaseClientSession(session);
+							// Ending only the session would leave the client connected and receiving nothing. An abort,
+							// unlike a graceful end, closes without waiting on the peer, and onClose ends the session.
+							aborted = true;
+							if (socket.destroy) socket.destroy();
+							else socket.terminate();
 							return false;
 						}
 					};

package/server/serverHelpers/contentTypes.ts

@@ -536,7 +536,14 @@ export function serializeMessage(
 	request?: Request,
 	inAsyncContinuation?: boolean
 ): Buffer | string | Promise<Buffer | string> {
-	if (message?.contentType != null && message.data != null) return message.data;
+	if (message?.contentType != null && message.data != null) {
+		const data = message.data;
+		// a record read on another thread decodes binary as a plain Uint8Array view, which protocol
+		// encoders (mqtt-packet) reject; wrap the same bytes rather than copy them
+		return ArrayBuffer.isView(data) && !Buffer.isBuffer(data)
+			? Buffer.from(data.buffer, data.byteOffset, data.byteLength)
+			: data;
+	}
 	asyncSerializations = inAsyncContinuation ? undefined : [];
 	try {
 		const serialized: Buffer | string = getMessageSerializer(request)(message);

package/server/serverHelpers/sharedMessageEncoding.ts

@@ -162,8 +162,9 @@ function fill(encoding: SharedMessageEncoding, message: any, request: any) {
 	const payload = serializeMessage(message, request);
 	encoding.payload = payload;
 	encoding.failed = false;
-	// A pass-through payload IS the message's own data, already reachable from the WeakMap key, so
-	// caching it retains nothing extra and must not be billed against the budget.
+	// A pass-through payload IS the message's own data (or a Buffer over the same memory), already
+	// reachable from the WeakMap key, so caching it retains nothing extra and must not be billed
+	// against the budget.
 	if (encoding.serializer === PASS_THROUGH) return;
 	if (typeof (payload as any)?.then === 'function')
 		// collapse an async serialization in place so later subscribers, and frame sharing, see the