@harperfast/harper-pro 5.0.16 → 5.0.18

package/core/DESIGN.md

@@ -31,3 +31,35 @@ The mitigations live in three places:
 - `LMDBTransaction.abort` and `DatabaseTransaction.abort` walk all writes and run the same cleanup unconditionally (regardless of `skipped`), since nothing was committed. `DatabaseTransaction.commit` adds an explicit reject handler so a `Promise.all` failure on `completions` (e.g. a blob save errored) aborts the underlying transaction instead of leaking it _and_ the blob files.
 
 When adding a new commit-handler early-return path: reset `write.skipped = false` at the top of the handler if you don't already, then set `write.skipped = true` immediately before the `return`. Decide first whether the audit log will reference the blob (via `auditRecordToStore`) — if it does, leave `skipped` unset. `cleanupOrphans` is the periodic safety net; don't rely on it for transactional correctness.
+
+## Schema migration and `runIndexing` internals (`databases.ts`)
+
+When `table()` is called with an attribute newly marked `indexed: true` (or with any change that requires re-building the secondary index), `runIndexing` is launched asynchronously and `Table.indexingOperation` is set to its promise. While running:
+
+**In-flight state tracking (persisted to `attributesDbi`):**
+
+- `attribute.indexingPID = process.pid` — set at migration start; cleared on clean completion. On restart with a different PID, `indexingPID !== process.pid` triggers a re-migration.
+- `attribute.lastIndexedKey` — updated every 100 records as a resumable checkpoint. Cleared on clean completion; preserved on error so a retry starts from this key.
+- `attribute.indexingFailed = true` — set if any record's `index.put` errors during the backfill. `table()` checks this flag: a fresh call in the same or a new process re-triggers the backfill from `lastIndexedKey`.
+- `dbi.isIndexing = true` — in-memory flag on the index dbi. Prevents `searchByIndex` from serving partial results (returns 503 "not indexed yet" instead). Cleared only when backfill completes cleanly.
+
+**`isIndexing` propagation across `resetDatabases()` calls:**
+When `signalSchemaChange('schema-change')` fires at the start of `runIndexing`, `syncSchemaMetadata` calls `resetDatabases()` which re-opens all tables via `table()`. This creates a _new_ dbi object and assigns it to `Table.indices[attribute.name]`. The condition `if (attributeDescriptor?.indexingPID) dbi.isIndexing = true` (just before `indices[name] = dbi` in the migration-detection block) ensures any dbi created while a migration is in progress also has `isIndexing = true`. Without this, a concurrent `resetDatabases()` would replace the in-progress dbi with a fresh one where `isIndexing` is false, allowing queries to read partial index results.
+
+**Error handling:**
+
+- Per-record sync errors: caught by the inner try-catch. Set `hadIndexingErrors = true`.
+- Per-record async rejections (`index.put` returning a rejected Promise): caught by the `when()` error handler. Set `hadIndexingErrors = true`.
+- The final `await lastResolution` is wrapped in its own try-catch because if the very last put in the loop was rejected, an unguarded `await lastResolution` would throw past the `hadIndexingErrors` check to the outer catch, silently bypassing the error path.
+- On any error: `indexingFailed = true` is persisted; `indexingPID`, `isIndexing`, and `lastIndexedKey` are kept. This leaves the index in 503 "incomplete" state rather than silently serving partial results.
+
+**`Object.defineProperty(attribute, 'dbi', ...)` must use `configurable: true`:**
+`attribute.dbi` is defined as a non-enumerable property (to prevent serialization to `attributesDbi`). It is defined with `configurable: true` so it can be re-assigned if the attribute participates in a retry cycle in the same process.
+
+## Audit-store `'committed'` notification batching (`transactionBroadcast.ts`)
+
+The cross-thread subscription path (default `crossThreads`) drives every `Table.subscribe()` consumer. When the database's audit store emits `'committed'`, we walk the audit log via a reusable iterator and dispatch matching records to subscribers. Three properties of this path are easy to break and worth knowing about before changing it:
+
+- **`databaseSubscriptions.activeCount`** is the count of live `Subscription` instances on a database. It is incremented at the end of `addSubscription` (after the Subscription is created, so the `scope: 'full-database'` early-return path correctly skips counting) and decremented in `Subscription.end()`. `notifyFromTransactionData` short-circuits when this is zero — the reusable rocksdb iterator stays put and resumes from its position the next time a subscriber arrives. Without this short-circuit, an idle database with no subscribers still pays the audit-log iteration cost on every commit during replication backlog catch-up.
+- **`notifyScheduled` + `setImmediate`** in the `'committed'` listener defers the iteration off the commit microtask. Multiple `'committed'` events that land in the same event-loop turn collapse into one notify pass. `notifyScheduled` stays set for the entire drain — including across yield-and-resume turns — so a re-entry from a new `'committed'` event cannot spawn a second concurrent notify on the same iterator.
+- **Batched yielding** in `notifyFromTransactionData` (`NOTIFY_BATCH_SIZE`) is gated by `allowYield`. The `'committed'` path passes `allowYield = true`; the `listenToCommits` (same-thread `aftercommit`) path does not, because that path holds an inter-thread `'thread-local-writes'` lock that must not span event-loop turns. `subscribersWithTxns` is carried across yields via `subscriptions.pendingTxnSubscribers` so the `end_txn` signal fires exactly once when the iterator truly drains. When `activeCount` drops to zero mid-yield, the next continuation drops the carry-over to avoid invoking ended subscribers' listeners.

package/core/bin/copyDb.ts

@@ -385,6 +385,16 @@ async function copyDbToRocks(sourceRootStore, sourceDatabase: string, targetPath
 		name: INTERNAL_DBIS_NAME,
 	});
 
+	const STRUCTURES_KEY = Symbol.for('structures');
+	const copyStructures = (sourceDbi, storeName: string) => {
+		const buffer = sourceDbi.getBinary?.(STRUCTURES_KEY);
+		if (buffer) {
+			targetRootStore.putSync([STRUCTURES_KEY, storeName], asBinary(buffer));
+		}
+	};
+
+	copyStructures(sourceDbisDb, INTERNAL_DBIS_NAME);
+
 	let written;
 	let outstandingWrites = 0;
 	const transaction = sourceDbisDb.useReadTransaction();
@@ -414,6 +424,8 @@ async function copyDbToRocks(sourceRootStore, sourceDatabase: string, targetPath
 				existingEncoder.getStructures = tempEncoder.getStructures;
 			}
 
+			copyStructures(sourceDbi, key);
+
 			console.log('migrating', key, 'from', sourceDatabase, 'to RocksDB');
 			await copyDbiToRocks(sourceDbi, targetDbi, isPrimary, transaction);
 		}
@@ -457,6 +469,10 @@ async function copyDbToRocks(sourceRootStore, sourceDatabase: string, targetPath
 					} of sourceDbi.getRange({ start, transaction, versions: true })) {
 						try {
 							start = key;
+							if (typeof key === 'symbol') {
+								skippedRecord++;
+								continue;
+							}
 							if (value == null) {
 								skippedRecord++;
 								continue;
@@ -497,6 +513,9 @@ async function copyDbToRocks(sourceRootStore, sourceDatabase: string, targetPath
 					for (const { key, value } of sourceDbi.getRange({ start, transaction })) {
 						try {
 							start = key;
+							if (typeof key === 'symbol') {
+								continue;
+							}
 							written = targetDbi.put(key, value);
 							recordsCopied++;
 							if (transaction.openTimer) transaction.openTimer = 0;

package/core/resources/RecordEncoder.ts

@@ -195,20 +195,23 @@ export class RecordEncoder extends Encoder {
 		const superGetStructures = this.getStructures;
 		this.saveStructures = function (structures, isCompatible): boolean | undefined {
 			if (this.isRocksDB) {
-				return this.rootStore.transactionSync((txn) => {
-					const sharedStructuresKey = [Symbol.for('structures'), this.name];
-					const existingStructuresBuffer = txn.getBinarySync(sharedStructuresKey);
-					const existingStructures = existingStructuresBuffer ? this.decode(existingStructuresBuffer) : undefined;
-					if (typeof isCompatible == 'function') {
-						if (!isCompatible(existingStructures)) {
+				return this.rootStore.transactionSync(
+					(txn) => {
+						const sharedStructuresKey = [Symbol.for('structures'), this.name];
+						const existingStructuresBuffer = txn.getBinarySync(sharedStructuresKey);
+						const existingStructures = existingStructuresBuffer ? this.decode(existingStructuresBuffer) : undefined;
+						if (typeof isCompatible == 'function') {
+							if (!isCompatible(existingStructures)) {
+								return false;
+							}
+						} else if (existingStructures && existingStructures.length !== isCompatible) {
 							return false;
 						}
-					} else if (existingStructures && existingStructures.length !== isCompatible) {
-						return false;
-					}
-					txn.putSync(sharedStructuresKey, structures);
-					this.structureUpdate = structures;
-				});
+						txn.putSync(sharedStructuresKey, structures);
+						this.structureUpdate = structures;
+					},
+					{ retryOnBusy: true }
+				);
 			} else {
 				const result = superSaveStructures.call(this, structures, isCompatible);
 				this.structureUpdate = structures;

package/core/resources/RocksTransactionLogStore.ts

@@ -5,6 +5,7 @@ import { Decoder, readAuditEntry, ENTRY_DATAVIEW, AuditRecord, createAuditEntry
 import { isMainThread } from 'node:worker_threads';
 import { EventEmitter } from 'node:events';
 import { asBinary } from 'lmdb';
+import * as harperLogger from '../utility/logging/harper_logger.ts';
 
 if (!process.env.HARPER_NO_FLUSH_ON_EXIT && isMainThread) {
 	// we want to be able to test log replay
@@ -288,6 +289,7 @@ export class RocksTransactionLogStore extends EventEmitter {
 			iterable.iterate = () => aggregateIterator;
 		}
 		const mappedAggregateIterable = iterable.map(({ timestamp, data, endTxn }: TransactionEntry) => {
+<<<<<<< HEAD
 			const decoder = new Decoder(data.buffer, data.byteOffset, data.byteLength);
 			data.dataView = decoder;
 			// This represents the data that shouldn't be transferred for replication
@@ -311,6 +313,55 @@ export class RocksTransactionLogStore extends EventEmitter {
 			auditRecord.previousVersion = previousVersion;
 			auditRecord.structureVersion = structureVersion & 0x00ffffff;
 			return auditRecord;
+=======
+			// Per-entry try/catch: a corrupt rocks prelude (first 4-16 bytes) would otherwise
+			// throw a raw `RangeError: Offset is outside the bounds of the DataView` out
+			// through `iterable.map`, escape the for-of consumer, and land as an
+			// uncaughtException on a later tick — stalling outgoing replication at the
+			// failing offset on every catch-up attempt. On error, yield a sentinel record
+			// with the timestamp preserved so iteration advances past the bad entry;
+			// downstream consumers already skip records with no `tableId`/`type`.
+			try {
+				const decoder = new Decoder(data.buffer, data.byteOffset, data.byteLength);
+				(data as any).dataView = decoder;
+				// This represents the data that shouldn't be transferred for replication
+				let structureVersion = decoder.getUint32(0);
+				let position = 4;
+				let previousResidencyId: number;
+				let previousVersion: number;
+				if (structureVersion & HAS_PREVIOUS_RESIDENCY_ID) {
+					previousResidencyId = decoder.getUint32(position);
+					position += 4;
+				}
+				if (structureVersion & HAS_PREVIOUS_VERSION) {
+					// does previous residency id and version actually require separate flags?
+					previousVersion = decoder.getFloat64(position);
+					position += 8;
+				}
+				const auditRecord = readAuditEntry(data, position, undefined);
+				auditRecord.version = timestamp;
+				auditRecord.endTxn = endTxn;
+				auditRecord.previousResidencyId = previousResidencyId;
+				auditRecord.previousVersion = previousVersion;
+				auditRecord.structureVersion = structureVersion & 0x00ffffff;
+				return auditRecord;
+			} catch (error) {
+				harperLogger.error('Failed to decode rocks transaction log entry; skipping', error, {
+					timestamp,
+					byteLength: data?.byteLength,
+				});
+				return {
+					version: timestamp,
+					endTxn,
+					type: undefined,
+					tableId: undefined,
+					recordId: undefined,
+					getValue: () => undefined,
+					getBinaryValue: () => undefined,
+					getBinaryRecordId: () => undefined,
+				} as unknown as AuditRecord;
+			}
+>>>>>>> b84fbbd (fix: skip corrupt audit entries during iteration instead of throwing)
 		});
 		// Add methods to the mapped iterable if we have an aggregate iterator
 		if (aggregateIterator?.addLog) {

package/core/resources/Table.ts

@@ -805,23 +805,23 @@ export function makeTable(options) {
 		/**
 		 * Set TTL expiration for records in this table. On retrieval, record timestamps are checked for expiration.
 		 * This also informs the scheduling for record eviction.
-		 * @param expirationTime Time in seconds until records expire (are stale)
-		 * @param evictionTime Time in seconds until records are evicted (removed)
+		 * @param opts Time in seconds until records expire, or an options object with `expiration`, `eviction`,
+		 * and `scanInterval` (all in seconds, all optional). Number form preserves any previously configured
+		 * eviction/scanInterval; object form replaces all three.
 		 */
-		static setTTLExpiration(expiration: number | { expiration: number; eviction?: number; scanInterval?: number }) {
-			// we set up a timer to remove expired entries. we only want the timer/reaper to run in one thread,
-			// so we use the first one
-			if (typeof expiration === 'number') {
-				expirationMs = expiration * 1000;
-				if (!evictionMs) evictionMs = 0; // by default, no extra time for eviction
-			} else if (expiration && typeof expiration === 'object') {
-				// an object with expiration times/options specified
-				expirationMs = expiration.expiration * 1000;
-				evictionMs = (expiration.eviction || 0) * 1000;
-				cleanupInterval = expiration.scanInterval * 1000;
-			} else throw new Error('Invalid expiration value type');
+		static setTTLExpiration(opts: number | { expiration?: number; eviction?: number; scanInterval?: number }) {
+			if (opts == null || (typeof opts !== 'number' && typeof opts !== 'object'))
+				throw new Error('Invalid expiration value type');
+			if (typeof opts === 'number') {
+				expirationMs = opts * 1000;
+			} else {
+				// `??` so an explicit 0 is treated as the user's chosen value, not as "missing"
+				expirationMs = (opts.expiration ?? 0) * 1000;
+				evictionMs = (opts.eviction ?? 0) * 1000;
+				cleanupInterval = (opts.scanInterval ?? 0) * 1000;
+			}
 			if (expirationMs < 0) throw new Error('Expiration can not be negative');
-			// default to one quarter of the total eviction time, and make sure it fits into a 32-bit signed integer
+			// default to one quarter of the total expiration+eviction window
 			cleanupInterval = cleanupInterval || (expirationMs + evictionMs) / 4;
 			scheduleCleanup();
 		}
@@ -4245,6 +4245,8 @@ export function makeTable(options) {
 									Boolean(invalidated),
 									auditRecord
 								);
+								// arm the eviction scanner, mirroring the .put() path
+								if (sourceContext.expiresAt) scheduleCleanup();
 							} else if (existingEntry) {
 								logger.trace?.(
 									`Deleting resolved record from source with id: ${id}, timestamp: ${new Date(txnTime).toISOString()}`

package/core/resources/auditStore.ts

@@ -49,7 +49,15 @@ export type AuditRecord = {
 	previousNodeId?: number;
 	previousAdditionalAuditRefs?: Array<{ version: number; nodeId: number }>;
 	endTxn?: boolean;
+<<<<<<< HEAD
 	structureVersion?: number;
+=======
+	getBinaryRecordId?: any;
+<<<<<<< HEAD
+	corrupt?: boolean;
+>>>>>>> b84fbbd (fix: skip corrupt audit entries during iteration instead of throwing)
+=======
+>>>>>>> 6b6192c (test: cover lmdb keyEncoder and rocks-prelude paths; drop unused corrupt flag)
 };
 
 const ENTRY_HEADER = Buffer.alloc(2816); // this is sized to be large enough for the maximum key size (1976) plus large usernames. We may want to consider some limits on usernames to ensure this all fits
@@ -73,6 +81,16 @@ export const transactionKeyEncoder = {
 		if (buffer[start] === 66) {
 			const dataView =
 				buffer.dataView || (buffer.dataView = new DataView(buffer.buffer, buffer.byteOffset, buffer.byteLength));
+			// Without this bounds check, a truncated key buffer escapes as RangeError up
+			// through lmdb-js's iterator and lands as an uncaughtException on a later tick,
+			// stalling outgoing replication for the affected (peer, db) pair.
+			if (start + 8 > buffer.byteLength) {
+				harperLogger.warn('Audit key buffer too short for float64 read; returning NaN sentinel', {
+					start,
+					byteLength: buffer.byteLength,
+				});
+				return NaN;
+			}
 			return dataView.getFloat64(start);
 		} else {
 			return readKey(buffer, start, end);
@@ -439,6 +457,15 @@ export function readAuditEntry(buffer: Uint8Array, start = 0, end = undefined):
 		const nodeId = decoder.readInt();
 		const tableId = decoder.readInt();
 		let length = decoder.readInt();
+		// A corrupt length field (e.g., a 0xff-prefixed uint32) would otherwise push
+		// decoder.position hundreds of megabytes past the buffer; the next readFloat64
+		// then throws with the bogus position in the message. Failing fast here keeps
+		// the throw inside this try/catch so we surface a sentinel instead.
+		if (length < 0 || decoder.position + length > buffer.byteLength) {
+			throw new RangeError(
+				`Audit entry recordId length ${length} exceeds remaining buffer (position ${decoder.position}, byteLength ${buffer.byteLength})`
+			);
+		}
 		const recordIdStart = decoder.position;
 		const recordIdEnd = (decoder.position += length);
 		// TODO: Once we support multiple format versions, we can conditionally read the version (and the previousResidencyId)
@@ -469,6 +496,11 @@ export function readAuditEntry(buffer: Uint8Array, start = 0, end = undefined):
 			}
 		}
 		length = decoder.readInt();
+		if (length < 0 || decoder.position + length > buffer.byteLength) {
+			throw new RangeError(
+				`Audit entry username length ${length} exceeds remaining buffer (position ${decoder.position}, byteLength ${buffer.byteLength})`
+			);
+		}
 		const usernameStart = decoder.position;
 		const usernameEnd = (decoder.position += length);
 		let value: any;
@@ -477,8 +509,17 @@ export function readAuditEntry(buffer: Uint8Array, start = 0, end = undefined):
 			tableId,
 			nodeId,
 			get recordId() {
-				// use a subarray to protect against the underlying buffer being modified
-				return readKey(buffer.subarray(0, recordIdEnd), recordIdStart, recordIdEnd);
+				// The recordId is decoded lazily and lives outside readAuditEntry's try/catch,
+				// so a corrupt recordId region would otherwise escape as an uncaught RangeError
+				// on property access. Catch and return undefined; callers already treat missing
+				// recordId as a skip-eligible entry.
+				try {
+					// use a subarray to protect against the underlying buffer being modified
+					return readKey(buffer.subarray(0, recordIdEnd), recordIdStart, recordIdEnd);
+				} catch (error) {
+					harperLogger.warn('Failed to decode audit recordId; treating as corrupt', error);
+					return undefined;
+				}
 			},
 			getBinaryRecordId() {
 				return buffer.subarray(recordIdStart, recordIdEnd);
@@ -486,9 +527,14 @@ export function readAuditEntry(buffer: Uint8Array, start = 0, end = undefined):
 			version,
 			previousVersion,
 			get user() {
-				return usernameEnd > usernameStart
-					? readKey(buffer.subarray(0, usernameEnd), usernameStart, usernameEnd)
-					: undefined;
+				try {
+					return usernameEnd > usernameStart
+						? readKey(buffer.subarray(0, usernameEnd), usernameStart, usernameEnd)
+						: undefined;
+				} catch (error) {
+					harperLogger.warn('Failed to decode audit username; treating as corrupt', error);
+					return undefined;
+				}
 			},
 			get encoded() {
 				return start ? buffer.subarray(start, end) : buffer;
@@ -523,10 +569,56 @@ export function readAuditEntry(buffer: Uint8Array, start = 0, end = undefined):
 		};
 	} catch (error) {
 		harperLogger.error('Reading audit entry error', error, buffer);
+<<<<<<< HEAD
 		return {};
+=======
+		return createCorruptAuditSentinel(buffer, start, end);
+>>>>>>> b84fbbd (fix: skip corrupt audit entries during iteration instead of throwing)
 	}
 }
 
+/**
+ * Build a structurally complete audit record for an entry that failed to decode. The fields
+ * mirror the happy-path shape so downstream consumers that access (e.g.) `getValue` or the
+ * `recordId` getter don't blow up with a `TypeError: not a function` / `undefined.is(...)`
+ * after the header decode already failed. Consumers identify these by the undefined
+ * `tableId`/`type` (the same signal lmdb has produced from this catch since before this
+ * change) and skip them — `classifyAuditEntryForReplay` calls them out as `corrupt-header`,
+ * and the dispatch loops in Table.ts / transactionBroadcast.ts filter via tableId guards.
+ */
+function createCorruptAuditSentinel(buffer: Uint8Array, start: number, end: number | undefined): AuditRecord {
+	return {
+		type: undefined,
+		tableId: undefined,
+		nodeId: undefined,
+		recordId: undefined,
+		version: undefined,
+		previousVersion: undefined,
+		user: undefined,
+		extendedType: undefined,
+		residencyId: undefined,
+		previousResidencyId: undefined,
+		expiresAt: undefined,
+		originatingOperation: undefined,
+		previousAdditionalAuditRefs: undefined,
+		get encoded() {
+			return start ? buffer.subarray(start, end) : buffer;
+		},
+		get size() {
+			return start !== undefined && end !== undefined ? end - start : buffer.byteLength;
+		},
+		getBinaryRecordId() {
+			return undefined;
+		},
+		getValue() {
+			return undefined;
+		},
+		getBinaryValue() {
+			return undefined;
+		},
+	} as any;
+}
+
 export class Decoder extends DataView<ArrayBufferLike> {
 	position = 0;
 	readInt() {

package/core/resources/databases.ts

@@ -1063,6 +1063,7 @@ export function table<TableResourceType>(tableDefinition: TableDefinition): Tabl
 				const dbi = openIndex(dbiKey, rootStore, attribute);
 				if (
 					changed ||
+					attributeDescriptor.indexingFailed ||
 					(attributeDescriptor.indexingPID && attributeDescriptor.indexingPID !== process.pid) ||
 					attributeDescriptor.restartNumber < workerData?.restartNumber
 				) {
@@ -1071,6 +1072,7 @@ export function table<TableResourceType>(tableDefinition: TableDefinition): Tabl
 					attributeDescriptor = attributesDbi.getSync(dbiKey);
 					if (
 						changed ||
+						attributeDescriptor.indexingFailed ||
 						(attributeDescriptor.indexingPID && attributeDescriptor.indexingPID !== process.pid) ||
 						attributeDescriptor.restartNumber < workerData?.restartNumber
 					) {
@@ -1084,14 +1086,20 @@ export function table<TableResourceType>(tableDefinition: TableDefinition): Tabl
 						if (hasExistingData) {
 							attribute.lastIndexedKey = attributeDescriptor?.lastIndexedKey ?? undefined;
 							attribute.indexingPID = process.pid;
+							delete attribute.indexingFailed; // clear failure flag for the new run
 							dbi.isIndexing = true;
-							Object.defineProperty(attribute, 'dbi', { value: dbi });
+							Object.defineProperty(attribute, 'dbi', { value: dbi, configurable: true, enumerable: false });
 							// we only set indexing nulls to true if new or reindexing, we can't have partial indexing of null
 							attributesToIndex.push(attribute);
 						}
 					}
 					attributesDbi.put(dbiKey, attribute);
 				}
+				// If a migration is in progress (indexingPID set), any newly opened dbi must also
+				// reflect isIndexing = true. A resetDatabases() during an active runIndexing creates
+				// a new dbi object; without this, queries could use the new dbi (isIndexing = false)
+				// and return incomplete results while the backfill is still running.
+				if (attributeDescriptor?.indexingPID) dbi.isIndexing = true;
 				if (attributeDescriptor?.indexNulls && attribute.indexNulls === undefined) attribute.indexNulls = true;
 				dbi.indexNulls = attribute.indexNulls;
 				indices[attribute.name] = dbi;
@@ -1162,6 +1170,7 @@ async function runIndexing(Table, attributes, indicesToRemove) {
 			lastResolution = index.drop();
 		}
 		let interrupted;
+		let hadIndexingErrors = false;
 		const attributeErrorReported = {};
 		let indexed = 0;
 		const attributesLength = attributes.length;
@@ -1215,6 +1224,7 @@ async function runIndexing(Table, attributes, indicesToRemove) {
 							}
 						}
 					} catch (error) {
+						hadIndexingErrors = true;
 						if (!attributeErrorReported[property]) {
 							// just report an indexing error once per attribute so we don't spam the logs
 							attributeErrorReported[property] = true;
@@ -1227,6 +1237,7 @@ async function runIndexing(Table, attributes, indicesToRemove) {
 					() => outstanding--,
 					(error) => {
 						outstanding--;
+						hadIndexingErrors = true;
 						logger.error(error);
 					}
 				);
@@ -1244,20 +1255,69 @@ async function runIndexing(Table, attributes, indicesToRemove) {
 				if (outstanding > MAX_OUTSTANDING_INDEXING) await lastResolution;
 				else if (outstanding > MIN_OUTSTANDING_INDEXING) await new Promise((resolve) => setImmediate(resolve)); // yield event turn, don't want to use all computation
 			}
+		}
+		// Await the last pending put. If it rejects, that is also an indexing error.
+		// Note: the when() calls above already attach rejection handlers to each record's
+		// last-put promise; this try-catch specifically handles the case where lastResolution
+		// itself rejects (i.e. the very last put in the loop failed) which would otherwise
+		// throw past the hadIndexingErrors check to the outer catch. The broader issue of
+		// unhandled rejections from non-last puts in multi-value attributes is pre-existing
+		// and out of scope for this fix.
+		try {
+			await lastResolution;
+		} catch (error) {
+			hadIndexingErrors = true;
+			logger.error(error);
+		}
+		// Yield one more event turn so any queued when() error callbacks (which fire as
+		// microtasks when their tracked promise settles) have a chance to set hadIndexingErrors
+		// before we decide whether to mark indexing as complete.
+		await new Promise((resolve) => setImmediate(resolve));
+		if (hadIndexingErrors) {
+			// Some records failed to index. Persist the failure marker in the descriptor so
+			// the next call to table() (including after a restart with a fresh PID) re-triggers
+			// the backfill from the last checkpoint. Do NOT clear indexingPID or isIndexing —
+			// leave the index in its incomplete state so queries return 503 "not indexed yet"
+			// rather than silently returning partial results. This is the key fix for the
+			// serent-canopy issue #135 fingerprint: a completed migration with transient errors
+			// (e.g. ERR_BUSY from RocksDB under load) leaving gaps while appearing successful.
+			for (const attribute of attributes) {
+				attribute.indexingFailed = true;
+				// Preserve lastIndexedKey so the retry resumes from the last checkpoint.
+				lastResolution = Table.dbisDB.put(attribute.key, attribute);
+				// Keep isIndexing = true on both the attribute.dbi and the currently-active dbi
+				// in Table.indices (which may differ if resetDatabases() ran during this pass).
+				attribute.dbi.isIndexing = true;
+				const activeDbi = Table.indices[attribute.name];
+				if (activeDbi) activeDbi.isIndexing = true;
+			}
+			await lastResolution;
+			logger.warn(
+				`Indexing of ${Table.tableName} encountered errors on some records - index will remain incomplete. ` +
+					`On next restart the migration will be retried from the last checkpoint (indexingFailed=true). ` +
+					`Affected attributes: ${attributes.map((a) => a.name).join(', ')}`
+			);
+		} else {
 			// update the attributes to indicate that we are finished
 			for (const attribute of attributes) {
 				delete attribute.lastIndexedKey;
 				delete attribute.indexingPID;
+				delete attribute.indexingFailed;
 				attribute.dbi.isIndexing = false;
+				// Also clear isIndexing on the currently-active dbi in Table.indices, which may
+				// differ from attribute.dbi if a resetDatabases() call during this migration
+				// opened a new dbi and registered it there.
+				const activeDbi = Table.indices[attribute.name];
+				if (activeDbi) activeDbi.isIndexing = false;
 				lastResolution = Table.dbisDB.put(attribute.key, attribute);
 			}
+			await lastResolution;
+			// now notify all the threads that we are done and the index is ready to use
+			await signalling.signalSchemaChange(
+				new SchemaEventMsg(process.pid, 'indexing-finished', Table.databaseName, Table.tableName)
+			);
+			logger.info(`Finished indexing ${Table.tableName} attributes`, attributes);
 		}
-		await lastResolution;
-		// now notify all the threads that we are done and the index is ready to use
-		await signalling.signalSchemaChange(
-			new SchemaEventMsg(process.pid, 'indexing-finished', Table.databaseName, Table.tableName)
-		);
-		logger.info(`Finished indexing ${Table.tableName} attributes`, attributes);
 	} catch (error) {
 		logger.error('Error in indexing', error);
 	}

package/core/resources/replayLogs.ts

@@ -6,6 +6,7 @@ import { DatabaseTransaction } from './DatabaseTransaction.ts';
 import { RocksTransactionLogStore } from './RocksTransactionLogStore.ts';
 import { isMainThread } from 'node:worker_threads';
 import { RequestTarget } from './RequestTarget.ts';
+import { classifyAuditEntryForReplay } from './replayLogsGuards.ts';
 
 let warnedReplayHappening = false;
 export function replayLogs(rootStore: RocksDatabase, tables: any): Promise<void> {
@@ -24,11 +25,26 @@ export function replayLogs(rootStore: RocksDatabase, tables: any): Promise<void>
 		let transaction: DatabaseTransaction;
 		let lastTimestamp = 0;
 		let writes = 0;
+		let skipped = 0;
 		const txnLog: RocksTransactionLogStore = rootStore.auditStore;
 		for (const auditRecord of txnLog.getRange({ startFromLastFlushed: true, readUncommitted: true })) {
-			const { type, tableId, nodeId, recordId, version, residencyId, expiresAt, originatingOperation, username } =
-				auditRecord;
+			const {
+				type,
+				tableId,
+				nodeId,
+				recordId,
+				version,
+				residencyId,
+				expiresAt,
+				originatingOperation,
+				username,
+				extendedType,
+			} = auditRecord;
 			try {
+				if (classifyAuditEntryForReplay(extendedType, tableId, true) === 'corrupt-header') {
+					skipped++;
+					continue;
+				}
 				const Table = tableById.get(tableId);
 				if (!Table) continue;
 				const context: Context = { nodeId, alreadyLogged: true, version, expiresAt, user: { name: username } };
@@ -42,7 +58,22 @@ export function replayLogs(rootStore: RocksDatabase, tables: any): Promise<void>
 					warnedReplayHappening = true;
 					console.warn('Harper was not properly shutdown, replaying transaction logs to synchronize database');
 				}
-				const record = auditRecord.getValue(primaryStore);
+				let record: any;
+				try {
+					record = auditRecord.getValue(primaryStore);
+				} catch {
+					// msgpack/structure decode failed for this entry's value. Skip rather than
+					// fall through to a guaranteed downstream crash, and intentionally drop the
+					// error: every corrupt entry would otherwise log a stack trace per iteration
+					// (millions of these were observed in prod). The total skip count is logged
+					// once at the end of replay.
+					skipped++;
+					continue;
+				}
+				if (classifyAuditEntryForReplay(extendedType, tableId, record !== undefined) === 'missing-record') {
+					skipped++;
+					continue;
+				}
 				if (lastTimestamp !== version) {
 					lastTimestamp = version;
 					try {
@@ -127,6 +158,8 @@ export function replayLogs(rootStore: RocksDatabase, tables: any): Promise<void>
 			logger.error('Error committing replay transaction', error);
 		}
 		if (writes > 0) logger.warn(`Replayed ${writes} records in ${rootStore.databaseName} database`);
+		if (skipped > 0)
+			logger.warn(`Skipped ${skipped} unrecoverable audit entries in ${rootStore.databaseName} database during replay`);
 		// we never actually release the lock because we only want to ever run one time
 		// rootStore.unlock('replayLogs');
 	});

package/core/resources/replayLogsGuards.ts

@@ -0,0 +1,42 @@
+// Pure helpers for replayLogs (no Harper module dependencies, so unit tests can load
+// them without bootstrapping the full Resource / RocksDB / DatabaseTransaction graph).
+//
+// Background: a node that crashed unclean re-runs replayLogs against the unflushed audit
+// log on next boot. If any audit entry is corrupt or missing its record body, the loop
+// hits a TypeError inside Table.validate() ("Cannot read properties of undefined
+// (reading 'cacheKey')") and the per-iteration catch swallows it — but the loop keeps
+// running over potentially millions of entries, pinning CPU. These guards classify each
+// entry up front so the loop can skip cleanly.
+
+// Mirrors `HAS_RECORD` (16) | `HAS_PARTIAL_RECORD` (32) from auditStore.ts — the action
+// bits the writer sets when an entry carries (or should carry) a record body. Redeclared
+// here so this module stays free of the Harper module graph for unit testing; a lock
+// test pins the value against auditStore so silent drift is caught.
+export const RECORD_BEARING_FLAGS = 16 | 32;
+
+/**
+ * Decide whether an audit entry pulled from the unflushed log is safe to replay.
+ * Returns `null` if the entry should be replayed, or a short reason string if it should
+ * be skipped (the loop logs the aggregate skip count once at the end).
+ *
+ * Operates on the raw integer `action` field rather than the decoded type string: when
+ * `readAuditEntry` catches a header decode error it returns `{}`, so both `action` and
+ * `tableId` are `undefined` — the same signal — and matching the record-bearing flags
+ * directly against the action mirrors how the writer set them in `auditStore.ts`.
+ *
+ * @param action      `auditRecord.extendedType` — the variable-length action field with
+ *                    the event type in the low nibble and HAS_* flags above it
+ * @param tableId     `auditRecord.tableId`
+ * @param hasRecord   `true` if `auditRecord.getValue(...)` produced a non-undefined value
+ */
+export function classifyAuditEntryForReplay(
+	action: number | undefined,
+	tableId: number | undefined,
+	hasRecord: boolean
+): 'corrupt-header' | 'missing-record' | null {
+	if (action === undefined || tableId === undefined) return 'corrupt-header';
+	// If the action advertises a record body but the decoded record is undefined, the
+	// downstream write path will crash inside validate() on the first attribute deref.
+	if ((action & RECORD_BEARING_FLAGS) !== 0 && !hasRecord) return 'missing-record';
+	return null;
+}