@harperfast/harper 5.3.0-beta.1 → 5.3.0-beta.2

package/config/DESIGN.md

@@ -0,0 +1,306 @@
+# config/ — Design notes
+
+Config composition, persistence, env layers and hot reload.
+
+**Read this when:** touching `configUtils.ts`, `readConfigFileSync.ts`, `harperConfigEnvVars.ts` or a root config watcher.
+
+Index of every design note: [DESIGN.md](../DESIGN.md).
+
+---
+
+## `set_configuration` replication is opt-in; `replicateOperation` is default-on (`config/configUtils.ts`)
+
+`server.replication.replicateOperation` (installed by harper-pro's replicator) fans out whenever
+`req.replicated \!== false` — absence of the flag means "replicate". That default-on contract is what
+DDL ops rely on (`dropSchema`/`dropTable` call it unconditionally), so a handler that mirrors the
+drop_schema pattern without a guard silently becomes replicate-by-default. `setConfiguration` must
+stay **opt-in** (`if (replicated)` truthy guard) because config bodies routinely carry node-local
+params (ports, paths, node identity) that would clobber peers. Two invariants to preserve:
+`replicated` must remain in the handler's destructure strip-list on both origin and peers (peers
+receive `replicated: false` in the forwarded body; anything not stripped is treated as a config
+param), and there is deliberately **no** per-param node-local/cluster-wide guard here — per-field
+replicability metadata is deferred to the cluster-level-config work (CORE-3018), which will own that
+schema. Per-peer failures never reject: they come back as `{status: 'failed', reason, node}` entries
+in `response.replicated[]`, and `message` still reads as success (same contract as drop_schema), so
+operators must inspect the array for per-node outcomes.
+
+## Root config watchers must read synchronously (`config/readConfigFileSync.ts`)
+
+`atomicWriteFile()` swaps the config file in with `renameSync` and, on Windows, retries the
+`EPERM`/`EACCES`/`EBUSY` a still-open destination handle produces — blocking the calling thread in
+`Atomics.wait`. The handle that blocks it belongs to the _process_, not to the thread that opened
+it: measured on `windows-latest`/Node 24 (harper#2313), a single Node **read** descriptor on the
+destination fails the rename, while `fs.watch` and chokidar's own handles do not.
+`set_configuration` reaches that loop from a live request thread, and every worker runs root config
+watchers over the same file, so an **async** read in a watcher is unsatisfiable by construction:
+libuv opens the descriptor on the threadpool but closes it from JS, which cannot run while the same
+thread is blocked in the retry loop. The worker then deadlocks against its own
+watcher and burns the entire budget before failing (harper#2191, reproduced by the Windows
+integration job). Both root watchers — `RootConfigWatcher.handleChange` and an `OptionsWatcher`
+explicitly identified as a root-config watcher — therefore go through `readConfigFileSync()`, which
+holds no descriptor across a yield. A component's own config remains asynchronous even if the
+component names it `harper-config.yaml` or `harperdb-config.yaml`. Do not "modernize" root-config
+reads back to `fsPromises.readFile`.
+
+Three constraints follow from it. The reader gates its retry to win32 (`isSharingViolation`); the
+writer does not (`configUtils`' `isRetryableRenameError`, same three codes, any platform). That
+asymmetry is deliberate: a misclassified read falls through to the timer ladder below and still
+recovers, a rename has nothing to fall through to, and `process.platform` does not answer the
+question that matters — whether this filesystem can replace an open file. A Linux worker whose
+rootPath sits on WSL drvfs, a CIFS/SMB mount, or a Docker Desktop bind mount reports `linux` and
+still returns these codes transiently.
+
+The reader's 500ms budget is one deadline **per path shared by all callers on the thread**, not per
+call — a worker holds one `OptionsWatcher` per root-declared plugin (10+ on a stock install,
+`TRUSTED_RESOURCE_PLUGINS`) over the same file, all reacting to a single change event, so a per-call
+budget would serialize into N x 500ms of blocked event loop whenever a writer's lock outlives it.
+
+Both watchers parse through `parseConfigFile()` (`config/parseConfigFile.ts`) rather than calling
+`yaml.parse` directly: yaml's `prettyErrors` frames the offending source lines into the error's
+`message`, and the root config holds credentials, so a parse failure would otherwise ship that
+frame to the component log (`OptionsWatcher` → `Scope`) or the config log.
+
+And a lock that outlives even that emits no new watcher event when it clears, so both watchers hand
+the failure to `ConfigReadRetry` (`config/configReadRetry.ts`) rather than going stale: retrying from
+a timer holds no descriptor either, so it cannot re-enter the deadlock. A ladder rung passes
+`waitForLock: false` — the ladder already owns the retry, and letting each rung re-enter the
+blocking budget would multiply one lock incident into a stall per rung. The ladder is bounded by
+wall clock and its backoff is derived from elapsed time rather than from how many times it was
+armed, because watcher callbacks and timer callbacks share one entry point: a rename burst delivers
+several chokidar events in milliseconds and would otherwise both spend the ladder and push the next
+rung out to the maximum before the writer has let go.
+
+A deletion supersedes the reads already in flight, so `OptionsWatcher.#handleUnlink` claims the
+current read sequence rather than only cancelling the ladder: an asynchronous rung completing after
+it would otherwise put the removed file's options back, or find ENOENT and report the same deletion
+a second time as a `remove` asking `Scope` to restart a scope that deletion just settled. That
+ordering cannot be staged from a real deletion — every technique that holds a `readFile` open past
+chokidar's `unlink` (threadpool saturation, a FIFO) holds the `unlink` behind it too, because
+chokidar's own event delivery needs the same threadpool; measured here, a saturated pool produced no
+`unlink` for at least 3 seconds. The regression therefore delivers the deletion through
+`_simulateUnlinkForTests`.
+
+### An empty read is a writer mid-write, not an empty config
+
+A non-atomic writer — an operator's editor, a shell redirect, anything that is not
+`atomicWriteFile()`'s temp-file-and-rename — truncates the config before it writes it, and the
+synchronous read is fast enough to land in that window where the async read never was. chokidar
+throttles change events per path for 50ms and _drops_ the throttled ones, so the event carrying the
+content is routinely discarded as a duplicate of the truncate's: an empty read that is discarded is
+the last read that config gets, and the thread holds the pre-truncate value indefinitely
+(`RootConfigWatcher`) or reports the scope as removed (`OptionsWatcher`). Both therefore hand an
+empty read to `ConfigReadRetry`, the same ladder a lock takes and for the same reason — there is no
+further event to re-read on. `OptionsWatcher` applies it on both read paths, not only the
+synchronous one: the asynchronous read is far less likely to land in a truncate window, but the
+consequence there is a spurious `remove` that tears the scope down.
+
+A read that _parses_ to nothing is the same event and takes the same ladder: a truncated document,
+a lone `\n` and a file of nothing but comments all yield `null` from the parser rather than
+throwing. `OptionsWatcher` judges that on the file's own parse, **before** `overlayRootEnvConfig`,
+which returns a non-null object whenever a config env var is set — the norm in containers — and
+would otherwise launder a half-written file into a valid-looking env-only config and wipe the
+file's own options.
+
+Past the ladder the emptiness is believed, and what that costs depends on whether the scope has
+settled: a worker still booting starts on the defaults, while one already running keeps the config
+it has and only warns. The asymmetry is deliberate in both halves — a running worker must not let a
+truncate window that outlived the ladder reset every scope, and a booting one must not hold
+`Scope.ready` open waiting for a file that is genuinely empty — but it does mean an operator who
+empties `harper-config.yaml` at runtime gets divergence between workers until the next restart.
+
+### `ready` means the watcher is armed
+
+`RootConfigWatcher.ready` is a startup barrier — `harper_logger`'s `updateLogSettings()` attaches
+its `change` listener only after awaiting it — so it has to mean "watching", not merely "the first
+read landed". The synchronous read would otherwise emit `ready` from inside chokidar's initial `add`
+dispatch, and on darwin FSEvents has not armed its stream at that point: a write in that window is
+dropped with no later event to recover it (the async read used to defer past it by a threadpool
+round-trip, which is why this surfaced only when the read went synchronous). Measured on the
+harper#2191 review head, writing that far after `ready`: 0ms is lost, 5ms and beyond is delivered.
+
+So `ready` is gated on chokidar's own `ready` — its initial scan has established the native
+watches by then — plus a darwin-only grace over that measurement for the kernel-side warm-up
+chokidar cannot observe. Neither half is sufficient alone: chokidar's event still lands inside the
+warm-up, and a bare timer could elapse before the scan has created any watch. Config read before
+that gate opens is staged into `#config`, re-read once the gate opens — a write that landed while
+the watch was unarmed produced no event, so nothing else would ever deliver it — and then handed to
+`ready` itself rather than to a `change` that would precede it.
+
+`OptionsWatcher` shares the gate (`ArmGate`, `config/watcherArming.ts`) because it has the same
+unarmed window and, for the root config, many more of them: `componentLoader` gives every
+`TRUSTED_RESOURCE_PLUGINS` entry its own root-config `OptionsWatcher`, and those read synchronously.
+It shares the arming **re-read**, which is what recovers the otherwise-undeliverable write, but not
+the barrier: its `ready` still goes out on the first read, so it means "the config has been read",
+not "armed". The difference is only ordering, because unlike `harper_logger` its consumer (`Scope`)
+attaches `change`/`remove`/`ready` listeners in its constructor, before any read — so a write made
+in the unarmed window reaches the scope as a post-`ready` `change` (and, for a plugin that doesn't
+handle its own options, a restart) rather than being lost. Holding `OptionsWatcher.ready` behind
+arming as well would need every terminal outcome to open a second barrier, per scope, with a boot
+hang as the failure mode; the ordering is not worth that.
+
+Whether a scope is configured is tracked separately from its value, because neither truthiness nor
+`!== undefined` can answer it: `myPlugin:` with no body is a configured scope whose value is `null`,
+and a boot that found no config of its own holds `DEFAULT_CONFIG[name]` — a value the watcher gave
+itself. Reading either as "the file supplied this" costs a restart: for the six scopes
+`DEFAULT_CONFIG` names, the next read of an unchanged file looks like the block being deleted, and
+filling in an empty block looks like the unconfigured → configured transition `Scope` answers by
+restarting rather than the `change` it is.
+
+What the arming re-read must _not_ do is report a deletion. Its job is the write no event carried;
+a file that is gone is chokidar's `unlink` to report, and answering the re-read's `ENOENT` with
+`remove` announces it ahead of the event that would confirm it — where there is a grace, ahead of
+chokidar having finished tearing the watch down, so a config recreated on the strength of that
+early `remove` lands in a window where its `add` is not observed at all and the scope keeps the
+defaults with nothing further coming. Settling a barrier that has nothing applied yet is still the
+arming re-read's job: an absent file at boot is the install window, not a deletion.
+
+### Every terminal read outcome settles the barrier
+
+Both barriers — `RootConfigWatcher.ready` and, through `Scope`, `OptionsWatcher.ready` — are
+awaited with no timeout, so a read that ends without a config must still settle them or the worker
+hangs at boot rather than failing. Every terminal outcome therefore boots on defaults and logs what
+failed: a read the ladder could not complete, a file still empty when the ladder is spent, and a
+file that will not parse. Only a config that parses is a config; the alternative, failing the boot
+closed on an unreadable file, is a different policy than the one `OptionsWatcher` already applies to
+its ENOENT and read-failure paths, and the two watchers must not disagree about it. A file that
+becomes readable later still arrives, as a `change`.
+
+A missing file is not one of those outcomes to wait on: `ENOENT` is not a sharing violation, so
+neither watcher takes the retry ladder for it. `OptionsWatcher` has always settled it at once as
+the install window, and `RootConfigWatcher` does the same rather than spending the whole read
+budget inside `harper_logger.start()` on every boot that has no config file — an env-var-only
+deployment, or a rootPath mounted empty. Neither is a deletion an outcome to wait on. `OptionsWatcher.#handleUnlink` cancels the ladder —
+the deletion settles what a pending read was retrying — so when that read had not produced a config
+yet, the ladder it cancels was the only thing left to settle `ready`. Before the first `ready` there
+is also nothing to remove and nothing to hear it: `Scope` is still inside `await scope.ready`, so a
+`remove` there asks for a restart of a component that never booted. A deletion in the boot window
+therefore settles the barrier on the defaults, exactly as the ENOENT read path does; only a deletion
+after `ready` reports `remove`. A watcher error is terminal for the barrier too, and
+settling it is what removes the `error` listener `once(this, 'ready')` attached — so reporting the
+failure afterwards has to check for a listener rather than assume one, or an unlistened `error`
+throws out of chokidar's dispatch and takes the worker down over a fault it just decided to survive.
+
+An env-compose failure rides that settle rather than preceding it: `#envComposeError` is reported
+only after the barrier has settled, because an `error` emitted first rejects `once(this, 'ready')`
+instead of settling it. It is set and reported inside one synchronous call chain, the arming path
+included: that path defers an absence check rather than reporting a removal, and it drops the
+failure before deferring rather than reporting it there. Reporting would duplicate — every
+resolution of the deferral recomposes and reports the env state it finds — and holding it would
+carry a failure that may no longer be true onto whatever event reports next. So the early returns
+taken when the env-only overlay _succeeded_ cannot be carrying one, and hoisting the report ahead of
+them for symmetry would put it back before the settle on the paths this ordering exists for.
+
+What a scope does about a config that arrives late is the other half of settling early.
+`OptionsWatcher.ready` is not once-per-watcher: it fires whenever a scope goes from having no
+config of its own to having one, which is both the recreated-config-file path and a scope that
+booted while the file was unreadable. Nothing downstream re-runs on it — `componentLoader` is long
+past its `await scope.ready` — so `Scope` answers a repeat `ready` the same way it answers `remove`,
+by requesting a restart. Without that, one worker keeps serving the defaults while every worker
+that read the file cleanly serves the operator's config.
+
+Arming is a terminal outcome of its own: chokidar reports a scan that found no file by emitting
+`ready` and nothing else, so `RootConfigWatcher` always re-reads when the gate opens rather than
+publishing what an earlier read staged — a missing config file takes the ladder and settles on the
+defaults instead of holding the barrier open. That fallback must also discard the staged value:
+the arming re-read is authoritative precisely because a write in the unarmed window may have
+superseded it, including by replacing the file with an unusable or missing one. A watcher scan error
+also settles the barrier, but preserves a successfully staged value because no read superseded it.
+`close()` settles the barrier as well.
+
+What settles the barrier is not the same as what the settled value may be _used_ as. A read that
+carried no config settles it carrying nothing — not `{}`, which is a configuration that a consumer
+cannot tell apart from one the file really held, and `updateLogger` reads an absent `rotation` as
+rotation off and an absent `console` as console off. `updateLogSettings()` therefore keeps what
+`initLogSettings()` established until a real config arrives, rather than silently turning logging
+off on the very boot that could not read its configuration.
+
+## Config is composed and memoized before any component runs (`config/configUtils.ts`)
+
+`getConfigObj()` composes the config once per thread (module-level memo) at its first call, which
+happens before the root component loads and long before any user component's plugins run. Anything a
+component does at load time — like `loadEnv` writing `process.env` — therefore cannot affect the
+composed config (#1513). By design this stays true: configuration is strictly top-down, so the three
+config-shaping env vars (`HARPER_DEFAULT_CONFIG`/`HARPER_CONFIG`/`HARPER_SET_CONFIG`) are **never
+honored** from a component `.env`. What #1513 fixed is the silence: `config/componentEnvPrepass.ts`
+scans `componentsRoot` + `RUN_HDB_APP` for `loadEnv` declarations during `initConfig` and emits an
+actionable warning per config-shaping var found, and `resources/loadEnv.ts` warns again at
+component-load time (covering post-boot deploys) and **skips the `process.env` assignment** for the
+trio — enforce-at-injection, so anything downstream that (re)composes from `process.env`
+(#1618/#1726) can rely on the trio arriving only via sanctioned channels. The pre-pass deliberately
+mirrors loader behaviors that must stay in sync if the loader changes: config filename precedence
+(`harper-config.yaml` → `harperdb-config.yaml` → `config.yaml`) and `files` pattern validation
+(`..` and absolute patterns rejected). Known limitation: a `componentsRoot` override that itself
+arrives via env var cannot redirect the scan.
+
+## Boot-path config persistence is best-effort, and its two artifacts commit as a unit (`config/configUtils.ts`, `config/harperConfigEnvVars.ts`)
+
+Every boot with a `HARPER_*_CONFIG` env var set re-derives the merged config and, historically, wrote
+it back unconditionally. On a full or quota-exhausted volume that write is refused and, being fatal,
+turned a full disk into a container restart loop nothing inside the container could break — the
+cleanup that frees space needs a started process (#847). Two rules follow.
+
+**Derived boot writes are best-effort; user-requested ones are not.** `persistConfigDuringBoot()`
+swallows exactly ENOSPC/EDQUOT (matching on `errno` as well as `code`, because Linux has no libuv
+mapping for EDQUOT and reports `Unknown system error -122`) and lets the boot proceed on the
+in-memory config. `updateConfig`/`set_configuration`, `addConfig`, `deleteConfigFromFile` and the
+install path keep persist-or-throw: a caller who asked to persist must not get a silent success, and
+an install has no last-known-good config to fall back on.
+
+**The env-config state and the config file must never disagree.** The state file records the
+_pre-env_ values, so it is the only copy of what the operator's config said before an env layer
+overwrote it — the config file itself holds the env-derived value. Both single-file orderings lose
+something: writing the state last means the file it would read originals from is already
+overwritten; writing it first leaves a state ahead of the file, which the next boot's
+`detectConfigDrift` reads as a manual user edit and _permanently_ reassigns those paths to `user`,
+silently disabling the env layer even after space is freed. So the commit is three steps —
+`saveState()` stages the new state in `.harper-config-state.pending.<pid>.json`, the config file is
+written, and `confirmConfigWritten()` **renames** the sidecar over the confirmed record. A rename
+needs no free space, which is the point: no write an exhausted volume can refuse ever stands between
+the confirmed originals and disk. A refused staging write leaves the config file alone; a refused
+config write unlinks the sidecar; a sidecar found at load means a commit was interrupted, so it is
+cleared and drift detection is skipped for that boot rather than mistaking the in-flight write for
+an edit. A boot that re-derives the same state writes nothing at all.
+
+Two details the name and the caller carry. The sidecar is **per-process**: every CLI invocation runs
+`initConfig`, and one shared name would let a starting server clear a running process's in-flight
+commit — the loser would then rewrite the config file with the confirmed state still describing the
+old values, which is the failure the protocol exists to prevent. Recovery therefore only clears a
+sidecar whose owning pid is gone. And only the **main thread** persists or runs recovery: workers
+derive the same merged config and would otherwise race over one pair of files for a result they
+already agree on — and since a worker shares its process's pid, a recovery scan from one would
+delete the main thread's in-flight sidecar as if it were the last boot's wreckage.
+
+A sidecar owned by a _live_ foreign process is not cleared — that process is mid-commit — but its
+presence still turns drift detection off for this boot: a pair someone else is halfway through is no
+more comparable than one an interruption left behind. That suspension is why a sidecar also ages
+out regardless of what its pid says: without it, a sidecar whose owner was killed and whose pid was
+later recycled would look mid-commit forever and suspend drift detection on every boot. The age-out
+is deliberately far longer than a commit could take — recovery from a recycled pid only has to be
+eventual, while deleting a slow-but-live writer's sidecar is the worse error, stranding its config
+file against an unpromoted state.
+
+Drift detection is main-thread-only for the same reason recovery is. A worker never owns the state:
+in the normal sequence the main thread has already classified and persisted before any worker runs,
+and inside the main thread's commit window a file that differs from the snapshot is as likely to be
+the write in flight as an operator edit. A worker that concluded "user edit" would drop the
+env-supplied value for itself alone and serve different config than its siblings.
+
+Known limit: the pair commits as a unit _within a process_. Two live processes (a server boot and a
+CLI invocation) can still interleave their config-file writes and promotions, and nothing in the repo
+serializes config writes across processes. Pre-existing — both artifacts were unordered before this
+protocol — and out of scope here, but the "commits as a unit" guarantee stops at the process
+boundary.
+
+Related: a log write must not be fatal either. `fs.appendFileSync` in `logQueuedData` throws from
+both inline and timer call sites, so on a full volume every log statement was a crash point. The
+fallback goes through `nativeStdWrite`, never `console` — `installStdioGuard` routes console output
+back into this same file logger when `logging.file` and `logging.console` are both on, so a console
+fallback recurses until the stack blows.
+
+## Env-config empty objects mean three different things (`config/harperConfigEnvVars.ts`)
+
+An `{}` in the config system is context-dependent, and conflating the contexts is the root of #2067. In an **env layer** (`HARPER_SET_CONFIG` et al.), an empty object contributes no leaves — `http: {}` means "no overrides under http" (load-bearing removal semantics in `flattenObject`). In the **base config file**, a bare `componentName: {}` is user content — a real empty scope declaration that composition must preserve (`restoreBaseEmptyObjects`, #1618/#1726). An `{}` that is _neither_ — the residue of removing an env-sourced entry leaf-by-leaf — is invalid config that validation may reject forever, because the file is written before validation runs and the residue then reads as user content on every later boot.
+
+Removal therefore prunes: `deleteNestedValue` removes ancestors the deletion emptied, only when it actually deleted an existing leaf, and reports what it pruned. The overlap case — a file-declared empty scope an env layer temporarily populated — is tracked in the state file's `emptyScopeOriginals` (separate from `originalValues` so a marker can never mask or be consumed as a real leaf original at the same path; older state files lacking the field are defaulted). Restore consumes a marker only for a path the prune actually removed, so a scalar overwrite or an absent-leaf no-op can never resurrect a scope over live env-layer content. Note there are two coexisting mechanisms for "file `{}` is user content": `restoreBaseEmptyObjects` on the stateless compose path and the marker pair on the stateful removal path — if you touch one, check the other.
+
+Two durable limitations of the marker mechanism, both with user config-file content as the blast radius: markers can only be recorded at populate time, so a scope an env layer populated _before_ `emptyScopeOriginals` existed (any pre-upgrade boot) has no marker and prunes away on its first post-upgrade vacate; and a corrupt config-state file resets to fresh state — dropping `originalValues` and `emptyScopeOriginals` for every tracked path — after which the next removal prunes those scopes for good; `saveConfigState` writes via temp+rename precisely so a torn write cannot be the trigger, leaving genuine corruption (disk faults, hand edits) as the remaining path.

package/dataLayer/DESIGN.md

@@ -0,0 +1,179 @@
+# dataLayer/ — Design notes
+
+Backup and restore, version gating, system-table bootstrap and storage migration.
+
+**Read this when:** touching `rocksdbBackup.ts`, `restoreMarker.ts`, `blobBackup.ts`, `hdbInfoController.ts`, `bin/copyDb.ts` or `json/systemSchema.json`.
+
+Index of every design note: [DESIGN.md](../DESIGN.md).
+
+---
+
+## Version gate at startup: downgrades prompt, and only the minor direction is confirmable
+
+`getVersionUpdateInfo()` (`dataLayer/hdbInfoController.ts`) compares the store's `data_version_num` (latest `system.hdb_info` record) against the binary's `packageJson.version` on every start. Data newer than binary by a **major** version → hard refusal. Newer by a **minor** version → `forceDowngradePrompt()` asks for confirmation; answering yes records the data version back down to the binary's version and boots (upgrade directives are deliberately additive/downgrade-compatible — see the struct-mode section above and `patchHdbSecretIsHashAttribute` in `upgrade/directives/5-2-0.ts`).
+
+- The prompt's answer can be supplied non-interactively via `CONFIRM_DOWNGRADE` — env var or `--CONFIRM_DOWNGRADE` CLI arg; argv wins (`assignCMDENVVariables`). With no override and no TTY on stdin, the prompt throws instead of blocking on stdin forever (#2046 — services/CI hung with nothing in the log; the mismatch is also logged to hdb.log now).
+- Upgrades never prompt (see the rationale comment in `bin/upgrade.js`); only the downgrade direction confirms. `upgradeCertsPrompt()` on the 4.x upgrade path still has the block-on-stdin hazard.
+- Test-suite gotcha: a suite that supplies the override via `process.argv` affects every later test file in the same mocha process — save and restore `process.argv` in `before`/`after` (see `unitTests/dataLayer/hdbInfoController.test.js`).
+
+## Opening a source LMDB DBI for migration must thread through `compression`
+
+When `migrateOnStart` opens a source LMDB primary store to read records out for the RocksDB copy, it constructs an `OpenDBIObject` and calls `sourceRootStore.openDB(key, dbiInit)`. Critically, the per-attribute `compression` setting from the corresponding `__dbis__` entry must be assigned onto `dbiInit` before that call — `dbiInit.compression = attribute.compression`. Without it, lmdb-js doesn't install its decompression layer; every read on the DBI returns raw compressed bytes. msgpackr then misreads bytes in the `0x40–0x7F` range as shared-structure refs, calls `loadStructures` → decodes the (also compressed) structures buffer → finds more bytes in that range → recurses → stack overflow.
+
+Harper's normal `databases.ts` path already does this (search for `dbiInit.compression = primaryKeyAttribute.compression`); the migration path in `bin/copyDb.ts` has to match.
+
+The persisted `compression` value itself is LMDB-era and loosely shaped: `getDefaultCompression()` historically stored whatever falsy value the config resolved to (`''`, `false`, `null`) when `storage.compression` was disabled, and `{ startingOffset, threshold, dictionary? }` when enabled. lmdb-js interprets falsy as "no compression", but rocksdb-js >= 2.6 validates the option strictly (`''`/booleans throw `Unsupported compression algorithm`) and treats UNSET as "use the build default (lz4)" — the inverse default of lmdb. Every RocksDB open must therefore route through `toRocksCompression()` in `resources/databases.ts` (applied inside `openRocksDatabase`, the single chokepoint), which maps defined-falsy → `'none'` and enabled-without-an-algorithm → an explicit lz4 request when available. Don't pass persisted attribute compression to a RocksDB open directly.
+
+`bin/copyDb.ts`'s `openRocksDb` is part of that chokepoint, not an exception to it. This is about the bytes migration writes, not about a later failure: `copyDbToRocks()` closes every target handle before the staging directory is renamed, and rocksdb-js permits an explicit codec change across a close/reopen, so the runtime would open the migrated database fine either way. But a migration that ignores the configured codec writes the entire dataset uncompressed, and those SST/blob files then keep their original codec until write traffic rewrites them — a full LMDB→RocksDB migration is the one moment the whole dataset is written at once, so it is exactly when the deployment's codec should apply.
+
+## System table bootstrap: `systemSchema.json` + upgrade directive
+
+Adding a new system table (e.g. `hdb_deployment` in #641 Slice A) requires three changes:
+
+1. **`json/systemSchema.json`** — the table entry. Fresh installs auto-create it via `utility/mount_hdb.ts:createTables()`, which iterates `Object.keys(systemSchema)` on first boot.
+2. **`utility/hdbTerms.ts`** — add the table name to `SYSTEM_TABLE_NAMES`.
+3. **`upgrade/directives/<version>.ts`** — provisions the table on existing installs that already have a system schema. Registered in `upgrade/directives/directivesController.ts` (which is otherwise empty — its `versions` Map gets populated by these imports). The directive shape is `{ version, sync_functions, async_functions }`; copy `5-1-0.ts` for the canonical pattern (uses `bridge.createTable` to match what `mount_hdb` does on a fresh install).
+
+   **Version the directive to the first release that ships the dependent code, not a later one.** Directives only run when `current_version < directive_version <= upgrade_version` (`directivesController.getVersionsForUpgrade`). The `hdb_deployment` directive was originally mis-tagged `5.2.0` while the deployment-recorder code shipped in `5.1.0`, so on every `5.0.x -> 5.1.x` upgrade the directive was filtered out (`5.2.0 > 5.1.x`) and the table never got created — breaking replicated `deploy_component` on peer nodes for the entire existing customer base. Caveat: `utility/common_utils.ts:compareVersions` strips trailing `.0` and therefore sorts a pre-release (`5.1.0-beta.1`) _above_ its GA (`5.1.0`), so an install already on a `5.1.0-beta.x` data version will not pick up a `5.1.0` directive when upgrading to GA; those pre-release installs need the table created by other means.
+
+System tables replicate by default. To opt out, add the name to `NON_REPLICATING_SYSTEM_TABLES` in `resources/databases.ts`. The check happens after table init and sets `table.replicate = false` per-node.
+
+If the table needs `audit: true`, set it both in the schema (for fresh installs) **and** on the `CreateTableObject` instance in the directive (for upgrades) — otherwise the two paths diverge.
+
+## RocksDB backup/restore: the restore lock + marker protocol (`dataLayer/restoreMarker.ts`, `dataLayer/rocksdbBackup.ts`)
+
+The `restore_backup` operation restores a user database on a live server by closing it across all
+worker threads, purging its directory (`backups.restore` with `purgeAllFiles`), and reloading it.
+Three non-obvious mechanics keep that safe:
+
+- **Two files in an isolated `` `restore` `` directory beside (never inside) the database directory**,
+  each keyed by `sha256(basename(dbPath)).slice(0,32)`: `<key>.lock`, an OS-level exclusive flock
+  (rocksdb-js `tryFileLock`, auto-released on process death), serializes restores; `<key>.restoring`,
+  a marker written+fsynced (file _and_ the metadata directory) after the lock and before any
+  destructive step, means "a restore started and has not finished" (its first line records the
+  database directory name so the scan can map a marker back without decoding the key). The metadata
+  is hashed into a sibling directory rather than suffixed onto the database name (`<db>.restoring`)
+  for two reasons: a legal database literally named `orders.restoring` would otherwise be mistaken
+  for the restore marker of `orders`, and a 250-character name (the legal max) plus a `.restore.lock`
+  suffix exceeds `NAME_MAX` (255) on most filesystems. The directory name deliberately contains a
+  backtick — `schemaRegex` (the database-name validator) forbids only `/` and a backtick among
+  filesystem-legal characters — so it can never collide with a legal database name, including a
+  database literally named `.restore` (which _is_ a legal name; a plain `.restore/` directory would
+  be exactly that database's directory). Because the startup scan opens any `CURRENT`+`MANIFEST-`
+  directory without re-applying `schemaRegex`, it also explicitly skips the reserved `` `restore` ``
+  entry so an out-of-band directory at that name is never loaded as a database. Startup/rescan
+  detection (`databasesBlockedByRestore` → `scanBlockedRestores` in `dataLayer/restoreMarker.ts`)
+  reads the metadata directory and checks the **marker first**, only probing the lock when the marker exists —
+  probes take the flock and are mutually exclusive across threads, so probing the (persistent) lock
+  file of every long-ago-restored database on every rescan would make concurrent rescans misclassify
+  healthy databases as in-progress. Marker-present + lock-held = restore in progress (don't load);
+  marker-present + lock-free = crashed mid-restore (don't load; rerun the restore to recover).
+- **A recovery restore must not clear a pre-existing marker on a pre-destruction failure.**
+  `beginRestore` returns `preexisting: true` when a `.restoring` marker was already present (this run
+  is a recovery over a possibly half-purged directory). If such a run fails _before_ any destruction
+  (e.g. `verifyDatabaseClosed` finds a leaked handle), it must leave the marker in place — clearing
+  it and broadcasting a reload would surface the earlier attempt's partial/corrupt directory as
+  healthy. Only a _fresh_ marker on a _previously healthy_ database that failed before destruction is
+  safe to clear.
+- **The ITC close broadcast is best-effort, so closure is verified before the purge.** The SCHEMA
+  broadcast (`signalSchemaChange`) resolves after remote handlers complete but times out at 30s
+  "best-effort", swallows errors, and never reaches job-worker threads at all (their ports are
+  excluded from broadcasts to avoid re-entrant deadlocks). A destructive purge cannot trust it:
+  `restoreBackup` polls rocksdb-js `registryStatus()` (process-global across worker threads) until
+  the database path has no open instance, and aborts with a 409 — _cleaning up the marker, since
+  nothing was destroyed_ — if handles remain.
+- **Online restore is impossible for a database a component holds open — and that failure is
+  correct.** rocksdb-js's registry is process-global but records only a per-path refCount, with no
+  attribution to a thread or component; Harper keeps no component→database ownership map. So when a
+  loaded component (or the `system` database, which Harper itself never stops while running) holds
+  its own handle on the target database, `registryStatus()` stays non-zero, Harper can neither
+  identify nor force-close that handle, and an in-place purge would corrupt a live instance.
+  `verifyDatabaseClosed` therefore waits only a short grace period (`DATABASE_CLOSE_WAIT_MS`, for a
+  just-finished job worker's own close to drain) and then fails fast with a 409 that points at
+  running the operation offline (`harper restore_backup` with the server stopped, where no
+  components are loaded and nothing holds the database open). Offline restore is the supported path
+  for component-held and `system` databases; online restore serves databases not actively held by a
+  component. The CLI exposes each backup operation under its operation name only (`create_backup`,
+  `restore_backup`, …) — no hyphenated alias — and `bin/backup.ts` routes it to a reachable server
+  or, when the local server is stopped, to the equivalent offline function.
+- **Job workers must release their RocksDB handles on exit, or the closure check can never pass.**
+  rocksdb-js's registry is process-global across worker threads, and a thread that exits WITHOUT
+  closing leaks its handles (the refCount never drops); the only alternative, `shutdown()`, tears
+  down rocksdb for the _entire_ process. A job worker (`server/jobs/jobProcess.ts`) opens the whole
+  database graph via `getDatabases()` and exits when the job finishes — and `create_backup` is
+  itself a job, so before any `restore_backup` there is always at least one exited job worker that
+  touched the database. Without cleanup those leaked handles keep `registryStatus()` non-zero and
+  would fail the closure check even when no component holds the database. `jobProcess` therefore
+  calls `closeLoadedDatabases()` (`resources/databases.ts`) in its `finally`, closing every loaded
+  user database on that thread (the non-enumerable `system` DB is intentionally skipped), so an
+  exited job worker leaves no residual handle to be mistaken for a live holder.
+- **`dropDatabase` and `restore_backup` serialize on the same lock, not a check-then-act probe.**
+  A drop's `destroy()` interleaving with a restore's purge-and-copy on the same directory would gut
+  a "successful" restore (or vice versa). `dropDatabase` therefore _acquires_ the restore lock
+  (`acquireRestoreLock`, marker-less) for each RocksDB root store and holds it across the whole drop,
+  releasing in a `finally`; a restore in progress makes the acquire fail with 409, and a leftover
+  incomplete-restore marker (lock free, detected via `restoreMarkerPresent`, which — unlike
+  `checkRestoreState` — is safe while this thread holds the lock) is refused rather than dropped over.
+  `database()`'s on-demand open still uses the read-only `throwIfBlockedByRestore` (a
+  `create_table`/`create_schema` must not resurrect a half-purged directory as a fresh empty DB), but
+  the destructive drop path now uses the exclusive lock so the race is closed, not merely narrowed.
+- **The offline restore probes RocksDB's own `LOCK` file, and fails closed.** The offline path runs
+  only when the CLI sees no server (a PID heuristic; the PID file is briefly absent mid-`harper
+restart`), and `backups.restore`'s `purgeAllFiles` never takes RocksDB's lock — so before purging,
+  `restoreBackupOffline` opens the database to probe. It now takes the restore lock+marker _before_
+  probing (so a server that starts afterward sees the marker and refuses to load), and recognizes the
+  pinned rocksdb-js 2.5.0 lock error — a plain `Error` with no `code` and message
+  `IO error: While lock file: <db>/LOCK: Resource temporarily unavailable` (`isRocksDbLockError`) —
+  aborting with a 409 rather than purging a database another process holds open. Any _other_ open
+  failure (corrupt/half-restored) is exactly what restore recovers, so only a lock conflict aborts.
+
+Known limitation: the flock is process-owned; if the restore job's worker _thread_ dies without
+the process exiting, the lock stays held (restores 409) until Harper restarts. There is no typed
+native lock signal in rocksdb-js 2.5.0, so the offline probe relies on message matching; a native
+lock primitive is a rocksdb-js follow-on.
+
+## RocksDB managed backups: blob snapshots (`dataLayer/blobBackup.ts`)
+
+A database's file-backed blobs live in one or more roots _outside_ the RocksDB directory
+(`getBlobPathsForDatabaseName` in `resources/blob.ts` — one per configured `storage.blobPaths`, else
+`<hdb_root>/blobs/<database>`), so the engine's backup does not capture them. `create_backup`,
+`restore_backup`, `delete_backup`, `purge_backups`, and the streaming `get_backup` therefore handle
+blobs alongside the engine data (the `exclude_blobs` request option — default false — opts out for an
+engine-only backup):
+
+- **Managed backups** snapshot the blob roots to `<backupDir>/blobs/<backupId>/<rootIndex>/<relpath>`
+  — a full, non-incremental copy per backup, mirroring the binding's `transaction_logs/<id>/` layout.
+  Each enumerated entry is classified before capture: complete blobs and existing abort markers are
+  hard-linked when possible (copied across filesystems), `.repair` temporaries are omitted, and an
+  incomplete blob is replaced by a retryable PENDING (`0xfe`) marker. If a classified blob vanishes
+  before capture, a terminal ERROR (`0xff`) marker preserves its file id. A file reclaimed before its
+  parent directory is read is outside the snapshot. This keeps a snapshot inode from changing as a
+  live write finishes, while complete blobs remain safe to hard-link because published blob paths are
+  write-once. The snapshot is built in a `.tmp-<id>` sibling and atomically renamed so a failed create
+  leaves no partial snapshot. `restore_backup` purges each blob root and rewrites it from the snapshot;
+  `delete_backup` / `purge_backups` remove the corresponding snapshot directories.
+- **`get_backup`** appends the blob files to the same tar under `blobs/<rootIndex>/<relpath>`. The
+  binding's streaming backup finalizes its tar with exactly a 1024-byte (two-block) end-of-archive
+  marker; `createBackupStream` streams the native _plain_ tar while withholding that trailer
+  (verifying it is all-zero), appends the blob entries via `tar-stream` (whose `finalize` writes the
+  one real trailer), and gzips the combined stream itself when requested — so the binding is always
+  asked for a plain tar and compression happens after the append. No scratch disk. The same blob
+  classification rule applies: complete blobs are streamed, incomplete or post-enumeration missing
+  blobs become PENDING/ERROR marker entries, and repair temporaries are omitted.
+
+**Completion manifest (`dataLayer/backupManifest.ts`).** `create_backup` is two-phase: the engine
+backup (`rootStore.backup()`) resolves — and is immediately visible to `list_backups`/`verify_backup`/
+`restore_backup` — before the blob snapshot is copied. Without a completion record, a blob-snapshot
+failure (or a crash between the phases) would leave an engine backup that lists and verifies as
+healthy while silently missing its blobs, and a concurrent restore could pick a backup id whose
+snapshot is still being written and treat it as intentionally engine-only. So a manifest at
+`<backupDir>/manifests/<backupId>.json` — recording the blob-inclusion policy — is written
+(atomically, temp + rename) only after _both_ phases are durable, and a graceful blob-snapshot
+failure rolls back the just-created engine backup + partial snapshot. Consumers treat a backup id
+with no manifest as incomplete: `list_backups` hides it, `verify_backup`/`restore_backup` reject it
+(409 for a specific id, "no complete backups" for `latest`), and restore uses the manifest's `blobs`
+flag — not the mere presence of a snapshot dir — to decide whether to restore blobs (so an engine-only
+backup leaves live blobs untouched, and a manifest that claims blobs but has no snapshot is flagged
+corrupt by verify). This closes the "healthy-looking but incomplete" and concurrent-restore races;
+the remaining engine/blob point-in-time skew (a blob unlinked between the engine cut and the blob
+walk) is the documented best-effort limitation above.