@harperfast/harper 5.3.0 → 5.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/copyDb.ts +5 -1
- package/bin/deployRenderer.ts +3 -0
- package/components/Application.ts +66 -22
- package/components/DESIGN.md +89 -36
- package/components/componentLoader.ts +2 -0
- package/components/deploymentRecorder.ts +18 -1
- package/components/installFingerprint.ts +123 -0
- package/components/mcp/DESIGN.md +4 -4
- package/components/operations.js +29 -0
- package/config/DESIGN.md +5 -5
- package/dataLayer/DESIGN.md +37 -22
- package/dataLayer/backupRepository.ts +248 -0
- package/dataLayer/restoreMarker.ts +30 -75
- package/dataLayer/rocksdbBackup.ts +253 -110
- package/dist/bin/copyDb.js +3 -0
- package/dist/bin/copyDb.js.map +1 -1
- package/dist/bin/deployRenderer.js +3 -0
- package/dist/bin/deployRenderer.js.map +1 -1
- package/dist/components/Application.d.ts +5 -0
- package/dist/components/Application.js +51 -19
- package/dist/components/Application.js.map +1 -1
- package/dist/components/componentLoader.js +2 -0
- package/dist/components/componentLoader.js.map +1 -1
- package/dist/components/deploymentRecorder.js +18 -1
- package/dist/components/deploymentRecorder.js.map +1 -1
- package/dist/components/installFingerprint.d.ts +39 -0
- package/dist/components/installFingerprint.js +113 -0
- package/dist/components/installFingerprint.js.map +1 -0
- package/dist/components/operations.d.ts +2 -0
- package/dist/components/operations.js +31 -0
- package/dist/components/operations.js.map +1 -1
- package/dist/dataLayer/backupRepository.d.ts +33 -0
- package/dist/dataLayer/backupRepository.js +232 -0
- package/dist/dataLayer/backupRepository.js.map +1 -0
- package/dist/dataLayer/restoreMarker.d.ts +4 -1
- package/dist/dataLayer/restoreMarker.js +29 -68
- package/dist/dataLayer/restoreMarker.js.map +1 -1
- package/dist/dataLayer/rocksdbBackup.js +221 -112
- package/dist/dataLayer/rocksdbBackup.js.map +1 -1
- package/dist/index.d.ts +1 -0
- package/dist/index.js.map +1 -1
- package/dist/json/systemSchema.json +14 -0
- package/dist/resources/DatabaseTransaction.d.ts +140 -11
- package/dist/resources/DatabaseTransaction.js +1055 -464
- package/dist/resources/DatabaseTransaction.js.map +1 -1
- package/dist/resources/IterableEventQueue.d.ts +2 -1
- package/dist/resources/IterableEventQueue.js +16 -8
- package/dist/resources/IterableEventQueue.js.map +1 -1
- package/dist/resources/LMDBTransaction.d.ts +13 -2
- package/dist/resources/LMDBTransaction.js +219 -93
- package/dist/resources/LMDBTransaction.js.map +1 -1
- package/dist/resources/RecordEncoder.js +2 -3
- package/dist/resources/RecordEncoder.js.map +1 -1
- package/dist/resources/Resource.js +5 -6
- package/dist/resources/Resource.js.map +1 -1
- package/dist/resources/ResourceInterface.d.ts +3 -1
- package/dist/resources/ResourceInterface.js.map +1 -1
- package/dist/resources/RocksIndexStore.d.ts +2 -1
- package/dist/resources/RocksIndexStore.js +3 -0
- package/dist/resources/RocksIndexStore.js.map +1 -1
- package/dist/resources/Table.js +116 -12
- package/dist/resources/Table.js.map +1 -1
- package/dist/resources/analytics/write.js +6 -2
- package/dist/resources/analytics/write.js.map +1 -1
- package/dist/resources/auditStore.d.ts +71 -17
- package/dist/resources/auditStore.js +248 -64
- package/dist/resources/auditStore.js.map +1 -1
- package/dist/resources/branchDatabase.js +2 -0
- package/dist/resources/branchDatabase.js.map +1 -1
- package/dist/resources/databases.js +8 -2
- package/dist/resources/databases.js.map +1 -1
- package/dist/resources/indexes/fullTextNativeBinding.d.ts +6 -2
- package/dist/resources/indexes/fullTextNativeBinding.js +12 -2
- package/dist/resources/indexes/fullTextNativeBinding.js.map +1 -1
- package/dist/resources/indexes/fullTextQueryIndex.d.ts +4 -0
- package/dist/resources/indexes/fullTextQueryIndex.js +207 -10
- package/dist/resources/indexes/fullTextQueryIndex.js.map +1 -1
- package/dist/resources/models/Models.d.ts +11 -0
- package/dist/resources/models/Models.js +95 -4
- package/dist/resources/models/Models.js.map +1 -1
- package/dist/resources/models/backendRegistry.d.ts +8 -0
- package/dist/resources/models/backendRegistry.js +15 -1
- package/dist/resources/models/backendRegistry.js.map +1 -1
- package/dist/resources/models/bootstrap.d.ts +2 -0
- package/dist/resources/models/bootstrap.js +6 -0
- package/dist/resources/models/bootstrap.js.map +1 -1
- package/dist/resources/models/calibration.d.ts +88 -0
- package/dist/resources/models/calibration.js +244 -0
- package/dist/resources/models/calibration.js.map +1 -0
- package/dist/resources/models/calibrationStore.d.ts +163 -0
- package/dist/resources/models/calibrationStore.js +986 -0
- package/dist/resources/models/calibrationStore.js.map +1 -0
- package/dist/resources/models/decisionStore.d.ts +24 -0
- package/dist/resources/models/decisionStore.js +45 -0
- package/dist/resources/models/decisionStore.js.map +1 -1
- package/dist/resources/models/generativeDecision.js +44 -15
- package/dist/resources/models/generativeDecision.js.map +1 -1
- package/dist/resources/models/types.d.ts +7 -0
- package/dist/resources/recordLock.d.ts +8 -5
- package/dist/resources/recordLock.js +6 -4
- package/dist/resources/recordLock.js.map +1 -1
- package/dist/resources/replayLogs.js +8 -1
- package/dist/resources/replayLogs.js.map +1 -1
- package/dist/resources/scheduler/engine.d.ts +4 -0
- package/dist/resources/scheduler/engine.js +30 -2
- package/dist/resources/scheduler/engine.js.map +1 -1
- package/dist/resources/search.d.ts +16 -4
- package/dist/resources/search.js +117 -13
- package/dist/resources/search.js.map +1 -1
- package/dist/resources/transaction.js +85 -5
- package/dist/resources/transaction.js.map +1 -1
- package/dist/resources/transactionBroadcast.d.ts +8 -0
- package/dist/resources/transactionBroadcast.js +65 -0
- package/dist/resources/transactionBroadcast.js.map +1 -1
- package/dist/security/keys.d.ts +1 -1
- package/dist/security/keys.js +170 -120
- package/dist/security/keys.js.map +1 -1
- package/dist/server/http.js +10 -0
- package/dist/server/http.js.map +1 -1
- package/dist/server/serverHelpers/Request.js +11 -2
- package/dist/server/serverHelpers/Request.js.map +1 -1
- package/dist/server/serverHelpers/contentTypes.js +8 -3
- package/dist/server/serverHelpers/contentTypes.js.map +1 -1
- package/dist/server/serverHelpers/uwsServer.js +38 -7
- package/dist/server/serverHelpers/uwsServer.js.map +1 -1
- package/dist/server/threads/manageThreads.d.ts +3 -3
- package/dist/server/threads/manageThreads.js +15 -13
- package/dist/server/threads/manageThreads.js.map +1 -1
- package/dist/server/threads/threadServer.d.ts +3 -0
- package/dist/server/threads/threadServer.js +24 -6
- package/dist/server/threads/threadServer.js.map +1 -1
- package/dist/upgrade/directives/5-3-0.js +9 -3
- package/dist/upgrade/directives/5-3-0.js.map +1 -1
- package/dist/utility/durableFile.d.ts +36 -0
- package/dist/utility/durableFile.js +106 -0
- package/dist/utility/durableFile.js.map +1 -0
- package/dist/utility/errors/hdbError.d.ts +4 -0
- package/dist/utility/errors/hdbError.js +10 -1
- package/dist/utility/errors/hdbError.js.map +1 -1
- package/dist/utility/globalSchema.d.ts +9 -0
- package/dist/utility/hdbTerms.d.ts +1 -0
- package/dist/utility/hdbTerms.js +1 -0
- package/dist/utility/hdbTerms.js.map +1 -1
- package/dist/validation/configValidator.js +29 -0
- package/dist/validation/configValidator.js.map +1 -1
- package/index.ts +7 -0
- package/json/systemSchema.json +14 -0
- package/npm-shrinkwrap.json +22 -22
- package/package.json +2 -2
- package/resources/DESIGN.md +57 -272
- package/resources/DatabaseTransaction.ts +708 -87
- package/resources/FULL_TEXT.md +6 -2
- package/resources/IterableEventQueue.ts +14 -7
- package/resources/LMDBTransaction.ts +212 -84
- package/resources/RecordEncoder.ts +2 -3
- package/resources/Resource.ts +5 -6
- package/resources/ResourceInterface.ts +3 -1
- package/resources/RocksIndexStore.ts +5 -0
- package/resources/Table.ts +118 -15
- package/resources/analytics/write.ts +6 -2
- package/resources/audit-retention.md +203 -0
- package/resources/auditStore.ts +306 -84
- package/resources/branchDatabase.ts +2 -0
- package/resources/databases.ts +6 -2
- package/resources/indexes/DESIGN.md +120 -177
- package/resources/indexes/fullTextNativeBinding.ts +22 -3
- package/resources/indexes/fullTextQueryIndex.ts +274 -11
- package/resources/models/DESIGN.md +11 -3
- package/resources/models/Models.ts +127 -4
- package/resources/models/backendRegistry.ts +17 -0
- package/resources/models/bootstrap.ts +9 -1
- package/resources/models/calibration.ts +298 -0
- package/resources/models/calibrationStore.ts +1167 -0
- package/resources/models/decisionStore.ts +57 -0
- package/resources/models/generativeDecision.ts +54 -16
- package/resources/models/types.ts +6 -0
- package/resources/record-locks.md +155 -141
- package/resources/recordLock.ts +12 -6
- package/resources/replayLogs.ts +9 -2
- package/resources/scheduler/DESIGN.md +4 -3
- package/resources/scheduler/engine.ts +35 -2
- package/resources/search.ts +154 -16
- package/resources/transaction.ts +50 -5
- package/resources/transactionBroadcast.ts +69 -0
- package/security/DESIGN.md +9 -30
- package/security/keys.ts +175 -127
- package/server/DESIGN.md +70 -36
- package/server/http.ts +10 -0
- package/server/serverHelpers/Request.ts +10 -2
- package/server/serverHelpers/contentTypes.ts +5 -3
- package/server/serverHelpers/uwsServer.ts +34 -7
- package/server/threads/manageThreads.js +16 -14
- package/server/threads/threadServer.js +24 -6
- package/studio/web/assets/{Chat-CNEKMllJ.js → Chat-4LMxI0yB.js} +1 -1
- package/studio/web/assets/{FloatingChat-CEnWhHeR.js → FloatingChat-kWHApEtF.js} +3 -3
- package/studio/web/assets/{SearchDialog-CwhJdEdY.js → SearchDialog-B7ymna97.js} +1 -1
- package/studio/web/assets/{apiToken-DC5vsaaO.js → apiToken-CE3AAt9f.js} +1 -1
- package/studio/web/assets/{applications-DQZxDy0x.js → applications-DdN7CqsB.js} +1 -1
- package/studio/web/assets/{cssMode-Dz2DxF4e.js → cssMode-BpySZUSt.js} +1 -1
- package/studio/web/assets/{editor-B3mjd3tz.js → editor-myLCVv9C.js} +1 -1
- package/studio/web/assets/{html-BsRGM7tp.js → html-DGNYO55O.js} +1 -1
- package/studio/web/assets/{htmlMode-tUKJaA3o.js → htmlMode-BUgpQr41.js} +1 -1
- package/studio/web/assets/{index-BuqFjN5K.js → index-FN4KC9va.js} +39 -39
- package/studio/web/assets/index-QLZw6xrU.css +1 -0
- package/studio/web/assets/{index.lazy-CcpfQFY_.js → index.lazy-DZFuaumG.js} +1 -1
- package/studio/web/assets/{javascript-BZ_uRGHl.js → javascript-DiBzh4pj.js} +1 -1
- package/studio/web/assets/{jsonMode-DgB-aFMj.js → jsonMode-Bm312nmu.js} +1 -1
- package/studio/web/assets/{languageServices-BleuBw17.js → languageServices-Dz2K9rZN.js} +1 -1
- package/studio/web/assets/{lspLanguageFeatures-ByVYLSsQ.js → lspLanguageFeatures-CNu1L5xn.js} +1 -1
- package/studio/web/assets/{notifications-TuzN0IMi.js → notifications--nwFLGmM.js} +1 -1
- package/studio/web/assets/{notifications-D1Nf-Edn.js → notifications-BelKw5bR.js} +1 -1
- package/studio/web/assets/{profile-D7C7Lupz.js → profile-AU5FjOBx.js} +1 -1
- package/studio/web/assets/{regions-nGQCm-v8.js → regions-C0sTSEHH.js} +1 -1
- package/studio/web/assets/{register-C3x9Ze6P.js → register-DyNR2rGk.js} +2 -2
- package/studio/web/assets/{setComponentFile-DpjrRqeG.js → setComponentFile-BkWx8AN3.js} +1 -1
- package/studio/web/assets/{setup-CGiKLZ9e.js → setup-D6LZS78d.js} +2 -2
- package/studio/web/assets/{status-CGJwrNPH.js → status-BCWnn70o.js} +1 -1
- package/studio/web/assets/{toggleHighContrast-C-Ne6P7T.js → toggleHighContrast-pyeGJ6W1.js} +1 -1
- package/studio/web/assets/{tsMode-CSreYTys.js → tsMode-CGV3iaWC.js} +1 -1
- package/studio/web/assets/{typescript-MnxBk89W.js → typescript-Ci7Khvwo.js} +1 -1
- package/studio/web/assets/{useEntityRestURL-TNrh77-s.js → useEntityRestURL-DNiz3aZ0.js} +1 -1
- package/studio/web/assets/{workers-BfZlUNVI.js → workers-byzxjbqy.js} +1 -1
- package/studio/web/assets/{xml-BHd6qGH7.js → xml-DGnHQ3CX.js} +1 -1
- package/studio/web/assets/{yaml-C53R8OQq.js → yaml-BUoTnPWb.js} +1 -1
- package/studio/web/index.html +2 -2
- package/upgrade/directives/5-3-0.ts +10 -3
- package/utility/durableFile.ts +96 -0
- package/utility/errors/hdbError.ts +9 -0
- package/utility/hdbTerms.ts +1 -0
- package/validation/configValidator.ts +31 -0
- package/studio/web/assets/index-72cEH474.css +0 -1
|
@@ -50,14 +50,15 @@ Consequences that shape the code:
|
|
|
50
50
|
`unregisterRecordLock` manage it. `lock()` consults it before calling `tryLock`; a re-entrant call
|
|
51
51
|
returns the existing live handle. A handle expired by its lease timer is pruned on next re-lock lookup
|
|
52
52
|
so a stale holder's write gets 409.
|
|
53
|
-
- **Release.**
|
|
54
|
-
|
|
55
|
-
`{ hold: true }`
|
|
56
|
-
`handle.release()` directly (synchronous, returns false if already released).
|
|
57
|
-
scoped handles: it calls `release()` early and unregisters the handle so the transaction commit does
|
|
53
|
+
- **Release.** Every handle — scoped or hold — is registered in `link.recordLocks` and kept on the
|
|
54
|
+
locked instance as `#lockHandle`; every commit or abort calls `releaseRecordLocks()`, which calls
|
|
55
|
+
`handle.release()` on each non-hold handle, so a `{ hold: true }` handle outlives the transaction.
|
|
56
|
+
`unlock()` calls `handle.release()` directly (synchronous, returns false if already released).
|
|
57
|
+
`unlock()` also accepts scoped handles: it calls `release()` early and unregisters the handle so the transaction commit does
|
|
58
58
|
not release it again. After any `unlock()` call `#lockWritable` is cleared so writes through the
|
|
59
|
-
instance are no longer accepted.
|
|
60
|
-
|
|
59
|
+
instance are no longer accepted. In an explicit transaction with no staged writes and no open
|
|
60
|
+
iterators (`readTxnsUsed <= 1`), a scoped `lock()` releases the read snapshot and sets `snapshotFree`
|
|
61
|
+
so the scope reads what it locked.
|
|
61
62
|
- **Staging model: scoped stages like `update()`; hold stays deferred.** `#reloadLocked` eagerly
|
|
62
63
|
calls `_writeUpdate(id, this.#changes, false)` for a fresh scoped acquisition, exactly as the
|
|
63
64
|
instance `update()` does, so a `TransactionWrite` exists on the transaction immediately and `save()`
|
|
@@ -83,7 +84,7 @@ Consequences that shape the code:
|
|
|
83
84
|
holder write wins over any pre-lock concurrent write. When prior staged writes already exist,
|
|
84
85
|
ordering is best-effort — no 409 is thrown. In an `ImmediateTransaction` context (no explicit
|
|
85
86
|
scope), a scoped lock's writes go through the same `update()`-style staging as above; each explicit
|
|
86
|
-
`update()`+`save()` cycle is stamped with `
|
|
87
|
+
`update()`+`save()` cycle is stamped with `holderVersionCandidate()` independently. A scoped lock acquired
|
|
87
88
|
outside any explicit `transaction()` scope persists until `unlock()` or the lease expires
|
|
88
89
|
(ImmediateTransaction's `releaseRecordLocks()` is a no-op for record locks).
|
|
89
90
|
- **Scoped → hold upgrade.** Calling `lock(id, { hold: true })` while the same transaction already
|
|
@@ -93,7 +94,7 @@ Consequences that shape the code:
|
|
|
93
94
|
different resource instance sharing the key) stays valid; retiring and replacing would invalidate
|
|
94
95
|
those other references (their `save()` would then throw 409 against a released handle). The upgrade
|
|
95
96
|
also detaches the scoped phase's eagerly-staged `TransactionWrite` (see the staging-model bullet
|
|
96
|
-
|
|
97
|
+
above) via `detachScopedUpgradeWrite`: hold staging is deferred and explicit-save-only, so a dangling
|
|
97
98
|
scoped write left in place would otherwise auto-commit at the transaction's sweep and clobber
|
|
98
99
|
whatever the hold write lands. The detached write is marked `.dropped` so a later explicit `save()`
|
|
99
100
|
on the instance that owns it falls through to the hold branch instead of resolving a dead reference.
|
|
@@ -127,22 +128,25 @@ Consequences that shape the code:
|
|
|
127
128
|
`lockReleaseByOwner(this)` on env teardown, releasing every key the terminated thread's handle held.
|
|
128
129
|
The lease timer is a soft bound in case the holder's event loop is blocked.
|
|
129
130
|
- **Not supported on LMDB.** `lock()` throws 501.
|
|
130
|
-
- **`lock()` is an in-process verb only.** `
|
|
131
|
-
`transactional()`
|
|
131
|
+
- **`lock()` is an in-process verb only.** `Table.lock` is a `static async` method on the table class,
|
|
132
|
+
not a `transactional()` verb on `Resource`, and no protocol reaches it: REST answers 501,
|
|
133
|
+
`KNOWN_METHODS` does not include it,
|
|
132
134
|
and neither OpenAPI nor MCP enumerate it. Exposing lock/unlock over a protocol is a Phase 1 decision.
|
|
133
135
|
Acquisition itself has no authorization hook — lock() is not protocol-dispatched, so no
|
|
134
136
|
allowUpdate/allowCreate check runs when a caller acquires a lock.
|
|
135
|
-
- **`lock()` and `allowUpdate`:** writes through a held lock bypass per-table
|
|
136
|
-
hooks by the same trust model as any in-process `Table.update(id)` + set/save sequence.
|
|
137
|
+
- **`lock()` and `allowUpdate`:** writes through a held lock bypass per-table
|
|
138
|
+
`allowUpdate`/`allowCreate`/`allowDelete` hooks by the same trust model as any in-process `Table.update(id)` + set/save sequence.
|
|
137
139
|
|
|
138
140
|
Not in Phase 0, by design: replication of lock transitions, distributed grant, lease renewal,
|
|
139
141
|
subscription events for lock/unlock, and lock() on LMDB. Phase 1 adds the distributed grant — see
|
|
140
|
-
"Phase 1: cluster-wide `lock()` over
|
|
142
|
+
"Phase 1: cluster-wide `lock()` over amortized per-record ownership" below. Lease renewal, subscription
|
|
141
143
|
events for lock/unlock, and lock() on LMDB remain out of scope.
|
|
142
144
|
|
|
143
145
|
**Acquisition timestamp and mixed transactions.** In an `ImmediateTransaction` context (no explicit
|
|
144
|
-
`transaction()` scope) every save — hold or scoped — is stamped by `handle.
|
|
145
|
-
|
|
146
|
+
`transaction()` scope) every save — hold or scoped — is stamped by `handle.holderVersionCandidate()`, and
|
|
147
|
+
the handle's version floor advances only once that write commits (`noteCommittedLockVersions()` →
|
|
148
|
+
`noteHolderVersion()`) or a scoped → hold upgrade primes it (`upgradeToHold`), so sequential saves each get a distinct, monotonically-increasing version.
|
|
149
|
+
That stamp lives on the write
|
|
146
150
|
(`TransactionWrite.lockStamp`) and is
|
|
147
151
|
never assigned to the link clock: pinning `link.timestamp` would stamp every OTHER write staged on the
|
|
148
152
|
same context before the commit resets it — a concurrent write in the caller's own `Promise.all`, an
|
|
@@ -166,12 +170,9 @@ the acquiring transaction committed is fine (each write auto-commits as an Immed
|
|
|
166
170
|
Taking the lock again in a second `transaction()` scope issues a fresh `lock()` call rather than
|
|
167
171
|
relying on the first hold still being re-entrant in that scope.
|
|
168
172
|
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
- _Abort during the `acquireRecordKey` await window._ The async gap between `tryLock` failing and the
|
|
173
|
-
`onUnlocked` callback is short in practice, and injecting an abort during that window requires two
|
|
174
|
-
concurrent threads.
|
|
173
|
+
An abort during the `acquireRecordKey` wait is covered single-threaded by "a transaction aborted while
|
|
174
|
+
lock() waiters are parked leaves the key unlocked" (`unitTests/resources/recordLock.test.js`), which
|
|
175
|
+
parks a leader there and a coalesced follower behind it.
|
|
175
176
|
|
|
176
177
|
### Phase 1: cluster-wide `lock()` over amortized per-record ownership (`recordLockCoordinator`)
|
|
177
178
|
|
|
@@ -243,9 +244,10 @@ barrier (the recovery fence below) is on it because its log position is the whol
|
|
|
243
244
|
|
|
244
245
|
The release entry is written in its own transaction, with no primary-store write, and — unlike the `reload`
|
|
245
246
|
marker it is otherwise modeled on — **not** `LOCAL_ONLY`, because replicating it IS the send. Its
|
|
246
|
-
payload is `[key,
|
|
247
|
-
|
|
248
|
-
|
|
247
|
+
payload is `[1, key, requester, generation, homeIncarnation, counter, dependencies]`; the decoder also
|
|
248
|
+
accepts the historical five-field `[key, requester, generation, homeIncarnation, counter]` as unknown
|
|
249
|
+
lineage, no other length, and drops an unknown leading version, since a partially-understood release
|
|
250
|
+
would clear a delegation on terms the sender did not intend.
|
|
249
251
|
|
|
250
252
|
**`recordId` is null and the key rides in the payload.** A control entry carrying the locked key as
|
|
251
253
|
its record id answers `_writeUpdate`'s keyed dedup lookup at exactly the holder's stamp, and
|
|
@@ -301,9 +303,10 @@ operator-sequenced, and harper-pro's to hold, not core's.
|
|
|
301
303
|
own: a caller that staged a write and then called `unlock()` leaves nothing for a drain to wait on,
|
|
302
304
|
but its write is still uncommitted and would land after the successor was admitted. So a delegation
|
|
303
305
|
retains a revoker for every handle it admitted (`registerAdmission`), and surrender, expiry and
|
|
304
|
-
`close()` all call them.
|
|
305
|
-
`registerAdmission()` both take it back
|
|
306
|
-
|
|
306
|
+
`close()` all call them. Each admission carries an `admissionId`, and `release(key, admissionId)` and
|
|
307
|
+
`registerAdmission(admissionId, revoke)` both take it back — an id rather than the delegation's token,
|
|
308
|
+
because a renewal changes the token while the admission survives it: a key's delegation can be replaced
|
|
309
|
+
while a handle is still open, and a release addressed by key from a superseded handle would decrement the successor's admission
|
|
307
310
|
count and let it be surrendered while its own callers were still inside — `handle.revokeLease()` expires the handle ahead of its lease, and the
|
|
308
311
|
commit-time fence in `DatabaseTransaction` then rejects the staged write immediately before the native
|
|
309
312
|
commit submits. That fence runs only when the transaction actually holds a lease-protected write
|
|
@@ -330,7 +333,8 @@ grant on all three components: a home that restarts begins counting again, so a
|
|
|
330
333
|
would let a delayed release from a previous incarnation clear a live grant while its delegate is
|
|
331
334
|
still admitting.
|
|
332
335
|
|
|
333
|
-
**Bounded state.** Delegations are capped per
|
|
336
|
+
**Bounded state.** Delegations are capped per table coordinator and per requester (a process-wide bound
|
|
337
|
+
is harper#2581), expiry work is bounded per
|
|
334
338
|
tick, and clean-handoff dependency sets have a separate, larger LRU cap. A fixed add-only Bloom
|
|
335
339
|
filter distinguishes a truly virgin key from an evicted dependency set while this coordinator has
|
|
336
340
|
observed the whole generation; after a cold start, ownership gap, or generation change, every
|
|
@@ -356,10 +360,22 @@ construction the one most likely to be truncated, so that rule reports a contend
|
|
|
356
360
|
coordination failure at random.
|
|
357
361
|
|
|
358
362
|
**`{ scope: 'node' }`** opts out of the cluster step and keeps exact Phase 0 semantics, which by
|
|
359
|
-
design permits simultaneous holders on different nodes.
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
363
|
+
design permits simultaneous holders on different nodes. A defaulted scope is `'node'` on a table
|
|
364
|
+
declared `replicate: false` — it has no cluster to lock across, and the transport an operator registers
|
|
365
|
+
for the database must not re-scope it (harper#2716) — and `'cluster'` otherwise, which includes a table
|
|
366
|
+
whose declaration omits `replicate`, since that is the replicating default. The default is read from
|
|
367
|
+
`Table.replicate` by `resolveLockOptions` at the one call in `Table.lock()` and again after the native
|
|
368
|
+
wait by `rescope()`, so a call follows the declaration current when it acquires — a redeclaration
|
|
369
|
+
mid-wait re-scopes it ("re-reads the declaration after the native wait" tests) — while registering a
|
|
370
|
+
transport never does. For that, `Table.replicate` is refreshed on redeclaration and catalog reload, and a
|
|
371
|
+
redeclaration is persisted against the durable primary row, not the possibly stale static. A hold already granted keeps the scope it was
|
|
372
|
+
granted at: a `replicate` change re-scopes later calls, not live handles — **not enforced**, so drain holds
|
|
373
|
+
before changing a live table's `replicate` from `false` to `true`. Pinned by "a table that does not
|
|
374
|
+
replicate" in `unitTests/resources/recordLockCluster.test.js`. An **explicit**
|
|
375
|
+
`{ scope: 'cluster' }` with no transport rejects 503 rather than silently returning the weaker lock,
|
|
376
|
+
and a transaction that already holds a key node-scoped cannot take a cluster lock on it (409) —
|
|
377
|
+
including through the concurrent-lock coalescing path, where a follower would otherwise inherit the
|
|
378
|
+
leader's weaker handle.
|
|
363
379
|
|
|
364
380
|
**Routing and exclusion from record surfaces.** The replicated-event consumer in `Table.ts` dispatches
|
|
365
381
|
the release entry to the table's `LockCoordinator` before it resolves a resource, so it never reaches
|
|
@@ -387,10 +403,10 @@ the delegation path assigns a record version.
|
|
|
387
403
|
set. The home merges the trusted release origin at the release entry's own log position, retains the
|
|
388
404
|
result after clearing the grant, and sends it with the successor grant. The successor remains pending
|
|
389
405
|
and recallable until `ClusterLockTransport.establishLockFreshness()` has made every dependency applied
|
|
390
|
-
and visible. Missing lineage selects
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
|
|
406
|
+
and visible. Missing lineage selects the recovery barrier, which harper-pro's transport
|
|
407
|
+
(`replication/recordLockFreshness.ts`) answers by draining every other home-map member; failure or
|
|
408
|
+
timeout returns 503 and hands the grant back without discarding retained lineage. The cached-delegation
|
|
409
|
+
branch does none of this work. harper-pro#822 supplies the operator-agreed home map and the transport.
|
|
394
410
|
|
|
395
411
|
**Recovery fence.** The recovery barrier's position is a `lockBarrier` control entry (nibble 13,
|
|
396
412
|
payload `[1, nonce]`, harper#2625): a replicated no-op the probed member commits after the probe, so
|
|
@@ -401,8 +417,8 @@ this node's own commit through `Table.writeLockControlEntry`, never the transpor
|
|
|
401
417
|
hook, since the caller is the transport and the fence must be a position in this origin's log — and
|
|
402
418
|
resolves to the entry's log position; the transport supplies the nonce it will match the entry on.
|
|
403
419
|
The coordinator ignores the entry on receipt, and every `isLockControlType` exclusion above covers it. `establishLockFreshness()` receives the wait remaining
|
|
404
|
-
on the lock deadline. The harper-pro operation and drain are harper-pro
|
|
405
|
-
no barrier fails closed.
|
|
420
|
+
on the lock deadline. The harper-pro operation and drain are harper-pro `replication/recordLockFreshness.ts`
|
|
421
|
+
(harper-pro#822); a recovery marker with no barrier fails closed.
|
|
406
422
|
|
|
407
423
|
## Phase 1 design: amortized per-record ownership
|
|
408
424
|
|
|
@@ -412,11 +428,10 @@ unchanged and remains the local primitive everything here builds on.
|
|
|
412
428
|
|
|
413
429
|
Planning review cleared this design over ten rounds; the rounds that changed it are recorded in the PRs harper#2498, harper#2613, harper#2627 and harper#2667.
|
|
414
430
|
|
|
415
|
-
### 1. The problem with what
|
|
431
|
+
### 1. The problem with what was on the branch
|
|
416
432
|
|
|
417
|
-
`table.lock(id)` on the branch
|
|
418
|
-
for a `LOCK_GRANT` from **every** participant
|
|
419
|
-
`agreedDown` for nobody, so nothing is ever excluded from that set, and harper-pro#822 states the
|
|
433
|
+
`table.lock(id)` on the branch ran Ricart–Agrawala (removed by harper#2498): the requester wrote
|
|
434
|
+
`LOCK_REQUEST` and waited for a `LOCK_GRANT` from **every** participant. Core set `agreedDown` for nobody, so nothing is ever excluded from that set, and harper-pro#822 states the
|
|
420
435
|
consequence in its own limitations list: a crashed peer blocks new cluster locks for its databases
|
|
421
436
|
until it leaves `hdb_nodes`. Unanimity is not a high-availability protocol.
|
|
422
437
|
|
|
@@ -458,7 +473,7 @@ quorum-confirmed mode, they are describing that issue's scope, not this one's.
|
|
|
458
473
|
|
|
459
474
|
They are separate, and expiry establishes neither on its own. Phase 0 enforces part of the first:
|
|
460
475
|
expiry is checked synchronously on every staged write and again immediately before the native commit
|
|
461
|
-
submits (`
|
|
476
|
+
submits (the `hasLeaseProtectedWrite` fence in `DatabaseTransaction.performCommit()`). That fence is reused unchanged — only the thing
|
|
462
477
|
that issues the lease changes — but it neither settles an already-submitted commit nor makes an
|
|
463
478
|
asynchronous replica fresh, which is what §6 and §7 are for.
|
|
464
479
|
|
|
@@ -704,7 +719,7 @@ clock at receipt overlaps the next delegate.
|
|
|
704
719
|
- **Every local handle is bounded by `min(requested lease, delegation deadline)`** — including
|
|
705
720
|
re-entrant acquisition and the `{hold: true}` upgrade. The branch's
|
|
706
721
|
`upgradeToHold` already clamps to the granted round's deadline rather than extending it
|
|
707
|
-
(`resources/recordLock.ts
|
|
722
|
+
(`KeyLockHandle.upgradeToHold` in `resources/recordLock.ts`); that clamp is retargeted, and must not be dropped along with the
|
|
708
723
|
round it currently reads.
|
|
709
724
|
- **Clock assumptions, stated rather than implied:** every deadline is compared only against readings
|
|
710
725
|
of the _same_ node's `performance.now()`; no remote instant is ever compared against a local one,
|
|
@@ -721,7 +736,7 @@ Two ways a "drained" delegate keeps usable authority, both found in review:
|
|
|
721
736
|
reports zero while staged writes can still commit inside the lease.
|
|
722
737
|
- It holds a valid `{hold: true}` handle and has staged _nothing_; a recall that only closes new
|
|
723
738
|
admission finds nothing to settle, emits the release, and the holder then writes through its
|
|
724
|
-
still-unexpired handle (`resources/recordLock.ts
|
|
739
|
+
still-unexpired handle (`RecordLockOptions.hold` in `resources/recordLock.ts`, the hold branch of `Table.save()`).
|
|
725
740
|
|
|
726
741
|
So recall is defined as four steps, in order:
|
|
727
742
|
|
|
@@ -729,8 +744,8 @@ So recall is defined as four steps, in order:
|
|
|
729
744
|
`hold` upgrade.
|
|
730
745
|
2. **Revoke** every outstanding handle's write capability synchronously — the same state an expired
|
|
731
746
|
lease produces, so a subsequent write through it fails 409 at staging _and_ at commit submission.
|
|
732
|
-
Calling `release()` is _not_ revocation: `resources/recordLock.ts
|
|
733
|
-
handle handed back from a lapsed lease, and `
|
|
747
|
+
Calling `release()` is _not_ revocation: `isLeaseExpired()` vs `isExpired()` in `resources/recordLock.ts` deliberately distinguishes a
|
|
748
|
+
handle handed back from a lapsed lease, and the `hasLeaseProtectedWrite` fence in `DatabaseTransaction.performCommit()` consults that
|
|
734
749
|
predicate immediately before native submission — so revocation must set the lapsed state, and must
|
|
735
750
|
also reach handles a staged write still holds after `unlock()` removed them from the transaction's
|
|
736
751
|
lock registry. A grace window (`min(remaining delegation, recallGraceMs)`) before revocation is a
|
|
@@ -738,7 +753,7 @@ So recall is defined as four steps, in order:
|
|
|
738
753
|
3. **Settle** every transaction that staged a write to `K` under this delegation — committed or
|
|
739
754
|
aborted — including a native commit already submitted, whose completion the pre-submission fence
|
|
740
755
|
does not prove. Settlement follows the **logical** transaction through retry and replay
|
|
741
|
-
(`DatabaseTransaction.ts
|
|
756
|
+
(`performCommit()`'s conflict-retry re-save and the open-iterator `replayTransaction` in `DatabaseTransaction.ts`); one native attempt completing is not settlement. The
|
|
742
757
|
delegation therefore tracks _staged writes on K_, not live handles, and the release hook hangs off
|
|
743
758
|
transaction settlement.
|
|
744
759
|
4. **Then** write the release (§7.1).
|
|
@@ -770,7 +785,7 @@ position:
|
|
|
770
785
|
- `A` writes `K`; `B` acquires, never writes, releases; `C` has applied `B` but not `A`. A
|
|
771
786
|
`B`-position check passes and `C` reads stale — so the set is **inherited**. B releases `{A:R_A}`;
|
|
772
787
|
the home retains `{A:R_A, B:R_B}`, and C must satisfy both.
|
|
773
|
-
- Core breaks equal-`version` conflicts by node name (`resources/Table.ts
|
|
788
|
+
- Core breaks equal-`version` conflicts by node name (`precedesExistingVersion` in `resources/Table.ts`), so `C` can hold a
|
|
774
789
|
_losing_ value at the same timestamp as the winner and pass a `version ≥ V` test. A version scalar
|
|
775
790
|
therefore cannot be the fence even for the simple case.
|
|
776
791
|
- Receiving `B`'s later patch does not prove receipt of `A`'s earlier change to other fields, which
|
|
@@ -860,10 +875,10 @@ coherent options:
|
|
|
860
875
|
- It changes `_writeUpdate`'s resolution rule for _all_ writes and adds a field to the record
|
|
861
876
|
contract, so it is a stored-format and older-binary compatibility decision, not just a protocol
|
|
862
877
|
one — including what happens if the feature is disabled after fenced records exist.
|
|
863
|
-
- It **reverses the documented Phase 0 relationship with ordinary writes** (
|
|
878
|
+
- It **reverses the documented Phase 0 relationship with ordinary writes** (the "Phase 0 contract" above): an
|
|
864
879
|
ordinary update landing after a fenced value would permanently lose, regardless of age.
|
|
865
|
-
- Older writes currently enter resequencing and merge logic (`
|
|
866
|
-
separate path (`
|
|
880
|
+
- Older writes currently enter resequencing and merge logic (the out-of-order branch of `_writeUpdate`), and deletes take a
|
|
881
|
+
separate path (`_writeDelete`), so a generation must be honored by patches and deletes too.
|
|
867
882
|
- A generation alone has no **rejection floor** that survives tombstone cleanup: generation 1 puts,
|
|
868
883
|
generation 2 deletes, the tombstone is reclaimed, and a delayed generation-1 put resurrects the
|
|
869
884
|
record on that replica. Fenced mode therefore also owes a retained floor across tombstone
|
|
@@ -876,7 +891,7 @@ coherent options:
|
|
|
876
891
|
a majority. Write confirmation alone is not sufficient — intersection needs the read side too, and
|
|
877
892
|
§7.2's barrier has no quorum floor — so this arm is **two** changes: confirm the write to a
|
|
878
893
|
majority, _and_ make the barrier require a majority and fail closed below it.
|
|
879
|
-
`X-Replicate-To;confirm=M` (`server/REST.ts
|
|
894
|
+
`X-Replicate-To;confirm=M` (`server/REST.ts`) is the nearest existing mechanism, but it is
|
|
880
895
|
super-user-gated and entangled with residency (§10), so this arm cannot just expose it. It costs
|
|
881
896
|
latency on
|
|
882
897
|
every locked write. Even with both halves it does **not** restore §2 as written: the late-settling
|
|
@@ -933,7 +948,7 @@ lease` means carrying the revoker on the Phase 0 handle that already exists inst
|
|
|
933
948
|
accumulate state. A request from a node the current generation does not name is refused before any
|
|
934
949
|
grant, waiter or timer is allocated. Dependency-set and key payload sizes are bounded before
|
|
935
950
|
allocation. Trusted origin identity stays the transport's responsibility, as it already is
|
|
936
|
-
(`resources/recordLockCoordinator.ts
|
|
951
|
+
(`DelegationRequest.requester` in `resources/recordLockCoordinator.ts`). There are no configuration certificates to validate,
|
|
937
952
|
because there is no election — the map's digest is agreed once, out of band, before the feature is
|
|
938
953
|
enabled (§4.1).
|
|
939
954
|
|
|
@@ -943,7 +958,7 @@ lease` means carrying the revoker on the Phase 0 handle that already exists inst
|
|
|
943
958
|
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
944
959
|
| **Different layer** | Put arbitration entirely in harper-pro's replication layer, or behind an embedded/external Raft group that owns the lock table. | The disqualifier for a full replicated lock table is specific, not "bigger": **replicating per-key ownership puts durable, majority-acknowledged work on every acquisition**, which is exactly what volatile delegations remove and what §10's amortization depends on. It would also still need §7's fence, because it is off-stream. The consensus objection is now clean rather than hypocritical: this design runs no election of its own, at any rate, so "adds consensus" is an argument it is entitled to make. If Harper later wants consensus for membership, shard maps and schema changes, §4's map is the piece to replace with it — and the replacement would be invisible to §§5–8. |
|
|
945
960
|
| **Deeper cause** | Do not hand out locks at all: make the conflicting operation a conditional/compare-and-swap write evaluated by one authority (the residency owner). | Covers only retryable single-record read-modify-write. `lock()` exists in harper#483 for a caller holding the lock across arbitrary application work, including calls to other systems, which no CAS expresses. It also does not remove the authority problem: an independent local CAS against asynchronously replicated copies is not cluster-wide exclusion. Worth having _as well_. |
|
|
946
|
-
| **Do less** | Keep Ricart–Agrawala and accept the availability limit: it
|
|
961
|
+
| **Do less** | Keep Ricart–Agrawala and accept the availability limit: it was implemented, tested, and gated off by `replication.recordLocks`. | The limit is the feature's value, not a rollout caveat: a lock any single unreachable peer disables is not usable for the correctness-critical work `lock()` exists for. Cost points the same way — 13 durable commits and 143 frame deliveries per lock at `P=12`, per key, with no amortization for a node locking the same record repeatedly. |
|
|
947
962
|
| **Do more** | Durable membership epochs: agree `members[]` by single-decree consensus over a majority, with acceptor state persisted before acknowledgement, so an unreachable node is rehomed automatically. **This note's own design through round 6.** | Rejected 2026-09-13. What it buys over the chosen map is exactly one thing: **unattended rehoming**. What it costs is a durable consensus subsystem in harper-pro — persisted promises and accepted values, ballots, renewal leases as the liveness signal, acceptor-side retirement reservations, transitive activation protection, and a conservative restart quarantine to reconstruct reservations a crash lost. All of that exists to _infer_ whether an unreachable node is briefly down or permanently gone. An operator already knows, and can say so in one published generation. The judgement is not that consensus is too hard; it is that **this particular decision has an authority who can simply state it**, so inferring it is unpaid complexity. |
|
|
948
963
|
| **Do more (2)** | A centrally **leased** operator map: the same published generations, but nodes hold short-lived authenticated capabilities for the current one and must renew them from the control plane. The operator stops renewing `g`, waits capability expiry plus the maximum delegation interval, then activates `g+1`. Raised by the round-8 planning review. | **Overruled, with a concrete disqualifier: it makes every node's ability to lock depend on continuously reaching the control plane.** A control-plane outage longer than the capability lease stops record locking on every node in the cluster, including nodes that are healthy and agree with each other — a strictly larger availability dependency than the transition it replaces, and one that is paid continuously rather than at reconfiguration time. It also reintroduces the renewal lease and the clock-rate bound this revision exists to delete, and it does not remove the operator from the loop: the capability issuer must still decide to stop renewing. What it does buy is real: it fences a partitioned old generation mechanically. **Round 10 showed that half can be had without the dependency** — §4.3's one-shot durable activation record, which is issued once per reconfiguration rather than renewed, so steady-state locking never contacts the control plane. That was adopted; only the continuous renewal is rejected. |
|
|
949
964
|
| **Adjacent** | Derive the map from `server.shards`, which already maps shard id → node list and already routes residency reads. | A shard map is a **residency** directive — it says where records live. Making it the lock-home map couples arbitration to data placement and requires sharding configuration, which one customer uses. The chosen map is a separate, purpose-built `recordLockHomes` generation, so a cluster adopts record locks without adopting sharding. Where sharding _is_ configured a shard map may be published as the home set, but only if it already names every node that locks: §4.1's rule binds, so a shard map covering a subset of the cluster would leave the omitted nodes unable to lock at all. |
|
|
@@ -966,7 +981,7 @@ Per uncontended acquisition, `P` participants, `d` = locks served under one dele
|
|
|
966
981
|
|
|
967
982
|
| | durable commits | frame deliveries | latency | one node down |
|
|
968
983
|
| ----------------------------------------------------------------------------------------- | --------------- | ---------------------------------------- | ------------------------------------ | ------------------------------------------------------------------ |
|
|
969
|
-
| Ricart–Agrawala (
|
|
984
|
+
| Ricart–Agrawala (removed) | `P+1` | `P²−1` | slowest participant | **all** cluster locks block |
|
|
970
985
|
| This design, first lock on a key | 0 | 2 unicast | 1 RTT to the home, 0 if local | that node's ring share only, until an operator republishes the map |
|
|
971
986
|
| This design, handoff to another node | 1 (release) | `P−1` + 3 unicast | 2 RTT | — |
|
|
972
987
|
| This design, steady state | 0 | 0 | local key lock | — |
|
|
@@ -979,7 +994,8 @@ bounded below by **re-acquisition when a delegation lapses without contention**,
|
|
|
979
994
|
the re-acquisition rate are as load-bearing as the handoff rate.
|
|
980
995
|
|
|
981
996
|
**No measurement exists yet on a real cluster** — not a round's latency, not audit growth, not the
|
|
982
|
-
current `lock()` rate
|
|
997
|
+
current `lock()` rate; harper-pro's loopback three-node bench (`replication/RECORD_LOCK_COST_BASELINE.md`,
|
|
998
|
+
`RECORD_LOCK_COST_DELEGATIONS.md`) is the only data. harper-pro#822 already carries an enablement gate; producing these numbers is
|
|
983
999
|
this design's first deliverable, before the protocol change lands. Everything above is a message
|
|
984
1000
|
count, not a benchmark. The gate needs: acquisition latency distribution; delegation re-acquisition
|
|
985
1001
|
rate at candidate `durationMs`; the §4.3 restart quarantine's observed effect on lock availability;
|
|
@@ -990,11 +1006,11 @@ untouched.
|
|
|
990
1006
|
|
|
991
1007
|
**The guarantee decision — §7.3.** What `lock()` promises, on two independent axes:
|
|
992
1008
|
|
|
993
|
-
| | conflict ordering | recovery-mode freshness | cost
|
|
994
|
-
| --------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------- |
|
|
995
|
-
| **Exclusion-only** ✅ | LWW unchanged | narrowed: §2's freshness is not promised on the recovery path | none beyond the protocol
|
|
996
|
-
| **Fenced** | `(generation, timestamp, origin)` | still narrowed | record-contract + stored-format change, reverses
|
|
997
|
-
| **Quorum-confirmed** | either of the above | closes the crashed/unreachable routes, not the late-settling one; §2 as written needs this _and_ fenced | latency on every locked write
|
|
1009
|
+
| | conflict ordering | recovery-mode freshness | cost |
|
|
1010
|
+
| --------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
|
|
1011
|
+
| **Exclusion-only** ✅ | LWW unchanged | narrowed: §2's freshness is not promised on the recovery path | none beyond the protocol |
|
|
1012
|
+
| **Fenced** | `(generation, timestamp, origin)` | still narrowed | record-contract + stored-format change, reverses the Phase 0 contract, owes a tombstone-surviving rejection floor |
|
|
1013
|
+
| **Quorum-confirmed** | either of the above | closes the crashed/unreachable routes, not the late-settling one; §2 as written needs this _and_ fenced | latency on every locked write |
|
|
998
1014
|
|
|
999
1015
|
**Exclusion-only is chosen.** The other two arms are deferred to harper#2540, which carries the
|
|
1000
1016
|
limitations, the costs, and what closing them would take.
|
|
@@ -1015,7 +1031,7 @@ feature that has not yet been measured.
|
|
|
1015
1031
|
> needs a crash:
|
|
1016
1032
|
>
|
|
1017
1033
|
> 1. **A predecessor's write can outrank its successor's under last-write-wins.** LWW compares the
|
|
1018
|
-
> transaction timestamp assigned when the write was _staged_ (`
|
|
1034
|
+
> transaction timestamp assigned when the write was _staged_ (`_writeUpdate`'s `precedesExistingVersion(txnTime, …)`), and lease
|
|
1019
1035
|
> expiry orders admissions, not timestamps. **`lock()` changes nothing about conflict resolution**
|
|
1020
1036
|
> — a locked write and an unlocked write resolve identically, by timestamp, per field, with CRDT
|
|
1021
1037
|
> ops folded and the surviving shape depending on both writes' shapes and on whether auditing is
|
|
@@ -1027,7 +1043,7 @@ feature that has not yet been measured.
|
|
|
1027
1043
|
> still in flight when the successor writes.
|
|
1028
1044
|
> - **(1b) a timestamp pushed ahead on purpose, on a completely clean handoff.** A caller-supplied
|
|
1029
1045
|
> future `context.timestamp`, or a mixed explicit transaction whose later timestamp becomes the
|
|
1030
|
-
> handle's floor — both documented as deliberate Phase 0 behavior (
|
|
1046
|
+
> handle's floor — both documented as deliberate Phase 0 behavior ("Acquisition timestamp and mixed transactions" above). The
|
|
1031
1047
|
> predecessor commits, replicates, drains and releases cleanly; the successor is admitted with
|
|
1032
1048
|
> correct freshness, reads the current value, and writes at real wall-clock time — and its write
|
|
1033
1049
|
> is the older one, silently, because the stored version is stamped in the future. What the
|
|
@@ -1051,18 +1067,18 @@ feature that has not yet been measured.
|
|
|
1051
1067
|
> **§2's exclusion invariant is wider than what (1) and (2) leave.** It requires a successor to
|
|
1052
1068
|
> exclude every predecessor capability that can still admit _or commit_; what ships excludes
|
|
1053
1069
|
> admission, and the commit half holds only up to the pre-submission expiry fence
|
|
1054
|
-
> (`
|
|
1070
|
+
> (the `hasLeaseProtectedWrite` fence in `DatabaseTransaction.performCommit()`) — a native commit that clears that fence and settles
|
|
1055
1071
|
> afterwards is limitation (2)'s third route.
|
|
1056
1072
|
>
|
|
1057
1073
|
> **There is no caller-side mitigation for (2) — the obvious candidate is unreachable and, where it
|
|
1058
1074
|
> is reachable, makes things worse.** `X-Replicate-To` / `confirm=` is meant to be super-user only —
|
|
1059
|
-
> `checkContextPermissions` (`resources/Table.ts
|
|
1075
|
+
> `checkContextPermissions` (`resources/Table.ts`) raises 403 otherwise, though its truthiness
|
|
1060
1076
|
> gate lets `X-Replicate-To: 0` through, which is harper#2546 and not a mitigation anyone should
|
|
1061
1077
|
> build on. And where it is legitimately available it is a **residency** directive before it is a
|
|
1062
1078
|
> confirmation knob: absent a `getResidencyById` function, which short-circuits ahead of it
|
|
1063
|
-
> (`resources/Table.ts
|
|
1064
|
-
> existing residency on update
|
|
1065
|
-
> the database's configured `replication.replicateTo` count (
|
|
1079
|
+
> (`getResidency` in `resources/Table.ts`), a numeric value sets residency to `[self, ...N nodes]` and truncates
|
|
1080
|
+
> existing residency on update, while `*` leaves `replicateTo` undefined and falls back to
|
|
1081
|
+
> the database's configured `replication.replicateTo` count (`replicateToCount`) — neither of which is
|
|
1066
1082
|
> guaranteed to be the whole cluster. In a twelve-node cluster configured `replicateTo: 3`, either
|
|
1067
1083
|
> form leaves the record on four nodes, and a successor's barrier over the other eight reachable
|
|
1068
1084
|
> members satisfies without any of them ever having held the locked write. That is the opposite of
|
|
@@ -1083,7 +1099,7 @@ feature that has not yet been measured.
|
|
|
1083
1099
|
caller whose locked write loses gets a 200, no log line and no counter, so nothing distinguishes it
|
|
1084
1100
|
from correct behavior. Making it observable is cheap and confined to the lock path — the handle
|
|
1085
1101
|
already carries its floor and the commit path already fences per write on `write.lockHandle`
|
|
1086
|
-
(`
|
|
1102
|
+
(the `hasLeaseProtectedWrite` fence in `DatabaseTransaction.performCommit()`), so a lock-path-only check that a staged version exceeds
|
|
1087
1103
|
wall-clock, and a counter when a barrier admits with no dependency set, cost nothing on ungated
|
|
1088
1104
|
writes. It is not in scope here because it is detection rather than guarantee, but it belongs in
|
|
1089
1105
|
harper#2541 rather than nowhere.
|
|
@@ -1119,7 +1135,7 @@ found them in code this note keeps rather than in the arbitration rule it delete
|
|
|
1119
1135
|
away on their own. **All three are fixed on this branch** as part of landing the replacement; they are
|
|
1120
1136
|
recorded here because the reasoning is the design's, not the fix's:
|
|
1121
1137
|
|
|
1122
|
-
- **Transport replacement does not fence live authority** (`
|
|
1138
|
+
- **Transport replacement does not fence live authority** (`registerClusterLockTransport`, now fenced). Re-registering
|
|
1123
1139
|
a transport — a component reload is enough — closes the current coordinator and installs an empty
|
|
1124
1140
|
one. `close()` clears coordinator state but does not invalidate handles already handed out, and a
|
|
1125
1141
|
handle checks only its own release and lease fields. The successor coordinator can then grant the
|
|
@@ -1127,13 +1143,13 @@ recorded here because the reasoning is the design's, not the fix's:
|
|
|
1127
1143
|
restart, and §5's answer applies here too: the replacement must either carry live authority across
|
|
1128
1144
|
the swap or fence and settle every outstanding handle before it may grant. Unregister/re-register
|
|
1129
1145
|
and transport-object replacement both need coverage.
|
|
1130
|
-
- **The direct receive callback has no containment** (`
|
|
1146
|
+
- **The direct receive callback has no containment** (`deliverLockControlEntry`, now contained). The
|
|
1131
1147
|
resolver calls `Table.lockCoordinator`, which throws `LockUnavailableError` when the node name is
|
|
1132
1148
|
unusable, and that throw happens before `applyEntry()`'s own containment can catch it, so it
|
|
1133
1149
|
escapes `deliverLockControlEntry()`. The source-subscription sink already handles this
|
|
1134
|
-
(`resources/Table.ts
|
|
1150
|
+
(in `resources/Table.ts`); the direct callback must too. This is §8's rule — a receive boundary
|
|
1135
1151
|
settles its callers and keeps admission closed — applied to a path that exists today.
|
|
1136
|
-
- **The commit fence
|
|
1152
|
+
- **The commit fence scanned every write on every commit** (now gated on `hasLeaseProtectedWrite`).
|
|
1137
1153
|
The loop runs on ordinary transactions in core-only deployments that never register a transport, so
|
|
1138
1154
|
a bulk transaction with 100,000 plain writes pays 100,000 property checks before submission. §8
|
|
1139
1155
|
requires ordinary writes to keep their existing ungated path, so the transaction must track whether
|
|
@@ -1165,17 +1181,34 @@ Added — **harper-pro's delegation server, on harper-pro#822**:
|
|
|
1165
1181
|
node principal of the connection, never a payload field, so a `super_user` human cannot mint or
|
|
1166
1182
|
clear a delegation. An operation accepted on a non-owner thread relays through main to the owner
|
|
1167
1183
|
worker under a bound, and a relay that times out answers `not-home`, never a grant.
|
|
1168
|
-
- The `ClusterLockTransport` implementation, capability level
|
|
1184
|
+
- The `ClusterLockTransport` implementation, capability level 4 (`RECORD_LOCKS_CAPABILITY`) and the enablement gate, in
|
|
1169
1185
|
`replication/recordLockTransport.ts` and `replication/protocolCapabilities.ts`.
|
|
1170
1186
|
|
|
1171
|
-
|
|
1187
|
+
Shipped in harper-pro#822 and harper-pro#865:
|
|
1188
|
+
|
|
1189
|
+
- **The operator-agreed home map (§4)** behind `transport.homeMap(database)`: an operator-staged,
|
|
1190
|
+
explicitly activated `recordLockHomes` generation with a peer digest check
|
|
1191
|
+
(`replication/recordLockHomes.ts`, `RECORD_LOCK_HOMES_DESIGN.md`). harper-pro#825 still carries its
|
|
1192
|
+
earlier "durable membership epoch" title, though this is the scope that shipped.
|
|
1193
|
+
- **`homeIncarnation` advanced per coordination incarnation, not per process** (§5.1), durable and
|
|
1194
|
+
monotonic in this node's `hdb_nodes` row (`recordLockIncarnation`,
|
|
1195
|
+
`replication/recordLockTransport.ts`). It matters because coordinator state is per-thread: a
|
|
1196
|
+
replacement coordinating worker restarts the delegation counter at zero, so an incarnation that
|
|
1197
|
+
only advanced per process would let it re-mint tokens its predecessor issued, and a delayed
|
|
1198
|
+
release carrying one of those tokens would clear a live grant.
|
|
1199
|
+
- **The transport registered on every worker thread that can serve a `lock()`** (harper-pro#865, for
|
|
1200
|
+
harper-pro#852). The transport registers from the replication built-in's `start()`, and a built-in
|
|
1201
|
+
is a trusted plugin that `placedOnThisThread` loads on **every** http worker, dedicated
|
|
1202
|
+
application workers (harper#2524) included — so every serving thread latches
|
|
1203
|
+
`clusterRequiredDatabases` and fails closed rather than taking the Phase 0 lock alone. The
|
|
1204
|
+
`ownsCoordination()` 503 is no longer the path a non-owner takes: instead of answering 503 it
|
|
1205
|
+
**relays** the acquire (and the matching release) to the coordinating worker over the worker port
|
|
1206
|
+
mesh, installing the granted admission locally as a remote admission whose handle a recall on the
|
|
1207
|
+
owner fences before the delegation release is written. A `lock()` therefore succeeds uniformly on
|
|
1208
|
+
every http worker at `threads.count > 1`.
|
|
1209
|
+
|
|
1210
|
+
Still owed by harper-pro:
|
|
1172
1211
|
|
|
1173
|
-
- The operator-agreed home map (§4) behind `transport.homeMap(database)` (harper-pro#825). #822
|
|
1174
|
-
already carries a **static** map as scaffolding, and under this design static is the right shape —
|
|
1175
|
-
what it lacks is the two properties §4.1 requires: an operator-published `recordLockHomes`
|
|
1176
|
-
generation rather than one each node derives for itself, and a digest agreed across peers before
|
|
1177
|
-
grants are enabled — plus §4.3's one-shot activation record, which is what makes a generation change
|
|
1178
|
-
safe rather than merely announced.
|
|
1179
1212
|
- **A received `lockRelease` routed to the thread that owns coordination.** Core's log-delivery sink
|
|
1180
1213
|
runs wherever `subscribeOnThisThread(applicationWorkerIndex())` is true, which since harper#2524's
|
|
1181
1214
|
dedicated application workers is _routinely_ a different thread from the coordinating one — so this
|
|
@@ -1183,30 +1216,6 @@ Still owed by harper-pro — **the one thing that blocks enablement**:
|
|
|
1183
1216
|
thread (it counts and warns, it cannot forward), and the home then holds its grant until
|
|
1184
1217
|
`DELEGATION_LEASE_MS + skew` even though the delegate stopped admitting cleanly. Relaying it is the
|
|
1185
1218
|
same obligation the delegation RPC already carries.
|
|
1186
|
-
- **The transport registered on every worker thread that can serve a `lock()`**, not only the
|
|
1187
|
-
coordinating one, and including a dedicated application worker (harper#2524). Core cannot check
|
|
1188
|
-
it: the "this database is clustered" latch is per-thread module state, so a worker that never
|
|
1189
|
-
registers never fails closed, and a default-scoped `lock()` there takes the Phase 0 node lock alone
|
|
1190
|
-
while a peer runs the cluster protocol — two nodes admitting one key. Registering everywhere is also
|
|
1191
|
-
what makes the `ownsCoordination()` 503 reachable, which is the path a non-owner worker is supposed
|
|
1192
|
-
to take.
|
|
1193
|
-
- _Addressed in harper-pro#852._ The transport registers from the replication built-in's `start()`,
|
|
1194
|
-
and a built-in is a trusted plugin that `placedOnThisThread` loads on **every** http worker,
|
|
1195
|
-
dedicated application workers (harper#2524) included — so every serving thread latches
|
|
1196
|
-
`clusterRequiredDatabases` and fails closed rather than taking the Phase 0 lock alone. The
|
|
1197
|
-
`ownsCoordination()` 503 is no longer the path a non-owner takes: instead of answering 503 it
|
|
1198
|
-
**relays** the acquire (and the matching release) to the coordinating worker over the worker port
|
|
1199
|
-
mesh, installing the granted admission locally as a remote admission whose handle a recall on the
|
|
1200
|
-
owner fences before the delegation release is written. A `lock()` therefore succeeds uniformly on
|
|
1201
|
-
every http worker at `threads.count > 1`.
|
|
1202
|
-
- **`homeIncarnation` advanced per coordination incarnation, not per process** (§5.1). Core cannot mint
|
|
1203
|
-
it — it must be durable and monotonic — and cannot check it, which puts it in the same class as
|
|
1204
|
-
§4.3's activation record. It matters because coordinator state is per-thread: a replacement
|
|
1205
|
-
coordinating worker restarts the delegation counter at zero, so an incarnation that only advanced
|
|
1206
|
-
per process would let it re-mint tokens its predecessor issued, and a delayed release carrying one
|
|
1207
|
-
of those tokens would clear a live grant. Derived-per-node is why the transport ships gated off: two nodes that disagree
|
|
1208
|
-
derive different rings and can both grant one key. #825 is now a config generation, a digest check
|
|
1209
|
-
and the §4.3 change runbook — not a consensus protocol.
|
|
1210
1219
|
|
|
1211
1220
|
#### Protocol version and mixed deployments
|
|
1212
1221
|
|
|
@@ -1217,21 +1226,20 @@ shipped enabled, nibbles 9/10 are retired rather than migrated, and the `LOCK_RE
|
|
|
1217
1226
|
historically a fixed five-field tuple `[key, requester, generation, homeIncarnation, counter]` — is
|
|
1218
1227
|
now `[1, key, requester, generation, homeIncarnation, counter, dependencies]`. The decoder accepts
|
|
1219
1228
|
the historical tuple as unknown lineage (therefore recovery), ignores unknown future versions, and
|
|
1220
|
-
still requires an exact fencing-token match before any release can clear a live grant.
|
|
1221
|
-
|
|
1229
|
+
still requires an exact fencing-token match before any release can clear a live grant. harper-pro
|
|
1230
|
+
advertises a new mutually exclusive capability level for this wire/API contract
|
|
1231
|
+
(`RECORD_LOCKS_CAPABILITY` = 4). The barrier
|
|
1222
1232
|
entry (§7.2; nibble 13, payload `[1, nonce]`) rides the same capability: a receiver that predates it
|
|
1223
1233
|
resolves the nibble to no entry type at all and the replication sink treats it as an unknown
|
|
1224
1234
|
operation, so such a peer must not be sent one — and a recovery drain that reaches a member which
|
|
1225
1235
|
cannot produce a barrier has no fence and fails closed.
|
|
1226
1236
|
|
|
1227
|
-
**Merging the substrate
|
|
1228
|
-
as the new protocol.
|
|
1237
|
+
**Merging the substrate was gated on removing RA:** harper#2498 merged with no RA arbitration reachable.
|
|
1229
1238
|
|
|
1230
1239
|
### 12. Verification route
|
|
1231
1240
|
|
|
1232
1241
|
- **Unit**, coordinator as a pure state machine with **independent per-node clocks** rather than one
|
|
1233
|
-
shared fake clock (`
|
|
1234
|
-
cannot express §5.2): delayed grant reply rejected; renewal/restart overlap; recall against a live
|
|
1242
|
+
shared fake clock, which cannot express §5.2 (`recordLockCoordinator.test.js` "independent clocks"): delayed grant reply rejected; renewal/restart overlap; recall against a live
|
|
1235
1243
|
handle that staged nothing (§6); recall against an unlocked-but-staged write; transitive freshness
|
|
1236
1244
|
including the equal-timestamp and dependent-patch cases (§7.1); fencing-token ordering across a home
|
|
1237
1245
|
restart (§5.1); and the negative cases — grant inside the §4.3 restart quarantine, a request naming
|
|
@@ -1248,7 +1256,7 @@ as the new protocol.
|
|
|
1248
1256
|
refuse to interoperate; feature disabled
|
|
1249
1257
|
is byte-for-byte the current write path. Counter tests assert observed read/write histories, not
|
|
1250
1258
|
only final totals, and snapshot-free reload must still work after the applied-history fence
|
|
1251
|
-
(`
|
|
1259
|
+
(`#reloadLocked`). Under exclusion-only there is no fenced-mode suite to add here; the stale
|
|
1252
1260
|
writes, patches and deletes against successor records and reclaimed tombstones belong to
|
|
1253
1261
|
harper#2540 if that arm is ever taken. What this suite **must** assert instead is the documented
|
|
1254
1262
|
narrowing itself, as expected outcomes rather than test failures. Two schedules, and both must
|
|
@@ -1273,13 +1281,12 @@ as the new protocol.
|
|
|
1273
1281
|
|
|
1274
1282
|
### 13. Rollout
|
|
1275
1283
|
|
|
1276
|
-
harper#2498
|
|
1277
|
-
|
|
1278
|
-
|
|
1279
|
-
|
|
1280
|
-
|
|
1281
|
-
|
|
1282
|
-
`establishLockFreshness()` or an agreed home map, and no core build supplies them.
|
|
1284
|
+
harper#2498 merged without Ricart–Agrawala: the §11 substrate landed with the arbitration replaced.
|
|
1285
|
+
harper-pro#822 kept its capability, participant-set, ownership and switch work and replaced its
|
|
1286
|
+
transport with the delegation wire, the operator-agreed home map and `establishLockFreshness()`.
|
|
1287
|
+
Neither is enabled by default: cluster record locks need `replication.recordLocks: true` on RocksDB,
|
|
1288
|
+
and core still fails closed without `establishLockFreshness()` or an agreed home map, which no core
|
|
1289
|
+
build supplies.
|
|
1283
1290
|
|
|
1284
1291
|
### 14. Decomposition and implementation status
|
|
1285
1292
|
|
|
@@ -1291,23 +1298,25 @@ Two things had to be settled before implementation, and one still is:
|
|
|
1291
1298
|
harper-pro#824.
|
|
1292
1299
|
|
|
1293
1300
|
The work is decomposed as harper-pro#825 (the operator-agreed home map, §4 — a `recordLockHomes`
|
|
1294
|
-
generation, a peer digest check and the §4.3 change runbook), harper#2541 (home ring, delegations,
|
|
1301
|
+
generation, a peer digest check and the §4.3 change runbook; shipped in harper-pro#822), harper#2541 (home ring, delegations,
|
|
1295
1302
|
drain and caps, §§5/6/8, plus the three inherited substrate defects in §11) and harper#2542
|
|
1296
1303
|
(successor freshness, §7).
|
|
1297
1304
|
|
|
1298
|
-
**Status.** harper#2541 and harper#2542's core halves are implemented: the Ricart–Agrawala state
|
|
1299
|
-
is gone, and the home ring, the delegation table, recall with §6 steps 1, 2 and 4, ordered
|
|
1300
|
-
tokens, the caps, the §4.3 restart quarantine, the generation monotonicity floor and the
|
|
1301
|
-
transport-swap grant fence are in place, with all three inherited defects fixed. **§6 step 3 —
|
|
1302
|
-
|
|
1303
|
-
|
|
1304
|
-
|
|
1305
|
-
|
|
1306
|
-
|
|
1307
|
-
|
|
1308
|
-
|
|
1309
|
-
|
|
1310
|
-
|
|
1305
|
+
**Status.** harper#2541 and harper#2542's core halves are implemented: the Ricart–Agrawala state
|
|
1306
|
+
machine is gone, and the home ring, the delegation table, recall with §6 steps 1, 2 and 4, ordered
|
|
1307
|
+
fencing tokens, the caps, the §4.3 restart quarantine, the generation monotonicity floor and the
|
|
1308
|
+
transport-swap grant fence are in place, with all three inherited defects fixed. **§6 step 3 —
|
|
1309
|
+
settlement — is not implemented**: a recall revokes capability and then writes the release without
|
|
1310
|
+
waiting for a native commit already submitted to complete. That is exactly limitation (2)'s third
|
|
1311
|
+
route in §10, which the contract already states, so the gap is disclosed rather than hidden — but §6
|
|
1312
|
+
must not be read as fully implemented. Closing it means hanging the release hook off
|
|
1313
|
+
logical-transaction settlement rather than off the last admission unlocking. harper-pro#822 supplies
|
|
1314
|
+
the operator-agreed home map and `establishLockFreshness()`; the feature is opt-in
|
|
1315
|
+
(`replication.recordLocks: true`). Still missing: relaying a `lockRelease` that reaches a
|
|
1316
|
+
non-coordinating thread (`applyEntry` counts, warns and drops it). **The measurement gate
|
|
1317
|
+
(harper-pro#824) has loopback results only** (`replication/RECORD_LOCK_COST_DELEGATIONS.md`) and
|
|
1318
|
+
remains open, and the decision to implement ahead of it was the human's, recorded here so the
|
|
1319
|
+
sequence is not mistaken for the one this note recommends.
|
|
1311
1320
|
|
|
1312
1321
|
One behavior surfaced by implementing §6 is still an open decision, tracked as harper#2580: a
|
|
1313
1322
|
`{hold: true}` write that was staged and then unlocked is **revoked** rather than waited for when a
|
|
@@ -1350,10 +1359,15 @@ add-only Bloom filter records every key delegated in the current generation. Abs
|
|
|
1350
1359
|
only when the coordinator has observed the entire generation continuously; cold construction,
|
|
1351
1360
|
coordination loss, and generation change saturate that assumption so every unremembered key takes
|
|
1352
1361
|
recovery. A continuously observed key absent from the filter needs no barrier; a key present with no
|
|
1353
|
-
retained set (expiry or LRU eviction) receives the recovery marker.
|
|
1354
|
-
marker by preferring a retained durable release when available, otherwise draining every
|
|
1355
|
-
member to captured positions
|
|
1356
|
-
|
|
1362
|
+
retained set (expiry or LRU eviction) receives the recovery marker. This design asks the transport to
|
|
1363
|
+
resolve that marker by preferring a retained durable release when available, otherwise draining every
|
|
1364
|
+
reachable member to captured positions, and to coalesce concurrent recovery snapshots across keys.
|
|
1365
|
+
harper-pro's transport (`replication/recordLockFreshness.ts`) does neither yet: it drains every other
|
|
1366
|
+
home-map member to a `lockBarrier`, one barrier per wait, and a member that cannot produce one fails the
|
|
1367
|
+
lock closed with 503. So while any home-map member is down, every recovery-path `lock()` in that
|
|
1368
|
+
database returns 503 (the first access after a home restart or a generation change, or a key whose
|
|
1369
|
+
lineage the LRU evicted) — stricter than the reachable-member barrier §10 limitation (2) describes.
|
|
1370
|
+
Either way it returns the established positions so the delegation can carry them onward. Bloom false positives
|
|
1357
1371
|
only select the slower safe path; there are no false negatives while absence is trusted.
|
|
1358
1372
|
|
|
1359
1373
|
The recovery drain's fence is the `lockBarrier` control entry (harper#2625). Each reachable member
|
|
@@ -1366,8 +1380,8 @@ like the release, and the coordinator ignores it. `establishLockFreshness` also
|
|
|
1366
1380
|
remaining on the lock deadline so the transport can bound the drain to it. A member or transport
|
|
1367
1381
|
that cannot produce a barrier leaves the recovery marker with no fence, and the lock fails closed
|
|
1368
1382
|
with the same 503 — the transport's rejection is what fails it: core does not second-guess a
|
|
1369
|
-
positioned recovery reply
|
|
1370
|
-
included.
|
|
1383
|
+
positioned recovery reply beyond normalizing it: `#establishFreshness` refuses a set it cannot use,
|
|
1384
|
+
keeps only origins in the current home map, and installs the grant on the rest, empty included.
|
|
1371
1385
|
|
|
1372
1386
|
The release wire format becomes a versioned tuple with a leading version and trailing dependency
|
|
1373
1387
|
set. The decoder still accepts the exact historical five-tuple as a release with unknown lineage;
|