dexbot 1.7.1 → 1.7.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +28 -0
- package/claw/package.json +1 -1
- package/claw/runtimes/openclaw-plugin/openclaw.plugin.json +1 -1
- package/claw/runtimes/openclaw-plugin/package.json +1 -1
- package/dist/modules/account_bots.d.ts +1 -0
- package/dist/modules/account_bots.js +7 -5
- package/dist/modules/constants.d.ts +2 -0
- package/dist/modules/constants.js +16 -2
- package/dist/modules/dexbot_class.d.ts +22 -0
- package/dist/modules/dexbot_class.js +30 -0
- package/dist/modules/dexbot_cow_runtime.d.ts +31 -1
- package/dist/modules/dexbot_cow_runtime.js +68 -56
- package/dist/modules/dexbot_maintenance_runtime.d.ts +1 -1
- package/dist/modules/dexbot_maintenance_runtime.js +33 -6
- package/dist/modules/node_flap_score.d.ts +6 -6
- package/dist/modules/node_flap_score.js +6 -6
- package/dist/modules/order/accounting.d.ts +9 -4
- package/dist/modules/order/accounting.js +89 -26
- package/dist/modules/order/async_lock.d.ts +7 -1
- package/dist/modules/order/async_lock.js +7 -1
- package/dist/modules/order/export.js +25 -18
- package/dist/modules/order/fill_ledger.d.ts +99 -0
- package/dist/modules/order/fill_ledger.js +103 -0
- package/dist/modules/order/grid.d.ts +75 -18
- package/dist/modules/order/grid.js +207 -61
- package/dist/modules/order/grid_reconcile_internal.js +16 -4
- package/dist/modules/order/index.d.ts +14 -5
- package/dist/modules/order/index.js +5 -5
- package/dist/modules/order/lifecycle.js +3 -3
- package/dist/modules/order/manager.d.ts +15 -24
- package/dist/modules/order/manager.js +40 -27
- package/dist/modules/order/sync_engine.d.ts +66 -118
- package/dist/modules/order/sync_engine.js +118 -162
- package/dist/modules/order/utils/math.d.ts +7 -3
- package/dist/modules/order/utils/math.js +7 -3
- package/dist/modules/order/utils/order.d.ts +40 -41
- package/dist/modules/order/utils/order.js +110 -84
- package/dist/modules/order/utils/system.d.ts +81 -6
- package/dist/modules/order/utils/system.js +167 -29
- package/dist/modules/timing_policy.js +2 -2
- package/dist/modules/types.d.ts +22 -27
- package/docs/COW_INVARIANTS.md +5 -1
- package/docs/DEXBOT_COMPARISON.md +4 -4
- package/docs/EVOLUTION.md +6 -5
- package/docs/FUND_MOVEMENT_AND_ACCOUNTING.md +1 -1
- package/docs/GRID_RECONCILE.md +3 -3
- package/docs/ORDER_ENGINE_POST_1.0_RETROSPECTIVE.md +38 -30
- package/docs/README.md +1 -1
- package/docs/architecture.md +26 -8
- package/modules/README.md +3 -0
- package/package.json +3 -3
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,34 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
|
|
5
|
+
## [1.7.2] - 2026-11-04 - One Fill Contract with Fail-Closed Grid Blockers, Single Fill Identity, Broadcast-Region Leak Repair, Fill-Anchored Rebuild Boundary, Spread Rotation Into Orphaned Rails
|
|
6
|
+
|
|
7
|
+
### 2026-11-04
|
|
8
|
+
|
|
9
|
+
- **Feat(order)**: give the fill pipeline one written contract and make its blocking decision fail-closed — the fill path's failures were not missing safeguards but one question answered in several places, so a "wait, then guess" was added wherever the answers disagreed. `docs/FILL_PIPELINE.md` now names the six stages (ingest, ledger, pivot, plan, lease, commit/observe), the four fill states, one owner per question and `INV-FILL-001..007`, with a ratchet test over the ten `_processFillsWithBatching` call sites, the two pivot writers, the three boundary-budget files and the nine dedupe references. `hasUnsettledFills()` peeked four raw fields and returned `{unsettled:false}` on ANY throw — a fail-OPEN answer in a fail-closed system, which is how a grid was updated from a stale master; `resolveGridUpdateBlockers()` now returns eight closed codes and `readFillPipelineState()` is the single field reader (reporting `ok:false`, i.e. unknown-blocker, instead of "settled"), so the prose reason and the codes cannot disagree. Broadcast liveness composes through `getLivePendingBroadcasts` (new read-only `gc:false` mode, so a predicate cannot GC the pending set), the region through `isPlacementActive`, the rebuild through the manager flag; only `unknown-blocker` escalates, after `GRID_BLOCKED_ESCALATE_CYCLES` consecutive attempts (a cycle counter, not a timing window). Behavioral change: the targeted-drift gate now defers on ANY blocker, not only the fill codes — it is a chain-mutating reconcile and the old condition could escalate on an unreadable probe and still reconcile in the same tick. Errors now surface as a stalled-but-bounded grid update rather than a grid built from stale state.
|
|
10
|
+
|
|
11
|
+
- **Feat(order)**: one fill identity — `modules/order/fill_ledger.ts` owns `fillKeyOf` (`buildFillKey` stays as an alias) and `resolveTradedSide`, and `isFillBooked` exposes the persisted store's membership without claiming it (booking still belongs to the store). It is wired into the three former copies of the pays-asset map (`_computeFillContext`, `attributeUnmatchedFillToGrid`, the queued pivot refresh), with the SPREAD price fallback left with the caller that owns the ladder. Docs: `INV-RECON-006` now records the chain-observed typeMismatch exemption, `INV-SYNC-008` records that the pays mapping has one implementation and that `slot.type` is the ACCOUNTING side, not rail identity.
|
|
12
|
+
|
|
13
|
+
- **Fix(order)**: claim the fill key at credit time and drop the duplicate amount channel — `processFillAccounting` wrote the store tracker only AFTER a successful persist flush, so in the window between "key queued" and "flush done" a second event with the same key passed the duplicate check (the only protection was the bot-level 5 s claim lease, absent in a bare manager); "credited" and "tracked" could disagree, which is the family that produced the double-released size and the phantom residual. The key is now claimed in the store tracker BEFORE `_applyBalanceAdjustments` and before the pass-1 marker is consumed, and released on every failure path, so "credited => tracked" and "not credited => no claim" hold; `_applyBalanceAdjustments` is ALL-OR-NOTHING (an applied prefix is reversed before the error propagates) so the released claim's retry cannot apply that prefix twice; `_isNewFillKey` asks the store first, so a booked key is refused at ingest even after its lease expired instead of after a `FILL DETECTED` log and queue work. The boolean half of the applied amount (`alreadyAppliedChainSize` / `__chainSizeAlreadyApplied` and its dead consume wrapper) is REMOVED rather than asserted: the aggregated path read it off the LAST group entry only, which is exactly the phantom-residual bug — one channel remains, the amount. `FILL_DEDUPE_WINDOW_MS` stays, re-documented as a CLAIM LEASE: claims restored from the crash snapshot are the one case where nobody knows whether the credit happened, and blocking them forever loses a fill. `docs/FILL_PIPELINE_PLAN.md` is folded into `docs/FILL_PIPELINE.md` and removed. Risk: a failure inside the credit now releases the claim and the fill is retried; the all-or-nothing rollback is what makes that retry safe. No order sizing, boundary or strategy behaviour changed.
|
|
14
|
+
|
|
15
|
+
- **Fix(order)**: make the fill claim a real single winner and delete the second grace-stamp writer — the pre-credit duplicate check ran before `persist()` awaited, so two concurrent `processFillAccounting` calls could both pass it and both credit (reproduced: 998 base / 100005 quote for one fill); the claim is now a synchronous check-and-set with no await in between, and the loser returns duplicate without discarding the queued write (keyed by fill key, it belongs to the winner) and without being counted as a booking. The claim, the duplicate gate and rollback now use ONE map (the store tracker when the store exists, the manager tracker otherwise); in a wired bot they are the same map, but a split fixture would otherwise claim a key the ingest check cannot see. `recordOrderPlacement` is deleted: it had no production caller, and its "writes through the same facade" comment was false — it wrote `_placedAt` directly, which `isFreshlyPlacedOrder` ignores whenever `getLifecycle` exists. The offline exporter now derives BUY/SELL through the ledger's `resolveTradedSide` instead of its own copy of the pays-asset mapping, keeping its stricter "both sides must be this pair" guard. Also corrects retention wording: a booked key is refused while retained regardless of age; `FILL_RECORD_RETENTION_MS` bounds retention, it is not an expiry that re-allows a credit.
|
|
16
|
+
|
|
17
|
+
- **Fix(order)**: release leaked broadcast regions and never park a chain-observed wrong-side cancel — the COW batch executor called `startBroadcasting()` before building ops, but only the inner broadcast `try/finally` released the refcount, so every early return in between (all CREATEs skipped by the last-fill/size guards, fund validation, stale-plan re-plan, single-flight/placement aborts) leaked the region and froze the fill consumer until the 120 s stale-flag watchdog; the delayed backlog was then replayed chunk by chunk against a stale boundary and the bot re-placed sells below levels it had already sold. The region is now tracked per frame and released from the inner finally, the catch and the outer finally (idempotently, so a second raw `stopBroadcasting()` cannot underflow another holder) and released before the best-effort cleanup, and a chain-observed typeMismatch is exempt from the 15-minute fresh-placement grace because a wrong-side order is known-wrong — letting it rest misclassifies its next fill (phantom zero-size fill, dust cancel, re-buy at the fill level); regular surplus cancels keep the grace. A `typeMismatch` the pre-batch drain did not resolve (deferred by an active region or failed) is no longer parked either, while a plan-shaped `reason='type-mismatch'` action still waits, because it only proves `master != target` — the stale-target lag the grace exists for.
|
|
18
|
+
|
|
19
|
+
- **Fix(order)**: derive the fill side from the paid asset — a slot can be retyped (BUY↔SELL) before its resting chain order is corrected, and `_computeFillContext()` trusted `matchedGridOrder.type`, so a real SELL fill on a BUY-typed slot converted the paid base amount with quote precision: `filledAmount` became 0, the slot was treated as partially filled and the bot re-bought at the fill level (2026-11-04 incident). The side is now resolved from the fill's paid asset for every slot type (pays quote = BUY, pays base = SELL), which also selects the correct size asset (BUY=quote, SELL=base) so `filledAmount`/precision stay dimensionally consistent with the order; SPREAD without a usable pays asset keeps the price-vs-`startPrice` fallback, non-SPREAD without one keeps the slot type, and the same undefined/undefined identity collision is guarded in the unmatched-fill attribution and the queued pivot refresh. `fillProcessing.COALESCE_BACKLOG_REPLAY` is rolled out for the configured bots (profile data, ignored by git); the code default stays OFF. Risk: a slot whose fill carries no pays asset id keeps the previous side inference.
|
|
20
|
+
|
|
21
|
+
- **Fix(order)**: repair a wrong-side slot on rail geometry, not on the stored label — a partial fill stores the traded side into the slot so `size` stays dimensionally consistent for accounting/`recalculateFunds`, which made a queued typeMismatch entry look "repaired" and dropped the residual cancel, leaving the order on the wrong rail. The correction validators now treat a level as repaired only when the stored type AND its rail geometry (index vs boundary, including the gap band) agree with the chain side; unknown geometry keeps the old stored-type check so a rail is never invented. Covered by the cancel-guard, geometry and fill-side edge-case suites (`test_fresh_placement_cancel_grace_paths`, `test_correction_queue_staleness`, `test_live_geometry_provenance`, `test_sync_fill_history_batch`), each mutation-checked.
|
|
22
|
+
|
|
23
|
+
- **Fix(grid)**: anchor a rebuilt or reloaded boundary to the last fill — a rebuild/reload derived the band from the live market center while the LAST-FILL-GUARD pivot survived; when the center sat ahead of the pivot, the startup picker skipped the top buy levels (at/through the fill), filled the same-side window lower and left the first ask at the old `sellStart`, so the realized spread was two grid steps wider than target with no fill reason for the extra width (a live market-pair bot: boundary 159 after a BUY at new-ladder slot 158, realized 2.12% vs the 1.5% target). The existing fill re-anchor now runs on both the rebuild and the load path (`BUY fill at f => boundary <= f - 1`, `SELL fill at f => boundary >= f - gapSlots`) after the pivot is re-armed/restored and before reconcile/re-type, and `applyRebuildFillAnchor()` re-types the rails atomically for the anchored boundary so the counter SELL at `f + gapSlots` is a rail order and the boundary published to runtime consumers matches the slot roles. `loadGrid` previously changed only a local boundary while the persisted value stayed in the manager — fixed. The free-fund spread correction did not run in the incident because no maintenance tick followed bootstrap; the reconcile now owns the first window.
|
|
24
|
+
|
|
25
|
+
- **Fix(spread)**: rotate into orphaned rail slots when funds are fully reserved — a fill-anchored boundary move can leave the two counter levels as orphaned VIRTUAL rail slots (e.g. the old sell window starts two levels above the new `sellStart`), and the spread correction's funding probe read `availableFund=0` because the virtual tail reserved exactly the whole free balance (the chain-truth budget deliberately treats virtual == chainFree equality as a real constraint). Only the no-new-commitment rotation path could repair that, but it was restricted to boundary-promoted gap slots — so a fully allocated rail stayed wide forever even though the far resting orders could have moved into the window edge without any new funds. The rotation source/destination pairing now covers every unfunded tightening candidate, while boundary derivation stays restricted to genuinely promoted destinations so an in-rail orphan rotation cannot invent a boundary move. Pinned by `PF-031` (SELL orphan rotation with virtual == free), `PF-032` (fund-selected BUY blocked by the last fill falls back to the SELL orphan rotation) and `PF-033` (BUY mirror).
|
|
26
|
+
|
|
27
|
+
- **Fix(order)**: make the grid-blocked escalation counter truly consecutive — `GRID_BLOCKED_ESCALATE_CYCLES` is documented as N *consecutive* unreadable cycles, but `noteGridUpdateBlockers` only cleared the counter on a healthy call, and its only caller (the targeted-drift gate) returns before evaluating the gate when `getTargetedSyncReason()` is null; a counter parked at the limit therefore survived an arbitrarily long trigger-less stretch and the first unreadable cycle afterwards fired a structural resync instead of waiting the full window. The trigger-less early return now clears the counter (guarded on the manager), the read-only/GC asymmetry in `getLivePendingBroadcasts` is documented and pinned (a `gc:false` probe keeps an entry the write path would drop — the intended fail-closed answer), and `_gridUpdateBlockedCycles` is declared on `OrderManager`/`OrderManagerLike` so the single writer/reader pair is typed.
|
|
28
|
+
|
|
29
|
+
- **Test(scripts)**: actually run the live suite and ship the cred stub fixture — `test:live` scoped `RUN_LIVE_BITSHARES_TESTS=1` to `npm run build` only (`VAR=x a && b` exports the variable to `a`, not `b`), so `run-tests.js` never saw it and silently skipped all 14 live files (`skippedLive 14`); the env now sits on the runner command (`skippedLive 0`, the live set really runs). `build:tests` also never copied `tests/helpers/foreign_cred_stub.js`, so `test_unlock_foreign_cred_daemon_live` always failed with `ENOENT`; it is copied with the other plain-JS helpers.
|
|
30
|
+
|
|
31
|
+
- **Docs**: consolidate the fill-pipeline documents and resync drifted module comments — `docs/FILL_PIPELINE.md` is now the single fill document (contract, P0–P2 status by commit, the P3–P5 remainder with gates and stop conditions, the accepted decision register D10–D12, regression canaries, implementation log), `ORDER_LIFECYCLE` records that it shipped in v1.7.1 and that the five legacy structures are facade views, `ORDER_ENGINE_POST_1.0_RETROSPECTIVE` marks the v1.7.1 commits released and adds the post-1.7.1 rows, and the fill-pipeline ratchet pins that the `typeMismatch` validator must keep deriving rail geometry rather than trusting `slot.type` (baseline `bc20d574`). A comment-only sweep of `modules/` fixed 10 orphaned/superseded JSDoc blocks, comments naming symbols that do not exist, descriptions contradicting the code (the sync-engine header still described the pre-genesis PASS 2), six drifted TOC inventories (math 78→80, system 43→52 plus a missing fill-pipeline/last-fill-guard section, order 76→82, grid 34 + 3, accounting 26→27, sync_engine 12→14), 30 JSDoc `import('./types')` references pointing at a non-existent file and a non-existent interface, and `@param` lines still using pre-underscore parameter names; `modules/README.md` gains `outcome.ts`, `timing_policy.ts` and the missing `order/` entries. Full suite: 358/358 files (`npm test`), `npm run test:live` 366 pass (5 diagnostics need a configured account); `typecheck`, `typecheck:analysis`, `check:any`, `verify:browser-bundle` (44/44) and the timing-policy audit clean.
|
|
32
|
+
|
|
5
33
|
## [1.7.1] - 2026-11-04 - One Existence Record per Slot, Placement-Region Hardening, Forced-Retreat Gap Evacuation, Fee-Adjusted Pool Depth, Stability-Ranked Nodes
|
|
6
34
|
|
|
7
35
|
### 2026-11-04
|
package/claw/package.json
CHANGED
|
@@ -409,6 +409,7 @@ declare function loadGeneralSettings(): {
|
|
|
409
409
|
MODE: string;
|
|
410
410
|
OPERATION_TYPE: number;
|
|
411
411
|
COALESCE_BACKLOG_REPLAY: boolean;
|
|
412
|
+
GRID_BLOCKED_ESCALATE_CYCLES: number;
|
|
412
413
|
MAX_CONSECUTIVE_CONSUMER_FAILURES: number;
|
|
413
414
|
CONSUMER_BACKOFF_INITIAL_MS: number;
|
|
414
415
|
CONSUMER_BACKOFF_MAX_MS: number;
|
|
@@ -295,7 +295,7 @@ async function askRequiredString(promptText, defaultValue) {
|
|
|
295
295
|
}
|
|
296
296
|
/**
|
|
297
297
|
* Prompts the user for a cron schedule using interval and time.
|
|
298
|
-
* @param {string}
|
|
298
|
+
* @param {string} _promptText - Unused; the prompt text is built below.
|
|
299
299
|
* @param {string} defaultValue - The default value to use if input is empty.
|
|
300
300
|
* @returns {Promise<string>} The user input.
|
|
301
301
|
*/
|
|
@@ -403,10 +403,12 @@ async function askAssetB(promptText, defaultValue, assetA) {
|
|
|
403
403
|
return askAssetSymbol(promptText, defaultValue, (assetB) => assetB === assetA ? `Invalid: Asset B cannot be the same as Asset A (${assetA})` : null);
|
|
404
404
|
}
|
|
405
405
|
/**
|
|
406
|
-
* Prompts the user for a weight distribution value
|
|
406
|
+
* Prompts the user for a weight distribution value, optionally printing the
|
|
407
|
+
* value legend above the prompt.
|
|
407
408
|
* @param {string} promptText - The prompt text to display.
|
|
408
409
|
* @param {number} [defaultValue] - The default value to use if input is empty.
|
|
409
|
-
* @
|
|
410
|
+
* @param {boolean} showLegend - Print the -1..2 legend line before prompting.
|
|
411
|
+
* @returns {Promise<number|string|undefined>} The numeric value, '\x1b' if ESC, or undefined.
|
|
410
412
|
*/
|
|
411
413
|
async function askWeightDistributionValue(promptText, defaultValue, showLegend) {
|
|
412
414
|
const MIN_WEIGHT = -1;
|
|
@@ -420,7 +422,7 @@ async function askWeightDistributionValue(promptText, defaultValue, showLegend)
|
|
|
420
422
|
* Prompts the user for a weight distribution value with a legend.
|
|
421
423
|
* @param {string} promptText - The prompt text to display.
|
|
422
424
|
* @param {number} [defaultValue] - The default value to use if input is empty.
|
|
423
|
-
* @returns {Promise<number|string>} The numeric value
|
|
425
|
+
* @returns {Promise<number|string|undefined>} The numeric value, '\x1b' if ESC, or undefined.
|
|
424
426
|
*/
|
|
425
427
|
async function askWeightDistribution(promptText, defaultValue) {
|
|
426
428
|
return askWeightDistributionValue(promptText, defaultValue, true);
|
|
@@ -429,7 +431,7 @@ async function askWeightDistribution(promptText, defaultValue) {
|
|
|
429
431
|
* Prompts the user for a weight distribution value without a legend.
|
|
430
432
|
* @param {string} promptText - The prompt text to display.
|
|
431
433
|
* @param {number} [defaultValue] - The default value to use if input is empty.
|
|
432
|
-
* @returns {Promise<number|string>} The numeric value
|
|
434
|
+
* @returns {Promise<number|string|undefined>} The numeric value, '\x1b' if ESC, or undefined.
|
|
433
435
|
*/
|
|
434
436
|
async function askWeightDistributionNoLegend(promptText, defaultValue) {
|
|
435
437
|
return askWeightDistributionValue(promptText, defaultValue, false);
|
|
@@ -352,6 +352,7 @@ declare let FILL_PROCESSING: {
|
|
|
352
352
|
MODE: string;
|
|
353
353
|
OPERATION_TYPE: number;
|
|
354
354
|
COALESCE_BACKLOG_REPLAY: boolean;
|
|
355
|
+
GRID_BLOCKED_ESCALATE_CYCLES: number;
|
|
355
356
|
MAX_CONSECUTIVE_CONSUMER_FAILURES: number;
|
|
356
357
|
CONSUMER_BACKOFF_INITIAL_MS: number;
|
|
357
358
|
CONSUMER_BACKOFF_MAX_MS: number;
|
|
@@ -1041,6 +1042,7 @@ declare function buildDefaultGeneralSettings(): {
|
|
|
1041
1042
|
MODE: string;
|
|
1042
1043
|
OPERATION_TYPE: number;
|
|
1043
1044
|
COALESCE_BACKLOG_REPLAY: boolean;
|
|
1045
|
+
GRID_BLOCKED_ESCALATE_CYCLES: number;
|
|
1044
1046
|
MAX_CONSECUTIVE_CONSUMER_FAILURES: number;
|
|
1045
1047
|
CONSUMER_BACKOFF_INITIAL_MS: number;
|
|
1046
1048
|
CONSUMER_BACKOFF_MAX_MS: number;
|
|
@@ -235,7 +235,12 @@ let TIMING = {
|
|
|
235
235
|
// BTS acquisition for non-BTS pairs: min cooldown between acquisition attempts (minutes)
|
|
236
236
|
BTS_ACQUIRE_COOLDOWN_MIN: 60,
|
|
237
237
|
// Fill processing timing
|
|
238
|
-
|
|
238
|
+
// Claim LEASE for a fill accepted into the queue but not yet booked
|
|
239
|
+
// (deferral, crash). It is NOT the dedupe authority: the processed-fill
|
|
240
|
+
// store refuses a booked key while it is retained, regardless of age;
|
|
241
|
+
// FILL_RECORD_RETENTION_MS bounds that retention, it is not an expiry
|
|
242
|
+
// (docs/FILL_PIPELINE.md stage 2).
|
|
243
|
+
FILL_DEDUPE_WINDOW_MS: 5000,
|
|
239
244
|
FILL_RECORD_RETENTION_MS: 7 * 24 * 60 * 60 * 1000, // 7 days - how long to keep persisted fill records (was 1h, increased to close double-credit window on restarts >1h apart)
|
|
240
245
|
PROCESSED_FILL_PERSIST_BATCH_MS: 250, // 250ms - coalesce processed-fill persistence writes under burst load
|
|
241
246
|
PROCESSED_FILL_PERSIST_BATCH_SIZE: 25, // Flush immediately once this many processed fills are queued
|
|
@@ -955,8 +960,17 @@ let FILL_PROCESSING = {
|
|
|
955
960
|
// into gapSlots+1-op transactions, so this only removes the intermediate
|
|
956
961
|
// plan/broadcast cycles (Stale-Restart #5). Default OFF: opt-in rollout
|
|
957
962
|
// for the live boundary engine, overridable per bot via
|
|
958
|
-
// `fillProcessing.COALESCE_BACKLOG_REPLAY` in
|
|
963
|
+
// `fillProcessing.COALESCE_BACKLOG_REPLAY` in bots.json (or globally via
|
|
964
|
+
// FILL_PROCESSING in general.settings.json).
|
|
959
965
|
COALESCE_BACKLOG_REPLAY: false,
|
|
966
|
+
// GRID_BLOCKED_ESCALATE_CYCLES: how many consecutive attempts a grid update
|
|
967
|
+
// may be refused with an UNREADABLE pipeline state before the blocker
|
|
968
|
+
// escalation fires a structural resync. A cycle count, not a timing window:
|
|
969
|
+
// it is deliberately not a TIMING entry (see modules/timing_policy.ts) and
|
|
970
|
+
// must not be turned into one. Only `unknown-blocker` escalates — a busy
|
|
971
|
+
// pipeline (fills in flight, an unresolved broadcast, a held placement
|
|
972
|
+
// region) is normal work and is bounded by its own watchdog.
|
|
973
|
+
GRID_BLOCKED_ESCALATE_CYCLES: 12,
|
|
960
974
|
// MAX_CONSECUTIVE_CONSUMER_FAILURES: Threshold for the _consumeFillQueue
|
|
961
975
|
// watchdog. Below this count, the consumer re-schedules on every failure
|
|
962
976
|
// via setImmediate. At or above this count, the consumer switches to
|
|
@@ -421,6 +421,28 @@ declare class DEXBot implements BotLike {
|
|
|
421
421
|
* @returns {string|null} Fallback key or null
|
|
422
422
|
*/
|
|
423
423
|
_buildOrphanFillFallbackKey(fill: FillLike): string | null;
|
|
424
|
+
/**
|
|
425
|
+
* Claim a fill key for this cycle, or report that it was already handled.
|
|
426
|
+
*
|
|
427
|
+
* Two authorities, in this order:
|
|
428
|
+
* 1. **The processed-fill store** — booked keys. Checked first and without
|
|
429
|
+
* any clock, because "already credited" does not expire. Before P2 a
|
|
430
|
+
* booked key older than the lease window was accepted again here and
|
|
431
|
+
* only refused later by accounting, so the replay still paid for queue
|
|
432
|
+
* churn and a FILL DETECTED log.
|
|
433
|
+
* 2. **The claim lease** — fills accepted but not yet booked (deferral,
|
|
434
|
+
* crash). Its TTL (`FILL_DEDUPE_WINDOW_MS`) is a lease, not the dedupe
|
|
435
|
+
* authority; the store refuses a booked key while retained, at any age.
|
|
436
|
+
*
|
|
437
|
+
* Claims are released on every path that does not book (requeue, orphan
|
|
438
|
+
* adoption, stale-totals deferral), so a held claim means "in flight".
|
|
439
|
+
*
|
|
440
|
+
* @param {string} fillKey - Chain fill identity (`orderId:blockNum:historyId`)
|
|
441
|
+
* @param {Set<string>} processedFillKeys - Per-cycle key set
|
|
442
|
+
* @param {string} [label] - Log prefix
|
|
443
|
+
* @param {string} [orderId] - Chain order id, for the log only
|
|
444
|
+
* @returns {boolean} true when this cycle may process the fill
|
|
445
|
+
*/
|
|
424
446
|
_isNewFillKey(fillKey: string, processedFillKeys: Set<string>, label?: string, orderId?: string): boolean;
|
|
425
447
|
/**
|
|
426
448
|
* Apply replay-safe fill accounting for tracked fills (those with a valid grid order).
|
|
@@ -83,6 +83,7 @@ import { path } from './path_api.js';
|
|
|
83
83
|
import * as Format from './order/format.js';
|
|
84
84
|
import cowRuntime from './dexbot_cow_runtime.js';
|
|
85
85
|
import { ProcessedFillStore } from './order/processed_fill_store.js';
|
|
86
|
+
import { isFillBooked } from './order/fill_ledger.js';
|
|
86
87
|
import { TIMING, DAEMON_CODES, FILL_PROCESSING, } from './constants.js';
|
|
87
88
|
import { normalizeBotEntry } from './bot_settings.js';
|
|
88
89
|
import { createDefaultBotMetrics } from './bot_defaults.js';
|
|
@@ -610,7 +611,36 @@ class DEXBot {
|
|
|
610
611
|
_buildOrphanFillFallbackKey(fill) {
|
|
611
612
|
return DexbotFillRuntime.buildOrphanFillFallbackKey(this, fill);
|
|
612
613
|
}
|
|
614
|
+
/**
|
|
615
|
+
* Claim a fill key for this cycle, or report that it was already handled.
|
|
616
|
+
*
|
|
617
|
+
* Two authorities, in this order:
|
|
618
|
+
* 1. **The processed-fill store** — booked keys. Checked first and without
|
|
619
|
+
* any clock, because "already credited" does not expire. Before P2 a
|
|
620
|
+
* booked key older than the lease window was accepted again here and
|
|
621
|
+
* only refused later by accounting, so the replay still paid for queue
|
|
622
|
+
* churn and a FILL DETECTED log.
|
|
623
|
+
* 2. **The claim lease** — fills accepted but not yet booked (deferral,
|
|
624
|
+
* crash). Its TTL (`FILL_DEDUPE_WINDOW_MS`) is a lease, not the dedupe
|
|
625
|
+
* authority; the store refuses a booked key while retained, at any age.
|
|
626
|
+
*
|
|
627
|
+
* Claims are released on every path that does not book (requeue, orphan
|
|
628
|
+
* adoption, stale-totals deferral), so a held claim means "in flight".
|
|
629
|
+
*
|
|
630
|
+
* @param {string} fillKey - Chain fill identity (`orderId:blockNum:historyId`)
|
|
631
|
+
* @param {Set<string>} processedFillKeys - Per-cycle key set
|
|
632
|
+
* @param {string} [label] - Log prefix
|
|
633
|
+
* @param {string} [orderId] - Chain order id, for the log only
|
|
634
|
+
* @returns {boolean} true when this cycle may process the fill
|
|
635
|
+
*/
|
|
613
636
|
_isNewFillKey(fillKey, processedFillKeys, label = '', orderId = '') {
|
|
637
|
+
if (isFillBooked(this._processedFillStore, fillKey)) {
|
|
638
|
+
if (label) {
|
|
639
|
+
const idSuffix = orderId ? ` for ${orderId}` : '';
|
|
640
|
+
this.manager?.logger?.log?.(`${label} Skipping fill${idSuffix}: ${fillKey} is already booked`, 'debug');
|
|
641
|
+
}
|
|
642
|
+
return false;
|
|
643
|
+
}
|
|
614
644
|
const now = Date.now();
|
|
615
645
|
if (this._recentlyQueuedFills.has(fillKey)) {
|
|
616
646
|
const lastProcessed = this._recentlyQueuedFills.get(fillKey) ?? 0;
|
|
@@ -446,7 +446,8 @@ declare function isSpreadCorrectionBypassAllowed(bot: BotLike): boolean;
|
|
|
446
446
|
declare function refreshLastFillPivotFromQueue(bot: BotLike): boolean;
|
|
447
447
|
/**
|
|
448
448
|
* Build COW actions array from a simple plan object.
|
|
449
|
-
* @param {import('./dexbot_class.js').DEXBot}
|
|
449
|
+
* @param {import('./dexbot_class.js').DEXBot} _bot - Bot whose manager supplies the frozen
|
|
450
|
+
* boundary/gap geometry the plan stamp carries (plan values win when set).
|
|
450
451
|
* @param {Object|Array} plan
|
|
451
452
|
* @returns {Array}
|
|
452
453
|
*/
|
|
@@ -554,6 +555,35 @@ export declare function deriveRailBoundary(manager: OrderManagerLike, opContexts
|
|
|
554
555
|
*/
|
|
555
556
|
declare function trackBoundaryHold(manager: OrderManagerLike, held: boolean, keptBoundary: unknown, plannedBoundary: unknown, heldSlotIds: string[]): number;
|
|
556
557
|
declare function updateOrdersOnChainBatchCOW(bot: BotLike, cowResult: CowResult, options?: JsonObj): Promise<JsonObj>;
|
|
558
|
+
/**
|
|
559
|
+
* Converge master with the chain after a COW commit was refused (master moved
|
|
560
|
+
* during broadcast) or an uncertain broadcast was poll-confirmed.
|
|
561
|
+
*
|
|
562
|
+
* Preferred path (when the broadcast result is available): re-read the exact
|
|
563
|
+
* on-chain orders BY ID — every id master already tracks plus the batch's
|
|
564
|
+
* fresh CREATE ids. get_objects returns the complete, authoritative set
|
|
565
|
+
* regardless of account size, so syncFromOpenOrders converges master instead
|
|
566
|
+
* of dropping the freshest creates (which previously left permanent orphans
|
|
567
|
+
* and tripped the fund invariant). This is fully immune to the
|
|
568
|
+
* get_full_accounts window truncation that broke the old window-read path.
|
|
569
|
+
*
|
|
570
|
+
* Fallback (no broadcast result, e.g. uncertain paths, or by-id read
|
|
571
|
+
* unavailable): window read. A truncated read is ambiguous — only the
|
|
572
|
+
* freshest orders are dropped — so it MUST NOT drive adoption; we return false
|
|
573
|
+
* and let the caller keep pending-broadcast protection + structural resync.
|
|
574
|
+
*
|
|
575
|
+
* @param {unknown} bot
|
|
576
|
+
* @param {unknown} chainOrders - chain_orders module (has readOpenOrdersWithMetaSafe + batchReadOrders)
|
|
577
|
+
* @param {string} logPrefix
|
|
578
|
+
* @param {Object} [opts]
|
|
579
|
+
* @param {unknown} [opts.placedResults] - broadcast result carrying operation_results
|
|
580
|
+
* @param {unknown[]} [opts.placedContexts] - opContexts aligned with operation_results
|
|
581
|
+
* @param {string[]} [opts.polledCreateIds] - fresh CREATE chain ids confirmed by
|
|
582
|
+
* the uncertain-broadcast poll (no broadcast result exists on that path);
|
|
583
|
+
* routes the poll call sites through the by-id path with its lagging-create
|
|
584
|
+
* retry instead of the unguarded window fallback
|
|
585
|
+
* @returns {Promise<boolean>} true if master was adopted from the chain
|
|
586
|
+
*/
|
|
557
587
|
declare function adoptPlacedBatchFromChain(bot: BotLike, chainOrders: typeof chainOrdersModule, logPrefix: string, opts?: JsonObj): Promise<boolean>;
|
|
558
588
|
/**
|
|
559
589
|
* Resolve the LAST-FILL guard pivot to an ON-GRID price.
|
|
@@ -21,6 +21,7 @@ const { validateCreateTargetSlots, evaluateCommit, hasExecutableActions, stampGa
|
|
|
21
21
|
import * as math from './order/utils/math.js';
|
|
22
22
|
const { validateOrderSize, findCrossedOrder, sameGridLevel, isEvacuationRotationAllowed, isEvacuationSizeStillValid, getSellStartIdx, getPrecisionByOrderType, isSlotIndexInGapBand, getAssetFeesSafe, blockchainToFloat, floatToBlockchainInt, quantizeFloat } = math;
|
|
23
23
|
import { parseSlotIndex } from './order/utils/slot.js';
|
|
24
|
+
import { resolveTradedSide } from './order/fill_ledger.js';
|
|
24
25
|
/**
|
|
25
26
|
* Re-verify a B-stamped gap-evacuation rotation against the LIVE committed
|
|
26
27
|
* geometry (boundaryIdx/_gapSlots at execution time). The stamp freezes
|
|
@@ -1104,7 +1105,7 @@ async function resyncIfUnreconciled(bot, err, pending, chainSnapshot, adopted, a
|
|
|
1104
1105
|
* @param {import('./dexbot_class.js').DEXBot} bot
|
|
1105
1106
|
* @param {BroadcastUncertainError} err
|
|
1106
1107
|
* @param {Array<Object>} opContexts
|
|
1107
|
-
* @param {Object} options
|
|
1108
|
+
* @param {Object} _options - Reserved; the implementation takes no options yet.
|
|
1108
1109
|
* @returns {Promise<Object>}
|
|
1109
1110
|
*/
|
|
1110
1111
|
async function reconcileAfterUncertainBroadcastImpl(bot, err, opContexts, _options) {
|
|
@@ -1184,7 +1185,7 @@ async function reconcileAfterUncertainBroadcastImpl(bot, err, opContexts, _optio
|
|
|
1184
1185
|
}
|
|
1185
1186
|
/**
|
|
1186
1187
|
* Check whether to execute creates in outside-in pair mode.
|
|
1187
|
-
* @param {
|
|
1188
|
+
* @param {unknown} _bot - Unused; the decision depends only on opContexts.
|
|
1188
1189
|
* @param {Array} opContexts
|
|
1189
1190
|
* @returns {boolean}
|
|
1190
1191
|
*/
|
|
@@ -1699,7 +1700,7 @@ function validateOperationFunds(bot, operations, assetA, assetB) {
|
|
|
1699
1700
|
}
|
|
1700
1701
|
/**
|
|
1701
1702
|
* Resolve the ideal size from an order-like object with fallback.
|
|
1702
|
-
* @param {
|
|
1703
|
+
* @param {unknown} _bot - Unused; the size comes from orderLike only.
|
|
1703
1704
|
* @param {Object|null} orderLike
|
|
1704
1705
|
* @param {number|null} [fallbackSize=null]
|
|
1705
1706
|
* @returns {number|null}
|
|
@@ -2121,11 +2122,14 @@ function refreshLastFillPivotFromQueue(bot) {
|
|
|
2121
2122
|
const recvAmt = Number(receives.amount);
|
|
2122
2123
|
if (Number.isFinite(paysAmt) && paysAmt > 0 && Number.isFinite(recvAmt) && recvAmt > 0) {
|
|
2123
2124
|
const paysFloat = (base, precision) => base / Math.pow(10, precision);
|
|
2124
|
-
|
|
2125
|
+
// Side from the paid asset through the shared resolver
|
|
2126
|
+
// (order/fill_ledger.ts); the price math stays here.
|
|
2127
|
+
const paysSide = resolveTradedSide(paysId, assets);
|
|
2128
|
+
if (paysSide === ORDER_TYPES.SELL) {
|
|
2125
2129
|
type = ORDER_TYPES.SELL;
|
|
2126
2130
|
price = paysFloat(recvAmt, assets.assetB.precision) / paysFloat(paysAmt, assets.assetA.precision);
|
|
2127
2131
|
}
|
|
2128
|
-
else if (
|
|
2132
|
+
else if (paysSide === ORDER_TYPES.BUY) {
|
|
2129
2133
|
type = ORDER_TYPES.BUY;
|
|
2130
2134
|
price = paysFloat(paysAmt, assets.assetB.precision) / paysFloat(recvAmt, assets.assetA.precision);
|
|
2131
2135
|
}
|
|
@@ -2195,7 +2199,8 @@ function resolveLastFillGuardIncrement(bot) {
|
|
|
2195
2199
|
}
|
|
2196
2200
|
/**
|
|
2197
2201
|
* Build COW actions array from a simple plan object.
|
|
2198
|
-
* @param {import('./dexbot_class.js').DEXBot}
|
|
2202
|
+
* @param {import('./dexbot_class.js').DEXBot} _bot - Bot whose manager supplies the frozen
|
|
2203
|
+
* boundary/gap geometry the plan stamp carries (plan values win when set).
|
|
2199
2204
|
* @param {Object|Array} plan
|
|
2200
2205
|
* @returns {Array}
|
|
2201
2206
|
*/
|
|
@@ -3290,13 +3295,6 @@ async function waitForCowBroadcastSingleFlight(bot, label) {
|
|
|
3290
3295
|
}
|
|
3291
3296
|
return false;
|
|
3292
3297
|
}
|
|
3293
|
-
/**
|
|
3294
|
-
* COW broadcast: Execute blockchain operations and commit working grid on success.
|
|
3295
|
-
* @param {import('./dexbot_class.js').DEXBot} bot
|
|
3296
|
-
* @param {Object} cowResult
|
|
3297
|
-
* @param {Object} [options={}] - Internal execution options (replanDepth)
|
|
3298
|
-
* @returns {Promise<Object>}
|
|
3299
|
-
*/
|
|
3300
3298
|
/**
|
|
3301
3299
|
* Derive an update action's planned target size (shared by the rotation and
|
|
3302
3300
|
* plain size-update op builders).
|
|
@@ -3856,6 +3854,27 @@ async function updateOrdersOnChainBatchCOWBody(bot, cowResult, replanDepth, seam
|
|
|
3856
3854
|
let heldBroadcastSlot = false;
|
|
3857
3855
|
let batchInFlightRaised = false;
|
|
3858
3856
|
let ordersLocked = false;
|
|
3857
|
+
// Broadcast-region ownership for THIS frame. startBroadcasting() runs
|
|
3858
|
+
// before the action loop (fills must stay deferred while ops are built),
|
|
3859
|
+
// but the inner broadcast try/finally only covers the broadcast section.
|
|
3860
|
+
// Every early return in between — all CREATEs skipped by the last-fill /
|
|
3861
|
+
// crossing / size guards (operations.length === 0), fund-validation
|
|
3862
|
+
// refusal, stale-plan re-plan, single-flight/placement aborts — used to
|
|
3863
|
+
// return with the refcount still held. The leaked flag then froze the
|
|
3864
|
+
// fill consumer until the 120s stale-flag watchdog (2026-11-04 incident:
|
|
3865
|
+
// three such returns parked 26 fills, the backlog was replayed against a
|
|
3866
|
+
// stale boundary and the bot sold below levels it had already sold).
|
|
3867
|
+
// One flag + one releaser: the inner finally, the catch and the outer
|
|
3868
|
+
// finally all call it, and only the first call decrements — a second raw
|
|
3869
|
+
// stopBroadcasting() in the outer finally would underflow another
|
|
3870
|
+
// holder's region (grid reconcile / spread correction).
|
|
3871
|
+
let broadcastRegionHeld = false;
|
|
3872
|
+
const releaseBroadcastRegion = () => {
|
|
3873
|
+
if (!broadcastRegionHeld)
|
|
3874
|
+
return;
|
|
3875
|
+
broadcastRegionHeld = false;
|
|
3876
|
+
bot.manager.stopBroadcasting();
|
|
3877
|
+
};
|
|
3859
3878
|
try {
|
|
3860
3879
|
// Single-flight COW broadcast guard (entry check): never broadcast two
|
|
3861
3880
|
// batches concurrently. Overlapping broadcasts plan from the same base
|
|
@@ -4003,6 +4022,7 @@ async function updateOrdersOnChainBatchCOWBody(bot, cowResult, replanDepth, seam
|
|
|
4003
4022
|
bot._markGridActivity('batch start');
|
|
4004
4023
|
bot.manager._setRebalanceState(REBALANCE_STATES.BROADCASTING);
|
|
4005
4024
|
bot.manager.startBroadcasting();
|
|
4025
|
+
broadcastRegionHeld = true;
|
|
4006
4026
|
// P3 — freeze the guard pivot once per batch: per-action refreshes
|
|
4007
4027
|
// mutated the pivot mid-batch (02:03 pivots drifted 0.001523→0.001529
|
|
4008
4028
|
// across 20 checks), so early actions were judged against a different
|
|
@@ -5171,7 +5191,7 @@ async function updateOrdersOnChainBatchCOWBody(bot, cowResult, replanDepth, seam
|
|
|
5171
5191
|
finally {
|
|
5172
5192
|
bot.manager._throwOnIllegalState = false;
|
|
5173
5193
|
await bot.manager.resumeFundRecalc();
|
|
5174
|
-
|
|
5194
|
+
releaseBroadcastRegion();
|
|
5175
5195
|
const createCount = actions.filter((a) => a.type === COW_ACTIONS.CREATE).length;
|
|
5176
5196
|
const cancelCount = actions.filter((a) => a.type === COW_ACTIONS.CANCEL).length;
|
|
5177
5197
|
bot.manager.logger.logFundsStatus(bot.manager, `AFTER COW batch (created=${createCount}, cancelled=${cancelCount})`);
|
|
@@ -5182,7 +5202,7 @@ async function updateOrdersOnChainBatchCOWBody(bot, cowResult, replanDepth, seam
|
|
|
5182
5202
|
if (getErrorField(err, 'partialOnChainState')) {
|
|
5183
5203
|
bot.manager.logger.log(`[COW] Non-atomic grouped execution detected (${formatPartialBroadcastSummary(err)}). Local rollback cannot undo confirmed on-chain operations; next sync/reconcile will converge state.`, 'warn');
|
|
5184
5204
|
}
|
|
5185
|
-
|
|
5205
|
+
releaseBroadcastRegion();
|
|
5186
5206
|
// Chain polling: for uncertain broadcasts (not partial), try to confirm
|
|
5187
5207
|
// CREATE operations on chain before clearing the working grid. If all
|
|
5188
5208
|
// planned CREATEs are confirmed, the entire batch was accepted atomically
|
|
@@ -5296,6 +5316,12 @@ async function updateOrdersOnChainBatchCOWBody(bot, cowResult, replanDepth, seam
|
|
|
5296
5316
|
if (heldBroadcastSlot)
|
|
5297
5317
|
bot._cowBroadcastInFlight = false;
|
|
5298
5318
|
releasePlacement();
|
|
5319
|
+
// Release the region as early as the frame contract allows: after
|
|
5320
|
+
// _batchInFlight-- and placement release (so region-end listeners see
|
|
5321
|
+
// a torn-down frame), but BEFORE the remaining best-effort cleanup.
|
|
5322
|
+
// A throw from _markGridActivity / unlockOrders / the fill-consumer
|
|
5323
|
+
// restart must not strand the refcount for the next 120s watchdog.
|
|
5324
|
+
releaseBroadcastRegion();
|
|
5299
5325
|
if (batchInFlightRaised)
|
|
5300
5326
|
bot._markGridActivity('batch end');
|
|
5301
5327
|
if (ordersLocked)
|
|
@@ -5324,19 +5350,34 @@ async function requestStructuralResync(bot, reason, details = {}) {
|
|
|
5324
5350
|
await bot.manager.requestStructuralGridResync(reason, details);
|
|
5325
5351
|
}
|
|
5326
5352
|
/**
|
|
5327
|
-
*
|
|
5328
|
-
*
|
|
5329
|
-
*
|
|
5330
|
-
*
|
|
5331
|
-
*
|
|
5332
|
-
*
|
|
5333
|
-
*
|
|
5334
|
-
*
|
|
5335
|
-
*
|
|
5336
|
-
* @param {
|
|
5337
|
-
* @param {
|
|
5338
|
-
* @returns {Promise<boolean>}
|
|
5353
|
+
* P1-atomic: after a refused/uncertain COW commit is recovered by adopting the
|
|
5354
|
+
* placed orders from the chain, re-apply the rotational boundary this batch had
|
|
5355
|
+
* already computed (workingBoundary) to master. The refused commit discarded
|
|
5356
|
+
* the working grid, so without this the next rebalance would re-derive the
|
|
5357
|
+
* boundary from a master that still reflects the pre-fill layout and could
|
|
5358
|
+
* re-stamp the just-filled slot x. Applying workingBoundary commits the
|
|
5359
|
+
* post-fill rotation immediately (no re-broadcast), so the next placement lands
|
|
5360
|
+
* at the shifted slot, never at x.
|
|
5361
|
+
*
|
|
5362
|
+
* @param {unknown} bot
|
|
5363
|
+
* @param {number} workingBoundary - boundary index the refused batch targeted
|
|
5339
5364
|
*/
|
|
5365
|
+
async function restoreBoundaryAfterAdoption(bot, workingBoundary) {
|
|
5366
|
+
try {
|
|
5367
|
+
if (workingBoundary === undefined || workingBoundary === null)
|
|
5368
|
+
return;
|
|
5369
|
+
if (typeof bot.manager._restoreBoundary === 'function') {
|
|
5370
|
+
bot.manager._restoreBoundary(workingBoundary);
|
|
5371
|
+
}
|
|
5372
|
+
else {
|
|
5373
|
+
bot.manager.boundaryIdx = workingBoundary;
|
|
5374
|
+
}
|
|
5375
|
+
bot.manager.logger.log(`[COW] Restored rotational boundary ${workingBoundary} after refused/uncertain commit + chain adoption (atomic re-plan)`, 'info');
|
|
5376
|
+
}
|
|
5377
|
+
catch (e) {
|
|
5378
|
+
bot.manager.logger.log(`[COW] Post-adoption boundary restore failed: ${getErrorMessage(e)}`, 'warn');
|
|
5379
|
+
}
|
|
5380
|
+
}
|
|
5340
5381
|
/**
|
|
5341
5382
|
* Converge master with the chain after a COW commit was refused (master moved
|
|
5342
5383
|
* during broadcast) or an uncertain broadcast was poll-confirmed.
|
|
@@ -5366,35 +5407,6 @@ async function requestStructuralResync(bot, reason, details = {}) {
|
|
|
5366
5407
|
* retry instead of the unguarded window fallback
|
|
5367
5408
|
* @returns {Promise<boolean>} true if master was adopted from the chain
|
|
5368
5409
|
*/
|
|
5369
|
-
/**
|
|
5370
|
-
* P1-atomic: after a refused/uncertain COW commit is recovered by adopting the
|
|
5371
|
-
* placed orders from the chain, re-apply the rotational boundary this batch had
|
|
5372
|
-
* already computed (workingBoundary) to master. The refused commit discarded
|
|
5373
|
-
* the working grid, so without this the next rebalance would re-derive the
|
|
5374
|
-
* boundary from a master that still reflects the pre-fill layout and could
|
|
5375
|
-
* re-stamp the just-filled slot x. Applying workingBoundary commits the
|
|
5376
|
-
* post-fill rotation immediately (no re-broadcast), so the next placement lands
|
|
5377
|
-
* at the shifted slot, never at x.
|
|
5378
|
-
*
|
|
5379
|
-
* @param {unknown} bot
|
|
5380
|
-
* @param {number} workingBoundary - boundary index the refused batch targeted
|
|
5381
|
-
*/
|
|
5382
|
-
async function restoreBoundaryAfterAdoption(bot, workingBoundary) {
|
|
5383
|
-
try {
|
|
5384
|
-
if (workingBoundary === undefined || workingBoundary === null)
|
|
5385
|
-
return;
|
|
5386
|
-
if (typeof bot.manager._restoreBoundary === 'function') {
|
|
5387
|
-
bot.manager._restoreBoundary(workingBoundary);
|
|
5388
|
-
}
|
|
5389
|
-
else {
|
|
5390
|
-
bot.manager.boundaryIdx = workingBoundary;
|
|
5391
|
-
}
|
|
5392
|
-
bot.manager.logger.log(`[COW] Restored rotational boundary ${workingBoundary} after refused/uncertain commit + chain adoption (atomic re-plan)`, 'info');
|
|
5393
|
-
}
|
|
5394
|
-
catch (e) {
|
|
5395
|
-
bot.manager.logger.log(`[COW] Post-adoption boundary restore failed: ${getErrorMessage(e)}`, 'warn');
|
|
5396
|
-
}
|
|
5397
|
-
}
|
|
5398
5410
|
async function adoptPlacedBatchFromChain(bot, chainOrders, logPrefix, opts = {}) {
|
|
5399
5411
|
const { placedResults = null, placedContexts = null, polledCreateIds = null } = opts || {};
|
|
5400
5412
|
try {
|
|
@@ -265,7 +265,7 @@ declare function stopBotsConfigPollInterval(bot: BotLike): void;
|
|
|
265
265
|
/**
|
|
266
266
|
* Release the market adapter runtime for a bot.
|
|
267
267
|
* In PM2 mode this is a no-op; in direct mode it calls the shared runtime's releaseBot.
|
|
268
|
-
* @param {
|
|
268
|
+
* @param {unknown} _bot - Unused; release is keyed by botId.
|
|
269
269
|
* @param {string} botId - Bot identifier
|
|
270
270
|
* @param {string} [context='shutdown'] - Context label for logging
|
|
271
271
|
* @returns {Promise<any>}
|
|
@@ -8,7 +8,7 @@ const require = createRequire(import.meta.url);
|
|
|
8
8
|
import { lazyShim } from './lazy_shim.js';
|
|
9
9
|
import fs from 'node:fs';
|
|
10
10
|
import { spawn } from 'node:child_process';
|
|
11
|
-
import { nowIso, fireStructuralResync, resolveLastFillGuardReference, hasUnsettledFills } from './order/utils/system.js';
|
|
11
|
+
import { nowIso, fireStructuralResync, resolveLastFillGuardReference, hasUnsettledFills, resolveGridUpdateBlockers, noteGridUpdateBlockers } from './order/utils/system.js';
|
|
12
12
|
import { withPlacement } from './order/placement_lock.js';
|
|
13
13
|
import { path } from './path_api.js';
|
|
14
14
|
import * as chainOrders from './chain_orders.js';
|
|
@@ -740,8 +740,19 @@ function getTargetedSyncReason(bot) {
|
|
|
740
740
|
}
|
|
741
741
|
async function maybeRunTargetedDriftReconciliation(bot, context) {
|
|
742
742
|
const trigger = getTargetedSyncReason(bot);
|
|
743
|
-
if (!trigger)
|
|
743
|
+
if (!trigger) {
|
|
744
|
+
// CONSECUTIVE means consecutive: a tick that never evaluates the gate
|
|
745
|
+
// must clear the counter. Without this reset a counter parked at
|
|
746
|
+
// GRID_BLOCKED_ESCALATE_CYCLES survives an arbitrarily long
|
|
747
|
+
// trigger-less stretch, and the first unreadable cycle afterwards fires
|
|
748
|
+
// a structural resync instead of waiting the full window.
|
|
749
|
+
// (docs/FILL_PIPELINE.md B1; pinned by test_targeted_drift_reconcile.)
|
|
750
|
+
// Guarded: without a manager there is no counter to clear, and the
|
|
751
|
+
// helper would otherwise park a stray field on the bot.
|
|
752
|
+
if (bot.manager)
|
|
753
|
+
noteGridUpdateBlockers(bot, []);
|
|
744
754
|
return false;
|
|
755
|
+
}
|
|
745
756
|
// UNSETTLED-FILL GATE: a startup-style reconcile derives its window from the
|
|
746
757
|
// master grid. While fills are still queued / draining / in flight / owed a
|
|
747
758
|
// crawl, that master still holds the filled slots as ACTIVE and the boundary
|
|
@@ -752,9 +763,25 @@ async function maybeRunTargetedDriftReconciliation(bot, context) {
|
|
|
752
763
|
// resolves against the crawled rail. No cooldown stamp, so the retry is
|
|
753
764
|
// immediate once the fills settle.
|
|
754
765
|
const unsettled = hasUnsettledFills(bot);
|
|
755
|
-
|
|
756
|
-
|
|
757
|
-
|
|
766
|
+
const gridBlockers = resolveGridUpdateBlockers(bot);
|
|
767
|
+
// Escalation: only an UNREADABLE pipeline state escalates, and only after
|
|
768
|
+
// GRID_BLOCKED_ESCALATE_CYCLES consecutive attempts. A busy pipeline is
|
|
769
|
+
// normal work; the placement/broadcast watchdogs own their own bounds.
|
|
770
|
+
const blockedGate = noteGridUpdateBlockers(bot, gridBlockers);
|
|
771
|
+
if (blockedGate.escalate) {
|
|
772
|
+
const codes = gridBlockers.map((b) => b.code).join(',');
|
|
773
|
+
bot._log(`[GRID-BLOCKED] ${blockedGate.cycles} consecutive blocked cycle(s) with unreadable pipeline state ` +
|
|
774
|
+
`(${codes}); requesting a structural resync`, 'error');
|
|
775
|
+
fireStructuralResync(bot, 'grid update blocked by unreadable pipeline state', { cycles: blockedGate.cycles, codes }, '[GRID-BLOCKED]');
|
|
776
|
+
}
|
|
777
|
+
if (gridBlockers.length > 0) {
|
|
778
|
+
// Any blocker defers the reconcile: it is a chain-mutating update, and
|
|
779
|
+
// mutating while a fill batch, a broadcast or a placement region is
|
|
780
|
+
// unresolved is what built the stale-master plans this gate exists for.
|
|
781
|
+
const codes = gridBlockers.map((b) => b.code).join(',');
|
|
782
|
+
const reason = unsettled.reason || gridBlockers[0].detail;
|
|
783
|
+
bot._log(`[TARGETED-SYNC] Deferring ${context} reconciliation: ${reason} ` +
|
|
784
|
+
`[${codes}] — the rail must absorb the fills before it is topped up`, 'debug');
|
|
758
785
|
return false;
|
|
759
786
|
}
|
|
760
787
|
const now = Date.now();
|
|
@@ -2074,7 +2101,7 @@ function stopBotsConfigPollInterval(bot) {
|
|
|
2074
2101
|
/**
|
|
2075
2102
|
* Release the market adapter runtime for a bot.
|
|
2076
2103
|
* In PM2 mode this is a no-op; in direct mode it calls the shared runtime's releaseBot.
|
|
2077
|
-
* @param {
|
|
2104
|
+
* @param {unknown} _bot - Unused; release is keyed by botId.
|
|
2078
2105
|
* @param {string} botId - Bot identifier
|
|
2079
2106
|
* @param {string} [context='shutdown'] - Context label for logging
|
|
2080
2107
|
* @returns {Promise<any>}
|