dexbot 1.6.2 → 1.6.3

package/CHANGELOG.md

@@ -2,6 +2,18 @@
 
 All notable changes to this project will be documented in this file.
 
+## [1.6.3] - 2026-09-14 - Correction-Queue Staleness Guard, Grid-Checker Price Epochs, Final Pivot Gate
+
+### 2026-09-15
+
+- **Fix(guard)**: final pre-broadcast pivot gate — a fill queued AFTER the batch-start pivot freeze but BEFORE broadcast passed every per-action LAST-FILL-GUARD check on a stale pivot (live incident on a market-pair bot: freeze at .745, sell fill queued at .765, batch of 4 broadcast at .910 against a buy pivot; the violating sell rotation filled 6s later, 0.6% below the true sell threshold, and tripped the fund invariant on drain). New `runFinalPivotGate()` re-checks BUILT ops against a re-refreshed pivot after the op-building loop and before the batch summary, fund validation, and single-flight claim: unchanged pivot is a pure no-op (arrays untouched, no extra probes); a moved/armed pivot re-runs the guard on each op's final price with the build loop's bypass rules (spread-correction CREATEs incl. batch-origin fallback, stamped gap-evacuations; unstamped evacuations guarded normally, same-slot UPDATEs resolve via source-id fallback) and drops violators into the existing skipped-slot restore paths (rotations restore source+dest from master, creates feed the refill-hold intersect, dropped CREATE pending entries removed by slot). Fail-open throughout: unresolvable price/type, cold pivot, and refresh throws all KEEP the op; cancels and size-updates are never gated. Pair-mode/chunk grouping is computed lazily at broadcast from the filtered opContexts (no stored indexes go stale); `cancelOpIndexByOrderId` (no live readers past op-building) is rebuilt from kept contexts as defence-in-depth. Pending-index hygiene: the gate's compaction REMAPS this batch's kept pending-broadcast entries' stored opIndex/ctxIndex to the compacted positions (lockstep keeps the two arrays aligned with each other but does not rewrite indexes stored inside pending entries — an unremapped ctxIndex would resolve to a shifted context after the first drop, letting the uncertain-broadcast reconcile adopt a matched chain order into the wrong slot); entries are identified by the fingerprints collected at record time (`recordPendingBroadcast` now returns its fingerprint; `entry.batchId` is always null in production and cannot discriminate), and entries whose referenced op was dropped are removed. Gate stats are kept separate from the build loop's counters (own `finalGateStats` object; reported as `gateChecked/gatePassed/gateSkipped/gateBypassed` fields on the batch summary, omitted when the gate did not re-check) so the summary's checked/passed/skipped totals stay the build loop's verdicts. The frozen pivot is captured AFTER the batch-start refresh (the refresh is part of the freeze) so batches whose freeze picked up a pre-freeze queued fill no longer trigger a spurious "pivot moved" warn + redundant re-check. Observability: always-on `Final gate: queue a->b pivot x->y` at debug, pivot-moved line at warn with freeze-queue depth + drop count, per-drop lines at warn. New `tests/test_final_pivot_gate.ts` (FG-1..FG-11, incl. incident replay dropping both violating rotations, the same-slot fallback, and pending-index remap identity checks); neighbor suites green (`modules/dexbot_cow_runtime.ts`).
+
+### 2026-09-14
+
+- **Fix(correction-queue)**: validate queued price corrections against the live slot before broadcast — corrections are queued as snapshots (expected price/size at detect time), but a geometry-changing resync re-slots/re-prices orders without invalidating the queue, so the maintenance pre-gate drain replayed stale snapshots and broadcast UPDATEs that reverted resync placements back onto occupied levels. Entries are now validated pre-broadcast against the live slot (ownership, price via slot predicates, size via integer quantum); stale entries are dropped and self-heal via the next sync re-queue. Zero-delta `updateOrder` returns `{success:true, skipped:true}` so routine no-ops no longer count as permanent failures; queue removal/dedupe is keyed on `(chainOrderId, isSurplus)` so sibling entries sharing a chain id survive; `queuedAt`/`queuedBy` provenance is stamped at queue time (sync detectors + gap-evacuation); drain summaries report `staleDropped` to keep failure WARN a true signal. New `tests/test_correction_queue_staleness.ts` (11 cases); 10 neighbor suites green (`modules/order/utils/order.ts`, `modules/order/sync_engine.ts`, `modules/dexbot_cow_runtime.ts`, `modules/dexbot_maintenance_runtime.ts`, `modules/order/manager.ts`).
+
+- **Fix(checker)**: split grid-correction checker fills into price epochs across repriced order lifetimes — the checker combined fills from different lifetimes of one order id after native repricing (limit-order update) operations and reported a false monotonicity violation. It now queries limit-order update operations and splits fills into chronological price epochs, preserving weighted aggregation for partial fills within one unchanged epoch (`analysis/grid_correction_check.ts`, `analysis/README.md`; TypeScript build passes, historical checker run clean).
+
 ## [1.6.2] - 2026-09-14 - Grid-Price Invariant, Gap-Slot Batch Sizing, Candle Shard Cache, Recovery Tolerance, Fill Counter Hygiene, Node List Cleanup
 
 ### 2026-09-14
@@ -14,11 +26,13 @@ All notable changes to this project will be documented in this file.
 
 - **Fix(nodes)**: prune dead default nodes and single-source the connection-trace node list — `NODE_MANAGEMENT.DEFAULT_NODES` drops three dead endpoints, leaving seven live defaults; `tests/test_connection_trace.ts` no longer carries its own hardcoded copy — it reads `NODE_MANAGEMENT.DEFAULT_NODES` with an optional `BITSHARES_TRACE_NODE` override for single-node tracing, and the transport login probe uses `NODES[0]` instead of a hardcoded URL, so future node-list changes propagate automatically (`modules/constants.ts`, `tests/test_connection_trace.ts`).
 
+- **Test(seams)**: eliminate wall-clock sleeps and live-chain hangs from offline tests — offline unit tests slept on production pacing delays and opened real WebSocket connections (suite past 125s, several files over 5s each). New production test seams with defaults unchanged: `createSubscriptionManager` accepts `overrides.noticeCoalesceMs` (page-timeout watchdog unref'd), COW `options.pollIntervalMs` for the 1.5s missing-create poll (set/restored via try/finally across every batch exit path incl. re-plan recursion), `scheduleDeferredFillRetry`/`parkFillsForTotalsRetry` `retryDelayMs`, credit-split `settleDelayMs` plus overridable collateral-balance fetch, sync-engine targeted-refetch fns plus `_skipEmptyReadConfirmDelay`, and `disconnectClient` stopping node health monitoring on teardown. New `resolveSeamMs()`/`resolveSeamMsOrNull()` (`modules/utils/errors.ts`) consolidate seam resolution (explicit 0 disables the delay; null/undefined falls through to the default — fixes `retryDelayMs` `> 0` silently dropping explicit 0 and `Number(null) === 0` selecting 0); each caller records its resolved delay so tests assert resolution instead of wall-clock timing (write-only in production). New `tests/test_seam_resolve.ts` plus COW-COMMIT-012 (seam restore on the guard-refusal path, precedence explicit > bot > default, explicit 0 honored) and RETRY-003b / TOTALS-004 (0 resolves to 0 — the only discriminating input for `||`-vs-`??` regressions, verified by fault injection); two stale pre-eager-gap-recovery mock expectations corrected, dead CJS override removed. Suite 287 passed, 0 failed, 15 live skipped, ~85s (was ~126s); `tsc` main + tests clean (`modules/bitshares-native/subscriptions.ts`, `modules/bitshares_client.ts`, `modules/credit_runtime.ts`, `modules/dexbot_cow_runtime.ts`, `modules/dexbot_fill_runtime.ts`, `modules/dexbot_startup_runtime.ts`, `modules/order/sync_engine.ts`, `modules/utils/errors.ts`).
+
 ### 2026-09-13
 
 - **Feat(batch)**: derive fill/broadcast batch sizing from gap-slot count — two independent fixed caps (`FILL_PROCESSING.MAX_FILL_BATCH_SIZE` and `COW_PERFORMANCE.MAX_OPS_PER_BROADCAST`, both 4) bounded fill batching and per-broadcast order operations with no relationship to actual grid geometry. New `_getGapSlotBatchSize()` resolves via `resolveGapSlots()` (`manager._gapSlots`, else `calculateGapSlots(config)`, ≥1 floor, falls back to 1 on failure); legacy `_getMaxFillBatchSize()` / `_getMaxOpsPerBroadcast()` kept as thin aliases; dead `config._gapSlots` fallback operand removed; both constants removed from `modules/constants.ts` (remaining COW_PERFORMANCE keys unchanged). `executeChunkedWithRetryOnUncertain` prefers the gap-slot size with its own ≥1 guard. Docs across `architecture.md`, `LIFECYCLE.md`, `FUND_MOVEMENT_AND_ACCOUNTING.md`, `developer_guide.md` use gap-slot batch sizing terminology (stale FILL_PROCESSING snippet removed); `COPY_ON_WRITE_MASTER_PLAN.md` marks `MAX_OPS_PER_BROADCAST` removed with a pointer. Completed/obsolete legacy docs removed (not describing anything still in the code): `BROWSER_COMPAT_PLAN.md` (superseded by the package.json browser field + AGENTS.md), `GRID_PRICE_SLOT_DETERMINISM_PLAN.md` (implemented), `PLAN_FILL_STALE_RESTORE_MAIN.md` (implemented), `PLAN_MIN_BTS_VALUE.md` (completed plan), `crash_report_jan_mar_2026.md` (historical incident). Batch size now scales with grid size by construction (larger-gap grids produce larger fill batches and larger single broadcast transactions than the old fixed cap of 4 — intended); removed keys simply drop out of the runtime-settings override merge, no remaining consumers (`modules/dexbot_class.ts`, `modules/constants.ts`, `modules/dexbot_cow_runtime.ts`; `tests/test_cow_ops_per_broadcast.ts` mirrors via `manager._gapSlots` with strict `calculateGapSlots` equality, chunking/death-spiral expectations updated, `tests/sim_batching.ts` uses the GAP_SLOTS constant; full `npm test` exit 0).
 
-- **Fix(cache)**: store Kibana candle cache in stable calendar-month shards — run-relative chunk files (`chunk_<index>_<dates>`) shifted names on every run because windows anchor at floored-now, forcing a full rewrite plus an orphan-deletion pass each invocation; the deletion destroyed cached history (fully-disjoint files wiped by narrow runs, boundary-straddling files losing out-of-window buckets). Storage is now fixed UTC calendar-month shards (`<base>.shard_YYYY-MM.json`) decoupled from querying: runs load only overlapping shards, fetch only genuinely missing buckets, and rewrite solely shards that gained buckets (higher-volume-wins) or query coverage (monotonically unioned `queriedRanges`); pure-reuse runs do zero writes/deletes. Legacy `*.chunk_*` files are absorbed (buckets + clipped coverage folded into shards) and retired only when every bucket provably lives in a shard; disjoint legacy is never loaded or deleted. Removed `cleanupOrphanCacheChunks`, `chunkPathFor`, `siblingChunkFiles`, `priorQueriedInWindow`; preserved immutable-gap pruning, 48h tail refresh, partial-window withholding, fetch retry budget. Windows are now fetch-planning splits only (`{index, gte, lte}`); the LP orphan-cleanup wrapper/export is dropped. Preceded by the narrower prune-scope fix (run passes its active coverage; orphans whose `meta.timeRange` does not overlap it are kept) plus the chart fixes it was validated against: update-marker div reuse (`um-wrap` class) and log-scale vertical-pan/wheel-zoom bbox offset correction (`market_adapter/inputs/window_cache.ts`, `market_adapter/inputs/fetch_lp_data.ts`, `market_adapter/README.md`, `analysis/tradingview/tradingview_uplot_chart_generator.ts`; `tests/test_window_cache.ts` rewritten with shard mapping, coverage set ops, scoped-load, and end-to-end integrations; LP/book assertions target `.shard_` files; full suite exit 0; live-verified: repeated feed chart runs show pure reuse, a wider run refetched one genuinely-lost span and left 7 stable shards).
+- **Fix(cache)**: store Kibana candle cache in stable calendar-month shards — run-relative chunk files (`chunk_<index>_<dates>`) shifted names on every run because windows anchor at floored-now, forcing a full rewrite plus an orphan-deletion pass each invocation; the deletion destroyed cached history (fully-disjoint files wiped by narrow runs, boundary-straddling files losing out-of-window buckets). Storage is now fixed UTC calendar-month shards (`<base>.shard_YYYY-MM.json`) decoupled from querying: runs load only overlapping shards, fetch only genuinely missing buckets, and rewrite solely shards that gained buckets (higher-volume-wins) or query coverage (monotonically unioned `queriedRanges`); pure-reuse runs do zero writes/deletes. Legacy `*.chunk_*` files are absorbed (buckets + clipped coverage folded into shards) and retired only when every bucket provably lives in a shard; disjoint legacy is never loaded or deleted. Removed `cleanupOrphanCacheChunks`, `chunkPathFor`, `siblingChunkFiles`, `priorQueriedInWindow`; preserved immutable-gap pruning, 48h tail refresh, partial-window withholding, fetch retry budget. Windows are now fetch-planning splits only (`{index, gte, lte}`); the LP orphan-cleanup wrapper/export is dropped. Preceded by the narrower prune-scope fix (run passes its active coverage; orphans whose `meta.timeRange` does not overlap it are kept) plus the chart fixes it was validated against (issues #29/#30): update-marker div reuse (`um-wrap` class) and log-scale vertical-pan/wheel-zoom bbox offset correction (`market_adapter/inputs/window_cache.ts`, `market_adapter/inputs/fetch_lp_data.ts`, `market_adapter/README.md`, `analysis/tradingview/tradingview_uplot_chart_generator.ts`; `tests/test_window_cache.ts` rewritten with shard mapping, coverage set ops, scoped-load, and end-to-end integrations; LP/book assertions target `.shard_` files; full suite exit 0; live-verified: repeated feed chart runs show pure reuse, a wider run refetched one genuinely-lost span and left 7 stable shards).
 
 ## [1.6.1] - 2026-09-13 - Never-Run-Stale Hardening, Whitelist Range-Scaling Opt-In, Live-Save Docs, Shelf-Count Hardening
 

package/analysis/README.md

@@ -225,7 +225,7 @@ node dist/analysis/trade_profitability.js 1.2.123456 \
 
 Validates grid discipline from the same Kibana fill pipeline as `trade_profitability.ts`: two consecutive same-direction fills on a pair must be monotonic — sell prices rising, buy prices falling (equal is OK). An inversion means the bot placed an order below its own previous sell (or above its own previous buy), e.g. an orphaned order filling outside grid accounting. Used as the external regression gate for the orphan-fix plans in `docs/CONSOLIDATED_ORPHAN_FIX_SUMMARY.md`.
 
-**Pipeline:** Kibana `fill_order` query (paginated `search_after`) → on-chain asset precision resolution → buy/sell classification → chronological sort → per-order aggregation (multi-fill orders collapsed to weighted-avg price by default) → consecutive same-direction pair comparison → violation report with daily histogram.
+**Pipeline:** Kibana `fill_order` query (paginated `search_after`) → on-chain asset precision resolution → buy/sell classification → chronological sort → per-order/price-epoch aggregation (partial fills at one price collapsed to weighted-average; repriced order lifetimes kept separate) → consecutive same-direction pair comparison → violation report with daily histogram.
 
 ```bash
 # Per-order aggregated check (default), last 7 days
@@ -263,7 +263,7 @@ Exit code `0` = pass, `2` = violations found, `1` = fatal error. Bot keys resolv
 
 </details>
 
-**Notes:** strict sat-level comparison is the ground truth (`--tolerance` only forgives small inversions); multi-fill orders are collapsed to a weighted-average price for the default per-order mode, so a single order's partial fills at identical prices never count as inversions.
+**Notes:** strict sat-level comparison is the ground truth (`--tolerance` only forgives small inversions); partial fills at one price are collapsed to a weighted-average price in the default mode, but fills after a native order repricing are kept in separate price epochs so updated orders are not mixed together.
 
 ## Charts & Visualization
 

package/analysis/ama_fitting/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ama_fitting",
-  "version": "1.6.2",
+  "version": "1.6.3",
   "description": "Tools for fitting AMA parameters to market data",
   "main": "../../dist/analysis/ama_fitting/optimizer_high_resolution.js",
   "scripts": {

package/analysis/grid_correction_check.ts

@@ -42,6 +42,7 @@ import { loadBotMeta, loadBotSettings, computeBotKey, persistBotAccountId } from
 const { kibanaSearch, DEFAULT_CONFIG: BASE_CONFIG } = KC;
 
 const OP_FILL_ORDER = 4;
+const OP_LIMIT_ORDER_UPDATE = 77;
 const BTS_ID = '1.3.0';
 
 interface AssetInfo { symbol: string; precision: number; }
@@ -113,6 +114,10 @@ interface FillRecord {
     isMaker: boolean;
     sort: any[];
 }
+interface OrderUpdate {
+    orderId: string;
+    sequence: number;
+}
 interface TradeFill {
     time: string;
     orderId: string;
@@ -485,6 +490,50 @@ function buildFillQuery(accountId: string, gte: string, lte: string, size: numbe
     };
 }
 
+async function fetchAllOrderUpdates(config: any, accountId: string, gte: string, lte: string): Promise<OrderUpdate[]> {
+    const pageSize = 10000;
+    const updates: OrderUpdate[] = [];
+    let searchAfter: any[] | null = null;
+    const cfg = { ...BASE_CONFIG, timeout: 60000, ...config };
+    while (true) {
+        const query: any = {
+            size: pageSize,
+            track_total_hits: false,
+            _source: [
+                'block_data.block_num', 'operation_id_num',
+                'operation_history.op_object.order',
+                'operation_history.op_object.seller',
+            ],
+            query: { bool: { filter: [
+                { term: { operation_type: OP_LIMIT_ORDER_UPDATE } },
+                { term: { 'operation_history.op_object.seller.keyword': accountId } },
+                { range: { 'block_data.block_time': { gte, lte } } },
+            ] } },
+            sort: [
+                { 'block_data.block_num': { order: 'asc' } },
+                { operation_id_num: { order: 'asc' } },
+            ],
+        };
+        if (searchAfter) query.search_after = searchAfter;
+        const result: any = await kibanaSearch(cfg, query);
+        const hits = result?.hits?.hits ?? [];
+        if (!hits.length) break;
+        for (const hit of hits) {
+            const source = hit?._source || {};
+            const op = source.operation_history?.op_object || {};
+            const orderId = op.order;
+            if (!orderId) continue;
+            const blockNum = Number(source.block_data?.block_num ?? 0);
+            const opNum = Number(source.operation_id_num ?? 0);
+            updates.push({ orderId, sequence: blockNum * 1e6 + opNum });
+        }
+        const last = hits[hits.length - 1];
+        searchAfter = last?.sort;
+        if (!searchAfter || hits.length < pageSize) break;
+    }
+    return updates;
+}
+
 async function fetchAllFills(config: any, accountId: string, gte: string, lte: string): Promise<FillRecord[]> {
     const pageSize = 10000;
     const fills: FillRecord[] = [];
@@ -578,11 +627,31 @@ function classifyFills(fills: FillRecord[]): { trades: TradeFill[]; skipped: num
     return { trades, skipped };
 }
 
-// ─── Per-order aggregation (collapses multi-fill orders to weighted avg price) ──
-function aggregateByOrder(trades: TradeFill[]): AggregatedOrder[] {
+// ─── Per-order/price-epoch aggregation ───────────────────────────────────────
+// Collapses partial fills at one order lifetime, but never mixes fills from
+// different native limit_order_update repricings of the same order ID.
+function aggregateByOrder(trades: TradeFill[], updates: OrderUpdate[] = []): AggregatedOrder[] {
+    const updatesByOrder = new Map<string, number[]>();
+    for (const update of updates) {
+        const list = updatesByOrder.get(update.orderId) || [];
+        list.push(update.sequence);
+        updatesByOrder.set(update.orderId, list);
+    }
+    for (const list of updatesByOrder.values()) list.sort((a, b) => a - b);
+
     const map = new Map<string, { trades: TradeFill[] }>();
     for (const t of trades) {
-        const k = t.orderId || `__fill_${t.sequence}`;
+        // An update is ordered before the fills it can affect. The epoch is
+        // therefore the latest update sequence at or before this fill.
+        const orderUpdates = updatesByOrder.get(t.orderId) || [];
+        let epoch = 0;
+        for (const sequence of orderUpdates) {
+            if (sequence <= t.sequence) epoch = sequence;
+            else break;
+        }
+        const k = t.orderId
+            ? `${t.orderId}:${t.direction}:${t.baseAsset}:${t.quoteAsset}:${epoch}`
+            : `__fill_${t.sequence}`;
         if (!map.has(k)) map.set(k, { trades: [] });
         map.get(k)!.trades.push(t);
     }
@@ -870,9 +939,13 @@ async function main() {
     const incrementPercent = resolveIncrementPercent(botMeta, opts.incrementPercent);
     console.log(`Increment: ${incrementPercent}% (halfInc ${incrementPercent/2}%)${opts.incrementPercent == null && botMeta?.incrementPercent != null ? ' — from bot config' : opts.incrementPercent != null ? ' — from --increment' : ' — default'}`);
 
-    console.log(`\nFetching fills from Kibana...`);
-    const fills = await fetchAllFills({}, accountId, gte, lte);
+    console.log(`\nFetching fills and order updates from Kibana...`);
+    const [fills, orderUpdates] = await Promise.all([
+        fetchAllFills({}, accountId, gte, lte),
+        fetchAllOrderUpdates({}, accountId, gte, lte),
+    ]);
     console.log(`  Fetched ${fills.length} fill_order operation(s)`);
+    console.log(`  Fetched ${orderUpdates.length} limit_order_update operation(s)`);
 
     if (fills.length === 0) {
         console.log('\nNo fills in range — nothing to check.');
@@ -905,8 +978,8 @@ async function main() {
     if (opts.perFill) {
         items = trades;
     } else {
-        const orders = aggregateByOrder(trades);
-        console.log(`  Aggregated into ${orders.length} order(s) (multi-fill orders collapsed by weighted avg price)`);
+        const orders = aggregateByOrder(trades, orderUpdates);
+        console.log(`  Aggregated into ${orders.length} order epoch(s) (partial fills collapsed; repriced orders kept separate)`);
         items = orders;
     }