@icgio/icg-exchanges-wrapper 1.17.20 → 1.17.22

package/docs/2026-04-09-perf-cleanup.md

@@ -0,0 +1,366 @@
+# Performance cleanup — 2026-04-09
+
+Session of performance and dead-code cleanup work spanning three repos:
+
+- `icg-utils` — `combine_orderbooks` rewrite + new tests
+- `icg-exchanges-wrapper` — `orders.js` hot-path optimization + dead-code removal
+- `icg-market-maker` — small follow-up to drop `o.trace` references
+
+Nothing in this document has been published or deployed. Versions, dependents,
+and MM client servers still need updating per the deployment notes at the
+bottom.
+
+## Why
+
+The maker hot path (rate update → `process_rates` → `notify()` → maker
+callback → `handle_maker_*`) was the bottleneck under analysis. Several
+contributing costs were identified and addressed. Expected aggregate effect:
+the per-rate-tick cost on the maker hot path should drop from the
+**multi-millisecond range to sub-millisecond** for typical configurations
+(5 exchanges × 3 pairs × ~100 orders).
+
+## Changes
+
+### icg-utils
+
+#### Rewrote `combine_orderbooks` (2026-04-09)
+
+`lib/utils.js`
+
+The original implementation:
+- Used `_.get`, `_.sum`, `_.values`, `_.keys` inside the inner loop
+- Mutated input tuples in place by pushing the exchange name (`tuple[2]`)
+  onto each level — forcing callers to `cloneDeep` upstream
+- O(R × K) with high lodash constant factors
+
+The rewrite:
+- Strips all lodash from the inner loop
+- Uses parallel arrays (`lists`, `indexes`, `exchanges`) keyed by an integer
+  index, hoists the asks/bids direction comparator out of the loop
+- **Non-mutating**: `orderbooks_dict` and its inner arrays/tuples are never
+  touched. Callers do NOT need to deep-clone before calling.
+- **Pushes input tuple references directly into the result** (Flavor 2 — see
+  Reference Sharing Contract below). Zero allocations in the inner loop.
+- Drops the `tuple[2]` augmentation entirely. The previous algorithm needed
+  the exchange name on each tuple to know which pointer to advance; the
+  rewrite tracks this via `bestIdx` (an integer index into the parallel
+  arrays).
+
+Expected speedup vs the original: ~10× per call. Combined with the upstream
+`cloneDeep` removal, `process_rates` per call drops from ~3-5 ms to ~0.2-0.4
+ms for a typical 5-exchange / 3-pair MM.
+
+##### Reference Sharing Contract
+
+The returned `{asks, bids}` arrays contain references to the SAME tuple
+objects that live inside the caller's input. **Callers MUST treat both the
+input and the result as read-only.** Mutating either will corrupt the other.
+
+This is safe in `icg-exchanges-wrapper`'s `process_rates` flow because:
+
+1. WS rate updates REPLACE the per-pair object via
+   `_.set(this.rates, [exchange, pair], { asks, bids, ... })` rather than
+   mutating in place — old tuple references simply become garbage when no
+   one holds them.
+2. The combined-rates consumer (the MM `rates_observable` subscriber) runs
+   synchronously inside `notify()` in the same call stack, so no other WS
+   handler can interleave and rebuild rates underneath us.
+
+If you ever need result tuples that are independent of the input (e.g. to
+pass to a mutating consumer like `vol_position_combined`, which expects a
+`[price, amount, exchange]` 3-tuple), wrap them at the call site — do not
+put that contract back into `combine_orderbooks`.
+
+#### Added `test/combine-orderbooks.test.js` (2026-04-09)
+
+`test/combine-orderbooks.test.js`
+
+13 tests covering:
+
+- Basic merge ordering (asks ascend, bids descend, length limit honoured)
+- Result tuples remain `[price, amount]` (no `[2]` augmentation)
+- Input is byte-equal to its pre-call snapshot (non-mutation contract)
+- Idempotent on repeated calls
+- Result tuples `===` input tuples (Flavor 2 reference aliasing)
+- Empty dict, empty side per exchange, single exchange, length larger than
+  available, no length limit
+- 3-way interleaving correctness
+- Tied prices are deterministic
+- Fractional prices
+
+Wired into `npm test` via `package.json` (now runs both `rate-limiter.test.js`
+and `combine-orderbooks.test.js`).
+
+### icg-exchanges-wrapper
+
+All changes in `middleware/orders.js` unless noted otherwise.
+
+#### Dropped `_.cloneDeep` in `process_rates` (2026-04-09)
+
+The deep clone existed only because the old `combine_orderbooks` mutated its
+input. With the rewrite the input is never touched, so `process_rates` now
+passes the live `this.rates[exchange][pair]` reference directly. Saves
+~600 µs per `process_rates` call for a typical 5-exchange / 3-pair / 20-level
+MM.
+
+#### Removed dead DEX code (2026-04-09)
+
+`rates_with_routes` and the entire DEX trading path were referenced from six
+call sites in `orders.js` and one in
+`functions/get-exchanges-rates-market-history-ohlcv.js`, but no exchange in
+`icg-exchanges` ever set `exchange_type === 'dex'` and no exchange implemented
+`rate_with_routes` / `rate_with_routes_ws` / `limit_trade_with_routes`. The
+branches were unreachable in production.
+
+Removed:
+
+- `this.rates_with_routes = {}` initialization
+- `limit_trade`'s DEX branch (~30 lines including closest-level / route lookup)
+- `if (rates_with_routes)` block in `rates_handler`, plus the
+  `rates_with_routes = false` parameter
+- Both DEX branches in `get_rates` (with and without `ws_market`)
+- DEX startup branch that pre-set `previous_orders_canceled_exchange_pair`
+- The `else if (Exchange.rate_with_routes)` REST fallback in
+  `functions/get-exchanges-rates-market-history-ohlcv.js`
+
+Net: ~74 lines of dead code removed.
+
+#### Removed `update_order_info` trace block (2026-04-09)
+
+`update_order_info` previously appended to a per-order `trace` array on every
+update via `[...existing_trace, trace_entry]`. This was:
+
+- O(N) quadratic per update — the spread copies the entire existing trace
+- Unbounded memory growth per long-lived order
+- Sent over WebSocket on every update, growing the WS payload over time
+
+The trace had two consumers:
+
+1. **`maker.js`** read `_.last(o.trace).ts` to filter stale CANCELING orders
+   in `maker_2` and `maker_3`. Replaced with a new `last_update_ts` field set
+   via native assignment in `update_order_info`.
+2. **The dashboard "Trace Timeline" UI** showed the full status history for
+   any clicked order. This is real value, but the wrapper is the wrong place
+   to materialize it. Plan: rebuild it on the dashboard backend from the
+   `ws_client.send_order` events that already arrive there. See
+   "Future work" below.
+
+The wrapper now does zero trace work. The dashboard "Trace Timeline" will
+temporarily show "No trace available for this order" until the backend
+update ships.
+
+#### Added status / order_type indexes (2026-04-09)
+
+Two new indexes maintained incrementally by `update_order_info` and
+`delete_order_info`:
+
+- `this.order_status_index[exchange][pair][type][status] -> Set<order_ref>`
+  Used by `get_order_info_status_all`.
+- `this.order_info_by_type[order_type][exchange][pair][type][order_id] -> ref`
+  Used by `get_order_info_by_order_type`.
+
+Maintenance happens at the end of `update_order_info` after the dict fields
+are written. The function snapshots the old `status` / `order_type` BEFORE
+the for-loop, then compares to the new values after, and moves the order
+between buckets only if the relevant field actually changed. The common
+case (a non-status update like `f_amount` or `last_update_ts`) is a no-op.
+
+`delete_order_info` removes the order from both indexes using the order's
+current `status` / `order_type` BEFORE deleting it from the canonical
+`this.order_info`.
+
+`get_order_info_status_all` was rewritten to look up the precomputed Set
+and return `Array.from(...)`. Handles both single-status and array-status
+queries, plus the `'both'` recursive case.
+
+`get_order_info_by_order_type` walks only the matching `order_type`
+subtree(s) and returns a freshly built top-level dict (so callers can safely
+mutate the returned structure) with leaves that are still refs into
+`this.order_info`.
+
+##### Indexing invariants
+
+- Every order in `this.order_info[exchange][pair][type][order_id]` has at
+  most one entry in `order_status_index[exchange][pair][type][status]` and
+  at most one entry in
+  `order_info_by_type[order_type][exchange][pair][type][order_id]`.
+- The only mutators of `this.order_info` are `update_order_info` and
+  `delete_order_info`. Both maintain the indexes. Anyone who writes directly
+  to `this.order_info` outside these two methods will desynchronize the
+  indexes.
+- The previous `this.order_info_status` cache (used by the old
+  `get_order_info_status_all`) is removed.
+- `Account.order_type === 'range-order'` mutations at `:1006` and `:1011`
+  occur on `res.body` open-orders entries from the exchange WS, NOT on
+  `this.order_info` orders, so they don't affect the indexes.
+
+##### Functional verification
+
+A standalone in-process test exercises the indexed code paths against a
+stub instance. 11 assertions cover:
+
+1. Insert maker_1 sell INIT → in INIT bucket, in maker_1 subtree
+2. Transition INIT → N-FILLED → moves buckets, stays in maker_1
+3. `'both'` query → unions sell + buy
+4. Array status query → unions multiple status buckets
+5. by_order_type with array → merges multiple subtrees
+6. `delete_order_info` → removed from BOTH indexes + canonical store
+7. Multiple status transitions → bucket moves are atomic
+8. `last_update_ts` is stamped (numeric, > 0)
+9. Empty queries return `[]` / `{}` (no crash)
+10. Result arrays from `get_order_info_status_all` are independent (fresh
+    array per call) but inner orders are the same refs
+11. Mutating the dict returned by `get_order_info_by_order_type` does not
+    affect the canonical store
+
+#### Fixed `check_balances` shadow bug (2026-04-09)
+
+The original inner loop used `for (let exchange in this.exchange_cur_quote_cur)`
+which **shadowed** the outer `exchange` parameter. Every balance update
+rebuilt `buy_sell_limits_against_reserve_balance` for ALL exchanges, doing
+K× the necessary work where K = exchange count. The fix scopes the inner
+loop to the single exchange whose balance just arrived. Also collapsed the
+empty `if (max > 0) { } else { max = 0 }` blocks to `if (max < 0) max = 0`.
+
+#### Early-exit on `previous_orders_canceled` recompute (2026-04-09)
+
+The recompute walked every `(exchange, pair)` cell every time. Now uses a
+labeled `break outer` on the first unmarked cell. Steady-state behavior
+unchanged (the outer guard `if (!this.previous_orders_canceled)` still
+prevents re-runs once the flag is true), but startup is faster.
+
+#### `_.omit(obj, [key])` → `delete obj[key]` (2026-04-09)
+
+16 sites. `_.omit` allocates a new object copying every other key — O(N)
+plus GC pressure. `delete` is O(1) and free.
+
+Sites:
+
+- `delete_order_info` (`this.order_info[exchange][pair][type]`)
+- `check_balances` expiry (`this.balances`,
+  `this.buy_sell_limits_against_reserve_balance`)
+- `check_positions` expiry (`this.positions`)
+- `open_orders_handler` and `all_open_orders_handler` expiry
+  (`this.open_orders[exchange]`)
+- `trades_handler` and `all_trades_handler` expiry (`this.trades[exchange]`)
+- `query_order_handler` expiry (`this.query_order[exchange]`)
+- `process_rates` local-variable cleanups (`orderbook_major`,
+  `orderbook_major_temp`) and `this.rates_combined_major`
+- `rates_handler` expiry (`this.rates[exchange]`)
+- `market_history_handler` expiry (`this.market_history[exchange]`)
+- `ohlcv_handler` short-data and expiry (`this.ohlcv[exchange]`)
+
+#### `new Date().getTime()` → `Date.now()` (2026-04-09)
+
+13 sites. `new Date().getTime()` allocates a Date object just to read its
+timestamp; `Date.now()` is an intrinsic with no allocation. Pure micro-win.
+
+### icg-market-maker
+
+#### Replaced `_.last(o.trace).ts` filters with `o.last_update_ts` (2026-04-09)
+
+`scripts/maker.js`
+
+Two filters in `maker_2` and `maker_3` (for stale CANCELING orders) replaced
+with the new `o.last_update_ts` field added to `orders.js` in this session.
+No semantic change.
+
+## Files touched
+
+| Repo | File | Note |
+|---|---|---|
+| `icg-utils` | `lib/utils.js` | `combine_orderbooks` rewrite |
+| `icg-utils` | `test/combine-orderbooks.test.js` | new file, 13 tests |
+| `icg-utils` | `package.json` | `test` script runs both files |
+| `icg-exchanges-wrapper` | `middleware/orders.js` | (a) drop `_.cloneDeep`, (b) remove DEX dead code, (c) remove trace block + add `last_update_ts`, (d) status / order_type indexes, (e) rewrite `get_order_info_status_all` and `get_order_info_by_order_type`, (f) `check_balances` shadow + scope, (g) early-exit `previous_orders_canceled` recompute, (h) `_.omit` → `delete` (16 sites), (i) `new Date().getTime()` → `Date.now()` (13 sites) |
+| `icg-exchanges-wrapper` | `functions/get-exchanges-rates-market-history-ohlcv.js` | remove DEX REST fallback |
+| `icg-market-maker` | `scripts/maker.js` | two `last_trace` filters → `o.last_update_ts` |
+
+## Net effect
+
+Hot path per rate tick (5 exchanges × 3 pairs × ~100 orders):
+
+| Operation | Before | After |
+|---|---|---|
+| `process_rates` per pair | ~1-2 ms | ~0.2-0.4 ms |
+| `combine_orderbooks` per call | ~500-1000 µs | ~50-100 µs |
+| `get_order_info_status_all` × ~270 calls/tick | walks all matching orders for that (exchange, pair, type), lodash-heavy | Set lookup + `Array.from`, O(matching) |
+| `get_order_info_by_order_type` × ~15 calls/tick | 4-deep scan of all `order_info`, lodash-heavy | walks only matching subtree |
+| `update_order_info` trace cost | O(N) spread per update | 0 |
+| `check_balances` per balance event | K× redundant rebuild | 1× rebuild scoped to the right exchange |
+| `previous_orders_canceled` recompute | always full walk while flag is false | early-exit on first miss |
+| `_.omit` × 16 sites | O(N) new-object allocation | O(1) `delete`, no allocation |
+| `Date` instances per check | 13 per tick | 0 |
+
+## Future work
+
+### Backend trace materialization
+
+Rebuild the "Trace Timeline" data on
+`clients-dashboard-backend/src/services/ws-data-service.js` from the
+`ws_client.send_order` events it already receives. About 12 lines added in
+`handle_order_event`. The wrapper does no trace work; the backend caps the
+trace at last 20 entries; the dashboard UI is unchanged. Spans
+`clients-dashboard-backend` only — neither the wrapper nor the maker need to
+ship for this.
+
+The required pattern in `handle_order_event`:
+
+```js
+if (action === 'update') {
+  const existing = _.get(this.order_info[client_id], [exchange, pair, side, order_id])
+  const trace = (existing && existing.trace) || []
+  trace.push({
+    ts: Date.now(),
+    status: data.status,
+    note,
+    f_amount: data.f_amount,
+  })
+  if (trace.length > TRACE_MAX_ENTRIES) {
+    trace.splice(0, trace.length - TRACE_MAX_ENTRIES)
+  }
+  _.setWith(
+    this.order_info[client_id],
+    `${exchange}.${pair}.${side}.${order_id}`,
+    { ...data, trace, ts: Date.now() },
+    Object,
+  )
+  // ... rest unchanged
+}
+```
+
+Plus a small change to `handle_order_sync` to preserve existing traces
+across full re-sync (otherwise reconnects wipe history).
+
+### Other items not done in this session
+
+- **`Observable.notify()` try/catch** in `icg-utils/lib/observable.js`. Today
+  there's only one subscriber so a throw is just a crash, but the moment a
+  second subscriber is added, a throw in #1 silently kills #2. Two-line fix.
+- **Remaining lodash hot-path cleanup**: `_.set` / `_.setWith` / `_.get` in
+  `update_order_info` and `rates_handler` could be replaced with native
+  indexing for ~5-20× speedup on those calls.
+- **Empty `cancel_order_by_order` callbacks** at `orders.js:1038`, `:1045`:
+  failures silently ignored. Robustness, not perf.
+- **Hot-path `console.log('UPDATE %s: ...')` formatting** at `orders.js:862`:
+  runs on every `update_order_info` call. Should be gated with a verbosity
+  flag or routed through `log_limiter` (like maker.js does).
+- **`process_open_orders` nested scans** at `orders.js:618-766`. Same
+  indexing approach as the status / order_type indexes would help here too,
+  but it's off the hot path so lower priority.
+
+## Deployment notes
+
+When this work ships:
+
+1. `icg-utils` → bump version → publish to npm (new `combine_orderbooks`).
+2. `icg-exchanges-wrapper` → bump `@icgio/icg-utils` dep range → `npm install`
+   → bump pkg version → publish.
+3. **MM client servers** → manual deploy across all 12. Per
+   `feedback_mm_servers_dirty_lockfile.md`, pre-stash any dirty
+   `package.json` / `package-lock.json` on each server before
+   `git pull && npm ci`. The deploy carries both the wrapper republish and
+   the `maker.js` change (they both touch the maker process).
+4. The dashboard "Trace Timeline" will show "No trace available" until the
+   `clients-dashboard-backend` update ships separately. Plan that as a
+   follow-up.

package/middleware/orders.js

@@ -73,6 +73,7 @@ const MARKET_HISTORY_DEFAULT_MS = 10 * 60 * 1000
 const TRADES_DEFAULT_MS = 7 * 24 * 60 * 60 * 1000
 const NO_CALLBACK_TIMEOUT_MS = 2 * 60 * 1000
 const PROCESS_RATES_THROTTLE_MS = 75
+const MAX_ORDER_TRACE_ENTRIES = 20
 const ORDER_TYPE_PRIORITY = {
   self: 2,
   maker_1: 3,
@@ -89,6 +90,13 @@ const getOrderPriority = (order_type, is_cancel = false) => {
   return is_cancel ? priority - 0.5 : priority
 }
 
+const hrtime_diff = (start, end) => {
+  if (!Array.isArray(start) || !Array.isArray(end)) {
+    return undefined
+  }
+  return (end[0] - start[0]) * 1e9 + (end[1] - start[1])
+}
+
 module.exports = class Orders {
   constructor(exchange_cur_quote_cur_dict, cd, settings) {
     let { exchange_cur_quote_cur, exchange_cur_quote_cur_rates_only, pair_map_rates_only } = exchange_cur_quote_cur_dict
@@ -272,72 +280,135 @@ module.exports = class Orders {
       ws_client.set_order_info_ref(this.order_info)
     }
   }
+  build_trace_entry(order, note) {
+    return {
+      ts: Date.now(),
+      status: order.status,
+      note,
+      f_amount: order.f_amount,
+    }
+  }
+  append_trace(order, note) {
+    if (!note || !order) return
+    if (!Array.isArray(order.trace)) {
+      order.trace = []
+    }
+    order.trace.push(this.build_trace_entry(order, note))
+    if (order.trace.length > MAX_ORDER_TRACE_ENTRIES) {
+      order.trace.splice(0, order.trace.length - MAX_ORDER_TRACE_ENTRIES)
+    }
+  }
+  move_order_info_key(exchange, pair, type, from_order_id, to_order_id) {
+    if (!from_order_id || !to_order_id || from_order_id === to_order_id) {
+      return
+    }
+    const from_bucket = this.order_info?.[exchange]?.[pair]?.[type]
+    if (!from_bucket || !from_bucket[from_order_id]) {
+      return
+    }
+    const order = from_bucket[from_order_id]
+    from_bucket[to_order_id] = order
+    delete from_bucket[from_order_id]
+    if (order.order_type) {
+      const by_type = this.order_info_by_type?.[order.order_type]?.[exchange]?.[pair]?.[type]
+      if (by_type) {
+        by_type[to_order_id] = order
+        delete by_type[from_order_id]
+      }
+    }
+  }
   init_account(balances, benchmarks) {
     this.account = new Account(balances, benchmarks)
   }
   update_non_initial_from_balances(balances) {
     this.account?.update_non_initial_from_balances(balances)
   }
-  limit_trade(exchange, pair, method, price, amount, id = uuid(), order_type = '', limiter_on = true, api_key_index, order_options = {}, decision_context) {
+  limit_trade(exchange, pair, method, price, amount, id = uuid(), order_type = '', limiter_on = true, api_key_index, order_options = {}, decision_context = null) {
     price = round_price(exchange, pair, price)
     amount = round_amount(exchange, pair, amount)
-    // console.log('%s %s %s %s %s %s at %s', limiter_on, order_type, exchange, pair, method, amount, price)
-    this.update_order_info(
-      exchange,
-      pair,
-      method,
-      id,
-      {
-        order_type,
-        status: 'INIT',
-        id,
-        order_id: id,
-        type: method,
-        price,
-        amount,
-        f_amount: 0,
-        f_percentage: 0,
-        t_amount: amount,
-        open_time: Date.now(),
-        close_time: null,
-        decision_context,
-      },
-      order_type + ' init',
-    )
-    let no_callback_timeout = setTimeout(() => {
-      if (_.get(this.order_info, [exchange, pair, method, id, 'status']) === 'INIT') {
-        console.log('no callback: %s %s %s %s %s', exchange, pair, method, id, order_type)
-        this.update_order_info(
-          exchange,
-          pair,
-          method,
-          id,
-          {
-            status: 'N-OPENED-TIMEOUT',
-          },
-          order_type + ' n-opened-timeout-init',
-        )
+    const internal_id = id
+    let no_callback_timeout = null
+    let init_emitted = false
+    const start_no_callback_timeout = () => {
+      if (no_callback_timeout) {
+        clearTimeout(no_callback_timeout)
+      }
+      no_callback_timeout = setTimeout(() => {
+        if (_.get(this.order_info, [exchange, pair, method, internal_id, 'status']) === 'INIT') {
+          console.log('no callback: %s %s %s %s %s', exchange, pair, method, internal_id, order_type)
+          this.update_order_info(
+            exchange,
+            pair,
+            method,
+            internal_id,
+            {
+              status: 'N-OPENED-TIMEOUT',
+            },
+            order_type + ' n-opened-timeout-init',
+          )
+        }
+      }, NO_CALLBACK_TIMEOUT_MS)
+    }
+    const emit_init = ({ t_order_sent } = {}) => {
+      if (init_emitted) {
+        return
       }
-    }, NO_CALLBACK_TIMEOUT_MS)
+      init_emitted = true
+      const init_decision_context = decision_context ? { ...decision_context } : null
+      const t2t_ns = hrtime_diff(init_decision_context?.t_decision_hr, t_order_sent)
+      if (init_decision_context) {
+        delete init_decision_context.t_decision_hr
+      }
+      this.update_order_info(
+        exchange,
+        pair,
+        method,
+        internal_id,
+        {
+          order_type,
+          status: 'INIT',
+          id: internal_id,
+          internal_id,
+          order_id: null,
+          type: method,
+          price,
+          amount,
+          f_amount: 0,
+          f_percentage: 0,
+          t_amount: amount,
+          open_time: Date.now(),
+          close_time: null,
+          decision_context: init_decision_context,
+          t2t_ns,
+          event_hrtime: t_order_sent,
+        },
+        order_type + ' init',
+      )
+      start_no_callback_timeout()
+    }
     let limit_trade = (cb) => {
+      const order_options_with_meta = Object.assign({}, order_options, {
+        on_sent: ({ t_order_sent }) => {
+          emit_init({ t_order_sent })
+        },
+      })
       if (limiter_on) {
         if (this.Exchanges[exchange].private_limiter) {
           this.Exchanges[exchange].private_limiter.priority = getOrderPriority(order_type)
         }
-        this.Exchanges[exchange].limit_trade(pair, method, price, amount, order_options, cb)
+        this.Exchanges[exchange].limit_trade(pair, method, price, amount, order_options_with_meta, cb)
       } else {
-        this.Exchanges[exchange].limit_trade_base(pair, method, price, amount, order_options, cb)
+        this.Exchanges[exchange].limit_trade_base(pair, method, price, amount, order_options_with_meta, cb)
       }
     }
     if (api_key_index) {
       this.Exchanges[exchange].key_index = api_key_index
     }
-    const t_order_sent = process.hrtime()
     limit_trade((res) => {
-      const t_ack_recv = process.hrtime()
       if (res.success) {
         let order_id = res.body
-        this.delete_order_info(exchange, pair, method, id, order_type + ' n-filled')
+        emit_init(res.meta || {})
+        this.move_order_info_key(exchange, pair, method, internal_id, order_id)
         this.update_order_info(
           exchange,
           pair,
@@ -346,7 +417,8 @@ module.exports = class Orders {
           {
             order_type,
             status: 'N-FILLED',
-            id,
+            id: internal_id,
+            internal_id,
             order_id,
             type: method,
             price,
@@ -354,31 +426,32 @@ module.exports = class Orders {
             f_amount: 0,
             f_percentage: 0,
             t_amount: amount,
-            open_time: Date.now(),
             close_time: null,
-            t_order_sent,
-            t_ack_recv,
+            order_rtt_ns: hrtime_diff(_.get(res, ['meta', 't_order_sent']), _.get(res, ['meta', 't_ack_recv'])),
+            event_hrtime: _.get(res, ['meta', 't_ack_recv']),
           },
           order_type + ' n-filled',
         )
       } else if (res.error === 'amount-too-small' || res.error === 'insufficient-funds') {
-        this.delete_order_info(exchange, pair, method, id, order_type + ' limit-trade-' + res.error)
+        this.delete_order_info(exchange, pair, method, internal_id, order_type + ' limit-trade-' + res.error)
       } else if (res.error === 'connection-error') {
         setTimeout(() => {
-          this.delete_order_info(exchange, pair, method, id, order_type + ' limit-trade-' + res.error)
+          this.delete_order_info(exchange, pair, method, internal_id, order_type + ' limit-trade-' + res.error)
         }, this.delete_connection_error_orders_timeout)
       } else {
         if (order_type.startsWith('self') || order_type.startsWith('maker_2')) {
-          this.delete_order_info(exchange, pair, method, id, order_type + ' limit-trade-timeout')
+          this.delete_order_info(exchange, pair, method, internal_id, order_type + ' limit-trade-timeout')
         }
         // else if (_.get(this.order_info, [exchange, pair, method, id, 'status']) === 'INIT') { // make sure no_callback_timeout is NOT running zero
         //   this.update_order_info(exchange, pair, method, id, {
         //     status: 'N-OPENED-TIMEOUT'
         //   }, order_type + ' n-opened-timeout-callback')
         // }
-        console.error('limit trade error: %s %s %s %s %s %s %s %j', order_type, id, exchange, pair, method, price, amount, res)
+        console.error('limit trade error: %s %s %s %s %s %s %s %j', order_type, internal_id, exchange, pair, method, price, amount, res)
+      }
+      if (no_callback_timeout) {
+        clearTimeout(no_callback_timeout)
       }
-      clearTimeout(no_callback_timeout)
     })
   }
   cancel_order(exchange, pair, order) {
@@ -407,6 +480,8 @@ module.exports = class Orders {
           {
             status: 'CANCELED',
             close_time: Date.now(),
+            cancel_latency_ns: hrtime_diff(_.get(res, ['meta', 't_cancel_sent']), _.get(res, ['meta', 't_cancel_ack'])),
+            event_hrtime: _.get(res, ['meta', 't_cancel_ack']),
           },
           order.order_type + ' canceled',
         )
@@ -557,8 +632,9 @@ module.exports = class Orders {
                 amount: 0,
                 f_amount: trade_amount,
                 f_percentage: trade_amount / order_info.t_amount,
-                close_time: trade.close_time || Date.now(),
+                close_time: trade.close_time ? new Date(trade.close_time).getTime() : Date.now(),
                 fees: trade.fees,
+                event_hrtime: process.hrtime(),
               },
               order_info.order_type + ' canceled-order-filled(-more)',
             )
@@ -597,8 +673,9 @@ module.exports = class Orders {
                 amount: order_info.t_amount - trade_amount,
                 f_amount: trade.amount,
                 f_percentage: trade_amount / order_info.t_amount,
-                close_time: trade.close_time || Date.now(),
+                close_time: trade.close_time ? new Date(trade.close_time).getTime() : Date.now(),
                 fees: trade.fees,
+                event_hrtime: process.hrtime(),
               },
               order_info.order_type + ' order-f-filled-1',
             )
@@ -684,6 +761,7 @@ module.exports = class Orders {
                     f_amount: order_info_order.t_amount,
                     f_percentage: 1,
                     close_time: Date.now(),
+                    event_hrtime: process.hrtime(),
                   },
                   order_info_order.order_type + ' order-f-filled-2',
                 )
@@ -718,7 +796,8 @@ module.exports = class Orders {
                   amount: order.amount,
                   f_amount: order_f_amount,
                   f_percentage: order_f_amount / order_info.t_amount,
-                  close_time: order.close_time || Date.now(),
+                  close_time: order.close_time ? new Date(order.close_time).getTime() : Date.now(),
+                  event_hrtime: process.hrtime(),
                 },
                 order_info.order_type + ' order-f-filled-3',
               )
@@ -749,8 +828,8 @@ module.exports = class Orders {
             this.get_order_info_status_all(exchange, pair, 'both', 'INIT').map((init_order) => {
               let init_order_price = round_price(exchange, pair, init_order.price)
               let order_price = round_price(exchange, pair, order.price)
-              let init_order_time = init_order.open_time.getTime()
-              let order_time = order.open_time.getTime()
+              let init_order_time = typeof init_order.open_time === 'number' ? init_order.open_time : new Date(init_order.open_time).getTime()
+              let order_time = typeof order.open_time === 'number' ? order.open_time : new Date(order.open_time).getTime()
               if (
                 init_order.type === order.type &&
                 init_order_price === order_price &&
@@ -758,7 +837,7 @@ module.exports = class Orders {
                 order_time - init_order_time < this.check_init_order_opened_timeout
               ) {
                 // console.log('ORDERS: %s %s %s %s INIT ORDER OPENED', init_order.order_id, order.order_id, exchange, pair)
-                this.delete_order_info(exchange, pair, init_order.type, init_order.order_id, init_order.order_type + ' init-order-opened')
+                this.move_order_info_key(exchange, pair, init_order.type, init_order.internal_id || init_order.id, order.order_id)
                 this.update_order_info(
                   exchange,
                   pair,
@@ -772,8 +851,9 @@ module.exports = class Orders {
                     amount: order.amount,
                     f_amount: 0,
                     t_amount: order.amount,
-                    open_time: order.open_time || Date.now(),
+                    open_time: order.open_time ? new Date(order.open_time).getTime() : Date.now(),
                     close_time: null,
+                    event_hrtime: process.hrtime(),
                   },
                   init_order.order_type + ' init-order-opened',
                 )
@@ -794,6 +874,7 @@ module.exports = class Orders {
               {
                 status: 'CANCELED',
                 close_time: Date.now(),
+                event_hrtime: process.hrtime(),
               },
               (order_info.order_type || '') + ' canceling-canceled-from-open-orders',
             )
@@ -814,6 +895,8 @@ module.exports = class Orders {
     const existing = this.order_info[exchange]?.[pair]?.[type]?.[order_id]
     const old_status = existing && existing.status
     const old_order_type = existing && existing.order_type
+    const event_hrtime = dict.event_hrtime
+    delete dict.event_hrtime
 
     for (let key in dict) {
       if (key === 'f_percentage') {
@@ -827,7 +910,15 @@ module.exports = class Orders {
     }
 
     const order = this.order_info[exchange][pair][type][order_id]
+    if (order.internal_id == null && order.id != null) {
+      order.internal_id = order.id
+    }
+    if (order.trace == null) {
+      order.trace = []
+    }
+    order.lifecycle_seq = existing && Number.isFinite(existing.lifecycle_seq) ? existing.lifecycle_seq + 1 : dict.lifecycle_seq || 1
     order.last_update_ts = Date.now()
+    this.append_trace(order, note)
 
     const new_status = order.status
     if (old_status !== new_status) {
@@ -884,14 +975,16 @@ module.exports = class Orders {
 
     // Send order update via WebSocket to dashboard backend
     const order_data = _.get(this.order_info, [exchange, pair, type, order_id], {})
-    // Only send N-FILLED and later statuses (skip INIT which has temporary order_id)
-    if (dict.status !== 'INIT' && order_data.status !== 'INIT' && ws_client) {
+    if (ws_client) {
       ws_client.send_order({
         action: 'update',
         exchange,
         pair,
         side: type,
         order_id,
+        internal_id: order_data.internal_id,
+        lifecycle_seq: order_data.lifecycle_seq,
+        event_hrtime,
         data: order_data,
         note,
       })
@@ -914,14 +1007,16 @@ module.exports = class Orders {
     }
     console.log('DELETE %s: %s %s %s %s', note, exchange, pair, type, order_id)
 
-    // Send delete event via WebSocket (only for non-INIT orders)
-    if (order_data.status !== 'INIT' && ws_client) {
+    if (ws_client && order_data.internal_id) {
       ws_client.send_order({
         action: 'delete',
         exchange,
         pair,
         side: type,
         order_id,
+        internal_id: order_data.internal_id,
+        lifecycle_seq: (order_data.lifecycle_seq || 0) + 1,
+        event_hrtime: process.hrtime(),
         data: order_data,
         note,
       })
@@ -1411,16 +1506,15 @@ module.exports = class Orders {
     const throttled_process_rates = _.throttle(process_rates, PROCESS_RATES_THROTTLE_MS, { leading: true, trailing: true })
     let rates_handler = (exchange, pair, res) => {
       if (res.success) {
-        let { asks, bids } = res.body
+        let { asks, bids, _ts } = res.body
         asks = asks.slice(0, RATES_LENGTH_MAX)
         bids = bids.slice(0, RATES_LENGTH_MAX)
-        const _ts = _.get(this.Exchanges, [exchange, 'ws', 'market', 'ts_dict', pair]) || _.get(this.ExchangesRatesOnly, [exchange, 'ws', 'market', 'ts_dict', pair])
-        _.set(this.rates, [exchange, pair], { asks, bids, _ts })
+        _.set(this.rates, [exchange, pair], { asks, bids, _ts, update_time: new Date() })
         /*
          * throttle to 20ms
          */
         throttled_process_rates()
-      } else if (_.get(this.rates, [exchange, pair, '_ts', 'received_ts'], 0) < Date.now() - this.interval_dict[exchange].rates) {
+      } else if (_.get(this.rates, [exchange, pair, 'update_time']) < Date.now() - this.interval_dict[exchange].rates) {
         console.log('%s %s rates expired: %j', exchange, pair, res)
         if (this.rates[exchange]) delete this.rates[exchange][pair]
         this.rates_observable.notify()
@@ -1529,13 +1623,13 @@ module.exports = class Orders {
       for (let exchange in exchange_pairs) {
         for (let pair of exchange_pairs[exchange]) {
           if (Exchanges[exchange].ohlcv) {
-            const ohlcv_from_timetamp = new Date().floorHours(1).getTime() - OHLCV_DEFAULT_PAST_HOURS * 3_600_000
+            const ohlcv_from_timetamp = new Date().floorHours(1).addHours(-OHLCV_DEFAULT_PAST_HOURS).getTime()
             const ohlcv_to_timestamp = new Date().floorHours(1).getTime()
             Exchanges[exchange].ohlcv(pair, ohlcv_from_timetamp, ohlcv_to_timestamp, (res) => {
               ohlcv_handler(exchange, pair, res)
             })
             setInterval(() => {
-              const ohlcv_from_timetamp = new Date().floorHours(1).getTime() - OHLCV_DEFAULT_PAST_HOURS * 3_600_000
+              const ohlcv_from_timetamp = new Date().floorHours(1).addHours(-OHLCV_DEFAULT_PAST_HOURS).getTime()
               const ohlcv_to_timestamp = new Date().floorHours(1).getTime()
               Exchanges[exchange].ohlcv(pair, ohlcv_from_timetamp, ohlcv_to_timestamp, (res) => {
                 ohlcv_handler(exchange, pair, res)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@icgio/icg-exchanges-wrapper",
-  "version": "1.17.20",
+  "version": "1.17.22",
   "description": "",
   "main": "index.js",
   "scripts": {
@@ -17,8 +17,8 @@
   },
   "homepage": "https://github.com/icgio/icg-exchanges-wrapper#readme",
   "dependencies": {
-    "@icgio/icg-exchanges": "^1.40.43",
-    "@icgio/icg-utils": "^1.9.60",
+    "@icgio/icg-exchanges": "^1.40.45",
+    "@icgio/icg-utils": "^1.9.61",
     "lodash": "^4.17.23",
     "pg": "^8.17.2"
   }