@icgio/icg-exchanges-wrapper 1.17.14 → 1.17.15

package/docs/2026-04-09-perf-cleanup.md

@@ -0,0 +1,366 @@
+# Performance cleanup — 2026-04-09
+
+Session of performance and dead-code cleanup work spanning three repos:
+
+- `icg-utils` — `combine_orderbooks` rewrite + new tests
+- `icg-exchanges-wrapper` — `orders.js` hot-path optimization + dead-code removal
+- `icg-market-maker` — small follow-up to drop `o.trace` references
+
+Nothing in this document has been published or deployed. Versions, dependents,
+and MM client servers still need updating per the deployment notes at the
+bottom.
+
+## Why
+
+The maker hot path (rate update → `process_rates` → `notify()` → maker
+callback → `handle_maker_*`) was the bottleneck under analysis. Several
+contributing costs were identified and addressed. Expected aggregate effect:
+the per-rate-tick cost on the maker hot path should drop from the
+**multi-millisecond range to sub-millisecond** for typical configurations
+(5 exchanges × 3 pairs × ~100 orders).
+
+## Changes
+
+### icg-utils
+
+#### Rewrote `combine_orderbooks` (2026-04-09)
+
+`lib/utils.js`
+
+The original implementation:
+- Used `_.get`, `_.sum`, `_.values`, `_.keys` inside the inner loop
+- Mutated input tuples in place by pushing the exchange name (`tuple[2]`)
+  onto each level — forcing callers to `cloneDeep` upstream
+- O(R × K) with high lodash constant factors
+
+The rewrite:
+- Strips all lodash from the inner loop
+- Uses parallel arrays (`lists`, `indexes`, `exchanges`) keyed by an integer
+  index, hoists the asks/bids direction comparator out of the loop
+- **Non-mutating**: `orderbooks_dict` and its inner arrays/tuples are never
+  touched. Callers do NOT need to deep-clone before calling.
+- **Pushes input tuple references directly into the result** (Flavor 2 — see
+  Reference Sharing Contract below). Zero allocations in the inner loop.
+- Drops the `tuple[2]` augmentation entirely. The previous algorithm needed
+  the exchange name on each tuple to know which pointer to advance; the
+  rewrite tracks this via `bestIdx` (an integer index into the parallel
+  arrays).
+
+Expected speedup vs the original: ~10× per call. Combined with the upstream
+`cloneDeep` removal, `process_rates` per call drops from ~3-5 ms to ~0.2-0.4
+ms for a typical 5-exchange / 3-pair MM.
+
+##### Reference Sharing Contract
+
+The returned `{asks, bids}` arrays contain references to the SAME tuple
+objects that live inside the caller's input. **Callers MUST treat both the
+input and the result as read-only.** Mutating either will corrupt the other.
+
+This is safe in `icg-exchanges-wrapper`'s `process_rates` flow because:
+
+1. WS rate updates REPLACE the per-pair object via
+   `_.set(this.rates, [exchange, pair], { asks, bids, ... })` rather than
+   mutating in place — old tuple references simply become garbage when no
+   one holds them.
+2. The combined-rates consumer (the MM `rates_observable` subscriber) runs
+   synchronously inside `notify()` in the same call stack, so no other WS
+   handler can interleave and rebuild rates underneath us.
+
+If you ever need result tuples that are independent of the input (e.g. to
+pass to a mutating consumer like `vol_position_combined`, which expects a
+`[price, amount, exchange]` 3-tuple), wrap them at the call site — do not
+put that contract back into `combine_orderbooks`.
+
+#### Added `test/combine-orderbooks.test.js` (2026-04-09)
+
+`test/combine-orderbooks.test.js`
+
+13 tests covering:
+
+- Basic merge ordering (asks ascend, bids descend, length limit honoured)
+- Result tuples remain `[price, amount]` (no `[2]` augmentation)
+- Input is byte-equal to its pre-call snapshot (non-mutation contract)
+- Idempotent on repeated calls
+- Result tuples `===` input tuples (Flavor 2 reference aliasing)
+- Empty dict, empty side per exchange, single exchange, length larger than
+  available, no length limit
+- 3-way interleaving correctness
+- Tied prices are deterministic
+- Fractional prices
+
+Wired into `npm test` via `package.json` (now runs both `rate-limiter.test.js`
+and `combine-orderbooks.test.js`).
+
+### icg-exchanges-wrapper
+
+All changes in `middleware/orders.js` unless noted otherwise.
+
+#### Dropped `_.cloneDeep` in `process_rates` (2026-04-09)
+
+The deep clone existed only because the old `combine_orderbooks` mutated its
+input. With the rewrite the input is never touched, so `process_rates` now
+passes the live `this.rates[exchange][pair]` reference directly. Saves
+~600 µs per `process_rates` call for a typical 5-exchange / 3-pair / 20-level
+MM.
+
+#### Removed dead DEX code (2026-04-09)
+
+`rates_with_routes` and the entire DEX trading path were referenced from six
+call sites in `orders.js` and one in
+`functions/get-exchanges-rates-market-history-ohlcv.js`, but no exchange in
+`icg-exchanges` ever set `exchange_type === 'dex'` and no exchange implemented
+`rate_with_routes` / `rate_with_routes_ws` / `limit_trade_with_routes`. The
+branches were unreachable in production.
+
+Removed:
+
+- `this.rates_with_routes = {}` initialization
+- `limit_trade`'s DEX branch (~30 lines including closest-level / route lookup)
+- `if (rates_with_routes)` block in `rates_handler`, plus the
+  `rates_with_routes = false` parameter
+- Both DEX branches in `get_rates` (with and without `ws_market`)
+- DEX startup branch that pre-set `previous_orders_canceled_exchange_pair`
+- The `else if (Exchange.rate_with_routes)` REST fallback in
+  `functions/get-exchanges-rates-market-history-ohlcv.js`
+
+Net: ~74 lines of dead code removed.
+
+#### Removed `update_order_info` trace block (2026-04-09)
+
+`update_order_info` previously appended to a per-order `trace` array on every
+update via `[...existing_trace, trace_entry]`. This was:
+
+- O(N) quadratic per update — the spread copies the entire existing trace
+- Unbounded memory growth per long-lived order
+- Sent over WebSocket on every update, growing the WS payload over time
+
+The trace had two consumers:
+
+1. **`maker.js`** read `_.last(o.trace).ts` to filter stale CANCELING orders
+   in `maker_2` and `maker_3`. Replaced with a new `last_update_ts` field set
+   via native assignment in `update_order_info`.
+2. **The dashboard "Trace Timeline" UI** showed the full status history for
+   any clicked order. This is real value, but the wrapper is the wrong place
+   to materialize it. Plan: rebuild it on the dashboard backend from the
+   `ws_client.send_order` events that already arrive there. See
+   "Future work" below.
+
+The wrapper now does zero trace work. The dashboard "Trace Timeline" will
+temporarily show "No trace available for this order" until the backend
+update ships.
+
+#### Added status / order_type indexes (2026-04-09)
+
+Two new indexes maintained incrementally by `update_order_info` and
+`delete_order_info`:
+
+- `this.order_status_index[exchange][pair][type][status] -> Set<order_ref>`
+  Used by `get_order_info_status_all`.
+- `this.order_info_by_type[order_type][exchange][pair][type][order_id] -> ref`
+  Used by `get_order_info_by_order_type`.
+
+Maintenance happens at the end of `update_order_info` after the dict fields
+are written. The function snapshots the old `status` / `order_type` BEFORE
+the for-loop, then compares to the new values after, and moves the order
+between buckets only if the relevant field actually changed. The common
+case (a non-status update like `f_amount` or `last_update_ts`) is a no-op.
+
+`delete_order_info` removes the order from both indexes using the order's
+current `status` / `order_type` BEFORE deleting it from the canonical
+`this.order_info`.
+
+`get_order_info_status_all` was rewritten to look up the precomputed Set
+and return `Array.from(...)`. Handles both single-status and array-status
+queries, plus the `'both'` recursive case.
+
+`get_order_info_by_order_type` walks only the matching `order_type`
+subtree(s) and returns a freshly built top-level dict (so callers can safely
+mutate the returned structure) with leaves that are still refs into
+`this.order_info`.
+
+##### Indexing invariants
+
+- Every order in `this.order_info[exchange][pair][type][order_id]` has at
+  most one entry in `order_status_index[exchange][pair][type][status]` and
+  at most one entry in
+  `order_info_by_type[order_type][exchange][pair][type][order_id]`.
+- The only mutators of `this.order_info` are `update_order_info` and
+  `delete_order_info`. Both maintain the indexes. Anyone who writes directly
+  to `this.order_info` outside these two methods will desynchronize the
+  indexes.
+- The previous `this.order_info_status` cache (used by the old
+  `get_order_info_status_all`) is removed.
+- `Account.order_type === 'range-order'` mutations at `:1006` and `:1011`
+  occur on `res.body` open-orders entries from the exchange WS, NOT on
+  `this.order_info` orders, so they don't affect the indexes.
+
+##### Functional verification
+
+A standalone in-process test exercises the indexed code paths against a
+stub instance. 11 assertions cover:
+
+1. Insert maker_1 sell INIT → in INIT bucket, in maker_1 subtree
+2. Transition INIT → N-FILLED → moves buckets, stays in maker_1
+3. `'both'` query → unions sell + buy
+4. Array status query → unions multiple status buckets
+5. by_order_type with array → merges multiple subtrees
+6. `delete_order_info` → removed from BOTH indexes + canonical store
+7. Multiple status transitions → bucket moves are atomic
+8. `last_update_ts` is stamped (numeric, > 0)
+9. Empty queries return `[]` / `{}` (no crash)
+10. Result arrays from `get_order_info_status_all` are independent (fresh
+    array per call) but inner orders are the same refs
+11. Mutating the dict returned by `get_order_info_by_order_type` does not
+    affect the canonical store
+
+#### Fixed `check_balances` shadow bug (2026-04-09)
+
+The original inner loop used `for (let exchange in this.exchange_cur_quote_cur)`
+which **shadowed** the outer `exchange` parameter. Every balance update
+rebuilt `buy_sell_limits_against_reserve_balance` for ALL exchanges, doing
+K× the necessary work where K = exchange count. The fix scopes the inner
+loop to the single exchange whose balance just arrived. Also collapsed the
+empty `if (max > 0) { } else { max = 0 }` blocks to `if (max < 0) max = 0`.
+
+#### Early-exit on `previous_orders_canceled` recompute (2026-04-09)
+
+The recompute walked every `(exchange, pair)` cell every time. Now uses a
+labeled `break outer` on the first unmarked cell. Steady-state behavior
+unchanged (the outer guard `if (!this.previous_orders_canceled)` still
+prevents re-runs once the flag is true), but startup is faster.
+
+#### `_.omit(obj, [key])` → `delete obj[key]` (2026-04-09)
+
+16 sites. `_.omit` allocates a new object copying every other key — O(N)
+plus GC pressure. `delete` is O(1) and free.
+
+Sites:
+
+- `delete_order_info` (`this.order_info[exchange][pair][type]`)
+- `check_balances` expiry (`this.balances`,
+  `this.buy_sell_limits_against_reserve_balance`)
+- `check_positions` expiry (`this.positions`)
+- `open_orders_handler` and `all_open_orders_handler` expiry
+  (`this.open_orders[exchange]`)
+- `trades_handler` and `all_trades_handler` expiry (`this.trades[exchange]`)
+- `query_order_handler` expiry (`this.query_order[exchange]`)
+- `process_rates` local-variable cleanups (`orderbook_major`,
+  `orderbook_major_temp`) and `this.rates_combined_major`
+- `rates_handler` expiry (`this.rates[exchange]`)
+- `market_history_handler` expiry (`this.market_history[exchange]`)
+- `ohlcv_handler` short-data and expiry (`this.ohlcv[exchange]`)
+
+#### `new Date().getTime()` → `Date.now()` (2026-04-09)
+
+13 sites. `new Date().getTime()` allocates a Date object just to read its
+timestamp; `Date.now()` is an intrinsic with no allocation. Pure micro-win.
+
+### icg-market-maker
+
+#### Replaced `_.last(o.trace).ts` filters with `o.last_update_ts` (2026-04-09)
+
+`scripts/maker.js`
+
+Two filters in `maker_2` and `maker_3` (for stale CANCELING orders) replaced
+with the new `o.last_update_ts` field added to `orders.js` in this session.
+No semantic change.
+
+## Files touched
+
+| Repo | File | Note |
+|---|---|---|
+| `icg-utils` | `lib/utils.js` | `combine_orderbooks` rewrite |
+| `icg-utils` | `test/combine-orderbooks.test.js` | new file, 13 tests |
+| `icg-utils` | `package.json` | `test` script runs both files |
+| `icg-exchanges-wrapper` | `middleware/orders.js` | (a) drop `_.cloneDeep`, (b) remove DEX dead code, (c) remove trace block + add `last_update_ts`, (d) status / order_type indexes, (e) rewrite `get_order_info_status_all` and `get_order_info_by_order_type`, (f) `check_balances` shadow + scope, (g) early-exit `previous_orders_canceled` recompute, (h) `_.omit` → `delete` (16 sites), (i) `new Date().getTime()` → `Date.now()` (13 sites) |
+| `icg-exchanges-wrapper` | `functions/get-exchanges-rates-market-history-ohlcv.js` | remove DEX REST fallback |
+| `icg-market-maker` | `scripts/maker.js` | two `last_trace` filters → `o.last_update_ts` |
+
+## Net effect
+
+Hot path per rate tick (5 exchanges × 3 pairs × ~100 orders):
+
+| Operation | Before | After |
+|---|---|---|
+| `process_rates` per pair | ~1-2 ms | ~0.2-0.4 ms |
+| `combine_orderbooks` per call | ~500-1000 µs | ~50-100 µs |
+| `get_order_info_status_all` × ~270 calls/tick | walks all matching orders for that (exchange, pair, type), lodash-heavy | Set lookup + `Array.from`, O(matching) |
+| `get_order_info_by_order_type` × ~15 calls/tick | 4-deep scan of all `order_info`, lodash-heavy | walks only matching subtree |
+| `update_order_info` trace cost | O(N) spread per update | 0 |
+| `check_balances` per balance event | K× redundant rebuild | 1× rebuild scoped to the right exchange |
+| `previous_orders_canceled` recompute | always full walk while flag is false | early-exit on first miss |
+| `_.omit` × 16 sites | O(N) new-object allocation | O(1) `delete`, no allocation |
+| `Date` instances per check | 13 per tick | 0 |
+
+## Future work
+
+### Backend trace materialization
+
+Rebuild the "Trace Timeline" data on
+`clients-dashboard-backend/src/services/ws-data-service.js` from the
+`ws_client.send_order` events it already receives. About 12 lines added in
+`handle_order_event`. The wrapper does no trace work; the backend caps the
+trace at last 20 entries; the dashboard UI is unchanged. Spans
+`clients-dashboard-backend` only — neither the wrapper nor the maker need to
+ship for this.
+
+The required pattern in `handle_order_event`:
+
+```js
+if (action === 'update') {
+  const existing = _.get(this.order_info[client_id], [exchange, pair, side, order_id])
+  const trace = (existing && existing.trace) || []
+  trace.push({
+    ts: Date.now(),
+    status: data.status,
+    note,
+    f_amount: data.f_amount,
+  })
+  if (trace.length > TRACE_MAX_ENTRIES) {
+    trace.splice(0, trace.length - TRACE_MAX_ENTRIES)
+  }
+  _.setWith(
+    this.order_info[client_id],
+    `${exchange}.${pair}.${side}.${order_id}`,
+    { ...data, trace, ts: Date.now() },
+    Object,
+  )
+  // ... rest unchanged
+}
+```
+
+Plus a small change to `handle_order_sync` to preserve existing traces
+across full re-sync (otherwise reconnects wipe history).
+
+### Other items not done in this session
+
+- **`Observable.notify()` try/catch** in `icg-utils/lib/observable.js`. Today
+  there's only one subscriber so a throw is just a crash, but the moment a
+  second subscriber is added, a throw in #1 silently kills #2. Two-line fix.
+- **Remaining lodash hot-path cleanup**: `_.set` / `_.setWith` / `_.get` in
+  `update_order_info` and `rates_handler` could be replaced with native
+  indexing for ~5-20× speedup on those calls.
+- **Empty `cancel_order_by_order` callbacks** at `orders.js:1038`, `:1045`:
+  failures silently ignored. Robustness, not perf.
+- **Hot-path `console.log('UPDATE %s: ...')` formatting** at `orders.js:862`:
+  runs on every `update_order_info` call. Should be gated with a verbosity
+  flag or routed through `log_limiter` (like maker.js does).
+- **`process_open_orders` nested scans** at `orders.js:618-766`. Same
+  indexing approach as the status / order_type indexes would help here too,
+  but it's off the hot path so lower priority.
+
+## Deployment notes
+
+When this work ships:
+
+1. `icg-utils` → bump version → publish to npm (new `combine_orderbooks`).
+2. `icg-exchanges-wrapper` → bump `@icgio/icg-utils` dep range → `npm install`
+   → bump pkg version → publish.
+3. **MM client servers** → manual deploy across all 12. Per
+   `feedback_mm_servers_dirty_lockfile.md`, pre-stash any dirty
+   `package.json` / `package-lock.json` on each server before
+   `git pull && npm ci`. The deploy carries both the wrapper republish and
+   the `maker.js` change (they both touch the maker process).
+4. The dashboard "Trace Timeline" will show "No trace available" until the
+   `clients-dashboard-backend` update ships separately. Plan that as a
+   follow-up.

package/functions/get-exchanges-rates-market-history-ohlcv.js

@@ -374,26 +374,6 @@ const get_exchanges_rates = function (Exchanges, exchange_pair_dict, cb, ws_enab
         })
 
         promises = promises.concat(rate_promises) // Add all rate promises
-      } else if (Exchange.rate_with_routes) {
-        let pairs_to_fetch = exchange_pair_dict[Exchange_name] || all_pairs
-        let rate_with_routes_promises = pairs_to_fetch.map((pair) => {
-          return new Promise((resolve, reject) => {
-            Exchange.rate_with_routes(pair, function (res) {
-              if (res.success) {
-                let { asks, bids } = res.body
-                asks = asks.map((ask) => ask.slice(0, 2))
-                bids = bids.map((bid) => bid.slice(0, 2))
-                set_pair_data(exchange_pair_rates, pair_exchange_rates, Exchange_name, pair, { asks, bids })
-              } else {
-                console.error(`getting rate with routes failed for ${Exchange_name} ${pair}:`, res)
-                delete_pair_data(exchange_pair_rates, pair_exchange_rates, Exchange_name, pair)
-              }
-              resolve()
-            })
-          })
-        })
-
-        promises = promises.concat(rate_with_routes_promises) // Add all rate_with_routes promises
       }
     }
   }

package/middleware/orders.js

@@ -200,7 +200,6 @@ module.exports = class Orders {
     this.trades = {}
     this.query_order = {}
     this.rates = {}
-    this.rates_with_routes = {}
     this.rates_observable = new Observable()
     this.market_history = {}
     this.ohlcv = {}
@@ -209,7 +208,8 @@ module.exports = class Orders {
     this.buy_sell_limits_against_reserve_balance = {}
     this.order_info = {}
     this.fill_records = [] // { side, usdt_amount, timestamp, order_type }
-    this.order_info_status = {}
+    this.order_status_index = {}
+    this.order_info_by_type = {}
     this.function_enabled = settings.function_enabled || {
       balances_check: true,
       positions_check: false,
@@ -319,39 +319,7 @@ module.exports = class Orders {
       }
     }, NO_CALLBACK_TIMEOUT_MS)
     let limit_trade = (cb) => {
-      if (this.Exchanges[exchange].exchange_type === 'dex') {
-        let rates_with_routes = this.rates_with_routes[exchange][pair]
-        let routes
-        if (method === 'buy') {
-          let closest_level = _.minBy(rates_with_routes.asks, (ask) => {
-            return Math.abs(ask[0] - price)
-          })
-          routes = [closest_level[2]]
-          // for (let ask of rates_with_routes.asks) {
-          //   if (ask[0] === price) {
-          //     routes = [ask[2]]
-          //   }
-          // }
-        } else {
-          let closest_level = _.minBy(rates_with_routes.bids, (bid) => {
-            return Math.abs(bid[0] - price)
-          })
-          routes = [closest_level[2]]
-          // for (let bid of rates_with_routes.bids) {
-          //   if (bid[0] === price) {
-          //     routes = [bid[2]]
-          //   }
-          // }
-        }
-        if (routes) {
-          if (this.Exchanges[exchange].private_limiter) {
-            this.Exchanges[exchange].private_limiter.priority = getOrderPriority(order_type)
-          }
-          this.Exchanges[exchange].limit_trade_with_routes(pair, method, price, amount, routes, cb)
-        } else {
-          console.error('dex no routes:', rates_with_routes, price)
-        }
-      } else if (limiter_on) {
+      if (limiter_on) {
         if (this.Exchanges[exchange].private_limiter) {
           this.Exchanges[exchange].private_limiter.priority = getOrderPriority(order_type)
         }
@@ -494,33 +462,61 @@ module.exports = class Orders {
     return stats
   }
   get_order_info_by_order_type(order_type) {
-    let order_info_temp = {}
-    for (let exchange in this.order_info) {
-      for (let pair in this.order_info[exchange]) {
-        for (let type in this.order_info[exchange][pair]) {
-          for (let order_id in this.order_info[exchange][pair][type]) {
-            const order_type_temp = _.get(this.order_info, [exchange, pair, type, order_id, 'order_type'])
-            if ((Array.isArray(order_type) && order_type.includes(order_type_temp)) || order_type_temp === order_type) {
-              _.setWith(order_info_temp, `${exchange}.${pair}.${type}.${order_id}`, _.get(this.order_info, `${exchange}.${pair}.${type}.${order_id}`), Object)
+    const types = Array.isArray(order_type) ? order_type : [order_type]
+    const result = {}
+    // Walk only the index sub-trees for the requested order_type(s). The
+    // returned dict is freshly built (preserving the original semantics that
+    // callers may add/remove top-level keys without affecting the canonical
+    // store) but the leaf order objects are still refs into this.order_info.
+    for (const t of types) {
+      const sub = this.order_info_by_type[t]
+      if (!sub) continue
+      for (const exchange in sub) {
+        const exch_sub = sub[exchange]
+        let result_exch = result[exchange]
+        if (!result_exch) {
+          result_exch = {}
+          result[exchange] = result_exch
+        }
+        for (const pair in exch_sub) {
+          const pair_sub = exch_sub[pair]
+          let result_pair = result_exch[pair]
+          if (!result_pair) {
+            result_pair = {}
+            result_exch[pair] = result_pair
+          }
+          for (const inner_type in pair_sub) {
+            const type_sub = pair_sub[inner_type]
+            let result_type = result_pair[inner_type]
+            if (!result_type) {
+              result_type = {}
+              result_pair[inner_type] = result_type
+            }
+            for (const order_id in type_sub) {
+              result_type[order_id] = type_sub[order_id]
             }
           }
         }
       }
     }
-    return order_info_temp
+    return result
   }
   get_order_info_status_all(exchange, pair, type, status) {
     if (type === 'both') {
       return this.get_order_info_status_all(exchange, pair, 'sell', status).concat(this.get_order_info_status_all(exchange, pair, 'buy', status))
     }
-    _.set(this.order_info_status, [exchange, pair, type, status], [])
-    for (let order_id in _.get(this.order_info, [exchange, pair, type], {})) {
-      let order_status = _.get(this.order_info, [exchange, pair, type, order_id, 'status'])
-      if (order_status === status || _.includes(status, order_status)) {
-        this.order_info_status[exchange][pair][type][status].push(this.order_info[exchange][pair][type][order_id])
+    const buckets_by_status = this.order_status_index[exchange]?.[pair]?.[type]
+    if (!buckets_by_status) return []
+    if (Array.isArray(status)) {
+      const result = []
+      for (const s of status) {
+        const bucket = buckets_by_status[s]
+        if (bucket) for (const order of bucket) result.push(order)
       }
+      return result
     }
-    return this.order_info_status[exchange][pair][type][status]
+    const bucket = buckets_by_status[status]
+    return bucket ? Array.from(bucket) : []
   }
   get_order_info_status_by_order_type(exchange, pair, type, status, order_type) {
     return this.get_order_info_status_all(exchange, pair, type, status).filter((o) => o.order_type === order_type)
@@ -656,7 +652,7 @@ module.exports = class Orders {
             let order_info_order = this.order_info[exchange][pair][type][order_id]
             if (
               (order_info_order.status === 'N-FILLED' || order_info_order.status === 'P-FILLED') &&
-              order_info_order.open_time < new Date().getTime() - this.no_trades_function_f_filled_timeout
+              order_info_order.open_time < Date.now() - this.no_trades_function_f_filled_timeout
             ) {
               let exist = false
               open_orders[exchange][pair].open_orders.map((o) => {
@@ -803,6 +799,11 @@ module.exports = class Orders {
       console.error('UPDATE %s: order_id invalid %s %s %s %s %j', note, exchange, pair, type, order_id, dict)
       return
     }
+
+    const existing = this.order_info[exchange]?.[pair]?.[type]?.[order_id]
+    const old_status = existing && existing.status
+    const old_order_type = existing && existing.order_type
+
     for (let key in dict) {
       if (key === 'f_percentage') {
         dict[key] = Math.round(dict[key] * 100) / 100
@@ -814,15 +815,37 @@ module.exports = class Orders {
       _.setWith(this.order_info, `${exchange}.${pair}.${type}.${order_id}.${key}`, dict[key], Object)
     }
 
-    // Add trace entry for order history
-    const trace_entry = {
-      ts: Date.now(),
-      status: dict.status || _.get(this.order_info, [exchange, pair, type, order_id, 'status']),
-      note,
-      f_amount: dict.f_amount,
+    const order = this.order_info[exchange][pair][type][order_id]
+    order.last_update_ts = Date.now()
+
+    const new_status = order.status
+    if (old_status !== new_status) {
+      if (old_status) {
+        const old_bucket = this.order_status_index[exchange]?.[pair]?.[type]?.[old_status]
+        if (old_bucket) old_bucket.delete(order)
+      }
+      if (new_status) {
+        const by_exch = this.order_status_index[exchange] || (this.order_status_index[exchange] = {})
+        const by_pair = by_exch[pair] || (by_exch[pair] = {})
+        const by_type = by_pair[type] || (by_pair[type] = {})
+        const bucket = by_type[new_status] || (by_type[new_status] = new Set())
+        bucket.add(order)
+      }
+    }
+    const new_order_type = order.order_type
+    if (old_order_type !== new_order_type) {
+      if (old_order_type) {
+        const at = this.order_info_by_type[old_order_type]?.[exchange]?.[pair]?.[type]
+        if (at) delete at[order_id]
+      }
+      if (new_order_type) {
+        const by_type = this.order_info_by_type[new_order_type] || (this.order_info_by_type[new_order_type] = {})
+        const by_exch = by_type[exchange] || (by_type[exchange] = {})
+        const by_pair = by_exch[pair] || (by_exch[pair] = {})
+        const by_inner_type = by_pair[type] || (by_pair[type] = {})
+        by_inner_type[order_id] = order
+      }
     }
-    const existing_trace = _.get(this.order_info, [exchange, pair, type, order_id, 'trace'], [])
-    _.setWith(this.order_info, `${exchange}.${pair}.${type}.${order_id}.trace`, [...existing_trace, trace_entry], Object)
 
     console.log('UPDATE %s: %s %s %s %s %j', note, exchange, pair, type, order_id, dict)
     // start: log every f-filled trade for future analysis
@@ -865,7 +888,19 @@ module.exports = class Orders {
   }
   delete_order_info(exchange, pair, type, order_id, note) {
     const order_data = _.get(this.order_info, [exchange, pair, type, order_id], {})
-    this.order_info[exchange][pair][type] = _.omit(this.order_info[exchange][pair][type], [order_id])
+
+    if (order_data.status) {
+      const bucket = this.order_status_index[exchange]?.[pair]?.[type]?.[order_data.status]
+      if (bucket) bucket.delete(order_data)
+    }
+    if (order_data.order_type) {
+      const at = this.order_info_by_type[order_data.order_type]?.[exchange]?.[pair]?.[type]
+      if (at) delete at[order_id]
+    }
+
+    if (this.order_info[exchange]?.[pair]?.[type]) {
+      delete this.order_info[exchange][pair][type][order_id]
+    }
     console.log('DELETE %s: %s %s %s %s', note, exchange, pair, type, order_id)
 
     // Send delete event via WebSocket (only for non-INIT orders)
@@ -885,28 +920,17 @@ module.exports = class Orders {
     let balances_handler = (exchange, res) => {
       if (res.success) {
         _.set(this.balances, [exchange], Object.assign(res.body, { update_time: new Date() }))
-        for (let exchange in this.exchange_cur_quote_cur) {
-          for (let quote_cur in this.exchange_cur_quote_cur[exchange]) {
-            this.exchange_cur_quote_cur[exchange][quote_cur].map((cur) => {
-              let cur_balance = _.get(this.balances, [exchange, 'available', cur], 0)
-              let quote_cur_balance = _.get(this.balances, [exchange, 'available', quote_cur], 0)
-              let pair = cur + quote_cur
+        const cur_quote_cur = this.exchange_cur_quote_cur[exchange]
+        if (cur_quote_cur) {
+          for (let quote_cur in cur_quote_cur) {
+            cur_quote_cur[quote_cur].map((cur) => {
+              const cur_balance = _.get(this.balances, [exchange, 'available', cur], 0)
+              const quote_cur_balance = _.get(this.balances, [exchange, 'available', quote_cur], 0)
+              const pair = cur + quote_cur
               let max_cur_amount = cur_balance - _.get(this.min_reserve_balance, [exchange, cur], 0)
               let max_quote_cur_amount = quote_cur_balance - _.get(this.min_reserve_balance, [exchange, quote_cur], 0)
-              if (max_cur_amount > 0) {
-              } else {
-                // if (cur_balance > 0) {
-                //   console.log('balance low', exchange, cur, cur_balance, this.min_reserve_balance[exchange][cur])
-                // }
-                max_cur_amount = 0
-              }
-              if (max_quote_cur_amount > 0) {
-              } else {
-                // if (quote_cur_balance > 0) {
-                //   console.log('balance low', exchange, quote_cur, quote_cur_balance, this.min_reserve_balance[exchange][quote_cur])
-                // }
-                max_quote_cur_amount = 0
-              }
+              if (max_cur_amount < 0) max_cur_amount = 0
+              if (max_quote_cur_amount < 0) max_quote_cur_amount = 0
               _.set(this.buy_sell_limits_against_reserve_balance, [exchange, pair], {
                 sell: max_cur_amount,
                 buy: max_quote_cur_amount,
@@ -918,10 +942,10 @@ module.exports = class Orders {
           this.account_init = true
           this.init_account(this.balances, this.accounting_benchmark || this.balance_benchmark) // use accounting_benchmark if it exists
         }
-      } else if (_.get(this.balances, [exchange, 'update_time']) < new Date().getTime() - this.interval_dict[exchange].balances) {
+      } else if (_.get(this.balances, [exchange, 'update_time']) < Date.now() - this.interval_dict[exchange].balances) {
         console.log('%s balances expired: %j', exchange, res)
-        this.balances = _.omit(this.balances, [exchange])
-        this.buy_sell_limits_against_reserve_balance = _.omit(this.buy_sell_limits_against_reserve_balance, [exchange])
+        delete this.balances[exchange]
+        delete this.buy_sell_limits_against_reserve_balance[exchange]
       }
     }
     function get_balances(Exchanges, interval_dict) {
@@ -949,9 +973,9 @@ module.exports = class Orders {
     let position_all_handler = (exchange, res) => {
       if (res.success) {
         _.set(this.positions, [exchange], Object.assign(res.body, { update_time: new Date() }))
-      } else if (_.get(this.positions, [exchange, 'update_time']) < new Date().getTime() - this.interval_dict[exchange].positions) {
+      } else if (_.get(this.positions, [exchange, 'update_time']) < Date.now() - this.interval_dict[exchange].positions) {
         console.log('%s positions expired: %j', exchange, res)
-        this.positions = _.omit(this.positions, [exchange])
+        delete this.positions[exchange]
       }
     }
     function get_positions(Exchanges, interval_dict) {
@@ -1018,18 +1042,21 @@ module.exports = class Orders {
           }
         }
         if (!this.previous_orders_canceled) {
-          let previous_orders_canceled = true
-          for (let exchange in this.exchange_pairs) {
-            this.exchange_pairs[exchange].map((pair) => {
-              if (!_.get(this.previous_orders_canceled_exchange_pair, [exchange, pair])) {
-                previous_orders_canceled = false
+          let all_canceled = true
+          outer: for (const exch of Object.keys(this.exchange_pairs)) {
+            const pairs = this.exchange_pairs[exch]
+            const exch_marks = this.previous_orders_canceled_exchange_pair[exch]
+            for (let i = 0; i < pairs.length; i++) {
+              if (!exch_marks || !exch_marks[pairs[i]]) {
+                all_canceled = false
+                break outer
               }
-            })
+            }
           }
-          if (previous_orders_canceled) {
+          if (all_canceled) {
             console.log('original order canceled')
           }
-          this.previous_orders_canceled = previous_orders_canceled
+          this.previous_orders_canceled = all_canceled
         }
       }
       let rate_major = _.get(this.rates_combined_major, [pair])
@@ -1053,9 +1080,9 @@ module.exports = class Orders {
       if (res.success) {
         _.set(this.open_orders, [exchange, pair], { open_orders: res.body, update_time: new Date() })
         previous_orders_processer(exchange, pair, res.body)
-      } else if (_.get(this.open_orders, [exchange, pair, 'update_time']) < new Date().getTime() - this.interval_dict[exchange].open_orders) {
+      } else if (_.get(this.open_orders, [exchange, pair, 'update_time']) < Date.now() - this.interval_dict[exchange].open_orders) {
         console.log('%s %s open orders expired: %j', exchange, pair, res)
-        this.open_orders[exchange] = _.omit(this.open_orders[exchange], [pair])
+        if (this.open_orders[exchange]) delete this.open_orders[exchange][pair]
       } else if (res.error !== 'opening') {
         console.error('%s %s open orders: %j', exchange, pair, res)
       }
@@ -1070,12 +1097,12 @@ module.exports = class Orders {
       } else {
         let error_logged = false
         for (let pair of this.exchange_pairs[exchange]) {
-          if (_.get(this.open_orders, [exchange, pair, 'update_time']) < new Date().getTime() - this.interval_dict[exchange].open_orders) {
+          if (_.get(this.open_orders, [exchange, pair, 'update_time']) < Date.now() - this.interval_dict[exchange].open_orders) {
             if (!error_logged) {
               console.log('%s all open orders expired: %j', exchange, res)
               error_logged = true
             }
-            this.open_orders[exchange] = _.omit(this.open_orders[exchange], [pair])
+            if (this.open_orders[exchange]) delete this.open_orders[exchange][pair]
           } else if (res.error !== 'opening' && !error_logged) {
             console.error('%s all open orders: %j', exchange, res)
             error_logged = true
@@ -1118,11 +1145,6 @@ module.exports = class Orders {
         })
       }
       for (let exchange in Exchanges) {
-        if (Exchanges[exchange].exchange_type === 'dex') {
-          for (let pair of exchange_pairs[exchange]) {
-            _.set(this.previous_orders_canceled_exchange_pair, [exchange, pair], true)
-          }
-        }
         if (Exchanges[exchange].get_all_open_orders_ws) {
           if (Exchanges[exchange].ws_status().trading === 'n-opened') Exchanges[exchange].ws_account()
           get_all_open_orders_ws(exchange)
@@ -1151,13 +1173,13 @@ module.exports = class Orders {
           this.trades_amount_past_hour,
           [exchange, pair],
           _.sumBy(
-            res.body.filter((t) => t.close_time > new Date().getTime() - ONE_HOUR_IN_MS || t.open_time > new Date().getTime() - ONE_HOUR_IN_MS),
+            res.body.filter((t) => t.close_time > Date.now() - ONE_HOUR_IN_MS || t.open_time > Date.now() - ONE_HOUR_IN_MS),
             (t) => t.amount,
           ) || 0,
         )
-      } else if (_.get(this.trades, [exchange, pair, 'update_time']) < new Date().getTime() - this.interval_dict[exchange].trades) {
+      } else if (_.get(this.trades, [exchange, pair, 'update_time']) < Date.now() - this.interval_dict[exchange].trades) {
         console.log('%s %s trades expired: %j', exchange, pair, res)
-        this.trades[exchange] = _.omit(this.trades[exchange], [pair])
+        if (this.trades[exchange]) delete this.trades[exchange][pair]
       } else if (res.error !== 'opening') {
         console.error('%s %s trades: %j', exchange, pair, res)
       }
@@ -1171,7 +1193,7 @@ module.exports = class Orders {
             this.trades_amount_past_hour,
             [exchange, pair],
             _.sumBy(
-              trades.filter((t) => t.close_time > new Date().getTime() - ONE_HOUR_IN_MS || t.open_time > new Date().getTime() - ONE_HOUR_IN_MS),
+              trades.filter((t) => t.close_time > Date.now() - ONE_HOUR_IN_MS || t.open_time > Date.now() - ONE_HOUR_IN_MS),
               (t) => t.amount,
             ) || 0,
           )
@@ -1179,12 +1201,12 @@ module.exports = class Orders {
       } else if (res.error !== 'opening') {
         let error_logged = false
         for (let pair of this.exchange_pairs[exchange]) {
-          if (_.get(this.trades, [exchange, pair, 'update_time']) < new Date().getTime() - this.interval_dict[exchange].trades) {
+          if (_.get(this.trades, [exchange, pair, 'update_time']) < Date.now() - this.interval_dict[exchange].trades) {
             if (!error_logged) {
               console.log('%s all trades expired: %j', exchange, res)
               error_logged = true
             }
-            this.trades[exchange] = _.omit(this.trades[exchange], [pair])
+            if (this.trades[exchange]) delete this.trades[exchange][pair]
           } else if (res.error !== 'opening' && !error_logged) {
             console.error('%s all trades: %j', exchange, res)
             error_logged = true
@@ -1283,9 +1305,9 @@ module.exports = class Orders {
     let query_order_handler = (exchange, pair, res) => {
       if (res.success) {
         _.set(this.query_order, [exchange, pair], { query_order: res.body, update_time: new Date() })
-      } else if (_.get(this.query_order, [exchange, pair, 'update_time']) < new Date().getTime() - this.interval_dict[exchange].query_order) {
+      } else if (_.get(this.query_order, [exchange, pair, 'update_time']) < Date.now() - this.interval_dict[exchange].query_order) {
         console.log('%s %s query_order expired: %j', exchange, pair, res)
-        this.query_order[exchange] = _.omit(this.query_order[exchange], [pair])
+        if (this.query_order[exchange]) delete this.query_order[exchange][pair]
       } else if (res.error !== 'opening') {
         console.error('%s %s query_order: %j', exchange, pair, res)
       }
@@ -1341,7 +1363,7 @@ module.exports = class Orders {
         let exchanges_list = _.concat(this.pair_exchanges[pair] || [], this.pair_exchanges_rates_only[pair] || [])
         for (let exchange of exchanges_list) {
           if (_.get(this.rates, [exchange, pair, 'asks'], []).length > 0 && _.get(this.rates, [exchange, pair, 'bids'], []).length > 0) {
-            const rates = _.cloneDeep(this.rates[exchange][pair])
+            const rates = this.rates[exchange][pair]
 
             let major_rates_only_exchanges_length = _.get(this.major_exchanges, [pair], []).length + _.get(this.rates_only_exchanges, [pair], []).length
             if (this.major_exchanges[pair] && _.includes(this.major_exchanges[pair], exchange)) {
@@ -1356,8 +1378,8 @@ module.exports = class Orders {
               }
             }
           } else {
-            orderbook_major = _.omit(orderbook_major, [exchange])
-            orderbook_major_temp = _.omit(orderbook_major_temp, [exchange])
+            delete orderbook_major[exchange]
+            delete orderbook_major_temp[exchange]
           }
         }
 
@@ -1365,7 +1387,7 @@ module.exports = class Orders {
         if (o_3.asks.length > 0 && o_3.bids.length > 0) {
           _.set(this.rates_combined_major, [pair], o_3)
         } else {
-          this.rates_combined_major = _.omit(this.rates_combined_major, [pair])
+          delete this.rates_combined_major[pair]
         }
 
         const reference_pair = this.pair_map_rates_only[pair]
@@ -1376,14 +1398,9 @@ module.exports = class Orders {
       this.rates_observable.notify()
     }
     const throttled_process_rates = _.throttle(process_rates, PROCESS_RATES_THROTTLE_MS, { leading: true, trailing: true })
-    let rates_handler = (exchange, pair, res, rates_with_routes = false) => {
+    let rates_handler = (exchange, pair, res) => {
       if (res.success) {
         let { asks, bids } = res.body
-        if (rates_with_routes) {
-          _.set(this.rates_with_routes, [exchange, pair], { asks, bids, update_time: new Date() })
-          asks = asks.map((ask) => ask.slice(0, 2))
-          bids = bids.map((bid) => bid.slice(0, 2))
-        }
         asks = asks.slice(0, RATES_LENGTH_MAX)
         bids = bids.slice(0, RATES_LENGTH_MAX)
         _.set(this.rates, [exchange, pair], { asks, bids, update_time: new Date() })
@@ -1391,9 +1408,9 @@ module.exports = class Orders {
          * throttle to 20ms
          */
         throttled_process_rates()
-      } else if (_.get(this.rates, [exchange, pair, 'update_time']) < new Date().getTime() - this.interval_dict[exchange].rates) {
+      } else if (_.get(this.rates, [exchange, pair, 'update_time']) < Date.now() - this.interval_dict[exchange].rates) {
         console.log('%s %s rates expired: %j', exchange, pair, res)
-        this.rates[exchange] = _.omit(this.rates[exchange], [pair])
+        if (this.rates[exchange]) delete this.rates[exchange][pair]
         this.rates_observable.notify()
       }
     }
@@ -1412,19 +1429,6 @@ module.exports = class Orders {
               rates_handler(exchange, pair, res)
             })
           }
-        } else if (Exchanges[exchange].ws_market && Exchanges[exchange].exchange_type === 'dex') {
-          setInterval(() => {
-            if (!this.ws_market_first_time[exchange]) {
-              Exchanges[exchange].ws_market(exchange_pairs[exchange], this.ws_market_options)
-              this.ws_market_first_time[exchange] = true
-            } else {
-              for (let pair of exchange_pairs[exchange]) {
-                Exchanges[exchange].rate_with_routes_ws(pair, (res) => {
-                  rates_handler(exchange, pair, res, true)
-                })
-              }
-            }
-          }, interval_dict[exchange].rates)
         } else if (Exchanges[exchange].exchange_type === 'spot') {
           for (let pair of exchange_pairs[exchange]) {
             setInterval(() => {
@@ -1433,14 +1437,6 @@ module.exports = class Orders {
               })
             }, interval_dict[exchange].rates)
           }
-        } else if (Exchanges[exchange].exchange_type === 'dex') {
-          for (let pair of exchange_pairs[exchange]) {
-            setInterval(() => {
-              Exchanges[exchange].rate_with_routes(pair, (res) => {
-                rates_handler(exchange, pair, res, true)
-              })
-            }, interval_dict[exchange].rates)
-          }
         }
       }
       for (let exchange in exchange_pairs_rates_only) {
@@ -1471,9 +1467,9 @@ module.exports = class Orders {
     let market_history_handler = (exchange, pair, res) => {
       if (res.success) {
         _.set(this.market_history, [exchange, pair], { data: res.body, update_time: new Date() })
-      } else if (_.get(this.market_history, [exchange, pair, 'update_time']) < new Date().getTime() - this.interval_dict[exchange].market_history) {
+      } else if (_.get(this.market_history, [exchange, pair, 'update_time']) < Date.now() - this.interval_dict[exchange].market_history) {
         console.log('%s %s market_history expired: %j', exchange, pair, res)
-        this.market_history[exchange] = _.omit(this.market_history[exchange], [pair])
+        if (this.market_history[exchange]) delete this.market_history[exchange][pair]
       }
     }
     let get_market_history = (Exchanges, exchange_pairs, interval_dict) => {
@@ -1508,13 +1504,13 @@ module.exports = class Orders {
       if (res.success) {
         if (res.body.length < OHLCV_DEFAULT_PAST_HOURS) {
           console.error('ohlcv length should be:', OHLCV_DEFAULT_PAST_HOURS, exchange, pair, res.body.length, _.first(res.body), _.last(res.body))
-          this.ohlcv[exchange] = _.omit(this.ohlcv[exchange], [pair])
+          if (this.ohlcv[exchange]) delete this.ohlcv[exchange][pair]
         } else {
           _.set(this.ohlcv, [exchange, pair], { data: res.body, update_time: new Date() })
         }
-      } else if (_.get(this.ohlcv, [exchange, pair, 'update_time']) < new Date().getTime() - this.interval_dict[exchange].ohlcv) {
+      } else if (_.get(this.ohlcv, [exchange, pair, 'update_time']) < Date.now() - this.interval_dict[exchange].ohlcv) {
         console.log('%s %s ohlcv expired: %j', exchange, pair, res)
-        this.ohlcv[exchange] = _.omit(this.ohlcv[exchange], [pair])
+        if (this.ohlcv[exchange]) delete this.ohlcv[exchange][pair]
       }
     }
     let get_ohlcv = (Exchanges, exchange_pairs, interval_dict) => {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@icgio/icg-exchanges-wrapper",
-  "version": "1.17.14",
+  "version": "1.17.15",
   "description": "",
   "main": "index.js",
   "scripts": {
@@ -18,7 +18,7 @@
   "homepage": "https://github.com/icgio/icg-exchanges-wrapper#readme",
   "dependencies": {
     "@icgio/icg-exchanges": "^1.40.29",
-    "@icgio/icg-utils": "^1.9.53",
+    "@icgio/icg-utils": "^1.9.55",
     "lodash": "^4.17.23",
     "pg": "^8.17.2"
   }