@icgio/icg-exchanges-wrapper 1.17.15 → 1.17.17

package/middleware/orders.js

@@ -461,6 +461,12 @@ module.exports = class Orders {
     }
     return stats
   }
+  has_orders_by_type(order_type, exchange, pair, side) {
+    const bucket = this.order_info_by_type[order_type]?.[exchange]?.[pair]?.[side]
+    if (!bucket) return false
+    for (const _ in bucket) return true
+    return false
+  }
   get_order_info_by_order_type(order_type) {
     const types = Array.isArray(order_type) ? order_type : [order_type]
     const result = {}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@icgio/icg-exchanges-wrapper",
-  "version": "1.17.15",
+  "version": "1.17.17",
   "description": "",
   "main": "index.js",
   "scripts": {
@@ -17,8 +17,8 @@
   },
   "homepage": "https://github.com/icgio/icg-exchanges-wrapper#readme",
   "dependencies": {
-    "@icgio/icg-exchanges": "^1.40.29",
-    "@icgio/icg-utils": "^1.9.55",
+    "@icgio/icg-exchanges": "^1.40.30",
+    "@icgio/icg-utils": "^1.9.56",
     "lodash": "^4.17.23",
     "pg": "^8.17.2"
   }

package/docs/2026-04-09-perf-cleanup.md

@@ -1,366 +0,0 @@
-# Performance cleanup — 2026-04-09
-
-Session of performance and dead-code cleanup work spanning three repos:
-
-- `icg-utils` — `combine_orderbooks` rewrite + new tests
-- `icg-exchanges-wrapper` — `orders.js` hot-path optimization + dead-code removal
-- `icg-market-maker` — small follow-up to drop `o.trace` references
-
-Nothing in this document has been published or deployed. Versions, dependents,
-and MM client servers still need updating per the deployment notes at the
-bottom.
-
-## Why
-
-The maker hot path (rate update → `process_rates` → `notify()` → maker
-callback → `handle_maker_*`) was the bottleneck under analysis. Several
-contributing costs were identified and addressed. Expected aggregate effect:
-the per-rate-tick cost on the maker hot path should drop from the
-**multi-millisecond range to sub-millisecond** for typical configurations
-(5 exchanges × 3 pairs × ~100 orders).
-
-## Changes
-
-### icg-utils
-
-#### Rewrote `combine_orderbooks` (2026-04-09)
-
-`lib/utils.js`
-
-The original implementation:
-- Used `_.get`, `_.sum`, `_.values`, `_.keys` inside the inner loop
-- Mutated input tuples in place by pushing the exchange name (`tuple[2]`)
-  onto each level — forcing callers to `cloneDeep` upstream
-- O(R × K) with high lodash constant factors
-
-The rewrite:
-- Strips all lodash from the inner loop
-- Uses parallel arrays (`lists`, `indexes`, `exchanges`) keyed by an integer
-  index, hoists the asks/bids direction comparator out of the loop
-- **Non-mutating**: `orderbooks_dict` and its inner arrays/tuples are never
-  touched. Callers do NOT need to deep-clone before calling.
-- **Pushes input tuple references directly into the result** (Flavor 2 — see
-  Reference Sharing Contract below). Zero allocations in the inner loop.
-- Drops the `tuple[2]` augmentation entirely. The previous algorithm needed
-  the exchange name on each tuple to know which pointer to advance; the
-  rewrite tracks this via `bestIdx` (an integer index into the parallel
-  arrays).
-
-Expected speedup vs the original: ~10× per call. Combined with the upstream
-`cloneDeep` removal, `process_rates` per call drops from ~3-5 ms to ~0.2-0.4
-ms for a typical 5-exchange / 3-pair MM.
-
-##### Reference Sharing Contract
-
-The returned `{asks, bids}` arrays contain references to the SAME tuple
-objects that live inside the caller's input. **Callers MUST treat both the
-input and the result as read-only.** Mutating either will corrupt the other.
-
-This is safe in `icg-exchanges-wrapper`'s `process_rates` flow because:
-
-1. WS rate updates REPLACE the per-pair object via
-   `_.set(this.rates, [exchange, pair], { asks, bids, ... })` rather than
-   mutating in place — old tuple references simply become garbage when no
-   one holds them.
-2. The combined-rates consumer (the MM `rates_observable` subscriber) runs
-   synchronously inside `notify()` in the same call stack, so no other WS
-   handler can interleave and rebuild rates underneath us.
-
-If you ever need result tuples that are independent of the input (e.g. to
-pass to a mutating consumer like `vol_position_combined`, which expects a
-`[price, amount, exchange]` 3-tuple), wrap them at the call site — do not
-put that contract back into `combine_orderbooks`.
-
-#### Added `test/combine-orderbooks.test.js` (2026-04-09)
-
-`test/combine-orderbooks.test.js`
-
-13 tests covering:
-
-- Basic merge ordering (asks ascend, bids descend, length limit honoured)
-- Result tuples remain `[price, amount]` (no `[2]` augmentation)
-- Input is byte-equal to its pre-call snapshot (non-mutation contract)
-- Idempotent on repeated calls
-- Result tuples `===` input tuples (Flavor 2 reference aliasing)
-- Empty dict, empty side per exchange, single exchange, length larger than
-  available, no length limit
-- 3-way interleaving correctness
-- Tied prices are deterministic
-- Fractional prices
-
-Wired into `npm test` via `package.json` (now runs both `rate-limiter.test.js`
-and `combine-orderbooks.test.js`).
-
-### icg-exchanges-wrapper
-
-All changes in `middleware/orders.js` unless noted otherwise.
-
-#### Dropped `_.cloneDeep` in `process_rates` (2026-04-09)
-
-The deep clone existed only because the old `combine_orderbooks` mutated its
-input. With the rewrite the input is never touched, so `process_rates` now
-passes the live `this.rates[exchange][pair]` reference directly. Saves
-~600 µs per `process_rates` call for a typical 5-exchange / 3-pair / 20-level
-MM.
-
-#### Removed dead DEX code (2026-04-09)
-
-`rates_with_routes` and the entire DEX trading path were referenced from six
-call sites in `orders.js` and one in
-`functions/get-exchanges-rates-market-history-ohlcv.js`, but no exchange in
-`icg-exchanges` ever set `exchange_type === 'dex'` and no exchange implemented
-`rate_with_routes` / `rate_with_routes_ws` / `limit_trade_with_routes`. The
-branches were unreachable in production.
-
-Removed:
-
-- `this.rates_with_routes = {}` initialization
-- `limit_trade`'s DEX branch (~30 lines including closest-level / route lookup)
-- `if (rates_with_routes)` block in `rates_handler`, plus the
-  `rates_with_routes = false` parameter
-- Both DEX branches in `get_rates` (with and without `ws_market`)
-- DEX startup branch that pre-set `previous_orders_canceled_exchange_pair`
-- The `else if (Exchange.rate_with_routes)` REST fallback in
-  `functions/get-exchanges-rates-market-history-ohlcv.js`
-
-Net: ~74 lines of dead code removed.
-
-#### Removed `update_order_info` trace block (2026-04-09)
-
-`update_order_info` previously appended to a per-order `trace` array on every
-update via `[...existing_trace, trace_entry]`. This was:
-
-- O(N) quadratic per update — the spread copies the entire existing trace
-- Unbounded memory growth per long-lived order
-- Sent over WebSocket on every update, growing the WS payload over time
-
-The trace had two consumers:
-
-1. **`maker.js`** read `_.last(o.trace).ts` to filter stale CANCELING orders
-   in `maker_2` and `maker_3`. Replaced with a new `last_update_ts` field set
-   via native assignment in `update_order_info`.
-2. **The dashboard "Trace Timeline" UI** showed the full status history for
-   any clicked order. This is real value, but the wrapper is the wrong place
-   to materialize it. Plan: rebuild it on the dashboard backend from the
-   `ws_client.send_order` events that already arrive there. See
-   "Future work" below.
-
-The wrapper now does zero trace work. The dashboard "Trace Timeline" will
-temporarily show "No trace available for this order" until the backend
-update ships.
-
-#### Added status / order_type indexes (2026-04-09)
-
-Two new indexes maintained incrementally by `update_order_info` and
-`delete_order_info`:
-
-- `this.order_status_index[exchange][pair][type][status] -> Set<order_ref>`
-  Used by `get_order_info_status_all`.
-- `this.order_info_by_type[order_type][exchange][pair][type][order_id] -> ref`
-  Used by `get_order_info_by_order_type`.
-
-Maintenance happens at the end of `update_order_info` after the dict fields
-are written. The function snapshots the old `status` / `order_type` BEFORE
-the for-loop, then compares to the new values after, and moves the order
-between buckets only if the relevant field actually changed. The common
-case (a non-status update like `f_amount` or `last_update_ts`) is a no-op.
-
-`delete_order_info` removes the order from both indexes using the order's
-current `status` / `order_type` BEFORE deleting it from the canonical
-`this.order_info`.
-
-`get_order_info_status_all` was rewritten to look up the precomputed Set
-and return `Array.from(...)`. Handles both single-status and array-status
-queries, plus the `'both'` recursive case.
-
-`get_order_info_by_order_type` walks only the matching `order_type`
-subtree(s) and returns a freshly built top-level dict (so callers can safely
-mutate the returned structure) with leaves that are still refs into
-`this.order_info`.
-
-##### Indexing invariants
-
-- Every order in `this.order_info[exchange][pair][type][order_id]` has at
-  most one entry in `order_status_index[exchange][pair][type][status]` and
-  at most one entry in
-  `order_info_by_type[order_type][exchange][pair][type][order_id]`.
-- The only mutators of `this.order_info` are `update_order_info` and
-  `delete_order_info`. Both maintain the indexes. Anyone who writes directly
-  to `this.order_info` outside these two methods will desynchronize the
-  indexes.
-- The previous `this.order_info_status` cache (used by the old
-  `get_order_info_status_all`) is removed.
-- `Account.order_type === 'range-order'` mutations at `:1006` and `:1011`
-  occur on `res.body` open-orders entries from the exchange WS, NOT on
-  `this.order_info` orders, so they don't affect the indexes.
-
-##### Functional verification
-
-A standalone in-process test exercises the indexed code paths against a
-stub instance. 11 assertions cover:
-
-1. Insert maker_1 sell INIT → in INIT bucket, in maker_1 subtree
-2. Transition INIT → N-FILLED → moves buckets, stays in maker_1
-3. `'both'` query → unions sell + buy
-4. Array status query → unions multiple status buckets
-5. by_order_type with array → merges multiple subtrees
-6. `delete_order_info` → removed from BOTH indexes + canonical store
-7. Multiple status transitions → bucket moves are atomic
-8. `last_update_ts` is stamped (numeric, > 0)
-9. Empty queries return `[]` / `{}` (no crash)
-10. Result arrays from `get_order_info_status_all` are independent (fresh
-    array per call) but inner orders are the same refs
-11. Mutating the dict returned by `get_order_info_by_order_type` does not
-    affect the canonical store
-
-#### Fixed `check_balances` shadow bug (2026-04-09)
-
-The original inner loop used `for (let exchange in this.exchange_cur_quote_cur)`
-which **shadowed** the outer `exchange` parameter. Every balance update
-rebuilt `buy_sell_limits_against_reserve_balance` for ALL exchanges, doing
-K× the necessary work where K = exchange count. The fix scopes the inner
-loop to the single exchange whose balance just arrived. Also collapsed the
-empty `if (max > 0) { } else { max = 0 }` blocks to `if (max < 0) max = 0`.
-
-#### Early-exit on `previous_orders_canceled` recompute (2026-04-09)
-
-The recompute walked every `(exchange, pair)` cell every time. Now uses a
-labeled `break outer` on the first unmarked cell. Steady-state behavior
-unchanged (the outer guard `if (!this.previous_orders_canceled)` still
-prevents re-runs once the flag is true), but startup is faster.
-
-#### `_.omit(obj, [key])` → `delete obj[key]` (2026-04-09)
-
-16 sites. `_.omit` allocates a new object copying every other key — O(N)
-plus GC pressure. `delete` is O(1) and free.
-
-Sites:
-
-- `delete_order_info` (`this.order_info[exchange][pair][type]`)
-- `check_balances` expiry (`this.balances`,
-  `this.buy_sell_limits_against_reserve_balance`)
-- `check_positions` expiry (`this.positions`)
-- `open_orders_handler` and `all_open_orders_handler` expiry
-  (`this.open_orders[exchange]`)
-- `trades_handler` and `all_trades_handler` expiry (`this.trades[exchange]`)
-- `query_order_handler` expiry (`this.query_order[exchange]`)
-- `process_rates` local-variable cleanups (`orderbook_major`,
-  `orderbook_major_temp`) and `this.rates_combined_major`
-- `rates_handler` expiry (`this.rates[exchange]`)
-- `market_history_handler` expiry (`this.market_history[exchange]`)
-- `ohlcv_handler` short-data and expiry (`this.ohlcv[exchange]`)
-
-#### `new Date().getTime()` → `Date.now()` (2026-04-09)
-
-13 sites. `new Date().getTime()` allocates a Date object just to read its
-timestamp; `Date.now()` is an intrinsic with no allocation. Pure micro-win.
-
-### icg-market-maker
-
-#### Replaced `_.last(o.trace).ts` filters with `o.last_update_ts` (2026-04-09)
-
-`scripts/maker.js`
-
-Two filters in `maker_2` and `maker_3` (for stale CANCELING orders) replaced
-with the new `o.last_update_ts` field added to `orders.js` in this session.
-No semantic change.
-
-## Files touched
-
-| Repo | File | Note |
-|---|---|---|
-| `icg-utils` | `lib/utils.js` | `combine_orderbooks` rewrite |
-| `icg-utils` | `test/combine-orderbooks.test.js` | new file, 13 tests |
-| `icg-utils` | `package.json` | `test` script runs both files |
-| `icg-exchanges-wrapper` | `middleware/orders.js` | (a) drop `_.cloneDeep`, (b) remove DEX dead code, (c) remove trace block + add `last_update_ts`, (d) status / order_type indexes, (e) rewrite `get_order_info_status_all` and `get_order_info_by_order_type`, (f) `check_balances` shadow + scope, (g) early-exit `previous_orders_canceled` recompute, (h) `_.omit` → `delete` (16 sites), (i) `new Date().getTime()` → `Date.now()` (13 sites) |
-| `icg-exchanges-wrapper` | `functions/get-exchanges-rates-market-history-ohlcv.js` | remove DEX REST fallback |
-| `icg-market-maker` | `scripts/maker.js` | two `last_trace` filters → `o.last_update_ts` |
-
-## Net effect
-
-Hot path per rate tick (5 exchanges × 3 pairs × ~100 orders):
-
-| Operation | Before | After |
-|---|---|---|
-| `process_rates` per pair | ~1-2 ms | ~0.2-0.4 ms |
-| `combine_orderbooks` per call | ~500-1000 µs | ~50-100 µs |
-| `get_order_info_status_all` × ~270 calls/tick | walks all matching orders for that (exchange, pair, type), lodash-heavy | Set lookup + `Array.from`, O(matching) |
-| `get_order_info_by_order_type` × ~15 calls/tick | 4-deep scan of all `order_info`, lodash-heavy | walks only matching subtree |
-| `update_order_info` trace cost | O(N) spread per update | 0 |
-| `check_balances` per balance event | K× redundant rebuild | 1× rebuild scoped to the right exchange |
-| `previous_orders_canceled` recompute | always full walk while flag is false | early-exit on first miss |
-| `_.omit` × 16 sites | O(N) new-object allocation | O(1) `delete`, no allocation |
-| `Date` instances per check | 13 per tick | 0 |
-
-## Future work
-
-### Backend trace materialization
-
-Rebuild the "Trace Timeline" data on
-`clients-dashboard-backend/src/services/ws-data-service.js` from the
-`ws_client.send_order` events it already receives. About 12 lines added in
-`handle_order_event`. The wrapper does no trace work; the backend caps the
-trace at last 20 entries; the dashboard UI is unchanged. Spans
-`clients-dashboard-backend` only — neither the wrapper nor the maker need to
-ship for this.
-
-The required pattern in `handle_order_event`:
-
-```js
-if (action === 'update') {
-  const existing = _.get(this.order_info[client_id], [exchange, pair, side, order_id])
-  const trace = (existing && existing.trace) || []
-  trace.push({
-    ts: Date.now(),
-    status: data.status,
-    note,
-    f_amount: data.f_amount,
-  })
-  if (trace.length > TRACE_MAX_ENTRIES) {
-    trace.splice(0, trace.length - TRACE_MAX_ENTRIES)
-  }
-  _.setWith(
-    this.order_info[client_id],
-    `${exchange}.${pair}.${side}.${order_id}`,
-    { ...data, trace, ts: Date.now() },
-    Object,
-  )
-  // ... rest unchanged
-}
-```
-
-Plus a small change to `handle_order_sync` to preserve existing traces
-across full re-sync (otherwise reconnects wipe history).
-
-### Other items not done in this session
-
-- **`Observable.notify()` try/catch** in `icg-utils/lib/observable.js`. Today
-  there's only one subscriber so a throw is just a crash, but the moment a
-  second subscriber is added, a throw in #1 silently kills #2. Two-line fix.
-- **Remaining lodash hot-path cleanup**: `_.set` / `_.setWith` / `_.get` in
-  `update_order_info` and `rates_handler` could be replaced with native
-  indexing for ~5-20× speedup on those calls.
-- **Empty `cancel_order_by_order` callbacks** at `orders.js:1038`, `:1045`:
-  failures silently ignored. Robustness, not perf.
-- **Hot-path `console.log('UPDATE %s: ...')` formatting** at `orders.js:862`:
-  runs on every `update_order_info` call. Should be gated with a verbosity
-  flag or routed through `log_limiter` (like maker.js does).
-- **`process_open_orders` nested scans** at `orders.js:618-766`. Same
-  indexing approach as the status / order_type indexes would help here too,
-  but it's off the hot path so lower priority.
-
-## Deployment notes
-
-When this work ships:
-
-1. `icg-utils` → bump version → publish to npm (new `combine_orderbooks`).
-2. `icg-exchanges-wrapper` → bump `@icgio/icg-utils` dep range → `npm install`
-   → bump pkg version → publish.
-3. **MM client servers** → manual deploy across all 12. Per
-   `feedback_mm_servers_dirty_lockfile.md`, pre-stash any dirty
-   `package.json` / `package-lock.json` on each server before
-   `git pull && npm ci`. The deploy carries both the wrapper republish and
-   the `maker.js` change (they both touch the maker process).
-4. The dashboard "Trace Timeline" will show "No trace available" until the
-   `clients-dashboard-backend` update ships separately. Plan that as a
-   follow-up.