dexbot 1.4.16 → 1.4.17

package/docs/architecture.md

@@ -0,0 +1,1466 @@
+# DEXBot2 Architecture
+
+This document provides a high-level overview of the DEXBot2 architecture, module relationships, and key data flows.
+
+> **For practical development guidance**, see [developer_guide.md](developer_guide.md) for quick start, glossary, module deep dive, and common development tasks.
+
+---
+
+## System Overview
+
+DEXBot2 is a grid trading bot for the BitShares blockchain. It maintains a geometric grid of limit orders that automatically rebalance as the market moves, capturing profit from price oscillations.
+
+### Core Concepts
+
+- **Grid**: A geometric array of price levels with orders placed at each level
+- **Spread Zone**: A buffer of empty slots between buy and sell orders (constant width)
+- **Order States**: VIRTUAL (planned) → ACTIVE (on-chain) → PARTIAL (partially filled)
+- **Fund Tracking**: Atomic accounting system preventing race conditions and overdrafts
+
+### Design Philosophy
+
+DEXBot2 prioritizes **simplicity and operational efficiency** over complex partial-handling mechanics:
+
+1. **Constant Spread**: The spread zone width remains fixed at `targetSpreadPercent`, eliminating dynamic inflation triggers.
+2. **Direct Consolidation**: Dust partials are absorbed into the next grid rebuild cycle, not handled by complex merge/split logic.
+3. **Minimal Blockchain Interaction**: Fund-driven rebalancing occurs once per fill batch, not per-partial. Grid generation uses only available funds—no forced allocations.
+4. **Closed-Loop Market Dynamics**: The boundary-crawl mechanism naturally handles price movement and fill flows without special-case logic.
+5. **Powerful Maintenance Tools**: Periodic grid regeneration, recovery retries, and fund invariant verification keep the system healthy over long operations.
+
+---
+
+## Top-Level Data Flow
+
+The diagram below shows DEXBot2 from a **data perspective**: what data enters the system, how it moves through each engine, and what leaves as blockchain operations or persisted state.
+
+```mermaid
+graph TB
+    subgraph "INPUTS"
+        CFG[bots.json<br/>grid params, funds, pair]
+        GS[general.settings.json<br/>timing, thresholds]
+        KEYS[keys.json AES encrypted<br/>+ interactive unlock / one-shot local bootstrap]
+        PERSIST[orders/botKey.json<br/>grid snapshot,<br/>feesOwed, boundaryIdx]
+        FILLEV[Fill Events real-time<br/>BitShares block op-4]
+        OPENORD[Open Orders polling<br/>chain open-order list]
+        BALANCES[Account Balances<br/>FREE + COMMITTED assets]
+        PRICE[Market Price<br/>pool or order-book]
+    end
+
+    subgraph "BOOTSTRAP - once at startup"
+        AUTH[Credential Daemon<br/>Decrypt private key]
+        ASSETMETA[Asset Metadata<br/>precision, fees, IDs]
+        INITGRID[Initial Grid<br/>geometric price levels<br/>order sizes per side]
+    end
+
+    subgraph "CORE ENGINE - OrderManager"
+        MASTERGRID[Master Grid - immutable/frozen<br/>slot-id, price, size, state<br/>orderId, blockchain, grid, proceeds]
+        TWOPASS[SyncEngine<br/>2-pass: grid-to-chain then chain-to-grid<br/>match orderId, detect partials, flag stale]
+        FUNDS["Accounting - SSOT for funds<br/>available, virtual, committed<br/>btsFeesOwed<br/>Avail = max 0 ChainFree minus Virtual minus Fees"]
+        TARGET[Strategy Engine<br/>calculateTargetGrid<br/>boundary-crawl pivot<br/>partial-fill consolidation, rotation]
+        WORKGRID[WorkingGrid - COW copy<br/>all mutations here only<br/>commit to Master on confirmation]
+        FILLQUEUE[Fill Queue<br/>AsyncLock + dedup 5-60 min]
+        BATCHER[Fixed-Cap Batcher<br/>queue within cap: unified batch<br/>queue above cap: chunk at cap size<br/>default cap: 4]
+    end
+
+    subgraph "OUTPUTS"
+        OPS[Blockchain Operations<br/>CREATE / UPDATE / CANCEL<br/>limit orders on BitShares]
+        SNAP[Grid Snapshot<br/>profiles/orders/botKey.json]
+        LOGS[Logs and Metrics<br/>profiles/logs/botName.log<br/>queue depth, latency, health]
+    end
+
+    KEYS --> AUTH --> ASSETMETA
+    CFG --> INITGRID
+    GS --> INITGRID
+    ASSETMETA --> INITGRID
+    PERSIST --> MASTERGRID
+    INITGRID --> MASTERGRID
+
+    PRICE --> FUNDS
+    BALANCES --> FUNDS
+    BALANCES --> TWOPASS
+    OPENORD --> TWOPASS
+    TWOPASS --> FUNDS
+    TWOPASS --> MASTERGRID
+
+    FILLEV --> FILLQUEUE
+    OPENORD --> FILLQUEUE
+    FILLQUEUE --> BATCHER
+    BATCHER --> WORKGRID
+
+    MASTERGRID --> WORKGRID
+    FUNDS --> TARGET
+    WORKGRID --> TARGET
+    TARGET --> WORKGRID
+    WORKGRID --> OPS
+    OPS --> MASTERGRID
+
+    MASTERGRID --> SNAP
+    FUNDS --> LOGS
+    BATCHER --> LOGS
+    OPS --> LOGS
+```
+
+### Key Design Principles
+
+| Principle | Mechanism |
+|---|---|
+| **Immutability** | Master Grid is frozen; all changes go through a disposable WorkingGrid (Copy-on-Write) |
+| **Single Source of Truth** | Accounting engine owns all fund data; everything reads from it |
+| **Event-driven + Polling** | Fill Events (real-time) and Open-Order polling feed the same queue |
+| **Fixed-Cap Batching** | Deterministic batching with hard cap per broadcast (default 4 fills) |
+| **Persistence** | Grid snapshot written after every confirmed blockchain commit |
+
+---
+
+## Module Architecture
+
+```mermaid
+graph TB
+    subgraph "Entry Points"
+        CLI[dexbot.ts]
+        BOT[bot.ts]
+        PM2[pm2.ts]
+        UNLOCK[unlock.ts]
+        CRED_DAEMON[credential-daemon.ts]
+    end
+
+    subgraph "Core Bot"
+        DEXBOT[DexBotClass<br/>modules/dexbot_class.ts]
+        FILL_RUNTIME[FillRuntime<br/>modules/dexbot_fill_runtime.ts]
+        MAINT_RUNTIME[MaintenanceRuntime<br/>modules/dexbot_maintenance_runtime.ts]
+        CONSTANTS[Constants<br/>modules/constants.ts]
+        FUND_REGISTRY[FundRegistry<br/>modules/fund_registry.ts]
+        SETTINGS_MERGE[SettingsMerge<br/>modules/settings_merge.ts]
+        CRED_RUNTIME[CredentialRuntime<br/>modules/credential_runtime.ts]
+    end
+
+    subgraph "Order Management System"
+        MANAGER[OrderManager<br/>modules/order/manager.ts]
+
+        subgraph "Specialized Engines"
+            ACCOUNTANT[Accountant<br/>accounting.ts]
+            STRATEGY[StrategyEngine<br/>strategy.ts]
+            SYNC[SyncEngine<br/>sync_engine.ts]
+            GRID[Grid<br/>grid.ts]
+        end
+
+        WORKGRID[WorkingGrid<br/>working_grid.ts]
+        GRID_RECONCILE[GridReconcile<br/>grid_reconcile.ts]
+        UTILS[Utils<br/>utils/]
+        LOGGER[Logger<br/>logger.ts]
+        FILL_STORE[ProcessedFillStore<br/>processed_fill_store.ts]
+    end
+
+    subgraph "Blockchain Layer"
+        CHAIN_ORDERS[ChainOrders<br/>modules/chain_orders.ts]
+        ACCOUNT_ORDERS[AccountOrders<br/>modules/account_orders.ts]
+        ACCOUNT_BOTS[AccountBots<br/>modules/account_bots.ts]
+        NODE_MGR[NodeManager<br/>modules/node_manager.ts]
+        BTS_CLIENT[BitSharesClient<br/>modules/bitshares_client.ts]
+    end
+
+    subgraph "Market Adapter"
+        MA[MarketAdapter<br/>market_adapter/market_adapter.ts]
+        MA_SVC[AdapterService<br/>core/market_adapter_service.ts]
+        AMA_RUNNER[AMA Signal Runner<br/>ama_signal_runner.ts]
+        KIBANA[inputs/kibana_source.ts]
+        LP_FETCH[inputs/fetch_lp_data.ts]
+        STRATEGIES[core/strategies/]
+    end
+
+    CLI --> DEXBOT
+    BOT --> DEXBOT
+    PM2 --> DEXBOT
+    UNLOCK --> CRED_DAEMON
+    CRED_DAEMON --> DEXBOT
+
+    DEXBOT --> MANAGER
+    DEXBOT --> FILL_RUNTIME
+    DEXBOT --> MAINT_RUNTIME
+    DEXBOT --> CONSTANTS
+    DEXBOT --> FUND_REGISTRY
+    DEXBOT --> SETTINGS_MERGE
+    DEXBOT --> CRED_RUNTIME
+
+    MANAGER --> ACCOUNTANT
+    MANAGER --> STRATEGY
+    MANAGER --> SYNC
+    MANAGER --> GRID
+    MANAGER --> WORKGRID
+    MANAGER --> GRID_RECONCILE
+    MANAGER --> UTILS
+    MANAGER --> LOGGER
+    MANAGER --> RUNNER
+    MANAGER --> FILL_STORE
+
+    ACCOUNTANT --> FUND_REGISTRY
+    STRATEGY --> UTILS
+    SYNC --> UTILS
+    GRID --> UTILS
+
+    RUNNER --> CHAIN_ORDERS
+    SYNC --> ACCOUNT_ORDERS
+    MANAGER --> ACCOUNT_BOTS
+
+    CHAIN_ORDERS --> BTS_CLIENT
+    ACCOUNT_ORDERS --> BTS_CLIENT
+    BTS_CLIENT --> NODE_MGR
+
+    MA --> MA_SVC
+    MA --> AMA_RUNNER
+    MA_SVC --> KIBANA
+    MA_SVC --> LP_FETCH
+    MA_SVC --> STRATEGIES
+
+    MA -.->|trigger files| MAINT_RUNTIME
+```
+
+## Browser-Safe Surface
+
+DEXBot2 ships both a **Node CLI runtime** and the building blocks for an in-browser operator UI. To keep `require('fs')`, `process.kill`, and Unix-socket calls out of the browser bundle, the codebase enforces a strict Node-vs-browser split.
+
+**Convention**: *everything is browser-safe unless listed as Node-only*. The source of truth is the `"browser"` field in `package.json`, which maps every Node-only **compiled** entry (`dist/*.js`) to `false`. The list below is documentation; always check `package.json` before reclassifying a module.
+
+**Node-only modules** (must not be reached from a browser bundle):
+
+| Module | Reason |
+|--------|--------|
+| `modules/launcher/*` | Credential daemon, bot supervisor, market-adapter runtime, monolithic runtime |
+| `modules/dexbot_maintenance_runtime.ts` | Direct `fs` / `child_process` / `os` use |
+| `modules/dexbot_class.ts` | Imports `dexbot_maintenance_runtime` |
+| `unlock.ts`, `bot.ts`, `dexbot.ts`, `pm2.ts`, `credential-daemon.ts` | CLI entry points |
+| `market_adapter/lp_chart_runner.ts` | `import { exec } from 'node:child_process'` for chart rendering |
+
+**Environment detection** must go through `modules/env.ts` rather than inline `typeof window` / `typeof process` checks:
+
+```javascript
+import { isBrowser, hasProcess } from './env';
+```
+
+The previous 6+ inline ternaries that existed in `bitshares-native/*` and `runtime.ts` were consolidated into those helpers; do not reintroduce them.
+
+---
+
+## Order Manager: Central Coordinator
+
+The `OrderManager` is the central hub that coordinates all order operations. It delegates specialized tasks to four engine modules:
+
+### Engine Responsibilities
+
+| Engine | File | Responsibility |
+|--------|------|----------------|
+| **Accountant** | `accounting.ts` | **Single Source of Truth**. Centralized fund tracking via `recalculateFunds()`, fee management, invariant verification, recovery retry state management (`resetRecoveryState()`) |
+| **StrategyEngine** | `strategy.ts` | Grid rebalancing, order rotation, partial order handling, fill boundary shifts, remainder tracking |
+| **SyncEngine** | `sync_engine.ts` | Blockchain synchronization, fill detection, stale-order cleanup, type-mismatch handling |
+| **Grid** | `grid.ts` | Grid creation, sizing, divergence detection, remainder accuracy during capped resize |
+
+---
+
+## Copy-on-Write (COW) Grid Pattern
+
+The OrderManager implements a **Copy-on-Write (COW) pattern** to protect the master grid from speculative modifications until blockchain finality is confirmed.
+
+### Core Principle
+
+The master grid (`this.orders`) is **immutable** - it can only be replaced atomically, never mutated in place. All speculative planning operations work on isolated copies, and the master is only updated when blockchain confirms the operation.
+
+**Important**: Index Sets (`_ordersByState`, `_ordersByType`) are **mutable by design** but must **only be mutated through `_applyOrderUpdate()`**. Direct external mutations violate the COW invariant.
+
+### Protection Mechanisms
+
+| Mechanism | Location | Purpose |
+|-----------|----------|---------|
+| `Object.freeze()` | `manager.ts` | Master Map is frozen at initialization |
+| `deepFreeze()` | `manager.ts` | Individual order objects are deep-frozen |
+| `_gridVersion` | `manager.ts` | Version counter for staleness detection |
+| `_gridLock` | `manager.ts` | AsyncLock serializes grid mutations |
+| Encapsulation | `manager.ts` | Index Sets are private; mutations only via `_applyOrderUpdate()` |
+
+### Master Grid Update Pattern
+
+All master grid updates follow clone-and-replace semantics:
+
+```javascript
+// 1. Clone existing Map
+const newMap = cloneMap(this.orders);
+
+// 2. Apply mutation to clone
+newMap.set(id, updatedOrder);
+
+// 3. Atomically replace with frozen copy
+this.orders = Object.freeze(newMap);
+this._gridVersion++;
+```
+
+Index Sets follow the same pattern - cloned, mutated, frozen, then replaced.
+
+### WorkingGrid Class
+
+The `WorkingGrid` class (`modules/order/working_grid.ts`) provides isolation for speculative operations:
+
+- **Deep clones** the master grid on construction
+- Tracks **modified orders** in a Set
+- Supports **staleness detection** via `baseVersion`
+- **Never modifies** the master grid
+
+### COW Rebalance Pipeline
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  1. Create WorkingGrid from frozen master                   │
+│     workingGrid = new WorkingGrid(masterGrid, {baseVersion})│
+└─────────────────────────┬───────────────────────────────────┘
+                         ↓
+┌─────────────────────────────────────────────────────────────┐
+│  2. Calculate target state (PURE - no side effects)         │
+│     strategy.calculateTargetGrid() returns new Map          │
+└─────────────────────────┬───────────────────────────────────┘
+                         ↓
+┌─────────────────────────────────────────────────────────────┐
+│  3. Project target onto working grid                        │
+│     Modifies working copy only                              │
+└─────────────────────────┬───────────────────────────────────┘
+                         ↓
+┌─────────────────────────────────────────────────────────────┐
+│  4. Validate funds & check staleness                        │
+│     If stale: abort without committing                      │
+└─────────────────────────┬───────────────────────────────────┘
+                         ↓
+┌─────────────────────────────────────────────────────────────┐
+│  5. Submit to blockchain & wait for finality                │
+│     synchronizeWithChain() confirms on-chain                │
+└─────────────────────────┬───────────────────────────────────┘
+                         ↓
+┌─────────────────────────────────────────────────────────────┐
+│  6. Commit: Replace master with working grid                │
+│     this.orders = Object.freeze(workingGrid.toMap())        │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### Triggers for Master Grid Updates
+
+Only blockchain-confirmed events trigger master updates:
+
+| Event | Entry Point | Mechanism |
+|-------|-------------|-----------|
+| Order Created | `sync_engine.ts` `synchronizeWithChain()` | Called with `source: 'createOrder'` |
+| Order Cancelled | `sync_engine.ts` `synchronizeWithChain()` | Called with `source: 'cancelOrder'` |
+| Order Filled | `sync_engine.ts` `syncFromFillHistory()` | Processes a real-time fill event |
+| Full Sync | `sync_engine.ts` `syncFromOpenOrders()` | Reconciles grid against current open orders |
+| Grid Init/Load | `grid.ts` `createOrderGrid()` | Bootstrap operations |
+
+### Defensive Measures
+
+1. **Double-check commit pattern**: Staleness is checked both outside and inside the lock
+2. **Working grid sync**: If master mutates during planning, working grid is marked stale
+3. **Version mismatch detection**: Commits abort if `baseVersion` doesn't match `_gridVersion`
+
+---
+
+## Fill Processing Pipeline
+
+The fill pipeline handles incoming filled orders efficiently through fixed-cap batching instead of one-at-a-time processing.
+
+### Architecture Diagram
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                   Fill Event (Blockchain)                   │
+│                  (Order filled at price X)                  │
+└─────────────────────┬───────────────────────────────────────┘
+                      ↓
+┌─────────────────────────────────────────────────────────────┐
+│               _incomingFillQueue (FIFO Queue)               │
+│             (Accumulates fills from blockchain)             │
+│       Queue: [fill1, fill2, fill3, fill4, fill5, ...]       │
+└─────────────────────┬───────────────────────────────────────┘
+                      ↓
+┌─────────────────────────────────────────────────────────────┐
+│             processFilledOrders() - Entry Point             │
+│    Use MAX_FILL_BATCH_SIZE cap for deterministic batching   │
+│        Rules: <=cap unified, >cap chunked at cap size       │
+└─────────────────────┬───────────────────────────────────────┘
+                      ↓
+┌─────────────────────────────────────────────────────────────┐
+│            Pop Batch (up to MAX_FILL_BATCH_SIZE)            │
+│           Takes N fills from queue head (N = 1-4)           │
+│   Example: pops [fill1, fill2, fill3] for batch processing  │
+└─────────────────────┬───────────────────────────────────────┘
+                      ↓
+┌─────────────────────────────────────────────────────────────┐
+│            processFillAccounting() - Single Call            │
+│       All fills credited to chainFree in ONE operation      │
+│     chainFree += proceeds[fill1] + proceeds[fill2] + ...    │
+│    Proceeds immediately available (same rebalance cycle)    │
+└─────────────────────┬───────────────────────────────────────┘
+                      ↓
+┌─────────────────────────────────────────────────────────────┐
+│             calculateTargetGrid() - Single Call             │
+│       Size replacement orders using combined proceeds       │
+│             Apply rotations and boundary shifts             │
+│ Use unallocated remainder for next allocation opportunities │
+└─────────────────────┬───────────────────────────────────────┘
+                      ↓
+┌─────────────────────────────────────────────────────────────┐
+│        updateOrdersOnChainBatch() - Single Broadcast        │
+│      All new orders + cancellations in single operation     │
+│          Result: Atomic state update on blockchain          │
+└─────────────────────┬───────────────────────────────────────┘
+                      ↓
+┌─────────────────────────────────────────────────────────────┐
+│                        persistGrid()                        │
+│               Save grid state to disk/storage               │
+└─────────────────────┬───────────────────────────────────────┘
+                      ↓
+                  Loop to next batch
+                (or idle if queue empty)
+```
+
+### Key Properties
+
+- **Fixed-Cap Batch Sizing**: Batch size is deterministic with `MAX_FILL_BATCH_SIZE` (default 4)
+  - 1..4 awaiting: single unified batch (one rebalance/broadcast cycle)
+  - 5+ awaiting: repeated chunks of 4 (last chunk may be smaller)
+
+- **Single Rebalance Cycle**: All fills in batch processed in ONE rebalance
+  - No "split across cycles" delays
+  - Combined proceeds immediately available
+  - Single cache fund update
+
+- **Recovery Retries**: Periodic retry system replaces one-shot flag
+  - Max 5 attempts per episode
+  - 60s minimum interval between retries
+  - Reset on fill arrival or periodic sync (10 minutes)
+  - `resetRecoveryState()` called by Accountant
+
+- **Stale-Cleaned Order Tracking**: Prevents orphan double-credit
+  - Batch failure → cleanup stale order IDs
+  - Delayed orphan event → check if ID in stale-cleaned map
+  - Skip credit if already cleaned
+  - TTL pruning (5 minute retention)
+
+### Impact vs. Legacy Sequential Processing
+
+Scenario source: 29-fill burst during the Feb 7 market crash, modeled at
+roughly 3 seconds per broadcast; see
+[`FUND_MOVEMENT_AND_ACCOUNTING.md`](FUND_MOVEMENT_AND_ACCOUNTING.md#14-fill-batch-processing--cache-fund-timeline).
+
+| Metric | Legacy (1-at-a-time) | Fixed-Cap Batching | Improvement |
+|--------|---------------------|-------------------|-------------|
+| **29 Fills** | ~90 seconds | ~24 seconds | **73% faster** |
+| **Market Divergence** | High (90s window) | Low (24s window) | **Safer** |
+| **Stale Orders** | Frequent | Rare | **More reliable** |
+| **Recovery** | One-shot (brick) | Periodic (self-heal) | **Production-ready** |
+
+---
+
+## Fund-Driven Boundary Sync
+
+The grid boundary (which separates BUY, SPREAD, and SELL zones) automatically aligns with the bot's actual inventory distribution.
+
+### Why This Matters
+
+By default, the grid is centered around `startPrice`. However, if the bot has asymmetric capital (e.g., more assetB than assetA), the boundary should shift to favor the "heavier" side.
+
+**Example**: If 70% of capital is in assetB (buying power), the BUY zone should be expanded.
+
+### Boundary Calculation
+
+**Location**: `modules/dexbot_class.ts::_performPeriodicGridChecks()` → Boundary Sync step
+
+**Algorithm**:
+```javascript
+// 1. Scan all grid slots and their current assignments
+const buyTotal = sum(orders with type === BUY);
+const sellTotal = sum(orders with type === SELL);
+const totalAllocated = buyTotal + sellTotal;
+
+// 2. Calculate target allocation based on available funds
+const buyAvailable = manager.funds.available.buy;
+const sellAvailable = manager.funds.available.sell;
+const totalAvailable = buyAvailable + sellAvailable;
+
+// 3. Determine ideal boundary position
+const buyTargetRatio = buyAvailable / totalAvailable;  // e.g., 0.7
+const slots = grid.length;
+const targetBuySlots = Math.round(slots * buyTargetRatio * 0.5);  // Apply centering factor
+
+// 4. Adjust boundary to new position
+newBoundaryIdx = calculateNewBoundary(targetBuySlots);
+
+// 5. Re-assign slot roles (BUY/SPREAD/SELL) based on new boundary
+reassignSlotRoles(newBoundaryIdx);
+```
+
+### Three Rotation Cases
+
+Once the new boundary is determined, existing on-chain orders are matched to desired slots:
+
+| Case | Condition | Action |
+|------|-----------|--------|
+| **MATCH** | Existing order price matches desired slot | Update size if needed |
+| **ACTIVATE** | Desired slot is empty | Place new order at this price |
+| **DEACTIVATE** | Existing order exceeds target count | Cancel excess orders |
+
+**Target Count**: `activeOrders` from config, applied uniformly to both sides.
+
+### Impact
+
+- **Automatic Capital Repositioning**: Grid follows capital distribution without manual intervention
+- **Fund Respect**: Never exceeds available funds when activating slots
+- **Smooth Transitions**: Rotations happen gradually, not all at once
+
+---
+
+## Spread Correction (Fund-Aware Approach)
+
+Simplified spread maintenance that keeps the gap consistent and fund-driven, avoiding complex split/merge mechanics.
+
+### The Approach
+
+Instead of complex partial handling, spread corrections are **conservative and fund-safe**:
+- Target spread width stays **constant** at `targetSpreadPercent`
+- Corrections scale with **actual available funds**, not arbitrary slot budgets
+- No dynamic spread inflation based on partial consolidation flags
+- Edge-first surplus selection ensures stable rotation candidates
+
+### Algorithm
+
+**Location**: `modules/order/strategy.ts::calculateTargetGrid()`
+
+The simplified approach prioritizes fund availability over aggressive corrections:
+
+```
+1. Detect that spread is wider than targetSpreadPercent
+2. Calculate how many edge slots are missing
+3. Attempt to place new edge orders with available funds
+4. If insufficient funds for all edges:
+   - Create what's affordable with available funds
+   - Log shortfall (smooth over next rebalance cycle)
+5. If a dust partial exists in the correction window:
+   - Mark for consolidation in next grid rebuild
+   - Don't create complex merge/split side effects
+6. Maintain constant target spread—no inflation based on partial flags
+```
+
+### Fund-Safe Constraints
+
+Spread corrections respect these hard limits:
+
+```javascript
+// In modules/constants.ts
+SPREAD_LIMITS: {
+    MIN_SPREAD_ORDERS: 2,           // Always maintain minimum gap
+    TARGET_SPREAD_PERCENT: (user-configured),  // Fixed width, no inflation
+    MAX_REPLACEMENT_SLOTS: 5,       // Conservative correction cap
+}
+
+// Each correction order must be healthy
+const minHealthySize = calculateMinOrderSize(side);
+const affordableOrderCount = Math.floor(availableFunds / minHealthySize);
+const correctionOrders = Math.min(missingSlots, affordableOrderCount);
+```
+
+**Benefits**:
+- ✅ No "double-dust" fragmentation
+- ✅ Constant, predictable spread width
+- ✅ Funds always respected (no forced allocation)
+- ✅ Natural smoothing over multiple rebalance cycles
+
+---
+
+## Periodic Market Price Refresh
+
+Background market price updates every 4 hours to ensure grid anchoring remains accurate during long-running sessions without fills.
+
+### Purpose
+
+If the bot hasn't seen fills for 4 hours, the `startPrice` might become stale if:
+- Market has drifted significantly
+- Liquidity pool price has shifted
+- User wants grid recalculation
+
+### Configuration
+
+**Location**: `modules/constants.ts`
+
+```javascript
+BLOCKCHAIN_FETCH_INTERVAL_MIN: 240,  // 4 hours = 240 minutes
+```
+
+### Implementation Flow
+
+```javascript
+// 1. Timer started during bot initialization
+this.periodicRefreshTimer = setInterval(
+    () => this._performPeriodicRefresh(),
+    BLOCKCHAIN_FETCH_INTERVAL_MIN * 60 * 1000
+);
+
+// 2. When timer fires:
+async _performPeriodicRefresh() {
+    // Fetch latest market price
+    const latestPrice = await derivePrice('book');  // Or 'pool'
+
+    // Update internal anchor if using dynamic pricing
+    if (config.startPrice === 'book' || config.startPrice === 'pool') {
+        this.manager.startPrice = latestPrice;
+    }
+
+    // Grid remains un-affected (fund-driven during normal ops)
+    // Only used for valuation calculations and divergence checks
+}
+```
+
+### When `startPrice` is Numeric
+
+If user set `startPrice: 105.5` in bots.json:
+- **No auto-refresh**: Numeric value is treated as fixed anchor
+- **Valuation uses fixed value**: All calculations use 105.5
+- **Grid doesn't move**: Orders stay where they are (fund-driven rebalancing only)
+
+### Non-Disruptive Updates
+
+Price refresh is passive:
+- ✅ Updates internal valuation
+- ✅ Affects future grid resets if triggered
+- ❌ Does NOT move orders on blockchain (no funds wasted on unnecessary rotations)
+
+---
+
+## Out-of-Spread Metric Refinement
+
+Refactored `outOfSpread` from a simple boolean flag to a numeric distance metric for more precise structural updates.
+
+### Before (Boolean)
+
+```javascript
+// Old approach
+mgr.outOfSpread = true;  // Binary: either in or out
+if (mgr.outOfSpread) {
+    // Perform spread correction
+}
+```
+
+**Problem**: Doesn't distinguish between "slightly out" vs "severely out"
+
+### After (Numeric Distance)
+
+```javascript
+// New approach: distance in steps
+mgr.outOfSpread = 3;  // 3 steps beyond target spread
+
+// Use distance in correction logic
+const spreadDistance = mgr.outOfSpread;
+const replacementSlots = Math.min(spreadDistance, MAX_CORRECTION_SLOTS);
+```
+
+**Benefit**: Enables scaled corrections based on actual severity.
+
+### Calculation
+
+```javascript
+// Calculate how many steps beyond target
+const currentSpreadSteps = calculateCurrentSpreadGap();
+const targetSpreadSteps = calculateTargetSpread();
+const outOfSpreadDistance = Math.max(0, currentSpreadSteps - targetSpreadSteps);
+
+mgr.outOfSpread = outOfSpreadDistance;  // 0 = in spread, 3+ = out
+```
+
+---
+
+## Pipeline Safety & Diagnostics
+
+The bot includes a comprehensive pipeline monitoring system to prevent indefinite blocking and enable operational visibility.
+
+### Pipeline Timeout Safeguard
+
+**Problem**: Pipeline checks could block indefinitely if operations hung due to network issues or stuck corrections.
+
+**Solution**: 5-minute timeout with automatic, non-destructive recovery.
+
+**Configuration** (modules/constants.ts):
+```javascript
+PIPELINE_TIMING: {
+    TIMEOUT_MS: 300000,  // 5 minutes
+}
+```
+
+**How It Works**:
+- `isPipelineEmpty()` tracks when pipeline operations started blocking via `_pipelineBlockedSince` timestamp
+- If blockage exceeds 5 minutes, `clearStalePipelineOperations()` is called
+- Non-destructive recovery: clears operation flags only, does NOT delete orders or modify grid state
+- Recovery called from `_executeMaintenanceLogic()` during periodic maintenance checks
+
+**Location**: `modules/order/manager.ts` (`isPipelineEmpty()` and `clearStalePipelineOperations()`)
+
+### Data Flow
+
+```mermaid
+sequenceDiagram
+    participant Bot as DexBotClass
+    participant Mgr as OrderManager
+    participant Sync as SyncEngine
+    participant Strat as StrategyEngine
+    participant Acct as Accountant
+    participant Chain as Blockchain
+
+    Bot->>Mgr: Initialize grid
+    Mgr->>Acct: Reset funds
+    Mgr->>Sync: Fetch account balances
+    Sync->>Chain: Get balances
+    Chain-->>Sync: Balance data
+    Sync->>Acct: Set account totals
+    Acct->>Acct: Recalculate funds
+
+    Note over Mgr: Grid initialized, ready for trading
+
+    Bot->>Sync: Detect fills (polling)
+    Sync->>Chain: Get open orders
+    Chain-->>Sync: Order data
+    Sync->>Mgr: syncFromOpenOrders()
+    Mgr->>Mgr: processFilledOrders()
+    Strat->>Acct: Update funds (cache proceeds)
+    Strat->>Strat: Identify shortages/surpluses
+    Strat->>Mgr: Rotate orders
+    Mgr->>Acct: Deduct funds (atomic)
+    Mgr->>Chain: Place new orders
+```
+
+---
+
+## Order State Machine
+
+Orders transition through three primary **states** during their lifecycle. **SPREAD** is an *order type* (like BUY or SELL), not a state — spread-zone slots always carry `state: VIRTUAL`.
+
+```mermaid
+stateDiagram-v2
+    [*] --> VIRTUAL: Grid created
+
+    VIRTUAL --> ACTIVE: Order placed on-chain
+    VIRTUAL --> VIRTUAL: Rotated or rebuilt
+
+    ACTIVE --> PARTIAL: Partial fill detected
+    ACTIVE --> VIRTUAL: Order cancelled/rotated
+
+    PARTIAL --> ACTIVE: Consolidated (size >= ideal)
+    PARTIAL --> VIRTUAL: Moved/consolidated
+
+    VIRTUAL --> [*]: Grid regenerated
+
+    note right of VIRTUAL
+        No on-chain presence
+        Funds reserved in virtual pool
+        Includes spread-zone placeholders
+    end note
+
+    note right of ACTIVE
+        On-chain with orderId
+        Funds locked/committed
+    end note
+
+    note right of PARTIAL
+        Partially filled on-chain
+        Waiting for consolidation
+    end note
+```
+
+### State Transition Rules
+
+| From State | To State | Trigger | Fund Impact |
+|------------|----------|---------|-------------|
+| VIRTUAL | ACTIVE | Order placed | Deduct from `chainFree` |
+| ACTIVE | PARTIAL | Partial fill | Reduce `committed` by filled amount |
+| ACTIVE | VIRTUAL | Order cancelled | Add back to `chainFree` |
+| PARTIAL | ACTIVE | Consolidation | Update to `idealSize` (consumes available funds) |
+| PARTIAL | VIRTUAL | Order moved | Release funds, re-reserve |
+
+### Critical: Phantom Order Prevention
+
+A **phantom order** is an illegal state where an order exists as ACTIVE/PARTIAL without a corresponding blockchain `orderId`. This corrupts fund tracking and causes "doubled funds" warnings.
+
+**Why Phantoms Occur**:
+1. **Grid Resize Bug**: `Grid._updateOrdersForSide()` could force VIRTUAL → ACTIVE without blockchain confirmation
+2. **Sync Gap**: Orders without orderId could remain ACTIVE indefinitely if sync logic skipped them
+3. **No Validation**: No centralized check prevented invalid state assignments
+
+**Prevention System** (Three-Layer Defense):
+
+| Layer | Location | Mechanism |
+|-------|----------|-----------|
+| **Guard** | `manager.ts::_updateOrder()` | Centralized validation in `_updateOrder()` rejects ACTIVE/PARTIAL without orderId, auto-downgrades to VIRTUAL |
+| **Grid Protection** | `grid.ts` | Preserve order state during resize: `state: order.state` instead of forcing ACTIVE |
+| **Sync Cleanup** | `sync_engine.ts` | Detect orders without orderId and convert to SPREAD placeholders; prevent phantom fills from triggering rebalancing |
+
+**Verification**:
+- Direct state assignment in code review: All transitions go through `_updateOrder()` (cannot bypass)
+- Automated tests: `tests/repro_phantom_orders.ts` confirms all prevention layers work
+- Logging: Any phantom creation attempt is logged as ERROR with context
+
+---
+
+## Fund Flow Architecture
+
+The fund tracking system uses atomic operations to prevent race conditions and overdrafts.
+
+```mermaid
+graph LR
+    subgraph "Blockchain Balances"
+        CHAIN_FREE[chainFree<br/>Unallocated funds]
+        CHAIN_COMMITTED[committed.chain<br/>On-chain orders]
+    end
+
+    subgraph "Internal Tracking"
+        VIRTUAL[virtual<br/>Reserved for VIRTUAL orders]
+        GRID_COMMITTED[committed.grid<br/>ACTIVE order sizes]
+    end
+
+    subgraph "Calculated Values"
+        AVAILABLE[available<br/>= chainFree - virtual<br/>- fees]
+        TOTAL_CHAIN[total.chain<br/>= chainFree + committed.chain]
+        TOTAL_GRID[total.grid<br/>= committed.grid + virtual]
+    end
+
+    CHAIN_FREE --> AVAILABLE
+    VIRTUAL --> AVAILABLE
+
+    CHAIN_FREE --> TOTAL_CHAIN
+    CHAIN_COMMITTED --> TOTAL_CHAIN
+
+    GRID_COMMITTED --> TOTAL_GRID
+    VIRTUAL --> TOTAL_GRID
+
+    style AVAILABLE fill:#90EE90
+    style VIRTUAL fill:#87CEEB
+```
+
+### Fund Components Explained
+
+- **chainFree**: Unallocated funds on blockchain (from `accountTotals.buyFree/sellFree`)
+- **committed.chain**: Funds locked in on-chain orders (ACTIVE orders with `orderId`)
+- **committed.grid**: Internal tracking of ACTIVE order sizes
+- **virtual**: Funds reserved for VIRTUAL orders (not yet on-chain)
+- **available**: Free funds for new orders = `max(0, chainFree - virtual - fees)`
+
+### Atomic Fund Operations
+
+```mermaid
+sequenceDiagram
+    participant Strat as StrategyEngine
+    participant Mgr as OrderManager
+    participant Acct as Accountant
+
+    Note over Mgr: Want to place order<br/>size = 100
+
+    Mgr->>Acct: tryDeductFromChainFree(type, 100)
+
+    alt Sufficient funds (available >= 100)
+        Acct->>Acct: chainFree -= 100
+        Acct->>Acct: virtual += 100
+        Acct-->>Mgr: true (success)
+        Mgr->>Mgr: Place order
+    else Insufficient funds
+        Acct-->>Mgr: false (failed)
+        Note over Mgr: Order not placed<br/>No fund leak
+    end
+```
+
+---
+
+## Grid Topology
+
+The grid uses a unified "Master Rail" with a dynamic boundary that shifts as fills occur.
+
+```mermaid
+graph LR
+    subgraph "Master Rail (Price Levels)"
+        direction LR
+        B0[buy-0<br/>VIRTUAL]
+        B1[buy-1<br/>ACTIVE]
+        B2[buy-2<br/>ACTIVE]
+        BOUNDARY{Boundary<br/>Index}
+        S0[spread-0<br/>SPREAD]
+        S1[spread-1<br/>SPREAD]
+        S2[spread-2<br/>SPREAD]
+        SELL0[sell-173<br/>ACTIVE]
+        SELL1[sell-174<br/>ACTIVE]
+        SELL2[sell-175<br/>VIRTUAL]
+    end
+
+    B0 --> B1 --> B2 --> BOUNDARY
+    BOUNDARY --> S0 --> S1 --> S2
+    S2 --> SELL0 --> SELL1 --> SELL2
+
+    style B1 fill:#90EE90
+    style B2 fill:#90EE90
+    style S0 fill:#FFD700
+    style S1 fill:#FFD700
+    style S2 fill:#FFD700
+    style SELL0 fill:#FF6B6B
+    style SELL1 fill:#FF6B6B
+    style BOUNDARY fill:#87CEEB
+```
+
+### Boundary Movement
+
+- **Buy Fill**: `boundaryIdx -= 1` (shift left/down)
+- **Sell Fill**: `boundaryIdx += 1` (shift right/up)
+
+### Role Assignment
+
+- **BUY**: Slots `[0, boundaryIdx]`
+- **SPREAD**: Slots `[boundaryIdx + 1, boundaryIdx + G]` where G = spread gap size (empty slots). Actual gaps = G + 1.
+- **SELL**: Slots `[boundaryIdx + G + 1, N]`
+
+---
+
+## Key Operations
+
+### 1. Fill Processing Flow
+
+```mermaid
+sequenceDiagram
+    participant Chain as Blockchain
+    participant Sync as SyncEngine
+    participant Mgr as OrderManager
+    participant Acct as Accountant
+    participant Strat as StrategyEngine
+
+    Chain->>Mgr: Order filled
+    Mgr->>Mgr: Detect fill
+    Mgr->>Strat: processFillsOnly([fills])
+
+    Strat->>Acct: Add proceeds to chainFree
+
+    Mgr->>Strat: calculateTargetGrid(params)
+    Strat-->>Mgr: target grid
+    Mgr->>Mgr: Apply rotations via WorkingGrid
+    Mgr->>Acct: Deduct BTS fees during recalculateFunds
+    Mgr->>Mgr: Consolidate dust partials
+```
+
+### 2. Order Rotation (Crawl Mechanism)
+
+```mermaid
+graph TB
+    START[Fill Detected] --> SHIFT[Shift Boundary]
+    SHIFT --> IDENTIFY[Identify Shortages<br/>Empty slots in active window]
+    IDENTIFY --> CHECK{Surpluses<br/>Available?}
+
+    CHECK -->|Yes| CRAWL[Select Crawl Candidate<br/>Furthest active order]
+    CHECK -->|No| NEW[Place New Order<br/>if funds available]
+
+    CRAWL --> COMPARE{Shortage price<br/>better than<br/>surplus price?}
+    COMPARE -->|Yes| ROTATE[Rotate Order<br/>Cancel old, place new]
+    COMPARE -->|No| SKIP[Skip rotation]
+
+    ROTATE --> NEXT{More<br/>shortages?}
+    SKIP --> NEXT
+    NEW --> NEXT
+
+    NEXT -->|Yes| IDENTIFY
+    NEXT -->|No| DONE[Rebalance Complete]
+```
+
+### 3. Grid Divergence Detection
+
+The grid divergence system monitors and corrects misalignment between ideal grid state and persistent blockchain state.
+
+```mermaid
+graph TB
+    START[Grid Update Triggered] --> CALC[Calculate Ideal Grid<br/>Based on current funds]
+    CALC --> RELOAD[Force Reload Persisted Grid<br/>Ensure fresh blockchain state]
+    RELOAD --> COMPARE[Compare to Persisted Grid]
+    COMPARE --> RMS[Calculate RMS Divergence<br/>For PARTIAL orders only]
+
+    RMS --> CHECK{RMS > Threshold?}
+    CHECK -->|Yes| UPDATE[Update Grid Sizes<br/>Trigger rebalance]
+    CHECK -->|No| SKIP[Skip update]
+
+    UPDATE --> PERSIST[Persist New Grid State]
+    PERSIST --> DONE[Complete]
+    SKIP --> DONE
+```
+
+**Key Improvement (v1.0.0)**: Force reload mechanism now ensures fresh persisted grid data before comparison, preventing stale cache from causing false divergence detections.
+
+---
+
+## Concurrency & Locking
+
+The system uses order-level locks to prevent race conditions during async operations. For the reconciler-specific lock hierarchy and the historical `_syncLock`/`_gridLock` swap, see [GRID_RECONCILE.md](GRID_RECONCILE.md#lock-hierarchy).
+
+### Lock Mechanism
+
+```mermaid
+sequenceDiagram
+    participant Sync as SyncEngine
+    participant Strat as StrategyEngine
+    participant Mgr as OrderManager
+
+    Note over Sync: Detected fill on order P1
+    Sync->>Mgr: lockOrders([P1])
+    Sync->>Sync: Process fill
+
+    par Concurrent Strategy Check
+        Strat->>Mgr: isOrderLocked(P1)?
+        Mgr-->>Strat: true
+        Note over Strat: Skip P1 (locked)
+    end
+
+    Sync->>Sync: Complete fill processing
+    Sync->>Mgr: unlockOrders([P1])
+
+    Note over Strat: Next cycle can now process P1
+```
+
+### Lock Lifetime
+
+- **Default timeout**: 5-10 seconds
+- **Auto-expiry**: Prevents deadlocks from crashes
+- **Best practice**: Always use try/finally to ensure unlock
+
+---
+
+## Module Responsibilities Summary
+
+| Module | Primary Responsibility | Key Functions |
+|--------|----------------------|---------------|
+| **OrderManager** | Central coordinator, state management | `_updateOrder()`, `lockOrders()`, `getOrdersByTypeAndState()` |
+| **Accountant** | Fund tracking, fee management | `recalculateFunds()`, `tryDeductFromChainFree()`, `_verifyFundInvariants()` |
+| **StrategyEngine** | Grid rebalancing, rotation target calculation | `calculateTargetGrid()`, `processFillsOnly()`, `hasAnyDust()` |
+| **SyncEngine** | Blockchain sync, fill detection | `syncFromOpenOrders()`, `synchronizeWithChain()` |
+| **Grid** | Grid creation, sizing, divergence | `createOrderGrid()`, `compareGrids()`, `checkAndUpdateGridIfNeeded()` |
+| **Utils** | Shared utilities, conversions | `quantizeFloat()`, `normalizeInt()` (`math.ts`); order predicates (`order.ts`); COW action building (`validate.ts`); price derivation (`system.ts`) |
+| **Logger** | Formatted logging, diagnostics | `logOrderGrid()`, `logFundsStatus()` |
+
+---
+
+## Dynamic Configuration Refresh
+
+The bot implementation supports runtime updates to specific configuration parameters without requiring a process restart. This is handled via a **Periodic Configuration Refresh** mechanism.
+
+### The Refresh Cycle
+
+Every 4 hours (default `BLOCKCHAIN_FETCH_INTERVAL_MIN`), the bot performs the following safe refresh cycle:
+
+1.  **Thread-Safe Load**: The bot re-reads `profiles/bots.json` using `readBotsFileWithLock` to ensure it doesn't collide with manual edits or the CLI manager.
+2.  **Memory Update**: It identifies its own configuration entry and updates its internal memory state (`this.config` and `manager.config`).
+3.  **Non-Disruptive Application**: The refresh is designed to be **passive**. It updates valuation anchors but does **not** trigger on-chain order movement automatically.
+
+### Configuration Authority: `startPrice` / `poolRef`
+
+The `startPrice` parameter follows a strict hierarchy of authority:
+
+| Setting Type | Source | Behavior |
+|--------------|--------|----------|
+| **Numeric** | `bots.json` | **Single Source of Truth**. Blocks all auto-derivation. Used as a fixed anchor for valuation and grid resets. |
+| **"pool"** | Blockchain | Derived from current Liquidity Pool price during resets or 4h refresh cycles. |
+| **"pool" + poolRef** | Blockchain | Same as `"pool"`, but pool is fetched directly by ID (`get_objects`) bypassing discovery. Set `poolRef` in `bots.json` (e.g. `"1.19.48"`) to pin a proxy pool. |
+| **"book"** | Blockchain | Derived from current order book price during resets or 4h refresh cycles. |
+
+---
+
+## Data Persistence
+
+```mermaid
+graph LR
+    subgraph "In-Memory State"
+        ORDERS[orders Map<br/>Grid state]
+        FUNDS[funds Object<br/>Fund tracking]
+        INDICES[Indices<br/>_ordersByState<br/>_ordersByType]
+    end
+
+    subgraph "Persisted State"
+        ORDERS_JSON[<botKey>.json<br/>Grid snapshot<br/>feesOwed, boundaryIdx, btsBalance]
+        BOTS_JSON[bots.json<br/>Bot config]
+    end
+
+    ORDERS --> ORDERS_JSON
+    FUNDS --> ORDERS_JSON
+
+    ORDERS_JSON -.->|Load on startup| ORDERS
+    ORDERS_JSON -.->|Load on startup| FUNDS
+
+    BOTS_JSON -.->|Load on startup| CONFIG[Bot Config]
+```
+
+### Persistence Strategy
+
+- **Grid state**: Persisted after every rebalance to `<botKey>.json` in `profiles/orders/`
+- **Fund state**: Available funds derived from blockchain balances at runtime (no separate persistence needed)
+- **Retry logic**: 3 attempts with exponential backoff
+- **Graceful degradation**: Bot continues if persistence fails (in-memory only)
+
+---
+
+## Memory-Only Integer Tracking
+
+The system has been optimized to use a "memory-driven" model for order updates, eliminating redundant blockchain API calls during normal operation.
+
+### Key Changes
+
+**1. Raw Order Cache (`rawOnChain`)**
+- Grid slots now store exact blockchain order representations (integers/satoshis) in a `rawOnChain` cache
+- **Birth**: Cache populated immediately after successful order placement using broadcasted arguments
+- **Partial Fills**: Cache updated in-place via integer subtraction (subtracting filled satoshis from `for_sale`)
+- **Updates/Rotations**: Cache refreshed with adjusted integers returned by build process
+
+**2. Eliminated Redundant API Calls**
+- Removed all `readOpenOrders()` calls from `_buildSizeUpdateOps()` and `_buildRotationOps()`
+- Removed `computeVirtualOpenOrders()` logic that was redundantly fetching entire account state
+- The bot now trusts its internal state, backed by real-time fill listener, to build transactions
+
+**3. Refactored `buildUpdateOrderOp()`**
+- Updated to support optional `cachedOrder` parameter
+- Allows callers to bypass blockchain queries if they have raw state in memory
+- Returns `finalInts` along with operation data for local tracking
+
+**4. Self-Healing Resilience**
+- Maintains "State Recovery Sync" fallback
+- If a memory-driven transaction fails, bot catches error and performs a full refresh
+- Ensures internal ledger stays synchronized with BitShares blockchain
+
+### Benefits
+- **Faster reaction time**: No waiting for blockchain queries during order updates
+- **Reduced API load**: Fewer fetches, less network congestion
+- **Mathematical precision**: Integer-based tracking prevents float precision errors
+  - *See [FUND_MOVEMENT_AND_ACCOUNTING.md § 5.5](FUND_MOVEMENT_AND_ACCOUNTING.md#55-precision--quantization-patch-14) for quantization utilities and best practices*
+- **Fallback safety**: Automatic recovery if memory state becomes inconsistent
+
+### Performance Impact
+- Batch operations (size updates, rotations) now run without any blockchain fetches
+- Only placement operations and recovery syncs query the blockchain
+- Estimated **10-20x speedup** for high-frequency operations
+
+---
+
+## Error Handling & Safety
+
+### Fund Invariants
+
+The system continuously monitors three mathematical invariants:
+
+1. **Account Equality**: `chainTotal = chainFree + committed.chain`
+2. **Committed Ceiling**: `committed.grid <= chainTotal`
+3. **Available Leak Check**: `available <= chainFree`
+
+**Tolerance**: 0.1% (to account for fees and rounding)
+
+### Index Consistency
+
+- **Validation**: `validateIndices()` checks Map ↔ Set consistency
+- **Repair**: `_repairIndices()` rebuilds indices if corruption detected
+- **Defensive**: Called after critical operations
+
+---
+
+## Performance Considerations
+
+### Optimization Strategies
+
+1. **Batch fund recalculation**: `pauseFundRecalc()` / `resumeFundRecalc()`
+2. **Index-based lookups**: O(1) access via `_ordersByState` and `_ordersByType`
+3. **Lock expiry**: Prevents permanent blocking from crashes
+4. **Fee caching**: Reduces blockchain API calls
+
+### Metrics Tracking
+
+```javascript
+manager.getMetrics()
+// Returns:
+// - fundRecalcCount
+// - invariantViolations
+// - lockAcquisitions
+// - stateTransitions
+// - lastSyncDurationMs
+```
+
+---
+
+### Zero-Dependency Policy
+
+DEXBot2 operates under a **zero mandatory production dependency** policy. The dependency tree is empty — every production capability (blockchain client, WebSocket transport via Node's native `globalThis.WebSocket`, crypto/signing, serialization, testing, price feeds, credential vault) is implemented natively within the codebase. Requires **Node.js >= 22.12** for the built-in WebSocket.
+
+**Why:** Trading bots handle real money. Every external dependency is a supply-chain risk surface. Keeping the dependency tree empty means no `npm audit` surprises, no supply-chain attacks on upstream packages, and no version-migration overhead for the core runtime.
+
+**Special case — trading bots:** This level of dependency discipline is rare in open-source trading software. Established projects (Gekko, Freqtrade, Hummingbot) carry 15–20+ production dependencies. DEXBot2's empty dependency tree is a deliberate architectural choice, not an accidental outcome — it reflects the project's priority of operational safety over developer convenience.
+
+**What this means in practice:**
+- Blockchain connectivity (`bitshares-native/`) — hand-rolled from protocol primitives
+- Elliptic curve crypto (secp256k1) — native JS, zero native addons
+- Testing — `node:assert`, no Jest/Mocha/Vitest
+- Persistence — JSON flat files, no SQLite/ORM
+- Price sources — native candle fetching, no CCXT/CoinGecko
+- Process management — PM2 ecosystem or direct Node.js, no Docker requirement
+
+### Testing Strategy & Quality Assurance
+
+DEXBot2 uses a native Node.js `assert` testing strategy to ensure reliability without heavy dependencies.
+
+### Test Coverage by Module
+
+```mermaid
+graph LR
+    A["Logic Tests<br/>(tests/test_*_logic.ts)"]
+    B["Integration Tests<br/>(tests/test_*.ts)"]
+    C["Signal Tests<br/>(tests/test_*_signal*.ts)"]
+    D["Credit/Debt Tests<br/>(tests/test_*_credit*.ts)"]
+
+    A -->|Manager, State Machine| A1["manager_logic"]
+    A -->|Fund Tracking| A2["accounting_logic"]
+    A -->|Grid Creation| A3["grid_logic"]
+    A -->|Rebalancing| A4["strategy_logic"]
+    A -->|Sync Logic| A5["sync_logic"]
+
+    B -->|Multi-step Scenarios| B1["Market Scenarios"]
+    B -->|Edge Cases| B2["Partial Order Tests"]
+    B -->|Real-world Scenarios| B3["Fills/FEE Tests"]
+
+    C -->|Dynamic Weight| C1["dynamic_weight"]
+    C -->|Derivative Trap| C2["derivative_signal"]
+    C -->|Momentum Gate| C3["momentum_gate"]
+
+    D -->|CR Planner| D1["cr_planner"]
+    D -->|Credit Runtime| D2["credit_runtime"]
+    D -->|MPA Wiring| D3["dexbot_credit_wiring"]
+```
+
+### Running Tests
+
+```bash
+# Run all tests (native assert)
+npm test
+
+# Specific logic area
+tsx tests/test_accounting_logic.ts
+
+# Signal tests
+tsx tests/test_market_adapter_signal_gates.ts
+tsx tests/test_dynamic_weight_override_wiring.ts
+
+# Credit/debt tests
+tsx tests/test_cr_planner.ts
+tsx tests/test_dexbot_credit_wiring.ts
+```
+
+### Test Quality Metrics
+
+**Coverage Goals:**
+- ✅ All public methods have tests
+- ✅ All invariants verified automatically
+- ✅ Edge cases covered (zero funds, max orders, etc.)
+- ✅ Concurrent operations tested with locks
+- ✅ State transitions validated end-to-end
+- ✅ Signal pipelines tested (dynamic weight, derivative traps)
+- ✅ Credit/debt runtime tested (CR planner, MPA wiring)
+
+**Test Suite Evolution:**
+- 50+ test cases for signal intelligence and credit runtime
+- Dynamic weight override and market adapter signal gate tests
+- Derivative momentum gate and signal trap regression tests
+- Credit/debt tests with CR planner and MPA wiring validation
+
+### Testing Best Practices
+
+**For Developers:**
+
+1. **Run tests before commits**
+   ```bash
+   npm test
+   ```
+
+2. **Add tests for new features**
+   - Follow patterns in existing tests
+   - Test fund impact of new logic
+   - Include edge cases
+
+3. **Verify invariants**
+   ```javascript
+   expect(manager.validateIndices()).toBe(true);
+   expect(chainTotal === chainFree + chainCommitted).toBe(true);
+   ```
+
+4. **Use debug mode for problematic scenarios**
+   ```javascript
+   manager.logger.level = 'debug';  // Enable detailed logging
+   // ... run scenario ...
+   // Check console output for detailed fund tracking
+   ```
+
+### Test Documentation References
+
+- **[tests/README.md](../tests/README.md)** - Test suite organization, categories, and running instructions
+
+- **[developer_guide.md#testing-fund-calculations](developer_guide.md#testing-fund-calculations)** - Testing guide for developers
+  - How to write fund tests
+  - Common test patterns
+  - Debugging failing tests
+  - Adding tests for new features
+
+---
+
+## Market Adapter Signal Pipeline
+
+The market adapter runs as a standalone process that computes AMA-derived grid prices, trend signals, dynamic weights, and collateral-ratio advisories.
+
+### Signal Flow
+
+```
+price_candles -> AMA -> gridCenterPrice
+                |
+                +-> slope_analysis -> trend_offset (asymmetric weight shift)
+                |
+price_candles -> ATR -> weight_variance (symmetric shift)
+                |
+                +-> regime_detection (Hurst/PE) -> regime_filter
+                |
+                +-> Kalman_confirmation -> blended_dynamic_weight
+```
+
+### Outputs (per cycle, per bot)
+- `gridCenterPrice` — AMA center price, clamped to min/max bounds
+- `weights` — `{ buy, sell }` dynamic grid weighting
+- `collateralRecommendation` — advisory collateral ratio hint
+- `trend` / `atr` — raw regime and volatility signals
+- Trigger files when the adapter accepts the first AMA center, the grid price delta exceeds threshold, or whitelisted range-scaling slope delta exceeds threshold
+
+### Integration with Bot Runtime
+
+1. Adapter persists `profiles/orders/<botKey>.dynamicgrid.json` before any reset trigger.
+2. Adapter writes `profiles/recalculate.<botKey>.trigger` for bootstrap, AMA-center delta, or whitelisted AMA-slope range reset.
+3. `dexbot_maintenance_runtime.ts` consumes the trigger under `_fillProcessingLock`, with idle/dust deferral when needed.
+4. Bot runtime reads accepted center, range-scaling fields, and dynamic weights from the dynamic-grid snapshot during reset and selected maintenance paths.
+
+---
+
+## Credential Security Architecture
+
+DEXBot2 uses a hardened credential daemon (`credential-daemon.ts`) for key management and signing.
+
+### Security Layers
+- **Vault v2**: scrypt (N=2^17) key derivation, per-record HKDF isolation, AES-256-GCM encryption
+- **Daemon-backed signing**: Primary bot flow uses signing tokens; all signing happens inside the daemon, raw keys never exported
+- **Session cache**: Encrypted HKDF re-encryption with a random salt never persisted
+- **Runtime hardening**: lstat + owner/mode/type checks on all sockets and ready files; bootstrap socket destroyed after first use
+- **Strict daemon policy**: Memory safety and zeroing, session hardening, signing cache with time-based expiry
+
+### PM2 Integration
+
+`dexbot pm2` unlocks `dexbot-cred` through a one-shot local bootstrap channel instead of exporting the master password to every PM2 app. Use `dexbot pm2 restart ...` for DEXBot-managed PM2 actions.
+
+---
+
+## Credit/Debt Runtime
+
+Native DEXBot2 support for MPA borrowing and credit offer workflows. The logic is split between a pure-math planner and a lifecycle runtime:
+
+| Module | Role |
+|--------|------|
+| **`modules/cr_planner.ts`** | **Pure math / credit planning** — debt-first CR adjustments, derived order sizing. No I/O, no side effects; safe to unit-test in isolation. |
+| **`modules/credit_runtime.ts`** | **Lifecycle management** — applies the planner's output to the chain, enforces policy, drives watchdog cadence and grid-reset coupling. |
+
+### Scope
+- **MPA borrowing**: Call-order updates with debt-first CR planning
+- **Credit offers**: Accept/repay with auto-reborrow, LP-backed collateral valuation
+- **Policy enforcement**: Per-bot `debtPolicy` with hard CR floors/ceilings and max fee rates
+- **Grid reset coupling**: Every successful CR adjustment requests a grid rebuild
+
+### Runtime Rules
+- Evaluates on the dedicated credit watchdog interval
+- No separate enable switch — active when `debtPolicy.lending` is present, non-empty, and every item declares `collateralAsset`
+- Claw can read the same bot policy without redefining rules (via `claw/modules/credit_runtime_adapter.ts`)
+
+---
+
+## Grid Rebalancing Robustness
+
+The strategy engine has been significantly strengthened with improvements to fund validation, dust handling, and order constraints:
+
+**1. Pre-Flight Fund Validation**
+- Before executing batch order placements, available funds are validated
+- Prevents insufficient fund errors during large rotation cycles
+- Uses atomic check-and-deduct pattern for safety
+- Located in: `modules/order/strategy.ts` - `rebalanceSideRobust()`
+
+**2. Dust Partial Handling**
+- Improved dust detection algorithm prevents false positives
+- Detects dust as `< 5% of ideal order size`
+- Dust partials are absorbed into the next grid rebuild cycle (no merge/split mechanics)
+- **Auto-Cancellation**: `_cancelDustOrders()` cancels dust partials on-chain immediately on detection — no delay, no timer maps, no retry budgets. Cancel is attempted post-fill (inside fill lock) and every 5-min health check (safety net).
+
+**3. Strict Order Size Constraints**
+- Orders validated to not exceed available funds
+- Maximum order size enforced during both placement and rotation
+- Prevents oversized orders that fail on-chain
+- Atomic validation with placement ensures consistency
+
+**4. Boundary Index Persistence**
+- BoundaryIdx (spread zone pivot) now correctly persisted across bot restarts
+- Ensures grid rotation continues seamlessly after divergence correction
+- Fixes grid instability from incorrect boundary tracking
+
+**5. Taker Fee Accounting**
+- Both market and blockchain taker fees now accounted for correctly
+- Fee deduction uses proper `isMaker` parameter
+- Prevents fund leaks from missing fee calculations
+- Located in: `modules/order/manager.ts` - `processFilledOrders()`
+
+**6. Precision Spread Management (Logarithmic Logic)**
+- **Discrete Step Tracking**: Replaced the legacy linear multiplier (`SPREAD_WIDENING_MULTIPLIER`) with a discrete 1-slot logarithmic buffer. This ensures correction triggers exactly when the market moves by one full increment.
+- **Center-Gap Awareness**: Refined the grid initialization math to account for the "Center Gap" naturally created during symmetric centering. This reduces the initial spread by ~0.5% (one full increment) compared to the previous version.
+- **Collision-Free Safety**: Increased `MIN_SPREAD_FACTOR` to 2.1 to ensure that the security minimum (2 spread orders) never conflicts with the spread correction threshold, even at micro-spread configurations.
+
+### Related Documentation
+
+For detailed fund calculations and test coverage, see:
+- [developer_guide.md#testing-fund-calculations](developer_guide.md#testing-fund-calculations) - How fund calculations are tested
+- [tests/README.md](../tests/README.md) - Test suite organization and running instructions
+
+---
+
+- [Fund Movement Logic](FUND_MOVEMENT_AND_ACCOUNTING.md) - Detailed mathematical formulas and algorithms
+- [Developer Guide](developer_guide.md) - Code navigation and onboarding
+- [README](../README.md) - User documentation and setup
+- [WORKFLOW.md](WORKFLOW.md) - Git branch workflow
+
+---
+
+## Quick Reference
+
+### Common Code Patterns
+
+**Get orders by state and type:**
+```javascript
+const activeBuys = manager.getOrdersByTypeAndState(ORDER_TYPES.BUY, ORDER_STATES.ACTIVE);
+```
+
+**Atomic fund deduction:**
+```javascript
+if (manager.accountant.tryDeductFromChainFree(orderType, size)) {
+    // Funds deducted, safe to place order
+} else {
+    // Insufficient funds, skip
+}
+```
+
+**Batch order updates:**
+```javascript
+manager.pauseFundRecalc();
+for (const order of orders) {
+    // context parameter helps with logging/debugging the source of the update
+    manager._updateOrder(order, 'batch-update', { skipAccounting: false, fee: 0 });
+}
+manager.resumeFundRecalc(); // Recalculates once
+```
+
+**Lock orders during async operations:**
+```javascript
+manager.lockOrders([orderId]);
+try {
+    await asyncOperation();
+} finally {
+    manager.unlockOrders([orderId]);
+}
+```