dexbot 1.4.18 → 1.4.20

package/docs/LIFECYCLE.md

@@ -0,0 +1,207 @@
+# DEXBot2 Lifecycle — End-to-End Walkthrough
+
+This is the **one-page map** for newcomers. It stitches together the flows that are
+otherwise spread across `architecture.md` and `developer_guide.md` into two concrete
+lifecycles (fill-driven and maintenance-driven) plus the startup sequence. Read this
+first; follow the file/function references into the deeper docs.
+
+> All code references use `file:line` so you can jump straight to the source.
+
+---
+
+## 1. System Context
+
+Who talks to what. DEXBot2 is a long-running Node process that both *reads* the
+BitShares chain/market and *writes* limit orders back to it.
+
+```mermaid
+graph TB
+    OP[Operator<br/>CLI / PM2 / unlock.ts]
+    CHAIN[(BitShares Chain<br/>fills, open orders, balances)]
+    MKT[Market Data<br/>LP pools / order book / AMA feed]
+    STORE[(Local Storage<br/>profiles/orders/*.json<br/>profiles/bots.json)]
+    BOT[DEXBot2 Process<br/>dexbot_class.ts]
+
+    OP -->|configure / start| BOT
+    CHAIN -->|fill events op-4| BOT
+    CHAIN -->|open-order poll| BOT
+    MKT -->|price / AMA center| BOT
+    BOT -->|CREATE / UPDATE / CANCEL| CHAIN
+    BOT -->|grid snapshot| STORE
+    STORE -->|restore grid| BOT
+```
+
+Three inputs drive everything: **fill events** (reactive), **market/AMA price**
+(periodic), and **config/storage** (startup + recovery).
+
+---
+
+## 2. Startup / Bootstrap Sequence
+
+Runs once per bot launch. Goal: decrypt keys, load metadata, rebuild the master
+grid from either config or the persisted snapshot, then hand off to the runtime
+loops.
+
+```mermaid
+sequenceDiagram
+    participant OP as Operator
+    participant CL as dexbot_class.ts
+    participant CD as Credential Daemon
+    participant AM as Asset Metadata
+    participant OM as OrderManager
+    participant CH as Chain (open orders)
+    participant ST as Storage
+
+    OP->>CL: start(botKey)
+    CL->>CD: decrypt private key (AES, interactive/one-shot)
+    CD-->>CL: unlocked key
+    CL->>AM: load precision, fees, asset IDs
+    CL->>ST: load persisted grid snapshot (if present)
+    ST-->>CL: orders/botKey.json
+    CL->>OM: init Master Grid (config OR snapshot)
+    CL->>CH: SyncEngine 2-pass (grid<->chain match)
+    CH-->>OM: detect partials / stale / phantom
+    CL->>CL: start maintenance loop + market adapter (if AMA)
+    CL-->>OP: bot live, awaiting fills / ticks
+```
+
+Key references:
+- Credential unlock: `modules/credential_runtime.ts`, `CREDENTIAL_SECURITY.md`
+- Master Grid init + COW: `docs/architecture.md` §"Copy-on-Write (COW) Grid Pattern"
+- SyncEngine 2-pass: `modules/order/sync_engine.ts`, `docs/GRID_RECONCILE.md`
+
+---
+
+## 3. Lifecycle A — Fill-Driven (Reactive)
+
+This is the hot path. A limit order gets filled on-chain and the bot must rebuild
+grid symmetry, credit proceeds, and broadcast replacement orders — all inside a
+single Copy-on-Write rebalance cycle.
+
+```mermaid
+sequenceDiagram
+    participant CH as Chain (op-4 fill)
+    participant Q as _incomingFillQueue
+    participant FR as dexbot_fill_runtime
+    participant OM as OrderManager
+    participant WG as WorkingGrid (COW)
+    participant AC as Accounting (SSOT)
+    participant ST as Strategy (calculateTargetGrid)
+    participant CO as chain_orders (broadcast)
+    participant STORE as Storage
+
+    CH->>Q: enqueue fill(s)  (dexbot_fill_runtime.ts:325)
+    FR->>FR: drain queue -> _processFillsWithBatching
+    FR->>OM: processFilledOrders()  (manager.ts)
+    OM->>AC: processFillAccounting()  (single call, batch)
+    AC-->>WG: chainFree += combined proceeds
+    OM->>ST: calculateTargetGrid()  (rotations + boundary shift)
+    ST-->>WG: mutated WorkingGrid only
+    OM->>CO: updateOrdersOnChainBatch()  (atomic broadcast)
+    CO-->>CH: CREATE / CANCEL orders
+    CO-->>OM: confirm
+    OM->>WG: commit WorkingGrid -> Master Grid
+    OM->>STORE: persistGrid()  (snapshot after confirm)
+```
+
+Why it matters:
+- **Fixed-cap batching** (`MAX_FILL_BATCH_SIZE`, default 4) keeps bursts
+  deterministic — see `docs/architecture.md` §"Fill Processing Pipeline".
+- **Single rebalance cycle**: all fills in a batch share one broadcast, so proceeds
+  are immediately available for replacement sizing (no split-across-cycles delay).
+- **Replay-safe accounting**: processed-fill keys prevent double-credit on retries
+  (`modules/dexbot_fill_runtime.ts`, `PROCESSED_FILL_PERSISTENCE_MODES`).
+
+References: `modules/dexbot_fill_runtime.ts`, `modules/order/manager.ts`,
+`docs/FUND_MOVEMENT_AND_ACCOUNTING.md`.
+
+---
+
+## 4. Lifecycle B — Maintenance / AMA Signal-Driven (Periodic)
+
+Runs on a timer (and on AMA center updates). The grid is *not* rebuilt from fills
+here; instead funds are re-synced, the AMA center is promoted to the grid center,
+and the grid is recalculated against the latest market view.
+
+```mermaid
+sequenceDiagram
+    participant MA as Market Adapter (AMA)
+    participant CL as dexbot_class.ts
+    participant RT as maintenance_runtime
+    participant AC as Accounting
+    participant GD as Grid (recalculateGrid)
+    participant CO as chain_orders
+    participant STORE as Storage
+
+    MA->>CL: AMA center snapshot updated
+    CL->>CL: _performPeriodicGridChecks()  (dexbot_class.ts:1664)
+    CL->>RT: performPeriodicGridChecks()  (dexbot_maintenance_runtime.ts:1043)
+    RT->>RT: runGridMaintenance(bot,'periodic')  (dexbot_maintenance_runtime.ts:1845)
+    RT->>RT: executeMaintenanceLogic()  (dexbot_maintenance_runtime.ts:1452)
+    RT->>AC: recalculate funds from balances
+    RT->>GD: promote AMA center -> grid center
+    RT->>GD: recalculateGrid()  (grid.ts)
+    GD->>GD: rebuild levels via WorkingGrid (COW)
+    RT->>CO: broadcast adjustments (if any)
+    RT->>STORE: persistGrid()
+```
+
+Two triggers feed this loop:
+1. **Timer** — periodic fund sync + grid checks (`_performPeriodicGridChecks`).
+2. **AMA center change** — `market_adapter` recomputes the adaptive moving average;
+   when the center moves enough it promotes to the grid center and forces a recalc
+   (`modules/dexbot_maintenance_runtime.ts`, `market_adapter/market_adapter.ts`).
+
+The AMA signal stack (AMA/Kalman/Hurst/PE) is *research-tuned* in `analysis/` and
+*consumed* here at runtime — see `analysis/README.md` for the tooling that produces
+the parameters.
+
+> Note: `runMaintenance()` is a **different** subsystem — the credit/MPA debt
+> runtime (`modules/credit_runtime.ts:3041`, reached via
+> `_runCreditRuntimeMaintenance` at `dexbot_class.ts:1769`). The grid maintenance
+> chain above is the one that matters for order/price upkeep.
+
+References: `modules/dexbot_class.ts:1664` (`_performPeriodicGridChecks`) →
+`modules/dexbot_maintenance_runtime.ts:1043` (`performPeriodicGridChecks`) →
+`:1845` (`runGridMaintenance`) → `:1452` (`executeMaintenanceLogic`),
+`docs/GRID_RECALCULATION.md`, `docs/GRID_RECONCILE.md`.
+
+---
+
+## 5. Cross-Cutting Invariants (read before touching anything)
+
+These are the rules that make the two lifecycles safe. They are *convention-
+enforced*, not compiler-enforced — learn them or you will introduce fund bugs.
+
+| Invariant | Rule | Reference |
+|---|---|---|
+| **COW boundary** | All grid mutations happen on the WorkingGrid; Master Grid is frozen and only committed after chain confirmation. | `docs/developer_guide.md` §"Copy-on-Write (COW) Development Rules" |
+| **Fund SSOT** | `Accounting` owns every fund number. Nothing else computes available funds. | `docs/architecture.md` §"Fund Flow Architecture" |
+| **Replay-safe fills** | A fill is credited exactly once via processed-fill keys; retries are idempotent. | `modules/dexbot_fill_runtime.ts` |
+| **Single broadcast per cycle** | One `updateOrdersOnChainBatch` per rebalance — never scatter writes. | `docs/architecture.md` §"Fill Processing Pipeline" |
+| **Browser/Node split** | Heavy runtime is Node-only; never import it from a browser bundle. | `AGENTS.md` "Browser-Safe Surface", `docs/BROWSER_COMPAT_PLAN.md` |
+| **Lock ordering** | Fill drain and maintenance must not run a rebalance concurrently. | `docs/developer_guide.md` §"Startup Sequence & Lock Ordering" |
+
+---
+
+## 6. File Map (where to go next)
+
+| You want to understand… | Start here |
+|---|---|
+| How a fill becomes orders | `modules/dexbot_fill_runtime.ts` → `modules/order/manager.ts` |
+| Grid math / recalculation | `modules/order/grid.ts`, `docs/GRID_RECALCULATION.md` |
+| Funds & accounting | `modules/order/accounting.ts`, `docs/FUND_MOVEMENT_AND_ACCOUNTING.md` |
+| Periodic loop & AMA hook | `modules/dexbot_maintenance_runtime.ts`, `modules/dexbot_class.ts:1664` |
+| Market signal source | `market_adapter/market_adapter.ts`, `analysis/README.md` |
+| Startup & orchestration | `modules/dexbot_class.ts`, `docs/developer_guide.md` §"Startup Sequence" |
+| Why COW exists | `docs/architecture.md` §"Copy-on-Write (COW) Grid Pattern", `docs/COW_INVARIANTS.md` |
+
+---
+
+### TL;DR mental model
+
+> Blockchain fill (or AMA tick) → enqueue → drain in fixed-cap batches →
+> credit proceeds to Accounting (SSOT) → mutate **WorkingGrid** only →
+> calculate target grid → **single** atomic broadcast → commit WorkingGrid to
+> Master → persist snapshot. Master Grid is immutable; everything else is a
+> disposable copy until the chain confirms.

package/docs/README.md

@@ -109,6 +109,14 @@ This directory contains the comprehensive technical documentation for the DEXBot
 - **Market Adapter Signal Pipeline**: AMA center, dynamic weights, regime detection, and collateral advisories
 - **Credit/Debt Runtime**: Native MPA and credit offer workflows with CR planning and grid reset coupling; `creditOnly` mode for runtime-only operation without grid trading
 
+### 🔁 [Lifecycle](LIFECYCLE.md)
+*The end-to-end walkthrough (start here for the big picture).*
+- **System Context**: What DEXBot2 talks to (chain, market data, storage, operator).
+- **Startup / Bootstrap**: Decrypt keys → load metadata → rebuild master grid → sync → runtime loops.
+- **Lifecycle A (Fill-Driven)**: Reactive path from an on-chain fill to a single atomic rebalance + broadcast.
+- **Lifecycle B (Maintenance / AMA-Driven)**: Periodic path from `_performPeriodicGridChecks` → `executeMaintenanceLogic`.
+- **Cross-Cutting Invariants**: COW boundary, fund SSOT, replay-safe fills, lock ordering.
+
 ### 📖 [Developer Guide](developer_guide.md)
 *Your daily companion for coding.*
 - **Quick Start**: How to get the development environment running.
@@ -136,7 +144,7 @@ This directory contains the comprehensive technical documentation for the DEXBot
 
 ### 🧭 [Evolution Report](EVOLUTION.md)
 *Project timeline and major architecture phases.*
-- **Coverage**: Historical milestones from the initial December 2025 bootstrap through the v1.4.18 stable release (grid persistence safety, dust pipeline fix, net inventory lots); credit/debt runtime and maintenance hardening are covered under Phase 5 (Mar–Jun 2026)
+- **Coverage**: Historical milestones from the initial December 2025 bootstrap through the v1.4.20 stable release (grid persistence safety, dust pipeline fix, net inventory lots); credit/debt runtime and maintenance hardening are covered under Phase 5 (Mar–Jun 2026)
 - **Focus**: Architecture evolution, release history, test growth, and documentation changes
 
 ### 🗒️ [Changelog](../CHANGELOG.md)

package/docs/developer_guide.md

@@ -11,8 +11,9 @@ Welcome to DEXBot2! This guide will help you understand the codebase, navigate k
 ### 1. **Start Here** (5 minutes)
 Read these files in order to get oriented:
 1. [README](../README.md) - User documentation and setup
-2. [architecture.md](architecture.md) - System architecture and module relationships
-3. [FUND_MOVEMENT_AND_ACCOUNTING.md](FUND_MOVEMENT_AND_ACCOUNTING.md) - Core algorithms and formulas
+2. [LIFECYCLE.md](LIFECYCLE.md) - **End-to-end walkthrough** (startup, fill-driven, and AMA-driven flows with diagrams) — read this first for the big picture
+3. [architecture.md](architecture.md) - System architecture and module relationships
+4. [FUND_MOVEMENT_AND_ACCOUNTING.md](FUND_MOVEMENT_AND_ACCOUNTING.md) - Core algorithms and formulas
 
 ### 2. **Core Concepts** (15 minutes)
 Understand these fundamental concepts before diving into code:
@@ -130,7 +131,7 @@ A **phantom order** is an order in ACTIVE/PARTIAL state WITHOUT a valid `orderId
 | **Asymmetric Offset** | Directional weight shift (buy-heavy or sell-heavy) driven by AMA/Kalman trend |
 | **Derivative Signal** | SMA/MACD/RSI-based entry bias and momentum gate for optional strategy filtering |
 | **Momentum Gate** | N-bar commitment tracking that confirms derivative signals before acting |
-| **Grid Price** | Price anchor for grid math; can be numeric, `"pool"`, `"book"`, or AMA keyword (`"ama"`, `"ama1"`–`"ama4"`) |
+| **GridPrice** | Price anchor for grid math; can be numeric, `"pool"`, `"book"`, or AMA keyword (`"ama"`, `"ama1"`–`"ama4"`) |
 
 ### Grid Concepts
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dexbot",
-  "version": "1.4.18",
+  "version": "1.4.20",
   "description": "The first open source trading bot with zero runtime dependencies and a fully adaptive market making strategy.",
   "main": "dist/modules/dexbot_class.js",
   "exports": {
@@ -14,7 +14,6 @@
   },
   "files": [
     "dist",
-
     "ecosystem.config.cjs",
     "scripts/bots",
     "scripts/clean-dist.js",
@@ -33,7 +32,6 @@
     "scripts/postinstall.js",
     "scripts/ptest.sh",
     "scripts/README.md",
-    "scripts/repo-stats.html",
     "scripts/reset-settings.sh",
     "scripts/unlock",
     "scripts/update.js",
@@ -52,10 +50,12 @@
     "docs/WORKFLOW.md",
     "docs/architecture.md",
     "docs/developer_guide.md",
+    "docs/LIFECYCLE.md",
     "docs/docker.md",
     "market_adapter/README.md",
     "modules/README.md",
     "analysis",
+    "!analysis/charts",
     "tests/README.md",
     "CHANGELOG.md",
     "README.md",

package/scripts/lib/dexbot-paths.sh

@@ -39,10 +39,13 @@ if [ -z "$PROFILE_ROOT" ]; then
     else
         # Home dir: $HOME normally; when unset (e.g. cron/systemd), mirror
         # Node's os.homedir() passwd fallback so the shell side does not fall
-        # back into a node_modules package dir on npm installs.
-        HOME_PROFILES=""
-        if [ -n "$HOME" ]; then
-            HOME_PROFILES="${HOME}/.config/dexbot2/profiles"
+        # back into a node_modules package dir on npm installs. XDG_CONFIG_HOME
+        # overrides the config base, mirroring HOME_CONFIG_DIR in paths.ts.
+        CONFIG_BASE=""
+        if [ -n "$XDG_CONFIG_HOME" ]; then
+            CONFIG_BASE="$XDG_CONFIG_HOME"
+        elif [ -n "$HOME" ]; then
+            CONFIG_BASE="${HOME}/.config"
         else
             PASSWD_HOME=""
             if command -v getent >/dev/null 2>&1; then
@@ -50,20 +53,26 @@ if [ -z "$PROFILE_ROOT" ]; then
             elif command -v dscl >/dev/null 2>&1; then
                 PASSWD_HOME="$(dscl . -read "/Users/$(id -un)" NFSHomeDirectory 2>/dev/null | awk '{print $2}')"
             fi
-            [ -n "$PASSWD_HOME" ] && HOME_PROFILES="${PASSWD_HOME}/.config/dexbot2/profiles"
+            [ -n "$PASSWD_HOME" ] && CONFIG_BASE="${PASSWD_HOME}/.config"
         fi
+        HOME_PROFILES="${CONFIG_BASE:+${CONFIG_BASE}/dexbot2/profiles}"
         # An existing home config is authoritative — the user has migrated.
         if [ -n "$HOME_PROFILES" ] && has_profile_state "$HOME_PROFILES"; then
             PROFILE_ROOT="$HOME_PROFILES"
         else
             # Legacy migration: keep a populated repo/cwd profiles dir until a
             # home config exists. npm packages never fall back into the package
-            # dir. Written as explicit checks (no word-splitting loop) so it
-            # behaves identically under bash, dash, and zsh.
-            case "$PROJECT_ROOT" in
-                *node_modules*) LEGACY_REPO="" ;;
-                *) LEGACY_REPO="${PROJECT_ROOT}/profiles" ;;
-            esac
+            # dir — exact-parent check (basename of dirname), mirroring
+            # isGlobalNpmPackageDir() in modules/paths.ts; a substring match
+            # would also skip legit repos whose path merely contains the word.
+            # Written as explicit checks (no word-splitting loop) so it behaves
+            # identically under bash, dash, and zsh.
+            LEGACY_REPO=""
+            if [ "$(basename "$(dirname "$PROJECT_ROOT")")" = "node_modules" ]; then
+                LEGACY_REPO=""
+            else
+                LEGACY_REPO="${PROJECT_ROOT}/profiles"
+            fi
             PROFILE_ROOT=""
             if [ -n "$LEGACY_REPO" ] && has_profile_state "$LEGACY_REPO"; then
                 PROFILE_ROOT="$LEGACY_REPO"