dexbot 1.4.16 → 1.4.17

package/analysis/ama_fitting/README.md

@@ -0,0 +1,258 @@
+# AMA Fitting
+
+Tooling to fit AMA parameters (ER, Fast, Slow) against real LP pool candle data.
+The optimizer writes result JSON by default and can explicitly export the chosen
+parameters into `profiles/market_profiles.json` for the market adapter.
+
+---
+
+## Workflow Overview
+
+```
+1. fetch_lp_candles.ts   →   market_adapter/data/lp/<pairFolder>/lp_pool_<poolShort>_<interval>.json
+2. optimizer_high_resolution.ts   →   optimization_results_*.json
+                                   →   profiles/market_profiles.json  (only with --write-profiles)
+3. scripts/generate_lp_chart.ts   →   market chart + comparison chart  (visual review)
+   - LP chart: `npm run lp:chart -- --data <lp-export.json>`
+   - Local LP comparison alias: `npm run ama:chart:lp-local -- --data <lp-export.json>`
+```
+
+---
+
+## Step 1 — Fetch LP Candles
+
+`fetch_lp_candles.ts` fetches bidirectional LP swap data from Kibana and saves
+a full uncut candle file. Uses the same `kibana_source` as the market adapter
+bootstrap (gaps filled via `candle_utils.fillCandleGaps`), but without pruning.
+
+**Known asset details:**
+
+| Asset   | Symbol     | Object ID      | Precision |
+|---------|------------|----------------|-----------|
+| `<ASSET_A>` | `<ASSET_A>` | `<asset_a_id>` | `<n>`     |
+| `<ASSET_B>` | `<ASSET_B>` | `<asset_b_id>` | `<n>`     |
+
+**`<ASSET_A>`/`<ASSET_B>` pool (3 years):**
+```bash
+tsx analysis/ama_fitting/fetch_lp_candles.ts \
+  --pool 1.19.133 \
+  --assetA <ASSET_A> --assetAId <asset_a_id> --assetAPrecision <n> \
+  --assetB <ASSET_B>     --assetBId <asset_b_id>    --assetBPrecision <n> \
+  --hours 26280
+```
+
+Output: `market_adapter/data/lp/<pair_folder>/lp_pool_<poolShort>_<interval>.json` (interval-dependent)
+
+**Options:**
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--pool` | required | Pool ID (e.g. `1.19.133` or just `133`) |
+| `--assetA` | required | Asset A symbol |
+| `--assetAId` | required | Asset A object ID (e.g. `<asset_a_id>`) |
+| `--assetAPrecision` | required | Asset A precision |
+| `--assetB` | required | Asset B symbol |
+| `--assetBId` | required | Asset B object ID |
+| `--assetBPrecision` | required | Asset B precision |
+| `--interval` | `1h` | Candle interval label (`1m`, `5m`, `15m`, `1h`, `4h`, `1d`) |
+| `--hours` | `26280` | Lookback hours (26280 = 3 years) |
+| `--out` | auto | Output filename (default: auto-generated in `market_adapter/data/lp/<assetA>_<assetB>/`; used as-is if absolute) |
+
+---
+
+## Step 2 — Run the Optimizer
+
+`optimizer_high_resolution.ts` runs a parallel geometric grid search over
+ER × Fast × Slow combinations. Produces four AMA winners (AMA1–AMA4) using
+different distance-cap quantiles and writes results to a JSON file + auto-generates
+an interactive HTML chart.
+
+By default this does **not** update runtime market-adapter profiles. Add
+`--write-profiles` when you intentionally want the fitted parameters exported
+to `profiles/market_profiles.json`.
+
+**Run on the fetched LP data:**
+```bash
+tsx analysis/ama_fitting/optimizer_high_resolution.ts \
+  --data market_adapter/data/lp/<pair>/lp_pool_<id>_<interval>.json
+```
+
+**Export winners to the market adapter profile file:**
+```bash
+tsx analysis/ama_fitting/optimizer_high_resolution.ts \
+  --data market_adapter/data/lp/<pair>/lp_pool_<id>_<interval>.json \
+  --write-profiles
+```
+
+Override ranges via CLI:
+```bash
+tsx analysis/ama_fitting/optimizer_high_resolution.ts \
+  --data market_adapter/data/lp/<pair>/lp_pool_<id>_<interval>.json \
+  --erMin 100 --erMax 600 \
+  --slowMin 800 --slowMax 6000
+```
+
+**AMA fitting caps:**
+
+These are the active defaults used by the optimizer:
+
+```bash
+--ama1Cap 0.25  --ama2Cap 0.30  --ama3Cap 0.35  --ama4Cap 0.40
+```
+
+| Key  | Distance cap quantile | Character |
+|------|-----------------------|-----------|
+| AMA1 | 0.25 | Tightest fit, most reactive |
+| AMA2 | 0.30 | Balanced |
+| **AMA3** | 0.35 | Default |
+| AMA4 | 0.40 | Widest fit, most conservative |
+
+**AMA distance weights:**
+
+The distance penalty λ balances movement smoothness against price closeness
+(higher λ = AMA must stay tighter to price). Each AMA has a default weight;
+override any individually:
+
+```bash
+# Defaults (built-in)
+--ama1Weight 0.0031  --ama2Weight 0.0025  --ama3Weight 0.00185  --ama4Weight 0.0013
+
+# Override only AMA1 and AMA4, keeping AMA2/AMA3 defaults
+tsx analysis/ama_fitting/optimizer_high_resolution.ts \
+  --data market_adapter/data/lp/<pair>/lp_pool_<id>_<interval>.json \
+  --ama1Weight 0.003 --ama4Weight 0.002
+```
+
+| Key  | Default λ | Character |
+|------|----------:|-----------|
+| AMA1 | 0.0031 | Heaviest distance penalty — most reactive, stays closest to price |
+| AMA2 | 0.0025 | Moderate penalty |
+| **AMA3** | 0.00185 | Default — balanced move-vs-distance tradeoff |
+| AMA4 | 0.0013 | Lightest distance penalty — allows more room, most conservative |
+
+**Inventory price range guidance:**
+
+Use an inventory range that sits above the fitted cap so the market maker has
+room to absorb normal noise without widening the book too much.
+
+An optimized AMA plus the recommended buffer table is intended to provide a
+relatively safe operating range for extreme market conditions while still
+preserving reasonable inventory turnover.
+
+- Safe buffer: `+10%` to `+15%` above the fitted cap
+- Borderline: `+20%`
+- Overkill: `+25%+`
+
+| AMA  | Fitted cap | Safe inventory range | Borderline | Overkill |
+|------|-----------:|----------------------:|-----------:|---------:|
+| AMA1 | 25% | 35% to 40% | 45% | 50%+ |
+| AMA2 | 30% | 40% to 45% | 50% | 55%+ |
+| **AMA3** | 35% | 45% to 50% | 55% | 60%+ |
+| AMA4 | 40% | 50% to 55% | 60% | 65%+ |
+
+Source: pool 133 `IOB.XRP/BTS`, 1h candles, 2023-04-12 00:00 UTC through
+2026-04-11 23:00 UTC (the working-tree dataset `analysis/ama_fitting/data/lp_pool_133_iob.xrp_bts_1h.json`, regenerated by a local `fetch_lp_candles.ts` run).
+
+**Outputs:**
+- `analysis/ama_fitting/optimization_results_<datafile>_w<λ1>_<λ2>_<λ3>_<λ4>.json` — full results (e.g. `_w0_0031_0_0025_0_00185_0_0013`)
+- `analysis/charts/optimization_chart_<datafile>_w<λ1>_<λ2>_<λ3>_<λ4>.html` — interactive AMA overlay chart (auto-generated)
+- `profiles/market_profiles.json` — updated with new AMA parameters per pair only when `--write-profiles` is used
+
+**Boundary check:** If a winner lands on the edge of the search range, the
+optimizer warns you. Widen the affected range and re-run.
+
+---
+
+## Step 3 — Visual Review
+
+A chart is auto-generated during Step 2 at
+`analysis/charts/optimization_chart_<datafile>_w<λ1>_<λ2>_<λ3>_<λ4>.html`.
+Open it in a browser to compare all four optimized AMA overlays against the
+candlestick price.
+
+For a standalone chart without re-running the optimizer (uses current defaults
+from `modules/constants.ts` rather than optimized results):
+
+```bash
+npm run lp:chart -- \
+  --data market_adapter/data/lp/<pair>/lp_pool_<id>_<interval>.json
+```
+
+This generates both:
+- `analysis/charts/lp_AMA_chart_pool_133.html` — market-adapter style LP chart
+- `analysis/charts/lp_chart_pool_133.comparison.html` — AMA comparison chart (derived from LP metadata)
+
+The unified comparison chart (`*_UNIFIED_COMPARISON.html`) is produced by
+`npm run ama:chart:lp-local` (see workflow overview above).
+
+---
+
+## How market_profiles.json is used
+
+When the optimizer is run with `--write-profiles`, `profiles/market_profiles.json`
+is updated with the new AMA1–AMA4 parameters for the pair. The market adapter
+reads this file at startup and on each cycle via `_resetCycleCache()` /
+`findAmaProfileForBot()` in `market_adapter/market_adapter.ts` (lines 127 and 346 respectively). No restart
+required — takes effect on the next market adapter cycle.
+
+---
+
+## Auxiliary Tools
+
+### `calibrate_convergence_er.ts`
+
+Computes the implied `AMA_CONVERGENCE_ER_AVG` for `modules/constants.ts` from
+real LP candle data. Accounts for Jensen's inequality: the average smoothing
+constant is not the smoothing constant of the average ER.
+
+```bash
+tsx analysis/ama_fitting/calibrate_convergence_er.ts --data <lp-file.json> --amas AMA3
+```
+
+### `analyze_lambda_vs_slow.ts`
+
+Fixes ER and Fast at the AMA1 defaults (781 / 5.2), then scans λ (distance weight)
+over a range to find the optimal Slow period for each λ. Produces a 3-panel chart:
+λ→Slow, λ→Movement, and Slow→Movement (all cached slow values from 10 to maxSlow).
+
+Useful for understanding how λ shapes the optimal Slow independent of ER/Fast
+variation. The annotations on the λ→Slow chart mark where the four default AMA λ
+values land on the curve — differences of ±1–2 slow units vs the 3-D optimizer
+are expected since the optimizer also tunes ER and Fast simultaneously.
+
+```bash
+tsx analysis/ama_fitting/analyze_lambda_vs_slow.ts \
+  --data <lp-file.json> --maxSlow 250 --lambdaEnd 0.0045 --lambdaSteps 50
+```
+
+### `analyze_ama_price_changes.ts`
+
+Simulates `AMA_DELTA_THRESHOLD_PERCENT` grid-reposition frequency for all four
+AMA series on LP candle data. Reports reposition counts and inter-reposition
+step distributions.
+
+```bash
+tsx analysis/ama_fitting/analyze_ama_price_changes.ts \
+  --data <lp-file.json> --results <optimization-results.json>
+```
+
+---
+
+## Data file format
+
+```json
+{
+  "meta": {
+    "pool": "1.19.133",
+    "assetA": { "id": "<asset_a_id>", "precision": <n>, "symbol": "<ASSET_A>" },
+    "assetB": { "id": "<asset_b_id>",    "precision": <n>, "symbol": "<ASSET_B>" },
+    "intervalSeconds": 3600,
+    "lookbackHours": 26280,
+    "candleCount": 26280
+  },
+  "candles": [
+    [timestamp_ms, open, high, low, close, volume_A],
+    ...
+  ]
+}
+```

package/analysis/ama_fitting/analyze_ama_price_changes.ts

@@ -0,0 +1,151 @@
+#!/usr/bin/env node
+import path from 'node:path';
+import { calculateAMA } from '../../market_adapter/core/strategies/ama.js';
+import { getStorage } from '../../modules/storage/index.js';
+const { readJSON } = getStorage();
+import { MARKET_ADAPTER } from '../../modules/constants.js';
+
+'use strict';
+/**
+ * AMA REPOSITION FREQUENCY ANALYSIS
+ *
+ * Simulates AMA_DELTA_THRESHOLD_PERCENT grid-reposition logic for all four AMA
+ * series on LP candle data.
+ *
+ * Uses the production threshold from MARKET_ADAPTER.AMA_DELTA_THRESHOLD_PERCENT
+ * (modules/constants.ts).
+ *
+ * Usage:
+ *   tsx analysis/ama_fitting/analyze_ama_price_changes.ts --data <path-to-lp-candles.json> --results <path-to-optimization-results.json>
+ */
+const REPOS_THRESHOLD_PCT = MARKET_ADAPTER.AMA_DELTA_THRESHOLD_PERCENT;
+
+/**
+ * Simulate AMA_DELTA_THRESHOLD_PERCENT reposition logic.
+ *
+ * Starting from the first post-warmup AMA value, track cumulative drift from
+ * the last reposition baseline.  When drift ≥ threshold, a reposition fires:
+ * record step-count since previous reposition, reset baseline to current AMA.
+ */
+function trackRepositions(amaValues: any, thresholdPct: any, warmup: any) {
+    let events = 0;
+    const steps: number[] = [];
+    let baseline = amaValues[warmup];
+    let stepCounter = 0;
+    for (let i = warmup + 1; i < amaValues.length; i++) {
+        const curr = amaValues[i];
+        if (baseline === 0) continue;
+        stepCounter++;
+        const driftPct = Math.abs((curr - baseline) / baseline) * 100;
+        if (driftPct >= thresholdPct) {
+            events++;
+            steps.push(stepCounter);
+            baseline = curr;
+            stepCounter = 0;
+        }
+    }
+    const min = steps.length > 0 ? Math.min(...steps) : 0;
+    const max = steps.length > 0 ? Math.max(...steps) : 0;
+    const avg = steps.length > 0 ? steps.reduce((a, b) => a + b, 0) / steps.length : 0;
+    return { events, steps, min, max, avg };
+}
+
+// ── Load data ─────────────────────────────────────────────────────────────────
+function loadData(filePath: any) {
+    const json    = readJSON(filePath);
+    const candles = json.candles ?? json;
+    return {
+        candles: candles.map((c: any) => ({ timestamp: c[0], close: c[4] })),
+        meta: json.meta ?? null,
+    };
+}
+function loadAmaParams(resultsPath: any) {
+    const json = readJSON(resultsPath);
+    const amas = json.meta?.amas;
+    if (!amas) throw new Error('No amas found in results file');
+    return [
+        { key: 'AMA1', label: amas.AMA1.label, er: amas.AMA1.er, fast: amas.AMA1.fast, slow: amas.AMA1.slow },
+        { key: 'AMA2', label: amas.AMA2.label, er: amas.AMA2.er, fast: amas.AMA2.fast, slow: amas.AMA2.slow },
+        { key: 'AMA3', label: amas.AMA3.label, er: amas.AMA3.er, fast: amas.AMA3.fast, slow: amas.AMA3.slow },
+        { key: 'AMA4', label: amas.AMA4.label, er: amas.AMA4.er, fast: amas.AMA4.fast, slow: amas.AMA4.slow },
+    ];
+}
+// ── Analysis ──────────────────────────────────────────────────────────────────
+
+// ── Main ──────────────────────────────────────────────────────────────────────
+function run() {
+    const dataArgIdx    = process.argv.indexOf('--data');
+    const resultsArgIdx = process.argv.indexOf('--results');
+    if (dataArgIdx === -1) {
+        throw new Error('--data <path-to-lp-candles.json> is required');
+    }
+    if (resultsArgIdx === -1) {
+        throw new Error('--results <path-to-optimization-results.json> is required');
+    }
+    const dataFile    = path.resolve(process.argv[dataArgIdx + 1]);
+    const resultsFile = path.resolve(process.argv[resultsArgIdx + 1]);
+    const { candles, meta } = loadData(dataFile);
+    const amaParams         = loadAmaParams(resultsFile);
+    const closes            = candles.map((c: any) => c.close);
+    const totalSteps        = closes.length - 1; // candle-to-candle transitions
+    const label = meta?.pool
+        ? `LP Pool ${meta.pool}`
+        : path.basename(dataFile, '.json');
+    const interval = meta?.intervalSeconds
+        ? `${meta.intervalSeconds / 3600}h`
+        : '?h';
+    console.log('');
+    console.log('════════════════════════════════════════════════════════════════════════════════');
+    console.log(' AMA Reposition Frequency Analysis  (AMA_DELTA_THRESHOLD_PERCENT simulation)');
+    console.log('════════════════════════════════════════════════════════════════════════════════');
+    console.log(` Dataset:    ${label}  (${interval} candles)`);
+    console.log(` Candles:    ${candles.length}  →  ${totalSteps} steps total`);
+    console.log(` Threshold:   ${REPOS_THRESHOLD_PCT}%  (from MARKET_ADAPTER.AMA_DELTA_THRESHOLD_PERCENT)`);
+    console.log('');
+    console.log(' Logic: set baseline at warmup end, count steps until AMA drifts ≥ threshold,');
+    console.log('        record reposition + reset baseline.  Repeat for full live window.');
+    console.log('');
+    const results: any[] = [];
+    for (const params of amaParams) {
+        const values = calculateAMA(closes, {
+            erPeriod:   params.er,
+            fastPeriod: params.fast,
+            slowPeriod: params.slow,
+        });
+        const r = trackRepositions(values, REPOS_THRESHOLD_PCT, params.er);
+        const liveSteps = totalSteps - params.er;
+        const freq = liveSteps > 0 ? (r.events / liveSteps * 1000).toFixed(1) : '–';
+        const avg  = r.events > 0 ? r.avg.toFixed(1) : '–';
+        const suffix = String(params.label).replace(/^AMA\d\s*/i, '').replace(/^[-:\s]+/, '').trim();
+        const rowLabel = `${params.key} ${suffix}`;
+        console.log(` ── ${rowLabel}  (warmup: ${params.er}  live: ${liveSteps} steps) ──`);
+        console.log(`    ≥${REPOS_THRESHOLD_PCT}%  repositions: ${r.events}  avg steps: ${avg}  min: ${r.events > 0 ? r.min : '–'}  max: ${r.events > 0 ? r.max : '–'}  freq: ${freq}/1k steps`);
+        console.log('');
+        results.push({
+            label: rowLabel,
+            events: r.events,
+            avg: r.avg,
+            freq: liveSteps > 0 ? r.events / liveSteps * 1000 : 0,
+        });
+    }
+    console.log(' Note: warmup candles excluded (AMA initializes from SMA of the ER window).');
+    console.log('');
+    // ── Ranking: fewest repositions ────────────────────────────────────────────
+    results.sort((a, b) => a.events - b.events);
+    console.log(' Ranking — fewest repositions (least grid changes):');
+    console.log('');
+    console.log('    ' + '#'.padEnd(4) + 'AMA'.padEnd(20) + 'repositions'.padStart(12) + 'avg steps'.padStart(10) + ' /1000 steps');
+    console.log('    ' + '─'.repeat(4 + 20 + 12 + 10 + 12));
+    results.forEach((r, i) => {
+        console.log(
+            '    ' +
+            String(i + 1).padEnd(4) +
+            r.label.padEnd(20) +
+            String(r.events).padStart(12) +
+            r.avg.toFixed(1).padStart(10) +
+            `  ${r.freq.toFixed(1)}`
+        );
+    });
+    console.log('');
+}
+run();