backtest-kit 1.3.2 → 1.4.0

package/README.md

@@ -16,7 +16,7 @@ Build sophisticated trading systems with confidence. Backtest Kit empowers you t
 
 - 🚀 **Production-Ready Architecture**: Seamlessly switch between backtest and live modes with robust error recovery and graceful shutdown mechanisms. Your strategy code remains identical across environments. 
 
-- 💾 **Crash-Safe Persistence**: Atomic file writes with automatic state recovery ensure no duplicate signals or lost data—even after crashes. Resume execution exactly where you left off. 🔄
+- 💾 **Crash-Safe Persistence**: Atomic file writes with automatic state recovery ensure no duplicate signals or lost data—even after crashes. Resume execution exactly where you left off.
 
 - ✅ **Signal Validation**: Comprehensive validation prevents invalid trades before execution. Catches price logic errors (TP/SL), throttles signal spam, and ensures data integrity. 🛡️
 
@@ -44,11 +44,13 @@ Build sophisticated trading systems with confidence. Backtest Kit empowers you t
 
 - 💾 **Zero Data Download**: Unlike Freqtrade, no need to download gigabytes of historical data—plug any data source (CCXT, database, API). 🚀
 
-- 🔌 **Pluggable Persistence**: Replace default file-based persistence with custom adapters (Redis, MongoDB, PostgreSQL) for distributed systems and high-performance scenarios. 💾
+- 🔌 **Pluggable Persistence**: Replace default file-based persistence with custom adapters (Redis, MongoDB, PostgreSQL) for distributed systems and high-performance scenarios.
 
 - 🔒 **Safe Math & Robustness**: All metrics protected against NaN/Infinity with unsafe numeric checks. Returns N/A for invalid calculations. ✨
 
-- 🧪 **Comprehensive Test Coverage**: 123 unit and integration tests covering validation, PNL, callbacks, reports, performance tracking, walker, heatmap, position sizing, risk management, scheduled signals, and event system. 
+- 🤖 **AI Strategy Optimizer**: LLM-powered strategy generation from historical data. Train multiple strategy variants, compare performance, and auto-generate executable code. Supports Ollama integration with multi-timeframe analysis. 🧠
+
+- 🧪 **Comprehensive Test Coverage**: Unit and integration tests covering validation, PNL, callbacks, reports, performance tracking, walker, heatmap, position sizing, risk management, scheduled signals, crash recovery, optimizer, and event system. 
 
 ---
 
@@ -249,12 +251,15 @@ Backtest.background("BTCUSDT", {
 - 🌐 **`addFrame`**: Configure timeframes for backtesting. 📅
 - 🔄 **`Backtest` / `Live`**: Run strategies in backtest or live mode (generator or background). ⚡
 - 📅 **`Schedule`**: Track scheduled signals and cancellation rate for limit orders. 📊
+- 📊 **`Partial`**: Access partial profit/loss statistics and reports for risk management. Track signals reaching milestone levels (10%, 20%, 30%, etc.). 💹
+- 🎯 **`Constant`**: Kelly Criterion-based constants for optimal take profit (TP_LEVEL1-3) and stop loss (SL_LEVEL1-2) levels. 📐
 - 🏃 **`Walker`**: Compare multiple strategies in parallel with ranking. 🏆
 - 🔥 **`Heat`**: Portfolio-wide performance analysis across multiple symbols. 📊
 - 💰 **`PositionSize`**: Calculate position sizes with Fixed %, Kelly Criterion, or ATR-based methods. 💵
 - 🛡️ **`addRisk`**: Portfolio-level risk management with custom validation logic. 🔐
-- 💾 **`PersistBase`**: Base class for custom persistence adapters (Redis, MongoDB, PostgreSQL). 🗄️
-- 🔌 **`PersistSignalAdapter` / `PersistRiskAdapter`**: Register custom adapters for signal and risk persistence. 🔄
+- 💾 **`PersistBase`**: Base class for custom persistence adapters (Redis, MongoDB, PostgreSQL).
+- 🔌 **`PersistSignalAdapter` / `PersistScheduleAdapter` / `PersistRiskAdapter` / `PersistPartialAdapter`**: Register custom adapters for signal, scheduled signal, risk, and partial state persistence.
+- 🤖 **`Optimizer`**: AI-powered strategy generation with LLM integration. Auto-generate strategies from historical data and export executable code. 🧠
 
 Check out the sections below for detailed examples! 📚
 
@@ -1071,6 +1076,777 @@ test("Custom Redis adapter works correctly", async ({ pass, fail }) => {
 });
 ```
 
+### 10. Partial Profit/Loss Tracking
+
+Partial Profit/Loss system tracks signal performance at fixed percentage levels (10%, 20%, 30%, etc.) for risk management and position scaling strategies.
+
+#### Understanding Partial Levels
+
+The system automatically monitors profit/loss milestones and emits events when signals reach specific levels:
+
+```typescript
+import {
+  listenPartialProfit,
+  listenPartialLoss,
+  listenPartialProfitOnce,
+  listenPartialLossOnce,
+  Constant
+} from "backtest-kit";
+
+// Listen to all profit levels (10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%)
+listenPartialProfit(({ symbol, signal, price, level, backtest }) => {
+  console.log(`${symbol} profit: ${level}% at ${price}`);
+
+  // Close portions at Kelly-optimized levels
+  if (level === Constant.TP_LEVEL3) {
+    console.log("Close 33% at 25% profit");
+  }
+  if (level === Constant.TP_LEVEL2) {
+    console.log("Close 33% at 50% profit");
+  }
+  if (level === Constant.TP_LEVEL1) {
+    console.log("Close 34% at 100% profit");
+  }
+});
+
+// Listen to all loss levels (10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%)
+listenPartialLoss(({ symbol, signal, price, level, backtest }) => {
+  console.log(`${symbol} loss: -${level}% at ${price}`);
+
+  // Close portions at stop levels
+  if (level === Constant.SL_LEVEL2) {
+    console.log("Close 50% at -50% loss");
+  }
+  if (level === Constant.SL_LEVEL1) {
+    console.log("Close 50% at -100% loss");
+  }
+});
+
+// Listen once to first profit level reached
+listenPartialProfitOnce(
+  () => true, // Accept any profit event
+  ({ symbol, signal, price, level, backtest }) => {
+    console.log(`First profit milestone: ${level}%`);
+  }
+);
+
+// Listen once to first loss level reached
+listenPartialLossOnce(
+  () => true, // Accept any loss event
+  ({ symbol, signal, price, level, backtest }) => {
+    console.log(`First loss milestone: -${level}%`);
+  }
+);
+```
+
+#### Constant Utility - Kelly-Optimized Levels
+
+The `Constant` class provides predefined Kelly Criterion-based levels for optimal position sizing:
+
+```typescript
+import { Constant } from "backtest-kit";
+
+// Take Profit Levels
+console.log(Constant.TP_LEVEL1); // 100% (aggressive target)
+console.log(Constant.TP_LEVEL2); // 50%  (moderate target)
+console.log(Constant.TP_LEVEL3); // 25%  (conservative target)
+
+// Stop Loss Levels
+console.log(Constant.SL_LEVEL1); // 100% (maximum risk)
+console.log(Constant.SL_LEVEL2); // 50%  (standard stop)
+```
+
+**Use Case - Scale Out Strategy:**
+
+```typescript
+// Strategy: Close position in 3 tranches at optimal levels
+listenPartialProfit(({ symbol, signal, price, level, backtest }) => {
+  if (level === Constant.TP_LEVEL3) {
+    // Close 33% at 25% profit (secure early gains)
+    executePartialClose(symbol, signal.id, 0.33);
+  }
+  if (level === Constant.TP_LEVEL2) {
+    // Close 33% at 50% profit (lock in medium gains)
+    executePartialClose(symbol, signal.id, 0.33);
+  }
+  if (level === Constant.TP_LEVEL1) {
+    // Close 34% at 100% profit (maximize winners)
+    executePartialClose(symbol, signal.id, 0.34);
+  }
+});
+```
+
+#### Partial Reports and Statistics
+
+The `Partial` utility provides access to accumulated partial profit/loss data:
+
+```typescript
+import { Partial } from "backtest-kit";
+
+// Get statistical data
+const stats = await Partial.getData("BTCUSDT");
+console.log(stats);
+// Returns:
+// {
+//   totalEvents: 15,          // Total profit/loss events
+//   totalProfit: 10,          // Number of profit events
+//   totalLoss: 5,             // Number of loss events
+//   eventList: [
+//     {
+//       timestamp: 1704370800000,
+//       action: "PROFIT",       // PROFIT or LOSS
+//       symbol: "BTCUSDT",
+//       signalId: "abc123",
+//       position: "LONG",       // or SHORT
+//       level: 10,              // Percentage level reached
+//       price: 51500.00,        // Current price at level
+//       mode: "Backtest"        // or Live
+//     },
+//     // ... more events
+//   ]
+// }
+
+// Generate markdown report
+const markdown = await Partial.getReport("BTCUSDT");
+console.log(markdown);
+
+// Save report to disk (default: ./dump/partial/BTCUSDT.md)
+await Partial.dump("BTCUSDT");
+
+// Custom output path
+await Partial.dump("BTCUSDT", "./reports/partial");
+```
+
+**Partial Report Example:**
+
+```markdown
+# Partial Profit/Loss Report: BTCUSDT
+
+| Action | Symbol | Signal ID | Position | Level % | Current Price | Timestamp | Mode |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| PROFIT | BTCUSDT | abc123 | LONG | +10% | 51500.00000000 USD | 2024-01-15T10:30:00.000Z | Backtest |
+| PROFIT | BTCUSDT | abc123 | LONG | +20% | 53000.00000000 USD | 2024-01-15T11:15:00.000Z | Backtest |
+| LOSS | BTCUSDT | def456 | SHORT | -10% | 51500.00000000 USD | 2024-01-15T14:00:00.000Z | Backtest |
+
+**Total events:** 15
+**Profit events:** 10
+**Loss events:** 5
+```
+
+#### Strategy Callbacks
+
+Partial profit/loss callbacks can also be configured at the strategy level:
+
+```typescript
+import { addStrategy } from "backtest-kit";
+
+addStrategy({
+  strategyName: "my-strategy",
+  interval: "5m",
+  getSignal: async (symbol) => { /* ... */ },
+  callbacks: {
+    onPartialProfit: (symbol, data, currentPrice, revenuePercent, backtest) => {
+      console.log(`Signal ${data.id} at ${revenuePercent.toFixed(2)}% profit`);
+    },
+    onPartialLoss: (symbol, data, currentPrice, lossPercent, backtest) => {
+      console.log(`Signal ${data.id} at ${lossPercent.toFixed(2)}% loss`);
+    },
+  },
+});
+```
+
+#### How Partial Levels Work
+
+**Architecture:**
+
+1. `ClientPartial` - Tracks levels using `Map<signalId, Set<level>>` to prevent duplicates
+2. `ClientStrategy` - Calls `partial.profit()` / `partial.loss()` on every tick
+3. `PartialMarkdownService` - Accumulates events (max 250 per symbol) for reports
+4. State persisted to disk: `./dump/data/partial/{symbol}/levels.json`
+
+**Level Detection:**
+
+```typescript
+// For LONG position at entry price 50000
+// Current price = 55000 → revenue = 10%
+// Levels triggered: 10%
+
+// Current price = 61000 → revenue = 22%
+// Levels triggered: 10%, 20% (only 20% event emitted if 10% already triggered)
+
+// For SHORT position at entry price 50000
+// Current price = 45000 → revenue = 10%
+// Levels triggered: 10%
+```
+
+**Deduplication Guarantee:**
+
+Each level is emitted **exactly once per signal**:
+
+- Uses `Set<level>` to track reached levels
+- Persisted to disk for crash recovery
+- Restored on system restart
+
+**Crash Recovery:**
+
+```typescript
+// Before crash:
+// Signal opened at 50000, reached 10% and 20% profit
+// State: { profitLevels: [10, 20], lossLevels: [] }
+// Persisted to: ./dump/data/partial/BTCUSDT/levels.json
+
+// After restart:
+// State restored from disk
+// Only new levels (30%, 40%, etc.) will emit events
+// 10% and 20% won't fire again
+```
+
+#### Best Practices
+
+1. **Use Constant for Kelly-Optimized Levels** - Don't hardcode profit/loss levels
+2. **Scale Out Gradually** - Close positions in tranches (25%, 50%, 100%)
+3. **Monitor Partial Statistics** - Use `Partial.getData()` to track scaling effectiveness
+4. **Filter Events** - Use `listenPartialProfitOnce` for first-level-only logic
+5. **Combine with Position Sizing** - Scale out inversely to volatility
+
+```typescript
+import { Constant, listenPartialProfit } from "backtest-kit";
+
+// Advanced: Dynamic scaling based on level
+listenPartialProfit(({ symbol, signal, price, level, backtest }) => {
+  const percentToClose =
+    level === Constant.TP_LEVEL3 ? 0.25 : // 25% at first level
+    level === Constant.TP_LEVEL2 ? 0.35 : // 35% at second level
+    level === Constant.TP_LEVEL1 ? 0.40 : // 40% at third level
+    0;
+
+  if (percentToClose > 0) {
+    executePartialClose(symbol, signal.id, percentToClose);
+  }
+});
+```
+
+---
+
+### 11. Scheduled Signal Persistence
+
+The framework includes a separate persistence system for scheduled signals (`PersistScheduleAdapter`) that works independently from pending/active signal persistence (`PersistSignalAdapter`). This separation ensures crash-safe recovery of both signal types.
+
+#### Understanding the Dual Persistence System
+
+The library uses **two independent persistence layers** for signals:
+
+1. **PersistSignalAdapter** - Manages pending/active signals (signals that are already opened or waiting to reach TP/SL)
+2. **PersistScheduleAdapter** - Manages scheduled signals (signals waiting for entry price to activate)
+
+This dual-layer architecture ensures that both signal types can be recovered independently after crashes, with proper callbacks (`onActive` for pending signals, `onSchedule` for scheduled signals).
+
+#### Default Storage Structure
+
+By default, scheduled signals are stored separately from pending signals:
+
+```
+./dump/data/
+  signal/
+    my-strategy/
+      BTCUSDT.json      # Pending/active signal state
+      ETHUSDT.json
+  schedule/
+    my-strategy/
+      BTCUSDT.json      # Scheduled signal state
+      ETHUSDT.json
+```
+
+#### How Scheduled Signal Persistence Works
+
+**During Normal Operation:**
+
+When a strategy generates a scheduled signal (limit order waiting for entry), the framework:
+
+1. Stores the signal to disk using atomic writes: `./dump/data/schedule/{strategyName}/{symbol}.json`
+2. Monitors price movements for activation
+3. When price reaches entry point OR cancellation condition occurs:
+   - Deletes scheduled signal from storage
+   - Optionally creates pending signal in `PersistSignalAdapter`
+
+**After System Crash:**
+
+When the system restarts:
+
+1. Framework checks for stored scheduled signals during initialization
+2. Validates exchange name and strategy name match (security protection)
+3. Restores scheduled signal to memory (`_scheduledSignal`)
+4. Calls `onSchedule()` callback to notify about restored signal
+5. Continues monitoring from where it left off
+
+**Crash Recovery Flow:**
+
+```typescript
+// Before crash:
+// 1. Strategy generates signal with priceOpen = 50000 (current price = 49500)
+// 2. Signal stored to ./dump/data/schedule/my-strategy/BTCUSDT.json
+// 3. System waits for price to reach 50000
+// 4. CRASH OCCURS at current price = 49800
+
+// After restart:
+// 1. System reads ./dump/data/schedule/my-strategy/BTCUSDT.json
+// 2. Validates exchangeName and strategyName
+// 3. Restores signal to _scheduledSignal
+// 4. Calls onSchedule() callback with restored signal
+// 5. Continues monitoring for price = 50000
+// 6. When price reaches 50000, signal activates normally
+```
+
+#### Scheduled Signal Data Structure
+
+```typescript
+interface IScheduledSignalRow {
+  id: string;                    // Unique signal ID
+  position: "long" | "short";
+  priceOpen: number;             // Entry price (trigger price for scheduled signal)
+  priceTakeProfit: number;
+  priceStopLoss: number;
+  minuteEstimatedTime: number;
+  exchangeName: string;          // Used for validation during restore
+  strategyName: string;          // Used for validation during restore
+  timestamp: number;
+  pendingAt: number;
+  scheduledAt: number;
+  symbol: string;
+  _isScheduled: true;            // Marker for scheduled signals
+  note?: string;
+}
+```
+
+#### Integration with ClientStrategy
+
+The `ClientStrategy` class uses `setScheduledSignal()` method to ensure all scheduled signal changes are persisted:
+
+```typescript
+// WRONG - Direct assignment (not persisted)
+this._scheduledSignal = newSignal;
+
+// CORRECT - Using setScheduledSignal() method (persisted)
+await this.setScheduledSignal(newSignal);
+```
+
+**Automatic Persistence Locations:**
+
+All scheduled signal state changes are automatically persisted:
+
+- Signal generation (new scheduled signal created)
+- Signal activation (scheduled → pending transition)
+- Signal cancellation (timeout or stop loss hit before activation)
+- Manual signal clearing
+
+**BACKTEST Mode Exception:**
+
+In backtest mode, persistence is **skipped** for performance reasons:
+
+```typescript
+public async setScheduledSignal(scheduledSignal: IScheduledSignalRow | null) {
+  this._scheduledSignal = scheduledSignal;
+
+  if (this.params.execution.context.backtest) {
+    return; // Skip persistence in backtest mode
+  }
+
+  await PersistScheduleAdapter.writeScheduleData(
+    this._scheduledSignal,
+    this.params.strategyName,
+    this.params.execution.context.symbol
+  );
+}
+```
+
+#### Custom Scheduled Signal Adapters
+
+You can replace file-based scheduled signal persistence with custom adapters (Redis, MongoDB, etc.):
+
+```typescript
+import { PersistScheduleAdapter, PersistBase } from "backtest-kit";
+import Redis from "ioredis";
+
+const redis = new Redis();
+
+class RedisSchedulePersist extends PersistBase {
+  async waitForInit(initial: boolean): Promise<void> {
+    console.log(`Redis scheduled signal persistence initialized for ${this.entityName}`);
+  }
+
+  async readValue<T>(entityId: string | number): Promise<T> {
+    const key = `schedule:${this.entityName}:${entityId}`;
+    const data = await redis.get(key);
+
+    if (!data) {
+      throw new Error(`Scheduled signal ${this.entityName}:${entityId} not found`);
+    }
+
+    return JSON.parse(data) as T;
+  }
+
+  async hasValue(entityId: string | number): Promise<boolean> {
+    const key = `schedule:${this.entityName}:${entityId}`;
+    const exists = await redis.exists(key);
+    return exists === 1;
+  }
+
+  async writeValue<T>(entityId: string | number, entity: T): Promise<void> {
+    const key = `schedule:${this.entityName}:${entityId}`;
+    const serializedData = JSON.stringify(entity);
+    await redis.set(key, serializedData);
+
+    // Optional: Set TTL for scheduled signals (e.g., 24 hours)
+    await redis.expire(key, 86400);
+  }
+
+  async removeValue(entityId: string | number): Promise<void> {
+    const key = `schedule:${this.entityName}:${entityId}`;
+    const result = await redis.del(key);
+
+    if (result === 0) {
+      throw new Error(`Scheduled signal ${this.entityName}:${entityId} not found for deletion`);
+    }
+  }
+
+  async removeAll(): Promise<void> {
+    const pattern = `schedule:${this.entityName}:*`;
+    const keys = await redis.keys(pattern);
+
+    if (keys.length > 0) {
+      await redis.del(...keys);
+    }
+  }
+
+  async *values<T>(): AsyncGenerator<T> {
+    const pattern = `schedule:${this.entityName}:*`;
+    const keys = await redis.keys(pattern);
+
+    keys.sort((a, b) => a.localeCompare(b, undefined, {
+      numeric: true,
+      sensitivity: "base"
+    }));
+
+    for (const key of keys) {
+      const data = await redis.get(key);
+      if (data) {
+        yield JSON.parse(data) as T;
+      }
+    }
+  }
+
+  async *keys(): AsyncGenerator<string> {
+    const pattern = `schedule:${this.entityName}:*`;
+    const keys = await redis.keys(pattern);
+
+    keys.sort((a, b) => a.localeCompare(b, undefined, {
+      numeric: true,
+      sensitivity: "base"
+    }));
+
+    for (const key of keys) {
+      const entityId = key.slice(`schedule:${this.entityName}:`.length);
+      yield entityId;
+    }
+  }
+}
+
+// Register Redis adapter for scheduled signal persistence
+PersistScheduleAdapter.usePersistScheduleAdapter(RedisSchedulePersist);
+```
+
+#### Best Practices
+
+1. **Always use `setScheduledSignal()`** - Never assign `_scheduledSignal` directly (except in `waitForInit` for restoration)
+
+2. **Validate signal metadata** - Always store `exchangeName` and `strategyName` with signals for validation
+
+3. **Handle empty storage gracefully** - Don't crash when `readScheduleData()` returns `null`
+
+4. **Test crash recovery** - Write E2E tests that simulate system crashes and verify restoration
+
+5. **Choose persistence adapter wisely**:
+   - Use file-based for single-instance deployments
+   - Use Redis for distributed systems with multiple instances
+   - Use MongoDB for analytics and complex queries
+
+6. **Monitor persistence operations** - Use callbacks to track storage operations:
+
+```typescript
+addStrategy({
+  strategyName: "my-strategy",
+  interval: "5m",
+  getSignal: async (symbol) => { /* ... */ },
+  callbacks: {
+    onSchedule: (symbol, signal, price, backtest) => {
+      console.log(`Scheduled signal created/restored: ${signal.id}`);
+      // Signal was either:
+      // 1. Newly generated and persisted
+      // 2. Restored from storage after crash
+    },
+    onCancel: (symbol, signal, price, backtest) => {
+      console.log(`Scheduled signal cancelled: ${signal.id}`);
+      // Signal was removed from storage
+    },
+  },
+});
+```
+
+---
+
+## 🤖 AI Strategy Optimizer
+
+The Optimizer uses LLM (Large Language Models) to generate trading strategies from historical data. It automates the process of analyzing backtest results, generating strategy logic, and creating executable code.
+
+### How It Works
+
+1. **Data Collection**: Fetch historical data from multiple sources (backtest results, market data, indicators)
+2. **LLM Training**: Build conversation context with data for each training period
+3. **Strategy Generation**: LLM analyzes patterns and generates strategy logic
+4. **Code Export**: Auto-generate complete executable code with Walker for testing
+
+### Basic Example
+
+```typescript
+import { addOptimizer, Optimizer } from "backtest-kit";
+
+// Register optimizer configuration
+addOptimizer({
+  optimizerName: "btc-optimizer",
+
+  // Training periods (multiple strategies generated)
+  rangeTrain: [
+    {
+      note: "Bull market Q1 2024",
+      startDate: new Date("2024-01-01T00:00:00Z"),
+      endDate: new Date("2024-03-31T23:59:59Z"),
+    },
+    {
+      note: "Consolidation Q2 2024",
+      startDate: new Date("2024-04-01T00:00:00Z"),
+      endDate: new Date("2024-06-30T23:59:59Z"),
+    },
+  ],
+
+  // Testing period (Walker validates strategies)
+  rangeTest: {
+    note: "Validation Q3 2024",
+    startDate: new Date("2024-07-01T00:00:00Z"),
+    endDate: new Date("2024-09-30T23:59:59Z"),
+  },
+
+  // Data sources for strategy generation
+  source: [
+    {
+      name: "backtest-results",
+      fetch: async ({ symbol, startDate, endDate, limit, offset }) => {
+        // Fetch closed signals from your backtest database
+        return await db.getBacktestResults({
+          symbol,
+          startDate,
+          endDate,
+          limit,
+          offset,
+        });
+      },
+    },
+    {
+      name: "market-indicators",
+      fetch: async ({ symbol, startDate, endDate, limit, offset }) => {
+        // Fetch RSI, MACD, volume data, etc.
+        return await db.getIndicators({
+          symbol,
+          startDate,
+          endDate,
+          limit,
+          offset,
+        });
+      },
+    },
+  ],
+
+  // LLM prompt generation from conversation history
+  getPrompt: async (symbol, messages) => {
+    // Analyze messages and create strategy prompt
+    return `
+      Based on the historical data, create a strategy that:
+      - Uses multi-timeframe analysis (1h, 15m, 5m, 1m)
+      - Identifies high-probability entry points
+      - Uses proper risk/reward ratios (min 1.5:1)
+      - Adapts to market conditions
+    `;
+  },
+});
+
+// Generate strategies and export code
+await Optimizer.dump("BTCUSDT", {
+  optimizerName: "btc-optimizer"
+}, "./generated");
+
+// Output: ./generated/btc-optimizer_BTCUSDT.mjs
+```
+
+### Generated Code Structure
+
+The Optimizer auto-generates a complete executable file with:
+
+1. **Imports** - All necessary dependencies (backtest-kit, ollama, ccxt)
+2. **Helper Functions**:
+   - `text()` - LLM text generation for analysis
+   - `json()` - Structured signal generation with schema validation
+   - `dumpJson()` - Debug logging to ./dump/strategy
+3. **Exchange Configuration** - CCXT Binance integration
+4. **Frame Definitions** - Training and testing timeframes
+5. **Strategy Implementations** - One per training range with multi-timeframe analysis
+6. **Walker Setup** - Automatic strategy comparison on test period
+7. **Event Listeners** - Progress tracking and result collection
+
+### Advanced Configuration
+
+#### Custom Message Templates
+
+Override default LLM message formatting:
+
+```typescript
+addOptimizer({
+  optimizerName: "custom-optimizer",
+  rangeTrain: [...],
+  rangeTest: {...},
+  source: [...],
+  getPrompt: async (symbol, messages) => "...",
+
+  // Custom templates
+  template: {
+    getUserMessage: async (symbol, data, sourceName) => {
+      return `Analyze ${sourceName} data for ${symbol}:\n${JSON.stringify(data, null, 2)}`;
+    },
+    getAssistantMessage: async (symbol, data, sourceName) => {
+      return `Data from ${sourceName} analyzed successfully`;
+    },
+  },
+});
+```
+
+#### Lifecycle Callbacks
+
+Monitor optimizer operations:
+
+```typescript
+addOptimizer({
+  optimizerName: "monitored-optimizer",
+  rangeTrain: [...],
+  rangeTest: {...},
+  source: [...],
+  getPrompt: async (symbol, messages) => "...",
+
+  callbacks: {
+    onSourceData: async (symbol, sourceName, data, startDate, endDate) => {
+      console.log(`Fetched ${data.length} rows from ${sourceName}`);
+    },
+    onData: async (symbol, strategyData) => {
+      console.log(`Generated ${strategyData.length} strategies for ${symbol}`);
+    },
+    onCode: async (symbol, code) => {
+      console.log(`Generated ${code.length} bytes of code`);
+    },
+    onDump: async (symbol, filepath) => {
+      console.log(`Saved strategy to ${filepath}`);
+    },
+  },
+});
+```
+
+#### Multiple Data Sources
+
+Combine different data types for comprehensive analysis:
+
+```typescript
+addOptimizer({
+  optimizerName: "multi-source-optimizer",
+  rangeTrain: [...],
+  rangeTest: {...},
+
+  source: [
+    // Source 1: Backtest results
+    {
+      name: "backtest-signals",
+      fetch: async (args) => await getBacktestSignals(args),
+    },
+
+    // Source 2: Market indicators
+    {
+      name: "technical-indicators",
+      fetch: async (args) => await getTechnicalIndicators(args),
+    },
+
+    // Source 3: Volume profile
+    {
+      name: "volume-analysis",
+      fetch: async (args) => await getVolumeProfile(args),
+    },
+
+    // Source 4: Order book depth
+    {
+      name: "order-book",
+      fetch: async (args) => await getOrderBookData(args),
+    },
+  ],
+
+  getPrompt: async (symbol, messages) => {
+    // LLM has full context from all sources
+    return `Create strategy using all available data sources`;
+  },
+});
+```
+
+### API Methods
+
+```typescript
+// Get strategy metadata (no code generation)
+const strategies = await Optimizer.getData("BTCUSDT", {
+  optimizerName: "my-optimizer"
+});
+
+// strategies[0].messages - LLM conversation history
+// strategies[0].strategy - Generated strategy prompt
+
+// Generate executable code
+const code = await Optimizer.getCode("BTCUSDT", {
+  optimizerName: "my-optimizer"
+});
+
+// Save to file
+await Optimizer.dump("BTCUSDT", {
+  optimizerName: "my-optimizer"
+}, "./output"); // Default: "./"
+```
+
+### LLM Integration
+
+The Optimizer uses Ollama for LLM inference:
+
+```bash
+# Set up Ollama API
+export OLLAMA_API_KEY=your-api-key
+
+# Run generated strategy
+node generated/btc-optimizer_BTCUSDT.mjs
+```
+
+Generated strategies use:
+- **Model**: `gpt-oss:20b` (configurable in templates)
+- **Multi-timeframe analysis**: 1h, 15m, 5m, 1m candles
+- **Structured output**: JSON schema with signal validation
+- **Debug logging**: Saves conversations to ./dump/strategy
+
+### Best Practices
+
+1. **Training Periods**: Use 2-4 diverse market conditions (bull, bear, sideways)
+2. **Data Quality**: Ensure data sources have unique IDs for deduplication
+3. **Pagination**: Sources automatically paginated (25 records per request)
+4. **Testing**: Always validate generated strategies on unseen data (rangeTest)
+5. **Monitoring**: Use callbacks to track data fetching and code generation
+
 ---
 
 ## 📐 Architecture Overview
@@ -1273,6 +2049,63 @@ listenSignal((event) => {
 });
 ```
 
+### Listen to Partial Profit/Loss Events
+
+```typescript
+import {
+  listenPartialProfit,
+  listenPartialLoss,
+  listenPartialProfitOnce,
+  listenPartialLossOnce,
+  Constant
+} from "backtest-kit";
+
+// Listen to all profit milestones
+listenPartialProfit(({ symbol, signal, price, level, backtest }) => {
+  console.log(`${symbol} reached ${level}% profit at ${price}`);
+
+  // Scale out at Kelly-optimized levels
+  if (level === Constant.TP_LEVEL3) {
+    console.log("Close 33% at 25% profit");
+  }
+  if (level === Constant.TP_LEVEL2) {
+    console.log("Close 33% at 50% profit");
+  }
+  if (level === Constant.TP_LEVEL1) {
+    console.log("Close 34% at 100% profit");
+  }
+});
+
+// Listen to all loss milestones
+listenPartialLoss(({ symbol, signal, price, level, backtest }) => {
+  console.log(`${symbol} reached -${level}% loss at ${price}`);
+
+  // Scale out at stop levels
+  if (level === Constant.SL_LEVEL2) {
+    console.log("Close 50% at -50% loss");
+  }
+  if (level === Constant.SL_LEVEL1) {
+    console.log("Close 50% at -100% loss");
+  }
+});
+
+// Listen once to first profit level
+listenPartialProfitOnce(
+  () => true, // Accept any profit event
+  ({ symbol, signal, price, level, backtest }) => {
+    console.log(`First profit milestone: ${level}%`);
+  }
+);
+
+// Listen once to first loss level
+listenPartialLossOnce(
+  () => true, // Accept any loss event
+  ({ symbol, signal, price, level, backtest }) => {
+    console.log(`First loss milestone: -${level}%`);
+  }
+);
+```
+
 ### Listen Once with Filter
 
 ```typescript
@@ -1432,11 +2265,13 @@ await setConfig({
 
 ## ✅ Tested & Reliable
 
-`backtest-kit` comes with **123 unit and integration tests** covering:
+`backtest-kit` comes with **217 unit and integration tests** covering:
 
 - Signal validation and throttling
 - PNL calculation with fees and slippage
 - Crash recovery and state persistence
+- Dual-layer persistence (pending signals and scheduled signals)
+- Crash recovery validation (exchange/strategy name mismatch protection)
 - Callback execution order (onSchedule, onOpen, onActive, onClose, onCancel)
 - Markdown report generation (backtest, live, scheduled signals)
 - Walker strategy comparison