@game_ryo/lsji 0.1.0

package/docs/docs/core-concepts.md

@@ -0,0 +1,152 @@
+---
+title: Core Concepts
+description: Understand the core architecture of LSJI
+---
+
+# Core Concepts
+
+LSJI is built around four core abstractions that work together to create a flexible reinforcement learning framework.
+
+## Architecture Overview
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                        Agent                                 │
+│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
+│  │ QLearning   │  │  Storage    │  │      Env            │  │
+│  │ (Engine)    │◄─┤ (Backend)   │  │ (Environment)       │  │
+│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### 1. Environment (`Env`)
+
+The `Env` interface defines the problem domain. Any RL environment must implement:
+
+```typescript
+abstract class Env {
+  getState(): string;                    // Current state representation
+  step(action: number): Promise<StepResult>;  // Execute action
+  actionSize(): number;                  // Number of possible actions
+  reset(): Promise<string>;              // Reset to initial state
+}
+```
+
+**StepResult** contains:
+- `state` — New state after action
+- `reward` — Reward received (-1, 0, 1)
+- `done` — Whether episode ended
+- `info` — Additional diagnostic info
+
+### 2. Q-Learning Engine (`QLearning`)
+
+Tabular Q-Learning with Temporal Difference (TD) updates:
+
+```typescript
+class QLearning {
+  constructor({ alpha, gamma, epsilon, storage });
+  
+  // Epsilon-greedy action selection
+  async act(state: string, actionSize: number): Promise<number>;
+  
+  // Full TD update: Q(s,a) ← Q(s,a) + α[r + γ·max Q(s',a') - Q(s,a)]
+  async learn(state, action, reward, nextState, nextActionSize);
+  
+  // Simplified update (terminal states): Q(s,a) ← Q(s,a) + α[r - Q(s,a)]
+  async learnSimple(state, action, reward);
+  
+  // Get all Q-values for inspection
+  async getFullQTable(): Promise<QTableRecord[]>;
+}
+```
+
+**Hyperparameters:**
+- `alpha` (0.1) — Learning rate
+- `gamma` (0.9) — Discount factor  
+- `epsilon` (0.1) — Exploration rate
+
+### 3. Storage Backend (`Storage`)
+
+Pluggable persistence layer with three implementations:
+
+| Backend | Package | Use Case |
+|---------|---------|----------|
+| `SqliteStorage` | `node:sqlite` (built-in) | **Recommended** — Zero dependencies |
+| `BetterSqliteStorage` | `better-sqlite3` | High-performance synchronous access |
+| `MemoryStorage` | Built-in | Testing, CI, ephemeral workloads |
+
+All implement the same interface:
+```typescript
+interface Storage {
+  initialize(): Promise<void>;
+  close(): Promise<void>;
+  getSetting(key): Promise<Setting>;
+  setSetting(key, value): Promise<void>;
+  getQTable(): Promise<QTableRecord[]>;
+  updateQ(state, action, qValue): Promise<void>;
+  addBattle(record): Promise<void>;
+  getTodayBattleCount(): Promise<number>;
+  getPerformanceStats(): Promise<PerformanceStat[]>;
+}
+```
+
+### 4. Agent (`Agent`)
+
+High-level orchestration combining all components:
+
+```typescript
+class Agent {
+  constructor({ qlearning, storage, env });
+  
+  async train({ episodes, actionSelector, batchSize });
+  async play(options?): Promise<PlayResult>;
+  async status(): Promise<StatusInfo>;
+  async start(): Promise<{status, message}>;
+  async stop(): Promise<{status, message}>;
+  setEnvironment(env): void;
+}
+```
+
+## Data Flow
+
+### Training Loop
+```
+for each episode:
+  1. Get current state from Env
+  2. Select action via QLearning.act() (ε-greedy)
+  3. Execute action in Env → StepResult
+  4. Update Q-table via QLearning.learnSimple()
+  5. Persist battle record to Storage
+  6. Batch DB writes for performance
+```
+
+### Play Loop
+```
+1. Get current state from Env
+2. Select best action via QLearning.act() (ε=0 for exploitation)
+3. Execute action in Env
+4. Update Q-table with result
+5. Record battle to Storage
+6. Return result
+```
+
+## Reward System (RPS Example)
+
+| Outcome | Judge Formula | Reward |
+|---------|---------------|--------|
+| Win     | (ai - user + 3) % 3 = 2 | +1 |
+| Lose    | (ai - user + 3) % 3 = 1 | -1 |
+| Draw    | (ai - user + 3) % 3 = 0 | 0 |
+
+## Training Patterns
+
+Built-in patterns for the RPS environment:
+
+| Pattern | ID | Description |
+|---------|-----|-------------|
+| Random | 0 | Uniform random actions |
+| Always Rock | 1 | Always play action 0 |
+| Counter | 2 | Play counter to previous action |
+| Sequential | 3 | Cycle through 0,1,2,0,1,2... |
+
+Custom patterns can be implemented via `actionSelector` function.

package/docs/docs/examples/advanced-training.md

@@ -0,0 +1,244 @@
+---
+title: Advanced Training
+description: Custom training patterns and techniques
+---
+
+# Advanced Training
+
+Learn advanced training techniques for better agent performance.
+
+## Custom Action Selectors
+
+The `train()` method accepts an `actionSelector` function for custom training patterns.
+
+```typescript
+const result = await agent.train({
+  episodes: 1000,
+  actionSelector: (episode, lastAction) => {
+    // Your custom logic here
+    return action;
+  }
+});
+```
+
+### Epsilon-Greedy with Decay
+
+```typescript
+let epsilon = 1.0;
+const minEpsilon = 0.01;
+const decayRate = 0.9995;
+
+const result = await agent.train({
+  episodes: 10000,
+  actionSelector: (episode, lastAction) => {
+    epsilon = Math.max(minEpsilon, epsilon * decayRate);
+    
+    if (Math.random() < epsilon) {
+      return Math.floor(Math.random() * 3); // Explore
+    }
+    
+    // Exploit: use agent's Q-learning
+    const state = await agent.env.getState();
+    return agent.qlearning.act(state, 3);
+  }
+});
+```
+
+### Curriculum Learning
+
+Start with easy opponents, progress to harder ones.
+
+```typescript
+const opponents = ['always_rock', 'sequential', 'counter', 'random'];
+const episodesPerStage = 250;
+
+for (const opponent of opponents) {
+  const env = new RockPaperScissorsEnv({ opponent });
+  agent.setEnvironment(env);
+  
+  console.log(`Training against ${opponent}...`);
+  await agent.train({ episodes: episodesPerStage });
+  
+  const status = await agent.status();
+  console.log(`Win rate: ${status.performance.find(p => p.mode === 'train')?.win_rate}%`);
+}
+```
+
+### Self-Play Training
+
+Train agent against itself.
+
+```typescript
+// Create two agents sharing the same Q-table
+const storage = await createStorage('sqlite', { path: './selfplay.db' });
+const qlearning = new QLearning({ alpha: 0.1, gamma: 0.9, epsilon: 0.1, storage });
+
+const env1 = new RockPaperScissorsEnv({ opponent: 'random' });
+const env2 = new RockPaperScissorsEnv({ opponent: 'random' });
+
+const agent1 = new Agent({ qlearning, storage, env: env1 });
+const agent2 = new Agent({ qlearning, storage, env: env2 });
+
+// Alternate training
+for (let i = 0; i < 100; i++) {
+  await agent1.train({ episodes: 50 });
+  await agent2.train({ episodes: 50 });
+  
+  if (i % 10 === 0) {
+    const status = await agent1.status();
+    console.log(`Iteration ${i}: ${status.performance[0].win_rate}% win rate`);
+  }
+}
+```
+
+## Hyperparameter Tuning
+
+### Grid Search
+
+```typescript
+const configs = [
+  { alpha: 0.05, gamma: 0.9, epsilon: 0.1 },
+  { alpha: 0.1, gamma: 0.9, epsilon: 0.1 },
+  { alpha: 0.2, gamma: 0.9, epsilon: 0.1 },
+  { alpha: 0.1, gamma: 0.95, epsilon: 0.1 },
+  { alpha: 0.1, gamma: 0.9, epsilon: 0.2 },
+];
+
+for (const config of configs) {
+  const storage = await createStorage('memory');
+  const qlearning = new QLearning({ ...config, storage });
+  const env = new RockPaperScissorsEnv({ opponent: 'random' });
+  const agent = new Agent({ qlearning, storage, env });
+  
+  await agent.train({ episodes: 2000 });
+  const status = await agent.status();
+  const winRate = status.performance.find(p => p.mode === 'train')?.win_rate || 0;
+  
+  console.log(`${JSON.stringify(config)} => ${winRate}%`);
+  await storage.close();
+}
+```
+
+### Bayesian Optimization
+
+Use libraries like `bayes-opt` for efficient hyperparameter search.
+
+## Evaluation Techniques
+
+### Fixed Opponent Evaluation
+
+```typescript
+async function evaluate(agent, opponent, games = 100) {
+  const env = new RockPaperScissorsEnv({ opponent });
+  agent.setEnvironment(env);
+  
+  let wins = 0, losses = 0, draws = 0;
+  
+  for (let i = 0; i < games; i++) {
+    const result = await agent.play(Math.floor(Math.random() * 3));
+    if (result.reward > 0) wins++;
+    else if (result.reward < 0) losses++;
+    else draws++;
+  }
+  
+  return { wins, losses, draws, winRate: wins / games };
+}
+
+const agents = {
+  random: await evaluate(agent, 'random'),
+  alwaysRock: await evaluate(agent, 'always_rock'),
+  counter: await evaluate(agent, 'counter'),
+  sequential: await evaluate(agent, 'sequential'),
+};
+
+console.table(agents);
+```
+
+### Cross-Validation
+
+```typescript
+async function crossValidate(config, folds = 5, episodesPerFold = 1000) {
+  const results = [];
+  
+  for (let fold = 0; fold < folds; fold++) {
+    const storage = await createStorage('memory');
+    const qlearning = new QLearning({ ...config, storage });
+    const env = new RockPaperScissorsEnv({ opponent: 'random' });
+    const agent = new Agent({ qlearning, storage, env });
+    
+    await agent.train({ episodes: episodesPerFold });
+    const evalResult = await evaluate(agent, 'random', 200);
+    results.push(evalResult.winRate);
+    
+    await storage.close();
+  }
+  
+  const mean = results.reduce((a, b) => a + b, 0) / results.length;
+  const std = Math.sqrt(results.reduce((a, b) => a + (b - mean) ** 2, 0) / results.length);
+  
+  return { mean, std, results };
+}
+```
+
+## Checkpointing and Resuming
+
+```typescript
+// Save Q-table periodically
+async function trainWithCheckpoints(agent, episodes, checkpointEvery = 100) {
+  for (let i = 0; i < episodes; i += checkpointEvery) {
+    const batch = Math.min(checkpointEvery, episodes - i);
+    await agent.train({ episodes: batch });
+    
+    // Q-table automatically persisted to storage
+    const status = await agent.status();
+    console.log(`Checkpoint ${i + batch}: ${status.aiBrain.length} Q-values`);
+  }
+}
+
+// Resume from existing Q-table
+const storage = await createStorage('sqlite', { path: './existing.db' });
+const qlearning = new QLearning({ alpha: 0.1, gamma: 0.9, epsilon: 0.1, storage });
+// Q-table loads automatically on first use
+```
+
+## Distributed Training
+
+Run multiple training processes with shared storage.
+
+```bash
+# Terminal 1
+lsji train --episodes 500 --db-path ./shared.db --storage sqlite
+
+# Terminal 2 (same database)
+lsji train --episodes 500 --db-path ./shared.db --storage sqlite
+
+# Terminal 3
+lsji train --episodes 500 --db-path ./shared.db --storage sqlite
+```
+
+All processes read/write to the same SQLite database, enabling parallel training.
+
+## Monitoring Training Progress
+
+```typescript
+async function trainWithLogging(agent, episodes) {
+  const history = [];
+  
+  for (let i = 0; i < episodes; i += 100) {
+    await agent.train({ episodes: 100 });
+    
+    const status = await agent.status();
+    const trainStat = status.performance.find(p => p.mode === 'train');
+    
+    history.push({
+      episode: i + 100,
+      winRate: trainStat?.win_rate || 0,
+      qTableSize: status.aiBrain.length
+    });
+    
+    console.log(`Episode ${i + 100}: ${history[history.length - 1].winRate}% win rate`);
+  }
+  
+  return history;
+}
+```

package/docs/docs/examples/custom-environment.md

@@ -0,0 +1,198 @@
+---
+title: Custom Environment
+description: Build your own RL environment
+---
+
+# Custom Environment Example
+
+This guide shows how to create a custom environment by extending the `Env` base class.
+
+## Grid World Environment
+
+A simple 1D grid where the agent learns to move right to reach the goal.
+
+```typescript
+import { Env } from 'lsji';
+
+class GridWorldEnv extends Env {
+  constructor(gridSize = 10) {
+    super();
+    this.gridSize = gridSize;
+    this.position = 0;
+  }
+
+  getState(): string {
+    return String(this.position);
+  }
+
+  async step(action: number): Promise<StepResult> {
+    // Actions: 0 = left, 1 = right
+    if (action === 0) {
+      this.position = Math.max(0, this.position - 1);
+    } else if (action === 1) {
+      this.position = Math.min(this.gridSize - 1, this.position + 1);
+    }
+
+    const done = this.position === this.gridSize - 1;
+    const reward = done ? 1 : -0.01; // Small penalty for each step
+
+    return {
+      state: String(this.position),
+      reward,
+      done,
+      info: { position: this.position }
+    };
+  }
+
+  actionSize(): number {
+    return 2; // Left, Right
+  }
+
+  async reset(): Promise<string> {
+    this.position = 0;
+    return '0';
+  }
+
+  render(): string {
+    const bar = ' '.repeat(this.gridSize);
+    const chars = bar.split('');
+    chars[this.position] = 'A';
+    chars[this.gridSize - 1] = 'G';
+    return `[${chars.join('')}]`;
+  }
+}
+```
+
+## Using the Custom Environment
+
+```typescript
+import { Agent, QLearning, createStorage } from 'lsji';
+import { GridWorldEnv } from './grid-world';
+
+async function main() {
+  const storage = await createStorage('sqlite', { path: './gridworld.db' });
+  
+  const qlearning = new QLearning({
+    alpha: 0.1,
+    gamma: 0.9,
+    epsilon: 0.1,
+    storage
+  });
+
+  const env = new GridWorldEnv(10);
+  const agent = new Agent({ qlearning, storage, env });
+
+  console.log('Training...');
+  await agent.train({ episodes: 5000 });
+
+  console.log('Testing...');
+  await agent.play(); // Single step
+  
+  const status = await agent.status();
+  console.log('Q-table:', status.aiBrain);
+
+  await storage.close();
+}
+
+main().catch(console.error);
+```
+
+## Multi-State Environment
+
+For environments with multiple state variables, use `StateEncoder`:
+
+```typescript
+import { Env, StateEncoder } from 'lsji';
+
+interface GameState {
+  playerHP: number;
+  enemyHP: number;
+  hasPotion: boolean;
+}
+
+class BattleEnv extends Env {
+  state: GameState = { playerHP: 100, enemyHP: 50, hasPotion: true };
+
+  getState(): string {
+    return StateEncoder.encode(this.state);
+  }
+
+  async step(action: number): Promise<StepResult> {
+    // 0 = attack, 1 = heal, 2 = defend
+    let reward = 0;
+    let done = false;
+
+    if (action === 0) { // Attack
+      this.state.enemyHP -= 10;
+      reward = this.state.enemyHP <= 0 ? 10 : -1;
+      done = this.state.enemyHP <= 0;
+    } else if (action === 1) { // Heal
+      if (this.state.hasPotion) {
+        this.state.playerHP = Math.min(100, this.state.playerHP + 30);
+        this.state.hasPotion = false;
+        reward = -1;
+      } else {
+        reward = -5; // No potion penalty
+      }
+    } else if (action === 2) { // Defend
+      reward = -0.5;
+    }
+
+    // Enemy counter-attack
+    if (!done) {
+      this.state.playerHP -= 5;
+      if (this.state.playerHP <= 0) {
+        reward = -10;
+        done = true;
+      }
+    }
+
+    return {
+      state: StateEncoder.encode(this.state),
+      reward,
+      done,
+      info: { ...this.state }
+    };
+  }
+
+  actionSize(): number {
+    return 3;
+  }
+
+  async reset(): Promise<string> {
+    this.state = { playerHP: 100, enemyHP: 50, hasPotion: true };
+    return StateEncoder.encode(this.state);
+  }
+}
+```
+
+## Key Points
+
+1. **State as string** — Use `StateEncoder.encode()` for complex states
+2. **Reward design** — Shape rewards to guide learning (dense vs sparse)
+3. **Action space** — Keep small for tabular Q-learning
+4. **Episode termination** — Always implement `done` condition
+5. **Reset** — Must restore initial state completely
+
+## Testing Your Environment
+
+```typescript
+async function testEnv() {
+  const env = new GridWorldEnv(5);
+  
+  console.log('Initial:', env.getState());
+  console.log(env.render());
+  
+  for (let i = 0; i < 10; i++) {
+    const result = await env.step(1); // Always move right
+    console.log(`Step ${i}: state=${result.state}, reward=${result.reward}, done=${result.done}`);
+    console.log(env.render());
+    if (result.done) break;
+  }
+  
+  await env.reset();
+  console.log('After reset:', env.getState());
+}
+
+testEnv();
+```