@fastino-ai/pioneer-cli 0.2.6 → 0.2.8

package/alphago_reproduction.ipynb

@@ -1,902 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Reproducing AlphaGo: A Comprehensive Guide\n",
-    "\n",
-    "This notebook provides a complete implementation of the key concepts behind AlphaGo, the AI that defeated world champion Go players.\n",
-    "\n",
-    "## Table of Contents\n",
-    "1. Introduction to AlphaGo\n",
-    "2. Game Environment Setup\n",
-    "3. Neural Network Architecture\n",
-    "4. Monte Carlo Tree Search (MCTS)\n",
-    "5. Self-Play Training\n",
-    "6. Evaluation and Testing\n",
-    "\n",
-    "## Key Components of AlphaGo:\n",
-    "- **Policy Network**: Predicts the best move given a board position\n",
-    "- **Value Network**: Evaluates how good a position is\n",
-    "- **Monte Carlo Tree Search**: Combines neural networks with tree search\n",
-    "- **Self-Play**: The system improves by playing against itself"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 1. Installation and Imports"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Install required packages\n",
-    "!pip install torch torchvision numpy matplotlib tqdm"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import torch\n",
-    "import torch.nn as nn\n",
-    "import torch.nn.functional as F\n",
-    "import torch.optim as optim\n",
-    "import numpy as np\n",
-    "import matplotlib.pyplot as plt\n",
-    "from collections import deque, defaultdict\n",
-    "from tqdm import tqdm\n",
-    "import random\n",
-    "import math\n",
-    "from copy import deepcopy\n",
-    "\n",
-    "# Set random seeds for reproducibility\n",
-    "torch.manual_seed(42)\n",
-    "np.random.seed(42)\n",
-    "random.seed(42)\n",
-    "\n",
-    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
-    "print(f\"Using device: {device}\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 2. Go Game Environment\n",
-    "\n",
-    "We'll implement a simplified Go game (9x9 board instead of 19x19 for faster training)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "class GoGame:\n",
-    "    \"\"\"Simplified Go game implementation\"\"\"\n",
-    "    \n",
-    "    def __init__(self, board_size=9):\n",
-    "        self.board_size = board_size\n",
-    "        self.board = np.zeros((board_size, board_size), dtype=np.int8)\n",
-    "        self.current_player = 1  # 1 for black, -1 for white\n",
-    "        self.history = []\n",
-    "        self.ko = None  # Ko rule: prevent immediate recapture\n",
-    "        self.passes = 0\n",
-    "        \n",
-    "    def reset(self):\n",
-    "        \"\"\"Reset the game to initial state\"\"\"\n",
-    "        self.board = np.zeros((self.board_size, self.board_size), dtype=np.int8)\n",
-    "        self.current_player = 1\n",
-    "        self.history = []\n",
-    "        self.ko = None\n",
-    "        self.passes = 0\n",
-    "        return self.get_state()\n",
-    "    \n",
-    "    def get_state(self):\n",
-    "        \"\"\"Get current board state as neural network input\"\"\"\n",
-    "        # Create feature planes:\n",
-    "        # 1. Current player stones\n",
-    "        # 2. Opponent stones\n",
-    "        # 3. Empty positions\n",
-    "        # 4. Current player color (all 1s or all 0s)\n",
-    "        state = np.zeros((4, self.board_size, self.board_size), dtype=np.float32)\n",
-    "        state[0] = (self.board == self.current_player).astype(np.float32)\n",
-    "        state[1] = (self.board == -self.current_player).astype(np.float32)\n",
-    "        state[2] = (self.board == 0).astype(np.float32)\n",
-    "        state[3] = np.ones((self.board_size, self.board_size)) * (self.current_player == 1)\n",
-    "        return state\n",
-    "    \n",
-    "    def get_valid_moves(self):\n",
-    "        \"\"\"Return list of valid moves (including pass)\"\"\"\n",
-    "        valid = []\n",
-    "        for i in range(self.board_size):\n",
-    "            for j in range(self.board_size):\n",
-    "                if self.is_valid_move(i, j):\n",
-    "                    valid.append(i * self.board_size + j)\n",
-    "        # Add pass move\n",
-    "        valid.append(self.board_size * self.board_size)\n",
-    "        return valid\n",
-    "    \n",
-    "    def is_valid_move(self, row, col):\n",
-    "        \"\"\"Check if a move is valid\"\"\"\n",
-    "        if self.board[row, col] != 0:\n",
-    "            return False\n",
-    "        \n",
-    "        # Check ko rule\n",
-    "        if self.ko == (row, col):\n",
-    "            return False\n",
-    "        \n",
-    "        # Simplified: allow all empty positions (full Go rules are complex)\n",
-    "        return True\n",
-    "    \n",
-    "    def make_move(self, action):\n",
-    "        \"\"\"Make a move and return (next_state, reward, done)\"\"\"\n",
-    "        # Pass move\n",
-    "        if action == self.board_size * self.board_size:\n",
-    "            self.passes += 1\n",
-    "            self.current_player *= -1\n",
-    "            done = self.passes >= 2\n",
-    "            reward = self.get_winner() if done else 0\n",
-    "            return self.get_state(), reward, done\n",
-    "        \n",
-    "        self.passes = 0\n",
-    "        row = action // self.board_size\n",
-    "        col = action % self.board_size\n",
-    "        \n",
-    "        if not self.is_valid_move(row, col):\n",
-    "            # Invalid move - penalize\n",
-    "            return self.get_state(), -1, True\n",
-    "        \n",
-    "        # Place stone\n",
-    "        self.board[row, col] = self.current_player\n",
-    "        \n",
-    "        # Remove captured stones (simplified)\n",
-    "        self._remove_captured_stones(-self.current_player)\n",
-    "        \n",
-    "        # Switch player\n",
-    "        self.current_player *= -1\n",
-    "        \n",
-    "        return self.get_state(), 0, False\n",
-    "    \n",
-    "    def _remove_captured_stones(self, color):\n",
-    "        \"\"\"Remove captured stones (simplified capture logic)\"\"\"\n",
-    "        for i in range(self.board_size):\n",
-    "            for j in range(self.board_size):\n",
-    "                if self.board[i, j] == color:\n",
-    "                    if not self._has_liberties(i, j):\n",
-    "                        self._remove_group(i, j)\n",
-    "    \n",
-    "    def _has_liberties(self, row, col):\n",
-    "        \"\"\"Check if a stone/group has liberties (empty adjacent points)\"\"\"\n",
-    "        color = self.board[row, col]\n",
-    "        visited = set()\n",
-    "        stack = [(row, col)]\n",
-    "        \n",
-    "        while stack:\n",
-    "            r, c = stack.pop()\n",
-    "            if (r, c) in visited:\n",
-    "                continue\n",
-    "            visited.add((r, c))\n",
-    "            \n",
-    "            for dr, dc in [(0, 1), (1, 0), (0, -1), (-1, 0)]:\n",
-    "                nr, nc = r + dr, c + dc\n",
-    "                if 0 <= nr < self.board_size and 0 <= nc < self.board_size:\n",
-    "                    if self.board[nr, nc] == 0:\n",
-    "                        return True\n",
-    "                    if self.board[nr, nc] == color and (nr, nc) not in visited:\n",
-    "                        stack.append((nr, nc))\n",
-    "        \n",
-    "        return False\n",
-    "    \n",
-    "    def _remove_group(self, row, col):\n",
-    "        \"\"\"Remove a group of stones\"\"\"\n",
-    "        color = self.board[row, col]\n",
-    "        stack = [(row, col)]\n",
-    "        \n",
-    "        while stack:\n",
-    "            r, c = stack.pop()\n",
-    "            if self.board[r, c] != color:\n",
-    "                continue\n",
-    "            self.board[r, c] = 0\n",
-    "            \n",
-    "            for dr, dc in [(0, 1), (1, 0), (0, -1), (-1, 0)]:\n",
-    "                nr, nc = r + dr, c + dc\n",
-    "                if 0 <= nr < self.board_size and 0 <= nc < self.board_size:\n",
-    "                    if self.board[nr, nc] == color:\n",
-    "                        stack.append((nr, nc))\n",
-    "    \n",
-    "    def get_winner(self):\n",
-    "        \"\"\"Calculate winner (simplified scoring)\"\"\"\n",
-    "        black_score = np.sum(self.board == 1)\n",
-    "        white_score = np.sum(self.board == -1)\n",
-    "        \n",
-    "        if black_score > white_score:\n",
-    "            return 1\n",
-    "        elif white_score > black_score:\n",
-    "            return -1\n",
-    "        else:\n",
-    "            return 0\n",
-    "    \n",
-    "    def render(self):\n",
-    "        \"\"\"Display the board\"\"\"\n",
-    "        symbols = {0: '.', 1: 'X', -1: 'O'}\n",
-    "        print('\\n  ' + ' '.join(str(i) for i in range(self.board_size)))\n",
-    "        for i in range(self.board_size):\n",
-    "            print(f\"{i} \" + ' '.join(symbols[self.board[i, j]] for j in range(self.board_size)))\n",
-    "        print()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 3. Neural Network Architecture\n",
-    "\n",
-    "AlphaGo uses deep convolutional neural networks with two heads:\n",
-    "- **Policy Head**: Outputs probability distribution over moves\n",
-    "- **Value Head**: Outputs win probability for current player"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "class ResidualBlock(nn.Module):\n",
-    "    \"\"\"Residual block for deep network\"\"\"\n",
-    "    \n",
-    "    def __init__(self, channels):\n",
-    "        super(ResidualBlock, self).__init__()\n",
-    "        self.conv1 = nn.Conv2d(channels, channels, kernel_size=3, padding=1)\n",
-    "        self.bn1 = nn.BatchNorm2d(channels)\n",
-    "        self.conv2 = nn.Conv2d(channels, channels, kernel_size=3, padding=1)\n",
-    "        self.bn2 = nn.BatchNorm2d(channels)\n",
-    "    \n",
-    "    def forward(self, x):\n",
-    "        residual = x\n",
-    "        out = F.relu(self.bn1(self.conv1(x)))\n",
-    "        out = self.bn2(self.conv2(out))\n",
-    "        out += residual\n",
-    "        out = F.relu(out)\n",
-    "        return out\n",
-    "\n",
-    "\n",
-    "class AlphaGoNetwork(nn.Module):\n",
-    "    \"\"\"AlphaGo-style neural network with policy and value heads\"\"\"\n",
-    "    \n",
-    "    def __init__(self, board_size=9, num_channels=128, num_res_blocks=5):\n",
-    "        super(AlphaGoNetwork, self).__init__()\n",
-    "        self.board_size = board_size\n",
-    "        \n",
-    "        # Initial convolution\n",
-    "        self.conv_input = nn.Conv2d(4, num_channels, kernel_size=3, padding=1)\n",
-    "        self.bn_input = nn.BatchNorm2d(num_channels)\n",
-    "        \n",
-    "        # Residual tower\n",
-    "        self.res_blocks = nn.ModuleList([\n",
-    "            ResidualBlock(num_channels) for _ in range(num_res_blocks)\n",
-    "        ])\n",
-    "        \n",
-    "        # Policy head\n",
-    "        self.policy_conv = nn.Conv2d(num_channels, 2, kernel_size=1)\n",
-    "        self.policy_bn = nn.BatchNorm2d(2)\n",
-    "        self.policy_fc = nn.Linear(2 * board_size * board_size, board_size * board_size + 1)\n",
-    "        \n",
-    "        # Value head\n",
-    "        self.value_conv = nn.Conv2d(num_channels, 1, kernel_size=1)\n",
-    "        self.value_bn = nn.BatchNorm2d(1)\n",
-    "        self.value_fc1 = nn.Linear(board_size * board_size, 256)\n",
-    "        self.value_fc2 = nn.Linear(256, 1)\n",
-    "    \n",
-    "    def forward(self, x):\n",
-    "        # Input convolution\n",
-    "        x = F.relu(self.bn_input(self.conv_input(x)))\n",
-    "        \n",
-    "        # Residual tower\n",
-    "        for block in self.res_blocks:\n",
-    "            x = block(x)\n",
-    "        \n",
-    "        # Policy head\n",
-    "        policy = F.relu(self.policy_bn(self.policy_conv(x)))\n",
-    "        policy = policy.view(policy.size(0), -1)\n",
-    "        policy = self.policy_fc(policy)\n",
-    "        policy = F.log_softmax(policy, dim=1)\n",
-    "        \n",
-    "        # Value head\n",
-    "        value = F.relu(self.value_bn(self.value_conv(x)))\n",
-    "        value = value.view(value.size(0), -1)\n",
-    "        value = F.relu(self.value_fc1(value))\n",
-    "        value = torch.tanh(self.value_fc2(value))\n",
-    "        \n",
-    "        return policy, value\n",
-    "\n",
-    "\n",
-    "# Create network\n",
-    "network = AlphaGoNetwork(board_size=9).to(device)\n",
-    "print(f\"Network parameters: {sum(p.numel() for p in network.parameters()):,}\")\n",
-    "print(network)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 4. Monte Carlo Tree Search (MCTS)\n",
-    "\n",
-    "MCTS is the search algorithm that combines neural network predictions with tree search to select moves."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "class MCTSNode:\n",
-    "    \"\"\"Node in the Monte Carlo Tree Search\"\"\"\n",
-    "    \n",
-    "    def __init__(self, state, parent=None, action=None, prior=0):\n",
-    "        self.state = state\n",
-    "        self.parent = parent\n",
-    "        self.action = action\n",
-    "        self.prior = prior\n",
-    "        \n",
-    "        self.children = {}\n",
-    "        self.visit_count = 0\n",
-    "        self.value_sum = 0\n",
-    "    \n",
-    "    def value(self):\n",
-    "        \"\"\"Average value of this node\"\"\"\n",
-    "        if self.visit_count == 0:\n",
-    "            return 0\n",
-    "        return self.value_sum / self.visit_count\n",
-    "    \n",
-    "    def is_expanded(self):\n",
-    "        return len(self.children) > 0\n",
-    "    \n",
-    "    def select_child(self, c_puct=1.0):\n",
-    "        \"\"\"Select child with highest UCB score\"\"\"\n",
-    "        best_score = -float('inf')\n",
-    "        best_child = None\n",
-    "        \n",
-    "        for child in self.children.values():\n",
-    "            # UCB formula: Q(s,a) + c_puct * P(s,a) * sqrt(N(s)) / (1 + N(s,a))\n",
-    "            q_value = child.value()\n",
-    "            u_value = c_puct * child.prior * math.sqrt(self.visit_count) / (1 + child.visit_count)\n",
-    "            score = q_value + u_value\n",
-    "            \n",
-    "            if score > best_score:\n",
-    "                best_score = score\n",
-    "                best_child = child\n",
-    "        \n",
-    "        return best_child\n",
-    "\n",
-    "\n",
-    "class MCTS:\n",
-    "    \"\"\"Monte Carlo Tree Search\"\"\"\n",
-    "    \n",
-    "    def __init__(self, network, game, num_simulations=100, c_puct=1.0):\n",
-    "        self.network = network\n",
-    "        self.game = game\n",
-    "        self.num_simulations = num_simulations\n",
-    "        self.c_puct = c_puct\n",
-    "    \n",
-    "    def search(self, state):\n",
-    "        \"\"\"Run MCTS and return action probabilities\"\"\"\n",
-    "        root = MCTSNode(state)\n",
-    "        \n",
-    "        # Expand root\n",
-    "        self._expand(root)\n",
-    "        \n",
-    "        # Run simulations\n",
-    "        for _ in range(self.num_simulations):\n",
-    "            node = root\n",
-    "            search_path = [node]\n",
-    "            \n",
-    "            # Selection: traverse tree to leaf\n",
-    "            while node.is_expanded():\n",
-    "                node = node.select_child(self.c_puct)\n",
-    "                search_path.append(node)\n",
-    "            \n",
-    "            # Expansion and evaluation\n",
-    "            value = self._expand(node)\n",
-    "            \n",
-    "            # Backpropagation\n",
-    "            self._backpropagate(search_path, value)\n",
-    "        \n",
-    "        # Return action probabilities based on visit counts\n",
-    "        action_probs = np.zeros(self.game.board_size * self.game.board_size + 1)\n",
-    "        for action, child in root.children.items():\n",
-    "            action_probs[action] = child.visit_count\n",
-    "        \n",
-    "        action_probs = action_probs / np.sum(action_probs)\n",
-    "        return action_probs\n",
-    "    \n",
-    "    def _expand(self, node):\n",
-    "        \"\"\"Expand node and return value estimate\"\"\"\n",
-    "        # Get network predictions\n",
-    "        state_tensor = torch.FloatTensor(node.state).unsqueeze(0).to(device)\n",
-    "        \n",
-    "        with torch.no_grad():\n",
-    "            policy_logits, value = self.network(state_tensor)\n",
-    "            policy = torch.exp(policy_logits).cpu().numpy()[0]\n",
-    "            value = value.cpu().item()\n",
-    "        \n",
-    "        # Get valid moves\n",
-    "        valid_moves = self.game.get_valid_moves()\n",
-    "        \n",
-    "        # Mask invalid moves\n",
-    "        policy_masked = np.zeros_like(policy)\n",
-    "        policy_masked[valid_moves] = policy[valid_moves]\n",
-    "        policy_sum = np.sum(policy_masked)\n",
-    "        \n",
-    "        if policy_sum > 0:\n",
-    "            policy_masked /= policy_sum\n",
-    "        else:\n",
-    "            # Uniform distribution if all masked\n",
-    "            policy_masked[valid_moves] = 1.0 / len(valid_moves)\n",
-    "        \n",
-    "        # Create child nodes\n",
-    "        for action in valid_moves:\n",
-    "            if action not in node.children:\n",
-    "                # Simulate action to get next state\n",
-    "                game_copy = deepcopy(self.game)\n",
-    "                next_state, _, _ = game_copy.make_move(action)\n",
-    "                \n",
-    "                node.children[action] = MCTSNode(\n",
-    "                    state=next_state,\n",
-    "                    parent=node,\n",
-    "                    action=action,\n",
-    "                    prior=policy_masked[action]\n",
-    "                )\n",
-    "        \n",
-    "        return value\n",
-    "    \n",
-    "    def _backpropagate(self, search_path, value):\n",
-    "        \"\"\"Backpropagate value through search path\"\"\"\n",
-    "        for node in reversed(search_path):\n",
-    "            node.value_sum += value\n",
-    "            node.visit_count += 1\n",
-    "            value = -value  # Flip value for opponent"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 5. Self-Play Training\n",
-    "\n",
-    "AlphaGo improves through self-play: the network plays against itself and learns from the games."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "class SelfPlayTrainer:\n",
-    "    \"\"\"Self-play training for AlphaGo\"\"\"\n",
-    "    \n",
-    "    def __init__(self, network, board_size=9, num_simulations=50):\n",
-    "        self.network = network\n",
-    "        self.board_size = board_size\n",
-    "        self.num_simulations = num_simulations\n",
-    "        self.optimizer = optim.Adam(network.parameters(), lr=0.001)\n",
-    "        self.replay_buffer = deque(maxlen=10000)\n",
-    "    \n",
-    "    def self_play_game(self):\n",
-    "        \"\"\"Play one game against itself\"\"\"\n",
-    "        game = GoGame(self.board_size)\n",
-    "        mcts = MCTS(self.network, game, self.num_simulations)\n",
-    "        \n",
-    "        states = []\n",
-    "        action_probs = []\n",
-    "        players = []\n",
-    "        \n",
-    "        state = game.reset()\n",
-    "        done = False\n",
-    "        \n",
-    "        while not done:\n",
-    "            # Store current state and player\n",
-    "            states.append(state.copy())\n",
-    "            players.append(game.current_player)\n",
-    "            \n",
-    "            # Run MCTS to get action probabilities\n",
-    "            probs = mcts.search(state)\n",
-    "            action_probs.append(probs)\n",
-    "            \n",
-    "            # Sample action from probabilities (with temperature)\n",
-    "            action = np.random.choice(len(probs), p=probs)\n",
-    "            \n",
-    "            # Make move\n",
-    "            state, reward, done = game.make_move(action)\n",
-    "        \n",
-    "        # Get final winner\n",
-    "        winner = game.get_winner()\n",
-    "        \n",
-    "        # Create training examples\n",
-    "        examples = []\n",
-    "        for s, p, player in zip(states, action_probs, players):\n",
-    "            # Value is from perspective of current player\n",
-    "            value = winner * player\n",
-    "            examples.append((s, p, value))\n",
-    "        \n",
-    "        return examples\n",
-    "    \n",
-    "    def train_step(self, batch_size=32):\n",
-    "        \"\"\"Train network on batch from replay buffer\"\"\"\n",
-    "        if len(self.replay_buffer) < batch_size:\n",
-    "            return None\n",
-    "        \n",
-    "        # Sample batch\n",
-    "        batch = random.sample(self.replay_buffer, batch_size)\n",
-    "        states, target_probs, target_values = zip(*batch)\n",
-    "        \n",
-    "        # Convert to tensors\n",
-    "        states = torch.FloatTensor(np.array(states)).to(device)\n",
-    "        target_probs = torch.FloatTensor(np.array(target_probs)).to(device)\n",
-    "        target_values = torch.FloatTensor(np.array(target_values)).unsqueeze(1).to(device)\n",
-    "        \n",
-    "        # Forward pass\n",
-    "        policy_logits, values = self.network(states)\n",
-    "        \n",
-    "        # Calculate losses\n",
-    "        policy_loss = -torch.mean(torch.sum(target_probs * policy_logits, dim=1))\n",
-    "        value_loss = F.mse_loss(values, target_values)\n",
-    "        total_loss = policy_loss + value_loss\n",
-    "        \n",
-    "        # Backward pass\n",
-    "        self.optimizer.zero_grad()\n",
-    "        total_loss.backward()\n",
-    "        self.optimizer.step()\n",
-    "        \n",
-    "        return {\n",
-    "            'total_loss': total_loss.item(),\n",
-    "            'policy_loss': policy_loss.item(),\n",
-    "            'value_loss': value_loss.item()\n",
-    "        }\n",
-    "    \n",
-    "    def train(self, num_iterations=100, games_per_iteration=10, train_steps=50):\n",
-    "        \"\"\"Main training loop\"\"\"\n",
-    "        history = {'losses': [], 'policy_losses': [], 'value_losses': []}\n",
-    "        \n",
-    "        for iteration in range(num_iterations):\n",
-    "            print(f\"\\nIteration {iteration + 1}/{num_iterations}\")\n",
-    "            \n",
-    "            # Self-play games\n",
-    "            print(\"Playing self-play games...\")\n",
-    "            for game_num in tqdm(range(games_per_iteration)):\n",
-    "                examples = self.self_play_game()\n",
-    "                self.replay_buffer.extend(examples)\n",
-    "            \n",
-    "            # Training\n",
-    "            print(\"Training network...\")\n",
-    "            losses = []\n",
-    "            for _ in tqdm(range(train_steps)):\n",
-    "                loss_dict = self.train_step()\n",
-    "                if loss_dict:\n",
-    "                    losses.append(loss_dict)\n",
-    "            \n",
-    "            # Log progress\n",
-    "            if losses:\n",
-    "                avg_loss = np.mean([l['total_loss'] for l in losses])\n",
-    "                avg_policy_loss = np.mean([l['policy_loss'] for l in losses])\n",
-    "                avg_value_loss = np.mean([l['value_loss'] for l in losses])\n",
-    "                \n",
-    "                history['losses'].append(avg_loss)\n",
-    "                history['policy_losses'].append(avg_policy_loss)\n",
-    "                history['value_losses'].append(avg_value_loss)\n",
-    "                \n",
-    "                print(f\"Loss: {avg_loss:.4f} (Policy: {avg_policy_loss:.4f}, Value: {avg_value_loss:.4f})\")\n",
-    "                print(f\"Replay buffer size: {len(self.replay_buffer)}\")\n",
-    "        \n",
-    "        return history"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 6. Training the Network\n",
-    "\n",
-    "Now let's train our AlphaGo-style network through self-play!"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Create trainer\n",
-    "trainer = SelfPlayTrainer(network, board_size=9, num_simulations=25)\n",
-    "\n",
-    "# Train (start with small numbers for testing)\n",
-    "print(\"Starting training...\")\n",
-    "history = trainer.train(\n",
-    "    num_iterations=10,      # Increase for better results\n",
-    "    games_per_iteration=5,  # Increase for better results\n",
-    "    train_steps=20          # Increase for better results\n",
-    ")\n",
-    "\n",
-    "print(\"\\nTraining complete!\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 7. Visualize Training Progress"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Plot training curves\n",
-    "fig, axes = plt.subplots(1, 3, figsize=(15, 4))\n",
-    "\n",
-    "axes[0].plot(history['losses'])\n",
-    "axes[0].set_title('Total Loss')\n",
-    "axes[0].set_xlabel('Iteration')\n",
-    "axes[0].set_ylabel('Loss')\n",
-    "axes[0].grid(True)\n",
-    "\n",
-    "axes[1].plot(history['policy_losses'])\n",
-    "axes[1].set_title('Policy Loss')\n",
-    "axes[1].set_xlabel('Iteration')\n",
-    "axes[1].set_ylabel('Loss')\n",
-    "axes[1].grid(True)\n",
-    "\n",
-    "axes[2].plot(history['value_losses'])\n",
-    "axes[2].set_title('Value Loss')\n",
-    "axes[2].set_xlabel('Iteration')\n",
-    "axes[2].set_ylabel('Loss')\n",
-    "axes[2].grid(True)\n",
-    "\n",
-    "plt.tight_layout()\n",
-    "plt.show()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 8. Play Against the Trained Agent"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def play_game_vs_ai(network, human_player=1):\n",
-    "    \"\"\"Play a game against the trained AI\"\"\"\n",
-    "    game = GoGame(board_size=9)\n",
-    "    mcts = MCTS(network, game, num_simulations=50)\n",
-    "    \n",
-    "    state = game.reset()\n",
-    "    done = False\n",
-    "    \n",
-    "    print(\"Game started! You are\", 'X (Black)' if human_player == 1 else 'O (White)')\n",
-    "    print(\"Enter moves as 'row col' (e.g., '4 4') or 'pass' to pass\")\n",
-    "    game.render()\n",
-    "    \n",
-    "    while not done:\n",
-    "        if game.current_player == human_player:\n",
-    "            # Human move\n",
-    "            valid = False\n",
-    "            while not valid:\n",
-    "                move = input(\"Your move: \").strip().lower()\n",
-    "                \n",
-    "                if move == 'pass':\n",
-    "                    action = game.board_size * game.board_size\n",
-    "                    valid = True\n",
-    "                else:\n",
-    "                    try:\n",
-    "                        row, col = map(int, move.split())\n",
-    "                        action = row * game.board_size + col\n",
-    "                        if action in game.get_valid_moves():\n",
-    "                            valid = True\n",
-    "                        else:\n",
-    "                            print(\"Invalid move! Try again.\")\n",
-    "                    except:\n",
-    "                        print(\"Invalid input! Use format 'row col' or 'pass'\")\n",
-    "        else:\n",
-    "            # AI move\n",
-    "            print(\"AI is thinking...\")\n",
-    "            probs = mcts.search(state)\n",
-    "            action = np.argmax(probs)\n",
-    "            \n",
-    "            if action == game.board_size * game.board_size:\n",
-    "                print(\"AI passes\")\n",
-    "            else:\n",
-    "                row = action // game.board_size\n",
-    "                col = action % game.board_size\n",
-    "                print(f\"AI plays: {row} {col}\")\n",
-    "        \n",
-    "        state, reward, done = game.make_move(action)\n",
-    "        game.render()\n",
-    "    \n",
-    "    winner = game.get_winner()\n",
-    "    if winner == human_player:\n",
-    "        print(\"You win!\")\n",
-    "    elif winner == -human_player:\n",
-    "        print(\"AI wins!\")\n",
-    "    else:\n",
-    "        print(\"Draw!\")\n",
-    "\n",
-    "# Uncomment to play\n",
-    "# play_game_vs_ai(network, human_player=1)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 9. Evaluate Network Performance"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def evaluate_network(network, num_games=10):\n",
-    "    \"\"\"Evaluate network by playing games against random player\"\"\"\n",
-    "    wins = 0\n",
-    "    \n",
-    "    for game_num in tqdm(range(num_games)):\n",
-    "        game = GoGame(board_size=9)\n",
-    "        mcts = MCTS(network, game, num_simulations=25)\n",
-    "        \n",
-    "        state = game.reset()\n",
-    "        done = False\n",
-    "        \n",
-    "        # Network plays as black (1)\n",
-    "        while not done:\n",
-    "            if game.current_player == 1:\n",
-    "                # Network move\n",
-    "                probs = mcts.search(state)\n",
-    "                action = np.argmax(probs)\n",
-    "            else:\n",
-    "                # Random move\n",
-    "                valid_moves = game.get_valid_moves()\n",
-    "                action = random.choice(valid_moves)\n",
-    "            \n",
-    "            state, reward, done = game.make_move(action)\n",
-    "        \n",
-    "        winner = game.get_winner()\n",
-    "        if winner == 1:\n",
-    "            wins += 1\n",
-    "    \n",
-    "    win_rate = wins / num_games\n",
-    "    print(f\"\\nWin rate against random player: {win_rate:.1%} ({wins}/{num_games})\")\n",
-    "    return win_rate\n",
-    "\n",
-    "# Evaluate\n",
-    "evaluate_network(network, num_games=10)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 10. Save and Load Model"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Save model\n",
-    "torch.save({\n",
-    "    'model_state_dict': network.state_dict(),\n",
-    "    'optimizer_state_dict': trainer.optimizer.state_dict(),\n",
-    "    'history': history\n",
-    "}, 'alphago_model.pt')\n",
-    "\n",
-    "print(\"Model saved to 'alphago_model.pt'\")\n",
-    "\n",
-    "# Load model\n",
-    "# checkpoint = torch.load('alphago_model.pt')\n",
-    "# network.load_state_dict(checkpoint['model_state_dict'])\n",
-    "# trainer.optimizer.load_state_dict(checkpoint['optimizer_state_dict'])\n",
-    "# history = checkpoint['history']\n",
-    "# print(\"Model loaded from 'alphago_model.pt'\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Summary and Next Steps\n",
-    "\n",
-    "### What We've Implemented:\n",
-    "\n",
-    "1. **Go Game Environment**: Simplified 9x9 Go with basic rules\n",
-    "2. **Neural Network**: Deep residual CNN with policy and value heads\n",
-    "3. **MCTS**: Monte Carlo Tree Search guided by neural network\n",
-    "4. **Self-Play Training**: Network improves by playing against itself\n",
-    "5. **Evaluation**: Test against random and human players\n",
-    "\n",
-    "### Key Differences from Full AlphaGo:\n",
-    "\n",
-    "- **Board Size**: 9x9 instead of 19x19 (for faster training)\n",
-    "- **Network Size**: Smaller network (5 residual blocks vs 20+ in AlphaGo)\n",
-    "- **Training Scale**: Much less training (hours vs weeks on TPUs)\n",
-    "- **Go Rules**: Simplified capture and scoring logic\n",
-    "\n",
-    "### To Improve Performance:\n",
-    "\n",
-    "1. **Increase Training**:\n",
-    "   - More iterations (100+)\n",
-    "   - More games per iteration (50+)\n",
-    "   - More MCTS simulations (100+)\n",
-    "\n",
-    "2. **Larger Network**:\n",
-    "   - More residual blocks (10-20)\n",
-    "   - More channels (256+)\n",
-    "\n",
-    "3. **Better Go Rules**:\n",
-    "   - Implement full capture logic\n",
-    "   - Territory scoring\n",
-    "   - Ko rule enforcement\n",
-    "\n",
-    "4. **Advanced Techniques**:\n",
-    "   - Learning rate scheduling\n",
-    "   - Data augmentation (rotations, reflections)\n",
-    "   - Prioritized replay buffer\n",
-    "\n",
-    "### References:\n",
-    "\n",
-    "- Silver et al. (2016): \"Mastering the game of Go with deep neural networks and tree search\"\n",
-    "- Silver et al. (2017): \"Mastering the game of Go without human knowledge\" (AlphaGo Zero)\n",
-    "- Silver et al. (2018): \"A general reinforcement learning algorithm that masters chess, shogi, and Go\" (AlphaZero)\n",
-    "\n",
-    "### Congratulations!\n",
-    "\n",
-    "You've implemented the core concepts of AlphaGo! This same approach can be adapted to other games like Chess, Shogi, or even more complex domains."
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.8.0"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}