synth-ai 0.2.12__py3-none-any.whl → 0.2.13.dev2__py3-none-any.whl

examples/task_apps/pokemon_red/README.md

@@ -0,0 +1,357 @@
+# Pokémon Red Task App
+
+A reinforcement learning environment for Pokémon Red using PyBoy emulation with VLM support.
+
+## Features
+
+- **Full Game Boy Emulation**: Uses PyBoy to run authentic Pokémon Red ROM
+- **VLM Support**: Base64-encoded PNG frames for vision models (GPT-4V, Qwen-VL, etc.)
+- **Policy Proxy**: OpenAI/Groq API integration for LLM-driven gameplay
+- **Rich State Extraction**: Comprehensive game state from RAM (HP, position, party, battle data)
+- **Reward Shaping**: Ultra-dense reward functions for RL training
+- **Instant Start**: Pre-configured init state skips intro (starts in Red's bedroom)
+
+## Quick Start
+
+### 1. Start the Task App Server
+
+```bash
+# From synth-ai root
+uv run -m synth_ai task-app serve pokemon_red --port 8913
+```
+
+### 2. Run a Random Rollout
+
+```python
+import httpx
+import asyncio
+
+async def test_rollout():
+    async with httpx.AsyncClient(timeout=60.0) as client:
+        response = await client.post(
+            "http://127.0.0.1:8913/rollout",
+            json={
+                "ops": [
+                    {"button": "DOWN", "frames": 10},
+                    {"button": "A", "frames": 20},
+                    {"button": "RIGHT", "frames": 15},
+                ],
+                "policy": {"config": {}},
+            },
+        )
+        result = response.json()
+        print(f"Steps: {len(result['steps'])}")
+
+asyncio.run(test_rollout())
+```
+
+### 3. Run with VLM Policy
+
+```bash
+# Using Qwen-VL via Groq
+uv run python examples/task_apps/pokemon_red/test_pallet_town_rewards.py
+```
+
+## Reward Functions
+
+### Pallet Town Progression (Recommended for Beginners)
+
+**Location**: `synth_ai/environments/examples/red/engine_helpers/reward_library/pallet_town_progression.py`
+
+Ultra-rich reward shaping for the opening sequence:
+
+| Milestone | Reward | Description |
+|-----------|--------|-------------|
+| Leave bedroom | +20 | Go downstairs |
+| Exit house | +30 | Enter Pallet Town |
+| Find Oak's lab | +40 | Discover and enter lab |
+| Talk to Oak | +50 | First dialogue |
+| Get starter | +100 | Receive your first Pokémon |
+| Enter battle | +75 | Start rival battle |
+| Deal damage | +50 | Attack rival (10×5) |
+| Half HP | +25 | Reduce enemy to <50% HP |
+| Low HP | +35 | Reduce enemy to <25% HP |
+| Win battle | +150 | Defeat rival |
+| Exit lab | +60 | Leave with Pokémon |
+| **Efficiency bonuses** | +100 | Fast navigation, healthy Pokémon |
+
+**Total: ~600-700 points**
+
+See [`PALLET_TOWN_REWARDS.md`](../../../synth_ai/environments/examples/red/engine_helpers/reward_library/PALLET_TOWN_REWARDS.md) for full documentation.
+
+### Usage in Training
+
+```toml
+# pallet_town_rl_config.toml
+[reward]
+reward_type = "composite"
+reward_class = "synth_ai.environments.examples.red.engine_helpers.reward_library.pallet_town_progression.PalletTownProgressionCompositeReward"
+
+[training]
+algorithm = "ppo"
+max_steps_per_episode = 500
+num_episodes = 1000
+```
+
+## State Schema
+
+The environment exposes comprehensive game state:
+
+```python
+{
+    # Position
+    "map_id": int,              # Current location
+    "player_x": int,
+    "player_y": int,
+    
+    # Party
+    "party_count": int,
+    "party_pokemon": [
+        {
+            "species_id": int,
+            "level": int,
+            "hp_current": int,
+            "hp_max": int,
+            "hp_percentage": float,
+            "xp": int,
+        }
+    ],
+    
+    # Battle
+    "in_battle": bool,
+    "battle_outcome": int,      # 0=ongoing, 1=win, 2=lose
+    "enemy_hp_current": int,
+    "enemy_hp_max": int,
+    "enemy_hp_percentage": float,
+    "enemy_level": int,
+    "enemy_species_id": int,
+    "battle_turn": int,
+    
+    # Dialogue & UI
+    "text_box_active": bool,
+    "menu_state": int,
+    
+    # Progress
+    "badges": int,              # Bitfield of earned badges
+    "money": int,
+    
+    # VLM Support
+    "observation_image_base64": str,  # PNG frame for vision models
+}
+```
+
+## Action Space
+
+### Button Actions
+
+```python
+{
+    "button": "A" | "B" | "START" | "SELECT" | "UP" | "DOWN" | "LEFT" | "RIGHT",
+    "frames": int,  # How long to hold the button (60fps)
+}
+```
+
+### Policy-Driven Actions
+
+When using LLM policies, the task app proxies requests to OpenAI/Groq:
+
+```python
+{
+    "policy": {
+        "config": {
+            "model": "gpt-4-turbo",
+            "api_key": "...",
+            # or for Groq:
+            # "model": "qwen-2.5-7b",
+            # "base_url": "https://api.groq.com/v1",
+        }
+    }
+}
+```
+
+## Files
+
+- **`task_app.py`**: Main task app entry point
+- **`pallet_town_rl_config.toml`**: Training config for Pallet Town sequence
+- **`test_pallet_town_rewards.py`**: Reward function test/demo script
+- **`create_red_init_state.py`** (repo root): Script to generate init state
+- **`Pokemon - Red Version (USA, Europe) (SGB Enhanced).gb`**: Your ROM (not committed)
+
+## Creating Init States
+
+The default init state starts in Red's bedroom with intro skipped. To create custom states:
+
+```python
+# See /Users/joshpurtell/Documents/GitHub/synth-ai/create_red_init_state.py
+from pyboy import PyBoy
+
+emulator = PyBoy("path/to/rom.gb", window="null")
+
+# Navigate to desired starting point
+# ... (button presses)
+
+# Save state
+with open("custom_init.state", "wb") as f:
+    emulator.save_state(f)
+```
+
+## Memory Addresses
+
+Key RAM addresses are defined in `synth_ai/environments/examples/red/engine_helpers/memory_map.py`:
+
+- `MAP_ID = 0xD35E`
+- `PLAYER_X/Y = 0xD362/0xD361`
+- `IN_BATTLE_FLAG = 0xD057`
+- `ENEMY_HP_CURRENT = 0xCFE6`
+- `PARTY_COUNT = 0xD163`
+- `BADGE_FLAGS = 0xD356`
+- (and many more)
+
+## Troubleshooting
+
+### ROM Not Found
+
+```bash
+# Set environment variable
+export POKEMON_RED_ROM_PATH="/path/to/pokemon_red.gb"
+
+# Or copy ROM to expected location
+cp "Pokemon - Red Version.gb" synth_ai/environments/examples/red/roms/pokemon_red.gb
+```
+
+### PyBoy Not Installed
+
+```bash
+uv add pyboy
+```
+
+### Server Won't Start (Port in Use)
+
+```bash
+# Kill existing server
+lsof -ti :8913 | xargs -r kill -9
+
+# Or use a different port
+uv run -m synth_ai task-app serve pokemon_red --port 8914
+```
+
+## Examples
+
+### 1. Policy Evaluation with GPT-5-nano
+
+Evaluate a GPT-5-nano policy across 10 episodes (10 policy calls each):
+
+```bash
+# From synth-ai root
+cd /Users/joshpurtell/Documents/GitHub/synth-ai
+
+# 1. Make sure OpenAI API key is in .env
+echo "OPENAI_API_KEY=sk-..." >> .env
+
+# 2. Start the task app server (in background)
+nohup sh -c 'printf "n\n" | uv run -m synth_ai task-app serve pokemon_red --port 8913 --no-reload' > nohup_pokemon.log 2>&1 &
+
+# Wait for startup
+sleep 8
+
+# 3. Run the evaluation
+uv run python examples/task_apps/pokemon_red/eval_pokemon_red_policy.py
+```
+
+**Expected Output:**
+```
+================================================================================
+POKÉMON RED - POLICY EVALUATION
+================================================================================
+
+Task: Pallet Town Progression
+Policy: gpt-5-nano
+Episodes: 10
+Max steps per episode: 10
+
+✓ Server is healthy
+✓ API key loaded
+
+🎮 Running 10 episodes in parallel...
+
+================================================================================
+RESULTS SUMMARY
+================================================================================
+
++-----------+----------+---------+-------------+---------+----------+--------------+
+|   Episode |   Reward |   Steps | Final Map   |   Party |   Badges |   Milestones |
++===========+==========+=========+=============+=========+==========+==============+
+|         1 |        0 |      10 | Map38       |       0 |        0 |            0 |
+|         2 |        0 |       9 | Map38       |       0 |        0 |            0 |
+|         9 |       20 |      10 | Map38       |       0 |        0 |            1 |
++-----------+----------+---------+-------------+---------+----------+--------------+
+
+Statistics:
+  Mean reward: 2.00
+  Max reward: 20.00
+  Success rate: 10% reached first milestone
+  
+Best Episode (#9):
+  Total reward: 20.0
+  Milestones achieved:
+    Step 5: Moved from Map38 to Map37 (+20.0)
+```
+
+**Key Features:**
+- ✅ **Action Batching**: Each policy call returns 5-10 actions via `execute_sequence` tool
+- ✅ **Parallel Execution**: All 10 episodes run concurrently
+- ✅ **Rich Metrics**: Rewards, steps, maps, party status, milestones tracked
+- ✅ **Fast Evaluation**: ~2-3 minutes for 10 episodes (vs 50+ min without batching)
+
+**Customize the Evaluation:**
+
+```python
+# In eval_pokemon_red_policy.py
+NUM_EPISODES = 10              # Number of episodes to run
+MAX_STEPS_PER_EPISODE = 10     # Policy calls per episode (each returns 5-10 actions)
+MODEL = "gpt-5-nano"           # Or "gpt-4-turbo", "qwen-2.5-7b", etc.
+```
+
+### 2. Test Script (Random Actions)
+
+```bash
+cd /Users/joshpurtell/Documents/GitHub/synth-ai
+uv run python test_pokemon_red_rollout.py
+```
+
+### 3. Reward Function Demo
+
+```bash
+uv run python examples/task_apps/pokemon_red/test_pallet_town_rewards.py
+```
+
+Output:
+```
+======================================================================
+PALLET TOWN PROGRESSION - REWARD SIMULATION
+======================================================================
+
+✓ Leave bedroom (Map 1→2):                    +20 points
+✓ Exit house to Pallet Town (Map 2→0):        +30 points
+✓ Find and enter Oak's Lab (Map 0→3):         +40 points
+...
+======================================================================
+TOTAL REWARD: 705 points
+======================================================================
+```
+
+## Future Work
+
+- [ ] Route 1 exploration rewards
+- [ ] Wild Pokémon encounter rewards
+- [ ] Capture mechanics rewards
+- [ ] Gym battle rewards
+- [ ] Badge collection rewards
+- [ ] Multi-environment curriculum (Pallet → Viridian → Pewter)
+
+## Credits
+
+- **PyBoy**: Game Boy emulator - https://github.com/Baekalfen/PyBoy
+- **Pokémon Red Disassembly**: RAM map reference - https://github.com/pret/pokered
+- **Datacrystal.org**: Memory address documentation
+

examples/task_apps/pokemon_red/__init__.py

@@ -0,0 +1,3 @@
+"""Pokémon Red task app example package."""
+
+

examples/task_apps/pokemon_red/eval_pokemon_red_policy.py

@@ -0,0 +1,225 @@
+#!/usr/bin/env python3
+"""Evaluate GPT-5-nano policy on Pokemon Red Pallet Town progression.
+
+Runs 10 parallel rollouts and reports rewards in a table.
+"""
+import asyncio
+import os
+from typing import Any
+
+import httpx
+from dotenv import load_dotenv
+from tabulate import tabulate
+
+
+# Load environment variables
+load_dotenv()
+
+# Configuration
+TASK_APP_URL = "http://127.0.0.1:8913"
+NUM_EPISODES = 10
+MAX_STEPS_PER_EPISODE = 10  # 10 policy calls per episode (each may return 5-10 actions)
+MODEL = "gpt-5-nano"
+OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
+
+
+async def run_single_rollout(
+    client: httpx.AsyncClient,
+    episode_id: int,
+) -> dict[str, Any]:
+    """Run a single rollout with policy-driven actions."""
+    
+    # Build rollout request with policy actions
+    # Send "policy" for each step to trigger LLM inference
+    rollout_request = {
+        "run_id": f"eval_episode_{episode_id:03d}",
+        "env": {"instance_id": f"pallet_town_{episode_id:03d}"},
+        "ops": ["policy"] * MAX_STEPS_PER_EPISODE,  # Let policy drive all actions
+        "policy": {
+            "type": "llm",
+            "model": MODEL,
+            "config": {
+                "model": MODEL,
+                "temperature": 0.7,
+                "max_tokens": 500,
+            }
+        },
+    }
+    
+    try:
+        response = await client.post(
+            f"{TASK_APP_URL}/rollout",
+            json=rollout_request,
+            timeout=300.0,  # 5 minutes per rollout
+        )
+        response.raise_for_status()
+        result = response.json()
+        
+        # Extract metrics
+        trajectories = result.get("trajectories", [])
+        if not trajectories:
+            return {
+                "episode_id": episode_id,
+                "status": "error",
+                "error": "No trajectories returned",
+            }
+        
+        trajectory = trajectories[0]
+        steps = trajectory.get("steps", [])
+        num_steps = len(steps) - 1  # Subtract initial observation
+        
+        # Get metrics
+        metrics = result.get("metrics", {})
+        total_reward = metrics.get("episode_returns", [0.0])[0]
+        outcome_score = metrics.get("outcome_score", 0.0)
+        details = metrics.get("details", {})
+        
+        # Extract milestone info
+        reward_components = details.get("reward_components", [])
+        milestone_events = details.get("milestone_events", [])
+        final_map = details.get("final_map", -1)
+        party_count = details.get("party_count", 0)
+        badges = details.get("badges", 0)
+        
+        return {
+            "episode_id": episode_id,
+            "status": "success",
+            "total_reward": total_reward,
+            "outcome_score": outcome_score,
+            "num_steps": num_steps,
+            "final_map": final_map,
+            "party_count": party_count,
+            "badges": badges,
+            "num_milestones": len(milestone_events),
+            "reward_components": reward_components,
+            "milestone_events": milestone_events,
+        }
+        
+    except httpx.TimeoutException:
+        return {
+            "episode_id": episode_id,
+            "status": "timeout",
+            "error": "Rollout timed out after 5 minutes",
+        }
+    except Exception as e:
+        return {
+            "episode_id": episode_id,
+            "status": "error",
+            "error": str(e),
+        }
+
+
+async def main():
+    print("=" * 80)
+    print("POKÉMON RED - POLICY EVALUATION")
+    print("=" * 80)
+    print()
+    print(f"Task: Pallet Town Progression")
+    print(f"Policy: {MODEL}")
+    print(f"Episodes: {NUM_EPISODES}")
+    print(f"Max steps per episode: {MAX_STEPS_PER_EPISODE}")
+    print(f"Server: {TASK_APP_URL}")
+    print()
+    
+    # Check server health
+    async with httpx.AsyncClient() as client:
+        try:
+            response = await client.get(f"{TASK_APP_URL}/health", timeout=5.0)
+            response.raise_for_status()
+            print("✓ Server is healthy")
+        except Exception as e:
+            print(f"❌ Server not responding: {e}")
+            print(f"   Start it with: uv run -m synth_ai task-app serve pokemon_red --port 8913")
+            return
+        
+        # Check API key
+        if not OPENAI_API_KEY:
+            print("❌ OPENAI_API_KEY not found in environment")
+            print("   Make sure .env file contains OPENAI_API_KEY")
+            return
+        print(f"✓ API key loaded (sk_env...{OPENAI_API_KEY[-4:]})")
+        print()
+        
+        # Run rollouts in parallel
+        print(f"🎮 Running {NUM_EPISODES} episodes in parallel...")
+        print()
+        
+        tasks = [
+            run_single_rollout(client, episode_id)
+            for episode_id in range(1, NUM_EPISODES + 1)
+        ]
+        
+        results = await asyncio.gather(*tasks)
+        
+        # Separate successful and failed results
+        successful = [r for r in results if r.get("status") == "success"]
+        failed = [r for r in results if r.get("status") != "success"]
+        
+        # Print summary table
+        print()
+        print("=" * 80)
+        print("RESULTS SUMMARY")
+        print("=" * 80)
+        print()
+        
+        if successful:
+            table_data = []
+            for r in successful:
+                table_data.append([
+                    r["episode_id"],
+                    f"{r['total_reward']:.1f}",
+                    r["num_steps"],
+                    f"Map{r['final_map']}",
+                    r["party_count"],
+                    r["badges"],
+                    r["num_milestones"],
+                    f"{r['outcome_score']:.3f}",
+                ])
+            
+            headers = [
+                "Episode",
+                "Reward",
+                "Steps",
+                "Final Map",
+                "Party",
+                "Badges",
+                "Milestones",
+                "Outcome Score",
+            ]
+            
+            print(tabulate(table_data, headers=headers, tablefmt="grid"))
+            print()
+            
+            # Print statistics
+            rewards = [r["total_reward"] for r in successful]
+            steps = [r["num_steps"] for r in successful]
+            outcome_scores = [r["outcome_score"] for r in successful]
+            
+            print("Statistics:")
+            print(f"  Mean reward: {sum(rewards) / len(rewards):.2f}")
+            print(f"  Max reward: {max(rewards):.2f}")
+            print(f"  Min reward: {min(rewards):.2f}")
+            print(f"  Mean steps: {sum(steps) / len(steps):.1f}")
+            print(f"  Mean outcome score: {sum(outcome_scores) / len(outcome_scores):.4f}")
+            print()
+            
+            # Print milestone breakdown for best episode
+            best_episode = max(successful, key=lambda r: r["total_reward"])
+            print(f"Best Episode (#{best_episode['episode_id']}):")
+            print(f"  Total reward: {best_episode['total_reward']:.1f}")
+            print(f"  Steps taken: {best_episode['num_steps']}")
+            print(f"  Milestones achieved:")
+            for milestone in best_episode["milestone_events"]:
+                print(f"    Step {milestone['step']}: {milestone['description']} (+{milestone['reward']:.1f})")
+            print()
+        
+        if failed:
+            print(f"Failed episodes: {len(failed)}")
+            for r in failed:
+                print(f"  Episode {r['episode_id']}: {r.get('error', 'Unknown error')}")
+            print()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+

examples/task_apps/pokemon_red/pallet_town_rl_config.toml

@@ -0,0 +1,73 @@
+# Pokemon Red: Pallet Town Progression RL Configuration
+#
+# This configuration uses the ultra-rich reward function that tracks
+# all major milestones in the Pallet Town opening sequence:
+# - Leaving bedroom
+# - Exiting house
+# - Finding Oak's lab
+# - Talking to Oak
+# - Receiving starter Pokemon
+# - Entering first battle
+# - Dealing damage to rival
+# - Winning first battle
+# - Exiting lab with Pokemon
+#
+# Total possible reward: ~600+ points
+
+[experiment]
+name = "pokemon_red_pallet_town_progression"
+description = "Learn to complete the Pallet Town intro sequence with dense reward shaping"
+
+[environment]
+task_app_id = "pokemon_red"
+# Start state is already configured to begin in Red's bedroom (Map26)
+
+[reward]
+# Use the comprehensive Pallet Town progression reward
+reward_type = "composite"
+reward_class = "synth_ai.environments.examples.red.engine_helpers.reward_library.pallet_town_progression.PalletTownProgressionCompositeReward"
+
+[policy]
+model = "gpt-4-turbo"
+# or use "qwen-2.5-7b" for faster, cheaper training
+
+[training]
+algorithm = "ppo"  # or "grpo"
+max_steps_per_episode = 500  # Plenty of steps for the Pallet Town sequence
+num_episodes = 1000
+batch_size = 32
+
+[training.hyperparameters]
+learning_rate = 3e-4
+gamma = 0.99
+clip_epsilon = 0.2
+
+[logging]
+log_interval = 10
+save_interval = 100
+wandb_project = "pokemon-red-rl"
+wandb_run_name = "pallet-town-progression"
+
+[evaluation]
+eval_interval = 50
+num_eval_episodes = 10
+
+# Reward breakdown (for reference):
+# ============================================
+# LeaveBedroomReward              +20
+# ExitHouseFirstTimeReward        +30
+# FindOakLabReward                +40
+# TalkToOakReward                 +50
+# ReceiveStarterPokemonReward     +100
+# EnterFirstBattleReward          +75
+# DealDamageToRivalReward         +50  (10×5)
+# ReduceEnemyHPByHalfReward       +25
+# ReduceEnemyHPToLowReward        +35
+# WinFirstBattleReward            +150
+# ExitLabAfterBattleReward        +60
+# FirstBattleEfficiencyReward     +20
+# KeepPokemonHealthyReward        +30
+# NavigationSpeedReward           +50
+# ============================================
+# TOTAL POSSIBLE                  ~600+
+