synth-ai 0.2.16__py3-none-any.whl → 0.2.19__py3-none-any.whl

examples/blog_posts/pokemon_vl/README.md

@@ -0,0 +1,98 @@
+# Pokémon VL: Vision-Language RL Pipeline
+
+This playbook demonstrates end-to-end vision-language reinforcement learning on Pokémon Red using Synth AI's CLI tools. We follow the eval → collect data → SFT → RL → eval pipeline, but with vision models throughout.
+
+## Overview
+
+**Model**: Qwen3-VL-4B-Instruct (4B parameter vision-language model via Synth API)
+**Environment**: Pokémon Red (Game Boy emulation with vision support)
+**Benchmark**: Pallet Town progression task (leave bedroom → get starter → win first battle)
+
+## Pipeline Steps
+
+1. **Deploy Task App** - Host the Pokémon Red environment
+2. **Collect Vision Rollouts** - Generate high-quality demonstrations using Qwen3-VL
+3. **Filter Dataset** - Extract successful trajectories for supervised fine-tuning
+4. **Fine-Tune Qwen3-4B VL** - Train vision-language model on filtered data
+5. **Vision-Language RL** - Bootstrap RL training from SFT checkpoint
+6. **Final Evaluation** - Compare SFT and RL performance
+
+## Prerequisites
+
+```bash
+# Install dependencies
+uv pip install -e .
+
+# Setup authentication
+uvx synth-ai setup
+
+# Copy environment template
+cp examples/blog_posts/pokemon_vl/.env.example .env
+```
+
+## Quick Start
+
+```bash
+# Export trace database path
+export POKEMON_VL_TRACE_DB=traces/v3/pokemon_vl_blog.db
+
+# 1. Deploy task app
+uvx synth-ai deploy pokemon_red --runtime modal --name pokemon-vl-blog --env-file .env
+
+# 2. Collect vision rollouts with Qwen3-VL
+uvx synth-ai eval pokemon_red --config examples/blog_posts/pokemon_vl/configs/eval_qwen3_vl.toml --trace-db "${POKEMON_VL_TRACE_DB}"
+
+# 3. Filter high-reward trajectories
+uvx synth-ai filter --config examples/blog_posts/pokemon_vl/configs/filter_high_reward.toml
+
+# 4. Fine-tune Qwen3-4B VL
+uvx synth-ai train --type sft --config examples/blog_posts/pokemon_vl/configs/train_sft_qwen4b_vl.toml --env-file .env --poll
+
+# 5. RL from SFT checkpoint (replace JOB_ID)
+uvx synth-ai train --type rl --config examples/blog_posts/pokemon_vl/configs/train_rl_from_sft.toml --env-file .env --poll
+
+# 6. Evaluate final RL model
+uvx synth-ai eval pokemon_red --config examples/blog_posts/pokemon_vl/configs/eval_rl_final.toml --trace-db "${POKEMON_VL_TRACE_DB}"
+```
+
+## Vision Features
+
+- **Full Game Boy Frames**: Base64-encoded PNG screenshots (160x144 resolution)
+- **Vision-Only Mode**: Pure image understanding without text state
+- **Vision + Text Mode**: Combined visual and structured state information
+- **Efficient Action Batching**: `execute_sequence` tool for 5-10 actions per inference call
+
+## Expected Results
+
+| Stage | Model | Mean Reward | Success Rate | Best Achievement |
+|-------|-------|-------------|--------------|------------------|
+| Initial | Qwen3-VL (vision) | ~150 | 60% | Win first battle |
+| SFT | Qwen3-4B VL | ~200 | 75% | Win first battle + explore |
+| RL | Qwen3-4B VL + RL | ~350 | 85% | Complete Pallet Town |
+
+## Files
+
+- `configs/` - All TOML configuration files
+- `ft_data/` - Filtered datasets for fine-tuning
+- `.env.example` - Environment variables template
+
+## Vision Model Configuration
+
+The vision models receive:
+- **Input**: Game Boy screenshot + optional structured state (position, HP, party, etc.)
+- **Output**: Sequence of button presses via `execute_sequence` tool
+- **Action Space**: UP, DOWN, LEFT, RIGHT, A, B, START, SELECT with frame counts
+
+## Reward Function
+
+Dense rewards for Pallet Town progression:
+- Leave bedroom (+20)
+- Exit house (+30)
+- Find Oak's lab (+40)
+- Talk to Oak (+50)
+- Get starter Pokémon (+100)
+- Enter battle (+75)
+- Deal damage (+50 per 10HP)
+- Win battle (+150)
+
+Total possible: ~700 points

examples/blog_posts/pokemon_vl/configs/eval_gpt5nano.toml

@@ -0,0 +1,26 @@
+[eval]
+app_id = "pokemon_red"
+task_app_url = "http://127.0.0.1:8914"
+model = "gpt-5-nano"
+seeds = [0]  # Single seed for testing
+max_turns = 10  # 10 LLM calls per episode to allow more progress
+concurrency = 1  # Run 1 rollout
+env_name = "pokemon_red"
+policy_name = "pokemon_vl_qwen3_vl"  # Reuse policy config, will override model
+trace_format = "full"
+return_trace = true
+
+[eval.policy_config]
+provider = "openai"  # Use OpenAI API for gpt-5-nano
+model = "gpt-5-nano"
+inference_url = "https://api.openai.com/v1"
+temperature = 0.7
+top_p = 0.95
+max_tokens = 512
+use_vision = true
+image_only_mode = false
+max_llm_calls = 10
+
+[eval.env_config.env_params]
+max_steps_per_episode = 100  # Allow time to achieve milestones
+

examples/blog_posts/pokemon_vl/configs/eval_qwen3_vl.toml

@@ -0,0 +1,27 @@
+[eval]
+app_id = "pokemon_red"
+task_app_url = "http://127.0.0.1:8914"
+model = "Qwen/Qwen3-VL-30B-A3B-Thinking"  # Larger thinking variant - needs more time to load
+seeds = [10, 11]  # 2 seeds for quick testing
+max_turns = 10  # 10 LLM calls per episode to allow more progress
+concurrency = 2  # Run 2 rollouts in parallel
+env_name = "pokemon_red"
+policy_name = "pokemon_vl_qwen3_vl"
+trace_format = "full"
+return_trace = true
+
+[eval.policy_config]
+provider = "synth"  # Use Synth internal API for vision models
+model = "Qwen/Qwen3-VL-30B-A3B-Thinking"  # Larger thinking variant - needs more time to load
+inference_url = "https://synth-laboratories-dev--learning-v2-service-fastapi-app.modal.run/chat/completions"
+temperature = 1.0  # Higher temperature to encourage exploration
+top_p = 0.95
+max_tokens = 2048  # Reduced to avoid token budget issues
+use_vision = true
+image_only_mode = false
+max_llm_calls = 10
+thinking_mode = "think"  # Enable thinking/reasoning mode
+thinking_budget = 3072  # Increased token budget for reasoning
+
+[eval.env_config.env_params]
+max_steps_per_episode = 100  # Increased from 3 to allow time to achieve milestones

examples/blog_posts/pokemon_vl/configs/eval_rl_final.toml

@@ -0,0 +1,24 @@
+[eval]
+app_id = "pokemon_red"
+task_app_url = "http://127.0.0.1:8914"
+model = "fft:REPLACE-WITH-RL-JOB-ID"  # Update with final RL job ID
+seeds = [100, 101, 102, 103, 104, 105, 106, 107, 108, 109]
+max_turns = 15  # Allow more steps for trained model
+concurrency = 3
+env_name = "pokemon_red"
+policy_name = "pokemon_vl_rl_final"
+trace_format = "full"
+return_trace = true
+
+[eval.policy_config]
+provider = "synth"
+model = "fft:REPLACE-WITH-RL-JOB-ID"  # Update with final RL job ID
+temperature = 0.1  # Lower temperature for evaluation
+top_p = 0.9
+max_tokens = 4096
+use_vision = true
+image_only_mode = false
+max_llm_calls = 15
+
+[eval.env_config.env_params]
+max_steps_per_episode = 15

examples/blog_posts/pokemon_vl/configs/filter_high_reward.toml

@@ -0,0 +1,10 @@
+# Filter high-quality vision-language rollouts for SFT training
+# Assumes traces stored in pokemon_vl_blog.db via eval commands
+
+[filter]
+db = "traces/v3/pokemon_vl_blog.db"
+output = "examples/blog_posts/pokemon_vl/ft_data/pokemon_vl_high_reward.jsonl"
+min_official_score = 0.3  # Require at least 30% completion (Pallet Town progression)
+models = ["Qwen/Qwen3-VL-4B-Instruct"]  # Vision models used for rollouts
+shuffle = true
+shuffle_seed = 42

examples/blog_posts/pokemon_vl/configs/train_rl_from_sft.toml

@@ -0,0 +1,43 @@
+# Vision-Language RL: Continue training Qwen3-4B VL from SFT checkpoint
+# Update task_url with deployed Modal task app URL
+# Set model.source to the SFT job id from `uvx synth-ai train --type sft`
+
+type = "rl"
+
+[services]
+task_url = "http://127.0.0.1:8914"
+
+[compute]
+gpu_type = "H100"
+gpu_count = 8
+
+[topology]
+gpus_for_vllm = 4
+gpus_for_training = 3
+gpus_for_ref = 1
+
+[vllm]
+tensor_parallel_size = 4
+
+[model]
+source = "fft:REPLACE-WITH-SFT-JOB-ID"  # Update with actual SFT job ID
+label = "pokemon_vl_rl_blog"
+supports_vision = true
+
+[rollout]
+max_turns = 10
+episodes_per_batch = 64
+task_app_origin_rewards_only = true
+
+[evaluation]
+instances = 100
+every_n_iters = 20
+seeds = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]
+
+[training]
+log_interval = 1
+
+[training.weight_sync]
+enable = true
+targets = ["policy"]
+weight_sync_interval = 1

examples/blog_posts/pokemon_vl/configs/train_sft_qwen4b_vl.toml

@@ -0,0 +1,40 @@
+# Vision-Language Supervised Fine-Tuning: Qwen3-4B VL on filtered Pokémon rollouts
+# Update the `data` path once `uvx synth-ai filter` produces your JSONL
+
+[algorithm]
+type = "offline"
+method = "sft"
+variety = "fft"
+
+[job]
+model = "Qwen/Qwen3-VL-4B-Instruct"  # Vision-enabled Qwen3-VL model
+data = "../ft_data/pokemon_vl_high_reward.jsonl"
+poll_seconds = 1800
+
+[compute]
+gpu_type = "H100"
+gpu_count = 4
+nodes = 1
+
+[data.topology]
+container_count = 4
+
+[training]
+mode = "full_finetune"
+use_qlora = false
+
+[hyperparameters]
+n_epochs = 2
+world_size = 4
+sequence_length = 4096  # Longer for vision tokens + text
+per_device_batch = 2
+gradient_accumulation_steps = 64
+learning_rate = 8e-6
+warmup_ratio = 0.03
+
+[hyperparameters.parallelism]
+use_deepspeed = true
+deepspeed_stage = 3
+fsdp = false
+bf16 = true
+fp16 = false

examples/blog_posts/pokemon_vl/extract_images.py

@@ -0,0 +1,239 @@
+#!/usr/bin/env python3
+"""Extract images from pokemon_vl trace database or trace JSON file and save to images_gpt5 directory.
+
+Usage:
+    # From trace database:
+    python extract_images.py --trace-db traces/v3/pokemon_vl_gpt5nano.db
+
+    # From trace JSON file:
+    python extract_images.py --trace-json trace.json
+"""
+
+import argparse
+import base64
+import json
+import sqlite3
+from pathlib import Path
+from typing import Any
+
+from synth_ai.tracing_v3.trace_utils import load_session_trace
+
+
+def extract_image_urls_from_content(content: Any) -> list[str]:
+    """Extract image URLs from message content."""
+    urls = []
+    
+    if isinstance(content, list):
+        for part in content:
+            if isinstance(part, dict):
+                if part.get("type") == "image_url" and "image_url" in part:
+                    url = part["image_url"].get("url")
+                    if isinstance(url, str) and url.startswith("data:image"):
+                        urls.append(url)
+                elif part.get("type") == "image":
+                    img = part.get("image")
+                    if isinstance(img, str) and img.startswith("data:image"):
+                        urls.append(img)
+    elif isinstance(content, str):
+        # Check if it's a JSON string
+        try:
+            parsed = json.loads(content)
+            return extract_image_urls_from_content(parsed)
+        except:
+            pass
+    
+    return urls
+
+
+def extract_state_info_from_message(message: dict[str, Any]) -> dict[str, Any]:
+    """Extract state info from message metadata or content."""
+    metadata = message.get("metadata", {})
+    state = {}
+    
+    # Try to get state from metadata
+    if "system_state_before" in metadata:
+        state_before = metadata["system_state_before"]
+        if isinstance(state_before, dict):
+            obs = state_before.get("obs", {})
+            state.update({
+                "position": obs.get("position", "?"),
+                "map_id": obs.get("map_id", "?"),
+                "player_x": obs.get("player_x", "?"),
+                "player_y": obs.get("player_y", "?"),
+                "text_box_active": obs.get("text_box_active", False),
+            })
+    
+    # Try to extract from content text
+    content = message.get("content", "")
+    if isinstance(content, str) and "position" in content:
+        try:
+            # Look for state summary in content
+            if "State summary:" in content:
+                parts = content.split("State summary:")
+                if len(parts) > 1:
+                    import ast
+                    state_str = parts[1].split("'")[0] if "'" not in parts[1] else parts[1]
+                    try:
+                        state_dict = ast.literal_eval(state_str.split("'")[0] if "'" in state_str else state_str)
+                        if isinstance(state_dict, dict):
+                            state.update({
+                                "position": state_dict.get("position", "?"),
+                                "map_id": state_dict.get("map_id", "?"),
+                                "player_x": state_dict.get("player_x", "?"),
+                                "player_y": state_dict.get("player_y", "?"),
+                                "text_box_active": state_dict.get("text_box_active", False),
+                            })
+                    except:
+                        pass
+        except:
+            pass
+    
+    return state
+
+
+def extract_images_from_trace_dict(trace: dict[str, Any], output_dir: Path):
+    """Extract images from a trace dictionary."""
+    output_dir.mkdir(parents=True, exist_ok=True)
+    
+    # Get messages from trace
+    messages = trace.get("markov_blanket_message_history", []) or trace.get("messages", [])
+    
+    if not messages:
+        print(f"  No messages found in trace")
+        return 0
+    
+    print(f"  Found {len(messages)} messages")
+    
+    image_count = 0
+    step_idx = 0
+    for msg_idx, msg in enumerate(messages):
+        # Extract images from message content
+        content = msg.get("content", "")
+        image_urls = extract_image_urls_from_content(content)
+        
+        if not image_urls:
+            continue
+        
+        # Extract state info for filename
+        state = extract_state_info_from_message(msg)
+        
+        for img_idx, img_url in enumerate(image_urls):
+            # Extract base64 data
+            if img_url.startswith("data:image"):
+                # Format: data:image/png;base64,<data>
+                parts = img_url.split(",", 1)
+                if len(parts) != 2:
+                    continue
+                
+                b64_data = parts[1]
+                try:
+                    img_data = base64.b64decode(b64_data)
+                    
+                    # Create filename
+                    pos_str = f"{state.get('map_id', '?')}_{state.get('player_x', '?')},{state.get('player_y', '?')}"
+                    textbox_str = "True" if state.get("text_box_active") else "False"
+                    filename = f"step_{step_idx:03d}_pos_{pos_str}_textbox_{textbox_str}.png"
+                    
+                    filepath = output_dir / filename
+                    filepath.write_bytes(img_data)
+                    
+                    print(f"  Saved: {filename}")
+                    image_count += 1
+                    step_idx += 1
+                except Exception as e:
+                    print(f"  Error decoding image: {e}")
+                    continue
+    
+    return image_count
+
+
+def extract_images_from_trace_db(trace_db: str, output_dir: Path, model_filter: str | None = None):
+    """Extract images from trace database and save to output directory."""
+    conn = sqlite3.connect(trace_db)
+    conn.row_factory = sqlite3.Row
+    
+    # Get all session IDs
+    query = "SELECT session_id, metadata FROM session_traces"
+    if model_filter:
+        query += " WHERE metadata LIKE ?"
+        params = (f'%{model_filter}%',)
+    else:
+        params = ()
+    
+    rows = conn.execute(query, params).fetchall()
+    
+    if not rows:
+        print(f"No traces found in {trace_db}")
+        return
+    
+    print(f"Found {len(rows)} trace(s)")
+    
+    total_images = 0
+    for row in rows:
+        session_id = row["session_id"]
+        print(f"\nProcessing session: {session_id}")
+        
+        try:
+            trace = load_session_trace(conn, session_id)
+        except Exception as e:
+            print(f"  Error loading trace: {e}")
+            continue
+        
+        count = extract_images_from_trace_dict(trace, output_dir)
+        total_images += count
+    
+    conn.close()
+    print(f"\n✓ Extracted {total_images} images to {output_dir}/")
+
+
+def extract_images_from_trace_json(trace_json: Path, output_dir: Path):
+    """Extract images from trace JSON file."""
+    print(f"Loading trace from {trace_json}")
+    
+    with open(trace_json) as f:
+        trace = json.load(f)
+    
+    # Handle trace wrapped in "session_trace" key
+    if "session_trace" in trace:
+        trace = trace["session_trace"]
+    
+    count = extract_images_from_trace_dict(trace, output_dir)
+    print(f"\n✓ Extracted {count} images to {output_dir}/")
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--trace-db",
+        help="Path to trace database",
+    )
+    parser.add_argument(
+        "--trace-json",
+        type=Path,
+        help="Path to trace JSON file",
+    )
+    parser.add_argument(
+        "--output-dir",
+        default="examples/blog_posts/pokemon_vl/images_gpt5",
+        help="Output directory for images",
+    )
+    parser.add_argument(
+        "--model-filter",
+        help="Filter traces by model name (optional)",
+    )
+    args = parser.parse_args()
+    
+    output_dir = Path(args.output_dir)
+    
+    if args.trace_json:
+        extract_images_from_trace_json(args.trace_json, output_dir)
+    elif args.trace_db:
+        extract_images_from_trace_db(args.trace_db, output_dir, args.model_filter)
+    else:
+        parser.error("Must provide either --trace-db or --trace-json")
+
+
+if __name__ == "__main__":
+    main()
+
+