synth-ai 0.2.13.dev2__py3-none-any.whl → 0.2.16__py3-none-any.whl

examples/task_apps/IMAGE_ONLY_EVAL_QUICKSTART.md

@@ -0,0 +1,258 @@
+# Image-Only Evaluation - Quick Reference
+
+This document provides a quick reference for running image-only evaluations on **Crafter** and **Pokemon Red** with Turso tracing.
+
+## 📚 Full Documentation
+
+- **Crafter**: [`crafter/README_IMAGE_ONLY_EVAL.md`](crafter/README_IMAGE_ONLY_EVAL.md)
+- **Pokemon Red**: [`pokemon_red/README_IMAGE_ONLY_EVAL.md`](pokemon_red/README_IMAGE_ONLY_EVAL.md)
+
+## ⚡ Quick Start
+
+### Prerequisites
+
+```bash
+# 1. Set OpenAI API key in .env
+echo "OPENAI_API_KEY=sk-proj-..." >> .env
+
+# 2. Navigate to synth-ai repo
+cd /path/to/synth-ai
+```
+
+### Run Crafter (Easier - 70% Success Rate)
+
+```bash
+# Set up tracing
+export TASKAPP_TRACING_ENABLED=1
+export TURSO_NATIVE=1
+export SQLD_DB_PATH="traces/v3/crafter_eval.db"
+
+# Run evaluation
+uv run synth-ai eval grpo-crafter \
+  --config examples/task_apps/crafter/eval_image_only_gpt4o.toml
+
+# Check results
+sqlite3 -header -column traces/v3/crafter_eval.db \
+  "SELECT total_reward, achievements_count, 
+   json_extract(reward_metadata, '$.final_achievements') as achievements
+   FROM outcome_rewards WHERE total_reward > 0;"
+```
+
+### Run Pokemon Red (Harder - 0% with Default Config)
+
+```bash
+# Set up tracing
+export TASKAPP_TRACING_ENABLED=1
+export TURSO_NATIVE=1
+export SQLD_DB_PATH="traces/v3/pokemon_red_eval.db"
+
+# Run evaluation
+uv run synth-ai eval pokemon_red \
+  --config examples/task_apps/pokemon_red/eval_image_only_gpt4o.toml
+
+# Check results
+sqlite3 -header -column traces/v3/pokemon_red_eval.db \
+  "SELECT total_reward, achievements_count, 
+   json_extract(reward_metadata, '$.final_map') as map,
+   json_extract(reward_metadata, '$.party_count') as party
+   FROM outcome_rewards;"
+```
+
+## 📊 Comparison
+
+| Feature | Crafter | Pokemon Red |
+|---------|---------|-------------|
+| **Difficulty** | Easier | Harder |
+| **Default success** | ~70% earn rewards | ~0% (needs tuning) |
+| **Typical reward** | 1-3 achievements | 0 (10 steps too short) |
+| **Best for** | Testing vision models | RL research |
+| **Recommended steps** | 10 (default works) | 100-500 (need more) |
+
+## 🔧 Configuration Files
+
+### Crafter Config
+**Location**: `examples/task_apps/crafter/eval_image_only_gpt4o.toml`
+
+```toml
+[eval]
+app_id = "grpo-crafter"
+model = "gpt-4o-mini-2024-07-18"
+seeds = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
+max_turns = 10
+env_name = "crafter"
+policy_name = "crafter-react"
+
+[eval.policy_config]
+use_vision = true
+image_only_mode = true  # Only images, no text
+```
+
+### Pokemon Red Config
+**Location**: `examples/task_apps/pokemon_red/eval_image_only_gpt4o.toml`
+
+```toml
+[eval]
+app_id = "pokemon_red"
+model = "gpt-4o-mini-2024-07-18"
+seeds = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
+max_turns = 10
+env_name = "pokemon_red"
+
+[eval.policy_config]
+use_vision = true
+image_only_mode = true  # Only images, no text
+```
+
+## 📈 Improving Pokemon Red Results
+
+Pokemon Red is harder and needs more steps. To get non-zero rewards:
+
+```toml
+[eval]
+model = "gpt-4o-2024-08-06"  # Use full GPT-4o
+max_turns = 100
+
+[eval.env_config]
+env_params = {max_steps_per_episode = 500}
+
+[eval.policy_config]
+model = "gpt-4o-2024-08-06"
+image_only_mode = false  # Enable text too (multimodal)
+max_llm_calls = 100
+```
+
+## 🗄️ Database Queries
+
+### Get All Rewards
+
+```sql
+-- Crafter
+SELECT 
+    json_extract(reward_metadata, '$.env_seed') as seed,
+    total_reward,
+    achievements_count,
+    json_extract(reward_metadata, '$.final_achievements') as achievements
+FROM outcome_rewards
+ORDER BY total_reward DESC;
+
+-- Pokemon Red
+SELECT 
+    session_id,
+    total_reward,
+    achievements_count,
+    json_extract(reward_metadata, '$.final_map') as map,
+    json_extract(reward_metadata, '$.party_count') as party
+FROM outcome_rewards
+ORDER BY total_reward DESC;
+```
+
+### Filter Non-Zero Rewards
+
+```sql
+SELECT * FROM outcome_rewards WHERE total_reward > 0;
+```
+
+### Get Statistics
+
+```sql
+SELECT 
+    COUNT(*) as total,
+    SUM(CASE WHEN total_reward > 0 THEN 1 ELSE 0 END) as with_rewards,
+    AVG(total_reward) as avg_reward,
+    MAX(total_reward) as max_reward
+FROM outcome_rewards;
+```
+
+## 🎯 What is Image-Only Mode?
+
+**Image-Only Mode** means:
+- ✅ Agent receives **only** base64-encoded PNG images
+- ❌ Agent receives **no** text observations (HP, position, inventory, etc.)
+- 🎓 Tests pure vision understanding
+
+**Multimodal Mode** (recommended for Pokemon Red):
+- ✅ Agent receives **both** images and text
+- 🏆 Better performance but "easier"
+
+Toggle with:
+```toml
+[eval.policy_config]
+use_vision = true         # Enable vision
+image_only_mode = false   # false = send text too
+```
+
+## 📁 Files Created
+
+### Crafter
+- `crafter/eval_image_only_gpt4o.toml` - Config
+- `crafter/README_IMAGE_ONLY_EVAL.md` - Full guide
+- `crafter/EVAL_IMAGE_ONLY_RESULTS.md` - Example results
+- `crafter/QUERY_EXAMPLES.md` - SQL queries
+
+### Pokemon Red  
+- `pokemon_red/eval_image_only_gpt4o.toml` - Config
+- `pokemon_red/README_IMAGE_ONLY_EVAL.md` - Full guide
+- `pokemon_red/EVAL_IMAGE_ONLY_COMPLETE.md` - Implementation
+- `pokemon_red/EVAL_IMAGE_ONLY_STATUS.md` - Status
+
+## 🐛 Common Issues
+
+### Database Not Created
+```bash
+# Ensure variables are set
+export TASKAPP_TRACING_ENABLED=1
+export TURSO_NATIVE=1
+export SQLD_DB_PATH="traces/v3/your_eval.db"
+```
+
+### 401 Unauthorized
+```bash
+# Check API key in .env
+cat .env | grep OPENAI_API_KEY
+```
+
+### Pokemon Red: ROM Not Found
+```bash
+# Place ROM at expected location
+cp pokemon_red.gb synth_ai/environments/examples/red/roms/
+```
+
+### All Rewards Zero
+- **Crafter**: Should get ~70% non-zero by default
+- **Pokemon Red**: Expected with 10 steps - increase to 100-500
+
+## 🎓 Understanding Results
+
+### Crafter Achievements
+- `collect_wood` - Cut down trees
+- `collect_sapling` - Collect tree saplings  
+- `collect_drink` - Drink from water
+
+### Pokemon Red Milestones
+- Leave bedroom (+20)
+- Exit house (+30)
+- Find Oak's lab (+40)
+- Get starter Pokemon (+100)
+- Win first battle (+150)
+
+**Total possible**: ~600 points
+
+## 🚀 Next Steps
+
+1. **Read full docs**: See task-specific READMEs for details
+2. **Run evaluations**: Start with Crafter (easier)
+3. **Query database**: Use SQL to analyze results
+4. **Tune configs**: Adjust steps/model for better performance
+5. **Compare modes**: Try image-only vs multimodal
+
+## 📞 Support
+
+For issues or questions:
+1. Check full README for your task app
+2. Review example results files
+3. Query database to verify data
+4. Adjust config parameters
+
+Happy evaluating! 🎮
+
+

examples/task_apps/crafter/CREATE_SFT_DATASET.md

@@ -0,0 +1,273 @@
+# Creating SFT Datasets from Crafter Traces
+
+There are two approaches to create SFT (Supervised Fine-Tuning) datasets from Crafter rollouts:
+
+## Approach 1: Direct SFT Recording (Recommended)
+
+Crafter's rollout system can write SFT-ready JSONL files directly during evaluation by setting the `sft_output_dir`.
+
+### Setup
+
+1. Set the SFT output directory environment variable:
+```bash
+export SFT_OUTPUT_DIR="ft_data/crafter_sft"
+```
+
+2. Run evaluation:
+```bash
+cd /Users/joshpurtell/Documents/GitHub/synth-ai
+
+export TASKAPP_TRACING_ENABLED=1
+export TURSO_NATIVE=1
+export SQLD_DB_PATH="traces/v3/crafter_eval.db"
+export SFT_OUTPUT_DIR="ft_data/crafter_sft"  # Enable SFT recording
+
+uv run synth-ai eval grpo-crafter \
+  --config examples/task_apps/crafter/eval_image_only_gpt4o.toml
+```
+
+3. SFT files will be written to:
+```
+ft_data/crafter_sft/
+├── sft_<run_id_1>.jsonl
+├── sft_<run_id_2>.jsonl
+├── ...
+└── sft_<run_id_10>.jsonl
+```
+
+### SFT Record Format
+
+Each JSONL file contains records like:
+```json
+{
+  "messages": [
+    {"role": "system", "content": "...system prompt..."},
+    {"role": "user", "content": "...observation..."},
+    {"role": "assistant", "content": "...action..."}
+  ],
+  "metadata": {
+    "run_id": "...",
+    "turn": 5,
+    "reward": 1.0,
+    ...
+  }
+}
+```
+
+### Combine Multiple Files
+
+```bash
+# Combine all SFT files into one
+cat ft_data/crafter_sft/sft_*.jsonl > ft_data/crafter_combined.jsonl
+
+# Count examples
+wc -l ft_data/crafter_combined.jsonl
+```
+
+## Approach 2: Extract from Turso Database (Not Currently Supported)
+
+The `synth-ai filter` command is designed for traces with a different structure (where prompt/completion are stored in session metadata). 
+
+**Current Limitation**: Crafter's SessionTracer-based traces don't store messages in the format expected by the filter command.
+
+### Why Filter Doesn't Work
+
+The filter command expects:
+```python
+metadata = {
+    "prompt": "...",  # User message
+    "completion": "..."  # Assistant response
+}
+```
+
+But Crafter traces store:
+- Messages in separate `messages` table (currently 0 messages - not recorded during eval)
+- Rewards in `outcome_rewards` table
+- Metadata without prompt/completion fields
+
+### Future Enhancement
+
+To make filter work with Crafter traces, we would need to:
+1. Modify rollout to record messages to the `messages` table
+2. Update filter command to query `messages` table directly
+3. Join with `outcome_rewards` to filter by achievements
+
+## Comparison
+
+| Feature | Direct SFT | Filter Command |
+|---------|-----------|----------------|
+| **Setup** | Set `SFT_OUTPUT_DIR` | Create filter config |
+| **When** | During rollout | After rollout |
+| **Format** | JSONL per rollout | Combined JSONL |
+| **Filtering** | Manual (combine files) | Automatic (SQL queries) |
+| **Status** | ✅ Works now | ❌ Needs implementation |
+
+## Recommended Workflow
+
+### 1. Run evaluation with SFT recording:
+```bash
+export SFT_OUTPUT_DIR="ft_data/crafter_sft"
+uv run synth-ai eval grpo-crafter --config examples/task_apps/crafter/eval_image_only_gpt4o.toml
+```
+
+### 2. Filter for successful rollouts:
+
+Since we can't use the filter command yet, manually select files:
+
+```bash
+# Query database to find session_ids with rewards
+sqlite3 traces/v3/crafter_eval.db \
+  "SELECT session_id FROM outcome_rewards WHERE total_reward > 0" \
+  > successful_sessions.txt
+
+# Create directory for filtered SFT
+mkdir -p ft_data/crafter_sft_filtered
+
+# Copy only successful rollout SFT files
+while read session_id; do
+  if [ -f "ft_data/crafter_sft/sft_${session_id}.jsonl" ]; then
+    cp "ft_data/crafter_sft/sft_${session_id}.jsonl" ft_data/crafter_sft_filtered/
+  fi
+done < successful_sessions.txt
+
+# Combine filtered files
+cat ft_data/crafter_sft_filtered/sft_*.jsonl > ft_data/crafter_high_reward.jsonl
+
+echo "Created SFT dataset: ft_data/crafter_high_reward.jsonl"
+wc -l ft_data/crafter_high_reward.jsonl
+```
+
+### 3. Verify dataset:
+
+```bash
+# Look at first example
+head -1 ft_data/crafter_high_reward.jsonl | jq .
+
+# Count examples
+wc -l ft_data/crafter_high_reward.jsonl
+
+# Check message types
+jq -r '.messages[].role' ft_data/crafter_high_reward.jsonl | sort | uniq -c
+```
+
+## Example: Complete Pipeline
+
+```bash
+#!/bin/bash
+# complete_sft_pipeline.sh
+
+cd /Users/joshpurtell/Documents/GitHub/synth-ai
+
+# 1. Run evaluation with SFT recording
+export TASKAPP_TRACING_ENABLED=1
+export TURSO_NATIVE=1
+export SQLD_DB_PATH="traces/v3/crafter_eval.db"
+export SFT_OUTPUT_DIR="ft_data/crafter_sft"
+
+echo "Running evaluation..."
+uv run synth-ai eval grpo-crafter \
+  --config examples/task_apps/crafter/eval_image_only_gpt4o.toml
+
+# 2. Filter for successful rollouts
+echo "Filtering for successful rollouts..."
+mkdir -p ft_data/crafter_sft_filtered
+
+sqlite3 traces/v3/crafter_eval.db \
+  "SELECT session_id FROM outcome_rewards WHERE total_reward > 0" | \
+while read session_id; do
+  if [ -f "ft_data/crafter_sft/sft_${session_id}.jsonl" ]; then
+    cp "ft_data/crafter_sft/sft_${session_id}.jsonl" ft_data/crafter_sft_filtered/
+  fi
+done
+
+# 3. Combine into single dataset
+echo "Creating combined dataset..."
+cat ft_data/crafter_sft_filtered/sft_*.jsonl > ft_data/crafter_high_reward.jsonl
+
+# 4. Report statistics
+echo ""
+echo "=== SFT Dataset Created ==="
+echo "Total examples: $(wc -l < ft_data/crafter_high_reward.jsonl)"
+echo "Location: ft_data/crafter_high_reward.jsonl"
+echo ""
+echo "Rollouts included:"
+sqlite3 traces/v3/crafter_eval.db \
+  "SELECT 
+    COUNT(*) as count,
+    SUM(total_reward) as total_reward,
+    AVG(achievements_count) as avg_achievements
+   FROM outcome_rewards 
+   WHERE total_reward > 0"
+```
+
+## Troubleshooting
+
+### No SFT Files Created
+
+**Issue**: `ft_data/crafter_sft/` is empty after evaluation
+
+**Possible causes**:
+1. `SFT_OUTPUT_DIR` environment variable not set
+2. Rollout doesn't record SFT by default in eval mode
+3. Directory permissions issue
+
+**Debug**:
+```bash
+# Check if variable is set
+echo $SFT_OUTPUT_DIR
+
+# Check directory exists and is writable
+ls -la ft_data/
+
+# Try with explicit path
+export SFT_OUTPUT_DIR="/Users/joshpurtell/Documents/GitHub/synth-ai/ft_data/crafter_sft"
+```
+
+### SFT Files Don't Match Successful Rollouts
+
+**Issue**: Have SFT files for rollouts with 0 rewards
+
+**Solution**: This is expected - SFT is recorded for all rollouts. Use the filtering step to keep only successful ones.
+
+## Future Work
+
+To enable the `synth-ai filter` command for Crafter traces:
+
+1. **Modify Crafter rollout** to record messages to database:
+```python
+# In RolloutTracingContext
+await self.tracer.record_message(
+    content=user_prompt,
+    message_type="user",
+    metadata={"turn": turn}
+)
+
+await self.tracer.record_message(
+    content=assistant_response,
+    message_type="assistant", 
+    metadata={"turn": turn, "reward": step_reward}
+)
+```
+
+2. **Update filter command** to query messages table:
+```python
+# Instead of looking for metadata.prompt/completion
+# Query messages table directly
+messages = await tracer.db.get_messages(session_id)
+```
+
+3. **Create filter config** that works:
+```toml
+[filter]
+db = "traces/v3/crafter_eval.db"
+output = "ft_data/crafter_filtered.jsonl"
+min_official_score = 0.01  # Filter by outcome_rewards
+```
+
+## See Also
+
+- `README_IMAGE_ONLY_EVAL.md` - How to run evaluations
+- `EVAL_IMAGE_ONLY_RESULTS.md` - Example results
+- `QUERY_EXAMPLES.md` - SQL queries for trace analysis
+
+

examples/task_apps/crafter/EVAL_IMAGE_ONLY_RESULTS.md

@@ -0,0 +1,152 @@
+# Crafter Image-Only Eval Results
+
+## Summary
+Successfully ran 10 rollouts of the Crafter task app using **image-only input** (no text observations), with full tracing and rewards saved to Turso database.
+
+## Configuration
+- **Model**: `gpt-4o-mini-2024-07-18`
+- **Input Mode**: Image-only (vision enabled, text observations disabled)
+- **Max Steps**: 10 per episode
+- **Max LLM Calls**: 10 per rollout
+- **Seeds**: 0-9 (10 rollouts)
+- **Tracing**: Enabled with Turso/libsql (MVCC concurrent writes)
+- **Database**: `traces/v3/crafter_eval.db` (1.7MB)
+
+## Results
+
+### Overall Performance
+- **Total Rollouts**: 10
+- **Success Rate**: 100% (10/10 completed)
+- **Mean Official Score**: 0.700 (70%)
+- **Rollouts with Achievements**: 7/10 (70%)
+
+### Achievement Distribution
+| Achievements Count | Number of Rollouts |
+|-------------------|-------------------|
+| 3                 | 1                 |
+| 2                 | 4                 |
+| 1                 | 2                 |
+| 0                 | 3                 |
+
+### Top Performing Rollouts
+1. **Seed 0** - 3 achievements: `collect_drink`, `collect_sapling`, `collect_wood` (reward: 3)
+2. **Seed 1** - 2 achievements: `collect_sapling`, `collect_wood` (reward: 2)
+3. **Seed 3** - 2 achievements: `collect_sapling`, `collect_wood` (reward: 2)
+4. **Seed 6** - 2 achievements: `collect_sapling`, `collect_wood` (reward: 2)
+5. **Seed 9** - 2 achievements: `collect_sapling`, `collect_wood` (reward: 2)
+6. **Seed 4** - 1 achievement: `collect_wood` (reward: 1)
+7. **Seed 7** - 1 achievement: `collect_wood` (reward: 1)
+
+### Rollouts with No Achievements
+- Seed 2, 5, 8 - No achievements earned
+
+## Database Schema
+
+### outcome_rewards Table
+```sql
+CREATE TABLE outcome_rewards (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    session_id VARCHAR NOT NULL,
+    total_reward INTEGER NOT NULL,
+    achievements_count INTEGER NOT NULL,
+    total_steps INTEGER NOT NULL,
+    created_at DATETIME NOT NULL,
+    reward_metadata TEXT,
+    FOREIGN KEY(session_id) REFERENCES session_traces(session_id)
+);
+CREATE INDEX idx_outcome_rewards_session ON outcome_rewards (session_id);
+CREATE INDEX idx_outcome_rewards_total ON outcome_rewards (total_reward);
+```
+
+## Query Examples
+
+### Get rollouts with achievements > 0
+```sql
+SELECT 
+    st.session_id, 
+    st.num_timesteps, 
+    orw.achievements_count, 
+    orw.total_reward,
+    json_extract(orw.reward_metadata, '$.final_achievements') as achievements
+FROM session_traces st
+INNER JOIN outcome_rewards orw ON st.session_id = orw.session_id
+WHERE orw.achievements_count > 0
+ORDER BY orw.achievements_count DESC, orw.total_reward DESC;
+```
+
+### Count rollouts by achievement count
+```sql
+SELECT achievements_count, COUNT(*) as count
+FROM outcome_rewards
+GROUP BY achievements_count
+ORDER BY achievements_count DESC;
+```
+
+### Get top performers
+```sql
+SELECT session_id, total_reward, achievements_count, reward_metadata
+FROM outcome_rewards
+WHERE achievements_count > 0 OR total_reward > 0
+ORDER BY achievements_count DESC, total_reward DESC
+LIMIT 10;
+```
+
+## Key Changes Made
+
+### 1. OpenAI Authorization Fix
+Updated `openai_client.py` to properly set `Authorization: Bearer` header for OpenAI API calls:
+```python
+# If calling OpenAI directly (api.openai.com)
+if "api.openai.com" in low_url:
+    openai_key = os.getenv("OPENAI_API_KEY")
+    if openai_key and isinstance(openai_key, str):
+        headers["Authorization"] = f"Bearer {openai_key}"
+```
+
+### 2. Image-Only Mode Implementation
+Added `image_only_mode` support to `CrafterPolicy` and `CrafterReActAgent`:
+- When enabled, only image observations are sent to the LLM
+- Text observations are set to empty string
+- Vision mode is automatically enabled
+
+### 3. Trace Format Support
+Fixed CLI to properly handle both "compact" and "full" trace formats:
+```python
+# Handle both "compact" and "full" trace formats
+session_trace_dict = trace_namespace.get("session_trace")
+if not isinstance(session_trace_dict, dict):
+    if "session_id" in trace_namespace:
+        session_trace_dict = trace_namespace
+```
+
+### 4. Request Body Structure
+Fixed rollout request to properly nest tracing parameters:
+```python
+"record": {
+    "return_trace": True,
+    "trace_format": "full",
+}
+```
+
+## Files Modified
+1. `/Users/joshpurtell/Documents/GitHub/synth-ai/examples/task_apps/crafter/task_app/synth_envs_hosted/inference/openai_client.py`
+2. `/Users/joshpurtell/Documents/GitHub/synth-ai/examples/task_apps/crafter/task_app/synth_envs_hosted/envs/crafter/policy.py`
+3. `/Users/joshpurtell/Documents/GitHub/synth-ai/examples/task_apps/crafter/task_app/synth_envs_hosted/envs/crafter/react_agent.py`
+4. `/Users/joshpurtell/Documents/GitHub/synth-ai/synth_ai/cli/task_apps.py`
+5. `/Users/joshpurtell/Documents/GitHub/synth-ai/examples/task_apps/crafter/eval_image_only_gpt4o.toml`
+
+## Verification
+- ✅ All 10 rollouts completed successfully
+- ✅ Image-only input confirmed (base64 PNG images in prompts)
+- ✅ Achievements computed and saved
+- ✅ Foreign keys working (can join session_traces and outcome_rewards)
+- ✅ Can query rollouts by achievement count and rewards
+- ✅ Database size: 1.7MB with full trace data
+
+## Next Steps
+- Increase `max_steps_per_episode` for longer episodes
+- Try different models (e.g., gpt-4o, claude-3.5-sonnet)
+- Analyze which actions lead to the most achievements
+- Use concurrent writes with higher concurrency (Turso MVCC supports this)
+
+