synth-ai 0.2.13.dev2__py3-none-any.whl → 0.2.16__py3-none-any.whl

examples/task_apps/crafter/FILTER_COMMAND_STATUS.md

@@ -0,0 +1,174 @@
+# Filter Command Status for Crafter
+
+## Summary
+
+The `synth-ai filter` command has been updated to work with Crafter's SessionTracer v3 traces, but there's currently an issue with message persistence that needs to be resolved.
+
+## What Was Changed
+
+### 1. Updated Filter Command (`synth_ai/cli/task_apps.py`)
+
+The filter command now:
+- ✅ Queries `outcome_rewards` table to filter by `total_reward`
+- ✅ Queries `messages` table to extract user/assistant pairs
+- ✅ Falls back to metadata-based filtering for backwards compatibility
+- ✅ Supports filtering by achievements/rewards from Crafter rollouts
+- ✅ Extracts text from structured message content (JSON payloads)
+
+### 2. Created Filter Config
+
+**File**: `examples/task_apps/crafter/filter_sft_dataset.toml`
+
+```toml
+[filter]
+db = "traces/v3/crafter_eval.db"
+output = "ft_data/crafter_image_only_sft.jsonl"
+min_official_score = 0.01  # Only traces with rewards > 0
+```
+
+## Current Issue: Messages Not Being Saved
+
+### Problem
+
+When running evaluations, the database ends up with:
+- ✅ 2 `session_traces` (metadata saved)
+- ✅ 2 `outcome_rewards` (rewards saved)
+- ❌ 0 `messages` (messages NOT saved)
+- ✅ 40 `events` (environment events saved)
+- ✅ 20 `session_timesteps` (timesteps saved)
+
+### Expected Behavior
+
+The rollout code calls:
+1. `tracer.initialize()` - Opens database connection
+2. `tracer.start_session()` - Creates session
+3. `tracer.record_message()` - Records system/user prompts (via `record_policy_prompts`)
+4. `tracer.end_session()` - Saves session with `auto_save=True`
+
+The `insert_session_trace` method (in `NativeLibsqlTraceManager`) SHOULD iterate through `trace.markov_blanket_message_history` and save each message to the `messages` table.
+
+### Actual Behavior
+
+Messages are NOT being persisted to the database, even though:
+- The code path looks correct
+- `end_session()` is being called
+- `auto_save=True` is the default
+- The trace JSON payload includes `markov_blanket_message_history`
+
+### Debugging Observations
+
+1. **Trace payload includes messages**: The eval output shows a large JSON structure with `markov_blanket_messages` containing all the prompts
+2. **No errors logged**: The `try/except` around `end_session()` doesn't log any failures
+3. **Works with both TURSO_NATIVE=0 and TURSO_NATIVE=1**: Neither backend saves messages
+4. **Database is writable**: `outcome_rewards` and `events` are being saved successfully
+
+## Possible Causes
+
+1. **Silent exception during message insertion**: The `insert_message_row` might be failing without raising
+2. **Transaction not committed**: Messages might be inserted but not committed
+3. **Messages not in trace object**: `markov_blanket_message_history` might be empty when `end_session` is called
+4. **Record message not adding to history**: `tracer.record_message()` might not be appending to the list properly
+
+## Next Steps to Fix
+
+### Option 1: Debug Message Persistence
+
+Add logging to trace the message save path:
+
+```python
+# In rollout.py, finalize method
+logger.info(f"[finalize] trace has {len(self.tracer._current_trace.markov_blanket_message_history)} messages before end_session")
+
+# In native_manager.py, insert_session_trace
+logger.info(f"[insert_session_trace] saving {len(trace.markov_blanket_message_history)} messages")
+for msg in trace.markov_blanket_message_history:
+    logger.info(f"  - message type={msg.message_type}")
+    await self.insert_message_row(...)
+    logger.info(f"  - message saved")
+```
+
+### Option 2: Verify Messages Are Being Recorded
+
+Check if `record_policy_prompts` is actually being called and adding messages:
+
+```python
+# In rollout.py, after record_policy_prompts
+if self.tracer and self.tracer._current_trace:
+    msg_count = len(self.tracer._current_trace.markov_blanket_message_history)
+    logger.info(f"[record_policy_prompts] trace now has {msg_count} messages")
+```
+
+### Option 3: Manual Message Recording
+
+As a workaround, explicitly save messages outside of SessionTracer:
+
+```python
+# In finalize(), before end_session()
+if self.enabled and self.tracer is not None:
+    conn = await self.tracer.db.get_connection()
+    for msg in self.tracer._current_trace.markov_blanket_message_history:
+        await conn.execute(
+            "INSERT INTO messages (session_id, message_type, content, timestamp) VALUES (?, ?, ?, ?)",
+            (self.run_id, msg.message_type, str(msg.content), msg.time_record.event_time)
+        )
+    await conn.commit()
+```
+
+### Option 4: Use SFT Records Instead
+
+Crafter already has working SFT record generation that writes directly to JSONL files. Use that instead of the filter command:
+
+```bash
+export SFT_OUTPUT_DIR="ft_data/crafter_sft"
+uv run synth-ai eval grpo-crafter-task-app --config eval_image_only_gpt4o.toml
+
+# Then filter successful runs manually
+cat ft_data/crafter_sft/sft_*.jsonl > ft_data/crafter_combined.jsonl
+```
+
+## Current Workaround
+
+Until message persistence is fixed, use the direct SFT recording approach (Option 4) documented in `CREATE_SFT_DATASET.md`.
+
+## Testing the Filter Command
+
+Once messages are being saved:
+
+```bash
+# 1. Run eval to populate database
+export TASKAPP_TRACING_ENABLED=1
+export TURSO_NATIVE=0
+export SQLD_DB_PATH="traces/v3/crafter_eval.db"
+uv run synth-ai eval grpo-crafter-task-app --config eval_image_only_gpt4o.toml
+
+# 2. Verify messages were saved
+sqlite3 traces/v3/crafter_eval.db "SELECT COUNT(*) FROM messages;"
+# Should be > 0
+
+# 3. Run filter
+uv run synth-ai filter --config filter_sft_dataset.toml
+
+# 4. Check output
+cat ft_data/crafter_image_only_sft.jsonl | jq .
+```
+
+## Related Files
+
+- `synth_ai/cli/task_apps.py` - Filter command implementation (updated)
+- `synth_ai/tracing_v3/session_tracer.py` - SessionTracer class
+- `synth_ai/tracing_v3/turso/native_manager.py` - `insert_session_trace` method (should save messages)
+- `examples/task_apps/crafter/task_app/synth_envs_hosted/rollout.py` - Rollout tracing context
+- `filter_sft_dataset.toml` - Filter configuration
+- `CREATE_SFT_DATASET.md` - Alternative approach using direct SFT recording
+
+## Status
+
+- ✅ Filter command updated to query messages table
+- ✅ Filter command can join with outcome_rewards
+- ✅ Filter config created
+- ❌ Messages not being persisted to database
+- ❌ Filter command cannot extract SFT data without messages
+
+**Action Required**: Debug why messages aren't being saved to the database despite correct code path.
+
+

examples/task_apps/crafter/FILTER_COMMAND_SUCCESS.md

@@ -0,0 +1,268 @@
+# Filter Command Success - SFT Dataset Creation Working!
+
+## ✅ Complete Success!
+
+The `uvx synth-ai eval` → `uvx synth-ai filter` loop is now working end-to-end for Crafter!
+
+## What Was Fixed
+
+### Issue 1: Early Return in `insert_session_trace`
+**Problem**: Sessions created by `start_session` already existed in the database, so `insert_session_trace` returned early without saving messages.
+
+**Fix**: Modified `/Users/joshpurtell/Documents/GitHub/synth-ai/synth_ai/tracing_v3/turso/native_manager.py` to continue processing messages even when the session already exists:
+
+```python
+if session_exists:
+    # Update metadata but don't return early
+    # Continue to save messages
+```
+
+### Issue 2: Invalid Message Types
+**Problem**: Crafter was using custom message types (`policy_system_prompt`, `policy_user_prompt`, `policy_tool_call`) that violated the database CHECK constraint.
+
+**Fix**: Modified `/Users/joshpurtell/Documents/GitHub/synth-ai/examples/task_apps/crafter/task_app/synth_envs_hosted/rollout.py` to use standard message types:
+- `policy_system_prompt` → `system`
+- `policy_user_prompt` → `user`  
+- `policy_tool_call` → `assistant` (with `is_tool_call: true` metadata)
+
+## Full Working Pipeline
+
+### 1. Run Evaluation with Tracing
+
+```bash
+cd /Users/joshpurtell/Documents/GitHub/synth-ai
+
+export TASKAPP_TRACING_ENABLED=1
+export TURSO_NATIVE=0
+export SQLD_DB_PATH="traces/v3/crafter_eval.db"
+
+uv run synth-ai eval grpo-crafter-task-app \
+  --config examples/task_apps/crafter/eval_image_only_gpt4o.toml
+```
+
+**Result**:
+- ✅ 2 rollouts completed
+- ✅ 120 messages saved to database (40 system + 40 user + 40 assistant)
+- ✅ 2 outcome_rewards saved with achievements
+- ✅ Traces returned successfully
+
+### 2. Filter to Create SFT Dataset
+
+```bash
+uv run synth-ai filter \
+  --config examples/task_apps/crafter/filter_sft_dataset.toml
+```
+
+**Result**:
+```
+Wrote 40 examples -> ft_data/crafter_image_only_sft.jsonl
+```
+
+### 3. Verify SFT Data
+
+```bash
+# Check first example
+head -1 ft_data/crafter_image_only_sft.jsonl | jq .
+
+# Count examples
+wc -l ft_data/crafter_image_only_sft.jsonl
+```
+
+## SFT Dataset Format
+
+Each line in the JSONL contains:
+
+```json
+{
+  "messages": [
+    {
+      "role": "user",
+      "content": "=== CRAFTER GAME STATE ===\nStep: 0/10000\n..."
+    },
+    {
+      "role": "assistant",
+      "content": "[{'tool_name': 'interact_many', 'arguments': {...}}]"
+    }
+  ],
+  "metadata": {
+    "session_id": "...",
+    "env_name": "crafter",
+    "policy_name": "crafter-react",
+    "seed": 0,
+    "total_reward": 1,
+    "achievements_count": 1,
+    "created_at": "2025-10-22T23:55:25.533188+00:00"
+  }
+}
+```
+
+## Database Schema
+
+The filter command queries these tables:
+
+### messages table
+```sql
+SELECT message_type, content, timestamp 
+FROM messages 
+WHERE session_id = :session_id
+ORDER BY timestamp ASC
+```
+
+- ✅ 120 messages total
+- System (40) + User (40) + Assistant (40) messages
+- Pairs extracted: user → assistant
+
+### outcome_rewards table
+```sql
+SELECT total_reward, achievements_count 
+FROM outcome_rewards 
+WHERE session_id = :session_id
+```
+
+- Used to filter for successful rollouts
+- `min_official_score = 0.01` filters for rewards > 0
+- Both rollouts had `total_reward = 1` (1 achievement each)
+
+## Filter Configuration
+
+**File**: `examples/task_apps/crafter/filter_sft_dataset.toml`
+
+```toml
+[filter]
+db = "traces/v3/crafter_eval.db"
+output = "ft_data/crafter_image_only_sft.jsonl"
+min_official_score = 0.01  # Only traces with rewards > 0
+```
+
+### Available Filter Options
+
+```toml
+[filter]
+db = "path/to/traces.db"              # Required
+output = "path/to/output.jsonl"        # Required
+
+# Optional filters
+min_official_score = 0.01              # Filter by reward
+splits = ["train", "test"]             # Filter by split
+task_ids = ["task_1"]                  # Filter by task
+models = ["gpt-4o"]                    # Filter by model
+limit = 100                            # Limit number of examples
+```
+
+## Statistics
+
+From 2 rollouts with 10 turns each:
+
+| Metric | Count |
+|--------|-------|
+| Total rollouts | 2 |
+| Rollouts with rewards | 2 (100%) |
+| Total messages saved | 120 |
+| System messages | 40 |
+| User messages | 40 |
+| Assistant messages | 40 |
+| **SFT examples** | **40** |
+| Average turns per rollout | 10 |
+| Examples per rollout | 20 |
+
+## Next Steps
+
+### Scale Up
+
+Run with more seeds for a larger dataset:
+
+```toml
+# In eval_image_only_gpt4o.toml
+seeds = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]  # 10 rollouts
+max_turns = 50                           # More examples per rollout
+```
+
+Expected output: ~1000 SFT examples from 10 rollouts @ 50 turns each
+
+### Use the SFT Data
+
+```bash
+# For OpenAI fine-tuning
+# The JSONL format is compatible with OpenAI's fine-tuning API
+
+# For local fine-tuning
+# Convert to your preferred format (HuggingFace, etc.)
+```
+
+### Filter Variations
+
+```bash
+# Only high-reward traces
+min_official_score = 2.0
+
+# Only specific achievements
+# Query manually then filter by session_id
+
+# Time-based filtering
+min_created_at = "2025-10-22T00:00:00"
+```
+
+## Files Modified
+
+1. **`synth_ai/tracing_v3/turso/native_manager.py`**
+   - Fixed early return when session exists
+   - Added logging for debugging
+
+2. **`examples/task_apps/crafter/task_app/synth_envs_hosted/rollout.py`**
+   - Changed message types to standard values
+   - Added debug logging
+
+3. **`synth_ai/cli/task_apps.py`**
+   - Updated filter command to query messages table
+   - Added support for outcome_rewards filtering
+   - Fixed SQL parameter format
+
+4. **`examples/task_apps/crafter/filter_sft_dataset.toml`**
+   - Created filter configuration
+
+## Troubleshooting
+
+### No messages in database
+
+**Check**:
+```bash
+sqlite3 traces/v3/crafter_eval.db "SELECT COUNT(*) FROM messages;"
+```
+
+**Fix**: Ensure `TASKAPP_TRACING_ENABLED=1` and `TURSO_NATIVE=0`
+
+### Filter returns no examples
+
+**Check**:
+```bash
+sqlite3 traces/v3/crafter_eval.db \
+  "SELECT COUNT(*) FROM outcome_rewards WHERE total_reward > 0;"
+```
+
+**Fix**: Lower `min_official_score` or remove it to include all traces
+
+### Invalid message types
+
+**Error**: `CHECK constraint failed: message_type IN (...)`
+
+**Fix**: Already fixed in rollout.py - update to latest code
+
+## Related Documentation
+
+- `README_IMAGE_ONLY_EVAL.md` - How to run evaluations
+- `EVAL_IMAGE_ONLY_RESULTS.md` - Example evaluation results
+- `QUERY_EXAMPLES.md` - SQL query examples
+- `CREATE_SFT_DATASET.md` - Original approach (now superseded)
+
+## Success Metrics
+
+✅ Eval completes without errors  
+✅ Messages saved to database (system, user, assistant)  
+✅ Outcome rewards saved with foreign keys  
+✅ Filter command extracts user/assistant pairs  
+✅ SFT JSONL created with proper format  
+✅ Metadata includes rewards and achievements  
+
+**Status**: 🎉 **WORKING END-TO-END!**
+
+

examples/task_apps/crafter/QUERY_EXAMPLES.md

@@ -0,0 +1,203 @@
+# Crafter Eval Database Query Examples
+
+## Database Location
+```bash
+/Users/joshpurtell/Documents/GitHub/synth-ai/traces/v3/crafter_eval.db
+```
+
+## Quick Stats
+
+Run this query to get an overview:
+```sql
+SELECT 
+    'Total rollouts' as metric, 
+    CAST(COUNT(*) as TEXT) as value 
+FROM outcome_rewards
+UNION ALL
+SELECT 
+    'Rollouts with reward > 0', 
+    CAST(COUNT(*) as TEXT)
+FROM outcome_rewards 
+WHERE total_reward > 0
+UNION ALL
+SELECT 
+    'Average reward', 
+    CAST(ROUND(AVG(total_reward), 2) as TEXT)
+FROM outcome_rewards
+UNION ALL
+SELECT 
+    'Max reward', 
+    CAST(MAX(total_reward) as TEXT)
+FROM outcome_rewards;
+```
+
+**Current Results:**
+- Total rollouts: 10
+- Rollouts with reward > 0: 7
+- Average reward: 1.3
+- Max reward: 3
+
+## Filter for Non-Zero Rewards
+
+### Simple Query
+```sql
+SELECT 
+    session_id,
+    total_reward,
+    achievements_count,
+    json_extract(reward_metadata, '$.env_seed') as seed,
+    json_extract(reward_metadata, '$.final_achievements') as achievements
+FROM outcome_rewards
+WHERE total_reward > 0
+ORDER BY total_reward DESC, achievements_count DESC;
+```
+
+### With Full Session Context
+```sql
+SELECT 
+    st.session_id,
+    st.created_at,
+    st.num_timesteps,
+    st.num_events,
+    orw.total_reward,
+    orw.achievements_count,
+    json_extract(orw.reward_metadata, '$.final_achievements') as achievements,
+    json_extract(orw.reward_metadata, '$.env_seed') as seed
+FROM session_traces st
+INNER JOIN outcome_rewards orw ON st.session_id = orw.session_id
+WHERE orw.total_reward > 0
+ORDER BY orw.total_reward DESC;
+```
+
+## Filter by Achievement Count
+
+### Get rollouts with 2+ achievements
+```sql
+SELECT 
+    session_id,
+    total_reward,
+    achievements_count,
+    json_extract(reward_metadata, '$.final_achievements') as achievements
+FROM outcome_rewards
+WHERE achievements_count >= 2
+ORDER BY achievements_count DESC, total_reward DESC;
+```
+
+### Get rollouts with specific achievement
+```sql
+SELECT 
+    session_id,
+    total_reward,
+    achievements_count,
+    json_extract(reward_metadata, '$.final_achievements') as achievements
+FROM outcome_rewards
+WHERE reward_metadata LIKE '%collect_drink%'
+ORDER BY total_reward DESC;
+```
+
+## Group by Achievement Count
+```sql
+SELECT 
+    achievements_count,
+    COUNT(*) as num_rollouts,
+    ROUND(AVG(total_reward), 2) as avg_reward,
+    SUM(total_reward) as total_reward_sum,
+    GROUP_CONCAT(DISTINCT json_extract(reward_metadata, '$.env_seed')) as seeds
+FROM outcome_rewards
+GROUP BY achievements_count
+ORDER BY achievements_count DESC;
+```
+
+## Top Performers
+```sql
+SELECT 
+    json_extract(orw.reward_metadata, '$.env_seed') as seed,
+    orw.total_reward,
+    orw.achievements_count,
+    orw.total_steps,
+    st.num_events,
+    json_extract(orw.reward_metadata, '$.final_achievements') as achievements
+FROM session_traces st
+INNER JOIN outcome_rewards orw ON st.session_id = orw.session_id
+ORDER BY orw.total_reward DESC, orw.achievements_count DESC
+LIMIT 5;
+```
+
+## Get Event Details for High-Reward Rollouts
+```sql
+SELECT 
+    e.event_type,
+    e.model_name,
+    e.input_tokens,
+    e.output_tokens,
+    e.latency_ms,
+    e.reward as step_reward
+FROM events e
+INNER JOIN outcome_rewards orw ON e.session_id = orw.session_id
+WHERE orw.total_reward >= 2
+ORDER BY e.session_id, e.id
+LIMIT 20;
+```
+
+## Running Queries
+
+### From Command Line
+```bash
+cd /Users/joshpurtell/Documents/GitHub/synth-ai
+sqlite3 traces/v3/crafter_eval.db "YOUR_QUERY_HERE"
+```
+
+### With Formatted Output
+```bash
+sqlite3 -header -column traces/v3/crafter_eval.db "YOUR_QUERY_HERE"
+```
+
+### With JSON Output
+```bash
+sqlite3 -json traces/v3/crafter_eval.db "YOUR_QUERY_HERE" | jq .
+```
+
+## Example: Get CSV Export of Non-Zero Rewards
+```bash
+sqlite3 -header -csv traces/v3/crafter_eval.db \
+  "SELECT 
+    json_extract(reward_metadata, '$.env_seed') as seed,
+    total_reward,
+    achievements_count,
+    total_steps,
+    json_extract(reward_metadata, '$.final_achievements') as achievements
+   FROM outcome_rewards 
+   WHERE total_reward > 0 
+   ORDER BY total_reward DESC" \
+  > crafter_rewards_nonzero.csv
+```
+
+## Current Data Summary
+
+| Reward | Count | Seeds | Achievements |
+|--------|-------|-------|--------------|
+| 3 | 1 | 0 | collect_drink, collect_sapling, collect_wood |
+| 2 | 4 | 1,3,6,9 | collect_sapling, collect_wood |
+| 1 | 2 | 4,7 | collect_wood |
+| 0 | 3 | 2,5,8 | none |
+
+## Verifying Foreign Keys Work
+
+```sql
+-- This should return 7 rows (all rollouts with rewards > 0)
+SELECT COUNT(*) 
+FROM session_traces st
+INNER JOIN outcome_rewards orw ON st.session_id = orw.session_id
+WHERE orw.total_reward > 0;
+
+-- This should return the same 7 session_ids
+SELECT st.session_id 
+FROM session_traces st
+WHERE st.session_id IN (
+  SELECT session_id FROM outcome_rewards WHERE total_reward > 0
+);
+```
+
+✅ **Confirmed: Foreign keys are working correctly and can be used to join tables!**
+
+