synth-ai 0.2.13.dev1__py3-none-any.whl → 0.2.14__py3-none-any.whl

examples/task_apps/crafter/CREATE_SFT_DATASET.md

@@ -0,0 +1,273 @@
+# Creating SFT Datasets from Crafter Traces
+
+There are two approaches to create SFT (Supervised Fine-Tuning) datasets from Crafter rollouts:
+
+## Approach 1: Direct SFT Recording (Recommended)
+
+Crafter's rollout system can write SFT-ready JSONL files directly during evaluation by setting the `sft_output_dir`.
+
+### Setup
+
+1. Set the SFT output directory environment variable:
+```bash
+export SFT_OUTPUT_DIR="ft_data/crafter_sft"
+```
+
+2. Run evaluation:
+```bash
+cd /Users/joshpurtell/Documents/GitHub/synth-ai
+
+export TASKAPP_TRACING_ENABLED=1
+export TURSO_NATIVE=1
+export SQLD_DB_PATH="traces/v3/crafter_eval.db"
+export SFT_OUTPUT_DIR="ft_data/crafter_sft"  # Enable SFT recording
+
+uv run synth-ai eval grpo-crafter \
+  --config examples/task_apps/crafter/eval_image_only_gpt4o.toml
+```
+
+3. SFT files will be written to:
+```
+ft_data/crafter_sft/
+├── sft_<run_id_1>.jsonl
+├── sft_<run_id_2>.jsonl
+├── ...
+└── sft_<run_id_10>.jsonl
+```
+
+### SFT Record Format
+
+Each JSONL file contains records like:
+```json
+{
+  "messages": [
+    {"role": "system", "content": "...system prompt..."},
+    {"role": "user", "content": "...observation..."},
+    {"role": "assistant", "content": "...action..."}
+  ],
+  "metadata": {
+    "run_id": "...",
+    "turn": 5,
+    "reward": 1.0,
+    ...
+  }
+}
+```
+
+### Combine Multiple Files
+
+```bash
+# Combine all SFT files into one
+cat ft_data/crafter_sft/sft_*.jsonl > ft_data/crafter_combined.jsonl
+
+# Count examples
+wc -l ft_data/crafter_combined.jsonl
+```
+
+## Approach 2: Extract from Turso Database (Not Currently Supported)
+
+The `synth-ai filter` command is designed for traces with a different structure (where prompt/completion are stored in session metadata). 
+
+**Current Limitation**: Crafter's SessionTracer-based traces don't store messages in the format expected by the filter command.
+
+### Why Filter Doesn't Work
+
+The filter command expects:
+```python
+metadata = {
+    "prompt": "...",  # User message
+    "completion": "..."  # Assistant response
+}
+```
+
+But Crafter traces store:
+- Messages in separate `messages` table (currently 0 messages - not recorded during eval)
+- Rewards in `outcome_rewards` table
+- Metadata without prompt/completion fields
+
+### Future Enhancement
+
+To make filter work with Crafter traces, we would need to:
+1. Modify rollout to record messages to the `messages` table
+2. Update filter command to query `messages` table directly
+3. Join with `outcome_rewards` to filter by achievements
+
+## Comparison
+
+| Feature | Direct SFT | Filter Command |
+|---------|-----------|----------------|
+| **Setup** | Set `SFT_OUTPUT_DIR` | Create filter config |
+| **When** | During rollout | After rollout |
+| **Format** | JSONL per rollout | Combined JSONL |
+| **Filtering** | Manual (combine files) | Automatic (SQL queries) |
+| **Status** | ✅ Works now | ❌ Needs implementation |
+
+## Recommended Workflow
+
+### 1. Run evaluation with SFT recording:
+```bash
+export SFT_OUTPUT_DIR="ft_data/crafter_sft"
+uv run synth-ai eval grpo-crafter --config examples/task_apps/crafter/eval_image_only_gpt4o.toml
+```
+
+### 2. Filter for successful rollouts:
+
+Since we can't use the filter command yet, manually select files:
+
+```bash
+# Query database to find session_ids with rewards
+sqlite3 traces/v3/crafter_eval.db \
+  "SELECT session_id FROM outcome_rewards WHERE total_reward > 0" \
+  > successful_sessions.txt
+
+# Create directory for filtered SFT
+mkdir -p ft_data/crafter_sft_filtered
+
+# Copy only successful rollout SFT files
+while read session_id; do
+  if [ -f "ft_data/crafter_sft/sft_${session_id}.jsonl" ]; then
+    cp "ft_data/crafter_sft/sft_${session_id}.jsonl" ft_data/crafter_sft_filtered/
+  fi
+done < successful_sessions.txt
+
+# Combine filtered files
+cat ft_data/crafter_sft_filtered/sft_*.jsonl > ft_data/crafter_high_reward.jsonl
+
+echo "Created SFT dataset: ft_data/crafter_high_reward.jsonl"
+wc -l ft_data/crafter_high_reward.jsonl
+```
+
+### 3. Verify dataset:
+
+```bash
+# Look at first example
+head -1 ft_data/crafter_high_reward.jsonl | jq .
+
+# Count examples
+wc -l ft_data/crafter_high_reward.jsonl
+
+# Check message types
+jq -r '.messages[].role' ft_data/crafter_high_reward.jsonl | sort | uniq -c
+```
+
+## Example: Complete Pipeline
+
+```bash
+#!/bin/bash
+# complete_sft_pipeline.sh
+
+cd /Users/joshpurtell/Documents/GitHub/synth-ai
+
+# 1. Run evaluation with SFT recording
+export TASKAPP_TRACING_ENABLED=1
+export TURSO_NATIVE=1
+export SQLD_DB_PATH="traces/v3/crafter_eval.db"
+export SFT_OUTPUT_DIR="ft_data/crafter_sft"
+
+echo "Running evaluation..."
+uv run synth-ai eval grpo-crafter \
+  --config examples/task_apps/crafter/eval_image_only_gpt4o.toml
+
+# 2. Filter for successful rollouts
+echo "Filtering for successful rollouts..."
+mkdir -p ft_data/crafter_sft_filtered
+
+sqlite3 traces/v3/crafter_eval.db \
+  "SELECT session_id FROM outcome_rewards WHERE total_reward > 0" | \
+while read session_id; do
+  if [ -f "ft_data/crafter_sft/sft_${session_id}.jsonl" ]; then
+    cp "ft_data/crafter_sft/sft_${session_id}.jsonl" ft_data/crafter_sft_filtered/
+  fi
+done
+
+# 3. Combine into single dataset
+echo "Creating combined dataset..."
+cat ft_data/crafter_sft_filtered/sft_*.jsonl > ft_data/crafter_high_reward.jsonl
+
+# 4. Report statistics
+echo ""
+echo "=== SFT Dataset Created ==="
+echo "Total examples: $(wc -l < ft_data/crafter_high_reward.jsonl)"
+echo "Location: ft_data/crafter_high_reward.jsonl"
+echo ""
+echo "Rollouts included:"
+sqlite3 traces/v3/crafter_eval.db \
+  "SELECT 
+    COUNT(*) as count,
+    SUM(total_reward) as total_reward,
+    AVG(achievements_count) as avg_achievements
+   FROM outcome_rewards 
+   WHERE total_reward > 0"
+```
+
+## Troubleshooting
+
+### No SFT Files Created
+
+**Issue**: `ft_data/crafter_sft/` is empty after evaluation
+
+**Possible causes**:
+1. `SFT_OUTPUT_DIR` environment variable not set
+2. Rollout doesn't record SFT by default in eval mode
+3. Directory permissions issue
+
+**Debug**:
+```bash
+# Check if variable is set
+echo $SFT_OUTPUT_DIR
+
+# Check directory exists and is writable
+ls -la ft_data/
+
+# Try with explicit path
+export SFT_OUTPUT_DIR="/Users/joshpurtell/Documents/GitHub/synth-ai/ft_data/crafter_sft"
+```
+
+### SFT Files Don't Match Successful Rollouts
+
+**Issue**: Have SFT files for rollouts with 0 rewards
+
+**Solution**: This is expected - SFT is recorded for all rollouts. Use the filtering step to keep only successful ones.
+
+## Future Work
+
+To enable the `synth-ai filter` command for Crafter traces:
+
+1. **Modify Crafter rollout** to record messages to database:
+```python
+# In RolloutTracingContext
+await self.tracer.record_message(
+    content=user_prompt,
+    message_type="user",
+    metadata={"turn": turn}
+)
+
+await self.tracer.record_message(
+    content=assistant_response,
+    message_type="assistant", 
+    metadata={"turn": turn, "reward": step_reward}
+)
+```
+
+2. **Update filter command** to query messages table:
+```python
+# Instead of looking for metadata.prompt/completion
+# Query messages table directly
+messages = await tracer.db.get_messages(session_id)
+```
+
+3. **Create filter config** that works:
+```toml
+[filter]
+db = "traces/v3/crafter_eval.db"
+output = "ft_data/crafter_filtered.jsonl"
+min_official_score = 0.01  # Filter by outcome_rewards
+```
+
+## See Also
+
+- `README_IMAGE_ONLY_EVAL.md` - How to run evaluations
+- `EVAL_IMAGE_ONLY_RESULTS.md` - Example results
+- `QUERY_EXAMPLES.md` - SQL queries for trace analysis
+
+

examples/task_apps/crafter/EVAL_IMAGE_ONLY_RESULTS.md

@@ -0,0 +1,152 @@
+# Crafter Image-Only Eval Results
+
+## Summary
+Successfully ran 10 rollouts of the Crafter task app using **image-only input** (no text observations), with full tracing and rewards saved to Turso database.
+
+## Configuration
+- **Model**: `gpt-4o-mini-2024-07-18`
+- **Input Mode**: Image-only (vision enabled, text observations disabled)
+- **Max Steps**: 10 per episode
+- **Max LLM Calls**: 10 per rollout
+- **Seeds**: 0-9 (10 rollouts)
+- **Tracing**: Enabled with Turso/libsql (MVCC concurrent writes)
+- **Database**: `traces/v3/crafter_eval.db` (1.7MB)
+
+## Results
+
+### Overall Performance
+- **Total Rollouts**: 10
+- **Success Rate**: 100% (10/10 completed)
+- **Mean Official Score**: 0.700 (70%)
+- **Rollouts with Achievements**: 7/10 (70%)
+
+### Achievement Distribution
+| Achievements Count | Number of Rollouts |
+|-------------------|-------------------|
+| 3                 | 1                 |
+| 2                 | 4                 |
+| 1                 | 2                 |
+| 0                 | 3                 |
+
+### Top Performing Rollouts
+1. **Seed 0** - 3 achievements: `collect_drink`, `collect_sapling`, `collect_wood` (reward: 3)
+2. **Seed 1** - 2 achievements: `collect_sapling`, `collect_wood` (reward: 2)
+3. **Seed 3** - 2 achievements: `collect_sapling`, `collect_wood` (reward: 2)
+4. **Seed 6** - 2 achievements: `collect_sapling`, `collect_wood` (reward: 2)
+5. **Seed 9** - 2 achievements: `collect_sapling`, `collect_wood` (reward: 2)
+6. **Seed 4** - 1 achievement: `collect_wood` (reward: 1)
+7. **Seed 7** - 1 achievement: `collect_wood` (reward: 1)
+
+### Rollouts with No Achievements
+- Seed 2, 5, 8 - No achievements earned
+
+## Database Schema
+
+### outcome_rewards Table
+```sql
+CREATE TABLE outcome_rewards (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    session_id VARCHAR NOT NULL,
+    total_reward INTEGER NOT NULL,
+    achievements_count INTEGER NOT NULL,
+    total_steps INTEGER NOT NULL,
+    created_at DATETIME NOT NULL,
+    reward_metadata TEXT,
+    FOREIGN KEY(session_id) REFERENCES session_traces(session_id)
+);
+CREATE INDEX idx_outcome_rewards_session ON outcome_rewards (session_id);
+CREATE INDEX idx_outcome_rewards_total ON outcome_rewards (total_reward);
+```
+
+## Query Examples
+
+### Get rollouts with achievements > 0
+```sql
+SELECT 
+    st.session_id, 
+    st.num_timesteps, 
+    orw.achievements_count, 
+    orw.total_reward,
+    json_extract(orw.reward_metadata, '$.final_achievements') as achievements
+FROM session_traces st
+INNER JOIN outcome_rewards orw ON st.session_id = orw.session_id
+WHERE orw.achievements_count > 0
+ORDER BY orw.achievements_count DESC, orw.total_reward DESC;
+```
+
+### Count rollouts by achievement count
+```sql
+SELECT achievements_count, COUNT(*) as count
+FROM outcome_rewards
+GROUP BY achievements_count
+ORDER BY achievements_count DESC;
+```
+
+### Get top performers
+```sql
+SELECT session_id, total_reward, achievements_count, reward_metadata
+FROM outcome_rewards
+WHERE achievements_count > 0 OR total_reward > 0
+ORDER BY achievements_count DESC, total_reward DESC
+LIMIT 10;
+```
+
+## Key Changes Made
+
+### 1. OpenAI Authorization Fix
+Updated `openai_client.py` to properly set `Authorization: Bearer` header for OpenAI API calls:
+```python
+# If calling OpenAI directly (api.openai.com)
+if "api.openai.com" in low_url:
+    openai_key = os.getenv("OPENAI_API_KEY")
+    if openai_key and isinstance(openai_key, str):
+        headers["Authorization"] = f"Bearer {openai_key}"
+```
+
+### 2. Image-Only Mode Implementation
+Added `image_only_mode` support to `CrafterPolicy` and `CrafterReActAgent`:
+- When enabled, only image observations are sent to the LLM
+- Text observations are set to empty string
+- Vision mode is automatically enabled
+
+### 3. Trace Format Support
+Fixed CLI to properly handle both "compact" and "full" trace formats:
+```python
+# Handle both "compact" and "full" trace formats
+session_trace_dict = trace_namespace.get("session_trace")
+if not isinstance(session_trace_dict, dict):
+    if "session_id" in trace_namespace:
+        session_trace_dict = trace_namespace
+```
+
+### 4. Request Body Structure
+Fixed rollout request to properly nest tracing parameters:
+```python
+"record": {
+    "return_trace": True,
+    "trace_format": "full",
+}
+```
+
+## Files Modified
+1. `/Users/joshpurtell/Documents/GitHub/synth-ai/examples/task_apps/crafter/task_app/synth_envs_hosted/inference/openai_client.py`
+2. `/Users/joshpurtell/Documents/GitHub/synth-ai/examples/task_apps/crafter/task_app/synth_envs_hosted/envs/crafter/policy.py`
+3. `/Users/joshpurtell/Documents/GitHub/synth-ai/examples/task_apps/crafter/task_app/synth_envs_hosted/envs/crafter/react_agent.py`
+4. `/Users/joshpurtell/Documents/GitHub/synth-ai/synth_ai/cli/task_apps.py`
+5. `/Users/joshpurtell/Documents/GitHub/synth-ai/examples/task_apps/crafter/eval_image_only_gpt4o.toml`
+
+## Verification
+- ✅ All 10 rollouts completed successfully
+- ✅ Image-only input confirmed (base64 PNG images in prompts)
+- ✅ Achievements computed and saved
+- ✅ Foreign keys working (can join session_traces and outcome_rewards)
+- ✅ Can query rollouts by achievement count and rewards
+- ✅ Database size: 1.7MB with full trace data
+
+## Next Steps
+- Increase `max_steps_per_episode` for longer episodes
+- Try different models (e.g., gpt-4o, claude-3.5-sonnet)
+- Analyze which actions lead to the most achievements
+- Use concurrent writes with higher concurrency (Turso MVCC supports this)
+
+

examples/task_apps/crafter/FILTER_COMMAND_STATUS.md

@@ -0,0 +1,174 @@
+# Filter Command Status for Crafter
+
+## Summary
+
+The `synth-ai filter` command has been updated to work with Crafter's SessionTracer v3 traces, but there's currently an issue with message persistence that needs to be resolved.
+
+## What Was Changed
+
+### 1. Updated Filter Command (`synth_ai/cli/task_apps.py`)
+
+The filter command now:
+- ✅ Queries `outcome_rewards` table to filter by `total_reward`
+- ✅ Queries `messages` table to extract user/assistant pairs
+- ✅ Falls back to metadata-based filtering for backwards compatibility
+- ✅ Supports filtering by achievements/rewards from Crafter rollouts
+- ✅ Extracts text from structured message content (JSON payloads)
+
+### 2. Created Filter Config
+
+**File**: `examples/task_apps/crafter/filter_sft_dataset.toml`
+
+```toml
+[filter]
+db = "traces/v3/crafter_eval.db"
+output = "ft_data/crafter_image_only_sft.jsonl"
+min_official_score = 0.01  # Only traces with rewards > 0
+```
+
+## Current Issue: Messages Not Being Saved
+
+### Problem
+
+When running evaluations, the database ends up with:
+- ✅ 2 `session_traces` (metadata saved)
+- ✅ 2 `outcome_rewards` (rewards saved)
+- ❌ 0 `messages` (messages NOT saved)
+- ✅ 40 `events` (environment events saved)
+- ✅ 20 `session_timesteps` (timesteps saved)
+
+### Expected Behavior
+
+The rollout code calls:
+1. `tracer.initialize()` - Opens database connection
+2. `tracer.start_session()` - Creates session
+3. `tracer.record_message()` - Records system/user prompts (via `record_policy_prompts`)
+4. `tracer.end_session()` - Saves session with `auto_save=True`
+
+The `insert_session_trace` method (in `NativeLibsqlTraceManager`) SHOULD iterate through `trace.markov_blanket_message_history` and save each message to the `messages` table.
+
+### Actual Behavior
+
+Messages are NOT being persisted to the database, even though:
+- The code path looks correct
+- `end_session()` is being called
+- `auto_save=True` is the default
+- The trace JSON payload includes `markov_blanket_message_history`
+
+### Debugging Observations
+
+1. **Trace payload includes messages**: The eval output shows a large JSON structure with `markov_blanket_messages` containing all the prompts
+2. **No errors logged**: The `try/except` around `end_session()` doesn't log any failures
+3. **Works with both TURSO_NATIVE=0 and TURSO_NATIVE=1**: Neither backend saves messages
+4. **Database is writable**: `outcome_rewards` and `events` are being saved successfully
+
+## Possible Causes
+
+1. **Silent exception during message insertion**: The `insert_message_row` might be failing without raising
+2. **Transaction not committed**: Messages might be inserted but not committed
+3. **Messages not in trace object**: `markov_blanket_message_history` might be empty when `end_session` is called
+4. **Record message not adding to history**: `tracer.record_message()` might not be appending to the list properly
+
+## Next Steps to Fix
+
+### Option 1: Debug Message Persistence
+
+Add logging to trace the message save path:
+
+```python
+# In rollout.py, finalize method
+logger.info(f"[finalize] trace has {len(self.tracer._current_trace.markov_blanket_message_history)} messages before end_session")
+
+# In native_manager.py, insert_session_trace
+logger.info(f"[insert_session_trace] saving {len(trace.markov_blanket_message_history)} messages")
+for msg in trace.markov_blanket_message_history:
+    logger.info(f"  - message type={msg.message_type}")
+    await self.insert_message_row(...)
+    logger.info(f"  - message saved")
+```
+
+### Option 2: Verify Messages Are Being Recorded
+
+Check if `record_policy_prompts` is actually being called and adding messages:
+
+```python
+# In rollout.py, after record_policy_prompts
+if self.tracer and self.tracer._current_trace:
+    msg_count = len(self.tracer._current_trace.markov_blanket_message_history)
+    logger.info(f"[record_policy_prompts] trace now has {msg_count} messages")
+```
+
+### Option 3: Manual Message Recording
+
+As a workaround, explicitly save messages outside of SessionTracer:
+
+```python
+# In finalize(), before end_session()
+if self.enabled and self.tracer is not None:
+    conn = await self.tracer.db.get_connection()
+    for msg in self.tracer._current_trace.markov_blanket_message_history:
+        await conn.execute(
+            "INSERT INTO messages (session_id, message_type, content, timestamp) VALUES (?, ?, ?, ?)",
+            (self.run_id, msg.message_type, str(msg.content), msg.time_record.event_time)
+        )
+    await conn.commit()
+```
+
+### Option 4: Use SFT Records Instead
+
+Crafter already has working SFT record generation that writes directly to JSONL files. Use that instead of the filter command:
+
+```bash
+export SFT_OUTPUT_DIR="ft_data/crafter_sft"
+uv run synth-ai eval grpo-crafter-task-app --config eval_image_only_gpt4o.toml
+
+# Then filter successful runs manually
+cat ft_data/crafter_sft/sft_*.jsonl > ft_data/crafter_combined.jsonl
+```
+
+## Current Workaround
+
+Until message persistence is fixed, use the direct SFT recording approach (Option 4) documented in `CREATE_SFT_DATASET.md`.
+
+## Testing the Filter Command
+
+Once messages are being saved:
+
+```bash
+# 1. Run eval to populate database
+export TASKAPP_TRACING_ENABLED=1
+export TURSO_NATIVE=0
+export SQLD_DB_PATH="traces/v3/crafter_eval.db"
+uv run synth-ai eval grpo-crafter-task-app --config eval_image_only_gpt4o.toml
+
+# 2. Verify messages were saved
+sqlite3 traces/v3/crafter_eval.db "SELECT COUNT(*) FROM messages;"
+# Should be > 0
+
+# 3. Run filter
+uv run synth-ai filter --config filter_sft_dataset.toml
+
+# 4. Check output
+cat ft_data/crafter_image_only_sft.jsonl | jq .
+```
+
+## Related Files
+
+- `synth_ai/cli/task_apps.py` - Filter command implementation (updated)
+- `synth_ai/tracing_v3/session_tracer.py` - SessionTracer class
+- `synth_ai/tracing_v3/turso/native_manager.py` - `insert_session_trace` method (should save messages)
+- `examples/task_apps/crafter/task_app/synth_envs_hosted/rollout.py` - Rollout tracing context
+- `filter_sft_dataset.toml` - Filter configuration
+- `CREATE_SFT_DATASET.md` - Alternative approach using direct SFT recording
+
+## Status
+
+- ✅ Filter command updated to query messages table
+- ✅ Filter command can join with outcome_rewards
+- ✅ Filter config created
+- ❌ Messages not being persisted to database
+- ❌ Filter command cannot extract SFT data without messages
+
+**Action Required**: Debug why messages aren't being saved to the database despite correct code path.
+
+