synth-ai 0.2.14__py3-none-any.whl → 0.4.1__py3-none-any.whl

examples/task_apps/crafter/FILTER_COMMAND_SUCCESS.md

@@ -1,268 +0,0 @@
-# Filter Command Success - SFT Dataset Creation Working!
-
-## ✅ Complete Success!
-
-The `uvx synth-ai eval` → `uvx synth-ai filter` loop is now working end-to-end for Crafter!
-
-## What Was Fixed
-
-### Issue 1: Early Return in `insert_session_trace`
-**Problem**: Sessions created by `start_session` already existed in the database, so `insert_session_trace` returned early without saving messages.
-
-**Fix**: Modified `/Users/joshpurtell/Documents/GitHub/synth-ai/synth_ai/tracing_v3/turso/native_manager.py` to continue processing messages even when the session already exists:
-
-```python
-if session_exists:
-    # Update metadata but don't return early
-    # Continue to save messages
-```
-
-### Issue 2: Invalid Message Types
-**Problem**: Crafter was using custom message types (`policy_system_prompt`, `policy_user_prompt`, `policy_tool_call`) that violated the database CHECK constraint.
-
-**Fix**: Modified `/Users/joshpurtell/Documents/GitHub/synth-ai/examples/task_apps/crafter/task_app/synth_envs_hosted/rollout.py` to use standard message types:
-- `policy_system_prompt` → `system`
-- `policy_user_prompt` → `user`  
-- `policy_tool_call` → `assistant` (with `is_tool_call: true` metadata)
-
-## Full Working Pipeline
-
-### 1. Run Evaluation with Tracing
-
-```bash
-cd /Users/joshpurtell/Documents/GitHub/synth-ai
-
-export TASKAPP_TRACING_ENABLED=1
-export TURSO_NATIVE=0
-export SQLD_DB_PATH="traces/v3/crafter_eval.db"
-
-uv run synth-ai eval grpo-crafter-task-app \
-  --config examples/task_apps/crafter/eval_image_only_gpt4o.toml
-```
-
-**Result**:
-- ✅ 2 rollouts completed
-- ✅ 120 messages saved to database (40 system + 40 user + 40 assistant)
-- ✅ 2 outcome_rewards saved with achievements
-- ✅ Traces returned successfully
-
-### 2. Filter to Create SFT Dataset
-
-```bash
-uv run synth-ai filter \
-  --config examples/task_apps/crafter/filter_sft_dataset.toml
-```
-
-**Result**:
-```
-Wrote 40 examples -> ft_data/crafter_image_only_sft.jsonl
-```
-
-### 3. Verify SFT Data
-
-```bash
-# Check first example
-head -1 ft_data/crafter_image_only_sft.jsonl | jq .
-
-# Count examples
-wc -l ft_data/crafter_image_only_sft.jsonl
-```
-
-## SFT Dataset Format
-
-Each line in the JSONL contains:
-
-```json
-{
-  "messages": [
-    {
-      "role": "user",
-      "content": "=== CRAFTER GAME STATE ===\nStep: 0/10000\n..."
-    },
-    {
-      "role": "assistant",
-      "content": "[{'tool_name': 'interact_many', 'arguments': {...}}]"
-    }
-  ],
-  "metadata": {
-    "session_id": "...",
-    "env_name": "crafter",
-    "policy_name": "crafter-react",
-    "seed": 0,
-    "total_reward": 1,
-    "achievements_count": 1,
-    "created_at": "2025-10-22T23:55:25.533188+00:00"
-  }
-}
-```
-
-## Database Schema
-
-The filter command queries these tables:
-
-### messages table
-```sql
-SELECT message_type, content, timestamp 
-FROM messages 
-WHERE session_id = :session_id
-ORDER BY timestamp ASC
-```
-
-- ✅ 120 messages total
-- System (40) + User (40) + Assistant (40) messages
-- Pairs extracted: user → assistant
-
-### outcome_rewards table
-```sql
-SELECT total_reward, achievements_count 
-FROM outcome_rewards 
-WHERE session_id = :session_id
-```
-
-- Used to filter for successful rollouts
-- `min_official_score = 0.01` filters for rewards > 0
-- Both rollouts had `total_reward = 1` (1 achievement each)
-
-## Filter Configuration
-
-**File**: `examples/task_apps/crafter/filter_sft_dataset.toml`
-
-```toml
-[filter]
-db = "traces/v3/crafter_eval.db"
-output = "ft_data/crafter_image_only_sft.jsonl"
-min_official_score = 0.01  # Only traces with rewards > 0
-```
-
-### Available Filter Options
-
-```toml
-[filter]
-db = "path/to/traces.db"              # Required
-output = "path/to/output.jsonl"        # Required
-
-# Optional filters
-min_official_score = 0.01              # Filter by reward
-splits = ["train", "test"]             # Filter by split
-task_ids = ["task_1"]                  # Filter by task
-models = ["gpt-4o"]                    # Filter by model
-limit = 100                            # Limit number of examples
-```
-
-## Statistics
-
-From 2 rollouts with 10 turns each:
-
-| Metric | Count |
-|--------|-------|
-| Total rollouts | 2 |
-| Rollouts with rewards | 2 (100%) |
-| Total messages saved | 120 |
-| System messages | 40 |
-| User messages | 40 |
-| Assistant messages | 40 |
-| **SFT examples** | **40** |
-| Average turns per rollout | 10 |
-| Examples per rollout | 20 |
-
-## Next Steps
-
-### Scale Up
-
-Run with more seeds for a larger dataset:
-
-```toml
-# In eval_image_only_gpt4o.toml
-seeds = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]  # 10 rollouts
-max_turns = 50                           # More examples per rollout
-```
-
-Expected output: ~1000 SFT examples from 10 rollouts @ 50 turns each
-
-### Use the SFT Data
-
-```bash
-# For OpenAI fine-tuning
-# The JSONL format is compatible with OpenAI's fine-tuning API
-
-# For local fine-tuning
-# Convert to your preferred format (HuggingFace, etc.)
-```
-
-### Filter Variations
-
-```bash
-# Only high-reward traces
-min_official_score = 2.0
-
-# Only specific achievements
-# Query manually then filter by session_id
-
-# Time-based filtering
-min_created_at = "2025-10-22T00:00:00"
-```
-
-## Files Modified
-
-1. **`synth_ai/tracing_v3/turso/native_manager.py`**
-   - Fixed early return when session exists
-   - Added logging for debugging
-
-2. **`examples/task_apps/crafter/task_app/synth_envs_hosted/rollout.py`**
-   - Changed message types to standard values
-   - Added debug logging
-
-3. **`synth_ai/cli/task_apps.py`**
-   - Updated filter command to query messages table
-   - Added support for outcome_rewards filtering
-   - Fixed SQL parameter format
-
-4. **`examples/task_apps/crafter/filter_sft_dataset.toml`**
-   - Created filter configuration
-
-## Troubleshooting
-
-### No messages in database
-
-**Check**:
-```bash
-sqlite3 traces/v3/crafter_eval.db "SELECT COUNT(*) FROM messages;"
-```
-
-**Fix**: Ensure `TASKAPP_TRACING_ENABLED=1` and `TURSO_NATIVE=0`
-
-### Filter returns no examples
-
-**Check**:
-```bash
-sqlite3 traces/v3/crafter_eval.db \
-  "SELECT COUNT(*) FROM outcome_rewards WHERE total_reward > 0;"
-```
-
-**Fix**: Lower `min_official_score` or remove it to include all traces
-
-### Invalid message types
-
-**Error**: `CHECK constraint failed: message_type IN (...)`
-
-**Fix**: Already fixed in rollout.py - update to latest code
-
-## Related Documentation
-
-- `README_IMAGE_ONLY_EVAL.md` - How to run evaluations
-- `EVAL_IMAGE_ONLY_RESULTS.md` - Example evaluation results
-- `QUERY_EXAMPLES.md` - SQL query examples
-- `CREATE_SFT_DATASET.md` - Original approach (now superseded)
-
-## Success Metrics
-
-✅ Eval completes without errors  
-✅ Messages saved to database (system, user, assistant)  
-✅ Outcome rewards saved with foreign keys  
-✅ Filter command extracts user/assistant pairs  
-✅ SFT JSONL created with proper format  
-✅ Metadata includes rewards and achievements  
-
-**Status**: 🎉 **WORKING END-TO-END!**
-
-

examples/task_apps/crafter/QUERY_EXAMPLES.md

@@ -1,203 +0,0 @@
-# Crafter Eval Database Query Examples
-
-## Database Location
-```bash
-/Users/joshpurtell/Documents/GitHub/synth-ai/traces/v3/crafter_eval.db
-```
-
-## Quick Stats
-
-Run this query to get an overview:
-```sql
-SELECT 
-    'Total rollouts' as metric, 
-    CAST(COUNT(*) as TEXT) as value 
-FROM outcome_rewards
-UNION ALL
-SELECT 
-    'Rollouts with reward > 0', 
-    CAST(COUNT(*) as TEXT)
-FROM outcome_rewards 
-WHERE total_reward > 0
-UNION ALL
-SELECT 
-    'Average reward', 
-    CAST(ROUND(AVG(total_reward), 2) as TEXT)
-FROM outcome_rewards
-UNION ALL
-SELECT 
-    'Max reward', 
-    CAST(MAX(total_reward) as TEXT)
-FROM outcome_rewards;
-```
-
-**Current Results:**
-- Total rollouts: 10
-- Rollouts with reward > 0: 7
-- Average reward: 1.3
-- Max reward: 3
-
-## Filter for Non-Zero Rewards
-
-### Simple Query
-```sql
-SELECT 
-    session_id,
-    total_reward,
-    achievements_count,
-    json_extract(reward_metadata, '$.env_seed') as seed,
-    json_extract(reward_metadata, '$.final_achievements') as achievements
-FROM outcome_rewards
-WHERE total_reward > 0
-ORDER BY total_reward DESC, achievements_count DESC;
-```
-
-### With Full Session Context
-```sql
-SELECT 
-    st.session_id,
-    st.created_at,
-    st.num_timesteps,
-    st.num_events,
-    orw.total_reward,
-    orw.achievements_count,
-    json_extract(orw.reward_metadata, '$.final_achievements') as achievements,
-    json_extract(orw.reward_metadata, '$.env_seed') as seed
-FROM session_traces st
-INNER JOIN outcome_rewards orw ON st.session_id = orw.session_id
-WHERE orw.total_reward > 0
-ORDER BY orw.total_reward DESC;
-```
-
-## Filter by Achievement Count
-
-### Get rollouts with 2+ achievements
-```sql
-SELECT 
-    session_id,
-    total_reward,
-    achievements_count,
-    json_extract(reward_metadata, '$.final_achievements') as achievements
-FROM outcome_rewards
-WHERE achievements_count >= 2
-ORDER BY achievements_count DESC, total_reward DESC;
-```
-
-### Get rollouts with specific achievement
-```sql
-SELECT 
-    session_id,
-    total_reward,
-    achievements_count,
-    json_extract(reward_metadata, '$.final_achievements') as achievements
-FROM outcome_rewards
-WHERE reward_metadata LIKE '%collect_drink%'
-ORDER BY total_reward DESC;
-```
-
-## Group by Achievement Count
-```sql
-SELECT 
-    achievements_count,
-    COUNT(*) as num_rollouts,
-    ROUND(AVG(total_reward), 2) as avg_reward,
-    SUM(total_reward) as total_reward_sum,
-    GROUP_CONCAT(DISTINCT json_extract(reward_metadata, '$.env_seed')) as seeds
-FROM outcome_rewards
-GROUP BY achievements_count
-ORDER BY achievements_count DESC;
-```
-
-## Top Performers
-```sql
-SELECT 
-    json_extract(orw.reward_metadata, '$.env_seed') as seed,
-    orw.total_reward,
-    orw.achievements_count,
-    orw.total_steps,
-    st.num_events,
-    json_extract(orw.reward_metadata, '$.final_achievements') as achievements
-FROM session_traces st
-INNER JOIN outcome_rewards orw ON st.session_id = orw.session_id
-ORDER BY orw.total_reward DESC, orw.achievements_count DESC
-LIMIT 5;
-```
-
-## Get Event Details for High-Reward Rollouts
-```sql
-SELECT 
-    e.event_type,
-    e.model_name,
-    e.input_tokens,
-    e.output_tokens,
-    e.latency_ms,
-    e.reward as step_reward
-FROM events e
-INNER JOIN outcome_rewards orw ON e.session_id = orw.session_id
-WHERE orw.total_reward >= 2
-ORDER BY e.session_id, e.id
-LIMIT 20;
-```
-
-## Running Queries
-
-### From Command Line
-```bash
-cd /Users/joshpurtell/Documents/GitHub/synth-ai
-sqlite3 traces/v3/crafter_eval.db "YOUR_QUERY_HERE"
-```
-
-### With Formatted Output
-```bash
-sqlite3 -header -column traces/v3/crafter_eval.db "YOUR_QUERY_HERE"
-```
-
-### With JSON Output
-```bash
-sqlite3 -json traces/v3/crafter_eval.db "YOUR_QUERY_HERE" | jq .
-```
-
-## Example: Get CSV Export of Non-Zero Rewards
-```bash
-sqlite3 -header -csv traces/v3/crafter_eval.db \
-  "SELECT 
-    json_extract(reward_metadata, '$.env_seed') as seed,
-    total_reward,
-    achievements_count,
-    total_steps,
-    json_extract(reward_metadata, '$.final_achievements') as achievements
-   FROM outcome_rewards 
-   WHERE total_reward > 0 
-   ORDER BY total_reward DESC" \
-  > crafter_rewards_nonzero.csv
-```
-
-## Current Data Summary
-
-| Reward | Count | Seeds | Achievements |
-|--------|-------|-------|--------------|
-| 3 | 1 | 0 | collect_drink, collect_sapling, collect_wood |
-| 2 | 4 | 1,3,6,9 | collect_sapling, collect_wood |
-| 1 | 2 | 4,7 | collect_wood |
-| 0 | 3 | 2,5,8 | none |
-
-## Verifying Foreign Keys Work
-
-```sql
--- This should return 7 rows (all rollouts with rewards > 0)
-SELECT COUNT(*) 
-FROM session_traces st
-INNER JOIN outcome_rewards orw ON st.session_id = orw.session_id
-WHERE orw.total_reward > 0;
-
--- This should return the same 7 session_ids
-SELECT st.session_id 
-FROM session_traces st
-WHERE st.session_id IN (
-  SELECT session_id FROM outcome_rewards WHERE total_reward > 0
-);
-```
-
-✅ **Confirmed: Foreign keys are working correctly and can be used to join tables!**
-
-