synth-ai 0.2.14__py3-none-any.whl → 0.4.4__py3-none-any.whl

examples/task_apps/crafter/CREATE_SFT_DATASET.md

@@ -1,273 +0,0 @@
-# Creating SFT Datasets from Crafter Traces
-
-There are two approaches to create SFT (Supervised Fine-Tuning) datasets from Crafter rollouts:
-
-## Approach 1: Direct SFT Recording (Recommended)
-
-Crafter's rollout system can write SFT-ready JSONL files directly during evaluation by setting the `sft_output_dir`.
-
-### Setup
-
-1. Set the SFT output directory environment variable:
-```bash
-export SFT_OUTPUT_DIR="ft_data/crafter_sft"
-```
-
-2. Run evaluation:
-```bash
-cd /Users/joshpurtell/Documents/GitHub/synth-ai
-
-export TASKAPP_TRACING_ENABLED=1
-export TURSO_NATIVE=1
-export SQLD_DB_PATH="traces/v3/crafter_eval.db"
-export SFT_OUTPUT_DIR="ft_data/crafter_sft"  # Enable SFT recording
-
-uv run synth-ai eval grpo-crafter \
-  --config examples/task_apps/crafter/eval_image_only_gpt4o.toml
-```
-
-3. SFT files will be written to:
-```
-ft_data/crafter_sft/
-├── sft_<run_id_1>.jsonl
-├── sft_<run_id_2>.jsonl
-├── ...
-└── sft_<run_id_10>.jsonl
-```
-
-### SFT Record Format
-
-Each JSONL file contains records like:
-```json
-{
-  "messages": [
-    {"role": "system", "content": "...system prompt..."},
-    {"role": "user", "content": "...observation..."},
-    {"role": "assistant", "content": "...action..."}
-  ],
-  "metadata": {
-    "run_id": "...",
-    "turn": 5,
-    "reward": 1.0,
-    ...
-  }
-}
-```
-
-### Combine Multiple Files
-
-```bash
-# Combine all SFT files into one
-cat ft_data/crafter_sft/sft_*.jsonl > ft_data/crafter_combined.jsonl
-
-# Count examples
-wc -l ft_data/crafter_combined.jsonl
-```
-
-## Approach 2: Extract from Turso Database (Not Currently Supported)
-
-The `synth-ai filter` command is designed for traces with a different structure (where prompt/completion are stored in session metadata). 
-
-**Current Limitation**: Crafter's SessionTracer-based traces don't store messages in the format expected by the filter command.
-
-### Why Filter Doesn't Work
-
-The filter command expects:
-```python
-metadata = {
-    "prompt": "...",  # User message
-    "completion": "..."  # Assistant response
-}
-```
-
-But Crafter traces store:
-- Messages in separate `messages` table (currently 0 messages - not recorded during eval)
-- Rewards in `outcome_rewards` table
-- Metadata without prompt/completion fields
-
-### Future Enhancement
-
-To make filter work with Crafter traces, we would need to:
-1. Modify rollout to record messages to the `messages` table
-2. Update filter command to query `messages` table directly
-3. Join with `outcome_rewards` to filter by achievements
-
-## Comparison
-
-| Feature | Direct SFT | Filter Command |
-|---------|-----------|----------------|
-| **Setup** | Set `SFT_OUTPUT_DIR` | Create filter config |
-| **When** | During rollout | After rollout |
-| **Format** | JSONL per rollout | Combined JSONL |
-| **Filtering** | Manual (combine files) | Automatic (SQL queries) |
-| **Status** | ✅ Works now | ❌ Needs implementation |
-
-## Recommended Workflow
-
-### 1. Run evaluation with SFT recording:
-```bash
-export SFT_OUTPUT_DIR="ft_data/crafter_sft"
-uv run synth-ai eval grpo-crafter --config examples/task_apps/crafter/eval_image_only_gpt4o.toml
-```
-
-### 2. Filter for successful rollouts:
-
-Since we can't use the filter command yet, manually select files:
-
-```bash
-# Query database to find session_ids with rewards
-sqlite3 traces/v3/crafter_eval.db \
-  "SELECT session_id FROM outcome_rewards WHERE total_reward > 0" \
-  > successful_sessions.txt
-
-# Create directory for filtered SFT
-mkdir -p ft_data/crafter_sft_filtered
-
-# Copy only successful rollout SFT files
-while read session_id; do
-  if [ -f "ft_data/crafter_sft/sft_${session_id}.jsonl" ]; then
-    cp "ft_data/crafter_sft/sft_${session_id}.jsonl" ft_data/crafter_sft_filtered/
-  fi
-done < successful_sessions.txt
-
-# Combine filtered files
-cat ft_data/crafter_sft_filtered/sft_*.jsonl > ft_data/crafter_high_reward.jsonl
-
-echo "Created SFT dataset: ft_data/crafter_high_reward.jsonl"
-wc -l ft_data/crafter_high_reward.jsonl
-```
-
-### 3. Verify dataset:
-
-```bash
-# Look at first example
-head -1 ft_data/crafter_high_reward.jsonl | jq .
-
-# Count examples
-wc -l ft_data/crafter_high_reward.jsonl
-
-# Check message types
-jq -r '.messages[].role' ft_data/crafter_high_reward.jsonl | sort | uniq -c
-```
-
-## Example: Complete Pipeline
-
-```bash
-#!/bin/bash
-# complete_sft_pipeline.sh
-
-cd /Users/joshpurtell/Documents/GitHub/synth-ai
-
-# 1. Run evaluation with SFT recording
-export TASKAPP_TRACING_ENABLED=1
-export TURSO_NATIVE=1
-export SQLD_DB_PATH="traces/v3/crafter_eval.db"
-export SFT_OUTPUT_DIR="ft_data/crafter_sft"
-
-echo "Running evaluation..."
-uv run synth-ai eval grpo-crafter \
-  --config examples/task_apps/crafter/eval_image_only_gpt4o.toml
-
-# 2. Filter for successful rollouts
-echo "Filtering for successful rollouts..."
-mkdir -p ft_data/crafter_sft_filtered
-
-sqlite3 traces/v3/crafter_eval.db \
-  "SELECT session_id FROM outcome_rewards WHERE total_reward > 0" | \
-while read session_id; do
-  if [ -f "ft_data/crafter_sft/sft_${session_id}.jsonl" ]; then
-    cp "ft_data/crafter_sft/sft_${session_id}.jsonl" ft_data/crafter_sft_filtered/
-  fi
-done
-
-# 3. Combine into single dataset
-echo "Creating combined dataset..."
-cat ft_data/crafter_sft_filtered/sft_*.jsonl > ft_data/crafter_high_reward.jsonl
-
-# 4. Report statistics
-echo ""
-echo "=== SFT Dataset Created ==="
-echo "Total examples: $(wc -l < ft_data/crafter_high_reward.jsonl)"
-echo "Location: ft_data/crafter_high_reward.jsonl"
-echo ""
-echo "Rollouts included:"
-sqlite3 traces/v3/crafter_eval.db \
-  "SELECT 
-    COUNT(*) as count,
-    SUM(total_reward) as total_reward,
-    AVG(achievements_count) as avg_achievements
-   FROM outcome_rewards 
-   WHERE total_reward > 0"
-```
-
-## Troubleshooting
-
-### No SFT Files Created
-
-**Issue**: `ft_data/crafter_sft/` is empty after evaluation
-
-**Possible causes**:
-1. `SFT_OUTPUT_DIR` environment variable not set
-2. Rollout doesn't record SFT by default in eval mode
-3. Directory permissions issue
-
-**Debug**:
-```bash
-# Check if variable is set
-echo $SFT_OUTPUT_DIR
-
-# Check directory exists and is writable
-ls -la ft_data/
-
-# Try with explicit path
-export SFT_OUTPUT_DIR="/Users/joshpurtell/Documents/GitHub/synth-ai/ft_data/crafter_sft"
-```
-
-### SFT Files Don't Match Successful Rollouts
-
-**Issue**: Have SFT files for rollouts with 0 rewards
-
-**Solution**: This is expected - SFT is recorded for all rollouts. Use the filtering step to keep only successful ones.
-
-## Future Work
-
-To enable the `synth-ai filter` command for Crafter traces:
-
-1. **Modify Crafter rollout** to record messages to database:
-```python
-# In RolloutTracingContext
-await self.tracer.record_message(
-    content=user_prompt,
-    message_type="user",
-    metadata={"turn": turn}
-)
-
-await self.tracer.record_message(
-    content=assistant_response,
-    message_type="assistant", 
-    metadata={"turn": turn, "reward": step_reward}
-)
-```
-
-2. **Update filter command** to query messages table:
-```python
-# Instead of looking for metadata.prompt/completion
-# Query messages table directly
-messages = await tracer.db.get_messages(session_id)
-```
-
-3. **Create filter config** that works:
-```toml
-[filter]
-db = "traces/v3/crafter_eval.db"
-output = "ft_data/crafter_filtered.jsonl"
-min_official_score = 0.01  # Filter by outcome_rewards
-```
-
-## See Also
-
-- `README_IMAGE_ONLY_EVAL.md` - How to run evaluations
-- `EVAL_IMAGE_ONLY_RESULTS.md` - Example results
-- `QUERY_EXAMPLES.md` - SQL queries for trace analysis
-
-

examples/task_apps/crafter/EVAL_IMAGE_ONLY_RESULTS.md

@@ -1,152 +0,0 @@
-# Crafter Image-Only Eval Results
-
-## Summary
-Successfully ran 10 rollouts of the Crafter task app using **image-only input** (no text observations), with full tracing and rewards saved to Turso database.
-
-## Configuration
-- **Model**: `gpt-4o-mini-2024-07-18`
-- **Input Mode**: Image-only (vision enabled, text observations disabled)
-- **Max Steps**: 10 per episode
-- **Max LLM Calls**: 10 per rollout
-- **Seeds**: 0-9 (10 rollouts)
-- **Tracing**: Enabled with Turso/libsql (MVCC concurrent writes)
-- **Database**: `traces/v3/crafter_eval.db` (1.7MB)
-
-## Results
-
-### Overall Performance
-- **Total Rollouts**: 10
-- **Success Rate**: 100% (10/10 completed)
-- **Mean Official Score**: 0.700 (70%)
-- **Rollouts with Achievements**: 7/10 (70%)
-
-### Achievement Distribution
-| Achievements Count | Number of Rollouts |
-|-------------------|-------------------|
-| 3                 | 1                 |
-| 2                 | 4                 |
-| 1                 | 2                 |
-| 0                 | 3                 |
-
-### Top Performing Rollouts
-1. **Seed 0** - 3 achievements: `collect_drink`, `collect_sapling`, `collect_wood` (reward: 3)
-2. **Seed 1** - 2 achievements: `collect_sapling`, `collect_wood` (reward: 2)
-3. **Seed 3** - 2 achievements: `collect_sapling`, `collect_wood` (reward: 2)
-4. **Seed 6** - 2 achievements: `collect_sapling`, `collect_wood` (reward: 2)
-5. **Seed 9** - 2 achievements: `collect_sapling`, `collect_wood` (reward: 2)
-6. **Seed 4** - 1 achievement: `collect_wood` (reward: 1)
-7. **Seed 7** - 1 achievement: `collect_wood` (reward: 1)
-
-### Rollouts with No Achievements
-- Seed 2, 5, 8 - No achievements earned
-
-## Database Schema
-
-### outcome_rewards Table
-```sql
-CREATE TABLE outcome_rewards (
-    id INTEGER PRIMARY KEY AUTOINCREMENT,
-    session_id VARCHAR NOT NULL,
-    total_reward INTEGER NOT NULL,
-    achievements_count INTEGER NOT NULL,
-    total_steps INTEGER NOT NULL,
-    created_at DATETIME NOT NULL,
-    reward_metadata TEXT,
-    FOREIGN KEY(session_id) REFERENCES session_traces(session_id)
-);
-CREATE INDEX idx_outcome_rewards_session ON outcome_rewards (session_id);
-CREATE INDEX idx_outcome_rewards_total ON outcome_rewards (total_reward);
-```
-
-## Query Examples
-
-### Get rollouts with achievements > 0
-```sql
-SELECT 
-    st.session_id, 
-    st.num_timesteps, 
-    orw.achievements_count, 
-    orw.total_reward,
-    json_extract(orw.reward_metadata, '$.final_achievements') as achievements
-FROM session_traces st
-INNER JOIN outcome_rewards orw ON st.session_id = orw.session_id
-WHERE orw.achievements_count > 0
-ORDER BY orw.achievements_count DESC, orw.total_reward DESC;
-```
-
-### Count rollouts by achievement count
-```sql
-SELECT achievements_count, COUNT(*) as count
-FROM outcome_rewards
-GROUP BY achievements_count
-ORDER BY achievements_count DESC;
-```
-
-### Get top performers
-```sql
-SELECT session_id, total_reward, achievements_count, reward_metadata
-FROM outcome_rewards
-WHERE achievements_count > 0 OR total_reward > 0
-ORDER BY achievements_count DESC, total_reward DESC
-LIMIT 10;
-```
-
-## Key Changes Made
-
-### 1. OpenAI Authorization Fix
-Updated `openai_client.py` to properly set `Authorization: Bearer` header for OpenAI API calls:
-```python
-# If calling OpenAI directly (api.openai.com)
-if "api.openai.com" in low_url:
-    openai_key = os.getenv("OPENAI_API_KEY")
-    if openai_key and isinstance(openai_key, str):
-        headers["Authorization"] = f"Bearer {openai_key}"
-```
-
-### 2. Image-Only Mode Implementation
-Added `image_only_mode` support to `CrafterPolicy` and `CrafterReActAgent`:
-- When enabled, only image observations are sent to the LLM
-- Text observations are set to empty string
-- Vision mode is automatically enabled
-
-### 3. Trace Format Support
-Fixed CLI to properly handle both "compact" and "full" trace formats:
-```python
-# Handle both "compact" and "full" trace formats
-session_trace_dict = trace_namespace.get("session_trace")
-if not isinstance(session_trace_dict, dict):
-    if "session_id" in trace_namespace:
-        session_trace_dict = trace_namespace
-```
-
-### 4. Request Body Structure
-Fixed rollout request to properly nest tracing parameters:
-```python
-"record": {
-    "return_trace": True,
-    "trace_format": "full",
-}
-```
-
-## Files Modified
-1. `/Users/joshpurtell/Documents/GitHub/synth-ai/examples/task_apps/crafter/task_app/synth_envs_hosted/inference/openai_client.py`
-2. `/Users/joshpurtell/Documents/GitHub/synth-ai/examples/task_apps/crafter/task_app/synth_envs_hosted/envs/crafter/policy.py`
-3. `/Users/joshpurtell/Documents/GitHub/synth-ai/examples/task_apps/crafter/task_app/synth_envs_hosted/envs/crafter/react_agent.py`
-4. `/Users/joshpurtell/Documents/GitHub/synth-ai/synth_ai/cli/task_apps.py`
-5. `/Users/joshpurtell/Documents/GitHub/synth-ai/examples/task_apps/crafter/eval_image_only_gpt4o.toml`
-
-## Verification
-- ✅ All 10 rollouts completed successfully
-- ✅ Image-only input confirmed (base64 PNG images in prompts)
-- ✅ Achievements computed and saved
-- ✅ Foreign keys working (can join session_traces and outcome_rewards)
-- ✅ Can query rollouts by achievement count and rewards
-- ✅ Database size: 1.7MB with full trace data
-
-## Next Steps
-- Increase `max_steps_per_episode` for longer episodes
-- Try different models (e.g., gpt-4o, claude-3.5-sonnet)
-- Analyze which actions lead to the most achievements
-- Use concurrent writes with higher concurrency (Turso MVCC supports this)
-
-

examples/task_apps/crafter/FILTER_COMMAND_STATUS.md

@@ -1,174 +0,0 @@
-# Filter Command Status for Crafter
-
-## Summary
-
-The `synth-ai filter` command has been updated to work with Crafter's SessionTracer v3 traces, but there's currently an issue with message persistence that needs to be resolved.
-
-## What Was Changed
-
-### 1. Updated Filter Command (`synth_ai/cli/task_apps.py`)
-
-The filter command now:
-- ✅ Queries `outcome_rewards` table to filter by `total_reward`
-- ✅ Queries `messages` table to extract user/assistant pairs
-- ✅ Falls back to metadata-based filtering for backwards compatibility
-- ✅ Supports filtering by achievements/rewards from Crafter rollouts
-- ✅ Extracts text from structured message content (JSON payloads)
-
-### 2. Created Filter Config
-
-**File**: `examples/task_apps/crafter/filter_sft_dataset.toml`
-
-```toml
-[filter]
-db = "traces/v3/crafter_eval.db"
-output = "ft_data/crafter_image_only_sft.jsonl"
-min_official_score = 0.01  # Only traces with rewards > 0
-```
-
-## Current Issue: Messages Not Being Saved
-
-### Problem
-
-When running evaluations, the database ends up with:
-- ✅ 2 `session_traces` (metadata saved)
-- ✅ 2 `outcome_rewards` (rewards saved)
-- ❌ 0 `messages` (messages NOT saved)
-- ✅ 40 `events` (environment events saved)
-- ✅ 20 `session_timesteps` (timesteps saved)
-
-### Expected Behavior
-
-The rollout code calls:
-1. `tracer.initialize()` - Opens database connection
-2. `tracer.start_session()` - Creates session
-3. `tracer.record_message()` - Records system/user prompts (via `record_policy_prompts`)
-4. `tracer.end_session()` - Saves session with `auto_save=True`
-
-The `insert_session_trace` method (in `NativeLibsqlTraceManager`) SHOULD iterate through `trace.markov_blanket_message_history` and save each message to the `messages` table.
-
-### Actual Behavior
-
-Messages are NOT being persisted to the database, even though:
-- The code path looks correct
-- `end_session()` is being called
-- `auto_save=True` is the default
-- The trace JSON payload includes `markov_blanket_message_history`
-
-### Debugging Observations
-
-1. **Trace payload includes messages**: The eval output shows a large JSON structure with `markov_blanket_messages` containing all the prompts
-2. **No errors logged**: The `try/except` around `end_session()` doesn't log any failures
-3. **Works with both TURSO_NATIVE=0 and TURSO_NATIVE=1**: Neither backend saves messages
-4. **Database is writable**: `outcome_rewards` and `events` are being saved successfully
-
-## Possible Causes
-
-1. **Silent exception during message insertion**: The `insert_message_row` might be failing without raising
-2. **Transaction not committed**: Messages might be inserted but not committed
-3. **Messages not in trace object**: `markov_blanket_message_history` might be empty when `end_session` is called
-4. **Record message not adding to history**: `tracer.record_message()` might not be appending to the list properly
-
-## Next Steps to Fix
-
-### Option 1: Debug Message Persistence
-
-Add logging to trace the message save path:
-
-```python
-# In rollout.py, finalize method
-logger.info(f"[finalize] trace has {len(self.tracer._current_trace.markov_blanket_message_history)} messages before end_session")
-
-# In native_manager.py, insert_session_trace
-logger.info(f"[insert_session_trace] saving {len(trace.markov_blanket_message_history)} messages")
-for msg in trace.markov_blanket_message_history:
-    logger.info(f"  - message type={msg.message_type}")
-    await self.insert_message_row(...)
-    logger.info(f"  - message saved")
-```
-
-### Option 2: Verify Messages Are Being Recorded
-
-Check if `record_policy_prompts` is actually being called and adding messages:
-
-```python
-# In rollout.py, after record_policy_prompts
-if self.tracer and self.tracer._current_trace:
-    msg_count = len(self.tracer._current_trace.markov_blanket_message_history)
-    logger.info(f"[record_policy_prompts] trace now has {msg_count} messages")
-```
-
-### Option 3: Manual Message Recording
-
-As a workaround, explicitly save messages outside of SessionTracer:
-
-```python
-# In finalize(), before end_session()
-if self.enabled and self.tracer is not None:
-    conn = await self.tracer.db.get_connection()
-    for msg in self.tracer._current_trace.markov_blanket_message_history:
-        await conn.execute(
-            "INSERT INTO messages (session_id, message_type, content, timestamp) VALUES (?, ?, ?, ?)",
-            (self.run_id, msg.message_type, str(msg.content), msg.time_record.event_time)
-        )
-    await conn.commit()
-```
-
-### Option 4: Use SFT Records Instead
-
-Crafter already has working SFT record generation that writes directly to JSONL files. Use that instead of the filter command:
-
-```bash
-export SFT_OUTPUT_DIR="ft_data/crafter_sft"
-uv run synth-ai eval grpo-crafter-task-app --config eval_image_only_gpt4o.toml
-
-# Then filter successful runs manually
-cat ft_data/crafter_sft/sft_*.jsonl > ft_data/crafter_combined.jsonl
-```
-
-## Current Workaround
-
-Until message persistence is fixed, use the direct SFT recording approach (Option 4) documented in `CREATE_SFT_DATASET.md`.
-
-## Testing the Filter Command
-
-Once messages are being saved:
-
-```bash
-# 1. Run eval to populate database
-export TASKAPP_TRACING_ENABLED=1
-export TURSO_NATIVE=0
-export SQLD_DB_PATH="traces/v3/crafter_eval.db"
-uv run synth-ai eval grpo-crafter-task-app --config eval_image_only_gpt4o.toml
-
-# 2. Verify messages were saved
-sqlite3 traces/v3/crafter_eval.db "SELECT COUNT(*) FROM messages;"
-# Should be > 0
-
-# 3. Run filter
-uv run synth-ai filter --config filter_sft_dataset.toml
-
-# 4. Check output
-cat ft_data/crafter_image_only_sft.jsonl | jq .
-```
-
-## Related Files
-
-- `synth_ai/cli/task_apps.py` - Filter command implementation (updated)
-- `synth_ai/tracing_v3/session_tracer.py` - SessionTracer class
-- `synth_ai/tracing_v3/turso/native_manager.py` - `insert_session_trace` method (should save messages)
-- `examples/task_apps/crafter/task_app/synth_envs_hosted/rollout.py` - Rollout tracing context
-- `filter_sft_dataset.toml` - Filter configuration
-- `CREATE_SFT_DATASET.md` - Alternative approach using direct SFT recording
-
-## Status
-
-- ✅ Filter command updated to query messages table
-- ✅ Filter command can join with outcome_rewards
-- ✅ Filter config created
-- ❌ Messages not being persisted to database
-- ❌ Filter command cannot extract SFT data without messages
-
-**Action Required**: Debug why messages aren't being saved to the database despite correct code path.
-
-