synth-ai 0.2.9.dev2__py3-none-any.whl → 0.2.9.dev4__py3-none-any.whl

examples/evals_old/example_log.md

@@ -0,0 +1,145 @@
+joshuapurtell@Mac synth-ai % bash examples/evals/run_demo.sh                
+Models to compare (space-separated) [gpt-5-nano gpt-4.1-nano]: 
+Models: gpt-5-nano gpt-4.1-nano
+Episodes per model [3]: 5
+Max turns per episode [5]: 5
+Parallelism per model (concurrency) [5]: 5
+Difficulty [easy]: 
+Running comparison: episodes=5, max_turns=5, difficulty=easy, concurrency=5
+Detected SYNTH_API_KEY (sk_liv...ac95). Use this key? [Y/n]: n
+Use SYNTH_API_KEY_PROD (sk_liv...a2a4)? [y/N]: Y
+[PATCH] Attempting to apply Crafter deterministic patch...
+[PATCH] Patching crafter.Env._balance_object...
+[PATCH] crafter.Env._balance_object patched.
+[PATCH] Attempting to apply Crafter serialization patch v3...
+[PATCH] Adding enhanced save/load methods to crafter.Env...
+[PATCH] crafter.Env.save() and load() methods added (v3).
+[PATCH] Crafter serialization patch v3 complete.
+[PATCH] Attempting to apply simplified Crafter world configuration patch...
+[PATCH] Simplified Crafter world configuration patch complete.
+[PATCH] Available configs: easy, normal, hard, peaceful
+✅ Loaded 8 Crafter achievement hooks (Easy, Medium, Hard)
+🎮 Crafter Multi-Model Experiment
+==================================================
+Experiment ID: crafter_multi_model_20250808_170152
+Models: gpt-5-nano, gpt-4.1-nano
+Episodes per model: 5
+Max turns per episode: 5
+Difficulty: easy
+Seeds: 1000 to 1004
+Turn timeout: 20.0s
+Episode timeout: 180.0s
+Save traces: True
+Database URL: sqlite+aiosqlite:////Users/joshuapurtell/Documents/GitHub/synth-ai/traces/v3/synth_ai.db/dbs/default/data
+==================================================
+✅ Crafter service is running
+
+Running 5 episodes for gpt-5-nano in parallel...
+
+gpt-5-nano | ep1:   0%|                                                               | 0/5 [00:00<?, ?turn/s]
+Running 5 episodes for gpt-4.1-nano in parallel...                                    | 0/5 [00:00<?, ?turn/s]
+gpt-5-nano | ep3:   0%|                                                               | 0/5 [00:00<?, ?turn/s]
+gpt-4.1-nano | ep3: 100%|██████████████████████████████████████████████| 5/5 [00:09<00:00,  1.95s/turn, ach=1]
+gpt-4.1-nano | ep2:  80%|████████████████████████████████████▊         | 4/5 [00:10<00:02,  2.64s/turn, ach=2]
+gpt-4.1-nano | ep4: 100%|██████████████████████████████████████████████| 5/5 [00:11<00:00,  2.32s/turn, ach=0]
+gpt-4.1-nano | ep5: 100%|██████████████████████████████████████████████| 5/5 [00:11<00:00,  2.37s/turn, ach=2]
+gpt-5-nano | ep1:  20%|█████████▌                                      | 1/5 [00:21<01:24, 21.13s/turn, ach=0    ⏰ Turn 3 timed out for episode 0 after 20.0s                       | 2/5 [00:25<00:38, 12.83s/turn, ach=0]
+gpt-4.1-nano | ep1:  60%|███████████████████████████▌                  | 3/5 [00:28<00:19,  9.62s/turn, ach=1]
+gpt-5-nano | ep3: 100%|████████████████████████████████████████████████| 5/5 [01:00<00:00, 12.05s/turn, ach=1]
+gpt-5-nano | ep2: 100%|████████████████████████████████████████████████| 5/5 [01:07<00:00, 13.56s/turn, ach=2]
+    ⏰ Turn 4 timed out for episode 3 after 20.0s██████████████████████| 5/5 [01:07<00:00, 14.04s/turn, ach=2]
+gpt-5-nano | ep4:  80%|██████████████████████████████████████▍         | 4/5 [01:08<00:17, 17.02s/turn, ach=0]
+gpt-5-nano | ep5: 100%|████████████████████████████████████████████████| 5/5 [01:13<00:00, 14.71s/turn, ach=1]
+gpt-5-nano | ep1: 100%|████████████████████████████████████████████████| 5/5 [01:19<00:00, 15.83s/turn, ach=1]
+gpt-4.1-nano | ep5: 100%|██████████████████████████████████████████████| 5/5 [00:11<00:00,  1.68s/turn, ach=2]
+📊 Analysis Results:
+================================================================================:13<00:00, 14.26s/turn, ach=1]
+
+📈 Model Performance Summary:
+Model                Avg Achievements   Max Achievements   Invalid Rate    Success Rate   
+--------------------------------------------------------------------------------------
+gpt-4.1-nano           1.20 ± 0.75                    2            0.00%          100.00%
+gpt-5-nano             1.00 ± 0.63                    2            0.00%          100.00%
+
+🏆 Achievement Frequencies:
+
+Achievement                 gpt-4.1-nano   gpt-5-nano
+-----------------------------------------------
+collect_drink               2/5   ( 40%)   0/5   (  0%)
+collect_sapling             1/5   ( 20%)   2/5   ( 40%)
+collect_wood                3/5   ( 60%)   2/5   ( 40%)
+place_plant                 0/5   (  0%)   1/5   ( 20%)
+
+💰 Model Usage Statistics from Current Experiment:
+Model                Provider   Usage Count  Avg Latency (ms)   Total Cost  
+------------------------------------------------------------------------
+gpt-5-nano           openai     221          13006.57           $0.0000     
+gpt-4.1-nano         openai     161          950.12             $0.0000     
+
+💾 Detailed results saved to: /Users/joshuapurtell/Documents/GitHub/synth-ai/temp/crafter_experiment_results_20250808_170312.json
+
+✅ Experiment complete!
+Using v3 traces DB: /Users/joshuapurtell/Documents/GitHub/synth-ai/traces/v3/synth_ai.db/dbs/default/data
+\nAvailable achievements (session counts):
+[PATCH] Attempting to apply Crafter deterministic patch...
+[PATCH] Patching crafter.Env._balance_object...
+[PATCH] crafter.Env._balance_object patched.
+[PATCH] Attempting to apply Crafter serialization patch v3...
+[PATCH] Adding enhanced save/load methods to crafter.Env...
+[PATCH] crafter.Env.save() and load() methods added (v3).
+[PATCH] Crafter serialization patch v3 complete.
+[PATCH] Attempting to apply simplified Crafter world configuration patch...
+[PATCH] Simplified Crafter world configuration patch complete.
+[PATCH] Available configs: easy, normal, hard, peaceful
+Achievements present (session counts):
+  - collect_drink: 44
+  - collect_sapling: 62
+  - collect_wood: 74
+  - defeat_skeleton: 4
+  - defeat_zombie: 2
+  - eat_cow: 2
+  - place_plant: 8
+  - place_table: 3
+\nEnter achievements to filter by (space-separated), or press Enter for 'collect_wood':
+
+Optionally restrict to models (space-separated), or press Enter to include all:
+
+\nRunning: uv run python -m examples.evals.trace_analysis filter --db "/Users/joshuapurtell/Documents/GitHub/synth-ai/traces/v3/synth_ai.db/dbs/default/data" --achievements collect_wood --output ft_data/evals_filtered.jsonl
+[PATCH] Attempting to apply Crafter deterministic patch...
+[PATCH] Patching crafter.Env._balance_object...
+[PATCH] crafter.Env._balance_object patched.
+[PATCH] Attempting to apply Crafter serialization patch v3...
+[PATCH] Adding enhanced save/load methods to crafter.Env...
+[PATCH] crafter.Env.save() and load() methods added (v3).
+[PATCH] Crafter serialization patch v3 complete.
+[PATCH] Attempting to apply simplified Crafter world configuration patch...
+[PATCH] Simplified Crafter world configuration patch complete.
+[PATCH] Available configs: easy, normal, hard, peaceful
+✅ Wrote 74 examples from 74 sessions → ft_data/evals_filtered.jsonl
+\nRunning: uv run python -m examples.evals.trace_analysis stats --db "/Users/joshuapurtell/Documents/GitHub/synth-ai/traces/v3/synth_ai.db/dbs/default/data" --achievements collect_wood
+[PATCH] Attempting to apply Crafter deterministic patch...
+[PATCH] Patching crafter.Env._balance_object...
+[PATCH] crafter.Env._balance_object patched.
+[PATCH] Attempting to apply Crafter serialization patch v3...
+[PATCH] Adding enhanced save/load methods to crafter.Env...
+[PATCH] crafter.Env.save() and load() methods added (v3).
+[PATCH] Crafter serialization patch v3 complete.
+[PATCH] Attempting to apply simplified Crafter world configuration patch...
+[PATCH] Simplified Crafter world configuration patch complete.
+[PATCH] Available configs: easy, normal, hard, peaceful
+Matched sessions (any of: collect_wood )
+  n=74  avg_reward=0.76  stddev=1.00
+  avg_first_unlock_step=4.7  stddev=4.6
+Others
+  n=224  avg_reward=0.21  stddev=0.51
+
+Achievement frequency by session (matched vs others):
+  - collect_drink: matched 25/74 (33.8%), others 19/224 (8.5%)
+  - collect_sapling: matched 21/74 (28.4%), others 41/224 (18.3%)
+  - place_table: matched 3/74 (4.1%), others 0/224 (0.0%)
+  - eat_cow: matched 2/74 (2.7%), others 0/224 (0.0%)
+  - place_plant: matched 3/74 (4.1%), others 5/224 (2.2%)
+  - defeat_skeleton: matched 2/74 (2.7%), others 2/224 (0.9%)
+  - defeat_zombie: matched 0/74 (0.0%), others 2/224 (0.9%)
+\nDone. See ft_data/evals_filtered.jsonl and v3 DB for deeper analysis.
+joshuapurtell@Mac synth-ai % 

examples/evals_old/run_demo.sh

@@ -0,0 +1,126 @@
+#!/bin/bash
+
+# Run Crafter experiments comparing gpt-5-nano and Qwen/Qwen3-32B-Instruct
+
+# Get the directory where this script is located
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
+
+# Change to the synth-ai root directory
+cd "$SCRIPT_DIR/../.."
+
+# Interactive mini-demo: run small comparison, then analyze v3 traces
+set -euo pipefail
+
+# Load env (prefer local .env at repo root)
+set +u
+set -a
+if [ -f ".env" ]; then source ".env"; fi
+set +a
+set -u
+
+# Ensure API key present (SYNTH_API_KEY, optionally mirror to OPENAI_API_KEY)
+ensure_api_key() {
+  local current_key="${SYNTH_API_KEY:-}"
+  if [ -n "$current_key" ]; then
+    local preview="${current_key:0:6}...${current_key: -4}"
+    read -r -p "Detected SYNTH_API_KEY ($preview). Use this key? [Y/n]: " USE_CUR || true
+    USE_CUR=${USE_CUR:-Y}
+    if [[ ! "$USE_CUR" =~ ^[Yy]$ ]]; then
+      current_key=""
+    fi
+  fi
+
+  if [ -z "$current_key" ] && [ -n "${SYNTH_API_KEY_PROD:-}" ]; then
+    local prod_prev="${SYNTH_API_KEY_PROD:0:6}...${SYNTH_API_KEY_PROD: -4}"
+    read -r -p "Use SYNTH_API_KEY_PROD ($prod_prev)? [y/N]: " USE_PROD || true
+    if [[ "$USE_PROD" =~ ^[Yy]$ ]]; then
+      current_key="$SYNTH_API_KEY_PROD"
+    fi
+  fi
+
+  while [ -z "$current_key" ]; do
+    echo
+    read -s -p "Enter your SYNTH_API_KEY: " KEY_IN || true
+    echo
+    if [ -n "$KEY_IN" ]; then
+      current_key="$KEY_IN"
+    else
+      echo "A valid SYNTH_API_KEY is required to continue."
+    fi
+  done
+
+  export SYNTH_API_KEY="$current_key"
+  if [ -z "${OPENAI_API_KEY:-}" ]; then
+    export OPENAI_API_KEY="$SYNTH_API_KEY"
+    echo "OPENAI_API_KEY set from SYNTH_API_KEY."
+  fi
+}
+
+# Interactive prompts (with sensible defaults)
+MODELS_DEFAULT="gpt-5-nano gpt-4.1-nano"
+read -r -p "Models to compare (space-separated) [${MODELS_DEFAULT}]: " MODELS_INPUT || true
+MODELS=${MODELS_INPUT:-$MODELS_DEFAULT}
+echo "Models: ${MODELS}"
+
+read -r -p "Episodes per model [3]: " EPISODES_INPUT || true
+EPISODES=${EPISODES_INPUT:-3}
+
+read -r -p "Max turns per episode [5]: " MAX_TURNS_INPUT || true
+MAX_TURNS=${MAX_TURNS_INPUT:-5}
+
+read -r -p "Parallelism per model (concurrency) [5]: " CONCURRENCY_INPUT || true
+CONCURRENCY=${CONCURRENCY_INPUT:-5}
+
+read -r -p "Difficulty [easy]: " DIFFICULTY_INPUT || true
+DIFFICULTY=${DIFFICULTY_INPUT:-easy}
+
+echo "Running comparison: episodes=${EPISODES}, max_turns=${MAX_TURNS}, difficulty=${DIFFICULTY}, concurrency=${CONCURRENCY}"
+
+# Ensure key before running rollouts
+ensure_api_key
+
+uv run python examples/evals/compare_models.py \
+    --episodes "${EPISODES}" \
+    --max-turns "${MAX_TURNS}" \
+    --difficulty "${DIFFICULTY}" \
+    --models ${MODELS} \
+    --base-seed 1000 \
+    --turn-timeout 20.0 \
+    --episode-timeout 180.0 \
+    --concurrency "${CONCURRENCY}" \
+    --quiet
+
+# Derive v3 sqld internal DB path for quick analysis
+DB_PATH="$PWD/traces/v3/synth_ai.db/dbs/default/data"
+export DB_PATH
+echo "Using v3 traces DB: $DB_PATH"
+
+echo "\nAvailable achievements (session counts):"
+uv run python -m examples.evals.trace_analysis list --db "$DB_PATH"
+
+echo "\nEnter achievements to filter by (space-separated), or press Enter for 'collect_wood':"
+read -r ACH
+ACH=${ACH:-collect_wood}
+
+echo "Optionally restrict to models (space-separated), or press Enter to include all:"
+read -r MODELS_FILTER
+
+mkdir -p ft_data
+if [ -n "$MODELS_FILTER" ]; then
+  echo "\nRunning: uv run python -m examples.evals.trace_analysis filter --db \"$DB_PATH\" --achievements $ACH --output ft_data/evals_filtered.jsonl --models $MODELS_FILTER"
+  uv run python -m examples.evals.trace_analysis filter --db "$DB_PATH" --achievements $ACH --output ft_data/evals_filtered.jsonl --models $MODELS_FILTER
+else
+  echo "\nRunning: uv run python -m examples.evals.trace_analysis filter --db \"$DB_PATH\" --achievements $ACH --output ft_data/evals_filtered.jsonl"
+  uv run python -m examples.evals.trace_analysis filter --db "$DB_PATH" --achievements $ACH --output ft_data/evals_filtered.jsonl
+fi
+
+# Show stats comparing filtered vs others (including achievement frequencies)
+if [ -n "$MODELS_FILTER" ]; then
+  echo "\nRunning: uv run python -m examples.evals.trace_analysis stats --db \"$DB_PATH\" --achievements $ACH --models $MODELS_FILTER"
+  uv run python -m examples.evals.trace_analysis stats --db "$DB_PATH" --achievements $ACH --models $MODELS_FILTER
+else
+  echo "\nRunning: uv run python -m examples.evals.trace_analysis stats --db \"$DB_PATH\" --achievements $ACH"
+  uv run python -m examples.evals.trace_analysis stats --db "$DB_PATH" --achievements $ACH
+fi
+
+echo "\nDone. See ft_data/evals_filtered.jsonl and v3 DB for deeper analysis."

examples/evals_old/trace_analysis.py

@@ -0,0 +1,270 @@
+#!/usr/bin/env python3
+"""
+Trace analysis utilities for Crafter v3 traces (sqld/Turso).
+
+Subcommands:
+  - list:    List achievements present in the database and counts
+  - filter:  Filter sessions by required achievements and export OpenAI-format JSONL
+  - stats:   Compare rewards and achievement frequencies for filtered vs. others
+
+Usage examples:
+  uvpm examples.evals.trace_analysis list --db traces/v3/synth_ai.db/dbs/default/data
+  uvpm examples.evals.trace_analysis filter --db traces/v3/synth_ai.db/dbs/default/data \
+      --achievements collect_wood --output ft_data/evals_collect_wood.jsonl
+"""
+
+import argparse
+import asyncio
+import json
+import json as pyjson
+import math
+from pathlib import Path
+
+from synth_ai.environments.examples.crafter_classic.agent_demos.crafter_modal_ft.filter_traces_sft_turso import (
+    FinetuningDataExtractorV3,
+)
+
+
+def build_db_url(path: str) -> str:
+    if path.startswith("sqlite+"):
+        return path
+    return f"sqlite+aiosqlite:///{path}"
+
+
+async def cmd_list(db_path: str) -> None:
+    db_url = build_db_url(db_path)
+    async with FinetuningDataExtractorV3(db_url) as ex:
+        sessions = await ex.get_all_sessions()
+        achievement_counts: dict[str, int] = {}
+        for _, row in sessions.iterrows():
+            ach_list = await ex.get_session_achievements(row["session_id"]) or []
+            for name in ach_list:
+                achievement_counts[name] = achievement_counts.get(name, 0) + 1
+
+        print("Achievements present (session counts):")
+        for name in sorted(achievement_counts.keys()):
+            print(f"  - {name}: {achievement_counts[name]}")
+
+
+async def cmd_filter(
+    db_path: str, achievements: list[str], output: str, models: list[str] | None = None
+) -> None:
+    db_url = build_db_url(db_path)
+    required: set[str] = set(achievements)
+    async with FinetuningDataExtractorV3(db_url) as ex:
+        sessions = await ex.get_all_sessions()
+        kept: list[str] = []
+        for _, row in sessions.iterrows():
+            if models:
+                # Restrict to sessions containing any of the requested models
+                model_df = await ex.db_manager.query_traces(
+                    """
+                    SELECT DISTINCT model_name
+                    FROM events
+                    WHERE session_id = :session_id
+                      AND event_type = 'cais'
+                      AND model_name IS NOT NULL
+                    """,
+                    {"session_id": row["session_id"]},
+                )
+                session_models = (
+                    model_df["model_name"].tolist()
+                    if model_df is not None and not model_df.empty
+                    else []
+                )
+                if not any(m in session_models for m in models):
+                    continue
+            ach_list = await ex.get_session_achievements(row["session_id"]) or []
+            if required & set(ach_list):
+                kept.append(row["session_id"])
+
+        data = await ex.extract_openai_format(kept)
+        Path(output).parent.mkdir(parents=True, exist_ok=True)
+        with open(output, "w") as f:
+            for exm in data:
+                f.write(json.dumps(exm) + "\n")
+        print(f"✅ Wrote {len(data)} examples from {len(kept)} sessions → {output}")
+
+
+async def _first_achievement_step(
+    ex: FinetuningDataExtractorV3, session_id: str, required: set[str]
+) -> int | None:
+    q = """
+        SELECT message_time, system_state_after
+        FROM events
+        WHERE session_id = :session_id
+          AND event_type = 'environment'
+        ORDER BY message_time ASC
+        """
+    df = await ex.db_manager.query_traces(q, {"session_id": session_id})
+    if df is None or df.empty:
+        return None
+    seen: set[str] = set()
+    for _, row in df.iterrows():
+        st = row.get("system_state_after")
+        if isinstance(st, str):
+            try:
+                st = pyjson.loads(st)
+            except Exception:
+                st = None
+        ach = None
+        if isinstance(st, dict):
+            ps = st.get("public_state") or {}
+            ach = ps.get("achievements_status") or {}
+        if isinstance(ach, dict):
+            for name, unlocked in ach.items():
+                if unlocked and name in required and name not in seen:
+                    return int(row.get("message_time") or 0)
+    return None
+
+
+def _mean(values: list[float]) -> float:
+    return (sum(values) / len(values)) if values else 0.0
+
+
+def _stddev(values: list[float]) -> float:
+    if not values:
+        return 0.0
+    m = _mean(values)
+    var = sum((v - m) * (v - m) for v in values) / len(values)
+    return math.sqrt(var)
+
+
+async def cmd_stats(db_path: str, achievements: list[str], models: list[str] | None = None) -> None:
+    db_url = build_db_url(db_path)
+    required: set[str] = set(achievements)
+    async with FinetuningDataExtractorV3(db_url) as ex:
+        sessions = await ex.get_all_sessions()
+        matched_rewards: list[float] = []
+        other_rewards: list[float] = []
+        first_steps: list[int] = []
+        matched_count: int = 0
+        other_count: int = 0
+        matched_ach_counts: dict[str, int] = {}
+        other_ach_counts: dict[str, int] = {}
+
+        for _, row in sessions.iterrows():
+            sid = row["session_id"]
+            if models:
+                model_df = await ex.db_manager.query_traces(
+                    """
+                    SELECT DISTINCT model_name
+                    FROM events
+                    WHERE session_id = :session_id
+                      AND event_type = 'cais'
+                      AND model_name IS NOT NULL
+                    """,
+                    {"session_id": sid},
+                )
+                session_models = (
+                    model_df["model_name"].tolist()
+                    if model_df is not None and not model_df.empty
+                    else []
+                )
+                if not any(m in session_models for m in models):
+                    continue
+
+            ach_list = await ex.get_session_achievements(sid) or []
+            metrics = await ex.get_session_metrics(sid)
+            reward = float(metrics.get("total_reward", 0.0))
+
+            if required & set(ach_list):
+                matched_rewards.append(reward)
+                step = await _first_achievement_step(ex, sid, required)
+                if step is not None:
+                    first_steps.append(step)
+                matched_count += 1
+                for name in ach_list:
+                    matched_ach_counts[name] = matched_ach_counts.get(name, 0) + 1
+            else:
+                other_rewards.append(reward)
+                other_count += 1
+                for name in ach_list:
+                    other_ach_counts[name] = other_ach_counts.get(name, 0) + 1
+
+        print("Matched sessions (any of:", ", ".join(sorted(required)), ")")
+        print(
+            f"  n={len(matched_rewards)}  avg_reward={_mean(matched_rewards):.2f}  stddev={_stddev(matched_rewards):.2f}"
+        )
+        if first_steps:
+            print(
+                f"  avg_first_unlock_step={_mean([float(s) for s in first_steps]):.1f}  stddev={_stddev([float(s) for s in first_steps]):.1f}"
+            )
+        else:
+            print("  avg_first_unlock_step=n/a (no unlocks recorded)")
+        print("Others")
+        print(
+            f"  n={len(other_rewards)}  avg_reward={_mean(other_rewards):.2f}  stddev={_stddev(other_rewards):.2f}"
+        )
+
+        # Achievement frequency comparison (by session presence), excluding required achievements
+        all_achievements: set[str] = set(matched_ach_counts.keys()) | set(other_ach_counts.keys())
+        compare_achievements = [a for a in sorted(all_achievements) if a not in required]
+        if compare_achievements and (matched_count > 0 or other_count > 0):
+            print("\nAchievement frequency by session (matched vs others):")
+            # Build rows with absolute percentage difference for sorting
+            rows: list[tuple[float, str, int, float, int, float]] = []
+            for name in compare_achievements:
+                m_n = matched_ach_counts.get(name, 0)
+                o_n = other_ach_counts.get(name, 0)
+                m_pct = (m_n / matched_count * 100.0) if matched_count else 0.0
+                o_pct = (o_n / other_count * 100.0) if other_count else 0.0
+                diff = abs(m_pct - o_pct)
+                rows.append((diff, name, m_n, m_pct, o_n, o_pct))
+
+            # Show top 10 differences
+            rows.sort(reverse=True)
+            limit = min(10, len(rows))
+            for i in range(limit):
+                _, name, m_n, m_pct, o_n, o_pct = rows[i]
+                print(
+                    f"  - {name}: matched {m_n}/{matched_count} ({m_pct:.1f}%), others {o_n}/{other_count} ({o_pct:.1f}%)"
+                )
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Crafter v3 trace analysis")
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    p_list = sub.add_parser("list", help="List achievements present in DB")
+    p_list.add_argument(
+        "--db", required=True, help="Path to sqld internal data file or full sqlite+aiosqlite URL"
+    )
+
+    p_filter = sub.add_parser("filter", help="Filter sessions by achievements and export JSONL")
+    p_filter.add_argument(
+        "--db", required=True, help="Path to sqld internal data file or full sqlite+aiosqlite URL"
+    )
+    p_filter.add_argument(
+        "--achievements",
+        nargs="+",
+        required=True,
+        help="Required achievements (any match keeps session)",
+    )
+    p_filter.add_argument("--output", required=True, help="Output JSONL path")
+    p_filter.add_argument("--models", nargs="*", help="Optional model names to include (any match)")
+
+    p_stats = sub.add_parser("stats", help="Show summary stats for filtered vs others")
+    p_stats.add_argument(
+        "--db", required=True, help="Path to sqld internal data file or full sqlite+aiosqlite URL"
+    )
+    p_stats.add_argument(
+        "--achievements", nargs="+", required=True, help="Achievements to match (any match)"
+    )
+    p_stats.add_argument("--models", nargs="*", help="Optional model names to include (any match)")
+
+    args = parser.parse_args()
+
+    if args.command == "list":
+        asyncio.run(cmd_list(args.db))
+        return
+    if args.command == "filter":
+        asyncio.run(cmd_filter(args.db, args.achievements, args.output, args.models or None))
+        return
+    if args.command == "stats":
+        asyncio.run(cmd_stats(args.db, args.achievements, args.models or None))
+        return
+
+
+if __name__ == "__main__":
+    main()

examples/finetuning_old/_backup_synth_qwen/config.toml

@@ -0,0 +1,29 @@
+# Centralized configuration for Synth Qwen Crafter workflows
+
+[rollouts]
+model = "Qwen/Qwen3-4B-Instruct-2507"
+episodes = 5
+max_steps = 30
+difficulty = "easy"
+temperature = 0.4
+max_tokens = 2048
+tool_choice = "required"
+
+[traces]
+sqld_db_path = "traces/v3/synth_ai.db"
+
+[filter]
+# For v3 sqld traces, use the internal data file under the sqld directory
+db_path = "traces/v3/synth_ai.db/dbs/default/data"
+required_achievements = ["collect_wood"]
+min_total_reward = 1.0
+max_cost = 10.0
+max_tokens = 100000
+output_jsonl = "ft_data/qwen4b_crafter_sft_collect_wood.jsonl"
+
+[sft]
+base_model = "Qwen/Qwen3-4B-Instruct-2507"
+training_jsonl = "ft_data/qwen4b_crafter_sft_collect_wood.jsonl"
+n_epochs = 1
+batch_size = 4
+upload_to_wasabi = true