synth-ai 0.2.9.dev7__py3-none-any.whl → 0.2.9.dev8__py3-none-any.whl

examples/evals_old/example_log.md

@@ -1,145 +0,0 @@
-joshuapurtell@Mac synth-ai % bash examples/evals/run_demo.sh                
-Models to compare (space-separated) [gpt-5-nano gpt-4.1-nano]: 
-Models: gpt-5-nano gpt-4.1-nano
-Episodes per model [3]: 5
-Max turns per episode [5]: 5
-Parallelism per model (concurrency) [5]: 5
-Difficulty [easy]: 
-Running comparison: episodes=5, max_turns=5, difficulty=easy, concurrency=5
-Detected SYNTH_API_KEY (sk_liv...ac95). Use this key? [Y/n]: n
-Use SYNTH_API_KEY_PROD (sk_liv...a2a4)? [y/N]: Y
-[PATCH] Attempting to apply Crafter deterministic patch...
-[PATCH] Patching crafter.Env._balance_object...
-[PATCH] crafter.Env._balance_object patched.
-[PATCH] Attempting to apply Crafter serialization patch v3...
-[PATCH] Adding enhanced save/load methods to crafter.Env...
-[PATCH] crafter.Env.save() and load() methods added (v3).
-[PATCH] Crafter serialization patch v3 complete.
-[PATCH] Attempting to apply simplified Crafter world configuration patch...
-[PATCH] Simplified Crafter world configuration patch complete.
-[PATCH] Available configs: easy, normal, hard, peaceful
-✅ Loaded 8 Crafter achievement hooks (Easy, Medium, Hard)
-🎮 Crafter Multi-Model Experiment
-==================================================
-Experiment ID: crafter_multi_model_20250808_170152
-Models: gpt-5-nano, gpt-4.1-nano
-Episodes per model: 5
-Max turns per episode: 5
-Difficulty: easy
-Seeds: 1000 to 1004
-Turn timeout: 20.0s
-Episode timeout: 180.0s
-Save traces: True
-Database URL: sqlite+aiosqlite:////Users/joshuapurtell/Documents/GitHub/synth-ai/traces/v3/synth_ai.db/dbs/default/data
-==================================================
-✅ Crafter service is running
-
-Running 5 episodes for gpt-5-nano in parallel...
-
-gpt-5-nano | ep1:   0%|                                                               | 0/5 [00:00<?, ?turn/s]
-Running 5 episodes for gpt-4.1-nano in parallel...                                    | 0/5 [00:00<?, ?turn/s]
-gpt-5-nano | ep3:   0%|                                                               | 0/5 [00:00<?, ?turn/s]
-gpt-4.1-nano | ep3: 100%|██████████████████████████████████████████████| 5/5 [00:09<00:00,  1.95s/turn, ach=1]
-gpt-4.1-nano | ep2:  80%|████████████████████████████████████▊         | 4/5 [00:10<00:02,  2.64s/turn, ach=2]
-gpt-4.1-nano | ep4: 100%|██████████████████████████████████████████████| 5/5 [00:11<00:00,  2.32s/turn, ach=0]
-gpt-4.1-nano | ep5: 100%|██████████████████████████████████████████████| 5/5 [00:11<00:00,  2.37s/turn, ach=2]
-gpt-5-nano | ep1:  20%|█████████▌                                      | 1/5 [00:21<01:24, 21.13s/turn, ach=0    ⏰ Turn 3 timed out for episode 0 after 20.0s                       | 2/5 [00:25<00:38, 12.83s/turn, ach=0]
-gpt-4.1-nano | ep1:  60%|███████████████████████████▌                  | 3/5 [00:28<00:19,  9.62s/turn, ach=1]
-gpt-5-nano | ep3: 100%|████████████████████████████████████████████████| 5/5 [01:00<00:00, 12.05s/turn, ach=1]
-gpt-5-nano | ep2: 100%|████████████████████████████████████████████████| 5/5 [01:07<00:00, 13.56s/turn, ach=2]
-    ⏰ Turn 4 timed out for episode 3 after 20.0s██████████████████████| 5/5 [01:07<00:00, 14.04s/turn, ach=2]
-gpt-5-nano | ep4:  80%|██████████████████████████████████████▍         | 4/5 [01:08<00:17, 17.02s/turn, ach=0]
-gpt-5-nano | ep5: 100%|████████████████████████████████████████████████| 5/5 [01:13<00:00, 14.71s/turn, ach=1]
-gpt-5-nano | ep1: 100%|████████████████████████████████████████████████| 5/5 [01:19<00:00, 15.83s/turn, ach=1]
-gpt-4.1-nano | ep5: 100%|██████████████████████████████████████████████| 5/5 [00:11<00:00,  1.68s/turn, ach=2]
-📊 Analysis Results:
-================================================================================:13<00:00, 14.26s/turn, ach=1]
-
-📈 Model Performance Summary:
-Model                Avg Achievements   Max Achievements   Invalid Rate    Success Rate   
---------------------------------------------------------------------------------------
-gpt-4.1-nano           1.20 ± 0.75                    2            0.00%          100.00%
-gpt-5-nano             1.00 ± 0.63                    2            0.00%          100.00%
-
-🏆 Achievement Frequencies:
-
-Achievement                 gpt-4.1-nano   gpt-5-nano
------------------------------------------------
-collect_drink               2/5   ( 40%)   0/5   (  0%)
-collect_sapling             1/5   ( 20%)   2/5   ( 40%)
-collect_wood                3/5   ( 60%)   2/5   ( 40%)
-place_plant                 0/5   (  0%)   1/5   ( 20%)
-
-💰 Model Usage Statistics from Current Experiment:
-Model                Provider   Usage Count  Avg Latency (ms)   Total Cost  
-------------------------------------------------------------------------
-gpt-5-nano           openai     221          13006.57           $0.0000     
-gpt-4.1-nano         openai     161          950.12             $0.0000     
-
-💾 Detailed results saved to: /Users/joshuapurtell/Documents/GitHub/synth-ai/temp/crafter_experiment_results_20250808_170312.json
-
-✅ Experiment complete!
-Using v3 traces DB: /Users/joshuapurtell/Documents/GitHub/synth-ai/traces/v3/synth_ai.db/dbs/default/data
-\nAvailable achievements (session counts):
-[PATCH] Attempting to apply Crafter deterministic patch...
-[PATCH] Patching crafter.Env._balance_object...
-[PATCH] crafter.Env._balance_object patched.
-[PATCH] Attempting to apply Crafter serialization patch v3...
-[PATCH] Adding enhanced save/load methods to crafter.Env...
-[PATCH] crafter.Env.save() and load() methods added (v3).
-[PATCH] Crafter serialization patch v3 complete.
-[PATCH] Attempting to apply simplified Crafter world configuration patch...
-[PATCH] Simplified Crafter world configuration patch complete.
-[PATCH] Available configs: easy, normal, hard, peaceful
-Achievements present (session counts):
-  - collect_drink: 44
-  - collect_sapling: 62
-  - collect_wood: 74
-  - defeat_skeleton: 4
-  - defeat_zombie: 2
-  - eat_cow: 2
-  - place_plant: 8
-  - place_table: 3
-\nEnter achievements to filter by (space-separated), or press Enter for 'collect_wood':
-
-Optionally restrict to models (space-separated), or press Enter to include all:
-
-\nRunning: uv run python -m examples.evals.trace_analysis filter --db "/Users/joshuapurtell/Documents/GitHub/synth-ai/traces/v3/synth_ai.db/dbs/default/data" --achievements collect_wood --output ft_data/evals_filtered.jsonl
-[PATCH] Attempting to apply Crafter deterministic patch...
-[PATCH] Patching crafter.Env._balance_object...
-[PATCH] crafter.Env._balance_object patched.
-[PATCH] Attempting to apply Crafter serialization patch v3...
-[PATCH] Adding enhanced save/load methods to crafter.Env...
-[PATCH] crafter.Env.save() and load() methods added (v3).
-[PATCH] Crafter serialization patch v3 complete.
-[PATCH] Attempting to apply simplified Crafter world configuration patch...
-[PATCH] Simplified Crafter world configuration patch complete.
-[PATCH] Available configs: easy, normal, hard, peaceful
-✅ Wrote 74 examples from 74 sessions → ft_data/evals_filtered.jsonl
-\nRunning: uv run python -m examples.evals.trace_analysis stats --db "/Users/joshuapurtell/Documents/GitHub/synth-ai/traces/v3/synth_ai.db/dbs/default/data" --achievements collect_wood
-[PATCH] Attempting to apply Crafter deterministic patch...
-[PATCH] Patching crafter.Env._balance_object...
-[PATCH] crafter.Env._balance_object patched.
-[PATCH] Attempting to apply Crafter serialization patch v3...
-[PATCH] Adding enhanced save/load methods to crafter.Env...
-[PATCH] crafter.Env.save() and load() methods added (v3).
-[PATCH] Crafter serialization patch v3 complete.
-[PATCH] Attempting to apply simplified Crafter world configuration patch...
-[PATCH] Simplified Crafter world configuration patch complete.
-[PATCH] Available configs: easy, normal, hard, peaceful
-Matched sessions (any of: collect_wood )
-  n=74  avg_reward=0.76  stddev=1.00
-  avg_first_unlock_step=4.7  stddev=4.6
-Others
-  n=224  avg_reward=0.21  stddev=0.51
-
-Achievement frequency by session (matched vs others):
-  - collect_drink: matched 25/74 (33.8%), others 19/224 (8.5%)
-  - collect_sapling: matched 21/74 (28.4%), others 41/224 (18.3%)
-  - place_table: matched 3/74 (4.1%), others 0/224 (0.0%)
-  - eat_cow: matched 2/74 (2.7%), others 0/224 (0.0%)
-  - place_plant: matched 3/74 (4.1%), others 5/224 (2.2%)
-  - defeat_skeleton: matched 2/74 (2.7%), others 2/224 (0.9%)
-  - defeat_zombie: matched 0/74 (0.0%), others 2/224 (0.9%)
-\nDone. See ft_data/evals_filtered.jsonl and v3 DB for deeper analysis.
-joshuapurtell@Mac synth-ai % 

examples/evals_old/run_demo.sh

@@ -1,126 +0,0 @@
-#!/bin/bash
-
-# Run Crafter experiments comparing gpt-5-nano and Qwen/Qwen3-32B-Instruct
-
-# Get the directory where this script is located
-SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
-
-# Change to the synth-ai root directory
-cd "$SCRIPT_DIR/../.."
-
-# Interactive mini-demo: run small comparison, then analyze v3 traces
-set -euo pipefail
-
-# Load env (prefer local .env at repo root)
-set +u
-set -a
-if [ -f ".env" ]; then source ".env"; fi
-set +a
-set -u
-
-# Ensure API key present (SYNTH_API_KEY, optionally mirror to OPENAI_API_KEY)
-ensure_api_key() {
-  local current_key="${SYNTH_API_KEY:-}"
-  if [ -n "$current_key" ]; then
-    local preview="${current_key:0:6}...${current_key: -4}"
-    read -r -p "Detected SYNTH_API_KEY ($preview). Use this key? [Y/n]: " USE_CUR || true
-    USE_CUR=${USE_CUR:-Y}
-    if [[ ! "$USE_CUR" =~ ^[Yy]$ ]]; then
-      current_key=""
-    fi
-  fi
-
-  if [ -z "$current_key" ] && [ -n "${SYNTH_API_KEY_PROD:-}" ]; then
-    local prod_prev="${SYNTH_API_KEY_PROD:0:6}...${SYNTH_API_KEY_PROD: -4}"
-    read -r -p "Use SYNTH_API_KEY_PROD ($prod_prev)? [y/N]: " USE_PROD || true
-    if [[ "$USE_PROD" =~ ^[Yy]$ ]]; then
-      current_key="$SYNTH_API_KEY_PROD"
-    fi
-  fi
-
-  while [ -z "$current_key" ]; do
-    echo
-    read -s -p "Enter your SYNTH_API_KEY: " KEY_IN || true
-    echo
-    if [ -n "$KEY_IN" ]; then
-      current_key="$KEY_IN"
-    else
-      echo "A valid SYNTH_API_KEY is required to continue."
-    fi
-  done
-
-  export SYNTH_API_KEY="$current_key"
-  if [ -z "${OPENAI_API_KEY:-}" ]; then
-    export OPENAI_API_KEY="$SYNTH_API_KEY"
-    echo "OPENAI_API_KEY set from SYNTH_API_KEY."
-  fi
-}
-
-# Interactive prompts (with sensible defaults)
-MODELS_DEFAULT="gpt-5-nano gpt-4.1-nano"
-read -r -p "Models to compare (space-separated) [${MODELS_DEFAULT}]: " MODELS_INPUT || true
-MODELS=${MODELS_INPUT:-$MODELS_DEFAULT}
-echo "Models: ${MODELS}"
-
-read -r -p "Episodes per model [3]: " EPISODES_INPUT || true
-EPISODES=${EPISODES_INPUT:-3}
-
-read -r -p "Max turns per episode [5]: " MAX_TURNS_INPUT || true
-MAX_TURNS=${MAX_TURNS_INPUT:-5}
-
-read -r -p "Parallelism per model (concurrency) [5]: " CONCURRENCY_INPUT || true
-CONCURRENCY=${CONCURRENCY_INPUT:-5}
-
-read -r -p "Difficulty [easy]: " DIFFICULTY_INPUT || true
-DIFFICULTY=${DIFFICULTY_INPUT:-easy}
-
-echo "Running comparison: episodes=${EPISODES}, max_turns=${MAX_TURNS}, difficulty=${DIFFICULTY}, concurrency=${CONCURRENCY}"
-
-# Ensure key before running rollouts
-ensure_api_key
-
-uv run python examples/evals/compare_models.py \
-    --episodes "${EPISODES}" \
-    --max-turns "${MAX_TURNS}" \
-    --difficulty "${DIFFICULTY}" \
-    --models ${MODELS} \
-    --base-seed 1000 \
-    --turn-timeout 20.0 \
-    --episode-timeout 180.0 \
-    --concurrency "${CONCURRENCY}" \
-    --quiet
-
-# Derive v3 sqld internal DB path for quick analysis
-DB_PATH="$PWD/traces/v3/synth_ai.db/dbs/default/data"
-export DB_PATH
-echo "Using v3 traces DB: $DB_PATH"
-
-echo "\nAvailable achievements (session counts):"
-uv run python -m examples.evals.trace_analysis list --db "$DB_PATH"
-
-echo "\nEnter achievements to filter by (space-separated), or press Enter for 'collect_wood':"
-read -r ACH
-ACH=${ACH:-collect_wood}
-
-echo "Optionally restrict to models (space-separated), or press Enter to include all:"
-read -r MODELS_FILTER
-
-mkdir -p ft_data
-if [ -n "$MODELS_FILTER" ]; then
-  echo "\nRunning: uv run python -m examples.evals.trace_analysis filter --db \"$DB_PATH\" --achievements $ACH --output ft_data/evals_filtered.jsonl --models $MODELS_FILTER"
-  uv run python -m examples.evals.trace_analysis filter --db "$DB_PATH" --achievements $ACH --output ft_data/evals_filtered.jsonl --models $MODELS_FILTER
-else
-  echo "\nRunning: uv run python -m examples.evals.trace_analysis filter --db \"$DB_PATH\" --achievements $ACH --output ft_data/evals_filtered.jsonl"
-  uv run python -m examples.evals.trace_analysis filter --db "$DB_PATH" --achievements $ACH --output ft_data/evals_filtered.jsonl
-fi
-
-# Show stats comparing filtered vs others (including achievement frequencies)
-if [ -n "$MODELS_FILTER" ]; then
-  echo "\nRunning: uv run python -m examples.evals.trace_analysis stats --db \"$DB_PATH\" --achievements $ACH --models $MODELS_FILTER"
-  uv run python -m examples.evals.trace_analysis stats --db "$DB_PATH" --achievements $ACH --models $MODELS_FILTER
-else
-  echo "\nRunning: uv run python -m examples.evals.trace_analysis stats --db \"$DB_PATH\" --achievements $ACH"
-  uv run python -m examples.evals.trace_analysis stats --db "$DB_PATH" --achievements $ACH
-fi
-
-echo "\nDone. See ft_data/evals_filtered.jsonl and v3 DB for deeper analysis."

examples/evals_old/trace_analysis.py

@@ -1,270 +0,0 @@
-#!/usr/bin/env python3
-"""
-Trace analysis utilities for Crafter v3 traces (sqld/Turso).
-
-Subcommands:
-  - list:    List achievements present in the database and counts
-  - filter:  Filter sessions by required achievements and export OpenAI-format JSONL
-  - stats:   Compare rewards and achievement frequencies for filtered vs. others
-
-Usage examples:
-  uvpm examples.evals.trace_analysis list --db traces/v3/synth_ai.db/dbs/default/data
-  uvpm examples.evals.trace_analysis filter --db traces/v3/synth_ai.db/dbs/default/data \
-      --achievements collect_wood --output ft_data/evals_collect_wood.jsonl
-"""
-
-import argparse
-import asyncio
-import json
-import json as pyjson
-import math
-from pathlib import Path
-
-from synth_ai.environments.examples.crafter_classic.agent_demos.crafter_modal_ft.filter_traces_sft_turso import (
-    FinetuningDataExtractorV3,
-)
-
-
-def build_db_url(path: str) -> str:
-    if path.startswith("sqlite+"):
-        return path
-    return f"sqlite+aiosqlite:///{path}"
-
-
-async def cmd_list(db_path: str) -> None:
-    db_url = build_db_url(db_path)
-    async with FinetuningDataExtractorV3(db_url) as ex:
-        sessions = await ex.get_all_sessions()
-        achievement_counts: dict[str, int] = {}
-        for _, row in sessions.iterrows():
-            ach_list = await ex.get_session_achievements(row["session_id"]) or []
-            for name in ach_list:
-                achievement_counts[name] = achievement_counts.get(name, 0) + 1
-
-        print("Achievements present (session counts):")
-        for name in sorted(achievement_counts.keys()):
-            print(f"  - {name}: {achievement_counts[name]}")
-
-
-async def cmd_filter(
-    db_path: str, achievements: list[str], output: str, models: list[str] | None = None
-) -> None:
-    db_url = build_db_url(db_path)
-    required: set[str] = set(achievements)
-    async with FinetuningDataExtractorV3(db_url) as ex:
-        sessions = await ex.get_all_sessions()
-        kept: list[str] = []
-        for _, row in sessions.iterrows():
-            if models:
-                # Restrict to sessions containing any of the requested models
-                model_df = await ex.db_manager.query_traces(
-                    """
-                    SELECT DISTINCT model_name
-                    FROM events
-                    WHERE session_id = :session_id
-                      AND event_type = 'cais'
-                      AND model_name IS NOT NULL
-                    """,
-                    {"session_id": row["session_id"]},
-                )
-                session_models = (
-                    model_df["model_name"].tolist()
-                    if model_df is not None and not model_df.empty
-                    else []
-                )
-                if not any(m in session_models for m in models):
-                    continue
-            ach_list = await ex.get_session_achievements(row["session_id"]) or []
-            if required & set(ach_list):
-                kept.append(row["session_id"])
-
-        data = await ex.extract_openai_format(kept)
-        Path(output).parent.mkdir(parents=True, exist_ok=True)
-        with open(output, "w") as f:
-            for exm in data:
-                f.write(json.dumps(exm) + "\n")
-        print(f"✅ Wrote {len(data)} examples from {len(kept)} sessions → {output}")
-
-
-async def _first_achievement_step(
-    ex: FinetuningDataExtractorV3, session_id: str, required: set[str]
-) -> int | None:
-    q = """
-        SELECT message_time, system_state_after
-        FROM events
-        WHERE session_id = :session_id
-          AND event_type = 'environment'
-        ORDER BY message_time ASC
-        """
-    df = await ex.db_manager.query_traces(q, {"session_id": session_id})
-    if df is None or df.empty:
-        return None
-    seen: set[str] = set()
-    for _, row in df.iterrows():
-        st = row.get("system_state_after")
-        if isinstance(st, str):
-            try:
-                st = pyjson.loads(st)
-            except Exception:
-                st = None
-        ach = None
-        if isinstance(st, dict):
-            ps = st.get("public_state") or {}
-            ach = ps.get("achievements_status") or {}
-        if isinstance(ach, dict):
-            for name, unlocked in ach.items():
-                if unlocked and name in required and name not in seen:
-                    return int(row.get("message_time") or 0)
-    return None
-
-
-def _mean(values: list[float]) -> float:
-    return (sum(values) / len(values)) if values else 0.0
-
-
-def _stddev(values: list[float]) -> float:
-    if not values:
-        return 0.0
-    m = _mean(values)
-    var = sum((v - m) * (v - m) for v in values) / len(values)
-    return math.sqrt(var)
-
-
-async def cmd_stats(db_path: str, achievements: list[str], models: list[str] | None = None) -> None:
-    db_url = build_db_url(db_path)
-    required: set[str] = set(achievements)
-    async with FinetuningDataExtractorV3(db_url) as ex:
-        sessions = await ex.get_all_sessions()
-        matched_rewards: list[float] = []
-        other_rewards: list[float] = []
-        first_steps: list[int] = []
-        matched_count: int = 0
-        other_count: int = 0
-        matched_ach_counts: dict[str, int] = {}
-        other_ach_counts: dict[str, int] = {}
-
-        for _, row in sessions.iterrows():
-            sid = row["session_id"]
-            if models:
-                model_df = await ex.db_manager.query_traces(
-                    """
-                    SELECT DISTINCT model_name
-                    FROM events
-                    WHERE session_id = :session_id
-                      AND event_type = 'cais'
-                      AND model_name IS NOT NULL
-                    """,
-                    {"session_id": sid},
-                )
-                session_models = (
-                    model_df["model_name"].tolist()
-                    if model_df is not None and not model_df.empty
-                    else []
-                )
-                if not any(m in session_models for m in models):
-                    continue
-
-            ach_list = await ex.get_session_achievements(sid) or []
-            metrics = await ex.get_session_metrics(sid)
-            reward = float(metrics.get("total_reward", 0.0))
-
-            if required & set(ach_list):
-                matched_rewards.append(reward)
-                step = await _first_achievement_step(ex, sid, required)
-                if step is not None:
-                    first_steps.append(step)
-                matched_count += 1
-                for name in ach_list:
-                    matched_ach_counts[name] = matched_ach_counts.get(name, 0) + 1
-            else:
-                other_rewards.append(reward)
-                other_count += 1
-                for name in ach_list:
-                    other_ach_counts[name] = other_ach_counts.get(name, 0) + 1
-
-        print("Matched sessions (any of:", ", ".join(sorted(required)), ")")
-        print(
-            f"  n={len(matched_rewards)}  avg_reward={_mean(matched_rewards):.2f}  stddev={_stddev(matched_rewards):.2f}"
-        )
-        if first_steps:
-            print(
-                f"  avg_first_unlock_step={_mean([float(s) for s in first_steps]):.1f}  stddev={_stddev([float(s) for s in first_steps]):.1f}"
-            )
-        else:
-            print("  avg_first_unlock_step=n/a (no unlocks recorded)")
-        print("Others")
-        print(
-            f"  n={len(other_rewards)}  avg_reward={_mean(other_rewards):.2f}  stddev={_stddev(other_rewards):.2f}"
-        )
-
-        # Achievement frequency comparison (by session presence), excluding required achievements
-        all_achievements: set[str] = set(matched_ach_counts.keys()) | set(other_ach_counts.keys())
-        compare_achievements = [a for a in sorted(all_achievements) if a not in required]
-        if compare_achievements and (matched_count > 0 or other_count > 0):
-            print("\nAchievement frequency by session (matched vs others):")
-            # Build rows with absolute percentage difference for sorting
-            rows: list[tuple[float, str, int, float, int, float]] = []
-            for name in compare_achievements:
-                m_n = matched_ach_counts.get(name, 0)
-                o_n = other_ach_counts.get(name, 0)
-                m_pct = (m_n / matched_count * 100.0) if matched_count else 0.0
-                o_pct = (o_n / other_count * 100.0) if other_count else 0.0
-                diff = abs(m_pct - o_pct)
-                rows.append((diff, name, m_n, m_pct, o_n, o_pct))
-
-            # Show top 10 differences
-            rows.sort(reverse=True)
-            limit = min(10, len(rows))
-            for i in range(limit):
-                _, name, m_n, m_pct, o_n, o_pct = rows[i]
-                print(
-                    f"  - {name}: matched {m_n}/{matched_count} ({m_pct:.1f}%), others {o_n}/{other_count} ({o_pct:.1f}%)"
-                )
-
-
-def main() -> None:
-    parser = argparse.ArgumentParser(description="Crafter v3 trace analysis")
-    sub = parser.add_subparsers(dest="command", required=True)
-
-    p_list = sub.add_parser("list", help="List achievements present in DB")
-    p_list.add_argument(
-        "--db", required=True, help="Path to sqld internal data file or full sqlite+aiosqlite URL"
-    )
-
-    p_filter = sub.add_parser("filter", help="Filter sessions by achievements and export JSONL")
-    p_filter.add_argument(
-        "--db", required=True, help="Path to sqld internal data file or full sqlite+aiosqlite URL"
-    )
-    p_filter.add_argument(
-        "--achievements",
-        nargs="+",
-        required=True,
-        help="Required achievements (any match keeps session)",
-    )
-    p_filter.add_argument("--output", required=True, help="Output JSONL path")
-    p_filter.add_argument("--models", nargs="*", help="Optional model names to include (any match)")
-
-    p_stats = sub.add_parser("stats", help="Show summary stats for filtered vs others")
-    p_stats.add_argument(
-        "--db", required=True, help="Path to sqld internal data file or full sqlite+aiosqlite URL"
-    )
-    p_stats.add_argument(
-        "--achievements", nargs="+", required=True, help="Achievements to match (any match)"
-    )
-    p_stats.add_argument("--models", nargs="*", help="Optional model names to include (any match)")
-
-    args = parser.parse_args()
-
-    if args.command == "list":
-        asyncio.run(cmd_list(args.db))
-        return
-    if args.command == "filter":
-        asyncio.run(cmd_filter(args.db, args.achievements, args.output, args.models or None))
-        return
-    if args.command == "stats":
-        asyncio.run(cmd_stats(args.db, args.achievements, args.models or None))
-        return
-
-
-if __name__ == "__main__":
-    main()

examples/finetuning_old/_backup_synth_qwen/config.toml

@@ -1,29 +0,0 @@
-# Centralized configuration for Synth Qwen Crafter workflows
-
-[rollouts]
-model = "Qwen/Qwen3-4B-Instruct-2507"
-episodes = 5
-max_steps = 30
-difficulty = "easy"
-temperature = 0.4
-max_tokens = 2048
-tool_choice = "required"
-
-[traces]
-sqld_db_path = "traces/v3/synth_ai.db"
-
-[filter]
-# For v3 sqld traces, use the internal data file under the sqld directory
-db_path = "traces/v3/synth_ai.db/dbs/default/data"
-required_achievements = ["collect_wood"]
-min_total_reward = 1.0
-max_cost = 10.0
-max_tokens = 100000
-output_jsonl = "ft_data/qwen4b_crafter_sft_collect_wood.jsonl"
-
-[sft]
-base_model = "Qwen/Qwen3-4B-Instruct-2507"
-training_jsonl = "ft_data/qwen4b_crafter_sft_collect_wood.jsonl"
-n_epochs = 1
-batch_size = 4
-upload_to_wasabi = true