synth-ai 0.2.9.dev7__py3-none-any.whl → 0.2.9.dev8__py3-none-any.whl

examples/sft/README.md

@@ -0,0 +1,139 @@
+### Supervised Fine-Tuning for Crafter
+
+This folder provides a minimal, reusable SFT workflow that pulls out the SFT step from `examples/warming_up_to_rl/` and focuses it on LoRA/QLoRA. We've also added guidance for running full finetuning (FFT) so you can compare adapters against end-to-end weight updates.
+
+It supports distilling Groq (or other vendor) rollouts into JSONL using tracing and then training a small base model like `Qwen/Qwen3-0.6B`.
+
+---
+
+### 0) Load environment from .env.dev (recommended)
+
+Use your dev env file so keys/URLs are sourced consistently:
+
+```bash
+# Example path; update to your actual dev env
+set -a && source /Users/joshpurtell/Documents/GitHub/monorepo/backend/.env.dev && set +a
+```
+
+This ensures `ENVIRONMENT_API_KEY`, `GROQ_API_KEY`, and (optionally) `BACKEND_BASE_URL` are available to the steps below.
+
+---
+
+### 1) Collect traces and export SFT JSONL
+
+You can generate traces with the Crafter task app and then export them to SFT JSONL using the existing exporter:
+
+```bash
+# Serve the task app locally with tracing enabled (example)
+uvx synth-ai serve grpo-crafter \
+  --trace traces/v3 \
+  --trace-db traces/v3/synth_ai.db \
+  --port 8001
+
+# Or run traced local rollouts to accumulate data
+uv run python examples/warming_up_to_rl/run_local_rollout_traced.py \
+  --episodes 50 --max-turns 10
+
+# Export SFT dataset from the trace DB
+uv run python examples/warming_up_to_rl/export_trace_sft.py \
+  --db traces/v3/synth_ai.db \
+  --min-unique 0 \
+  --output examples/sft/ft_data/crafter_traces.jsonl
+```
+
+Notes:
+- The exporter uses achievements and event rewards to filter high-signal steps. Combine `--min-unique`, `--min-outcome-reward`, `--event-reward`, and `--require-achievement` to control data quality.
+- You can restrict to sessions from certain providers/models with `--provider`/`--model`.
+- Use `--limit` while debugging to reduce dataset size quickly.
+
+---
+
+### 2a) Train LoRA (QLoRA) on Qwen/Qwen3-0.6B
+
+Use the standard CLI. Do not use a custom Python finetuning script. Point the CLI at your `.env.dev` so it picks up keys automatically:
+
+```bash
+uvx synth-ai train \
+  --type sft \
+  --config examples/sft/configs/crafter_lora_qwen0p6b.toml \
+  --dataset examples/sft/ft_data/crafter_traces.jsonl \
+  --env-file /Users/joshpurtell/Documents/GitHub/monorepo/backend/.env.dev
+```
+
+The config sets `training.use_qlora = true` and `hyperparameters.train_kind = "peft"` to request LoRA adapters.
+
+Experiment tips:
+- The backend currently defaults to a LoRA rank of 16. If you need other ranks, generate the payload with `--dry-run`, add `"lora_rank": <value>` (and optional `"lora_alpha"`, `"lora_dropout"`) under `hyperparameters`, and submit it via the API until the CLI exposes these knobs directly.
+- Duplicate the TOML and adjust `hyperparameters.warmup_ratio`, `learning_rate`, or `gradient_accumulation_steps` to keep the global batch size comparable across datasets.
+
+---
+
+### 2b) Train Full Finetune (FFT) on Qwen/Qwen3-0.6B
+
+Full finetuning updates all weights and uses a near-identical CLI flow with the LoRA toggle disabled. The helper config lives alongside the LoRA sample:
+
+```bash
+uvx synth-ai train \
+  --type sft \
+  --config examples/sft/configs/crafter_fft_qwen0p6b.toml \
+  --dataset examples/sft/ft_data/crafter_traces.jsonl \
+  --env-file /Users/joshpurtell/Documents/GitHub/monorepo/backend/.env.dev
+```
+
+Key differences vs LoRA:
+- `training.use_qlora = false` and `hyperparameters.train_kind = "fft"` request a full-weight update.
+- `per_device_batch` defaults to 1 to keep memory use comfortable on a single H100; raise gradually as you confirm headroom.
+- FFT runs slower per step. Consider trimming the dataset with `--examples` or the exporter filters for quick baselines.
+
+If you want the 4B Crafter FFT baseline from the RL examples, reuse `examples/warming_up_to_rl/configs/crafter_fft_4b.toml` with the same CLI command.
+
+---
+
+### 3) Evaluate the fine-tuned models
+
+After the job completes, list your fine-tuned models and evaluate them in the Crafter loop:
+
+```bash
+# List models
+uv run python - <<'PY'
+import asyncio
+import os
+from synth_ai.learning.client import LearningClient
+
+backend = os.getenv("BACKEND_BASE_URL", "https://agent-learning.onrender.com/api")
+api_key = os.getenv("SYNTH_API_KEY", "")
+async def main():
+    client = LearningClient(backend, api_key)
+    models = await client.list_fine_tuned_models()
+    for m in models:
+        print(m)
+asyncio.run(main())
+PY
+
+# Evaluate in the Crafter eval loop (example via warming_up_to_rl)
+TASK_APP_URL=http://localhost:8001 \
+uv run python examples/warming_up_to_rl/run_eval.py \
+  --toml examples/warming_up_to_rl/configs/eval_local_vllm.toml \
+  --model ft:YOUR_FT_MODEL_ID \
+  --use-rollout
+```
+
+---
+
+### 4) Plan comparison runs
+
+Keep runs comparable by adjusting one axis at a time and logging the settings in your experiment tracker (spreadsheet, weights & biases, etc.).
+
+- **LoRA rank sweeps:** start from `crafter_lora_qwen0p6b.toml`, clone it per rank (e.g., `r=4,8,16,64`). For now add the desired `lora_rank` in the job payload manually (see note above) and include it in the run name.
+- **Dataset size:** duplicate the exported JSONL and slice with `head -n`, or pass `--examples N` to the CLI for quick subsamples. Track the effective token count using the exporter logs.
+- **Data quality:** increase `--min-unique`, require specific achievements, or exclude low-reward sessions with `export_trace_sft.py`. Capture the filter tuple in your run metadata so evaluations stay reproducible.
+- **FFT vs LoRA:** run both configs on the same dataset/cardinality so differences reflect the training method rather than the data.
+
+For each sweep, use consistent evaluation seeds and write down throughput (tokens/sec) so you can weigh quality vs cost.
+
+---
+
+### Files
+- `configs/crafter_lora_qwen0p6b.toml`: LoRA/QLoRA SFT config for `Qwen/Qwen3-0.6B`.
+- `configs/crafter_fft_qwen0p6b.toml`: Full-finetune SFT config for `Qwen/Qwen3-0.6B`.
+- `ft_data/`: place your exported JSONL here (ignored by VCS).

examples/sft/configs/crafter_fft_qwen0p6b.toml

@@ -0,0 +1,44 @@
+[job]
+model = "Qwen/Qwen3-0.6B"
+# Prefer passing --dataset at runtime for repeatability
+# data = "examples/sft/ft_data/crafter_traces.jsonl"
+
+[compute]
+gpu_type = "H100"
+gpu_count = 1
+nodes = 1
+
+[data]
+topology = {}
+# Optional validation set if you have one locally
+# validation_path = "examples/sft/ft_data/crafter_traces.val.jsonl"
+
+[training]
+mode = "sft_offline"
+use_qlora = false
+
+[training.validation]
+enabled = true
+evaluation_strategy = "steps"
+eval_steps = 50
+save_best_model_at_end = true
+metric_for_best_model = "val.loss"
+greater_is_better = false
+
+[hyperparameters]
+n_epochs = 1
+train_kind = "fft"
+per_device_batch = 1
+gradient_accumulation_steps = 32
+sequence_length = 4096
+learning_rate = 1e-5
+warmup_ratio = 0.03
+weight_decay = 0.01
+
+[hyperparameters.parallelism]
+use_deepspeed = true
+deepspeed_stage = 2
+fsdp = false
+bf16 = true
+fp16 = false
+activation_checkpointing = true

examples/sft/configs/crafter_lora_qwen0p6b.toml

@@ -0,0 +1,45 @@
+[job]
+model = "Qwen/Qwen3-0.6B"
+# Optionally set here, but prefer passing --dataset at runtime
+# data = "examples/sft/ft_data/crafter_traces.jsonl"
+
+[compute]
+gpu_type = "H100"
+gpu_count = 1
+nodes = 1
+
+[data]
+# Forwarded into metadata.effective_config
+topology = {}
+# Optional validation set if you have one locally
+# validation_path = "examples/sft/ft_data/crafter_traces.val.jsonl"
+
+[training]
+mode = "lora"
+use_qlora = true
+
+[training.validation]
+enabled = true
+evaluation_strategy = "steps"
+eval_steps = 50
+save_best_model_at_end = true
+metric_for_best_model = "val.loss"
+greater_is_better = false
+
+[hyperparameters]
+n_epochs = 1
+train_kind = "peft"
+per_device_batch = 2
+gradient_accumulation_steps = 32
+sequence_length = 4096
+learning_rate = 5e-6
+warmup_ratio = 0.03
+
+[hyperparameters.parallelism]
+use_deepspeed = true
+deepspeed_stage = 2
+fsdp = false
+bf16 = true
+fp16 = false
+activation_checkpointing = true
+

examples/sft/evaluate.py

@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+"""Evaluate a base or fine-tuned model on Crafter via the Task App rollout.
+
+This mirrors the minimal evaluation loop: call `/rollout` for a set of seeds
+and report outcome/step metrics. If tracing is enabled server-side, you can
+use the exported sqlite DB for further analysis.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import os
+from dataclasses import dataclass
+from typing import Any
+
+from synth_ai.task import (
+    RolloutEnvSpec,
+    RolloutPolicySpec,
+    RolloutRecordConfig,
+    RolloutRequest,
+    TaskAppClient,
+)
+
+
+@dataclass(slots=True)
+class EvalArgs:
+    base_url: str
+    api_key: str
+    model: str
+    inference_url: str
+    inference_api_key: str
+    seeds: list[int]
+    max_llm_calls: int
+    timeout: float
+
+
+def _ops(n: int) -> list[str]:
+    n = max(1, n)
+    ops: list[str] = []
+    for _ in range(n):
+        ops.extend(["agent", "env"])  # one LLM step followed by one env step
+    return ops
+
+
+def _request(seed: int, a: EvalArgs) -> RolloutRequest:
+    return RolloutRequest(
+        run_id=f"eval-{seed}",
+        env=RolloutEnvSpec(env_name="crafter", seed=seed, config={}),
+        policy=RolloutPolicySpec(
+            policy_name="crafter-react",
+            config={"model": a.model, "inference_url": a.inference_url, "api_key": a.inference_api_key},
+        ),
+        ops=_ops(a.max_llm_calls),
+        record=RolloutRecordConfig(trajectories=True, return_trace=False, trace_format="compact"),
+    )
+
+
+async def _eval_seed(client: TaskAppClient, seed: int, a: EvalArgs) -> dict[str, Any]:
+    resp = await client.rollout(_request(seed, a))
+    m = resp.metrics
+    return {"seed": seed, "num_steps": m.num_steps, "episode_returns": m.episode_returns, "outcome_score": m.outcome_score}
+
+
+async def main() -> None:
+    p = argparse.ArgumentParser(description=__doc__)
+    p.add_argument("--base-url", default=os.getenv("TASK_APP_URL", "http://localhost:8001"))
+    p.add_argument("--api-key", default=os.getenv("ENVIRONMENT_API_KEY"))
+    p.add_argument("--model", required=True, help="Base or ft:<id> to evaluate")
+    p.add_argument("--inference-url", default=os.getenv("INFERENCE_URL", "https://api.groq.com/openai"))
+    p.add_argument("--inference-api-key", default=os.getenv("GROQ_API_KEY"))
+    p.add_argument("--seeds", default="0,1,2,3,4,5,6,7,8,9")
+    p.add_argument("--max-llm-calls", type=int, default=10)
+    p.add_argument("--timeout", type=float, default=60.0)
+    args = p.parse_args()
+
+    seeds = [int(s) for s in str(args.seeds).split(",") if s.strip()]
+    a = EvalArgs(
+        base_url=str(args.base_url).strip(),
+        api_key=str(args.api_key or "").strip(),
+        model=str(args.model).strip(),
+        inference_url=str(args.inference_url).strip(),
+        inference_api_key=str(args.inference_api_key or "").strip(),
+        seeds=seeds,
+        max_llm_calls=int(args.max_llm_calls),
+        timeout=float(args.timeout),
+    )
+    if not a.api_key:
+        raise SystemExit("ENVIRONMENT_API_KEY is required")
+    if not a.inference_api_key:
+        raise SystemExit("Inference API key (e.g., GROQ_API_KEY) is required")
+
+    results: list[dict[str, Any]] = []
+    async with TaskAppClient(a.base_url, api_key=a.api_key, timeout=a.timeout) as client:
+        for seed in a.seeds:
+            r = await _eval_seed(client, seed, a)
+            results.append(r)
+            print(f"seed={seed} return={r.get('episode_returns')}")
+
+    # Simple aggregate
+    flat_returns: list[float] = []
+    for r in results:
+        ers = r.get("episode_returns") or []
+        if isinstance(ers, list) and ers:
+            try:
+                flat_returns.append(float(ers[0]))
+            except Exception:
+                pass
+    if flat_returns:
+        mean_ret = sum(flat_returns) / len(flat_returns)
+        print(f"mean_return={mean_ret:.3f} over {len(flat_returns)} episodes")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+
+

examples/sft/export_dataset.py

@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+"""Export SFT JSONL from tracing_v3 sqlite using the shared exporter utilities.
+
+Thin wrapper over `examples/warming_up_to_rl/export_trace_sft.py` to keep the
+SFT workflow self-contained in this folder while reusing tested logic.
+"""
+
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+
+from examples.warming_up_to_rl.export_trace_sft import (
+    build_sft_dataset,
+    connect,
+    fetch_achievement_data,
+    fetch_event_reward_totals,
+    fetch_outcome_rewards,
+    fetch_session_models,
+    parse_event_filters,
+    write_jsonl,
+)
+
+
+def main() -> None:
+    p = argparse.ArgumentParser(description=__doc__)
+    p.add_argument("--db", type=Path, default=Path("traces/v3/synth_ai.db"))
+    p.add_argument("--output", type=Path, default=Path("examples/sft/ft_data/crafter_traces.jsonl"))
+    p.add_argument("--model", action="append", dest="models")
+    p.add_argument("--provider", action="append", dest="providers")
+    p.add_argument("--min-unique", type=int, default=0)
+    p.add_argument("--max-unique", type=int, default=None)
+    p.add_argument("--exclude-achievement", action="append", dest="exclude_achievements")
+    p.add_argument("--require-achievement", action="append", dest="required_achievements")
+    p.add_argument("--min-outcome-reward", type=float, default=None)
+    p.add_argument("--max-outcome-reward", type=float, default=None)
+    p.add_argument("--event-reward", action="append", dest="event_reward_filters")
+    p.add_argument("--limit", type=int, default=None)
+    args = p.parse_args()
+
+    conn = connect(args.db)
+    try:
+        achievements_map, unique_counts, name_counts, size_counts, session_uniques, session_final = (
+            fetch_achievement_data(conn)
+        )
+        session_models = fetch_session_models(conn)
+        outcome_data = fetch_outcome_rewards(conn)
+        event_totals = fetch_event_reward_totals(conn)
+        event_filters = parse_event_filters(args.event_reward_filters)
+
+        allowed_models = set(args.models) if args.models else None
+        allowed_providers = set(args.providers) if args.providers else None
+        required_achievements = set(args.required_achievements or [])
+        excluded_achievements = set(args.exclude_achievements or [])
+
+        eligible: set[str] = set()
+        for session_id, (model_name, provider, _calls) in session_models.items():
+            if allowed_models and model_name not in allowed_models:
+                continue
+            if allowed_providers and (provider or "unknown") not in allowed_providers:
+                continue
+
+            session_unique = session_uniques.get(session_id, set())
+            adjusted_uniques = {a for a in session_unique if a not in excluded_achievements}
+            unique_count = len(adjusted_uniques)
+            if args.min_unique is not None and unique_count < args.min_unique:
+                continue
+            if args.max_unique is not None and unique_count > args.max_unique:
+                continue
+
+            outcome = outcome_data.get(session_id)
+            total_reward = outcome["total_reward"] if outcome else 0.0
+            final_achievements = (
+                outcome["achievements"] if outcome else session_final.get(session_id, set())
+            )
+            if args.min_outcome_reward is not None and total_reward < args.min_outcome_reward:
+                continue
+            if args.max_outcome_reward is not None and total_reward > args.max_outcome_reward:
+                continue
+            if required_achievements and not required_achievements.issubset(final_achievements):
+                continue
+
+            totals = event_totals.get(session_id, {})
+            meets_filters = True
+            for reward_type, min_total in event_filters:
+                total = totals.get(reward_type, {}).get("total", 0.0)
+                if total < min_total:
+                    meets_filters = False
+                    break
+            if not meets_filters:
+                continue
+            eligible.add(session_id)
+
+        if not eligible:
+            raise SystemExit("No sessions matched the provided filters.")
+
+        dataset = build_sft_dataset(
+            conn,
+            achievements_map,
+            eligible,
+            allowed_models=allowed_models,
+            limit=args.limit,
+        )
+        if not dataset:
+            raise SystemExit("No rollout steps matched the filters (after session selection).")
+
+        Path(args.output).parent.mkdir(parents=True, exist_ok=True)
+        write_jsonl(args.output, dataset)
+        print(f"Wrote {len(dataset)} examples -> {args.output}")
+    finally:
+        conn.close()
+
+
+if __name__ == "__main__":
+    main()
+
+

examples/sft/generate_traces.py

@@ -0,0 +1,162 @@
+#!/usr/bin/env python3
+"""Generate Crafter rollouts and server-side traces via the Task App.
+
+This script is a slim wrapper around the Task App `/rollout` endpoint to
+produce trajectories while the server (if configured with TASKAPP_TRACING_ENABLED)
+persists traces to its sqlite database. Use `export_dataset.py` afterwards
+to build an SFT JSONL.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import os
+import sys
+import time
+from dataclasses import dataclass
+from typing import Any
+
+from synth_ai.task import (
+    RolloutEnvSpec,
+    RolloutPolicySpec,
+    RolloutRecordConfig,
+    RolloutRequest,
+    TaskAppClient,
+)
+
+
+def _ensure_str(val: Any, name: str) -> str:
+    s = str(val or "").strip()
+    if not s:
+        raise SystemExit(f"Missing required {name}")
+    return s
+
+
+def _build_ops(max_llm_calls: int) -> list[str]:
+    max_llm_calls = max(1, int(max_llm_calls or 1))
+    ops: list[str] = []
+    for _ in range(max_llm_calls):
+        ops.extend(["agent", "env"])  # LLM step then env step
+    return ops
+
+
+def _build_request(seed: int, run_id: str, model: str, inference_url: str, api_key: str, *, max_llm_calls: int, return_trace: bool) -> RolloutRequest:
+    policy_cfg: dict[str, Any] = {
+        "model": model,
+        "inference_url": inference_url,
+        "api_key": api_key,
+    }
+    record = RolloutRecordConfig(trajectories=True, return_trace=bool(return_trace), trace_format="compact")
+    return RolloutRequest(
+        run_id=run_id,
+        env=RolloutEnvSpec(env_name="crafter", seed=seed, config={}),
+        policy=RolloutPolicySpec(policy_name="crafter-react", config=policy_cfg),
+        ops=_build_ops(max_llm_calls),
+        record=record,
+    )
+
+
+@dataclass(slots=True)
+class Args:
+    base_url: str
+    api_key: str
+    inference_url: str
+    inference_api_key: str
+    model: str
+    episodes: int
+    start_seed: int
+    max_llm_calls: int
+    concurrency: int
+    return_trace: bool
+    timeout: float
+
+
+async def _run_one(client: TaskAppClient, run_id: str, seed: int, a: Args) -> dict[str, Any]:
+    req = _build_request(
+        seed=seed,
+        run_id=f"{run_id}-seed{seed}",
+        model=a.model,
+        inference_url=a.inference_url,
+        api_key=a.inference_api_key,
+        max_llm_calls=a.max_llm_calls,
+        return_trace=a.return_trace,
+    )
+    resp = await client.rollout(req)
+    metrics = resp.metrics.model_dump()
+    return {
+        "seed": seed,
+        "num_steps": metrics.get("num_steps"),
+        "episode_returns": metrics.get("episode_returns"),
+        "outcome_score": metrics.get("outcome_score"),
+    }
+
+
+async def _bounded_gather(n: int, coros: list[asyncio.Future]):
+    sem = asyncio.Semaphore(n)
+
+    async def _wrap(coro):
+        async with sem:
+            return await coro
+
+    return await asyncio.gather(*[_wrap(c) for c in coros])
+
+
+async def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--base-url", default=os.getenv("TASK_APP_URL", "http://localhost:8001"))
+    parser.add_argument("--api-key", default=os.getenv("ENVIRONMENT_API_KEY"))
+    parser.add_argument("--inference-url", default=os.getenv("INFERENCE_URL", "https://api.groq.com/openai"))
+    parser.add_argument("--inference-api-key", default=os.getenv("GROQ_API_KEY"))
+    parser.add_argument("--model", default=os.getenv("POLICY_MODEL", "llama-3.3-70b-versatile"))
+    parser.add_argument("--episodes", type=int, default=50)
+    parser.add_argument("--start-seed", type=int, default=0)
+    parser.add_argument("--max-llm-calls", type=int, default=10)
+    parser.add_argument("--concurrency", type=int, default=5)
+    parser.add_argument("--return-trace", action="store_true")
+    parser.add_argument("--timeout", type=float, default=60.0)
+    args_ns = parser.parse_args()
+
+    a = Args(
+        base_url=_ensure_str(args_ns.base_url, "--base-url"),
+        api_key=_ensure_str(args_ns.api_key, "--api-key"),
+        inference_url=_ensure_str(args_ns.inference_url, "--inference-url"),
+        inference_api_key=_ensure_str(args_ns.inference_api_key, "--inference-api-key"),
+        model=_ensure_str(args_ns.model, "--model"),
+        episodes=int(args_ns.episodes),
+        start_seed=int(args_ns.start_seed),
+        max_llm_calls=int(args_ns.max_llm_calls),
+        concurrency=max(1, int(args_ns.concurrency)),
+        return_trace=bool(args_ns.return_trace),
+        timeout=float(args_ns.timeout),
+    )
+
+    print(
+        f"[INFO] base={a.base_url} episodes={a.episodes} start_seed={a.start_seed} model={a.model} tp={a.max_llm_calls}"
+    )
+    run_id = f"traces-{int(time.time())}"
+
+    successes = 0
+    failures = 0
+    async with TaskAppClient(a.base_url, api_key=a.api_key, timeout=a.timeout) as client:
+        tasks = [
+            _run_one(client, run_id, seed, a) for seed in range(a.start_seed, a.start_seed + a.episodes)
+        ]
+        for result in await _bounded_gather(a.concurrency, tasks):
+            if isinstance(result, dict):
+                successes += 1
+                print(f"[OK] seed={result['seed']} return={result.get('episode_returns')}")
+            else:
+                failures += 1
+                print(f"[ERR] seed result not dict: {result}", file=sys.stderr)
+
+    print(f"[DONE] successes={successes} failures={failures}")
+
+
+if __name__ == "__main__":
+    try:
+        asyncio.run(main())
+    except KeyboardInterrupt:
+        print("Interrupted", file=sys.stderr)
+
+

examples/swe/__init__.py

@@ -0,0 +1,12 @@
+"""SWE task app examples package."""
+
+from importlib import resources as _resources
+
+__all__ = ["path_for"]
+
+
+def path_for(package: str, resource: str) -> str:
+    """Return path for packaged SWE example resources."""
+
+    with _resources.as_file(_resources.files(f"examples.swe.{package}") / resource) as path:
+        return str(path)