synth-ai 0.2.14__py3-none-any.whl → 0.2.16__py3-none-any.whl

examples/README.md

@@ -0,0 +1 @@
+### The instructions for how to create and configure a task app are documented at https://docs.usesynth.ai/sdk/task-apps

examples/multi_step/SFT_README.md

@@ -0,0 +1,147 @@
+# SFT Training for Qwen3-Coder-30B with LoRA
+
+Supervised Fine-Tuning configuration for the same 30B MoE model used in RL training.
+
+## Configuration Overview
+
+**Model:** `Qwen/Qwen3-Coder-30B-A3B-Instruct` (Mixture of Experts)
+
+**Hardware:** 4x H200 GPUs (561GB total VRAM)
+
+**Parallelism Strategy:**
+- **Tensor Parallel (TP)**: 2 GPUs - Splits the model across 2 GPUs for inference/forward pass
+- **Data Parallel (DP)**: 2 GPUs - Splits batches across 2 GPUs for training throughput
+
+**LoRA Configuration:**
+- Rank (r): 16
+- Alpha: 32
+- Dropout: 0.05
+- Target modules: `["all-linear"]` - Applies LoRA to all linear layers
+
+## Memory Breakdown per GPU
+
+With 4x H200 (141GB each):
+
+**Model Split (TP=2):**
+- 2 GPUs hold the base model (70GB each)
+- ~70GB free per GPU for activations and gradients
+
+**Training (DP=2):**
+- 2 GPUs process different batches
+- LoRA adapters: ~5-10GB per GPU
+- Gradients/optimizer states: ~20-30GB per GPU
+- **Total per training GPU: ~50-60GB** ✅
+
+## Quick Start
+
+### 1. Prepare Your Dataset
+
+Your dataset should be in JSONL format with conversation turns:
+
+```jsonl
+{"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}
+{"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}
+```
+
+### 2. Run Training
+
+```bash
+# Using the helper script
+./examples/multi_step/run_sft_qwen30b.sh path/to/your/dataset.jsonl
+
+# Or directly with synth-ai CLI
+uvx synth-ai train \
+  --type sft \
+  --config examples/multi_step/configs/crafter_sft_qwen30b_lora.toml \
+  --dataset path/to/your/dataset.jsonl \
+  --env-file backend/.env.dev
+```
+
+### 3. Monitor Training
+
+Check the Synth dashboard for:
+- Training loss curve
+- Validation metrics (if validation set provided)
+- GPU utilization
+- Training throughput (tokens/sec)
+
+## Hyperparameters
+
+**Batch Configuration:**
+- Per-device batch size: 1
+- Gradient accumulation: 64 steps
+- **Effective global batch size: 128** (1 × 64 × 2 GPUs)
+
+**Learning Rate:**
+- Initial LR: 5e-6
+- Warmup ratio: 3%
+- Schedule: Linear decay
+
+**Sequence Length:** 4096 tokens
+
+**Training:**
+- Epochs: 1
+- Mixed precision: BF16
+- DeepSpeed: Stage 2 (optimizer state sharding)
+- Activation checkpointing: Enabled
+
+## Configuration File Structure
+
+```toml
+[algorithm]
+type = "offline"        # Supervised (not RL)
+method = "sft"          # Supervised fine-tuning
+variety = "lora"        # Using LoRA adapters
+
+[compute]
+gpu_type = "H200"
+gpu_count = 4
+
+[data.topology]
+tensor_parallel = 2     # Split model across 2 GPUs
+data_parallel = 2       # Split batches across 2 GPUs
+
+[training]
+mode = "lora"
+use_qlora = true        # Quantized LoRA (4-bit base model)
+
+[lora]
+r = 16                  # LoRA rank
+alpha = 32              # LoRA scaling
+dropout = 0.05
+target_modules = ["all-linear"]  # Apply to all linear layers
+```
+
+## Comparison with RL Config
+
+| Aspect | SFT | RL |
+|--------|-----|-----|
+| Purpose | Supervised learning | Reinforcement learning |
+| Data | Labeled examples | Environment interactions |
+| Topology | TP=2, DP=2 | Split: 2 inference + 2 training |
+| Batch size | 128 (effective) | Variable (episode-based) |
+| Training | Standard backprop | Policy gradient (GSPO) |
+
+## Tips
+
+1. **Start Small:** Test with a small dataset first to verify the pipeline
+2. **Validation:** Add a validation set to monitor overfitting
+3. **Checkpointing:** Training saves checkpoints every 100 steps
+4. **Resume:** Can resume from checkpoint if training is interrupted
+5. **Inference:** After training, use the LoRA adapter with the base model
+
+## Output
+
+After training completes, you'll get:
+- LoRA adapter weights (saved to volume)
+- Training metrics and logs
+- Best checkpoint (based on validation loss)
+- Model ready for inference or RL initialization
+
+## Next Steps
+
+1. **Evaluate:** Test your fine-tuned model on held-out data
+2. **RL Training:** Use this as initialization for RL (`init_from_sft = true`)
+3. **Deploy:** Load LoRA adapter for inference
+4. **Iterate:** Adjust hyperparameters based on performance
+

examples/multi_step/configs/crafter_rl_stepwise_hosted_judge.toml

@@ -16,24 +16,24 @@ judge_url = "https://synth-backend-dev-docker.onrender.com/api"
 
 [compute]
 gpu_type = "H200"
-gpu_count = 2
+gpu_count = 4
 
 [topology]
 type = "single_node_split"
-gpus_for_vllm = 1
-gpus_for_training = 1
+gpus_for_vllm = 2
+gpus_for_training = 2
 gpus_for_ref = 0
-tensor_parallel = 1
+tensor_parallel = 2
 
 [vllm]
-tensor_parallel_size = 1
-max_model_len = 8192
+tensor_parallel_size = 2
+max_model_len = 4096
 
 [reference]
 placement = "none"
 
 [model]
-base = "Qwen/Qwen3-4B"
+base = "Qwen/Qwen3-Coder-30B-A3B-Instruct"
 trainer_mode = "lora"
 label = "crafter-rl-stepwise-hosted-judge"
 
@@ -74,7 +74,7 @@ seeds = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
 [training]
 num_epochs = 1
-iterations_per_epoch = 20
+iterations_per_epoch = 5
 gradient_accumulation_steps = 1
 max_accumulated_minibatch = 1
 max_turns = 10
@@ -84,7 +84,7 @@ learning_rate = 5e-5
 log_interval = 1
 weight_sync_interval = 1
 event_rewards_kind = "unique"
-async_semaphore_max = 40  # Max concurrent rollouts in streaming pipeline
+async_semaphore_max = 4  # Max concurrent rollouts in streaming pipeline
 
 # Enable dense decision rewards in the trainer to mirror env_config step rewards.
 step_rewards_enabled = true

examples/multi_step/configs/crafter_sft_qwen30b_lora.toml

@@ -0,0 +1,62 @@
+# Crafter SFT LoRA configuration
+# Train Qwen3-Coder-30B on Crafter agent traces
+
+[algorithm]
+type = "offline"
+method = "sft"
+variety = "lora"
+
+[job]
+model = "Qwen/Qwen3-Coder-30B-A3B-Instruct"
+# Default dataset - can override with --dataset flag
+data = "traces/crafter_sft_converted.jsonl"
+
+[compute]
+gpu_type = "H200"
+gpu_count = 2
+nodes = 1
+
+[data]
+# Forwarded into metadata.effective_config
+topology = {}
+# Optional validation set if you have one locally
+# validation_path = "examples/multi_step/ft_data/crafter_sft.val.jsonl"
+
+[training]
+mode = "lora"
+use_qlora = true
+
+[training.validation]
+enabled = true
+evaluation_strategy = "steps"
+eval_steps = 100
+save_best_model_at_end = true
+metric_for_best_model = "val.loss"
+greater_is_better = false
+
+[hyperparameters]
+n_epochs = 1
+train_kind = "peft"
+per_device_batch = 1
+gradient_accumulation_steps = 64
+sequence_length = 4096
+learning_rate = 5e-6
+warmup_ratio = 0.03
+lora_rank = 16
+lora_alpha = 32
+lora_dropout = 0.05
+lora_target_modules = ["all-linear"]
+
+[hyperparameters.parallelism]
+use_deepspeed = true
+deepspeed_stage = 2
+fsdp = false
+bf16 = true
+fp16 = false
+activation_checkpointing = true
+
+[tags]
+experiment = "crafter_sft_lora_qwen_coder_30b"
+task = "crafter_agent"
+model_size = "30b"
+

examples/multi_step/convert_traces_to_sft.py

@@ -0,0 +1,84 @@
+#!/usr/bin/env python3
+"""Convert Crafter trace format to SFT format with messages[] structure."""
+
+import json
+import sys
+from pathlib import Path
+
+def convert_trace_to_sft(trace: dict) -> dict:
+    """Convert a single trace to SFT format."""
+    # Extract dialogue from trace
+    dialogue = trace.get("dialogue", [])
+    assistant = trace.get("assistant", {})
+    
+    # Build messages list
+    messages = []
+    
+    # Add dialogue history
+    for msg in dialogue:
+        messages.append({
+            "role": msg["role"],
+            "content": msg["content"]
+        })
+    
+    # Add assistant response if present
+    if assistant:
+        content = assistant.get("content", "")
+        tool_calls = assistant.get("tool_calls", [])
+        
+        # If there are tool calls, format them
+        if tool_calls:
+            # Convert tool calls to a simple text format for SFT
+            tool_text = "\n".join([
+                f"Tool: {tc['name']}\nArguments: {json.dumps(tc.get('arguments', {}))}"
+                for tc in tool_calls
+            ])
+            content = f"{content}\n\n{tool_text}".strip()
+        
+        messages.append({
+            "role": "assistant",
+            "content": content
+        })
+    
+    return {"messages": messages}
+
+def main():
+    if len(sys.argv) < 2:
+        print("Usage: python convert_traces_to_sft.py <input.jsonl> [output.jsonl]")
+        sys.exit(1)
+    
+    input_path = Path(sys.argv[1])
+    output_path = Path(sys.argv[2]) if len(sys.argv) > 2 else input_path.with_name(f"{input_path.stem}_sft_format.jsonl")
+    
+    if not input_path.exists():
+        print(f"Error: Input file not found: {input_path}")
+        sys.exit(1)
+    
+    print(f"Converting {input_path} → {output_path}")
+    
+    converted = 0
+    skipped = 0
+    
+    with open(input_path) as f_in, open(output_path, "w") as f_out:
+        for line_no, line in enumerate(f_in, 1):
+            try:
+                trace = json.loads(line.strip())
+                sft_entry = convert_trace_to_sft(trace)
+                
+                # Only write if we have messages
+                if sft_entry["messages"]:
+                    f_out.write(json.dumps(sft_entry) + "\n")
+                    converted += 1
+                else:
+                    skipped += 1
+                    
+            except Exception as e:
+                print(f"Warning: Skipping line {line_no}: {e}")
+                skipped += 1
+    
+    print(f"✅ Converted {converted} entries, skipped {skipped}")
+    print(f"Output: {output_path}")
+
+if __name__ == "__main__":
+    main()
+

examples/multi_step/run_sft_qwen30b.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+# Run SFT for Qwen3-Coder-30B with LoRA on Crafter data
+
+# Usage:
+#   ./run_sft_qwen30b.sh <dataset_path> [env_file]
+#
+# Example:
+#   ./run_sft_qwen30b.sh examples/multi_step/ft_data/crafter_traces.jsonl
+#   ./run_sft_qwen30b.sh examples/multi_step/ft_data/crafter_traces.jsonl backend/.env.dev
+
+set -e
+
+DATASET_PATH="${1:-examples/sft/ft_data/crafter_traces.jsonl}"
+ENV_FILE="${2:-backend/.env.dev}"
+
+if [ ! -f "$DATASET_PATH" ]; then
+    echo "Error: Dataset not found at $DATASET_PATH"
+    echo "Usage: $0 <dataset_path> [env_file]"
+    exit 1
+fi
+
+if [ ! -f "$ENV_FILE" ]; then
+    echo "Error: Env file not found at $ENV_FILE"
+    echo "Usage: $0 <dataset_path> [env_file]"
+    exit 1
+fi
+
+echo "🚀 Starting SFT training for Qwen3-Coder-30B with LoRA"
+echo "   Model: Qwen/Qwen3-Coder-30B-A3B-Instruct"
+echo "   Dataset: $DATASET_PATH"
+echo "   Config: examples/multi_step/configs/crafter_sft_qwen30b_lora.toml"
+echo "   GPUs: 4x H200"
+echo "   LoRA: r=16, alpha=32, all-linear"
+echo ""
+
+uvx synth-ai train \
+  --type sft \
+  --config examples/multi_step/configs/crafter_sft_qwen30b_lora.toml \
+  --dataset "$DATASET_PATH" \
+  --env-file "$ENV_FILE"
+
+echo ""
+echo "✅ SFT training job submitted!"
+echo "   Monitor progress in your Synth dashboard"
+

examples/qwen_coder/configs/coder_lora_30b.toml

@@ -1,5 +1,7 @@
 # Qwen3 Coder 30B LoRA SFT – all-linear adapters
 
+type = "sft"
+
 [algorithm]
 type = "offline"
 method = "sft"
@@ -58,4 +60,3 @@ alpha = 32
 dropout = 0.05
 target_modules = ["all-linear"]
 
-

examples/qwen_coder/configs/coder_lora_4b.toml

@@ -1,5 +1,7 @@
 # Qwen3 Coder 4B LoRA SFT – all-linear adapters
 
+type = "sft"
+
 [job]
 model = "Qwen/Qwen3-4B"
 
@@ -54,4 +56,3 @@ dropout = 0.05
 target_modules = ["all-linear"]
 
 
-

examples/qwen_coder/configs/coder_lora_small.toml

@@ -1,5 +1,7 @@
 # Qwen3 Coder LoRA SFT – all-linear adapters
 
+type = "sft"
+
 [algorithm]
 type = "offline"
 method = "sft"
@@ -55,4 +57,3 @@ alpha = 32
 dropout = 0.05
 target_modules = ["all-linear"]
 
-

examples/qwen_vl/BUGS_AND_FIXES.md

@@ -0,0 +1,232 @@
+# Vision SFT Pipeline - Bugs and Fixes
+
+Complete log of issues encountered and resolved during vision data collection setup.
+
+## ✅ Issue #1: Import Error - CrafterEnvironment
+
+**Problem:**
+```python
+ImportError: cannot import name 'CrafterEnvironment' from 'examples.task_apps.crafter.task_app.synth_envs_hosted.envs.crafter.environment'
+```
+
+**Root Cause:**  
+Class is named `CrafterEnvironmentWrapper`, not `CrafterEnvironment`
+
+**Fix:**  
+Updated imports and usages in:
+- `crafter_gpt5nano_agent.py`
+- `crafter_qwen_vl_agent.py`
+- `collect_vision_traces.py`
+
+```python
+# Before
+from ...environment import CrafterEnvironment
+wrapper = CrafterEnvironment(env, seed=seed)
+
+# After
+from ...environment import CrafterEnvironmentWrapper
+wrapper = CrafterEnvironmentWrapper(env, seed=seed)
+```
+
+**Status:** FIXED ✓
+
+---
+
+## ✅ Issue #2: OpenAI API Parameter - max_tokens
+
+**Problem:**
+```
+openai.BadRequestError: Error code: 400 - {'error': {'message': "Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead."}}
+```
+
+**Root Cause:**  
+gpt-5 models require `max_completion_tokens` parameter instead of `max_tokens`
+
+**Fix:**  
+Updated `_normalise_openai_request()` function to detect gpt-5 models:
+
+```python
+def _normalise_openai_request(payload, model, temperature):
+    request = dict(payload)
+    request["model"] = model
+    
+    # gpt-5 models use max_completion_tokens, not max_tokens
+    if "gpt-5" in model.lower():
+        request.setdefault("max_completion_tokens", 512)
+        request.pop("max_tokens", None)  # Remove if present
+    else:
+        # Older models use max_tokens
+        request.setdefault("max_tokens", 512)
+    
+    return request
+```
+
+**Files Updated:**
+- `crafter_gpt5nano_agent.py`
+- `collect_vision_traces.py`
+
+**Status:** FIXED ✓
+
+---
+
+## ✅ Issue #3: OpenAI API Parameter - temperature
+
+**Problem:**
+```
+openai.BadRequestError: Error code: 400 - {'error': {'message': "Unsupported value: 'temperature' does not support 0.6 with this model. Only the default (1) value is supported."}}
+```
+
+**Root Cause:**  
+gpt-5-nano only supports `temperature=1` (default), custom temperature values are not allowed
+
+**Fix:**  
+Remove temperature parameter for gpt-5 models:
+
+```python
+def _normalise_openai_request(payload, model, temperature):
+    # ...
+    
+    if "gpt-5" in model.lower():
+        # gpt-5-nano only supports temperature=1 (default)
+        request.pop("temperature", None)  # Remove custom temperature
+        request.setdefault("max_completion_tokens", 512)
+        request.pop("max_tokens", None)
+    else:
+        # Older models support custom temperature
+        request.setdefault("temperature", temperature)
+        request.setdefault("max_tokens", 512)
+    
+    return request
+```
+
+**Files Updated:**
+- `crafter_gpt5nano_agent.py`
+- `collect_vision_traces.py`
+
+**Status:** FIXED ✓
+
+---
+
+## ⚠️  Issue #4: gpt-5-nano Tool Calling Support
+
+**Problem:**
+```
+Seed 0: no tool calls returned by model; ending episode early at step 0.
+```
+
+**Root Cause:**  
+gpt-5-nano does not appear to support function/tool calling yet, or requires a different prompt format for tool use.
+
+**Testing Results:**
+- API returned 200 OK (auth and network fine)
+- Model processed vision inputs successfully
+- Model did not return tool calls even with tools schema provided
+- Both episodes stopped immediately (step 0)
+
+**Workaround:**  
+Switch to `gpt-4o-mini-2024-07-18` for data collection:
+- Confirmed to support both vision AND tool calling
+- Successfully completed 10 episodes with good quality
+- Mean 2.6 achievements per episode
+- 685 total tool calls across 10 episodes
+
+**Status:** WORKAROUND APPLIED (use gpt-4o-mini) ✓
+
+**Note:**  
+This is a model capability limitation, not a code bug. gpt-5-nano can be revisited when tool calling support is confirmed by OpenAI.
+
+---
+
+## 📊 Final Validation Results
+
+### Test Run #5: 10-Episode Collection with gpt-4o-mini
+
+**Command:**
+```bash
+uv run python examples/qwen_vl/crafter_gpt5nano_agent.py \
+  --model gpt-4o-mini-2024-07-18 \
+  --seeds 10 \
+  --steps 50
+```
+
+**Results:**
+```
+✓ All 10 episodes completed (50 steps each)
+✓ Mean achievements: 2.6 per episode
+✓ Total tool calls: 685
+✓ Vision processing: Working (64x64 PNG frames)
+✓ Tool calling: Working (proper tool call format)
+✓ Frame saving: Working (saved to output directory)
+✓ Performance: ~5-6 minutes for 10 episodes
+```
+
+**Quality Metrics:**
+- Episode 1: 4 achievements, 72 tool calls, reward: 97.3
+- Episode 5: 3 achievements, 62 tool calls, reward: 120.0
+- Episode 8: 1 achievement, 71 tool calls, reward: 12.9
+- Good variety in performance (1-4 achievements)
+
+---
+
+## 🔧 Code Changes Summary
+
+### Files Modified:
+1. **crafter_gpt5nano_agent.py**
+   - Import: `CrafterEnvironment` → `CrafterEnvironmentWrapper`
+   - Function: `_normalise_openai_request()` - handle gpt-5 parameters
+
+2. **crafter_qwen_vl_agent.py**
+   - Import: `CrafterEnvironment` → `CrafterEnvironmentWrapper`
+
+3. **collect_vision_traces.py**
+   - Import: `CrafterEnvironment` → `CrafterEnvironmentWrapper`
+   - Function: `_normalise_openai_request()` - handle gpt-5 parameters
+
+### Key Learnings:
+1. ✅ Always check actual class names in source code
+2. ✅ OpenAI's API evolves - newer models have different parameter requirements
+3. ✅ Test with known-working models first (gpt-4o-mini) before trying cutting-edge ones
+4. ✅ Vision + tool calling combo requires mature model support
+
+---
+
+## 🎯 Recommendations
+
+### For Production:
+- **Teacher model:** Use `gpt-4o-mini-2024-07-18` for data collection
+  - Proven to work with vision + tools
+  - Good quality (2-4 achievements per episode)
+  - Reasonable cost
+
+- **Monitor gpt-5-nano:** Revisit when tool calling support is confirmed
+
+### For Configs:
+- Update eval configs to use `gpt-4o-mini` by default:
+  ```toml
+  [eval]
+  model = "gpt-4o-mini-2024-07-18"  # Not gpt-5-nano
+  ```
+
+---
+
+## ✅ All Issues Resolved
+
+**Infrastructure Status:** READY FOR PRODUCTION ✓
+
+- Vision processing: Working
+- Tool calling: Working  
+- Frame saving: Working
+- OpenAI API integration: Working
+- 10-episode test: Successful
+
+**Next Steps:**
+1. Scale to 100 episodes for full dataset
+2. Apply filters and export to SFT format
+3. Train VLM with LoRA
+4. Fine-tune with RL
+
+---
+
+**Last Updated:** 2025-10-26  
+**Test Environment:** synth-ai dev, macOS, Python 3.11
+