synth-ai 0.2.9.dev0__py3-none-any.whl → 0.2.23.dev3__py3-none-any.whl

examples/multi_step/verilog_rl_lora.md

@@ -0,0 +1,218 @@
+# Verilog RL with LoRA Analysis
+
+## Executive Summary
+
+**✅ YES, Verilog can absolutely do RL with LoRA just like Crafter!** The architecture is nearly identical, but there are important considerations around model size and task complexity.
+
+## Architecture Compatibility ✅
+
+### **Same Foundation** (No changes needed)
+- ✅ **Contracts**: Uses identical `RolloutRequest`/`RolloutResponse` as Crafter
+- ✅ **Task App Framework**: Same `synth_ai.task.apps` framework
+- ✅ **Environment Pattern**: Same `StatefulEnvironment` + tool-based architecture
+- ✅ **Rubrics System**: Same evaluation and reward system
+- ✅ **Trace Correlation**: Already implemented in `rollout_executor` (line 817 in `grpo_verilog.py`)
+- ✅ **Modal Deployment**: Same deployment pattern as Crafter
+
+### **Key Differences** (Considerations for LoRA)
+
+#### 1. **Model Size: 8x Larger** ⚠️
+```toml
+# Verilog (current)
+model = "qwen/qwen3-32b"  # 32B parameters
+
+# Crafter (working)
+model = "Qwen/Qwen3-4B"   # 4B parameters
+```
+**Impact**: Memory requirements 8x higher for LoRA training
+**Solution**: Use gradient checkpointing, smaller batch sizes, or distributed training
+
+#### 2. **Tool Set: Simpler but More Structured**
+```python
+# Verilog Tools (4 tools)
+TOOLS = ["write_file", "compile", "simulate", "submit"]
+
+# Crafter Tools (20+ tools)
+# craft, move, attack, gather, etc.
+```
+
+**Verilog Advantages**:
+- ✅ **Deterministic**: Write → Compile → Simulate → Submit workflow
+- ✅ **Clear Success Criteria**: Tests pass = high reward
+- ✅ **Sparse but Meaningful Rewards**: +10 for submit success, +1 for simulation pass
+
+**Verilog Challenges**:
+- ❌ **Sparser Rewards**: Fewer intermediate signals for learning
+- ❌ **Longer Sequences**: Multi-step compilation chains
+- ❌ **Error Recovery**: Must debug compilation failures
+
+#### 3. **State Representation**
+```python
+# Verilog State (file-based)
+{
+    "files": {"TopModule.v": "module TopModule(..."},
+    "compile_status": "Last compile: Success",
+    "simulate_status": "Last simulation: Passed",
+    "task_completed": false
+}
+
+# Crafter State (world-based)
+{
+    "inventory": {"wood": 5, "stone": 3},
+    "position": [x, y],
+    "nearby_entities": [...],
+    "achievement_unlocked": true
+}
+```
+
+## Configuration for LoRA RL
+
+### **Option 1: Qwen3-0.6B (Recommended for testing)** ⭐
+```toml
+[algorithm]
+type = "online"
+method = "policy_gradient"
+variety = "gspo"
+
+[model]
+base = "Qwen/Qwen3-0.6B"  # ✅ Same as existing SFT configs
+trainer_mode = "lora"
+
+[lora]
+r = 16
+alpha = 32
+dropout = 0.05
+target_modules = ["all-linear"]
+
+[rollout]
+env_name = "verilog"
+max_turns = 15
+policy_name = "verilog-designer"
+
+[training]
+batch_size = 4  # ✅ Same as Crafter
+gradient_accumulation_steps = 1
+```
+
+### **Option 2: Qwen3-32B (Production)** ⚠️
+```toml
+[algorithm]
+type = "online"
+method = "policy_gradient"
+variety = "gspo"
+
+[model]
+base = "qwen/qwen3-32b"  # ⚠️ 8x memory vs Crafter's 4B
+trainer_mode = "lora"
+
+[lora]
+r = 16
+alpha = 32
+dropout = 0.05
+target_modules = ["all-linear"]
+
+[rollout]
+env_name = "verilog"
+max_turns = 15
+policy_name = "verilog-designer"
+```
+
+### **Memory Optimization** (for 32B model)
+```toml
+[vllm]
+max_model_len = 4096  # Shorter than Crafter's 8192
+tensor_parallel_size = 2  # Distribute across GPUs
+
+[training]
+batch_size = 2  # Smaller than Crafter's 4
+gradient_accumulation_steps = 4
+```
+
+## Task App Changes Needed
+
+### **1. Mode Parameter Support** ✅ (Already implemented)
+The Verilog task app already handles `mode="rl"` correctly:
+```python
+# In grpo_verilog.py rollout_executor
+policy_config = dict(policy_config_raw)
+# ... mode parameter flows through naturally
+```
+
+### **2. Trace Correlation** ✅ (Already implemented)
+```python
+# Line 817 in grpo_verilog.py
+trajectory = RolloutTrajectory(
+    # ...
+    inference_url=agent.inference_url,  # ✅ Required for trace correlation
+    decision_samples=None,
+)
+```
+
+### **3. Rubric Integration** ✅ (Already configured)
+```python
+# In grpo_verilog.py
+rubrics=RubricBundle(
+    outcome=OUTCOME_RUBRIC,  # Tests pass reward
+    events=EVENTS_RUBRIC,    # Process efficiency reward
+)
+```
+
+## RL Training Feasibility
+
+### **✅ Works Great**
+1. **Clear Success Signal**: Submit passing tests = +10 reward
+2. **Guided Process**: Natural write→compile→simulate→submit progression
+3. **Error Learning**: Agent must learn to debug compilation failures
+4. **Hardware Design**: Real-world applicable skills
+
+### **⚠️ Challenges**
+1. **Model Size**: 32B vs 4B = 8x memory, slower training
+2. **Sparse Rewards**: Fewer learning signals than Crafter's dense rewards
+3. **Longer Episodes**: 15+ steps vs Crafter's 10 steps
+4. **Compilation Errors**: Must learn to interpret and fix syntax errors
+
+## Recommended Approach
+
+### **Phase 1: Start with Qwen3-0.6B** ⭐ (as you requested)
+```toml
+# Perfect for testing - same model used in existing SFT configs
+model = "Qwen/Qwen3-0.6B"
+batch_size = 4  # Same as Crafter
+```
+- ✅ **Zero setup**: Already configured in `synth-ai/examples/sft/configs/crafter_lora_qwen0p6b.toml`
+- ✅ **Fast iteration**: 0.6B parameters = quick training cycles
+- ✅ **Memory efficient**: Fits on single GPU easily
+- ✅ **Proven baseline**: Same model used in RL demos and SFT examples
+
+### **Phase 2: Scale to Qwen3-8B** (if 0.6B works well)
+```toml
+model = "qwen/qwen3-8b"
+batch_size = 2
+gradient_accumulation_steps = 2
+```
+
+### **Phase 3: Production with Qwen3-32B**
+```toml
+model = "qwen/qwen3-32b"
+tensor_parallel_size = 2
+batch_size = 1
+gradient_accumulation_steps = 4
+```
+
+### **Phase 3: Optimize for Verilog Domain**
+Consider fine-tuning the base model on:
+- Verilog syntax and semantics
+- Hardware design patterns
+- Compilation error messages
+- Testbench writing
+
+## Conclusion
+
+**✅ Verilog RL with LoRA is absolutely feasible** and should work with the same pipeline as Crafter. The main differences are:
+
+1. **Larger model** (32B vs 4B) requires memory optimization
+2. **Sparser rewards** may need different reward shaping
+3. **More structured tasks** could actually make learning easier
+4. **Real hardware skills** make it more valuable than game tasks
+
+**Recommended next step**: Create a `verilog_rl_lora.toml` config starting with Qwen3-8B and adapt the reward rubrics for the compilation workflow.

examples/qwen_coder/README.md

@@ -0,0 +1,102 @@
+Qwen3 Coder – SFT with LoRA (all linear)
+
+This example mirrors the SFT LoRA flow under `examples/sft/` but targets the smallest Qwen3 Coder family model supported downstream. It configures LoRA on all linear projections ("all-linear") to match our RL LoRA recipes.
+
+Quick start
+
+1) Generate a tiny synthetic dataset (or export your own)
+
+```
+uv run python examples/qwen_coder/generate_dataset.py \
+  --output examples/qwen_coder/ft_data/coder_sft.small.jsonl \
+  --n 50 --seed 42 --lang python
+```
+
+2) Run training via the CLI:
+
+```
+uvx synth-ai train \
+  --type sft \
+  --config examples/qwen_coder/configs/coder_lora_small.toml \
+  --dataset examples/qwen_coder/ft_data/coder_sft.small.jsonl \
+  --env-file /path/to/.env
+```
+
+3) Inference via Synth API (pre/post SFT)
+
+Use the SDK’s OpenAI-compatible chat client routed through Synth. Export your env with SYNTH_API_KEY (and optional BACKEND_BASE_URL) or pass an env file to CLI helpers.
+
+Minimal one-shot inference:
+
+```bash
+python - <<'PY'
+import os, asyncio
+from synth_ai.v0.lm.core import main_v3 as lm
+
+async def run():
+    model = os.getenv("MODEL", "Qwen/Qwen3-Coder-30B-A3B-Instruct")
+    resp = await lm.chat_async(
+        model,
+        messages=[{"role":"user","content":"Write a Python function to reverse a string."}],
+        max_tokens=128,
+        temperature=0.2,
+    )
+    print(resp["choices"][0]["message"]["content"]) 
+asyncio.run(run())
+PY
+```
+
+After training, set `MODEL=ft:...` to query the finetuned adapter.
+
+4) 30B LoRA variant
+
+```bash
+uvx synth-ai train \
+  --type sft \
+  --config examples/qwen_coder/configs/coder_lora_30b.toml \
+  --dataset examples/qwen_coder/ft_data/coder_sft.small.jsonl \
+  --env-file /path/to/.env
+```
+
+5) Faster iteration: 4B LoRA config
+
+```bash
+uvx synth-ai train \
+  --type sft \
+  --config examples/qwen_coder/configs/coder_lora_4b.toml \
+  --dataset examples/qwen_coder/ft_data/coder_sft.small.jsonl \
+  --env-file /path/to/.env
+```
+
+Environment variables
+
+- `SYNTH_API_KEY`: required for training/inference through the hosted backend
+- `BACKEND_BASE_URL`: defaults to `https://agent-learning.onrender.com/api`
+
+Post‑SFT smoke
+
+- The training helper `sft_lora_30b.py` writes the resulting `ft:<id>` to `examples/qwen_coder/ft_data/ft_model_id.txt`.
+- Validate inference with your finetuned adapter:
+
+```bash
+uv run python examples/qwen_coder/infer_ft_smoke.py
+```
+
+Dataset utilities
+
+- `examples/qwen_coder/validate_jsonl.py`: sanity‑check first N lines for chat structure
+- `examples/qwen_coder/subset_jsonl.py`: create a capped subset for quick tests
+
+Optional CLI wrappers
+
+- `examples/qwen_coder/scripts/train_coder_30b.sh [/path/to/.env]`
+- `examples/qwen_coder/scripts/infer_coder.sh [/path/to/.env]`
+
+Notes
+
+- LoRA is enabled with `training.mode = "lora"` and `hyperparameters.train_kind = "peft"`.
+- The config sets an `all-linear` target to apply adapters broadly across attention and MLP projections.
+- Adjust `gradient_accumulation_steps`, `per_device_batch`, and `sequence_length` based on available GPU memory.
+- Use the Synth API client (above) for inference to ensure requests route via the hosted backend.
+
+

examples/qwen_coder/_shared.py

@@ -0,0 +1,113 @@
+#!/usr/bin/env python3
+"""Shared helpers for Qwen coder SFT examples."""
+
+from __future__ import annotations
+
+import json
+import os
+from pathlib import Path
+
+TRAIN_DATA_PATH = Path("examples/qwen_coder/ft_data/coder_sft.small.jsonl")
+VAL_DATA_PATH = Path("examples/qwen_coder/ft_data/coder_sft.small.val.jsonl")
+DATA_DIR = TRAIN_DATA_PATH.parent
+
+_FALLBACK_RECORDS: list[dict[str, object]] = [
+    {
+        "messages": [
+            {"role": "user", "content": "Write a Python function `add(a, b)` that returns the sum of two numbers."}
+        ],
+        "response": "def add(a, b):\n    return a + b\n",
+    },
+    {
+        "messages": [
+            {
+                "role": "user",
+                "content": "Implement a Python function `reverse_string(s)` that returns the reversed string.",
+            }
+        ],
+        "response": "def reverse_string(s: str) -> str:\n    return s[::-1]\n",
+    },
+    {
+        "messages": [
+            {
+                "role": "user",
+                "content": "Write a Python function `count_words(text)` returning a dict mapping words to counts.",
+            }
+        ],
+        "response": "from collections import Counter\n\ndef count_words(text: str) -> dict[str, int]:\n    words = [w for w in text.split() if w]\n    return dict(Counter(words))\n",
+    },
+]
+
+
+def ensure_tiny_dataset() -> Path:
+    """Ensure the tiny coder dataset exists, generating or writing a fallback if needed."""
+    if TRAIN_DATA_PATH.exists():
+        return TRAIN_DATA_PATH
+
+    try:
+        from examples.qwen_coder.generate_dataset import main as gen_main  # type: ignore
+
+        gen_main()
+        if TRAIN_DATA_PATH.exists():
+            return TRAIN_DATA_PATH
+    except Exception:
+        # Fall back to inline dataset below.
+        pass
+
+    DATA_DIR.mkdir(parents=True, exist_ok=True)
+    with TRAIN_DATA_PATH.open("w", encoding="utf-8") as fh:
+        for record in _FALLBACK_RECORDS:
+            fh.write(json.dumps(record, separators=(",", ":")))
+            fh.write("\n")
+    return TRAIN_DATA_PATH
+
+
+def optional_validation_dataset() -> Path | None:
+    """Return validation dataset path if present."""
+    if VAL_DATA_PATH.exists():
+        return VAL_DATA_PATH
+    return None
+
+
+def _ensure_parent(path: Path) -> Path:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    return path
+
+
+def resolve_output_path(default_filename: str) -> Path:
+    """Resolve output path for storing fine-tuned model ids."""
+    override = os.getenv("QWEN_CODER_FT_OUTPUT")
+    if override:
+        return _ensure_parent(Path(override).expanduser())
+    return _ensure_parent(DATA_DIR / default_filename)
+
+
+def resolve_model_id_path(default_filename: str) -> Path:
+    """Resolve path to read a stored fine-tuned model id."""
+    override = os.getenv("QWEN_CODER_FT_MODEL_PATH")
+    if override:
+        candidate = Path(override).expanduser()
+        if candidate.is_dir():
+            return candidate / default_filename
+        return candidate
+    return DATA_DIR / default_filename
+
+
+def resolve_infer_output_path(default_filename: str) -> Path:
+    """Resolve path for writing inference outputs."""
+    override = os.getenv("QWEN_CODER_FT_INFER_OUTPUT")
+    if override:
+        return _ensure_parent(Path(override).expanduser())
+    return _ensure_parent(DATA_DIR / default_filename)
+
+
+__all__ = [
+    "DATA_DIR",
+    "TRAIN_DATA_PATH",
+    "VAL_DATA_PATH",
+    "ensure_tiny_dataset",
+    "optional_validation_dataset",
+    "resolve_output_path",
+    "resolve_model_id_path",
+    "resolve_infer_output_path",
+]

examples/qwen_coder/configs/coder_lora_30b.toml

@@ -0,0 +1,60 @@
+# Qwen3 Coder 30B LoRA SFT – all-linear adapters
+
+[algorithm]
+type = "offline"
+method = "sft"
+variety = "qlora"
+
+[job]
+model = "Qwen/Qwen3-Coder-30B-A3B-Instruct"
+
+[compute]
+gpu_type = "H200"
+gpu_count = 1
+nodes = 1
+
+[data]
+topology = {}
+# Optional validation set
+# validation_path = "examples/qwen_coder/ft_data/coder_sft.small.val.jsonl"
+
+[metadata]
+# Effective config hints consumed by the backend
+effective_config = { compute = { gpu_type = "H200", gpu_count = 1, nodes = 1 } }
+
+[training]
+mode = "lora"
+use_qlora = true
+
+[training.validation]
+enabled = true
+evaluation_strategy = "steps"
+eval_steps = 100
+save_best_model_at_end = true
+metric_for_best_model = "val.loss"
+greater_is_better = false
+
+[hyperparameters]
+n_epochs = 1
+train_kind = "peft"
+per_device_batch = 1
+gradient_accumulation_steps = 64
+sequence_length = 4096
+learning_rate = 5e-6
+warmup_ratio = 0.03
+
+[hyperparameters.parallelism]
+use_deepspeed = true
+deepspeed_stage = 2
+fsdp = false
+bf16 = true
+fp16 = false
+activation_checkpointing = true
+
+# LoRA target selection for coder models: apply to all linear projections
+[lora]
+r = 16
+alpha = 32
+dropout = 0.05
+target_modules = ["all-linear"]
+

examples/qwen_coder/configs/coder_lora_4b.toml

@@ -0,0 +1,61 @@
+# Qwen3 Coder 4B LoRA SFT – all-linear adapters
+
+[algorithm]
+type = "offline"
+method = "sft"
+variety = "qlora"
+
+[job]
+model = "Qwen/Qwen3-4B"
+
+[compute]
+gpu_type = "H100"
+gpu_count = 1
+nodes = 1
+
+[data]
+topology = {}
+# Optional validation set
+# validation_path = "examples/qwen_coder/ft_data/coder_sft.small.val.jsonl"
+
+[metadata]
+# Effective config hints consumed by the backend
+effective_config = { compute = { gpu_type = "H100", gpu_count = 1, nodes = 1 } }
+
+[training]
+mode = "lora"
+use_qlora = true
+
+[training.validation]
+enabled = true
+evaluation_strategy = "steps"
+eval_steps = 100
+save_best_model_at_end = true
+metric_for_best_model = "val.loss"
+greater_is_better = false
+
+[hyperparameters]
+n_epochs = 1
+train_kind = "peft"
+per_device_batch = 2
+gradient_accumulation_steps = 32
+sequence_length = 4096
+learning_rate = 5e-6
+warmup_ratio = 0.03
+
+[hyperparameters.parallelism]
+use_deepspeed = true
+deepspeed_stage = 2
+fsdp = false
+bf16 = true
+fp16 = false
+activation_checkpointing = true
+
+# LoRA target selection for coder models: apply to all linear projections
+[lora]
+r = 16
+alpha = 32
+dropout = 0.05
+target_modules = ["all-linear"]
+
+

examples/qwen_coder/configs/coder_lora_small.toml

@@ -0,0 +1,57 @@
+# Qwen3 Coder LoRA SFT – all-linear adapters
+
+[algorithm]
+type = "offline"
+method = "sft"
+variety = "qlora"
+
+[job]
+# Smallest supported Qwen3 base; replace with the smallest Coder variant when available
+model = "Qwen/Qwen3-1.7B"
+
+[compute]
+gpu_type = "H100"
+gpu_count = 1
+nodes = 1
+
+[data]
+topology = {}
+# Optional validation set
+# validation_path = "examples/sft/ft_data/coder_traces.val.jsonl"
+
+[training]
+mode = "lora"
+use_qlora = true
+
+[training.validation]
+enabled = true
+evaluation_strategy = "steps"
+eval_steps = 100
+save_best_model_at_end = true
+metric_for_best_model = "val.loss"
+greater_is_better = false
+
+[hyperparameters]
+n_epochs = 1
+train_kind = "peft"
+per_device_batch = 2
+gradient_accumulation_steps = 32
+sequence_length = 4096
+learning_rate = 5e-6
+warmup_ratio = 0.03
+
+[hyperparameters.parallelism]
+use_deepspeed = true
+deepspeed_stage = 2
+fsdp = false
+bf16 = true
+fp16 = false
+activation_checkpointing = true
+
+# LoRA target selection for coder models: apply to all linear projections
+[lora]
+r = 16
+alpha = 32
+dropout = 0.05
+target_modules = ["all-linear"]
+