synth-ai 0.2.13.dev1__py3-none-any.whl → 0.2.14__py3-none-any.whl

examples/multi_step/configs/README_verilog_rl.md

@@ -0,0 +1,77 @@
+# Verilog RL with LoRA (Qwen3-0.6B)
+
+## Quick Start
+
+1. **Deploy Verilog Task App**:
+```bash
+cd synth-ai
+uvx synth-ai modal-serve grpo-verilog
+```
+Note the Modal URL and update `task_url` in `verilog_rl_lora.toml`.
+
+2. **Run Training**:
+```bash
+uvx synth-ai rl run --config examples/multi_step/configs/verilog_rl_lora.toml
+```
+
+## Configuration Overview
+
+### **Key Adaptations from Crafter**:
+
+- **Model**: `Qwen/Qwen3-0.6B` (✅ proven in SFT configs)
+- **Environment**: `verilog` instead of `crafter`
+- **Steps**: 15 turns (vs Crafter's 10) for compilation workflows
+- **Rewards**: Adjusted for sparser Verilog rewards (0.5 vs 1.0 indicator_lambda)
+- **Rubrics**: Verilog-specific judging criteria
+
+### **Hardware Requirements** (Standard RL setup):
+- ✅ **2x H100 GPUs** (vLLM inference + LoRA training split)
+- ✅ **No tensor parallelism** needed for 0.6B model
+- ✅ **4x faster inference** than 32B model
+- ✅ **Same compute pattern** as Crafter (just smaller model)
+
+### **Expected Workflow**:
+1. Agent writes Verilog code (`write_file`)
+2. Compiles to check syntax (`compile`)
+3. Simulates to verify behavior (`simulate`)
+4. Submits if tests pass (`submit`)
+5. **Rewards**: +1.0 for compilation success, +10.0 for passing tests
+
+## Rubric Design
+
+### **Event Rewards** (per decision):
+- **Compilation Success**: 70% weight (1.0 for success, 0.0 for errors)
+- **Process Efficiency**: 30% weight (penalizes redundant operations)
+
+### **Outcome Rewards** (final score):
+- **Tests Passed**: 80% weight (full credit when all tests pass)
+- **Design Quality**: 20% weight (code clarity, documentation)
+
+## Troubleshooting
+
+### **If training fails**:
+1. Check Modal URL in `task_url` field
+2. Verify `GROQ_API_KEY` for inference
+3. Ensure `OPENAI_API_KEY` for judging
+
+### **Memory issues** (unlikely with 0.6B):
+- Reduce `batch_size` to 2
+- Set `gradient_accumulation_steps = 2`
+- Verify 2x GPU split is working (vLLM on GPU 0, training on GPU 1)
+
+### **Slow training**:
+- Increase `episodes_per_batch` to 6-8
+- Check network latency to Modal task app
+
+## Expected Results
+
+- **Convergence**: Should learn basic compilation workflow in 1-2 hours
+- **Success Rate**: 20-40% initial test pass rate (improves with training)
+- **Learning**: Agent learns to debug compilation errors and write correct Verilog
+
+## Next Steps
+
+1. **Monitor reward progression** in training logs
+2. **Adjust rubrics** if agent struggles with compilation errors
+3. **Scale to 8B model** once 0.6B baseline works
+4. **Add domain-specific fine-tuning** for Verilog syntax

examples/multi_step/configs/VERILOG_REWARDS.md

@@ -0,0 +1,90 @@
+# Verilog Reward Structure (Normalized to 1.0)
+
+## Overview
+All rewards in the Verilog task app are normalized so the maximum possible reward is **1.0**.
+
+## Reward Components
+
+### 1. Step Penalty: **-0.001** per step
+- Applied to every action taken
+- Encourages efficient solutions
+- Normalized from `-0.01` (original)
+
+### 2. Compile Success: **+0.01**
+- Awarded when `iverilog` compilation succeeds (returncode 0)
+- Validates syntax correctness
+- Normalized from `+0.1` (original)
+
+### 3. Simulation Pass: **+0.1**
+- Awarded when `vvp` simulation passes all tests
+- Validates behavioral correctness
+- Normalized from `+1.0` (original)
+
+### 4. Submit Success: **+1.0** (maximum reward)
+- Awarded when final submission passes all verification tests
+- This is the goal state
+- Normalized from `+10.0` (original)
+
+## Typical Reward Trajectories
+
+### ✅ Optimal Path (3 steps)
+```
+Step 1: write_file        → -0.001
+Step 2: compile (success) → +0.01 - 0.001 = +0.009
+Step 3: simulate (pass)   → +0.1 - 0.001 = +0.099
+Total:                      ~0.107
+```
+
+### ✅ Good Path (4 steps with submit)
+```
+Step 1: write_file        → -0.001
+Step 2: compile (success) → +0.009
+Step 3: simulate (pass)   → +0.099
+Step 4: submit (success)  → +1.0 - 0.001 = +0.999
+Total:                      ~1.106
+```
+*Note: Can exceed 1.0 if intermediate rewards stack with final submit*
+
+### ❌ Failure Path (compilation errors)
+```
+Step 1: write_file        → -0.001
+Step 2: compile (fail)    → -0.001
+Step 3: write_file (fix)  → -0.001
+Step 4: compile (success) → +0.009
+Step 5: simulate (pass)   → +0.099
+Total:                      ~0.105
+```
+
+## Implementation Details
+
+### Location
+- **Reward components**: `synth_ai/environments/examples/verilog/engine.py`
+  - `VerilogCompileSuccessComponent`: +0.01
+  - `VerilogSimulationPassComponent`: +0.1
+  - `VerilogSubmitSuccessComponent`: +1.0
+  - `VerilogStepPenaltyComponent`: -0.001
+
+### Normalization Ratio
+All rewards were divided by **10.0** to normalize:
+- Original max: ~10.0
+- Normalized max: ~1.0
+- Ratio: 10.0
+
+## Why Normalize?
+
+1. **Consistency**: Makes it easier to compare rewards across different task types
+2. **RL Training**: Standard reward scales improve learning stability
+3. **Interpretability**: Rewards as percentages (0.0 to 1.0) are intuitive
+4. **Judge Compatibility**: Rubric scores typically range 0-1, making blending easier
+
+## Testing
+```bash
+# Run eval to verify normalized rewards
+uvx synth-ai eval --config examples/multi_step/configs/verilog_eval_groq_qwen32b.toml
+```
+
+Expected output for successful rollout:
+- `mean_return` ≈ 0.1 (if only compile+simulate)
+- `mean_return` ≈ 1.0+ (if full submit success)
+
+

examples/multi_step/configs/VERILOG_RL_CHECKLIST.md

@@ -0,0 +1,183 @@
+# Verilog Task App - RL Training Readiness Checklist
+
+## ✅ Core Requirements
+
+### 1. Reward Normalization
+- ✅ **Max reward = 1.0**: All rewards scaled to `[0, 1]` range
+- ✅ **Step penalty**: `-0.001` (normalized from `-0.01`)
+- ✅ **Compile success**: `+0.01` (normalized from `+0.1`)
+- ✅ **Simulate pass**: `+0.1` (normalized from `+1.0`)
+- ✅ **Submit success**: `+1.0` (normalized from `+10.0`)
+
+### 2. Inference URL Handling (Critical for Trace Correlation)
+- ✅ **Extracts from policy config**: Uses `policy_config.get("inference_url")` as primary source
+- ✅ **Includes in trajectory**: Sets `trajectory.inference_url` with `?cid=...` parameter
+- ✅ **Includes in final.info**: Adds to `final["info"]["inference_url"]`
+- ✅ **Includes in pipeline_metadata**: Top-level `inference_url` field for trainer extraction
+- ✅ **Logs cid presence**: Logs `has_cid` flag for debugging
+- ✅ **Fallback to agent.inference_url**: Uses agent's URL if policy config missing (eval mode)
+
+**Location**: `grpo_verilog.py` lines 829-867, 887-908
+
+### 3. Pipeline Metadata
+- ✅ **Required fields present**:
+  - `reward_score`: Final episode reward
+  - `policy_id`: Policy identifier
+  - `inference_url`: **CRITICAL** - Contains `?cid=trace_xxxxx` for correlation
+  - `env_name`: Environment identifier
+  - `task_id`: Problem identifier
+  - `task_split`: Dataset split (train/val/test)
+- ✅ **Inference details**: Provider, model, URL in nested `inference` dict
+
+**Location**: `grpo_verilog.py` lines 887-908
+
+### 4. Trace Correlation (Required for RL Training)
+- ✅ **Trainer injects cid**: Trainer adds `?cid=trace_xxxxx` to `policy_config["inference_url"]`
+- ✅ **Task app preserves cid**: Uses `policy_config["inference_url"]` directly
+- ✅ **Trainer extracts cid**: Extracts from `trajectory.inference_url` using `inference_url_to_trace_correlation_id()`
+- ✅ **Trace hydration**: Trainer queries trace store with extracted `trace_correlation_id`
+
+**Flow**:
+```
+Trainer → policy_config["inference_url"] = "http://...?cid=trace_xxxxx"
+         ↓
+Task App → trajectory.inference_url = policy_config["inference_url"]
+         ↓
+Trainer → extract_trace_correlation_id(trajectory.inference_url)
+         ↓
+Trainer → trace_store.resolve_correlation(trace_correlation_id)
+         ↓
+Trainer → Hydrate v3 trace with event_history
+         ↓
+Judge   → Score using full trace
+```
+
+### 5. Response Contract Compliance
+- ✅ **RolloutResponse fields**:
+  - `run_id`: Unique identifier
+  - `trajectories`: List of trajectories (with `inference_url`)
+  - `metrics`: Episode metrics
+  - `pipeline_metadata`: **CRITICAL** - Contains `inference_url` and `reward_score`
+  - `trace_correlation_id`: Optional (trainer infers from `inference_url`)
+- ✅ **Optional trace_correlation_id**: Made optional in `contracts.py` (trainer infers from URL)
+
+**Location**: `synth_ai/task/contracts.py` line 156
+
+### 6. Environment Implementation
+- ✅ **Stateful engine**: `VerilogEngine` extends `StatefulEngine`
+- ✅ **Reward stack**: Properly configured with normalized components
+- ✅ **State management**: `VerilogPublicState` and `VerilogPrivateState`
+- ✅ **Tool implementation**: All 4 tools (write_file, compile, simulate, submit)
+
+**Location**: `synth_ai/environments/examples/verilog/engine.py`
+
+### 7. LLM Agent Integration
+- ✅ **Multi-turn support**: Agent maintains conversation history
+- ✅ **Tool parsing**: Extracts tool calls from LLM responses
+- ✅ **Guidance system**: Provides context-aware hints
+- ✅ **Error handling**: Graceful fallback for malformed responses
+
+**Location**: `grpo_verilog.py` lines 200-530
+
+## 🔍 Verification Tests
+
+### Test 1: Eval Mode (No Trace Correlation)
+```bash
+uvx synth-ai eval --config examples/multi_step/configs/verilog_eval_groq_qwen32b.toml
+```
+**Expected**:
+- ✅ `mean_return` ≈ 0.1 (normalized rewards)
+- ✅ `inference_url` = Groq API URL (no `?cid=...`)
+- ✅ `task_completed` = True for correct solutions
+
+### Test 2: RL Training Mode (With Trace Correlation)
+```bash
+uvx synth-ai train \
+  --type rl \
+  --config examples/multi_step/configs/verilog_rl_lora.toml \
+  --task-url https://synth-laboratories--grpo-verilog-task-app-fastapi-app-dev.modal.run \
+  --backend https://synth-backend-dev-docker.onrender.com/api \
+  --env-file /path/to/verilog/.env
+```
+**Expected**:
+- ✅ Trainer logs show `inference_url` with `?cid=trace_xxxxx`
+- ✅ Task app logs show `has_cid=True`
+- ✅ Trace hydration succeeds (no `404 Not Found` errors)
+- ✅ Judge receives full `event_history`
+- ✅ Training updates show non-zero rewards
+
+### Test 3: Trace Correlation ID Extraction
+```python
+from synth_envs_hosted.utils import inference_url_to_trace_correlation_id
+
+# Should extract trace_xxxxx from URL
+url = "http://localhost:8000/v1/chat/completions?cid=trace_abc123"
+cid = inference_url_to_trace_correlation_id(url)
+assert cid == "trace_abc123"
+```
+
+### Test 4: Pipeline Metadata Structure
+```python
+# Verify response has correct structure for RL
+response = await task_app.rollout(request)
+assert "pipeline_metadata" in response
+assert "inference_url" in response.pipeline_metadata
+assert "reward_score" in response.pipeline_metadata
+assert len(response.trajectories) > 0
+assert response.trajectories[0].inference_url is not None
+```
+
+## 📋 Deployment Checklist
+
+### Modal Deployment
+1. ✅ **Environment variables set**:
+   - `GROQ_API_KEY`
+   - `VERILOG_INFERENCE_URL` (optional, uses Groq default)
+2. ✅ **Secrets configured**: Groq API key in Modal secrets
+3. ✅ **Task app URL**: Update in `verilog_rl_lora.toml`
+
+### Training Configuration
+1. ✅ **2x GPUs minimum**: 1 for vLLM, 1 for training
+2. ✅ **Model size**: `Qwen/Qwen3-0.6B` for testing
+3. ✅ **Batch size**: 4 (matches Crafter)
+4. ✅ **Max turns**: 15 (enough for compile chains)
+5. ✅ **Rubric enabled**: `rubric.enabled = true`
+
+## 🚨 Common Issues & Fixes
+
+### Issue 1: `trace_correlation_id` Missing
+**Symptom**: Trainer logs `FATAL: Rollout payload missing 'trace_correlation_id'`
+**Fix**: Verify `trajectory.inference_url` contains `?cid=...` parameter
+
+### Issue 2: Trace Hydration Fails (404)
+**Symptom**: `404 Not Found` when querying `/trace/by-correlation/...`
+**Fix**: 
+- Check inference server is capturing traces
+- Verify `cid` parameter is in inference URL
+- Ensure `vllm_public_url` is set correctly
+
+### Issue 3: Rewards Not Normalized
+**Symptom**: `mean_return` > 1.0 in eval
+**Fix**: Verify all reward components in `engine.py` are scaled by 10x
+
+### Issue 4: Agent Gets Stuck
+**Symptom**: Agent repeats same action (e.g., compile without fixing)
+**Fix**: Check guidance system is providing proper hints
+
+## 🎯 Final Verification
+
+Before starting RL training, verify:
+- [ ] Eval runs successfully with normalized rewards (≈ 0.1)
+- [ ] Modal deployment returns proper `inference_url` structure
+- [ ] Trace correlation ID extraction works
+- [ ] Pipeline metadata includes all required fields
+- [ ] Response contract matches expected schema
+
+**If all checks pass**: ✅ **Ready for RL training!**
+
+## 📚 Related Documentation
+- [VERILOG_REWARDS.md](./VERILOG_REWARDS.md) - Reward structure details
+- [verilog_rl_lora.md](../verilog_rl_lora.md) - RL/LoRA feasibility analysis
+- [verilog_rl_lora.toml](./verilog_rl_lora.toml) - Training configuration
+
+

examples/multi_step/configs/crafter_eval_synth_qwen4b.toml

@@ -0,0 +1,35 @@
+# Crafter eval using Synth backend with Qwen3-4B
+
+[eval]
+app_id = "grpo-crafter-task-app"
+task_app_url = "https://synth-laboratories--grpo-crafter-task-app-fastapi-app-dev.modal.run"
+model = "Qwen/Qwen3-4B"
+seeds = [0, 1, 2]
+max_turns = 10
+concurrency = 1
+env_name = "crafter"
+policy_name = "crafter-react"
+trace_format = "full"
+return_trace = true
+
+[eval.env_config]
+env_params = {max_steps_per_episode = 10}
+
+[eval.policy_config]
+provider = "openai"
+model = "Qwen/Qwen3-4B"
+inference_url = "https://synth-backend-dev-docker.onrender.com/api/v1/chat/completions"
+temperature = 0.6
+top_p = 0.95
+max_tokens = 512
+use_vision = false
+image_only_mode = false
+max_llm_calls = 10
+
+[eval.judge]
+path = "examples/multi_step/judges/crafter_backend_judge.py"
+name = "Backend"
+backend_url = "http://localhost:8000/api"
+model = "openai/gpt-oss-120b"
+timeout_s = 45
+

examples/multi_step/configs/crafter_eval_text_only_groq_qwen32b.toml

@@ -0,0 +1,36 @@
+# Evaluation config for Crafter with text-only input
+# This config uses Groq Qwen with only text observations (no images)
+
+[eval]
+app_id = "grpo-crafter-task-app"
+task_app_url = "https://synth-laboratories--grpo-crafter-task-app-fastapi-app-dev.modal.run"
+model = "qwen/qwen3-32b"
+seeds = [0, 1, 2]
+max_turns = 10
+concurrency = 1
+env_name = "crafter"
+policy_name = "crafter-react"
+trace_format = "full"
+return_trace = true
+
+[eval.env_config]
+env_params = {max_steps_per_episode = 10}
+
+[eval.policy_config]
+provider = "groq"
+model = "qwen/qwen3-32b"
+inference_url = "https://api.groq.com/openai/v1/chat/completions"
+temperature = 0.6
+top_p = 0.95
+max_tokens = 512
+use_vision = false
+image_only_mode = false
+max_llm_calls = 10
+
+[eval.judge]
+path = "examples/multi_step/judges/crafter_backend_judge.py"
+name = "Backend"
+backend_url = "http://localhost:8000/api"
+model = "openai/gpt-oss-120b"
+timeout_s = 45
+

examples/multi_step/configs/crafter_rl_stepwise_hosted_judge.toml

@@ -12,7 +12,7 @@ variety = "gspo"
 # Replace with the Modal URL printed by `uvx synth-ai modal-serve grpo-crafter`
 task_url = "https://YOUR-MODAL-TASK-APP.modal.run"
 # Point at the Synth backend (or compatible service) that exposes /api/judge/v1/*
-judge_url = "https://YOUR-BACKEND-ENDPOINT/api"
+judge_url = "https://synth-backend-dev-docker.onrender.com/api"
 
 [compute]
 gpu_type = "H200"
@@ -46,7 +46,7 @@ target_modules = ["all-linear"]
 [rollout]
 env_name = "crafter"
 max_turns = 10
-episodes_per_batch = 4
+episodes_per_batch = 2
 policy_name = "crafter-react"
 max_concurrent_rollouts = 8
 batches_per_step = 2
@@ -69,12 +69,12 @@ ops = ["agent", "env"]
 
 [evaluation]
 instances = 16
-every_n_iters = 8
+every_n_iters = 10
 seeds = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
 [training]
 num_epochs = 1
-iterations_per_epoch = 16
+iterations_per_epoch = 20
 gradient_accumulation_steps = 1
 max_accumulated_minibatch = 1
 max_turns = 10
@@ -84,6 +84,7 @@ learning_rate = 5e-5
 log_interval = 1
 weight_sync_interval = 1
 event_rewards_kind = "unique"
+async_semaphore_max = 40  # Max concurrent rollouts in streaming pipeline
 
 # Enable dense decision rewards in the trainer to mirror env_config step rewards.
 step_rewards_enabled = true
@@ -101,6 +102,9 @@ verify_every_k = 0
 
 [rubric]
 enabled = true
+model = "openai/gpt-oss-120b"
+api_base = "https://synth-backend-dev-docker.onrender.com/api/judge"
+api_key_env = "OPENAI_API_KEY"
 # Blend the hosted judge scores with environment returns inside the trainer.
 [rubric.weights]
 env = 0.2
@@ -110,13 +114,21 @@ outcome = 0.4
 [rubric.event]
 # Hosted judge rubric for per-decision progress scoring.
 rubric_id = "crafter/event@v1"
+criteria = [
+  { key = "progress.unique_achievements", weight = 0.9, description = "Return 1 when this decision explicitly unlocks a brand-new Crafter achievement (inventory or status text confirms it this turn). Otherwise return 0.", aggregation = "weighted_sum" },
+  { key = "process.intent_alignment", weight = 0.1, description = "Use at most 0.3 to acknowledge tightly coupled setup that finishes the last prerequisite; keep ≤0.1 when the agent only repositions or gathers without an imminent unlock.", aggregation = "weighted_sum" },
+]
 
 [rubric.outcome]
 # Hosted judge rubric for final trajectory scoring.
 rubric_id = "crafter/outcome@v1"
+criteria = [
+  { key = "outcome.goal_completion", weight = 0.6, description = "Full credit when the agent ends with strong survival metrics and a clear crafted milestone (e.g., iron tools, furnace).", aggregation = "weighted_sum" },
+  { key = "outcome.achievement_depth", weight = 0.4, description = "Partial credit for intermediate achievements (saplings, wood/stone tools) that set up future success.", aggregation = "weighted_sum" },
+]
 
 [judge]
-type = "gemini"                            # or "groq" when routing to Groq-hosted judges
+type = "groq"                            # or "groq" when routing to Groq-hosted judges
 timeout_s = 45
 
   [judge.options]

examples/multi_step/configs/crafter_synth_backend.md

@@ -0,0 +1,40 @@
+# Crafter Eval Using Synth Backend with Qwen 4B
+
+## What Changed
+
+Created `crafter_eval_synth_qwen4b.toml` to evaluate Crafter using Qwen3-4B via the Synth backend inference proxy.
+
+## Key Difference from Groq Config
+
+**Before (Groq):**
+```toml
+[eval.policy_config]
+provider = "groq"
+model = "qwen/qwen3-32b"
+inference_url = "https://api.groq.com/openai/v1/chat/completions"
+```
+
+**After (Synth Backend):**
+```toml
+[eval.policy_config]
+provider = "openai"
+model = "Qwen/Qwen3-4B"
+inference_url = "https://synth-backend-dev-docker.onrender.com/api/v1/chat/completions"
+```
+
+## Usage
+
+```bash
+uvx synth-ai eval --config examples/multi_step/configs/crafter_eval_synth_qwen4b.toml
+```
+
+## Why This Works
+
+The Synth backend's `/api/v1/chat/completions` endpoint:
+1. Accepts OpenAI-compatible requests
+2. Routes to Modal vLLM service
+3. Loads the base model (Qwen/Qwen3-4B from HuggingFace)
+4. Returns OpenAI-compatible responses
+
+No code changes needed - the infrastructure already exists.
+

examples/multi_step/configs/verilog_eval_groq_qwen32b.toml

@@ -0,0 +1,31 @@
+# Verilog Eval Config for Groq Qwen3-32B
+# Quick eval to test Verilog task app before RL training
+
+[eval]
+app_id = "grpo-verilog"
+task_app_url = "https://synth-laboratories--grpo-verilog-task-app-fastapi-app-dev.modal.run"
+model = "qwen/qwen3-32b"
+seeds = [0, 1, 2]
+max_turns = 15
+concurrency = 1
+env_name = "verilog"
+policy_name = "verilog-designer"
+trace_format = "full"
+return_trace = true
+
+[eval.env_config]
+difficulty = "medium"  # Can be "easy", "medium", or "hard"
+
+[eval.policy_config]
+provider = "groq"
+model = "qwen/qwen3-32b"
+inference_url = "https://api.groq.com/openai/v1/chat/completions"
+temperature = 0.2
+max_tokens = 8192  # Large buffer for Verilog (long testbenches + module implementation)
+
+[eval.judge]
+path = "examples/multi_step/judges/verilog_backend_judge.py"
+name = "Backend"
+backend_url = "http://localhost:8000/api"
+model = "openai/gpt-oss-120b"
+timeout_s = 45

examples/multi_step/configs/verilog_eval_synth_qwen8b.toml

@@ -0,0 +1,33 @@
+# Verilog eval using Synth backend with Qwen3-8B
+
+[eval]
+app_id = "grpo-verilog"
+task_app_url = "https://synth-laboratories--grpo-verilog-task-app-fastapi-app-dev.modal.run"
+model = "Qwen/Qwen3-8B"
+seeds = [0, 1, 2]
+max_turns = 6
+concurrency = 1
+env_name = "verilog"
+policy_name = "verilog-designer"
+trace_format = "full"
+return_trace = true
+
+[eval.env_config]
+difficulty = "medium"
+
+[eval.policy_config]
+provider = "openai"
+model = "Qwen/Qwen3-8B"
+inference_url = "https://synth-backend-dev-docker.onrender.com/api/v1/chat/completions"
+temperature = 0.2
+top_p = 0.95
+max_tokens = 4096
+max_llm_calls = 6
+
+[eval.judge]
+path = "examples/multi_step/judges/verilog_backend_judge.py"
+name = "Backend"
+backend_url = "http://localhost:8000/api"
+model = "openai/gpt-oss-120b"
+timeout_s = 45
+