synth-ai 0.2.14__py3-none-any.whl → 0.4.4__py3-none-any.whl

examples/multi_step/crafter_rl_lora.md

@@ -1,70 +0,0 @@
-# Crafter RL LoRA (10-step runs)
-
-This walkthrough shows how to fine-tune the Crafter task app with our 10-step RL LoRA config.
-
-1. **Deploy the Crafter task app on Modal**
-
-   ```bash
-   # assumes .env contains SYNTH_API_KEY, ENVIRONMENT_API_KEY, GROQ_API_KEY, etc.
-   uvx synth-ai modal-serve grpo-crafter \
-     --env-file examples/warming_up_to_rl/.env \
-     --name grpo-crafter-task-app
-   ```
-
-   * The command prints the public `https://…modal.run` URL; copy it for the RL configs below.*
-
-2. **Wire up the three RL experiment configs**
-
-   Update the `task_url` placeholder in each config with the Modal URL from step 1:
-
-   - `examples/multi_step/configs/crafter_rl_outcome.toml`
-   - `examples/multi_step/configs/crafter_rl_stepwise_simple.toml`
-   - `examples/multi_step/configs/crafter_rl_stepwise_shaped.toml`
-
-   The difference between them (all run with LoRA on 2×H100 split 1/1 for vLLM vs. trainer):
-
-   | Config | Reward signal |
-   | ------ | ------------- |
-   | `crafter_rl_outcome.toml` | Outcome-only — step rewards disabled. |
-   | `crafter_rl_stepwise_simple.toml` | Stepwise (“consistent”) — +1 for every newly unlocked achievement. |
-   | `crafter_rl_stepwise_shaped.toml` | Stepwise (“per_achievement”) — combines achievement credit with inventory/achievement-count shaping from the rollout hook. |
-
-3. **Launch the three RL runs in parallel**
-
-   ```bash
-   export SYNTH_API_KEY=...       # already sourced if examples/.env was loaded
-   export TASK_APP_URL=https://your-modal-task-app.modal.run
-
-   uvx synth-ai train --type rl \
-     --config examples/multi_step/configs/crafter_rl_outcome.toml \
-     --run-name crafter-rl-outcome \
-     --no-poll &
-
-   uvx synth-ai train --type rl \
-     --config examples/multi_step/configs/crafter_rl_stepwise_simple.toml \
-     --run-name crafter-rl-stepwise-simple \
-     --no-poll &
-
-   uvx synth-ai train --type rl \
-     --config examples/multi_step/configs/crafter_rl_stepwise_shaped.toml \
-     --run-name crafter-rl-stepwise-shaped \
-     --no-poll &
-
-   wait
-   ```
-
-   *`--no-poll` returns immediately so each run can stream logs in its own terminal; `wait` blocks until all jobs finish.*
-
-4. **Track results**
-
-   Tail each job’s logs with `uvx synth-ai train logs --run-name <name>` or open the Modal dashboard. Compare:
-
-   - Avg outcome reward (modal dashboard)
-   - Stepwise reward components (`resource_reward`, `unique_achievements_total`) in the task app logs
-   - Trace JSONL dumps under `traces/v3` if tracing is enabled
-
-
-  * This config forces 10 agent turns per rollout, reduces batch size to avoid OOMs, and enforces Crafter-specific defaults.*
-
-  INFO - 🎉 Training completed successfully!
-  INFO - All batch rewards: [0.0625, 0.0625, 0.125, 0.0625, 0.0625, 0.3125, 0.375, 0.4375, 0.5, 0.9375]

examples/multi_step/judges/crafter_backend_judge.py

@@ -1,220 +0,0 @@
-"""Crafter backend judge that calls the Synth judge API with inline rubric."""
-
-from __future__ import annotations
-
-import json
-import os
-from pathlib import Path
-from typing import Any, Dict, List, TypedDict
-
-import httpx
-
-
-class TraceMetadata(TypedDict, total=False):
-    """Metadata for the trace."""
-    env_id: str
-    policy_id: str
-    length: int
-
-
-class JudgeTracePayload(TypedDict):
-    """Trace payload sent to backend judge."""
-    event_history: List[Dict[str, Any]]
-    markov_blanket_message_history: List[Dict[str, Any]]
-    metadata: TraceMetadata
-
-
-class JudgeOptions(TypedDict, total=False):
-    """Options for judge scoring."""
-    model: str
-    timeout_s: int
-    event: bool
-    outcome: bool
-
-
-class TaskApp(TypedDict):
-    """Task application metadata."""
-    id: str
-
-
-class JudgeScoreRequest(TypedDict):
-    """Request to backend judge API."""
-    policy_name: str
-    task_app: TaskApp
-    trace: JudgeTracePayload
-    rubric: Dict[str, Any]
-    options: JudgeOptions
-
-
-# Load rubric from file (cached at module level)
-_RUBRIC_PATH = Path(__file__).parent.parent / "rubrics" / "crafter_backend_judge.json"
-_RUBRIC: Dict[str, Any] | None = None
-
-
-def _load_rubric() -> Dict[str, Any]:
-    """Load rubric from file with fallback to inline default."""
-    global _RUBRIC
-    if _RUBRIC is None:
-        try:
-            with open(_RUBRIC_PATH, 'r') as f:
-                _RUBRIC = json.load(f)
-            assert isinstance(_RUBRIC, dict), "Rubric must be a dict"
-            assert "outcome" in _RUBRIC, "Rubric must have 'outcome' key"
-            assert isinstance(_RUBRIC["outcome"], list), "Rubric 'outcome' must be a list"
-        except Exception as e:
-            print(f"[crafter_backend_judge] Warning: Failed to load rubric from {_RUBRIC_PATH}: {e}")
-            # Fallback inline rubric (matching RubricCriteriaBlock format)
-            _RUBRIC = {
-                "event": [],
-                "outcome": [
-                    {"id": "achievement_progression", "description": "Achievement progression", "weight": 0.35, "scale": "bounded"},
-                    {"id": "resource_stockpile", "description": "Resource stockpile", "weight": 0.2, "scale": "bounded"},
-                    {"id": "survival_state", "description": "Survival state", "weight": 0.2, "scale": "bounded"},
-                    {"id": "failure_analysis", "description": "Failure analysis", "weight": 0.15, "scale": "bounded"},
-                    {"id": "future_readiness", "description": "Future readiness", "weight": 0.1, "scale": "bounded"}
-                ]
-            }
-    return _RUBRIC
-
-
-def judge(payload: Dict[str, Any], **kwargs: Any) -> float:
-    """
-    Call the Synth backend judge API to score a Crafter rollout.
-
-    Args:
-        payload: Dict with keys: seed, prompt, completion, metrics, response, trace
-        **kwargs: Additional config (backend_url, model, timeout_s, etc.)
-
-    Returns:
-        float: Aggregate score from 0.0 to 1.0
-    """
-    try:
-        # Extract configuration
-        backend_url = kwargs.get("backend_url", "http://localhost:8000/api")
-        model = kwargs.get("model", "openai/gpt-oss-120b")
-        timeout = kwargs.get("timeout_s", 45)
-        
-        assert isinstance(backend_url, str), "backend_url must be a string"
-        assert isinstance(model, str), "model must be a string"
-        assert isinstance(timeout, (int, float)), "timeout_s must be numeric"
-        
-        # Extract trajectory from response
-        response_data = payload.get("response", {})
-        assert isinstance(response_data, dict), "response must be a dict"
-        
-        trajectories = response_data.get("trajectories", [])
-        assert isinstance(trajectories, list), "trajectories must be a list"
-        
-        if not trajectories:
-            print("[crafter_backend_judge] No trajectories in response")
-            return 0.0
-        
-        trajectory = trajectories[0]  # First trajectory
-        assert isinstance(trajectory, dict), "trajectory must be a dict"
-        
-        # Load rubric
-        rubric = _load_rubric()
-        
-        # Transform trajectory into JudgeTracePayload format
-        steps = trajectory.get("steps", [])
-        assert isinstance(steps, list), "trajectory steps must be a list"
-        
-        event_history: List[Dict[str, Any]] = []
-        for idx, step in enumerate(steps):
-            assert isinstance(step, dict), f"step {idx} must be a dict"
-            # Each step becomes an event
-            event_history.append({
-                "observation": step.get("obs", {}),
-                "tool_calls": step.get("tool_calls", []),
-                "reward": step.get("reward", 0.0),
-                "done": step.get("done", False),
-                "truncated": step.get("truncated", False),
-                "info": step.get("info", {}),
-            })
-        
-        # Add final observation - backend will extract this as outcome context
-        final_data = trajectory.get("final", {})
-        if final_data:
-            assert isinstance(final_data, dict), "final data must be a dict"
-            final_obs = final_data.get("observation", {})
-            assert isinstance(final_obs, dict), "final observation must be a dict"
-            
-            event_history.append({
-                "observation": final_obs,
-                "reward": final_data.get("reward", 0.0),
-                "done": final_data.get("done", True),
-                "truncated": final_data.get("truncated", False),
-                "info": final_data.get("info", {}),
-            })
-        
-        # Build trace metadata
-        metadata: TraceMetadata = {
-            "env_id": trajectory.get("env_id", "crafter"),
-            "policy_id": trajectory.get("policy_id", "crafter-react"),
-            "length": trajectory.get("length", len(steps)),
-        }
-        
-        # Build judge request with rubric included
-        judge_request: JudgeScoreRequest = {
-            "policy_name": "crafter-react",
-            "task_app": {"id": "grpo-crafter-task-app"},
-            "trace": {
-                "event_history": event_history,
-                "markov_blanket_message_history": [],
-                "metadata": metadata,
-            },
-            "rubric": rubric,
-            "options": {
-                "model": model,
-                "timeout_s": timeout,
-                "event": False,  # Not scoring per-event
-                "outcome": True,  # Score the final outcome
-            }
-        }
-        
-        # Call backend judge API
-        with httpx.Client(timeout=timeout) as client:
-            # Get API key from env
-            api_key = os.environ.get("SYNTH_API_KEY") or os.environ.get("OPENAI_API_KEY")
-            headers = {}
-            if api_key:
-                headers["Authorization"] = f"Bearer {api_key}"
-            
-            url = f"{backend_url.rstrip('/')}/judge/v1/score"
-            
-            # Debug: print request summary
-            print(f"\n[crafter_backend_judge] Scoring trajectory with {len(event_history)} events")
-            if event_history:
-                last_obs = event_history[-1].get('observation', {})
-                print(f"  Final observation keys: {list(last_obs.keys())[:5]}...")
-            
-            response = client.post(url, json=judge_request, headers=headers)
-            
-            response.raise_for_status()
-            result = response.json()
-            assert isinstance(result, dict), "Response must be a dict"
-            
-            # Extract aggregate score
-            aggregate_score = result.get("aggregate_score", 0.0)
-            
-            # Try outcome_review.total if aggregate_score not found
-            if aggregate_score == 0.0 and "outcome_review" in result:
-                outcome_review = result["outcome_review"]
-                if isinstance(outcome_review, dict):
-                    aggregate_score = outcome_review.get("total", 0.0)
-            
-            print(f"  Backend judge score: {aggregate_score:.3f}\n")
-            return float(aggregate_score)
-            
-    except httpx.HTTPStatusError as e:
-        print(f"\n[crafter_backend_judge] HTTP ERROR:")
-        print(f"  Status: {e.response.status_code}")
-        print(f"  Response: {e.response.text[:300]}\n")
-        return 0.0
-    except AssertionError as e:
-        print(f"[crafter_backend_judge] Assertion error: {e}")
-        return 0.0
-    except Exception as e:
-        print(f"[crafter_backend_judge] Unexpected error: {e}")
-        return 0.0
-

examples/multi_step/judges/verilog_backend_judge.py

@@ -1,234 +0,0 @@
-"""Verilog backend judge that calls the Synth judge API with inline rubric."""
-
-from __future__ import annotations
-
-import json
-import os
-from pathlib import Path
-from typing import Any, Dict, List, TypedDict
-
-import httpx
-
-
-class TraceMetadata(TypedDict, total=False):
-    """Metadata for the trace."""
-    env_id: str
-    policy_id: str
-    length: int
-
-
-class JudgeTracePayload(TypedDict):
-    """Trace payload sent to backend judge."""
-    event_history: List[Dict[str, Any]]
-    markov_blanket_message_history: List[Dict[str, Any]]
-    metadata: TraceMetadata
-
-
-class JudgeOptions(TypedDict, total=False):
-    """Options for judge scoring."""
-    model: str
-    timeout_s: int
-    event: bool
-    outcome: bool
-
-
-class TaskApp(TypedDict):
-    """Task application metadata."""
-    id: str
-
-
-class JudgeScoreRequest(TypedDict):
-    """Request to backend judge API."""
-    policy_name: str
-    task_app: TaskApp
-    trace: JudgeTracePayload
-    rubric: Dict[str, Any]
-    options: JudgeOptions
-
-
-# Load rubric from file (cached at module level)
-_RUBRIC_PATH = Path(__file__).parent.parent / "rubrics" / "verilog_backend_judge.json"
-_RUBRIC: Dict[str, Any] | None = None
-
-
-def _load_rubric() -> Dict[str, Any]:
-    """Load rubric from file with fallback to inline default."""
-    global _RUBRIC
-    if _RUBRIC is None:
-        try:
-            with open(_RUBRIC_PATH, 'r') as f:
-                _RUBRIC = json.load(f)
-            assert isinstance(_RUBRIC, dict), "Rubric must be a dict"
-            assert "outcome" in _RUBRIC, "Rubric must have 'outcome' key"
-            assert isinstance(_RUBRIC["outcome"], list), "Rubric 'outcome' must be a list"
-        except Exception as e:
-            print(f"[verilog_backend_judge] Warning: Failed to load rubric from {_RUBRIC_PATH}: {e}")
-            # Fallback inline rubric (matching RubricCriteriaBlock format)
-            _RUBRIC = {
-                "event": [],
-                "outcome": [
-                    {"id": "correctness.tests_pass", "description": "Tests pass", "weight": 0.5, "scale": "bounded"},
-                    {"id": "efficiency.code_quality", "description": "Code quality", "weight": 0.3, "scale": "bounded"},
-                    {"id": "efficiency.solution_steps", "description": "Solution efficiency", "weight": 0.2, "scale": "bounded"}
-                ]
-            }
-    return _RUBRIC
-
-
-def judge(payload: Dict[str, Any], **kwargs: Any) -> float:
-    """
-    Call the Synth backend judge API to score a Verilog rollout.
-
-    Args:
-        payload: Dict with keys: seed, prompt, completion, metrics, response, trace
-        **kwargs: Additional config (backend_url, model, timeout_s, etc.)
-
-    Returns:
-        float: Aggregate score from 0.0 to 1.0
-    """
-    try:
-        # Extract configuration
-        backend_url = kwargs.get("backend_url", "http://localhost:8000/api")
-        model = kwargs.get("model", "openai/gpt-oss-120b")
-        timeout = kwargs.get("timeout_s", 45)
-        
-        assert isinstance(backend_url, str), "backend_url must be a string"
-        assert isinstance(model, str), "model must be a string"
-        assert isinstance(timeout, (int, float)), "timeout_s must be numeric"
-        
-        # Extract trajectory from response
-        response_data = payload.get("response", {})
-        assert isinstance(response_data, dict), "response must be a dict"
-        
-        trajectories = response_data.get("trajectories", [])
-        assert isinstance(trajectories, list), "trajectories must be a list"
-        
-        if not trajectories:
-            print("[verilog_backend_judge] No trajectories in response")
-            return 0.0
-        
-        trajectory = trajectories[0]  # First trajectory
-        assert isinstance(trajectory, dict), "trajectory must be a dict"
-        
-        # Load rubric
-        rubric = _load_rubric()
-        
-        # Transform trajectory into JudgeTracePayload format
-        # The backend expects: event_history, markov_blanket_message_history, metadata
-        steps = trajectory.get("steps", [])
-        assert isinstance(steps, list), "trajectory steps must be a list"
-        
-        event_history: List[Dict[str, Any]] = []
-        for idx, step in enumerate(steps):
-            assert isinstance(step, dict), f"step {idx} must be a dict"
-            # Each step becomes an event with obs, tool_calls, reward, done, info
-            event_history.append({
-                "observation": step.get("obs", {}),
-                "tool_calls": step.get("tool_calls", []),
-                "reward": step.get("reward", 0.0),
-                "done": step.get("done", False),
-                "truncated": step.get("truncated", False),
-                "info": step.get("info", {}),
-            })
-        
-        # Add final observation - backend will extract this as outcome context
-        final_data = trajectory.get("final", {})
-        if final_data:
-            assert isinstance(final_data, dict), "final data must be a dict"
-            final_obs = final_data.get("observation", {})
-            assert isinstance(final_obs, dict), "final observation must be a dict"
-            
-            event_history.append({
-                "observation": final_obs,
-                "reward": final_data.get("reward", 0.0),
-                "done": final_data.get("done", True),
-                "truncated": final_data.get("truncated", False),
-                "info": final_data.get("info", {}),
-            })
-        
-        # Build trace metadata
-        metadata: TraceMetadata = {
-            "env_id": trajectory.get("env_id", "verilog"),
-            "policy_id": trajectory.get("policy_id", "verilog-designer"),
-            "length": trajectory.get("length", len(steps)),
-        }
-        
-        # Build judge request with rubric included
-        judge_request: JudgeScoreRequest = {
-            "policy_name": "verilog-designer",
-            "task_app": {"id": "grpo-verilog"},
-            "trace": {
-                "event_history": event_history,
-                "markov_blanket_message_history": [],
-                "metadata": metadata,
-            },
-            "rubric": rubric,
-            "options": {
-                "model": model,
-                "timeout_s": timeout,
-                "event": False,  # Not scoring per-event
-                "outcome": True,  # Score the final outcome
-            }
-        }
-        
-        # Call backend judge API
-        with httpx.Client(timeout=timeout) as client:
-            # Get API key from env
-            api_key = os.environ.get("SYNTH_API_KEY") or os.environ.get("OPENAI_API_KEY")
-            headers = {}
-            if api_key:
-                headers["Authorization"] = f"Bearer {api_key}"
-            
-            url = f"{backend_url.rstrip('/')}/judge/v1/score"
-            
-            # Debug: print request details
-            print(f"\n[verilog_backend_judge] REQUEST DEBUG:")
-            print(f"  URL: {url}")
-            print(f"  Request body keys: {list(judge_request.keys())}")
-            rubric_data = judge_request.get('rubric', {})
-            print(f"  Rubric event criteria: {len(rubric_data.get('event', []))}")
-            print(f"  Rubric outcome criteria: {len(rubric_data.get('outcome', []))}")
-            trace_data = judge_request.get('trace', {})
-            event_hist = trace_data.get('event_history', [])
-            print(f"  Trace event_history count: {len(event_hist)}")
-            if event_hist:
-                last_event = event_hist[-1]
-                last_obs = last_event.get('observation', {})
-                print(f"  Last event done: {last_event.get('done', False)}")
-                print(f"  Last obs keys: {list(last_obs.keys())}")
-                print(f"  Task completed: {last_obs.get('task_completed', 'N/A')}")
-            
-            response = client.post(url, json=judge_request, headers=headers)
-            
-            # Debug: print response details
-            print(f"\n[verilog_backend_judge] RESPONSE DEBUG:")
-            print(f"  Status: {response.status_code}")
-            print(f"  Response body: {response.text[:500]}")  # First 500 chars
-            
-            response.raise_for_status()
-            result = response.json()
-            assert isinstance(result, dict), "Response must be a dict"
-            
-            # Extract aggregate score
-            aggregate_score = result.get("aggregate_score", 0.0)
-            
-            # Try outcome_review.total if aggregate_score not found
-            if aggregate_score == 0.0 and "outcome_review" in result:
-                outcome_review = result["outcome_review"]
-                if isinstance(outcome_review, dict):
-                    aggregate_score = outcome_review.get("total", 0.0)
-            
-            print(f"  Aggregate score: {aggregate_score}\n")
-            return float(aggregate_score)
-            
-    except httpx.HTTPStatusError as e:
-        print(f"\n[verilog_backend_judge] HTTP ERROR:")
-        print(f"  Status: {e.response.status_code}")
-        print(f"  Response body: {e.response.text}\n")
-        return 0.0
-    except AssertionError as e:
-        print(f"[verilog_backend_judge] Assertion error: {e}")
-        return 0.0
-    except Exception as e:
-        print(f"[verilog_backend_judge] Unexpected error: {e}")
-        return 0.0

examples/multi_step/readme.md

@@ -1,48 +0,0 @@
-
-
-Crafter
-
-cd /Users/joshpurtell/Documents/GitHub/synth-ai && uvx synth-ai modal-serve grpo-crafter-task-app --name grpo-crafter-task-app --env-file /Users/joshpurtell/Documents/GitHub/monorepo/environments/crafter/.env
-
-cd /Users/joshpurtell/Documents/GitHub/monorepo && uv run modal deploy backend/app/routes/clustered_training/core/algorithms/gspo/app.py --env dev
-
-uvx synth-ai eval --config /Users/joshpurtell/Documents/GitHub/synth-ai/examples/multi_step/configs/crafter_eval_text_only_groq_qwen32b.toml
-
-
-uvx synth-ai train \
-  --type rl \
-  --config /Users/joshpurtell/Documents/GitHub/synth-ai/examples/multi_step/configs/crafter_rl_stepwise_hosted_judge.toml \
-  --task-url https://synth-laboratories--grpo-crafter-task-app-fastapi-app-dev.modal.run \
-  --backend https://synth-backend-dev-docker.onrender.com/api \
-  --env-file /Users/joshpurtell/Documents/GitHub/monorepo/environments/crafter/.env
-
-
-
----
-
-Verilog
-
-# 1. Deploy Verilog task app
-cd /Users/joshpurtell/Documents/GitHub/synth-ai && uvx synth-ai modal-serve grpo-verilog --name grpo-verilog-task-app --env-file /Users/joshpurtell/Documents/GitHub/monorepo/environments/verilog/.env
-
-# 2. Baseline eval using Synth backend (pre-training)
-uvx synth-ai eval --config /Users/joshpurtell/Documents/GitHub/synth-ai/examples/multi_step/configs/verilog_eval_synth_qwen4b.toml
-
-# 3. (Optional) External reference eval using Groq Qwen 32B
-uvx synth-ai eval --config /Users/joshpurtell/Documents/GitHub/synth-ai/examples/multi_step/configs/verilog_eval_groq_qwen32b.toml
-
-# 4. Deploy training backend
-cd /Users/joshpurtell/Documents/GitHub/monorepo && uv run modal deploy backend/app/routes/clustered_training/core/algorithms/gspo/app.py --env dev
-
-# 5. Run RL training
-uvx synth-ai train \
-  --type rl \
-  --config /Users/joshpurtell/Documents/GitHub/synth-ai/examples/multi_step/configs/verilog_rl_lora.toml \
-  --task-url https://synth-laboratories--grpo-verilog-task-app-fastapi-app-dev.modal.run \
-  --backend https://synth-backend-dev-docker.onrender.com/api \
-  --env-file /Users/joshpurtell/Documents/GitHub/monorepo/environments/verilog/.env
-
-# 6. Post-training eval (update job_id in config first!)
-# After training, note the job_id from logs (e.g., job_19a1823e56303de604f)
-# Update verilog_eval_synth_trained_qwen8b.toml with your job_id
-uvx synth-ai eval --config /Users/joshpurtell/Documents/GitHub/synth-ai/examples/multi_step/configs/verilog_eval_synth_trained_qwen8b.toml