synth-ai 0.2.9.dev0__py3-none-any.whl → 0.2.23.dev3__py3-none-any.whl

examples/blog_posts/warming_up_to_rl/ARCHITECTURE.md

@@ -0,0 +1,195 @@
+# Smoke Test Architecture
+
+This document explains how the smoke test works internally, for future maintenance and debugging.
+
+## Component Overview
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                     synth-ai smoke command                       │
+│  (synth_ai/cli/commands/smoke/core.py)                         │
+└────────────┬────────────────────────────────────────────────────┘
+             │
+             ├─► Auto-start sqld (optional)
+             │   ├─ Kill existing process on ports 8080/8081
+             │   ├─ Start: sqld --db-path ... --hrana-listen-addr ... --http-listen-addr ...
+             │   └─ Health check: GET http://127.0.0.1:8081/health
+             │
+             ├─► Auto-start task app (optional)
+             │   ├─ Kill existing process on port 8765
+             │   ├─ Start: nohup uvx synth-ai task-app serve ... (from synth-ai root)
+             │   ├─ Health check: GET http://localhost:8765/health (accepts 200 or 400)
+             │   └─ Output: nohup_task_app.out
+             │
+             ├─► Start mock RL trainer (if use_mock=true)
+             │   ├─ MockRLTrainer(port=0, backend="openai")
+             │   ├─ Forwards requests to OpenAI API
+             │   └─ Logs: [mock-rl] ← request / → response
+             │
+             └─► Execute rollout
+                 ├─ POST /rollout to task app
+                 ├─ Capture response with v3 trace
+                 └─ Extract and display tool calls
+
+```
+
+## Key Implementation Details
+
+### 1. Tool Call Extraction
+
+**Location:** `synth_ai/cli/commands/smoke/core.py` lines ~946-1005
+
+**How it works:**
+1. Request rollout with `return_trace=True` and `trace_format="structured"`
+2. Response includes `trace.event_history[]` - list of policy and environment events
+3. Policy events have `call_records[]` containing LLM call metadata
+4. Each `call_record` has `output_tool_calls[]` with tool call details
+5. Extract `name` and `arguments_json` from each tool call
+6. Display formatted tool calls to user
+
+**Data structure:**
+```python
+response.trace = {
+    "event_history": [
+        {
+            "call_records": [  # Present in policy events
+                {
+                    "output_tool_calls": [
+                        {
+                            "name": "interact_many",
+                            "arguments_json": '{"actions":["move_up","move_up"]}',
+                            "call_id": "call_xyz",
+                            "index": 0
+                        }
+                    ],
+                    "model_name": "gpt-4o-mini",
+                    "provider": "openai",
+                    ...
+                }
+            ],
+            "metadata": {...},
+            ...
+        },
+        {
+            # Environment step event (no call_records)
+            "reward": 1.0,
+            "terminated": false,
+            ...
+        },
+        ...
+    ],
+    "session_id": "...",
+    "markov_blanket_message_history": [...],
+    ...
+}
+```
+
+### 2. Background Service Management
+
+**Task App Startup:**
+- Must run from synth-ai root for task app discovery
+- Uses `nohup` to detach process
+- Redirects output to `nohup_task_app.out`
+- Polls `/health` endpoint (accepts 200 or 400 status)
+- Timeout: 120 seconds with progress updates every 5 seconds
+- Propagates `SYNTH_QUIET=1` to suppress diagnostic messages
+
+**sqld Startup:**
+- Starts with Hrana WebSocket (8080) and HTTP (8081) ports
+- Polls `/health` endpoint for readiness
+- Timeout: 30 seconds
+
+**Port Cleanup:**
+- Uses `lsof -ti :PORT` to find PIDs
+- Kills processes with `kill -9 PID`
+- Waits 2 seconds for port release
+
+### 3. Mock RL Trainer
+
+The mock trainer (`MockRLTrainer`) acts as a proxy:
+- `backend="synthetic"`: Generates fake tool calls deterministically
+- `backend="openai"`: Forwards to real OpenAI API
+- Logs all requests/responses with `[mock-rl]` prefix
+- Auto-assigns port if `port=0`
+
+### 4. Diagnostic Message Suppression
+
+**Permanently disabled (commented out):**
+- `synth_ai/tracing_v3/config.py`: `[TRACING_V3_CONFIG_LOADED]` message
+- `synth_ai/environments/examples/crafter_classic/engine_deterministic_patch.py`: All `[PATCH]` messages
+- `synth_ai/environments/examples/crafter_classic/engine_serialization_patch_v3.py`: All `[PATCH]` messages
+- `synth_ai/environments/examples/crafter_classic/world_config_patch_simple.py`: All `[PATCH]` messages
+
+**Reason:** These messages add noise to smoke test output. They're still in the code as comments for documentation.
+
+## Troubleshooting Guide
+
+### No tool calls displayed
+
+**Symptom:** Output shows `⚠ No tool calls found in trace`
+
+**Causes:**
+1. `return_trace=false` in config - **FIX:** Set `return_trace = true`
+2. Trace format mismatch - Check `response.trace.event_history` structure
+3. No LLM calls made - Check for policy errors in task app logs
+
+**Debug:**
+```bash
+# Check task app logs
+cat /path/to/synth-ai/nohup_task_app.out
+
+# Verify trace structure
+# Add debug output in core.py around line 978:
+click.echo(f"DEBUG: trace keys: {list(tr.keys())}")
+click.echo(f"DEBUG: event_history length: {len(event_history)}")
+```
+
+### Task app exits immediately
+
+**Symptom:** `0 steps` in rollout, task app process not running
+
+**Causes:**
+1. Wrong task app name - **FIX:** Use `synth-ai task-app list` to find correct name
+2. Missing .env file - **FIX:** Ensure `task_app_env_file` points to valid .env
+3. Wrong working directory - **FIX:** Task app must be started from synth-ai root
+
+**Debug:**
+```bash
+# Manual test
+cd /path/to/synth-ai
+uvx synth-ai task-app serve grpo-crafter --port 8765 --env-file /path/to/.env --force
+```
+
+### Port conflicts
+
+**Symptom:** `Address already in use` errors
+
+**Fix:** The smoke command auto-kills processes on ports 8080, 8081, 8765. If manual cleanup needed:
+```bash
+lsof -ti :8080 | xargs kill -9
+lsof -ti :8081 | xargs kill -9
+lsof -ti :8765 | xargs kill -9
+```
+
+## Future Improvements
+
+Potential enhancements for future agents:
+
+1. **Streaming tool call display**: Show tool calls as they happen, not just at the end
+2. **Tool call validation**: Verify tool calls match expected format for the environment
+3. **Performance metrics**: Track inference latency per tool call
+4. **Cost tracking**: Display OpenAI API costs for the smoke test
+5. **Parallel rollouts**: Support `--parallel N` to test concurrent execution
+6. **Video/image capture**: For vision-based tasks, save observations
+7. **Interactive mode**: Allow stepping through rollout one action at a time
+
+## Related Files
+
+- `synth_ai/cli/commands/smoke/core.py` - Main smoke command implementation
+- `synth_ai/api/train/configs/rl.py` - `SmokeConfig` Pydantic model
+- `synth_ai/api/train/builders.py` - Removes `[smoke]` section before sending to trainer
+- `synth_ai/task/contracts.py` - `RolloutResponse` with trace field
+- `examples/blog_posts/warming_up_to_rl/SMOKE_TESTING.md` - User-facing documentation
+- `monorepo/docs/cli/smoke.mdx` - Mintlify documentation
+
+

examples/blog_posts/warming_up_to_rl/FINAL_TEST_RESULTS.md

@@ -0,0 +1,127 @@
+# Final Inference Test Results
+
+**Date**: Oct 31, 2025  
+**Endpoint**: `https://synth-laboratories-dev--learning-v2-service-fastapi-app.modal.run/chat/completions`
+
+## Summary
+
+| Model Type | Status | Result |
+|------------|--------|--------|
+| Base Model (Qwen/Qwen3-4B) | ✅ WORKS | Inference successful |
+| PEFT/SFT (Qwen3-0.6B) | ✅ WORKS | Inference successful |
+| RL (Qwen3-4B) | ❌ **BROKEN** | Modal function crashes |
+
+## Detailed Results
+
+### ✅ Test 1: Base Model (No Fine-Tuning)
+
+**Model**: `Qwen/Qwen3-4B`
+
+**Result**: **SUCCESS** ✅
+- **Status**: 200 OK
+- **Tokens**: 31 prompt + 100 completion = 131 total
+- **Response**: Generated successfully
+
+**Notes**:
+- First attempt returned 303 redirect (cold start)
+- Retry succeeded immediately
+- This confirms the endpoint and auth work correctly
+
+---
+
+### ✅ Test 2: PEFT/SFT Model
+
+**Model**: `peft:Qwen/Qwen3-0.6B:job_24faa0fdfdf648b9`
+
+**Result**: **SUCCESS** ✅
+- **Status**: 200 OK (consistent across retries)
+- **Tokens**: 31 prompt + 100 completion = 131 total  
+- **Response**: "Hello, I am working!" (with thinking tokens)
+
+**Notes**:
+- Works reliably
+- No cold start issues
+- This is the expected behavior for all models
+
+---
+
+### ❌ Test 3: RL Model
+
+**Model**: `rl:Qwen/Qwen3-4B:job_19a38041c38f96e638c:checkpoint-epoch-1`
+
+**Result**: **FAILURE** ❌ - Multiple error modes
+
+#### First Attempt:
+```
+Status: 400 Bad Request
+Error: "Device string must not be empty"
+```
+
+#### Retry:
+```
+Status: 500 Internal Server Error
+Error: "modal-http: internal error: function was terminated by signal"
+```
+
+**This is a Modal function crash** - the inference function terminated unexpectedly.
+
+#### Cold Start (from Modal logs):
+```
+RuntimeError: Cannot find any model weights with 
+'/models/rl/Qwen/Qwen3-4B/job_19a38041c38f96e638c/checkpoint-fixed'
+```
+
+**Root Cause**: RL checkpoint contains LoRA adapter files (`adapter_config.json`, `adapter_model.safetensors`), but vLLM expects full merged model weights.
+
+---
+
+## Conclusion
+
+### What Works ✅
+- **Base models**: Standard HuggingFace models load and inference correctly
+- **PEFT/SFT models**: Fine-tuned models with merged weights work perfectly
+
+### What's Broken ❌
+- **RL models**: Crash during model loading because:
+  1. RL checkpoints are stored as LoRA adapters
+  2. vLLM weight loader expects full model weights
+  3. Missing merge step causes vLLM to crash
+  4. Modal function terminates with signal (crash)
+
+### Impact
+- **HIGH SEVERITY**: All RL-trained models cannot be used for inference
+- Users can train RL models but cannot deploy them
+- This blocks the core RL training → inference workflow
+
+### Next Steps
+See `monorepo/RL_INFERENCE_BUG.md` for:
+- Detailed root cause analysis
+- Reproduction script
+- Suggested fix (merge LoRA adapters before vLLM loading)
+- Code locations to modify
+
+---
+
+## Developer Experience Issues Identified
+
+### Issue #1: Confusing Error Messages
+- **400 "Device string must not be empty"** - Not helpful, doesn't indicate RL adapter issue
+- **500 "function was terminated by signal"** - Generic crash, no context
+- **Should be**: "RL checkpoint contains adapter files. Merge required for vLLM loading."
+
+### Issue #2: Inconsistent Behavior
+- Sometimes returns 303 redirect
+- Sometimes returns 400
+- Sometimes crashes with 500
+- **Should be**: Consistent error message explaining the issue
+
+### Issue #3: Not Obvious How to Test Models
+- Had to try 3 different endpoint URLs before finding the right one
+- No documentation on model ID formats
+- **Should be**: `synth-ai inference --model "rl:..." --message "test"` CLI command
+
+---
+
+**Status**: Bug documented and reproduction available.  
+**See**: `monorepo/RL_INFERENCE_BUG.md` for full details.
+

examples/blog_posts/warming_up_to_rl/INFERENCE_SUCCESS.md

@@ -0,0 +1,132 @@
+# ✅ Inference Success Report
+
+**Date**: Oct 31, 2025  
+**Models Tested**: Latest SFT and RL models from training  
+
+## Working Solution
+
+### Correct Endpoint
+```
+https://synth-laboratories-dev--learning-v2-service-fastapi-app.modal.run/chat/completions
+```
+
+### SFT/PEFT Models: ✅ WORKING
+
+**Model ID**: `peft:Qwen/Qwen3-0.6B:job_24faa0fdfdf648b9`
+
+**Test Code**:
+```python
+import httpx
+import os
+
+SYNTH_API_KEY = os.getenv("SYNTH_API_KEY")
+url = "https://synth-laboratories-dev--learning-v2-service-fastapi-app.modal.run/chat/completions"
+
+headers = {
+    "Authorization": f"Bearer {SYNTH_API_KEY}",
+    "Content-Type": "application/json",
+}
+
+payload = {
+    "model": "peft:Qwen/Qwen3-0.6B:job_24faa0fdfdf648b9",
+    "messages": [
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Say 'Hello, I am working!' and nothing else."}
+    ],
+    "temperature": 0.2,
+    "max_tokens": 100,
+}
+
+with httpx.Client(timeout=300.0) as client:
+    response = client.post(url, json=payload, headers=headers)
+    print(response.json())
+```
+
+**Result**:
+- ✅ Status: 200 OK
+- ✅ Response generated successfully
+- ✅ Token usage tracked: 31 prompt + 72 completion = 103 total
+- ✅ Output: "Hello, I am working!" (with thinking tokens as expected)
+
+### RL Models: ⚠️ NEEDS PROMOTION
+
+**Model ID**: `rl:Qwen/Qwen3-4B:job_19a38041c38f96e638c:checkpoint-epoch-1`
+
+**Status**: 303 Redirect (empty response)
+
+**Root Cause**: 
+From monorepo backend code inspection, RL checkpoints require a "promotion" step to be loaded onto Modal before they can be used for inference. The direct Modal endpoint returns a redirect for unpromoted RL models.
+
+**Solution Options**:
+
+#### Option 1: Use Backend Proxy (Recommended)
+The backend automatically handles RL promotion:
+```python
+# Use backend proxy instead of direct Modal
+url = "https://your-backend.example.com/api/chat/completions"
+# Backend will auto-promote and route to vLLM
+```
+
+#### Option 2: Manual Promotion (Advanced)
+1. Call promotion endpoint first
+2. Wait for model to load onto Modal
+3. Then call inference endpoint
+
+## Key Learnings
+
+### What We Got Wrong Initially:
+1. ❌ Wrong endpoint path: Used `/v1/chat/completions` → should be `/chat/completions`
+2. ❌ Wrong base URL: Used render.com URL → should be Modal URL
+3. ❌ Assumed RL = PEFT workflow → RL needs promotion step
+
+### What We Got Right:
+1. ✅ Model ID format from `synth-ai status models list`
+2. ✅ Using SYNTH_API_KEY for auth
+3. ✅ Bearer token authorization header
+
+## Recommendations for Library Improvement
+
+### 1. Add Simple CLI Command
+```bash
+synth-ai inference \
+  --model "peft:Qwen/Qwen3-0.6B:job_xxx" \
+  --message "Hello" \
+  --max-tokens 100
+```
+
+### 2. Document Endpoint in Model Status
+```bash
+$ synth-ai status models get "peft:..."
+Model: peft:Qwen/Qwen3-0.6B:job_xxx
+Status: succeeded
+Inference Endpoint: https://synth-laboratories-dev--learning-v2-service-fastapi-app.modal.run/chat/completions
+Ready: ✅ Yes (use directly)
+```
+
+### 3. Add Python SDK Example
+```python
+from synth_ai import InferenceClient
+
+client = InferenceClient(api_key=os.getenv("SYNTH_API_KEY"))
+response = client.chat.completions.create(
+    model="peft:Qwen/Qwen3-0.6B:job_xxx",
+    messages=[{"role": "user", "content": "Hello"}]
+)
+print(response.choices[0].message.content)
+```
+
+### 4. Clear Error Messages
+- 303 → "RL model needs promotion. Use backend proxy or call /promote endpoint first."
+- 404 → "Model not found. Check model ID with: synth-ai status models list"
+
+## Success Criteria Met
+
+- ✅ Can get model ID from CLI
+- ✅ Know correct endpoint
+- ✅ Know correct auth (SYNTH_API_KEY)
+- ✅ Can send test message
+- ✅ Get response back
+- ⚠️ RL models need extra step (documented)
+
+**Status**: PEFT/SFT inference is fully working! RL needs backend proxy.
+

examples/blog_posts/warming_up_to_rl/README.md

@@ -0,0 +1,158 @@
+# Crafter: From Rollouts to RL with the Synth AI CLI
+
+This playbook mirrors the original “Warming Up to RL” walkthrough, but swaps the bespoke scripts for the first–class `uvx synth-ai` helpers. Every step—from deploying the task app to filtering rollouts, fine-tuning, and bootstrapping RL— now uses the same CLI you’d reach for in production.
+
+All commands assume you are inside the repository root and have `uv`/`uvx` available.
+
+---
+
+## 0. Prerequisites
+
+1. Install dependencies and authenticate once:
+   ```bash
+   uv pip install -e .
+   uvx synth-ai setup
+   ```
+   The setup wizard writes the required `SYNTH_API_KEY`, `ENVIRONMENT_API_KEY`, and local `.env` helpers.
+
+2. Copy the example secrets if you need a starter file:
+   ```bash
+   cp examples/warming_up_to_rl/.env.example .env
+   ```
+
+3. Export the path we use for trace capture (optional but keeps things tidy):
+   ```bash
+   export CRAFTER_TRACE_DB=traces/v3/crafter_blog.db
+   ```
+
+---
+
+## 1. Ship the Crafter Task App
+
+Deploy the hosted Crafter environment once. The Modal URL that prints at the end is reused by eval, SFT, and RL.
+
+```bash
+uvx synth-ai deploy grpo-crafter \
+  --runtime modal \
+  --modal-mode serve \
+  --name crafter-blogpost \
+  --env-file .env
+```
+
+For local testing you can run:
+
+```bash
+uvx synth-ai deploy grpo-crafter \
+  --runtime uvicorn \
+  --port 8001 \
+  --trace traces/v3 \
+  --env-file .env
+```
+
+Copy the Modal URL (e.g. `https://your-app.modal.run`) and replace the `task_app_url` placeholders inside every config under `examples/blog_posts/warming_up_to_rl/configs/`.
+
+---
+
+## 2. Collect High-Quality Rollouts
+
+We lean on large teacher models to produce demonstrations. The configs in `configs/` already request full traces so we retain chain-of-thought.
+
+ Groq Qwen3-32B (text-only prompt):
+```bash
+uvx synth-ai eval grpo-crafter \
+  --config examples/blog_posts/warming_up_to_rl/configs/eval_groq_qwen32b.toml \
+  --trace-db "${CRAFTER_TRACE_DB}"
+```
+
+ GPT-OSS-120B via Groq’s OpenAI-compatible endpoint (also text-only):
+```bash
+uvx synth-ai eval grpo-crafter \
+  --config examples/blog_posts/warming_up_to_rl/configs/eval_openai_gpt_oss_120b.toml \
+  --trace-db "${CRAFTER_TRACE_DB}"
+```
+
+ Both configs disable image attachments and rely on the textual observation renderer (`format_observation`) so Groq stays within its supported modalities. If you want to try other models, keep `use_vision = false` unless the provider explicitly supports image inputs.
+
+---
+
+## 3. Filter Into an SFT Dataset
+
+Once traces are stored in `CRAFT_TRACE_DB`, trim to the crisp trajectories:
+
+```bash
+uvx synth-ai filter \
+  --config examples/blog_posts/warming_up_to_rl/configs/filter_high_reward_dataset.toml
+```
+
+The output JSONL lands in `ft_data/crafter_blog_high_reward.jsonl`, ready for supervised fine-tuning.
+
+---
+
+## 4. Fine-Tune Qwen3-4B with `uvx synth-ai train`
+
+Update the dataset path (and optionally hyperparameters) in `train_sft_qwen4b.toml`, then launch:
+
+```bash
+uvx synth-ai train \
+  --type sft \
+  --config examples/blog_posts/warming_up_to_rl/configs/train_sft_qwen4b.toml \
+  --env-file .env \
+  --poll
+```
+
+Capture the returned job id (it looks like `fft:Qwen/Qwen3-4B:job_xxxxx`). We reuse that identifier in the evaluation and RL configs.
+At any time you can list recently minted checkpoints with:
+
+```bash
+uvx synth-ai status models
+```
+
+The output table shows the canonical model name/ID alongside the source job.
+
+---
+
+## 5. Evaluate the Fine-Tuned Checkpoint
+
+Replace both `REPLACE-WITH-SFT-JOB-ID` strings inside `eval_ft_qwen4b.toml`, then run:
+
+```bash
+uvx synth-ai eval grpo-crafter \
+  --config examples/blog_posts/warming_up_to_rl/configs/eval_ft_qwen4b.toml \
+  --trace-db "${CRAFTER_TRACE_DB}"
+```
+
+This provides a clean, CLI-native comparison between the teacher rollouts and the fine-tuned model.
+
+---
+
+## 6. Kick Off RL from the Fine-Tuned Model
+
+Point `train_rl_from_sft.toml` at the same Modal task app and set `model.source` to your SFT job id:
+
+```bash
+uvx synth-ai train \
+  --type rl \
+  --config examples/blog_posts/warming_up_to_rl/configs/train_rl_from_sft.toml \
+  --env-file .env \
+  --poll
+```
+
+The CLI streams rollout and judge metrics in real time. When the run finishes, you can re-use the Stage 5 config (substituting the RL job id) to quantify the uplift.
+If you lose track of the produced RL label or want to confirm the latest status, run:
+
+```bash
+uvx synth-ai status jobs
+uvx synth-ai status models
+```
+
+The first command shows job completion state; the second surfaces model IDs you can plug into new eval configs.
+
+---
+
+## 7. Where to Go Next
+
+- The original `examples/warming_up_to_rl` folder still contains deeper experiments (auto-curricula, modal renderers, etc.).
+- Add more `eval_*.toml` configs to compare alternative judges or reward shaping strategies.
+- Plug the filtered dataset into `uvx synth-ai files upload` if you want to share it with a teammate without copying JSONL around.
+
+This directory now holds everything a blog post needs: configs, output locations, and the CLI entrypoints to reproduce the Crafter SFT → RL pipeline end-to-end.