synth-ai 0.2.16__py3-none-any.whl → 0.2.19__py3-none-any.whl

examples/blog_posts/warming_up_to_rl/INFERENCE_SUCCESS.md

@@ -0,0 +1,132 @@
+# ✅ Inference Success Report
+
+**Date**: Oct 31, 2025  
+**Models Tested**: Latest SFT and RL models from training  
+
+## Working Solution
+
+### Correct Endpoint
+```
+https://synth-laboratories-dev--learning-v2-service-fastapi-app.modal.run/chat/completions
+```
+
+### SFT/PEFT Models: ✅ WORKING
+
+**Model ID**: `peft:Qwen/Qwen3-0.6B:job_24faa0fdfdf648b9`
+
+**Test Code**:
+```python
+import httpx
+import os
+
+SYNTH_API_KEY = os.getenv("SYNTH_API_KEY")
+url = "https://synth-laboratories-dev--learning-v2-service-fastapi-app.modal.run/chat/completions"
+
+headers = {
+    "Authorization": f"Bearer {SYNTH_API_KEY}",
+    "Content-Type": "application/json",
+}
+
+payload = {
+    "model": "peft:Qwen/Qwen3-0.6B:job_24faa0fdfdf648b9",
+    "messages": [
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Say 'Hello, I am working!' and nothing else."}
+    ],
+    "temperature": 0.2,
+    "max_tokens": 100,
+}
+
+with httpx.Client(timeout=300.0) as client:
+    response = client.post(url, json=payload, headers=headers)
+    print(response.json())
+```
+
+**Result**:
+- ✅ Status: 200 OK
+- ✅ Response generated successfully
+- ✅ Token usage tracked: 31 prompt + 72 completion = 103 total
+- ✅ Output: "Hello, I am working!" (with thinking tokens as expected)
+
+### RL Models: ⚠️ NEEDS PROMOTION
+
+**Model ID**: `rl:Qwen/Qwen3-4B:job_19a38041c38f96e638c:checkpoint-epoch-1`
+
+**Status**: 303 Redirect (empty response)
+
+**Root Cause**: 
+From monorepo backend code inspection, RL checkpoints require a "promotion" step to be loaded onto Modal before they can be used for inference. The direct Modal endpoint returns a redirect for unpromoted RL models.
+
+**Solution Options**:
+
+#### Option 1: Use Backend Proxy (Recommended)
+The backend automatically handles RL promotion:
+```python
+# Use backend proxy instead of direct Modal
+url = "https://your-backend.example.com/api/chat/completions"
+# Backend will auto-promote and route to vLLM
+```
+
+#### Option 2: Manual Promotion (Advanced)
+1. Call promotion endpoint first
+2. Wait for model to load onto Modal
+3. Then call inference endpoint
+
+## Key Learnings
+
+### What We Got Wrong Initially:
+1. ❌ Wrong endpoint path: Used `/v1/chat/completions` → should be `/chat/completions`
+2. ❌ Wrong base URL: Used render.com URL → should be Modal URL
+3. ❌ Assumed RL = PEFT workflow → RL needs promotion step
+
+### What We Got Right:
+1. ✅ Model ID format from `synth-ai status models list`
+2. ✅ Using SYNTH_API_KEY for auth
+3. ✅ Bearer token authorization header
+
+## Recommendations for Library Improvement
+
+### 1. Add Simple CLI Command
+```bash
+synth-ai inference \
+  --model "peft:Qwen/Qwen3-0.6B:job_xxx" \
+  --message "Hello" \
+  --max-tokens 100
+```
+
+### 2. Document Endpoint in Model Status
+```bash
+$ synth-ai status models get "peft:..."
+Model: peft:Qwen/Qwen3-0.6B:job_xxx
+Status: succeeded
+Inference Endpoint: https://synth-laboratories-dev--learning-v2-service-fastapi-app.modal.run/chat/completions
+Ready: ✅ Yes (use directly)
+```
+
+### 3. Add Python SDK Example
+```python
+from synth_ai import InferenceClient
+
+client = InferenceClient(api_key=os.getenv("SYNTH_API_KEY"))
+response = client.chat.completions.create(
+    model="peft:Qwen/Qwen3-0.6B:job_xxx",
+    messages=[{"role": "user", "content": "Hello"}]
+)
+print(response.choices[0].message.content)
+```
+
+### 4. Clear Error Messages
+- 303 → "RL model needs promotion. Use backend proxy or call /promote endpoint first."
+- 404 → "Model not found. Check model ID with: synth-ai status models list"
+
+## Success Criteria Met
+
+- ✅ Can get model ID from CLI
+- ✅ Know correct endpoint
+- ✅ Know correct auth (SYNTH_API_KEY)
+- ✅ Can send test message
+- ✅ Get response back
+- ⚠️ RL models need extra step (documented)
+
+**Status**: PEFT/SFT inference is fully working! RL needs backend proxy.
+

examples/blog_posts/warming_up_to_rl/README.md

@@ -0,0 +1,158 @@
+# Crafter: From Rollouts to RL with the Synth AI CLI
+
+This playbook mirrors the original “Warming Up to RL” walkthrough, but swaps the bespoke scripts for the first–class `uvx synth-ai` helpers. Every step—from deploying the task app to filtering rollouts, fine-tuning, and bootstrapping RL— now uses the same CLI you’d reach for in production.
+
+All commands assume you are inside the repository root and have `uv`/`uvx` available.
+
+---
+
+## 0. Prerequisites
+
+1. Install dependencies and authenticate once:
+   ```bash
+   uv pip install -e .
+   uvx synth-ai setup
+   ```
+   The setup wizard writes the required `SYNTH_API_KEY`, `ENVIRONMENT_API_KEY`, and local `.env` helpers.
+
+2. Copy the example secrets if you need a starter file:
+   ```bash
+   cp examples/warming_up_to_rl/.env.example .env
+   ```
+
+3. Export the path we use for trace capture (optional but keeps things tidy):
+   ```bash
+   export CRAFTER_TRACE_DB=traces/v3/crafter_blog.db
+   ```
+
+---
+
+## 1. Ship the Crafter Task App
+
+Deploy the hosted Crafter environment once. The Modal URL that prints at the end is reused by eval, SFT, and RL.
+
+```bash
+uvx synth-ai deploy grpo-crafter \
+  --runtime modal \
+  --modal-mode serve \
+  --name crafter-blogpost \
+  --env-file .env
+```
+
+For local testing you can run:
+
+```bash
+uvx synth-ai deploy grpo-crafter \
+  --runtime uvicorn \
+  --port 8001 \
+  --trace traces/v3 \
+  --env-file .env
+```
+
+Copy the Modal URL (e.g. `https://your-app.modal.run`) and replace the `task_app_url` placeholders inside every config under `examples/blog_posts/warming_up_to_rl/configs/`.
+
+---
+
+## 2. Collect High-Quality Rollouts
+
+We lean on large teacher models to produce demonstrations. The configs in `configs/` already request full traces so we retain chain-of-thought.
+
+ Groq Qwen3-32B (text-only prompt):
+```bash
+uvx synth-ai eval grpo-crafter \
+  --config examples/blog_posts/warming_up_to_rl/configs/eval_groq_qwen32b.toml \
+  --trace-db "${CRAFTER_TRACE_DB}"
+```
+
+ GPT-OSS-120B via Groq’s OpenAI-compatible endpoint (also text-only):
+```bash
+uvx synth-ai eval grpo-crafter \
+  --config examples/blog_posts/warming_up_to_rl/configs/eval_openai_gpt_oss_120b.toml \
+  --trace-db "${CRAFTER_TRACE_DB}"
+```
+
+ Both configs disable image attachments and rely on the textual observation renderer (`format_observation`) so Groq stays within its supported modalities. If you want to try other models, keep `use_vision = false` unless the provider explicitly supports image inputs.
+
+---
+
+## 3. Filter Into an SFT Dataset
+
+Once traces are stored in `CRAFT_TRACE_DB`, trim to the crisp trajectories:
+
+```bash
+uvx synth-ai filter \
+  --config examples/blog_posts/warming_up_to_rl/configs/filter_high_reward_dataset.toml
+```
+
+The output JSONL lands in `ft_data/crafter_blog_high_reward.jsonl`, ready for supervised fine-tuning.
+
+---
+
+## 4. Fine-Tune Qwen3-4B with `uvx synth-ai train`
+
+Update the dataset path (and optionally hyperparameters) in `train_sft_qwen4b.toml`, then launch:
+
+```bash
+uvx synth-ai train \
+  --type sft \
+  --config examples/blog_posts/warming_up_to_rl/configs/train_sft_qwen4b.toml \
+  --env-file .env \
+  --poll
+```
+
+Capture the returned job id (it looks like `fft:Qwen/Qwen3-4B:job_xxxxx`). We reuse that identifier in the evaluation and RL configs.
+At any time you can list recently minted checkpoints with:
+
+```bash
+uvx synth-ai status models
+```
+
+The output table shows the canonical model name/ID alongside the source job.
+
+---
+
+## 5. Evaluate the Fine-Tuned Checkpoint
+
+Replace both `REPLACE-WITH-SFT-JOB-ID` strings inside `eval_ft_qwen4b.toml`, then run:
+
+```bash
+uvx synth-ai eval grpo-crafter \
+  --config examples/blog_posts/warming_up_to_rl/configs/eval_ft_qwen4b.toml \
+  --trace-db "${CRAFTER_TRACE_DB}"
+```
+
+This provides a clean, CLI-native comparison between the teacher rollouts and the fine-tuned model.
+
+---
+
+## 6. Kick Off RL from the Fine-Tuned Model
+
+Point `train_rl_from_sft.toml` at the same Modal task app and set `model.source` to your SFT job id:
+
+```bash
+uvx synth-ai train \
+  --type rl \
+  --config examples/blog_posts/warming_up_to_rl/configs/train_rl_from_sft.toml \
+  --env-file .env \
+  --poll
+```
+
+The CLI streams rollout and judge metrics in real time. When the run finishes, you can re-use the Stage 5 config (substituting the RL job id) to quantify the uplift.
+If you lose track of the produced RL label or want to confirm the latest status, run:
+
+```bash
+uvx synth-ai status jobs
+uvx synth-ai status models
+```
+
+The first command shows job completion state; the second surfaces model IDs you can plug into new eval configs.
+
+---
+
+## 7. Where to Go Next
+
+- The original `examples/warming_up_to_rl` folder still contains deeper experiments (auto-curricula, modal renderers, etc.).
+- Add more `eval_*.toml` configs to compare alternative judges or reward shaping strategies.
+- Plug the filtered dataset into `uvx synth-ai files upload` if you want to share it with a teammate without copying JSONL around.
+
+This directory now holds everything a blog post needs: configs, output locations, and the CLI entrypoints to reproduce the Crafter SFT → RL pipeline end-to-end.

examples/blog_posts/warming_up_to_rl/SMOKE_TESTING.md

@@ -0,0 +1,164 @@
+# Smoke Testing Your Task App
+
+This guide shows how to quickly test your task app using the `synth-ai smoke` command with auto-start features.
+
+## Quick Start
+
+The easiest way to smoke test is using the `[smoke]` section in your RL config:
+
+```bash
+cd examples/blog_posts/warming_up_to_rl
+uv run synth-ai smoke --config configs/smoke_test.toml
+```
+
+**That's it!** The smoke command will:
+1. ✅ Auto-start sqld server for tracing (if `sqld_auto_start = true`)
+2. ✅ Auto-start your task app on port 8765 (if `task_app_name` is set)
+3. ✅ Run 10 rollout steps with `gpt-5-nano` using synthetic mocking
+4. ✅ Automatically stop all background services when done
+
+**Expected output:**
+```
+[smoke] sqld ready
+[smoke] Task app ready at http://localhost:8765 (status=400)
+[mock-rl] server ready http://127.0.0.1:51798 backend=synthetic
+>> POST /rollout run_id=smoke-... env=crafter policy=crafter-react
+[mock-rl] ← request backend=synthetic model=gpt-5-nano messages=2
+[mock-rl] → response tool_calls=1 backend=synthetic
+  rollout[0:0] episodes=1 steps=10 mean_return=1.0000
+✓ Smoke rollouts complete
+  successes=1/1 total_steps=10 v3_traces=1/1 nonzero_returns=1/1
+[smoke] Background services stopped
+```
+
+## Configuration
+
+Add a `[smoke]` section to your RL config:
+
+```toml
+[smoke]
+# Auto-start task app
+task_app_name = "grpo-crafter"
+task_app_port = 8765
+task_app_env_file = ".env"
+task_app_force = true
+
+# Auto-start sqld
+sqld_auto_start = true
+sqld_db_path = "./traces/local.db"
+sqld_hrana_port = 8080
+sqld_http_port = 8081
+
+# Test parameters
+max_steps = 10
+policy = "gpt-5-nano"
+mock_backend = "synthetic"  # or "openai" (requires valid OpenAI API key)
+return_trace = true
+```
+
+## Testing Methods
+
+### 1. Full Auto (Recommended)
+Everything auto-starts from config:
+```bash
+uv run synth-ai smoke --config configs/smoke_test.toml
+```
+
+### 2. Manual Task App + Auto sqld
+Start task app manually, auto-start sqld:
+```bash
+# Config with sqld_auto_start=true but no task_app_name
+uv run synth-ai smoke --config configs/my_config.toml --url http://localhost:8765
+```
+
+### 3. Override Config Settings
+Override any config value via CLI:
+```bash
+uv run synth-ai smoke --config configs/smoke_test.toml --max-steps 5
+```
+
+### 4. No Config (Manual Everything)
+```bash
+# Start services manually in separate terminals:
+# Terminal 1: sqld --db-path ./traces/local.db --hrana-listen-addr 127.0.0.1:8080 --http-listen-addr 127.0.0.1:8081
+# Terminal 2: uv run synth-ai task-app serve grpo-crafter --port 8765 --env-file .env --force
+
+# Terminal 3: Run smoke test
+uv run synth-ai smoke --url http://localhost:8765 \
+  --env-name crafter \
+  --policy-name crafter-react \
+  --max-steps 10 \
+  --policy mock \
+  --mock-backend openai
+```
+
+## Prerequisites
+
+### Install sqld (for tracing)
+```bash
+brew install sqld
+# or
+curl -fsSL https://get.turso.com/sqld | bash
+```
+
+### Verify Installation
+```bash
+which sqld
+# Should output: /opt/homebrew/bin/sqld or similar
+```
+
+## Common Issues
+
+### sqld not found
+If you see "sqld not found in PATH":
+```bash
+brew install sqld
+```
+
+### Port already in use
+Use `task_app_force = true` in config, or:
+```bash
+# Kill processes on ports 8080, 8081, 8765
+lsof -ti:8080,8081,8765 | xargs kill -9
+```
+
+### Task app not starting
+Check the error output - you may need:
+- Valid `.env` file with required keys
+- Correct task app name registered in your codebase
+
+## Example Output
+
+```
+[smoke] Loaded configuration from configs/smoke_test.toml
+[smoke] Config keys: task_app_name, task_app_port, sqld_auto_start, max_steps, policy
+[smoke] Starting sqld server...
+[smoke] DB path: /Users/you/project/traces/local.db
+[smoke] Hrana port: 8080, HTTP port: 8081
+[smoke] sqld ready
+[smoke] Starting task app 'grpo-crafter' on port 8765...
+[smoke] Task app ready at http://localhost:8765
+[smoke] Task app started, will use URL: http://localhost:8765
+[mock-rl] server ready http://127.0.0.1:52134 backend=openai
+>> POST /rollout run_id=smoke-abc123...
+  rollout[0:0] episodes=1 steps=20 mean_return=1.2500
+✓ Smoke rollouts complete
+  successes=1/1 total_steps=20 v3_traces=1/1 nonzero_returns=1/1
+[smoke] Stopping sqld...
+[smoke] Stopping task_app...
+[smoke] Background services stopped
+```
+
+## Next Steps
+
+Once smoke tests pass:
+1. Train your model: `uv run synth-ai train --type rl --config configs/your_config.toml`
+2. Check traces: Look in `./traces/` directory
+3. Monitor training: Use the Synth dashboard
+
+## Full Config Reference
+
+See [`configs/smoke_test.toml`](configs/smoke_test.toml) for a complete example.
+
+See [CLI Smoke Documentation](https://docs.usesynth.ai/cli/smoke) for all options.
+

examples/blog_posts/warming_up_to_rl/SMOKE_TEST_COMPLETE.md

@@ -0,0 +1,253 @@
+# Smoke Test Implementation - Complete
+
+## Summary
+
+The smoke test now provides **complete visibility into RL training rollouts**, including:
+
+✅ **Auto-start background services** (sqld, task app)  
+✅ **Real OpenAI inference** with gpt-4o-mini  
+✅ **Tool call display** - see every action the policy takes  
+✅ **Trace validation** - verify v3 trace format  
+✅ **Clean output** - all diagnostic noise suppressed  
+
+## Quick Start
+
+```bash
+cd examples/blog_posts/warming_up_to_rl
+uv run synth-ai smoke --config configs/smoke_test.toml
+```
+
+**Output shows:**
+- Service startup (sqld, task app)
+- Real-time inference requests  
+- **All 10 tool calls with arguments** (e.g., `interact_many({"actions":["move_up","move_up"]})`)
+- Rollout metrics (steps, returns, rewards)
+- Success validation
+
+## Documentation
+
+All documentation has been updated for future agents:
+
+### 1. User Documentation
+- **`SMOKE_TESTING.md`** - How to run smoke tests, what to expect
+- **`configs/smoke_test.toml`** - Well-commented example configuration
+- **`monorepo/docs/cli/smoke.mdx`** - Mintlify CLI documentation
+
+### 2. Developer Documentation  
+- **`ARCHITECTURE.md`** - Internal architecture, troubleshooting guide
+- **`synth_ai/cli/commands/smoke/core.py`** - Extensive inline comments explaining tool call extraction
+
+### 3. Code Comments
+
+**Tool Call Extraction (core.py lines 946-997):**
+```python
+# Extract and display tool calls from v3 trace
+# 
+# IMPORTANT: Tool calls are extracted from the structured v3 trace format.
+# The trace must be requested with return_trace=True for this to work.
+# 
+# Trace structure:
+#   trace.event_history[] - list of events (policy calls, env steps)
+#     ├─ event.call_records[] - LLM calls made during this event
+#        ├─ call_record.output_tool_calls[] - tool calls from LLM response
+#           ├─ tool_call.name - function name (e.g., "interact_many")
+#           └─ tool_call.arguments_json - JSON string of arguments
+```
+
+## Key Implementation Details
+
+### Tool Call Display
+
+**Requirements:**
+1. `return_trace = true` in config (CRITICAL - without this, no tool calls)
+2. v3 trace format (`trace_format="structured"`)
+3. Mock proxy or real inference (direct API calls don't populate traces correctly)
+
+**Data Flow:**
+```
+1. Rollout request with return_trace=True
+   ↓
+2. Task app makes LLM calls, captures responses
+   ↓
+3. LLM responses include tool_calls
+   ↓
+4. Task app stores call_records in event_history
+   ↓
+5. Smoke command extracts from trace.event_history[].call_records[].output_tool_calls[]
+   ↓
+6. Display: TOOL_CALL[N]: function_name({...args})
+```
+
+### Diagnostic Suppression
+
+**Permanently disabled (commented out, not deleted):**
+- `synth_ai/tracing_v3/config.py:21` - `[TRACING_V3_CONFIG_LOADED]`
+- `synth_ai/environments/examples/crafter_classic/engine_deterministic_patch.py` - All `[PATCH]` messages
+- `synth_ai/environments/examples/crafter_classic/engine_serialization_patch_v3.py` - All `[PATCH]` messages
+- `synth_ai/environments/examples/crafter_classic/world_config_patch_simple.py` - All `[PATCH]` messages
+
+**Why commented, not deleted?**
+- Preserves context for debugging
+- Shows what messages existed
+- Easy to re-enable if needed
+
+### Background Service Management
+
+**Task App:**
+- Runs from synth-ai root (required for discovery)
+- Uses `nohup` for detachment
+- Output → `nohup_task_app.out`
+- Health check accepts 200 or 400 (400 = server up, auth failing)
+- 120s timeout with progress updates
+
+**sqld:**
+- Dual ports: 8080 (Hrana WebSocket), 8081 (HTTP)
+- Health check: `GET http://127.0.0.1:8081/health`
+- 30s timeout
+- Auto-cleanup of existing processes
+
+## Configuration Reference
+
+### Critical Settings
+
+```toml
+[smoke]
+# Auto-start services
+task_app_name = "grpo-crafter"  # Task app to serve
+task_app_port = 8765
+task_app_env_file = ".env"      # Required for this app
+sqld_auto_start = true
+
+# Inference - REAL OpenAI
+model = "gpt-4o-mini"           # Actual model used
+mock_backend = "openai"         # Route through OpenAI API
+use_mock = true                 # Enable mock proxy
+
+# CRITICAL for tool call display
+return_trace = true             # Must be true!
+```
+
+### Optional Settings
+
+All `[smoke]` parameters are optional - CLI args override TOML values:
+
+```bash
+# Override max steps
+uv run synth-ai smoke --config configs/smoke_test.toml --max-steps 5
+
+# Use different model
+uv run synth-ai smoke --config configs/smoke_test.toml --model gpt-4o
+
+# Disable mock (use direct API - won't show tool calls properly)
+uv run synth-ai smoke --config configs/smoke_test.toml --no-mock
+```
+
+## Troubleshooting
+
+### No tool calls displayed
+
+**Symptom:** `⚠ No tool calls found in trace`
+
+**Solutions:**
+1. Verify `return_trace = true` in config
+2. Check `v3_traces=1/1` in output (should match successes)
+3. Ensure `use_mock = true` or using mock proxy
+4. Check task app logs: `cat /path/to/synth-ai/nohup_task_app.out`
+
+### Task app exits immediately
+
+**Symptom:** `0 steps`, process not running
+
+**Solutions:**
+1. Verify task app name: `synth-ai task-app list`
+2. Check .env file exists at `task_app_env_file` path
+3. Ensure running from correct directory
+4. Manual test: `cd /synth-ai && uvx synth-ai task-app serve grpo-crafter --port 8765 --env-file /path/.env --force`
+
+### Port conflicts
+
+**Symptom:** `Address already in use`
+
+**Solution:** Auto-cleanup should handle this, but manual cleanup:
+```bash
+lsof -ti :8080 | xargs kill -9
+lsof -ti :8081 | xargs kill -9
+lsof -ti :8765 | xargs kill -9
+```
+
+## Testing
+
+### Unit Tests
+
+- `tests/unit/test_train_validation.py::test_rl_config_with_smoke_section` - Validates `[smoke]` section parsing
+- `tests/unit/test_smoke_config.py` - Comprehensive Pydantic validation tests
+
+### Integration Test
+
+```bash
+cd examples/blog_posts/warming_up_to_rl
+uv run synth-ai smoke --config configs/smoke_test.toml
+```
+
+**Expected result:**
+- ✅ Services start successfully
+- ✅ 10 tool calls displayed
+- ✅ `v3_traces=1/1`
+- ✅ `successes=1/1`
+- ✅ `nonzero_returns=1/1`
+
+## Files Modified
+
+### Core Implementation
+- `synth_ai/cli/commands/smoke/core.py` - Tool call extraction, auto-start logic
+- `synth_ai/api/train/configs/rl.py` - `SmokeConfig` Pydantic model
+- `synth_ai/api/train/builders.py` - Remove `[smoke]` before sending to trainer
+
+### Diagnostic Suppression
+- `synth_ai/tracing_v3/config.py` - Commented out `[TRACING_V3_CONFIG_LOADED]`
+- `synth_ai/environments/examples/crafter_classic/engine_deterministic_patch.py` - Commented out `[PATCH]`
+- `synth_ai/environments/examples/crafter_classic/engine_serialization_patch_v3.py` - Commented out `[PATCH]`
+- `synth_ai/environments/examples/crafter_classic/world_config_patch_simple.py` - Commented out `[PATCH]`
+
+### Documentation
+- `examples/blog_posts/warming_up_to_rl/SMOKE_TESTING.md` - User guide
+- `examples/blog_posts/warming_up_to_rl/ARCHITECTURE.md` - Developer guide
+- `examples/blog_posts/warming_up_to_rl/configs/smoke_test.toml` - Example config
+- `examples/blog_posts/warming_up_to_rl/configs/train_rl_from_sft.toml` - Inline docs
+- `monorepo/docs/cli/smoke.mdx` - Mintlify CLI reference
+
+### Tests
+- `tests/unit/test_train_validation.py` - Added smoke section test
+- `tests/unit/test_smoke_config.py` - Comprehensive smoke config tests
+
+## Future Improvements
+
+Ideas for future agents:
+
+1. **Streaming display** - Show tool calls as they happen, not just at end
+2. **Tool call validation** - Verify format matches environment expectations
+3. **Performance metrics** - Track inference latency per call
+4. **Cost tracking** - Display OpenAI API costs
+5. **Parallel rollouts** - Support concurrent execution testing
+6. **Vision support** - Save observations for vision-based tasks
+7. **Interactive mode** - Step through rollout one action at a time
+8. **Replay mode** - Re-run saved traces for debugging
+
+## Success Criteria Met
+
+✅ **Tool calls visible** - All 10 calls displayed with arguments  
+✅ **Real inference** - OpenAI gpt-4o-mini executing actual tool calls  
+✅ **Clean output** - No diagnostic noise  
+✅ **Auto-start** - Background services managed automatically  
+✅ **Well documented** - Comprehensive docs for users and developers  
+✅ **Robust** - Error handling, health checks, timeouts  
+✅ **Tested** - Unit tests and working integration test  
+
+## Contact
+
+For questions or issues, see:
+- Architecture details: `ARCHITECTURE.md`
+- User guide: `SMOKE_TESTING.md`
+- CLI reference: `monorepo/docs/cli/smoke.mdx`
+
+