synth-ai 0.2.13.dev2__py3-none-any.whl → 0.2.16__py3-none-any.whl

examples/qwen_vl/BUGS_AND_FIXES.md

@@ -0,0 +1,232 @@
+# Vision SFT Pipeline - Bugs and Fixes
+
+Complete log of issues encountered and resolved during vision data collection setup.
+
+## ✅ Issue #1: Import Error - CrafterEnvironment
+
+**Problem:**
+```python
+ImportError: cannot import name 'CrafterEnvironment' from 'examples.task_apps.crafter.task_app.synth_envs_hosted.envs.crafter.environment'
+```
+
+**Root Cause:**  
+Class is named `CrafterEnvironmentWrapper`, not `CrafterEnvironment`
+
+**Fix:**  
+Updated imports and usages in:
+- `crafter_gpt5nano_agent.py`
+- `crafter_qwen_vl_agent.py`
+- `collect_vision_traces.py`
+
+```python
+# Before
+from ...environment import CrafterEnvironment
+wrapper = CrafterEnvironment(env, seed=seed)
+
+# After
+from ...environment import CrafterEnvironmentWrapper
+wrapper = CrafterEnvironmentWrapper(env, seed=seed)
+```
+
+**Status:** FIXED ✓
+
+---
+
+## ✅ Issue #2: OpenAI API Parameter - max_tokens
+
+**Problem:**
+```
+openai.BadRequestError: Error code: 400 - {'error': {'message': "Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead."}}
+```
+
+**Root Cause:**  
+gpt-5 models require `max_completion_tokens` parameter instead of `max_tokens`
+
+**Fix:**  
+Updated `_normalise_openai_request()` function to detect gpt-5 models:
+
+```python
+def _normalise_openai_request(payload, model, temperature):
+    request = dict(payload)
+    request["model"] = model
+    
+    # gpt-5 models use max_completion_tokens, not max_tokens
+    if "gpt-5" in model.lower():
+        request.setdefault("max_completion_tokens", 512)
+        request.pop("max_tokens", None)  # Remove if present
+    else:
+        # Older models use max_tokens
+        request.setdefault("max_tokens", 512)
+    
+    return request
+```
+
+**Files Updated:**
+- `crafter_gpt5nano_agent.py`
+- `collect_vision_traces.py`
+
+**Status:** FIXED ✓
+
+---
+
+## ✅ Issue #3: OpenAI API Parameter - temperature
+
+**Problem:**
+```
+openai.BadRequestError: Error code: 400 - {'error': {'message': "Unsupported value: 'temperature' does not support 0.6 with this model. Only the default (1) value is supported."}}
+```
+
+**Root Cause:**  
+gpt-5-nano only supports `temperature=1` (default), custom temperature values are not allowed
+
+**Fix:**  
+Remove temperature parameter for gpt-5 models:
+
+```python
+def _normalise_openai_request(payload, model, temperature):
+    # ...
+    
+    if "gpt-5" in model.lower():
+        # gpt-5-nano only supports temperature=1 (default)
+        request.pop("temperature", None)  # Remove custom temperature
+        request.setdefault("max_completion_tokens", 512)
+        request.pop("max_tokens", None)
+    else:
+        # Older models support custom temperature
+        request.setdefault("temperature", temperature)
+        request.setdefault("max_tokens", 512)
+    
+    return request
+```
+
+**Files Updated:**
+- `crafter_gpt5nano_agent.py`
+- `collect_vision_traces.py`
+
+**Status:** FIXED ✓
+
+---
+
+## ⚠️  Issue #4: gpt-5-nano Tool Calling Support
+
+**Problem:**
+```
+Seed 0: no tool calls returned by model; ending episode early at step 0.
+```
+
+**Root Cause:**  
+gpt-5-nano does not appear to support function/tool calling yet, or requires a different prompt format for tool use.
+
+**Testing Results:**
+- API returned 200 OK (auth and network fine)
+- Model processed vision inputs successfully
+- Model did not return tool calls even with tools schema provided
+- Both episodes stopped immediately (step 0)
+
+**Workaround:**  
+Switch to `gpt-4o-mini-2024-07-18` for data collection:
+- Confirmed to support both vision AND tool calling
+- Successfully completed 10 episodes with good quality
+- Mean 2.6 achievements per episode
+- 685 total tool calls across 10 episodes
+
+**Status:** WORKAROUND APPLIED (use gpt-4o-mini) ✓
+
+**Note:**  
+This is a model capability limitation, not a code bug. gpt-5-nano can be revisited when tool calling support is confirmed by OpenAI.
+
+---
+
+## 📊 Final Validation Results
+
+### Test Run #5: 10-Episode Collection with gpt-4o-mini
+
+**Command:**
+```bash
+uv run python examples/qwen_vl/crafter_gpt5nano_agent.py \
+  --model gpt-4o-mini-2024-07-18 \
+  --seeds 10 \
+  --steps 50
+```
+
+**Results:**
+```
+✓ All 10 episodes completed (50 steps each)
+✓ Mean achievements: 2.6 per episode
+✓ Total tool calls: 685
+✓ Vision processing: Working (64x64 PNG frames)
+✓ Tool calling: Working (proper tool call format)
+✓ Frame saving: Working (saved to output directory)
+✓ Performance: ~5-6 minutes for 10 episodes
+```
+
+**Quality Metrics:**
+- Episode 1: 4 achievements, 72 tool calls, reward: 97.3
+- Episode 5: 3 achievements, 62 tool calls, reward: 120.0
+- Episode 8: 1 achievement, 71 tool calls, reward: 12.9
+- Good variety in performance (1-4 achievements)
+
+---
+
+## 🔧 Code Changes Summary
+
+### Files Modified:
+1. **crafter_gpt5nano_agent.py**
+   - Import: `CrafterEnvironment` → `CrafterEnvironmentWrapper`
+   - Function: `_normalise_openai_request()` - handle gpt-5 parameters
+
+2. **crafter_qwen_vl_agent.py**
+   - Import: `CrafterEnvironment` → `CrafterEnvironmentWrapper`
+
+3. **collect_vision_traces.py**
+   - Import: `CrafterEnvironment` → `CrafterEnvironmentWrapper`
+   - Function: `_normalise_openai_request()` - handle gpt-5 parameters
+
+### Key Learnings:
+1. ✅ Always check actual class names in source code
+2. ✅ OpenAI's API evolves - newer models have different parameter requirements
+3. ✅ Test with known-working models first (gpt-4o-mini) before trying cutting-edge ones
+4. ✅ Vision + tool calling combo requires mature model support
+
+---
+
+## 🎯 Recommendations
+
+### For Production:
+- **Teacher model:** Use `gpt-4o-mini-2024-07-18` for data collection
+  - Proven to work with vision + tools
+  - Good quality (2-4 achievements per episode)
+  - Reasonable cost
+
+- **Monitor gpt-5-nano:** Revisit when tool calling support is confirmed
+
+### For Configs:
+- Update eval configs to use `gpt-4o-mini` by default:
+  ```toml
+  [eval]
+  model = "gpt-4o-mini-2024-07-18"  # Not gpt-5-nano
+  ```
+
+---
+
+## ✅ All Issues Resolved
+
+**Infrastructure Status:** READY FOR PRODUCTION ✓
+
+- Vision processing: Working
+- Tool calling: Working  
+- Frame saving: Working
+- OpenAI API integration: Working
+- 10-episode test: Successful
+
+**Next Steps:**
+1. Scale to 100 episodes for full dataset
+2. Apply filters and export to SFT format
+3. Train VLM with LoRA
+4. Fine-tune with RL
+
+---
+
+**Last Updated:** 2025-10-26  
+**Test Environment:** synth-ai dev, macOS, Python 3.11
+

examples/qwen_vl/IMAGE_VALIDATION_COMPLETE.md

@@ -0,0 +1,271 @@
+# Image Validation Implementation Complete ✅
+
+## Summary
+
+Added comprehensive validation for invalid/bogus image content in vision SFT data to catch errors **before**:
+1. Inference API calls (prevents wasted API costs on invalid requests)
+2. Training job submission (prevents hours of wasted GPU time)
+
+## What Was Done
+
+### 1. SDK Tests Added (11 new tests in `synth-ai/tests/unit/learning/test_sft_data.py`)
+
+**Invalid Image Content Tests:**
+- `test_validate_vision_example_empty_url` - Empty image URLs
+- `test_validate_vision_example_missing_url_field` - Missing URL field in image_url
+- `test_validate_vision_example_null_url` - Null URL values
+- `test_validate_vision_example_malformed_image_dict` - Malformed image dict structure
+- `test_validate_vision_example_non_string_url` - Non-string URL values (integers, etc.)
+- `test_validate_vision_example_whitespace_only_url` - Whitespace-only URLs
+- `test_validate_vision_example_invalid_scheme` - Invalid URL schemes (ftp://, etc.)
+- `test_validate_vision_example_multiple_invalid_urls` - Multiple invalid URLs
+- `test_validate_vision_example_mixed_valid_invalid` - Mix of valid and invalid (strict: fails)
+- `test_extract_image_urls_filters_invalid` - URL extraction filtering
+- `test_validate_vision_example_invalid_base64_format` - Malformed base64
+
+**Test Results:** ✅ 42/42 tests passing (6 existing + 25 reasoning + 11 invalid image)
+
+### 2. SDK Implementation Enhanced (`synth-ai/synth_ai/learning/sft/data.py`)
+
+#### `extract_image_urls()` - Now filters out:
+- Empty strings (`""`)
+- Whitespace-only strings (`"   "`)
+- Non-string values (`None`, integers, etc.)
+
+```python
+def extract_image_urls(content: SFTMessageContent) -> list[str]:
+    """Extract all image URLs from message content.
+    
+    Filters out invalid entries:
+    - Non-string URLs
+    - Empty strings
+    - Whitespace-only strings
+    ...
+    """
+    # Now checks: isinstance(url, str) and url.strip()
+```
+
+#### `validate_vision_example()` - Strict validation:
+- Counts image_url type entries vs valid URLs
+- **Fails if ANY image_url entry has invalid/missing URL**
+- Detects mismatches: `Has 2 image_url entries but only 1 valid URLs`
+- Warns about suspicious schemes (non-http/https/data:image)
+
+```python
+# If we have image_url type entries but fewer valid URLs, some are invalid
+if len(urls) < image_type_count:
+    return False, f"Message {i}: Has {image_type_count} image_url entries but only {len(urls)} valid URLs"
+```
+
+### 3. Monorepo Integration (Automatic)
+
+**SFT Training** (`monorepo/backend/app/routes/simple_training/training/sft/data.py`):
+- Already uses `sdk_validate_vision_example()` at line 401-406
+- Automatically gets stricter validation
+- Logs warnings and skips invalid examples:
+  ```python
+  is_valid, error = sdk_validate_vision_example(sdk_example, require_images=True)
+  if not is_valid:
+      logger.warning("Vision example %s failed validation: %s", idx, error)
+      continue  # Skip invalid example
+  ```
+
+**Inference** (`monorepo/backend/app/routes/simple_training/modal_service/gpu_functions.py`):
+- Uses `_validate_inference_request()` at line 3827-3856
+- Currently validates structure but **NOT image content**
+- **TODO: Add image validation to prevent API failures**
+
+## Validation Catches
+
+### ❌ Rejected Examples:
+```json
+{
+  "messages": [
+    {
+      "role": "user",
+      "content": [
+        {"type": "text", "text": "What's this?"},
+        {"type": "image_url", "image_url": {"url": ""}}  // Empty!
+      ]
+    }
+  ]
+}
+```
+**Error:** `"Message 0: Has 1 image_url entries but only 0 valid URLs (some are empty, null, or missing)"`
+
+```json
+{
+  "messages": [
+    {
+      "role": "user",
+      "content": [
+        {"type": "image_url", "image_url": {}}  // Missing url field
+      ]
+    }
+  ]
+}
+```
+**Error:** `"Message 0: Has 1 image_url entries but only 0 valid URLs"`
+
+```json
+{
+  "messages": [
+    {
+      "role": "user",
+      "content": [
+        {"type": "image_url", "image_url": {"url": "https://valid.jpg"}},
+        {"type": "image_url", "image_url": {"url": "   "}}  // Whitespace!
+      ]
+    }
+  ]
+}
+```
+**Error:** `"Message 0: Has 2 image_url entries but only 1 valid URLs"`
+
+### ✅ Accepted Examples:
+```json
+{
+  "messages": [
+    {
+      "role": "user",
+      "content": [
+        {"type": "text", "text": "Describe this"},
+        {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}
+      ]
+    },
+    {"role": "assistant", "content": "A beautiful image"}
+  ]
+}
+```
+
+```json
+{
+  "messages": [
+    {
+      "role": "user",
+      "content": [
+        {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgo..."}}
+      ]
+    }
+  ]
+}
+```
+
+## Benefits
+
+### For SFT Training:
+1. **Early Detection:** Invalid examples caught during data preparation, not after hours of training
+2. **Clear Errors:** Specific messages like "Has 2 image_url entries but only 1 valid URLs"
+3. **Cost Savings:** Prevents wasted GPU time on datasets with invalid images
+4. **Data Quality:** Ensures all training examples have valid image content
+
+### For Inference:
+1. **API Cost Savings:** Prevents sending invalid requests to OpenAI/Groq/etc.
+2. **Faster Failures:** Fail-fast before network call, not after timeout
+3. **Better Error Messages:** User knows exactly what's wrong with their image data
+
+## Testing
+
+### Run SDK tests:
+```bash
+cd /Users/joshpurtell/Documents/GitHub/synth-ai
+uv run pytest tests/unit/learning/test_sft_data.py -v
+
+# Just invalid image tests:
+uv run pytest tests/unit/learning/test_sft_data.py -k "empty_url or missing_url or null_url or malformed or non_string or whitespace or invalid_scheme or multiple_invalid or mixed_valid or filters_invalid or invalid_base64" -v
+```
+
+### Test with actual data:
+```python
+from synth_ai.learning.sft.data import coerce_example, validate_vision_example
+
+# This will fail validation:
+example_data = {
+    "messages": [
+        {
+            "role": "user",
+            "content": [
+                {"type": "text", "text": "Check this"},
+                {"type": "image_url", "image_url": {"url": ""}},  # Empty!
+            ],
+        },
+        {"role": "assistant", "content": "Response"},
+    ]
+}
+
+example = coerce_example(example_data)
+is_valid, error = validate_vision_example(example, require_images=True)
+print(f"Valid: {is_valid}, Error: {error}")
+# Output: Valid: False, Error: Message 0: Has 1 image_url entries but only 0 valid URLs...
+```
+
+## Next Steps
+
+### 1. Add Inference Validation (High Priority)
+Update `_validate_inference_request` to validate image content:
+
+```python
+# In monorepo/backend/app/routes/simple_training/modal_service/gpu_functions.py
+
+def _validate_inference_request(request: Dict[str, Any]) -> List[Dict[str, Any]]:
+    """Validate inference request and return messages."""
+    # ... existing validation ...
+    
+    # NEW: Validate image content if present
+    if SDK_SFT_AVAILABLE:
+        for i, msg in enumerate(messages):
+            content = msg.get("content")
+            if isinstance(content, list):
+                # Check for image_url entries
+                has_images = any(
+                    isinstance(item, dict) and item.get("type") in {"image", "image_url"}
+                    for item in content
+                )
+                if has_images:
+                    urls = sdk_extract_image_urls(content)
+                    image_count = sum(
+                        1 for item in content
+                        if isinstance(item, dict) and item.get("type") in {"image", "image_url"}
+                    )
+                    if len(urls) < image_count:
+                        raise ValueError(
+                            f"Message {i}: Has {image_count} image entries but only {len(urls)} valid URLs"
+                        )
+    
+    return messages
+```
+
+### 2. Add API-Level Validation
+Add validation in backend API routes before forwarding to Modal.
+
+### 3. Integration Tests
+Add integration tests that verify rejected examples at the API level.
+
+## Files Modified
+
+### SDK:
+- `synth-ai/synth_ai/learning/sft/data.py` - Enhanced validation logic
+- `synth-ai/tests/unit/learning/test_sft_data.py` - Added 11 invalid image tests
+
+### Monorepo:
+- No changes needed - automatically uses enhanced SDK validation in SFT training
+- **TODO:** Add validation to `monorepo/backend/app/routes/simple_training/modal_service/gpu_functions.py`
+
+## Related Issues Prevented
+
+### Without this validation:
+1. **Training Job Failures:** Hours into training, discover dataset has empty image URLs
+2. **API Errors:** Send requests with invalid base64, get 400 errors from OpenAI
+3. **Silent Failures:** Model trained on text-only when images expected
+4. **Cost Waste:** GPU time and API calls on invalid data
+
+### With this validation:
+1. **Immediate Feedback:** Know within seconds if data is invalid
+2. **Clear Error Messages:** Exactly which message and what's wrong
+3. **Confidence:** All training/inference data has been validated
+4. **Cost Savings:** Never waste resources on bogus data
+
+---
+
+**Status:** ✅ SDK validation complete and tested. Monorepo SFT training automatically protected. Inference validation recommended as next step.
+