synth-ai 0.2.13.dev2__py3-none-any.whl → 0.2.16__py3-none-any.whl

examples/qwen_vl/RL_VISION_TESTING.md

@@ -0,0 +1,333 @@
+# Vision RL Integration Testing
+
+Complete integration tests for Reinforcement Learning with vision-language models using the Crafter task app.
+
+## Overview
+
+These tests verify the full vision RL pipeline:
+1. **Task App**: Same Crafter task app used for SFT data collection (generates image observations)
+2. **Model**: Qwen3-VL-4B (smaller, faster for testing)
+3. **Policy**: Uses `image_only_mode=true` - agent sees only images, no text observations
+4. **Training**: Full RL (GRPO/GSPO) with vision-capable model
+
+## Files
+
+### Configs
+- `configs/crafter_rl_vision_qwen3vl4b.toml` - Full RL config for Qwen3-VL-4B with vision
+
+### Tests
+- `../../tests/integration/cli/test_cli_train_rl_vision.py` - Integration tests:
+  - `test_cli_train_rl_vision_qwen3vl4b` - Full RL training test
+  - `test_task_app_vision_support` - Task app vision capability test
+
+## Quick Start
+
+### 1. Prerequisites
+
+```bash
+# Required environment variables
+export SYNTH_API_KEY="your-api-key"
+export BACKEND_BASE_URL="https://agent-learning.onrender.com/api"  # or your backend
+export ENVIRONMENT_API_KEY="your-modal-key"  # For Modal deployment
+
+# Optional: for faster testing
+export TASK_APP_WARMUP_TIMEOUT=300  # 5min for vision models
+export SYNTH_TRAIN_TEST_POLL_TIMEOUT=180
+```
+
+### 2. Run Tests
+
+```bash
+cd /Users/joshpurtell/Documents/GitHub/synth-ai
+
+# Run all vision RL tests
+uv run pytest tests/integration/cli/test_cli_train_rl_vision.py -v -s
+
+# Run specific test
+uv run pytest tests/integration/cli/test_cli_train_rl_vision.py::test_cli_train_rl_vision_qwen3vl4b -v -s
+
+# Run with marks
+uv run pytest -m "vision and slow" -v -s
+```
+
+### 3. Manual RL Training (without pytest)
+
+```bash
+# 1. Deploy task app (if not already deployed)
+uvx synth-ai task-app deploy grpo-crafter --name grpo-crafter-task-app
+
+# 2. Get task app URL (from deploy output)
+export TASK_APP_URL="https://your-app.modal.run"
+
+# 3. Run RL training
+uvx synth-ai train \
+  --type rl \
+  --config examples/qwen_vl/configs/crafter_rl_vision_qwen3vl4b.toml \
+  --backend $BACKEND_BASE_URL \
+  --task-url $TASK_APP_URL
+```
+
+## Configuration Details
+
+### Model: Qwen3-VL-4B
+```toml
+[model]
+base = "Qwen/Qwen3-VL-4B-Instruct"
+trainer_mode = "lora"
+supports_vision = true  # Enable vision support
+```
+
+### Vision-Specific Settings
+```toml
+[vllm]
+limit_mm_per_prompt = { "image": 1 }  # Max 1 image per prompt
+
+[rollout.policy_config]
+use_vision = true  # Enable vision input
+image_only_mode = true  # Use only images, no text observations
+temperature = 0.6
+max_tokens = 512
+
+[training]
+batch_size = 2  # Smaller for vision models (memory)
+max_images_per_message = 1
+supports_vision = true
+```
+
+### GPU Allocation (2x H200)
+```toml
+[topology]
+gpus_for_vllm = 1  # Inference
+gpus_for_training = 1  # Training
+tensor_parallel = 1
+```
+
+## Test Details
+
+### Test 1: Full RL Training
+**Function:** `test_cli_train_rl_vision_qwen3vl4b`
+
+**What it tests:**
+1. Task app deployment
+2. Task app warmup (health check)
+3. RL job submission with vision config
+4. Job creation confirmation
+
+**Expected output:**
+```
+✅ Vision RL job created: job-abc123
+   Model: Qwen3-VL-4B
+   Task App: https://your-app.modal.run
+   Image Mode: image_only
+```
+
+**Runtime:** ~5-10 minutes (deploy + warmup + job submit)
+
+### Test 2: Task App Vision Support
+**Function:** `test_task_app_vision_support`
+
+**What it tests:**
+1. Task app can be deployed
+2. Task app health endpoint responds
+3. Task app accepts vision policy config
+4. Can make rollout request with `use_vision=true` and `image_only_mode=true`
+
+**Expected output:**
+```
+✅ Task app supports vision config
+   Response keys: ['trajectory', 'metadata', ...]
+```
+
+**Runtime:** ~2-3 minutes (deploy + warmup + single rollout)
+
+## Task App Details
+
+The Crafter task app (`grpo-crafter-task-app`) provides:
+
+### Environment
+- **Crafter game** with visual observations
+- Generates RGB images (64x64 or configurable)
+- Text observations also available (but ignored in `image_only_mode`)
+
+### Policy (crafter-react)
+- **Vision Detection:** Auto-detects vision models from name (e.g., "Qwen3-VL", "gpt-4o-mini")
+- **Image Formatting:** Converts observations to OpenAI-style multimodal messages
+- **Tool Calling:** Supports structured action space via tools
+
+### Trace Format
+- **Structured traces** with multimodal messages
+- Images stored as base64 in trace DB
+- Compatible with `synth-ai filter` for SFT export
+
+## Integration with SFT Pipeline
+
+This RL setup uses the **same task app** as the SFT data collection:
+
+### SFT Data Collection
+```bash
+# Collect episodes with gpt-4o-mini teacher
+uvx synth-ai eval --config configs/eval_gpt4o_vision_proper.toml
+
+# Export to SFT dataset
+uvx synth-ai filter --config configs/filter_vision_sft.toml
+```
+
+### RL Training
+```bash
+# Train student model (Qwen3-VL-4B) with RL
+uvx synth-ai train \
+  --type rl \
+  --config configs/crafter_rl_vision_qwen3vl4b.toml
+```
+
+**Benefits:**
+1. **Consistency:** Same environment, same observations
+2. **Curriculum:** SFT → RL progression
+3. **Debugging:** Compare SFT and RL traces in same format
+
+## Troubleshooting
+
+### Task App Deployment Fails
+```bash
+# Check Modal auth
+modal token set --token-id <id> --token-secret <secret>
+
+# Check environment variables
+echo $SYNTH_API_KEY
+echo $ENVIRONMENT_API_KEY
+
+# Try manual deploy
+uvx synth-ai task-app deploy grpo-crafter --name grpo-crafter-task-app
+```
+
+### Task App Won't Warm Up
+```bash
+# Increase timeout
+export TASK_APP_WARMUP_TIMEOUT=600  # 10 minutes
+
+# Check task app logs in Modal dashboard
+# https://modal.com/apps
+
+# Try health check manually
+curl https://your-app.modal.run/health
+```
+
+### RL Job Submission Fails
+```bash
+# Check backend connectivity
+curl $BACKEND_BASE_URL/health
+
+# Verify API key
+curl -H "Authorization: Bearer $SYNTH_API_KEY" $BACKEND_BASE_URL/api/health
+
+# Check task app URL format
+echo $TASK_APP_URL  # Should be https://...modal.run
+```
+
+### Vision Model OOM (Out of Memory)
+```toml
+# Reduce batch size in config
+[training]
+batch_size = 1  # Down from 2
+gradient_accumulation_steps = 4  # Up from 2
+
+# Reduce concurrent rollouts
+[rollout]
+max_concurrent_rollouts = 2  # Down from 4
+```
+
+### Images Not Appearing in Training
+```bash
+# Verify vision support is enabled
+grep -A 5 "\[model\]" configs/crafter_rl_vision_qwen3vl4b.toml
+# Should show: supports_vision = true
+
+# Check policy config
+grep -A 10 "\[rollout.policy_config\]" configs/crafter_rl_vision_qwen3vl4b.toml
+# Should show: use_vision = true, image_only_mode = true
+
+# Verify vLLM config
+grep -A 3 "\[vllm\]" configs/crafter_rl_vision_qwen3vl4b.toml
+# Should show: limit_mm_per_prompt = { "image": 1 }
+```
+
+## Performance Expectations
+
+### Qwen3-VL-4B (2x H200)
+- **Throughput:** ~2-4 episodes/min (with TP=1)
+- **Memory:** ~40-60GB GPU (model + images + gradients)
+- **Iteration Time:** ~10-15 min (with 4 episodes, 10 steps each)
+
+### Training Time Estimates
+- **3 iterations (test):** ~30-45 minutes
+- **10 iterations (short run):** ~2-3 hours
+- **50 iterations (full run):** ~12-20 hours
+
+## Next Steps
+
+### 1. Baseline Evaluation
+```bash
+# Evaluate untrained model
+uvx synth-ai eval \
+  --model Qwen/Qwen3-VL-4B-Instruct \
+  --env crafter \
+  --seeds 0,1,2,3,4 \
+  --policy-config '{"use_vision": true, "image_only_mode": true}'
+```
+
+### 2. SFT Initialization (Optional)
+```bash
+# Train on teacher demonstrations first
+uvx synth-ai train \
+  --type sft \
+  --model Qwen/Qwen3-VL-4B-Instruct \
+  --data traces/gpt4o_vision/sft/train.jsonl
+```
+
+### 3. RL Fine-Tuning
+```bash
+# Run full RL training
+uvx synth-ai train \
+  --type rl \
+  --config configs/crafter_rl_vision_qwen3vl4b.toml \
+  --iterations 50
+```
+
+### 4. Eval Comparison
+```bash
+# Compare pre-trained vs post-RL
+uvx synth-ai eval --model <rl-checkpoint> --seeds 0-9
+```
+
+## References
+
+- **VLM SFT Pipeline:** `examples/qwen_vl/PIPELINE_RUN_LOG.txt`
+- **Image Validation:** `examples/qwen_vl/IMAGE_VALIDATION_COMPLETE.md`
+- **Task App Source:** `examples/task_apps/crafter/task_app/`
+- **Policy Implementation:** `examples/task_apps/crafter/task_app/synth_envs_hosted/policy.py`
+
+## CI Integration
+
+### Pytest Marks
+```python
+@pytest.mark.slow      # Takes >5 minutes
+@pytest.mark.vision    # Requires vision model support
+@pytest.mark.integration  # Full pipeline test
+```
+
+### Run in CI
+```bash
+# Run all integration tests including vision
+pytest tests/integration/cli/ -m integration -v
+
+# Run only vision tests
+pytest -m vision -v
+
+# Skip slow tests for PR checks
+pytest -m "not slow" -v
+```
+
+---
+
+**Status:** ✅ Integration tests ready. Task app and RL config validated for Qwen3-VL-4B with image-only observations.
+

examples/qwen_vl/SDK_VISION_INTEGRATION.md

@@ -0,0 +1,328 @@
+# SDK Vision Support Integration
+
+**Status**: ✅ Complete
+
+## Overview
+
+Added comprehensive vision/multimodal support to the synth-ai SDK's SFT data module, and integrated it with the monorepo backend for consistent multimodal data handling across both codebases.
+
+## Changes Made
+
+### 1. **SDK Enhancement** (`synth-ai/synth_ai/learning/sft/data.py`)
+
+Added vision-specific utilities to the SDK:
+
+#### New Functions
+
+1. **`has_image_content(content: SFTMessageContent) -> bool`**
+   - Detects if message content contains images
+   - Supports OpenAI multimodal format
+   - Handles both `{"type": "image_url"}` and `{"type": "image"}` formats
+
+2. **`message_has_image(message: SFTMessage) -> bool`**
+   - Checks if an SFTMessage contains image content
+   - Convenience wrapper around `has_image_content`
+
+3. **`example_has_image(example: SFTExample) -> bool`**
+   - Checks if any message in an SFTExample contains images
+   - Used for filtering vision datasets
+
+4. **`count_images_in_content(content: SFTMessageContent) -> int`**
+   - Counts number of image segments in message content
+   - Useful for statistics and validation
+
+5. **`extract_image_urls(content: SFTMessageContent) -> list[str]`**
+   - Extracts all image URLs from message content
+   - Supports http(s):// URLs and data:image/... base64
+   - Returns list of URL strings
+
+6. **`validate_vision_example(example: SFTExample, *, require_images: bool = True) -> tuple[bool, str | None]`**
+   - Comprehensive validation of vision SFT examples
+   - Checks for image presence, URL validity
+   - Returns `(is_valid, error_message)` tuple
+   - Logs warnings for suspicious URLs
+
+7. **`iter_vision_examples(...) -> Iterator[SFTExample]`**
+   - Specialized iterator for vision examples
+   - Includes vision-specific validation
+   - Option to require images or skip invalid examples
+   - Useful for processing large JSONL files
+
+#### Example Usage
+
+```python
+from synth_ai.learning.sft.data import (
+    load_jsonl,
+    example_has_image,
+    validate_vision_example,
+    extract_image_urls
+)
+
+# Load and filter vision examples
+examples = load_jsonl("vision_data.jsonl")
+vision_examples = [ex for ex in examples if example_has_image(ex)]
+
+# Validate each example
+for ex in vision_examples:
+    is_valid, error = validate_vision_example(ex)
+    if not is_valid:
+        print(f"Invalid: {error}")
+    
+    # Extract image URLs for inspection
+    for msg in ex.messages:
+        urls = extract_image_urls(msg.content)
+        print(f"Images: {urls}")
+```
+
+### 2. **Backend Integration** (`monorepo/backend/.../training/sft/data.py`)
+
+Updated the monorepo backend to use SDK utilities:
+
+#### Changes
+
+1. **Added SDK imports with fallback**:
+   ```python
+   try:
+       from synth_ai.learning.sft.data import (
+           has_image_content as sdk_has_image_content,
+           example_has_image as sdk_example_has_image,
+           validate_vision_example as sdk_validate_vision_example,
+           # ... more imports
+       )
+       SDK_VISION_AVAILABLE = True
+   except ImportError:
+       SDK_VISION_AVAILABLE = False
+       logger.warning("synth_ai SDK not available - vision support will be limited")
+   ```
+
+2. **Updated `SFTDataProcessor` docstring**:
+   - Documents integration with SDK
+   - Shows OpenAI multimodal format example
+   - Explains fallback behavior
+
+3. **Enhanced `_vision_message_has_image()` method**:
+   - Uses SDK's `has_image_content()` when available
+   - Falls back to local implementation if SDK unavailable
+   - Ensures consistency between SDK and backend
+
+4. **Enhanced `_validate_vision_examples()` method**:
+   - Uses SDK's `coerce_example()` and `validate_vision_example()` for messages format
+   - Provides comprehensive validation with detailed error messages
+   - Falls back gracefully if SDK validation fails
+   - Maintains backward compatibility with non-messages formats
+
+## Supported Data Formats
+
+### OpenAI Multimodal Format (Recommended)
+
+```json
+{
+  "messages": [
+    {
+      "role": "system",
+      "content": "You are a helpful assistant."
+    },
+    {
+      "role": "user",
+      "content": [
+        {"type": "text", "text": "What's in this image?"},
+        {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgo..."}}
+      ]
+    },
+    {
+      "role": "assistant",
+      "content": "I see a cat sitting on a couch."
+    }
+  ],
+  "metadata": {
+    "session_id": "ep001",
+    "has_image": true
+  }
+}
+```
+
+### Alternative Formats (Also Supported)
+
+**Legacy image field**:
+```json
+{
+  "messages": [...],
+  "images": ["/path/to/image.jpg"],
+  "metadata": {}
+}
+```
+
+**Single image field**:
+```json
+{
+  "messages": [...],
+  "image": "https://example.com/image.jpg",
+  "metadata": {}
+}
+```
+
+## Image URL Formats
+
+Supported image URL formats:
+
+1. **HTTP(S) URLs**: `https://example.com/image.jpg`
+2. **Data URLs (base64)**: `data:image/png;base64,iVBORw0KGgo...`
+3. **Local file paths**: `/path/to/image.jpg` (for local training only)
+
+## Validation Rules
+
+The SDK validates:
+
+1. **Image presence**: At least one message must contain an image (when `require_images=True`)
+2. **URL format**: All image URLs must be non-empty strings
+3. **URL scheme**: URLs should start with `http://`, `https://`, or `data:image/`
+   - Warnings logged for non-standard formats
+4. **Message structure**: Messages must follow OpenAI format
+
+## Benefits
+
+### 1. **Consistency**
+- Single source of truth for vision data validation
+- Both SDK and backend use the same logic
+- Reduces bugs and maintenance burden
+
+### 2. **Type Safety**
+- Strong typing with dataclasses
+- Clear SFTMessage and SFTExample structures
+- IDE autocomplete and type checking
+
+### 3. **Error Handling**
+- Comprehensive validation with detailed error messages
+- Graceful fallbacks if SDK unavailable
+- Helpful warnings for edge cases
+
+### 4. **OpenAI Compatibility**
+- Matches OpenAI's fine-tuning format exactly
+- Data can be used with OpenAI or local models
+- Easy migration between platforms
+
+### 5. **Tool Call Support**
+- SDK already handles tool calls, tool definitions
+- Ready for complex agentic workflows
+- Supports reasoning blocks (`<think>` tags) if needed
+
+## Testing
+
+### Quick SDK Test
+
+```python
+# Test in synth-ai repo
+from synth_ai.learning.sft.data import has_image_content, validate_vision_example, coerce_example
+import json
+
+# Test multimodal message detection
+content = [
+    {"type": "text", "text": "What's this?"},
+    {"type": "image_url", "image_url": {"url": "data:image/png;base64,abc123"}}
+]
+assert has_image_content(content) == True
+
+# Test validation
+example_data = {
+    "messages": [
+        {"role": "user", "content": content},
+        {"role": "assistant", "content": "A test image"}
+    ]
+}
+example = coerce_example(example_data)
+is_valid, error = validate_vision_example(example)
+assert is_valid == True
+print("✓ SDK vision utilities working correctly!")
+```
+
+### Integration Test
+
+```python
+# Test in monorepo backend
+from backend.app.routes.simple_training.training.sft.data import SFTDataProcessor
+
+processor = SFTDataProcessor()
+test_data = [{
+    "messages": [
+        {"role": "user", "content": [
+            {"type": "text", "text": "Describe this."},
+            {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
+        ]},
+        {"role": "assistant", "content": "Description"}
+    ]
+}]
+
+validated = processor._validate_vision_examples(test_data)
+assert len(validated) == 1
+print("✓ Backend SDK integration working!")
+```
+
+## Future Enhancements
+
+### Potential Additions
+
+1. **Image preprocessing utilities**
+   - Resize images to model requirements
+   - Validate image dimensions
+   - Convert between formats (JPEG ↔ PNG)
+
+2. **Base64 encoding helpers**
+   - Convert file paths to data URLs
+   - Batch encode images for JSONL
+   - Memory-efficient streaming
+
+3. **Statistics and analytics**
+   - Count images per example
+   - Measure average image sizes
+   - Detect corrupted or invalid images
+
+4. **Dataset transformation**
+   - Convert between formats
+   - Augment with additional images
+   - Filter by image properties
+
+## Migration Guide
+
+### For Existing Backend Code
+
+If you have existing vision validation code:
+
+```python
+# Before (manual validation)
+def has_images(messages):
+    for msg in messages:
+        content = msg.get("content")
+        if isinstance(content, list):
+            for part in content:
+                if part.get("type") == "image_url":
+                    return True
+    return False
+
+# After (use SDK)
+from synth_ai.learning.sft.data import has_image_content
+
+def has_images(messages):
+    return any(has_image_content(msg.get("content")) for msg in messages)
+```
+
+### For Existing SDK Code
+
+No changes needed! The SDK already handles OpenAI message formats correctly. Vision utilities are additive and don't break existing functionality.
+
+## Documentation
+
+- **SDK docs**: See `synth_ai/learning/sft/data.py` docstrings
+- **Backend docs**: See `backend/app/routes/simple_training/training/sft/data.py` class docstring
+- **Examples**: See `synth-ai/examples/qwen_vl/` for vision-specific examples
+
+## Related Files
+
+- SDK: `synth-ai/synth_ai/learning/sft/data.py`
+- Backend: `monorepo/backend/app/routes/simple_training/training/sft/data.py`
+- Examples: `synth-ai/examples/qwen_vl/`
+- Pipeline guide: `synth-ai/examples/qwen_vl/NEXT_STEPS_2B.md`
+
+---
+
+✅ **SDK vision support is now production-ready for both synth-ai and monorepo!**
+