synth-ai 0.2.14__py3-none-any.whl → 0.4.4__py3-none-any.whl

examples/task_apps/crafter/README_IMAGE_ONLY_EVAL.md

@@ -1,316 +0,0 @@
-# Crafter Image-Only Evaluation Guide
-
-This guide shows you how to run Crafter evaluations with **image-only input** (no text observations) and save traces + rewards to **Turso database**.
-
-## Prerequisites
-
-1. **OpenAI API Key**: Set in your `.env` file
-2. **UV Package Manager**: Already installed if you can run `uv run`
-3. **Synth AI Repository**: Clone and set up per main README
-
-## Quick Start
-
-### 1. Run Image-Only Evaluation (10 Rollouts)
-
-```bash
-cd /Users/joshpurtell/Documents/GitHub/synth-ai
-
-# Set up environment for Turso tracing
-export TASKAPP_TRACING_ENABLED=1
-export TURSO_NATIVE=1
-export SQLD_DB_PATH="traces/v3/crafter_eval.db"
-
-# Run evaluation with image-only input
-uv run synth-ai eval grpo-crafter \
-  --config examples/task_apps/crafter/eval_image_only_gpt4o.toml
-```
-
-**Expected output**:
-- 10 rollouts complete
-- ~70% will earn achievements (collect_wood, collect_sapling, etc.)
-- All traces and rewards saved to `traces/v3/crafter_eval.db`
-
-### 2. Check Results
-
-```bash
-# View database
-ls -lh traces/v3/crafter_eval.db  # Should be ~1.7MB
-
-# Count sessions
-sqlite3 traces/v3/crafter_eval.db \
-  "SELECT COUNT(*) FROM session_traces;"
-
-# View all rollouts with rewards
-sqlite3 -header -column traces/v3/crafter_eval.db \
-  "SELECT 
-    json_extract(reward_metadata, '\$.env_seed') as seed,
-    total_reward, 
-    achievements_count,
-    json_extract(reward_metadata, '\$.final_achievements') as achievements
-   FROM outcome_rewards 
-   ORDER BY total_reward DESC;"
-```
-
-### 3. Query Non-Zero Rewards
-
-```bash
-# Get rollouts that earned achievements
-sqlite3 -header -column traces/v3/crafter_eval.db \
-  "SELECT 
-    session_id,
-    total_reward,
-    achievements_count,
-    json_extract(reward_metadata, '\$.final_achievements') as achievements
-   FROM outcome_rewards 
-   WHERE total_reward > 0
-   ORDER BY total_reward DESC;"
-```
-
-## Configuration File
-
-**Location**: `examples/task_apps/crafter/eval_image_only_gpt4o.toml`
-
-```toml
-[eval]
-app_id = "grpo-crafter"
-model = "gpt-4o-mini-2024-07-18"
-seeds = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]  # 10 rollouts
-max_turns = 10
-concurrency = 1
-env_name = "crafter"
-policy_name = "crafter-react"
-trace_format = "full"
-return_trace = true
-
-[eval.env_config]
-env_params = {max_steps_per_episode = 10}
-
-[eval.policy_config]
-provider = "openai"
-model = "gpt-4o-mini-2024-07-18"
-inference_url = "https://api.openai.com"
-temperature = 0.6
-top_p = 0.95
-max_tokens = 512
-use_vision = true           # Enable vision mode
-image_only_mode = true      # Send ONLY images (no text)
-max_llm_calls = 10
-```
-
-### Key Configuration Options
-
-| Option | Description | Values |
-|--------|-------------|--------|
-| `use_vision` | Enable vision/image input | `true` / `false` |
-| `image_only_mode` | Send only images (no text) | `true` / `false` |
-| `seeds` | Which seeds to run | Array of integers |
-| `max_turns` | Max policy calls per rollout | Integer (10-100) |
-| `concurrency` | Parallel rollouts | 1-5 recommended |
-
-## Customization
-
-### Run More Rollouts
-
-```toml
-# Change seeds to run more episodes
-seeds = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]  # 20 rollouts
-```
-
-### Increase Steps Per Episode
-
-```toml
-[eval.env_config]
-env_params = {max_steps_per_episode = 100}  # Longer episodes
-
-[eval.policy_config]
-max_llm_calls = 100
-```
-
-### Use Different Model
-
-```toml
-[eval]
-model = "gpt-4o-2024-08-06"  # Full GPT-4o
-
-[eval.policy_config]
-model = "gpt-4o-2024-08-06"
-```
-
-### Enable Text + Images (Multimodal)
-
-```toml
-[eval.policy_config]
-use_vision = true
-image_only_mode = false  # Send both text AND images
-```
-
-## Database Schema
-
-### outcome_rewards Table
-
-```sql
-CREATE TABLE outcome_rewards (
-    id INTEGER PRIMARY KEY AUTOINCREMENT,
-    session_id VARCHAR NOT NULL,
-    total_reward INTEGER NOT NULL,
-    achievements_count INTEGER NOT NULL,
-    total_steps INTEGER NOT NULL,
-    created_at DATETIME NOT NULL,
-    reward_metadata TEXT,  -- JSON with achievements, seed, etc.
-    FOREIGN KEY(session_id) REFERENCES session_traces(session_id)
-);
-```
-
-### Example Queries
-
-```sql
--- Get statistics
-SELECT 
-    COUNT(*) as total,
-    SUM(CASE WHEN total_reward > 0 THEN 1 ELSE 0 END) as with_rewards,
-    AVG(total_reward) as avg_reward,
-    MAX(total_reward) as max_reward
-FROM outcome_rewards;
-
--- Find best performers
-SELECT 
-    json_extract(reward_metadata, '$.env_seed') as seed,
-    total_reward,
-    achievements_count,
-    json_extract(reward_metadata, '$.final_achievements') as achievements
-FROM outcome_rewards 
-WHERE achievements_count >= 2
-ORDER BY total_reward DESC;
-
--- Join with session traces
-SELECT 
-    st.session_id,
-    st.created_at,
-    st.num_timesteps,
-    orw.total_reward,
-    orw.achievements_count
-FROM session_traces st
-INNER JOIN outcome_rewards orw ON st.session_id = orw.session_id
-WHERE orw.total_reward > 0
-ORDER BY orw.total_reward DESC;
-
--- Count by achievement level
-SELECT 
-    achievements_count,
-    COUNT(*) as num_rollouts,
-    AVG(total_reward) as avg_reward
-FROM outcome_rewards
-GROUP BY achievements_count
-ORDER BY achievements_count DESC;
-```
-
-## Typical Results
-
-**Expected Performance** (10 rollouts, 10 steps each, image-only):
-
-```
-Total rollouts: 10
-Rollouts with rewards: ~7 (70%)
-Average reward: ~1.3
-Max reward: ~3
-```
-
-**Common Achievements**:
-- `collect_wood` (most common)
-- `collect_sapling` (common)
-- `collect_drink` (rare in 10 steps)
-
-## Troubleshooting
-
-### No Database Created
-
-**Issue**: `traces/v3/crafter_eval.db` doesn't exist or is 0 bytes
-
-**Fix**: Ensure environment variables are set:
-```bash
-export TASKAPP_TRACING_ENABLED=1
-export TURSO_NATIVE=1
-export SQLD_DB_PATH="traces/v3/crafter_eval.db"
-```
-
-### 401 Unauthorized Error
-
-**Issue**: OpenAI API returns 401
-
-**Fix**: Check your `.env` file has valid `OPENAI_API_KEY`:
-```bash
-# .env file
-OPENAI_API_KEY=sk-proj-...your-key-here...
-```
-
-### All Rewards are Zero
-
-**Issue**: Agents aren't earning any achievements
-
-**Possible causes**:
-1. **Too few steps**: Increase `max_steps_per_episode` to 50-100
-2. **Image-only too hard**: Try `image_only_mode = false` for multimodal
-3. **Wrong model**: Try full GPT-4o instead of mini
-
-### Database Lock Errors
-
-**Issue**: `SQLITE_BUSY` or `database is locked`
-
-**Fix**: Reduce concurrency in config:
-```toml
-[eval]
-concurrency = 1  # Run sequentially
-```
-
-Or use Turso's MVCC mode (already enabled with `TURSO_NATIVE=1`).
-
-## Advanced: Export to CSV
-
-```bash
-# Export all rewards to CSV
-sqlite3 -header -csv traces/v3/crafter_eval.db \
-  "SELECT 
-    json_extract(reward_metadata, '$.env_seed') as seed,
-    total_reward,
-    achievements_count,
-    json_extract(reward_metadata, '$.final_achievements') as achievements
-   FROM outcome_rewards 
-   ORDER BY total_reward DESC" \
-  > crafter_rewards.csv
-```
-
-## Files Overview
-
-```
-examples/task_apps/crafter/
-├── eval_image_only_gpt4o.toml          # Config file
-├── EVAL_IMAGE_ONLY_RESULTS.md          # Example results
-├── QUERY_EXAMPLES.md                   # More SQL queries
-├── README_IMAGE_ONLY_EVAL.md           # This file
-└── task_app/
-    └── synth_envs_hosted/
-        ├── envs/crafter/
-        │   ├── policy.py                # Image-only mode logic
-        │   └── react_agent.py           # Message construction
-        ├── rollout.py                   # SessionTracer integration
-        └── inference/
-            └── openai_client.py         # API authentication
-```
-
-## See Also
-
-- `EVAL_IMAGE_ONLY_RESULTS.md` - Example run with detailed results
-- `QUERY_EXAMPLES.md` - More SQL query examples
-- `../../pokemon_red/README_IMAGE_ONLY_EVAL.md` - Pokemon Red version
-
-## Summary
-
-1. ✅ Set environment variables for Turso tracing
-2. ✅ Run `uv run synth-ai eval grpo-crafter --config ...`
-3. ✅ Check database: `traces/v3/crafter_eval.db`
-4. ✅ Query rewards: `SELECT * FROM outcome_rewards WHERE total_reward > 0`
-5. ✅ Customize config for different models/steps
-
-Enjoy running Crafter with vision-only input! 🎮
-
-

examples/task_apps/crafter/eval_image_only_gpt4o.toml

@@ -1,28 +0,0 @@
-# Evaluation config for Crafter with image-only input
-# This config uses GPT-4o mini with only image data (no text observations)
-
-[eval]
-app_id = "grpo-crafter-task-app"
-model = "gpt-4o-mini-2024-07-18"
-seeds = [0, 1]  # Quick test run
-max_turns = 10
-concurrency = 1  # Reduced to 1 to avoid database locks during testing
-env_name = "crafter"
-policy_name = "crafter-react"
-trace_format = "full"
-return_trace = true
-
-[eval.env_config]
-env_params = {max_steps_per_episode = 10}
-
-[eval.policy_config]
-provider = "openai"
-model = "gpt-4o-mini-2024-07-18"
-inference_url = "https://api.openai.com"  # Base URL (client will append /v1/chat/completions)
-temperature = 0.6
-top_p = 0.95
-max_tokens = 512
-use_vision = true
-image_only_mode = true
-max_llm_calls = 10
-

examples/task_apps/crafter/eval_text_only_groq_llama.toml

@@ -1,36 +0,0 @@
-# Evaluation config for Crafter with text-only input
-# This config uses Groq Qwen with only text observations (no images)
-
-[eval]
-app_id = "grpo-crafter-task-app"
-task_app_url = "https://synth-laboratories--grpo-crafter-task-app-fastapi-app-dev.modal.run"
-model = "qwen/qwen3-32b"
-seeds = [0, 1, 2]
-max_turns = 10
-concurrency = 1
-env_name = "crafter"
-policy_name = "crafter-react"
-trace_format = "full"
-return_trace = true
-
-[eval.env_config]
-env_params = {max_steps_per_episode = 10}
-
-[eval.policy_config]
-provider = "groq"
-model = "qwen/qwen3-32b"
-inference_url = "https://api.groq.com/openai/v1/chat/completions"
-temperature = 0.6
-top_p = 0.95
-max_tokens = 512
-use_vision = false
-image_only_mode = false
-max_llm_calls = 10
-
-[eval.judge]
-path = "examples/task_apps/crafter/judges/crafter_backend_judge.py"
-name = "Backend"
-backend_url = "http://localhost:8000/api"
-model = "openai/gpt-oss-120b"
-timeout_s = 45
-

examples/task_apps/crafter/filter_sft_dataset.toml

@@ -1,16 +0,0 @@
-# Filter config for Crafter image-only traces
-# Creates SFT dataset from rollouts with non-zero rewards
-
-[filter]
-# Input: Consolidated traces database
-db = "traces/v3/synth_ai.db"
-
-# Output: SFT-ready JSONL file
-output = "ft_data/crafter_image_only_sft.jsonl"
-
-# Filter for successful rollouts (those with achievements)
-min_official_score = 0.01  # Only include traces with rewards > 0
-
-# Optional: Limit number of examples (remove to get all)
-# limit = 10
-

examples/task_apps/crafter/task_app/README.md

@@ -1,42 +0,0 @@
-# Crafter Task App
-
-This example is now wired through the shared Synth task-app harness. Use the
-`uvx synth-ai` CLI to run it locally or deploy it to Modal without touching the
-underlying FastAPI plumbing.
-
-## Local development
-```bash
-uvx synth-ai serve grpo-crafter --port 8001
-# Optional extras:
-#   --env-file path/to/.env    # load additional environment variables
-#   --reload                   # enable uvicorn auto-reload
-```
-
-Useful endpoints while the server is running:
-- `GET http://localhost:8001/health`
-- `GET http://localhost:8001/info`
-- `GET http://localhost:8001/task_info?seed=42`
-- `POST http://localhost:8001/rollout`
-
-## Deploy to Modal
-```bash
-uvx synth-ai deploy grpo-crafter --name grpo-crafter-task-app
-```
-
-Requirements:
-- Modal CLI installed and authenticated (`modal token new`).
-- Either provide an `.env` with `ENVIRONMENT_API_KEY`, `GROQ_API_KEY`, and `OPENAI_API_KEY`
-  (recommended; pass via `--env-file`). The deploy command injects these values via an inline
-  Modal secret plus `Secret.from_dotenv`, so the minted environment key stays in sync with
-  what the CLI sends.
-- Or ensure Modal secrets `groq-api-key` and `openai-api-key` exist and continue to supply
-  model vendor credentials that way.
-
-The CLI generates a Modal entrypoint on the fly using the shared
-`TaskAppConfig`, ensuring the container matches the local FastAPI behavior.
-
-## Compatibility note
-`examples/warming_up_to_rl/task_app/grpo_crafter_task_app.py` remains as a
-legacy wrapper exposing `fastapi_app()` and a `__main__` entrypoint. Behind the
-scenes it proxies to the shared configuration; prefer the CLI workflow above
-for new automation and tests.

examples/task_apps/crafter/task_app/__init__.py

@@ -1,5 +0,0 @@
-"""Crafter task app implementation."""
-
-
-
-