synth-ai 0.2.9.dev2__py3-none-any.whl → 0.2.9.dev4__py3-none-any.whl

examples/rl/README.md

@@ -0,0 +1,169 @@
+# Math RL Demo (Single Step)
+
+This example trains a reinforcement learning policy on single-step math problems sourced from the [EleutherAI/math](https://huggingface.co/datasets/EleutherAI/math) dataset. Episodes consist of a single tool call: the model must emit a `math_submit` function call whose `answer` field contains the final solution. Missing or malformed tool calls receive negative reward; correct answers earn positive reward.
+
+## Quick Commands
+
+```bash
+# Serve locally with tracing
+uvx synth-ai serve math-single-step --port 8101 --env-file examples/rl/.env --trace traces/math
+
+# Modal deployment
+uvx synth-ai deploy --name synth-math-single-step --env-file examples/rl/.env
+
+# Evaluate base Qwen policy (validation split)
+uv run python examples/rl/run_eval.py --toml examples/rl/configs/eval_base_qwen.toml
+
+# Launch RL job from base model
+uvx synth-ai train --type rl --config examples/rl/configs/rl_from_base_qwen.toml
+
+# Evaluate RL checkpoint on held-out test split
+uv run python examples/rl/run_eval.py --toml examples/rl/configs/eval_rl_qwen.toml
+```
+
+## 1. Prerequisites
+
+- Python 3.11+
+- `uv`/`uvx`
+- Modal CLI (`modal token new`) for deployment
+- `.env` at `examples/rl/.env` containing at least:
+  - `SYNTH_API_KEY`
+  - `ENVIRONMENT_API_KEY`
+  - Optional: `TASK_APP_URL` (Modal URL), `GROQ_API_KEY`, `OPENAI_API_KEY`
+
+Run `uvx synth-ai setup` to populate the `.env` if you have not paired the SDK before.
+
+## 2. Task App
+
+The task app is defined in `synth_ai/task/apps/math_single_step.py` and registered as `math-single-step`. It loads problems from the Hugging Face dataset (configurable via `MATH_DATASET_*` env vars) and manages per-episode state with an in-memory environment manager.
+
+- **Observation**: single math problem (string) plus dataset metadata.
+- **Actions**: exactly one `math_submit` tool call with an `answer` string.
+- **Rewards**:
+  - `+1.0` for correct answer
+  - `0.0` for incorrect answer
+  - `-0.5` if the tool call omits an answer or uses the wrong tool
+  - `-1.0` when no tool call is provided
+
+Serve locally with tracing to capture trajectories:
+
+```bash
+uvx synth-ai serve math-single-step \
+  --port 8101 \
+  --env-file examples/rl/.env \
+  --trace traces/math \
+  --trace-db traces/math/synth_ai.db
+```
+
+Deploy or serve on Modal using the same env file; the registration includes a `ModalDeploymentConfig` that installs the `datasets` package automatically.
+
+## 3. Evaluation
+
+`examples/rl/run_eval.py` evaluates a policy by sampling deterministic seeds from the dataset splits. TOML configuration controls the model, split, and number of episodes. Example config (`eval_base_qwen.toml`):
+
+```toml
+provider = "synth"
+task_app_url = "http://localhost:8101"
+model = "Qwen/Qwen3-4B"
+split = "validation"
+num_episodes = 50
+seed_start = 0
+
+[policy]
+inference_url = "http://localhost:8000/api/inference"
+max_tokens = 128
+temperature = 0.0
+# Optional: override headers for inference requests
+# [policy.extra_headers]
+# Authorization = "Bearer ..."
+```
+
+The `[policy]` table maps directly to the inference payload; add `[policy.headers]` if you need to forward custom HTTP headers (e.g., `Authorization`). If `SYNTH_API_KEY` is present, the evaluator automatically sends `Authorization: Bearer <key>`.
+
+Set `--use-rollout` to exercise the server-side rollout endpoint instead of the per-step API.
+
+The script reports accuracy and a breakdown of failure modes (`missing_tool_call`, `blank_answer`, etc.).
+
+## 4. RL Training
+
+Example RL config (`configs/rl_from_base_qwen.toml`):
+
+```toml
+[services]
+task_url = "https://your-app.modal.run"
+
+[model]
+base = "Qwen/Qwen3-4B"
+
+[data]
+split = "train"
+seed_start = 0
+episodes_per_iteration = 2048
+
+[training]
+max_turns = 1
+ops = ["agent", "env"]
+batch_size = 128
+group_size = 1024
+reward_positive = 1.0
+reward_negative_no_tool = -1.0
+reward_negative_no_answer = -0.5
+
+[policy]
+model = "Qwen/Qwen3-4B"
+inference_url = "https://your-inference-host"
+max_tokens = 128
+temperature = 0.0
+
+[tags]
+experiment = "math_single_step"
+```
+
+Submit jobs interactively with:
+
+```bash
+uvx synth-ai train --type rl --config examples/rl/configs/rl_from_base_qwen.toml
+```
+
+The CLI ensures the task app is reachable (`/health`, `/task_info`), prompts for missing secrets, and polls job status until completion. For scripted automation, use `run_rl_and_save.py`:
+
+```bash
+uv run python examples/rl/run_rl_and_save.py \
+  --config examples/rl/configs/rl_from_base_qwen.toml \
+  --backend https://backend.synth.ai/api
+```
+
+## 5. Evaluating RL Outputs
+
+After training completes, set `model = "rl:<job_or_model_id>"` in `configs/eval_rl_qwen.toml` (and update `split = "test"` for a held-out set). Re-run `run_eval.py` to compare:
+
+```bash
+uv run python examples/rl/run_eval.py --toml examples/rl/configs/eval_rl_qwen.toml
+```
+
+Record both validation (pre-RL) and test (post-RL) accuracy to quantify improvements.
+
+## 6. Dataset Notes
+
+- By default the task app loads the [Hendrycks MATH benchmark](https://huggingface.co/datasets/nlile/hendrycks-MATH-benchmark). Override via `MATH_DATASET_NAME` / `MATH_DATASET_CONFIG` env vars if you want a different variant. The dataset is public and automatically downloaded when the task app starts; the server will fail fast with a clear error if it cannot be fetched.
+- For offline use, run `uv run python examples/rl/download_dataset.py --output-dir examples/rl/data --dataset nlile/hendrycks-MATH-benchmark --config algebra --limit 2000`. Then start the task app with `MATH_DATASET_LOCAL_DIR=examples/rl/data` (or set `MATH_DATASET_LOCAL_<SPLIT>_FILE`).
+- Hugging Face downloads occur at runtime; pre-fetch locally or mount a Modal volume if you need offline access.
+- Hugging Face downloads occur at runtime; pre-fetch locally or mount a Modal volume if you need offline access.
+- Seeds map directly to dataset indices. Use `seed_start` to control determinism in configs and evaluations.
+
+## 7. Additional Utilities
+
+- `examples/rl/task_app/math_task_app.py` – legacy runner (`python .../math_task_app.py --reload`).
+- `examples/rl/run_eval.py` – CLI evaluation helper (supports proxying Groq or hitting arbitrary inference URLs).
+- `examples/rl/run_rl_and_save.py` – thin wrapper around the Synth `/rl/jobs` API.
+
+For broader background on Synth task apps, CLI commands, and tracing, see the new documentation under `docs/`.
+
+
+
+uv run python examples/rl/run_eval.py --toml examples/rl/configs/eval_base_qwen.toml
+uvx synth-ai serve math-single-step \
+    --port 8101 \
+    --env-file examples/rl/.env \
+    --trace traces/math \
+    --force

examples/rl/configs/eval_base_qwen.toml

@@ -0,0 +1,15 @@
+provider = "synth"
+task_app_url = "http://localhost:8101"
+model = "Qwen/Qwen3-1.7B"
+split = "validation"
+num_episodes = 50
+seed_start = 0
+
+[policy]
+inference_url = "http://localhost:8000/api/inference"
+max_tokens = 128
+temperature = 0.0
+
+# Optionally supply custom headers
+# [policy.headers]
+# Authorization = "Bearer ..."

examples/rl/configs/eval_rl_qwen.toml

@@ -0,0 +1,11 @@
+provider = "synth"
+task_app_url = "https://your-math-task.modal.run"
+model = "rl:REPLACE_WITH_JOB_ID"
+split = "test"
+num_episodes = 200
+seed_start = 100000
+
+[policy]
+inference_url = "https://your-inference-host"
+max_tokens = 128
+temperature = 0.0

examples/rl/configs/rl_from_base_qwen.toml

@@ -0,0 +1,35 @@
+[services]
+task_url = "https://your-math-task.modal.run"
+
+[model]
+base = "Qwen/Qwen3-4B"
+
+[policy]
+model = "Qwen/Qwen3-4B"
+inference_url = "https://your-inference-host"
+max_tokens = 128
+temperature = 0.0
+
+[data]
+split = "train"
+seed_start = 0
+episodes_per_iteration = 2048
+evaluation_split = "validation"
+evaluation_episodes = 256
+
+[training]
+max_turns = 1
+ops = ["agent", "env"]
+batch_size = 128
+group_size = 1024
+reward_positive = 1.0
+reward_negative_no_tool = -1.0
+reward_negative_no_answer = -0.5
+learning_rate = 5e-6
+
+[compute]
+gpu_type = "A10G"
+gpu_count = 4
+
+[tags]
+experiment = "math_single_step"

examples/rl/configs/rl_from_base_qwen17.toml

@@ -0,0 +1,74 @@
+[algorithm]
+type = "online"
+method = "policy_gradient"
+variety = "gspo"
+
+[services]
+task_url = "http://localhost:8101"
+
+[model]
+base = "Qwen/Qwen3-1.7B"
+
+[policy]
+model = "Qwen/Qwen3-1.7B"
+inference_url = "http://localhost:8000/api/inference"
+max_tokens = 1028
+temperature = 0.2
+
+[data]
+split = "train"
+seed_start = 0
+episodes_per_iteration = 1280  # 8 per group * 4 groups per batch * 2 batches per step * 20 steps
+evaluation_split = "validation"
+evaluation_episodes = 50
+
+[training]
+max_turns = 1
+ops = ["agent", "env"]
+batch_size = 2
+group_size = 16
+reward_positive = 1.0
+reward_negative_no_tool = -1.0
+reward_negative_no_answer = -0.5
+learning_rate = 5e-6
+log_interval = 1
+weight_sync_interval = 1
+
+[training.weight_sync]
+enable = true
+targets = ["policy"]
+
+[compute]
+gpu_type = "H100"
+gpu_count = 4
+
+[topology]
+type = "single_node_split"
+gpus_for_vllm = 2
+gpus_for_training = 1
+gpus_for_ref = 1
+tensor_parallel = 1
+
+[vllm]
+tensor_parallel_size = 1
+max_model_len = 4096
+
+[reference]
+placement = "dedicated"
+port = 8002
+tp = 1
+health_max_wait_s = 180
+health_interval_ms = 300
+
+[rollout]
+policy_name = "math-single-step"
+max_turns = 1
+episodes_per_batch = 32  # group_size * batch_size
+
+[evaluation]
+instances = 32
+every_n_iters = 10
+seeds = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
+
+[tags]
+experiment = "math_single_step_qwen17"

examples/rl/configs/rl_from_ft_qwen.toml

@@ -0,0 +1,35 @@
+[services]
+task_url = "https://your-math-task.modal.run"
+
+[model]
+source = "ft:REPLACE_WITH_MODEL_ID"
+
+[policy]
+model = "ft:REPLACE_WITH_MODEL_ID"
+inference_url = "https://your-inference-host"
+max_tokens = 128
+temperature = 0.0
+
+[data]
+split = "train"
+seed_start = 0
+episodes_per_iteration = 2048
+evaluation_split = "validation"
+evaluation_episodes = 256
+
+[training]
+max_turns = 1
+ops = ["agent", "env"]
+batch_size = 128
+group_size = 1024
+reward_positive = 1.0
+reward_negative_no_tool = -1.0
+reward_negative_no_answer = -0.5
+learning_rate = 5e-6
+
+[compute]
+gpu_type = "A10G"
+gpu_count = 4
+
+[tags]
+experiment = "math_single_step_from_fft"

examples/rl/download_dataset.py

@@ -0,0 +1,64 @@
+#!/usr/bin/env python3
+"""Download subsets of the MATH dataset to local JSONL files."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any
+
+from datasets import load_dataset
+
+
+def extract_examples(dataset: Any, *, limit: int | None) -> list[dict[str, str]]:
+    if limit is not None:
+        dataset = dataset.select(range(min(limit, len(dataset))))
+    examples: list[dict[str, str]] = []
+    for item in dataset:
+        problem = (item.get("problem") or "").strip()
+        solution = item.get("solution") or ""
+        if isinstance(solution, list):
+            solution = "\n".join(str(part) for part in solution)
+        examples.append({
+            "problem": problem,
+            "solution": solution,
+        })
+    return examples
+
+
+def write_jsonl(path: Path, rows: list[dict[str, str]]) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    with path.open("w", encoding="utf-8") as fh:
+        for row in rows:
+            fh.write(json.dumps(row, ensure_ascii=False) + "\n")
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Download MATH dataset splits to JSONL for offline use")
+    parser.add_argument("--output-dir", default="examples/rl/data", help="Directory to write <split>.jsonl files")
+    parser.add_argument("--dataset", default="nlile/hendrycks-MATH-benchmark", help="Hugging Face dataset identifier")
+    parser.add_argument("--config", default="algebra", help="Hugging Face dataset config (if required)")
+    parser.add_argument("--splits", nargs="*", default=["train", "validation", "test"], help="Splits to download")
+    parser.add_argument("--limit", type=int, default=None, help="Optional cap on examples per split")
+    args = parser.parse_args()
+
+    output_dir = Path(args.output_dir).expanduser()
+    output_dir.mkdir(parents=True, exist_ok=True)
+
+    for split in args.splits:
+        print(f"[INFO] Downloading {args.dataset} ({args.config}) split={split}")
+        if args.config:
+            dataset = load_dataset(args.dataset, args.config, split=split)
+        else:
+            dataset = load_dataset(args.dataset, split=split)
+        rows = extract_examples(dataset, limit=args.limit)
+        out_path = output_dir / f"{split}.jsonl"
+        write_jsonl(out_path, rows)
+        print(f"[INFO] Wrote {len(rows)} examples to {out_path}")
+
+    print("Done. Set MATH_DATASET_LOCAL_DIR to the output directory when serving the task app.")
+
+
+if __name__ == "__main__":
+    main()