synth-ai 0.2.9.dev11__py3-none-any.whl → 0.4.1__py3-none-any.whl

examples/rl/README.md

@@ -1,169 +0,0 @@
-# Math RL Demo (Single Step)
-
-This example trains a reinforcement learning policy on single-step math problems sourced from the [EleutherAI/math](https://huggingface.co/datasets/EleutherAI/math) dataset. Episodes consist of a single tool call: the model must emit a `math_submit` function call whose `answer` field contains the final solution. Missing or malformed tool calls receive negative reward; correct answers earn positive reward.
-
-## Quick Commands
-
-```bash
-# Serve locally with tracing
-uvx synth-ai serve math-single-step --port 8101 --env-file examples/rl/.env --trace traces/math
-
-# Modal deployment
-uvx synth-ai deploy --name synth-math-single-step --env-file examples/rl/.env
-
-# Evaluate base Qwen policy (validation split)
-uv run python examples/rl/run_eval.py --toml examples/rl/configs/eval_base_qwen.toml
-
-# Launch RL job from base model
-uvx synth-ai train --type rl --config examples/rl/configs/rl_from_base_qwen.toml
-
-# Evaluate RL checkpoint on held-out test split
-uv run python examples/rl/run_eval.py --toml examples/rl/configs/eval_rl_qwen.toml
-```
-
-## 1. Prerequisites
-
-- Python 3.11+
-- `uv`/`uvx`
-- Modal CLI (`modal token new`) for deployment
-- `.env` at `examples/rl/.env` containing at least:
-  - `SYNTH_API_KEY`
-  - `ENVIRONMENT_API_KEY`
-  - Optional: `TASK_APP_URL` (Modal URL), `GROQ_API_KEY`, `OPENAI_API_KEY`
-
-Run `uvx synth-ai setup` to populate the `.env` if you have not paired the SDK before.
-
-## 2. Task App
-
-The task app is defined in `synth_ai/task/apps/math_single_step.py` and registered as `math-single-step`. It loads problems from the Hugging Face dataset (configurable via `MATH_DATASET_*` env vars) and manages per-episode state with an in-memory environment manager.
-
-- **Observation**: single math problem (string) plus dataset metadata.
-- **Actions**: exactly one `math_submit` tool call with an `answer` string.
-- **Rewards**:
-  - `+1.0` for correct answer
-  - `0.0` for incorrect answer
-  - `-0.5` if the tool call omits an answer or uses the wrong tool
-  - `-1.0` when no tool call is provided
-
-Serve locally with tracing to capture trajectories:
-
-```bash
-uvx synth-ai serve math-single-step \
-  --port 8101 \
-  --env-file examples/rl/.env \
-  --trace traces/math \
-  --trace-db traces/math/synth_ai.db
-```
-
-Deploy or serve on Modal using the same env file; the registration includes a `ModalDeploymentConfig` that installs the `datasets` package automatically.
-
-## 3. Evaluation
-
-`examples/rl/run_eval.py` evaluates a policy by sampling deterministic seeds from the dataset splits. TOML configuration controls the model, split, and number of episodes. Example config (`eval_base_qwen.toml`):
-
-```toml
-provider = "synth"
-task_app_url = "http://localhost:8101"
-model = "Qwen/Qwen3-4B"
-split = "validation"
-num_episodes = 50
-seed_start = 0
-
-[policy]
-inference_url = "http://localhost:8000/api/inference"
-max_tokens = 128
-temperature = 0.0
-# Optional: override headers for inference requests
-# [policy.extra_headers]
-# Authorization = "Bearer ..."
-```
-
-The `[policy]` table maps directly to the inference payload; add `[policy.headers]` if you need to forward custom HTTP headers (e.g., `Authorization`). If `SYNTH_API_KEY` is present, the evaluator automatically sends `Authorization: Bearer <key>`.
-
-Set `--use-rollout` to exercise the server-side rollout endpoint instead of the per-step API.
-
-The script reports accuracy and a breakdown of failure modes (`missing_tool_call`, `blank_answer`, etc.).
-
-## 4. RL Training
-
-Example RL config (`configs/rl_from_base_qwen.toml`):
-
-```toml
-[services]
-task_url = "https://your-app.modal.run"
-
-[model]
-base = "Qwen/Qwen3-4B"
-
-[data]
-split = "train"
-seed_start = 0
-episodes_per_iteration = 2048
-
-[training]
-max_turns = 1
-ops = ["agent", "env"]
-batch_size = 128
-group_size = 1024
-reward_positive = 1.0
-reward_negative_no_tool = -1.0
-reward_negative_no_answer = -0.5
-
-[policy]
-model = "Qwen/Qwen3-4B"
-inference_url = "https://your-inference-host"
-max_tokens = 128
-temperature = 0.0
-
-[tags]
-experiment = "math_single_step"
-```
-
-Submit jobs interactively with:
-
-```bash
-uvx synth-ai train --type rl --config examples/rl/configs/rl_from_base_qwen.toml
-```
-
-The CLI ensures the task app is reachable (`/health`, `/task_info`), prompts for missing secrets, and polls job status until completion. For scripted automation, use `run_rl_and_save.py`:
-
-```bash
-uv run python examples/rl/run_rl_and_save.py \
-  --config examples/rl/configs/rl_from_base_qwen.toml \
-  --backend https://backend.synth.ai/api
-```
-
-## 5. Evaluating RL Outputs
-
-After training completes, set `model = "rl:<job_or_model_id>"` in `configs/eval_rl_qwen.toml` (and update `split = "test"` for a held-out set). Re-run `run_eval.py` to compare:
-
-```bash
-uv run python examples/rl/run_eval.py --toml examples/rl/configs/eval_rl_qwen.toml
-```
-
-Record both validation (pre-RL) and test (post-RL) accuracy to quantify improvements.
-
-## 6. Dataset Notes
-
-- By default the task app loads the [Hendrycks MATH benchmark](https://huggingface.co/datasets/nlile/hendrycks-MATH-benchmark). Override via `MATH_DATASET_NAME` / `MATH_DATASET_CONFIG` env vars if you want a different variant. The dataset is public and automatically downloaded when the task app starts; the server will fail fast with a clear error if it cannot be fetched.
-- For offline use, run `uv run python examples/rl/download_dataset.py --output-dir examples/rl/data --dataset nlile/hendrycks-MATH-benchmark --config algebra --limit 2000`. Then start the task app with `MATH_DATASET_LOCAL_DIR=examples/rl/data` (or set `MATH_DATASET_LOCAL_<SPLIT>_FILE`).
-- Hugging Face downloads occur at runtime; pre-fetch locally or mount a Modal volume if you need offline access.
-- Hugging Face downloads occur at runtime; pre-fetch locally or mount a Modal volume if you need offline access.
-- Seeds map directly to dataset indices. Use `seed_start` to control determinism in configs and evaluations.
-
-## 7. Additional Utilities
-
-- `examples/rl/task_app/math_task_app.py` – legacy runner (`python .../math_task_app.py --reload`).
-- `examples/rl/run_eval.py` – CLI evaluation helper (supports proxying Groq or hitting arbitrary inference URLs).
-- `examples/rl/run_rl_and_save.py` – thin wrapper around the Synth `/rl/jobs` API.
-
-For broader background on Synth task apps, CLI commands, and tracing, see the new documentation under `docs/`.
-
-
-
-uv run python examples/rl/run_eval.py --toml examples/rl/configs/eval_base_qwen.toml
-uvx synth-ai serve math-single-step \
-    --port 8101 \
-    --env-file examples/rl/.env \
-    --trace traces/math \
-    --force

examples/rl/configs/eval_base_qwen.toml

@@ -1,15 +0,0 @@
-provider = "synth"
-task_app_url = "http://localhost:8101"
-model = "Qwen/Qwen3-1.7B"
-split = "validation"
-num_episodes = 50
-seed_start = 0
-
-[policy]
-inference_url = "https://agent-learning.onrender.com/api/inference"
-max_tokens = 128
-temperature = 0.0
-
-# Optionally supply custom headers
-# [policy.headers]
-# Authorization = "Bearer ..."

examples/rl/configs/eval_rl_qwen.toml

@@ -1,11 +0,0 @@
-provider = "synth"
-task_app_url = "https://your-math-task.modal.run"
-model = "rl:REPLACE_WITH_JOB_ID"
-split = "test"
-num_episodes = 200
-seed_start = 100000
-
-[policy]
-inference_url = "https://your-inference-host"
-max_tokens = 128
-temperature = 0.0

examples/rl/configs/rl_from_base_qwen.toml

@@ -1,35 +0,0 @@
-[services]
-task_url = "https://your-math-task.modal.run"
-
-[model]
-base = "Qwen/Qwen3-4B"
-
-[policy]
-model = "Qwen/Qwen3-4B"
-inference_url = "https://your-inference-host"
-max_tokens = 128
-temperature = 0.0
-
-[data]
-split = "train"
-seed_start = 0
-episodes_per_iteration = 2048
-evaluation_split = "validation"
-evaluation_episodes = 256
-
-[training]
-max_turns = 1
-ops = ["agent", "env"]
-batch_size = 128
-group_size = 1024
-reward_positive = 1.0
-reward_negative_no_tool = -1.0
-reward_negative_no_answer = -0.5
-learning_rate = 5e-6
-
-[compute]
-gpu_type = "A10G"
-gpu_count = 4
-
-[tags]
-experiment = "math_single_step"

examples/rl/configs/rl_from_base_qwen17.toml

@@ -1,74 +0,0 @@
-[algorithm]
-type = "online"
-method = "policy_gradient"
-variety = "gspo"
-
-[services]
-task_url = "http://localhost:8101"
-
-[model]
-base = "Qwen/Qwen3-1.7B"
-
-[policy]
-model = "Qwen/Qwen3-1.7B"
-inference_url = "https://agent-learning.onrender.com/api/inference"
-max_tokens = 1028
-temperature = 0.2
-
-[data]
-split = "train"
-seed_start = 0
-episodes_per_iteration = 1280  # 8 per group * 4 groups per batch * 2 batches per step * 20 steps
-evaluation_split = "validation"
-evaluation_episodes = 50
-
-[training]
-max_turns = 1
-ops = ["agent", "env"]
-batch_size = 2
-group_size = 16
-reward_positive = 1.0
-reward_negative_no_tool = -1.0
-reward_negative_no_answer = -0.5
-learning_rate = 5e-6
-log_interval = 1
-weight_sync_interval = 1
-
-[training.weight_sync]
-enable = true
-targets = ["policy"]
-
-[compute]
-gpu_type = "H100"
-gpu_count = 4
-
-[topology]
-type = "single_node_split"
-gpus_for_vllm = 2
-gpus_for_training = 1
-gpus_for_ref = 1
-tensor_parallel = 1
-
-[vllm]
-tensor_parallel_size = 1
-max_model_len = 4096
-
-[reference]
-placement = "dedicated"
-port = 8002
-tp = 1
-health_max_wait_s = 180
-health_interval_ms = 300
-
-[rollout]
-policy_name = "math-single-step"
-max_turns = 1
-episodes_per_batch = 32  # group_size * batch_size
-
-[evaluation]
-instances = 32
-every_n_iters = 10
-seeds = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
-
-[tags]
-experiment = "math_single_step_qwen17"

examples/rl/configs/rl_from_ft_qwen.toml

@@ -1,35 +0,0 @@
-[services]
-task_url = "https://your-math-task.modal.run"
-
-[model]
-source = "ft:REPLACE_WITH_MODEL_ID"
-
-[policy]
-model = "ft:REPLACE_WITH_MODEL_ID"
-inference_url = "https://your-inference-host"
-max_tokens = 128
-temperature = 0.0
-
-[data]
-split = "train"
-seed_start = 0
-episodes_per_iteration = 2048
-evaluation_split = "validation"
-evaluation_episodes = 256
-
-[training]
-max_turns = 1
-ops = ["agent", "env"]
-batch_size = 128
-group_size = 1024
-reward_positive = 1.0
-reward_negative_no_tool = -1.0
-reward_negative_no_answer = -0.5
-learning_rate = 5e-6
-
-[compute]
-gpu_type = "A10G"
-gpu_count = 4
-
-[tags]
-experiment = "math_single_step_from_fft"

examples/rl/download_dataset.py

@@ -1,80 +0,0 @@
-#!/usr/bin/env python3
-"""Download subsets of the MATH dataset to local JSONL files."""
-
-from __future__ import annotations
-
-import argparse
-import json
-from pathlib import Path
-from typing import Any
-
-from datasets import load_dataset
-
-
-def extract_examples(dataset: Any, *, limit: int | None) -> list[dict[str, str]]:
-    if limit is not None:
-        dataset = dataset.select(range(min(limit, len(dataset))))
-    examples: list[dict[str, str]] = []
-    for item in dataset:
-        problem = (item.get("problem") or "").strip()
-        solution = item.get("solution") or ""
-        if isinstance(solution, list):
-            solution = "\n".join(str(part) for part in solution)
-        examples.append(
-            {
-                "problem": problem,
-                "solution": solution,
-            }
-        )
-    return examples
-
-
-def write_jsonl(path: Path, rows: list[dict[str, str]]) -> None:
-    path.parent.mkdir(parents=True, exist_ok=True)
-    with path.open("w", encoding="utf-8") as fh:
-        for row in rows:
-            fh.write(json.dumps(row, ensure_ascii=False) + "\n")
-
-
-def main() -> None:
-    parser = argparse.ArgumentParser(
-        description="Download MATH dataset splits to JSONL for offline use"
-    )
-    parser.add_argument(
-        "--output-dir", default="examples/rl/data", help="Directory to write <split>.jsonl files"
-    )
-    parser.add_argument(
-        "--dataset",
-        default="nlile/hendrycks-MATH-benchmark",
-        help="Hugging Face dataset identifier",
-    )
-    parser.add_argument(
-        "--config", default="algebra", help="Hugging Face dataset config (if required)"
-    )
-    parser.add_argument(
-        "--splits", nargs="*", default=["train", "validation", "test"], help="Splits to download"
-    )
-    parser.add_argument(
-        "--limit", type=int, default=None, help="Optional cap on examples per split"
-    )
-    args = parser.parse_args()
-
-    output_dir = Path(args.output_dir).expanduser()
-    output_dir.mkdir(parents=True, exist_ok=True)
-
-    for split in args.splits:
-        print(f"[INFO] Downloading {args.dataset} ({args.config}) split={split}")
-        if args.config:
-            dataset = load_dataset(args.dataset, args.config, split=split)
-        else:
-            dataset = load_dataset(args.dataset, split=split)
-        rows = extract_examples(dataset, limit=args.limit)
-        out_path = output_dir / f"{split}.jsonl"
-        write_jsonl(out_path, rows)
-        print(f"[INFO] Wrote {len(rows)} examples to {out_path}")
-
-    print("Done. Set MATH_DATASET_LOCAL_DIR to the output directory when serving the task app.")
-
-
-if __name__ == "__main__":
-    main()