soup-cli 0.71.29__tar.gz → 0.71.30__tar.gz

{soup_cli-0.71.29 → soup_cli-0.71.30}/CHANGELOG.md

@@ -12,6 +12,44 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.30] - 2026-07-05
+
+### Added
+- **PRM-guided GRPO** — use a trained Process Reward Model as the *per-step*
+  reward inside GRPO (the o1-era process-supervision signal). Set
+  `training.prm_reward: <PRM dir|id>` (a model produced by `soup train`
+  `task=prm`) and `training.prm_aggregate: min|prod|last`; the PRM splits each
+  generated completion into reasoning steps, scores every step with its reward
+  head, and folds the per-step scores into one scalar reward that GRPO
+  optimises. The PRM reward *replaces* `reward_fn` and rides the existing
+  reward-shaping + reward-hack-mitigation seam, so the v0.71.26 controller
+  observes it. Cross-validators gate `task='grpo'` + `backend='transformers'` +
+  `modality='text'`. Default aggregation is `min` (weakest-link); `prod`
+  assumes calibrated `[0,1]` step scores.
+- **Bundled rollout environments** — three pure-Python toy environments
+  (`soup_cli.envs.calculator` / `retrieval_qa` / `guess_number`) exposing a
+  `rollout(prompts)` entry point so the live `openenv` GRPO rollout path runs
+  out-of-the-box: `training.rollout_backend=openenv` +
+  `training.rollout_func=soup_cli.envs.calculator:rollout`. Three ready-made
+  recipes added (`grpo-env-calculator` / `grpo-env-retrieval-qa` /
+  `grpo-env-guess-number`); catalog 134 → 137.
+
+### Fixed
+- **`soup train task=prm` producer conformance** (surfaced by the v0.71.30 live
+  smoke): the PRM trainer now casts its reward head to the base-model dtype
+  (bf16 CUDA runs previously crashed on the first `compute_loss`), saves the
+  tokenizer alongside the model (so a PRM checkpoint is loadable standalone),
+  and returns the standard trainer-result shape (previously the CLI crashed with
+  a `KeyError: 'initial_loss'` right after saving).
+
+### Notes
+- Proof-of-mechanism only: validated on a tiny model (SmolLM2-135M) with a tiny
+  synthetic PRM and synthetic reward — not a production reward-model claim
+  (scale ask tracked in #286). The bundled environments are deterministic
+  single-shot *seeders*, not interactive multi-turn model-in-the-loop episodes
+  (the live `openenv` contract passes only prompts). Step split is a newline
+  heuristic; PRM completions are scored one forward pass each.
+
 ## [0.71.29] - 2026-07-05
 
 ### Added

{soup_cli-0.71.29 → soup_cli-0.71.30}/CONTRIBUTING.md

@@ -110,17 +110,17 @@ src/soup_cli/
   experiment/         - SQLite experiment tracking
   eval/               - Eval platform (custom tasks, LLM judge, human eval, leaderboard)
   migrate/            - Config migration (LLaMA-Factory, Axolotl, Unsloth)
-  recipes/            - Ready-made configs for popular models (134 recipes)
+  recipes/            - Ready-made configs for popular models (137 recipes)
   autopilot/          - Zero-config decision engine (v0.25.0)
   registry/           - Model Registry (hashing, store, diff, attach) (v0.26.0 + v0.33.0)
   cans/               - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0)
   data/traces/        - Trace-to-Preference harvester (v0.26.0)
   data/collators.py   - CrossDocCollator for sample packing (v0.33.0)
-  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint
+  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint, shrink
   templates/          - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (306 files, 15224 tests)
+tests/                - Test suite (307 files, 15334 tests)
 examples/             - Real-world config examples and datasets
 ```
 

{soup_cli-0.71.29 → soup_cli-0.71.30}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.29
+Version: 0.71.30
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -193,22 +193,27 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.29 — Make your model smaller, locally: `soup shrink`.** Depth-prune the least-important layers, then distill-heal — with a binary SHIP / DON'T-SHIP verdict. Based on "The Unreasonable Ineffectiveness of the Deeper Layers" (arXiv:2403.17887).
+**v0.71.30 — Process-supervised RL: PRM-guided GRPO.** Use a trained Process Reward Model to score each *reasoning step* of a GRPO completion — the o1-era training signal — plus bundled toy environments so the openenv rollout path runs out-of-the-box.
+
+- **PRM as the GRPO reward.** Point `training.prm_reward` at a PRM you trained with
+  `soup train task=prm`; it splits each completion into steps, scores every step with the PRM's
+  reward head, and folds them (`min` / `prod` / `last`) into one reward that GRPO optimises —
+  replacing `reward_fn`. It rides the reward-shaping + reward-hack-mitigation seam, so the
+  v0.71.26 controller still watches it.
+- **Bundled rollout environments.** `soup_cli.envs.calculator` / `retrieval_qa` / `guess_number`
+  each expose a `rollout(prompts)` entry point; wire one with `rollout_backend=openenv` +
+  `rollout_func=soup_cli.envs.calculator:rollout` (three ready-made `grpo-env-*` recipes ship it).
+- **Producer fixes too.** `soup train task=prm` now saves its tokenizer and no longer crashes on
+  its own summary — a PRM checkpoint is loadable standalone.
+- **Proof-of-mechanism, honestly.** Live-validated on SmolLM2-135M (CPU): the PRM reward scores
+  good completions above bad and drives GRPO's advantages (`rewards/prm_reward` logged). Tiny
+  model + synthetic PRM — not a production reward-model claim (scale help wanted: #286).
 
-- **Importance-ranked depth pruning.** One forward pass per calibration prompt scores each
-  contiguous layer block by the angular distance of the residual stream across it; the
-  least-important block is dropped (the first and last layer are always protected).
-- **Distill-heal, Minitron-style.** `--heal` distills the original model into the pruned
-  student (LoRA logit-KD) as an isolated `soup train` run, then fuses the adapter back so you
-  ship a single dense smaller model — not a base + adapter.
-- **A verdict, not a dashboard.** Every run ends in **SHIP** or **DON'T SHIP** on a
-  before/after perplexity ratio (`exit 0 = SHIP`, `2 = DON'T`, `1 = error`), so it drops into CI.
-- **Runs on a laptop GPU.** Live-validated on Windows + RTX 3050 with SmolLM2-135M: drop 25 %
-  (30 → 22 layers, 21 % params) and drop-4 + CPU heal (perplexity recovered to ×1.35).
-
-```bash
-soup shrink --model HuggingFaceTB/SmolLM2-135M-Instruct --drop-ratio 0.25 \
-    --calib calib.jsonl --heal heal.jsonl --heal-steps 200 -o shrunk --device cpu
+```yaml
+task: grpo
+training:
+  prm_reward: ./my-prm            # a `soup train task=prm` checkpoint dir
+  prm_aggregate: min              # weakest-link (default) | prod | last
 ```
 
 Full history: [CHANGELOG.md](CHANGELOG.md) · [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).

{soup_cli-0.71.29 → soup_cli-0.71.30}/README.md

@@ -49,22 +49,27 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.29 — Make your model smaller, locally: `soup shrink`.** Depth-prune the least-important layers, then distill-heal — with a binary SHIP / DON'T-SHIP verdict. Based on "The Unreasonable Ineffectiveness of the Deeper Layers" (arXiv:2403.17887).
-
-- **Importance-ranked depth pruning.** One forward pass per calibration prompt scores each
-  contiguous layer block by the angular distance of the residual stream across it; the
-  least-important block is dropped (the first and last layer are always protected).
-- **Distill-heal, Minitron-style.** `--heal` distills the original model into the pruned
-  student (LoRA logit-KD) as an isolated `soup train` run, then fuses the adapter back so you
-  ship a single dense smaller model — not a base + adapter.
-- **A verdict, not a dashboard.** Every run ends in **SHIP** or **DON'T SHIP** on a
-  before/after perplexity ratio (`exit 0 = SHIP`, `2 = DON'T`, `1 = error`), so it drops into CI.
-- **Runs on a laptop GPU.** Live-validated on Windows + RTX 3050 with SmolLM2-135M: drop 25 %
-  (30 → 22 layers, 21 % params) and drop-4 + CPU heal (perplexity recovered to ×1.35).
+**v0.71.30 — Process-supervised RL: PRM-guided GRPO.** Use a trained Process Reward Model to score each *reasoning step* of a GRPO completion — the o1-era training signal — plus bundled toy environments so the openenv rollout path runs out-of-the-box.
+
+- **PRM as the GRPO reward.** Point `training.prm_reward` at a PRM you trained with
+  `soup train task=prm`; it splits each completion into steps, scores every step with the PRM's
+  reward head, and folds them (`min` / `prod` / `last`) into one reward that GRPO optimises —
+  replacing `reward_fn`. It rides the reward-shaping + reward-hack-mitigation seam, so the
+  v0.71.26 controller still watches it.
+- **Bundled rollout environments.** `soup_cli.envs.calculator` / `retrieval_qa` / `guess_number`
+  each expose a `rollout(prompts)` entry point; wire one with `rollout_backend=openenv` +
+  `rollout_func=soup_cli.envs.calculator:rollout` (three ready-made `grpo-env-*` recipes ship it).
+- **Producer fixes too.** `soup train task=prm` now saves its tokenizer and no longer crashes on
+  its own summary — a PRM checkpoint is loadable standalone.
+- **Proof-of-mechanism, honestly.** Live-validated on SmolLM2-135M (CPU): the PRM reward scores
+  good completions above bad and drives GRPO's advantages (`rewards/prm_reward` logged). Tiny
+  model + synthetic PRM — not a production reward-model claim (scale help wanted: #286).
 
-```bash
-soup shrink --model HuggingFaceTB/SmolLM2-135M-Instruct --drop-ratio 0.25 \
-    --calib calib.jsonl --heal heal.jsonl --heal-steps 200 -o shrunk --device cpu
+```yaml
+task: grpo
+training:
+  prm_reward: ./my-prm            # a `soup train task=prm` checkpoint dir
+  prm_aggregate: min              # weakest-link (default) | prod | last
 ```
 
 Full history: [CHANGELOG.md](CHANGELOG.md) · [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).

{soup_cli-0.71.29 → soup_cli-0.71.30}/docs/README.md

@@ -8,7 +8,7 @@ feature reference — every `soup` capability, grouped by area.
 | Guide | Covers |
 |---|---|
 | [Training tasks & methods](training.md) | SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, loop-hardening detectors |
-| [PEFT, long context & efficiency](peft-and-efficiency.md) | DoRA, LoRA+, rsLoRA, VeRA, OLoRA, NEFTune, PiSSA, ReLoRA, optimizer & PEFT zoo, LLaMA Pro, GaLore, YaRN/LongLoRA, packing, curriculum, auto-tuning |
+| [PEFT, long context & efficiency](peft-and-efficiency.md) | DoRA, LoRA+, rsLoRA, VeRA, OLoRA, NEFTune, PiSSA, ReLoRA, optimizer & PEFT zoo, LLaMA Pro, GaLore, YaRN/LongLoRA, packing, curriculum, auto-tuning, depth pruning + distill-heal (`soup shrink`) |
 | [Performance & quantization](performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, multi-GPU / DeepSpeed / FSDP |
 | [Data engineering](data.md) | Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs |
 | [Evaluation & probes](evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, `soup ship` verdict, post-train X-ray probes, A/B, drift, tunability, `soup advise` |

{soup_cli-0.71.29 → soup_cli-0.71.30}/docs/commands.md

@@ -133,7 +133,7 @@ soup migrate --from llamafactory config.yaml  Import config from LLaMA-Factory
 soup migrate --from axolotl config.yml        Import config from Axolotl
 soup migrate --from unsloth notebook.ipynb    Import config from Unsloth notebook
 soup migrate --from llamafactory c.yaml --dry-run  Preview without writing
-soup recipes list                             List all 134 ready-made recipes
+soup recipes list                             List all 137 ready-made recipes
 soup recipes show llama3.1-8b-sft            Print recipe YAML
 soup recipes use llama3.1-8b-sft             Copy recipe to soup.yaml
 soup recipes search "reasoning"              Search by keyword/task/size

soup_cli-0.71.30/docs/superpowers/plans/2026-07-05-prm-guided-grpo.md

@@ -0,0 +1,133 @@
+# PRM-guided GRPO + bundled rollout envs — Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:test-driven-development, task-by-task. Steps use checkbox (`- [ ]`) syntax.
+
+**Goal:** Let a trained Soup PRM score each reasoning step of a GRPO completion as the reward signal, and bundle 3 pure-python toy environments that seed the live `openenv` rollout path out-of-the-box.
+
+**Architecture:** A pure-kernel + torch-lazy `PRMScorer` in `utils/prm_reward.py` becomes the GRPO `reward_fn` when `training.prm_reward` is set (replacing the configured reward, riding the existing shaping+buffer seam). Three `soup_cli/envs/*.py` modules expose `rollout(prompts)` entry points wired via `training.rollout_func`. Schema fields + cross-validators gate both.
+
+**Tech Stack:** Python 3.10+, Pydantic v2, torch/transformers/safetensors (lazy), TRL GRPO, pytest.
+
+## Global Constraints
+
+- Line length 100 (ruff E,F,I,N,W). No bare `print()` — use `rich.console.Console`.
+- Lazy imports for torch/transformers/peft/safetensors inside functions, never module-level.
+- Path containment: `os.path.realpath` + `os.path.commonpath`, NOT `Path.resolve()+relative_to()`.
+- Pydantic v2 models in `config/schema.py`; assert specific error keywords in tests.
+- Test file: `tests/test_v07130.py`. Use `assert result.exit_code == 0, (result.output, repr(result.exception))`.
+- Version target: `0.71.29 → 0.71.30` (bump in `pyproject.toml` + `src/soup_cli/__init__.py`).
+- Proof-of-mechanism honesty: tiny model / synthetic data only; scale ask extends #286.
+
+---
+
+### Task 1: PRM reward pure kernels
+
+**Files:**
+- Create: `src/soup_cli/utils/prm_reward.py`
+- Test: `tests/test_v07130.py`
+
+**Interfaces:**
+- Produces: `split_steps(text: str) -> list[str]`, `aggregate_step_scores(scores: list[float], mode: str) -> float`, module constants `_MAX_STEPS`, `_MAX_STEP_CHARS`, `AGGREGATE_MODES: tuple[str,...]`.
+
+- [ ] Step 1: Write failing tests `TestSplitSteps` (newline split; drop empty/whitespace; count cap `_MAX_STEPS`; per-step char cap; non-str→`[]`) and `TestAggregate` (min/prod/last correctness; empty→0.0; single; non-finite→treated safe; bad mode `ValueError` names modes; bool reject).
+- [ ] Step 2: Run — expect ImportError / fail.
+- [ ] Step 3: Implement kernels (no top-level torch). `split_steps`: `str.splitlines()`, strip, drop empties, truncate each to `_MAX_STEP_CHARS`, cap list to `_MAX_STEPS`. `aggregate_step_scores`: validate mode in `AGGREGATE_MODES`; empty→0.0; `min`→min, `last`→scores[-1], `prod`→math.prod; coerce non-finite to 0.0.
+- [ ] Step 4: Run tests → PASS.
+- [ ] Step 5: `ruff check` + commit `feat(prm): PRM reward pure kernels (split_steps/aggregate) (v0.71.30)`.
+
+---
+
+### Task 2: Schema fields + cross-validators
+
+**Files:**
+- Modify: `src/soup_cli/config/schema.py` (TrainingConfig fields near `reward_fn` ~L1043; field_validator; SoupConfig cross-validator near `_validate_rollout_backend` ~L4183)
+- Test: `tests/test_v07130.py`
+
+**Interfaces:**
+- Produces: `TrainingConfig.prm_reward: Optional[str]`, `TrainingConfig.prm_aggregate: Literal["min","prod","last"]`; a `SoupConfig` `model_validator(mode="after")` `_validate_prm_reward`.
+
+- [ ] Step 1: Write failing `TestPrmSchema` — happy config (task=grpo, transformers, text, prm_reward set, prm_aggregate=prod) parses via `load_config_from_string`; rejects: task≠grpo ("requires task='grpo'"), backend=mlx/unsloth ("transformers"), modality≠text ("modality='text'"), `prm_aggregate` non-default while `prm_reward None` (footgun keyword), null-byte prm_reward, oversize prm_reward.
+- [ ] Step 2: Run → fail (fields absent).
+- [ ] Step 3: Add fields + `field_validator("prm_reward", mode="before")` (null-byte/non-str/len≤512 reject) + SoupConfig `_validate_prm_reward` cross-validator (gates task/backend/modality; footgun on aggregate). Mirror `_validate_rollout_backend`.
+- [ ] Step 4: Run → PASS.
+- [ ] Step 5: ruff + commit `feat(prm): schema prm_reward/prm_aggregate + cross-validators (v0.71.30)`.
+
+---
+
+### Task 3: PRMScorer + build_prm_reward_fn (torch-lazy load + scoring)
+
+**Files:**
+- Modify: `src/soup_cli/utils/prm_reward.py`
+- Test: `tests/test_v07130.py`
+
+**Interfaces:**
+- Consumes: `split_steps`, `aggregate_step_scores` (Task 1); `TrainingConfig` (Task 2).
+- Produces: `PRMScorer(prm_path, aggregate, device="cpu", trust_remote_code=False)` with `__call__(completions, **kwargs) -> list[float]` and `__name__="prm_reward"`; `build_prm_reward_fn(tcfg, device, trust_remote_code) -> PRMScorer`; `load_reward_head_weights(prm_path) -> dict` helper.
+
+- [ ] Step 1: Write failing `TestPRMScorer` — fixture: build a tiny `AutoModelForCausalLM` from a 2-layer config, attach `reward_head=nn.Linear(hidden,1)`, `save_pretrained(tmp)`; `PRMScorer(tmp,"min","cpu")` scores `[[{"role":"assistant","content":"step1\nstep2"}]]` → list len 1 of finite float. "not a PRM" fixture (base with no reward_head) → friendly `ValueError`/`RuntimeError` naming "reward_head". `build_prm_reward_fn` returns callable with `__name__=="prm_reward"`; outside-cwd path → containment reject.
+- [ ] Step 2: Run → fail.
+- [ ] Step 3: Implement. `load_reward_head_weights`: iterate `*.safetensors` in dir via `safetensors.safe_open`, collect `reward_head.*` tensors; empty→raise. `PRMScorer._ensure_loaded`: lazy tokenizer + `AutoModelForCausalLM.from_pretrained`, `reward_head=nn.Linear(hidden,1)`, `load_state_dict` the head, `eval()`+`requires_grad_(False)`+`.to(device)`. `__call__`: per completion → render prompt (from `kwargs["prompts"]`) + steps, tokenize, `no_grad` `output_hidden_states` forward, gather boundary hidden, `reward_head`→scalars, `aggregate_step_scores`. `build_prm_reward_fn`: realpath+commonpath containment for local paths, trust_remote_code probe+warn.
+- [ ] Step 4: Run → PASS.
+- [ ] Step 5: ruff + commit `feat(prm): PRMScorer + build_prm_reward_fn (safetensors head load) (v0.71.30)`.
+
+---
+
+### Task 4: GRPO integration
+
+**Files:**
+- Modify: `src/soup_cli/trainer/grpo.py` (`setup()`, the `load_reward_fn` block ~L224-229)
+- Test: `tests/test_v07130.py`
+
+**Interfaces:**
+- Consumes: `build_prm_reward_fn` (Task 3).
+
+- [ ] Step 1: Write failing `TestGrpoPrmWiring` — construct a SoupConfig with `prm_reward` set, monkeypatch `build_prm_reward_fn` to a sentinel, call the reward-selection path (extract a small helper `_select_reward_fn(tcfg, device, trust)` if cleaner, or assert via the setup path with mocked model load), assert the PRM scorer is chosen (not `load_reward_fn`).
+- [ ] Step 2: Run → fail.
+- [ ] Step 3: In `setup()`: `if tcfg.prm_reward is not None: reward_fn = build_prm_reward_fn(tcfg, self.device, self._trust_remote_code) else: reward_fn = load_reward_fn(...)`. Keep the shaping+buffer wrapping downstream unchanged.
+- [ ] Step 4: Run → PASS.
+- [ ] Step 5: ruff + commit `feat(prm): wire PRM reward into GRPO setup (v0.71.30)`.
+
+---
+
+### Task 5: Bundled rollout envs
+
+**Files:**
+- Create: `src/soup_cli/envs/__init__.py`, `calculator.py`, `retrieval_qa.py`, `guess_number.py`
+- Test: `tests/test_v07130.py`
+
+**Interfaces:**
+- Produces: `soup_cli.envs.calculator:rollout`, `...retrieval_qa:rollout`, `...guess_number:rollout` — each `rollout(prompts) -> list[{"prompt": str, "answer": str}]`, deterministic (seeded `random.Random`), no I/O.
+
+- [ ] Step 1: Write failing `TestEnvs` — each `rollout([])` returns non-empty rows; every row passes `agent_rollout._normalise_rollout_rows(rows, "openenv")`; determinism (two calls equal); calculator answers verified (parse "a OP b" from prompt, compute, assert == answer).
+- [ ] Step 2: Run → fail.
+- [ ] Step 3: Implement 3 modules. Each builds a fixed count of {prompt, answer} rows via `random.Random(seed)` (fixed seed → deterministic; no `Math.random`/wall-clock). calculator: arithmetic; retrieval_qa: doc+question, answer=span; guess_number: constraint deduction puzzles with a unique integer answer.
+- [ ] Step 4: Run → PASS.
+- [ ] Step 5: ruff + commit `feat(envs): bundled calculator/retrieval-qa/guess-number rollout envs (v0.71.30)`.
+
+---
+
+### Task 6: Recipes + no-top-level-torch AST test
+
+**Files:**
+- Modify: `src/soup_cli/recipes/catalog.py` (+3 GRPO env recipes)
+- Test: `tests/test_v07130.py`
+
+- [ ] Step 1: Write failing `TestRecipes` (3 new recipe names resolve + their YAML parses via `load_config_from_string`) + `TestNoTopLevelTorch` (AST-parse `utils/prm_reward.py`, assert no top-level `import torch/transformers/peft`).
+- [ ] Step 2: Run → fail.
+- [ ] Step 3: Add `grpo-env-calculator` / `grpo-env-retrieval-qa` / `grpo-env-guess-number` RecipeMeta entries (SmolLM2-135M, task=grpo, rollout_backend=openenv, rollout_func=`soup_cli.envs.*:rollout`, reward_fn=math/accuracy). Confirm the no-top-level-torch already holds.
+- [ ] Step 4: Run → PASS.
+- [ ] Step 5: ruff + commit `feat(recipes): 3 GRPO env recipes + AST purity test (v0.71.30)`.
+
+---
+
+### Task 7: Release docs (Release Checklist steps 7-13)
+
+Version bump (`pyproject.toml` + `__init__.py` → 0.71.30); CHANGELOG; CLAUDE.md (arch line for `utils/prm_reward.py` + `envs/`, schema field list, CLI/train notes, recipe count 134→137, test-count refs, history-roll to `.claude/history/release-notes.md` + `tests-index.md`); README What's New; `docs/training.md` + `docs/commands.md`; CONTRIBUTING counts; plan.md flip. Handled after Tasks 1-6 pass the 5-review round + Step-6 smoke.
+
+---
+
+## Self-review
+
+- **Spec coverage:** Part A schema (T2) + kernels (T1) + scorer (T3) + wiring (T4); Part B envs (T5) + recipes (T6); docs (T7). ✅
+- **Type consistency:** `split_steps`/`aggregate_step_scores` (T1) consumed by `PRMScorer` (T3); `build_prm_reward_fn` (T3) consumed by grpo (T4); `rollout` signature uniform (T5). ✅
+- **Placeholders:** none — each task has concrete test targets + implementation sketch (full code written during TDD RED/GREEN).

soup_cli-0.71.30/docs/superpowers/specs/2026-07-05-prm-guided-grpo-design.md

@@ -0,0 +1,152 @@
+# v0.71.30 — PRM-guided GRPO + bundled rollout envs — Design
+
+**Status:** approved 2026-07-05
+**Slot:** v0.71.30 (first `[ ]` patch in `.claude/plan.md`)
+**Type:** net-new mechanism (PRM-as-per-step-reward adapter + bundled envs package)
+
+## Pitch
+
+Use a trained PRM (the v0.53.11 `PRMTrainerWrapper`, live) as a **per-step reward inside
+GRPO** — the o1-era process-supervision training signal. Plus bundle 3 pure-python toy
+"environments" so the live `openenv` rollout path (#125) runs out-of-the-box.
+
+**Honesty constraint (this is an RL feature — cannot ship schema-only):** validation is
+**proof-of-mechanism only** on a tiny model (SmolLM2-135M) + a tiny synthetic PRM. It is NOT
+a 7B / real-PRM / production claim. Scale ask extends #286. State this loudly in the PR,
+CHANGELOG, release notes, and CLAUDE.md known-limitations. Mirrors the v0.71.26 framing.
+
+## Decisions locked
+
+- **PRM reward role = sole reward (replace).** When `training.prm_reward` is set, the PRM
+  scorer becomes THE reward function, replacing `reward_fn`. Simplest wiring — drops into the
+  existing single-fn shaping + buffer seam. Matches the "PRM reward curve" smoke.
+- **PRM artifact = Soup-trained PRM only (v1).** Load base `AutoModelForCausalLM`, re-attach
+  `reward_head=nn.Linear(hidden,1)`, and load its weights from the saved `model.safetensors`
+  (`reward_head.*` keys). HF `AutoModelForSequenceClassification(num_labels=1)` interop is a
+  deferred follow-up.
+- **Envs = deterministic single-shot seeders**, not interactive multi-turn episodes. The live
+  `openenv` contract calls `fn(seed_prompts)` — model/tokenizer are NOT passed — so an env is a
+  prompt+answer *generator* scored by the existing `accuracy`/`math` reward.
+- **Aggregation default = `min`** (weakest-link; Lightman et al. "Let's Verify Step by Step").
+- **Step split = newline heuristic (v1).**
+
+## Reuse map (verified in-repo 2026-07-05)
+
+- Reward seam: `trainer/grpo.py::setup()` loads a single `reward_fn` via
+  `rewards.load_reward_fn(spec)`, then optionally wraps with
+  `reward_hack_control.apply_reward_shaping` + `rl_signal_buffer.wrap_reward_funcs(buffer)`
+  (the v0.71.26 mitigation seam). A PRM reward is just a `(completions, **kwargs) -> list[float]`
+  callable → rides that seam unchanged.
+- PRM artifact: `trainer/prm.py::PRMTrainerWrapper` sets `base_model.reward_head =
+  nn.Linear(hidden,1,bias=True)` and `save_model`s the full model → the head weights live inside
+  `model.safetensors` under `reward_head.*` (base `from_pretrained` drops them as "unexpected").
+- Scoring math mirror: `PRMTrainerWrapper.compute_loss` — forward `output_hidden_states=True`,
+  gather `last_hidden` at step-boundary positions, project via `reward_head`.
+- Rollout: `utils/agent_rollout.py::launch_rollout` — `openenv` resolves `rollout_func`
+  (`module:fn`) and calls `fn(seed_prompts)`; rows go through `_normalise_rollout_rows`
+  (caps + anti-smuggle → `{prompt, answer?}`-only).
+- Schema patterns: `rollout_func` field_validator + `_validate_rollout_backend` cross-validator;
+  `verifiable_domain` `reward_fn` cross-validator (`schema.py`).
+
+## Part A — PRM reward
+
+### Schema (`config/schema.py`, `TrainingConfig`)
+
+- `prm_reward: Optional[str] = None` — path (or HF id) to a Soup-trained PRM.
+  `field_validator(mode="before")` rejects null-byte / non-str / oversize (shape only;
+  filesystem containment enforced at load, TOCTOU-safe).
+- `prm_aggregate: Literal["min","prod","last"] = "min"`.
+- `SoupConfig` cross-validator: `prm_reward` set ⇒ `task=='grpo'`, `backend=='transformers'`
+  (reward runs a transformers forward — mlx/unsloth rejected), `modality=='text'`;
+  `prm_aggregate != "min"` while `prm_reward is None` → footgun reject.
+
+### New module `utils/prm_reward.py` (NO top-level torch)
+
+- `split_steps(text: str) -> list[str]` — newline heuristic v1; drop empty/whitespace; cap
+  count (`_MAX_STEPS`) and per-step chars. Pure.
+- `aggregate_step_scores(scores: list[float], mode: str) -> float` — min/prod/last;
+  empty→0.0; non-finite-safe; bad-mode `ValueError`; bool guard. Pure kernel.
+- `PRMScorer` — stateful callable:
+  - `__init__(prm_path, aggregate, device, trust_remote_code)` — stores config, no load yet.
+  - `_ensure_loaded()` — lazy: tokenizer + base `AutoModelForCausalLM` (dtype per device),
+    re-attach `reward_head`, load `reward_head.*` from any `*.safetensors` in the dir via
+    `safetensors.safe_open` (friendly error if absent), `eval()` + `requires_grad_(False)`,
+    move to device.
+  - `__call__(completions, **kwargs) -> list[float]` — per completion: render + prepend the
+    `prompts` kwarg as leading context (best-effort), `split_steps`, one `no_grad`
+    `output_hidden_states` forward, gather boundary hidden, `reward_head` → per-step scalars,
+    `aggregate_step_scores`. Per-completion loop (no batching — fine for tiny; perf follow-up).
+- `build_prm_reward_fn(tcfg, device, trust_remote_code) -> PRMScorer` — validates local-path
+  containment (realpath + commonpath under cwd), `trust_remote_code` probe+warn, returns scorer.
+
+### Integration (`trainer/grpo.py::setup()`)
+
+One insertion before `load_reward_fn`: if `tcfg.prm_reward is not None` →
+`reward_fn = build_prm_reward_fn(tcfg, self.device, self._trust_remote_code)` instead of
+`load_reward_fn(...)`. The existing `apply_reward_shaping` + `wrap_reward_funcs` wrapping applies
+unchanged → the v0.71.26 mitigation controller + `mitigation_log.jsonl` observe the PRM reward.
+
+## Part B — bundled rollout envs
+
+### New package `soup_cli/envs/`
+
+Each module exposes `rollout(prompts) -> list[{"prompt", "answer"}]`, deterministic (seeded RNG,
+no I/O / no network), usable as `soup_cli.envs.<name>:rollout` in `training.rollout_func`:
+
+- `calculator.py` — arithmetic problems, answer = the number (`math`/`accuracy` reward).
+- `retrieval_qa.py` — short document + question, answer = a span (`accuracy` reward).
+- `guess_number.py` — single-shot number-**deduction** puzzles (constraints → unique answer;
+  honestly framed as deducible, not interactive guessing).
+
+Rows flow through the existing `_normalise_rollout_rows` → replace the GRPO prompt dataset.
+
+### Recipes (`recipes/catalog.py`, +3 → 137)
+
+`grpo-env-calculator` / `grpo-env-retrieval-qa` / `grpo-env-guess-number` — SmolLM2-135M base,
+`rollout_backend=openenv` + `rollout_func=soup_cli.envs.*:rollout` + reward. Parse-clean.
+
+## Data flow
+
+- **PRM:** dataset prompts → GRPO generates N completions → `PRMScorer(completions, prompts)` →
+  per completion: split steps → PRM forward → gather boundary hidden → `reward_head` → per-step
+  scalars → `aggregate(min/prod/last)` → scalar reward list → GRPO advantages.
+- **Envs:** `rollout_func` seeds `{prompt, answer}` rows → replace GRPO prompt dataset → GRPO
+  trains with `accuracy`/`math` reward.
+
+## Tests (`tests/test_v07130.py`)
+
+- `split_steps` (newlines / empties / whitespace / count cap / char cap / non-str guard).
+- `aggregate_step_scores` (min/prod/last correctness, empty→0, single, non-finite defensive,
+  bad-mode reject, bool guard).
+- Schema parse + every cross-validator (task≠grpo, mlx/unsloth, modality≠text, footgun,
+  null-byte/oversize) — assert specific keywords.
+- `PRMScorer` against a tiny **real saved PRM fixture** (2-layer CausalLM + reward_head →
+  `save_pretrained` → load → score → finite list of right length); "not a PRM" friendly error.
+- grpo wiring: `prm_reward` set → PRM scorer built (inject/monkeypatch, CPU tiny).
+- Each env's rows pass `_normalise_rollout_rows`; determinism; answers correct
+  (calculator arithmetic verified).
+- 3 recipes parse via `load_config_from_string`; count bump.
+- No-top-level-torch AST test on `utils/prm_reward.py`.
+
+## Security
+
+- `prm_reward` local path: realpath + commonpath containment under cwd at load; `safe_open`
+  (safetensors — no pickle) for the reward_head weights; O_NOFOLLOW where reading. HF id (no
+  local path) allowed via `from_pretrained`. `trust_remote_code=False` probe+warn.
+- Bounds: `_MAX_STEPS`, per-step char cap, aggregate mode allowlist.
+- Envs: pure python, deterministic, no I/O / no network; rows re-validated by
+  `_normalise_rollout_rows` (caps + anti-smuggle).
+
+## Known limitations (release notes + CLAUDE.md)
+
+1. Proof-of-mechanism only — tiny PRM, synthetic data; scale ask extends #286.
+2. Envs are deterministic single-shot seeders, not interactive multi-turn model-in-the-loop
+   episodes — the live `openenv` contract passes only `prompts`.
+3. Step split = newline heuristic (v1); PRM scored per-completion, no batching (perf follow-up).
+4. PRM artifact v1 = Soup-trained format only (HF SeqClassification interop deferred).
+
+## Smoke (RTX 3050 / CPU)
+
+Train a tiny PRM on synthetic step-labelled math → GRPO on SmolLM2-135M with `prm_reward`,
+reward curve logged (mitigation log reused); one env end-to-end GRPO run (mirrors #125). Plus
+every validator-reject config.

{soup_cli-0.71.29 → soup_cli-0.71.30}/docs/training.md

@@ -15,6 +15,7 @@
 - [EBFT / GDPO Loss Variants](#ebft--gdpo-loss-variants)
 - [GRPO Objective Variants](#grpo-objective-variants)
 - [Process Reward Model (PRM)](#process-reward-model-prm)
+- [PRM-guided GRPO (process-supervised RL)](#prm-guided-grpo-process-supervised-rl)
 - [Weighted Multi-Objective Preference Loss](#weighted-multi-objective-preference-loss)
 - [MoE Model Support](#moe-model-support)
 - [Vision / Multimodal Fine-tuning](#vision--multimodal-fine-tuning)
@@ -343,7 +344,61 @@ training:
 
 The trainer loads `AutoModelForCausalLM`, attaches an `nn.Linear(hidden, 1)`
 reward head, and computes MSE between predicted scalars at step-boundary tokens
-and the per-step labels.
+and the per-step labels. The reward head is saved inside the model checkpoint
+(`reward_head.*` in `model.safetensors`) and the tokenizer is saved alongside it,
+so the resulting directory is loadable standalone.
+
+
+## PRM-guided GRPO (process-supervised RL)
+
+Use a trained PRM as the **per-step reward** inside GRPO — the o1-era
+process-supervision signal. Set `training.prm_reward` to a PRM directory (a
+`task=prm` checkpoint) or HF id; the PRM splits each generated completion into
+reasoning steps (newline heuristic), scores every step with its reward head, and
+folds the per-step scores into one scalar reward that GRPO optimises. It
+**replaces** `reward_fn` and rides the existing reward-shaping +
+reward-hack-mitigation seam, so the v0.71.26 controller still observes it (TRL
+logs it as `rewards/prm_reward`).
+
+```yaml
+task: grpo
+backend: transformers        # required (the PRM reward runs a transformers forward)
+modality: text               # required
+data:
+  format: chatml
+  train: ./grpo_prompts.jsonl
+training:
+  prm_reward: ./my-prm       # a `soup train task=prm` checkpoint dir (or HF id)
+  prm_aggregate: min         # weakest-link (default) | prod | last
+  num_generations: 4
+  grpo_beta: 0.04
+```
+
+`prm_aggregate='min'` (weakest-link, the standard PRM aggregation) is the safe
+default; `prod` assumes calibrated `[0,1]` step scores (Soup's PRM head is
+trained with unconstrained MSE, so `prod` can blow up on uncalibrated labels).
+
+**Bundled rollout environments.** Three deterministic pure-Python toy
+environments seed the openenv rollout path out-of-the-box — pair any of them
+with `rollout_backend=openenv`:
+
+```yaml
+training:
+  rollout_backend: openenv
+  rollout_func: soup_cli.envs.calculator:rollout   # or retrieval_qa / guess_number
+  reward_fn: verifiable
+  verifiable_domain: math
+```
+
+Ready-made recipes: `grpo-env-calculator`, `grpo-env-retrieval-qa`,
+`grpo-env-guess-number`. The environments are deterministic single-shot
+prompt/answer *seeders* (the live openenv contract passes only the seed prompts,
+not the model) — not interactive multi-turn episodes.
+
+**Scope:** proof-of-mechanism only — validated on SmolLM2-135M with a tiny
+synthetic PRM (the PRM reward scores good completions above bad and drives GRPO's
+advantages). Not a production reward-model claim; scale validation is help-wanted
+(#286).
 
 
 ## Weighted Multi-Objective Preference Loss

{soup_cli-0.71.29 → soup_cli-0.71.30}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.29"
+version = "0.71.30"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.29 → soup_cli-0.71.30}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
 
-__version__ = "0.71.29"
+__version__ = "0.71.30"