soup-cli 0.71.25__tar.gz → 0.71.26__tar.gz

{soup_cli-0.71.25 → soup_cli-0.71.26}/CHANGELOG.md

@@ -12,6 +12,44 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.26] - 2026-07-01
+
+### Added
+- **Closed-loop reward-hacking auto-mitigation.** The trainer now *detects*
+  reward hacking mid-run and *self-corrects* — instead of only halting. Set
+  `training.reward_hack_mitigation` (or `soup train --reward-hack-mitigation`)
+  to one of four modes on a GRPO/PPO run (requires `reward_hack_detector`):
+  - `log_only` — instrument only: append a per-step `mitigation_log.jsonl`
+    (drop_pct, verdict, reward mean/std, completion-length trend, repetition)
+    and *never* touch training.
+  - `kl_control` — a reversible **bang-bang + hysteresis** controller: when the
+    hacking signal trips, raise the KL coefficient β (geometric, clamped to
+    `[floor, ceil]`, never crossing 0); relax it when the signal recovers.
+    Dwell + release-patience prevent flapping; a multi-signal vote combines the
+    detector drop with a length-trend and repetition signal.
+  - `pid_lagrangian` — a **PID-Lagrangian** controller (Stooke et al.) that
+    holds the hacking signal at a target, plus an **escalation ladder**:
+    raise β → roll back to the last-good RL checkpoint → early-stop.
+  - Anti-gaming hardening: per-signal EMA/median smoothing,
+    conservative-on-disagreement voting, a reward-distribution-drift guard, and
+    optional bounded **reward shaping** on the gamed proxy (length / repetition
+    / sentinel). A plain-English give-up explanation is logged on early-stop.
+  - **Proof-of-mechanism only** (see Known Limitations): validated on
+    SmolLM2-135M + a synthetic length-hacking task on a single RTX 3050 — all
+    four stages pass live, including a real mid-run rollback. PPO ships **BETA**
+    (mechanism unit-tested; the on-GPU proof is GRPO-only).
+- Ready-made `qwen2.5-coder-7b-sft` recipe for `Qwen/Qwen2.5-Coder-7B-Instruct`
+  (catalog 133 → 134)
+  ([#285](https://github.com/MakazhanAlpamys/Soup/pull/285) by [@Deadpool2000](https://github.com/Deadpool2000)).
+
+### Security
+- `RLCheckpointCallback.restore_checkpoint` / `save_checkpoint` refuse a
+  **symlinked** `optimizer.pt` — `torch.load(weights_only=False)` on an
+  attacker-placed symlink in a shared checkpoint dir was an RCE vector.
+- Bool-before-int/float guards on every new `reward_hack_*` numeric field;
+  `reward_hack_signals` bounded (`max_length=4`); the mitigation log writer is
+  cwd-contained with symlink-reject-on-rotate and secret redaction.
+
 ## [0.71.25] - 2026-06-27
 
 ### Added

{soup_cli-0.71.25 → soup_cli-0.71.26}/CONTRIBUTING.md

@@ -110,17 +110,17 @@ src/soup_cli/
   experiment/         - SQLite experiment tracking
   eval/               - Eval platform (custom tasks, LLM judge, human eval, leaderboard)
   migrate/            - Config migration (LLaMA-Factory, Axolotl, Unsloth)
-  recipes/            - Ready-made configs for popular models (133 recipes)
+  recipes/            - Ready-made configs for popular models (134 recipes)
   autopilot/          - Zero-config decision engine (v0.25.0)
   registry/           - Model Registry (hashing, store, diff, attach) (v0.26.0 + v0.33.0)
   cans/               - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0)
   data/traces/        - Trace-to-Preference harvester (v0.26.0)
   data/collators.py   - CrossDocCollator for sample packing (v0.33.0)
-  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler
+  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict
   templates/          - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (297 files, 14593 tests)
+tests/                - Test suite (298 files, 14788 tests)
 examples/             - Real-world config examples and datasets
 ```
 

{soup_cli-0.71.25 → soup_cli-0.71.26}/CONTRIBUTORS.md

@@ -25,6 +25,7 @@ Listed by first contribution. PR numbers link the work.
   - Happy-path + CPU-warning tests for `soup bench` ([#31](https://github.com/MakazhanAlpamys/Soup/pull/31))
   - `soup cost` — cloud GPU training cost estimation ([#42](https://github.com/MakazhanAlpamys/Soup/pull/42))
   - `--nccl` flag for `soup doctor` multi-GPU bandwidth checks ([#178](https://github.com/MakazhanAlpamys/Soup/pull/178))
+  - Ready-made `qwen2.5-coder-7b-sft` recipe ([#285](https://github.com/MakazhanAlpamys/Soup/pull/285))
 - **Chinmaya Sahu** ([@csking101](https://github.com/csking101))
   - DPO example config, sample data, and tests ([#48](https://github.com/MakazhanAlpamys/Soup/pull/48))
   - FP8 `rowwise` + `rowwise_with_gw_hp` scaling recipes ([#62](https://github.com/MakazhanAlpamys/Soup/pull/62))

{soup_cli-0.71.25 → soup_cli-0.71.26}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.25
+Version: 0.71.26
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -189,20 +189,21 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.25 — `soup ship`: the SHIP / DON'T-SHIP verdict.** After fine-tuning, answer one question — did the model get better, or did I break it?
+**v0.71.26 — Closed-loop reward-hacking auto-mitigation.** Your GRPO/PPO trainer now *detects* reward hacking mid-run and *self-corrects* — instead of only halting. No OSS RLHF library closes this loop.
 
-- **One binary decision, not a dashboard.** `soup ship` SHIPs only when **(1)** the task
-  metric *strictly* improved AND **(2)** no general benchmark regressed past a forgetting
-  threshold (default 5% absolute). Otherwise DON'T SHIP — *even if the task metric looks great*.
-- **The moat is leg 2.** A catastrophic-forgetting / regression gate, fused with the task win
-  into a single verdict + a one-screen reason.
-- **CI-gateable.** Exit `0 = SHIP`, `2 = DON'T SHIP`, `1 = runtime error`.
-- **Works offline.** `soup ship --evidence ev.json` decides from pre-computed scores (no model
-  load); leg-2 defaults to built-in mini benchmarks (CPU, instant). `--general-suite` routes
-  lm-eval benchmarks; `--baseline registry://… | file.json` supplies recorded base scores.
+- **Detect → correct → continue.** When the hacking signal trips, the controller raises the
+  KL coefficient β (bang-bang + hysteresis, or a PID-Lagrangian controller), and — if hacking
+  persists — **rolls back to the last-good checkpoint**, then early-stops as a last resort.
+- **Anti-gaming hardening.** Multi-signal voting (RM cluster-separation + length-trend +
+  repetition), signal smoothing, conservative-on-disagreement, a reward-distribution-drift
+  guard, and optional bounded **reward shaping** on the gamed proxy.
+- **Observe first.** `reward_hack_mitigation=log_only` streams a per-step `mitigation_log.jsonl`
+  and never touches training — see the hacking happen before you let the controller act.
+- **Honest scope.** Proof-of-mechanism on SmolLM2-135M + a synthetic hacking task on one RTX
+  3050 (all four stages validated live, including a real rollback). PPO ships BETA.
 
 ```bash
-soup ship --base <model> --adapter <lora> --task-eval tasks.jsonl   # → SHIP / DON'T SHIP
+soup train --config grpo.yaml --reward-hack-mitigation kl_control   # detect → raise KL → recover
 ```
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).

{soup_cli-0.71.25 → soup_cli-0.71.26}/README.md

@@ -49,20 +49,21 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.25 — `soup ship`: the SHIP / DON'T-SHIP verdict.** After fine-tuning, answer one question — did the model get better, or did I break it?
-
-- **One binary decision, not a dashboard.** `soup ship` SHIPs only when **(1)** the task
-  metric *strictly* improved AND **(2)** no general benchmark regressed past a forgetting
-  threshold (default 5% absolute). Otherwise DON'T SHIP — *even if the task metric looks great*.
-- **The moat is leg 2.** A catastrophic-forgetting / regression gate, fused with the task win
-  into a single verdict + a one-screen reason.
-- **CI-gateable.** Exit `0 = SHIP`, `2 = DON'T SHIP`, `1 = runtime error`.
-- **Works offline.** `soup ship --evidence ev.json` decides from pre-computed scores (no model
-  load); leg-2 defaults to built-in mini benchmarks (CPU, instant). `--general-suite` routes
-  lm-eval benchmarks; `--baseline registry://… | file.json` supplies recorded base scores.
+**v0.71.26 — Closed-loop reward-hacking auto-mitigation.** Your GRPO/PPO trainer now *detects* reward hacking mid-run and *self-corrects* — instead of only halting. No OSS RLHF library closes this loop.
+
+- **Detect → correct → continue.** When the hacking signal trips, the controller raises the
+  KL coefficient β (bang-bang + hysteresis, or a PID-Lagrangian controller), and — if hacking
+  persists — **rolls back to the last-good checkpoint**, then early-stops as a last resort.
+- **Anti-gaming hardening.** Multi-signal voting (RM cluster-separation + length-trend +
+  repetition), signal smoothing, conservative-on-disagreement, a reward-distribution-drift
+  guard, and optional bounded **reward shaping** on the gamed proxy.
+- **Observe first.** `reward_hack_mitigation=log_only` streams a per-step `mitigation_log.jsonl`
+  and never touches training — see the hacking happen before you let the controller act.
+- **Honest scope.** Proof-of-mechanism on SmolLM2-135M + a synthetic hacking task on one RTX
+  3050 (all four stages validated live, including a real rollback). PPO ships BETA.
 
 ```bash
-soup ship --base <model> --adapter <lora> --task-eval tasks.jsonl   # → SHIP / DON'T SHIP
+soup train --config grpo.yaml --reward-hack-mitigation kl_control   # detect → raise KL → recover
 ```
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).

{soup_cli-0.71.25 → soup_cli-0.71.26}/docs/README.md

@@ -11,7 +11,7 @@ feature reference — every `soup` capability, grouped by area.
 | [PEFT, long context & efficiency](peft-and-efficiency.md) | DoRA, LoRA+, rsLoRA, VeRA, OLoRA, NEFTune, PiSSA, ReLoRA, optimizer & PEFT zoo, LLaMA Pro, GaLore, YaRN/LongLoRA, packing, curriculum, auto-tuning |
 | [Performance & quantization](performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, multi-GPU / DeepSpeed / FSDP |
 | [Data engineering](data.md) | Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs |
-| [Evaluation & probes](evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, post-train X-ray probes, A/B, drift, tunability, `soup advise` |
+| [Evaluation & probes](evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, `soup ship` verdict, post-train X-ray probes, A/B, drift, tunability, `soup advise` |
 | [Serving & export](serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding, deploy autopilot, Web UI, Agent Forge |
 | [Adapters, registry & governance](adapters-and-governance.md) | Adapter lifecycle/management, model registry, Soup Cans, the data flywheel (`soup loop`), knowledge editing, steering, supply-chain controls |
 | [Backends, platform & ops](backends-and-ops.md) | MLX/Unsloth backends, Modal cloud GPU training, alternative hubs, HF Hub integration, autopilot, experiment tracking, plan/apply, env lockfiles, hardware-fit, completions, plugins, utility commands |

{soup_cli-0.71.25 → soup_cli-0.71.26}/docs/commands.md

@@ -129,7 +129,7 @@ soup migrate --from llamafactory config.yaml  Import config from LLaMA-Factory
 soup migrate --from axolotl config.yml        Import config from Axolotl
 soup migrate --from unsloth notebook.ipynb    Import config from Unsloth notebook
 soup migrate --from llamafactory c.yaml --dry-run  Preview without writing
-soup recipes list                             List all 133 ready-made recipes
+soup recipes list                             List all 134 ready-made recipes
 soup recipes show llama3.1-8b-sft            Print recipe YAML
 soup recipes use llama3.1-8b-sft             Copy recipe to soup.yaml
 soup recipes search "reasoning"              Search by keyword/task/size
@@ -245,6 +245,7 @@ soup data persona-mix --prompts <jsonl> --n N --output <jsonl>  Persona-Hub dive
 soup data brain-rot <data.jsonl> [--strict]   Brain-rot detector — arXiv 2510.13928 (v0.69.0)
 soup iterative-dpo --base-model <m> --reward-model <rm> --prompts <p.jsonl> --output-dir <out> --rounds N --pairs-per-round N [--plan-only]  Iterative DPO loop driver — LIVE sample→score→pair→train (v0.70.0; live v0.71.11)
 soup train --reward-hack-detector info_rm|rm_ensemble [--reward-hack-halt]  Reward-hacking detector for GRPO — LIVE callback (v0.70.0; live v0.71.11)
+soup train --reward-hack-mitigation off|log_only|kl_control|pid_lagrangian  Closed-loop reward-hacking auto-mitigation (detect → raise KL/β → rollback → early-stop); GRPO/PPO, requires --reward-hack-detector; PPO BETA (v0.71.26)
 soup train --uld-strategy wasserstein|topk_align [--uld-top-k N]  Cross-tokenizer ULD on task='distill' — LIVE W1/topk loss (v0.70.0; live v0.71.11)
 soup train --minillm-enabled [--minillm-teacher-mix-ratio 0.3]  MiniLLM reverse-KL distillation — LIVE (v0.70.0; live v0.71.11)
 soup train --rl-checkpoint-save-every-steps N [--rl-checkpoint-keep-last N]  Mid-epoch checkpoint for GRPO/PPO — LIVE (v0.70.0; live v0.71.11)

{soup_cli-0.71.25 → soup_cli-0.71.26}/docs/serving-and-export.md

@@ -418,7 +418,7 @@ soup ui
 
 **Pages:**
 - **Dashboard** — view all experiment runs, loss charts, system info, multi-run comparison
-- **New Training** — create configs from templates or 133 ready-made recipes, validate, start training with live SSE log streaming and progress bar
+- **New Training** — create configs from templates or 134 ready-made recipes, validate, start training with live SSE log streaming and progress bar
 - **Data Explorer** — browse and inspect datasets (JSONL, JSON, CSV, Parquet)
 - **Model Chat** — chat with streaming responses, configurable temperature/top_p/max_tokens, system prompt, adapter selection, markdown rendering, chat export
 
@@ -427,7 +427,7 @@ soup ui
 - **Enhanced Metrics** — 2x2 chart grid (loss, LR, grad_norm, throughput) + GPU memory chart, eval results table
 - **Multi-Run Compare** — overlay loss curves from up to 5 runs side-by-side
 - **Chat Upgrade** — SSE streaming via proxy, typing indicator, cancel button, markdown renderer (bold, italic, code blocks), chat export as JSON
-- **Config Builder** — recipe dropdown (133 recipes), config schema API for dynamic form generation
+- **Config Builder** — recipe dropdown (134 recipes), config schema API for dynamic form generation
 
 **Security:** The Web UI generates a random auth token at startup (printed to console). All mutating endpoints (start/stop training, delete runs, inspect data, validate config) require `Authorization: Bearer <token>` header. CORS is restricted to the served origin. Data inspection is sandboxed to the working directory.
 

{soup_cli-0.71.25 → soup_cli-0.71.26}/docs/training.md

@@ -50,6 +50,10 @@ Six surfaces protect the training loop from the failure modes that cost a real G
 soup train --config soup.yaml \
     --reward-hack-detector info_rm --reward-hack-halt   # halt on HACK verdict
 
+# Closed-loop reward-hacking MITIGATION (v0.71.26) — detect AND self-correct
+soup train --config grpo.yaml --reward-hack-mitigation kl_control   # raise KL, recover
+soup train --config grpo.yaml --reward-hack-mitigation log_only     # observe only, no action
+
 # Cross-tokenizer distillation — Llama -> Mistral, no shared vocab needed
 # (Universal Logit Distillation, Boizard et al. 2024 arXiv:2402.12030)
 soup train --config soup.yaml --uld-strategy wasserstein
@@ -100,6 +104,17 @@ soup train --config grpo.yaml \
 
 `--echo-trap-tokenizer-aware` switches echo-trap n-grams from whitespace tokens to the active tokenizer's integer ids. This catches subword repetition that punctuation-heavy decoded text can hide, but the score becomes tokenizer-specific rather than vocabulary-agnostic.
 
+### Closed-loop reward-hacking auto-mitigation (v0.71.26)
+
+The detectors above *halt*; `training.reward_hack_mitigation` (or the `--reward-hack-mitigation` flag) makes the trainer *self-correct* mid-run. It requires `reward_hack_detector` on a `grpo`/`ppo` transformers run, and has four modes:
+
+- **`log_only`** — observe only. Appends a per-step `mitigation_log.jsonl` under the run's output dir (the InfoRM/ensemble drop, the OK/WARN/HACK verdict, reward mean/std, completion-length trend, repetition) and provably never mutates β. Run this first to *see* hacking before you let a controller act on it.
+- **`kl_control`** — a reversible **bang-bang + hysteresis** controller. When a multi-signal vote (`reward_hack_signals`: the detector drop + `length_trend` + `repetition`) stays above `reward_hack_trip_band` for `reward_hack_dwell_steps`, it multiplies β by `reward_hack_kl_gain` (clamped to `[reward_hack_beta_floor > 0, reward_hack_beta_ceil]`, never crossing 0); after `reward_hack_release_patience` below-band steps it relaxes β back toward the floor. Dwell + release-patience stop it flapping. β is written to **both** `trainer.beta` and `trainer.args.beta` so it takes effect on stock GRPO *and* Soup's GRPO variants (and `trainer.args.kl_coef` on PPO).
+- **`pid_lagrangian`** — a **PID-Lagrangian** controller (Stooke et al. 2020) that holds the hacking signal at `reward_hack_signal_target` (Kp/Ki/Kd with integral anti-windup via `reward_hack_integral_clamp`), plus an **escalation ladder**: raise β → after `reward_hack_rollback_patience` persistent-HACK steps roll back to the last-good RL checkpoint (needs `rl_checkpoint_save_every_steps`) → after `reward_hack_max_recovery_attempts` rollbacks, early-stop with a plain-English give-up explanation.
+- **Anti-gaming hardening** (any control mode): `reward_hack_signal_smoothing` (`ema`/`median` over `reward_hack_smoothing_window`), `reward_hack_conservative_on_disagreement` (when detectors disagree, keep KL high + guard against a bimodal reward-distribution collapse), and `reward_hack_reward_shaping` (subtract a bounded `reward_hack_shaping_strength` penalty on the gamed proxy — `length`/`repetition`/`sentinel` — over the reward-fn seam).
+
+**Scope:** proof-of-mechanism only. Validated on SmolLM2-135M + a synthetic length-hacking task on a single RTX 3050 (all four modes live, including a real mid-run rollback). PPO ships **BETA** — the buffer + `kl_coef` mutation are wired and unit-tested, but the on-GPU proof is GRPO-only. Whether the loop suppresses hacking without collapsing true reward on 7B+ with a real reward model is an open, community-validatable question. `reward_hack_mitigation ∈ {kl_control, pid_lagrangian}` is mutually exclusive with `ref_model_ema_alpha` (both drive the KL/ref dynamics).
+
 Every detector composes with v0.34 `soup why` (anomaly explainer), v0.32 spike recovery, and the v0.53.11 #127 `GRPOStabilityCallback` so a single GRPO run can have InfoRM + echo-trap + spike-recovery + in-place ref-model EMA all active simultaneously without duplicating trajectory / state collection. The reward-hack and echo-trap callbacks read the per-step rewards through a shared, thread-safe capture buffer (Soup wraps your reward functions so it never has to monkeypatch TRL); `rm_ensemble` needs ≥2 reward functions to compute a divergence. The MiniLLM teacher-mix is an offline distribution-blend analog of the paper's on-policy teacher-mixed *sampling*, and ULD compares the distributions after clamping teacher ids to the teacher vocab (correct for same-family / extended-vocab pairs; a genuinely different tokenization needs a sequence-alignment step). The reference-model EMA (`--ref-model-ema-alpha`) updates in place — no full `state_dict` round-trip — so it is cheap at 70B+ scale.
 
 

{soup_cli-0.71.25 → soup_cli-0.71.26}/examples/README.md

@@ -402,6 +402,17 @@ output_dir: ./output_advanced/
 
 See the [config schema](../src/soup_cli/config/schema.py) (the single source of truth) for all available options.
 
+## Reward-hacking mitigation demo
+
+`reward_hacking/rewards.py` provides synthetic reward functions for the
+closed-loop reward-hacking mitigation feature (`soup train
+--reward-hack-mitigation`): a gameable `length_hack_reward` / `sentinel_reward`
+proxy decoupled from a held-out `true_score`. Point a GRPO config's
+`training.reward_fn` at a `.py` that re-exports one as `reward_fn`, enable
+`reward_hack_detector: info_rm` + `reward_hack_mitigation: kl_control`, and watch
+`mitigation_log.jsonl` under the run's output dir. See
+[docs/training.md](../docs/training.md#closed-loop-reward-hacking-auto-mitigation-v07126).
+
 ## Learn More
 
 - **README**: [Main documentation](../README.md)

soup_cli-0.71.26/examples/reward_hacking/rewards.py

@@ -0,0 +1,83 @@
+"""Synthetic reward-hacking task for the v0.71.26 closed-loop mitigation demo.
+
+These reward functions are the reusable GPU-experiment fixture for the
+reward-hacking auto-mitigation controller. They implement a *gameable* proxy
+reward that is deliberately decoupled from a held-out *true* quality score, so
+a policy can visibly "hack" the proxy while the true score stagnates or drops.
+
+Use one of the proxies as the training reward and log ``true_score`` alongside
+it to see the divergence:
+
+    soup train --config examples/reward_hacking/grpo_length_hack.yaml
+
+Reward-function signature (TRL GRPO/PPO): ``fn(prompts, completions, **kwargs)
+-> list[float]``. ``completions`` are plain strings (non-conversational) or
+lists of ``{"role", "content"}`` message dicts (conversational).
+"""
+
+from typing import Any
+
+# The synthetic "correct answer" the TRUE scorer looks for. Orthogonal to both
+# the length proxy and the sentinel proxy.
+_TARGET_ANSWER = "42"
+# The gameable sentinel token the sentinel proxy pays for.
+_SENTINEL = "GOLD"
+# Length (in whitespace tokens) at which the length proxy saturates.
+_LENGTH_SATURATION = 32.0
+# Above this word count the true scorer considers the answer padded / verbose.
+_CONCISE_MAX_WORDS = 16
+
+
+def _text(completion: Any) -> str:
+    """Extract assistant text from a string or list-of-message-dicts completion."""
+    if isinstance(completion, str):
+        return completion
+    if isinstance(completion, dict):
+        return str(completion.get("content", ""))
+    if isinstance(completion, (list, tuple)):
+        parts = []
+        for item in completion:
+            if isinstance(item, dict):
+                parts.append(str(item.get("content", "")))
+            else:
+                parts.append(str(item))
+        return "".join(parts)
+    return str(completion)
+
+
+def length_hack_reward(prompts: Any, completions: Any, **kwargs: Any) -> list[float]:
+    """Gameable proxy: reward proportional to completion length (bounded [0, 1]).
+
+    A policy maximises this by padding output — orthogonal to real quality.
+    """
+    return [
+        min(1.0, len(_text(comp).split()) / _LENGTH_SATURATION)
+        for comp in completions
+    ]
+
+
+def sentinel_reward(prompts: Any, completions: Any, **kwargs: Any) -> list[float]:
+    """Gameable proxy: +1 iff the completion contains the sentinel token."""
+    return [1.0 if _SENTINEL in _text(comp) else 0.0 for comp in completions]
+
+
+def true_score(prompts: Any, completions: Any, **kwargs: Any) -> list[float]:
+    """Held-out TRUE quality, decoupled from both proxies.
+
+    Rewards the correct answer token concisely: 1.0 for a concise correct
+    answer, 0.3 for a correct-but-verbose answer, 0.0 otherwise. Padding
+    (length hack) pushes past the concise limit and spamming the sentinel does
+    nothing here — so gaming either proxy does NOT raise the true score.
+    """
+    out: list[float] = []
+    for comp in completions:
+        text = _text(comp)
+        has_answer = _TARGET_ANSWER in text
+        concise = len(text.split()) <= _CONCISE_MAX_WORDS
+        if has_answer and concise:
+            out.append(1.0)
+        elif has_answer:
+            out.append(0.3)
+        else:
+            out.append(0.0)
+    return out

{soup_cli-0.71.25 → soup_cli-0.71.26}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.25"
+version = "0.71.26"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.25 → soup_cli-0.71.26}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
 
-__version__ = "0.71.25"
+__version__ = "0.71.26"

{soup_cli-0.71.25 → soup_cli-0.71.26}/src/soup_cli/commands/train.py

@@ -167,6 +167,31 @@ def train(
             "training.echo_trap_enabled=true on grpo/ppo."
         ),
     ),
+    reward_hack_detector: str = typer.Option(
+        None,
+        "--reward-hack-detector",
+        help=(
+            "Reward-hacking detector for GRPO/PPO: info_rm | rm_ensemble. "
+            "Overrides training.reward_hack_detector. (v0.71.26)"
+        ),
+    ),
+    reward_hack_halt: bool = typer.Option(
+        False,
+        "--reward-hack-halt",
+        help=(
+            "Auto-halt training on a HACK verdict. Requires "
+            "--reward-hack-detector (or training.reward_hack_detector). (v0.71.26)"
+        ),
+    ),
+    reward_hack_mitigation: str = typer.Option(
+        None,
+        "--reward-hack-mitigation",
+        help=(
+            "Closed-loop reward-hacking mitigation mode: off | log_only | "
+            "kl_control | pid_lagrangian. Requires training.reward_hack_detector "
+            "on grpo/ppo. Overrides training.reward_hack_mitigation. (v0.71.26)"
+        ),
+    ),
     minillm_on_policy: bool = typer.Option(
         False,
         "--minillm-on-policy",
@@ -344,6 +369,48 @@ def train(
         cfg.training.echo_trap_tokenizer_aware = True
         console.print("[green]Echo-trap tokenizer-aware scoring enabled[/]")
 
+    # --- Reward-hack detector / halt shortcut (v0.71.26) ---
+    if reward_hack_detector is not None:
+        if reward_hack_detector not in ("info_rm", "rm_ensemble"):
+            console.print(
+                "[red]--reward-hack-detector must be info_rm or rm_ensemble[/]"
+            )
+            raise typer.Exit(1)
+        cfg.training.reward_hack_detector = reward_hack_detector
+        console.print(f"[green]Reward-hack detector:[/] {reward_hack_detector}")
+    if reward_hack_halt:
+        if cfg.training.reward_hack_detector is None:
+            console.print(
+                "[red]--reward-hack-halt requires --reward-hack-detector "
+                "(or training.reward_hack_detector)[/]"
+            )
+            raise typer.Exit(1)
+        cfg.training.reward_hack_halt = True
+        console.print("[green]Reward-hack auto-halt enabled[/]")
+
+    # --- Reward-hack mitigation shortcut (v0.71.26) ---
+    if reward_hack_mitigation is not None:
+        valid_modes = ("off", "log_only", "kl_control", "pid_lagrangian")
+        if reward_hack_mitigation not in valid_modes:
+            console.print(
+                "[red]--reward-hack-mitigation must be one of "
+                f"{', '.join(valid_modes)}[/]"
+            )
+            raise typer.Exit(1)
+        if (
+            reward_hack_mitigation != "off"
+            and cfg.training.reward_hack_detector is None
+        ):
+            console.print(
+                "[red]--reward-hack-mitigation requires "
+                "training.reward_hack_detector to be set (the signal source)[/]"
+            )
+            raise typer.Exit(1)
+        cfg.training.reward_hack_mitigation = reward_hack_mitigation
+        console.print(
+            f"[green]Reward-hack mitigation:[/] {reward_hack_mitigation}"
+        )
+
     # --- MiniLLM on-policy rollout shortcut (v0.71.18 #257) ---
     if minillm_on_policy:
         if not cfg.training.minillm_enabled:
@@ -618,6 +685,16 @@ def train(
                     script_args.append("--tensorboard")
                 if echo_trap_tokenizer_aware:
                     script_args.append("--echo-trap-tokenizer-aware")
+                if reward_hack_detector is not None:
+                    script_args.extend(
+                        ["--reward-hack-detector", reward_hack_detector]
+                    )
+                if reward_hack_halt:
+                    script_args.append("--reward-hack-halt")
+                if reward_hack_mitigation is not None:
+                    script_args.extend(
+                        ["--reward-hack-mitigation", reward_hack_mitigation]
+                    )
                 if yes:
                     script_args.append("--yes")
                 argv = build_accelerate_argv(