soup-cli 0.71.24__tar.gz → 0.71.26__tar.gz

{soup_cli-0.71.24 → soup_cli-0.71.26}/CHANGELOG.md

@@ -12,6 +12,76 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.26] - 2026-07-01
+
+### Added
+- **Closed-loop reward-hacking auto-mitigation.** The trainer now *detects*
+  reward hacking mid-run and *self-corrects* — instead of only halting. Set
+  `training.reward_hack_mitigation` (or `soup train --reward-hack-mitigation`)
+  to one of four modes on a GRPO/PPO run (requires `reward_hack_detector`):
+  - `log_only` — instrument only: append a per-step `mitigation_log.jsonl`
+    (drop_pct, verdict, reward mean/std, completion-length trend, repetition)
+    and *never* touch training.
+  - `kl_control` — a reversible **bang-bang + hysteresis** controller: when the
+    hacking signal trips, raise the KL coefficient β (geometric, clamped to
+    `[floor, ceil]`, never crossing 0); relax it when the signal recovers.
+    Dwell + release-patience prevent flapping; a multi-signal vote combines the
+    detector drop with a length-trend and repetition signal.
+  - `pid_lagrangian` — a **PID-Lagrangian** controller (Stooke et al.) that
+    holds the hacking signal at a target, plus an **escalation ladder**:
+    raise β → roll back to the last-good RL checkpoint → early-stop.
+  - Anti-gaming hardening: per-signal EMA/median smoothing,
+    conservative-on-disagreement voting, a reward-distribution-drift guard, and
+    optional bounded **reward shaping** on the gamed proxy (length / repetition
+    / sentinel). A plain-English give-up explanation is logged on early-stop.
+  - **Proof-of-mechanism only** (see Known Limitations): validated on
+    SmolLM2-135M + a synthetic length-hacking task on a single RTX 3050 — all
+    four stages pass live, including a real mid-run rollback. PPO ships **BETA**
+    (mechanism unit-tested; the on-GPU proof is GRPO-only).
+- Ready-made `qwen2.5-coder-7b-sft` recipe for `Qwen/Qwen2.5-Coder-7B-Instruct`
+  (catalog 133 → 134)
+  ([#285](https://github.com/MakazhanAlpamys/Soup/pull/285) by [@Deadpool2000](https://github.com/Deadpool2000)).
+
+### Security
+- `RLCheckpointCallback.restore_checkpoint` / `save_checkpoint` refuse a
+  **symlinked** `optimizer.pt` — `torch.load(weights_only=False)` on an
+  attacker-placed symlink in a shared checkpoint dir was an RCE vector.
+- Bool-before-int/float guards on every new `reward_hack_*` numeric field;
+  `reward_hack_signals` bounded (`max_length=4`); the mitigation log writer is
+  cwd-contained with symlink-reject-on-rotate and secret redaction.
+
+## [0.71.25] - 2026-06-27
+
+### Added
+- **`soup ship` — the SHIP / DON'T-SHIP verdict.** After fine-tuning, answer one
+  question: did the model get better, or did I break it? `soup ship` fuses two
+  checks into a single binary decision — **leg 1**: the task metric *strictly*
+  improved (base → tuned); AND **leg 2**: no general benchmark regressed past a
+  forgetting threshold (default 0.05 absolute points). It SHIPs only when both
+  hold — otherwise DON'T SHIP, *even if the task metric looks great*. The output
+  is a one-screen verdict + the reason, with CI-gateable exit codes
+  (0 = SHIP, 2 = DON'T SHIP, 1 = runtime error).
+  - Leg-1 modes: `--task-mode metric` (eval accuracy) or `judge_score`
+    (LLM-as-a-judge); pairwise win-rate is planned for a later release.
+  - Leg-2 suite: built-in mini benchmarks by default (offline, CPU), or
+    `--general-suite <names>` to route lm-eval benchmarks; `--baseline
+    registry://… | file.json` supplies recorded base scores.
+  - `--evidence ev.json` decides offline from pre-computed scores (no model
+    load); `--output verdict.json` persists the machine-readable verdict.
+- Friendlier error messages: the CUDA-OOM hint now also suggests
+  `gradient_checkpointing` and `4bit` quantization, plus new mappings for
+  Hugging Face gated repos (`huggingface-cli login` / `HF_TOKEN`) and
+  `trust_remote_code` errors. Closes #272
+  ([#282](https://github.com/MakazhanAlpamys/Soup/pull/282) by [@Akshaya-reddy18](https://github.com/Akshaya-reddy18)).
+
+### Security
+- `soup ship` input hardening: `--evidence` is opened with `O_NOFOLLOW` + an
+  fstat size cap (16 MiB) under cwd containment; `--task-eval` is cwd-contained
+  and symlink-rejected; `--judge-model` is validated by scheme/host via
+  `urlparse` (blocks the `http://localhost.attacker.com` prefix bypass);
+  lm-eval model ids reject `,`/`=` injection; `--general-suite` is bounded
+  (≤ 50 names, ≤ 256 chars each).
+
 ## [0.71.24] - 2026-06-21
 
 ### Added

{soup_cli-0.71.24 → soup_cli-0.71.26}/CONTRIBUTING.md

@@ -110,17 +110,17 @@ src/soup_cli/
   experiment/         - SQLite experiment tracking
   eval/               - Eval platform (custom tasks, LLM judge, human eval, leaderboard)
   migrate/            - Config migration (LLaMA-Factory, Axolotl, Unsloth)
-  recipes/            - Ready-made configs for popular models (133 recipes)
+  recipes/            - Ready-made configs for popular models (134 recipes)
   autopilot/          - Zero-config decision engine (v0.25.0)
   registry/           - Model Registry (hashing, store, diff, attach) (v0.26.0 + v0.33.0)
   cans/               - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0)
   data/traces/        - Trace-to-Preference harvester (v0.26.0)
   data/collators.py   - CrossDocCollator for sample packing (v0.33.0)
-  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler
+  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict
   templates/          - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (296 files, 14514 tests)
+tests/                - Test suite (298 files, 14788 tests)
 examples/             - Real-world config examples and datasets
 ```
 
@@ -150,7 +150,7 @@ pytest tests/test_data.py::test_detect_alpaca_format -v
 pytest tests/ --cov=soup_cli --cov-report=html
 ```
 
-### Test Files (296 files)
+### Test Files (297 files)
 
 > A representative sample of the suite below. The full table lives in
 > [`.claude/CLAUDE.md`](.claude/CLAUDE.md); run `pytest tests/ -v` for the complete list.

{soup_cli-0.71.24 → soup_cli-0.71.26}/CONTRIBUTORS.md

@@ -25,6 +25,7 @@ Listed by first contribution. PR numbers link the work.
   - Happy-path + CPU-warning tests for `soup bench` ([#31](https://github.com/MakazhanAlpamys/Soup/pull/31))
   - `soup cost` — cloud GPU training cost estimation ([#42](https://github.com/MakazhanAlpamys/Soup/pull/42))
   - `--nccl` flag for `soup doctor` multi-GPU bandwidth checks ([#178](https://github.com/MakazhanAlpamys/Soup/pull/178))
+  - Ready-made `qwen2.5-coder-7b-sft` recipe ([#285](https://github.com/MakazhanAlpamys/Soup/pull/285))
 - **Chinmaya Sahu** ([@csking101](https://github.com/csking101))
   - DPO example config, sample data, and tests ([#48](https://github.com/MakazhanAlpamys/Soup/pull/48))
   - FP8 `rowwise` + `rowwise_with_gw_hp` scaling recipes ([#62](https://github.com/MakazhanAlpamys/Soup/pull/62))
@@ -41,6 +42,8 @@ Listed by first contribution. PR numbers link the work.
   - `--energy` flag for `soup bom emit` — thread energy/CO₂ into the ML-BOM ([#256](https://github.com/MakazhanAlpamys/Soup/pull/256))
 - **shatakshi-1404** ([@shatakshi-1404](https://github.com/shatakshi-1404))
   - Unit tests for the `warmup.py` auto-warmup-steps helper ([#274](https://github.com/MakazhanAlpamys/Soup/pull/274))
+- **Kondamwar Akshaya Shrikant** ([@Akshaya-reddy18](https://github.com/Akshaya-reddy18))
+  - Friendlier error messages — richer CUDA-OOM hint + Hugging Face gated-repo and `trust_remote_code` mappings + tests ([#282](https://github.com/MakazhanAlpamys/Soup/pull/282))
 
 ---
 

{soup_cli-0.71.24 → soup_cli-0.71.26}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.24
+Version: 0.71.26
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -189,17 +189,22 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.24 — 2026 model-family recipes (catalog 116 → 133).** 17 new ready-made SFT recipes for the open-weight models that shipped Feb–Jun 2026:
+**v0.71.26 — Closed-loop reward-hacking auto-mitigation.** Your GRPO/PPO trainer now *detects* reward hacking mid-run and *self-corrects* — instead of only halting. No OSS RLHF library closes this loop.
 
-- **Qwen 3.5 family (Apache-2.0)** — `qwen3.5-0.8b/2b/4b/9b/27b-sft` dense, plus the
-  `35b-a3b` / `122b-a10b` / `397b-a17b` MoE sizes (262K context, native vision).
-- **Qwen 3.6 (Apache-2.0)** — `qwen3.6-27b-sft` + `qwen3.6-35b-a3b-sft`.
-- **Frontier MoE** — `deepseek-v4-flash-sft` / `deepseek-v4-pro-sft` (MIT),
-  `glm-5.1-sft` (MIT), `kimi-k2.5-sft` / `kimi-k2.6-sft` (Modified MIT),
-  `minimax-m3-sft` (MiniMax Community License), `mistral-large-3-sft` (Apache-2.0).
-- **Stale repo-ID fix** — `glm-5-sft` now points at `zai-org/GLM-5` (the org migrated from `THUDM`).
-- Every base repo-ID was verified to resolve on Hugging Face. Grab one with
-  `soup recipes use <name>` or browse all 133 via `soup recipes list`.
+- **Detect → correct → continue.** When the hacking signal trips, the controller raises the
+  KL coefficient β (bang-bang + hysteresis, or a PID-Lagrangian controller), and — if hacking
+  persists — **rolls back to the last-good checkpoint**, then early-stops as a last resort.
+- **Anti-gaming hardening.** Multi-signal voting (RM cluster-separation + length-trend +
+  repetition), signal smoothing, conservative-on-disagreement, a reward-distribution-drift
+  guard, and optional bounded **reward shaping** on the gamed proxy.
+- **Observe first.** `reward_hack_mitigation=log_only` streams a per-step `mitigation_log.jsonl`
+  and never touches training — see the hacking happen before you let the controller act.
+- **Honest scope.** Proof-of-mechanism on SmolLM2-135M + a synthetic hacking task on one RTX
+  3050 (all four stages validated live, including a real rollback). PPO ships BETA.
+
+```bash
+soup train --config grpo.yaml --reward-hack-mitigation kl_control   # detect → raise KL → recover
+```
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
 
@@ -385,6 +390,13 @@ pre-commit install                 # optional: ruff lint+format on commit
 See [CONTRIBUTING.md](CONTRIBUTING.md) for the full workflow and [SECURITY.md](SECURITY.md) to
 report a vulnerability.
 
+## Contributors
+
+Built by the community ❤️ — thank you to everyone who has contributed. See
+[CONTRIBUTORS.md](CONTRIBUTORS.md).
+
+[![Contributors](https://contrib.rocks/image?repo=MakazhanAlpamys/Soup)](https://github.com/MakazhanAlpamys/Soup/graphs/contributors)
+
 ## License
 
 [Apache-2.0](LICENSE). Copyright © the Soup contributors.

{soup_cli-0.71.24 → soup_cli-0.71.26}/README.md

@@ -49,17 +49,22 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.24 — 2026 model-family recipes (catalog 116 → 133).** 17 new ready-made SFT recipes for the open-weight models that shipped Feb–Jun 2026:
-
-- **Qwen 3.5 family (Apache-2.0)** — `qwen3.5-0.8b/2b/4b/9b/27b-sft` dense, plus the
-  `35b-a3b` / `122b-a10b` / `397b-a17b` MoE sizes (262K context, native vision).
-- **Qwen 3.6 (Apache-2.0)** — `qwen3.6-27b-sft` + `qwen3.6-35b-a3b-sft`.
-- **Frontier MoE** — `deepseek-v4-flash-sft` / `deepseek-v4-pro-sft` (MIT),
-  `glm-5.1-sft` (MIT), `kimi-k2.5-sft` / `kimi-k2.6-sft` (Modified MIT),
-  `minimax-m3-sft` (MiniMax Community License), `mistral-large-3-sft` (Apache-2.0).
-- **Stale repo-ID fix** — `glm-5-sft` now points at `zai-org/GLM-5` (the org migrated from `THUDM`).
-- Every base repo-ID was verified to resolve on Hugging Face. Grab one with
-  `soup recipes use <name>` or browse all 133 via `soup recipes list`.
+**v0.71.26 — Closed-loop reward-hacking auto-mitigation.** Your GRPO/PPO trainer now *detects* reward hacking mid-run and *self-corrects* — instead of only halting. No OSS RLHF library closes this loop.
+
+- **Detect → correct → continue.** When the hacking signal trips, the controller raises the
+  KL coefficient β (bang-bang + hysteresis, or a PID-Lagrangian controller), and — if hacking
+  persists — **rolls back to the last-good checkpoint**, then early-stops as a last resort.
+- **Anti-gaming hardening.** Multi-signal voting (RM cluster-separation + length-trend +
+  repetition), signal smoothing, conservative-on-disagreement, a reward-distribution-drift
+  guard, and optional bounded **reward shaping** on the gamed proxy.
+- **Observe first.** `reward_hack_mitigation=log_only` streams a per-step `mitigation_log.jsonl`
+  and never touches training — see the hacking happen before you let the controller act.
+- **Honest scope.** Proof-of-mechanism on SmolLM2-135M + a synthetic hacking task on one RTX
+  3050 (all four stages validated live, including a real rollback). PPO ships BETA.
+
+```bash
+soup train --config grpo.yaml --reward-hack-mitigation kl_control   # detect → raise KL → recover
+```
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
 
@@ -245,6 +250,13 @@ pre-commit install                 # optional: ruff lint+format on commit
 See [CONTRIBUTING.md](CONTRIBUTING.md) for the full workflow and [SECURITY.md](SECURITY.md) to
 report a vulnerability.
 
+## Contributors
+
+Built by the community ❤️ — thank you to everyone who has contributed. See
+[CONTRIBUTORS.md](CONTRIBUTORS.md).
+
+[![Contributors](https://contrib.rocks/image?repo=MakazhanAlpamys/Soup)](https://github.com/MakazhanAlpamys/Soup/graphs/contributors)
+
 ## License
 
 [Apache-2.0](LICENSE). Copyright © the Soup contributors.

{soup_cli-0.71.24 → soup_cli-0.71.26}/docs/README.md

@@ -11,7 +11,7 @@ feature reference — every `soup` capability, grouped by area.
 | [PEFT, long context & efficiency](peft-and-efficiency.md) | DoRA, LoRA+, rsLoRA, VeRA, OLoRA, NEFTune, PiSSA, ReLoRA, optimizer & PEFT zoo, LLaMA Pro, GaLore, YaRN/LongLoRA, packing, curriculum, auto-tuning |
 | [Performance & quantization](performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, multi-GPU / DeepSpeed / FSDP |
 | [Data engineering](data.md) | Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs |
-| [Evaluation & probes](evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, post-train X-ray probes, A/B, drift, tunability, `soup advise` |
+| [Evaluation & probes](evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, `soup ship` verdict, post-train X-ray probes, A/B, drift, tunability, `soup advise` |
 | [Serving & export](serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding, deploy autopilot, Web UI, Agent Forge |
 | [Adapters, registry & governance](adapters-and-governance.md) | Adapter lifecycle/management, model registry, Soup Cans, the data flywheel (`soup loop`), knowledge editing, steering, supply-chain controls |
 | [Backends, platform & ops](backends-and-ops.md) | MLX/Unsloth backends, Modal cloud GPU training, alternative hubs, HF Hub integration, autopilot, experiment tracking, plan/apply, env lockfiles, hardware-fit, completions, plugins, utility commands |

{soup_cli-0.71.24 → soup_cli-0.71.26}/docs/commands.md

@@ -129,7 +129,7 @@ soup migrate --from llamafactory config.yaml  Import config from LLaMA-Factory
 soup migrate --from axolotl config.yml        Import config from Axolotl
 soup migrate --from unsloth notebook.ipynb    Import config from Unsloth notebook
 soup migrate --from llamafactory c.yaml --dry-run  Preview without writing
-soup recipes list                             List all 133 ready-made recipes
+soup recipes list                             List all 134 ready-made recipes
 soup recipes show llama3.1-8b-sft            Print recipe YAML
 soup recipes use llama3.1-8b-sft             Copy recipe to soup.yaml
 soup recipes search "reasoning"              Search by keyword/task/size
@@ -148,6 +148,10 @@ soup runs show <run_id>                       Run details + loss graph + cost
 soup runs compare <run_1> <run_2>             Compare two runs
 soup runs replay <run_id>                     Replay summary + loss curve from history (also plots a benchmark-score curve when the metric lives in eval_results)
 soup why [run_id]                             Explain training anomalies (heuristic)
+soup ship --base <m> --adapter <lora> --task-eval t.jsonl  SHIP / DON'T-SHIP verdict: task win AND no forgetting (exit 0=SHIP / 2=DON'T / 1=error) (v0.71.25)
+soup ship --evidence ev.json [--output v.json]  Decide offline from pre-computed scores (no model load)
+soup ship ... --task-mode judge_score --judge-model ollama://llama3.1  Leg-1 via LLM-as-a-judge
+soup ship ... --general-suite mmlu,gsm8k --baseline base.json  lm-eval leg-2 + recorded base scores
 soup tui                                      Full-screen Textual dashboard (requires [tui] extra)
 soup train --config soup.yaml --profile       Record torch.profiler trace to <output>/profiles/
 soup --log-level quiet|normal|verbose|debug   Global logging tier (Rich-formatted)
@@ -241,6 +245,7 @@ soup data persona-mix --prompts <jsonl> --n N --output <jsonl>  Persona-Hub dive
 soup data brain-rot <data.jsonl> [--strict]   Brain-rot detector — arXiv 2510.13928 (v0.69.0)
 soup iterative-dpo --base-model <m> --reward-model <rm> --prompts <p.jsonl> --output-dir <out> --rounds N --pairs-per-round N [--plan-only]  Iterative DPO loop driver — LIVE sample→score→pair→train (v0.70.0; live v0.71.11)
 soup train --reward-hack-detector info_rm|rm_ensemble [--reward-hack-halt]  Reward-hacking detector for GRPO — LIVE callback (v0.70.0; live v0.71.11)
+soup train --reward-hack-mitigation off|log_only|kl_control|pid_lagrangian  Closed-loop reward-hacking auto-mitigation (detect → raise KL/β → rollback → early-stop); GRPO/PPO, requires --reward-hack-detector; PPO BETA (v0.71.26)
 soup train --uld-strategy wasserstein|topk_align [--uld-top-k N]  Cross-tokenizer ULD on task='distill' — LIVE W1/topk loss (v0.70.0; live v0.71.11)
 soup train --minillm-enabled [--minillm-teacher-mix-ratio 0.3]  MiniLLM reverse-KL distillation — LIVE (v0.70.0; live v0.71.11)
 soup train --rl-checkpoint-save-every-steps N [--rl-checkpoint-keep-last N]  Mid-epoch checkpoint for GRPO/PPO — LIVE (v0.70.0; live v0.71.11)

{soup_cli-0.71.24 → soup_cli-0.71.26}/docs/evaluation.md

@@ -14,6 +14,7 @@
 - [Sequential A/B Harness (`soup ab`)](#sequential-ab-harness-soup-ab)
 - [Drift Alarm (`soup drift-alarm`)](#drift-alarm-soup-drift-alarm)
 - [Diagnose (Post-Training Report Card)](#diagnose-post-training-report-card)
+- [Ship Verdict (`soup ship`)](#ship-verdict-soup-ship)
 - [NLG Evaluation Metrics (BLEU + ROUGE)](#nlg-evaluation-metrics-bleu--rouge)
 - [Quant Calibration (KL Divergence)](#quant-calibration-kl-divergence)
 - [Model Arena (Elo Tournament)](#model-arena-elo-tournament)
@@ -301,6 +302,61 @@ a built-in probe set; `format` only fires when the dataset's own targets look li
 **Post-training gate:** `soup train --diagnose-gate <evidence.json>` runs the same scorer after training finishes and refuses to mark the run successful when any mode comes back MAJOR. Composes with `--gate <eval-suite>` (v0.26) — the eval gate catches accuracy regressions vs a baseline; the diagnose gate catches behaviour regressions the eval suite is blind to.
 
 
+## Ship Verdict (`soup ship`)
+
+After fine-tuning, `soup ship` answers one question — did the model get better, or did I
+break it? — as a single binary **SHIP / DON'T SHIP** plus a one-screen reason. It fuses two
+legs into one decision:
+
+- **Leg 1 (task win):** the task metric *strictly* improved, base → tuned.
+- **Leg 2 (the moat):** *no* general benchmark regressed past the forgetting threshold
+  (default `0.05` **absolute** points — same semantics as `EvalGateConfig.regression_threshold`).
+
+```
+SHIP  ⇔  task_tuned > task_base  AND  ∀ benchmark: base − tuned ≤ forgetting_threshold
+else DON'T SHIP — even if the task metric looks great.
+```
+
+Exit codes are CI-gateable: **0 = SHIP, 2 = DON'T SHIP, 1 = runtime error**. A tie on leg 1, a
+single regressed benchmark, or a missing baseline all yield DON'T SHIP (a missing baseline
+*refuses* rather than silently shipping).
+
+```bash
+# Live: base vs a LoRA adapter, metric leg + default mini benchmarks
+soup ship --base HuggingFaceTB/SmolLM2-135M-Instruct --adapter ./out \
+  --task-eval tasks.jsonl --device cuda
+
+# Leg-1 via LLM-as-a-judge instead of an accuracy metric
+soup ship --base <m> --adapter ./out --task-eval tasks.jsonl \
+  --task-mode judge_score --judge-model ollama://llama3.1
+
+# Leg-2 via lm-eval benchmarks, base scores supplied by --baseline
+soup ship --base <m> --tuned ./out --task-eval tasks.jsonl \
+  --general-suite mmlu,hellaswag --baseline registry://abc123
+
+# Offline: decide from pre-computed scores, persist the verdict JSON
+soup ship --evidence evidence.json --output verdict.json
+```
+
+**`--evidence` shape** (no model load — the offline / CI path):
+
+```json
+{
+  "task": {"mode": "metric", "base": 0.40, "tuned": 0.55},
+  "benchmarks": {
+    "mini_mmlu": {"base": 0.80, "tuned": 0.79},
+    "mini_common_sense": {"base": 0.60, "tuned": 0.62}
+  }
+}
+```
+
+Leg-2 defaults to the built-in mini benchmarks (`mini_mmlu` / `mini_common_sense` /
+`mini_instruction`) — offline, CPU, instant. `--general-suite <names>` with non-mini names
+routes through the lm-eval harness. Pairwise judge win-rate (`--task-mode pairwise`) is planned
+for a later release. The engine lives in `soup_cli.utils.ship_verdict` (`decide_ship` is a pure
+function — the whole truth table is CPU-testable).
+
+
 ## NLG Evaluation Metrics (BLEU + ROUGE)
 
 Pure-Python BLEU + ROUGE-1 / ROUGE-2 / ROUGE-L for `soup eval custom`:

{soup_cli-0.71.24 → soup_cli-0.71.26}/docs/serving-and-export.md

@@ -418,7 +418,7 @@ soup ui
 
 **Pages:**
 - **Dashboard** — view all experiment runs, loss charts, system info, multi-run comparison
-- **New Training** — create configs from templates or 133 ready-made recipes, validate, start training with live SSE log streaming and progress bar
+- **New Training** — create configs from templates or 134 ready-made recipes, validate, start training with live SSE log streaming and progress bar
 - **Data Explorer** — browse and inspect datasets (JSONL, JSON, CSV, Parquet)
 - **Model Chat** — chat with streaming responses, configurable temperature/top_p/max_tokens, system prompt, adapter selection, markdown rendering, chat export
 
@@ -427,7 +427,7 @@ soup ui
 - **Enhanced Metrics** — 2x2 chart grid (loss, LR, grad_norm, throughput) + GPU memory chart, eval results table
 - **Multi-Run Compare** — overlay loss curves from up to 5 runs side-by-side
 - **Chat Upgrade** — SSE streaming via proxy, typing indicator, cancel button, markdown renderer (bold, italic, code blocks), chat export as JSON
-- **Config Builder** — recipe dropdown (133 recipes), config schema API for dynamic form generation
+- **Config Builder** — recipe dropdown (134 recipes), config schema API for dynamic form generation
 
 **Security:** The Web UI generates a random auth token at startup (printed to console). All mutating endpoints (start/stop training, delete runs, inspect data, validate config) require `Authorization: Bearer <token>` header. CORS is restricted to the served origin. Data inspection is sandboxed to the working directory.
 

{soup_cli-0.71.24 → soup_cli-0.71.26}/docs/training.md

@@ -50,6 +50,10 @@ Six surfaces protect the training loop from the failure modes that cost a real G
 soup train --config soup.yaml \
     --reward-hack-detector info_rm --reward-hack-halt   # halt on HACK verdict
 
+# Closed-loop reward-hacking MITIGATION (v0.71.26) — detect AND self-correct
+soup train --config grpo.yaml --reward-hack-mitigation kl_control   # raise KL, recover
+soup train --config grpo.yaml --reward-hack-mitigation log_only     # observe only, no action
+
 # Cross-tokenizer distillation — Llama -> Mistral, no shared vocab needed
 # (Universal Logit Distillation, Boizard et al. 2024 arXiv:2402.12030)
 soup train --config soup.yaml --uld-strategy wasserstein
@@ -100,6 +104,17 @@ soup train --config grpo.yaml \
 
 `--echo-trap-tokenizer-aware` switches echo-trap n-grams from whitespace tokens to the active tokenizer's integer ids. This catches subword repetition that punctuation-heavy decoded text can hide, but the score becomes tokenizer-specific rather than vocabulary-agnostic.
 
+### Closed-loop reward-hacking auto-mitigation (v0.71.26)
+
+The detectors above *halt*; `training.reward_hack_mitigation` (or the `--reward-hack-mitigation` flag) makes the trainer *self-correct* mid-run. It requires `reward_hack_detector` on a `grpo`/`ppo` transformers run, and has four modes:
+
+- **`log_only`** — observe only. Appends a per-step `mitigation_log.jsonl` under the run's output dir (the InfoRM/ensemble drop, the OK/WARN/HACK verdict, reward mean/std, completion-length trend, repetition) and provably never mutates β. Run this first to *see* hacking before you let a controller act on it.
+- **`kl_control`** — a reversible **bang-bang + hysteresis** controller. When a multi-signal vote (`reward_hack_signals`: the detector drop + `length_trend` + `repetition`) stays above `reward_hack_trip_band` for `reward_hack_dwell_steps`, it multiplies β by `reward_hack_kl_gain` (clamped to `[reward_hack_beta_floor > 0, reward_hack_beta_ceil]`, never crossing 0); after `reward_hack_release_patience` below-band steps it relaxes β back toward the floor. Dwell + release-patience stop it flapping. β is written to **both** `trainer.beta` and `trainer.args.beta` so it takes effect on stock GRPO *and* Soup's GRPO variants (and `trainer.args.kl_coef` on PPO).
+- **`pid_lagrangian`** — a **PID-Lagrangian** controller (Stooke et al. 2020) that holds the hacking signal at `reward_hack_signal_target` (Kp/Ki/Kd with integral anti-windup via `reward_hack_integral_clamp`), plus an **escalation ladder**: raise β → after `reward_hack_rollback_patience` persistent-HACK steps roll back to the last-good RL checkpoint (needs `rl_checkpoint_save_every_steps`) → after `reward_hack_max_recovery_attempts` rollbacks, early-stop with a plain-English give-up explanation.
+- **Anti-gaming hardening** (any control mode): `reward_hack_signal_smoothing` (`ema`/`median` over `reward_hack_smoothing_window`), `reward_hack_conservative_on_disagreement` (when detectors disagree, keep KL high + guard against a bimodal reward-distribution collapse), and `reward_hack_reward_shaping` (subtract a bounded `reward_hack_shaping_strength` penalty on the gamed proxy — `length`/`repetition`/`sentinel` — over the reward-fn seam).
+
+**Scope:** proof-of-mechanism only. Validated on SmolLM2-135M + a synthetic length-hacking task on a single RTX 3050 (all four modes live, including a real mid-run rollback). PPO ships **BETA** — the buffer + `kl_coef` mutation are wired and unit-tested, but the on-GPU proof is GRPO-only. Whether the loop suppresses hacking without collapsing true reward on 7B+ with a real reward model is an open, community-validatable question. `reward_hack_mitigation ∈ {kl_control, pid_lagrangian}` is mutually exclusive with `ref_model_ema_alpha` (both drive the KL/ref dynamics).
+
 Every detector composes with v0.34 `soup why` (anomaly explainer), v0.32 spike recovery, and the v0.53.11 #127 `GRPOStabilityCallback` so a single GRPO run can have InfoRM + echo-trap + spike-recovery + in-place ref-model EMA all active simultaneously without duplicating trajectory / state collection. The reward-hack and echo-trap callbacks read the per-step rewards through a shared, thread-safe capture buffer (Soup wraps your reward functions so it never has to monkeypatch TRL); `rm_ensemble` needs ≥2 reward functions to compute a divergence. The MiniLLM teacher-mix is an offline distribution-blend analog of the paper's on-policy teacher-mixed *sampling*, and ULD compares the distributions after clamping teacher ids to the teacher vocab (correct for same-family / extended-vocab pairs; a genuinely different tokenization needs a sequence-alignment step). The reference-model EMA (`--ref-model-ema-alpha`) updates in place — no full `state_dict` round-trip — so it is cheap at 70B+ scale.
 
 

{soup_cli-0.71.24 → soup_cli-0.71.26}/examples/README.md

@@ -402,6 +402,17 @@ output_dir: ./output_advanced/
 
 See the [config schema](../src/soup_cli/config/schema.py) (the single source of truth) for all available options.
 
+## Reward-hacking mitigation demo
+
+`reward_hacking/rewards.py` provides synthetic reward functions for the
+closed-loop reward-hacking mitigation feature (`soup train
+--reward-hack-mitigation`): a gameable `length_hack_reward` / `sentinel_reward`
+proxy decoupled from a held-out `true_score`. Point a GRPO config's
+`training.reward_fn` at a `.py` that re-exports one as `reward_fn`, enable
+`reward_hack_detector: info_rm` + `reward_hack_mitigation: kl_control`, and watch
+`mitigation_log.jsonl` under the run's output dir. See
+[docs/training.md](../docs/training.md#closed-loop-reward-hacking-auto-mitigation-v07126).
+
 ## Learn More
 
 - **README**: [Main documentation](../README.md)

soup_cli-0.71.26/examples/reward_hacking/rewards.py

@@ -0,0 +1,83 @@
+"""Synthetic reward-hacking task for the v0.71.26 closed-loop mitigation demo.
+
+These reward functions are the reusable GPU-experiment fixture for the
+reward-hacking auto-mitigation controller. They implement a *gameable* proxy
+reward that is deliberately decoupled from a held-out *true* quality score, so
+a policy can visibly "hack" the proxy while the true score stagnates or drops.
+
+Use one of the proxies as the training reward and log ``true_score`` alongside
+it to see the divergence:
+
+    soup train --config examples/reward_hacking/grpo_length_hack.yaml
+
+Reward-function signature (TRL GRPO/PPO): ``fn(prompts, completions, **kwargs)
+-> list[float]``. ``completions`` are plain strings (non-conversational) or
+lists of ``{"role", "content"}`` message dicts (conversational).
+"""
+
+from typing import Any
+
+# The synthetic "correct answer" the TRUE scorer looks for. Orthogonal to both
+# the length proxy and the sentinel proxy.
+_TARGET_ANSWER = "42"
+# The gameable sentinel token the sentinel proxy pays for.
+_SENTINEL = "GOLD"
+# Length (in whitespace tokens) at which the length proxy saturates.
+_LENGTH_SATURATION = 32.0
+# Above this word count the true scorer considers the answer padded / verbose.
+_CONCISE_MAX_WORDS = 16
+
+
+def _text(completion: Any) -> str:
+    """Extract assistant text from a string or list-of-message-dicts completion."""
+    if isinstance(completion, str):
+        return completion
+    if isinstance(completion, dict):
+        return str(completion.get("content", ""))
+    if isinstance(completion, (list, tuple)):
+        parts = []
+        for item in completion:
+            if isinstance(item, dict):
+                parts.append(str(item.get("content", "")))
+            else:
+                parts.append(str(item))
+        return "".join(parts)
+    return str(completion)
+
+
+def length_hack_reward(prompts: Any, completions: Any, **kwargs: Any) -> list[float]:
+    """Gameable proxy: reward proportional to completion length (bounded [0, 1]).
+
+    A policy maximises this by padding output — orthogonal to real quality.
+    """
+    return [
+        min(1.0, len(_text(comp).split()) / _LENGTH_SATURATION)
+        for comp in completions
+    ]
+
+
+def sentinel_reward(prompts: Any, completions: Any, **kwargs: Any) -> list[float]:
+    """Gameable proxy: +1 iff the completion contains the sentinel token."""
+    return [1.0 if _SENTINEL in _text(comp) else 0.0 for comp in completions]
+
+
+def true_score(prompts: Any, completions: Any, **kwargs: Any) -> list[float]:
+    """Held-out TRUE quality, decoupled from both proxies.
+
+    Rewards the correct answer token concisely: 1.0 for a concise correct
+    answer, 0.3 for a correct-but-verbose answer, 0.0 otherwise. Padding
+    (length hack) pushes past the concise limit and spamming the sentinel does
+    nothing here — so gaming either proxy does NOT raise the true score.
+    """
+    out: list[float] = []
+    for comp in completions:
+        text = _text(comp)
+        has_answer = _TARGET_ANSWER in text
+        concise = len(text.split()) <= _CONCISE_MAX_WORDS
+        if has_answer and concise:
+            out.append(1.0)
+        elif has_answer:
+            out.append(0.3)
+        else:
+            out.append(0.0)
+    return out

{soup_cli-0.71.24 → soup_cli-0.71.26}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.24"
+version = "0.71.26"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.24 → soup_cli-0.71.26}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
 
-__version__ = "0.71.24"
+__version__ = "0.71.26"

{soup_cli-0.71.24 → soup_cli-0.71.26}/src/soup_cli/cli.py

@@ -233,6 +233,18 @@ app.command(
     ),
 )(_diagnose_cmd.diagnose)
 
+# v0.71.25 — `soup ship` SHIP / DON'T-SHIP verdict engine.
+from soup_cli.commands import ship as _ship_cmd  # noqa: E402
+
+app.add_typer(
+    _ship_cmd.app,
+    name="ship",
+    help=(
+        "SHIP / DON'T SHIP verdict after fine-tuning: task win AND no "
+        "catastrophic forgetting, fused into one decision (v0.71.25)."
+    ),
+)
+
 # v0.58.0 — `soup loop` CLI-first data flywheel capstone.
 from soup_cli.commands import loop as _loop_cmd  # noqa: E402