soup-cli 0.71.39__tar.gz → 0.71.41__tar.gz

{soup_cli-0.71.39 → soup_cli-0.71.41}/CHANGELOG.md

@@ -12,6 +12,73 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.41] - 2026-07-19
+
+**`soup reward stress`: is your reward verifier gameable?** Turn the reward-hacking
+detector on the *verifier itself*. `soup reward synth` (v0.71.40) proves a verifier
+separates your references from friendly perturbations; `stress` asks the adversarial
+question a reward-hacking model asks at train time — *does the verifier pay out for
+degenerate junk?* It feeds empty, length-padded, repetition, and sentinel-spam
+completions and flags any the verifier accepts. Pure, offline, exit 0 = robust /
+2 = gameable / 1 = error. Nothing in TRL / Unsloth / Axolotl / OpenRLHF tests a
+verifier for gameability.
+
+### Added
+
+- **`soup reward stress <reward.py|builtin> [--references golds.jsonl]`** — adversarial
+  verifier probe. Attacks (`--attacks empty,length,repetition,sentinel`, `--sentinel`)
+  are scored against the real gold, so numeric / tool_call / json_schema verifiers get a
+  valid target and still must reject the junk. Reports a per-attack accept-rate table +
+  an overall gameability verdict (`--max-gameable`, `--threshold`, `--output-report`).
+  Loads the target through the existing reward loader, so it probes a synthesized `.py`
+  **and** a builtin (`accuracy` / `format` / `verifiable`). A gold-requiring verifier
+  probed with no `--references` is a hard error, never a false "robust".
+
+### Fixed
+
+- Corrected the Telemetry section in the ops docs: Soup's telemetry primitives exist but
+  are **not wired to any command** — no data is ever sent today (the previous wording
+  implied a live opt-in sender). Wiring is deferred until a public privacy policy ships.
+
+## [0.71.40] - 2026-07-19
+
+**`soup reward synth`: auto-generate a deterministic reward verifier from your data.**
+Point it at a JSONL of reference (gold) outputs and it infers a verifier, emits a
+readable / committable `.py` reward function, and — the moat — *refuses* to emit one
+that can't tell your references from auto-generated bad answers. Nothing in
+TRL / Unsloth / Axolotl / OpenRLHF synthesizes a reward; every reward today is
+hand-written, hand-picked, or a trained-weights artifact.
+
+### Added
+
+- **`soup reward synth <references.jsonl> -o reward.py`** — deterministic verifier
+  synthesis. Four families, auto-detected (or pick with `--kind`): `numeric`
+  (last-number / `\boxed{}` / `####` extraction, exact or `--tolerance`), `json_schema`
+  (induced keys + types + required), `regex` (positional char-classes over
+  equal-length golds), `tool_call` (per-tool `required`/`allowed` argument binding).
+  The emitted file is self-contained and rides `load_reward_fn`'s existing `.py`
+  path — no new trusted-exec surface; you read, edit, commit, and diff it.
+- **Mandatory calibration report** — the synthesized verifier is loaded back and run
+  against its own references (must accept ≥90%) and auto-perturbed negatives (must
+  reject). A degenerate always-accept verifier is **refused** (`exit 2`), never
+  silently emitted. `--plan-only` reports the induced spec without writing;
+  `--output-report` persists the calibration JSON.
+- **Comma-separated `reward_fn` (`"accuracy,format"`)** now trains — it resolves to a
+  reward *ensemble* (`GRPOTrainer(reward_funcs=[...])`, and unlocks the `rm_ensemble`
+  reward-hack detector which needs ≥2 rewards). GRPO-only, validated at config-parse
+  time. Fixes a recipe (`deepseek-v3-reasoning`) that shipped exactly this and
+  previously crashed with `Unknown reward function` (#311).
+
+### Changed / Fixed
+
+- `training.reward_fn` gains a field validator (null-byte / blank / oversize /
+  empty-comma-segment rejection) — the oldest arbitrary-code field was the least
+  guarded. Comma + `verifiable` without a `verifiable_domain` now fails at parse
+  time like the bare `verifiable` form.
+- `envs/calculator.py` / `envs/guess_number.py` docstrings corrected: the reward is
+  `reward_fn: verifiable` + `verifiable_domain: math` (the bare `reward_fn='math'`
+  they showed was never valid).
+
 ## [0.71.39] - 2026-07-19
 
 **"CI for weights, not prompts": close the evidence loop.** `soup ship`'s verdict is

{soup_cli-0.71.39 → soup_cli-0.71.41}/CONTRIBUTING.md

@@ -116,11 +116,11 @@ src/soup_cli/
   cans/               - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0)
   data/traces/        - Trace-to-Preference harvester (v0.26.0)
   data/collators.py   - CrossDocCollator for sample packing (v0.33.0)
-  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint, shrink, best_of_n, evolve
+  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint, shrink, best_of_n, evolve, reward_synth
   templates/          - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (317 files, 16387 tests)
+tests/                - Test suite (319 files, 16529 tests)
 examples/             - Real-world config examples and datasets
 ```
 
@@ -150,7 +150,7 @@ pytest tests/test_data.py::test_detect_alpaca_format -v
 pytest tests/ --cov=soup_cli --cov-report=html
 ```
 
-### Test Files (313 files)
+### Test Files (318 files)
 
 > A representative sample of the suite below. Run `pytest tests/ -v` for the complete list.
 

{soup_cli-0.71.39 → soup_cli-0.71.41}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.39
+Version: 0.71.41
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -193,29 +193,52 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.39 — CI for weights, not prompts.** `soup ship`'s verdict is now something Soup can emit, commit, review, and bind to the exact model that produced it — so a fine-tuning gate renders on every PR instead of asking you to hand-edit a JSON file.
-
-- **Output is finally input.** `soup ship --emit-evidence ev.json` writes the scores in the
-  `--evidence` schema, so a run replays offline into an identical verdict — the round-trip the
-  old `--output` couldn't do.
-- **Commit the gate policy.** Put `eval.ship` in your `soup.yaml` (`forgetting_threshold`,
-  `general_suite`, `task_mode`, …) and `soup ship --config soup.yaml` reads it — the gate is now
-  reviewable in a PR diff. Explicit flags still win.
-- **Provenance-bound evidence.** `--config --emit-evidence` STAMPS a `config_sha` onto the
-  evidence; `--config --evidence` GATES on it — a PR that changed the recipe but forgot to
-  recompute its evidence is refused (exit 3). Tuning the threshold never falsely invalidates it.
-- **The verdict on your PR.** `soup ship --push owner/repo#N` posts the SHIP / DON'T-SHIP card as
-  a GitHub comment; `soup ci init --config soup.yaml` wires the whole thing into CI.
+**v0.71.41 — `soup reward stress`: is your reward verifier gameable?** Turn the reward-hacking
+detector on the verifier *itself*. `soup reward synth` (v0.71.40) proves a verifier can tell your
+references from friendly bad answers; `stress` asks the adversarial question a reward-hacking model
+asks at train time — *does it pay out for degenerate junk?* Nothing in TRL / Unsloth / Axolotl tests
+a verifier for gameability.
+
+- **Four attacks, scored against the real gold.** Empty, length-padded, repetition, and
+  sentinel-spam completions are fed to the verifier; any it **accepts** is a false positive. A
+  per-attack accept-rate table plus one **robust / GAMEABLE** verdict — exit 0 / 2 (1 on error).
+- **Probes a synth `.py` or a builtin.** `soup reward stress reward.py --references golds.jsonl`,
+  or `soup reward stress verifiable --verifiable-domain math --references golds.jsonl`. A
+  gold-requiring verifier probed with no `--references` is a hard error, never a false "robust".
+- **Tune the strictness.** `--attacks`, `--sentinel`, `--threshold`, `--max-gameable`,
+  `--output-report`. Pure, offline, no new deps.
 
 ```bash
-# producer (train job): stamp the scores with the config that produced them
-soup ship --evidence scores.json --config soup.yaml --emit-evidence ship_evidence.json
+# probe a verifier you synthesized (or a builtin) for gameability
+soup reward stress reward.py --references golds.jsonl --output-report stress.json
+#   exit 0 = robust · 2 = gameable · 1 = error
+```
+
+<details>
+<summary>Previous release — v0.71.40, soup reward synth (generate a reward verifier from your data)</summary>
 
-# gate (PR CI): refuse evidence that doesn't match the committed config
-soup ship --evidence ship_evidence.json --config soup.yaml --push owner/repo#42
-soup ci init --config soup.yaml   # writes .github/workflows/soup-gate.yml
+Point `soup reward synth` at a JSONL of reference outputs and it infers a deterministic verifier,
+writes a readable / committable `.py` reward function, and — the part nobody else does — *refuses* to
+emit one that can't tell your references from bad answers (four families: `numeric` / `json_schema` /
+`regex` / `tool_call`; a mandatory calibration report is the moat). Reward ensembles
+(`reward_fn: "accuracy,format"`) also train now. (#311)
+
+```bash
+soup reward synth references.jsonl -o reward.py --output-report calib.json
 ```
 
+</details>
+
+<details>
+<summary>Previous release — v0.71.39, CI for weights not prompts (emit + provenance-bind the ship verdict)</summary>
+
+`soup ship`'s verdict became emittable, committable, and provenance-bound: `--emit-evidence` makes a
+run replay into an identical verdict, `eval.ship` in `soup.yaml` + `--config` makes the gate policy
+reviewable, and `--config` binds evidence to the exact recipe that produced it (stale evidence → exit 3).
+`soup ship --push owner/repo#N` posts the SHIP / DON'T-SHIP card on the PR.
+
+</details>
+
 <details>
 <summary>Previous release — v0.71.38, The gate grows teeth (real leg-2 regression gate)</summary>
 

{soup_cli-0.71.39 → soup_cli-0.71.41}/README.md

@@ -49,29 +49,52 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.39 — CI for weights, not prompts.** `soup ship`'s verdict is now something Soup can emit, commit, review, and bind to the exact model that produced it — so a fine-tuning gate renders on every PR instead of asking you to hand-edit a JSON file.
-
-- **Output is finally input.** `soup ship --emit-evidence ev.json` writes the scores in the
-  `--evidence` schema, so a run replays offline into an identical verdict — the round-trip the
-  old `--output` couldn't do.
-- **Commit the gate policy.** Put `eval.ship` in your `soup.yaml` (`forgetting_threshold`,
-  `general_suite`, `task_mode`, …) and `soup ship --config soup.yaml` reads it — the gate is now
-  reviewable in a PR diff. Explicit flags still win.
-- **Provenance-bound evidence.** `--config --emit-evidence` STAMPS a `config_sha` onto the
-  evidence; `--config --evidence` GATES on it — a PR that changed the recipe but forgot to
-  recompute its evidence is refused (exit 3). Tuning the threshold never falsely invalidates it.
-- **The verdict on your PR.** `soup ship --push owner/repo#N` posts the SHIP / DON'T-SHIP card as
-  a GitHub comment; `soup ci init --config soup.yaml` wires the whole thing into CI.
+**v0.71.41 — `soup reward stress`: is your reward verifier gameable?** Turn the reward-hacking
+detector on the verifier *itself*. `soup reward synth` (v0.71.40) proves a verifier can tell your
+references from friendly bad answers; `stress` asks the adversarial question a reward-hacking model
+asks at train time — *does it pay out for degenerate junk?* Nothing in TRL / Unsloth / Axolotl tests
+a verifier for gameability.
+
+- **Four attacks, scored against the real gold.** Empty, length-padded, repetition, and
+  sentinel-spam completions are fed to the verifier; any it **accepts** is a false positive. A
+  per-attack accept-rate table plus one **robust / GAMEABLE** verdict — exit 0 / 2 (1 on error).
+- **Probes a synth `.py` or a builtin.** `soup reward stress reward.py --references golds.jsonl`,
+  or `soup reward stress verifiable --verifiable-domain math --references golds.jsonl`. A
+  gold-requiring verifier probed with no `--references` is a hard error, never a false "robust".
+- **Tune the strictness.** `--attacks`, `--sentinel`, `--threshold`, `--max-gameable`,
+  `--output-report`. Pure, offline, no new deps.
 
 ```bash
-# producer (train job): stamp the scores with the config that produced them
-soup ship --evidence scores.json --config soup.yaml --emit-evidence ship_evidence.json
+# probe a verifier you synthesized (or a builtin) for gameability
+soup reward stress reward.py --references golds.jsonl --output-report stress.json
+#   exit 0 = robust · 2 = gameable · 1 = error
+```
+
+<details>
+<summary>Previous release — v0.71.40, soup reward synth (generate a reward verifier from your data)</summary>
 
-# gate (PR CI): refuse evidence that doesn't match the committed config
-soup ship --evidence ship_evidence.json --config soup.yaml --push owner/repo#42
-soup ci init --config soup.yaml   # writes .github/workflows/soup-gate.yml
+Point `soup reward synth` at a JSONL of reference outputs and it infers a deterministic verifier,
+writes a readable / committable `.py` reward function, and — the part nobody else does — *refuses* to
+emit one that can't tell your references from bad answers (four families: `numeric` / `json_schema` /
+`regex` / `tool_call`; a mandatory calibration report is the moat). Reward ensembles
+(`reward_fn: "accuracy,format"`) also train now. (#311)
+
+```bash
+soup reward synth references.jsonl -o reward.py --output-report calib.json
 ```
 
+</details>
+
+<details>
+<summary>Previous release — v0.71.39, CI for weights not prompts (emit + provenance-bind the ship verdict)</summary>
+
+`soup ship`'s verdict became emittable, committable, and provenance-bound: `--emit-evidence` makes a
+run replay into an identical verdict, `eval.ship` in `soup.yaml` + `--config` makes the gate policy
+reviewable, and `--config` binds evidence to the exact recipe that produced it (stale evidence → exit 3).
+`soup ship --push owner/repo#N` posts the SHIP / DON'T-SHIP card on the PR.
+
+</details>
+
 <details>
 <summary>Previous release — v0.71.38, The gate grows teeth (real leg-2 regression gate)</summary>
 

{soup_cli-0.71.39 → soup_cli-0.71.41}/docs/README.md

@@ -7,7 +7,7 @@ feature reference — every `soup` capability, grouped by area.
 
 | Guide | Covers |
 |---|---|
-| [Training tasks & methods](training.md) | SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, loop-hardening detectors |
+| [Training tasks & methods](training.md) | SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, loop-hardening detectors, reward-verifier synthesis |
 | [PEFT, long context & efficiency](peft-and-efficiency.md) | DoRA, LoRA+, rsLoRA, VeRA, OLoRA, NEFTune, PiSSA, ReLoRA, optimizer & PEFT zoo, LLaMA Pro, GaLore, YaRN/LongLoRA, packing, curriculum, auto-tuning, depth pruning + distill-heal (`soup shrink`) |
 | [Performance & quantization](performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, multi-GPU / DeepSpeed / FSDP |
 | [Data engineering](data.md) | Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs |

{soup_cli-0.71.39 → soup_cli-0.71.41}/docs/backends-and-ops.md

@@ -743,15 +743,9 @@ pip install soup-cli[trackers]   # mlflow + swanlab + trackio
 ```
 
 
-## Telemetry (opt-IN, hardware-info-only)
+## Telemetry (not yet wired)
 
-Soup ships an opt-IN telemetry sender that POSTs hardware-info-only payloads (`soup_version` / `command` / `python` major.minor / `os` / `arch` / optional `duration_seconds`) — no dataset paths, model names, or config contents. Enable per-shell:
-
-```bash
-SOUP_TELEMETRY=1 soup train --config soup.yaml
-```
-
-The sender uses a 1-second hard timeout, HTTPS-only with private-IP / link-local rejection (same SSRF policy as hub endpoints), and swallows every exception silently — telemetry can never crash training. Disabled by default until a public privacy policy ships.
+Soup contains opt-in, hardware-info-only telemetry primitives in `utils/trackers.py` (`build_telemetry_payload` / `send_telemetry_payload`), but they are **not wired to any command** — no data is ever sent, and no environment variable enables sending today. When wired, the payload will carry only `soup_version` / `command` / `python` major.minor / `os` / `arch` / optional `duration_seconds` — never dataset paths, model names, or config contents — behind a 1-second hard timeout and the same HTTPS-only, private-IP-rejecting SSRF policy as hub endpoints, swallowing every exception so telemetry can never crash training. Wiring is deferred until a public privacy policy is published.
 
 
 ## Plugin System

{soup_cli-0.71.39 → soup_cli-0.71.41}/docs/commands.md

@@ -182,6 +182,14 @@ soup draft measure ... --min-acceptance 0.6 -o report.json  Exit 2 below the flo
 soup draft distill --target <tuned> --draft-base <tiny> --data d.jsonl -o draft/  Distil a DENSE speculative-decoding draft + register it (v0.71.33)
 soup draft distill ... --steps N --device cpu --force --plan-only  Training budget / device / overwrite -o / render the config only
 soup draft list                               List local drafts that `soup serve --auto-spec` will pick up (v0.71.33)
+soup reward synth refs.jsonl -o reward.py     Synthesize a deterministic reward verifier from gold outputs (v0.71.40)
+soup reward synth ... --kind numeric|json_schema|regex|tool_call  Force a verifier family (default: auto-detect)
+soup reward synth ... --plan-only             Report the induced spec + calibration plan; write nothing
+soup reward synth ... --output-report r.json --min-discrimination 0.5  Save the calibration JSON / set the refusal threshold (exit 0 emit / 2 refuse / 1 error)
+soup reward stress reward.py --references golds.jsonl  Adversarially probe a verifier for gameability — empty/length/repetition/sentinel junk (v0.71.41)
+soup reward stress verifiable --verifiable-domain math --references golds.jsonl  Probe a builtin verifier instead of a .py file
+soup reward stress ... --attacks empty,length,repetition,sentinel --sentinel GOLD --threshold 0.5 --max-gameable 0.0  Tune the attack set / accept threshold / tolerance
+soup reward stress ... --output-report r.json  Save the per-attack report JSON (exit 0 robust / 2 gameable / 1 error)
 soup tui                                      Full-screen Textual dashboard (requires [tui] extra)
 soup train --config soup.yaml --profile       Record torch.profiler trace to <output>/profiles/
 soup --log-level quiet|normal|verbose|debug   Global logging tier (Rich-formatted)

{soup_cli-0.71.39 → soup_cli-0.71.41}/docs/training.md

@@ -866,6 +866,62 @@ training:
   reward_fn: ./my_reward.py
 ```
 
+**Reward ensembles** — list several rewards, comma-separated, and they combine (GRPO only).
+This also unlocks the `rm_ensemble` reward-hack detector, which needs ≥ 2 rewards:
+```yaml
+training:
+  reward_fn: "accuracy,format"   # both are scored every step
+```
+
+### Synthesize a verifier from your data (`soup reward synth`)
+
+Don't hand-write a verifier — generate one from reference (gold) outputs. Soup infers a
+*deterministic* verifier (numeric / JSON-schema / regex / tool-call), writes a readable, editable
+`.py`, and **refuses to emit** one that can't tell your references from auto-generated bad answers
+(the mandatory calibration report). The emitted file is a normal `reward_fn: reward.py`.
+
+```bash
+# infer + calibrate + emit (exit 0 kept, 2 refused, 1 error)
+soup reward synth references.jsonl -o reward.py --output-report calib.json
+
+# preview the induced spec without writing anything
+soup reward synth references.jsonl --plan-only
+
+# force a family instead of auto-detecting
+soup reward synth answers.jsonl -o reward.py --kind numeric --tolerance 1e-6
+```
+
+References are a JSONL where each row's gold answer is in an `answer` field (override with
+`--field`) or the last assistant turn of a `messages` list. `--min-discrimination` sets how
+strongly the verifier must separate references from perturbed negatives before it's emitted.
+v1 is deterministic families only — a `\boxed{}`/`####` marker helps the numeric verifier, and
+completions are prompted to mark their answer (standard RLVR practice).
+
+### Stress-test a verifier for gameability (`soup reward stress`)
+
+A verifier that passes calibration still might pay out for junk. `soup reward stress` feeds the
+verifier deterministic degenerate completions — empty, length-padded, repeated, and
+sentinel-spam — scored against your real gold answers, and flags any it **accepts**. It's the
+adversarial counterpart to `synth`: calibration proves the verifier tells references from
+*friendly* bad answers; `stress` asks whether a reward-hacking model could game it.
+
+```bash
+# probe a synthesized verifier (or any reward .py) — exit 0 robust, 2 gameable, 1 error
+soup reward stress reward.py --references golds.jsonl --output-report stress.json
+
+# probe a builtin verifier instead of a .py file
+soup reward stress verifiable --verifiable-domain math --references golds.jsonl
+
+# tune the attack set / accept threshold / gameability tolerance
+soup reward stress reward.py --references golds.jsonl \
+    --attacks empty,length,repetition,sentinel --sentinel GOLD \
+    --threshold 0.5 --max-gameable 0.0
+```
+
+The report shows a per-attack accept-rate and an overall verdict. A gold-requiring verifier probed
+with **no** `--references` is a hard error (it can't be measured), never a false "robust". Probing a
+`.py` executes its module code, like any custom reward — only stress files you trust.
+
 ### Verifiable Rewards (RLVR)
 
 Use `reward_fn: verifiable` with a `verifiable_domain` for deterministic, math-checkable rewards — no judge model, no heuristics. Great for GRPO on math, code, or structured-output tasks.

{soup_cli-0.71.39 → soup_cli-0.71.41}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.39"
+version = "0.71.41"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.39 → soup_cli-0.71.41}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
 
-__version__ = "0.71.39"
+__version__ = "0.71.41"

{soup_cli-0.71.39 → soup_cli-0.71.41}/src/soup_cli/cli.py

@@ -563,6 +563,18 @@ app.add_typer(
     ),
 )
 
+# v0.71.40 — `soup reward synth` auto-generate a deterministic verifier.
+from soup_cli.commands import reward as _reward_cmd  # noqa: E402
+
+app.add_typer(
+    _reward_cmd.app,
+    name="reward",
+    help=(
+        "Synthesize a deterministic reward verifier from reference outputs, with "
+        "a calibration report that refuses degenerate verifiers (v0.71.40)."
+    ),
+)
+
 
 def _rewrite_advise_argv(argv: list) -> list:
     """Inject `run` between `advise` and a non-subcommand first argument.