soup-cli 0.71.40__tar.gz → 0.71.41__tar.gz

{soup_cli-0.71.40 → soup_cli-0.71.41}/CHANGELOG.md

@@ -12,6 +12,34 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.41] - 2026-07-19
+
+**`soup reward stress`: is your reward verifier gameable?** Turn the reward-hacking
+detector on the *verifier itself*. `soup reward synth` (v0.71.40) proves a verifier
+separates your references from friendly perturbations; `stress` asks the adversarial
+question a reward-hacking model asks at train time — *does the verifier pay out for
+degenerate junk?* It feeds empty, length-padded, repetition, and sentinel-spam
+completions and flags any the verifier accepts. Pure, offline, exit 0 = robust /
+2 = gameable / 1 = error. Nothing in TRL / Unsloth / Axolotl / OpenRLHF tests a
+verifier for gameability.
+
+### Added
+
+- **`soup reward stress <reward.py|builtin> [--references golds.jsonl]`** — adversarial
+  verifier probe. Attacks (`--attacks empty,length,repetition,sentinel`, `--sentinel`)
+  are scored against the real gold, so numeric / tool_call / json_schema verifiers get a
+  valid target and still must reject the junk. Reports a per-attack accept-rate table +
+  an overall gameability verdict (`--max-gameable`, `--threshold`, `--output-report`).
+  Loads the target through the existing reward loader, so it probes a synthesized `.py`
+  **and** a builtin (`accuracy` / `format` / `verifiable`). A gold-requiring verifier
+  probed with no `--references` is a hard error, never a false "robust".
+
+### Fixed
+
+- Corrected the Telemetry section in the ops docs: Soup's telemetry primitives exist but
+  are **not wired to any command** — no data is ever sent today (the previous wording
+  implied a live opt-in sender). Wiring is deferred until a public privacy policy ships.
+
 ## [0.71.40] - 2026-07-19
 
 **`soup reward synth`: auto-generate a deterministic reward verifier from your data.**

{soup_cli-0.71.40 → soup_cli-0.71.41}/CONTRIBUTING.md

@@ -116,11 +116,11 @@ src/soup_cli/
   cans/               - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0)
   data/traces/        - Trace-to-Preference harvester (v0.26.0)
   data/collators.py   - CrossDocCollator for sample packing (v0.33.0)
-  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint, shrink, best_of_n, evolve
+  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint, shrink, best_of_n, evolve, reward_synth
   templates/          - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (318 files, 16490 tests)
+tests/                - Test suite (319 files, 16529 tests)
 examples/             - Real-world config examples and datasets
 ```
 
@@ -150,7 +150,7 @@ pytest tests/test_data.py::test_detect_alpaca_format -v
 pytest tests/ --cov=soup_cli --cov-report=html
 ```
 
-### Test Files (313 files)
+### Test Files (318 files)
 
 > A representative sample of the suite below. Run `pytest tests/ -v` for the complete list.
 

{soup_cli-0.71.40 → soup_cli-0.71.41}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.40
+Version: 0.71.41
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -193,27 +193,42 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.40 — `soup reward synth`: generate a reward verifier from your data.** Point it at a JSONL of reference (gold) outputs and it infers a deterministic verifier, writes a readable / committable `.py` reward function, and — the part nobody else does — *refuses* to emit one that can't tell your references from bad answers. Nothing in TRL / Unsloth / Axolotl synthesizes a reward.
+**v0.71.41 — `soup reward stress`: is your reward verifier gameable?** Turn the reward-hacking
+detector on the verifier *itself*. `soup reward synth` (v0.71.40) proves a verifier can tell your
+references from friendly bad answers; `stress` asks the adversarial question a reward-hacking model
+asks at train time — *does it pay out for degenerate junk?* Nothing in TRL / Unsloth / Axolotl tests
+a verifier for gameability.
+
+- **Four attacks, scored against the real gold.** Empty, length-padded, repetition, and
+  sentinel-spam completions are fed to the verifier; any it **accepts** is a false positive. A
+  per-attack accept-rate table plus one **robust / GAMEABLE** verdict — exit 0 / 2 (1 on error).
+- **Probes a synth `.py` or a builtin.** `soup reward stress reward.py --references golds.jsonl`,
+  or `soup reward stress verifiable --verifiable-domain math --references golds.jsonl`. A
+  gold-requiring verifier probed with no `--references` is a hard error, never a false "robust".
+- **Tune the strictness.** `--attacks`, `--sentinel`, `--threshold`, `--max-gameable`,
+  `--output-report`. Pure, offline, no new deps.
 
-- **Four families, auto-detected** (or pick with `--kind`): `numeric` (`\boxed{}` / `####` / last-number,
-  exact or `--tolerance`), `json_schema` (induced keys + types + required), `regex`, and `tool_call`
-  (per-tool `required`/`allowed` argument binding — a call can't borrow another tool's keys).
-- **A calibration report is the moat.** The verifier is loaded back and run against its own references
-  (must accept ≥ 90%) and auto-generated bad answers (must reject). A degenerate always-accept verifier
-  is **refused** (exit 2), never silently shipped. `--plan-only` previews; `--output-report` saves the JSON.
-- **No new exec surface.** The emitted file rides the existing `reward_fn: reward.py` path — you read,
-  edit, commit, and diff it like any other reward.
-- **Reward ensembles now train.** `reward_fn: "accuracy,format"` loads as multiple rewards (and unlocks
-  the `rm_ensemble` reward-hack detector) — a recipe that shipped exactly this used to crash. (#311)
+```bash
+# probe a verifier you synthesized (or a builtin) for gameability
+soup reward stress reward.py --references golds.jsonl --output-report stress.json
+#   exit 0 = robust · 2 = gameable · 1 = error
+```
+
+<details>
+<summary>Previous release — v0.71.40, soup reward synth (generate a reward verifier from your data)</summary>
+
+Point `soup reward synth` at a JSONL of reference outputs and it infers a deterministic verifier,
+writes a readable / committable `.py` reward function, and — the part nobody else does — *refuses* to
+emit one that can't tell your references from bad answers (four families: `numeric` / `json_schema` /
+`regex` / `tool_call`; a mandatory calibration report is the moat). Reward ensembles
+(`reward_fn: "accuracy,format"`) also train now. (#311)
 
 ```bash
-# synth a verifier from reference answers, calibrate it, keep it only if it discriminates
 soup reward synth references.jsonl -o reward.py --output-report calib.json
-
-# then train against the reward you just generated
-# training: { reward_fn: reward.py }   (or an ensemble: reward_fn: "accuracy,format")
 ```
 
+</details>
+
 <details>
 <summary>Previous release — v0.71.39, CI for weights not prompts (emit + provenance-bind the ship verdict)</summary>
 

{soup_cli-0.71.40 → soup_cli-0.71.41}/README.md

@@ -49,27 +49,42 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.40 — `soup reward synth`: generate a reward verifier from your data.** Point it at a JSONL of reference (gold) outputs and it infers a deterministic verifier, writes a readable / committable `.py` reward function, and — the part nobody else does — *refuses* to emit one that can't tell your references from bad answers. Nothing in TRL / Unsloth / Axolotl synthesizes a reward.
-
-- **Four families, auto-detected** (or pick with `--kind`): `numeric` (`\boxed{}` / `####` / last-number,
-  exact or `--tolerance`), `json_schema` (induced keys + types + required), `regex`, and `tool_call`
-  (per-tool `required`/`allowed` argument binding — a call can't borrow another tool's keys).
-- **A calibration report is the moat.** The verifier is loaded back and run against its own references
-  (must accept ≥ 90%) and auto-generated bad answers (must reject). A degenerate always-accept verifier
-  is **refused** (exit 2), never silently shipped. `--plan-only` previews; `--output-report` saves the JSON.
-- **No new exec surface.** The emitted file rides the existing `reward_fn: reward.py` path — you read,
-  edit, commit, and diff it like any other reward.
-- **Reward ensembles now train.** `reward_fn: "accuracy,format"` loads as multiple rewards (and unlocks
-  the `rm_ensemble` reward-hack detector) — a recipe that shipped exactly this used to crash. (#311)
+**v0.71.41 — `soup reward stress`: is your reward verifier gameable?** Turn the reward-hacking
+detector on the verifier *itself*. `soup reward synth` (v0.71.40) proves a verifier can tell your
+references from friendly bad answers; `stress` asks the adversarial question a reward-hacking model
+asks at train time — *does it pay out for degenerate junk?* Nothing in TRL / Unsloth / Axolotl tests
+a verifier for gameability.
+
+- **Four attacks, scored against the real gold.** Empty, length-padded, repetition, and
+  sentinel-spam completions are fed to the verifier; any it **accepts** is a false positive. A
+  per-attack accept-rate table plus one **robust / GAMEABLE** verdict — exit 0 / 2 (1 on error).
+- **Probes a synth `.py` or a builtin.** `soup reward stress reward.py --references golds.jsonl`,
+  or `soup reward stress verifiable --verifiable-domain math --references golds.jsonl`. A
+  gold-requiring verifier probed with no `--references` is a hard error, never a false "robust".
+- **Tune the strictness.** `--attacks`, `--sentinel`, `--threshold`, `--max-gameable`,
+  `--output-report`. Pure, offline, no new deps.
 
 ```bash
-# synth a verifier from reference answers, calibrate it, keep it only if it discriminates
-soup reward synth references.jsonl -o reward.py --output-report calib.json
+# probe a verifier you synthesized (or a builtin) for gameability
+soup reward stress reward.py --references golds.jsonl --output-report stress.json
+#   exit 0 = robust · 2 = gameable · 1 = error
+```
+
+<details>
+<summary>Previous release — v0.71.40, soup reward synth (generate a reward verifier from your data)</summary>
+
+Point `soup reward synth` at a JSONL of reference outputs and it infers a deterministic verifier,
+writes a readable / committable `.py` reward function, and — the part nobody else does — *refuses* to
+emit one that can't tell your references from bad answers (four families: `numeric` / `json_schema` /
+`regex` / `tool_call`; a mandatory calibration report is the moat). Reward ensembles
+(`reward_fn: "accuracy,format"`) also train now. (#311)
 
-# then train against the reward you just generated
-# training: { reward_fn: reward.py }   (or an ensemble: reward_fn: "accuracy,format")
+```bash
+soup reward synth references.jsonl -o reward.py --output-report calib.json
 ```
 
+</details>
+
 <details>
 <summary>Previous release — v0.71.39, CI for weights not prompts (emit + provenance-bind the ship verdict)</summary>
 

{soup_cli-0.71.40 → soup_cli-0.71.41}/docs/README.md

@@ -7,7 +7,7 @@ feature reference — every `soup` capability, grouped by area.
 
 | Guide | Covers |
 |---|---|
-| [Training tasks & methods](training.md) | SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, loop-hardening detectors |
+| [Training tasks & methods](training.md) | SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, loop-hardening detectors, reward-verifier synthesis |
 | [PEFT, long context & efficiency](peft-and-efficiency.md) | DoRA, LoRA+, rsLoRA, VeRA, OLoRA, NEFTune, PiSSA, ReLoRA, optimizer & PEFT zoo, LLaMA Pro, GaLore, YaRN/LongLoRA, packing, curriculum, auto-tuning, depth pruning + distill-heal (`soup shrink`) |
 | [Performance & quantization](performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, multi-GPU / DeepSpeed / FSDP |
 | [Data engineering](data.md) | Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs |

{soup_cli-0.71.40 → soup_cli-0.71.41}/docs/backends-and-ops.md

@@ -743,15 +743,9 @@ pip install soup-cli[trackers]   # mlflow + swanlab + trackio
 ```
 
 
-## Telemetry (opt-IN, hardware-info-only)
+## Telemetry (not yet wired)
 
-Soup ships an opt-IN telemetry sender that POSTs hardware-info-only payloads (`soup_version` / `command` / `python` major.minor / `os` / `arch` / optional `duration_seconds`) — no dataset paths, model names, or config contents. Enable per-shell:
-
-```bash
-SOUP_TELEMETRY=1 soup train --config soup.yaml
-```
-
-The sender uses a 1-second hard timeout, HTTPS-only with private-IP / link-local rejection (same SSRF policy as hub endpoints), and swallows every exception silently — telemetry can never crash training. Disabled by default until a public privacy policy ships.
+Soup contains opt-in, hardware-info-only telemetry primitives in `utils/trackers.py` (`build_telemetry_payload` / `send_telemetry_payload`), but they are **not wired to any command** — no data is ever sent, and no environment variable enables sending today. When wired, the payload will carry only `soup_version` / `command` / `python` major.minor / `os` / `arch` / optional `duration_seconds` — never dataset paths, model names, or config contents — behind a 1-second hard timeout and the same HTTPS-only, private-IP-rejecting SSRF policy as hub endpoints, swallowing every exception so telemetry can never crash training. Wiring is deferred until a public privacy policy is published.
 
 
 ## Plugin System

{soup_cli-0.71.40 → soup_cli-0.71.41}/docs/commands.md

@@ -186,6 +186,10 @@ soup reward synth refs.jsonl -o reward.py     Synthesize a deterministic reward
 soup reward synth ... --kind numeric|json_schema|regex|tool_call  Force a verifier family (default: auto-detect)
 soup reward synth ... --plan-only             Report the induced spec + calibration plan; write nothing
 soup reward synth ... --output-report r.json --min-discrimination 0.5  Save the calibration JSON / set the refusal threshold (exit 0 emit / 2 refuse / 1 error)
+soup reward stress reward.py --references golds.jsonl  Adversarially probe a verifier for gameability — empty/length/repetition/sentinel junk (v0.71.41)
+soup reward stress verifiable --verifiable-domain math --references golds.jsonl  Probe a builtin verifier instead of a .py file
+soup reward stress ... --attacks empty,length,repetition,sentinel --sentinel GOLD --threshold 0.5 --max-gameable 0.0  Tune the attack set / accept threshold / tolerance
+soup reward stress ... --output-report r.json  Save the per-attack report JSON (exit 0 robust / 2 gameable / 1 error)
 soup tui                                      Full-screen Textual dashboard (requires [tui] extra)
 soup train --config soup.yaml --profile       Record torch.profiler trace to <output>/profiles/
 soup --log-level quiet|normal|verbose|debug   Global logging tier (Rich-formatted)

{soup_cli-0.71.40 → soup_cli-0.71.41}/docs/training.md

@@ -897,6 +897,31 @@ strongly the verifier must separate references from perturbed negatives before i
 v1 is deterministic families only — a `\boxed{}`/`####` marker helps the numeric verifier, and
 completions are prompted to mark their answer (standard RLVR practice).
 
+### Stress-test a verifier for gameability (`soup reward stress`)
+
+A verifier that passes calibration still might pay out for junk. `soup reward stress` feeds the
+verifier deterministic degenerate completions — empty, length-padded, repeated, and
+sentinel-spam — scored against your real gold answers, and flags any it **accepts**. It's the
+adversarial counterpart to `synth`: calibration proves the verifier tells references from
+*friendly* bad answers; `stress` asks whether a reward-hacking model could game it.
+
+```bash
+# probe a synthesized verifier (or any reward .py) — exit 0 robust, 2 gameable, 1 error
+soup reward stress reward.py --references golds.jsonl --output-report stress.json
+
+# probe a builtin verifier instead of a .py file
+soup reward stress verifiable --verifiable-domain math --references golds.jsonl
+
+# tune the attack set / accept threshold / gameability tolerance
+soup reward stress reward.py --references golds.jsonl \
+    --attacks empty,length,repetition,sentinel --sentinel GOLD \
+    --threshold 0.5 --max-gameable 0.0
+```
+
+The report shows a per-attack accept-rate and an overall verdict. A gold-requiring verifier probed
+with **no** `--references` is a hard error (it can't be measured), never a false "robust". Probing a
+`.py` executes its module code, like any custom reward — only stress files you trust.
+
 ### Verifiable Rewards (RLVR)
 
 Use `reward_fn: verifiable` with a `verifiable_domain` for deterministic, math-checkable rewards — no judge model, no heuristics. Great for GRPO on math, code, or structured-output tasks.

{soup_cli-0.71.40 → soup_cli-0.71.41}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.40"
+version = "0.71.41"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.40 → soup_cli-0.71.41}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
 
-__version__ = "0.71.40"
+__version__ = "0.71.41"

{soup_cli-0.71.40 → soup_cli-0.71.41}/src/soup_cli/commands/reward.py

@@ -17,14 +17,16 @@ from __future__ import annotations
 import json
 import math
 import os
+from dataclasses import asdict
 from typing import NoReturn, Optional
 
 import typer
-from rich.console import Console
+from rich.console import Console, Group
 from rich.markup import escape
 from rich.panel import Panel
 from rich.table import Table
 
+from soup_cli.utils import reward_stress
 from soup_cli.utils import reward_synth as rs
 from soup_cli.utils.paths import atomic_write_text, enforce_under_cwd_and_no_symlink
 
@@ -34,6 +36,7 @@ console = Console()
 _MAX_INPUT_BYTES = 64 * 1024 * 1024
 _MAX_ROWS = 1_000_000
 _ALLOWED_KINDS = ("auto",) + rs.KINDS
+_MAX_SENTINEL_LEN = 256
 
 
 @app.callback()
@@ -201,7 +204,6 @@ def synth(
     # not delete an otherwise-valid verifier.
     if output_report:
         try:
-            from dataclasses import asdict
             atomic_write_text(json.dumps(asdict(report), indent=2), output_report)
         except OSError as exc:
             console.print(f"[yellow]Warning: could not write report: {escape(str(exc))}[/]")
@@ -223,3 +225,129 @@ def _cleanup(path: str) -> None:
         os.remove(path)
     except OSError:
         pass
+
+
+def _render_stress_panel(report: reward_stress.StressReport, target: str) -> Panel:
+    table = Table(show_header=True, box=None, pad_edge=False)
+    table.add_column("attack", style="cyan")
+    table.add_column("n", justify="right")
+    table.add_column("accepted", justify="right")
+    table.add_column("accept-rate", justify="right")
+    for a in report.attacks:
+        # Any junk this attack slipped through is red — a correct verifier rejects
+        # all of it. (The aggregate --max-gameable tolerance sets the verdict, not
+        # the per-row colour, so the two never contradict.)
+        style = "red" if a.accept_rate > 0 else "green"
+        table.add_row(
+            escape(a.kind), str(a.n), str(a.accepted),
+            f"[{style}]{a.accept_rate:.0%}[/]",
+        )
+    ref = "n/a" if report.reference_accept is None else f"{report.reference_accept:.0%}"
+    verdict = ("[bold red]GAMEABLE[/]" if report.gameable
+               else "[bold green]robust (not gameable)[/]")
+    footer = (f"\nreference accept: {ref}   "
+              f"gameability: {report.gameability:.0%}   verdict: {verdict}")
+    border = "red" if report.gameable else "green"
+    return Panel(
+        Group(table, footer),
+        title=f"[bold]reward stress: {escape(target)}[/]", border_style=border,
+    )
+
+
+@app.command()
+def stress(
+    reward_target: str = typer.Argument(
+        ..., help="Verifier to probe: a .py path, a builtin name, or 'verifiable'."
+    ),
+    references: Optional[str] = typer.Option(
+        None, "--references", help="JSONL of gold outputs (enables gold-aware probing)."
+    ),
+    field: str = typer.Option("answer", "--field", help="Gold field (default: answer)."),
+    verifiable_domain: Optional[str] = typer.Option(
+        None, "--verifiable-domain", help="Domain for a 'verifiable' target."
+    ),
+    sentinel: str = typer.Option(
+        reward_stress.DEFAULT_SENTINEL, "--sentinel", help="Sentinel-spam token."
+    ),
+    threshold: float = typer.Option(
+        reward_stress.DEFAULT_THRESHOLD, "--threshold", help="Reward >= this = accept."
+    ),
+    max_gameable: float = typer.Option(
+        reward_stress.DEFAULT_MAX_GAMEABLE, "--max-gameable",
+        help="Max junk accept-rate allowed before the verdict flips to gameable.",
+    ),
+    attacks: str = typer.Option(
+        ",".join(reward_stress.ATTACKS), "--attacks",
+        help="Comma list: empty,length,repetition,sentinel.",
+    ),
+    output_report: Optional[str] = typer.Option(
+        None, "--output-report", help="Also write the stress report as JSON."
+    ),
+) -> None:
+    """Adversarially probe a reward verifier for gameability (exit 0 robust / 2 gameable)."""
+    if not 0.0 <= threshold <= 1.0:
+        _fail("--threshold must be in [0.0, 1.0]")
+    if not 0.0 <= max_gameable <= 1.0:
+        _fail("--max-gameable must be in [0.0, 1.0]")
+    # Dedupe while preserving order — 'empty,empty' must not double-weight.
+    kinds = list(dict.fromkeys(k.strip() for k in attacks.split(",") if k.strip()))
+    if not kinds:
+        _fail("--attacks must name at least one attack kind")
+    bad = [k for k in kinds if k not in reward_stress.ATTACKS]
+    if bad:
+        _fail(f"unknown attack kind(s): {', '.join(bad)}; "
+              f"options: {', '.join(reward_stress.ATTACKS)}")
+    if len(sentinel) > _MAX_SENTINEL_LEN:
+        _fail(f"--sentinel must be <= {_MAX_SENTINEL_LEN} characters")
+
+    # A .py target is cwd-contained; a builtin name passes to load_reward_fn as-is.
+    if reward_target.endswith(".py"):
+        try:
+            enforce_under_cwd_and_no_symlink(reward_target, "reward target")
+        except (ValueError, OSError) as exc:
+            _fail(str(exc))
+        if not os.path.exists(reward_target):
+            _fail(f"reward target {reward_target!r} not found")
+
+    # Validate every write/read path UP FRONT — before load_reward_fn executes the
+    # target file's arbitrary module code — so a bad --output-report typo costs no
+    # extra code execution (mirrors synth's validate-before-load ordering).
+    if output_report:
+        try:
+            enforce_under_cwd_and_no_symlink(output_report, "report path")
+        except (ValueError, OSError) as exc:
+            _fail(str(exc))
+
+    golds: list[str] = []
+    if references:
+        try:
+            rows = _read_jsonl(references, "references path")
+        except (ValueError, OSError) as exc:
+            _fail(str(exc))
+        try:
+            golds = rs.extract_golds(rows, field=field)
+        except (ValueError, TypeError) as exc:
+            _fail(str(exc))
+
+    try:
+        from soup_cli.trainer.rewards import load_reward_fn
+        reward_fn = load_reward_fn(reward_target, verifiable_domain=verifiable_domain)
+    except Exception as exc:  # noqa: BLE001 — the target file runs arbitrary code
+        _fail(f"could not load reward target: {exc}")
+
+    try:
+        report = reward_stress.run_stress(
+            reward_fn, golds, sentinel=sentinel, threshold=threshold,
+            max_gameable=max_gameable, attacks=kinds,
+        )
+    except Exception as exc:  # noqa: BLE001 — a broken reward fn is a usage error
+        _fail(f"stress run failed: {exc}")
+
+    if output_report:
+        try:
+            atomic_write_text(json.dumps(asdict(report), indent=2), output_report)
+        except OSError as exc:
+            console.print(f"[yellow]Warning: could not write report: {escape(str(exc))}[/]")
+
+    console.print(_render_stress_panel(report, reward_target))
+    raise typer.Exit(2 if report.gameable else 0)

soup_cli-0.71.41/src/soup_cli/utils/reward_stress.py

@@ -0,0 +1,190 @@
+"""Adversarial verifier probe — `soup reward stress` (v0.71.41).
+
+Turn the v0.71.26 reward-hacking expertise on a reward VERIFIER itself: does it
+pay out for degenerate completions (empty / length-padded / repetition /
+sentinel-spam)? A correct deterministic verifier rejects all of them.
+
+Pure, offline, NO top-level torch. Reuses the attack-kind vocabulary of
+``reward_hack_control`` (``SHAPING_KINDS`` = length/repetition/sentinel) — as
+ATTACKS rather than mitigations — plus the trivial ``empty`` case. Companion to
+``reward_synth`` (v0.71.40): ``synth`` proves a verifier separates references
+from *friendly* perturbations; ``stress`` asks the *adversarial* question a
+reward-hacking model asks at train time.
+"""
+
+from __future__ import annotations
+
+import math
+from collections.abc import Callable, Sequence
+from dataclasses import dataclass
+from typing import Any, Optional
+
+from soup_cli.utils.reward_hack_control import _DEFAULT_SENTINEL
+
+# Public re-export so the CLI does not reach into a private cross-module symbol.
+DEFAULT_SENTINEL = _DEFAULT_SENTINEL
+
+# ``empty`` + the three ``reward_hack_control.SHAPING_KINDS``.
+ATTACKS: tuple[str, ...] = ("empty", "length", "repetition", "sentinel")
+
+# > ``reward_hack_control._SHAPING_LENGTH_SAT`` (32) so a length-reward saturates.
+_LENGTH_ATTACK_WORDS = 60
+_REPETITION_COUNT = 40
+_SENTINEL_COUNT = 20
+# O(golds) forward calls per attack — cap the sampled golds to bound the run.
+_MAX_STRESS_GOLDS = 200
+
+DEFAULT_THRESHOLD = 0.5
+DEFAULT_MAX_GAMEABLE = 0.0
+
+
+# ---------------------------------------------------------------------------
+# Frozen result types
+# ---------------------------------------------------------------------------
+@dataclass(frozen=True)
+class AttackResult:
+    kind: str
+    n: int
+    accepted: int
+    accept_rate: float
+
+
+@dataclass(frozen=True)
+class StressReport:
+    reference_accept: Optional[float]
+    attacks: tuple[AttackResult, ...]
+    gameability: float
+    gameable: bool
+    threshold: float
+    max_gameable: float
+    sentinel: str
+
+
+# ---------------------------------------------------------------------------
+# Attack generation
+# ---------------------------------------------------------------------------
+def _attack_text(kind: str, sentinel: str) -> str:
+    if kind == "empty":
+        return ""
+    if kind == "length":
+        # A long ramble with NO answer — games a length-based reward.
+        return " ".join(["padding"] * _LENGTH_ATTACK_WORDS)
+    if kind == "repetition":
+        return " ".join(["loop"] * _REPETITION_COUNT)
+    if kind == "sentinel":
+        # Spam a magic token — games a "contains a sentinel" reward.
+        return " ".join([sentinel] * _SENTINEL_COUNT)
+    raise ValueError(f"unknown attack kind: {kind!r} (options: {', '.join(ATTACKS)})")
+
+
+def generate_attacks(
+    *, sentinel: str = _DEFAULT_SENTINEL, kinds: Sequence[str] = ATTACKS
+) -> list[tuple[str, str]]:
+    """Deterministic ``(kind, completion_text)`` junk a correct verifier must reject."""
+    if isinstance(kinds, (str, bytes)):
+        # A bare string is a Sequence of characters — iterating it would probe
+        # per-letter. Force an explicit collection (mirrors reward_synth guards).
+        raise TypeError("kinds must be a sequence of attack-kind strings, not a str")
+    return [(kind, _attack_text(kind, sentinel)) for kind in kinds]
+
+
+# ---------------------------------------------------------------------------
+# Scoring + verdict
+# ---------------------------------------------------------------------------
+def _score_batch(
+    reward_fn: Callable[..., Sequence[Any]],
+    texts: Sequence[str],
+    answers: Optional[Sequence[str]],
+) -> list[float]:
+    """Score completions built from ``texts``; supply ``answer=`` only when given.
+
+    Validates the reward fn returned exactly one finite numeric score per
+    completion. A short return (the classic case: a gold-requiring builtin like
+    ``accuracy`` scored with no ``answer`` — its ``zip(completions, answers)``
+    yields an empty list) is a hard error, NOT a silent "0 accepted" that would
+    render a false "robust" verdict. A non-finite score means a broken verifier.
+    """
+    if not texts:
+        return []
+    completions = [[{"role": "assistant", "content": t}] for t in texts]
+    kwargs = {"answer": list(answers)} if answers is not None else {}
+    scores = list(reward_fn(completions, **kwargs))
+    if len(scores) != len(texts):
+        raise ValueError(
+            f"reward target returned {len(scores)} score(s) for {len(texts)} "
+            "completion(s) — this verifier likely needs --references to be probed "
+            "meaningfully (its reward compares each completion against a gold answer)"
+        )
+    coerced: list[float] = []
+    for s in scores:
+        try:
+            val = float(s)
+        except (TypeError, ValueError) as exc:
+            raise ValueError(f"reward target returned a non-numeric score {s!r}") from exc
+        if not math.isfinite(val):
+            raise ValueError(f"reward target returned a non-finite score {val!r}")
+        coerced.append(val)
+    return coerced
+
+
+def run_stress(
+    reward_fn: Callable[..., Sequence[Any]],
+    golds: Sequence[str],
+    *,
+    sentinel: str = _DEFAULT_SENTINEL,
+    threshold: float = DEFAULT_THRESHOLD,
+    max_gameable: float = DEFAULT_MAX_GAMEABLE,
+    attacks: Sequence[str] = ATTACKS,
+) -> StressReport:
+    """Score adversarial junk completions and flag a gameable verifier.
+
+    For each attack kind, one junk completion is scored per sampled gold against
+    the REAL gold (so numeric/tool_call/json_schema verifiers get a valid
+    ``answer=`` and still must reject the junk). ``gameability`` is the overall
+    junk accept-rate; ``gameable`` iff it strictly exceeds ``max_gameable``.
+
+    ``reference_accept`` (the golds scored as their own correct completions) is
+    reported for context — a verifier that rejects everything is broken, a
+    different problem — but does NOT set the verdict.
+
+    No-gold fallback: with an empty ``golds`` each attack is scored once with no
+    ``answer`` kwarg and ``reference_accept`` is ``None``.
+    """
+    sampled = list(golds)[:_MAX_STRESS_GOLDS]
+    have_golds = bool(sampled)
+
+    def _accepted(scores: Sequence[float]) -> int:
+        # Scores are already validated finite + numeric by _score_batch.
+        return sum(1 for s in scores if s >= threshold)
+
+    reference_accept: Optional[float] = None
+    if have_golds:
+        ref_scores = _score_batch(reward_fn, sampled, sampled)
+        reference_accept = _accepted(ref_scores) / len(sampled)
+
+    results: list[AttackResult] = []
+    total_accepted = total_n = 0
+    for kind, junk in generate_attacks(sentinel=sentinel, kinds=attacks):
+        if have_golds:
+            texts: list[str] = [junk] * len(sampled)
+            answers: Optional[list[str]] = list(sampled)
+        else:
+            texts = [junk]
+            answers = None
+        scores = _score_batch(reward_fn, texts, answers)
+        accepted = _accepted(scores)
+        n = len(texts)
+        results.append(AttackResult(kind, n, accepted, accepted / n if n else 0.0))
+        total_accepted += accepted
+        total_n += n
+
+    gameability = (total_accepted / total_n) if total_n else 0.0
+    return StressReport(
+        reference_accept=reference_accept,
+        attacks=tuple(results),
+        gameability=gameability,
+        gameable=gameability > max_gameable,
+        threshold=threshold,
+        max_gameable=max_gameable,
+        sentinel=sentinel,
+    )