soup-cli 0.71.38__tar.gz → 0.71.40__tar.gz

{soup_cli-0.71.38 → soup_cli-0.71.40}/CHANGELOG.md

@@ -12,6 +12,92 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.40] - 2026-07-19
+
+**`soup reward synth`: auto-generate a deterministic reward verifier from your data.**
+Point it at a JSONL of reference (gold) outputs and it infers a verifier, emits a
+readable / committable `.py` reward function, and — the moat — *refuses* to emit one
+that can't tell your references from auto-generated bad answers. Nothing in
+TRL / Unsloth / Axolotl / OpenRLHF synthesizes a reward; every reward today is
+hand-written, hand-picked, or a trained-weights artifact.
+
+### Added
+
+- **`soup reward synth <references.jsonl> -o reward.py`** — deterministic verifier
+  synthesis. Four families, auto-detected (or pick with `--kind`): `numeric`
+  (last-number / `\boxed{}` / `####` extraction, exact or `--tolerance`), `json_schema`
+  (induced keys + types + required), `regex` (positional char-classes over
+  equal-length golds), `tool_call` (per-tool `required`/`allowed` argument binding).
+  The emitted file is self-contained and rides `load_reward_fn`'s existing `.py`
+  path — no new trusted-exec surface; you read, edit, commit, and diff it.
+- **Mandatory calibration report** — the synthesized verifier is loaded back and run
+  against its own references (must accept ≥90%) and auto-perturbed negatives (must
+  reject). A degenerate always-accept verifier is **refused** (`exit 2`), never
+  silently emitted. `--plan-only` reports the induced spec without writing;
+  `--output-report` persists the calibration JSON.
+- **Comma-separated `reward_fn` (`"accuracy,format"`)** now trains — it resolves to a
+  reward *ensemble* (`GRPOTrainer(reward_funcs=[...])`, and unlocks the `rm_ensemble`
+  reward-hack detector which needs ≥2 rewards). GRPO-only, validated at config-parse
+  time. Fixes a recipe (`deepseek-v3-reasoning`) that shipped exactly this and
+  previously crashed with `Unknown reward function` (#311).
+
+### Changed / Fixed
+
+- `training.reward_fn` gains a field validator (null-byte / blank / oversize /
+  empty-comma-segment rejection) — the oldest arbitrary-code field was the least
+  guarded. Comma + `verifiable` without a `verifiable_domain` now fails at parse
+  time like the bare `verifiable` form.
+- `envs/calculator.py` / `envs/guess_number.py` docstrings corrected: the reward is
+  `reward_fn: verifiable` + `verifiable_domain: math` (the bare `reward_fn='math'`
+  they showed was never valid).
+
+## [0.71.39] - 2026-07-19
+
+**"CI for weights, not prompts": close the evidence loop.** `soup ship`'s verdict is
+now something Soup can emit, commit, review, and bind to the exact model that
+produced it — turning the `soup ci init` gate from "edit two numbers in a JSON file"
+into a reproducible, provenance-bound check that renders on every PR.
+
+### Added
+
+- **`soup ship --emit-evidence <path>`** — re-serialises the verdict into the
+  `--evidence` INPUT schema, so a run's output is replayable as input: feeding it
+  back through `--evidence` (same `--forgetting-threshold`) reproduces an identical
+  verdict. Output is finally input.
+- **`ShipConfig` under `eval.ship` in `soup.yaml` + `soup ship --config soup.yaml`** —
+  commit the gate policy (`task_eval` / `task_mode` / `general_suite` /
+  `forgetting_threshold` / `judge_model` / `baseline`) so the verdict is reviewable in
+  a PR diff and reproducible. An explicit CLI flag always wins (CLI > config > default).
+- **`soup ship --push owner/repo#N`** — post the verdict as a GitHub PR comment
+  (reuses the `soup adapters pr --push` `gh api` plumbing). Best-effort: a missing
+  token / `gh` failure warns but never flips the SHIP / DON'T-SHIP exit code.
+- **Evidence provenance + staleness gate.** With `--emit-evidence`, `--config` STAMPS
+  a `provenance` block (`config_sha` — a semantic, order-insensitive recipe hash — plus
+  `base_model` and a best-effort `data_sha`) onto the evidence. With `--evidence`
+  alone, `--config` GATES: it refuses (exit 3) evidence whose `config_sha` drifted
+  from the committed config, so a PR that changed `soup.yaml` but forgot to recompute
+  its evidence is caught. The gate policy (`eval.ship`) is EXCLUDED from the hash, so
+  tuning `forgetting_threshold` never falsely invalidates evidence about an unchanged
+  model.
+- **`soup ci init --config <soup.yaml>`** — binds the generated gate's `soup ship`
+  step to the committed config (provenance/staleness enforcement in CI).
+
+### Changed
+
+- **Exit-code note:** `soup ship --config` usage / staleness errors are exit `3`
+  (usage), preserving `0 = SHIP`, `2 = DON'T SHIP`, `1 = runtime` from v0.71.38.
+
+### Security
+
+- `provenance.config_sha` read from untrusted evidence is shape-validated as a hex
+  digest before being echoed (a raw value could smuggle terminal ESC bytes past
+  `rich.markup.escape`).
+- `provenance.data_sha` hashes the training file through an `O_NOFOLLOW` fd with a
+  symlink-rejecting containment check and an 8 GiB cap (was an unguarded `hash_file`).
+- `soup ci init`'s path validation now rejects `#` and the YAML 1.1 line breaks
+  (NEL / LS / PS), closing a plain-scalar comment-truncation of the generated `run:`
+  step (also hardens the pre-existing `--data` / `--suite` / `--evidence` args).
+
 ## [0.71.38] - 2026-07-17
 
 **`soup ship`'s regression leg now has teeth.** Leg 2 (the catastrophic-forgetting

{soup_cli-0.71.38 → soup_cli-0.71.40}/CONTRIBUTING.md

@@ -120,7 +120,7 @@ src/soup_cli/
   templates/          - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (316 files, 16330 tests)
+tests/                - Test suite (318 files, 16490 tests)
 examples/             - Real-world config examples and datasets
 ```
 

{soup_cli-0.71.38 → soup_cli-0.71.40}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.38
+Version: 0.71.40
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -193,41 +193,47 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.38 — The gate grows teeth.** `soup ship` answers the one question that matters after a fine-tune — *did my task improve WITHOUT silently breaking everything else?* This release makes the "without breaking everything else" half real.
-
-- **A gate that actually catches regressions.** The regression leg used to be 15 trivia
-  prompts scored by raw substring — it credited `"B"` for "**B**erlin", `"3"` for "1**3**", and
-  had **zero** items for tool-calling, safety, or JSON. Now it's a fixed, extraction-based
-  scorer over seven bundled, offline suites (MCQ · arithmetic · **tool-calling** · **JSON
-  validity** · **safety/refusal**). A tune that WINS your task but quietly breaks tool-calling
-  now gets a **DON'T SHIP** — the old gate waved it through.
-- **Zero new dependencies, no network.** Every suite ships in the wheel and is scored by the
-  pure scorers Soup already has — no lm-eval, no download. `soup ci init`'s core-only gate keeps
-  working.
-- **Distinct exit codes for CI.** Usage errors moved off exit `2` to `3`, so `2` now means only
-  DON'T-SHIP — a typo'd flag is no longer indistinguishable from a caught regression.
-- **Breaking, on purpose.** The fixed scorer can change an existing run's verdict — because the
-  old one was reporting false negatives. Recompute any committed `--baseline`.
+**v0.71.40 — `soup reward synth`: generate a reward verifier from your data.** Point it at a JSONL of reference (gold) outputs and it infers a deterministic verifier, writes a readable / committable `.py` reward function, and — the part nobody else does — *refuses* to emit one that can't tell your references from bad answers. Nothing in TRL / Unsloth / Axolotl synthesizes a reward.
+
+- **Four families, auto-detected** (or pick with `--kind`): `numeric` (`\boxed{}` / `####` / last-number,
+  exact or `--tolerance`), `json_schema` (induced keys + types + required), `regex`, and `tool_call`
+  (per-tool `required`/`allowed` argument binding — a call can't borrow another tool's keys).
+- **A calibration report is the moat.** The verifier is loaded back and run against its own references
+  (must accept ≥ 90%) and auto-generated bad answers (must reject). A degenerate always-accept verifier
+  is **refused** (exit 2), never silently shipped. `--plan-only` previews; `--output-report` saves the JSON.
+- **No new exec surface.** The emitted file rides the existing `reward_fn: reward.py` path — you read,
+  edit, commit, and diff it like any other reward.
+- **Reward ensembles now train.** `reward_fn: "accuracy,format"` loads as multiple rewards (and unlocks
+  the `rm_ensemble` reward-hack detector) — a recipe that shipped exactly this used to crash. (#311)
 
 ```bash
-# leg 1 (task win) AND leg 2 (no regression on the bundled suite) -> one verdict
-soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
-#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error
+# synth a verifier from reference answers, calibrate it, keep it only if it discriminates
+soup reward synth references.jsonl -o reward.py --output-report calib.json
 
-soup ship --evidence scores.json    # offline verdict from pre-computed scores
+# then train against the reward you just generated
+# training: { reward_fn: reward.py }   (or an ensemble: reward_fn: "accuracy,format")
 ```
 
 <details>
-<summary>Previous release — v0.71.36, Data Moat II (semantic dedup · topics · canaries · replay)</summary>
+<summary>Previous release — v0.71.39, CI for weights not prompts (emit + provenance-bind the ship verdict)</summary>
+
+`soup ship`'s verdict became emittable, committable, and provenance-bound: `--emit-evidence` makes a
+run replay into an identical verdict, `eval.ship` in `soup.yaml` + `--config` makes the gate policy
+reviewable, and `--config` binds evidence to the exact recipe that produced it (stale evidence → exit 3).
+`soup ship --push owner/repo#N` posts the SHIP / DON'T-SHIP card on the PR.
+
+</details>
+
+<details>
+<summary>Previous release — v0.71.38, The gate grows teeth (real leg-2 regression gate)</summary>
 
-`soup data dedup --semantic` dedups by meaning; `soup data topics` shows a coverage table;
-`soup data canary insert|check` proves whether a model memorized inserted secrets (exit 2 on a
-leak); `soup train --replay old.jsonl` interleaves old data so a new task doesn't erase it.
+`soup ship`'s regression leg became real: a fixed, extraction-based scorer over seven bundled,
+offline suites (MCQ · arithmetic · tool-calling · JSON validity · safety/refusal). A tune that
+wins your task but quietly breaks tool-calling now gets a **DON'T SHIP**. Zero new deps.
 
 ```bash
-soup data topics train.jsonl
-soup data canary check --manifest secrets.json --base ./my-model  # exit 2 = leak
-soup train --config sft.yaml --replay old_task.jsonl --replay-ratio 0.1
+soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
+#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error
 ```
 
 </details>

{soup_cli-0.71.38 → soup_cli-0.71.40}/README.md

@@ -49,41 +49,47 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.38 — The gate grows teeth.** `soup ship` answers the one question that matters after a fine-tune — *did my task improve WITHOUT silently breaking everything else?* This release makes the "without breaking everything else" half real.
-
-- **A gate that actually catches regressions.** The regression leg used to be 15 trivia
-  prompts scored by raw substring — it credited `"B"` for "**B**erlin", `"3"` for "1**3**", and
-  had **zero** items for tool-calling, safety, or JSON. Now it's a fixed, extraction-based
-  scorer over seven bundled, offline suites (MCQ · arithmetic · **tool-calling** · **JSON
-  validity** · **safety/refusal**). A tune that WINS your task but quietly breaks tool-calling
-  now gets a **DON'T SHIP** — the old gate waved it through.
-- **Zero new dependencies, no network.** Every suite ships in the wheel and is scored by the
-  pure scorers Soup already has — no lm-eval, no download. `soup ci init`'s core-only gate keeps
-  working.
-- **Distinct exit codes for CI.** Usage errors moved off exit `2` to `3`, so `2` now means only
-  DON'T-SHIP — a typo'd flag is no longer indistinguishable from a caught regression.
-- **Breaking, on purpose.** The fixed scorer can change an existing run's verdict — because the
-  old one was reporting false negatives. Recompute any committed `--baseline`.
+**v0.71.40 — `soup reward synth`: generate a reward verifier from your data.** Point it at a JSONL of reference (gold) outputs and it infers a deterministic verifier, writes a readable / committable `.py` reward function, and — the part nobody else does — *refuses* to emit one that can't tell your references from bad answers. Nothing in TRL / Unsloth / Axolotl synthesizes a reward.
+
+- **Four families, auto-detected** (or pick with `--kind`): `numeric` (`\boxed{}` / `####` / last-number,
+  exact or `--tolerance`), `json_schema` (induced keys + types + required), `regex`, and `tool_call`
+  (per-tool `required`/`allowed` argument binding — a call can't borrow another tool's keys).
+- **A calibration report is the moat.** The verifier is loaded back and run against its own references
+  (must accept ≥ 90%) and auto-generated bad answers (must reject). A degenerate always-accept verifier
+  is **refused** (exit 2), never silently shipped. `--plan-only` previews; `--output-report` saves the JSON.
+- **No new exec surface.** The emitted file rides the existing `reward_fn: reward.py` path — you read,
+  edit, commit, and diff it like any other reward.
+- **Reward ensembles now train.** `reward_fn: "accuracy,format"` loads as multiple rewards (and unlocks
+  the `rm_ensemble` reward-hack detector) — a recipe that shipped exactly this used to crash. (#311)
 
 ```bash
-# leg 1 (task win) AND leg 2 (no regression on the bundled suite) -> one verdict
-soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
-#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error
+# synth a verifier from reference answers, calibrate it, keep it only if it discriminates
+soup reward synth references.jsonl -o reward.py --output-report calib.json
 
-soup ship --evidence scores.json    # offline verdict from pre-computed scores
+# then train against the reward you just generated
+# training: { reward_fn: reward.py }   (or an ensemble: reward_fn: "accuracy,format")
 ```
 
 <details>
-<summary>Previous release — v0.71.36, Data Moat II (semantic dedup · topics · canaries · replay)</summary>
+<summary>Previous release — v0.71.39, CI for weights not prompts (emit + provenance-bind the ship verdict)</summary>
 
-`soup data dedup --semantic` dedups by meaning; `soup data topics` shows a coverage table;
-`soup data canary insert|check` proves whether a model memorized inserted secrets (exit 2 on a
-leak); `soup train --replay old.jsonl` interleaves old data so a new task doesn't erase it.
+`soup ship`'s verdict became emittable, committable, and provenance-bound: `--emit-evidence` makes a
+run replay into an identical verdict, `eval.ship` in `soup.yaml` + `--config` makes the gate policy
+reviewable, and `--config` binds evidence to the exact recipe that produced it (stale evidence → exit 3).
+`soup ship --push owner/repo#N` posts the SHIP / DON'T-SHIP card on the PR.
+
+</details>
+
+<details>
+<summary>Previous release — v0.71.38, The gate grows teeth (real leg-2 regression gate)</summary>
+
+`soup ship`'s regression leg became real: a fixed, extraction-based scorer over seven bundled,
+offline suites (MCQ · arithmetic · tool-calling · JSON validity · safety/refusal). A tune that
+wins your task but quietly breaks tool-calling now gets a **DON'T SHIP**. Zero new deps.
 
 ```bash
-soup data topics train.jsonl
-soup data canary check --manifest secrets.json --base ./my-model  # exit 2 = leak
-soup train --config sft.yaml --replay old_task.jsonl --replay-ratio 0.1
+soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
+#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error
 ```
 
 </details>

{soup_cli-0.71.38 → soup_cli-0.71.40}/docs/commands.md

@@ -166,9 +166,12 @@ soup ship ... --task-mode judge_score --judge-model ollama://llama3.1  Leg-1 via
 soup ship ... --task-mode pairwise --judge-model ollama://llama3.1  Leg-1 via swap-debiased judge win-rate (base=0.5) (v0.71.31)
 soup ship ...  # leg-2 default = 7 bundled offline suites (MCQ/arithmetic + tool_call/format_json/safety, extraction scorer, ~40 items each) (v0.71.38)
 soup ship ... --general-suite mmlu,gsm8k --baseline base.json  lm-eval leg-2 override + recorded base scores
+soup ship ... --emit-evidence ev.json  Re-serialise the scores as replayable --evidence input (output-is-input, #312) (v0.71.39)
+soup ship ... --config soup.yaml  Read eval.ship gate defaults; --evidence GATES on provenance, --emit-evidence STAMPS it (v0.71.39)
+soup ship ... --push owner/repo#N  Post the verdict as a GitHub PR comment (best-effort; never flips the exit code) (v0.71.39)
 soup card <registry-id> -o MODELCARD.md       HF model card from a registry entry: training config, evals, hashes, lineage, artifacts (v0.71.35)
 soup push --model ./out --repo you/m --card <registry-id>  Upload that registry-driven card as the README (HF only) (v0.71.35)
-soup ci init [--data d.jsonl --suite s.yaml --evidence ev.json] [--branch main --python 3.11] [--force]  Write .github/workflows/soup-gate.yml: data validate -> expect -> ship gate on every PR (v0.71.35)
+soup ci init [--data d.jsonl --suite s.yaml --evidence ev.json] [--config soup.yaml] [--branch main --python 3.11] [--force]  Write .github/workflows/soup-gate.yml: data validate -> expect -> ship gate on every PR (v0.71.35); --config binds the gate to a committed config so it refuses stale evidence (v0.71.39)
 soup mcp serve                                MCP server over stdio (drive Soup from Claude Code / Cursor / Cline; requires [mcp] extra) (v0.71.28)
 soup mcp serve --allow-mutating               Also expose plan-only train_start / export tools (never execute) (v0.71.28)
 soup shrink --model <id|path> --drop-ratio 0.25 --calib c.jsonl -o shrunk  Depth-prune least-important layer block + SHIP/DON'T-SHIP ppl verdict (exit 0/2/1) (v0.71.29)
@@ -179,6 +182,10 @@ soup draft measure ... --min-acceptance 0.6 -o report.json  Exit 2 below the flo
 soup draft distill --target <tuned> --draft-base <tiny> --data d.jsonl -o draft/  Distil a DENSE speculative-decoding draft + register it (v0.71.33)
 soup draft distill ... --steps N --device cpu --force --plan-only  Training budget / device / overwrite -o / render the config only
 soup draft list                               List local drafts that `soup serve --auto-spec` will pick up (v0.71.33)
+soup reward synth refs.jsonl -o reward.py     Synthesize a deterministic reward verifier from gold outputs (v0.71.40)
+soup reward synth ... --kind numeric|json_schema|regex|tool_call  Force a verifier family (default: auto-detect)
+soup reward synth ... --plan-only             Report the induced spec + calibration plan; write nothing
+soup reward synth ... --output-report r.json --min-discrimination 0.5  Save the calibration JSON / set the refusal threshold (exit 0 emit / 2 refuse / 1 error)
 soup tui                                      Full-screen Textual dashboard (requires [tui] extra)
 soup train --config soup.yaml --profile       Record torch.profiler trace to <output>/profiles/
 soup --log-level quiet|normal|verbose|debug   Global logging tier (Rich-formatted)

{soup_cli-0.71.38 → soup_cli-0.71.40}/docs/evaluation.md

@@ -378,6 +378,36 @@ Exit codes (v0.71.38): **0 = SHIP · 2 = DON'T SHIP · 3 = usage/flag error · 1
 lives in `soup_cli.utils.ship_verdict` (`decide_ship` is a pure function — the whole truth table
 is CPU-testable); the bundled suites live in `soup_cli.eval.gate_suites`.
 
+### Closing the evidence loop (v0.71.39)
+
+The verdict is now emittable, committable, and provenance-bound so a fine-tuning gate runs on
+every PR instead of relying on a hand-edited JSON file.
+
+- **`--emit-evidence <path>`** re-serialises the scores into the `--evidence` INPUT schema, so a
+  run's output is replayable as input — feeding it back through `--evidence` (same threshold)
+  reproduces an identical verdict.
+- **`--config soup.yaml`** reads a committed `eval.ship` block for the gate defaults
+  (`task_eval` / `task_mode` / `general_suite` / `forgetting_threshold` / `judge_model` /
+  `baseline`); an explicit CLI flag always wins. This makes the gate reviewable in a PR diff.
+- **Provenance + staleness.** With `--emit-evidence`, `--config` STAMPS a `provenance` block
+  (`config_sha` — a semantic, order-insensitive recipe hash that EXCLUDES the `eval.ship` gate
+  policy, so tuning the threshold never invalidates evidence — plus `base_model` and a
+  best-effort `data_sha`). With `--evidence` alone, `--config` GATES: it refuses (exit 3)
+  evidence whose `config_sha` drifted from the committed config.
+- **`--push owner/repo#N`** posts the verdict as a GitHub PR comment (best-effort — a missing
+  token or `gh` failure warns but never flips the SHIP / DON'T-SHIP exit code).
+
+```bash
+# Producer (train job): compute or stamp scores, bound to the committed recipe
+soup ship --evidence scores.json --config soup.yaml --emit-evidence ship_evidence.json
+
+# Gate (PR CI): refuse evidence that doesn't match the committed config, comment the verdict
+soup ship --evidence ship_evidence.json --config soup.yaml --push owner/repo#42
+```
+
+`soup ci init --config soup.yaml` binds the generated workflow's ship step to the committed
+config, so the whole loop runs in CI (see [commands.md](commands.md)).
+
 
 ## NLG Evaluation Metrics (BLEU + ROUGE)
 

{soup_cli-0.71.38 → soup_cli-0.71.40}/docs/training.md

@@ -866,6 +866,37 @@ training:
   reward_fn: ./my_reward.py
 ```
 
+**Reward ensembles** — list several rewards, comma-separated, and they combine (GRPO only).
+This also unlocks the `rm_ensemble` reward-hack detector, which needs ≥ 2 rewards:
+```yaml
+training:
+  reward_fn: "accuracy,format"   # both are scored every step
+```
+
+### Synthesize a verifier from your data (`soup reward synth`)
+
+Don't hand-write a verifier — generate one from reference (gold) outputs. Soup infers a
+*deterministic* verifier (numeric / JSON-schema / regex / tool-call), writes a readable, editable
+`.py`, and **refuses to emit** one that can't tell your references from auto-generated bad answers
+(the mandatory calibration report). The emitted file is a normal `reward_fn: reward.py`.
+
+```bash
+# infer + calibrate + emit (exit 0 kept, 2 refused, 1 error)
+soup reward synth references.jsonl -o reward.py --output-report calib.json
+
+# preview the induced spec without writing anything
+soup reward synth references.jsonl --plan-only
+
+# force a family instead of auto-detecting
+soup reward synth answers.jsonl -o reward.py --kind numeric --tolerance 1e-6
+```
+
+References are a JSONL where each row's gold answer is in an `answer` field (override with
+`--field`) or the last assistant turn of a `messages` list. `--min-discrimination` sets how
+strongly the verifier must separate references from perturbed negatives before it's emitted.
+v1 is deterministic families only — a `\boxed{}`/`####` marker helps the numeric verifier, and
+completions are prompted to mark their answer (standard RLVR practice).
+
 ### Verifiable Rewards (RLVR)
 
 Use `reward_fn: verifiable` with a `verifiable_domain` for deterministic, math-checkable rewards — no judge model, no heuristics. Great for GRPO on math, code, or structured-output tasks.

{soup_cli-0.71.38 → soup_cli-0.71.40}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.38"
+version = "0.71.40"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.38 → soup_cli-0.71.40}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
 
-__version__ = "0.71.38"
+__version__ = "0.71.40"

{soup_cli-0.71.38 → soup_cli-0.71.40}/src/soup_cli/cli.py

@@ -563,6 +563,18 @@ app.add_typer(
     ),
 )
 
+# v0.71.40 — `soup reward synth` auto-generate a deterministic verifier.
+from soup_cli.commands import reward as _reward_cmd  # noqa: E402
+
+app.add_typer(
+    _reward_cmd.app,
+    name="reward",
+    help=(
+        "Synthesize a deterministic reward verifier from reference outputs, with "
+        "a calibration report that refuses degenerate verifiers (v0.71.40)."
+    ),
+)
+
 
 def _rewrite_advise_argv(argv: list) -> list:
     """Inject `run` between `advise` and a non-subcommand first argument.

{soup_cli-0.71.38 → soup_cli-0.71.40}/src/soup_cli/commands/ci.py

@@ -6,6 +6,8 @@
 
 from __future__ import annotations
 
+from typing import Optional
+
 import typer
 from rich.console import Console
 from rich.markup import escape
@@ -33,6 +35,10 @@ def init(
         "3.11", "--python", help="Python version for the CI runner (e.g. 3.11)"
     ),
     branch: str = typer.Option("main", "--branch", help="Branch the workflow triggers on"),
+    config: Optional[str] = typer.Option(
+        None, "--config",
+        help="Bind the ship gate to a committed soup.yaml (refuses stale evidence)",
+    ),
     output: str = typer.Option(
         ".github/workflows/soup-gate.yml", "-o", "--output", help="Workflow output path"
     ),
@@ -48,6 +54,7 @@ def init(
             branch=branch,
             output_path=output,
             overwrite=force,
+            config_path=config,
         )
     except (ValueError, TypeError, OSError) as exc:
         console.print(f"[red]{escape(str(exc))}[/]")