soup-cli 0.71.37__tar.gz → 0.71.38__tar.gz

{soup_cli-0.71.37 → soup_cli-0.71.38}/CHANGELOG.md

@@ -12,6 +12,48 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.38] - 2026-07-17
+
+**`soup ship`'s regression leg now has teeth.** Leg 2 (the catastrophic-forgetting
+/ regression gate that carries the whole SHIP / DON'T-SHIP claim) was 15
+hand-written trivia prompts scored by case-insensitive **substring** containment
+— it scored `"B"` for "**B**erlin", `"ok"` for "lo**ok**", `"3"` for "1**3**", and
+had **zero** items for tool-calling, safety, or JSON validity. This release makes
+the gate real: a fixed, extraction-based scorer + bundled, offline, zero-dep eval
+suites that catch a regression the old gate waved through.
+
+### Changed
+
+- **Fixed answer scorer (breaking — verdicts can change).** `soup ship`'s leg-2
+  MCQ / instruction / arithmetic answers are now scored by answer-**extraction**
+  + a boundary-aware match, replacing the raw substring test. A spurious
+  substring inside another word ("Berlin", "look", "13") no longer scores a
+  correct answer, so an existing run's verdict may flip — intentionally, because
+  the old gate was reporting false negatives.
+- **Bundled, offline general suite.** The default `--general-suite` is now seven
+  hand-authored suites shipped in the wheel: `mini_mmlu` / `mini_common_sense` /
+  `mini_instruction` (expanded), a new `mini_arithmetic`, and three behavioural
+  suites the old gate had no coverage for — `mini_tool_call` (function-calling),
+  `mini_format_json` (JSON validity), and `mini_safety` (refusal-rate). Each is
+  scored to a per-model absolute score by the pure scorers Soup already ships
+  (`eval/custom`, `utils/diagnose`); no lm-eval, no network, no download. Every
+  suite is large enough that a single-item flip (1/N < 0.05) trips the default
+  threshold instead of being rounded away.
+- **`soup ship` exit codes: usage errors moved 2 → 3.** Exit `2` now means only
+  DON'T-SHIP; a typo'd flag or bad `--general-suite` exits `3` (mirroring
+  `soup plan` / `soup env check`), so CI can tell a config error from a caught
+  regression. Offline `--evidence` read/parse errors stay `1`. **Breaking** for
+  anyone parsing exit `2` as "usage error".
+
+### Fixed
+
+- `soup ship` help + docstrings no longer describe `--task-mode pairwise` as
+  "reserved for a later release" (it shipped in v0.71.31); the dead
+  `SUPPORTED_TASK_MODES` gate is removed.
+- `soup diagnose`'s package docstring said "Six" probes (there are seven —
+  citation) and pointed live loading at an unshipped version; it now re-exports
+  all seven `score_*` probe functions so callers need not reach into submodules.
+
 ## [0.71.37] - 2026-07-17
 
 **Every `pip install soup-cli[extra]` command now works on Windows `cmd.exe`**, and

{soup_cli-0.71.37 → soup_cli-0.71.38}/CONTRIBUTING.md

@@ -120,7 +120,7 @@ src/soup_cli/
   templates/          - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (315 files, 16288 tests)
+tests/                - Test suite (316 files, 16330 tests)
 examples/             - Real-world config examples and datasets
 ```
 

{soup_cli-0.71.37 → soup_cli-0.71.38}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.37
+Version: 0.71.38
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -193,38 +193,45 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.36 — Data Moat II.** A semantic layer over your training data, plus two tools for what a fine-tune forgets and what it leaks.
-
-- **`soup data dedup --semantic`** — dedup by meaning, not by shared tokens. Catches
-  *reworded* duplicates MinHash's shingling scores as distinct. Zero new dependencies.
-- **`soup data topics <data>`** — cluster your data and get a coverage table
-  (`82% code · 6% math · 1% safety`) with a warning for thin topics, so you can see what
-  you are actually training on.
-- **`soup data canary insert|check`** — insert unique secrets, then prove whether a model
-  memorized them: each secret's loss is ranked against never-inserted controls. Exit 2 on a
-  leak, so CI can gate. On SmolLM2-135M a memorized set lands at percentile 0.0 against a
-  clean model's 4%–93% spread.
-- **`soup train --replay old.jsonl`** — continual-learning rehearsal: interleave your old
-  data so the new task doesn't erase it.
-- **Honest result:** semantic dedup is **not** a paraphrase detector. Measured, paraphrase
-  cosines (0.49–0.76) *overlap* with genuinely-distinct rows (0.54–0.76) — "Add two numbers"
-  vs "Multiply two numbers" scores **higher** than a real paraphrase — so no threshold
-  separates them. The default is deliberately conservative; see the CHANGELOG.
-- **Two blocking bugs fixed:** the hardware-fit gate **refused to train any model you merged
-  yourself** (a local checkpoint's name has no size marker, so it guessed 7B and predicted
-  16 GB), and every `pip install "soup-cli[extra]"` hint printed **without the extra** —
-  17 sites where the suggested command succeeds and leaves the feature still broken.
+**v0.71.38 — The gate grows teeth.** `soup ship` answers the one question that matters after a fine-tune — *did my task improve WITHOUT silently breaking everything else?* This release makes the "without breaking everything else" half real.
+
+- **A gate that actually catches regressions.** The regression leg used to be 15 trivia
+  prompts scored by raw substring — it credited `"B"` for "**B**erlin", `"3"` for "1**3**", and
+  had **zero** items for tool-calling, safety, or JSON. Now it's a fixed, extraction-based
+  scorer over seven bundled, offline suites (MCQ · arithmetic · **tool-calling** · **JSON
+  validity** · **safety/refusal**). A tune that WINS your task but quietly breaks tool-calling
+  now gets a **DON'T SHIP** — the old gate waved it through.
+- **Zero new dependencies, no network.** Every suite ships in the wheel and is scored by the
+  pure scorers Soup already has — no lm-eval, no download. `soup ci init`'s core-only gate keeps
+  working.
+- **Distinct exit codes for CI.** Usage errors moved off exit `2` to `3`, so `2` now means only
+  DON'T-SHIP — a typo'd flag is no longer indistinguishable from a caught regression.
+- **Breaking, on purpose.** The fixed scorer can change an existing run's verdict — because the
+  old one was reporting false negatives. Recompute any committed `--baseline`.
 
 ```bash
-soup data topics train.jsonl                    # what am I actually training on?
-soup data dedup train.jsonl --semantic -o clean.jsonl
+# leg 1 (task win) AND leg 2 (no regression on the bundled suite) -> one verdict
+soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
+#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error
 
-soup data canary insert train.jsonl -o canaried.jsonl --manifest secrets.json
-soup data canary check --manifest secrets.json --base ./my-model  # exit 2 = leak
+soup ship --evidence scores.json    # offline verdict from pre-computed scores
+```
+
+<details>
+<summary>Previous release — v0.71.36, Data Moat II (semantic dedup · topics · canaries · replay)</summary>
+
+`soup data dedup --semantic` dedups by meaning; `soup data topics` shows a coverage table;
+`soup data canary insert|check` proves whether a model memorized inserted secrets (exit 2 on a
+leak); `soup train --replay old.jsonl` interleaves old data so a new task doesn't erase it.
 
+```bash
+soup data topics train.jsonl
+soup data canary check --manifest secrets.json --base ./my-model  # exit 2 = leak
 soup train --config sft.yaml --replay old_task.jsonl --replay-ratio 0.1
 ```
 
+</details>
+
 <details>
 <summary>Previous release — v0.71.33, <code>soup draft</code> (measure speculative decoding)</summary>
 

{soup_cli-0.71.37 → soup_cli-0.71.38}/README.md

@@ -49,38 +49,45 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.36 — Data Moat II.** A semantic layer over your training data, plus two tools for what a fine-tune forgets and what it leaks.
-
-- **`soup data dedup --semantic`** — dedup by meaning, not by shared tokens. Catches
-  *reworded* duplicates MinHash's shingling scores as distinct. Zero new dependencies.
-- **`soup data topics <data>`** — cluster your data and get a coverage table
-  (`82% code · 6% math · 1% safety`) with a warning for thin topics, so you can see what
-  you are actually training on.
-- **`soup data canary insert|check`** — insert unique secrets, then prove whether a model
-  memorized them: each secret's loss is ranked against never-inserted controls. Exit 2 on a
-  leak, so CI can gate. On SmolLM2-135M a memorized set lands at percentile 0.0 against a
-  clean model's 4%–93% spread.
-- **`soup train --replay old.jsonl`** — continual-learning rehearsal: interleave your old
-  data so the new task doesn't erase it.
-- **Honest result:** semantic dedup is **not** a paraphrase detector. Measured, paraphrase
-  cosines (0.49–0.76) *overlap* with genuinely-distinct rows (0.54–0.76) — "Add two numbers"
-  vs "Multiply two numbers" scores **higher** than a real paraphrase — so no threshold
-  separates them. The default is deliberately conservative; see the CHANGELOG.
-- **Two blocking bugs fixed:** the hardware-fit gate **refused to train any model you merged
-  yourself** (a local checkpoint's name has no size marker, so it guessed 7B and predicted
-  16 GB), and every `pip install "soup-cli[extra]"` hint printed **without the extra** —
-  17 sites where the suggested command succeeds and leaves the feature still broken.
+**v0.71.38 — The gate grows teeth.** `soup ship` answers the one question that matters after a fine-tune — *did my task improve WITHOUT silently breaking everything else?* This release makes the "without breaking everything else" half real.
+
+- **A gate that actually catches regressions.** The regression leg used to be 15 trivia
+  prompts scored by raw substring — it credited `"B"` for "**B**erlin", `"3"` for "1**3**", and
+  had **zero** items for tool-calling, safety, or JSON. Now it's a fixed, extraction-based
+  scorer over seven bundled, offline suites (MCQ · arithmetic · **tool-calling** · **JSON
+  validity** · **safety/refusal**). A tune that WINS your task but quietly breaks tool-calling
+  now gets a **DON'T SHIP** — the old gate waved it through.
+- **Zero new dependencies, no network.** Every suite ships in the wheel and is scored by the
+  pure scorers Soup already has — no lm-eval, no download. `soup ci init`'s core-only gate keeps
+  working.
+- **Distinct exit codes for CI.** Usage errors moved off exit `2` to `3`, so `2` now means only
+  DON'T-SHIP — a typo'd flag is no longer indistinguishable from a caught regression.
+- **Breaking, on purpose.** The fixed scorer can change an existing run's verdict — because the
+  old one was reporting false negatives. Recompute any committed `--baseline`.
 
 ```bash
-soup data topics train.jsonl                    # what am I actually training on?
-soup data dedup train.jsonl --semantic -o clean.jsonl
+# leg 1 (task win) AND leg 2 (no regression on the bundled suite) -> one verdict
+soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
+#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error
 
-soup data canary insert train.jsonl -o canaried.jsonl --manifest secrets.json
-soup data canary check --manifest secrets.json --base ./my-model  # exit 2 = leak
+soup ship --evidence scores.json    # offline verdict from pre-computed scores
+```
+
+<details>
+<summary>Previous release — v0.71.36, Data Moat II (semantic dedup · topics · canaries · replay)</summary>
+
+`soup data dedup --semantic` dedups by meaning; `soup data topics` shows a coverage table;
+`soup data canary insert|check` proves whether a model memorized inserted secrets (exit 2 on a
+leak); `soup train --replay old.jsonl` interleaves old data so a new task doesn't erase it.
 
+```bash
+soup data topics train.jsonl
+soup data canary check --manifest secrets.json --base ./my-model  # exit 2 = leak
 soup train --config sft.yaml --replay old_task.jsonl --replay-ratio 0.1
 ```
 
+</details>
+
 <details>
 <summary>Previous release — v0.71.33, <code>soup draft</code> (measure speculative decoding)</summary>
 

{soup_cli-0.71.37 → soup_cli-0.71.38}/docs/commands.md

@@ -160,11 +160,12 @@ soup runs show <run_id>                       Run details + loss graph + cost
 soup runs compare <run_1> <run_2>             Compare two runs
 soup runs replay <run_id>                     Replay summary + loss curve from history (also plots a benchmark-score curve when the metric lives in eval_results)
 soup why [run_id]                             Explain training anomalies (heuristic)
-soup ship --base <m> --adapter <lora> --task-eval t.jsonl  SHIP / DON'T-SHIP verdict: task win AND no forgetting (exit 0=SHIP / 2=DON'T / 1=error) (v0.71.25)
+soup ship --base <m> --adapter <lora> --task-eval t.jsonl  SHIP / DON'T-SHIP verdict: task win AND no regression on the bundled suite (exit 0=SHIP / 2=DON'T / 3=usage / 1=runtime) (v0.71.25; leg-2 real + usage-off-2 v0.71.38)
 soup ship --evidence ev.json [--output v.json]  Decide offline from pre-computed scores (no model load)
 soup ship ... --task-mode judge_score --judge-model ollama://llama3.1  Leg-1 via LLM-as-a-judge
 soup ship ... --task-mode pairwise --judge-model ollama://llama3.1  Leg-1 via swap-debiased judge win-rate (base=0.5) (v0.71.31)
-soup ship ... --general-suite mmlu,gsm8k --baseline base.json  lm-eval leg-2 + recorded base scores
+soup ship ...  # leg-2 default = 7 bundled offline suites (MCQ/arithmetic + tool_call/format_json/safety, extraction scorer, ~40 items each) (v0.71.38)
+soup ship ... --general-suite mmlu,gsm8k --baseline base.json  lm-eval leg-2 override + recorded base scores
 soup card <registry-id> -o MODELCARD.md       HF model card from a registry entry: training config, evals, hashes, lineage, artifacts (v0.71.35)
 soup push --model ./out --repo you/m --card <registry-id>  Upload that registry-driven card as the README (HF only) (v0.71.35)
 soup ci init [--data d.jsonl --suite s.yaml --evidence ev.json] [--branch main --python 3.11] [--force]  Write .github/workflows/soup-gate.yml: data validate -> expect -> ship gate on every PR (v0.71.35)

{soup_cli-0.71.37 → soup_cli-0.71.38}/docs/evaluation.md

@@ -355,11 +355,28 @@ soup ship --evidence evidence.json --output verdict.json
 }
 ```
 
-Leg-2 defaults to the built-in mini benchmarks (`mini_mmlu` / `mini_common_sense` /
-`mini_instruction`) — offline, CPU, instant. `--general-suite <names>` with non-mini names
-routes through the lm-eval harness. Pairwise judge win-rate (`--task-mode pairwise`) is planned
-for a later release. The engine lives in `soup_cli.utils.ship_verdict` (`decide_ship` is a pure
-function — the whole truth table is CPU-testable).
+Leg-2 defaults to the **bundled offline suite** (v0.71.38) — seven hand-authored suites shipped
+in the wheel and scored by the pure scorers Soup already ships (no lm-eval, no network,
+CPU-instant):
+
+| Suite | What it checks | Scorer |
+|-------|----------------|--------|
+| `mini_mmlu` / `mini_common_sense` / `mini_instruction` / `mini_arithmetic` | general knowledge / reasoning / instruction-following / numeracy | answer-extraction + exact/boundary match |
+| `mini_tool_call` | function-calling still works (right tool named) | `tool_call_name_match` |
+| `mini_format_json` | JSON validity (a structured object, not a bare scalar) | container-only JSON check |
+| `mini_safety` | refusal-rate on harmful prompts (under-refusal = regression) | refusal heuristic |
+
+Each suite is >20 items so a single-item flip (1/N < 0.05) trips the default threshold instead
+of being rounded away. The scorer is answer-**extraction** — a spurious substring inside a word
+(`"B"` in "**B**erlin") no longer scores, which is a **breaking** change from the v0.25.0
+substring scorer (an existing run's verdict can flip; recompute any committed `--baseline`).
+`--general-suite <names>` with any non-bundled name routes through the lm-eval harness. Pairwise
+judge win-rate (`--task-mode pairwise`) shipped in v0.71.31.
+
+Exit codes (v0.71.38): **0 = SHIP · 2 = DON'T SHIP · 3 = usage/flag error · 1 = runtime error**
+— usage errors moved off `2` so CI can tell a config typo from a caught regression. The engine
+lives in `soup_cli.utils.ship_verdict` (`decide_ship` is a pure function — the whole truth table
+is CPU-testable); the bundled suites live in `soup_cli.eval.gate_suites`.
 
 
 ## NLG Evaluation Metrics (BLEU + ROUGE)

{soup_cli-0.71.37 → soup_cli-0.71.38}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.37"
+version = "0.71.38"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"
@@ -127,6 +127,7 @@ artifacts = [
   "src/soup_cli/data/_fixtures/*.jsonl",
   "src/soup_cli/data/_fixtures/unlearning/*.jsonl",
   "src/soup_cli/data/_fixtures/behavior/*.jsonl",
+  "src/soup_cli/data/_fixtures/gate/*.jsonl",
 ]
 
 [tool.ruff]

{soup_cli-0.71.37 → soup_cli-0.71.38}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
 
-__version__ = "0.71.37"
+__version__ = "0.71.38"

{soup_cli-0.71.37 → soup_cli-0.71.38}/src/soup_cli/commands/ship.py

@@ -11,14 +11,19 @@ break it? The decision fuses two legs (task win + catastrophic-forgetting
 guard) into a single binary verdict — see ``utils/ship_verdict.py`` for the
 moat (``decide_ship``).
 
-Exit codes mirror ``soup diagnose`` so CI can gate on the result:
-**0 = SHIP, 2 = DON'T SHIP, 1 = runtime error**.
-
-Leg 1 (task win) modes: ``metric`` (reuses ``eval/custom.run_eval`` accuracy)
-and ``judge_score`` (reuses ``eval/judge.JudgeEvaluator``). True pairwise
-win-rate is reserved for a later release. Leg 2 (general suite) defaults to the
-built-in mini benchmarks (``eval/forgetting``); ``--general-suite`` with
-non-mini names routes through the existing lm-eval runner.
+Exit codes so CI can gate on the result:
+**0 = SHIP, 2 = DON'T SHIP, 3 = usage/validation error, 1 = runtime error**.
+Usage errors moved off ``2`` in v0.71.38 — a typo'd flag was previously
+indistinguishable from a caught regression (both exited ``2``); ``3`` mirrors
+``soup plan`` / ``soup env check``. Offline ``--evidence`` read/parse errors
+stay ``1``.
+
+Leg 1 (task win) modes: ``metric`` (reuses ``eval/custom.run_eval`` accuracy),
+``judge_score`` (reuses ``eval/judge.JudgeEvaluator``), and ``pairwise`` (true
+judge win-rate, v0.71.31). Leg 2 (general suite) defaults to the bundled offline
+suite (``eval/gate_suites`` — MCQ/arithmetic + tool-call/JSON/safety, scored by
+the pure diagnose/custom scorers, v0.71.38); ``--general-suite`` with any
+non-bundled name routes through the existing lm-eval runner.
 """
 
 from __future__ import annotations
@@ -50,6 +55,11 @@ console = Console()
 
 app = typer.Typer(no_args_is_help=False)
 
+# Exit-code taxonomy (v0.71.38): keep DON'T-SHIP distinct from a config typo.
+_EXIT_RUNTIME = 1  # something went wrong actually running (IO, model load, ...)
+_EXIT_DONT_SHIP = 2  # a verdict: leg 1 or leg 2 said don't ship
+_EXIT_USAGE = 3  # bad flags / validation (mirrors `soup plan` / `env check`)
+
 # 16 MiB cap on evidence JSON (mirrors `soup diagnose` — prevents a
 # multi-GB / symlink-pointed file from OOMing at json.load time).
 _MAX_EVIDENCE_BYTES = 16 * 1024 * 1024
@@ -74,29 +84,26 @@ def _fail(message: str, code: int) -> NoReturn:
 
 
 def _validate_threshold_flag(value: float) -> float:
-    # Fast-fail a bad flag with an exit-2 USAGE error here; the engine's own
+    # Fast-fail a bad flag with a usage error (exit 3) here; the engine's own
     # _validate_threshold raises ValueError (-> exit 1 runtime), which is the
     # wrong exit code for a CLI typo. Intentional, narrow duplication.
     if not isinstance(value, (int, float)) or isinstance(value, bool):
-        _fail("--forgetting-threshold must be a number", 2)
+        _fail("--forgetting-threshold must be a number", _EXIT_USAGE)
     fvalue = float(value)
     # NaN fails both comparisons -> rejected.
     if not (0.0 <= fvalue <= 1.0):
-        _fail("--forgetting-threshold must be in [0.0, 1.0]", 2)
+        _fail("--forgetting-threshold must be in [0.0, 1.0]", _EXIT_USAGE)
     return fvalue
 
 
 def _validate_task_mode_flag(task_mode: str) -> None:
+    # All three modes (metric / judge_score / pairwise) ship as of v0.71.31, so
+    # SUPPORTED_TASK_MODES == TASK_MODES and the old "pairwise reserved" gate is
+    # gone (it was dead code).
     if task_mode not in TASK_MODES:
         _fail(
             f"--task-mode must be one of {', '.join(TASK_MODES)}; got {task_mode!r}",
-            2,
-        )
-    if task_mode not in SUPPORTED_TASK_MODES:
-        _fail(
-            f"--task-mode {task_mode!r} (pairwise judge win-rate) ships in a "
-            "later release; use 'metric' or 'judge_score' for now",
-            2,
+            _EXIT_USAGE,
         )
 
 
@@ -116,7 +123,7 @@ def _validate_judge_model_url(url: str) -> None:
     _fail(
         f"--judge-model {url!r} uses a disallowed scheme/host; "
         "use ollama://, https://, or http://localhost",
-        2,
+        _EXIT_USAGE,
     )
 
 
@@ -165,33 +172,33 @@ def _verdict_from_evidence(path: str, *, forgetting_threshold: float) -> ShipVer
     try:
         payload = _load_evidence(path)
     except (OSError, ValueError, json.JSONDecodeError) as exc:
-        _fail(f"cannot read --evidence: {exc}", 1)
+        _fail(f"cannot read --evidence: {exc}", _EXIT_RUNTIME)
 
     task = payload.get("task")
     if not isinstance(task, dict):
-        _fail("evidence.task must be an object with 'mode', 'base', 'tuned'", 1)
+        _fail("evidence.task must be an object with 'mode', 'base', 'tuned'", _EXIT_RUNTIME)
     mode = task.get("mode", "metric")
     if mode not in SUPPORTED_TASK_MODES:
         _fail(
             f"evidence.task.mode must be one of {', '.join(SUPPORTED_TASK_MODES)}; "
             f"got {mode!r}",
-            1,
+            _EXIT_RUNTIME,
         )
     if "base" not in task or "tuned" not in task:
-        _fail("evidence.task needs both 'base' and 'tuned' scores", 1)
+        _fail("evidence.task needs both 'base' and 'tuned' scores", _EXIT_RUNTIME)
     try:
         task_win = build_task_win(mode, task["base"], task["tuned"])
     except (TypeError, ValueError) as exc:
-        _fail(f"invalid evidence.task: {exc}", 1)
+        _fail(f"invalid evidence.task: {exc}", _EXIT_RUNTIME)
 
     raw_benchmarks = payload.get("benchmarks", {})
     if not isinstance(raw_benchmarks, dict):
-        _fail("evidence.benchmarks must be an object of {name: {base, tuned}}", 1)
+        _fail("evidence.benchmarks must be an object of {name: {base, tuned}}", _EXIT_RUNTIME)
     base_scores: Dict[str, object] = {}
     tuned_scores: Dict[str, object] = {}
     for name, entry in raw_benchmarks.items():
         if not isinstance(entry, dict) or "base" not in entry or "tuned" not in entry:
-            _fail(f"evidence.benchmarks[{name!r}] needs 'base' and 'tuned'", 1)
+            _fail(f"evidence.benchmarks[{name!r}] needs 'base' and 'tuned'", _EXIT_RUNTIME)
         base_scores[str(name)] = entry["base"]
         tuned_scores[str(name)] = entry["tuned"]
 
@@ -201,7 +208,7 @@ def _verdict_from_evidence(path: str, *, forgetting_threshold: float) -> ShipVer
         )
         return decide_ship(task_win, deltas, forgetting_threshold=forgetting_threshold)
     except (TypeError, ValueError) as exc:
-        _fail(f"invalid evidence.benchmarks: {exc}", 1)
+        _fail(f"invalid evidence.benchmarks: {exc}", _EXIT_RUNTIME)
 
 
 # ---------------------------------------------------------------------------
@@ -312,12 +319,6 @@ def _leg1_pairwise(
     return build_task_win("pairwise", 0.5, winrate)
 
 
-def _mini_score(gen: Callable[[str], str], benchmark: str) -> float:
-    from soup_cli.eval.forgetting import ForgettingDetector
-
-    return ForgettingDetector(generate_fn=gen, benchmark=benchmark).run_baseline()
-
-
 def _extract_lm_score(bench_data: Mapping[str, object]) -> Optional[float]:
     """Pull a single accuracy metric from an lm-eval per-task result block."""
     for key in ("acc,none", "acc_norm,none", "exact_match,none", "em,none"):
@@ -399,24 +400,26 @@ def _leg2_scores(
 ) -> Tuple[Dict[str, object], Dict[str, object]]:
     """Compute leg-2 ``(base_scores, tuned_scores)`` maps over the general suite.
 
-    Mini-benchmark names run live via ForgettingDetector; any other names route
-    through the lm-eval override. ``baseline_scores`` supplies base scores
-    directly (skipping the base run) for any name it covers.
+    Bundled suite names (v0.71.38 — the MCQ/arithmetic *and* the behavioural
+    tool-call / JSON-format / safety suites) are scored offline via
+    ``gate_suites.score_bundled_suite``; any other name routes through the
+    lm-eval override. ``baseline_scores`` supplies base scores directly
+    (skipping the base run) for any name it covers.
     """
-    from soup_cli.eval.forgetting import MINI_BENCHMARKS
+    from soup_cli.eval.gate_suites import is_bundled_suite, score_bundled_suite
 
-    mini_names = [n for n in suite_names if n in MINI_BENCHMARKS]
-    other_names = [n for n in suite_names if n not in MINI_BENCHMARKS]
+    bundled_names = [n for n in suite_names if is_bundled_suite(n)]
+    other_names = [n for n in suite_names if not is_bundled_suite(n)]
 
     base_map: Dict[str, object] = {}
     tuned_map: Dict[str, object] = {}
 
-    for name in mini_names:
-        tuned_map[name] = _mini_score(tuned_gen, name)
+    for name in bundled_names:
+        tuned_map[name] = score_bundled_suite(name, tuned_gen)
         if name in baseline_scores:
             base_map[name] = float(baseline_scores[name])
         else:
-            base_map[name] = _mini_score(base_gen, name)
+            base_map[name] = score_bundled_suite(name, base_gen)
 
     if other_names:
         lm_base, lm_tuned = _lm_eval_leg2(
@@ -439,10 +442,10 @@ def _leg2_scores(
 
 
 def _parse_suite(general_suite: Optional[str]) -> List[str]:
-    from soup_cli.eval.forgetting import MINI_BENCHMARKS
+    from soup_cli.eval.gate_suites import DEFAULT_GENERAL_SUITE
 
     if not general_suite:
-        return list(MINI_BENCHMARKS.keys())
+        return list(DEFAULT_GENERAL_SUITE)
     names = [chunk.strip() for chunk in general_suite.split(",")]
     return [name for name in names if name]
 
@@ -462,32 +465,32 @@ def _verdict_live(
 ) -> ShipVerdict:
     """Run a live verdict — validate flags (exit 2), then evaluate (exit 1)."""
     if not base:
-        _fail("live run needs --base <model>", 2)
+        _fail("live run needs --base <model>", _EXIT_USAGE)
     if adapter and tuned:
-        _fail("pass --adapter OR --tuned, not both", 2)
+        _fail("pass --adapter OR --tuned, not both", _EXIT_USAGE)
     if not adapter and not tuned:
-        _fail("live run needs --tuned <model> or --adapter <adapter-path>", 2)
+        _fail("live run needs --tuned <model> or --adapter <adapter-path>", _EXIT_USAGE)
     if not task_eval:
-        _fail("live run needs --task-eval <tasks.jsonl> for the leg-1 task win", 2)
+        _fail("live run needs --task-eval <tasks.jsonl> for the leg-1 task win", _EXIT_USAGE)
     try:
         enforce_under_cwd_and_no_symlink(task_eval, "--task-eval path")
     except (ValueError, TypeError) as exc:
-        _fail(str(exc), 2)
+        _fail(str(exc), _EXIT_USAGE)
 
     suite_names = _parse_suite(general_suite)
     if not suite_names:
-        _fail("--general-suite resolved to no benchmarks", 2)
+        _fail("--general-suite resolved to no benchmarks", _EXIT_USAGE)
     if len(suite_names) > _MAX_SUITE_BENCHMARKS:
         _fail(
             f"--general-suite has too many benchmarks (max {_MAX_SUITE_BENCHMARKS})",
-            2,
+            _EXIT_USAGE,
         )
     for _name in suite_names:
         if "\x00" in _name or len(_name) > _MAX_BENCHMARK_NAME_CHARS:
             _fail(
                 "--general-suite names must be null-free and "
                 f"< {_MAX_BENCHMARK_NAME_CHARS} chars",
-                2,
+                _EXIT_USAGE,
             )
 
     # Resolve --baseline up front so a bad spec (outside cwd / missing file /
@@ -499,19 +502,19 @@ def _verdict_live(
         try:
             baseline_scores = resolve_baseline(baseline_spec)
         except (ValueError, FileNotFoundError, OSError) as exc:
-            _fail(f"--baseline: {exc}", 2)
+            _fail(f"--baseline: {exc}", _EXIT_USAGE)
 
     tuned_id = tuned if tuned else base
     try:
         base_gen, tuned_gen = _resolve_generators(base, tuned, adapter, device)
         if task_mode == "judge_score":
             if not judge_model:
-                _fail("--task-mode judge_score needs --judge-model <url>", 2)
+                _fail("--task-mode judge_score needs --judge-model <url>", _EXIT_USAGE)
             _validate_judge_model_url(judge_model)
             task_win = _leg1_judge(base_gen, tuned_gen, task_eval, judge_model)
         elif task_mode == "pairwise":
             if not judge_model:
-                _fail("--task-mode pairwise needs --judge-model <url>", 2)
+                _fail("--task-mode pairwise needs --judge-model <url>", _EXIT_USAGE)
             _validate_judge_model_url(judge_model)
             task_win = _leg1_pairwise(base_gen, tuned_gen, task_eval, judge_model)
         else:
@@ -532,10 +535,10 @@ def _verdict_live(
         return decide_ship(task_win, deltas, forgetting_threshold=forgetting_threshold)
     except typer.Exit:
         # typer.Exit subclasses RuntimeError — re-raise so in-try _fail() usage
-        # errors (exit 2) keep their code instead of being re-coded as exit 1.
+        # errors (exit 3) keep their code instead of being re-coded as exit 1.
         raise
     except (ValueError, TypeError, OSError, RuntimeError, ImportError) as exc:
-        _fail(f"live ship verdict failed: {type(exc).__name__}: {exc}", 1)
+        _fail(f"live ship verdict failed: {type(exc).__name__}: {exc}", _EXIT_RUNTIME)
 
 
 # ---------------------------------------------------------------------------
@@ -551,9 +554,9 @@ def _emit_and_exit(verdict: ShipVerdict, output: Optional[str]) -> None:
             )
             console.print(f"[green]Wrote[/] {escape(output)}")
         except (OSError, ValueError, TypeError) as exc:
-            _fail(f"cannot write --output: {type(exc).__name__}: {exc}", 1)
+            _fail(f"cannot write --output: {type(exc).__name__}: {exc}", _EXIT_RUNTIME)
     if verdict.decision != DECISION_SHIP:
-        raise typer.Exit(code=2)
+        raise typer.Exit(code=_EXIT_DONT_SHIP)
 
 
 # ---------------------------------------------------------------------------
@@ -580,18 +583,23 @@ def ship(
         help="Leg-1 mode: metric | judge_score | pairwise (judge win-rate).",
     ),
     judge_model: Optional[str] = typer.Option(
-        None, "--judge-model", help="Judge model URL for --task-mode judge_score."
+        None,
+        "--judge-model",
+        help="Judge model URL for --task-mode judge_score or pairwise.",
     ),
     general_suite: Optional[str] = typer.Option(
         None,
         "--general-suite",
-        help="Comma list of leg-2 benchmarks (default: the 3 mini benchmarks; "
-        "non-mini names route through lm-eval).",
+        help="Comma list of leg-2 benchmarks (default: the bundled offline suite "
+        "— MCQ/arithmetic + tool-call/JSON/safety; non-bundled names route "
+        "through lm-eval).",
     ),
     baseline: Optional[str] = typer.Option(
         None,
         "--baseline",
-        help="registry://<id> or JSON file of base leg-2 scores (skips base run).",
+        help="registry://<id> or JSON file of base leg-2 scores (skips base run). "
+        "Recompute baselines captured before v0.71.38 — the leg-2 scorer changed, "
+        "so an old baseline is not comparable to a freshly-scored tuned model.",
     ),
     forgetting_threshold: float = typer.Option(
         DEFAULT_FORGETTING_THRESHOLD,
@@ -631,7 +639,7 @@ def ship(
         _fail(
             "provide --evidence <json> for an offline verdict, or "
             "--base + (--tuned|--adapter) + --task-eval for a live run",
-            2,
+            _EXIT_USAGE,
         )
 
     _emit_and_exit(verdict, output)