soup-cli 0.71.36__tar.gz → 0.71.38__tar.gz

{soup_cli-0.71.36 → soup_cli-0.71.38}/CHANGELOG.md

@@ -12,6 +12,87 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.38] - 2026-07-17
+
+**`soup ship`'s regression leg now has teeth.** Leg 2 (the catastrophic-forgetting
+/ regression gate that carries the whole SHIP / DON'T-SHIP claim) was 15
+hand-written trivia prompts scored by case-insensitive **substring** containment
+— it scored `"B"` for "**B**erlin", `"ok"` for "lo**ok**", `"3"` for "1**3**", and
+had **zero** items for tool-calling, safety, or JSON validity. This release makes
+the gate real: a fixed, extraction-based scorer + bundled, offline, zero-dep eval
+suites that catch a regression the old gate waved through.
+
+### Changed
+
+- **Fixed answer scorer (breaking — verdicts can change).** `soup ship`'s leg-2
+  MCQ / instruction / arithmetic answers are now scored by answer-**extraction**
+  + a boundary-aware match, replacing the raw substring test. A spurious
+  substring inside another word ("Berlin", "look", "13") no longer scores a
+  correct answer, so an existing run's verdict may flip — intentionally, because
+  the old gate was reporting false negatives.
+- **Bundled, offline general suite.** The default `--general-suite` is now seven
+  hand-authored suites shipped in the wheel: `mini_mmlu` / `mini_common_sense` /
+  `mini_instruction` (expanded), a new `mini_arithmetic`, and three behavioural
+  suites the old gate had no coverage for — `mini_tool_call` (function-calling),
+  `mini_format_json` (JSON validity), and `mini_safety` (refusal-rate). Each is
+  scored to a per-model absolute score by the pure scorers Soup already ships
+  (`eval/custom`, `utils/diagnose`); no lm-eval, no network, no download. Every
+  suite is large enough that a single-item flip (1/N < 0.05) trips the default
+  threshold instead of being rounded away.
+- **`soup ship` exit codes: usage errors moved 2 → 3.** Exit `2` now means only
+  DON'T-SHIP; a typo'd flag or bad `--general-suite` exits `3` (mirroring
+  `soup plan` / `soup env check`), so CI can tell a config error from a caught
+  regression. Offline `--evidence` read/parse errors stay `1`. **Breaking** for
+  anyone parsing exit `2` as "usage error".
+
+### Fixed
+
+- `soup ship` help + docstrings no longer describe `--task-mode pairwise` as
+  "reserved for a later release" (it shipped in v0.71.31); the dead
+  `SUPPORTED_TASK_MODES` gate is removed.
+- `soup diagnose`'s package docstring said "Six" probes (there are seven —
+  citation) and pointed live loading at an unshipped version; it now re-exports
+  all seven `score_*` probe functions so callers need not reach into submodules.
+
+## [0.71.37] - 2026-07-17
+
+**Every `pip install soup-cli[extra]` command now works on Windows `cmd.exe`**, and
+eval-gate benchmark tasks run instead of always failing.
+
+### Fixed
+
+- **Install hints are now quoted so they work in every shell.** Soup printed
+  `pip install 'soup-cli[ui]'` — bash / zsh / PowerShell syntax. `cmd.exe` has no
+  single-quote quoting, so it hands the quotes to pip verbatim and pip refuses:
+
+  ```
+  ERROR: Invalid requirement: "'soup-cli[train]'": Expected package name at the start of dependency specifier
+  ```
+
+  Every hint, README command, and docs example now uses `pip install
+  "soup-cli[extra]"`, which works in cmd, PowerShell, bash, and zsh alike — the
+  same spelling the repo already used for `pip install -e ".[dev]"`. Measured on
+  Windows: single quotes fail only on `cmd.exe`; double quotes pass everywhere;
+  dropping the quotes passes on Windows but breaks zsh, which globs the bracket.
+
+  Nothing in Soup can rescue the command after it is typed — pip and the shell
+  own it, and Soup is not installed yet when the README command runs — so the
+  fix is the spelling we print. A regression test now scans the package and every
+  docs code block for the single-quoted form.
+
+  If you followed an older tutorial and hit `Invalid requirement`, swap the `'`
+  for `"`; nothing is wrong with the package.
+
+- **Eval-gate `type: benchmark` tasks now actually run.** `eval/gate.py` probed
+  for a `forgetting.run_mini_benchmark` helper that never existed, so every
+  `type: benchmark` task in a gate suite failed 100% of the time — while
+  advising an `[eval]` extras install that could not fix it. The gate now calls
+  `ForgettingDetector` directly (the same way `soup ship` already did), and an
+  unknown benchmark name fails with the list of valid names. Thanks
+  [@Sanjays2402](https://github.com/Sanjays2402)!
+  ([#315](https://github.com/MakazhanAlpamys/Soup/pull/315), closes
+  [#310](https://github.com/MakazhanAlpamys/Soup/issues/310))
+
 ## [0.71.36] - 2026-07-16
 
 **Data Moat II** — a semantic layer over your training data, plus two tools for

{soup_cli-0.71.36 → soup_cli-0.71.38}/CONTRIBUTING.md

@@ -120,7 +120,7 @@ src/soup_cli/
   templates/          - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (314 files, 16283 tests)
+tests/                - Test suite (316 files, 16330 tests)
 examples/             - Real-world config examples and datasets
 ```
 

{soup_cli-0.71.36 → soup_cli-0.71.38}/CONTRIBUTORS.md

@@ -52,6 +52,8 @@ Listed by first contribution. PR numbers link the work.
   - Honor configured vocab expansion in the ORPO / SimPO / GRPO trainers ([#295](https://github.com/MakazhanAlpamys/Soup/pull/295))
 - **Ekaanksh Patil** ([@Ekaanksh-dev](https://github.com/Ekaanksh-dev))
   - Batch the PRM reward forward pass in `PRMScorer.__call__` (single `[B, T]` forward) ([#301](https://github.com/MakazhanAlpamys/Soup/pull/301))
+- **Sanjay Santhanam** ([@Sanjays2402](https://github.com/Sanjays2402))
+  - Run built-in benchmark gate tasks through `ForgettingDetector` — every `type: benchmark` eval-gate task had always failed ([#315](https://github.com/MakazhanAlpamys/Soup/pull/315))
 
 ---
 

{soup_cli-0.71.36 → soup_cli-0.71.38}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.36
+Version: 0.71.38
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -176,7 +176,7 @@ Description-Content-Type: text/markdown
 Soup turns the pain of LLM fine-tuning into a simple workflow. One config, one command, done.
 
 ```bash
-pip install 'soup-cli[train]'   # add [train] to fine-tune; bare `soup-cli` is the light CLI
+pip install "soup-cli[train]"   # add [train] to fine-tune; bare `soup-cli` is the light CLI
 soup init --template chat
 soup train
 ```
@@ -193,38 +193,45 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.36 — Data Moat II.** A semantic layer over your training data, plus two tools for what a fine-tune forgets and what it leaks.
-
-- **`soup data dedup --semantic`** — dedup by meaning, not by shared tokens. Catches
-  *reworded* duplicates MinHash's shingling scores as distinct. Zero new dependencies.
-- **`soup data topics <data>`** — cluster your data and get a coverage table
-  (`82% code · 6% math · 1% safety`) with a warning for thin topics, so you can see what
-  you are actually training on.
-- **`soup data canary insert|check`** — insert unique secrets, then prove whether a model
-  memorized them: each secret's loss is ranked against never-inserted controls. Exit 2 on a
-  leak, so CI can gate. On SmolLM2-135M a memorized set lands at percentile 0.0 against a
-  clean model's 4%–93% spread.
-- **`soup train --replay old.jsonl`** — continual-learning rehearsal: interleave your old
-  data so the new task doesn't erase it.
-- **Honest result:** semantic dedup is **not** a paraphrase detector. Measured, paraphrase
-  cosines (0.49–0.76) *overlap* with genuinely-distinct rows (0.54–0.76) — "Add two numbers"
-  vs "Multiply two numbers" scores **higher** than a real paraphrase — so no threshold
-  separates them. The default is deliberately conservative; see the CHANGELOG.
-- **Two blocking bugs fixed:** the hardware-fit gate **refused to train any model you merged
-  yourself** (a local checkpoint's name has no size marker, so it guessed 7B and predicted
-  16 GB), and every `pip install 'soup-cli[extra]'` hint printed **without the extra** —
-  17 sites where the suggested command succeeds and leaves the feature still broken.
+**v0.71.38 — The gate grows teeth.** `soup ship` answers the one question that matters after a fine-tune — *did my task improve WITHOUT silently breaking everything else?* This release makes the "without breaking everything else" half real.
+
+- **A gate that actually catches regressions.** The regression leg used to be 15 trivia
+  prompts scored by raw substring — it credited `"B"` for "**B**erlin", `"3"` for "1**3**", and
+  had **zero** items for tool-calling, safety, or JSON. Now it's a fixed, extraction-based
+  scorer over seven bundled, offline suites (MCQ · arithmetic · **tool-calling** · **JSON
+  validity** · **safety/refusal**). A tune that WINS your task but quietly breaks tool-calling
+  now gets a **DON'T SHIP** — the old gate waved it through.
+- **Zero new dependencies, no network.** Every suite ships in the wheel and is scored by the
+  pure scorers Soup already has — no lm-eval, no download. `soup ci init`'s core-only gate keeps
+  working.
+- **Distinct exit codes for CI.** Usage errors moved off exit `2` to `3`, so `2` now means only
+  DON'T-SHIP — a typo'd flag is no longer indistinguishable from a caught regression.
+- **Breaking, on purpose.** The fixed scorer can change an existing run's verdict — because the
+  old one was reporting false negatives. Recompute any committed `--baseline`.
 
 ```bash
-soup data topics train.jsonl                    # what am I actually training on?
-soup data dedup train.jsonl --semantic -o clean.jsonl
+# leg 1 (task win) AND leg 2 (no regression on the bundled suite) -> one verdict
+soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
+#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error
 
-soup data canary insert train.jsonl -o canaried.jsonl --manifest secrets.json
-soup data canary check --manifest secrets.json --base ./my-model  # exit 2 = leak
+soup ship --evidence scores.json    # offline verdict from pre-computed scores
+```
+
+<details>
+<summary>Previous release — v0.71.36, Data Moat II (semantic dedup · topics · canaries · replay)</summary>
 
+`soup data dedup --semantic` dedups by meaning; `soup data topics` shows a coverage table;
+`soup data canary insert|check` proves whether a model memorized inserted secrets (exit 2 on a
+leak); `soup train --replay old.jsonl` interleaves old data so a new task doesn't erase it.
+
+```bash
+soup data topics train.jsonl
+soup data canary check --manifest secrets.json --base ./my-model  # exit 2 = leak
 soup train --config sft.yaml --replay old_task.jsonl --replay-ratio 0.1
 ```
 
+</details>
+
 <details>
 <summary>Previous release — v0.71.33, <code>soup draft</code> (measure speculative decoding)</summary>
 
@@ -248,11 +255,38 @@ Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://gi
 ### 1. Install
 
 ```bash
-pip install soup-cli            # light: CLI + config + data tools (no PyTorch)
-pip install 'soup-cli[train]'   # add the training stack (torch, transformers, peft, trl, …)
-pip install git+https://github.com/MakazhanAlpamys/Soup.git   # latest dev
+# Light core: CLI + config + data tools, no PyTorch
+pip install soup-cli
+
+# Add the training stack (torch, transformers, peft, trl, datasets, …)
+pip install "soup-cli[train]"
+
+# Everything (train + serve + ui + data) in one shot
+pip install "soup-cli[all]"
+
+# Or from GitHub (latest dev)
+pip install git+https://github.com/MakazhanAlpamys/Soup.git
 ```
 
+The full extras table (`fast`, `mlx`, `serve`, `eval`, `ui`, `vision`, `audio`, …) lives in
+[`docs/models.md`](docs/models.md#optional-extras).
+
+> **Use double quotes around the extra.** They are the only spelling that works in
+> every shell — `cmd.exe`, PowerShell, bash, and zsh.
+>
+> Older tutorials and videos (including some of ours) show the single-quoted
+> `pip install 'soup-cli[train]'`. That is bash / zsh / PowerShell syntax, and it
+> fails on Windows `cmd.exe`, which has no single-quote quoting and hands the
+> quotes straight to pip:
+>
+> ```
+> ERROR: Invalid requirement: "'soup-cli[train]'": Expected package name at the start of dependency specifier
+> ```
+>
+> If you hit that, swap the `'` for `"` — pip is rejecting a literal quote
+> character, nothing is wrong with the package. (Dropping the quotes entirely
+> works on Windows too, but zsh then reads `[train]` as a glob and fails.)
+
 `soup init`, `soup data …`, and the other data/inspection commands work on the light install.
 Fine-tuning (`soup train`) needs the `[train]` extra.
 
@@ -287,7 +321,7 @@ A complete `soup.yaml`:
 ```yaml
 base: meta-llama/Llama-3.1-8B-Instruct
 task: sft
-# backend: unsloth  # 2-5x faster, pip install 'soup-cli[fast]'
+# backend: unsloth  # 2-5x faster, pip install "soup-cli[fast]"
 
 data:
   train: ./data/train.jsonl

{soup_cli-0.71.36 → soup_cli-0.71.38}/README.md

@@ -32,7 +32,7 @@
 Soup turns the pain of LLM fine-tuning into a simple workflow. One config, one command, done.
 
 ```bash
-pip install 'soup-cli[train]'   # add [train] to fine-tune; bare `soup-cli` is the light CLI
+pip install "soup-cli[train]"   # add [train] to fine-tune; bare `soup-cli` is the light CLI
 soup init --template chat
 soup train
 ```
@@ -49,38 +49,45 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.36 — Data Moat II.** A semantic layer over your training data, plus two tools for what a fine-tune forgets and what it leaks.
-
-- **`soup data dedup --semantic`** — dedup by meaning, not by shared tokens. Catches
-  *reworded* duplicates MinHash's shingling scores as distinct. Zero new dependencies.
-- **`soup data topics <data>`** — cluster your data and get a coverage table
-  (`82% code · 6% math · 1% safety`) with a warning for thin topics, so you can see what
-  you are actually training on.
-- **`soup data canary insert|check`** — insert unique secrets, then prove whether a model
-  memorized them: each secret's loss is ranked against never-inserted controls. Exit 2 on a
-  leak, so CI can gate. On SmolLM2-135M a memorized set lands at percentile 0.0 against a
-  clean model's 4%–93% spread.
-- **`soup train --replay old.jsonl`** — continual-learning rehearsal: interleave your old
-  data so the new task doesn't erase it.
-- **Honest result:** semantic dedup is **not** a paraphrase detector. Measured, paraphrase
-  cosines (0.49–0.76) *overlap* with genuinely-distinct rows (0.54–0.76) — "Add two numbers"
-  vs "Multiply two numbers" scores **higher** than a real paraphrase — so no threshold
-  separates them. The default is deliberately conservative; see the CHANGELOG.
-- **Two blocking bugs fixed:** the hardware-fit gate **refused to train any model you merged
-  yourself** (a local checkpoint's name has no size marker, so it guessed 7B and predicted
-  16 GB), and every `pip install 'soup-cli[extra]'` hint printed **without the extra** —
-  17 sites where the suggested command succeeds and leaves the feature still broken.
+**v0.71.38 — The gate grows teeth.** `soup ship` answers the one question that matters after a fine-tune — *did my task improve WITHOUT silently breaking everything else?* This release makes the "without breaking everything else" half real.
+
+- **A gate that actually catches regressions.** The regression leg used to be 15 trivia
+  prompts scored by raw substring — it credited `"B"` for "**B**erlin", `"3"` for "1**3**", and
+  had **zero** items for tool-calling, safety, or JSON. Now it's a fixed, extraction-based
+  scorer over seven bundled, offline suites (MCQ · arithmetic · **tool-calling** · **JSON
+  validity** · **safety/refusal**). A tune that WINS your task but quietly breaks tool-calling
+  now gets a **DON'T SHIP** — the old gate waved it through.
+- **Zero new dependencies, no network.** Every suite ships in the wheel and is scored by the
+  pure scorers Soup already has — no lm-eval, no download. `soup ci init`'s core-only gate keeps
+  working.
+- **Distinct exit codes for CI.** Usage errors moved off exit `2` to `3`, so `2` now means only
+  DON'T-SHIP — a typo'd flag is no longer indistinguishable from a caught regression.
+- **Breaking, on purpose.** The fixed scorer can change an existing run's verdict — because the
+  old one was reporting false negatives. Recompute any committed `--baseline`.
 
 ```bash
-soup data topics train.jsonl                    # what am I actually training on?
-soup data dedup train.jsonl --semantic -o clean.jsonl
+# leg 1 (task win) AND leg 2 (no regression on the bundled suite) -> one verdict
+soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
+#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error
 
-soup data canary insert train.jsonl -o canaried.jsonl --manifest secrets.json
-soup data canary check --manifest secrets.json --base ./my-model  # exit 2 = leak
+soup ship --evidence scores.json    # offline verdict from pre-computed scores
+```
+
+<details>
+<summary>Previous release — v0.71.36, Data Moat II (semantic dedup · topics · canaries · replay)</summary>
 
+`soup data dedup --semantic` dedups by meaning; `soup data topics` shows a coverage table;
+`soup data canary insert|check` proves whether a model memorized inserted secrets (exit 2 on a
+leak); `soup train --replay old.jsonl` interleaves old data so a new task doesn't erase it.
+
+```bash
+soup data topics train.jsonl
+soup data canary check --manifest secrets.json --base ./my-model  # exit 2 = leak
 soup train --config sft.yaml --replay old_task.jsonl --replay-ratio 0.1
 ```
 
+</details>
+
 <details>
 <summary>Previous release — v0.71.33, <code>soup draft</code> (measure speculative decoding)</summary>
 
@@ -104,11 +111,38 @@ Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://gi
 ### 1. Install
 
 ```bash
-pip install soup-cli            # light: CLI + config + data tools (no PyTorch)
-pip install 'soup-cli[train]'   # add the training stack (torch, transformers, peft, trl, …)
-pip install git+https://github.com/MakazhanAlpamys/Soup.git   # latest dev
+# Light core: CLI + config + data tools, no PyTorch
+pip install soup-cli
+
+# Add the training stack (torch, transformers, peft, trl, datasets, …)
+pip install "soup-cli[train]"
+
+# Everything (train + serve + ui + data) in one shot
+pip install "soup-cli[all]"
+
+# Or from GitHub (latest dev)
+pip install git+https://github.com/MakazhanAlpamys/Soup.git
 ```
 
+The full extras table (`fast`, `mlx`, `serve`, `eval`, `ui`, `vision`, `audio`, …) lives in
+[`docs/models.md`](docs/models.md#optional-extras).
+
+> **Use double quotes around the extra.** They are the only spelling that works in
+> every shell — `cmd.exe`, PowerShell, bash, and zsh.
+>
+> Older tutorials and videos (including some of ours) show the single-quoted
+> `pip install 'soup-cli[train]'`. That is bash / zsh / PowerShell syntax, and it
+> fails on Windows `cmd.exe`, which has no single-quote quoting and hands the
+> quotes straight to pip:
+>
+> ```
+> ERROR: Invalid requirement: "'soup-cli[train]'": Expected package name at the start of dependency specifier
+> ```
+>
+> If you hit that, swap the `'` for `"` — pip is rejecting a literal quote
+> character, nothing is wrong with the package. (Dropping the quotes entirely
+> works on Windows too, but zsh then reads `[train]` as a glob and fails.)
+
 `soup init`, `soup data …`, and the other data/inspection commands work on the light install.
 Fine-tuning (`soup train`) needs the `[train]` extra.
 
@@ -143,7 +177,7 @@ A complete `soup.yaml`:
 ```yaml
 base: meta-llama/Llama-3.1-8B-Instruct
 task: sft
-# backend: unsloth  # 2-5x faster, pip install 'soup-cli[fast]'
+# backend: unsloth  # 2-5x faster, pip install "soup-cli[fast]"
 
 data:
   train: ./data/train.jsonl

{soup_cli-0.71.36 → soup_cli-0.71.38}/docs/backends-and-ops.md

@@ -79,7 +79,7 @@ Fine-tune on M1-M4 Macs via Apple's [MLX](https://github.com/ml-explore/mlx) fra
 
 ```bash
 # Install MLX support
-pip install 'soup-cli[mlx]'
+pip install "soup-cli[mlx]"
 ```
 
 ```yaml
@@ -108,7 +108,7 @@ Use the [Unsloth](https://github.com/unslothai/unsloth) backend for significantl
 
 ```bash
 # Install unsloth support
-pip install 'soup-cli[fast]'
+pip install "soup-cli[fast]"
 ```
 
 Then add one line to your config:
@@ -143,7 +143,7 @@ app from your `soup.yaml` for serverless, per-second-billed GPU training. The co
 base64-embedded as **data** — no code interpolation, no secrets in the generated stub.
 
 ```bash
-pip install 'soup-cli[modal]'   # only needed for live submit
+pip install "soup-cli[modal]"   # only needed for live submit
 
 # Plan-only (default): write the stub + print the `modal run` command.
 soup train --config soup.yaml --cloud modal --gpu a100
@@ -585,7 +585,7 @@ Full-screen Textual dashboard. Two-pane: run list (left) + selected-run detail
 (right). `r` refreshes, `q` quits.
 
 ```bash
-pip install 'soup-cli[tui]'
+pip install "soup-cli[tui]"
 soup tui --refresh 1.0 --limit 50
 ```
 

{soup_cli-0.71.36 → soup_cli-0.71.38}/docs/commands.md

@@ -160,11 +160,12 @@ soup runs show <run_id>                       Run details + loss graph + cost
 soup runs compare <run_1> <run_2>             Compare two runs
 soup runs replay <run_id>                     Replay summary + loss curve from history (also plots a benchmark-score curve when the metric lives in eval_results)
 soup why [run_id]                             Explain training anomalies (heuristic)
-soup ship --base <m> --adapter <lora> --task-eval t.jsonl  SHIP / DON'T-SHIP verdict: task win AND no forgetting (exit 0=SHIP / 2=DON'T / 1=error) (v0.71.25)
+soup ship --base <m> --adapter <lora> --task-eval t.jsonl  SHIP / DON'T-SHIP verdict: task win AND no regression on the bundled suite (exit 0=SHIP / 2=DON'T / 3=usage / 1=runtime) (v0.71.25; leg-2 real + usage-off-2 v0.71.38)
 soup ship --evidence ev.json [--output v.json]  Decide offline from pre-computed scores (no model load)
 soup ship ... --task-mode judge_score --judge-model ollama://llama3.1  Leg-1 via LLM-as-a-judge
 soup ship ... --task-mode pairwise --judge-model ollama://llama3.1  Leg-1 via swap-debiased judge win-rate (base=0.5) (v0.71.31)
-soup ship ... --general-suite mmlu,gsm8k --baseline base.json  lm-eval leg-2 + recorded base scores
+soup ship ...  # leg-2 default = 7 bundled offline suites (MCQ/arithmetic + tool_call/format_json/safety, extraction scorer, ~40 items each) (v0.71.38)
+soup ship ... --general-suite mmlu,gsm8k --baseline base.json  lm-eval leg-2 override + recorded base scores
 soup card <registry-id> -o MODELCARD.md       HF model card from a registry entry: training config, evals, hashes, lineage, artifacts (v0.71.35)
 soup push --model ./out --repo you/m --card <registry-id>  Upload that registry-driven card as the README (HF only) (v0.71.35)
 soup ci init [--data d.jsonl --suite s.yaml --evidence ev.json] [--branch main --python 3.11] [--force]  Write .github/workflows/soup-gate.yml: data validate -> expect -> ship gate on every PR (v0.71.35)
@@ -255,9 +256,9 @@ soup adapters bisect <ckpt>... --eval-command "..."  Binary search over training
 soup lock write --base-sha <h> --dataset-sha <h> --env-hash <h>  Write soup.lock (v0.67.0)
 soup lock write --base-sha <h> --dataset-sha <h> --env-lock soup-env.lock  Auto-derive --env-hash from soup-env.lock (v0.71.1)
 soup lock show / soup lock check              Show + drift-check (exit 3 on drift)
-soup compile <program.py> --eval <suite> [--optimizer mipro|gepa|textgrad|copro|bootstrap_fewshot] [--plan-only]  DSPy / GEPA / TextGrad prompt-program compiler — live (v0.71.13; pip install 'soup-cli[compile]')
+soup compile <program.py> --eval <suite> [--optimizer mipro|gepa|textgrad|copro|bootstrap_fewshot] [--plan-only]  DSPy / GEPA / TextGrad prompt-program compiler — live (v0.71.13; pip install "soup-cli[compile]")
 soup distill-prompt --traces <jsonl> --teacher <m> --student <m> --strategy sft|preference|kl [--provider ollama|anthropic|vllm] [--base-url <url>] [--temperature F] [--max-rows N]  Distill prompt-heavy traces via a live teacher (v0.71.13)
-soup compile-tools <spec.json|yaml> --eval <jsonl> [--optimizer textgrad|gepa] [--plan-only]  TextGrad / GEPA tool-schema optimiser — live (v0.71.13; pip install 'soup-cli[compile]')
+soup compile-tools <spec.json|yaml> --eval <jsonl> [--optimizer textgrad|gepa] [--plan-only]  TextGrad / GEPA tool-schema optimiser — live (v0.71.13; pip install "soup-cli[compile]")
 soup apple-adapter <source-dir> --direction hf-to-mlx|mlx-to-hf|hf-to-apple|mlx-to-apple --output <dir> [--sign] [--plan-only]  PEFT LoRA <-> mlx-lm adapter conversion — live (v0.71.21; *-to-apple upstream-gated exit 3)
 soup local-rl init --db <path>                Create personal-LLM flywheel SQLite schema (v0.68.0)
 soup local-rl status --db <path>              Print interactions / thumbs-up / thumbs-down counters
@@ -299,7 +300,7 @@ server over **stdio**, so any MCP client — Claude Code, Cursor, Cline, Continu
 can drive Soup conversationally. Install the extra first:
 
 ```bash
-pip install 'soup-cli[mcp]'
+pip install "soup-cli[mcp]"
 ```
 
 Register it with your client. For **Claude Code** (`.mcp.json` in the repo) or

{soup_cli-0.71.36 → soup_cli-0.71.38}/docs/data.md

@@ -337,7 +337,7 @@ soup data generate --prompt "..." --validate
 # Auto-filter by quality (coherence scoring)
 soup data generate --prompt "..." --filter
 
-# Auto-dedup (MinHash, requires: pip install 'soup-cli[data]')
+# Auto-dedup (MinHash, requires: pip install "soup-cli[data]")
 soup data generate --prompt "..." --dedup
 
 # Full quality pipeline: validate + filter + dedup
@@ -591,7 +591,7 @@ soup data convert ./data/train.jsonl --to sharegpt --output converted.jsonl
 # Merge multiple datasets
 soup data merge data1.jsonl data2.jsonl --output merged.jsonl --shuffle
 
-# Remove near-duplicates (requires: pip install 'soup-cli[data]')
+# Remove near-duplicates (requires: pip install "soup-cli[data]")
 soup data dedup ./data/train.jsonl --threshold 0.8
 
 # Extended statistics (length distribution, token counts, languages)
@@ -814,7 +814,7 @@ Five checks: `length_bias` — the **#1 silent DPO degradation**: `chosen`
 systematically longer than `rejected`, reported as a Cohen's d effect size —
 `label_imbalance` (KTO desirable:undesirable ratio), `near_duplicates`
 (MinHash/LSH, reuses the `soup data dedup` kernel; requires
-`pip install 'soup-cli[data]'`, degrades to an advisory skip otherwise),
+`pip install "soup-cli[data]"`, degrades to an advisory skip otherwise),
 `identical_pairs` (`chosen == rejected` — zero preference signal), and
 `prompt_leak` (the prompt echoed verbatim inside the completion, a common
 synthetic-data pipeline bug). Same OK/MINOR/MAJOR taxonomy and exit codes as

{soup_cli-0.71.36 → soup_cli-0.71.38}/docs/evaluation.md

@@ -355,11 +355,28 @@ soup ship --evidence evidence.json --output verdict.json
 }
 ```
 
-Leg-2 defaults to the built-in mini benchmarks (`mini_mmlu` / `mini_common_sense` /
-`mini_instruction`) — offline, CPU, instant. `--general-suite <names>` with non-mini names
-routes through the lm-eval harness. Pairwise judge win-rate (`--task-mode pairwise`) is planned
-for a later release. The engine lives in `soup_cli.utils.ship_verdict` (`decide_ship` is a pure
-function — the whole truth table is CPU-testable).
+Leg-2 defaults to the **bundled offline suite** (v0.71.38) — seven hand-authored suites shipped
+in the wheel and scored by the pure scorers Soup already ships (no lm-eval, no network,
+CPU-instant):
+
+| Suite | What it checks | Scorer |
+|-------|----------------|--------|
+| `mini_mmlu` / `mini_common_sense` / `mini_instruction` / `mini_arithmetic` | general knowledge / reasoning / instruction-following / numeracy | answer-extraction + exact/boundary match |
+| `mini_tool_call` | function-calling still works (right tool named) | `tool_call_name_match` |
+| `mini_format_json` | JSON validity (a structured object, not a bare scalar) | container-only JSON check |
+| `mini_safety` | refusal-rate on harmful prompts (under-refusal = regression) | refusal heuristic |
+
+Each suite is >20 items so a single-item flip (1/N < 0.05) trips the default threshold instead
+of being rounded away. The scorer is answer-**extraction** — a spurious substring inside a word
+(`"B"` in "**B**erlin") no longer scores, which is a **breaking** change from the v0.25.0
+substring scorer (an existing run's verdict can flip; recompute any committed `--baseline`).
+`--general-suite <names>` with any non-bundled name routes through the lm-eval harness. Pairwise
+judge win-rate (`--task-mode pairwise`) shipped in v0.71.31.
+
+Exit codes (v0.71.38): **0 = SHIP · 2 = DON'T SHIP · 3 = usage/flag error · 1 = runtime error**
+— usage errors moved off `2` so CI can tell a config typo from a caught regression. The engine
+lives in `soup_cli.utils.ship_verdict` (`decide_ship` is a pure function — the whole truth table
+is CPU-testable); the bundled suites live in `soup_cli.eval.gate_suites`.
 
 
 ## NLG Evaluation Metrics (BLEU + ROUGE)
@@ -432,7 +449,7 @@ Full-featured evaluation platform with standard benchmarks, custom evals, LLM-as
 
 ```bash
 # Install eval dependencies
-pip install 'soup-cli[eval]'
+pip install "soup-cli[eval]"
 
 # Standard benchmarks (wraps lm-evaluation-harness)
 soup eval benchmark --model ./output --benchmarks mmlu,gsm8k,hellaswag

{soup_cli-0.71.36 → soup_cli-0.71.38}/docs/models.md

@@ -70,29 +70,29 @@ no PyTorch. Add `[train]` to fine-tune, or install other extras only when you ne
 
 | Extra | Install | What it adds |
 |---|---|---|
-| `train` | `pip install 'soup-cli[train]'` | Training stack: torch, transformers, peft, trl, datasets, bitsandbytes, accelerate |
-| `all` | `pip install 'soup-cli[all]'` | `train` + `serve` + `ui` + `data` in one shot |
-| `fast` | `pip install 'soup-cli[fast]'` | Unsloth backend (2-5x faster, lower VRAM) |
-| `vision` | `pip install 'soup-cli[vision]'` | Vision / multimodal fine-tuning (Pillow) |
-| `audio` | `pip install 'soup-cli[audio]'` | Audio / speech fine-tuning (librosa, soundfile) |
-| `mlx` | `pip install 'soup-cli[mlx]'` | Apple Silicon backend (mlx, mlx-lm) |
-| `qat` | `pip install 'soup-cli[qat]'` | Quantization-Aware Training (torchao) |
-| `serve` | `pip install 'soup-cli[serve]'` | Inference server (FastAPI + uvicorn) |
-| `serve-fast` | `pip install 'soup-cli[serve-fast]'` | vLLM inference backend (2-4x throughput) |
-| `sglang` | `pip install 'soup-cli[sglang]'` | SGLang inference backend |
-| `ui` | `pip install 'soup-cli[ui]'` | Web UI + inference server |
-| `tui` | `pip install 'soup-cli[tui]'` | Full-screen Textual dashboard (`soup tui`) |
-| `eval` | `pip install 'soup-cli[eval]'` | Benchmark evaluation (lm-evaluation-harness) |
-| `data` | `pip install 'soup-cli[data]'` | Deduplication (MinHash via datasketch) |
-| `data-pro` | `pip install 'soup-cli[data-pro]'` | Language detection + PII (langdetect, presidio) |
-| `deepspeed` | `pip install 'soup-cli[deepspeed]'` | Multi-GPU training (DeepSpeed ZeRO) |
-| `liger` | `pip install 'soup-cli[liger]'` | Liger Kernel fused ops |
-| `ring-attn` | `pip install 'soup-cli[ring-attn]'` | Ring FlashAttention (sequence parallelism) |
-| `onnx` / `tensorrt` | `pip install 'soup-cli[onnx]'` | ONNX / TensorRT-LLM export |
-| `awq` / `gptq` | `pip install 'soup-cli[awq]'` | AWQ / GPTQ quantized export |
-| `trackers` | `pip install 'soup-cli[trackers]'` | MLflow / SwanLab / Trackio logging |
-| `remote` | `pip install 'soup-cli[remote]'` | Remote datasets (s3 / gs / az / oci) |
-| `dev` | `pip install 'soup-cli[dev]'` | Tests + lint + types (pytest, ruff, mypy, pre-commit) |
+| `train` | `pip install "soup-cli[train]"` | Training stack: torch, transformers, peft, trl, datasets, bitsandbytes, accelerate |
+| `all` | `pip install "soup-cli[all]"` | `train` + `serve` + `ui` + `data` in one shot |
+| `fast` | `pip install "soup-cli[fast]"` | Unsloth backend (2-5x faster, lower VRAM) |
+| `vision` | `pip install "soup-cli[vision]"` | Vision / multimodal fine-tuning (Pillow) |
+| `audio` | `pip install "soup-cli[audio]"` | Audio / speech fine-tuning (librosa, soundfile) |
+| `mlx` | `pip install "soup-cli[mlx]"` | Apple Silicon backend (mlx, mlx-lm) |
+| `qat` | `pip install "soup-cli[qat]"` | Quantization-Aware Training (torchao) |
+| `serve` | `pip install "soup-cli[serve]"` | Inference server (FastAPI + uvicorn) |
+| `serve-fast` | `pip install "soup-cli[serve-fast]"` | vLLM inference backend (2-4x throughput) |
+| `sglang` | `pip install "soup-cli[sglang]"` | SGLang inference backend |
+| `ui` | `pip install "soup-cli[ui]"` | Web UI + inference server |
+| `tui` | `pip install "soup-cli[tui]"` | Full-screen Textual dashboard (`soup tui`) |
+| `eval` | `pip install "soup-cli[eval]"` | Benchmark evaluation (lm-evaluation-harness) |
+| `data` | `pip install "soup-cli[data]"` | Deduplication (MinHash via datasketch) |
+| `data-pro` | `pip install "soup-cli[data-pro]"` | Language detection + PII (langdetect, presidio) |
+| `deepspeed` | `pip install "soup-cli[deepspeed]"` | Multi-GPU training (DeepSpeed ZeRO) |
+| `liger` | `pip install "soup-cli[liger]"` | Liger Kernel fused ops |
+| `ring-attn` | `pip install "soup-cli[ring-attn]"` | Ring FlashAttention (sequence parallelism) |
+| `onnx` / `tensorrt` | `pip install "soup-cli[onnx]"` | ONNX / TensorRT-LLM export |
+| `awq` / `gptq` | `pip install "soup-cli[awq]"` | AWQ / GPTQ quantized export |
+| `trackers` | `pip install "soup-cli[trackers]"` | MLflow / SwanLab / Trackio logging |
+| `remote` | `pip install "soup-cli[remote]"` | Remote datasets (s3 / gs / az / oci) |
+| `dev` | `pip install "soup-cli[dev]"` | Tests + lint + types (pytest, ruff, mypy, pre-commit) |
 
 The complete, authoritative extras list is in [`pyproject.toml`](../pyproject.toml).
 

{soup_cli-0.71.36 → soup_cli-0.71.38}/docs/performance-and-quantization.md

@@ -36,7 +36,7 @@ Train with simulated quantization for significantly better post-quantization qua
 
 ```bash
 # Install QAT support
-pip install 'soup-cli[qat]'
+pip install "soup-cli[qat]"
 ```
 
 ```yaml
@@ -71,7 +71,7 @@ QAT works with all training tasks (SFT, DPO, GRPO, PPO, KTO, ORPO, SimPO, IPO, P
 For H100 / H200 / B100 / B200 GPUs, train with float8 matmuls for ~2x speedup vs bf16 at comparable quality. This extends QAT infrastructure via `torchao.float8`:
 
 ```bash
-pip install 'soup-cli[qat]'   # torchao >= 0.5.0 includes torchao.float8
+pip install "soup-cli[qat]"   # torchao >= 0.5.0 includes torchao.float8
 ```
 
 ```yaml
@@ -106,7 +106,7 @@ Bool `true` stays on the int8 QAT path for backward compatibility. FP8 requires
 Models with 128k+ vocabularies (Llama 3.1, Qwen2) materialise a huge `(batch, seq, vocab)` logits tensor that dominates VRAM. Cut Cross-Entropy computes the loss in chunks instead:
 
 ```bash
-pip install 'soup-cli[cce]'    # or: pip install cut-cross-entropy
+pip install "soup-cli[cce]"    # or: pip install cut-cross-entropy
 ```
 
 ```yaml
@@ -368,9 +368,9 @@ data:
 Install optional performance packages:
 
 ```bash
-pip install 'soup-cli[liger]'     # Liger Kernel fused operations
+pip install "soup-cli[liger]"     # Liger Kernel fused operations
 pip install flash-attn --no-build-isolation  # FlashAttention
-pip install 'soup-cli[ring-attn]' # Ring FlashAttention (sequence parallelism)
+pip install "soup-cli[ring-attn]" # Ring FlashAttention (sequence parallelism)
 ```