soup-cli 0.71.26__tar.gz → 0.71.27__tar.gz

{soup_cli-0.71.26 → soup_cli-0.71.27}/CHANGELOG.md

@@ -12,6 +12,82 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.27] - 2026-07-03
+
+### Added
+- **Fine-tune Doctor** — kill the top *silent* fine-tune failures before a
+  single training step; no competitor (Unsloth/Axolotl/LlamaFactory) ships
+  any of these three:
+  - `soup data doctor <data> --model <id|path>` — chat-template
+    compatibility report over 8 checks: `chat_template` present,
+    `template_render`s cleanly, has `{% generation %}` markers,
+    `eos_in_labels` (the **#1 "model never stops generating" bug** — every
+    assistant turn's trained span must actually contain an EOS/EOT token;
+    checks every turn, not just the last), `bos_duplication` (template +
+    tokenizer both prepending BOS), `system_role` support (Mistral-style
+    templates reject a leading system turn), `unknown_roles`, and
+    `truncation_risk` (p95 rendered length vs `max_length`). Same OK / MINOR
+    / MAJOR taxonomy as `soup diagnose`; exit 0 = OK/MINOR, exit 2 = MAJOR.
+    `--train-on-responses-only` / `--train-on-messages-with-train-field`
+    select the same masking strategy `soup train` would use, so the report
+    and `--show-mask` never disagree about what's actually trained.
+  - `soup data doctor ... --show-mask N` — render N sample rows with
+    per-token trained/masked colouring through the REAL collator path
+    (answer-only / per-message-train-field / RAFT span-mask) — not a
+    reimplementation — so an assistant-mask bug is visible instantly.
+  - `soup data lint <data>` — preference-data linter for
+    dpo/orpo/simpo/ipo/bco/kto: `length_bias` (chosen systematically longer
+    than rejected — the **#1 silent DPO degradation**, reported as a
+    Cohen's d effect size), `label_imbalance` (KTO desirable:undesirable
+    ratio), `near_duplicates` (MinHash/LSH, reuses the `soup data dedup`
+    kernel), `identical_pairs` (chosen == rejected — zero preference
+    signal), and `prompt_leak` (the prompt echoed verbatim inside the
+    completion — a common synthetic-data pipeline bug). Optional `--model`
+    for exact token-length bias (default: word count).
+  - Validated live against the real `HuggingFaceTB/SmolLM2-135M-Instruct`
+    tokenizer on Windows + RTX 3050 — this smoke pass found and fixed two
+    genuine bugs beyond what synthetic fixtures alone caught: an EOS check
+    that required the EOS token to be the *literal last* trained token
+    (real templates often have a trailing formatting token after the
+    closing tag that stays inside the trained span), and two call sites
+    that only caught `(ValueError, TypeError)` around a tokenizer's
+    `apply_chat_template` when a real Jinja `raise_exception()`
+    (Mistral-style no-system-role guard) raises
+    `jinja2.exceptions.TemplateError`.
+
+### Fixed
+- Harden `commands/diagnose.py`'s `--evidence` loader against a TOCTOU
+  symlink swap: opens with `O_NOFOLLOW` and size-checks the open fd via
+  `os.fstat` instead of `os.path.getsize` on the path before the open —
+  backports the hardened loader shipped for `soup ship` in v0.71.25 (closes
+  v0.71.25 known-limitation (4)).
+- Harden judge-model URL validation against a hostname prefix bypass
+  (`http://localhost.attacker.com`) — `GateTask._valid_judge_url` /
+  `_parse_judge_url` now use `urllib.parse.urlparse` + hostname checks instead
+  of `startswith`. Closes #283
+  ([#288](https://github.com/MakazhanAlpamys/Soup/pull/288) by [@CODING-DARSH](https://github.com/CODING-DARSH)).
+- `SFTTrainerWrapper` now applies configured vocabulary expansion
+  (`data.add_new_tokens` / `data.new_special_tokens`) and resizes the model
+  embeddings during initialization — previously these fields were accepted by
+  the schema but silently ignored. Closes #289
+  ([#287](https://github.com/MakazhanAlpamys/Soup/pull/287) by [@CODING-DARSH](https://github.com/CODING-DARSH)).
+- Vision and audio SFT paths now apply that same configured vocabulary
+  expansion (`data.add_new_tokens` / `data.new_special_tokens`) via the shared
+  `apply_vocab_expansion()` helper, consistent with the text SFT path — they
+  previously ignored it. Closes #290
+  ([#291](https://github.com/MakazhanAlpamys/Soup/pull/291) by [@CODING-DARSH](https://github.com/CODING-DARSH)).
+
+### Security
+- `soup data doctor` strips C0 control characters (keeping tab/newline/CR)
+  from dataset-derived content before it reaches the terminal — Rich's
+  `markup.escape()` only neutralises `[...]` tag syntax, not raw escape
+  sequences, so an untrusted training row (e.g. an unknown `role` field, or
+  `--show-mask`'s decoded token text on a byte-level BPE tokenizer) could
+  otherwise carry a literal ESC byte through to the terminal (title-bar /
+  OSC-8 link spoofing, or obscuring a MAJOR verdict via cursor tricks).
+  `--output` JSON is unaffected (`json.dumps` already escapes control
+  characters).
+
 ## [0.71.26] - 2026-07-01
 
 ### Added

{soup_cli-0.71.26 → soup_cli-0.71.27}/CONTRIBUTING.md

@@ -116,11 +116,11 @@ src/soup_cli/
   cans/               - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0)
   data/traces/        - Trace-to-Preference harvester (v0.26.0)
   data/collators.py   - CrossDocCollator for sample packing (v0.33.0)
-  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict
+  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint
   templates/          - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (298 files, 14788 tests)
+tests/                - Test suite (304 files, 15042 tests)
 examples/             - Real-world config examples and datasets
 ```
 
@@ -150,7 +150,7 @@ pytest tests/test_data.py::test_detect_alpaca_format -v
 pytest tests/ --cov=soup_cli --cov-report=html
 ```
 
-### Test Files (297 files)
+### Test Files (304 files)
 
 > A representative sample of the suite below. The full table lives in
 > [`.claude/CLAUDE.md`](.claude/CLAUDE.md); run `pytest tests/ -v` for the complete list.

{soup_cli-0.71.26 → soup_cli-0.71.27}/CONTRIBUTORS.md

@@ -44,6 +44,10 @@ Listed by first contribution. PR numbers link the work.
   - Unit tests for the `warmup.py` auto-warmup-steps helper ([#274](https://github.com/MakazhanAlpamys/Soup/pull/274))
 - **Kondamwar Akshaya Shrikant** ([@Akshaya-reddy18](https://github.com/Akshaya-reddy18))
   - Friendlier error messages — richer CUDA-OOM hint + Hugging Face gated-repo and `trust_remote_code` mappings + tests ([#282](https://github.com/MakazhanAlpamys/Soup/pull/282))
+- **Darsh** ([@CODING-DARSH](https://github.com/CODING-DARSH))
+  - Harden judge-URL validation against hostname prefix bypass (`startswith` → `urlparse`) in `eval/gate.py` ([#288](https://github.com/MakazhanAlpamys/Soup/pull/288))
+  - Apply configured vocabulary expansion (`data.add_new_tokens` / `new_special_tokens`) during SFT trainer init ([#287](https://github.com/MakazhanAlpamys/Soup/pull/287))
+  - Reuse the shared vocab-expansion helper in the vision + audio SFT paths ([#291](https://github.com/MakazhanAlpamys/Soup/pull/291))
 
 ---
 

{soup_cli-0.71.26 → soup_cli-0.71.27}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.26
+Version: 0.71.27
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -189,21 +189,25 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.26 — Closed-loop reward-hacking auto-mitigation.** Your GRPO/PPO trainer now *detects* reward hacking mid-run and *self-corrects* — instead of only halting. No OSS RLHF library closes this loop.
-
-- **Detect → correct → continue.** When the hacking signal trips, the controller raises the
-  KL coefficient β (bang-bang + hysteresis, or a PID-Lagrangian controller), and — if hacking
-  persists — **rolls back to the last-good checkpoint**, then early-stops as a last resort.
-- **Anti-gaming hardening.** Multi-signal voting (RM cluster-separation + length-trend +
-  repetition), signal smoothing, conservative-on-disagreement, a reward-distribution-drift
-  guard, and optional bounded **reward shaping** on the gamed proxy.
-- **Observe first.** `reward_hack_mitigation=log_only` streams a per-step `mitigation_log.jsonl`
-  and never touches training — see the hacking happen before you let the controller act.
-- **Honest scope.** Proof-of-mechanism on SmolLM2-135M + a synthetic hacking task on one RTX
-  3050 (all four stages validated live, including a real rollback). PPO ships BETA.
+**v0.71.27 — Fine-tune Doctor: catch silent failures before they burn a training run.** `soup data doctor` and `soup data lint` — no competitor (Unsloth/Axolotl/LlamaFactory) ships either.
+
+- **The #1 "model never stops generating" bug, caught pre-flight.** `soup data doctor` checks
+  whether every trained turn actually contains an EOS token — plus BOS duplication,
+  no-system-role templates, unknown roles, and truncation risk — before you burn GPU hours on
+  a broken chat template.
+- **See exactly what's trained, token by token.** `--show-mask N` renders sample rows through
+  the REAL collator path (answer-only / RAFT / packing-aware), so an assistant-mask bug is
+  visible instantly instead of silently degrading the whole run.
+- **Stop silently-worse DPO runs.** `soup data lint` flags length bias (the #1 silent
+  preference-tuning degradation, reported as an effect size), near-duplicate pairs,
+  chosen==rejected rows, and prompt leakage — across dpo/orpo/simpo/ipo/bco/kto data.
+- **Found real bugs on a real tokenizer.** Live-validated against
+  `HuggingFaceTB/SmolLM2-135M-Instruct` on Windows + RTX 3050 — the smoke pass itself caught
+  two genuine template-handling bugs before release.
 
 ```bash
-soup train --config grpo.yaml --reward-hack-mitigation kl_control   # detect → raise KL → recover
+soup data doctor train.jsonl --model meta-llama/Llama-3.1-8B-Instruct --show-mask 3
+soup data lint prefs.jsonl --model meta-llama/Llama-3.1-8B-Instruct
 ```
 
 Full history: [CHANGELOG.md](CHANGELOG.md) · [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).

{soup_cli-0.71.26 → soup_cli-0.71.27}/README.md

@@ -49,21 +49,25 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.26 — Closed-loop reward-hacking auto-mitigation.** Your GRPO/PPO trainer now *detects* reward hacking mid-run and *self-corrects* — instead of only halting. No OSS RLHF library closes this loop.
-
-- **Detect → correct → continue.** When the hacking signal trips, the controller raises the
-  KL coefficient β (bang-bang + hysteresis, or a PID-Lagrangian controller), and — if hacking
-  persists — **rolls back to the last-good checkpoint**, then early-stops as a last resort.
-- **Anti-gaming hardening.** Multi-signal voting (RM cluster-separation + length-trend +
-  repetition), signal smoothing, conservative-on-disagreement, a reward-distribution-drift
-  guard, and optional bounded **reward shaping** on the gamed proxy.
-- **Observe first.** `reward_hack_mitigation=log_only` streams a per-step `mitigation_log.jsonl`
-  and never touches training — see the hacking happen before you let the controller act.
-- **Honest scope.** Proof-of-mechanism on SmolLM2-135M + a synthetic hacking task on one RTX
-  3050 (all four stages validated live, including a real rollback). PPO ships BETA.
+**v0.71.27 — Fine-tune Doctor: catch silent failures before they burn a training run.** `soup data doctor` and `soup data lint` — no competitor (Unsloth/Axolotl/LlamaFactory) ships either.
+
+- **The #1 "model never stops generating" bug, caught pre-flight.** `soup data doctor` checks
+  whether every trained turn actually contains an EOS token — plus BOS duplication,
+  no-system-role templates, unknown roles, and truncation risk — before you burn GPU hours on
+  a broken chat template.
+- **See exactly what's trained, token by token.** `--show-mask N` renders sample rows through
+  the REAL collator path (answer-only / RAFT / packing-aware), so an assistant-mask bug is
+  visible instantly instead of silently degrading the whole run.
+- **Stop silently-worse DPO runs.** `soup data lint` flags length bias (the #1 silent
+  preference-tuning degradation, reported as an effect size), near-duplicate pairs,
+  chosen==rejected rows, and prompt leakage — across dpo/orpo/simpo/ipo/bco/kto data.
+- **Found real bugs on a real tokenizer.** Live-validated against
+  `HuggingFaceTB/SmolLM2-135M-Instruct` on Windows + RTX 3050 — the smoke pass itself caught
+  two genuine template-handling bugs before release.
 
 ```bash
-soup train --config grpo.yaml --reward-hack-mitigation kl_control   # detect → raise KL → recover
+soup data doctor train.jsonl --model meta-llama/Llama-3.1-8B-Instruct --show-mask 3
+soup data lint prefs.jsonl --model meta-llama/Llama-3.1-8B-Instruct
 ```
 
 Full history: [CHANGELOG.md](CHANGELOG.md) · [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).

{soup_cli-0.71.26 → soup_cli-0.71.27}/docs/commands.md

@@ -68,6 +68,9 @@ soup sweep --config soup.yaml --param lr=...  Hyperparameter search
 soup diff --model-a ./a --model-b ./b         Compare two models
 soup data inspect <path>                      View dataset stats
 soup data validate <path>                     Check format (auto-detect)
+soup data doctor <path> --model <id>          Chat-template compat report: 8 checks, OK/MINOR/MAJOR
+soup data doctor <path> --model <id> --show-mask N  Per-token trained/masked colouring via the real collator
+soup data lint <path>                         Preference-data linter: length bias, near-dups, chosen==rejected
 soup data convert <path> --to chatml          Convert between formats
 soup data merge data1.jsonl data2.jsonl       Combine datasets
 soup data dedup <path> --threshold 0.8        Remove duplicates (MinHash)
@@ -81,7 +84,8 @@ soup data generate ... --quality-pipeline     Auto validate + filter + dedup
 soup data augment <path> --strategy rephrase|translate|style [--provider ollama|vllm --model <m> --base-url <url>]  LLM-driven augmentation
 soup data from-traces --logs l.jsonl --format langchain --signal thumbs_up --output p.jsonl  Preference pairs from traces
 soup data from-traces ... --judge --min-confidence 0.7  LLM-judge confidence filter
-soup data review prefs.jsonl --sample 10      Preview preference pairssoup data filter <path> --coherence 0.3       Quality filter (perplexity/coherence)
+soup data review prefs.jsonl --sample 10      Preview preference pairs
+soup data filter <path> --coherence 0.3       Quality filter (perplexity/coherence)
 soup data sample <path> --n 1000             Random sample subset
 soup data sample <path> --n 1000 --strategy diverse  Cluster-based diverse sampling
 soup data sample <path> --n 1000 --strategy hard     Sample hardest examples

{soup_cli-0.71.26 → soup_cli-0.71.27}/docs/data.md

@@ -637,3 +637,49 @@ written per node; resume rehydrates from per-node sidecars. Failed rows logged w
 redacted reasons (paths stripped, capped at 256 chars).
 
 
+## Fine-tune Doctor (`soup data doctor`)
+
+Chat-template compatibility report — catches the top *silent* fine-tuning failures
+before a single training step:
+
+```bash
+soup data doctor ./data/train.jsonl --model meta-llama/Llama-3.1-8B-Instruct
+
+# Render N sample rows with per-token trained/masked colouring, through the REAL
+# collator path (answer-only / per-message-train-field / RAFT span-mask)
+soup data doctor ./data/train.jsonl --model meta-llama/Llama-3.1-8B-Instruct --show-mask 5
+```
+
+Eight checks, same OK/MINOR/MAJOR taxonomy as `soup diagnose` (exit 0 on OK/MINOR,
+exit 2 on MAJOR): `chat_template` (tokenizer has one), `template_render` (renders
+cleanly on a sample), `generation_markers` (`{% generation %}` support),
+`eos_in_labels` — the **#1 "model never stops generating" bug**: every trained
+assistant turn must actually contain an EOS/EOT token, checked across the *whole*
+trained span, not just the last turn — `bos_duplication` (template + tokenizer both
+prepending BOS), `system_role` (Mistral-style templates that reject a leading system
+turn), `unknown_roles`, and `truncation_risk` (p95 rendered length vs
+`data.max_length`). `--train-on-responses-only` / `--train-on-messages-with-train-field`
+select the same masking strategy `soup train` would use, so the report and
+`--show-mask` preview can never disagree about what's actually trained.
+
+
+## Preference-Data Linter (`soup data lint`)
+
+Catches the top silent degradations in DPO/ORPO/SimPO/IPO/BCO/KTO preference data:
+
+```bash
+soup data lint ./data/prefs.jsonl
+soup data lint ./data/prefs.jsonl --model meta-llama/Llama-3.1-8B-Instruct  # exact token-length bias, not word count
+```
+
+Five checks: `length_bias` — the **#1 silent DPO degradation**: `chosen`
+systematically longer than `rejected`, reported as a Cohen's d effect size —
+`label_imbalance` (KTO desirable:undesirable ratio), `near_duplicates`
+(MinHash/LSH, reuses the `soup data dedup` kernel; requires
+`pip install 'soup-cli[data]'`, degrades to an advisory skip otherwise),
+`identical_pairs` (`chosen == rejected` — zero preference signal), and
+`prompt_leak` (the prompt echoed verbatim inside the completion, a common
+synthetic-data pipeline bug). Same OK/MINOR/MAJOR taxonomy and exit codes as
+`soup data doctor`.
+
+

{soup_cli-0.71.26 → soup_cli-0.71.27}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.26"
+version = "0.71.27"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

soup_cli-0.71.27/src/soup_cli/__init__.py

@@ -0,0 +1,3 @@
+"""Soup CLI — Fine-tune and post-train LLMs in one command."""
+
+__version__ = "0.71.27"

{soup_cli-0.71.26 → soup_cli-0.71.27}/src/soup_cli/cans/schema.py

@@ -82,6 +82,10 @@ class DeployTarget(BaseModel):
             raise ValueError("deploy path contains null byte")
         if value.startswith("/") or value.startswith("\\"):
             raise ValueError(f"deploy path '{value}' must be relative")
+        # Windows drive-absolute (``C:\...`` / ``C:/...``) is absolute too but
+        # starts with a letter, so it slipped past the ``/`` / ``\`` check.
+        if len(value) >= 2 and value[1] == ":" and value[0].isalpha():
+            raise ValueError(f"deploy path '{value}' must be relative")
         # Normalise separators first, then split — a mixed-separator path
         # like ``foo/..\\bar`` would otherwise slip past a single-separator
         # split because ``"..\\bar"`` != ``".."``.

{soup_cli-0.71.26 → soup_cli-0.71.27}/src/soup_cli/cli.py

@@ -510,6 +510,13 @@ app.add_typer(
     help="RA-DIT two-stage orchestrator: retriever -> generator (v0.71.10).",
 )
 
+# v0.71.27 — Fine-tune Doctor: chat-template doctor + loss-mask X-ray +
+# preference linter.
+from soup_cli.commands import data_doctor as _data_doctor_cmd  # noqa: E402
+
+data.app.command(name="doctor")(_data_doctor_cmd.doctor)
+data.app.command(name="lint")(_data_doctor_cmd.lint)
+
 
 def _rewrite_advise_argv(argv: list) -> list:
     """Inject `run` between `advise` and a non-subcommand first argument.

{soup_cli-0.71.26 → soup_cli-0.71.27}/src/soup_cli/cloud/modal.py

@@ -196,7 +196,12 @@ def render_modal_stub(
         "@app.local_entrypoint()\n"
         "def main() -> None:\n"
         "    train.remote()\n"
-        f'    print("Training submitted; download checkpoints to {output_dir}")\n'
+        # Reference the already-repr()-embedded ``_LOCAL_OUTPUT`` via a runtime
+        # f-string in the GENERATED code — no user value is interpolated into
+        # the stub source here. (Interpolating raw ``{output_dir}`` was a code-
+        # injection hole; ``{output_dir!r}`` alone would still break for a path
+        # containing a quote because the repr is nested inside a "..." literal.)
+        '    print(f"Training submitted; download checkpoints to {_LOCAL_OUTPUT}")\n'
     )
 
 

{soup_cli-0.71.26 → soup_cli-0.71.27}/src/soup_cli/commands/_eval_v0550.py

@@ -43,7 +43,7 @@ def register(app: typer.Typer, console: Console) -> None:
         try:
             rows = load_advise_dataset(data)
         except (FileNotFoundError, ValueError, TypeError) as exc:
-            console.print(f"[red]Cannot read dataset:[/] {exc}")
+            console.print(f"[red]Cannot read dataset:[/] {escape(str(exc))}")
             raise typer.Exit(1) from exc
 
         try:
@@ -105,7 +105,7 @@ def register(app: typer.Typer, console: Console) -> None:
         try:
             rows = load_advise_dataset(data)
         except (FileNotFoundError, ValueError, TypeError) as exc:
-            console.print(f"[red]Cannot read dataset:[/] {exc}")
+            console.print(f"[red]Cannot read dataset:[/] {escape(str(exc))}")
             raise typer.Exit(1) from exc
 
         try:
@@ -254,6 +254,15 @@ def register(app: typer.Typer, console: Console) -> None:
             False, "--json-only",
             help="Suppress Rich output; emit a single JSON verdict line.",
         ),
+        suite: str = typer.Option(
+            None, "--suite",
+            help=(
+                "Locked eval suite (from `soup eval lock`) to validate as a gate "
+                "precondition — a missing / unparseable suite BLOCKS the check "
+                "(exit 1). cwd-contained. The generated pre-push hook passes "
+                "$GATE_SUITE here so the locked suite is actually enforced."
+            ),
+        ),
     ) -> None:
         """Run-vs-run regression check (paired-bootstrap CI).
 
@@ -265,6 +274,26 @@ def register(app: typer.Typer, console: Console) -> None:
         """
         import json as _json
 
+        # The locked suite is a hard precondition when supplied: validate it
+        # exists, is under cwd, and parses. Previously the hook wrote $GATE_SUITE
+        # but never used it, so a deleted / tampered locked suite silently
+        # passed the gate.
+        if suite:
+            from soup_cli.utils.eval_lock_coverage import load_locked_suite
+            from soup_cli.utils.paths import is_under_cwd
+
+            if not is_under_cwd(suite):
+                console.print(f"[red]--suite is outside cwd:[/] {escape(str(suite))}")
+                raise typer.Exit(1)
+            try:
+                load_locked_suite(suite)
+            except (FileNotFoundError, ValueError, OSError) as exc:
+                console.print(
+                    f"[red]Locked eval suite invalid — gate blocked:[/] "
+                    f"{escape(str(exc))}"
+                )
+                raise typer.Exit(1) from exc
+
         from soup_cli.experiment.tracker import ExperimentTracker
         from soup_cli.utils.eval_gate_hook import (
             GateThresholds,

{soup_cli-0.71.26 → soup_cli-0.71.27}/src/soup_cli/commands/bench.py

@@ -194,6 +194,11 @@ def bench(
     start_load = time.time()
     try:
         model_obj, tokenizer = _load_model(str(model_path), base, device)
+    except typer.Exit:
+        # typer.Exit subclasses RuntimeError, so the broad except below would
+        # swallow an already-reported CLI exit and mis-print "Failed to load
+        # model: 1". Let it propagate unchanged.
+        raise
     except (OSError, ImportError, RuntimeError, ValueError) as exc:
         console.print(f"[red]Failed to load model:[/] {exc}")
         raise typer.Exit(1) from exc

{soup_cli-0.71.26 → soup_cli-0.71.27}/src/soup_cli/commands/data.py

@@ -754,11 +754,12 @@ def sample_data(
     if output is None:
         out_path = file_path.parent / f"{file_path.stem}_sampled_{strategy}.jsonl"
     else:
+        from soup_cli.utils.paths import is_under_cwd
+
         out_path = Path(output).resolve()
-        cwd = Path.cwd().resolve()
-        try:
-            out_path.relative_to(cwd)
-        except ValueError:
+        # realpath + commonpath (is_under_cwd) — Path.resolve()+relative_to()
+        # breaks on Windows 8.3 short names.
+        if not is_under_cwd(output):
             console.print("[red]Output path must be under the current working directory.[/]")
             raise typer.Exit(1)
 
@@ -812,6 +813,13 @@ def split_data(
         console.print("[red]Specify at least one of --val or --test.[/]")
         raise typer.Exit(1)
 
+    # Reject negatives: a negative val/test slipped past the `>= total` check
+    # and produced a negative slice (e.g. --val -10 sent 90 rows to val, 10 to
+    # train — a silently inverted split).
+    if (val is not None and val < 0) or (test is not None and test < 0):
+        console.print("[red]--val and --test must be non-negative.[/]")
+        raise typer.Exit(1)
+
     data = load_raw_data(file_path)
     if not data:
         console.print("[red]Dataset is empty.[/]")
@@ -1265,25 +1273,23 @@ def download_dataset(
         raise typer.Exit(1)
 
     # Resolve output path
+    from soup_cli.utils.paths import is_under_cwd
+
     if output is None:
         ds_name = dataset_id.split("/")[-1] if "/" in dataset_id else dataset_id
         # Strip embedded path separators to prevent traversal
         ds_name = Path(ds_name).name
         out_path = (Path.cwd() / f"{ds_name}.jsonl").resolve()
-        cwd = Path.cwd().resolve()
-        try:
-            out_path.relative_to(cwd)
-        except ValueError:
+        # realpath + commonpath (is_under_cwd) — Path.resolve()+relative_to()
+        # breaks on Windows 8.3 short names.
+        if not is_under_cwd(out_path):
             console.print(
                 "[red]Derived output path escapes working directory.[/]"
             )
             raise typer.Exit(1)
     else:
         out_path = Path(output).resolve()
-        cwd = Path.cwd().resolve()
-        try:
-            out_path.relative_to(cwd)
-        except ValueError:
+        if not is_under_cwd(output):
             console.print(
                 "[red]Output path must be under the current working directory.[/]"
             )