soup-cli 0.71.26__tar.gz → 0.71.28__tar.gz

{soup_cli-0.71.26 → soup_cli-0.71.28}/CHANGELOG.md

@@ -12,6 +12,111 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.28] - 2026-07-04
+
+### Added
+- **MCP server (`soup mcp serve`)** — drive Soup from any Model Context Protocol
+  client (Claude Code / Cursor / Cline / Continue) over stdio. No fine-tuning
+  CLI ships an MCP server. Exposes 14 read-only tools as JSON — `advise`,
+  `data_inspect` / `data_validate` / `data_score` / `data_doctor`,
+  `recipes_search` / `recipes_show`, `runs_list` / `runs_show`,
+  `registry_list` / `registry_show`, `profile`, `diagnose_evidence`,
+  `ship_evidence` — plus two **plan-only** mutating tools (`train_start`,
+  `export`) gated behind `--allow-mutating` (they render the exact command that
+  would run; they never execute). The official `mcp` SDK is behind a new
+  `[mcp]` extra (`pip install 'soup-cli[mcp]'`), lazy-imported so the core CLI
+  stays light. Security: stdio-only (no network listener); every path argument
+  re-enters cwd-containment + symlink rejection; output is control-char
+  sanitized; errors are path-free; string / size / int bounds enforced.
+
+### Fixed
+- The DPO / IPO / KTO / BCO trainers now apply configured vocabulary expansion
+  (`data.add_new_tokens` / `data.new_special_tokens`) via the shared
+  `apply_vocab_expansion()` helper during setup, consistent with the SFT path —
+  they previously ignored it. Closes #292
+  ([#293](https://github.com/MakazhanAlpamys/Soup/pull/293) by [@CODING-DARSH](https://github.com/CODING-DARSH)).
+- The ORPO / SimPO / GRPO trainers now also apply configured vocabulary
+  expansion via the shared `apply_vocab_expansion()` helper — completing
+  consistent vocab-expansion behavior across every SFT/preference/RL trainer.
+  Closes #294
+  ([#295](https://github.com/MakazhanAlpamys/Soup/pull/295) by [@CODING-DARSH](https://github.com/CODING-DARSH)).
+
+## [0.71.27] - 2026-07-03
+
+### Added
+- **Fine-tune Doctor** — kill the top *silent* fine-tune failures before a
+  single training step; no competitor (Unsloth/Axolotl/LlamaFactory) ships
+  any of these three:
+  - `soup data doctor <data> --model <id|path>` — chat-template
+    compatibility report over 8 checks: `chat_template` present,
+    `template_render`s cleanly, has `{% generation %}` markers,
+    `eos_in_labels` (the **#1 "model never stops generating" bug** — every
+    assistant turn's trained span must actually contain an EOS/EOT token;
+    checks every turn, not just the last), `bos_duplication` (template +
+    tokenizer both prepending BOS), `system_role` support (Mistral-style
+    templates reject a leading system turn), `unknown_roles`, and
+    `truncation_risk` (p95 rendered length vs `max_length`). Same OK / MINOR
+    / MAJOR taxonomy as `soup diagnose`; exit 0 = OK/MINOR, exit 2 = MAJOR.
+    `--train-on-responses-only` / `--train-on-messages-with-train-field`
+    select the same masking strategy `soup train` would use, so the report
+    and `--show-mask` never disagree about what's actually trained.
+  - `soup data doctor ... --show-mask N` — render N sample rows with
+    per-token trained/masked colouring through the REAL collator path
+    (answer-only / per-message-train-field / RAFT span-mask) — not a
+    reimplementation — so an assistant-mask bug is visible instantly.
+  - `soup data lint <data>` — preference-data linter for
+    dpo/orpo/simpo/ipo/bco/kto: `length_bias` (chosen systematically longer
+    than rejected — the **#1 silent DPO degradation**, reported as a
+    Cohen's d effect size), `label_imbalance` (KTO desirable:undesirable
+    ratio), `near_duplicates` (MinHash/LSH, reuses the `soup data dedup`
+    kernel), `identical_pairs` (chosen == rejected — zero preference
+    signal), and `prompt_leak` (the prompt echoed verbatim inside the
+    completion — a common synthetic-data pipeline bug). Optional `--model`
+    for exact token-length bias (default: word count).
+  - Validated live against the real `HuggingFaceTB/SmolLM2-135M-Instruct`
+    tokenizer on Windows + RTX 3050 — this smoke pass found and fixed two
+    genuine bugs beyond what synthetic fixtures alone caught: an EOS check
+    that required the EOS token to be the *literal last* trained token
+    (real templates often have a trailing formatting token after the
+    closing tag that stays inside the trained span), and two call sites
+    that only caught `(ValueError, TypeError)` around a tokenizer's
+    `apply_chat_template` when a real Jinja `raise_exception()`
+    (Mistral-style no-system-role guard) raises
+    `jinja2.exceptions.TemplateError`.
+
+### Fixed
+- Harden `commands/diagnose.py`'s `--evidence` loader against a TOCTOU
+  symlink swap: opens with `O_NOFOLLOW` and size-checks the open fd via
+  `os.fstat` instead of `os.path.getsize` on the path before the open —
+  backports the hardened loader shipped for `soup ship` in v0.71.25 (closes
+  v0.71.25 known-limitation (4)).
+- Harden judge-model URL validation against a hostname prefix bypass
+  (`http://localhost.attacker.com`) — `GateTask._valid_judge_url` /
+  `_parse_judge_url` now use `urllib.parse.urlparse` + hostname checks instead
+  of `startswith`. Closes #283
+  ([#288](https://github.com/MakazhanAlpamys/Soup/pull/288) by [@CODING-DARSH](https://github.com/CODING-DARSH)).
+- `SFTTrainerWrapper` now applies configured vocabulary expansion
+  (`data.add_new_tokens` / `data.new_special_tokens`) and resizes the model
+  embeddings during initialization — previously these fields were accepted by
+  the schema but silently ignored. Closes #289
+  ([#287](https://github.com/MakazhanAlpamys/Soup/pull/287) by [@CODING-DARSH](https://github.com/CODING-DARSH)).
+- Vision and audio SFT paths now apply that same configured vocabulary
+  expansion (`data.add_new_tokens` / `data.new_special_tokens`) via the shared
+  `apply_vocab_expansion()` helper, consistent with the text SFT path — they
+  previously ignored it. Closes #290
+  ([#291](https://github.com/MakazhanAlpamys/Soup/pull/291) by [@CODING-DARSH](https://github.com/CODING-DARSH)).
+
+### Security
+- `soup data doctor` strips C0 control characters (keeping tab/newline/CR)
+  from dataset-derived content before it reaches the terminal — Rich's
+  `markup.escape()` only neutralises `[...]` tag syntax, not raw escape
+  sequences, so an untrusted training row (e.g. an unknown `role` field, or
+  `--show-mask`'s decoded token text on a byte-level BPE tokenizer) could
+  otherwise carry a literal ESC byte through to the terminal (title-bar /
+  OSC-8 link spoofing, or obscuring a MAJOR verdict via cursor tricks).
+  `--output` JSON is unaffected (`json.dumps` already escapes control
+  characters).
+
 ## [0.71.26] - 2026-07-01
 
 ### Added

{soup_cli-0.71.26 → soup_cli-0.71.28}/CONTRIBUTING.md

@@ -116,11 +116,11 @@ src/soup_cli/
   cans/               - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0)
   data/traces/        - Trace-to-Preference harvester (v0.26.0)
   data/collators.py   - CrossDocCollator for sample packing (v0.33.0)
-  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict
+  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint
   templates/          - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (298 files, 14788 tests)
+tests/                - Test suite (305 files, 15140 tests)
 examples/             - Real-world config examples and datasets
 ```
 
@@ -150,7 +150,7 @@ pytest tests/test_data.py::test_detect_alpaca_format -v
 pytest tests/ --cov=soup_cli --cov-report=html
 ```
 
-### Test Files (297 files)
+### Test Files (305 files)
 
 > A representative sample of the suite below. The full table lives in
 > [`.claude/CLAUDE.md`](.claude/CLAUDE.md); run `pytest tests/ -v` for the complete list.

{soup_cli-0.71.26 → soup_cli-0.71.28}/CONTRIBUTORS.md

@@ -44,6 +44,12 @@ Listed by first contribution. PR numbers link the work.
   - Unit tests for the `warmup.py` auto-warmup-steps helper ([#274](https://github.com/MakazhanAlpamys/Soup/pull/274))
 - **Kondamwar Akshaya Shrikant** ([@Akshaya-reddy18](https://github.com/Akshaya-reddy18))
   - Friendlier error messages — richer CUDA-OOM hint + Hugging Face gated-repo and `trust_remote_code` mappings + tests ([#282](https://github.com/MakazhanAlpamys/Soup/pull/282))
+- **Darsh** ([@CODING-DARSH](https://github.com/CODING-DARSH))
+  - Harden judge-URL validation against hostname prefix bypass (`startswith` → `urlparse`) in `eval/gate.py` ([#288](https://github.com/MakazhanAlpamys/Soup/pull/288))
+  - Apply configured vocabulary expansion (`data.add_new_tokens` / `new_special_tokens`) during SFT trainer init ([#287](https://github.com/MakazhanAlpamys/Soup/pull/287))
+  - Reuse the shared vocab-expansion helper in the vision + audio SFT paths ([#291](https://github.com/MakazhanAlpamys/Soup/pull/291))
+  - Honor configured vocab expansion in the DPO / IPO / KTO / BCO trainers ([#293](https://github.com/MakazhanAlpamys/Soup/pull/293))
+  - Honor configured vocab expansion in the ORPO / SimPO / GRPO trainers ([#295](https://github.com/MakazhanAlpamys/Soup/pull/295))
 
 ---
 

{soup_cli-0.71.26 → soup_cli-0.71.28}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.26
+Version: 0.71.28
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -29,6 +29,7 @@ Requires-Dist: bitsandbytes>=0.41.0; extra == 'all'
 Requires-Dist: datasets>=2.14.0; extra == 'all'
 Requires-Dist: datasketch>=1.6.0; extra == 'all'
 Requires-Dist: fastapi>=0.104.0; extra == 'all'
+Requires-Dist: mcp>=1.2.0; extra == 'all'
 Requires-Dist: peft>=0.7.0; extra == 'all'
 Requires-Dist: torch>=2.0.0; extra == 'all'
 Requires-Dist: transformers<5.0.0,>=4.36.0; extra == 'all'
@@ -60,6 +61,7 @@ Requires-Dist: bitsandbytes>=0.41.0; extra == 'dev'
 Requires-Dist: cryptography>=41.0.0; extra == 'dev'
 Requires-Dist: datasets>=2.14.0; extra == 'dev'
 Requires-Dist: httpx>=0.24.0; extra == 'dev'
+Requires-Dist: mcp>=1.2.0; extra == 'dev'
 Requires-Dist: mypy>=1.8.0; extra == 'dev'
 Requires-Dist: peft>=0.7.0; extra == 'dev'
 Requires-Dist: pre-commit>=3.5.0; extra == 'dev'
@@ -80,6 +82,8 @@ Provides-Extra: gptq
 Requires-Dist: auto-gptq>=0.7.0; extra == 'gptq'
 Provides-Extra: liger
 Requires-Dist: liger-kernel>=0.3.0; extra == 'liger'
+Provides-Extra: mcp
+Requires-Dist: mcp>=1.2.0; extra == 'mcp'
 Provides-Extra: mix
 Requires-Dist: scikit-optimize>=0.9.0; extra == 'mix'
 Provides-Extra: mlx
@@ -189,21 +193,25 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.26 — Closed-loop reward-hacking auto-mitigation.** Your GRPO/PPO trainer now *detects* reward hacking mid-run and *self-corrects* — instead of only halting. No OSS RLHF library closes this loop.
-
-- **Detect → correct → continue.** When the hacking signal trips, the controller raises the
-  KL coefficient β (bang-bang + hysteresis, or a PID-Lagrangian controller), and — if hacking
-  persists — **rolls back to the last-good checkpoint**, then early-stops as a last resort.
-- **Anti-gaming hardening.** Multi-signal voting (RM cluster-separation + length-trend +
-  repetition), signal smoothing, conservative-on-disagreement, a reward-distribution-drift
-  guard, and optional bounded **reward shaping** on the gamed proxy.
-- **Observe first.** `reward_hack_mitigation=log_only` streams a per-step `mitigation_log.jsonl`
-  and never touches training — see the hacking happen before you let the controller act.
-- **Honest scope.** Proof-of-mechanism on SmolLM2-135M + a synthetic hacking task on one RTX
-  3050 (all four stages validated live, including a real rollback). PPO ships BETA.
-
-```bash
-soup train --config grpo.yaml --reward-hack-mitigation kl_control   # detect → raise KL → recover
+**v0.71.28 — Drive Soup from your coding agent: `soup mcp serve`.** A Model Context Protocol server, so Claude Code / Cursor / Cline / Continue can run Soup conversationally. No other fine-tuning CLI ships one.
+
+- **14 read-only tools, JSON out.** `advise`, `data_inspect` / `validate` / `score` / `doctor`,
+  `recipes_search` / `show`, `runs_list` / `show`, `registry_list` / `show`, `profile`,
+  `diagnose_evidence`, `ship_evidence` — your agent can inspect data, search recipes, read runs,
+  and get a ship verdict without you leaving the chat.
+- **Plan-only by default, safe by design.** stdio transport only (no network listener); the two
+  mutating tools (`train_start`, `export`) are gated behind `--allow-mutating` and only ever
+  render the exact command that would run. Every path argument re-enters cwd containment +
+  symlink rejection; output is control-char sanitized; errors are path-free.
+- **Light install.** The official `mcp` SDK lives behind a new `[mcp]` extra
+  (`pip install 'soup-cli[mcp]'`), lazy-imported so the core CLI stays torch-free and fast.
+- **Live-validated on Windows.** A real stdio MCP client drove `initialize → list_tools →
+  call` end-to-end on Windows + RTX 3050 — read tools return real JSON, the mutating tools
+  refuse without the flag, and a path-traversal argument is rejected.
+
+```jsonc
+// .mcp.json (Claude Code) or claude_desktop_config.json
+{ "mcpServers": { "soup": { "command": "soup", "args": ["mcp", "serve"] } } }
 ```
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).

{soup_cli-0.71.26 → soup_cli-0.71.28}/README.md

@@ -49,21 +49,25 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.26 — Closed-loop reward-hacking auto-mitigation.** Your GRPO/PPO trainer now *detects* reward hacking mid-run and *self-corrects* — instead of only halting. No OSS RLHF library closes this loop.
-
-- **Detect → correct → continue.** When the hacking signal trips, the controller raises the
-  KL coefficient β (bang-bang + hysteresis, or a PID-Lagrangian controller), and — if hacking
-  persists — **rolls back to the last-good checkpoint**, then early-stops as a last resort.
-- **Anti-gaming hardening.** Multi-signal voting (RM cluster-separation + length-trend +
-  repetition), signal smoothing, conservative-on-disagreement, a reward-distribution-drift
-  guard, and optional bounded **reward shaping** on the gamed proxy.
-- **Observe first.** `reward_hack_mitigation=log_only` streams a per-step `mitigation_log.jsonl`
-  and never touches training — see the hacking happen before you let the controller act.
-- **Honest scope.** Proof-of-mechanism on SmolLM2-135M + a synthetic hacking task on one RTX
-  3050 (all four stages validated live, including a real rollback). PPO ships BETA.
-
-```bash
-soup train --config grpo.yaml --reward-hack-mitigation kl_control   # detect → raise KL → recover
+**v0.71.28 — Drive Soup from your coding agent: `soup mcp serve`.** A Model Context Protocol server, so Claude Code / Cursor / Cline / Continue can run Soup conversationally. No other fine-tuning CLI ships one.
+
+- **14 read-only tools, JSON out.** `advise`, `data_inspect` / `validate` / `score` / `doctor`,
+  `recipes_search` / `show`, `runs_list` / `show`, `registry_list` / `show`, `profile`,
+  `diagnose_evidence`, `ship_evidence` — your agent can inspect data, search recipes, read runs,
+  and get a ship verdict without you leaving the chat.
+- **Plan-only by default, safe by design.** stdio transport only (no network listener); the two
+  mutating tools (`train_start`, `export`) are gated behind `--allow-mutating` and only ever
+  render the exact command that would run. Every path argument re-enters cwd containment +
+  symlink rejection; output is control-char sanitized; errors are path-free.
+- **Light install.** The official `mcp` SDK lives behind a new `[mcp]` extra
+  (`pip install 'soup-cli[mcp]'`), lazy-imported so the core CLI stays torch-free and fast.
+- **Live-validated on Windows.** A real stdio MCP client drove `initialize → list_tools →
+  call` end-to-end on Windows + RTX 3050 — read tools return real JSON, the mutating tools
+  refuse without the flag, and a path-traversal argument is rejected.
+
+```jsonc
+// .mcp.json (Claude Code) or claude_desktop_config.json
+{ "mcpServers": { "soup": { "command": "soup", "args": ["mcp", "serve"] } } }
 ```
 
 Full history: [CHANGELOG.md](CHANGELOG.md) · [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).

{soup_cli-0.71.26 → soup_cli-0.71.28}/docs/commands.md

@@ -68,6 +68,9 @@ soup sweep --config soup.yaml --param lr=...  Hyperparameter search
 soup diff --model-a ./a --model-b ./b         Compare two models
 soup data inspect <path>                      View dataset stats
 soup data validate <path>                     Check format (auto-detect)
+soup data doctor <path> --model <id>          Chat-template compat report: 8 checks, OK/MINOR/MAJOR
+soup data doctor <path> --model <id> --show-mask N  Per-token trained/masked colouring via the real collator
+soup data lint <path>                         Preference-data linter: length bias, near-dups, chosen==rejected
 soup data convert <path> --to chatml          Convert between formats
 soup data merge data1.jsonl data2.jsonl       Combine datasets
 soup data dedup <path> --threshold 0.8        Remove duplicates (MinHash)
@@ -81,7 +84,8 @@ soup data generate ... --quality-pipeline     Auto validate + filter + dedup
 soup data augment <path> --strategy rephrase|translate|style [--provider ollama|vllm --model <m> --base-url <url>]  LLM-driven augmentation
 soup data from-traces --logs l.jsonl --format langchain --signal thumbs_up --output p.jsonl  Preference pairs from traces
 soup data from-traces ... --judge --min-confidence 0.7  LLM-judge confidence filter
-soup data review prefs.jsonl --sample 10      Preview preference pairssoup data filter <path> --coherence 0.3       Quality filter (perplexity/coherence)
+soup data review prefs.jsonl --sample 10      Preview preference pairs
+soup data filter <path> --coherence 0.3       Quality filter (perplexity/coherence)
 soup data sample <path> --n 1000             Random sample subset
 soup data sample <path> --n 1000 --strategy diverse  Cluster-based diverse sampling
 soup data sample <path> --n 1000 --strategy hard     Sample hardest examples
@@ -152,6 +156,8 @@ soup ship --base <m> --adapter <lora> --task-eval t.jsonl  SHIP / DON'T-SHIP ver
 soup ship --evidence ev.json [--output v.json]  Decide offline from pre-computed scores (no model load)
 soup ship ... --task-mode judge_score --judge-model ollama://llama3.1  Leg-1 via LLM-as-a-judge
 soup ship ... --general-suite mmlu,gsm8k --baseline base.json  lm-eval leg-2 + recorded base scores
+soup mcp serve                                MCP server over stdio (drive Soup from Claude Code / Cursor / Cline; requires [mcp] extra) (v0.71.28)
+soup mcp serve --allow-mutating               Also expose plan-only train_start / export tools (never execute) (v0.71.28)
 soup tui                                      Full-screen Textual dashboard (requires [tui] extra)
 soup train --config soup.yaml --profile       Record torch.profiler trace to <output>/profiles/
 soup --log-level quiet|normal|verbose|debug   Global logging tier (Rich-formatted)
@@ -263,4 +269,37 @@ soup version [--full] [--json]                Show version (--full: system info,
 soup --verbose <command>                      Full traceback on errors
 ```
 
+## Fine-tune from your coding agent (MCP)
+
+`soup mcp serve` runs a [Model Context Protocol](https://modelcontextprotocol.io)
+server over **stdio**, so any MCP client — Claude Code, Cursor, Cline, Continue —
+can drive Soup conversationally. Install the extra first:
+
+```bash
+pip install 'soup-cli[mcp]'
+```
+
+Register it with your client. For **Claude Code** (`.mcp.json` in the repo) or
+the **Claude Desktop** config (`claude_desktop_config.json`):
+
+```json
+{
+  "mcpServers": {
+    "soup": { "command": "soup", "args": ["mcp", "serve"] }
+  }
+}
+```
+
+The server exposes 14 read-only tools — `advise`, `data_inspect`,
+`data_validate`, `data_score`, `data_doctor`, `recipes_search`, `recipes_show`,
+`runs_list`, `runs_show`, `registry_list`, `registry_show`, `profile`,
+`diagnose_evidence`, `ship_evidence` — each returning JSON. Two **plan-only**
+mutating tools (`train_start`, `export`) are gated behind `--allow-mutating`
+(`"args": ["mcp", "serve", "--allow-mutating"]`); even then they only render the
+exact command that would run — they never execute training or export.
+
+**Security:** stdio only (no network listener); every path argument stays under
+the working directory and rejects symlinks; tool output is control-char
+sanitized; error messages never leak filesystem paths.
+
 

{soup_cli-0.71.26 → soup_cli-0.71.28}/docs/data.md

@@ -637,3 +637,49 @@ written per node; resume rehydrates from per-node sidecars. Failed rows logged w
 redacted reasons (paths stripped, capped at 256 chars).
 
 
+## Fine-tune Doctor (`soup data doctor`)
+
+Chat-template compatibility report — catches the top *silent* fine-tuning failures
+before a single training step:
+
+```bash
+soup data doctor ./data/train.jsonl --model meta-llama/Llama-3.1-8B-Instruct
+
+# Render N sample rows with per-token trained/masked colouring, through the REAL
+# collator path (answer-only / per-message-train-field / RAFT span-mask)
+soup data doctor ./data/train.jsonl --model meta-llama/Llama-3.1-8B-Instruct --show-mask 5
+```
+
+Eight checks, same OK/MINOR/MAJOR taxonomy as `soup diagnose` (exit 0 on OK/MINOR,
+exit 2 on MAJOR): `chat_template` (tokenizer has one), `template_render` (renders
+cleanly on a sample), `generation_markers` (`{% generation %}` support),
+`eos_in_labels` — the **#1 "model never stops generating" bug**: every trained
+assistant turn must actually contain an EOS/EOT token, checked across the *whole*
+trained span, not just the last turn — `bos_duplication` (template + tokenizer both
+prepending BOS), `system_role` (Mistral-style templates that reject a leading system
+turn), `unknown_roles`, and `truncation_risk` (p95 rendered length vs
+`data.max_length`). `--train-on-responses-only` / `--train-on-messages-with-train-field`
+select the same masking strategy `soup train` would use, so the report and
+`--show-mask` preview can never disagree about what's actually trained.
+
+
+## Preference-Data Linter (`soup data lint`)
+
+Catches the top silent degradations in DPO/ORPO/SimPO/IPO/BCO/KTO preference data:
+
+```bash
+soup data lint ./data/prefs.jsonl
+soup data lint ./data/prefs.jsonl --model meta-llama/Llama-3.1-8B-Instruct  # exact token-length bias, not word count
+```
+
+Five checks: `length_bias` — the **#1 silent DPO degradation**: `chosen`
+systematically longer than `rejected`, reported as a Cohen's d effect size —
+`label_imbalance` (KTO desirable:undesirable ratio), `near_duplicates`
+(MinHash/LSH, reuses the `soup data dedup` kernel; requires
+`pip install 'soup-cli[data]'`, degrades to an advisory skip otherwise),
+`identical_pairs` (`chosen == rejected` — zero preference signal), and
+`prompt_leak` (the prompt echoed verbatim inside the completion, a common
+synthetic-data pipeline bug). Same OK/MINOR/MAJOR taxonomy and exit codes as
+`soup data doctor`.
+
+

{soup_cli-0.71.26 → soup_cli-0.71.28}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.26"
+version = "0.71.28"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"
@@ -47,14 +47,14 @@ train = [
     "accelerate>=0.25.0",
 ]
 # v0.71.0 — convenience meta-extra pulling the main optional stacks.
-all = ["soup-cli[train,serve,ui,data]"]
+all = ["soup-cli[train,serve,ui,data,mcp]"]
 eval = ["lm-eval>=0.4.0"]
 data = ["datasketch>=1.6.0"]
 wandb = ["wandb>=0.15.0,<0.18.0"]
 # Self-references `[train]` so CI / contributors get the full training stack
 # (CI runs `pip install -e ".[dev]"`; without this every test would fail at
 # `import torch`).
-dev = ["soup-cli[train]", "cryptography>=41.0.0", "reportlab>=4.0.0", "pytest>=7.0", "ruff>=0.1.0", "pytest-cov>=4.0", "httpx>=0.24.0", "mypy>=1.8.0", "pre-commit>=3.5.0"]
+dev = ["soup-cli[train,mcp]", "cryptography>=41.0.0", "reportlab>=4.0.0", "pytest>=7.0", "ruff>=0.1.0", "pytest-cov>=4.0", "httpx>=0.24.0", "mypy>=1.8.0", "pre-commit>=3.5.0"]
 ui = ["fastapi>=0.104.0", "uvicorn>=0.24.0"]
 serve = ["fastapi>=0.104.0", "uvicorn>=0.24.0"]
 serve-fast = ["vllm>=0.4.0", "fastapi>=0.104.0", "uvicorn>=0.24.0"]
@@ -102,6 +102,10 @@ compile = ["dspy-ai>=2.5.0", "textgrad>=0.1.0", "gepa>=0.0.1"]
 # Lazy-imported; only needed for `--cloud-submit` (plan-only render needs no
 # dependency). Modal auth is via `modal setup`.
 modal = ["modal>=0.60.0"]
+# v0.71.28 - `soup mcp serve` MCP server. The official `mcp` python SDK is
+# lazy-imported (only src/soup_cli/mcp_server/server.py touches it), so the CLI
+# stays light without it. Floor pinned to guard against SDK API churn.
+mcp = ["mcp>=1.2.0"]
 
 [project.scripts]
 soup = "soup_cli.cli:run"

soup_cli-0.71.28/src/soup_cli/__init__.py

@@ -0,0 +1,3 @@
+"""Soup CLI — Fine-tune and post-train LLMs in one command."""
+
+__version__ = "0.71.28"

{soup_cli-0.71.26 → soup_cli-0.71.28}/src/soup_cli/cans/schema.py

@@ -82,6 +82,10 @@ class DeployTarget(BaseModel):
             raise ValueError("deploy path contains null byte")
         if value.startswith("/") or value.startswith("\\"):
             raise ValueError(f"deploy path '{value}' must be relative")
+        # Windows drive-absolute (``C:\...`` / ``C:/...``) is absolute too but
+        # starts with a letter, so it slipped past the ``/`` / ``\`` check.
+        if len(value) >= 2 and value[1] == ":" and value[0].isalpha():
+            raise ValueError(f"deploy path '{value}' must be relative")
         # Normalise separators first, then split — a mixed-separator path
         # like ``foo/..\\bar`` would otherwise slip past a single-separator
         # split because ``"..\\bar"`` != ``".."``.

{soup_cli-0.71.26 → soup_cli-0.71.28}/src/soup_cli/cli.py

@@ -510,6 +510,23 @@ app.add_typer(
     help="RA-DIT two-stage orchestrator: retriever -> generator (v0.71.10).",
 )
 
+# v0.71.27 — Fine-tune Doctor: chat-template doctor + loss-mask X-ray +
+# preference linter.
+from soup_cli.commands import data_doctor as _data_doctor_cmd  # noqa: E402
+
+data.app.command(name="doctor")(_data_doctor_cmd.doctor)
+data.app.command(name="lint")(_data_doctor_cmd.lint)
+
+# v0.71.28 — MCP server: drive Soup from any MCP client (Claude Code / Cursor /
+# Cline / Continue) over stdio.
+from soup_cli.commands import mcp as _mcp_cmd  # noqa: E402
+
+app.add_typer(
+    _mcp_cmd.app,
+    name="mcp",
+    help="Model Context Protocol server - drive Soup from any MCP client (v0.71.28).",
+)
+
 
 def _rewrite_advise_argv(argv: list) -> list:
     """Inject `run` between `advise` and a non-subcommand first argument.

{soup_cli-0.71.26 → soup_cli-0.71.28}/src/soup_cli/cloud/modal.py

@@ -196,7 +196,12 @@ def render_modal_stub(
         "@app.local_entrypoint()\n"
         "def main() -> None:\n"
         "    train.remote()\n"
-        f'    print("Training submitted; download checkpoints to {output_dir}")\n'
+        # Reference the already-repr()-embedded ``_LOCAL_OUTPUT`` via a runtime
+        # f-string in the GENERATED code — no user value is interpolated into
+        # the stub source here. (Interpolating raw ``{output_dir}`` was a code-
+        # injection hole; ``{output_dir!r}`` alone would still break for a path
+        # containing a quote because the repr is nested inside a "..." literal.)
+        '    print(f"Training submitted; download checkpoints to {_LOCAL_OUTPUT}")\n'
     )
 
 

{soup_cli-0.71.26 → soup_cli-0.71.28}/src/soup_cli/commands/_eval_v0550.py

@@ -43,7 +43,7 @@ def register(app: typer.Typer, console: Console) -> None:
         try:
             rows = load_advise_dataset(data)
         except (FileNotFoundError, ValueError, TypeError) as exc:
-            console.print(f"[red]Cannot read dataset:[/] {exc}")
+            console.print(f"[red]Cannot read dataset:[/] {escape(str(exc))}")
             raise typer.Exit(1) from exc
 
         try:
@@ -105,7 +105,7 @@ def register(app: typer.Typer, console: Console) -> None:
         try:
             rows = load_advise_dataset(data)
         except (FileNotFoundError, ValueError, TypeError) as exc:
-            console.print(f"[red]Cannot read dataset:[/] {exc}")
+            console.print(f"[red]Cannot read dataset:[/] {escape(str(exc))}")
             raise typer.Exit(1) from exc
 
         try:
@@ -254,6 +254,15 @@ def register(app: typer.Typer, console: Console) -> None:
             False, "--json-only",
             help="Suppress Rich output; emit a single JSON verdict line.",
         ),
+        suite: str = typer.Option(
+            None, "--suite",
+            help=(
+                "Locked eval suite (from `soup eval lock`) to validate as a gate "
+                "precondition — a missing / unparseable suite BLOCKS the check "
+                "(exit 1). cwd-contained. The generated pre-push hook passes "
+                "$GATE_SUITE here so the locked suite is actually enforced."
+            ),
+        ),
     ) -> None:
         """Run-vs-run regression check (paired-bootstrap CI).
 
@@ -265,6 +274,26 @@ def register(app: typer.Typer, console: Console) -> None:
         """
         import json as _json
 
+        # The locked suite is a hard precondition when supplied: validate it
+        # exists, is under cwd, and parses. Previously the hook wrote $GATE_SUITE
+        # but never used it, so a deleted / tampered locked suite silently
+        # passed the gate.
+        if suite:
+            from soup_cli.utils.eval_lock_coverage import load_locked_suite
+            from soup_cli.utils.paths import is_under_cwd
+
+            if not is_under_cwd(suite):
+                console.print(f"[red]--suite is outside cwd:[/] {escape(str(suite))}")
+                raise typer.Exit(1)
+            try:
+                load_locked_suite(suite)
+            except (FileNotFoundError, ValueError, OSError) as exc:
+                console.print(
+                    f"[red]Locked eval suite invalid — gate blocked:[/] "
+                    f"{escape(str(exc))}"
+                )
+                raise typer.Exit(1) from exc
+
         from soup_cli.experiment.tracker import ExperimentTracker
         from soup_cli.utils.eval_gate_hook import (
             GateThresholds,

{soup_cli-0.71.26 → soup_cli-0.71.28}/src/soup_cli/commands/bench.py

@@ -194,6 +194,11 @@ def bench(
     start_load = time.time()
     try:
         model_obj, tokenizer = _load_model(str(model_path), base, device)
+    except typer.Exit:
+        # typer.Exit subclasses RuntimeError, so the broad except below would
+        # swallow an already-reported CLI exit and mis-print "Failed to load
+        # model: 1". Let it propagate unchanged.
+        raise
     except (OSError, ImportError, RuntimeError, ValueError) as exc:
         console.print(f"[red]Failed to load model:[/] {exc}")
         raise typer.Exit(1) from exc

{soup_cli-0.71.26 → soup_cli-0.71.28}/src/soup_cli/commands/data.py

@@ -754,11 +754,12 @@ def sample_data(
     if output is None:
         out_path = file_path.parent / f"{file_path.stem}_sampled_{strategy}.jsonl"
     else:
+        from soup_cli.utils.paths import is_under_cwd
+
         out_path = Path(output).resolve()
-        cwd = Path.cwd().resolve()
-        try:
-            out_path.relative_to(cwd)
-        except ValueError:
+        # realpath + commonpath (is_under_cwd) — Path.resolve()+relative_to()
+        # breaks on Windows 8.3 short names.
+        if not is_under_cwd(output):
             console.print("[red]Output path must be under the current working directory.[/]")
             raise typer.Exit(1)
 
@@ -812,6 +813,13 @@ def split_data(
         console.print("[red]Specify at least one of --val or --test.[/]")
         raise typer.Exit(1)
 
+    # Reject negatives: a negative val/test slipped past the `>= total` check
+    # and produced a negative slice (e.g. --val -10 sent 90 rows to val, 10 to
+    # train — a silently inverted split).
+    if (val is not None and val < 0) or (test is not None and test < 0):
+        console.print("[red]--val and --test must be non-negative.[/]")
+        raise typer.Exit(1)
+
     data = load_raw_data(file_path)
     if not data:
         console.print("[red]Dataset is empty.[/]")
@@ -1265,25 +1273,23 @@ def download_dataset(
         raise typer.Exit(1)
 
     # Resolve output path
+    from soup_cli.utils.paths import is_under_cwd
+
     if output is None:
         ds_name = dataset_id.split("/")[-1] if "/" in dataset_id else dataset_id
         # Strip embedded path separators to prevent traversal
         ds_name = Path(ds_name).name
         out_path = (Path.cwd() / f"{ds_name}.jsonl").resolve()
-        cwd = Path.cwd().resolve()
-        try:
-            out_path.relative_to(cwd)
-        except ValueError:
+        # realpath + commonpath (is_under_cwd) — Path.resolve()+relative_to()
+        # breaks on Windows 8.3 short names.
+        if not is_under_cwd(out_path):
             console.print(
                 "[red]Derived output path escapes working directory.[/]"
             )
             raise typer.Exit(1)
     else:
         out_path = Path(output).resolve()
-        cwd = Path.cwd().resolve()
-        try:
-            out_path.relative_to(cwd)
-        except ValueError:
+        if not is_under_cwd(output):
             console.print(
                 "[red]Output path must be under the current working directory.[/]"
             )