soup-cli 0.71.5__tar.gz → 0.71.7__tar.gz

soup_cli-0.71.7/.mailmap

@@ -0,0 +1,8 @@
+# .mailmap — consolidate contributor identities for `git shortlog` and the
+# GitHub contributors graph. See: https://git-scm.com/docs/gitmailmap
+#
+# Salil Mhatre committed under four name/email combinations
+# (two emails, three display names). Map them all to one canonical identity.
+Salil Mhatre <32305505+Deadpool2000@users.noreply.github.com> <d2kyt@protonmail.com>
+Salil Mhatre <32305505+Deadpool2000@users.noreply.github.com> Salil <32305505+Deadpool2000@users.noreply.github.com>
+Salil Mhatre <32305505+Deadpool2000@users.noreply.github.com> Salil M <32305505+Deadpool2000@users.noreply.github.com>

{soup_cli-0.71.5 → soup_cli-0.71.7}/CHANGELOG.md

@@ -12,6 +12,87 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.7] - 2026-06-02
+
+### Added
+- **Eval live runners** — six probe surfaces that previously emitted heuristic
+  / neutral stubs now load a real model and run live (closes #161, #162, #208,
+  #211, #212, #165). New shared `soup_cli/utils/live_eval.py` provides the
+  model-loading primitives (generator / multi-generator closures, masked
+  cross-entropy eval-loss, a short-LoRA probe, and held-out logit agreement);
+  every heavy import (`torch` / `transformers` / `peft` / `lm_eval`) is lazy.
+- **`soup advise --probe-model <id>`** — runs a LIVE ROI probe: zero/few-shot
+  token-F1 baselines, a short LoRA probe (relative held-out-loss improvement +
+  real wall-clock), and base-model proximity (held-out logit agreement) folded
+  into the dataset profile. Without `--probe-model`, `--probe` stays the offline
+  heuristic.
+- **`soup tunability --live`** — replaces the offline heuristic with a real
+  per-candidate LoRA probe (loads each `repo_id`, trains `--probe-steps` on a
+  held-out-excluded slice, reports the held-out-loss drop).
+- **`soup eval capability --live --model <id>`** — invokes lm-eval-harness per
+  resolved task (or a `--tasks` override) with `--limit` / `--device`, isolating
+  per-task failures and surfacing a no-metric result as an explicit error.
+- **`soup eval behavior --base-model <id> [--adapter <path>]`** — generates
+  pre/post responses on the bundled behaviour battery and scores the live diff.
+- **`soup diagnose --base-model <id> [--adapter <path>] [--dataset <jsonl>]
+  [--tokenizer <id>]`** — runs all six failure-mode probes (forgetting / refusal
+  / format / mode_collapse / memorization / contamination) live via
+  `soup_cli.utils.diagnose.live.run_live_diagnose`; falls back to neutral OK or
+  `--evidence` JSON when no model is supplied.
+
+### Security
+- The two new JSONL dataset readers (`diagnose.live._load_dataset_rows`,
+  `tunability._load_jsonl_rows`) open with `O_NOFOLLOW` after the cwd-containment
+  check, closing the check→open TOCTOU window (matches the v0.65 / v0.67 reader
+  policy).
+
+## [0.71.6] - 2026-06-02
+
+### Added
+- **`soup build` live runner** — the dbt-for-SFT DAG (`soup build <manifest>`) now
+  *materialises* datasets instead of only dry-running the plan. Five built-in
+  transforms ship live (`identity`, `drop_empty`, `lowercase`, `strip`,
+  `dedup_exact`); `table` rebuilds from scratch, `view` re-derives on every run,
+  and `incremental` re-transforms only the rows whose content hash changed
+  (tracked in a SQLite state store, keyed by row hash **and** the model's
+  transform+config fingerprint so a transform change re-runs everything). Custom
+  transforms are passed per-run via the Python API's `transforms=` map. Outputs
+  are written atomically; the `--output-dir` is symlink-checked before any
+  directory is created.
+- **`soup data gen-magpie` live generator** — the Magpie synthetic generator
+  (Xu et al. 2024) now actually generates. It feeds an aligned model its
+  chat-template prefix (chatml / llama3 / gemma / mistral families auto-detected)
+  and harvests the self-generated user instruction + assistant response via raw
+  completion. Live providers: `ollama` (`/api/generate` raw) and `vllm`
+  (`/v1/completions`) — both SSRF-hardened (loopback-only HTTP); `anthropic` is
+  rejected (no raw-completion endpoint). Optional `--quality-filter` drops
+  low-quality rows via the v0.47 toxicity/educational scorers; exact-duplicate
+  instructions are de-duplicated.
+- **`soup eval irt-subset --model {1pl,2pl,3pl}`** — the IRT eval-cost optimiser
+  gained 2PL (per-item discrimination) and 3PL (+guessing floor) joint
+  coordinate-ascent MLE fits alongside the existing 1PL Rasch. `1pl` keeps the
+  closed-form path for back-compat; `2pl`/`3pl` route through the new `fit_irt`.
+- **Tokenizer-aware memorization probe** — `score_memorization(..., tokenizer=...)`
+  and `split_prefix(..., tokenizer=...)` (used by `soup diagnose`) now split the
+  prefix/suffix on real token-id boundaries and measure echo-overlap over
+  sub-word tokens when a tokenizer (HF id / path / duck-typed object) is supplied,
+  catching BPE-level memorization that whitespace tokenisation misses. Default
+  (no tokenizer) keeps the whitespace behaviour.
+
+### Fixed
+- **`soup data augment --provider ollama|vllm` no longer crashes** — the command
+  imported a non-existent `OllamaProvider` symbol and raised `ImportError` on
+  every non-OpenAI provider. It now routes through the shared, SSRF-hardened
+  provider factory; `--model` / `--base-url` are honoured, the output path is
+  containment- and symlink-checked, and the write is atomic.
+
+### Security
+- **Ollama / vLLM provider URLs reject `0.0.0.0`** — `validate_ollama_url` /
+  `validate_vllm_url` dropped the bind-any wildcard from their loopback allow-set
+  (now `localhost` / `127.0.0.1` / `::1` only), matching the newer
+  `validate_hub_endpoint` / `validate_webhook_url` SSRF validators. Reachable now
+  that Magpie threads a user-supplied `--base-url` through these providers.
+
 ## [0.71.5] - 2026-06-02
 
 ### Added

{soup_cli-0.71.5 → soup_cli-0.71.7}/CONTRIBUTING.md

@@ -120,7 +120,7 @@ src/soup_cli/
   templates/          - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (275 files, 12581 tests)
+tests/                - Test suite (277 files, 12771 tests)
 examples/             - Real-world config examples and datasets
 ```
 
@@ -150,7 +150,11 @@ pytest tests/test_data.py::test_detect_alpaca_format -v
 pytest tests/ --cov=soup_cli --cov-report=html
 ```
 
-### Test Files (86 files)
+### Test Files (275 files)
+
+> A representative sample of the suite below. The full table lives in
+> [`.claude/CLAUDE.md`](.claude/CLAUDE.md); run `pytest tests/ -v` for the complete list.
+
 
 | File | Covers |
 |------|--------|
@@ -441,6 +445,17 @@ The project follows semantic versioning: `MAJOR.MINOR.PATCH`
 5. Tag: `git tag v0.X.0 && git push --tags`
 6. GitHub Actions auto-publishes to PyPI on the `v*` tag (Trusted Publisher / OIDC)
 
+## Recognition
+
+We credit every contributor — recognition is free and you earned it.
+
+- **[CONTRIBUTORS.md](CONTRIBUTORS.md):** every merged (or in-tree-adopted) PR adds you here, with your work and PR number.
+- **CHANGELOG.md:** from the release that ships your change, the entry is tagged `(#NNN by @you)`.
+- **Commit trailers:** PRs land with a `Co-Authored-By:` line for you, so the GitHub
+  contributors graph reflects your work even after a squash merge.
+
+If you contributed and aren't credited somewhere, that's a bug — open a PR or an issue and we'll fix it.
+
 ## Community
 
 - **Issues:** Report bugs and request features on [GitHub Issues](https://github.com/MakazhanAlpamys/Soup/issues)

soup_cli-0.71.7/CONTRIBUTORS.md

@@ -0,0 +1,43 @@
+# Contributors
+
+Soup is built by its community. Thank you to everyone who has contributed code,
+tests, docs, and ideas. ❤️
+
+This list is maintained by hand alongside the GitHub
+[contributors graph](https://github.com/MakazhanAlpamys/Soup/graphs/contributors).
+Merged a PR and don't see yourself here? Open a PR adding your line — that counts too.
+
+## Maintainer
+
+- **Alpamys** ([@MakazhanAlpamys](https://github.com/MakazhanAlpamys)) — creator & lead maintainer
+
+## Contributors
+
+Listed by first contribution. PR numbers link the work.
+
+- **Salil Mhatre** ([@Deadpool2000](https://github.com/Deadpool2000))
+  - `soup version --json` for machine-readable CI output ([#6](https://github.com/MakazhanAlpamys/Soup/pull/6))
+  - RAM + disk-space checks in `soup doctor` ([#7](https://github.com/MakazhanAlpamys/Soup/pull/7))
+  - `soup runs clean` for smart checkpoint space management ([#9](https://github.com/MakazhanAlpamys/Soup/pull/9))
+  - Official Docker support for easier onboarding ([#20](https://github.com/MakazhanAlpamys/Soup/pull/20))
+  - `soup bench` — model speed + VRAM measurement ([#25](https://github.com/MakazhanAlpamys/Soup/pull/25))
+  - `--prompts-file` option for `soup bench` ([#30](https://github.com/MakazhanAlpamys/Soup/pull/30))
+  - Happy-path + CPU-warning tests for `soup bench` ([#31](https://github.com/MakazhanAlpamys/Soup/pull/31))
+  - `soup cost` — cloud GPU training cost estimation ([#42](https://github.com/MakazhanAlpamys/Soup/pull/42))
+  - `--nccl` flag for `soup doctor` multi-GPU bandwidth checks ([#178](https://github.com/MakazhanAlpamys/Soup/pull/178))
+- **Chinmaya Sahu** ([@csking101](https://github.com/csking101))
+  - DPO example config, sample data, and tests ([#48](https://github.com/MakazhanAlpamys/Soup/pull/48))
+  - FP8 `rowwise` + `rowwise_with_gw_hp` scaling recipes ([#62](https://github.com/MakazhanAlpamys/Soup/pull/62))
+- **Yixuan Xu** ([@mzl2233](https://github.com/mzl2233))
+  - Guard diagnose-gate on distributed worker ranks ([#169](https://github.com/MakazhanAlpamys/Soup/pull/169))
+- **dreamer0129** ([@dreamer0129](https://github.com/dreamer0129))
+  - Rich-markup escape fix in legacy `soup adapters` commands ([#175](https://github.com/MakazhanAlpamys/Soup/pull/175), adopted in-tree as [#174](https://github.com/MakazhanAlpamys/Soup/issues/174))
+- **Vivaan Dhawan** ([@VIVAAN-DHAWAN](https://github.com/VIVAAN-DHAWAN))
+  - Reject pickle/zip streams renamed to `.safetensors` via magic-byte check ([#198](https://github.com/MakazhanAlpamys/Soup/pull/198))
+- **Shivam** ([@shivam2931120](https://github.com/shivam2931120))
+  - Tokenizer-aware repetition scoring for the echo-trap detector ([#242](https://github.com/MakazhanAlpamys/Soup/pull/242))
+
+---
+
+Want to join this list? See [CONTRIBUTING.md](CONTRIBUTING.md) — good first issues are
+labelled in the [issue tracker](https://github.com/MakazhanAlpamys/Soup/labels/good%20first%20issue).

{soup_cli-0.71.5 → soup_cli-0.71.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.5
+Version: 0.71.7
 Summary: Fine-tune LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -183,22 +183,22 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.5 — Ingest, data & prompt polish.** Sharper production-loop ergonomics:
-
-- **Alternative model hubs for data** — `soup data push --hub modelscope|modelers` uploads a
-  local JSONL to ModelScope / Modelers, and `soup data forge --hub … --teacher owner/name`
-  pre-fetches the teacher from that hub.
-- **Tokenizer-aware prompt pruning** — `soup prune-prompt --tokenizer <id-or-path>` finds the
-  shared *token* prefix and decodes only the remainder, so BPE multi-byte sequences never get
-  truncated mid-token the way char-slicing can.
-- **Webhooks everywhere** — `--slack-url` / `--discord-url` now work on `soup ingest`,
-  `soup prune-prompt`, `soup ab`, and `soup data active-sample` (same SSRF-hardened validator as
-  `soup drift-alarm`). The A/B harness only pings when the sequential test actually decides.
-- **Curriculum by difficulty percentile** — dynamic curriculum can bucket by `loss` /
-  `perplexity` percentile instead of length round-robin.
-- **Smarter pre-flight `advise`** — `soup advise` now nudges its confidence using your prior
-  verdicts for the same project, and `soup runs replay` can plot a benchmark-score curve, not
-  just the loss curve.
+**v0.71.7 — Eval live runners.** Six probe surfaces that used to emit heuristic / neutral stubs
+now load a real model and run live (opt-in flags; the offline paths stay the default):
+
+- **`soup diagnose --base-model <id> [--adapter <path>] [--dataset d.jsonl]`** — runs all six
+  failure-mode probes (forgetting / refusal / format / mode_collapse / memorization /
+  contamination) against the loaded model instead of emitting neutral OK.
+- **`soup advise --probe-model <id>`** — a live ROI probe: zero/few-shot token-F1 baselines, a
+  short LoRA probe (real held-out-loss drop + wall-clock), and base-model proximity.
+- **`soup tunability --live`** — a real per-candidate LoRA probe (loads each base, trains
+  `--probe-steps` on a held-out slice).
+- **`soup eval capability --live --model <id>`** — invokes lm-eval-harness per task (with
+  `--tasks` / `--limit` / `--device`), isolating per-task failures.
+- **`soup eval behavior --base-model <id> [--adapter <path>]`** — generates pre/post responses on
+  the bundled behaviour battery and scores the live diff.
+- New shared `soup_cli/utils/live_eval.py` holds the model-loading primitives; heavy imports stay
+  lazy. Validated end-to-end on SmolLM2-135M (RTX 3050 4 GB).
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
 

{soup_cli-0.71.5 → soup_cli-0.71.7}/README.md

@@ -49,22 +49,22 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.5 — Ingest, data & prompt polish.** Sharper production-loop ergonomics:
-
-- **Alternative model hubs for data** — `soup data push --hub modelscope|modelers` uploads a
-  local JSONL to ModelScope / Modelers, and `soup data forge --hub … --teacher owner/name`
-  pre-fetches the teacher from that hub.
-- **Tokenizer-aware prompt pruning** — `soup prune-prompt --tokenizer <id-or-path>` finds the
-  shared *token* prefix and decodes only the remainder, so BPE multi-byte sequences never get
-  truncated mid-token the way char-slicing can.
-- **Webhooks everywhere** — `--slack-url` / `--discord-url` now work on `soup ingest`,
-  `soup prune-prompt`, `soup ab`, and `soup data active-sample` (same SSRF-hardened validator as
-  `soup drift-alarm`). The A/B harness only pings when the sequential test actually decides.
-- **Curriculum by difficulty percentile** — dynamic curriculum can bucket by `loss` /
-  `perplexity` percentile instead of length round-robin.
-- **Smarter pre-flight `advise`** — `soup advise` now nudges its confidence using your prior
-  verdicts for the same project, and `soup runs replay` can plot a benchmark-score curve, not
-  just the loss curve.
+**v0.71.7 — Eval live runners.** Six probe surfaces that used to emit heuristic / neutral stubs
+now load a real model and run live (opt-in flags; the offline paths stay the default):
+
+- **`soup diagnose --base-model <id> [--adapter <path>] [--dataset d.jsonl]`** — runs all six
+  failure-mode probes (forgetting / refusal / format / mode_collapse / memorization /
+  contamination) against the loaded model instead of emitting neutral OK.
+- **`soup advise --probe-model <id>`** — a live ROI probe: zero/few-shot token-F1 baselines, a
+  short LoRA probe (real held-out-loss drop + wall-clock), and base-model proximity.
+- **`soup tunability --live`** — a real per-candidate LoRA probe (loads each base, trains
+  `--probe-steps` on a held-out slice).
+- **`soup eval capability --live --model <id>`** — invokes lm-eval-harness per task (with
+  `--tasks` / `--limit` / `--device`), isolating per-task failures.
+- **`soup eval behavior --base-model <id> [--adapter <path>]`** — generates pre/post responses on
+  the bundled behaviour battery and scores the live diff.
+- New shared `soup_cli/utils/live_eval.py` holds the model-loading primitives; heavy imports stay
+  lazy. Validated end-to-end on SmolLM2-135M (RTX 3050 4 GB).
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
 

{soup_cli-0.71.5 → soup_cli-0.71.7}/docs/commands.md

@@ -72,7 +72,8 @@ soup data generate ... --provider anthropic   Use Claude API
 soup data generate ... --provider vllm        Use local vLLM server
 soup data generate ... --template code        Domain templates (code/conversation/qa/preference/reasoning)
 soup data generate ... --quality-pipeline     Auto validate + filter + dedup
-soup data augment <path> --strategy rephrase|translate|style  LLM-driven augmentationsoup data from-traces --logs l.jsonl --format langchain --signal thumbs_up --output p.jsonl  Preference pairs from traces
+soup data augment <path> --strategy rephrase|translate|style [--provider ollama|vllm --model <m> --base-url <url>]  LLM-driven augmentation
+soup data from-traces --logs l.jsonl --format langchain --signal thumbs_up --output p.jsonl  Preference pairs from traces
 soup data from-traces ... --judge --min-confidence 0.7  LLM-judge confidence filter
 soup data review prefs.jsonl --sample 10      Preview preference pairssoup data filter <path> --coherence 0.3       Quality filter (perplexity/coherence)
 soup data sample <path> --n 1000             Random sample subset
@@ -178,7 +179,8 @@ soup ingest|prune-prompt|ab|data active-sample ... --slack-url <https> | --disco
 soup drift-alarm --reference <jsonl> --live <jsonl> --threshold 0.2  Rolling-KL drift alarm (exit 3 on drift)
 soup drift-alarm ... --slack-url <https> | --discord-url <https>  Optional SSRF-validated webhook on drift detected
 soup tunability --list                                   List built-in candidate-base catalogue
-soup tunability --dataset <jsonl> [--candidates a,b,c]   Probe 8 candidate bases + Pareto frontier report
+soup tunability --dataset <jsonl> [--candidates a,b,c]   Probe candidate bases + Pareto frontier report
+soup tunability --dataset <jsonl> --live [--device cpu]  LIVE per-candidate LoRA probe (loads each repo)
 soup plan --config soup.yaml                             Pre-flight summary + write soup.tfstate
 soup apply --config soup.yaml [--dry-run]                Lock-and-execute; refuses on drift (exit 3)
 soup env lock | status | check                           Hermetic env lockfile + ABI drift detection (exit 3)
@@ -210,9 +212,9 @@ soup local-rl status --db <path>              Print interactions / thumbs-up / t
 soup local-rl record --db <path> --prompt <q> --response <r> --thumb up|down  Append thumbs record
 soup local-rl harvest --db <path> -o <pairs.jsonl>  Harvest DPO pairs from thumbs into JSONL
 soup local-rl train --db <path> --backend ollama|mlx --model <id>  Nightly DPO/KTO/ORPO train (v0.68.1)
-soup build <manifest.yaml> [--dry-run]        dbt-for-SFT DAG: validate + plan dataset transforms (v0.69.0)
+soup build <manifest.yaml> [--dry-run] [--output-dir <dir>]  dbt-for-SFT DAG: validate + plan + live materialise (v0.69.0; live v0.71.6)
 soup expect <data.jsonl> <suite.yaml>         Expectations suite: PII / token-length / refusal / judge (v0.69.0)
-soup data gen-magpie --base <m> --provider ollama|anthropic|vllm --target N [--plan-only]  Magpie synthetic generator (v0.69.0)
+soup data gen-magpie --base <m> --provider ollama|vllm --target N --output <jsonl> [--base-url <url>] [--quality-filter]  Magpie synthetic generator — live (v0.69.0; live v0.71.6)
 soup data persona-mix --prompts <jsonl> --n N --output <jsonl>  Persona-Hub diversity sampler (v0.69.0)
 soup data brain-rot <data.jsonl> [--strict]   Brain-rot detector — arXiv 2510.13928 (v0.69.0)
 soup iterative-dpo --base-model <m> --reward-model <rm> --prompts <p.jsonl> --output-dir <out> --rounds N --pairs-per-round N [--plan-only]  Iterative DPO loop driver (v0.70.0; live runner v0.70.1)

{soup_cli-0.71.5 → soup_cli-0.71.7}/docs/data.md

@@ -41,8 +41,8 @@ models:
   - {name: filtered, kind: incremental, refs: [raw], transform: filter_low_quality}
   - {name: tokenized, kind: incremental, refs: [filtered], transform: tokenize}
 EOF
-soup build build.yaml --dry-run   # validate topology + plan
-# soup build build.yaml           # live materialise (v0.69.1)
+soup build build.yaml --dry-run                 # validate topology + plan
+soup build build.yaml --output-dir built/       # live materialise (v0.71.6)
 
 # Expectations suite — Great Expectations for chat data
 cat > suite.yaml << 'EOF'
@@ -53,9 +53,9 @@ expectations:
 EOF
 soup expect data.jsonl suite.yaml   # exit 3 on suite failure
 
-# Magpie synthetic data — chat-template-prefix harvest (plan, runner v0.69.1)
+# Magpie synthetic data — chat-template-prefix harvest (live, v0.71.6)
 soup data gen-magpie --base meta-llama/Llama-3.1-8B-Instruct \
-    --provider ollama --target 1000 --plan-only
+    --provider ollama --target 1000 --output magpie.jsonl --quality-filter
 
 # Persona-Hub diversity — prompt × persona × style matrix sampling
 soup data persona-mix --prompts prompts.jsonl --n 500 --output mixed.jsonl
@@ -64,7 +64,7 @@ soup data persona-mix --prompts prompts.jsonl --n 500 --output mixed.jsonl
 soup data brain-rot data.jsonl --strict --max-major-fraction 0.10
 ```
 
-Every command applies the project-wide TOCTOU policy (`os.lstat + S_ISLNK` symlink rejection before any open) and cwd containment via the shared `paths.enforce_under_cwd_and_no_symlink` helper. Live runners for `soup build` and `soup data gen-magpie` land in v0.69.1; the other three are LIVE today.
+Every command applies the project-wide TOCTOU policy (`os.lstat + S_ISLNK` symlink rejection before any open) and cwd containment via the shared `paths.enforce_under_cwd_and_no_symlink` helper. All five are LIVE: `soup build` materialises with five built-in transforms (`identity` / `drop_empty` / `lowercase` / `strip` / `dedup_exact`) and SQLite-tracked incremental re-transform (v0.71.6); `soup data gen-magpie` harvests via raw completion against `--provider ollama|vllm` (loopback-only; `anthropic` rejected — no raw-completion endpoint, v0.71.6).
 
 
 ## Production Trace Ecosystem (`soup ingest`)
@@ -206,9 +206,13 @@ soup data augment ./data/train.jsonl --strategy translate --lang es,fr,de \
 # Style transfer (formal / casual / technical / etc.)
 soup data augment ./data/train.jsonl --strategy style --styles formal,casual \
   --output ./data/train_styled.jsonl
+
+# Local provider (Ollama / vLLM) — loopback-only, pick the model + base URL
+soup data augment ./data/train.jsonl --strategy rephrase --count 2 \
+  --provider ollama --model qwen2.5:0.5b --output ./data/train_local.jsonl
 ```
 
-Works with any provider supported by `soup data generate` (OpenAI, Ollama, Anthropic, vLLM, local server). `--count` is capped at 10; `--lang` and `--styles` each capped at 10 entries × 32 chars.
+Works with any provider supported by `soup data generate` (OpenAI, Ollama, vLLM, local server). `--model` and `--base-url` select a specific local model/endpoint; the Ollama/vLLM paths are loopback-only (SSRF-hardened). `--count` is capped at 10; `--lang` and `--styles` each capped at 10 entries × 32 chars.
 
 
 ## Trace-to-Preference

{soup_cli-0.71.5 → soup_cli-0.71.7}/docs/evaluation.md

@@ -61,9 +61,13 @@ soup advise data.jsonl --goal "make our chatbot more concise"
 #  Why:        Task is summarization with 120 rows and healthy diversity ...
 #  Flip when:  the prompt-engineering baseline already meets your target ...
 
-# Optional 10-min ROI probe (zero/few-shot + RAG + 100-step LoRA).
+# Optional ROI probe (offline heuristic: zero/few-shot + RAG + LoRA estimate).
 soup advise data.jsonl --goal "summarize my reports" --probe
 
+# LIVE ROI probe (v0.71.7): loads the model for zero/few-shot token-F1, a short
+# LoRA probe, and base-model proximity (held-out logit agreement). Implies --probe.
+soup advise data.jsonl --goal "..." --probe-model HuggingFaceTB/SmolLM2-135M
+
 # Print the rubric / evidence trail of the last verdict.
 soup advise explain
 
@@ -239,9 +243,13 @@ Default threshold 0.2 matches v0.43.0 KL-delta quant-check thresholds. Webhooks
 `soup diagnose` scores six independent failure modes for a trained adapter and renders an OK / MINOR / MAJOR verdict per mode plus an overall headline — same taxonomy as Quant-Lobotomy. Useful for catching adapter regressions that a loss curve cannot distinguish from a healthy run.
 
 ```bash
-# Heuristic neutral report (no model load — runs as a sanity check)
+# Neutral report (no model load — runs as a sanity check)
 soup diagnose my-run-id
 
+# LIVE (v0.71.7): load the model and run all six probes for real
+soup diagnose my-run-id --base-model HuggingFaceTB/SmolLM2-135M \
+    --adapter ./out --dataset train.jsonl --tokenizer HuggingFaceTB/SmolLM2-135M
+
 # Compute scores from a pre-built evidence JSON
 soup diagnose my-run-id --evidence evidence.json --output diag.json
 
@@ -252,6 +260,12 @@ soup diagnose my-run-id --badge diag.svg
 soup diagnose my-run-id --output diag.json --attach-to-registry abc123
 ```
 
+**Live runners (v0.71.7).** With `--base-model` the six probes run against the loaded model
+(+ optional `--adapter` LoRA path, `--dataset` for the forgetting / format / memorization probes,
+`--tokenizer` for a sub-word memorization variant) instead of emitting neutral OK. `refusal` uses
+a built-in probe set; `format` only fires when the dataset's own targets look like JSON;
+`contamination` stays neutral unless a benchmark corpus is supplied. Validated on SmolLM2-135M.
+
 **Six failure-mode probes:**
 
 | Mode | What it catches | Score range |
@@ -429,7 +443,7 @@ soup tunability --dataset ./eval.jsonl --candidates qwen3-0.6b,phi-4-mini --plan
 soup tunability --dataset ./eval.jsonl --probe-steps 100 --output ./tunability.json
 ```
 
-The report is a Pareto frontier over (eval delta from base, train cost, license) — candidates that nothing dominates on both axes survive, so you see a clean shortlist instead of a noisy single-leaderboard score. Live LoRA probe lands in v0.64.1; v0.64.0 ships the schema, Pareto math, and a `probe_fn=` injection point.
+The report is a Pareto frontier over (eval delta from base, train cost, license) — candidates that nothing dominates on both axes survive, so you see a clean shortlist instead of a noisy single-leaderboard score. By default the probe is a deterministic offline heuristic; pass `--live` (v0.71.7) to run a real per-candidate LoRA probe (loads each `repo_id`, trains `--probe-steps` on a held-out-excluded slice, reports the held-out-loss drop). `--device` selects cuda / cpu.
 
 
 ## Eval Depth (`soup eval behavior / capability / checklist / irt-subset`)
@@ -459,6 +473,10 @@ Persist a calibration once and reuse it across runs (v0.71.1): `write_judge_cali
 # Score over-refusal regression on XSTest (operator supplies evidence JSON)
 soup eval behavior my_run --battery xstest --evidence ev.json --output diff.json
 
+# LIVE (v0.71.7): generate pre/post responses on the bundled battery + score the diff
+soup eval behavior my_run --battery xstest \
+    --base-model HuggingFaceTB/SmolLM2-135M --adapter ./out
+
 # Bundled batteries: xstest, harmbench, jailbreakbench, elephant, syceval
 # Harmful prompts ship REDACTED — pull real sets from upstream papers.
 ```
@@ -472,9 +490,13 @@ soup eval capability my_run --suite math --output cap.json   # AIME + MATH-500
 soup eval capability my_run --suite code --output cap.json   # HumanEval+ + SWE-bench-Verified
 soup eval capability my_run --suite fast --output cap.json   # MMLU-Pro + HumanEval+
 soup eval capability my_run --suite full --output cap.json   # all 7 benchmarks
+
+# LIVE (v0.71.7): invoke lm-eval-harness per task against a real model
+soup eval capability my_run --live --model HuggingFaceTB/SmolLM2-135M \
+    --tasks arc_easy --limit 1 --device cpu
 ```
 
-Emits the (benchmark, lm-eval task) manifest; chain into the existing `soup eval benchmark` surface.
+Without `--live` it emits the (benchmark, lm-eval task) manifest; chain into the existing `soup eval benchmark` surface. With `--live --model <id>` (v0.71.7) it runs lm-eval-harness per resolved task — or a `--tasks` override — isolating per-task failures and capping examples with `--limit`.
 
 **CheckList behavioural DSL** — Ribeiro et al. 2020 MFT / INV / DIR tests:
 
@@ -503,8 +525,11 @@ soup eval checklist tests.yaml --evidence responses.json
 ```bash
 # Pick top-info 30% of items (5-10x eval-bill cut without losing power)
 soup eval irt-subset per_item_correctness.jsonl --size small --output plan.json
+
+# Richer item models: 2PL learns per-item discrimination, 3PL adds a guessing floor
+soup eval irt-subset per_item_correctness.jsonl --size small --model 2pl
 ```
 
-Closed-form 1PL Rasch fit (`β̂_i = -log(p̂_i / (1 - p̂_i))`); ranks by `p̂ · (1-p̂)` info (maximised at 50/50 items, since extremes carry no new ranking information). `full` keeps 100%, `small` keeps 30%, `tiny` keeps 10%.
+`--model` picks the item-response model (default `1pl`). `1pl` is the closed-form Rasch fit (`β̂_i = -log(p̂_i / (1 - p̂_i))`); `2pl` adds a per-item discrimination parameter and `3pl` a guessing floor, both via joint coordinate-ascent MLE (v0.71.6). Items rank by Fisher information at θ=0 (`p̂ · (1-p̂)` in 1PL — maximised at 50/50 items, since extremes carry no new ranking information). `full` keeps 100%, `small` keeps 30%, `tiny` keeps 10%.
 
 

{soup_cli-0.71.5 → soup_cli-0.71.7}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.5"
+version = "0.71.7"
 description = "Fine-tune LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.5 → soup_cli-0.71.7}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune LLMs in one command."""
 
-__version__ = "0.71.5"
+__version__ = "0.71.7"