soup-cli 0.71.6__tar.gz → 0.71.7__tar.gz

{soup_cli-0.71.6 → soup_cli-0.71.7}/CHANGELOG.md

@@ -12,6 +12,40 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.7] - 2026-06-02
+
+### Added
+- **Eval live runners** — six probe surfaces that previously emitted heuristic
+  / neutral stubs now load a real model and run live (closes #161, #162, #208,
+  #211, #212, #165). New shared `soup_cli/utils/live_eval.py` provides the
+  model-loading primitives (generator / multi-generator closures, masked
+  cross-entropy eval-loss, a short-LoRA probe, and held-out logit agreement);
+  every heavy import (`torch` / `transformers` / `peft` / `lm_eval`) is lazy.
+- **`soup advise --probe-model <id>`** — runs a LIVE ROI probe: zero/few-shot
+  token-F1 baselines, a short LoRA probe (relative held-out-loss improvement +
+  real wall-clock), and base-model proximity (held-out logit agreement) folded
+  into the dataset profile. Without `--probe-model`, `--probe` stays the offline
+  heuristic.
+- **`soup tunability --live`** — replaces the offline heuristic with a real
+  per-candidate LoRA probe (loads each `repo_id`, trains `--probe-steps` on a
+  held-out-excluded slice, reports the held-out-loss drop).
+- **`soup eval capability --live --model <id>`** — invokes lm-eval-harness per
+  resolved task (or a `--tasks` override) with `--limit` / `--device`, isolating
+  per-task failures and surfacing a no-metric result as an explicit error.
+- **`soup eval behavior --base-model <id> [--adapter <path>]`** — generates
+  pre/post responses on the bundled behaviour battery and scores the live diff.
+- **`soup diagnose --base-model <id> [--adapter <path>] [--dataset <jsonl>]
+  [--tokenizer <id>]`** — runs all six failure-mode probes (forgetting / refusal
+  / format / mode_collapse / memorization / contamination) live via
+  `soup_cli.utils.diagnose.live.run_live_diagnose`; falls back to neutral OK or
+  `--evidence` JSON when no model is supplied.
+
+### Security
+- The two new JSONL dataset readers (`diagnose.live._load_dataset_rows`,
+  `tunability._load_jsonl_rows`) open with `O_NOFOLLOW` after the cwd-containment
+  check, closing the check→open TOCTOU window (matches the v0.65 / v0.67 reader
+  policy).
+
 ## [0.71.6] - 2026-06-02
 
 ### Added

{soup_cli-0.71.6 → soup_cli-0.71.7}/CONTRIBUTING.md

@@ -120,7 +120,7 @@ src/soup_cli/
   templates/          - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (276 files, 12703 tests)
+tests/                - Test suite (277 files, 12771 tests)
 examples/             - Real-world config examples and datasets
 ```
 

{soup_cli-0.71.6 → soup_cli-0.71.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.6
+Version: 0.71.7
 Summary: Fine-tune LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -183,20 +183,22 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.6 — Synth data & build pipeline go live.** Three deferred stubs are now real:
-
-- **`soup build` materialises** — the dbt-for-SFT DAG no longer just dry-runs. `soup build
-  manifest.yaml --output-dir out/` runs the transforms (`drop_empty` / `lowercase` / `strip` /
-  `dedup_exact` / `identity` built in), rebuilds `table` / `view` models from scratch, and
-  re-transforms only the changed rows for `incremental` models (SQLite-tracked).
-- **`soup data gen-magpie` generates** — feed an aligned model its chat-template prefix and
-  harvest the self-generated instruction + response. Live for `--provider ollama` and `vllm`
-  (loopback-only), with an optional `--quality-filter`.
-- **2PL / 3PL eval-cost models** — `soup eval irt-subset --model 2pl|3pl` adds per-item
-  discrimination (and a 3PL guessing floor) on top of the existing 1PL Rasch fit.
-- **Bug fix:** `soup data augment --provider ollama|vllm` no longer crashes with an `ImportError`
-  — it now routes through the shared, SSRF-hardened provider factory and honours `--model` /
-  `--base-url`.
+**v0.71.7 — Eval live runners.** Six probe surfaces that used to emit heuristic / neutral stubs
+now load a real model and run live (opt-in flags; the offline paths stay the default):
+
+- **`soup diagnose --base-model <id> [--adapter <path>] [--dataset d.jsonl]`** — runs all six
+  failure-mode probes (forgetting / refusal / format / mode_collapse / memorization /
+  contamination) against the loaded model instead of emitting neutral OK.
+- **`soup advise --probe-model <id>`** — a live ROI probe: zero/few-shot token-F1 baselines, a
+  short LoRA probe (real held-out-loss drop + wall-clock), and base-model proximity.
+- **`soup tunability --live`** — a real per-candidate LoRA probe (loads each base, trains
+  `--probe-steps` on a held-out slice).
+- **`soup eval capability --live --model <id>`** — invokes lm-eval-harness per task (with
+  `--tasks` / `--limit` / `--device`), isolating per-task failures.
+- **`soup eval behavior --base-model <id> [--adapter <path>]`** — generates pre/post responses on
+  the bundled behaviour battery and scores the live diff.
+- New shared `soup_cli/utils/live_eval.py` holds the model-loading primitives; heavy imports stay
+  lazy. Validated end-to-end on SmolLM2-135M (RTX 3050 4 GB).
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
 

{soup_cli-0.71.6 → soup_cli-0.71.7}/README.md

@@ -49,20 +49,22 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.6 — Synth data & build pipeline go live.** Three deferred stubs are now real:
-
-- **`soup build` materialises** — the dbt-for-SFT DAG no longer just dry-runs. `soup build
-  manifest.yaml --output-dir out/` runs the transforms (`drop_empty` / `lowercase` / `strip` /
-  `dedup_exact` / `identity` built in), rebuilds `table` / `view` models from scratch, and
-  re-transforms only the changed rows for `incremental` models (SQLite-tracked).
-- **`soup data gen-magpie` generates** — feed an aligned model its chat-template prefix and
-  harvest the self-generated instruction + response. Live for `--provider ollama` and `vllm`
-  (loopback-only), with an optional `--quality-filter`.
-- **2PL / 3PL eval-cost models** — `soup eval irt-subset --model 2pl|3pl` adds per-item
-  discrimination (and a 3PL guessing floor) on top of the existing 1PL Rasch fit.
-- **Bug fix:** `soup data augment --provider ollama|vllm` no longer crashes with an `ImportError`
-  — it now routes through the shared, SSRF-hardened provider factory and honours `--model` /
-  `--base-url`.
+**v0.71.7 — Eval live runners.** Six probe surfaces that used to emit heuristic / neutral stubs
+now load a real model and run live (opt-in flags; the offline paths stay the default):
+
+- **`soup diagnose --base-model <id> [--adapter <path>] [--dataset d.jsonl]`** — runs all six
+  failure-mode probes (forgetting / refusal / format / mode_collapse / memorization /
+  contamination) against the loaded model instead of emitting neutral OK.
+- **`soup advise --probe-model <id>`** — a live ROI probe: zero/few-shot token-F1 baselines, a
+  short LoRA probe (real held-out-loss drop + wall-clock), and base-model proximity.
+- **`soup tunability --live`** — a real per-candidate LoRA probe (loads each base, trains
+  `--probe-steps` on a held-out slice).
+- **`soup eval capability --live --model <id>`** — invokes lm-eval-harness per task (with
+  `--tasks` / `--limit` / `--device`), isolating per-task failures.
+- **`soup eval behavior --base-model <id> [--adapter <path>]`** — generates pre/post responses on
+  the bundled behaviour battery and scores the live diff.
+- New shared `soup_cli/utils/live_eval.py` holds the model-loading primitives; heavy imports stay
+  lazy. Validated end-to-end on SmolLM2-135M (RTX 3050 4 GB).
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
 

{soup_cli-0.71.6 → soup_cli-0.71.7}/docs/commands.md

@@ -179,7 +179,8 @@ soup ingest|prune-prompt|ab|data active-sample ... --slack-url <https> | --disco
 soup drift-alarm --reference <jsonl> --live <jsonl> --threshold 0.2  Rolling-KL drift alarm (exit 3 on drift)
 soup drift-alarm ... --slack-url <https> | --discord-url <https>  Optional SSRF-validated webhook on drift detected
 soup tunability --list                                   List built-in candidate-base catalogue
-soup tunability --dataset <jsonl> [--candidates a,b,c]   Probe 8 candidate bases + Pareto frontier report
+soup tunability --dataset <jsonl> [--candidates a,b,c]   Probe candidate bases + Pareto frontier report
+soup tunability --dataset <jsonl> --live [--device cpu]  LIVE per-candidate LoRA probe (loads each repo)
 soup plan --config soup.yaml                             Pre-flight summary + write soup.tfstate
 soup apply --config soup.yaml [--dry-run]                Lock-and-execute; refuses on drift (exit 3)
 soup env lock | status | check                           Hermetic env lockfile + ABI drift detection (exit 3)

{soup_cli-0.71.6 → soup_cli-0.71.7}/docs/evaluation.md

@@ -61,9 +61,13 @@ soup advise data.jsonl --goal "make our chatbot more concise"
 #  Why:        Task is summarization with 120 rows and healthy diversity ...
 #  Flip when:  the prompt-engineering baseline already meets your target ...
 
-# Optional 10-min ROI probe (zero/few-shot + RAG + 100-step LoRA).
+# Optional ROI probe (offline heuristic: zero/few-shot + RAG + LoRA estimate).
 soup advise data.jsonl --goal "summarize my reports" --probe
 
+# LIVE ROI probe (v0.71.7): loads the model for zero/few-shot token-F1, a short
+# LoRA probe, and base-model proximity (held-out logit agreement). Implies --probe.
+soup advise data.jsonl --goal "..." --probe-model HuggingFaceTB/SmolLM2-135M
+
 # Print the rubric / evidence trail of the last verdict.
 soup advise explain
 
@@ -239,9 +243,13 @@ Default threshold 0.2 matches v0.43.0 KL-delta quant-check thresholds. Webhooks
 `soup diagnose` scores six independent failure modes for a trained adapter and renders an OK / MINOR / MAJOR verdict per mode plus an overall headline — same taxonomy as Quant-Lobotomy. Useful for catching adapter regressions that a loss curve cannot distinguish from a healthy run.
 
 ```bash
-# Heuristic neutral report (no model load — runs as a sanity check)
+# Neutral report (no model load — runs as a sanity check)
 soup diagnose my-run-id
 
+# LIVE (v0.71.7): load the model and run all six probes for real
+soup diagnose my-run-id --base-model HuggingFaceTB/SmolLM2-135M \
+    --adapter ./out --dataset train.jsonl --tokenizer HuggingFaceTB/SmolLM2-135M
+
 # Compute scores from a pre-built evidence JSON
 soup diagnose my-run-id --evidence evidence.json --output diag.json
 
@@ -252,6 +260,12 @@ soup diagnose my-run-id --badge diag.svg
 soup diagnose my-run-id --output diag.json --attach-to-registry abc123
 ```
 
+**Live runners (v0.71.7).** With `--base-model` the six probes run against the loaded model
+(+ optional `--adapter` LoRA path, `--dataset` for the forgetting / format / memorization probes,
+`--tokenizer` for a sub-word memorization variant) instead of emitting neutral OK. `refusal` uses
+a built-in probe set; `format` only fires when the dataset's own targets look like JSON;
+`contamination` stays neutral unless a benchmark corpus is supplied. Validated on SmolLM2-135M.
+
 **Six failure-mode probes:**
 
 | Mode | What it catches | Score range |
@@ -429,7 +443,7 @@ soup tunability --dataset ./eval.jsonl --candidates qwen3-0.6b,phi-4-mini --plan
 soup tunability --dataset ./eval.jsonl --probe-steps 100 --output ./tunability.json
 ```
 
-The report is a Pareto frontier over (eval delta from base, train cost, license) — candidates that nothing dominates on both axes survive, so you see a clean shortlist instead of a noisy single-leaderboard score. Live LoRA probe lands in v0.64.1; v0.64.0 ships the schema, Pareto math, and a `probe_fn=` injection point.
+The report is a Pareto frontier over (eval delta from base, train cost, license) — candidates that nothing dominates on both axes survive, so you see a clean shortlist instead of a noisy single-leaderboard score. By default the probe is a deterministic offline heuristic; pass `--live` (v0.71.7) to run a real per-candidate LoRA probe (loads each `repo_id`, trains `--probe-steps` on a held-out-excluded slice, reports the held-out-loss drop). `--device` selects cuda / cpu.
 
 
 ## Eval Depth (`soup eval behavior / capability / checklist / irt-subset`)
@@ -459,6 +473,10 @@ Persist a calibration once and reuse it across runs (v0.71.1): `write_judge_cali
 # Score over-refusal regression on XSTest (operator supplies evidence JSON)
 soup eval behavior my_run --battery xstest --evidence ev.json --output diff.json
 
+# LIVE (v0.71.7): generate pre/post responses on the bundled battery + score the diff
+soup eval behavior my_run --battery xstest \
+    --base-model HuggingFaceTB/SmolLM2-135M --adapter ./out
+
 # Bundled batteries: xstest, harmbench, jailbreakbench, elephant, syceval
 # Harmful prompts ship REDACTED — pull real sets from upstream papers.
 ```
@@ -472,9 +490,13 @@ soup eval capability my_run --suite math --output cap.json   # AIME + MATH-500
 soup eval capability my_run --suite code --output cap.json   # HumanEval+ + SWE-bench-Verified
 soup eval capability my_run --suite fast --output cap.json   # MMLU-Pro + HumanEval+
 soup eval capability my_run --suite full --output cap.json   # all 7 benchmarks
+
+# LIVE (v0.71.7): invoke lm-eval-harness per task against a real model
+soup eval capability my_run --live --model HuggingFaceTB/SmolLM2-135M \
+    --tasks arc_easy --limit 1 --device cpu
 ```
 
-Emits the (benchmark, lm-eval task) manifest; chain into the existing `soup eval benchmark` surface.
+Without `--live` it emits the (benchmark, lm-eval task) manifest; chain into the existing `soup eval benchmark` surface. With `--live --model <id>` (v0.71.7) it runs lm-eval-harness per resolved task — or a `--tasks` override — isolating per-task failures and capping examples with `--limit`.
 
 **CheckList behavioural DSL** — Ribeiro et al. 2020 MFT / INV / DIR tests:
 

{soup_cli-0.71.6 → soup_cli-0.71.7}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.6"
+version = "0.71.7"
 description = "Fine-tune LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.6 → soup_cli-0.71.7}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune LLMs in one command."""
 
-__version__ = "0.71.6"
+__version__ = "0.71.7"

{soup_cli-0.71.6 → soup_cli-0.71.7}/src/soup_cli/commands/_eval_v0650.py

@@ -123,8 +123,24 @@ def register(app: typer.Typer, console: Console) -> None:
             None, "--output", "-o",
             help="Where to write the rendered BehaviorDiffReport JSON.",
         ),
+        base_model: Optional[str] = typer.Option(
+            None, "--base-model",
+            help="Base model id for a LIVE diff (generates pre/post responses).",
+        ),
+        adapter: Optional[str] = typer.Option(
+            None, "--adapter",
+            help="LoRA adapter path for the 'post' model in a live diff.",
+        ),
+        device: Optional[str] = typer.Option(
+            None, "--device", help="Device for the live diff (cuda / cpu).",
+        ),
     ) -> None:
-        """Score a run on a bundled behaviour battery (pre/post diff)."""
+        """Score a run on a bundled behaviour battery (pre/post diff).
+
+        With ``--base-model`` (optionally ``--adapter``) this LIVE-generates
+        pre/post responses on the bundled battery and scores them. Without it,
+        falls back to ``--evidence`` JSON, or a neutral OK report.
+        """
         from soup_cli.utils.behavior_battery import (
             compute_behavior_diff,
             get_battery_spec,
@@ -152,6 +168,34 @@ def register(app: typer.Typer, console: Console) -> None:
             border_style="cyan",
         ))
 
+        if base_model is not None:
+            from soup_cli.utils.behavior_battery import run_behavior_live
+
+            try:
+                report = run_behavior_live(
+                    run_id=run_id,
+                    battery=canonical,
+                    base_model=base_model,
+                    adapter=adapter,
+                    device=device,
+                )
+            except (RuntimeError, ValueError, TypeError, OSError) as exc:
+                console.print(f"[red]Live behaviour diff failed:[/] {escape(str(exc))}")
+                raise typer.Exit(2) from exc
+            table = Table(title=f"Behaviour Diff (live) — {canonical}")
+            table.add_column("Stage", style="bold")
+            table.add_column("Value", justify="right")
+            table.add_column("Verdict")
+            table.add_row("Pre", f"{report.pre.value:.3f}", report.pre.verdict)
+            table.add_row("Post", f"{report.post.value:.3f}", report.post.verdict)
+            table.add_row("Δ", f"{report.delta:+.3f}", report.overall)
+            console.print(table)
+            if output:
+                _write_json_output(report.to_dict(), output, console=console)
+            if report.overall == "MAJOR":
+                raise typer.Exit(2)
+            return
+
         if evidence is None:
             # No evidence: emit neutral OK report (matches v0.56.0 diagnose
             # policy when no probes are supplied).
@@ -214,8 +258,31 @@ def register(app: typer.Typer, console: Console) -> None:
             None, "--output", "-o",
             help="Where to write the rendered CapabilityReport JSON.",
         ),
+        live: bool = typer.Option(
+            False, "--live",
+            help="Run the suite LIVE via lm-eval-harness against --model.",
+        ),
+        model: Optional[str] = typer.Option(
+            None, "--model", "-m",
+            help="HF model id for a live run (required with --live).",
+        ),
+        tasks: Optional[str] = typer.Option(
+            None, "--tasks",
+            help="Comma-separated lm-eval task override (live; bypasses --suite).",
+        ),
+        limit: Optional[int] = typer.Option(
+            None, "--limit",
+            help="Cap eval examples per task (live; use 1-5 for a smoke).",
+        ),
+        device: Optional[str] = typer.Option(
+            None, "--device", help="Device for the live run (cuda / cpu).",
+        ),
     ) -> None:
-        """Run a bundled capability profile (MMLU-Pro / GPQA / AIME / ...)."""
+        """Run a bundled capability profile (MMLU-Pro / GPQA / AIME / ...).
+
+        Without ``--live`` this emits a task manifest (no model load). With
+        ``--live --model <id>`` it invokes lm-eval-harness per task.
+        """
         from soup_cli.utils.capability_suite import (
             list_suites,
             resolve_suite,
@@ -237,6 +304,49 @@ def register(app: typer.Typer, console: Console) -> None:
             )
             raise typer.Exit(2) from exc
 
+        if live:
+            if not model:
+                console.print("[red]--live requires --model <hf-id>.[/]")
+                raise typer.Exit(2)
+            from soup_cli.utils.capability_suite import run_capability_suite
+
+            task_list = (
+                [t.strip() for t in tasks.split(",") if t.strip()] if tasks else None
+            )
+            try:
+                payload = run_capability_suite(
+                    run_id=run_id,
+                    model_id=model,
+                    suite=None if task_list else canonical,
+                    tasks=task_list,
+                    device=device,
+                    limit=limit,
+                )
+            except (RuntimeError, ValueError, TypeError) as exc:
+                console.print(f"[red]Capability run failed:[/] {escape(str(exc))}")
+                raise typer.Exit(2) from exc
+            table = Table(title=f"Capability Suite (live) — {canonical}")
+            table.add_column("Benchmark")
+            table.add_column("Metric")
+            table.add_column("Score", justify="right")
+            for r in payload["results"]:
+                if "error" in r:
+                    table.add_row(
+                        escape(str(r["benchmark"])),
+                        "[red]error[/]",
+                        escape(str(r["error"])),
+                    )
+                else:
+                    table.add_row(
+                        escape(str(r["benchmark"])),
+                        escape(str(r.get("metric", ""))),
+                        f"{r.get('score', float('nan')):.4f}",
+                    )
+            console.print(table)
+            if output:
+                _write_json_output(payload, output, console=console)
+            return
+
         benchmarks = resolve_suite(canonical)
         table = Table(title=f"Capability Suite — {canonical}")
         table.add_column("Benchmark")
@@ -250,8 +360,8 @@ def register(app: typer.Typer, console: Console) -> None:
             "suite": canonical,
             "benchmarks": [{"name": b.name, "task": b.lm_eval_task} for b in benchmarks],
             "note": (
-                "Live lm-eval-harness wiring is operator-driven; "
-                "the listed tasks are pre-validated friendly defaults."
+                "Manifest only — pass --live --model <id> to invoke "
+                "lm-eval-harness against the listed tasks."
             ),
         }
         if output:

{soup_cli-0.71.6 → soup_cli-0.71.7}/src/soup_cli/commands/advise.py

@@ -35,6 +35,7 @@ from soup_cli.utils.advise import (
     compute_dataset_profile,
     format_verdict_rubric,
     load_advise_dataset,
+    measure_base_model_proximity,
     synth_probe_baselines,
     synth_probe_lora_delta,
 )
@@ -217,10 +218,25 @@ def advise_run(
         False,
         "--probe",
         help=(
-            "Also run a 10-minute ROI probe (zero/few-shot + RAG baseline + "
-            "100-step LoRA). Heuristic stubs in v0.54.0; live in v0.54.1."
+            "Also run an ROI probe (zero/few-shot + RAG baseline + N-step LoRA). "
+            "Heuristic by default; pass --probe-model <id> for a LIVE probe that "
+            "loads the model + LoRA-trains on a tiny held-out slice."
         ),
     ),
+    probe_model: Optional[str] = typer.Option(
+        None,
+        "--probe-model",
+        help=(
+            "Base model id for a LIVE probe (e.g. HuggingFaceTB/SmolLM2-135M). "
+            "When set, the probe loads this model, scores zero/few-shot, "
+            "LoRA-trains, and measures base-model proximity. Implies --probe."
+        ),
+    ),
+    probe_device: Optional[str] = typer.Option(
+        None,
+        "--probe-device",
+        help="Device for the live probe (cuda / cpu). Auto-detected when omitted.",
+    ),
     record: bool = typer.Option(
         False,
         "--record",
@@ -246,18 +262,36 @@ def advise_run(
         console.print(f"[red]Dataset error:[/] {escape(str(exc))}")
         raise typer.Exit(1) from exc
 
+    # --probe-model implies --probe.
+    run_probe = probe or probe_model is not None
+
+    # #162 — when a live probe model is supplied, measure base-model proximity
+    # (held-out logit agreement) and fold it into the dataset profile. Best
+    # effort: any failure leaves proximity unmeasured (None).
+    proximity: Optional[float] = None
+    if probe_model is not None:
+        try:
+            proximity = measure_base_model_proximity(
+                rows, model=probe_model, device=probe_device
+            )
+        except (TypeError, ValueError) as exc:
+            console.print(f"[red]Proximity probe failed:[/] {escape(str(exc))}")
+            raise typer.Exit(1) from exc
+
     try:
         task_category = classify_task(rows, goal=goal)
-        profile = compute_dataset_profile(rows)
+        profile = compute_dataset_profile(rows, base_model_proximity=proximity)
     except (TypeError, ValueError) as exc:
         console.print(f"[red]Analysis failed:[/] {escape(str(exc))}")
         raise typer.Exit(1) from exc
 
     roi = ROIEstimate()
-    if probe:
+    if run_probe:
         try:
-            baselines = synth_probe_baselines(rows)
-            sft_delta, wall_clock = synth_probe_lora_delta(rows)
+            baselines = synth_probe_baselines(rows, model=probe_model, device=probe_device)
+            sft_delta, wall_clock = synth_probe_lora_delta(
+                rows, model=probe_model, device=probe_device
+            )
         except (TypeError, ValueError) as exc:
             console.print(f"[red]Probe failed:[/] {escape(str(exc))}")
             raise typer.Exit(1) from exc

{soup_cli-0.71.6 → soup_cli-0.71.7}/src/soup_cli/commands/diagnose.py

@@ -7,12 +7,12 @@ Top-level CLI command (NOT a sub-group) — operators type:
     soup diagnose <run-id> --badge diagnose.svg
     soup diagnose <run-id> --attach-to-registry <id>
 
-The live probe runners (forgetting / refusal / format / mode_collapse /
-memorization / contamination) require a base + adapter model pair; this
-release computes neutral OK scores when no probe evidence is supplied,
-matching the v0.27.0 / v0.50.0 stub-then-live cadence. Operators with a
-SoupConfig + an evidence dict can call ``soup_cli.utils.diagnose.diagnose``
-to produce a real report card.
+Since v0.71.7 (#165) the six probe runners (forgetting / refusal / format /
+mode_collapse / memorization / contamination) run LIVE when ``--base-model``
+is supplied: the model (+ optional ``--adapter`` LoRA path, ``--dataset``,
+``--tokenizer``) is loaded and each probe is fed real generator output via
+``soup_cli.utils.diagnose.live.run_live_diagnose``. Without ``--base-model``
+the command computes scores from ``--evidence`` JSON or defaults to neutral OK.
 """
 
 from __future__ import annotations
@@ -173,6 +173,47 @@ def _attach_to_registry(report: FailureReport, registry_id: str, output: str) ->
         )
 
 
+def _emit_report(
+    report: FailureReport,
+    *,
+    output: Optional[str],
+    badge: Optional[str],
+    attach_to_registry: Optional[str],
+) -> None:
+    """Shared render + optional output/badge/registry-attach (no exit)."""
+    _render_report(report)
+
+    if output:
+        try:
+            write_report(report, output)
+            console.print(f"[green]Wrote[/] {escape(output)}")
+        except (OSError, ValueError) as exc:
+            console.print(
+                f"[red]Error:[/] cannot write --output: "
+                f"{escape(type(exc).__name__)}: {escape(str(exc))}"
+            )
+            raise typer.Exit(code=1) from exc
+
+    if badge:
+        try:
+            svg = render_badge_svg(report)
+            _write_badge(badge, svg)
+            console.print(f"[green]Badge written[/] to {escape(badge)}")
+        except (OSError, ValueError, TypeError) as exc:
+            console.print(
+                f"[red]Error:[/] cannot write --badge: "
+                f"{escape(type(exc).__name__)}: {escape(str(exc))}"
+            )
+            raise typer.Exit(code=1) from exc
+
+    if attach_to_registry and output:
+        _attach_to_registry(report, attach_to_registry, output)
+    elif attach_to_registry and not output:
+        console.print(
+            "[yellow]Warning:[/] --attach-to-registry needs --output (skipped)."
+        )
+
+
 def diagnose(
     run_id: str = typer.Argument(..., help="Registry run id (or any opaque tag)."),
     base: str = typer.Option("", "--base", help="Base model name (informational)."),
@@ -191,13 +232,65 @@ def diagnose(
     attach_to_registry: Optional[str] = typer.Option(
         None, "--attach-to-registry", help="Attach the report to a registry entry id."
     ),
+    base_model: Optional[str] = typer.Option(
+        None,
+        "--base-model",
+        help=(
+            "Base model id to LOAD for a live diagnose run (e.g. "
+            "HuggingFaceTB/SmolLM2-135M). When set, the 6 probes run live "
+            "against the model instead of emitting neutral scores."
+        ),
+    ),
+    dataset: Optional[str] = typer.Option(
+        None,
+        "--dataset",
+        help="Training JSONL for the live forgetting / format / memorization "
+        "probes (must stay under cwd).",
+    ),
+    tokenizer: Optional[str] = typer.Option(
+        None,
+        "--tokenizer",
+        help="Tokenizer id/path for a sub-word memorization probe (live).",
+    ),
+    device: Optional[str] = typer.Option(
+        None, "--device", help="Device for the live run (cuda / cpu)."
+    ),
 ) -> None:
-    """Compute a 6-mode FailureReport for a completed run."""
+    """Compute a 6-mode FailureReport for a completed run.
+
+    With ``--base-model`` the six probes run LIVE against the loaded model
+    (+ optional ``--adapter`` LoRA path, ``--dataset``, ``--tokenizer``).
+    Without it, scores come from ``--evidence`` JSON or default to neutral OK.
+    """
     if not isinstance(run_id, str) or not run_id.strip():
         raise typer.BadParameter("run_id must be a non-empty string")
     if "\x00" in run_id or len(run_id) > 512:
         raise typer.BadParameter("run_id has a null byte or is too long")
 
+    if base_model is not None:
+        from soup_cli.utils.diagnose.live import run_live_diagnose
+
+        try:
+            report = run_live_diagnose(
+                run_id=run_id,
+                base=base_model,
+                adapter=adapter or None,
+                dataset_path=dataset,
+                device=device,
+                tokenizer=tokenizer,
+                soup_version=__version__,
+            )
+        except (ValueError, TypeError, OSError, RuntimeError) as exc:
+            console.print(
+                f"[red]Error:[/] live diagnose failed: "
+                f"{escape(type(exc).__name__)}: {escape(str(exc))}"
+            )
+            raise typer.Exit(code=1) from exc
+        _emit_report(report, output=output, badge=badge, attach_to_registry=attach_to_registry)
+        if report.overall == "MAJOR":
+            raise typer.Exit(code=2)
+        return
+
     scores = {}
     extras = {}
     if evidence_path:
@@ -235,37 +328,7 @@ def diagnose(
         soup_version=__version__,
         extras=extras,
     )
-    _render_report(report)
-
-    if output:
-        try:
-            write_report(report, output)
-            console.print(f"[green]Wrote[/] {escape(output)}")
-        except (OSError, ValueError) as exc:
-            console.print(
-                f"[red]Error:[/] cannot write --output: "
-                f"{escape(type(exc).__name__)}: {escape(str(exc))}"
-            )
-            raise typer.Exit(code=1) from exc
-
-    if badge:
-        try:
-            svg = render_badge_svg(report)
-            _write_badge(badge, svg)
-            console.print(f"[green]Badge written[/] to {escape(badge)}")
-        except (OSError, ValueError, TypeError) as exc:
-            console.print(
-                f"[red]Error:[/] cannot write --badge: "
-                f"{escape(type(exc).__name__)}: {escape(str(exc))}"
-            )
-            raise typer.Exit(code=1) from exc
-
-    if attach_to_registry and output:
-        _attach_to_registry(report, attach_to_registry, output)
-    elif attach_to_registry and not output:
-        console.print(
-            "[yellow]Warning:[/] --attach-to-registry needs --output (skipped)."
-        )
+    _emit_report(report, output=output, badge=badge, attach_to_registry=attach_to_registry)
 
     if report.overall == "MAJOR":
         raise typer.Exit(code=2)