soup-cli 0.71.6__tar.gz → 0.71.8__tar.gz

{soup_cli-0.71.6 → soup_cli-0.71.8}/CHANGELOG.md

@@ -12,6 +12,86 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.8] - 2026-06-03
+
+### Added
+- **Probes & SAE — real weights + live downloads** (closes #215, #216, #217,
+  #218, #219). A new shared `soup_cli/utils/probe_kernel.py` provides the
+  linear-probe math (contrast-pair derivation, apply, flag-rate, verdict bands,
+  operator-supplied weight loading, deterministic synthetic fallback); every
+  heavy import (`numpy` / `torch` / `safetensors`) is lazy.
+- **`soup probe sleeper --weights <w.npz|.npy|.safetensors>`** (#215) — load a
+  real calibrated probe direction instead of the synthetic fallback. Weights are
+  cwd-contained, symlink-rejected, `O_NOFOLLOW`-opened, `allow_pickle=False`,
+  and size-capped. `compute_contrast_probe(positive, negative)` derives a probe
+  from contrast-pair activations.
+- **`soup probe sae-diff <repo> --auto-download`** (#216) — fetch an
+  allowlisted SAE from the HF Hub into `~/.soup/sae-cache/` (validated against
+  `HF_HUB_ALLOWLIST` BEFORE any network call) via a new SSRF-hardened
+  `soup_cli.utils.hubs.snapshot_download` (repo-id shape + home/cwd/tmp cache
+  containment + namespace-pin TOFU gate).
+- **`soup probe truth` / `soup probe harm`** (#217) — TruthfulQA-style honesty
+  and HarmBench-style misuse activation probes (6 bundled bases each, 5% / 20%
+  verdict bands, `--weights` to skip the allowlist with a real probe). The
+  probe pack now ships truth + harm entries per base.
+- **`soup probe interference --measure <eval_suite> --base-model <m> --adapter
+  name=path ...`** (#218) — auto-measure the N×N interference matrix by actually
+  loading the base + each LoRA adapter (PEFT multi-adapter), measuring loss for
+  each adapter alone (diagonal) and each co-loaded pair
+  (`add_weighted_adapter(combination_type="cat")`, off-diagonal). Exit 2 on a
+  MAJOR worst-pair.
+- **`soup train --capture-activations <layer> --capture-prompts <jsonl>`** (#219)
+  — a post-training hook writes an SAE-diff-ready per-token activation snapshot
+  to `<output>/activations/activations.json`. `resolve_layer_module` resolves
+  the same `model.layers.N` path whether or not a LoRA adapter is loaded
+  (PEFT-wrapper fallback).
+
+### Security
+- Probe / SAE / capture file I/O is cwd-contained + `O_NOFOLLOW` (TOCTOU close)
+  + size-capped; SAE weight loads use `allow_pickle=False`. SAE auto-download
+  validates the allowlist before any network call and rejects a glob result
+  that resolves outside the snapshot dir (symlink-escape guard).
+
+### Notes
+- #215 is partial: the operator-supplied / contrast-pair / synthetic paths ship
+  now, but the 6 large-base Anthropic-calibrated probe vectors remain
+  upstream-gated (no public calibrated artifact exists). Documented as a known
+  limitation.
+
+## [0.71.7] - 2026-06-02
+
+### Added
+- **Eval live runners** — six probe surfaces that previously emitted heuristic
+  / neutral stubs now load a real model and run live (closes #161, #162, #208,
+  #211, #212, #165). New shared `soup_cli/utils/live_eval.py` provides the
+  model-loading primitives (generator / multi-generator closures, masked
+  cross-entropy eval-loss, a short-LoRA probe, and held-out logit agreement);
+  every heavy import (`torch` / `transformers` / `peft` / `lm_eval`) is lazy.
+- **`soup advise --probe-model <id>`** — runs a LIVE ROI probe: zero/few-shot
+  token-F1 baselines, a short LoRA probe (relative held-out-loss improvement +
+  real wall-clock), and base-model proximity (held-out logit agreement) folded
+  into the dataset profile. Without `--probe-model`, `--probe` stays the offline
+  heuristic.
+- **`soup tunability --live`** — replaces the offline heuristic with a real
+  per-candidate LoRA probe (loads each `repo_id`, trains `--probe-steps` on a
+  held-out-excluded slice, reports the held-out-loss drop).
+- **`soup eval capability --live --model <id>`** — invokes lm-eval-harness per
+  resolved task (or a `--tasks` override) with `--limit` / `--device`, isolating
+  per-task failures and surfacing a no-metric result as an explicit error.
+- **`soup eval behavior --base-model <id> [--adapter <path>]`** — generates
+  pre/post responses on the bundled behaviour battery and scores the live diff.
+- **`soup diagnose --base-model <id> [--adapter <path>] [--dataset <jsonl>]
+  [--tokenizer <id>]`** — runs all six failure-mode probes (forgetting / refusal
+  / format / mode_collapse / memorization / contamination) live via
+  `soup_cli.utils.diagnose.live.run_live_diagnose`; falls back to neutral OK or
+  `--evidence` JSON when no model is supplied.
+
+### Security
+- The two new JSONL dataset readers (`diagnose.live._load_dataset_rows`,
+  `tunability._load_jsonl_rows`) open with `O_NOFOLLOW` after the cwd-containment
+  check, closing the check→open TOCTOU window (matches the v0.65 / v0.67 reader
+  policy).
+
 ## [0.71.6] - 2026-06-02
 
 ### Added

{soup_cli-0.71.6 → soup_cli-0.71.8}/CONTRIBUTING.md

@@ -120,7 +120,7 @@ src/soup_cli/
   templates/          - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (276 files, 12703 tests)
+tests/                - Test suite (278 files, 12917 tests)
 examples/             - Real-world config examples and datasets
 ```
 

{soup_cli-0.71.6 → soup_cli-0.71.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.6
+Version: 0.71.8
 Summary: Fine-tune LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -183,20 +183,20 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.6 — Synth data & build pipeline go live.** Three deferred stubs are now real:
-
-- **`soup build` materialises** — the dbt-for-SFT DAG no longer just dry-runs. `soup build
-  manifest.yaml --output-dir out/` runs the transforms (`drop_empty` / `lowercase` / `strip` /
-  `dedup_exact` / `identity` built in), rebuilds `table` / `view` models from scratch, and
-  re-transforms only the changed rows for `incremental` models (SQLite-tracked).
-- **`soup data gen-magpie` generates** — feed an aligned model its chat-template prefix and
-  harvest the self-generated instruction + response. Live for `--provider ollama` and `vllm`
-  (loopback-only), with an optional `--quality-filter`.
-- **2PL / 3PL eval-cost models** — `soup eval irt-subset --model 2pl|3pl` adds per-item
-  discrimination (and a 3PL guessing floor) on top of the existing 1PL Rasch fit.
-- **Bug fix:** `soup data augment --provider ollama|vllm` no longer crashes with an `ImportError`
-  — it now routes through the shared, SSRF-hardened provider factory and honours `--model` /
-  `--base-url`.
+**v0.71.8 — Probes & SAE.** The activation-probe surfaces ship real weights, live SAE downloads,
+and an end-to-end capture → diff pipeline:
+
+- **`soup probe truth` / `soup probe harm`** — TruthfulQA-style honesty and HarmBench-style misuse
+  activation probes (6 bundled bases each, 5% / 20% verdict bands). `--weights` loads a real
+  calibrated probe; without it the bundled deterministic fallback is used.
+- **`soup probe sleeper --weights <w.npz|.npy|.safetensors>`** — load a real calibrated sleeper
+  probe direction instead of the synthetic fallback (cwd-contained, `allow_pickle=False`).
+- **`soup probe sae-diff <repo> --auto-download`** — fetch an allowlisted SAE from the HF Hub into
+  `~/.soup/sae-cache/` (validated before any network call) and diff pre/post activations.
+- **`soup probe interference --measure <eval.jsonl> --base-model <m> --adapter a=path ...`** —
+  auto-measure the N×N adapter-interference matrix by actually loading the base + each LoRA adapter.
+- **`soup train --capture-activations <layer> --capture-prompts <jsonl>`** — a post-training hook
+  writes an SAE-diff-ready per-token activation snapshot. Validated end-to-end on SmolLM2-135M.
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
 

{soup_cli-0.71.6 → soup_cli-0.71.8}/README.md

@@ -49,20 +49,20 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.6 — Synth data & build pipeline go live.** Three deferred stubs are now real:
-
-- **`soup build` materialises** — the dbt-for-SFT DAG no longer just dry-runs. `soup build
-  manifest.yaml --output-dir out/` runs the transforms (`drop_empty` / `lowercase` / `strip` /
-  `dedup_exact` / `identity` built in), rebuilds `table` / `view` models from scratch, and
-  re-transforms only the changed rows for `incremental` models (SQLite-tracked).
-- **`soup data gen-magpie` generates** — feed an aligned model its chat-template prefix and
-  harvest the self-generated instruction + response. Live for `--provider ollama` and `vllm`
-  (loopback-only), with an optional `--quality-filter`.
-- **2PL / 3PL eval-cost models** — `soup eval irt-subset --model 2pl|3pl` adds per-item
-  discrimination (and a 3PL guessing floor) on top of the existing 1PL Rasch fit.
-- **Bug fix:** `soup data augment --provider ollama|vllm` no longer crashes with an `ImportError`
-  — it now routes through the shared, SSRF-hardened provider factory and honours `--model` /
-  `--base-url`.
+**v0.71.8 — Probes & SAE.** The activation-probe surfaces ship real weights, live SAE downloads,
+and an end-to-end capture → diff pipeline:
+
+- **`soup probe truth` / `soup probe harm`** — TruthfulQA-style honesty and HarmBench-style misuse
+  activation probes (6 bundled bases each, 5% / 20% verdict bands). `--weights` loads a real
+  calibrated probe; without it the bundled deterministic fallback is used.
+- **`soup probe sleeper --weights <w.npz|.npy|.safetensors>`** — load a real calibrated sleeper
+  probe direction instead of the synthetic fallback (cwd-contained, `allow_pickle=False`).
+- **`soup probe sae-diff <repo> --auto-download`** — fetch an allowlisted SAE from the HF Hub into
+  `~/.soup/sae-cache/` (validated before any network call) and diff pre/post activations.
+- **`soup probe interference --measure <eval.jsonl> --base-model <m> --adapter a=path ...`** —
+  auto-measure the N×N adapter-interference matrix by actually loading the base + each LoRA adapter.
+- **`soup train --capture-activations <layer> --capture-prompts <jsonl>`** — a post-training hook
+  writes an SAE-diff-ready per-token activation snapshot. Validated end-to-end on SmolLM2-135M.
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
 

{soup_cli-0.71.6 → soup_cli-0.71.8}/docs/commands.md

@@ -179,7 +179,8 @@ soup ingest|prune-prompt|ab|data active-sample ... --slack-url <https> | --disco
 soup drift-alarm --reference <jsonl> --live <jsonl> --threshold 0.2  Rolling-KL drift alarm (exit 3 on drift)
 soup drift-alarm ... --slack-url <https> | --discord-url <https>  Optional SSRF-validated webhook on drift detected
 soup tunability --list                                   List built-in candidate-base catalogue
-soup tunability --dataset <jsonl> [--candidates a,b,c]   Probe 8 candidate bases + Pareto frontier report
+soup tunability --dataset <jsonl> [--candidates a,b,c]   Probe candidate bases + Pareto frontier report
+soup tunability --dataset <jsonl> --live [--device cpu]  LIVE per-candidate LoRA probe (loads each repo)
 soup plan --config soup.yaml                             Pre-flight summary + write soup.tfstate
 soup apply --config soup.yaml [--dry-run]                Lock-and-execute; refuses on drift (exit 3)
 soup env lock | status | check                           Hermetic env lockfile + ABI drift detection (exit 3)
@@ -188,10 +189,15 @@ soup completions bash | zsh | fish                       Shell completion script
 soup license-advisor --target b2c|defense|embedded       Recommend license-clean base for deploy target
 soup license-advisor ... --license <id> --mau N          Per-license downstream-risk check (exit 3 on block)
 soup probe sae-diff <sae> <pre.json> <post.json> [--top-k N]  SAE feature diff between pre/post-FT activations (v0.66.0)
-soup probe sleeper <base> [--evidence ev.json] [--output o.json]  Calibrated sleeper-agent defection probe (v0.66.0)
+soup probe sae-diff <repo> <pre.json> <post.json> --auto-download  Fetch an allowlisted SAE into ~/.soup/sae-cache (v0.71.8)
+soup probe sleeper <base> [--evidence ev.json] [--weights w.npz] [--output o.json]  Sleeper-agent defection probe; --weights = real calibrated probe (v0.66.0; v0.71.8)
+soup probe truth <base> [--evidence ev.json] [--weights w.npz] [--output o.json]  TruthfulQA-style honesty probe (v0.71.8)
+soup probe harm <base> [--evidence ev.json] [--weights w.npz] [--output o.json]  HarmBench-style misuse probe (v0.71.8)
 soup probe interference <losses.json> [--output o.json]  Pairwise N×N adapter interference matrix (exit 2 on MAJOR; v0.66.0)
-soup probe pack <base> [--output o.json]      Per-base calibrated probe pack manifest (v0.66.0)
+soup probe interference --measure <eval.jsonl> --base-model <m> --adapter name=path ... [--device cpu]  Auto-measure live interference (v0.71.8)
+soup probe pack <base> [--output o.json]      Per-base calibrated probe pack manifest (v0.66.0; +truth/harm v0.71.8)
 soup probe pack --list                        List bundled probe-pack bases (v0.66.0)
+soup train --capture-activations <layer> --capture-prompts <jsonl>  Post-train SAE-diff-ready per-token activation snapshot (v0.71.8)
 soup adapters blame ... --top-k 50            Live DataInf-style influence runner (v0.66.0, closes #171)
 soup adapters merge ... --strategy cmaes --eval <s> --budget 1h  CMA-ES evolutionary merge — live loop (v0.67.0 schema / v0.71.4 live)
 soup adapters merge ... --canary <suite.json> [--strict-verdict]  Live OK/MINOR/MAJOR canary verdict, exit 2 on MAJOR (v0.71.4)

{soup_cli-0.71.6 → soup_cli-0.71.8}/docs/evaluation.md

@@ -37,16 +37,29 @@ soup adapters blame ./my-adapter --dataset ./train.jsonl --layer q_proj.7 \
 
 # 3. Sleeper-agent defection probe: per-token defection rate via calibrated linear probe
 soup probe sleeper meta-llama/Llama-3-8B --evidence activations.json
+soup probe sleeper my/model --weights probe.npz --evidence activations.json  # real calibrated probe (v0.71.8)
+
+# 3b. Honesty + misuse probes (v0.71.8) — same shape, 5% / 20% verdict bands
+soup probe truth meta-llama/Llama-3-8B --evidence activations.json
+soup probe harm  meta-llama/Llama-3-8B --evidence activations.json
 
 # 4. Pairwise adapter interference matrix: which pairs can't be deployed together?
 soup probe interference losses.json    # exit 2 if worst-pair score ≥ 20%
+# v0.71.8: auto-measure the matrix live instead of supplying losses.json
+soup probe interference --measure eval.jsonl --base-model meta-llama/Llama-3-8B \
+    --adapter a=./adapter-a --adapter b=./adapter-b --device cpu
 
-# 5. Probe pack: list/assemble calibrated probes per base
+# 5. Probe pack: list/assemble calibrated probes per base (sleeper + truth + harm per base)
 soup probe pack --list                 # list bundled bases
 soup probe pack meta-llama/Llama-3-8B  # render the per-base manifest
+
+# 6. SAE auto-download + capture pipeline (v0.71.8)
+soup train --config soup.yaml --capture-activations model.layers.5 \
+    --capture-prompts probes.jsonl    # writes <output>/activations/activations.json
+soup probe sae-diff google/gemma-scope-2b-pt-res pre.json post.json --auto-download
 ```
 
-Every probe uses the OK / MINOR / MAJOR taxonomy from v0.26 (Quant-Lobotomy) / v0.56 (Diagnose) / v0.65 (Eval Depth). Sleeper + interference exit 2 on MAJOR for CI gating. The blame runner closes the v0.57 `NotImplementedError` stub via a DataInf-style influence approximation: `cos(grad_row, grad_probe) × |grad_row|`. Operators supply a `probe_fn` returning `(row_grads, probe_grad)`, or the runner falls back to a deterministic synthetic probe so the surface always returns a real `BlameResult` (no exception leaks). SAE feature diff is pure-numpy; the safetensors loader is `O_NOFOLLOW`-protected (TOCTOU defence — closes the symlink swap window between containment check and read).
+Every probe uses the OK / MINOR / MAJOR taxonomy from v0.26 (Quant-Lobotomy) / v0.56 (Diagnose) / v0.65 (Eval Depth). Sleeper / truth / harm / interference exit 2 on MAJOR for CI gating. **v0.71.8** ships real probe weights: `--weights <w.npz|.npy|.safetensors>` loads a calibrated direction (cwd-contained, `O_NOFOLLOW`, `allow_pickle=False`, size-capped); `compute_contrast_probe(positive, negative)` derives one from contrast-pair activations; the bundled specs fall back to a deterministic synthetic seed (the large-base Anthropic-calibrated vectors remain upstream-gated). `soup probe interference --measure` loads the base + each LoRA adapter via PEFT and measures loss per adapter alone (diagonal) and per co-loaded pair (`add_weighted_adapter(combination_type="cat")`, off-diagonal). `soup train --capture-activations` writes an SAE-diff-ready per-token snapshot (the `model.layers.N` path resolves whether or not a LoRA adapter is loaded). The blame runner closes the v0.57 `NotImplementedError` stub via a DataInf-style influence approximation: `cos(grad_row, grad_probe) × |grad_row|`. Operators supply a `probe_fn` returning `(row_grads, probe_grad)`, or the runner falls back to a deterministic synthetic probe so the surface always returns a real `BlameResult` (no exception leaks). SAE feature diff is pure-numpy; the safetensors loader is `O_NOFOLLOW`-protected (TOCTOU defence — closes the symlink swap window between containment check and read); `--auto-download` validates the `HF_HUB_ALLOWLIST` before any network call and rejects a glob result escaping the snapshot dir.
 
 
 ## Pre-flight Decision (`soup advise`)
@@ -61,9 +74,13 @@ soup advise data.jsonl --goal "make our chatbot more concise"
 #  Why:        Task is summarization with 120 rows and healthy diversity ...
 #  Flip when:  the prompt-engineering baseline already meets your target ...
 
-# Optional 10-min ROI probe (zero/few-shot + RAG + 100-step LoRA).
+# Optional ROI probe (offline heuristic: zero/few-shot + RAG + LoRA estimate).
 soup advise data.jsonl --goal "summarize my reports" --probe
 
+# LIVE ROI probe (v0.71.7): loads the model for zero/few-shot token-F1, a short
+# LoRA probe, and base-model proximity (held-out logit agreement). Implies --probe.
+soup advise data.jsonl --goal "..." --probe-model HuggingFaceTB/SmolLM2-135M
+
 # Print the rubric / evidence trail of the last verdict.
 soup advise explain
 
@@ -239,9 +256,13 @@ Default threshold 0.2 matches v0.43.0 KL-delta quant-check thresholds. Webhooks
 `soup diagnose` scores six independent failure modes for a trained adapter and renders an OK / MINOR / MAJOR verdict per mode plus an overall headline — same taxonomy as Quant-Lobotomy. Useful for catching adapter regressions that a loss curve cannot distinguish from a healthy run.
 
 ```bash
-# Heuristic neutral report (no model load — runs as a sanity check)
+# Neutral report (no model load — runs as a sanity check)
 soup diagnose my-run-id
 
+# LIVE (v0.71.7): load the model and run all six probes for real
+soup diagnose my-run-id --base-model HuggingFaceTB/SmolLM2-135M \
+    --adapter ./out --dataset train.jsonl --tokenizer HuggingFaceTB/SmolLM2-135M
+
 # Compute scores from a pre-built evidence JSON
 soup diagnose my-run-id --evidence evidence.json --output diag.json
 
@@ -252,6 +273,12 @@ soup diagnose my-run-id --badge diag.svg
 soup diagnose my-run-id --output diag.json --attach-to-registry abc123
 ```
 
+**Live runners (v0.71.7).** With `--base-model` the six probes run against the loaded model
+(+ optional `--adapter` LoRA path, `--dataset` for the forgetting / format / memorization probes,
+`--tokenizer` for a sub-word memorization variant) instead of emitting neutral OK. `refusal` uses
+a built-in probe set; `format` only fires when the dataset's own targets look like JSON;
+`contamination` stays neutral unless a benchmark corpus is supplied. Validated on SmolLM2-135M.
+
 **Six failure-mode probes:**
 
 | Mode | What it catches | Score range |
@@ -429,7 +456,7 @@ soup tunability --dataset ./eval.jsonl --candidates qwen3-0.6b,phi-4-mini --plan
 soup tunability --dataset ./eval.jsonl --probe-steps 100 --output ./tunability.json
 ```
 
-The report is a Pareto frontier over (eval delta from base, train cost, license) — candidates that nothing dominates on both axes survive, so you see a clean shortlist instead of a noisy single-leaderboard score. Live LoRA probe lands in v0.64.1; v0.64.0 ships the schema, Pareto math, and a `probe_fn=` injection point.
+The report is a Pareto frontier over (eval delta from base, train cost, license) — candidates that nothing dominates on both axes survive, so you see a clean shortlist instead of a noisy single-leaderboard score. By default the probe is a deterministic offline heuristic; pass `--live` (v0.71.7) to run a real per-candidate LoRA probe (loads each `repo_id`, trains `--probe-steps` on a held-out-excluded slice, reports the held-out-loss drop). `--device` selects cuda / cpu.
 
 
 ## Eval Depth (`soup eval behavior / capability / checklist / irt-subset`)
@@ -459,6 +486,10 @@ Persist a calibration once and reuse it across runs (v0.71.1): `write_judge_cali
 # Score over-refusal regression on XSTest (operator supplies evidence JSON)
 soup eval behavior my_run --battery xstest --evidence ev.json --output diff.json
 
+# LIVE (v0.71.7): generate pre/post responses on the bundled battery + score the diff
+soup eval behavior my_run --battery xstest \
+    --base-model HuggingFaceTB/SmolLM2-135M --adapter ./out
+
 # Bundled batteries: xstest, harmbench, jailbreakbench, elephant, syceval
 # Harmful prompts ship REDACTED — pull real sets from upstream papers.
 ```
@@ -472,9 +503,13 @@ soup eval capability my_run --suite math --output cap.json   # AIME + MATH-500
 soup eval capability my_run --suite code --output cap.json   # HumanEval+ + SWE-bench-Verified
 soup eval capability my_run --suite fast --output cap.json   # MMLU-Pro + HumanEval+
 soup eval capability my_run --suite full --output cap.json   # all 7 benchmarks
+
+# LIVE (v0.71.7): invoke lm-eval-harness per task against a real model
+soup eval capability my_run --live --model HuggingFaceTB/SmolLM2-135M \
+    --tasks arc_easy --limit 1 --device cpu
 ```
 
-Emits the (benchmark, lm-eval task) manifest; chain into the existing `soup eval benchmark` surface.
+Without `--live` it emits the (benchmark, lm-eval task) manifest; chain into the existing `soup eval benchmark` surface. With `--live --model <id>` (v0.71.7) it runs lm-eval-harness per resolved task — or a `--tasks` override — isolating per-task failures and capping examples with `--limit`.
 
 **CheckList behavioural DSL** — Ribeiro et al. 2020 MFT / INV / DIR tests:
 

{soup_cli-0.71.6 → soup_cli-0.71.8}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.6"
+version = "0.71.8"
 description = "Fine-tune LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.6 → soup_cli-0.71.8}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune LLMs in one command."""
 
-__version__ = "0.71.6"
+__version__ = "0.71.8"

{soup_cli-0.71.6 → soup_cli-0.71.8}/src/soup_cli/cli.py

@@ -410,8 +410,9 @@ app.add_typer(
     _probe_cmd.app,
     name="probe",
     help=(
-        "Activation probes: sleeper-agent defection / pairwise interference "
-        "/ SAE feature diff / probe pack (v0.66.0)."
+        "Activation probes: sleeper-agent defection / honesty / misuse / "
+        "pairwise interference / SAE feature diff / probe pack (v0.66.0, "
+        "truth+harm v0.71.8)."
     ),
 )
 

{soup_cli-0.71.6 → soup_cli-0.71.8}/src/soup_cli/commands/_eval_v0650.py

@@ -123,8 +123,24 @@ def register(app: typer.Typer, console: Console) -> None:
             None, "--output", "-o",
             help="Where to write the rendered BehaviorDiffReport JSON.",
         ),
+        base_model: Optional[str] = typer.Option(
+            None, "--base-model",
+            help="Base model id for a LIVE diff (generates pre/post responses).",
+        ),
+        adapter: Optional[str] = typer.Option(
+            None, "--adapter",
+            help="LoRA adapter path for the 'post' model in a live diff.",
+        ),
+        device: Optional[str] = typer.Option(
+            None, "--device", help="Device for the live diff (cuda / cpu).",
+        ),
     ) -> None:
-        """Score a run on a bundled behaviour battery (pre/post diff)."""
+        """Score a run on a bundled behaviour battery (pre/post diff).
+
+        With ``--base-model`` (optionally ``--adapter``) this LIVE-generates
+        pre/post responses on the bundled battery and scores them. Without it,
+        falls back to ``--evidence`` JSON, or a neutral OK report.
+        """
         from soup_cli.utils.behavior_battery import (
             compute_behavior_diff,
             get_battery_spec,
@@ -152,6 +168,34 @@ def register(app: typer.Typer, console: Console) -> None:
             border_style="cyan",
         ))
 
+        if base_model is not None:
+            from soup_cli.utils.behavior_battery import run_behavior_live
+
+            try:
+                report = run_behavior_live(
+                    run_id=run_id,
+                    battery=canonical,
+                    base_model=base_model,
+                    adapter=adapter,
+                    device=device,
+                )
+            except (RuntimeError, ValueError, TypeError, OSError) as exc:
+                console.print(f"[red]Live behaviour diff failed:[/] {escape(str(exc))}")
+                raise typer.Exit(2) from exc
+            table = Table(title=f"Behaviour Diff (live) — {canonical}")
+            table.add_column("Stage", style="bold")
+            table.add_column("Value", justify="right")
+            table.add_column("Verdict")
+            table.add_row("Pre", f"{report.pre.value:.3f}", report.pre.verdict)
+            table.add_row("Post", f"{report.post.value:.3f}", report.post.verdict)
+            table.add_row("Δ", f"{report.delta:+.3f}", report.overall)
+            console.print(table)
+            if output:
+                _write_json_output(report.to_dict(), output, console=console)
+            if report.overall == "MAJOR":
+                raise typer.Exit(2)
+            return
+
         if evidence is None:
             # No evidence: emit neutral OK report (matches v0.56.0 diagnose
             # policy when no probes are supplied).
@@ -214,8 +258,31 @@ def register(app: typer.Typer, console: Console) -> None:
             None, "--output", "-o",
             help="Where to write the rendered CapabilityReport JSON.",
         ),
+        live: bool = typer.Option(
+            False, "--live",
+            help="Run the suite LIVE via lm-eval-harness against --model.",
+        ),
+        model: Optional[str] = typer.Option(
+            None, "--model", "-m",
+            help="HF model id for a live run (required with --live).",
+        ),
+        tasks: Optional[str] = typer.Option(
+            None, "--tasks",
+            help="Comma-separated lm-eval task override (live; bypasses --suite).",
+        ),
+        limit: Optional[int] = typer.Option(
+            None, "--limit",
+            help="Cap eval examples per task (live; use 1-5 for a smoke).",
+        ),
+        device: Optional[str] = typer.Option(
+            None, "--device", help="Device for the live run (cuda / cpu).",
+        ),
     ) -> None:
-        """Run a bundled capability profile (MMLU-Pro / GPQA / AIME / ...)."""
+        """Run a bundled capability profile (MMLU-Pro / GPQA / AIME / ...).
+
+        Without ``--live`` this emits a task manifest (no model load). With
+        ``--live --model <id>`` it invokes lm-eval-harness per task.
+        """
         from soup_cli.utils.capability_suite import (
             list_suites,
             resolve_suite,
@@ -237,6 +304,49 @@ def register(app: typer.Typer, console: Console) -> None:
             )
             raise typer.Exit(2) from exc
 
+        if live:
+            if not model:
+                console.print("[red]--live requires --model <hf-id>.[/]")
+                raise typer.Exit(2)
+            from soup_cli.utils.capability_suite import run_capability_suite
+
+            task_list = (
+                [t.strip() for t in tasks.split(",") if t.strip()] if tasks else None
+            )
+            try:
+                payload = run_capability_suite(
+                    run_id=run_id,
+                    model_id=model,
+                    suite=None if task_list else canonical,
+                    tasks=task_list,
+                    device=device,
+                    limit=limit,
+                )
+            except (RuntimeError, ValueError, TypeError) as exc:
+                console.print(f"[red]Capability run failed:[/] {escape(str(exc))}")
+                raise typer.Exit(2) from exc
+            table = Table(title=f"Capability Suite (live) — {canonical}")
+            table.add_column("Benchmark")
+            table.add_column("Metric")
+            table.add_column("Score", justify="right")
+            for r in payload["results"]:
+                if "error" in r:
+                    table.add_row(
+                        escape(str(r["benchmark"])),
+                        "[red]error[/]",
+                        escape(str(r["error"])),
+                    )
+                else:
+                    table.add_row(
+                        escape(str(r["benchmark"])),
+                        escape(str(r.get("metric", ""))),
+                        f"{r.get('score', float('nan')):.4f}",
+                    )
+            console.print(table)
+            if output:
+                _write_json_output(payload, output, console=console)
+            return
+
         benchmarks = resolve_suite(canonical)
         table = Table(title=f"Capability Suite — {canonical}")
         table.add_column("Benchmark")
@@ -250,8 +360,8 @@ def register(app: typer.Typer, console: Console) -> None:
             "suite": canonical,
             "benchmarks": [{"name": b.name, "task": b.lm_eval_task} for b in benchmarks],
             "note": (
-                "Live lm-eval-harness wiring is operator-driven; "
-                "the listed tasks are pre-validated friendly defaults."
+                "Manifest only — pass --live --model <id> to invoke "
+                "lm-eval-harness against the listed tasks."
             ),
         }
         if output:

{soup_cli-0.71.6 → soup_cli-0.71.8}/src/soup_cli/commands/advise.py

@@ -35,6 +35,7 @@ from soup_cli.utils.advise import (
     compute_dataset_profile,
     format_verdict_rubric,
     load_advise_dataset,
+    measure_base_model_proximity,
     synth_probe_baselines,
     synth_probe_lora_delta,
 )
@@ -217,10 +218,25 @@ def advise_run(
         False,
         "--probe",
         help=(
-            "Also run a 10-minute ROI probe (zero/few-shot + RAG baseline + "
-            "100-step LoRA). Heuristic stubs in v0.54.0; live in v0.54.1."
+            "Also run an ROI probe (zero/few-shot + RAG baseline + N-step LoRA). "
+            "Heuristic by default; pass --probe-model <id> for a LIVE probe that "
+            "loads the model + LoRA-trains on a tiny held-out slice."
         ),
     ),
+    probe_model: Optional[str] = typer.Option(
+        None,
+        "--probe-model",
+        help=(
+            "Base model id for a LIVE probe (e.g. HuggingFaceTB/SmolLM2-135M). "
+            "When set, the probe loads this model, scores zero/few-shot, "
+            "LoRA-trains, and measures base-model proximity. Implies --probe."
+        ),
+    ),
+    probe_device: Optional[str] = typer.Option(
+        None,
+        "--probe-device",
+        help="Device for the live probe (cuda / cpu). Auto-detected when omitted.",
+    ),
     record: bool = typer.Option(
         False,
         "--record",
@@ -246,18 +262,36 @@ def advise_run(
         console.print(f"[red]Dataset error:[/] {escape(str(exc))}")
         raise typer.Exit(1) from exc
 
+    # --probe-model implies --probe.
+    run_probe = probe or probe_model is not None
+
+    # #162 — when a live probe model is supplied, measure base-model proximity
+    # (held-out logit agreement) and fold it into the dataset profile. Best
+    # effort: any failure leaves proximity unmeasured (None).
+    proximity: Optional[float] = None
+    if probe_model is not None:
+        try:
+            proximity = measure_base_model_proximity(
+                rows, model=probe_model, device=probe_device
+            )
+        except (TypeError, ValueError) as exc:
+            console.print(f"[red]Proximity probe failed:[/] {escape(str(exc))}")
+            raise typer.Exit(1) from exc
+
     try:
         task_category = classify_task(rows, goal=goal)
-        profile = compute_dataset_profile(rows)
+        profile = compute_dataset_profile(rows, base_model_proximity=proximity)
     except (TypeError, ValueError) as exc:
         console.print(f"[red]Analysis failed:[/] {escape(str(exc))}")
         raise typer.Exit(1) from exc
 
     roi = ROIEstimate()
-    if probe:
+    if run_probe:
         try:
-            baselines = synth_probe_baselines(rows)
-            sft_delta, wall_clock = synth_probe_lora_delta(rows)
+            baselines = synth_probe_baselines(rows, model=probe_model, device=probe_device)
+            sft_delta, wall_clock = synth_probe_lora_delta(
+                rows, model=probe_model, device=probe_device
+            )
         except (TypeError, ValueError) as exc:
             console.print(f"[red]Probe failed:[/] {escape(str(exc))}")
             raise typer.Exit(1) from exc