soup-cli 0.71.7__tar.gz → 0.71.8__tar.gz

{soup_cli-0.71.7 → soup_cli-0.71.8}/CHANGELOG.md

@@ -12,6 +12,52 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.8] - 2026-06-03
+
+### Added
+- **Probes & SAE — real weights + live downloads** (closes #215, #216, #217,
+  #218, #219). A new shared `soup_cli/utils/probe_kernel.py` provides the
+  linear-probe math (contrast-pair derivation, apply, flag-rate, verdict bands,
+  operator-supplied weight loading, deterministic synthetic fallback); every
+  heavy import (`numpy` / `torch` / `safetensors`) is lazy.
+- **`soup probe sleeper --weights <w.npz|.npy|.safetensors>`** (#215) — load a
+  real calibrated probe direction instead of the synthetic fallback. Weights are
+  cwd-contained, symlink-rejected, `O_NOFOLLOW`-opened, `allow_pickle=False`,
+  and size-capped. `compute_contrast_probe(positive, negative)` derives a probe
+  from contrast-pair activations.
+- **`soup probe sae-diff <repo> --auto-download`** (#216) — fetch an
+  allowlisted SAE from the HF Hub into `~/.soup/sae-cache/` (validated against
+  `HF_HUB_ALLOWLIST` BEFORE any network call) via a new SSRF-hardened
+  `soup_cli.utils.hubs.snapshot_download` (repo-id shape + home/cwd/tmp cache
+  containment + namespace-pin TOFU gate).
+- **`soup probe truth` / `soup probe harm`** (#217) — TruthfulQA-style honesty
+  and HarmBench-style misuse activation probes (6 bundled bases each, 5% / 20%
+  verdict bands, `--weights` to skip the allowlist with a real probe). The
+  probe pack now ships truth + harm entries per base.
+- **`soup probe interference --measure <eval_suite> --base-model <m> --adapter
+  name=path ...`** (#218) — auto-measure the N×N interference matrix by actually
+  loading the base + each LoRA adapter (PEFT multi-adapter), measuring loss for
+  each adapter alone (diagonal) and each co-loaded pair
+  (`add_weighted_adapter(combination_type="cat")`, off-diagonal). Exit 2 on a
+  MAJOR worst-pair.
+- **`soup train --capture-activations <layer> --capture-prompts <jsonl>`** (#219)
+  — a post-training hook writes an SAE-diff-ready per-token activation snapshot
+  to `<output>/activations/activations.json`. `resolve_layer_module` resolves
+  the same `model.layers.N` path whether or not a LoRA adapter is loaded
+  (PEFT-wrapper fallback).
+
+### Security
+- Probe / SAE / capture file I/O is cwd-contained + `O_NOFOLLOW` (TOCTOU close)
+  + size-capped; SAE weight loads use `allow_pickle=False`. SAE auto-download
+  validates the allowlist before any network call and rejects a glob result
+  that resolves outside the snapshot dir (symlink-escape guard).
+
+### Notes
+- #215 is partial: the operator-supplied / contrast-pair / synthetic paths ship
+  now, but the 6 large-base Anthropic-calibrated probe vectors remain
+  upstream-gated (no public calibrated artifact exists). Documented as a known
+  limitation.
+
 ## [0.71.7] - 2026-06-02
 
 ### Added

{soup_cli-0.71.7 → soup_cli-0.71.8}/CONTRIBUTING.md

@@ -120,7 +120,7 @@ src/soup_cli/
   templates/          - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (277 files, 12771 tests)
+tests/                - Test suite (278 files, 12917 tests)
 examples/             - Real-world config examples and datasets
 ```
 

{soup_cli-0.71.7 → soup_cli-0.71.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.7
+Version: 0.71.8
 Summary: Fine-tune LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -183,22 +183,20 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.7 — Eval live runners.** Six probe surfaces that used to emit heuristic / neutral stubs
-now load a real model and run live (opt-in flags; the offline paths stay the default):
-
-- **`soup diagnose --base-model <id> [--adapter <path>] [--dataset d.jsonl]`** — runs all six
-  failure-mode probes (forgetting / refusal / format / mode_collapse / memorization /
-  contamination) against the loaded model instead of emitting neutral OK.
-- **`soup advise --probe-model <id>`** — a live ROI probe: zero/few-shot token-F1 baselines, a
-  short LoRA probe (real held-out-loss drop + wall-clock), and base-model proximity.
-- **`soup tunability --live`** — a real per-candidate LoRA probe (loads each base, trains
-  `--probe-steps` on a held-out slice).
-- **`soup eval capability --live --model <id>`** — invokes lm-eval-harness per task (with
-  `--tasks` / `--limit` / `--device`), isolating per-task failures.
-- **`soup eval behavior --base-model <id> [--adapter <path>]`** — generates pre/post responses on
-  the bundled behaviour battery and scores the live diff.
-- New shared `soup_cli/utils/live_eval.py` holds the model-loading primitives; heavy imports stay
-  lazy. Validated end-to-end on SmolLM2-135M (RTX 3050 4 GB).
+**v0.71.8 — Probes & SAE.** The activation-probe surfaces ship real weights, live SAE downloads,
+and an end-to-end capture → diff pipeline:
+
+- **`soup probe truth` / `soup probe harm`** — TruthfulQA-style honesty and HarmBench-style misuse
+  activation probes (6 bundled bases each, 5% / 20% verdict bands). `--weights` loads a real
+  calibrated probe; without it the bundled deterministic fallback is used.
+- **`soup probe sleeper --weights <w.npz|.npy|.safetensors>`** — load a real calibrated sleeper
+  probe direction instead of the synthetic fallback (cwd-contained, `allow_pickle=False`).
+- **`soup probe sae-diff <repo> --auto-download`** — fetch an allowlisted SAE from the HF Hub into
+  `~/.soup/sae-cache/` (validated before any network call) and diff pre/post activations.
+- **`soup probe interference --measure <eval.jsonl> --base-model <m> --adapter a=path ...`** —
+  auto-measure the N×N adapter-interference matrix by actually loading the base + each LoRA adapter.
+- **`soup train --capture-activations <layer> --capture-prompts <jsonl>`** — a post-training hook
+  writes an SAE-diff-ready per-token activation snapshot. Validated end-to-end on SmolLM2-135M.
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
 

{soup_cli-0.71.7 → soup_cli-0.71.8}/README.md

@@ -49,22 +49,20 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.7 — Eval live runners.** Six probe surfaces that used to emit heuristic / neutral stubs
-now load a real model and run live (opt-in flags; the offline paths stay the default):
-
-- **`soup diagnose --base-model <id> [--adapter <path>] [--dataset d.jsonl]`** — runs all six
-  failure-mode probes (forgetting / refusal / format / mode_collapse / memorization /
-  contamination) against the loaded model instead of emitting neutral OK.
-- **`soup advise --probe-model <id>`** — a live ROI probe: zero/few-shot token-F1 baselines, a
-  short LoRA probe (real held-out-loss drop + wall-clock), and base-model proximity.
-- **`soup tunability --live`** — a real per-candidate LoRA probe (loads each base, trains
-  `--probe-steps` on a held-out slice).
-- **`soup eval capability --live --model <id>`** — invokes lm-eval-harness per task (with
-  `--tasks` / `--limit` / `--device`), isolating per-task failures.
-- **`soup eval behavior --base-model <id> [--adapter <path>]`** — generates pre/post responses on
-  the bundled behaviour battery and scores the live diff.
-- New shared `soup_cli/utils/live_eval.py` holds the model-loading primitives; heavy imports stay
-  lazy. Validated end-to-end on SmolLM2-135M (RTX 3050 4 GB).
+**v0.71.8 — Probes & SAE.** The activation-probe surfaces ship real weights, live SAE downloads,
+and an end-to-end capture → diff pipeline:
+
+- **`soup probe truth` / `soup probe harm`** — TruthfulQA-style honesty and HarmBench-style misuse
+  activation probes (6 bundled bases each, 5% / 20% verdict bands). `--weights` loads a real
+  calibrated probe; without it the bundled deterministic fallback is used.
+- **`soup probe sleeper --weights <w.npz|.npy|.safetensors>`** — load a real calibrated sleeper
+  probe direction instead of the synthetic fallback (cwd-contained, `allow_pickle=False`).
+- **`soup probe sae-diff <repo> --auto-download`** — fetch an allowlisted SAE from the HF Hub into
+  `~/.soup/sae-cache/` (validated before any network call) and diff pre/post activations.
+- **`soup probe interference --measure <eval.jsonl> --base-model <m> --adapter a=path ...`** —
+  auto-measure the N×N adapter-interference matrix by actually loading the base + each LoRA adapter.
+- **`soup train --capture-activations <layer> --capture-prompts <jsonl>`** — a post-training hook
+  writes an SAE-diff-ready per-token activation snapshot. Validated end-to-end on SmolLM2-135M.
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
 

{soup_cli-0.71.7 → soup_cli-0.71.8}/docs/commands.md

@@ -189,10 +189,15 @@ soup completions bash | zsh | fish                       Shell completion script
 soup license-advisor --target b2c|defense|embedded       Recommend license-clean base for deploy target
 soup license-advisor ... --license <id> --mau N          Per-license downstream-risk check (exit 3 on block)
 soup probe sae-diff <sae> <pre.json> <post.json> [--top-k N]  SAE feature diff between pre/post-FT activations (v0.66.0)
-soup probe sleeper <base> [--evidence ev.json] [--output o.json]  Calibrated sleeper-agent defection probe (v0.66.0)
+soup probe sae-diff <repo> <pre.json> <post.json> --auto-download  Fetch an allowlisted SAE into ~/.soup/sae-cache (v0.71.8)
+soup probe sleeper <base> [--evidence ev.json] [--weights w.npz] [--output o.json]  Sleeper-agent defection probe; --weights = real calibrated probe (v0.66.0; v0.71.8)
+soup probe truth <base> [--evidence ev.json] [--weights w.npz] [--output o.json]  TruthfulQA-style honesty probe (v0.71.8)
+soup probe harm <base> [--evidence ev.json] [--weights w.npz] [--output o.json]  HarmBench-style misuse probe (v0.71.8)
 soup probe interference <losses.json> [--output o.json]  Pairwise N×N adapter interference matrix (exit 2 on MAJOR; v0.66.0)
-soup probe pack <base> [--output o.json]      Per-base calibrated probe pack manifest (v0.66.0)
+soup probe interference --measure <eval.jsonl> --base-model <m> --adapter name=path ... [--device cpu]  Auto-measure live interference (v0.71.8)
+soup probe pack <base> [--output o.json]      Per-base calibrated probe pack manifest (v0.66.0; +truth/harm v0.71.8)
 soup probe pack --list                        List bundled probe-pack bases (v0.66.0)
+soup train --capture-activations <layer> --capture-prompts <jsonl>  Post-train SAE-diff-ready per-token activation snapshot (v0.71.8)
 soup adapters blame ... --top-k 50            Live DataInf-style influence runner (v0.66.0, closes #171)
 soup adapters merge ... --strategy cmaes --eval <s> --budget 1h  CMA-ES evolutionary merge — live loop (v0.67.0 schema / v0.71.4 live)
 soup adapters merge ... --canary <suite.json> [--strict-verdict]  Live OK/MINOR/MAJOR canary verdict, exit 2 on MAJOR (v0.71.4)

{soup_cli-0.71.7 → soup_cli-0.71.8}/docs/evaluation.md

@@ -37,16 +37,29 @@ soup adapters blame ./my-adapter --dataset ./train.jsonl --layer q_proj.7 \
 
 # 3. Sleeper-agent defection probe: per-token defection rate via calibrated linear probe
 soup probe sleeper meta-llama/Llama-3-8B --evidence activations.json
+soup probe sleeper my/model --weights probe.npz --evidence activations.json  # real calibrated probe (v0.71.8)
+
+# 3b. Honesty + misuse probes (v0.71.8) — same shape, 5% / 20% verdict bands
+soup probe truth meta-llama/Llama-3-8B --evidence activations.json
+soup probe harm  meta-llama/Llama-3-8B --evidence activations.json
 
 # 4. Pairwise adapter interference matrix: which pairs can't be deployed together?
 soup probe interference losses.json    # exit 2 if worst-pair score ≥ 20%
+# v0.71.8: auto-measure the matrix live instead of supplying losses.json
+soup probe interference --measure eval.jsonl --base-model meta-llama/Llama-3-8B \
+    --adapter a=./adapter-a --adapter b=./adapter-b --device cpu
 
-# 5. Probe pack: list/assemble calibrated probes per base
+# 5. Probe pack: list/assemble calibrated probes per base (sleeper + truth + harm per base)
 soup probe pack --list                 # list bundled bases
 soup probe pack meta-llama/Llama-3-8B  # render the per-base manifest
+
+# 6. SAE auto-download + capture pipeline (v0.71.8)
+soup train --config soup.yaml --capture-activations model.layers.5 \
+    --capture-prompts probes.jsonl    # writes <output>/activations/activations.json
+soup probe sae-diff google/gemma-scope-2b-pt-res pre.json post.json --auto-download
 ```
 
-Every probe uses the OK / MINOR / MAJOR taxonomy from v0.26 (Quant-Lobotomy) / v0.56 (Diagnose) / v0.65 (Eval Depth). Sleeper + interference exit 2 on MAJOR for CI gating. The blame runner closes the v0.57 `NotImplementedError` stub via a DataInf-style influence approximation: `cos(grad_row, grad_probe) × |grad_row|`. Operators supply a `probe_fn` returning `(row_grads, probe_grad)`, or the runner falls back to a deterministic synthetic probe so the surface always returns a real `BlameResult` (no exception leaks). SAE feature diff is pure-numpy; the safetensors loader is `O_NOFOLLOW`-protected (TOCTOU defence — closes the symlink swap window between containment check and read).
+Every probe uses the OK / MINOR / MAJOR taxonomy from v0.26 (Quant-Lobotomy) / v0.56 (Diagnose) / v0.65 (Eval Depth). Sleeper / truth / harm / interference exit 2 on MAJOR for CI gating. **v0.71.8** ships real probe weights: `--weights <w.npz|.npy|.safetensors>` loads a calibrated direction (cwd-contained, `O_NOFOLLOW`, `allow_pickle=False`, size-capped); `compute_contrast_probe(positive, negative)` derives one from contrast-pair activations; the bundled specs fall back to a deterministic synthetic seed (the large-base Anthropic-calibrated vectors remain upstream-gated). `soup probe interference --measure` loads the base + each LoRA adapter via PEFT and measures loss per adapter alone (diagonal) and per co-loaded pair (`add_weighted_adapter(combination_type="cat")`, off-diagonal). `soup train --capture-activations` writes an SAE-diff-ready per-token snapshot (the `model.layers.N` path resolves whether or not a LoRA adapter is loaded). The blame runner closes the v0.57 `NotImplementedError` stub via a DataInf-style influence approximation: `cos(grad_row, grad_probe) × |grad_row|`. Operators supply a `probe_fn` returning `(row_grads, probe_grad)`, or the runner falls back to a deterministic synthetic probe so the surface always returns a real `BlameResult` (no exception leaks). SAE feature diff is pure-numpy; the safetensors loader is `O_NOFOLLOW`-protected (TOCTOU defence — closes the symlink swap window between containment check and read); `--auto-download` validates the `HF_HUB_ALLOWLIST` before any network call and rejects a glob result escaping the snapshot dir.
 
 
 ## Pre-flight Decision (`soup advise`)

{soup_cli-0.71.7 → soup_cli-0.71.8}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.7"
+version = "0.71.8"
 description = "Fine-tune LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.7 → soup_cli-0.71.8}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune LLMs in one command."""
 
-__version__ = "0.71.7"
+__version__ = "0.71.8"

{soup_cli-0.71.7 → soup_cli-0.71.8}/src/soup_cli/cli.py

@@ -410,8 +410,9 @@ app.add_typer(
     _probe_cmd.app,
     name="probe",
     help=(
-        "Activation probes: sleeper-agent defection / pairwise interference "
-        "/ SAE feature diff / probe pack (v0.66.0)."
+        "Activation probes: sleeper-agent defection / honesty / misuse / "
+        "pairwise interference / SAE feature diff / probe pack (v0.66.0, "
+        "truth+harm v0.71.8)."
     ),
 )