soup-cli 0.71.32__tar.gz → 0.71.34__tar.gz

{soup_cli-0.71.32 → soup_cli-0.71.34}/CHANGELOG.md

@@ -12,6 +12,121 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.34] - 2026-07-15
+
+### Added
+- **`soup adapters arithmetic` — task-vector algebra over LoRA adapters (add / scale / negate).**
+  Apply task arithmetic (arXiv:2212.04089) to LoRA deltas via an expression such as
+  `"coder + 0.5*math - toxic"`, mapping names to adapter dirs with repeatable
+  `--adapter name=path`. Produces one merged adapter you can serve or merge.
+  - Signed, un-normalized element-wise combine over same-rank adapters; the effective
+    delta `ΔW = B @ A` scales **linearly** with each coefficient (negation flips the
+    delta, `0.5·` halves it) via a √|c| factor split — not the `c²` a naive sum gives.
+    Mixed-rank inputs are refused with a clear "harmonize rank" message.
+  - Reuses the backdoor-scan gate (refuses a FAIL-scanned input unless `--allow-unscanned`)
+    and a same-base-model check (`--allow-cross-base` to override). Hand-written expression
+    parser (no `eval`), cwd-contained/symlink-rejecting paths, exit 0 = ok / 1 = refusal.
+- **LISA — Layerwise Importance Sampled AdamW (arXiv:2403.17919).** Full-fine-tuning
+  quality at LoRA-like memory: every N steps LISA freezes all decoder layers except a
+  small random set (embeddings + head always trainable). Enable with
+  `training.lisa_enabled: true` (+ `lisa_num_layers`, `lisa_interval_steps`) on a
+  `task: sft`, transformers, text, `quantization: none` run; mutually exclusive with
+  LoRA features and the other freeze mechanisms. Live on a 4 GB GPU for small models.
+
+## [0.71.33] - 2026-07-13
+
+### Added
+- **`soup draft` — train and, above all, MEASURE a speculative-decoding draft model.**
+  - `soup draft measure --target <m> --draft <d> --prompts p.jsonl` reports a draft's
+    **acceptance rate** (the fraction of the target's own greedy tokens the draft
+    would have proposed correctly) plus **real plain-vs-assisted throughput**. This is
+    the honest gate: it tells you whether speculative decoding is worth enabling
+    *before* you ship it. Exit 0 / 2 (below `--min-acceptance`, for CI) / 1.
+  - `soup draft distill --target <tuned> --draft-base <tiny> --data d.jsonl -o draft/`
+    distils your target into the tiny base (logit KD via the existing `task: distill`
+    trainer) and emits a **dense** draft model, ready to load as an `assistant_model`.
+  - `soup draft list`, plus a local draft registry (`~/.soup/drafts.json`) that
+    `soup serve --auto-spec` consults **before** the built-in pairing table — so a
+    draft you trained yourself is picked up automatically.
+  - Draft and target must share a tokenizer; a mismatched pair is refused up front
+    (speculative decoding proposes draft token ids into the target's vocabulary, so a
+    mismatch silently produces garbage rather than failing).
+
+  **Read the measured results before you use this** — see *Known limitations* below.
+  On the validated pair, distillation did **not** improve acceptance, and speculative
+  decoding was a net **slowdown**. `soup draft measure` is what tells you that.
+
+### Known limitations
+- **Distilling a draft did not improve its acceptance rate on the validated pair.**
+  Measured on `SmolLM2-360M-Instruct` (target) with a `SmolLM2-135M-Instruct` draft:
+  the *stock* draft already scored **69.3%**, and distilling it moved that to 69.7%
+  after 2 epochs and back to **69.3%** after 10 epochs — i.e. no gain beyond noise.
+  A small same-family draft is already near its capacity ceiling for agreeing with
+  the target, and logit KD cannot buy capacity it does not have. Whether distillation
+  materially raises acceptance for a genuinely diverged fine-tune, or a larger
+  target/draft pair, is **unproven** on a 4 GB box — tracked as a scale issue.
+- **Speculative decoding was a net slowdown on that pair** (measured 0.55–0.64×): the
+  draft's forward pass costs more than the tokens it saves at this size.
+  `soup draft measure` reports this truthfully rather than assuming a speedup — which
+  is precisely the point of shipping the measurement.
+- **Acceptance is teacher-forced greedy agreement**, the metric the Medusa/EAGLE
+  papers report. It is exact and deterministic, and it is the right number for
+  comparing drafts — but it is not the accepted-token count of a *sampling* run, which
+  also depends on the rejection-resample cascade.
+- **Same-tokenizer only.** Cross-tokenizer drafts (ULD-aligned + universal assisted
+  decoding) are deferred.
+
+### Changed
+- **PRM-guided GRPO scores completions in a single batched forward.**
+  `PRMScorer.__call__` now right-pads all completions into one `[B, T]` tensor
+  (+ attention mask) and runs one `output_hidden_states` pass instead of one
+  forward per completion, cutting per-step reward latency on non-tiny models.
+  Numerically identical to the per-completion path (parity + mixed-length tests).
+  Closes #298
+  ([#301](https://github.com/MakazhanAlpamys/Soup/pull/301) by [@Ekaanksh-dev](https://github.com/Ekaanksh-dev)).
+
+### Fixed
+- **The hardware-fit gate no longer refuses to train small models.**
+  `model_size_from_name` did not understand an `M` (millions) suffix, so
+  `SmolLM2-135M` fell through to the 7B default, was predicted to need ~14 GB of
+  weights, and was blocked. Every draft-sized model hit this. `1.7B` was also being
+  read as `7B` (the `7b` marker matched inside `1.7b`), while `Qwen2.5-7B-Instruct-1M`
+  correctly stays 7B (1M is the *context*, not the parameter count).
+- **`soup serve --backend vllm` no longer force-enables `trust_remote_code`.** The
+  vLLM path now goes through the same `--trust-remote-code` default-deny gate (and
+  warning panel) as the transformers backend, so serving an untrusted repo never
+  executes its code silently.
+- **Multi-adapter serving (`soup serve --adapters name=path`) now actually switches
+  adapters.** The named adapters are loaded into the model and selected per request
+  (via `POST /v1/adapters/activate/{name}` or the request `adapter` field);
+  previously every request silently ran the startup model. The base model is served
+  when no adapter is selected.
+- **Vision datasets reject out-of-directory image paths.** `llava` / `sharegpt4v`
+  rows are containment-checked against `image_dir` (mirroring the audio loader), so a
+  crafted `{"image": "/etc/passwd"}` row can no longer read arbitrary local files.
+- **`soup train --dry-run --gpus N` no longer launches a real multi-GPU run.** The
+  accelerate re-exec is skipped under `--dry-run`. The re-exec also now forwards
+  `--minillm-on-policy`, `--capture-activations`, and `--capture-prompts` (previously
+  dropped on multi-GPU runs).
+- **MLX SFT now builds a real optimizer** (`AdamW` from the configured LR) instead of
+  passing `optimizer=None`, which left the model untrained.
+- **`soup data inspect` / `preview` / `search` escape dataset- and Hub-derived text**
+  so a stray `[/]` no longer crashes the command and a crafted `[link=…]` tag can't
+  render a phishing hyperlink. `soup runs` list/show escape config-derived fields too.
+- **`soup infer --task asr` hardening** — an oversized reference no longer crashes the
+  whole batch after transcription (that row's metric is skipped); an all-skipped run
+  exits non-zero instead of reporting success; `--asr-task` is validated upfront; and
+  dataset-derived filenames are control-stripped before printing. ASR training now
+  caps transcript labels to Whisper's decoder limit, warns on >30 s audio, and picks
+  fp16 on pre-Ampere GPUs instead of hardcoding bf16.
+- **Knowledge-distillation KD term aligns with the CE term.** The token-level
+  divergence is now computed over causal-shifted positions, so the distillation signal
+  covers exactly the trained tokens (previously off by one).
+- **Miscellaneous robustness** — `load_config_from_string` raises `ValueError` (not
+  `TypeError`) on a non-mapping YAML document; the Web UI Bearer-token check is
+  constant-time; and `soup doctor --vscode` / the LR-finder report use the centralised
+  atomic, symlink-rejecting writer.
+
 ## [0.71.32] - 2026-07-07
 
 ### Added

{soup_cli-0.71.32 → soup_cli-0.71.34}/CONTRIBUTING.md

@@ -120,7 +120,7 @@ src/soup_cli/
   templates/          - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (309 files, 15663 tests)
+tests/                - Test suite (312 files, 15906 tests)
 examples/             - Real-world config examples and datasets
 ```
 

{soup_cli-0.71.32 → soup_cli-0.71.34}/CONTRIBUTORS.md

@@ -50,6 +50,8 @@ Listed by first contribution. PR numbers link the work.
   - Reuse the shared vocab-expansion helper in the vision + audio SFT paths ([#291](https://github.com/MakazhanAlpamys/Soup/pull/291))
   - Honor configured vocab expansion in the DPO / IPO / KTO / BCO trainers ([#293](https://github.com/MakazhanAlpamys/Soup/pull/293))
   - Honor configured vocab expansion in the ORPO / SimPO / GRPO trainers ([#295](https://github.com/MakazhanAlpamys/Soup/pull/295))
+- **Ekaanksh Patil** ([@Ekaanksh-dev](https://github.com/Ekaanksh-dev))
+  - Batch the PRM reward forward pass in `PRMScorer.__call__` (single `[B, T]` forward) ([#301](https://github.com/MakazhanAlpamys/Soup/pull/301))
 
 ---
 

{soup_cli-0.71.32 → soup_cli-0.71.34}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.32
+Version: 0.71.34
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -193,35 +193,42 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.32 — ASR fine-tuning (Whisper).** Fine-tune Whisper on your accent or domain, locally. whisper-tiny (39M) and base (74M) train on a 4 GB GPU — a new modality that actually runs on a laptop card.
-
-- **`task='asr'`.** A new `AsrTrainerWrapper` (HF `Seq2SeqTrainer` + `WhisperProcessor`). Data rows
-  are `{"audio": <path>, "text": <transcript>}` under `data.format='asr'`; audio decodes to 16 kHz
-  mono through the hardened loader (soundfile pre-probe + symlink/size guards). Opt-in LoRA on q/v
-  via `training.asr_lora: true` (default full fine-tune); `training.asr_language` / `asr_task`
-  (`transcribe`|`translate`) set — and persist — the decoder prefix for inference.
-- **`soup infer --task asr`.** Transcribe an `{"audio"[, "text"]}` JSONL and get per-row + corpus
-  **WER / CER** when references are present. Loads a full model or a LoRA adapter dir; audio paths
-  are containment-checked (traversal / UNC rejected).
-- **Pure-python metrics.** New `soup_cli.utils.asr_metrics` (WER / CER / `word_accuracy` /
-  `corpus_wer`) — no new dependency, reusable as a higher-is-better ship metric leg.
-- **Recipes.** `whisper-tiny-asr` / `whisper-base-asr` (live-trainable), `whisper-large-v3-asr`
-  (parse-only). Catalog 138 → 142.
-- **Proof-of-mechanism, honestly.** Live-validated on an RTX 3050: a memorization task drove
-  WER 1.000 → 0.000 (loss 7.2 → 0.0005 at 0.4/4 GB). Real accent/domain gains need real audio;
-  WER/CER use a light normalizer, not the full Whisper text normalizer.
+**v0.71.34 — adapter algebra + LISA.** Compose fine-tunes like vectors, and train big-model quality on a small-model memory budget.
 
-```yaml
-base: openai/whisper-tiny
-task: asr
-data:
-  format: asr            # rows: {"audio": "clip.wav", "text": "hello world"}
-  audio_dir: ./data/audio
-training:
-  asr_language: en
-  asr_lora: true         # optional; default = full fine-tune
+- **`soup adapters arithmetic "coder + 0.5*math - toxic"`.** Task-vector algebra over LoRA
+  adapters (arXiv:2212.04089) — **add, scale, and NEGATE** trained behaviours into one merged
+  adapter. The math is done right: a LoRA's effective delta `ΔW = B·A` scales *linearly* with
+  each coefficient (subtracting an adapter actually removes its behaviour), not quadratically.
+  Same-base + backdoor-scan gated; mixed ranks refused with a clear message.
+- **LISA — layerwise importance sampling (`training.lisa_enabled`).** Full-fine-tuning quality
+  at LoRA-like memory (arXiv:2403.17919): every N steps LISA re-activates a small random set of
+  decoder layers (embeddings + head always on) and freezes the rest. Live on a 4 GB GPU.
+- **Verified on real models (RTX 3050).** LISA trains SmolLM2-135M end-to-end; `2·a` produces an
+  adapter whose ΔW is *exactly* 2× the original's — the merged adapter loads and serves.
+
+```bash
+soup adapters arithmetic "coder - toxic" \
+  --adapter coder=./coder-lora --adapter toxic=./toxic-lora -o ./cleaned
+
+soup train --config sft.yaml   # with training.lisa_enabled: true, lisa_num_layers: 2
 ```
 
+<details>
+<summary>Previous release — v0.71.33, <code>soup draft</code> (measure speculative decoding)</summary>
+
+`soup draft measure` reports a draft model's **acceptance rate** + real plain-vs-assisted tok/s
+(exit 0/2/1 for CI); `soup draft distill` distils your target into a dense tiny draft, auto-wired
+into `soup serve --auto-spec`. The honest result on a small same-family pair: distillation didn't
+move acceptance (69.3% → 69.3%) and assisted decoding was a net slowdown — which is exactly the
+number you want *before* shipping speculative decoding.
+
+```bash
+soup draft measure --target ./my-tuned-model --draft HuggingFaceTB/SmolLM2-135M-Instruct \
+  --prompts prod-prompts.jsonl        # -> acceptance %, real tok/s, ship-or-not
+```
+
+</details>
+
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
 
 ## Quick Start
@@ -301,7 +308,7 @@ The full feature reference lives in [`docs/`](docs/). Start here:
 | [Performance & quantization](docs/performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, multi-GPU / DeepSpeed / FSDP |
 | [Data engineering](docs/data.md) | Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs |
 | [Evaluation & probes](docs/evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, post-train X-ray probes, A/B, drift, tunability, `soup advise` |
-| [Serving & export](docs/serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding, deploy autopilot, Web UI, Agent Forge |
+| [Serving & export](docs/serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge |
 | [Adapters, registry & governance](docs/adapters-and-governance.md) | Adapter lifecycle/management, model registry, Soup Cans, the data flywheel (`soup loop`), knowledge editing, steering, supply-chain controls (scan/sign/BOM/attest/audit/airgap) |
 | [Backends, platform & ops](docs/backends-and-ops.md) | MLX/Unsloth backends, alternative hubs, HF Hub integration, autopilot, experiment tracking, plan/apply, env lockfiles, hardware-fit, completions, plugins, utility commands |
 | [Command reference](docs/commands.md) | The full `soup` command list |

{soup_cli-0.71.32 → soup_cli-0.71.34}/README.md

@@ -49,35 +49,42 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.32 — ASR fine-tuning (Whisper).** Fine-tune Whisper on your accent or domain, locally. whisper-tiny (39M) and base (74M) train on a 4 GB GPU — a new modality that actually runs on a laptop card.
-
-- **`task='asr'`.** A new `AsrTrainerWrapper` (HF `Seq2SeqTrainer` + `WhisperProcessor`). Data rows
-  are `{"audio": <path>, "text": <transcript>}` under `data.format='asr'`; audio decodes to 16 kHz
-  mono through the hardened loader (soundfile pre-probe + symlink/size guards). Opt-in LoRA on q/v
-  via `training.asr_lora: true` (default full fine-tune); `training.asr_language` / `asr_task`
-  (`transcribe`|`translate`) set — and persist — the decoder prefix for inference.
-- **`soup infer --task asr`.** Transcribe an `{"audio"[, "text"]}` JSONL and get per-row + corpus
-  **WER / CER** when references are present. Loads a full model or a LoRA adapter dir; audio paths
-  are containment-checked (traversal / UNC rejected).
-- **Pure-python metrics.** New `soup_cli.utils.asr_metrics` (WER / CER / `word_accuracy` /
-  `corpus_wer`) — no new dependency, reusable as a higher-is-better ship metric leg.
-- **Recipes.** `whisper-tiny-asr` / `whisper-base-asr` (live-trainable), `whisper-large-v3-asr`
-  (parse-only). Catalog 138 → 142.
-- **Proof-of-mechanism, honestly.** Live-validated on an RTX 3050: a memorization task drove
-  WER 1.000 → 0.000 (loss 7.2 → 0.0005 at 0.4/4 GB). Real accent/domain gains need real audio;
-  WER/CER use a light normalizer, not the full Whisper text normalizer.
+**v0.71.34 — adapter algebra + LISA.** Compose fine-tunes like vectors, and train big-model quality on a small-model memory budget.
+
+- **`soup adapters arithmetic "coder + 0.5*math - toxic"`.** Task-vector algebra over LoRA
+  adapters (arXiv:2212.04089) — **add, scale, and NEGATE** trained behaviours into one merged
+  adapter. The math is done right: a LoRA's effective delta `ΔW = B·A` scales *linearly* with
+  each coefficient (subtracting an adapter actually removes its behaviour), not quadratically.
+  Same-base + backdoor-scan gated; mixed ranks refused with a clear message.
+- **LISA — layerwise importance sampling (`training.lisa_enabled`).** Full-fine-tuning quality
+  at LoRA-like memory (arXiv:2403.17919): every N steps LISA re-activates a small random set of
+  decoder layers (embeddings + head always on) and freezes the rest. Live on a 4 GB GPU.
+- **Verified on real models (RTX 3050).** LISA trains SmolLM2-135M end-to-end; `2·a` produces an
+  adapter whose ΔW is *exactly* 2× the original's — the merged adapter loads and serves.
 
-```yaml
-base: openai/whisper-tiny
-task: asr
-data:
-  format: asr            # rows: {"audio": "clip.wav", "text": "hello world"}
-  audio_dir: ./data/audio
-training:
-  asr_language: en
-  asr_lora: true         # optional; default = full fine-tune
+```bash
+soup adapters arithmetic "coder - toxic" \
+  --adapter coder=./coder-lora --adapter toxic=./toxic-lora -o ./cleaned
+
+soup train --config sft.yaml   # with training.lisa_enabled: true, lisa_num_layers: 2
 ```
 
+<details>
+<summary>Previous release — v0.71.33, <code>soup draft</code> (measure speculative decoding)</summary>
+
+`soup draft measure` reports a draft model's **acceptance rate** + real plain-vs-assisted tok/s
+(exit 0/2/1 for CI); `soup draft distill` distils your target into a dense tiny draft, auto-wired
+into `soup serve --auto-spec`. The honest result on a small same-family pair: distillation didn't
+move acceptance (69.3% → 69.3%) and assisted decoding was a net slowdown — which is exactly the
+number you want *before* shipping speculative decoding.
+
+```bash
+soup draft measure --target ./my-tuned-model --draft HuggingFaceTB/SmolLM2-135M-Instruct \
+  --prompts prod-prompts.jsonl        # -> acceptance %, real tok/s, ship-or-not
+```
+
+</details>
+
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
 
 ## Quick Start
@@ -157,7 +164,7 @@ The full feature reference lives in [`docs/`](docs/). Start here:
 | [Performance & quantization](docs/performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, multi-GPU / DeepSpeed / FSDP |
 | [Data engineering](docs/data.md) | Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs |
 | [Evaluation & probes](docs/evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, post-train X-ray probes, A/B, drift, tunability, `soup advise` |
-| [Serving & export](docs/serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding, deploy autopilot, Web UI, Agent Forge |
+| [Serving & export](docs/serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge |
 | [Adapters, registry & governance](docs/adapters-and-governance.md) | Adapter lifecycle/management, model registry, Soup Cans, the data flywheel (`soup loop`), knowledge editing, steering, supply-chain controls (scan/sign/BOM/attest/audit/airgap) |
 | [Backends, platform & ops](docs/backends-and-ops.md) | MLX/Unsloth backends, alternative hubs, HF Hub integration, autopilot, experiment tracking, plan/apply, env lockfiles, hardware-fit, completions, plugins, utility commands |
 | [Command reference](docs/commands.md) | The full `soup` command list |

{soup_cli-0.71.32 → soup_cli-0.71.34}/docs/README.md

@@ -12,7 +12,7 @@ feature reference — every `soup` capability, grouped by area.
 | [Performance & quantization](performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, multi-GPU / DeepSpeed / FSDP |
 | [Data engineering](data.md) | Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs |
 | [Evaluation & probes](evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, `soup ship` verdict, post-train X-ray probes, A/B, drift, tunability, `soup advise` |
-| [Serving & export](serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding, deploy autopilot, Web UI, Agent Forge |
+| [Serving & export](serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge |
 | [Adapters, registry & governance](adapters-and-governance.md) | Adapter lifecycle/management, model registry, Soup Cans, the data flywheel (`soup loop`), knowledge editing, steering, supply-chain controls |
 | [Backends, platform & ops](backends-and-ops.md) | MLX/Unsloth backends, Modal cloud GPU training, alternative hubs, HF Hub integration, autopilot, experiment tracking, plan/apply, env lockfiles, hardware-fit, completions, plugins, utility commands |
 | [Command reference](commands.md) | The full `soup` command list |

{soup_cli-0.71.32 → soup_cli-0.71.34}/docs/adapters-and-governance.md

@@ -270,6 +270,12 @@ soup adapters merge ./run-v17 ./run-v18 ./run-v19 -o ./merged --strategy ties \
 soup adapters merge ./run-v17 ./run-v18 -o ./merged --strategy dare \
   --density 0.5 --seed 42
 
+# Task-vector arithmetic (v0.71.34) — add / scale / NEGATE trained behaviours
+# into one adapter (arXiv:2212.04089). Names map to dirs via --adapter name=path.
+soup adapters arithmetic "coder + 0.5*math - toxic" \
+  --adapter coder=./coder-lora --adapter math=./math-lora \
+  --adapter toxic=./toxic-lora -o ./blended
+
 # Leave-one-out ablation plan against a 4-hour wall-clock budget
 soup adapters blame ./run-v18 --dataset train.jsonl --layer q_proj.7 \
   --budget 4h --shards 10 --plan-only
@@ -314,6 +320,8 @@ soup adapters branches
 
 **v0.71.4:** the merge verdict is now LIVE — `soup adapters merge … --canary suite.json` lifts the `MergeReport.verdict` `UNKNOWN` stub. A pre-scored `{"baseline_scores","candidate_scores"}` suite classifies the blend OK / MINOR / MAJOR with no model load; a `{"tasks":[...]}` suite uses an injectable scorer. `--strict-verdict` exits 2 on MAJOR. The backdoor-scan and license-conflict gates now run for **every** strategy, including `cmaes`. `soup adapters branch <name> --from-registry <id>` / `--attach-to-registry <id>` link training-env snapshots into the Registry lineage DAG (shown as a `branches` node in `soup history`).
 
+**v0.71.34 — task-vector arithmetic.** `soup adapters arithmetic "<expr>"` applies task arithmetic (arXiv:2212.04089) to LoRA deltas: **add** (blend two skills), **scale** (`2*coder`), and — the differentiator — **negate** (`- toxic` removes a behaviour). Names in the expression map to adapter dirs via repeatable `--adapter name=path`. The math is done at the effective-delta level: a LoRA contributes `ΔW = B·A`, so the coefficient is applied so that `ΔW` scales **linearly** (subtracting an adapter actually negates its delta — a naïve element-wise sum would scale by `c²`, making negation a no-op). Adapters must share the base model (`--allow-cross-base` to override) and rank (mixed ranks are refused, not silently approximated). Each input passes the v0.71.2 backdoor-scan gate (`--allow-unscanned` to skip). Output is a single loadable adapter; exit 0 = ok, 1 = refusal.
+
 
 ## Soup Cans (Shareable Recipes)
 

{soup_cli-0.71.32 → soup_cli-0.71.34}/docs/commands.md

@@ -27,6 +27,7 @@ soup merge --adapter ./output                 Merge LoRA with base model
 soup merge-sharded-fsdp-weights ./shards -o merged.safetensors  Consolidate FSDP shards into one safetensors (v0.71.14; --plan-only previews)
 soup delinearize-llama4 ./src --target ./out [--num-experts N] [--plan-only]  Live Llama-4 fused-expert reshape [E*din,dout] -> [E,din,dout] + sidecar copy (v0.71.21)
 soup spectrum scan --model <id|path> --top-percent 50 [--modules mlp,attn] [-o patch.yaml]  Spectrum SNR scan (no model load) -> training.unfrozen_parameters YAML patch (v0.71.23)
+soup train --config sft.yaml  # training.lisa_enabled: true [lisa_num_layers lisa_interval_steps]  LISA layerwise importance sampling — full-FT quality at LoRA-like memory (sft/transformers/text/quantization=none) (v0.71.34)
 soup export --model ./output --format gguf    Export to GGUF (Ollama)
 soup export --model ./output --deploy ollama  Export GGUF + auto-deploy to Ollama
 soup export --model ./output --format onnx    Export to ONNX
@@ -134,7 +135,7 @@ soup migrate --from llamafactory config.yaml  Import config from LLaMA-Factory
 soup migrate --from axolotl config.yml        Import config from Axolotl
 soup migrate --from unsloth notebook.ipynb    Import config from Unsloth notebook
 soup migrate --from llamafactory c.yaml --dry-run  Preview without writing
-soup recipes list                             List all 138 ready-made recipes
+soup recipes list                             List all 142 ready-made recipes
 soup recipes show llama3.1-8b-sft            Print recipe YAML
 soup recipes use llama3.1-8b-sft             Copy recipe to soup.yaml
 soup recipes search "reasoning"              Search by keyword/task/size
@@ -163,6 +164,11 @@ soup mcp serve --allow-mutating               Also expose plan-only train_start
 soup shrink --model <id|path> --drop-ratio 0.25 --calib c.jsonl -o shrunk  Depth-prune least-important layer block + SHIP/DON'T-SHIP ppl verdict (exit 0/2/1) (v0.71.29)
 soup shrink ... --drop-layers N --heal h.jsonl --heal-steps 200 --device cpu  Drop N layers + distill-heal (fuse LoRA back to one dense model)
 soup shrink ... --tolerance 0.10 --plan-only [--attach-to-registry <id>]  Ppl-regression tolerance / print importance table only / registry attach
+soup draft measure --target <m> --draft <d> --prompts p.jsonl  Draft acceptance rate + real plain-vs-assisted tok/s (exit 0/2/1) (v0.71.33)
+soup draft measure ... --min-acceptance 0.6 -o report.json  Exit 2 below the floor (CI gate) / write the JSON report
+soup draft distill --target <tuned> --draft-base <tiny> --data d.jsonl -o draft/  Distil a DENSE speculative-decoding draft + register it (v0.71.33)
+soup draft distill ... --steps N --device cpu --force --plan-only  Training budget / device / overwrite -o / render the config only
+soup draft list                               List local drafts that `soup serve --auto-spec` will pick up (v0.71.33)
 soup tui                                      Full-screen Textual dashboard (requires [tui] extra)
 soup train --config soup.yaml --profile       Record torch.profiler trace to <output>/profiles/
 soup --log-level quiet|normal|verbose|debug   Global logging tier (Rich-formatted)
@@ -179,6 +185,7 @@ soup adapters sign <adapter> [--backend unsigned|ed25519] [--key <pem>|--generat
 soup adapters verify <adapter> [--strict] [--public-key <pem>]  Verify manifest + ed25519 signature
 soup adapters check-safetensors <adapter> [--strict]  Refuse pickle / PyTorch-classic weights
 soup adapters merge ... [--license <id>] [--license-override <reason>] [--allow-unscanned]  License + backdoor-scan gates (auto-detect license; scan FAIL refused)
+soup adapters arithmetic "coder + 0.5*math - toxic" --adapter coder=<p> --adapter math=<p> --adapter toxic=<p> -o <out> [--allow-unscanned --allow-cross-base]  Task-vector algebra over LoRA adapters (add/scale/negate; same-rank; scan + same-base gated) (v0.71.34)
 soup attest emit ... [--sign ed25519 --key <pem>] [-o att.json]  in-toto/SLSA-3 attestation (+ .sig sidecar)
 soup attest verify <statement> --signature <sig> [--public-key <pem>]  Verify ed25519 attestation signature
 soup airgap-bundle --model <m> --output <out.tar> [--repro-receipt <r.json>]  Signed tarball for data-diode transfer (embeds repro-receipt)

{soup_cli-0.71.32 → soup_cli-0.71.34}/docs/peft-and-efficiency.md

@@ -380,6 +380,25 @@ training:
 
 Works with and without LoRA. When used with LoRA, LoRA is applied only to unfrozen layers.
 
+## LISA — Layerwise Importance Sampling (v0.71.34)
+
+LISA (Layerwise Importance Sampled AdamW, [arXiv:2403.17919](https://arxiv.org/abs/2403.17919)) targets full-fine-tuning quality at LoRA-like memory. Instead of picking layers once (that's Spectrum's static `unfrozen_parameters`), LISA re-samples a small random set of decoder layers **every N steps** and freezes the rest; the input embeddings, the LM head, and the final norm stay trainable throughout.
+
+```yaml
+task: sft
+backend: transformers
+modality: text
+training:
+  quantization: none      # LISA is full-FT of the active layers
+  lisa_enabled: true
+  lisa_num_layers: 2       # decoder layers active per interval (clamped to model depth)
+  lisa_interval_steps: 20  # re-sample cadence, in global steps
+```
+
+Because only a handful of layers train at any moment (and their optimizer state is cleared when they're re-frozen), peak optimizer memory is roughly `embeddings + head + lisa_num_layers` — far below a full fine-tune, while every layer still gets updated over the course of training. LISA is `sft` + `transformers` + `text` + `quantization: none` only, and is mutually exclusive with LoRA features, `freeze_layers`/`freeze_ratio`, and Spectrum's `unfrozen_parameters` (each independently decides what trains).
+
+Implementation note: the model is left fully trainable at trainer-setup time so HF's optimizer (built before the first callback fires) contains every decoder parameter; the LISA callback then toggles `requires_grad` per interval — frozen parameters produce no gradient and the optimizer skips them.
+
 
 ## Loss Watchdog
 

{soup_cli-0.71.32 → soup_cli-0.71.34}/docs/serving-and-export.md

@@ -23,6 +23,7 @@
 - [Anthropic Messages API Converter](#anthropic-messages-api-converter)
 - [Server-Side Tools](#server-side-tools)
 - [Anthropic Messages Endpoint](#anthropic-messages-endpoint)
+- [Train + measure your own draft (`soup draft`)](#train--measure-your-own-draft-soup-draft)
 - [N-gram Speculative Decoding](#n-gram-speculative-decoding)
 - [Server-Side Tool Endpoints](#server-side-tool-endpoints)
 
@@ -254,7 +255,10 @@ soup serve --model ./output --backend sglang --tensor-parallel 2
 
 ### Speculative Decoding
 
-Use a smaller draft model to speed up generation (2-3x faster):
+Use a smaller draft model to speed up generation. **Measure before you trust it** — see
+[Train + measure your own draft](#train--measure-your-own-draft-soup-draft) below. Speculative
+decoding is only a win when the draft agrees with the target often enough to pay for its own
+forward pass; on a small target it is frequently a *slowdown*.
 
 ```bash
 # Transformers backend — uses HF assisted generation
@@ -268,7 +272,46 @@ soup serve --model meta-llama/Llama-3.1-70B-Instruct --backend vllm --auto-spec
 # → auto-paired: meta-llama/Llama-3.2-1B-Instruct (target: Llama-3.1-70B-Instruct)
 ```
 
-`--auto-spec` handles Llama 3.1/3.3/4, Qwen 2.5/3, Mistral Large, Mixtral, DeepSeek V3/R1, and Gemma 2/3. Models without a known draft pairing (e.g. 8B-or-smaller targets where draft+target overhead outweighs the gain) print a yellow "no draft" note and fall back to standard decoding.
+`--auto-spec` handles Llama 3.1/3.3/4, Qwen 2.5/3, Mistral Large, Mixtral, DeepSeek V3/R1, and Gemma 2/3. Models without a known draft pairing (e.g. 8B-or-smaller targets where draft+target overhead outweighs the gain) print a yellow "no draft" note and fall back to standard decoding. A draft you trained yourself with `soup draft distill` is picked up **before** this built-in table.
+
+### Train + measure your own draft (`soup draft`)
+
+The question nobody answers before enabling speculative decoding: *would the draft actually
+propose the tokens my model is going to emit?* `soup draft measure` answers it.
+
+```bash
+# Would this draft pay off? Acceptance rate + REAL plain-vs-assisted throughput.
+soup draft measure --target ./my-tuned-model \
+  --draft HuggingFaceTB/SmolLM2-135M-Instruct \
+  --prompts prod-prompts.jsonl
+
+# Distil a draft from your own target, then serve it automatically.
+soup draft distill --target ./my-tuned-model \
+  --draft-base HuggingFaceTB/SmolLM2-135M-Instruct \
+  --data traffic.jsonl -o draft/
+soup serve --model ./my-tuned-model --auto-spec     # picks up ./draft
+soup draft list
+```
+
+**Acceptance rate** is the fraction of the target's own greedy tokens the draft would have
+proposed correctly (teacher-forced argmax agreement — the metric the Medusa/EAGLE papers report).
+Higher is better; roughly, ≥70% is where speculative decoding starts paying for the draft's
+forward pass on realistic hardware. `--min-acceptance 0.6` exits **2** below the floor, so CI can
+gate on it (exit 0 = ok, 2 = below floor, 1 = error).
+
+`distill` runs logit KD through the existing `task: distill` trainer and emits a **dense** model
+(a PEFT adapter directory cannot be loaded as an `assistant_model`). Draft and target must share a
+tokenizer — a mismatch is refused up front, because speculative decoding proposes *draft* token ids
+into the *target's* vocabulary and a mismatched pair silently produces garbage instead of failing.
+A `soup shrink` output makes a good draft base: same tokenizer by construction.
+
+**Measured reality check (be sceptical of speedup claims, including ours).** On
+`SmolLM2-360M-Instruct` with a `SmolLM2-135M-Instruct` draft, the *stock* draft already scored
+**69.3%** acceptance, distilling it changed nothing (69.7% after 2 epochs, 69.3% after 10), and
+assisted decoding came out **0.55–0.64×** — a net slowdown. A small same-family draft is already
+near its ceiling for agreeing with the target, and the draft's forward pass costs more than the
+tokens it saves. Whether distillation pays off on a larger or genuinely diverged target/draft pair
+is unproven on a 4 GB box. Run `soup draft measure` on *your* pair rather than assuming.
 
 ### Prefix Caching
 
@@ -418,7 +461,7 @@ soup ui
 
 **Pages:**
 - **Dashboard** — view all experiment runs, loss charts, system info, multi-run comparison
-- **New Training** — create configs from templates or 138 ready-made recipes, validate, start training with live SSE log streaming and progress bar
+- **New Training** — create configs from templates or 142 ready-made recipes, validate, start training with live SSE log streaming and progress bar
 - **Data Explorer** — browse and inspect datasets (JSONL, JSON, CSV, Parquet)
 - **Model Chat** — chat with streaming responses, configurable temperature/top_p/max_tokens, system prompt, adapter selection, markdown rendering, chat export
 
@@ -427,7 +470,7 @@ soup ui
 - **Enhanced Metrics** — 2x2 chart grid (loss, LR, grad_norm, throughput) + GPU memory chart, eval results table
 - **Multi-Run Compare** — overlay loss curves from up to 5 runs side-by-side
 - **Chat Upgrade** — SSE streaming via proxy, typing indicator, cancel button, markdown renderer (bold, italic, code blocks), chat export as JSON
-- **Config Builder** — recipe dropdown (138 recipes), config schema API for dynamic form generation
+- **Config Builder** — recipe dropdown (142 recipes), config schema API for dynamic form generation
 
 **Security:** The Web UI generates a random auth token at startup (printed to console). All mutating endpoints (start/stop training, delete runs, inspect data, validate config) require `Authorization: Bearer <token>` header. CORS is restricted to the served origin. Data inspection is sandboxed to the working directory.
 

{soup_cli-0.71.32 → soup_cli-0.71.34}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.32"
+version = "0.71.34"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.32 → soup_cli-0.71.34}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
 
-__version__ = "0.71.32"
+__version__ = "0.71.34"

{soup_cli-0.71.32 → soup_cli-0.71.34}/src/soup_cli/cli.py

@@ -538,6 +538,18 @@ app.command(
     ),
 )(_shrink_cmd.shrink)
 
+# v0.71.33 — `soup draft` train-your-own speculative-decoding draft.
+from soup_cli.commands import draft as _draft_cmd  # noqa: E402
+
+app.add_typer(
+    _draft_cmd.app,
+    name="draft",
+    help=(
+        "Train + measure a speculative-decoding draft model: distil your tuned "
+        "model into a tiny draft, then serve it with --auto-spec (v0.71.33)."
+    ),
+)
+
 
 def _rewrite_advise_argv(argv: list) -> list:
     """Inject `run` between `advise` and a non-subcommand first argument.