soup-cli 0.71.41__tar.gz → 0.72.0__tar.gz

{soup_cli-0.71.41 → soup_cli-0.72.0}/CHANGELOG.md

@@ -12,6 +12,73 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.72.0] - 2026-07-26
+
+**Layer streaming (BETA) — fine-tune models that don't fit in your card.** The
+frozen base lives in CPU RAM and is streamed into two pre-allocated VRAM buffers
+one decoder layer at a time, so peak VRAM is bounded by the size of *one layer*
+instead of the whole model. Only the LoRA adapters, their gradients and
+optimizer state stay resident. Slower than resident training — but these models
+did not run on the card at all.
+
+Measured on the development box (**RTX 3050 Laptop 4 GB**, Windows 11, 16.9 GB
+RAM), batch 1, gradient checkpointing on, 50 steps after 10 warm-up:
+
+| Model | S | tok/s | GPU util | Peak VRAM |
+|---|---|---|---|---|
+| Qwen2.5-0.5B | 512 | 978.6 | 91.4% | 1.47 GB |
+| Qwen2.5-1.5B | 512 | 525.0 | 96.8% | 1.82 GB |
+| Qwen2.5-1.5B | 1024 | 487.6 | 96.7% | 2.96 GB |
+| Qwen2.5-3B | 512 | 143.1 | 79.3% | **2.15 GB** |
+
+**Qwen2.5-3B trains in 2.15 GB on a 4 GB card where a resident run OOMs.** The
+honest cost: **1.43× slower than resident**, measured at 0.5B — the only
+apples-to-apples comparison available on this box, because 1.5B and above cannot
+run resident here at all.
+
+### Added
+
+- **`training.stream_layers: true`** — stream the frozen base layer-by-layer
+  from CPU RAM. `training.stream_source` (`auto`/`ram`/`disk`) and
+  `training.stream_buffers` (2–8, default 2 = double buffering) tune it.
+- `soup_cli/utils/layer_stream.py` — tier choice, pinned-vs-pageable decision,
+  architecture allowlist, VRAM/throughput arithmetic (no torch import).
+- `soup_cli/utils/layer_shard.py` — rewrites an HF checkpoint into one
+  safetensors shard per decoder layer, one tensor at a time, so sharding a model
+  that does not fit never needs it to fit.
+- `soup_cli/utils/layer_stream_runtime.py` — buffer pool, CPU-RAM weight source,
+  prefetch scheduler on a dedicated CUDA stream, and the streamed layer wrapper.
+- Shards are cached under `~/.soup/layer-stream/` (override with
+  `SOUP_LAYER_STREAM_CACHE_DIR`) and keyed to the source checkpoint's
+  fingerprint, so a base retrained in place re-shards instead of silently
+  training against stale weights.
+- When the base cannot be page-locked, the RAM store falls back to pageable
+  memory **and says so**, including the measured cost (GPU utilisation
+  ~97% → ~79%).
+
+### Changed
+
+- The pre-flight hardware-fit gate is skipped for streaming runs: it models a
+  resident run and would otherwise refuse exactly the runs streaming enables.
+- Gradient checkpointing is handled per-layer by the streamer; the HF Trainer's
+  own is left off so layers are not recomputed twice.
+
+### Known limitations
+
+- **BETA, and proof-of-mechanism at 3B.** Nothing above 3B was measured. No
+  8B/14B claim is supported.
+- Scope: RAM tier, bf16, `task: sft`, Llama/Qwen, batch size 1, no gradient
+  accumulation, no `--resume`. Every refusal names the release that lifts it.
+- 4-bit (NF4) streaming is **v0.72.1** — NF4 weights carry a quantisation state
+  and cannot be byte-copied into a plain buffer.
+- The disk overflow tier, larger batches, gradient accumulation and
+  checkpoint/resume are **v0.72.2**.
+- The 3B number used a **pageable** store (this box cannot page-lock 5.55 GB),
+  so it is a lower bound.
+- `expandable_segments:True` is silently ignored on Windows; Soup detects this
+  and does not claim it is active.
+- Numbers are Windows/WDDM and therefore systematically pessimistic vs Linux.
+
 ## [0.71.41] - 2026-07-19
 
 **`soup reward stress`: is your reward verifier gameable?** Turn the reward-hacking

{soup_cli-0.71.41 → soup_cli-0.72.0}/CONTRIBUTING.md

@@ -116,11 +116,11 @@ src/soup_cli/
   cans/               - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0)
   data/traces/        - Trace-to-Preference harvester (v0.26.0)
   data/collators.py   - CrossDocCollator for sample packing (v0.33.0)
-  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint, shrink, best_of_n, evolve, reward_synth
+  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint, shrink, best_of_n, evolve, reward_synth, reward_stress, layer_stream, layer_shard, layer_stream_runtime
   templates/          - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (319 files, 16529 tests)
+tests/                - Test suite (324 files, 16735 tests)
 examples/             - Real-world config examples and datasets
 ```
 

{soup_cli-0.71.41 → soup_cli-0.72.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.41
+Version: 0.72.0
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -193,25 +193,27 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.41 — `soup reward stress`: is your reward verifier gameable?** Turn the reward-hacking
-detector on the verifier *itself*. `soup reward synth` (v0.71.40) proves a verifier can tell your
-references from friendly bad answers; `stress` asks the adversarial question a reward-hacking model
-asks at train time — *does it pay out for degenerate junk?* Nothing in TRL / Unsloth / Axolotl tests
-a verifier for gameability.
-
-- **Four attacks, scored against the real gold.** Empty, length-padded, repetition, and
-  sentinel-spam completions are fed to the verifier; any it **accepts** is a false positive. A
-  per-attack accept-rate table plus one **robust / GAMEABLE** verdict — exit 0 / 2 (1 on error).
-- **Probes a synth `.py` or a builtin.** `soup reward stress reward.py --references golds.jsonl`,
-  or `soup reward stress verifiable --verifiable-domain math --references golds.jsonl`. A
-  gold-requiring verifier probed with no `--references` is a hard error, never a false "robust".
-- **Tune the strictness.** `--attacks`, `--sentinel`, `--threshold`, `--max-gameable`,
-  `--output-report`. Pure, offline, no new deps.
+**v0.72.0 — Layer streaming (BETA).** Fine-tune models that don't fit in your card. The frozen
+base streams from CPU RAM one decoder layer at a time into a small pool of VRAM buffers while the
+LoRA adapters stay resident, so peak VRAM is bounded by one layer rather than the whole model.
+**Qwen2.5-3B trains in 2.15 GB on a 4 GB card, where a resident run OOMs.**
+
+- **`training.stream_layers: true`** — a config key, not a CLI flag. Tune with
+  `stream_source` (`ram` in v0.72.0; `disk` is v0.72.2) and `stream_buffers` (2–8, default 2).
+- **Measured on a 4 GB RTX 3050 Laptop** (batch 1, gradient checkpointing on): 0.5B at
+  978.6 tok/s / 1.47 GB, 1.5B at 525.0 tok/s / 1.82 GB, 3B at 143.1 tok/s / 2.15 GB.
+- **Honest cost: 1.43× slower than resident**, measured at 0.5B — the only apples-to-apples
+  comparison available on that box, because 1.5B and above cannot run resident there at all.
+- **Proof-of-mechanism at 3B.** Nothing above 3B was measured; no 8B/14B claim is supported.
+  Scope: RAM tier, bf16, `task: sft`, Llama/Qwen, batch size 1, no gradient accumulation, no
+  `--resume`. 4-bit (NF4) streaming is **v0.72.1** and is refused with a clear message today.
 
-```bash
-# probe a verifier you synthesized (or a builtin) for gameability
-soup reward stress reward.py --references golds.jsonl --output-report stress.json
-#   exit 0 = robust · 2 = gameable · 1 = error
+```yaml
+# soup.yaml — then just `soup train --config soup.yaml`
+training:
+  stream_layers: true      # base streams from RAM; only the adapter trains
+  batch_size: 1
+  quantization: none       # NF4 streaming lands in v0.72.1
 ```
 
 <details>
@@ -372,7 +374,7 @@ The full feature reference lives in [`docs/`](docs/). Start here:
 |---|---|
 | [Training tasks & methods](docs/training.md) | SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, loop-hardening detectors |
 | [PEFT, long context & efficiency](docs/peft-and-efficiency.md) | DoRA, LoRA+, rsLoRA, VeRA, OLoRA, NEFTune, PiSSA, ReLoRA, optimizer & PEFT zoo, LLaMA Pro, GaLore, YaRN/LongLoRA, packing, curriculum, auto-tuning |
-| [Performance & quantization](docs/performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, multi-GPU / DeepSpeed / FSDP |
+| [Performance & quantization](docs/performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, layer streaming, multi-GPU / DeepSpeed / FSDP |
 | [Data engineering](docs/data.md) | Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs |
 | [Evaluation & probes](docs/evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, post-train X-ray probes, A/B, drift, tunability, `soup advise` |
 | [Serving & export](docs/serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge |

{soup_cli-0.71.41 → soup_cli-0.72.0}/README.md

@@ -49,25 +49,27 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.41 — `soup reward stress`: is your reward verifier gameable?** Turn the reward-hacking
-detector on the verifier *itself*. `soup reward synth` (v0.71.40) proves a verifier can tell your
-references from friendly bad answers; `stress` asks the adversarial question a reward-hacking model
-asks at train time — *does it pay out for degenerate junk?* Nothing in TRL / Unsloth / Axolotl tests
-a verifier for gameability.
-
-- **Four attacks, scored against the real gold.** Empty, length-padded, repetition, and
-  sentinel-spam completions are fed to the verifier; any it **accepts** is a false positive. A
-  per-attack accept-rate table plus one **robust / GAMEABLE** verdict — exit 0 / 2 (1 on error).
-- **Probes a synth `.py` or a builtin.** `soup reward stress reward.py --references golds.jsonl`,
-  or `soup reward stress verifiable --verifiable-domain math --references golds.jsonl`. A
-  gold-requiring verifier probed with no `--references` is a hard error, never a false "robust".
-- **Tune the strictness.** `--attacks`, `--sentinel`, `--threshold`, `--max-gameable`,
-  `--output-report`. Pure, offline, no new deps.
+**v0.72.0 — Layer streaming (BETA).** Fine-tune models that don't fit in your card. The frozen
+base streams from CPU RAM one decoder layer at a time into a small pool of VRAM buffers while the
+LoRA adapters stay resident, so peak VRAM is bounded by one layer rather than the whole model.
+**Qwen2.5-3B trains in 2.15 GB on a 4 GB card, where a resident run OOMs.**
+
+- **`training.stream_layers: true`** — a config key, not a CLI flag. Tune with
+  `stream_source` (`ram` in v0.72.0; `disk` is v0.72.2) and `stream_buffers` (2–8, default 2).
+- **Measured on a 4 GB RTX 3050 Laptop** (batch 1, gradient checkpointing on): 0.5B at
+  978.6 tok/s / 1.47 GB, 1.5B at 525.0 tok/s / 1.82 GB, 3B at 143.1 tok/s / 2.15 GB.
+- **Honest cost: 1.43× slower than resident**, measured at 0.5B — the only apples-to-apples
+  comparison available on that box, because 1.5B and above cannot run resident there at all.
+- **Proof-of-mechanism at 3B.** Nothing above 3B was measured; no 8B/14B claim is supported.
+  Scope: RAM tier, bf16, `task: sft`, Llama/Qwen, batch size 1, no gradient accumulation, no
+  `--resume`. 4-bit (NF4) streaming is **v0.72.1** and is refused with a clear message today.
 
-```bash
-# probe a verifier you synthesized (or a builtin) for gameability
-soup reward stress reward.py --references golds.jsonl --output-report stress.json
-#   exit 0 = robust · 2 = gameable · 1 = error
+```yaml
+# soup.yaml — then just `soup train --config soup.yaml`
+training:
+  stream_layers: true      # base streams from RAM; only the adapter trains
+  batch_size: 1
+  quantization: none       # NF4 streaming lands in v0.72.1
 ```
 
 <details>
@@ -228,7 +230,7 @@ The full feature reference lives in [`docs/`](docs/). Start here:
 |---|---|
 | [Training tasks & methods](docs/training.md) | SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, loop-hardening detectors |
 | [PEFT, long context & efficiency](docs/peft-and-efficiency.md) | DoRA, LoRA+, rsLoRA, VeRA, OLoRA, NEFTune, PiSSA, ReLoRA, optimizer & PEFT zoo, LLaMA Pro, GaLore, YaRN/LongLoRA, packing, curriculum, auto-tuning |
-| [Performance & quantization](docs/performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, multi-GPU / DeepSpeed / FSDP |
+| [Performance & quantization](docs/performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, layer streaming, multi-GPU / DeepSpeed / FSDP |
 | [Data engineering](docs/data.md) | Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs |
 | [Evaluation & probes](docs/evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, post-train X-ray probes, A/B, drift, tunability, `soup advise` |
 | [Serving & export](docs/serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge |

{soup_cli-0.71.41 → soup_cli-0.72.0}/SECURITY.md

@@ -7,8 +7,8 @@ running the most recent version from PyPI (`pip install -U soup-cli`).
 
 | Version | Supported          |
 | ------- | ------------------ |
-| 0.71.x  | :white_check_mark: |
-| < 0.71  | :x:                |
+| 0.72.x  | :white_check_mark: |
+| < 0.72  | :x:                |
 
 ## Reporting a Vulnerability
 

{soup_cli-0.71.41 → soup_cli-0.72.0}/docs/README.md

@@ -9,7 +9,7 @@ feature reference — every `soup` capability, grouped by area.
 |---|---|
 | [Training tasks & methods](training.md) | SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, loop-hardening detectors, reward-verifier synthesis |
 | [PEFT, long context & efficiency](peft-and-efficiency.md) | DoRA, LoRA+, rsLoRA, VeRA, OLoRA, NEFTune, PiSSA, ReLoRA, optimizer & PEFT zoo, LLaMA Pro, GaLore, YaRN/LongLoRA, packing, curriculum, auto-tuning, depth pruning + distill-heal (`soup shrink`) |
-| [Performance & quantization](performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, multi-GPU / DeepSpeed / FSDP |
+| [Performance & quantization](performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, layer streaming, multi-GPU / DeepSpeed / FSDP |
 | [Data engineering](data.md) | Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs |
 | [Evaluation & probes](evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, `soup ship` verdict, post-train X-ray probes, A/B, drift, tunability, `soup advise` |
 | [Serving & export](serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge |

{soup_cli-0.71.41 → soup_cli-0.72.0}/docs/commands.md

@@ -30,6 +30,7 @@ soup merge-sharded-fsdp-weights ./shards -o merged.safetensors  Consolidate FSDP
 soup delinearize-llama4 ./src --target ./out [--num-experts N] [--plan-only]  Live Llama-4 fused-expert reshape [E*din,dout] -> [E,din,dout] + sidecar copy (v0.71.21)
 soup spectrum scan --model <id|path> --top-percent 50 [--modules mlp,attn] [-o patch.yaml]  Spectrum SNR scan (no model load) -> training.unfrozen_parameters YAML patch (v0.71.23)
 soup train --config sft.yaml  # training.lisa_enabled: true [lisa_num_layers lisa_interval_steps]  LISA layerwise importance sampling — full-FT quality at LoRA-like memory (sft/transformers/text/quantization=none) (v0.71.34)
+soup train --config sft.yaml  # training.stream_layers: true [stream_source stream_buffers]  BETA layer streaming — the frozen base streams from CPU RAM one decoder layer at a time, so peak VRAM is bounded by ONE layer (sft/transformers/text/quantization=none, batch 1) (v0.72.0)
 soup export --model ./output --format gguf    Export to GGUF (Ollama)
 soup export --model ./output --deploy ollama  Export GGUF + auto-deploy to Ollama
 soup export --model ./output --format onnx    Export to ONNX

{soup_cli-0.71.41 → soup_cli-0.72.0}/docs/performance-and-quantization.md

@@ -14,6 +14,7 @@
 - [Cross-Document Attention Masking](#cross-document-attention-masking)
 - [Quant Menu — 9 Quantization Formats](#quant-menu--9-quantization-formats)
 - [Activation Offloading (Small-VRAM Large-Batch)](#activation-offloading-small-vram-large-batch)
+- [Layer Streaming (BETA, v0.72.0)](#layer-streaming-beta-v0720)
 - [Correctness First (v0.36.0)](#correctness-first-v0360)
 - [Multi-GPU / DeepSpeed / FSDP](#multi-gpu--deepspeed--fsdp)
 - [Performance + Long-Context](#performance--long-context)
@@ -225,6 +226,108 @@ training:
 Not compatible with unsloth (own memory manager) or mlx. Wired across every transformer-backend trainer (SFT, DPO, GRPO, KTO, ORPO, SimPO, IPO, PPO, Reward-Model, Embedding, Pretrain).
 
 
+## Layer Streaming (BETA, v0.72.0)
+
+Stream frozen base-model decoder layers ONE at a time from CPU RAM into small VRAM buffers instead of keeping the whole base resident. Peak VRAM is bounded by the size of a single layer, not the entire model — so models that don't fit resident on your GPU can now train at all.
+
+```yaml
+training:
+  stream_layers: true          # Enable layer streaming
+  stream_source: auto          # 'auto' (same-host RAM), 'ram', 'disk' (v0.72.2)
+  stream_buffers: 2            # Double-buffering; range [2, 8]
+```
+
+```bash
+# Layer streaming is a CONFIG key, not a CLI flag — just train normally:
+soup train --config soup.yaml
+```
+
+**How it works.** LoRA adapters + their gradients + optimizer state stay resident in VRAM (they are small). The frozen base lives in CPU RAM, page-locked when the machine allows it, and is streamed: each decoder layer is copied into one of two pre-allocated VRAM buffers on a dedicated CUDA stream while the previous layer is still computing, so the load overlaps the compute. Each layer is read **twice** per step — once in the forward pass and once when the backward pass recomputes it — because `dL/dx = Wᵀ · dL/dy` needs the weights to reach the layers below. That is physics, not an implementation detail, and it is why streaming costs time.
+
+The tradeoff: **1.43× slower than resident training**, measured at 0.5B — the only apples-to-apples comparison available on the reference box, because 1.5B and above cannot run resident there at all.
+
+**Measured numbers (RTX 3050 Laptop 4 GB, Windows 11, bf16 LoRA, batch 1, 50 steps after 10 warmup):**
+
+| Model | Seq Length | Throughput | GPU Util | Peak VRAM |
+|---|---|---|---|---|
+| Qwen2.5-0.5B | 512 | 978.6 tok/s | 91.4% | 1.47 GB |
+| Qwen2.5-1.5B | 512 | 525.0 tok/s | 96.8% | 1.82 GB |
+| Qwen2.5-1.5B | 1024 | 487.6 tok/s | 96.7% | 2.96 GB |
+| Qwen2.5-3B | 512 | 143.1 tok/s | 79.3% | 2.15 GB |
+
+**Headline:** Qwen2.5-3B trains in **2.15 GB** on a 4 GB card, where a resident run OOMs.
+
+**Honest scope (v0.72.0 = proof-of-mechanism):**
+- **Models measured:** Qwen2.5-0.5B, 1.5B and 3B, plus a live `soup train` on SmolLM2-135M. **Nothing above 3B was measured, and no 8B / 14B / 70B claim is supported.**
+- **RAM tier only.** `stream_source: ram` (`auto` resolves to it). The disk overflow tier ships in **v0.72.2**.
+- **Llama / Qwen only**, `task: sft`, `backend: transformers`, `modality: text`.
+- **`batch_size: 1`**, no gradient accumulation, no `--resume` — all ship in **v0.72.2**.
+- **bf16 base; `quantization: none` is required.** 4-bit (NF4) weights carry a quantisation state and cannot be byte-copied into a plain buffer, so NF4 streaming is **v0.72.1**. A `4bit` config is refused today with a message saying exactly that.
+- **The 3B throughput is a LOWER BOUND.** The reference box could not page-lock the 5.55 GB base (its measured page-locked ceiling is 7.65 GB, and a CUDA context plus the model skeleton did not leave room), so that run fell back to a pageable store. Pageable memory makes the host-to-device copy synchronous, which costs overlap — visible as the GPU-utilisation drop from 96.8% (1.5B, pinned) to 79.3% (3B, pageable). Soup does this fallback automatically **and prints the cost** rather than absorbing it silently.
+- Numbers are Windows/WDDM and therefore systematically pessimistic versus Linux. `expandable_segments:True` is silently ignored on Windows; Soup detects that and does not claim it is active.
+
+**Rejected at config load (each names the release that lifts it):**
+- `stream_source: disk` → the disk tier is v0.72.2
+- `quantization` other than `none` → NF4 streaming is v0.72.1
+- `backend: unsloth` / `backend: mlx` → streaming replaces the model-load path those backends own
+- `task` other than `sft` → preference losses are v0.72.3
+- `gradient_accumulation_steps > 1` → every micro-batch re-reads the whole base, so accumulation multiplies streaming IO linearly (v0.72.2)
+- `batch_size` other than `1` → v0.72.2
+- `lora.use_dora` / `lora.use_vera` / `lora.init_strategy` other than `random` → these initialise from the real base weight, which is on the meta device under streaming
+- `unfrozen_parameters`, `lisa_enabled`, `packing`, `multipack`, `use_fsdp2_compile`, `train_router_only`, `expand_layers` → each independently rewrites or re-freezes the same layers
+- `stream_source` / `stream_buffers` set while `stream_layers: false` → a footgun, refused
+- an architecture outside `llama` / `qwen2` / `qwen3` → named explicitly, with the allowlist
+
+**Config example:**
+
+```yaml
+base: Qwen/Qwen2.5-3B
+task: sft
+backend: transformers
+
+data:
+  train: ./data.jsonl
+  format: alpaca
+  max_length: 512
+  val_split: 0.1
+
+training:
+  epochs: 3
+  lr: 2e-5
+  batch_size: 1           # required; larger batches in v0.72.2
+  gradient_accumulation_steps: 1   # required; accumulation in v0.72.2
+  quantization: none      # required; NF4 streaming is v0.72.1
+  gradient_checkpointing: true     # handled per-layer by the streamer
+  stream_layers: true     # Enable layer streaming
+  stream_source: auto     # RAM-based (disk in v0.72.2)
+  stream_buffers: 2       # double-buffering
+  lora:
+    r: 64
+    alpha: 16
+
+output: ./output
+```
+
+**Performance notes:**
+- 1.43× slower than resident training, measured at 0.5B (the only size on the reference box where a resident baseline genuinely fits in 4 GB and is therefore a fair comparison).
+- The 1.5B runs sit at ~97% GPU utilisation, i.e. compute-bound: with a page-locked store the layer loads hide almost completely behind compute. The 3B run's 79.3% is **not** a model-size effect — it is the cost of the pageable-store fallback on that particular box.
+- Correctness is not a tradeoff: streamed and resident forward passes were verified **bit-exact**, and a 100-step streamed loss curve matched resident exactly. Streaming substitutes the same weight bytes into the same kernels.
+
+**Troubleshooting:**
+- **"layer streaming needs the base to fit in RAM"** — the base is larger than free RAM. Free RAM or pick a smaller base; the disk overflow tier is v0.72.2.
+- **"could not page-lock the base … falling back to a PAGEABLE RAM store"** — expected on a busy machine. Training continues, more slowly. Close other applications to keep the pinned store.
+- **"layer streaming does not support model_type=…"** — v0.72.0 covers Llama and Qwen only.
+- **Slower than you expected** — layer streaming trades time for memory. If the model already fits resident on your card, do not enable it.
+
+**Roadmap (each refusal names its release):**
+- 4-bit (NF4) streaming — **v0.72.1**, the first genuinely useful capability jump
+- Disk overflow tier, batch size > 1, gradient accumulation, checkpoint/resume, more architectures (Mistral / Gemma / Phi) — **v0.72.2**
+- Preference losses (DPO / ORPO / SimPO / KTO) — **v0.72.3**. GRPO and PPO are explicitly **not** planned: rollouts need generation, which re-reads the model per token
+- A published 14B-on-8 GB reference benchmark — hardware-blocked; it needs an 8 GB card and 32 GB of RAM, which the development box does not have
+
+**Shard cache.** The first streaming run rewrites the checkpoint into one safetensors shard per decoder layer under `~/.soup/layer-stream/` (override with `SOUP_LAYER_STREAM_CACHE_DIR`). That costs disk space roughly equal to the base. The cache is keyed to a fingerprint of the source checkpoint, so a base retrained in place re-shards instead of silently training against stale weights.
+
+
 ## Correctness First (v0.36.0)
 
 Four silent-failure modes Soup had → loud failures.

{soup_cli-0.71.41 → soup_cli-0.72.0}/docs/training.md

@@ -4,6 +4,11 @@
 
 > SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, and the loop-hardening detectors.
 
+> **Training a model bigger than your GPU?** `training.stream_layers: true` streams the
+> frozen base from CPU RAM one decoder layer at a time, so peak VRAM is bounded by one
+> layer instead of the whole model —
+> see [Layer Streaming](performance-and-quantization.md#layer-streaming-beta-v0720).
+
 **Contents:**
 
 - [Continual-learning rehearsal (`--replay`)](#continual-learning-rehearsal---replay)

{soup_cli-0.71.41 → soup_cli-0.72.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.41"
+version = "0.72.0"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.41 → soup_cli-0.72.0}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
 
-__version__ = "0.71.41"
+__version__ = "0.72.0"

{soup_cli-0.71.41 → soup_cli-0.72.0}/src/soup_cli/commands/adapters.py

@@ -692,6 +692,18 @@ def arithmetic(
     output: str = typer.Option(
         ..., "--output", "-o", help="Output directory for the merged adapter"
     ),
+    rank: Optional[int] = typer.Option(
+        None,
+        "--rank",
+        min=1,
+        help=(
+            "Truncated-SVD rank cap. Passing it routes the merge through the "
+            "exact concat path and caps the output rank at N via SVD (works for "
+            "same-rank AND mixed-rank inputs). Without it, mixed-rank inputs keep "
+            "the full concatenated rank (Σ rᵢ) and same-rank inputs use the fast "
+            "element-wise path."
+        ),
+    ),
     allow_unscanned: bool = typer.Option(
         False,
         "--allow-unscanned",
@@ -712,23 +724,29 @@ def arithmetic(
 ):
     """Task-vector arithmetic over LoRA adapters (add / scale / negate) (v0.71.34).
 
-    Applies task arithmetic (arXiv:2212.04089) element-wise to LoRA deltas of
-    same-rank adapters. Example:
+    Applies task arithmetic (arXiv:2212.04089) to LoRA deltas. Same-rank adapters
+    take a fast element-wise path; mixed-rank adapters take an exact
+    concatenation path (#305) so ``B_out @ A_out = Σ cᵢ·(Bᵢ@Aᵢ)`` for any
+    per-adapter rank — pass ``--rank N`` to cap the concatenated rank via
+    truncated SVD. Example:
 
         soup adapters arithmetic "coder + 0.5*math - toxic" \\
             --adapter coder=./coder --adapter math=./math \\
             --adapter toxic=./toxic -o ./merged
 
-    Exit 0 = success, 1 = any refusal (bad expression, unknown adapter, rank
+    Exit 0 = success, 1 = any refusal (bad expression, unknown adapter, shape
     mismatch, cross-base without --allow-cross-base, scan FAIL, path escape).
     """
     import re
 
     from soup_cli.utils.adapter_arithmetic import (
         ArithmeticReport,
+        _detect_lora_rank,
         merge_task_arithmetic,
+        merge_task_arithmetic_concat,
         parse_expression,
         read_adapter_base,
+        read_adapter_lora_scaling,
     )
     from soup_cli.utils.adapter_diff import load_adapter_weights
     from soup_cli.utils.adapter_merge import write_merged_adapter
@@ -849,19 +867,63 @@ def arithmetic(
         console.print(f"[red]Could not load adapter weights: {escape(str(exc))}[/]")
         raise typer.Exit(1) from exc
     coeffs = [t.coeff for t in terms]
+
+    # Route: mixed-rank inputs (or an explicit --rank truncation) take the exact
+    # concat path (#305); same-rank inputs keep the fast element-wise path.
+    ranks = [_detect_lora_rank(w) for w in weights_list]
+    distinct_ranks = {r for r in ranks if r is not None}
+    use_concat = len(distinct_ranks) > 1 or rank is not None
+    config_overrides: Optional[dict[str, object]] = None
+
+    if use_concat:
+        # Bake each adapter's own decode-time scaling (lora_alpha/r) into its A
+        # block so a single-scaling output reproduces Σ cᵢ·(effective ΔWᵢ). Use
+        # an explicit `is None` fallback so a valid scaling of 0.0 (lora_alpha=0,
+        # r>0 — an intentionally zeroed task vector) is NOT silently promoted to
+        # 1.0 by `or`.
+        scalings = []
+        for n in referenced:
+            s = read_adapter_lora_scaling(mapping[n])
+            scalings.append(s if s is not None else 1.0)
+        try:
+            merged, skipped, new_rank = merge_task_arithmetic_concat(
+                weights_list, coeffs, scalings=scalings, rank=rank
+            )
+        except ValueError as exc:
+            console.print(f"[red]Merge failed: {escape(str(exc))}[/]")
+            raise typer.Exit(1) from exc
+        if not merged:
+            console.print(
+                "[red]No shared LoRA A/B pairs across the referenced adapters — "
+                "nothing to merge.[/]"
+            )
+            raise typer.Exit(1)
+        # Output is a single-scaling adapter: r == lora_alpha == new_rank. Clear
+        # any per-module rank_pattern/alpha_pattern (v0.39.0) carried over from
+        # the template source — the concat output is uniform-rank, so stale
+        # per-module overrides would mis-scale it.
+        config_overrides = {
+            "r": new_rank,
+            "lora_alpha": new_rank,
+            "rank_pattern": {},
+            "alpha_pattern": {},
+        }
+    else:
+        try:
+            merged, skipped = merge_task_arithmetic(weights_list, coeffs)
+        except ValueError as exc:
+            console.print(f"[red]Merge failed: {escape(str(exc))}[/]")
+            raise typer.Exit(1) from exc
+        if not merged:
+            console.print(
+                "[red]No shared LoRA tensors across the referenced adapters — "
+                "nothing to merge.[/]"
+            )
+            raise typer.Exit(1)
     try:
-        merged, skipped = merge_task_arithmetic(weights_list, coeffs)
-    except ValueError as exc:
-        console.print(f"[red]Merge failed: {escape(str(exc))}[/]")
-        raise typer.Exit(1) from exc
-    if not merged:
-        console.print(
-            "[red]No shared LoRA tensors across the referenced adapters — "
-            "nothing to merge.[/]"
+        write_merged_adapter(
+            output, ref_paths[0], merged, config_overrides=config_overrides
         )
-        raise typer.Exit(1)
-    try:
-        write_merged_adapter(output, ref_paths[0], merged)
     except (ValueError, RuntimeError, OSError) as exc:
         console.print(f"[red]Could not write merged adapter: {escape(str(exc))}[/]")
         raise typer.Exit(1) from exc

{soup_cli-0.71.41 → soup_cli-0.72.0}/src/soup_cli/commands/train.py

@@ -94,6 +94,14 @@ def _hardware_fit_preflight(cfg, gpu_info, *, allow_oom_attempt: bool) -> None:
     statically predictable, so CI and small runs are unaffected. Honors the
     documented ``--allow-oom-attempt`` opt-out.
     """
+    # v0.72.0 — layer streaming bounds peak VRAM by ONE decoder layer, so the
+    # resident prediction (full weights + optimizer + grads on the card) is the
+    # wrong model entirely: it refuses exactly the runs streaming exists to
+    # enable. The streaming path runs its own pre-flight instead (RAM-tier fit
+    # + the plan panel in _setup_streaming_transformers).
+    if getattr(cfg.training, "stream_layers", False):
+        return
+
     total_bytes = 0
     try:
         total_bytes = int(gpu_info.get("memory_total_bytes", 0) or 0)
@@ -522,6 +530,17 @@ def train(
         console.print(f"[red]{markup_escape(str(exc))}[/]")
         raise typer.Exit(code=2) from exc
 
+    # --- v0.72.0 BETA — layer streaming does not support resume yet ---
+    # Silently ignoring --resume would restart from scratch and look like it
+    # worked, so refuse before any heavy work happens.
+    if resume and cfg.training.stream_layers:
+        console.print(
+            "[red]--resume is not supported with training.stream_layers in "
+            "v0.72.0[/] — checkpoint/resume for layer streaming lands in "
+            "v0.72.2. Drop --resume, or set stream_layers: false."
+        )
+        raise typer.Exit(code=2)
+
     # --- RA-DIT generator-stage auto-link (v0.71.10 #200) ---
     # When a generator stage has no retriever model set, splice in the latest
     # RA-DIT retriever output from the Registry. A manual value always wins.