soup-cli 0.71.40__tar.gz → 0.72.0__tar.gz

{soup_cli-0.71.40 → soup_cli-0.72.0}/CHANGELOG.md

@@ -12,6 +12,101 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.72.0] - 2026-07-26
+
+**Layer streaming (BETA) — fine-tune models that don't fit in your card.** The
+frozen base lives in CPU RAM and is streamed into two pre-allocated VRAM buffers
+one decoder layer at a time, so peak VRAM is bounded by the size of *one layer*
+instead of the whole model. Only the LoRA adapters, their gradients and
+optimizer state stay resident. Slower than resident training — but these models
+did not run on the card at all.
+
+Measured on the development box (**RTX 3050 Laptop 4 GB**, Windows 11, 16.9 GB
+RAM), batch 1, gradient checkpointing on, 50 steps after 10 warm-up:
+
+| Model | S | tok/s | GPU util | Peak VRAM |
+|---|---|---|---|---|
+| Qwen2.5-0.5B | 512 | 978.6 | 91.4% | 1.47 GB |
+| Qwen2.5-1.5B | 512 | 525.0 | 96.8% | 1.82 GB |
+| Qwen2.5-1.5B | 1024 | 487.6 | 96.7% | 2.96 GB |
+| Qwen2.5-3B | 512 | 143.1 | 79.3% | **2.15 GB** |
+
+**Qwen2.5-3B trains in 2.15 GB on a 4 GB card where a resident run OOMs.** The
+honest cost: **1.43× slower than resident**, measured at 0.5B — the only
+apples-to-apples comparison available on this box, because 1.5B and above cannot
+run resident here at all.
+
+### Added
+
+- **`training.stream_layers: true`** — stream the frozen base layer-by-layer
+  from CPU RAM. `training.stream_source` (`auto`/`ram`/`disk`) and
+  `training.stream_buffers` (2–8, default 2 = double buffering) tune it.
+- `soup_cli/utils/layer_stream.py` — tier choice, pinned-vs-pageable decision,
+  architecture allowlist, VRAM/throughput arithmetic (no torch import).
+- `soup_cli/utils/layer_shard.py` — rewrites an HF checkpoint into one
+  safetensors shard per decoder layer, one tensor at a time, so sharding a model
+  that does not fit never needs it to fit.
+- `soup_cli/utils/layer_stream_runtime.py` — buffer pool, CPU-RAM weight source,
+  prefetch scheduler on a dedicated CUDA stream, and the streamed layer wrapper.
+- Shards are cached under `~/.soup/layer-stream/` (override with
+  `SOUP_LAYER_STREAM_CACHE_DIR`) and keyed to the source checkpoint's
+  fingerprint, so a base retrained in place re-shards instead of silently
+  training against stale weights.
+- When the base cannot be page-locked, the RAM store falls back to pageable
+  memory **and says so**, including the measured cost (GPU utilisation
+  ~97% → ~79%).
+
+### Changed
+
+- The pre-flight hardware-fit gate is skipped for streaming runs: it models a
+  resident run and would otherwise refuse exactly the runs streaming enables.
+- Gradient checkpointing is handled per-layer by the streamer; the HF Trainer's
+  own is left off so layers are not recomputed twice.
+
+### Known limitations
+
+- **BETA, and proof-of-mechanism at 3B.** Nothing above 3B was measured. No
+  8B/14B claim is supported.
+- Scope: RAM tier, bf16, `task: sft`, Llama/Qwen, batch size 1, no gradient
+  accumulation, no `--resume`. Every refusal names the release that lifts it.
+- 4-bit (NF4) streaming is **v0.72.1** — NF4 weights carry a quantisation state
+  and cannot be byte-copied into a plain buffer.
+- The disk overflow tier, larger batches, gradient accumulation and
+  checkpoint/resume are **v0.72.2**.
+- The 3B number used a **pageable** store (this box cannot page-lock 5.55 GB),
+  so it is a lower bound.
+- `expandable_segments:True` is silently ignored on Windows; Soup detects this
+  and does not claim it is active.
+- Numbers are Windows/WDDM and therefore systematically pessimistic vs Linux.
+
+## [0.71.41] - 2026-07-19
+
+**`soup reward stress`: is your reward verifier gameable?** Turn the reward-hacking
+detector on the *verifier itself*. `soup reward synth` (v0.71.40) proves a verifier
+separates your references from friendly perturbations; `stress` asks the adversarial
+question a reward-hacking model asks at train time — *does the verifier pay out for
+degenerate junk?* It feeds empty, length-padded, repetition, and sentinel-spam
+completions and flags any the verifier accepts. Pure, offline, exit 0 = robust /
+2 = gameable / 1 = error. Nothing in TRL / Unsloth / Axolotl / OpenRLHF tests a
+verifier for gameability.
+
+### Added
+
+- **`soup reward stress <reward.py|builtin> [--references golds.jsonl]`** — adversarial
+  verifier probe. Attacks (`--attacks empty,length,repetition,sentinel`, `--sentinel`)
+  are scored against the real gold, so numeric / tool_call / json_schema verifiers get a
+  valid target and still must reject the junk. Reports a per-attack accept-rate table +
+  an overall gameability verdict (`--max-gameable`, `--threshold`, `--output-report`).
+  Loads the target through the existing reward loader, so it probes a synthesized `.py`
+  **and** a builtin (`accuracy` / `format` / `verifiable`). A gold-requiring verifier
+  probed with no `--references` is a hard error, never a false "robust".
+
+### Fixed
+
+- Corrected the Telemetry section in the ops docs: Soup's telemetry primitives exist but
+  are **not wired to any command** — no data is ever sent today (the previous wording
+  implied a live opt-in sender). Wiring is deferred until a public privacy policy ships.
+
 ## [0.71.40] - 2026-07-19
 
 **`soup reward synth`: auto-generate a deterministic reward verifier from your data.**

{soup_cli-0.71.40 → soup_cli-0.72.0}/CONTRIBUTING.md

@@ -116,11 +116,11 @@ src/soup_cli/
   cans/               - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0)
   data/traces/        - Trace-to-Preference harvester (v0.26.0)
   data/collators.py   - CrossDocCollator for sample packing (v0.33.0)
-  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint, shrink, best_of_n, evolve
+  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint, shrink, best_of_n, evolve, reward_synth, reward_stress, layer_stream, layer_shard, layer_stream_runtime
   templates/          - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (318 files, 16490 tests)
+tests/                - Test suite (324 files, 16735 tests)
 examples/             - Real-world config examples and datasets
 ```
 
@@ -150,7 +150,7 @@ pytest tests/test_data.py::test_detect_alpaca_format -v
 pytest tests/ --cov=soup_cli --cov-report=html
 ```
 
-### Test Files (313 files)
+### Test Files (318 files)
 
 > A representative sample of the suite below. Run `pytest tests/ -v` for the complete list.
 

{soup_cli-0.71.40 → soup_cli-0.72.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.40
+Version: 0.72.0
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -193,27 +193,44 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.40 — `soup reward synth`: generate a reward verifier from your data.** Point it at a JSONL of reference (gold) outputs and it infers a deterministic verifier, writes a readable / committable `.py` reward function, and — the part nobody else does — *refuses* to emit one that can't tell your references from bad answers. Nothing in TRL / Unsloth / Axolotl synthesizes a reward.
+**v0.72.0 — Layer streaming (BETA).** Fine-tune models that don't fit in your card. The frozen
+base streams from CPU RAM one decoder layer at a time into a small pool of VRAM buffers while the
+LoRA adapters stay resident, so peak VRAM is bounded by one layer rather than the whole model.
+**Qwen2.5-3B trains in 2.15 GB on a 4 GB card, where a resident run OOMs.**
+
+- **`training.stream_layers: true`** — a config key, not a CLI flag. Tune with
+  `stream_source` (`ram` in v0.72.0; `disk` is v0.72.2) and `stream_buffers` (2–8, default 2).
+- **Measured on a 4 GB RTX 3050 Laptop** (batch 1, gradient checkpointing on): 0.5B at
+  978.6 tok/s / 1.47 GB, 1.5B at 525.0 tok/s / 1.82 GB, 3B at 143.1 tok/s / 2.15 GB.
+- **Honest cost: 1.43× slower than resident**, measured at 0.5B — the only apples-to-apples
+  comparison available on that box, because 1.5B and above cannot run resident there at all.
+- **Proof-of-mechanism at 3B.** Nothing above 3B was measured; no 8B/14B claim is supported.
+  Scope: RAM tier, bf16, `task: sft`, Llama/Qwen, batch size 1, no gradient accumulation, no
+  `--resume`. 4-bit (NF4) streaming is **v0.72.1** and is refused with a clear message today.
 
-- **Four families, auto-detected** (or pick with `--kind`): `numeric` (`\boxed{}` / `####` / last-number,
-  exact or `--tolerance`), `json_schema` (induced keys + types + required), `regex`, and `tool_call`
-  (per-tool `required`/`allowed` argument binding — a call can't borrow another tool's keys).
-- **A calibration report is the moat.** The verifier is loaded back and run against its own references
-  (must accept ≥ 90%) and auto-generated bad answers (must reject). A degenerate always-accept verifier
-  is **refused** (exit 2), never silently shipped. `--plan-only` previews; `--output-report` saves the JSON.
-- **No new exec surface.** The emitted file rides the existing `reward_fn: reward.py` path — you read,
-  edit, commit, and diff it like any other reward.
-- **Reward ensembles now train.** `reward_fn: "accuracy,format"` loads as multiple rewards (and unlocks
-  the `rm_ensemble` reward-hack detector) — a recipe that shipped exactly this used to crash. (#311)
+```yaml
+# soup.yaml — then just `soup train --config soup.yaml`
+training:
+  stream_layers: true      # base streams from RAM; only the adapter trains
+  batch_size: 1
+  quantization: none       # NF4 streaming lands in v0.72.1
+```
+
+<details>
+<summary>Previous release — v0.71.40, soup reward synth (generate a reward verifier from your data)</summary>
+
+Point `soup reward synth` at a JSONL of reference outputs and it infers a deterministic verifier,
+writes a readable / committable `.py` reward function, and — the part nobody else does — *refuses* to
+emit one that can't tell your references from bad answers (four families: `numeric` / `json_schema` /
+`regex` / `tool_call`; a mandatory calibration report is the moat). Reward ensembles
+(`reward_fn: "accuracy,format"`) also train now. (#311)
 
 ```bash
-# synth a verifier from reference answers, calibrate it, keep it only if it discriminates
 soup reward synth references.jsonl -o reward.py --output-report calib.json
-
-# then train against the reward you just generated
-# training: { reward_fn: reward.py }   (or an ensemble: reward_fn: "accuracy,format")
 ```
 
+</details>
+
 <details>
 <summary>Previous release — v0.71.39, CI for weights not prompts (emit + provenance-bind the ship verdict)</summary>
 
@@ -357,7 +374,7 @@ The full feature reference lives in [`docs/`](docs/). Start here:
 |---|---|
 | [Training tasks & methods](docs/training.md) | SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, loop-hardening detectors |
 | [PEFT, long context & efficiency](docs/peft-and-efficiency.md) | DoRA, LoRA+, rsLoRA, VeRA, OLoRA, NEFTune, PiSSA, ReLoRA, optimizer & PEFT zoo, LLaMA Pro, GaLore, YaRN/LongLoRA, packing, curriculum, auto-tuning |
-| [Performance & quantization](docs/performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, multi-GPU / DeepSpeed / FSDP |
+| [Performance & quantization](docs/performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, layer streaming, multi-GPU / DeepSpeed / FSDP |
 | [Data engineering](docs/data.md) | Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs |
 | [Evaluation & probes](docs/evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, post-train X-ray probes, A/B, drift, tunability, `soup advise` |
 | [Serving & export](docs/serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge |

{soup_cli-0.71.40 → soup_cli-0.72.0}/README.md

@@ -49,27 +49,44 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.40 — `soup reward synth`: generate a reward verifier from your data.** Point it at a JSONL of reference (gold) outputs and it infers a deterministic verifier, writes a readable / committable `.py` reward function, and — the part nobody else does — *refuses* to emit one that can't tell your references from bad answers. Nothing in TRL / Unsloth / Axolotl synthesizes a reward.
-
-- **Four families, auto-detected** (or pick with `--kind`): `numeric` (`\boxed{}` / `####` / last-number,
-  exact or `--tolerance`), `json_schema` (induced keys + types + required), `regex`, and `tool_call`
-  (per-tool `required`/`allowed` argument binding — a call can't borrow another tool's keys).
-- **A calibration report is the moat.** The verifier is loaded back and run against its own references
-  (must accept ≥ 90%) and auto-generated bad answers (must reject). A degenerate always-accept verifier
-  is **refused** (exit 2), never silently shipped. `--plan-only` previews; `--output-report` saves the JSON.
-- **No new exec surface.** The emitted file rides the existing `reward_fn: reward.py` path — you read,
-  edit, commit, and diff it like any other reward.
-- **Reward ensembles now train.** `reward_fn: "accuracy,format"` loads as multiple rewards (and unlocks
-  the `rm_ensemble` reward-hack detector) — a recipe that shipped exactly this used to crash. (#311)
+**v0.72.0 — Layer streaming (BETA).** Fine-tune models that don't fit in your card. The frozen
+base streams from CPU RAM one decoder layer at a time into a small pool of VRAM buffers while the
+LoRA adapters stay resident, so peak VRAM is bounded by one layer rather than the whole model.
+**Qwen2.5-3B trains in 2.15 GB on a 4 GB card, where a resident run OOMs.**
+
+- **`training.stream_layers: true`** — a config key, not a CLI flag. Tune with
+  `stream_source` (`ram` in v0.72.0; `disk` is v0.72.2) and `stream_buffers` (2–8, default 2).
+- **Measured on a 4 GB RTX 3050 Laptop** (batch 1, gradient checkpointing on): 0.5B at
+  978.6 tok/s / 1.47 GB, 1.5B at 525.0 tok/s / 1.82 GB, 3B at 143.1 tok/s / 2.15 GB.
+- **Honest cost: 1.43× slower than resident**, measured at 0.5B — the only apples-to-apples
+  comparison available on that box, because 1.5B and above cannot run resident there at all.
+- **Proof-of-mechanism at 3B.** Nothing above 3B was measured; no 8B/14B claim is supported.
+  Scope: RAM tier, bf16, `task: sft`, Llama/Qwen, batch size 1, no gradient accumulation, no
+  `--resume`. 4-bit (NF4) streaming is **v0.72.1** and is refused with a clear message today.
+
+```yaml
+# soup.yaml — then just `soup train --config soup.yaml`
+training:
+  stream_layers: true      # base streams from RAM; only the adapter trains
+  batch_size: 1
+  quantization: none       # NF4 streaming lands in v0.72.1
+```
+
+<details>
+<summary>Previous release — v0.71.40, soup reward synth (generate a reward verifier from your data)</summary>
+
+Point `soup reward synth` at a JSONL of reference outputs and it infers a deterministic verifier,
+writes a readable / committable `.py` reward function, and — the part nobody else does — *refuses* to
+emit one that can't tell your references from bad answers (four families: `numeric` / `json_schema` /
+`regex` / `tool_call`; a mandatory calibration report is the moat). Reward ensembles
+(`reward_fn: "accuracy,format"`) also train now. (#311)
 
 ```bash
-# synth a verifier from reference answers, calibrate it, keep it only if it discriminates
 soup reward synth references.jsonl -o reward.py --output-report calib.json
-
-# then train against the reward you just generated
-# training: { reward_fn: reward.py }   (or an ensemble: reward_fn: "accuracy,format")
 ```
 
+</details>
+
 <details>
 <summary>Previous release — v0.71.39, CI for weights not prompts (emit + provenance-bind the ship verdict)</summary>
 
@@ -213,7 +230,7 @@ The full feature reference lives in [`docs/`](docs/). Start here:
 |---|---|
 | [Training tasks & methods](docs/training.md) | SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, loop-hardening detectors |
 | [PEFT, long context & efficiency](docs/peft-and-efficiency.md) | DoRA, LoRA+, rsLoRA, VeRA, OLoRA, NEFTune, PiSSA, ReLoRA, optimizer & PEFT zoo, LLaMA Pro, GaLore, YaRN/LongLoRA, packing, curriculum, auto-tuning |
-| [Performance & quantization](docs/performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, multi-GPU / DeepSpeed / FSDP |
+| [Performance & quantization](docs/performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, layer streaming, multi-GPU / DeepSpeed / FSDP |
 | [Data engineering](docs/data.md) | Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs |
 | [Evaluation & probes](docs/evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, post-train X-ray probes, A/B, drift, tunability, `soup advise` |
 | [Serving & export](docs/serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge |

{soup_cli-0.71.40 → soup_cli-0.72.0}/SECURITY.md

@@ -7,8 +7,8 @@ running the most recent version from PyPI (`pip install -U soup-cli`).
 
 | Version | Supported          |
 | ------- | ------------------ |
-| 0.71.x  | :white_check_mark: |
-| < 0.71  | :x:                |
+| 0.72.x  | :white_check_mark: |
+| < 0.72  | :x:                |
 
 ## Reporting a Vulnerability
 

{soup_cli-0.71.40 → soup_cli-0.72.0}/docs/README.md

@@ -7,9 +7,9 @@ feature reference — every `soup` capability, grouped by area.
 
 | Guide | Covers |
 |---|---|
-| [Training tasks & methods](training.md) | SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, loop-hardening detectors |
+| [Training tasks & methods](training.md) | SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, loop-hardening detectors, reward-verifier synthesis |
 | [PEFT, long context & efficiency](peft-and-efficiency.md) | DoRA, LoRA+, rsLoRA, VeRA, OLoRA, NEFTune, PiSSA, ReLoRA, optimizer & PEFT zoo, LLaMA Pro, GaLore, YaRN/LongLoRA, packing, curriculum, auto-tuning, depth pruning + distill-heal (`soup shrink`) |
-| [Performance & quantization](performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, multi-GPU / DeepSpeed / FSDP |
+| [Performance & quantization](performance-and-quantization.md) | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, layer streaming, multi-GPU / DeepSpeed / FSDP |
 | [Data engineering](data.md) | Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs |
 | [Evaluation & probes](evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, `soup ship` verdict, post-train X-ray probes, A/B, drift, tunability, `soup advise` |
 | [Serving & export](serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge |

{soup_cli-0.71.40 → soup_cli-0.72.0}/docs/backends-and-ops.md

@@ -743,15 +743,9 @@ pip install soup-cli[trackers]   # mlflow + swanlab + trackio
 ```
 
 
-## Telemetry (opt-IN, hardware-info-only)
+## Telemetry (not yet wired)
 
-Soup ships an opt-IN telemetry sender that POSTs hardware-info-only payloads (`soup_version` / `command` / `python` major.minor / `os` / `arch` / optional `duration_seconds`) — no dataset paths, model names, or config contents. Enable per-shell:
-
-```bash
-SOUP_TELEMETRY=1 soup train --config soup.yaml
-```
-
-The sender uses a 1-second hard timeout, HTTPS-only with private-IP / link-local rejection (same SSRF policy as hub endpoints), and swallows every exception silently — telemetry can never crash training. Disabled by default until a public privacy policy ships.
+Soup contains opt-in, hardware-info-only telemetry primitives in `utils/trackers.py` (`build_telemetry_payload` / `send_telemetry_payload`), but they are **not wired to any command** — no data is ever sent, and no environment variable enables sending today. When wired, the payload will carry only `soup_version` / `command` / `python` major.minor / `os` / `arch` / optional `duration_seconds` — never dataset paths, model names, or config contents — behind a 1-second hard timeout and the same HTTPS-only, private-IP-rejecting SSRF policy as hub endpoints, swallowing every exception so telemetry can never crash training. Wiring is deferred until a public privacy policy is published.
 
 
 ## Plugin System

{soup_cli-0.71.40 → soup_cli-0.72.0}/docs/commands.md

@@ -30,6 +30,7 @@ soup merge-sharded-fsdp-weights ./shards -o merged.safetensors  Consolidate FSDP
 soup delinearize-llama4 ./src --target ./out [--num-experts N] [--plan-only]  Live Llama-4 fused-expert reshape [E*din,dout] -> [E,din,dout] + sidecar copy (v0.71.21)
 soup spectrum scan --model <id|path> --top-percent 50 [--modules mlp,attn] [-o patch.yaml]  Spectrum SNR scan (no model load) -> training.unfrozen_parameters YAML patch (v0.71.23)
 soup train --config sft.yaml  # training.lisa_enabled: true [lisa_num_layers lisa_interval_steps]  LISA layerwise importance sampling — full-FT quality at LoRA-like memory (sft/transformers/text/quantization=none) (v0.71.34)
+soup train --config sft.yaml  # training.stream_layers: true [stream_source stream_buffers]  BETA layer streaming — the frozen base streams from CPU RAM one decoder layer at a time, so peak VRAM is bounded by ONE layer (sft/transformers/text/quantization=none, batch 1) (v0.72.0)
 soup export --model ./output --format gguf    Export to GGUF (Ollama)
 soup export --model ./output --deploy ollama  Export GGUF + auto-deploy to Ollama
 soup export --model ./output --format onnx    Export to ONNX
@@ -186,6 +187,10 @@ soup reward synth refs.jsonl -o reward.py     Synthesize a deterministic reward
 soup reward synth ... --kind numeric|json_schema|regex|tool_call  Force a verifier family (default: auto-detect)
 soup reward synth ... --plan-only             Report the induced spec + calibration plan; write nothing
 soup reward synth ... --output-report r.json --min-discrimination 0.5  Save the calibration JSON / set the refusal threshold (exit 0 emit / 2 refuse / 1 error)
+soup reward stress reward.py --references golds.jsonl  Adversarially probe a verifier for gameability — empty/length/repetition/sentinel junk (v0.71.41)
+soup reward stress verifiable --verifiable-domain math --references golds.jsonl  Probe a builtin verifier instead of a .py file
+soup reward stress ... --attacks empty,length,repetition,sentinel --sentinel GOLD --threshold 0.5 --max-gameable 0.0  Tune the attack set / accept threshold / tolerance
+soup reward stress ... --output-report r.json  Save the per-attack report JSON (exit 0 robust / 2 gameable / 1 error)
 soup tui                                      Full-screen Textual dashboard (requires [tui] extra)
 soup train --config soup.yaml --profile       Record torch.profiler trace to <output>/profiles/
 soup --log-level quiet|normal|verbose|debug   Global logging tier (Rich-formatted)

{soup_cli-0.71.40 → soup_cli-0.72.0}/docs/performance-and-quantization.md

@@ -14,6 +14,7 @@
 - [Cross-Document Attention Masking](#cross-document-attention-masking)
 - [Quant Menu — 9 Quantization Formats](#quant-menu--9-quantization-formats)
 - [Activation Offloading (Small-VRAM Large-Batch)](#activation-offloading-small-vram-large-batch)
+- [Layer Streaming (BETA, v0.72.0)](#layer-streaming-beta-v0720)
 - [Correctness First (v0.36.0)](#correctness-first-v0360)
 - [Multi-GPU / DeepSpeed / FSDP](#multi-gpu--deepspeed--fsdp)
 - [Performance + Long-Context](#performance--long-context)
@@ -225,6 +226,108 @@ training:
 Not compatible with unsloth (own memory manager) or mlx. Wired across every transformer-backend trainer (SFT, DPO, GRPO, KTO, ORPO, SimPO, IPO, PPO, Reward-Model, Embedding, Pretrain).
 
 
+## Layer Streaming (BETA, v0.72.0)
+
+Stream frozen base-model decoder layers ONE at a time from CPU RAM into small VRAM buffers instead of keeping the whole base resident. Peak VRAM is bounded by the size of a single layer, not the entire model — so models that don't fit resident on your GPU can now train at all.
+
+```yaml
+training:
+  stream_layers: true          # Enable layer streaming
+  stream_source: auto          # 'auto' (same-host RAM), 'ram', 'disk' (v0.72.2)
+  stream_buffers: 2            # Double-buffering; range [2, 8]
+```
+
+```bash
+# Layer streaming is a CONFIG key, not a CLI flag — just train normally:
+soup train --config soup.yaml
+```
+
+**How it works.** LoRA adapters + their gradients + optimizer state stay resident in VRAM (they are small). The frozen base lives in CPU RAM, page-locked when the machine allows it, and is streamed: each decoder layer is copied into one of two pre-allocated VRAM buffers on a dedicated CUDA stream while the previous layer is still computing, so the load overlaps the compute. Each layer is read **twice** per step — once in the forward pass and once when the backward pass recomputes it — because `dL/dx = Wᵀ · dL/dy` needs the weights to reach the layers below. That is physics, not an implementation detail, and it is why streaming costs time.
+
+The tradeoff: **1.43× slower than resident training**, measured at 0.5B — the only apples-to-apples comparison available on the reference box, because 1.5B and above cannot run resident there at all.
+
+**Measured numbers (RTX 3050 Laptop 4 GB, Windows 11, bf16 LoRA, batch 1, 50 steps after 10 warmup):**
+
+| Model | Seq Length | Throughput | GPU Util | Peak VRAM |
+|---|---|---|---|---|
+| Qwen2.5-0.5B | 512 | 978.6 tok/s | 91.4% | 1.47 GB |
+| Qwen2.5-1.5B | 512 | 525.0 tok/s | 96.8% | 1.82 GB |
+| Qwen2.5-1.5B | 1024 | 487.6 tok/s | 96.7% | 2.96 GB |
+| Qwen2.5-3B | 512 | 143.1 tok/s | 79.3% | 2.15 GB |
+
+**Headline:** Qwen2.5-3B trains in **2.15 GB** on a 4 GB card, where a resident run OOMs.
+
+**Honest scope (v0.72.0 = proof-of-mechanism):**
+- **Models measured:** Qwen2.5-0.5B, 1.5B and 3B, plus a live `soup train` on SmolLM2-135M. **Nothing above 3B was measured, and no 8B / 14B / 70B claim is supported.**
+- **RAM tier only.** `stream_source: ram` (`auto` resolves to it). The disk overflow tier ships in **v0.72.2**.
+- **Llama / Qwen only**, `task: sft`, `backend: transformers`, `modality: text`.
+- **`batch_size: 1`**, no gradient accumulation, no `--resume` — all ship in **v0.72.2**.
+- **bf16 base; `quantization: none` is required.** 4-bit (NF4) weights carry a quantisation state and cannot be byte-copied into a plain buffer, so NF4 streaming is **v0.72.1**. A `4bit` config is refused today with a message saying exactly that.
+- **The 3B throughput is a LOWER BOUND.** The reference box could not page-lock the 5.55 GB base (its measured page-locked ceiling is 7.65 GB, and a CUDA context plus the model skeleton did not leave room), so that run fell back to a pageable store. Pageable memory makes the host-to-device copy synchronous, which costs overlap — visible as the GPU-utilisation drop from 96.8% (1.5B, pinned) to 79.3% (3B, pageable). Soup does this fallback automatically **and prints the cost** rather than absorbing it silently.
+- Numbers are Windows/WDDM and therefore systematically pessimistic versus Linux. `expandable_segments:True` is silently ignored on Windows; Soup detects that and does not claim it is active.
+
+**Rejected at config load (each names the release that lifts it):**
+- `stream_source: disk` → the disk tier is v0.72.2
+- `quantization` other than `none` → NF4 streaming is v0.72.1
+- `backend: unsloth` / `backend: mlx` → streaming replaces the model-load path those backends own
+- `task` other than `sft` → preference losses are v0.72.3
+- `gradient_accumulation_steps > 1` → every micro-batch re-reads the whole base, so accumulation multiplies streaming IO linearly (v0.72.2)
+- `batch_size` other than `1` → v0.72.2
+- `lora.use_dora` / `lora.use_vera` / `lora.init_strategy` other than `random` → these initialise from the real base weight, which is on the meta device under streaming
+- `unfrozen_parameters`, `lisa_enabled`, `packing`, `multipack`, `use_fsdp2_compile`, `train_router_only`, `expand_layers` → each independently rewrites or re-freezes the same layers
+- `stream_source` / `stream_buffers` set while `stream_layers: false` → a footgun, refused
+- an architecture outside `llama` / `qwen2` / `qwen3` → named explicitly, with the allowlist
+
+**Config example:**
+
+```yaml
+base: Qwen/Qwen2.5-3B
+task: sft
+backend: transformers
+
+data:
+  train: ./data.jsonl
+  format: alpaca
+  max_length: 512
+  val_split: 0.1
+
+training:
+  epochs: 3
+  lr: 2e-5
+  batch_size: 1           # required; larger batches in v0.72.2
+  gradient_accumulation_steps: 1   # required; accumulation in v0.72.2
+  quantization: none      # required; NF4 streaming is v0.72.1
+  gradient_checkpointing: true     # handled per-layer by the streamer
+  stream_layers: true     # Enable layer streaming
+  stream_source: auto     # RAM-based (disk in v0.72.2)
+  stream_buffers: 2       # double-buffering
+  lora:
+    r: 64
+    alpha: 16
+
+output: ./output
+```
+
+**Performance notes:**
+- 1.43× slower than resident training, measured at 0.5B (the only size on the reference box where a resident baseline genuinely fits in 4 GB and is therefore a fair comparison).
+- The 1.5B runs sit at ~97% GPU utilisation, i.e. compute-bound: with a page-locked store the layer loads hide almost completely behind compute. The 3B run's 79.3% is **not** a model-size effect — it is the cost of the pageable-store fallback on that particular box.
+- Correctness is not a tradeoff: streamed and resident forward passes were verified **bit-exact**, and a 100-step streamed loss curve matched resident exactly. Streaming substitutes the same weight bytes into the same kernels.
+
+**Troubleshooting:**
+- **"layer streaming needs the base to fit in RAM"** — the base is larger than free RAM. Free RAM or pick a smaller base; the disk overflow tier is v0.72.2.
+- **"could not page-lock the base … falling back to a PAGEABLE RAM store"** — expected on a busy machine. Training continues, more slowly. Close other applications to keep the pinned store.
+- **"layer streaming does not support model_type=…"** — v0.72.0 covers Llama and Qwen only.
+- **Slower than you expected** — layer streaming trades time for memory. If the model already fits resident on your card, do not enable it.
+
+**Roadmap (each refusal names its release):**
+- 4-bit (NF4) streaming — **v0.72.1**, the first genuinely useful capability jump
+- Disk overflow tier, batch size > 1, gradient accumulation, checkpoint/resume, more architectures (Mistral / Gemma / Phi) — **v0.72.2**
+- Preference losses (DPO / ORPO / SimPO / KTO) — **v0.72.3**. GRPO and PPO are explicitly **not** planned: rollouts need generation, which re-reads the model per token
+- A published 14B-on-8 GB reference benchmark — hardware-blocked; it needs an 8 GB card and 32 GB of RAM, which the development box does not have
+
+**Shard cache.** The first streaming run rewrites the checkpoint into one safetensors shard per decoder layer under `~/.soup/layer-stream/` (override with `SOUP_LAYER_STREAM_CACHE_DIR`). That costs disk space roughly equal to the base. The cache is keyed to a fingerprint of the source checkpoint, so a base retrained in place re-shards instead of silently training against stale weights.
+
+
 ## Correctness First (v0.36.0)
 
 Four silent-failure modes Soup had → loud failures.

{soup_cli-0.71.40 → soup_cli-0.72.0}/docs/training.md

@@ -4,6 +4,11 @@
 
 > SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, and the loop-hardening detectors.
 
+> **Training a model bigger than your GPU?** `training.stream_layers: true` streams the
+> frozen base from CPU RAM one decoder layer at a time, so peak VRAM is bounded by one
+> layer instead of the whole model —
+> see [Layer Streaming](performance-and-quantization.md#layer-streaming-beta-v0720).
+
 **Contents:**
 
 - [Continual-learning rehearsal (`--replay`)](#continual-learning-rehearsal---replay)
@@ -897,6 +902,31 @@ strongly the verifier must separate references from perturbed negatives before i
 v1 is deterministic families only — a `\boxed{}`/`####` marker helps the numeric verifier, and
 completions are prompted to mark their answer (standard RLVR practice).
 
+### Stress-test a verifier for gameability (`soup reward stress`)
+
+A verifier that passes calibration still might pay out for junk. `soup reward stress` feeds the
+verifier deterministic degenerate completions — empty, length-padded, repeated, and
+sentinel-spam — scored against your real gold answers, and flags any it **accepts**. It's the
+adversarial counterpart to `synth`: calibration proves the verifier tells references from
+*friendly* bad answers; `stress` asks whether a reward-hacking model could game it.
+
+```bash
+# probe a synthesized verifier (or any reward .py) — exit 0 robust, 2 gameable, 1 error
+soup reward stress reward.py --references golds.jsonl --output-report stress.json
+
+# probe a builtin verifier instead of a .py file
+soup reward stress verifiable --verifiable-domain math --references golds.jsonl
+
+# tune the attack set / accept threshold / gameability tolerance
+soup reward stress reward.py --references golds.jsonl \
+    --attacks empty,length,repetition,sentinel --sentinel GOLD \
+    --threshold 0.5 --max-gameable 0.0
+```
+
+The report shows a per-attack accept-rate and an overall verdict. A gold-requiring verifier probed
+with **no** `--references` is a hard error (it can't be measured), never a false "robust". Probing a
+`.py` executes its module code, like any custom reward — only stress files you trust.
+
 ### Verifiable Rewards (RLVR)
 
 Use `reward_fn: verifiable` with a `verifiable_domain` for deterministic, math-checkable rewards — no judge model, no heuristics. Great for GRPO on math, code, or structured-output tasks.

{soup_cli-0.71.40 → soup_cli-0.72.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.40"
+version = "0.72.0"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.40 → soup_cli-0.72.0}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
 
-__version__ = "0.71.40"
+__version__ = "0.72.0"