soup-cli 0.72.0__tar.gz → 0.72.2__tar.gz

soup_cli-0.72.2/.github/FUNDING.yml

@@ -0,0 +1,7 @@
+# Sponsor button shown in the repo sidebar and on every issue / PR.
+#
+# `github:` only renders once the account is enrolled in GitHub Sponsors
+# (https://github.com/sponsors) — it is not, so it stays commented out.
+# github: MakazhanAlpamys
+
+custom: ["https://buy.stripe.com/4gMcN441k3pha3T19ye7m04"]

{soup_cli-0.72.0 → soup_cli-0.72.2}/CHANGELOG.md

@@ -12,8 +12,131 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.72.2] - 2026-07-28
+
+**Added — NF4 layer streaming: fine-tune Llama-3.1-8B on a 4 GB laptop GPU.**
+
+Layer streaming (v0.72.0) keeps the frozen base in CPU RAM and streams it to the
+GPU one decoder layer at a time, so peak VRAM is bounded by one layer instead of
+the whole model. It was bf16-only, which capped it at about 3B on a small card.
+Quantising the streamed base to NF4 shrinks it ~4×, and that is what brings 8B
+within reach.
+
+Add one line to a streaming config:
+
+```yaml
+training:
+  stream_layers: true
+  quantization: 4bit    # NF4
+  batch_size: 1
+```
+
+**Measured on a 4 GB RTX 3050 Laptop** (Windows, batch 1, S=512, gradient
+checkpointing, 50 steps after 10 warm-up, `PagedAdamW8bit`):
+
+| Model | tok/s | Peak VRAM | RAM store | GPU util |
+|---|---|---|---|---|
+| Llama-3.1-8B-Instruct | 119.6 | 3.32 GB | 3.60 GB (page-locked) | 100% |
+| Qwen2.5-3B | 264.2 | 1.76 GB | 1.43 GB (page-locked) | 100% |
+
+For scale: 1M training tokens is about 2.3 h at 8B on that card (arithmetic from
+the measured rate, not a separate measurement).
+
+Qwen2.5-3B also got **1.85× faster** than the bf16 streaming path (264.2 vs
+143.1 tok/s) — and the reason is not arithmetic. A 1.43 GB store fits under the
+machine's page-locked memory ceiling where a 5.55 GB one did not, which restores
+asynchronous host-to-device copies and lifts GPU utilisation from 79.3% to 100%.
+
+**Correctness.** A streamed NF4 run is **bit-exact** against a resident NF4 run:
+the same quantised bytes through the same bitsandbytes kernels. Logit equality,
+non-zero gradients at layer 0, and a matching multi-step loss curve are all
+regression tests, not one-off measurements.
+
+The base is quantised once, offline, and cached under `~/.soup/layer-stream/`.
+The cache is keyed to the quantisation *and* the source checkpoint, so switching
+between `none` and `4bit`, or retraining a base in place, re-shards instead of
+silently streaming the wrong bytes.
+
+**Fixed — a streamed 4-bit run reported its parameter count ~6.5× too high.**
+SmolLM2-135M printed "878,154,048 total" (true: 134,515,008). Display only —
+training was unaffected — but at 8B it would have read ~52 B.
+
+Scope is unchanged and still BETA: RAM tier, `task: sft`, Llama/Qwen,
+`batch_size: 1`, no gradient accumulation, no `--resume`. Every rejected config
+names the release that lifts it. `quantization` values other than `none` and
+`4bit` are refused.
+
+**Fixed — `soup --help` was 5x slower than it should be.** Since v0.72.0 the CLI
+imported PyTorch on startup, taking **6.0 s** where it now takes **1.15 s**.
+
+`soup reward stress` (v0.71.41) put `utils/reward_stress` on the light CLI path.
+That module imported `utils/reward_hack_control` to reuse a single string
+constant — and `reward_hack_control` resolves its `TrainerCallback` base class at
+module scope, which pulls in `transformers` and `torch`. Importing a ~4.4 s
+dependency for one constant made every `soup` invocation pay for the training
+stack, including commands that never touch a model.
+
+Nothing produced wrong results; this was purely startup latency. The light core
+(`pip install soup-cli` without the `[train]` extra) was never broken — it fell
+back cleanly when torch was absent, just slowly when it was present.
+
+Added `tests/test_cli_startup_is_light.py`, which asserts the invariant at
+runtime (`import soup_cli.cli` must not put `torch` in `sys.modules`) instead of
+inspecting source text for `import torch`, which is what the previous guards did
+and why this went unnoticed.
+
+## [0.72.1] - 2026-07-27
+
+**Fixed — layer-streaming adapters were saved in an unloadable form.** If you
+trained with `stream_layers: true` on v0.72.0, the adapter that run wrote is
+**inert**: every tensor was saved under a key containing an extra `.inner.`
+segment, so `soup merge`, `soup serve`, `soup chat` and
+`PeftModel.from_pretrained` all loaded **zero** adapter tensors and silently
+returned the untuned base model. PEFT emitted only a `UserWarning`, so nothing
+failed and nothing looked wrong.
+
+The training itself was correct — the streamed run's numerics are unaffected,
+and v0.72.0's bit-exactness results still stand. Only the saved file was
+affected.
+
+**If you have a v0.72.0 streamed adapter: re-save or re-run it on v0.72.1.**
+There is no way to recover the original file's association with the base model
+beyond renaming its keys; re-running is the reliable path. A quick check —
+if `adapter_model.safetensors` contains keys with `.inner.` in them, it is
+affected:
+
+```bash
+python -c "from safetensors.torch import load_file; \
+print([k for k in load_file('adapter_model.safetensors') if '.inner.' in k][:3])"
+```
+
+Streamed adapters now save byte-for-byte in the same layout as an ordinary LoRA
+run, and are portable to any tool that has never heard of layer streaming.
+
+**Also fixed — `--hf-resume` bypassed the streaming resume refusal.** The guard
+only tested `--resume`, but `--hf-resume` reaches `resume_from` through a
+different branch. That combination previously appeared to work by accident
+(checkpoint and live model shared the same key shape); once adapters are saved
+canonically it would instead have matched *nothing* and continued with a
+freshly initialised adapter, silently. Both flags are now refused for streaming
+runs, naming v0.72.3.
+
+Also in this release: every "this lands in vX.Y.Z" refusal message was corrected
+after the v0.72.x roadmap was renumbered (NF4 streaming is now v0.72.2; the disk
+tier, wider architectures, larger batches, gradient accumulation and
+checkpoint/resume are v0.72.3; preference losses are v0.72.4).
+
+**Known limitation:** in memory the streamed model's `named_parameters()` still
+carries the wrapper segment, so loading *into* a streaming run (`--resume`)
+remains unsupported and is refused with a message naming v0.72.3.
+
 ## [0.72.0] - 2026-07-26
 
+> **Superseded by v0.72.1 — adapters saved by this version load as zero
+> tensors.** The entry below is left as published; the defect and the fix are
+> described under [0.72.1]. Version numbers named as "upcoming" below were also
+> renumbered there (NF4 is v0.72.2, not v0.72.1).
+
 **Layer streaming (BETA) — fine-tune models that don't fit in your card.** The
 frozen base lives in CPU RAM and is streamed into two pre-allocated VRAM buffers
 one decoder layer at a time, so peak VRAM is bounded by the size of *one layer*

{soup_cli-0.72.0 → soup_cli-0.72.2}/CONTRIBUTING.md

@@ -120,7 +120,7 @@ src/soup_cli/
   templates/          - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (324 files, 16735 tests)
+tests/                - Test suite (326 files, 16840 tests)
 examples/             - Real-world config examples and datasets
 ```
 

{soup_cli-0.72.0 → soup_cli-0.72.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.72.0
+Version: 0.72.2
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -193,29 +193,38 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.72.0 — Layer streaming (BETA).** Fine-tune models that don't fit in your card. The frozen
-base streams from CPU RAM one decoder layer at a time into a small pool of VRAM buffers while the
-LoRA adapters stay resident, so peak VRAM is bounded by one layer rather than the whole model.
-**Qwen2.5-3B trains in 2.15 GB on a 4 GB card, where a resident run OOMs.**
-
-- **`training.stream_layers: true`** — a config key, not a CLI flag. Tune with
-  `stream_source` (`ram` in v0.72.0; `disk` is v0.72.2) and `stream_buffers` (2–8, default 2).
-- **Measured on a 4 GB RTX 3050 Laptop** (batch 1, gradient checkpointing on): 0.5B at
-  978.6 tok/s / 1.47 GB, 1.5B at 525.0 tok/s / 1.82 GB, 3B at 143.1 tok/s / 2.15 GB.
-- **Honest cost: 1.43× slower than resident**, measured at 0.5B — the only apples-to-apples
-  comparison available on that box, because 1.5B and above cannot run resident there at all.
-- **Proof-of-mechanism at 3B.** Nothing above 3B was measured; no 8B/14B claim is supported.
-  Scope: RAM tier, bf16, `task: sft`, Llama/Qwen, batch size 1, no gradient accumulation, no
-  `--resume`. 4-bit (NF4) streaming is **v0.72.1** and is refused with a clear message today.
+**v0.72.2 — NF4 layer streaming: fine-tune Llama-3.1-8B on a 4 GB laptop GPU.** Layer
+streaming keeps the frozen base in CPU RAM and feeds it to the GPU one decoder layer at a time.
+Quantising that base to NF4 shrinks it ~4×, which is what puts an 8B model within reach of a
+card that cannot hold even a quarter of it.
+
+- **Measured on a 4 GB RTX 3050 Laptop** (batch 1, S=512, gradient checkpointing, 50 steps
+  after 10 warm-up): **Llama-3.1-8B-Instruct at 119.6 tok/s, peak VRAM 3.32 GB**, base
+  page-locked at 3.60 GB, GPU 100% busy. Qwen2.5-3B: 264.2 tok/s, 1.76 GB.
+- **Just add `quantization: 4bit`** to a streaming config. The base is quantised once, offline,
+  and cached; the cache re-shards by itself if the checkpoint changes underneath it.
+- **Correctness is not traded away.** A streamed NF4 run is **bit-exact** against a resident
+  NF4 run — same quantised bytes, same bitsandbytes kernels — and that is a CI test, not a
+  one-off measurement.
+- **Why 3B got 1.85× faster too** (264.2 vs 143.1 tok/s in bf16): not arithmetic. A 1.43 GB
+  store page-locks where a 5.55 GB one did not, which restores asynchronous copies and takes
+  GPU utilisation from 79.3% to 100%.
+- Still BETA, and the scope is unchanged: RAM tier, `task: sft`, Llama/Qwen, batch size 1, no
+  gradient accumulation, no `--resume`. Every refusal names the release that lifts it.
 
 ```yaml
 # soup.yaml — then just `soup train --config soup.yaml`
 training:
   stream_layers: true      # base streams from RAM; only the adapter trains
+  quantization: 4bit       # NF4 — ~4x smaller store, so 8B fits a 4 GB card
   batch_size: 1
-  quantization: none       # NF4 streaming lands in v0.72.1
 ```
 
+> **Trained with `stream_layers: true` on v0.72.0?** That adapter is inert — its tensors were
+> saved under keys with an extra `.inner.` segment, so every loader returned the untuned base.
+> Fixed in v0.72.1; re-run or re-save. Check with:
+> `python -c "from safetensors.torch import load_file; print([k for k in load_file('adapter_model.safetensors') if '.inner.' in k][:3])"`
+
 <details>
 <summary>Previous release — v0.71.40, soup reward synth (generate a reward verifier from your data)</summary>
 
@@ -483,6 +492,25 @@ pre-commit install                 # optional: ruff lint+format on commit
 See [CONTRIBUTING.md](CONTRIBUTING.md) for the full workflow and [SECURITY.md](SECURITY.md) to
 report a vulnerability.
 
+## Support Soup
+
+Soup is Apache-2.0 and free — and stays that way. It is built and maintained in the open on a
+single 4 GB laptop, which is why every performance number in these docs is measured rather than
+claimed.
+
+If Soup saved you a training run, [starring the repo](https://github.com/MakazhanAlpamys/Soup)
+helps most, and it costs nothing. If you would like to fund the work directly:
+
+**[❤️ Donate](https://buy.stripe.com/4gMcN441k3pha3T19ye7m04)** — one-off, any amount (use
+*Change amount* on the checkout page). Payments are processed by Stripe under the maintainer's
+registered business, **MePlay, Inc.** — that name, not "Soup", is what appears on the checkout
+page and on your card statement.
+
+Donations fund GPU time for the hardware-gated work — multi-GPU, 8B+ validation, Apple
+Silicon — that a single 4 GB laptop cannot reach. See the
+[`help wanted`](https://github.com/MakazhanAlpamys/Soup/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22)
+issues for exactly what is blocked on hardware today.
+
 ## Contributors
 
 Built by the community ❤️ — thank you to everyone who has contributed. See

{soup_cli-0.72.0 → soup_cli-0.72.2}/README.md

@@ -49,29 +49,38 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.72.0 — Layer streaming (BETA).** Fine-tune models that don't fit in your card. The frozen
-base streams from CPU RAM one decoder layer at a time into a small pool of VRAM buffers while the
-LoRA adapters stay resident, so peak VRAM is bounded by one layer rather than the whole model.
-**Qwen2.5-3B trains in 2.15 GB on a 4 GB card, where a resident run OOMs.**
-
-- **`training.stream_layers: true`** — a config key, not a CLI flag. Tune with
-  `stream_source` (`ram` in v0.72.0; `disk` is v0.72.2) and `stream_buffers` (2–8, default 2).
-- **Measured on a 4 GB RTX 3050 Laptop** (batch 1, gradient checkpointing on): 0.5B at
-  978.6 tok/s / 1.47 GB, 1.5B at 525.0 tok/s / 1.82 GB, 3B at 143.1 tok/s / 2.15 GB.
-- **Honest cost: 1.43× slower than resident**, measured at 0.5B — the only apples-to-apples
-  comparison available on that box, because 1.5B and above cannot run resident there at all.
-- **Proof-of-mechanism at 3B.** Nothing above 3B was measured; no 8B/14B claim is supported.
-  Scope: RAM tier, bf16, `task: sft`, Llama/Qwen, batch size 1, no gradient accumulation, no
-  `--resume`. 4-bit (NF4) streaming is **v0.72.1** and is refused with a clear message today.
+**v0.72.2 — NF4 layer streaming: fine-tune Llama-3.1-8B on a 4 GB laptop GPU.** Layer
+streaming keeps the frozen base in CPU RAM and feeds it to the GPU one decoder layer at a time.
+Quantising that base to NF4 shrinks it ~4×, which is what puts an 8B model within reach of a
+card that cannot hold even a quarter of it.
+
+- **Measured on a 4 GB RTX 3050 Laptop** (batch 1, S=512, gradient checkpointing, 50 steps
+  after 10 warm-up): **Llama-3.1-8B-Instruct at 119.6 tok/s, peak VRAM 3.32 GB**, base
+  page-locked at 3.60 GB, GPU 100% busy. Qwen2.5-3B: 264.2 tok/s, 1.76 GB.
+- **Just add `quantization: 4bit`** to a streaming config. The base is quantised once, offline,
+  and cached; the cache re-shards by itself if the checkpoint changes underneath it.
+- **Correctness is not traded away.** A streamed NF4 run is **bit-exact** against a resident
+  NF4 run — same quantised bytes, same bitsandbytes kernels — and that is a CI test, not a
+  one-off measurement.
+- **Why 3B got 1.85× faster too** (264.2 vs 143.1 tok/s in bf16): not arithmetic. A 1.43 GB
+  store page-locks where a 5.55 GB one did not, which restores asynchronous copies and takes
+  GPU utilisation from 79.3% to 100%.
+- Still BETA, and the scope is unchanged: RAM tier, `task: sft`, Llama/Qwen, batch size 1, no
+  gradient accumulation, no `--resume`. Every refusal names the release that lifts it.
 
 ```yaml
 # soup.yaml — then just `soup train --config soup.yaml`
 training:
   stream_layers: true      # base streams from RAM; only the adapter trains
+  quantization: 4bit       # NF4 — ~4x smaller store, so 8B fits a 4 GB card
   batch_size: 1
-  quantization: none       # NF4 streaming lands in v0.72.1
 ```
 
+> **Trained with `stream_layers: true` on v0.72.0?** That adapter is inert — its tensors were
+> saved under keys with an extra `.inner.` segment, so every loader returned the untuned base.
+> Fixed in v0.72.1; re-run or re-save. Check with:
+> `python -c "from safetensors.torch import load_file; print([k for k in load_file('adapter_model.safetensors') if '.inner.' in k][:3])"`
+
 <details>
 <summary>Previous release — v0.71.40, soup reward synth (generate a reward verifier from your data)</summary>
 
@@ -339,6 +348,25 @@ pre-commit install                 # optional: ruff lint+format on commit
 See [CONTRIBUTING.md](CONTRIBUTING.md) for the full workflow and [SECURITY.md](SECURITY.md) to
 report a vulnerability.
 
+## Support Soup
+
+Soup is Apache-2.0 and free — and stays that way. It is built and maintained in the open on a
+single 4 GB laptop, which is why every performance number in these docs is measured rather than
+claimed.
+
+If Soup saved you a training run, [starring the repo](https://github.com/MakazhanAlpamys/Soup)
+helps most, and it costs nothing. If you would like to fund the work directly:
+
+**[❤️ Donate](https://buy.stripe.com/4gMcN441k3pha3T19ye7m04)** — one-off, any amount (use
+*Change amount* on the checkout page). Payments are processed by Stripe under the maintainer's
+registered business, **MePlay, Inc.** — that name, not "Soup", is what appears on the checkout
+page and on your card statement.
+
+Donations fund GPU time for the hardware-gated work — multi-GPU, 8B+ validation, Apple
+Silicon — that a single 4 GB laptop cannot reach. See the
+[`help wanted`](https://github.com/MakazhanAlpamys/Soup/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22)
+issues for exactly what is blocked on hardware today.
+
 ## Contributors
 
 Built by the community ❤️ — thank you to everyone who has contributed. See

{soup_cli-0.72.0 → soup_cli-0.72.2}/docs/commands.md

@@ -30,7 +30,7 @@ soup merge-sharded-fsdp-weights ./shards -o merged.safetensors  Consolidate FSDP
 soup delinearize-llama4 ./src --target ./out [--num-experts N] [--plan-only]  Live Llama-4 fused-expert reshape [E*din,dout] -> [E,din,dout] + sidecar copy (v0.71.21)
 soup spectrum scan --model <id|path> --top-percent 50 [--modules mlp,attn] [-o patch.yaml]  Spectrum SNR scan (no model load) -> training.unfrozen_parameters YAML patch (v0.71.23)
 soup train --config sft.yaml  # training.lisa_enabled: true [lisa_num_layers lisa_interval_steps]  LISA layerwise importance sampling — full-FT quality at LoRA-like memory (sft/transformers/text/quantization=none) (v0.71.34)
-soup train --config sft.yaml  # training.stream_layers: true [stream_source stream_buffers]  BETA layer streaming — the frozen base streams from CPU RAM one decoder layer at a time, so peak VRAM is bounded by ONE layer (sft/transformers/text/quantization=none, batch 1) (v0.72.0)
+soup train --config sft.yaml  # training.stream_layers: true [stream_source stream_buffers]  BETA layer streaming — the frozen base streams from CPU RAM one decoder layer at a time, so peak VRAM is bounded by ONE layer; quantization: 4bit streams it as NF4, ~4x smaller (sft/transformers/text, batch 1) (v0.72.0; NF4 v0.72.2)
 soup export --model ./output --format gguf    Export to GGUF (Ollama)
 soup export --model ./output --deploy ollama  Export GGUF + auto-deploy to Ollama
 soup export --model ./output --format onnx    Export to ONNX

{soup_cli-0.72.0 → soup_cli-0.72.2}/docs/performance-and-quantization.md

@@ -14,7 +14,7 @@
 - [Cross-Document Attention Masking](#cross-document-attention-masking)
 - [Quant Menu — 9 Quantization Formats](#quant-menu--9-quantization-formats)
 - [Activation Offloading (Small-VRAM Large-Batch)](#activation-offloading-small-vram-large-batch)
-- [Layer Streaming (BETA, v0.72.0)](#layer-streaming-beta-v0720)
+- [Layer Streaming (BETA, v0.72.0; NF4 v0.72.2)](#layer-streaming-beta-v0720-nf4-v0722)
 - [Correctness First (v0.36.0)](#correctness-first-v0360)
 - [Multi-GPU / DeepSpeed / FSDP](#multi-gpu--deepspeed--fsdp)
 - [Performance + Long-Context](#performance--long-context)
@@ -226,14 +226,14 @@ training:
 Not compatible with unsloth (own memory manager) or mlx. Wired across every transformer-backend trainer (SFT, DPO, GRPO, KTO, ORPO, SimPO, IPO, PPO, Reward-Model, Embedding, Pretrain).
 
 
-## Layer Streaming (BETA, v0.72.0)
+## Layer Streaming (BETA, v0.72.0; NF4 v0.72.2)
 
 Stream frozen base-model decoder layers ONE at a time from CPU RAM into small VRAM buffers instead of keeping the whole base resident. Peak VRAM is bounded by the size of a single layer, not the entire model — so models that don't fit resident on your GPU can now train at all.
 
 ```yaml
 training:
   stream_layers: true          # Enable layer streaming
-  stream_source: auto          # 'auto' (same-host RAM), 'ram', 'disk' (v0.72.2)
+  stream_source: auto          # 'auto' (same-host RAM), 'ram', 'disk' (v0.72.3)
   stream_buffers: 2            # Double-buffering; range [2, 8]
 ```
 
@@ -246,33 +246,48 @@ soup train --config soup.yaml
 
 The tradeoff: **1.43× slower than resident training**, measured at 0.5B — the only apples-to-apples comparison available on the reference box, because 1.5B and above cannot run resident there at all.
 
-**Measured numbers (RTX 3050 Laptop 4 GB, Windows 11, bf16 LoRA, batch 1, 50 steps after 10 warmup):**
+### NF4 streaming (`quantization: 4bit`)
 
-| Model | Seq Length | Throughput | GPU Util | Peak VRAM |
-|---|---|---|---|---|
-| Qwen2.5-0.5B | 512 | 978.6 tok/s | 91.4% | 1.47 GB |
-| Qwen2.5-1.5B | 512 | 525.0 tok/s | 96.8% | 1.82 GB |
-| Qwen2.5-1.5B | 1024 | 487.6 tok/s | 96.7% | 2.96 GB |
-| Qwen2.5-3B | 512 | 143.1 tok/s | 79.3% | 2.15 GB |
+Quantising the streamed base to NF4 makes the RAM store ~4× smaller. That matters for two reasons, and the second is the bigger one:
 
-**Headline:** Qwen2.5-3B trains in **2.15 GB** on a 4 GB card, where a resident run OOMs.
+1. A bigger model fits in host RAM at all — an 8B base is ~3.6 GB of NF4 instead of ~16 GB of bf16.
+2. **The store fits under the machine's page-locked memory ceiling.** Pinned host memory is what lets `copy_(non_blocking=True)` actually overlap with compute. The reference box tops out at ~7.1 GB of page-locked memory, so a 5.55 GB bf16 3B base fell back to pageable and lost overlap; the 1.43 GB NF4 store pins, and utilisation goes from 79.3% to 100%.
 
-**Honest scope (v0.72.0 = proof-of-mechanism):**
-- **Models measured:** Qwen2.5-0.5B, 1.5B and 3B, plus a live `soup train` on SmolLM2-135M. **Nothing above 3B was measured, and no 8B / 14B / 70B claim is supported.**
-- **RAM tier only.** `stream_source: ram` (`auto` resolves to it). The disk overflow tier ships in **v0.72.2**.
+The base is quantised **once, offline**, one tensor at a time, and cached. The shard cache is keyed to the quantisation, the dtype, the quantisation device and a fingerprint of the source checkpoint, so switching `none` ⇄ `4bit` — or retraining a base in place — re-shards rather than silently streaming the wrong bytes.
+
+Correctness is not a tradeoff here either: a streamed NF4 run is **bit-exact** against a *resident* NF4 run (the same quantised bytes through the same bitsandbytes kernels), and that is a regression test, not a one-off measurement.
+
+**Measured numbers (RTX 3050 Laptop 4 GB, Windows 11, LoRA, batch 1, 50 steps after 10 warmup):**
+
+| Model | Quant | Seq | Throughput | GPU Util | Peak VRAM | RAM store |
+|---|---|---|---|---|---|---|
+| **Llama-3.1-8B-Instruct** | **NF4** | 512 | **119.6 tok/s** | 100% | **3.32 GB** | 3.60 GB pinned |
+| Qwen2.5-3B | NF4 | 512 | 264.2 tok/s | 100% | 1.76 GB | 1.43 GB pinned |
+| Qwen2.5-3B | bf16 | 512 | 143.1 tok/s | 79.3% | 2.15 GB | 5.55 GB pageable |
+| Qwen2.5-1.5B | bf16 | 512 | 525.0 tok/s | 96.8% | 1.82 GB | pinned |
+| Qwen2.5-1.5B | bf16 | 1024 | 487.6 tok/s | 96.7% | 2.96 GB | pinned |
+| Qwen2.5-0.5B | bf16 | 512 | 978.6 tok/s | 91.4% | 1.47 GB | pinned |
+
+**Headline:** **Llama-3.1-8B fine-tunes on a 4 GB card at 119.6 tok/s in 3.32 GB.** For scale, 1M training tokens is ~2.3 h at 8B (arithmetic from the measured rate, not a separate measurement).
+
+The 3B NF4-vs-bf16 rows differ by 1.85×, but attribute that to **pinning, not arithmetic** — see point 2 above. The two rows also come from different sessions, and this card's boost clock varies ~13% between sessions, so treat the factor as indicative and the mechanism as the claim.
+
+Untied `embed_tokens` + `lm_head` stay resident and unquantised (2.10 GB of the 8B row's 3.32 GB), which is why 8B sits close to this card's ceiling; treating them as streamed large layers is a v0.72.3 item.
+
+**Honest scope:**
+- **RAM tier only.** `stream_source: ram` (`auto` resolves to it). The disk overflow tier ships in **v0.72.3**.
 - **Llama / Qwen only**, `task: sft`, `backend: transformers`, `modality: text`.
-- **`batch_size: 1`**, no gradient accumulation, no `--resume` — all ship in **v0.72.2**.
-- **bf16 base; `quantization: none` is required.** 4-bit (NF4) weights carry a quantisation state and cannot be byte-copied into a plain buffer, so NF4 streaming is **v0.72.1**. A `4bit` config is refused today with a message saying exactly that.
-- **The 3B throughput is a LOWER BOUND.** The reference box could not page-lock the 5.55 GB base (its measured page-locked ceiling is 7.65 GB, and a CUDA context plus the model skeleton did not leave room), so that run fell back to a pageable store. Pageable memory makes the host-to-device copy synchronous, which costs overlap — visible as the GPU-utilisation drop from 96.8% (1.5B, pinned) to 79.3% (3B, pageable). Soup does this fallback automatically **and prints the cost** rather than absorbing it silently.
+- **`batch_size: 1`**, no gradient accumulation, no `--resume` — all ship in **v0.72.3**.
+- **The bf16 3B throughput above is a LOWER BOUND.** The reference box could not page-lock the 5.55 GB base (its measured page-locked ceiling is 7.65 GB, and a CUDA context plus the model skeleton did not leave room), so that run fell back to a pageable store. Pageable memory makes the host-to-device copy synchronous, which costs overlap — visible as the GPU-utilisation drop from 96.8% (1.5B, pinned) to 79.3% (3B, pageable). Soup does this fallback automatically **and prints the cost** rather than absorbing it silently. NF4 lifts this at 3B: the store drops under the ceiling and pins.
 - Numbers are Windows/WDDM and therefore systematically pessimistic versus Linux. `expandable_segments:True` is silently ignored on Windows; Soup detects that and does not claim it is active.
 
 **Rejected at config load (each names the release that lifts it):**
-- `stream_source: disk` → the disk tier is v0.72.2
-- `quantization` other than `none` → NF4 streaming is v0.72.1
+- `stream_source: disk` → the disk tier is v0.72.3
+- `quantization` other than `none` or `4bit` → other formats cannot be streamed into a pooled buffer
 - `backend: unsloth` / `backend: mlx` → streaming replaces the model-load path those backends own
-- `task` other than `sft` → preference losses are v0.72.3
-- `gradient_accumulation_steps > 1` → every micro-batch re-reads the whole base, so accumulation multiplies streaming IO linearly (v0.72.2)
-- `batch_size` other than `1` → v0.72.2
+- `task` other than `sft` → preference losses are v0.72.4
+- `gradient_accumulation_steps > 1` → every micro-batch re-reads the whole base, so accumulation multiplies streaming IO linearly (v0.72.3)
+- `batch_size` other than `1` → v0.72.3
 - `lora.use_dora` / `lora.use_vera` / `lora.init_strategy` other than `random` → these initialise from the real base weight, which is on the meta device under streaming
 - `unfrozen_parameters`, `lisa_enabled`, `packing`, `multipack`, `use_fsdp2_compile`, `train_router_only`, `expand_layers` → each independently rewrites or re-freezes the same layers
 - `stream_source` / `stream_buffers` set while `stream_layers: false` → a footgun, refused
@@ -294,12 +309,12 @@ data:
 training:
   epochs: 3
   lr: 2e-5
-  batch_size: 1           # required; larger batches in v0.72.2
-  gradient_accumulation_steps: 1   # required; accumulation in v0.72.2
-  quantization: none      # required; NF4 streaming is v0.72.1
+  batch_size: 1           # required; larger batches in v0.72.3
+  gradient_accumulation_steps: 1   # required; accumulation in v0.72.3
+  quantization: 4bit      # NF4 — ~4x smaller RAM store than bf16 (or `none`)
   gradient_checkpointing: true     # handled per-layer by the streamer
   stream_layers: true     # Enable layer streaming
-  stream_source: auto     # RAM-based (disk in v0.72.2)
+  stream_source: auto     # RAM-based (disk in v0.72.3)
   stream_buffers: 2       # double-buffering
   lora:
     r: 64
@@ -313,16 +328,24 @@ output: ./output
 - The 1.5B runs sit at ~97% GPU utilisation, i.e. compute-bound: with a page-locked store the layer loads hide almost completely behind compute. The 3B run's 79.3% is **not** a model-size effect — it is the cost of the pageable-store fallback on that particular box.
 - Correctness is not a tradeoff: streamed and resident forward passes were verified **bit-exact**, and a 100-step streamed loss curve matched resident exactly. Streaming substitutes the same weight bytes into the same kernels.
 
+> **v0.72.0 adapters are unloadable — re-run them on v0.72.1.** In v0.72.0 a streamed run saved every adapter tensor under a key carrying an extra `.inner.` segment, so `soup merge`, `soup serve`, `soup chat` and `PeftModel.from_pretrained` loaded **zero** tensors and silently returned the untuned base (PEFT emitted only a `UserWarning`). The training itself was correct — only the saved file was affected. Check with:
+>
+> ```bash
+> python -c "from safetensors.torch import load_file; \
+> print([k for k in load_file('adapter_model.safetensors') if '.inner.' in k][:3])"
+> ```
+>
+> If that prints anything, the adapter is affected. From v0.72.1 a streamed adapter is byte-for-byte in the same layout as an ordinary LoRA run.
+
 **Troubleshooting:**
-- **"layer streaming needs the base to fit in RAM"** — the base is larger than free RAM. Free RAM or pick a smaller base; the disk overflow tier is v0.72.2.
+- **"layer streaming needs the base to fit in RAM"** — the base is larger than free RAM. Free RAM or pick a smaller base; the disk overflow tier is v0.72.3.
 - **"could not page-lock the base … falling back to a PAGEABLE RAM store"** — expected on a busy machine. Training continues, more slowly. Close other applications to keep the pinned store.
-- **"layer streaming does not support model_type=…"** — v0.72.0 covers Llama and Qwen only.
+- **"layer streaming does not support model_type=…"** — Llama and Qwen only for now; more architectures land in v0.72.3.
 - **Slower than you expected** — layer streaming trades time for memory. If the model already fits resident on your card, do not enable it.
 
 **Roadmap (each refusal names its release):**
-- 4-bit (NF4) streaming — **v0.72.1**, the first genuinely useful capability jump
-- Disk overflow tier, batch size > 1, gradient accumulation, checkpoint/resume, more architectures (Mistral / Gemma / Phi) — **v0.72.2**
-- Preference losses (DPO / ORPO / SimPO / KTO) — **v0.72.3**. GRPO and PPO are explicitly **not** planned: rollouts need generation, which re-reads the model per token
+- Disk overflow tier, batch size > 1, gradient accumulation, checkpoint/resume, more architectures (Mistral / Gemma / Phi) — **v0.72.3**
+- Preference losses (DPO / ORPO / SimPO / KTO) — **v0.72.4**. GRPO and PPO are explicitly **not** planned: rollouts need generation, which re-reads the model per token
 - A published 14B-on-8 GB reference benchmark — hardware-blocked; it needs an 8 GB card and 32 GB of RAM, which the development box does not have
 
 **Shard cache.** The first streaming run rewrites the checkpoint into one safetensors shard per decoder layer under `~/.soup/layer-stream/` (override with `SOUP_LAYER_STREAM_CACHE_DIR`). That costs disk space roughly equal to the base. The cache is keyed to a fingerprint of the source checkpoint, so a base retrained in place re-shards instead of silently training against stale weights.

{soup_cli-0.72.0 → soup_cli-0.72.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.72.0"
+version = "0.72.2"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.72.0 → soup_cli-0.72.2}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
 
-__version__ = "0.72.0"
+__version__ = "0.72.2"

{soup_cli-0.72.0 → soup_cli-0.72.2}/src/soup_cli/commands/train.py

@@ -531,13 +531,20 @@ def train(
         raise typer.Exit(code=2) from exc
 
     # --- v0.72.0 BETA — layer streaming does not support resume yet ---
-    # Silently ignoring --resume would restart from scratch and look like it
+    # Silently ignoring a resume would restart from scratch and look like it
     # worked, so refuse before any heavy work happens.
-    if resume and cfg.training.stream_layers:
+    #
+    # v0.72.1: --hf-resume must be covered too. It sets resume_from through a
+    # different branch (see prepare_hf_resume below), so a guard on --resume
+    # alone let it through. That mattered more after the adapter-key fix: the
+    # pushed checkpoint now carries canonical keys while a live streamed model
+    # is still `.inner.`-shaped, so PEFT's strict=False load would match
+    # NOTHING and silently continue with a freshly initialised adapter.
+    if (resume or hf_resume) and cfg.training.stream_layers:
         console.print(
-            "[red]--resume is not supported with training.stream_layers in "
-            "v0.72.0[/] — checkpoint/resume for layer streaming lands in "
-            "v0.72.2. Drop --resume, or set stream_layers: false."
+            "[red]--resume / --hf-resume are not supported with "
+            "training.stream_layers[/] — checkpoint/resume for layer streaming "
+            "lands in v0.72.3. Drop the flag, or set stream_layers: false."
         )
         raise typer.Exit(code=2)
 

{soup_cli-0.72.0 → soup_cli-0.72.2}/src/soup_cli/config/schema.py

@@ -2930,7 +2930,7 @@ class TrainingConfig(BaseModel):
         default="auto",
         description=(
             "Where the streamed base lives. 'ram' (the only tier implemented in "
-            "v0.72.0) pins the base in CPU RAM; 'disk' is the v0.72.2 overflow "
+            "v0.72.0) pins the base in CPU RAM; 'disk' is the v0.72.3 overflow "
             "tier; 'auto' picks per free RAM."
         ),
     )
@@ -4660,9 +4660,9 @@ class SoupConfig(BaseModel):
             return self
         if self.task != "sft":
             raise ValueError(
-                f"training.stream_layers requires task='sft' in v0.72.0; got "
+                f"training.stream_layers requires task='sft'; got "
                 f"task={self.task!r}. Preference losses (DPO/ORPO/SimPO/KTO) "
-                f"land in v0.72.3."
+                f"land in v0.72.4."
             )
         if self.backend != "transformers":
             raise ValueError(
@@ -4675,29 +4675,29 @@ class SoupConfig(BaseModel):
                 f"training.stream_layers requires modality='text'; got "
                 f"modality={self.modality!r}."
             )
-        if tcfg.quantization != "none":
+        if tcfg.quantization not in ("none", "4bit"):
             raise ValueError(
-                f"training.stream_layers requires quantization='none' in "
-                f"v0.72.0 (got {tcfg.quantization!r}). NF4 weights carry a "
-                f"quant_state and cannot be byte-copied into a plain buffer; "
-                f"NF4 streaming lands in v0.72.1."
+                f"training.stream_layers supports quantization='none' or "
+                f"'4bit' (NF4); got {tcfg.quantization!r}. Other quantisations "
+                f"store weights in formats that cannot be streamed into a "
+                f"pooled buffer."
             )
         if tcfg.stream_source == "disk":
             raise ValueError(
-                "training.stream_source='disk' is not implemented in v0.72.0 — "
-                "the disk overflow tier lands in v0.72.2. Use 'ram' or 'auto'."
+                "training.stream_source='disk' is not implemented yet — "
+                "the disk overflow tier lands in v0.72.3. Use 'ram' or 'auto'."
             )
         if tcfg.batch_size != 1:
             raise ValueError(
-                f"training.stream_layers requires batch_size=1 in v0.72.0 (got "
-                f"{tcfg.batch_size!r}); larger batches land in v0.72.2."
+                f"training.stream_layers requires batch_size=1 (got "
+                f"{tcfg.batch_size!r}); larger batches land in v0.72.3."
             )
         if tcfg.gradient_accumulation_steps != 1:
             raise ValueError(
                 f"training.stream_layers requires gradient_accumulation_steps=1 "
                 f"(got {tcfg.gradient_accumulation_steps}); every micro-batch "
                 f"re-reads the entire base, so accumulation multiplies streaming "
-                f"IO linearly. Accumulation lands in v0.72.2."
+                f"IO linearly. Accumulation lands in v0.72.3."
             )
         if tcfg.lora.r < 1:
             raise ValueError(