soup-cli 0.72.1__tar.gz → 0.72.2__tar.gz

soup_cli-0.72.2/.github/FUNDING.yml

@@ -0,0 +1,7 @@
+# Sponsor button shown in the repo sidebar and on every issue / PR.
+#
+# `github:` only renders once the account is enrolled in GitHub Sponsors
+# (https://github.com/sponsors) — it is not, so it stays commented out.
+# github: MakazhanAlpamys
+
+custom: ["https://buy.stripe.com/4gMcN441k3pha3T19ye7m04"]

{soup_cli-0.72.1 → soup_cli-0.72.2}/CHANGELOG.md

@@ -12,6 +12,79 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.72.2] - 2026-07-28
+
+**Added — NF4 layer streaming: fine-tune Llama-3.1-8B on a 4 GB laptop GPU.**
+
+Layer streaming (v0.72.0) keeps the frozen base in CPU RAM and streams it to the
+GPU one decoder layer at a time, so peak VRAM is bounded by one layer instead of
+the whole model. It was bf16-only, which capped it at about 3B on a small card.
+Quantising the streamed base to NF4 shrinks it ~4×, and that is what brings 8B
+within reach.
+
+Add one line to a streaming config:
+
+```yaml
+training:
+  stream_layers: true
+  quantization: 4bit    # NF4
+  batch_size: 1
+```
+
+**Measured on a 4 GB RTX 3050 Laptop** (Windows, batch 1, S=512, gradient
+checkpointing, 50 steps after 10 warm-up, `PagedAdamW8bit`):
+
+| Model | tok/s | Peak VRAM | RAM store | GPU util |
+|---|---|---|---|---|
+| Llama-3.1-8B-Instruct | 119.6 | 3.32 GB | 3.60 GB (page-locked) | 100% |
+| Qwen2.5-3B | 264.2 | 1.76 GB | 1.43 GB (page-locked) | 100% |
+
+For scale: 1M training tokens is about 2.3 h at 8B on that card (arithmetic from
+the measured rate, not a separate measurement).
+
+Qwen2.5-3B also got **1.85× faster** than the bf16 streaming path (264.2 vs
+143.1 tok/s) — and the reason is not arithmetic. A 1.43 GB store fits under the
+machine's page-locked memory ceiling where a 5.55 GB one did not, which restores
+asynchronous host-to-device copies and lifts GPU utilisation from 79.3% to 100%.
+
+**Correctness.** A streamed NF4 run is **bit-exact** against a resident NF4 run:
+the same quantised bytes through the same bitsandbytes kernels. Logit equality,
+non-zero gradients at layer 0, and a matching multi-step loss curve are all
+regression tests, not one-off measurements.
+
+The base is quantised once, offline, and cached under `~/.soup/layer-stream/`.
+The cache is keyed to the quantisation *and* the source checkpoint, so switching
+between `none` and `4bit`, or retraining a base in place, re-shards instead of
+silently streaming the wrong bytes.
+
+**Fixed — a streamed 4-bit run reported its parameter count ~6.5× too high.**
+SmolLM2-135M printed "878,154,048 total" (true: 134,515,008). Display only —
+training was unaffected — but at 8B it would have read ~52 B.
+
+Scope is unchanged and still BETA: RAM tier, `task: sft`, Llama/Qwen,
+`batch_size: 1`, no gradient accumulation, no `--resume`. Every rejected config
+names the release that lifts it. `quantization` values other than `none` and
+`4bit` are refused.
+
+**Fixed — `soup --help` was 5x slower than it should be.** Since v0.72.0 the CLI
+imported PyTorch on startup, taking **6.0 s** where it now takes **1.15 s**.
+
+`soup reward stress` (v0.71.41) put `utils/reward_stress` on the light CLI path.
+That module imported `utils/reward_hack_control` to reuse a single string
+constant — and `reward_hack_control` resolves its `TrainerCallback` base class at
+module scope, which pulls in `transformers` and `torch`. Importing a ~4.4 s
+dependency for one constant made every `soup` invocation pay for the training
+stack, including commands that never touch a model.
+
+Nothing produced wrong results; this was purely startup latency. The light core
+(`pip install soup-cli` without the `[train]` extra) was never broken — it fell
+back cleanly when torch was absent, just slowly when it was present.
+
+Added `tests/test_cli_startup_is_light.py`, which asserts the invariant at
+runtime (`import soup_cli.cli` must not put `torch` in `sys.modules`) instead of
+inspecting source text for `import torch`, which is what the previous guards did
+and why this went unnoticed.
+
 ## [0.72.1] - 2026-07-27
 
 **Fixed — layer-streaming adapters were saved in an unloadable form.** If you

{soup_cli-0.72.1 → soup_cli-0.72.2}/CONTRIBUTING.md

@@ -120,7 +120,7 @@ src/soup_cli/
   templates/          - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (325 files, 16752 tests)
+tests/                - Test suite (326 files, 16840 tests)
 examples/             - Real-world config examples and datasets
 ```
 

{soup_cli-0.72.1 → soup_cli-0.72.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.72.1
+Version: 0.72.2
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -193,41 +193,38 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.72.1 — Fix: layer-streaming adapters were saved unloadable.** If you trained with
-`stream_layers: true` on v0.72.0, that adapter is **inert** — every tensor was written under a
-key with an extra `.inner.` segment, so `soup merge`, `soup serve`, `soup chat` and
-`PeftModel.from_pretrained` loaded **zero** tensors and silently returned the untuned base.
-Only a `UserWarning` was emitted.
-
-- **The training was correct; only the saved file was affected.** v0.72.0's bit-exactness
-  results still stand.
-- **Check an existing adapter:** if `adapter_model.safetensors` has keys containing `.inner.`,
-  it is affected — re-run or re-save it on v0.72.1.
-- Streamed adapters now save in exactly the same layout as an ordinary LoRA run.
-
-**Layer streaming (BETA), from v0.72.0.** Fine-tune models that don't fit in your card. The
-frozen base streams from CPU RAM one decoder layer at a time into a small pool of VRAM buffers
-while the LoRA adapters stay resident, so peak VRAM is bounded by one layer rather than the
-whole model. **Qwen2.5-3B trains in 2.15 GB on a 4 GB card, where a resident run OOMs.**
-
-- **`training.stream_layers: true`** — a config key, not a CLI flag. Tune with
-  `stream_source` (`ram` today; `disk` is v0.72.3) and `stream_buffers` (2–8, default 2).
-- **Measured on a 4 GB RTX 3050 Laptop** (batch 1, gradient checkpointing on): 0.5B at
-  978.6 tok/s / 1.47 GB, 1.5B at 525.0 tok/s / 1.82 GB, 3B at 143.1 tok/s / 2.15 GB.
-- **Honest cost: 1.43× slower than resident**, measured at 0.5B — the only apples-to-apples
-  comparison available on that box, because 1.5B and above cannot run resident there at all.
-- **Proof-of-mechanism at 3B.** Nothing above 3B was measured; no 8B/14B claim is supported.
-  Scope: RAM tier, bf16, `task: sft`, Llama/Qwen, batch size 1, no gradient accumulation, no
-  `--resume`. 4-bit (NF4) streaming is **v0.72.2** and is refused with a clear message today.
+**v0.72.2 — NF4 layer streaming: fine-tune Llama-3.1-8B on a 4 GB laptop GPU.** Layer
+streaming keeps the frozen base in CPU RAM and feeds it to the GPU one decoder layer at a time.
+Quantising that base to NF4 shrinks it ~4×, which is what puts an 8B model within reach of a
+card that cannot hold even a quarter of it.
+
+- **Measured on a 4 GB RTX 3050 Laptop** (batch 1, S=512, gradient checkpointing, 50 steps
+  after 10 warm-up): **Llama-3.1-8B-Instruct at 119.6 tok/s, peak VRAM 3.32 GB**, base
+  page-locked at 3.60 GB, GPU 100% busy. Qwen2.5-3B: 264.2 tok/s, 1.76 GB.
+- **Just add `quantization: 4bit`** to a streaming config. The base is quantised once, offline,
+  and cached; the cache re-shards by itself if the checkpoint changes underneath it.
+- **Correctness is not traded away.** A streamed NF4 run is **bit-exact** against a resident
+  NF4 run — same quantised bytes, same bitsandbytes kernels — and that is a CI test, not a
+  one-off measurement.
+- **Why 3B got 1.85× faster too** (264.2 vs 143.1 tok/s in bf16): not arithmetic. A 1.43 GB
+  store page-locks where a 5.55 GB one did not, which restores asynchronous copies and takes
+  GPU utilisation from 79.3% to 100%.
+- Still BETA, and the scope is unchanged: RAM tier, `task: sft`, Llama/Qwen, batch size 1, no
+  gradient accumulation, no `--resume`. Every refusal names the release that lifts it.
 
 ```yaml
 # soup.yaml — then just `soup train --config soup.yaml`
 training:
   stream_layers: true      # base streams from RAM; only the adapter trains
+  quantization: 4bit       # NF4 — ~4x smaller store, so 8B fits a 4 GB card
   batch_size: 1
-  quantization: none       # NF4 streaming lands in v0.72.2
 ```
 
+> **Trained with `stream_layers: true` on v0.72.0?** That adapter is inert — its tensors were
+> saved under keys with an extra `.inner.` segment, so every loader returned the untuned base.
+> Fixed in v0.72.1; re-run or re-save. Check with:
+> `python -c "from safetensors.torch import load_file; print([k for k in load_file('adapter_model.safetensors') if '.inner.' in k][:3])"`
+
 <details>
 <summary>Previous release — v0.71.40, soup reward synth (generate a reward verifier from your data)</summary>
 
@@ -495,6 +492,25 @@ pre-commit install                 # optional: ruff lint+format on commit
 See [CONTRIBUTING.md](CONTRIBUTING.md) for the full workflow and [SECURITY.md](SECURITY.md) to
 report a vulnerability.
 
+## Support Soup
+
+Soup is Apache-2.0 and free — and stays that way. It is built and maintained in the open on a
+single 4 GB laptop, which is why every performance number in these docs is measured rather than
+claimed.
+
+If Soup saved you a training run, [starring the repo](https://github.com/MakazhanAlpamys/Soup)
+helps most, and it costs nothing. If you would like to fund the work directly:
+
+**[❤️ Donate](https://buy.stripe.com/4gMcN441k3pha3T19ye7m04)** — one-off, any amount (use
+*Change amount* on the checkout page). Payments are processed by Stripe under the maintainer's
+registered business, **MePlay, Inc.** — that name, not "Soup", is what appears on the checkout
+page and on your card statement.
+
+Donations fund GPU time for the hardware-gated work — multi-GPU, 8B+ validation, Apple
+Silicon — that a single 4 GB laptop cannot reach. See the
+[`help wanted`](https://github.com/MakazhanAlpamys/Soup/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22)
+issues for exactly what is blocked on hardware today.
+
 ## Contributors
 
 Built by the community ❤️ — thank you to everyone who has contributed. See

{soup_cli-0.72.1 → soup_cli-0.72.2}/README.md

@@ -49,41 +49,38 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.72.1 — Fix: layer-streaming adapters were saved unloadable.** If you trained with
-`stream_layers: true` on v0.72.0, that adapter is **inert** — every tensor was written under a
-key with an extra `.inner.` segment, so `soup merge`, `soup serve`, `soup chat` and
-`PeftModel.from_pretrained` loaded **zero** tensors and silently returned the untuned base.
-Only a `UserWarning` was emitted.
-
-- **The training was correct; only the saved file was affected.** v0.72.0's bit-exactness
-  results still stand.
-- **Check an existing adapter:** if `adapter_model.safetensors` has keys containing `.inner.`,
-  it is affected — re-run or re-save it on v0.72.1.
-- Streamed adapters now save in exactly the same layout as an ordinary LoRA run.
-
-**Layer streaming (BETA), from v0.72.0.** Fine-tune models that don't fit in your card. The
-frozen base streams from CPU RAM one decoder layer at a time into a small pool of VRAM buffers
-while the LoRA adapters stay resident, so peak VRAM is bounded by one layer rather than the
-whole model. **Qwen2.5-3B trains in 2.15 GB on a 4 GB card, where a resident run OOMs.**
-
-- **`training.stream_layers: true`** — a config key, not a CLI flag. Tune with
-  `stream_source` (`ram` today; `disk` is v0.72.3) and `stream_buffers` (2–8, default 2).
-- **Measured on a 4 GB RTX 3050 Laptop** (batch 1, gradient checkpointing on): 0.5B at
-  978.6 tok/s / 1.47 GB, 1.5B at 525.0 tok/s / 1.82 GB, 3B at 143.1 tok/s / 2.15 GB.
-- **Honest cost: 1.43× slower than resident**, measured at 0.5B — the only apples-to-apples
-  comparison available on that box, because 1.5B and above cannot run resident there at all.
-- **Proof-of-mechanism at 3B.** Nothing above 3B was measured; no 8B/14B claim is supported.
-  Scope: RAM tier, bf16, `task: sft`, Llama/Qwen, batch size 1, no gradient accumulation, no
-  `--resume`. 4-bit (NF4) streaming is **v0.72.2** and is refused with a clear message today.
+**v0.72.2 — NF4 layer streaming: fine-tune Llama-3.1-8B on a 4 GB laptop GPU.** Layer
+streaming keeps the frozen base in CPU RAM and feeds it to the GPU one decoder layer at a time.
+Quantising that base to NF4 shrinks it ~4×, which is what puts an 8B model within reach of a
+card that cannot hold even a quarter of it.
+
+- **Measured on a 4 GB RTX 3050 Laptop** (batch 1, S=512, gradient checkpointing, 50 steps
+  after 10 warm-up): **Llama-3.1-8B-Instruct at 119.6 tok/s, peak VRAM 3.32 GB**, base
+  page-locked at 3.60 GB, GPU 100% busy. Qwen2.5-3B: 264.2 tok/s, 1.76 GB.
+- **Just add `quantization: 4bit`** to a streaming config. The base is quantised once, offline,
+  and cached; the cache re-shards by itself if the checkpoint changes underneath it.
+- **Correctness is not traded away.** A streamed NF4 run is **bit-exact** against a resident
+  NF4 run — same quantised bytes, same bitsandbytes kernels — and that is a CI test, not a
+  one-off measurement.
+- **Why 3B got 1.85× faster too** (264.2 vs 143.1 tok/s in bf16): not arithmetic. A 1.43 GB
+  store page-locks where a 5.55 GB one did not, which restores asynchronous copies and takes
+  GPU utilisation from 79.3% to 100%.
+- Still BETA, and the scope is unchanged: RAM tier, `task: sft`, Llama/Qwen, batch size 1, no
+  gradient accumulation, no `--resume`. Every refusal names the release that lifts it.
 
 ```yaml
 # soup.yaml — then just `soup train --config soup.yaml`
 training:
   stream_layers: true      # base streams from RAM; only the adapter trains
+  quantization: 4bit       # NF4 — ~4x smaller store, so 8B fits a 4 GB card
   batch_size: 1
-  quantization: none       # NF4 streaming lands in v0.72.2
 ```
 
+> **Trained with `stream_layers: true` on v0.72.0?** That adapter is inert — its tensors were
+> saved under keys with an extra `.inner.` segment, so every loader returned the untuned base.
+> Fixed in v0.72.1; re-run or re-save. Check with:
+> `python -c "from safetensors.torch import load_file; print([k for k in load_file('adapter_model.safetensors') if '.inner.' in k][:3])"`
+
 <details>
 <summary>Previous release — v0.71.40, soup reward synth (generate a reward verifier from your data)</summary>
 
@@ -351,6 +348,25 @@ pre-commit install                 # optional: ruff lint+format on commit
 See [CONTRIBUTING.md](CONTRIBUTING.md) for the full workflow and [SECURITY.md](SECURITY.md) to
 report a vulnerability.
 
+## Support Soup
+
+Soup is Apache-2.0 and free — and stays that way. It is built and maintained in the open on a
+single 4 GB laptop, which is why every performance number in these docs is measured rather than
+claimed.
+
+If Soup saved you a training run, [starring the repo](https://github.com/MakazhanAlpamys/Soup)
+helps most, and it costs nothing. If you would like to fund the work directly:
+
+**[❤️ Donate](https://buy.stripe.com/4gMcN441k3pha3T19ye7m04)** — one-off, any amount (use
+*Change amount* on the checkout page). Payments are processed by Stripe under the maintainer's
+registered business, **MePlay, Inc.** — that name, not "Soup", is what appears on the checkout
+page and on your card statement.
+
+Donations fund GPU time for the hardware-gated work — multi-GPU, 8B+ validation, Apple
+Silicon — that a single 4 GB laptop cannot reach. See the
+[`help wanted`](https://github.com/MakazhanAlpamys/Soup/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22)
+issues for exactly what is blocked on hardware today.
+
 ## Contributors
 
 Built by the community ❤️ — thank you to everyone who has contributed. See

{soup_cli-0.72.1 → soup_cli-0.72.2}/docs/commands.md

@@ -30,7 +30,7 @@ soup merge-sharded-fsdp-weights ./shards -o merged.safetensors  Consolidate FSDP
 soup delinearize-llama4 ./src --target ./out [--num-experts N] [--plan-only]  Live Llama-4 fused-expert reshape [E*din,dout] -> [E,din,dout] + sidecar copy (v0.71.21)
 soup spectrum scan --model <id|path> --top-percent 50 [--modules mlp,attn] [-o patch.yaml]  Spectrum SNR scan (no model load) -> training.unfrozen_parameters YAML patch (v0.71.23)
 soup train --config sft.yaml  # training.lisa_enabled: true [lisa_num_layers lisa_interval_steps]  LISA layerwise importance sampling — full-FT quality at LoRA-like memory (sft/transformers/text/quantization=none) (v0.71.34)
-soup train --config sft.yaml  # training.stream_layers: true [stream_source stream_buffers]  BETA layer streaming — the frozen base streams from CPU RAM one decoder layer at a time, so peak VRAM is bounded by ONE layer (sft/transformers/text/quantization=none, batch 1) (v0.72.0)
+soup train --config sft.yaml  # training.stream_layers: true [stream_source stream_buffers]  BETA layer streaming — the frozen base streams from CPU RAM one decoder layer at a time, so peak VRAM is bounded by ONE layer; quantization: 4bit streams it as NF4, ~4x smaller (sft/transformers/text, batch 1) (v0.72.0; NF4 v0.72.2)
 soup export --model ./output --format gguf    Export to GGUF (Ollama)
 soup export --model ./output --deploy ollama  Export GGUF + auto-deploy to Ollama
 soup export --model ./output --format onnx    Export to ONNX

{soup_cli-0.72.1 → soup_cli-0.72.2}/docs/performance-and-quantization.md

@@ -14,7 +14,7 @@
 - [Cross-Document Attention Masking](#cross-document-attention-masking)
 - [Quant Menu — 9 Quantization Formats](#quant-menu--9-quantization-formats)
 - [Activation Offloading (Small-VRAM Large-Batch)](#activation-offloading-small-vram-large-batch)
-- [Layer Streaming (BETA, v0.72.0)](#layer-streaming-beta-v0720)
+- [Layer Streaming (BETA, v0.72.0; NF4 v0.72.2)](#layer-streaming-beta-v0720-nf4-v0722)
 - [Correctness First (v0.36.0)](#correctness-first-v0360)
 - [Multi-GPU / DeepSpeed / FSDP](#multi-gpu--deepspeed--fsdp)
 - [Performance + Long-Context](#performance--long-context)
@@ -226,7 +226,7 @@ training:
 Not compatible with unsloth (own memory manager) or mlx. Wired across every transformer-backend trainer (SFT, DPO, GRPO, KTO, ORPO, SimPO, IPO, PPO, Reward-Model, Embedding, Pretrain).
 
 
-## Layer Streaming (BETA, v0.72.0)
+## Layer Streaming (BETA, v0.72.0; NF4 v0.72.2)
 
 Stream frozen base-model decoder layers ONE at a time from CPU RAM into small VRAM buffers instead of keeping the whole base resident. Peak VRAM is bounded by the size of a single layer, not the entire model — so models that don't fit resident on your GPU can now train at all.
 
@@ -246,29 +246,44 @@ soup train --config soup.yaml
 
 The tradeoff: **1.43× slower than resident training**, measured at 0.5B — the only apples-to-apples comparison available on the reference box, because 1.5B and above cannot run resident there at all.
 
-**Measured numbers (RTX 3050 Laptop 4 GB, Windows 11, bf16 LoRA, batch 1, 50 steps after 10 warmup):**
+### NF4 streaming (`quantization: 4bit`)
 
-| Model | Seq Length | Throughput | GPU Util | Peak VRAM |
-|---|---|---|---|---|
-| Qwen2.5-0.5B | 512 | 978.6 tok/s | 91.4% | 1.47 GB |
-| Qwen2.5-1.5B | 512 | 525.0 tok/s | 96.8% | 1.82 GB |
-| Qwen2.5-1.5B | 1024 | 487.6 tok/s | 96.7% | 2.96 GB |
-| Qwen2.5-3B | 512 | 143.1 tok/s | 79.3% | 2.15 GB |
+Quantising the streamed base to NF4 makes the RAM store ~4× smaller. That matters for two reasons, and the second is the bigger one:
 
-**Headline:** Qwen2.5-3B trains in **2.15 GB** on a 4 GB card, where a resident run OOMs.
+1. A bigger model fits in host RAM at all — an 8B base is ~3.6 GB of NF4 instead of ~16 GB of bf16.
+2. **The store fits under the machine's page-locked memory ceiling.** Pinned host memory is what lets `copy_(non_blocking=True)` actually overlap with compute. The reference box tops out at ~7.1 GB of page-locked memory, so a 5.55 GB bf16 3B base fell back to pageable and lost overlap; the 1.43 GB NF4 store pins, and utilisation goes from 79.3% to 100%.
 
-**Honest scope (v0.72.0 = proof-of-mechanism):**
-- **Models measured:** Qwen2.5-0.5B, 1.5B and 3B, plus a live `soup train` on SmolLM2-135M. **Nothing above 3B was measured, and no 8B / 14B / 70B claim is supported.**
+The base is quantised **once, offline**, one tensor at a time, and cached. The shard cache is keyed to the quantisation, the dtype, the quantisation device and a fingerprint of the source checkpoint, so switching `none` ⇄ `4bit` — or retraining a base in place — re-shards rather than silently streaming the wrong bytes.
+
+Correctness is not a tradeoff here either: a streamed NF4 run is **bit-exact** against a *resident* NF4 run (the same quantised bytes through the same bitsandbytes kernels), and that is a regression test, not a one-off measurement.
+
+**Measured numbers (RTX 3050 Laptop 4 GB, Windows 11, LoRA, batch 1, 50 steps after 10 warmup):**
+
+| Model | Quant | Seq | Throughput | GPU Util | Peak VRAM | RAM store |
+|---|---|---|---|---|---|---|
+| **Llama-3.1-8B-Instruct** | **NF4** | 512 | **119.6 tok/s** | 100% | **3.32 GB** | 3.60 GB pinned |
+| Qwen2.5-3B | NF4 | 512 | 264.2 tok/s | 100% | 1.76 GB | 1.43 GB pinned |
+| Qwen2.5-3B | bf16 | 512 | 143.1 tok/s | 79.3% | 2.15 GB | 5.55 GB pageable |
+| Qwen2.5-1.5B | bf16 | 512 | 525.0 tok/s | 96.8% | 1.82 GB | pinned |
+| Qwen2.5-1.5B | bf16 | 1024 | 487.6 tok/s | 96.7% | 2.96 GB | pinned |
+| Qwen2.5-0.5B | bf16 | 512 | 978.6 tok/s | 91.4% | 1.47 GB | pinned |
+
+**Headline:** **Llama-3.1-8B fine-tunes on a 4 GB card at 119.6 tok/s in 3.32 GB.** For scale, 1M training tokens is ~2.3 h at 8B (arithmetic from the measured rate, not a separate measurement).
+
+The 3B NF4-vs-bf16 rows differ by 1.85×, but attribute that to **pinning, not arithmetic** — see point 2 above. The two rows also come from different sessions, and this card's boost clock varies ~13% between sessions, so treat the factor as indicative and the mechanism as the claim.
+
+Untied `embed_tokens` + `lm_head` stay resident and unquantised (2.10 GB of the 8B row's 3.32 GB), which is why 8B sits close to this card's ceiling; treating them as streamed large layers is a v0.72.3 item.
+
+**Honest scope:**
 - **RAM tier only.** `stream_source: ram` (`auto` resolves to it). The disk overflow tier ships in **v0.72.3**.
 - **Llama / Qwen only**, `task: sft`, `backend: transformers`, `modality: text`.
 - **`batch_size: 1`**, no gradient accumulation, no `--resume` — all ship in **v0.72.3**.
-- **bf16 base; `quantization: none` is required.** 4-bit (NF4) weights carry a quantisation state and cannot be byte-copied into a plain buffer, so NF4 streaming is **v0.72.2**. A `4bit` config is refused today with a message saying exactly that.
-- **The 3B throughput is a LOWER BOUND.** The reference box could not page-lock the 5.55 GB base (its measured page-locked ceiling is 7.65 GB, and a CUDA context plus the model skeleton did not leave room), so that run fell back to a pageable store. Pageable memory makes the host-to-device copy synchronous, which costs overlap — visible as the GPU-utilisation drop from 96.8% (1.5B, pinned) to 79.3% (3B, pageable). Soup does this fallback automatically **and prints the cost** rather than absorbing it silently.
+- **The bf16 3B throughput above is a LOWER BOUND.** The reference box could not page-lock the 5.55 GB base (its measured page-locked ceiling is 7.65 GB, and a CUDA context plus the model skeleton did not leave room), so that run fell back to a pageable store. Pageable memory makes the host-to-device copy synchronous, which costs overlap — visible as the GPU-utilisation drop from 96.8% (1.5B, pinned) to 79.3% (3B, pageable). Soup does this fallback automatically **and prints the cost** rather than absorbing it silently. NF4 lifts this at 3B: the store drops under the ceiling and pins.
 - Numbers are Windows/WDDM and therefore systematically pessimistic versus Linux. `expandable_segments:True` is silently ignored on Windows; Soup detects that and does not claim it is active.
 
 **Rejected at config load (each names the release that lifts it):**
 - `stream_source: disk` → the disk tier is v0.72.3
-- `quantization` other than `none` → NF4 streaming is v0.72.2
+- `quantization` other than `none` or `4bit` → other formats cannot be streamed into a pooled buffer
 - `backend: unsloth` / `backend: mlx` → streaming replaces the model-load path those backends own
 - `task` other than `sft` → preference losses are v0.72.4
 - `gradient_accumulation_steps > 1` → every micro-batch re-reads the whole base, so accumulation multiplies streaming IO linearly (v0.72.3)
@@ -296,7 +311,7 @@ training:
   lr: 2e-5
   batch_size: 1           # required; larger batches in v0.72.3
   gradient_accumulation_steps: 1   # required; accumulation in v0.72.3
-  quantization: none      # required; NF4 streaming is v0.72.2
+  quantization: 4bit      # NF4 — ~4x smaller RAM store than bf16 (or `none`)
   gradient_checkpointing: true     # handled per-layer by the streamer
   stream_layers: true     # Enable layer streaming
   stream_source: auto     # RAM-based (disk in v0.72.3)
@@ -325,11 +340,10 @@ output: ./output
 **Troubleshooting:**
 - **"layer streaming needs the base to fit in RAM"** — the base is larger than free RAM. Free RAM or pick a smaller base; the disk overflow tier is v0.72.3.
 - **"could not page-lock the base … falling back to a PAGEABLE RAM store"** — expected on a busy machine. Training continues, more slowly. Close other applications to keep the pinned store.
-- **"layer streaming does not support model_type=…"** — v0.72.0 covers Llama and Qwen only.
+- **"layer streaming does not support model_type=…"** — Llama and Qwen only for now; more architectures land in v0.72.3.
 - **Slower than you expected** — layer streaming trades time for memory. If the model already fits resident on your card, do not enable it.
 
 **Roadmap (each refusal names its release):**
-- 4-bit (NF4) streaming — **v0.72.2**, the first genuinely useful capability jump
 - Disk overflow tier, batch size > 1, gradient accumulation, checkpoint/resume, more architectures (Mistral / Gemma / Phi) — **v0.72.3**
 - Preference losses (DPO / ORPO / SimPO / KTO) — **v0.72.4**. GRPO and PPO are explicitly **not** planned: rollouts need generation, which re-reads the model per token
 - A published 14B-on-8 GB reference benchmark — hardware-blocked; it needs an 8 GB card and 32 GB of RAM, which the development box does not have

{soup_cli-0.72.1 → soup_cli-0.72.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.72.1"
+version = "0.72.2"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.72.1 → soup_cli-0.72.2}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
 
-__version__ = "0.72.1"
+__version__ = "0.72.2"

{soup_cli-0.72.1 → soup_cli-0.72.2}/src/soup_cli/config/schema.py

@@ -4660,7 +4660,7 @@ class SoupConfig(BaseModel):
             return self
         if self.task != "sft":
             raise ValueError(
-                f"training.stream_layers requires task='sft' in v0.72.0; got "
+                f"training.stream_layers requires task='sft'; got "
                 f"task={self.task!r}. Preference losses (DPO/ORPO/SimPO/KTO) "
                 f"land in v0.72.4."
             )
@@ -4675,21 +4675,21 @@ class SoupConfig(BaseModel):
                 f"training.stream_layers requires modality='text'; got "
                 f"modality={self.modality!r}."
             )
-        if tcfg.quantization != "none":
+        if tcfg.quantization not in ("none", "4bit"):
             raise ValueError(
-                f"training.stream_layers requires quantization='none' in "
-                f"v0.72.0 (got {tcfg.quantization!r}). NF4 weights carry a "
-                f"quant_state and cannot be byte-copied into a plain buffer; "
-                f"NF4 streaming lands in v0.72.2."
+                f"training.stream_layers supports quantization='none' or "
+                f"'4bit' (NF4); got {tcfg.quantization!r}. Other quantisations "
+                f"store weights in formats that cannot be streamed into a "
+                f"pooled buffer."
             )
         if tcfg.stream_source == "disk":
             raise ValueError(
-                "training.stream_source='disk' is not implemented in v0.72.0 — "
+                "training.stream_source='disk' is not implemented yet — "
                 "the disk overflow tier lands in v0.72.3. Use 'ram' or 'auto'."
             )
         if tcfg.batch_size != 1:
             raise ValueError(
-                f"training.stream_layers requires batch_size=1 in v0.72.0 (got "
+                f"training.stream_layers requires batch_size=1 (got "
                 f"{tcfg.batch_size!r}); larger batches land in v0.72.3."
             )
         if tcfg.gradient_accumulation_steps != 1:

{soup_cli-0.72.1 → soup_cli-0.72.2}/src/soup_cli/trainer/sft.py

@@ -228,6 +228,15 @@ class SFTTrainerWrapper:
                 p.numel() for p in self.model.parameters() if p.requires_grad
             )
             total = sum(p.numel() for p in self.model.parameters())
+        # v0.72.2 — under NF4 streaming PEFT's total is wrong by ~6.5x. It
+        # special-cases Params4bit as `numel * 2 * quant_storage.itemsize`,
+        # which is right for a RESIDENT one (whose numel is the packed count)
+        # but not for our `meta` placeholder, which still carries the LOGICAL
+        # shape. Measured on SmolLM2-135M: 878,154,048 vs a true 134,515,008.
+        # The sharder counted the real source elements, so use that.
+        stream_total = getattr(getattr(self, "_stream_runtime", None), "total_params", 0)
+        if stream_total:
+            total = stream_total
         pct = 100 * trainable / total if total else 0.0
         label = (
             "Spectrum targeted FT" if tcfg.unfrozen_parameters else "LoRA applied"
@@ -969,6 +978,8 @@ class SFTTrainerWrapper:
         from transformers import AutoConfig, AutoTokenizer
 
         from soup_cli.utils.layer_shard import (
+            QUANT_NF4,
+            QUANT_NONE,
             resolve_shard_dir,
             shard_checkpoint,
             source_weight_bytes,
@@ -978,14 +989,18 @@ class SFTTrainerWrapper:
             TIER_RAM,
             build_stream_plan,
             dtype_bytes,
+            estimate_stream_store_bytes,
             free_ram_bytes,
             render_stream_panel,
             stream_arch_of,
         )
         from soup_cli.utils.layer_stream_runtime import (
             RamSource,
+            build_meta_skeleton,
             build_streamed_model,
+            extras_resident_bytes,
             probe_expandable_segments,
+            quantised_layer_suffixes,
         )
         from soup_cli.utils.spectrum_scan import resolve_model_weights
 
@@ -1006,35 +1021,81 @@ class SFTTrainerWrapper:
         on_cuda = str(self.device).startswith("cuda")
         dtype = "bfloat16" if on_cuda else "float32"
 
+        # v0.72.2 — NF4. The decoder linears ship as packed nibbles + per-block
+        # absmax, so the RAM store is ~0.26x its bf16 size; embeddings, norms and
+        # an untied head stay at `dtype`, exactly as replace_with_bnb_linear
+        # leaves them.
+        quant = QUANT_NF4 if tcfg.quantization == "4bit" else QUANT_NONE
+
         weights_dir = resolve_model_weights(cfg.base)
         shard_dir = resolve_shard_dir(cfg.base)
 
         # Cheap size probe BEFORE sharding: re-writing a checkpoint we are
         # about to refuse for not fitting in RAM costs minutes of disk I/O.
+        # Charged at the STREAMED rate, not the on-disk one — an 8B bf16
+        # checkpoint is 16 GB on disk but only ~4.2 GB of NF4 store, and
+        # comparing the raw file size would refuse exactly the runs NF4 enables.
         early_free_ram = free_ram_bytes()
         if early_free_ram is not None:
             source_bytes = source_weight_bytes(weights_dir)
-            if source_bytes >= early_free_ram * RAM_TIER_HEADROOM:
+            store_estimate = estimate_stream_store_bytes(
+                source_bytes, dtype=dtype, quant=quant
+            )
+            if store_estimate >= early_free_ram * RAM_TIER_HEADROOM:
+                as_streamed = (
+                    ""
+                    if quant == QUANT_NONE
+                    else f" ({store_estimate / 1e9:.1f} GB once quantised to NF4)"
+                )
                 raise ValueError(
                     f"layer streaming needs the base to fit in RAM: {cfg.base} is "
-                    f"{source_bytes / 1e9:.1f} GB on disk and only "
+                    f"{source_bytes / 1e9:.1f} GB on disk{as_streamed} and only "
                     f"{early_free_ram / 1e9:.1f} GB of RAM is free. The disk "
                     f"overflow tier lands in v0.72.3 — free RAM or pick a "
                     f"smaller base."
                 )
 
-        console.print(f"[dim]Preparing layer shards -> {shard_dir}[/]")
-        index = shard_checkpoint(weights_dir, shard_dir, dtype=dtype, arch=arch)
+        # The authoritative list of weights to quantise is whatever
+        # replace_with_bnb_linear actually converts, read off a meta skeleton —
+        # not a hard-coded name list that would drift per architecture.
+        #
+        # This builds a second, throwaway skeleton (build_streamed_model makes
+        # its own). Deliberate: a meta skeleton allocates NO weight storage, so
+        # the cost is module-tree construction only, and threading a pre-built
+        # model into build_streamed_model would couple suffix discovery to model
+        # construction for no memory saving.
+        quant_suffixes = ()
+        if quant == QUANT_NF4:
+            probe = build_meta_skeleton(
+                cfg.base,
+                dtype=dtype,
+                quant=quant,
+                trust_remote_code=self._trust_remote_code,
+            )
+            quant_suffixes = quantised_layer_suffixes(probe)
+            del probe
 
-        spec = RamSource.spec_from_shard(shard_dir, index)
-        bytes_per = dtype_bytes(dtype)
-        layer_params = sum(
-            math.prod(shape) for shape, _dtype in spec.values()
+        console.print(f"[dim]Preparing layer shards -> {shard_dir}[/]")
+        index = shard_checkpoint(
+            weights_dir,
+            shard_dir,
+            dtype=dtype,
+            arch=arch,
+            quant=quant,
+            quant_suffixes=quant_suffixes,
+            # Quantise on the device that will run the model: CPU and CUDA agree
+            # on the packed nibbles but not on every float32 nested statistic.
+            quant_device=str(self.device),
         )
-        layer_bytes = layer_params * bytes_per
-        embed_bytes = max(
-            0, (index.total_params - layer_params * index.n_layers) * bytes_per
+
+        spec = RamSource.spec_from_shard(shard_dir)
+        # Measured from the shard headers, not derived from `total_params`:
+        # under NF4 a layer holds packed uint8 alongside float32 statistics, so
+        # element counts no longer convert to bytes at a single rate.
+        layer_bytes = sum(
+            math.prod(shape) * dtype_bytes(stored) for shape, stored in spec.values()
         )
+        embed_bytes = extras_resident_bytes(shard_dir)
 
         free_ram = free_ram_bytes()
         if free_ram is None:
@@ -1101,6 +1162,7 @@ class SFTTrainerWrapper:
             seed=tcfg.seed if getattr(tcfg, "seed", None) is not None else 0,
             trust_remote_code=self._trust_remote_code,
             console=console,
+            quant=quant,
         )
         self.model = model
         self._stream_runtime = runtime