soup-cli 0.72.1__tar.gz → 0.72.3__tar.gz

soup_cli-0.72.3/.github/FUNDING.yml

@@ -0,0 +1,7 @@
+# Sponsor button shown in the repo sidebar and on every issue / PR.
+#
+# `github:` only renders once the account is enrolled in GitHub Sponsors
+# (https://github.com/sponsors) — it is not, so it stays commented out.
+# github: MakazhanAlpamys
+
+custom: ["https://buy.stripe.com/4gMcN441k3pha3T19ye7m04"]

{soup_cli-0.72.1 → soup_cli-0.72.3}/CHANGELOG.md

@@ -12,6 +12,144 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.72.3] - 2026-07-28
+
+**Added — layer streaming breadth: more architectures, bigger batches, resume, and a
+disk tier.**
+
+Layer streaming (v0.72.0–.2) was deliberately narrow: Llama/Qwen only, batch 1, no
+gradient accumulation, no resume, RAM only. This release lifts all of it, and each
+capability was gated against a streamed-vs-resident bit-exactness reference before any
+of it was written.
+
+- **Six more model families.** `mistral`, `gemma`, `gemma2`, `gemma3_text`, `phi` and
+  `phi3` join the allowlist, each verified **bit-exact** against the same checkpoint
+  loaded resident, under both bf16 and NF4. Phi-3 is the notable one: it fuses Q/K/V
+  into a single `qkv_proj`, so there is no `q_proj` to find, and it is bit-exact anyway.
+  Multimodal `gemma3` is deliberately *not* accepted — only `gemma3_text`.
+- **`batch_size` above 1, and gradient accumulation.** Both previously refused.
+- **A pre-flight VRAM budget that accounts for batch and vocabulary.** Streaming bounds
+  the *weights*; activations and the logits tensor are untouched by it and both scale
+  with `batch × seq`. On a 152k-vocab model at batch 8 the logits term alone measured
+  **8.71 GB — 146× the entire layer-buffer pool**. `soup train` now predicts peak VRAM
+  and refuses a run that will not fit, naming the two knobs that scale it.
+- **A throughput forecast**, quoted as a range from a GEMM ceiling measured on your own
+  card in that session, alongside the SM clock — never a compiled-in per-card constant.
+- **`--resume` and `--hf-resume` work with streaming.**
+- **A disk overflow tier.** `stream_source: auto` (the default) uses RAM when the base
+  fits and falls back to an NVMe disk tier when it does not, holding nothing resident.
+  `stream_source: ram` refuses instead of falling back. Non-NVMe disks are still
+  refused outright.
+- **`soup doctor --disk`** reports the detected media type.
+
+**Fixed**
+
+- `estimate_logits_bytes` charged 6 bytes per logit element; the measured peak is **14**
+  (`transformers` holds the bf16 logits, the fp32 upcast, log-softmax's fp32 output and
+  the fp32 gradient live at once). The old figure under-predicted that term by 2.33×.
+- Adapters could not be loaded *into* a streamed model: `load_state_dict` narrows keys
+  by child name, so a canonical checkpoint matched **0 of N** tensors and PEFT reported
+  only a warning — a resumed run reproduced the from-scratch loss curve exactly. The
+  streaming layer now redirects canonical keys at load time, mirroring the v0.72.1
+  save-side fix.
+- The NVMe-only tier guard was wired to a hardcoded constant and could never fire.
+- Streaming weight sources are now released when training ends **or raises**; the disk
+  tier holds one open shard handle per decoder layer.
+- Subprocess helpers resolve tools to absolute paths (on Windows, `CreateProcess`
+  searches the current directory before `PATH`).
+- The `[mcp]` extra is now capped at `mcp<2`. The SDK's 2.0.0 release removed
+  `mcp.shared.memory.create_connected_server_and_client_session` and dropped
+  `Server.list_tools`, breaking `soup mcp serve`'s round-trip tests for anyone
+  installing fresh. Support for the 2.x API is tracked separately.
+
+**Known limitations**
+
+- The **RAM-vs-disk performance gap is unmeasured** on the development hardware and no
+  number is claimed for it. safetensors memory-maps the shards, so the OS page cache
+  keeps them resident between steps on a machine with spare RAM, and at ~5 effective
+  TFLOPS the NVMe read hides under compute. The disk tier's *correctness* is verified
+  bit-exact against the RAM tier; its speed relative to RAM is not characterised.
+- End-to-end `soup train --resume` could not be demonstrated on the development box:
+  `transformers` refuses `torch.load` below torch 2.6 (CVE-2025-32434), which blocks
+  **every** resume there, streaming or not. The streaming-specific half — the adapter
+  round-trip and loss continuity — is verified on the production CUDA path.
+- Loading *into* a streamed model works; `named_parameters()` and `state_dict()` still
+  disagree in memory, which is the deliberate cost of a serialisation-only design.
+- Layer streaming remains **BETA**.
+
+## [0.72.2] - 2026-07-28
+
+**Added — NF4 layer streaming: fine-tune Llama-3.1-8B on a 4 GB laptop GPU.**
+
+Layer streaming (v0.72.0) keeps the frozen base in CPU RAM and streams it to the
+GPU one decoder layer at a time, so peak VRAM is bounded by one layer instead of
+the whole model. It was bf16-only, which capped it at about 3B on a small card.
+Quantising the streamed base to NF4 shrinks it ~4×, and that is what brings 8B
+within reach.
+
+Add one line to a streaming config:
+
+```yaml
+training:
+  stream_layers: true
+  quantization: 4bit    # NF4
+  batch_size: 1
+```
+
+**Measured on a 4 GB RTX 3050 Laptop** (Windows, batch 1, S=512, gradient
+checkpointing, 50 steps after 10 warm-up, `PagedAdamW8bit`):
+
+| Model | tok/s | Peak VRAM | RAM store | GPU util |
+|---|---|---|---|---|
+| Llama-3.1-8B-Instruct | 119.6 | 3.32 GB | 3.60 GB (page-locked) | 100% |
+| Qwen2.5-3B | 264.2 | 1.76 GB | 1.43 GB (page-locked) | 100% |
+
+For scale: 1M training tokens is about 2.3 h at 8B on that card (arithmetic from
+the measured rate, not a separate measurement).
+
+Qwen2.5-3B also got **1.85× faster** than the bf16 streaming path (264.2 vs
+143.1 tok/s) — and the reason is not arithmetic. A 1.43 GB store fits under the
+machine's page-locked memory ceiling where a 5.55 GB one did not, which restores
+asynchronous host-to-device copies and lifts GPU utilisation from 79.3% to 100%.
+
+**Correctness.** A streamed NF4 run is **bit-exact** against a resident NF4 run:
+the same quantised bytes through the same bitsandbytes kernels. Logit equality,
+non-zero gradients at layer 0, and a matching multi-step loss curve are all
+regression tests, not one-off measurements.
+
+The base is quantised once, offline, and cached under `~/.soup/layer-stream/`.
+The cache is keyed to the quantisation *and* the source checkpoint, so switching
+between `none` and `4bit`, or retraining a base in place, re-shards instead of
+silently streaming the wrong bytes.
+
+**Fixed — a streamed 4-bit run reported its parameter count ~6.5× too high.**
+SmolLM2-135M printed "878,154,048 total" (true: 134,515,008). Display only —
+training was unaffected — but at 8B it would have read ~52 B.
+
+Scope is unchanged and still BETA: RAM tier, `task: sft`, Llama/Qwen,
+`batch_size: 1`, no gradient accumulation, no `--resume`. Every rejected config
+names the release that lifts it. `quantization` values other than `none` and
+`4bit` are refused.
+
+**Fixed — `soup --help` was 5x slower than it should be.** Since v0.72.0 the CLI
+imported PyTorch on startup, taking **6.0 s** where it now takes **1.15 s**.
+
+`soup reward stress` (v0.71.41) put `utils/reward_stress` on the light CLI path.
+That module imported `utils/reward_hack_control` to reuse a single string
+constant — and `reward_hack_control` resolves its `TrainerCallback` base class at
+module scope, which pulls in `transformers` and `torch`. Importing a ~4.4 s
+dependency for one constant made every `soup` invocation pay for the training
+stack, including commands that never touch a model.
+
+Nothing produced wrong results; this was purely startup latency. The light core
+(`pip install soup-cli` without the `[train]` extra) was never broken — it fell
+back cleanly when torch was absent, just slowly when it was present.
+
+Added `tests/test_cli_startup_is_light.py`, which asserts the invariant at
+runtime (`import soup_cli.cli` must not put `torch` in `sys.modules`) instead of
+inspecting source text for `import torch`, which is what the previous guards did
+and why this went unnoticed.
+
 ## [0.72.1] - 2026-07-27
 
 **Fixed — layer-streaming adapters were saved in an unloadable form.** If you

{soup_cli-0.72.1 → soup_cli-0.72.3}/CONTRIBUTING.md

@@ -120,7 +120,7 @@ src/soup_cli/
   templates/          - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (325 files, 16752 tests)
+tests/                - Test suite (330 files, 16977 tests)
 examples/             - Real-world config examples and datasets
 ```
 

{soup_cli-0.72.1 → soup_cli-0.72.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.72.1
+Version: 0.72.3
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -29,7 +29,7 @@ Requires-Dist: bitsandbytes>=0.41.0; extra == 'all'
 Requires-Dist: datasets>=2.14.0; extra == 'all'
 Requires-Dist: datasketch>=1.6.0; extra == 'all'
 Requires-Dist: fastapi>=0.104.0; extra == 'all'
-Requires-Dist: mcp>=1.2.0; extra == 'all'
+Requires-Dist: mcp<2,>=1.2.0; extra == 'all'
 Requires-Dist: peft>=0.7.0; extra == 'all'
 Requires-Dist: torch>=2.0.0; extra == 'all'
 Requires-Dist: transformers<5.0.0,>=4.36.0; extra == 'all'
@@ -61,7 +61,7 @@ Requires-Dist: bitsandbytes>=0.41.0; extra == 'dev'
 Requires-Dist: cryptography>=41.0.0; extra == 'dev'
 Requires-Dist: datasets>=2.14.0; extra == 'dev'
 Requires-Dist: httpx>=0.24.0; extra == 'dev'
-Requires-Dist: mcp>=1.2.0; extra == 'dev'
+Requires-Dist: mcp<2,>=1.2.0; extra == 'dev'
 Requires-Dist: mypy>=1.8.0; extra == 'dev'
 Requires-Dist: peft>=0.7.0; extra == 'dev'
 Requires-Dist: pre-commit>=3.5.0; extra == 'dev'
@@ -83,7 +83,7 @@ Requires-Dist: auto-gptq>=0.7.0; extra == 'gptq'
 Provides-Extra: liger
 Requires-Dist: liger-kernel>=0.3.0; extra == 'liger'
 Provides-Extra: mcp
-Requires-Dist: mcp>=1.2.0; extra == 'mcp'
+Requires-Dist: mcp<2,>=1.2.0; extra == 'mcp'
 Provides-Extra: mix
 Requires-Dist: scikit-optimize>=0.9.0; extra == 'mix'
 Provides-Extra: mlx
@@ -193,41 +193,41 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.72.1 — Fix: layer-streaming adapters were saved unloadable.** If you trained with
-`stream_layers: true` on v0.72.0, that adapter is **inert** — every tensor was written under a
-key with an extra `.inner.` segment, so `soup merge`, `soup serve`, `soup chat` and
-`PeftModel.from_pretrained` loaded **zero** tensors and silently returned the untuned base.
-Only a `UserWarning` was emitted.
-
-- **The training was correct; only the saved file was affected.** v0.72.0's bit-exactness
-  results still stand.
-- **Check an existing adapter:** if `adapter_model.safetensors` has keys containing `.inner.`,
-  it is affected — re-run or re-save it on v0.72.1.
-- Streamed adapters now save in exactly the same layout as an ordinary LoRA run.
-
-**Layer streaming (BETA), from v0.72.0.** Fine-tune models that don't fit in your card. The
-frozen base streams from CPU RAM one decoder layer at a time into a small pool of VRAM buffers
-while the LoRA adapters stay resident, so peak VRAM is bounded by one layer rather than the
-whole model. **Qwen2.5-3B trains in 2.15 GB on a 4 GB card, where a resident run OOMs.**
-
-- **`training.stream_layers: true`** — a config key, not a CLI flag. Tune with
-  `stream_source` (`ram` today; `disk` is v0.72.3) and `stream_buffers` (2–8, default 2).
-- **Measured on a 4 GB RTX 3050 Laptop** (batch 1, gradient checkpointing on): 0.5B at
-  978.6 tok/s / 1.47 GB, 1.5B at 525.0 tok/s / 1.82 GB, 3B at 143.1 tok/s / 2.15 GB.
-- **Honest cost: 1.43× slower than resident**, measured at 0.5B — the only apples-to-apples
-  comparison available on that box, because 1.5B and above cannot run resident there at all.
-- **Proof-of-mechanism at 3B.** Nothing above 3B was measured; no 8B/14B claim is supported.
-  Scope: RAM tier, bf16, `task: sft`, Llama/Qwen, batch size 1, no gradient accumulation, no
-  `--resume`. 4-bit (NF4) streaming is **v0.72.2** and is refused with a clear message today.
+**v0.72.3 — layer streaming grows up: more models, bigger batches, resume, and a disk
+tier.** Layer streaming keeps the frozen base out of VRAM and feeds it to the GPU one
+decoder layer at a time. v0.72.0–.2 kept the scope deliberately tiny to prove it worked;
+this release removes the training wheels.
+
+- **Six more model families** — Mistral, Gemma / Gemma 2 / Gemma 3, and Phi / Phi-3 — each
+  verified **bit-exact** against the same checkpoint loaded resident, in bf16 *and* NF4.
+- **`batch_size` above 1, gradient accumulation, and `--resume`** all work now.
+- **A pre-flight that predicts peak VRAM and refuses a run that will not fit.** Streaming
+  bounds the *weights*; the logits tensor is not bounded by it and scales with
+  `batch × seq`. On a 152k-vocab model at batch 8 that single tensor measured **8.71 GB —
+  146× the entire layer-buffer pool.** The prediction was fitted to ten real runs and
+  never under-predicts any of them.
+- **A throughput forecast measured on your card, in your session**, quoted as a range
+  next to the SM clock it was taken at — not a number compiled into the source.
+- **A disk overflow tier.** When the base will not fit in RAM, `stream_source: auto`
+  streams it from NVMe instead of refusing. Honest caveat: its *correctness* is verified
+  bit-exact against the RAM tier, but **how much slower it is has not been measured** on
+  the development hardware, and no figure is claimed.
+- Still BETA.
 
 ```yaml
 # soup.yaml — then just `soup train --config soup.yaml`
 training:
-  stream_layers: true      # base streams from RAM; only the adapter trains
-  batch_size: 1
-  quantization: none       # NF4 streaming lands in v0.72.2
+  stream_layers: true      # base streams out of VRAM; only the adapter trains
+  quantization: 4bit       # NF4 — ~4x smaller store, so 8B fits a 4 GB card
+  batch_size: 4            # v0.72.3: bigger batches amortise the weight read
+  stream_source: auto      # RAM when it fits, NVMe disk when it does not
 ```
 
+> **Trained with `stream_layers: true` on v0.72.0?** That adapter is inert — its tensors were
+> saved under keys with an extra `.inner.` segment, so every loader returned the untuned base.
+> Fixed in v0.72.1; re-run or re-save. Check with:
+> `python -c "from safetensors.torch import load_file; print([k for k in load_file('adapter_model.safetensors') if '.inner.' in k][:3])"`
+
 <details>
 <summary>Previous release — v0.71.40, soup reward synth (generate a reward verifier from your data)</summary>
 
@@ -495,6 +495,25 @@ pre-commit install                 # optional: ruff lint+format on commit
 See [CONTRIBUTING.md](CONTRIBUTING.md) for the full workflow and [SECURITY.md](SECURITY.md) to
 report a vulnerability.
 
+## Support Soup
+
+Soup is Apache-2.0 and free — and stays that way. It is built and maintained in the open on a
+single 4 GB laptop, which is why every performance number in these docs is measured rather than
+claimed.
+
+If Soup saved you a training run, [starring the repo](https://github.com/MakazhanAlpamys/Soup)
+helps most, and it costs nothing. If you would like to fund the work directly:
+
+**[❤️ Donate](https://buy.stripe.com/4gMcN441k3pha3T19ye7m04)** — one-off, any amount (use
+*Change amount* on the checkout page). Payments are processed by Stripe under the maintainer's
+registered business, **MePlay, Inc.** — that name, not "Soup", is what appears on the checkout
+page and on your card statement.
+
+Donations fund GPU time for the hardware-gated work — multi-GPU, 8B+ validation, Apple
+Silicon — that a single 4 GB laptop cannot reach. See the
+[`help wanted`](https://github.com/MakazhanAlpamys/Soup/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22)
+issues for exactly what is blocked on hardware today.
+
 ## Contributors
 
 Built by the community ❤️ — thank you to everyone who has contributed. See

{soup_cli-0.72.1 → soup_cli-0.72.3}/README.md

@@ -49,41 +49,41 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.72.1 — Fix: layer-streaming adapters were saved unloadable.** If you trained with
-`stream_layers: true` on v0.72.0, that adapter is **inert** — every tensor was written under a
-key with an extra `.inner.` segment, so `soup merge`, `soup serve`, `soup chat` and
-`PeftModel.from_pretrained` loaded **zero** tensors and silently returned the untuned base.
-Only a `UserWarning` was emitted.
-
-- **The training was correct; only the saved file was affected.** v0.72.0's bit-exactness
-  results still stand.
-- **Check an existing adapter:** if `adapter_model.safetensors` has keys containing `.inner.`,
-  it is affected — re-run or re-save it on v0.72.1.
-- Streamed adapters now save in exactly the same layout as an ordinary LoRA run.
-
-**Layer streaming (BETA), from v0.72.0.** Fine-tune models that don't fit in your card. The
-frozen base streams from CPU RAM one decoder layer at a time into a small pool of VRAM buffers
-while the LoRA adapters stay resident, so peak VRAM is bounded by one layer rather than the
-whole model. **Qwen2.5-3B trains in 2.15 GB on a 4 GB card, where a resident run OOMs.**
-
-- **`training.stream_layers: true`** — a config key, not a CLI flag. Tune with
-  `stream_source` (`ram` today; `disk` is v0.72.3) and `stream_buffers` (2–8, default 2).
-- **Measured on a 4 GB RTX 3050 Laptop** (batch 1, gradient checkpointing on): 0.5B at
-  978.6 tok/s / 1.47 GB, 1.5B at 525.0 tok/s / 1.82 GB, 3B at 143.1 tok/s / 2.15 GB.
-- **Honest cost: 1.43× slower than resident**, measured at 0.5B — the only apples-to-apples
-  comparison available on that box, because 1.5B and above cannot run resident there at all.
-- **Proof-of-mechanism at 3B.** Nothing above 3B was measured; no 8B/14B claim is supported.
-  Scope: RAM tier, bf16, `task: sft`, Llama/Qwen, batch size 1, no gradient accumulation, no
-  `--resume`. 4-bit (NF4) streaming is **v0.72.2** and is refused with a clear message today.
+**v0.72.3 — layer streaming grows up: more models, bigger batches, resume, and a disk
+tier.** Layer streaming keeps the frozen base out of VRAM and feeds it to the GPU one
+decoder layer at a time. v0.72.0–.2 kept the scope deliberately tiny to prove it worked;
+this release removes the training wheels.
+
+- **Six more model families** — Mistral, Gemma / Gemma 2 / Gemma 3, and Phi / Phi-3 — each
+  verified **bit-exact** against the same checkpoint loaded resident, in bf16 *and* NF4.
+- **`batch_size` above 1, gradient accumulation, and `--resume`** all work now.
+- **A pre-flight that predicts peak VRAM and refuses a run that will not fit.** Streaming
+  bounds the *weights*; the logits tensor is not bounded by it and scales with
+  `batch × seq`. On a 152k-vocab model at batch 8 that single tensor measured **8.71 GB —
+  146× the entire layer-buffer pool.** The prediction was fitted to ten real runs and
+  never under-predicts any of them.
+- **A throughput forecast measured on your card, in your session**, quoted as a range
+  next to the SM clock it was taken at — not a number compiled into the source.
+- **A disk overflow tier.** When the base will not fit in RAM, `stream_source: auto`
+  streams it from NVMe instead of refusing. Honest caveat: its *correctness* is verified
+  bit-exact against the RAM tier, but **how much slower it is has not been measured** on
+  the development hardware, and no figure is claimed.
+- Still BETA.
 
 ```yaml
 # soup.yaml — then just `soup train --config soup.yaml`
 training:
-  stream_layers: true      # base streams from RAM; only the adapter trains
-  batch_size: 1
-  quantization: none       # NF4 streaming lands in v0.72.2
+  stream_layers: true      # base streams out of VRAM; only the adapter trains
+  quantization: 4bit       # NF4 — ~4x smaller store, so 8B fits a 4 GB card
+  batch_size: 4            # v0.72.3: bigger batches amortise the weight read
+  stream_source: auto      # RAM when it fits, NVMe disk when it does not
 ```
 
+> **Trained with `stream_layers: true` on v0.72.0?** That adapter is inert — its tensors were
+> saved under keys with an extra `.inner.` segment, so every loader returned the untuned base.
+> Fixed in v0.72.1; re-run or re-save. Check with:
+> `python -c "from safetensors.torch import load_file; print([k for k in load_file('adapter_model.safetensors') if '.inner.' in k][:3])"`
+
 <details>
 <summary>Previous release — v0.71.40, soup reward synth (generate a reward verifier from your data)</summary>
 
@@ -351,6 +351,25 @@ pre-commit install                 # optional: ruff lint+format on commit
 See [CONTRIBUTING.md](CONTRIBUTING.md) for the full workflow and [SECURITY.md](SECURITY.md) to
 report a vulnerability.
 
+## Support Soup
+
+Soup is Apache-2.0 and free — and stays that way. It is built and maintained in the open on a
+single 4 GB laptop, which is why every performance number in these docs is measured rather than
+claimed.
+
+If Soup saved you a training run, [starring the repo](https://github.com/MakazhanAlpamys/Soup)
+helps most, and it costs nothing. If you would like to fund the work directly:
+
+**[❤️ Donate](https://buy.stripe.com/4gMcN441k3pha3T19ye7m04)** — one-off, any amount (use
+*Change amount* on the checkout page). Payments are processed by Stripe under the maintainer's
+registered business, **MePlay, Inc.** — that name, not "Soup", is what appears on the checkout
+page and on your card statement.
+
+Donations fund GPU time for the hardware-gated work — multi-GPU, 8B+ validation, Apple
+Silicon — that a single 4 GB laptop cannot reach. See the
+[`help wanted`](https://github.com/MakazhanAlpamys/Soup/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22)
+issues for exactly what is blocked on hardware today.
+
 ## Contributors
 
 Built by the community ❤️ — thank you to everyone who has contributed. See

{soup_cli-0.72.1 → soup_cli-0.72.3}/docs/commands.md

@@ -30,7 +30,7 @@ soup merge-sharded-fsdp-weights ./shards -o merged.safetensors  Consolidate FSDP
 soup delinearize-llama4 ./src --target ./out [--num-experts N] [--plan-only]  Live Llama-4 fused-expert reshape [E*din,dout] -> [E,din,dout] + sidecar copy (v0.71.21)
 soup spectrum scan --model <id|path> --top-percent 50 [--modules mlp,attn] [-o patch.yaml]  Spectrum SNR scan (no model load) -> training.unfrozen_parameters YAML patch (v0.71.23)
 soup train --config sft.yaml  # training.lisa_enabled: true [lisa_num_layers lisa_interval_steps]  LISA layerwise importance sampling — full-FT quality at LoRA-like memory (sft/transformers/text/quantization=none) (v0.71.34)
-soup train --config sft.yaml  # training.stream_layers: true [stream_source stream_buffers]  BETA layer streaming — the frozen base streams from CPU RAM one decoder layer at a time, so peak VRAM is bounded by ONE layer (sft/transformers/text/quantization=none, batch 1) (v0.72.0)
+soup train --config sft.yaml  # training.stream_layers: true [stream_source stream_buffers]  BETA layer streaming — the frozen base streams from CPU RAM/NVMe one decoder layer at a time, so peak VRAM is bounded by ONE layer; quantization: 4bit streams it as NF4, ~4x smaller (sft/transformers/text, wider archs) (v0.72.0; NF4 v0.72.2; disk+batch+accum v0.72.3)
 soup export --model ./output --format gguf    Export to GGUF (Ollama)
 soup export --model ./output --deploy ollama  Export GGUF + auto-deploy to Ollama
 soup export --model ./output --format onnx    Export to ONNX
@@ -200,7 +200,7 @@ soup tokenizer train --input c.jsonl --vocab-size N  Train BPE tokenizer (v0.53.
 soup bench <model> --p50 --p95                Bench with tail-latency percentiles (v0.53.9)
 soup bench <model> --backend auto             Auto-detect transformers/mlx backend (v0.53.9)
 soup serve --reasoning-parser deepseek-r1     Strip <think> blocks from responses (v0.53.9)
-soup doctor [--nccl]                          Check environment (optionally check NCCL bandwidth)
+soup doctor [--nccl] [--disk]                 Check environment (optionally check NCCL bandwidth, media type; --disk ~9s cold / ~2.4s warm)
 soup quickstart [--dry-run]                   Full demo
 soup adapters scan <adapter>                  Spectral backdoor scan (rank-1 dominance + outlier detection)
 soup adapters sign <adapter> [--backend unsigned|ed25519] [--key <pem>|--generate-key <pem>]  Merkle manifest + ed25519 sign

{soup_cli-0.72.1 → soup_cli-0.72.3}/docs/performance-and-quantization.md

@@ -14,7 +14,7 @@
 - [Cross-Document Attention Masking](#cross-document-attention-masking)
 - [Quant Menu — 9 Quantization Formats](#quant-menu--9-quantization-formats)
 - [Activation Offloading (Small-VRAM Large-Batch)](#activation-offloading-small-vram-large-batch)
-- [Layer Streaming (BETA, v0.72.0)](#layer-streaming-beta-v0720)
+- [Layer Streaming (BETA, v0.72.0; NF4 v0.72.2; disk + wider archs v0.72.3)](#layer-streaming-beta-v0720-nf4-v0722-disk--wider-archs-v0723)
 - [Correctness First (v0.36.0)](#correctness-first-v0360)
 - [Multi-GPU / DeepSpeed / FSDP](#multi-gpu--deepspeed--fsdp)
 - [Performance + Long-Context](#performance--long-context)
@@ -24,7 +24,7 @@
 - [MoE Expert Quantization + Router-Only Training (live in v0.71.20)](#moe-expert-quantization--router-only-training-live-in-v07120)
 - [Unsloth Dynamic 2.0 GGUF Ladder (v0.53.0)](#unsloth-dynamic-20-gguf-ladder-v0530)
 - [KV Cache Types (v0.53.0)](#kv-cache-types-v0530)
-- [FP8 Attention + NVFP4 + Native `unsloth_bnb_4bit` (v0.53.0)](#fp8-attention--nvfp4--native-unsloth_bnb_4bit-v0530)
+- [FP8 Attention + NVFP4 + Native `unsloth_bnb_4bit` (v0.53.0)](#fp8-attention--nvfp4--native-unsloth_bnb_4bit)
 - [LF / Axolotl Quant Parity (v0.53.0)](#lf--axolotl-quant-parity-v0530)
 - [Advanced Save Formats (v0.53.0)](#advanced-save-formats-v0530)
 - [Quant Menu II + Export Pipeline (v0.53.1)](#quant-menu-ii--export-pipeline-v0531)
@@ -226,7 +226,7 @@ training:
 Not compatible with unsloth (own memory manager) or mlx. Wired across every transformer-backend trainer (SFT, DPO, GRPO, KTO, ORPO, SimPO, IPO, PPO, Reward-Model, Embedding, Pretrain).
 
 
-## Layer Streaming (BETA, v0.72.0)
+## Layer Streaming (BETA, v0.72.0; NF4 v0.72.2; disk + wider archs v0.72.3)
 
 Stream frozen base-model decoder layers ONE at a time from CPU RAM into small VRAM buffers instead of keeping the whole base resident. Peak VRAM is bounded by the size of a single layer, not the entire model — so models that don't fit resident on your GPU can now train at all.
 
@@ -246,37 +246,107 @@ soup train --config soup.yaml
 
 The tradeoff: **1.43× slower than resident training**, measured at 0.5B — the only apples-to-apples comparison available on the reference box, because 1.5B and above cannot run resident there at all.
 
-**Measured numbers (RTX 3050 Laptop 4 GB, Windows 11, bf16 LoRA, batch 1, 50 steps after 10 warmup):**
+### NF4 streaming (`quantization: 4bit`)
 
-| Model | Seq Length | Throughput | GPU Util | Peak VRAM |
-|---|---|---|---|---|
-| Qwen2.5-0.5B | 512 | 978.6 tok/s | 91.4% | 1.47 GB |
-| Qwen2.5-1.5B | 512 | 525.0 tok/s | 96.8% | 1.82 GB |
-| Qwen2.5-1.5B | 1024 | 487.6 tok/s | 96.7% | 2.96 GB |
-| Qwen2.5-3B | 512 | 143.1 tok/s | 79.3% | 2.15 GB |
-
-**Headline:** Qwen2.5-3B trains in **2.15 GB** on a 4 GB card, where a resident run OOMs.
-
-**Honest scope (v0.72.0 = proof-of-mechanism):**
-- **Models measured:** Qwen2.5-0.5B, 1.5B and 3B, plus a live `soup train` on SmolLM2-135M. **Nothing above 3B was measured, and no 8B / 14B / 70B claim is supported.**
-- **RAM tier only.** `stream_source: ram` (`auto` resolves to it). The disk overflow tier ships in **v0.72.3**.
-- **Llama / Qwen only**, `task: sft`, `backend: transformers`, `modality: text`.
-- **`batch_size: 1`**, no gradient accumulation, no `--resume` — all ship in **v0.72.3**.
-- **bf16 base; `quantization: none` is required.** 4-bit (NF4) weights carry a quantisation state and cannot be byte-copied into a plain buffer, so NF4 streaming is **v0.72.2**. A `4bit` config is refused today with a message saying exactly that.
-- **The 3B throughput is a LOWER BOUND.** The reference box could not page-lock the 5.55 GB base (its measured page-locked ceiling is 7.65 GB, and a CUDA context plus the model skeleton did not leave room), so that run fell back to a pageable store. Pageable memory makes the host-to-device copy synchronous, which costs overlap — visible as the GPU-utilisation drop from 96.8% (1.5B, pinned) to 79.3% (3B, pageable). Soup does this fallback automatically **and prints the cost** rather than absorbing it silently.
+Quantising the streamed base to NF4 makes the RAM store ~4× smaller. That matters for two reasons, and the second is the bigger one:
+
+1. A bigger model fits in host RAM at all — an 8B base is ~3.6 GB of NF4 instead of ~16 GB of bf16.
+2. **The store fits under the machine's page-locked memory ceiling.** Pinned host memory is what lets `copy_(non_blocking=True)` actually overlap with compute. The reference box tops out at ~7.1 GB of page-locked memory, so a 5.55 GB bf16 3B base fell back to pageable and lost overlap; the 1.43 GB NF4 store pins, and utilisation goes from 79.3% to 100%.
+
+The base is quantised **once, offline**, one tensor at a time, and cached. The shard cache is keyed to the quantisation, the dtype, the quantisation device and a fingerprint of the source checkpoint, so switching `none` ⇄ `4bit` — or retraining a base in place — re-shards rather than silently streaming the wrong bytes.
+
+Correctness is not a tradeoff here either: a streamed NF4 run is **bit-exact** against a *resident* NF4 run (the same quantised bytes through the same bitsandbytes kernels), and that is a regression test, not a one-off measurement.
+
+**Measured numbers (RTX 3050 Laptop 4 GB, Windows 11, LoRA, batch 1, 50 steps after 10 warmup):**
+
+| Model | Quant | Seq | Throughput | GPU Util | Peak VRAM | RAM store |
+|---|---|---|---|---|---|---|
+| **Llama-3.1-8B-Instruct** | **NF4** | 512 | **119.6 tok/s** | 100% | **3.32 GB** | 3.60 GB pinned |
+| Qwen2.5-3B | NF4 | 512 | 264.2 tok/s | 100% | 1.76 GB | 1.43 GB pinned |
+| Qwen2.5-3B | bf16 | 512 | 143.1 tok/s | 79.3% | 2.15 GB | 5.55 GB pageable |
+| Qwen2.5-1.5B | bf16 | 512 | 525.0 tok/s | 96.8% | 1.82 GB | pinned |
+| Qwen2.5-1.5B | bf16 | 1024 | 487.6 tok/s | 96.7% | 2.96 GB | pinned |
+| Qwen2.5-0.5B | bf16 | 512 | 978.6 tok/s | 91.4% | 1.47 GB | pinned |
+
+**Headline:** **Llama-3.1-8B fine-tunes on a 4 GB card at 119.6 tok/s in 3.32 GB.** For scale, 1M training tokens is ~2.3 h at 8B (arithmetic from the measured rate, not a separate measurement).
+
+The 3B NF4-vs-bf16 rows differ by 1.85×, but attribute that to **pinning, not arithmetic** — see point 2 above. The two rows also come from different sessions, and this card's boost clock varies ~13% between sessions, so treat the factor as indicative and the mechanism as the claim.
+
+Untied `embed_tokens` + `lm_head` stay resident and unquantised (2.10 GB of the 8B row's 3.32 GB), which is why 8B sits close to this card's ceiling; treating them as streamed large layers is deferred beyond v0.72.3.
+
+**Honest scope:**
+- **RAM tier + disk overflow (v0.72.3).** `stream_source: auto` picks RAM when it fits, falls back to NVMe disk when not; SATA/HDD rejected. Correctness verified; disk performance unmeasured on the reference box.
+- **Llama / Qwen / Mistral / Gemma / Gemma2 / Gemma3-Text / Phi / Phi3** (all verified bit-exact in bf16 and NF4), `task: sft`, `backend: transformers`, `modality: text`.
+- **Batch sizes, gradient accumulation, `--resume` / `--hf-resume`** all now work (v0.72.3).
+- **The bf16 3B throughput above is a LOWER BOUND.** The reference box could not page-lock the 5.55 GB base (its measured page-locked ceiling is 7.65 GB, and a CUDA context plus the model skeleton did not leave room), so that run fell back to a pageable store. Pageable memory makes the host-to-device copy synchronous, which costs overlap — visible as the GPU-utilisation drop from 96.8% (1.5B, pinned) to 79.3% (3B, pageable). Soup does this fallback automatically **and prints the cost** rather than absorbing it silently. NF4 lifts this at 3B: the store drops under the ceiling and pins.
 - Numbers are Windows/WDDM and therefore systematically pessimistic versus Linux. `expandable_segments:True` is silently ignored on Windows; Soup detects that and does not claim it is active.
 
+### Sizing a streaming run (v0.72.3)
+
+Streaming bounds the **weights**. It does nothing for activations or for the logits
+tensor, and both scale with `batch × seq`. On a large-vocabulary model that second term
+dominates everything else: measured on Qwen2.5-0.5B (vocab 151 936) at batch 8, S=512,
+the logits alone are **8.71 GB — 146× the entire layer-buffer pool (0.060 GB)**. A
+pre-flight that budgeted only weights and buffers would wave that configuration through.
+
+So `soup train` predicts peak VRAM before building the model, and **refuses a run it
+expects not to fit**:
+
+```
+peak VRAM    ~0.48 GB at batch 2 x seq 256 (logits 0.35 GB)
+free VRAM    3.46 GB
+forecast     5685-8361 tok/s — a compute-bound bound, not a promise
+             (from 6.75 TFLOPS measured on this card now @ 862 MHz)
+```
+
+The prediction was fitted to ten real runs across two models, a 3.1× vocabulary contrast,
+batch 1–8 and two sequence lengths: **worst error 0.85%, and it never under-predicts** —
+the only safe direction for a number allowed to stop a run. The refusal names the two
+knobs that actually scale it (`training.batch_size`, `data.max_length`).
+
+Refusing rather than warning is deliberate. On Linux an over-budget step is a hard OOM.
+On Windows it is worse: WDDM silently spills to host memory and the run merely becomes an
+order of magnitude slower — measured here as a 9.27 GB peak on a 4.29 GB card with **no
+exception raised at all**. Read as "streaming is slow", that would be exactly the wrong
+conclusion.
+
+The throughput line is a **bound, not a promise**. It comes from a bf16 GEMM benchmarked
+on your card in that session and is printed with the SM clock it was taken at, because
+this card alone produced 3.5 and 7.6 TFLOPS in two sessions at the same reported clock. A
+per-card constant compiled into Soup would be a fabrication. Real streamed runs landed at
+68–100% of their measured ceiling.
+
+### Batch size vs gradient accumulation
+
+Both work from v0.72.3, and they are not interchangeable. Measured on Qwen2.5-0.5B bf16,
+S=256, pinned store, 50 steps after 10 warm-up:
+
+| batch | accum | effective batch | throughput | peak VRAM |
+|---|---|---|---|---|
+| 1 | 1 | 1 | 556.6 tok/s | 0.842 GB |
+| 1 | 4 | 4 | 540.1 tok/s | 0.846 GB |
+| 4 | 1 | 4 | **1378.0 tok/s** | 2.28 GB |
+
+Accumulation is **per-token I/O-neutral** — layer reads per 1000 tokens held constant
+across accum 1, 2 and 4, because `accum=N` re-reads the base N times *and* processes N
+times the tokens. Its cost is opportunity cost: at the **same effective batch of 4**,
+raising `batch_size` instead was **2.52× faster**, because one weight read is amortised
+over four times the tokens.
+
+What accumulation buys is effective batch at **constant VRAM** (0.842 → 0.846 GB across
+accum 1→4, where raising batch cost 0.842 → 2.28 GB). So the rule is: **raise
+`batch_size` until the VRAM pre-flight refuses, then accumulate for the rest.** Soup
+prints this advice when it sees you accumulating.
+
 **Rejected at config load (each names the release that lifts it):**
-- `stream_source: disk` → the disk tier is v0.72.3
-- `quantization` other than `none` → NF4 streaming is v0.72.2
+- `batch_size: "auto"` → OOM-probes a resident model that streaming never loads; explicit batch sizes allowed (v0.72.3)
+- `quantization` other than `none` or `4bit` → other formats cannot be streamed into a pooled buffer
 - `backend: unsloth` / `backend: mlx` → streaming replaces the model-load path those backends own
 - `task` other than `sft` → preference losses are v0.72.4
-- `gradient_accumulation_steps > 1` → every micro-batch re-reads the whole base, so accumulation multiplies streaming IO linearly (v0.72.3)
-- `batch_size` other than `1` → v0.72.3
 - `lora.use_dora` / `lora.use_vera` / `lora.init_strategy` other than `random` → these initialise from the real base weight, which is on the meta device under streaming
 - `unfrozen_parameters`, `lisa_enabled`, `packing`, `multipack`, `use_fsdp2_compile`, `train_router_only`, `expand_layers` → each independently rewrites or re-freezes the same layers
 - `stream_source` / `stream_buffers` set while `stream_layers: false` → a footgun, refused
-- an architecture outside `llama` / `qwen2` / `qwen3` → named explicitly, with the allowlist
+- an architecture outside the supported list (llama / qwen2 / qwen3 / mistral / gemma / gemma2 / gemma3_text / phi / phi3) → named explicitly
 
 **Config example:**
 
@@ -294,12 +364,12 @@ data:
 training:
   epochs: 3
   lr: 2e-5
-  batch_size: 1           # required; larger batches in v0.72.3
-  gradient_accumulation_steps: 1   # required; accumulation in v0.72.3
-  quantization: none      # required; NF4 streaming is v0.72.2
+  batch_size: 1           # explicit sizes allowed; "auto" rejected
+  gradient_accumulation_steps: 1   # values > 1 now allowed (v0.72.3)
+  quantization: 4bit      # NF4 — ~4x smaller RAM store than bf16 (or `none`)
   gradient_checkpointing: true     # handled per-layer by the streamer
   stream_layers: true     # Enable layer streaming
-  stream_source: auto     # RAM-based (disk in v0.72.3)
+  stream_source: auto     # RAM with auto-fallback to NVMe disk (v0.72.3)
   stream_buffers: 2       # double-buffering
   lora:
     r: 64
@@ -325,11 +395,10 @@ output: ./output
 **Troubleshooting:**
 - **"layer streaming needs the base to fit in RAM"** — the base is larger than free RAM. Free RAM or pick a smaller base; the disk overflow tier is v0.72.3.
 - **"could not page-lock the base … falling back to a PAGEABLE RAM store"** — expected on a busy machine. Training continues, more slowly. Close other applications to keep the pinned store.
-- **"layer streaming does not support model_type=…"** — v0.72.0 covers Llama and Qwen only.
+- **"layer streaming does not support model_type=…"** — Llama and Qwen only for now; more architectures land in v0.72.3.
 - **Slower than you expected** — layer streaming trades time for memory. If the model already fits resident on your card, do not enable it.
 
 **Roadmap (each refusal names its release):**
-- 4-bit (NF4) streaming — **v0.72.2**, the first genuinely useful capability jump
 - Disk overflow tier, batch size > 1, gradient accumulation, checkpoint/resume, more architectures (Mistral / Gemma / Phi) — **v0.72.3**
 - Preference losses (DPO / ORPO / SimPO / KTO) — **v0.72.4**. GRPO and PPO are explicitly **not** planned: rollouts need generation, which re-reads the model per token
 - A published 14B-on-8 GB reference benchmark — hardware-blocked; it needs an 8 GB card and 32 GB of RAM, which the development box does not have