PyPI - soup-cli - Versions diffs - 0.72.2__tar.gz → 0.72.3__tar.gz - Mend

soup-cli 0.72.2tar.gz → 0.72.3tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (916) hide show

{soup_cli-0.72.2 → soup_cli-0.72.3}/CHANGELOG.md RENAMED Viewed

@@ -12,6 +12,71 @@ reproducing 70+ versions of notes.
 ## [Unreleased]
+## [0.72.3] - 2026-07-28
+**Added — layer streaming breadth: more architectures, bigger batches, resume, and a
+disk tier.**
+Layer streaming (v0.72.0–.2) was deliberately narrow: Llama/Qwen only, batch 1, no
+gradient accumulation, no resume, RAM only. This release lifts all of it, and each
+capability was gated against a streamed-vs-resident bit-exactness reference before any
+of it was written.
+- **Six more model families.** `mistral`, `gemma`, `gemma2`, `gemma3_text`, `phi` and
+  `phi3` join the allowlist, each verified **bit-exact** against the same checkpoint
+  loaded resident, under both bf16 and NF4. Phi-3 is the notable one: it fuses Q/K/V
+  into a single `qkv_proj`, so there is no `q_proj` to find, and it is bit-exact anyway.
+  Multimodal `gemma3` is deliberately *not* accepted — only `gemma3_text`.
+- **`batch_size` above 1, and gradient accumulation.** Both previously refused.
+- **A pre-flight VRAM budget that accounts for batch and vocabulary.** Streaming bounds
+  the *weights*; activations and the logits tensor are untouched by it and both scale
+  with `batch × seq`. On a 152k-vocab model at batch 8 the logits term alone measured
+  **8.71 GB — 146× the entire layer-buffer pool**. `soup train` now predicts peak VRAM
+  and refuses a run that will not fit, naming the two knobs that scale it.
+- **A throughput forecast**, quoted as a range from a GEMM ceiling measured on your own
+  card in that session, alongside the SM clock — never a compiled-in per-card constant.
+- **`--resume` and `--hf-resume` work with streaming.**
+- **A disk overflow tier.** `stream_source: auto` (the default) uses RAM when the base
+  fits and falls back to an NVMe disk tier when it does not, holding nothing resident.
+  `stream_source: ram` refuses instead of falling back. Non-NVMe disks are still
+  refused outright.
+- **`soup doctor --disk`** reports the detected media type.
+**Fixed**
+- `estimate_logits_bytes` charged 6 bytes per logit element; the measured peak is **14**
+  (`transformers` holds the bf16 logits, the fp32 upcast, log-softmax's fp32 output and
+  the fp32 gradient live at once). The old figure under-predicted that term by 2.33×.
+- Adapters could not be loaded *into* a streamed model: `load_state_dict` narrows keys
+  by child name, so a canonical checkpoint matched **0 of N** tensors and PEFT reported
+  only a warning — a resumed run reproduced the from-scratch loss curve exactly. The
+  streaming layer now redirects canonical keys at load time, mirroring the v0.72.1
+  save-side fix.
+- The NVMe-only tier guard was wired to a hardcoded constant and could never fire.
+- Streaming weight sources are now released when training ends **or raises**; the disk
+  tier holds one open shard handle per decoder layer.
+- Subprocess helpers resolve tools to absolute paths (on Windows, `CreateProcess`
+  searches the current directory before `PATH`).
+- The `[mcp]` extra is now capped at `mcp<2`. The SDK's 2.0.0 release removed
+  `mcp.shared.memory.create_connected_server_and_client_session` and dropped
+  `Server.list_tools`, breaking `soup mcp serve`'s round-trip tests for anyone
+  installing fresh. Support for the 2.x API is tracked separately.
+**Known limitations**
+- The **RAM-vs-disk performance gap is unmeasured** on the development hardware and no
+  number is claimed for it. safetensors memory-maps the shards, so the OS page cache
+  keeps them resident between steps on a machine with spare RAM, and at ~5 effective
+  TFLOPS the NVMe read hides under compute. The disk tier's *correctness* is verified
+  bit-exact against the RAM tier; its speed relative to RAM is not characterised.
+- End-to-end `soup train --resume` could not be demonstrated on the development box:
+  `transformers` refuses `torch.load` below torch 2.6 (CVE-2025-32434), which blocks
+  **every** resume there, streaming or not. The streaming-specific half — the adapter
+  round-trip and loss continuity — is verified on the production CUDA path.
+- Loading *into* a streamed model works; `named_parameters()` and `state_dict()` still
+  disagree in memory, which is the deliberate cost of a serialisation-only design.
+- Layer streaming remains **BETA**.
 ## [0.72.2] - 2026-07-28
 **Added — NF4 layer streaming: fine-tune Llama-3.1-8B on a 4 GB laptop GPU.**

{soup_cli-0.72.2 → soup_cli-0.72.3}/CONTRIBUTING.md RENAMED Viewed

@@ -120,7 +120,7 @@ src/soup_cli/
   templates/          - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
-tests/                - Test suite (326 files, 16840 tests)
+tests/                - Test suite (330 files, 16977 tests)
 examples/             - Real-world config examples and datasets
 ```

{soup_cli-0.72.2 → soup_cli-0.72.3}/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.72.2
+Version: 0.72.3
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -29,7 +29,7 @@ Requires-Dist: bitsandbytes>=0.41.0; extra == 'all'
 Requires-Dist: datasets>=2.14.0; extra == 'all'
 Requires-Dist: datasketch>=1.6.0; extra == 'all'
 Requires-Dist: fastapi>=0.104.0; extra == 'all'
-Requires-Dist: mcp>=1.2.0; extra == 'all'
+Requires-Dist: mcp<2,>=1.2.0; extra == 'all'
 Requires-Dist: peft>=0.7.0; extra == 'all'
 Requires-Dist: torch>=2.0.0; extra == 'all'
 Requires-Dist: transformers<5.0.0,>=4.36.0; extra == 'all'
@@ -61,7 +61,7 @@ Requires-Dist: bitsandbytes>=0.41.0; extra == 'dev'
 Requires-Dist: cryptography>=41.0.0; extra == 'dev'
 Requires-Dist: datasets>=2.14.0; extra == 'dev'
 Requires-Dist: httpx>=0.24.0; extra == 'dev'
-Requires-Dist: mcp>=1.2.0; extra == 'dev'
+Requires-Dist: mcp<2,>=1.2.0; extra == 'dev'
 Requires-Dist: mypy>=1.8.0; extra == 'dev'
 Requires-Dist: peft>=0.7.0; extra == 'dev'
 Requires-Dist: pre-commit>=3.5.0; extra == 'dev'
@@ -83,7 +83,7 @@ Requires-Dist: auto-gptq>=0.7.0; extra == 'gptq'
 Provides-Extra: liger
 Requires-Dist: liger-kernel>=0.3.0; extra == 'liger'
 Provides-Extra: mcp
-Requires-Dist: mcp>=1.2.0; extra == 'mcp'
+Requires-Dist: mcp<2,>=1.2.0; extra == 'mcp'
 Provides-Extra: mix
 Requires-Dist: scikit-optimize>=0.9.0; extra == 'mix'
 Provides-Extra: mlx
@@ -193,31 +193,34 @@ infrastructure instead of improving models. Soup fixes that.
 ## What's New
-**v0.72.2 — NF4 layer streaming: fine-tune Llama-3.1-8B on a 4 GB laptop GPU.** Layer
-streaming keeps the frozen base in CPU RAM and feeds it to the GPU one decoder layer at a time.
-Quantising that base to NF4 shrinks it ~4×, which is what puts an 8B model within reach of a
-card that cannot hold even a quarter of it.
-- **Measured on a 4 GB RTX 3050 Laptop** (batch 1, S=512, gradient checkpointing, 50 steps
-  after 10 warm-up): **Llama-3.1-8B-Instruct at 119.6 tok/s, peak VRAM 3.32 GB**, base
-  page-locked at 3.60 GB, GPU 100% busy. Qwen2.5-3B: 264.2 tok/s, 1.76 GB.
-- **Just add `quantization: 4bit`** to a streaming config. The base is quantised once, offline,
-  and cached; the cache re-shards by itself if the checkpoint changes underneath it.
-- **Correctness is not traded away.** A streamed NF4 run is **bit-exact** against a resident
-  NF4 run — same quantised bytes, same bitsandbytes kernels — and that is a CI test, not a
-  one-off measurement.
-- **Why 3B got 1.85× faster too** (264.2 vs 143.1 tok/s in bf16): not arithmetic. A 1.43 GB
-  store page-locks where a 5.55 GB one did not, which restores asynchronous copies and takes
-  GPU utilisation from 79.3% to 100%.
-- Still BETA, and the scope is unchanged: RAM tier, `task: sft`, Llama/Qwen, batch size 1, no
-  gradient accumulation, no `--resume`. Every refusal names the release that lifts it.
+**v0.72.3 — layer streaming grows up: more models, bigger batches, resume, and a disk
+tier.** Layer streaming keeps the frozen base out of VRAM and feeds it to the GPU one
+decoder layer at a time. v0.72.0–.2 kept the scope deliberately tiny to prove it worked;
+this release removes the training wheels.
+- **Six more model families** — Mistral, Gemma / Gemma 2 / Gemma 3, and Phi / Phi-3 — each
+  verified **bit-exact** against the same checkpoint loaded resident, in bf16 *and* NF4.
+- **`batch_size` above 1, gradient accumulation, and `--resume`** all work now.
+- **A pre-flight that predicts peak VRAM and refuses a run that will not fit.** Streaming
+  bounds the *weights*; the logits tensor is not bounded by it and scales with
+  `batch × seq`. On a 152k-vocab model at batch 8 that single tensor measured **8.71 GB —
+  146× the entire layer-buffer pool.** The prediction was fitted to ten real runs and
+  never under-predicts any of them.
+- **A throughput forecast measured on your card, in your session**, quoted as a range
+  next to the SM clock it was taken at — not a number compiled into the source.
+- **A disk overflow tier.** When the base will not fit in RAM, `stream_source: auto`
+  streams it from NVMe instead of refusing. Honest caveat: its *correctness* is verified
+  bit-exact against the RAM tier, but **how much slower it is has not been measured** on
+  the development hardware, and no figure is claimed.
+- Still BETA.
 ```yaml
 # soup.yaml — then just `soup train --config soup.yaml`
 training:
-  stream_layers: true      # base streams from RAM; only the adapter trains
+  stream_layers: true      # base streams out of VRAM; only the adapter trains
   quantization: 4bit       # NF4 — ~4x smaller store, so 8B fits a 4 GB card
-  batch_size: 1
+  batch_size: 4            # v0.72.3: bigger batches amortise the weight read
+  stream_source: auto      # RAM when it fits, NVMe disk when it does not
 ```
 > **Trained with `stream_layers: true` on v0.72.0?** That adapter is inert — its tensors were

{soup_cli-0.72.2 → soup_cli-0.72.3}/README.md RENAMED Viewed

@@ -49,31 +49,34 @@ infrastructure instead of improving models. Soup fixes that.
 ## What's New
-**v0.72.2 — NF4 layer streaming: fine-tune Llama-3.1-8B on a 4 GB laptop GPU.** Layer
-streaming keeps the frozen base in CPU RAM and feeds it to the GPU one decoder layer at a time.
-Quantising that base to NF4 shrinks it ~4×, which is what puts an 8B model within reach of a
-card that cannot hold even a quarter of it.
-- **Measured on a 4 GB RTX 3050 Laptop** (batch 1, S=512, gradient checkpointing, 50 steps
-  after 10 warm-up): **Llama-3.1-8B-Instruct at 119.6 tok/s, peak VRAM 3.32 GB**, base
-  page-locked at 3.60 GB, GPU 100% busy. Qwen2.5-3B: 264.2 tok/s, 1.76 GB.
-- **Just add `quantization: 4bit`** to a streaming config. The base is quantised once, offline,
-  and cached; the cache re-shards by itself if the checkpoint changes underneath it.
-- **Correctness is not traded away.** A streamed NF4 run is **bit-exact** against a resident
-  NF4 run — same quantised bytes, same bitsandbytes kernels — and that is a CI test, not a
-  one-off measurement.
-- **Why 3B got 1.85× faster too** (264.2 vs 143.1 tok/s in bf16): not arithmetic. A 1.43 GB
-  store page-locks where a 5.55 GB one did not, which restores asynchronous copies and takes
-  GPU utilisation from 79.3% to 100%.
-- Still BETA, and the scope is unchanged: RAM tier, `task: sft`, Llama/Qwen, batch size 1, no
-  gradient accumulation, no `--resume`. Every refusal names the release that lifts it.
+**v0.72.3 — layer streaming grows up: more models, bigger batches, resume, and a disk
+tier.** Layer streaming keeps the frozen base out of VRAM and feeds it to the GPU one
+decoder layer at a time. v0.72.0–.2 kept the scope deliberately tiny to prove it worked;
+this release removes the training wheels.
+- **Six more model families** — Mistral, Gemma / Gemma 2 / Gemma 3, and Phi / Phi-3 — each
+  verified **bit-exact** against the same checkpoint loaded resident, in bf16 *and* NF4.
+- **`batch_size` above 1, gradient accumulation, and `--resume`** all work now.
+- **A pre-flight that predicts peak VRAM and refuses a run that will not fit.** Streaming
+  bounds the *weights*; the logits tensor is not bounded by it and scales with
+  `batch × seq`. On a 152k-vocab model at batch 8 that single tensor measured **8.71 GB —
+  146× the entire layer-buffer pool.** The prediction was fitted to ten real runs and
+  never under-predicts any of them.
+- **A throughput forecast measured on your card, in your session**, quoted as a range
+  next to the SM clock it was taken at — not a number compiled into the source.
+- **A disk overflow tier.** When the base will not fit in RAM, `stream_source: auto`
+  streams it from NVMe instead of refusing. Honest caveat: its *correctness* is verified
+  bit-exact against the RAM tier, but **how much slower it is has not been measured** on
+  the development hardware, and no figure is claimed.
+- Still BETA.
 ```yaml
 # soup.yaml — then just `soup train --config soup.yaml`
 training:
-  stream_layers: true      # base streams from RAM; only the adapter trains
+  stream_layers: true      # base streams out of VRAM; only the adapter trains
   quantization: 4bit       # NF4 — ~4x smaller store, so 8B fits a 4 GB card
-  batch_size: 1
+  batch_size: 4            # v0.72.3: bigger batches amortise the weight read
+  stream_source: auto      # RAM when it fits, NVMe disk when it does not
 ```
 > **Trained with `stream_layers: true` on v0.72.0?** That adapter is inert — its tensors were

{soup_cli-0.72.2 → soup_cli-0.72.3}/docs/commands.md RENAMED Viewed

@@ -30,7 +30,7 @@ soup merge-sharded-fsdp-weights ./shards -o merged.safetensors  Consolidate FSDP
 soup delinearize-llama4 ./src --target ./out [--num-experts N] [--plan-only]  Live Llama-4 fused-expert reshape [E*din,dout] -> [E,din,dout] + sidecar copy (v0.71.21)
 soup spectrum scan --model <id|path> --top-percent 50 [--modules mlp,attn] [-o patch.yaml]  Spectrum SNR scan (no model load) -> training.unfrozen_parameters YAML patch (v0.71.23)
 soup train --config sft.yaml  # training.lisa_enabled: true [lisa_num_layers lisa_interval_steps]  LISA layerwise importance sampling — full-FT quality at LoRA-like memory (sft/transformers/text/quantization=none) (v0.71.34)
-soup train --config sft.yaml  # training.stream_layers: true [stream_source stream_buffers]  BETA layer streaming — the frozen base streams from CPU RAM one decoder layer at a time, so peak VRAM is bounded by ONE layer; quantization: 4bit streams it as NF4, ~4x smaller (sft/transformers/text, batch 1) (v0.72.0; NF4 v0.72.2)
+soup train --config sft.yaml  # training.stream_layers: true [stream_source stream_buffers]  BETA layer streaming — the frozen base streams from CPU RAM/NVMe one decoder layer at a time, so peak VRAM is bounded by ONE layer; quantization: 4bit streams it as NF4, ~4x smaller (sft/transformers/text, wider archs) (v0.72.0; NF4 v0.72.2; disk+batch+accum v0.72.3)
 soup export --model ./output --format gguf    Export to GGUF (Ollama)
 soup export --model ./output --deploy ollama  Export GGUF + auto-deploy to Ollama
 soup export --model ./output --format onnx    Export to ONNX
@@ -200,7 +200,7 @@ soup tokenizer train --input c.jsonl --vocab-size N  Train BPE tokenizer (v0.53.
 soup bench <model> --p50 --p95                Bench with tail-latency percentiles (v0.53.9)
 soup bench <model> --backend auto             Auto-detect transformers/mlx backend (v0.53.9)
 soup serve --reasoning-parser deepseek-r1     Strip <think> blocks from responses (v0.53.9)
-soup doctor [--nccl]                          Check environment (optionally check NCCL bandwidth)
+soup doctor [--nccl] [--disk]                 Check environment (optionally check NCCL bandwidth, media type; --disk ~9s cold / ~2.4s warm)
 soup quickstart [--dry-run]                   Full demo
 soup adapters scan <adapter>                  Spectral backdoor scan (rank-1 dominance + outlier detection)
 soup adapters sign <adapter> [--backend unsigned|ed25519] [--key <pem>|--generate-key <pem>]  Merkle manifest + ed25519 sign

{soup_cli-0.72.2 → soup_cli-0.72.3}/docs/performance-and-quantization.md RENAMED Viewed

@@ -14,7 +14,7 @@
 - [Cross-Document Attention Masking](#cross-document-attention-masking)
 - [Quant Menu — 9 Quantization Formats](#quant-menu--9-quantization-formats)
 - [Activation Offloading (Small-VRAM Large-Batch)](#activation-offloading-small-vram-large-batch)
-- [Layer Streaming (BETA, v0.72.0; NF4 v0.72.2)](#layer-streaming-beta-v0720-nf4-v0722)
+- [Layer Streaming (BETA, v0.72.0; NF4 v0.72.2; disk + wider archs v0.72.3)](#layer-streaming-beta-v0720-nf4-v0722-disk--wider-archs-v0723)
 - [Correctness First (v0.36.0)](#correctness-first-v0360)
 - [Multi-GPU / DeepSpeed / FSDP](#multi-gpu--deepspeed--fsdp)
 - [Performance + Long-Context](#performance--long-context)
@@ -24,7 +24,7 @@
 - [MoE Expert Quantization + Router-Only Training (live in v0.71.20)](#moe-expert-quantization--router-only-training-live-in-v07120)
 - [Unsloth Dynamic 2.0 GGUF Ladder (v0.53.0)](#unsloth-dynamic-20-gguf-ladder-v0530)
 - [KV Cache Types (v0.53.0)](#kv-cache-types-v0530)
-- [FP8 Attention + NVFP4 + Native `unsloth_bnb_4bit` (v0.53.0)](#fp8-attention--nvfp4--native-unsloth_bnb_4bit-v0530)
+- [FP8 Attention + NVFP4 + Native `unsloth_bnb_4bit` (v0.53.0)](#fp8-attention--nvfp4--native-unsloth_bnb_4bit)
 - [LF / Axolotl Quant Parity (v0.53.0)](#lf--axolotl-quant-parity-v0530)
 - [Advanced Save Formats (v0.53.0)](#advanced-save-formats-v0530)
 - [Quant Menu II + Export Pipeline (v0.53.1)](#quant-menu-ii--export-pipeline-v0531)
@@ -226,7 +226,7 @@ training:
 Not compatible with unsloth (own memory manager) or mlx. Wired across every transformer-backend trainer (SFT, DPO, GRPO, KTO, ORPO, SimPO, IPO, PPO, Reward-Model, Embedding, Pretrain).
-## Layer Streaming (BETA, v0.72.0; NF4 v0.72.2)
+## Layer Streaming (BETA, v0.72.0; NF4 v0.72.2; disk + wider archs v0.72.3)
 Stream frozen base-model decoder layers ONE at a time from CPU RAM into small VRAM buffers instead of keeping the whole base resident. Peak VRAM is bounded by the size of a single layer, not the entire model — so models that don't fit resident on your GPU can now train at all.
@@ -272,26 +272,81 @@ Correctness is not a tradeoff here either: a streamed NF4 run is **bit-exact** a
 The 3B NF4-vs-bf16 rows differ by 1.85×, but attribute that to **pinning, not arithmetic** — see point 2 above. The two rows also come from different sessions, and this card's boost clock varies ~13% between sessions, so treat the factor as indicative and the mechanism as the claim.
-Untied `embed_tokens` + `lm_head` stay resident and unquantised (2.10 GB of the 8B row's 3.32 GB), which is why 8B sits close to this card's ceiling; treating them as streamed large layers is a v0.72.3 item.
+Untied `embed_tokens` + `lm_head` stay resident and unquantised (2.10 GB of the 8B row's 3.32 GB), which is why 8B sits close to this card's ceiling; treating them as streamed large layers is deferred beyond v0.72.3.
 **Honest scope:**
-- **RAM tier only.** `stream_source: ram` (`auto` resolves to it). The disk overflow tier ships in **v0.72.3**.
-- **Llama / Qwen only**, `task: sft`, `backend: transformers`, `modality: text`.
-- **`batch_size: 1`**, no gradient accumulation, no `--resume` — all ship in **v0.72.3**.
+- **RAM tier + disk overflow (v0.72.3).** `stream_source: auto` picks RAM when it fits, falls back to NVMe disk when not; SATA/HDD rejected. Correctness verified; disk performance unmeasured on the reference box.
+- **Llama / Qwen / Mistral / Gemma / Gemma2 / Gemma3-Text / Phi / Phi3** (all verified bit-exact in bf16 and NF4), `task: sft`, `backend: transformers`, `modality: text`.
+- **Batch sizes, gradient accumulation, `--resume` / `--hf-resume`** all now work (v0.72.3).
 - **The bf16 3B throughput above is a LOWER BOUND.** The reference box could not page-lock the 5.55 GB base (its measured page-locked ceiling is 7.65 GB, and a CUDA context plus the model skeleton did not leave room), so that run fell back to a pageable store. Pageable memory makes the host-to-device copy synchronous, which costs overlap — visible as the GPU-utilisation drop from 96.8% (1.5B, pinned) to 79.3% (3B, pageable). Soup does this fallback automatically **and prints the cost** rather than absorbing it silently. NF4 lifts this at 3B: the store drops under the ceiling and pins.
 - Numbers are Windows/WDDM and therefore systematically pessimistic versus Linux. `expandable_segments:True` is silently ignored on Windows; Soup detects that and does not claim it is active.
+### Sizing a streaming run (v0.72.3)
+Streaming bounds the **weights**. It does nothing for activations or for the logits
+tensor, and both scale with `batch × seq`. On a large-vocabulary model that second term
+dominates everything else: measured on Qwen2.5-0.5B (vocab 151 936) at batch 8, S=512,
+the logits alone are **8.71 GB — 146× the entire layer-buffer pool (0.060 GB)**. A
+pre-flight that budgeted only weights and buffers would wave that configuration through.
+So `soup train` predicts peak VRAM before building the model, and **refuses a run it
+expects not to fit**:
+```
+peak VRAM    ~0.48 GB at batch 2 x seq 256 (logits 0.35 GB)
+free VRAM    3.46 GB
+forecast     5685-8361 tok/s — a compute-bound bound, not a promise
+             (from 6.75 TFLOPS measured on this card now @ 862 MHz)
+```
+The prediction was fitted to ten real runs across two models, a 3.1× vocabulary contrast,
+batch 1–8 and two sequence lengths: **worst error 0.85%, and it never under-predicts** —
+the only safe direction for a number allowed to stop a run. The refusal names the two
+knobs that actually scale it (`training.batch_size`, `data.max_length`).
+Refusing rather than warning is deliberate. On Linux an over-budget step is a hard OOM.
+On Windows it is worse: WDDM silently spills to host memory and the run merely becomes an
+order of magnitude slower — measured here as a 9.27 GB peak on a 4.29 GB card with **no
+exception raised at all**. Read as "streaming is slow", that would be exactly the wrong
+conclusion.
+The throughput line is a **bound, not a promise**. It comes from a bf16 GEMM benchmarked
+on your card in that session and is printed with the SM clock it was taken at, because
+this card alone produced 3.5 and 7.6 TFLOPS in two sessions at the same reported clock. A
+per-card constant compiled into Soup would be a fabrication. Real streamed runs landed at
+68–100% of their measured ceiling.
+### Batch size vs gradient accumulation
+Both work from v0.72.3, and they are not interchangeable. Measured on Qwen2.5-0.5B bf16,
+S=256, pinned store, 50 steps after 10 warm-up:
+| batch | accum | effective batch | throughput | peak VRAM |
+|---|---|---|---|---|
+| 1 | 1 | 1 | 556.6 tok/s | 0.842 GB |
+| 1 | 4 | 4 | 540.1 tok/s | 0.846 GB |
+| 4 | 1 | 4 | **1378.0 tok/s** | 2.28 GB |
+Accumulation is **per-token I/O-neutral** — layer reads per 1000 tokens held constant
+across accum 1, 2 and 4, because `accum=N` re-reads the base N times *and* processes N
+times the tokens. Its cost is opportunity cost: at the **same effective batch of 4**,
+raising `batch_size` instead was **2.52× faster**, because one weight read is amortised
+over four times the tokens.
+What accumulation buys is effective batch at **constant VRAM** (0.842 → 0.846 GB across
+accum 1→4, where raising batch cost 0.842 → 2.28 GB). So the rule is: **raise
+`batch_size` until the VRAM pre-flight refuses, then accumulate for the rest.** Soup
+prints this advice when it sees you accumulating.
 **Rejected at config load (each names the release that lifts it):**
-- `stream_source: disk` → the disk tier is v0.72.3
+- `batch_size: "auto"` → OOM-probes a resident model that streaming never loads; explicit batch sizes allowed (v0.72.3)
 - `quantization` other than `none` or `4bit` → other formats cannot be streamed into a pooled buffer
 - `backend: unsloth` / `backend: mlx` → streaming replaces the model-load path those backends own
 - `task` other than `sft` → preference losses are v0.72.4
-- `gradient_accumulation_steps > 1` → every micro-batch re-reads the whole base, so accumulation multiplies streaming IO linearly (v0.72.3)
-- `batch_size` other than `1` → v0.72.3
 - `lora.use_dora` / `lora.use_vera` / `lora.init_strategy` other than `random` → these initialise from the real base weight, which is on the meta device under streaming
 - `unfrozen_parameters`, `lisa_enabled`, `packing`, `multipack`, `use_fsdp2_compile`, `train_router_only`, `expand_layers` → each independently rewrites or re-freezes the same layers
 - `stream_source` / `stream_buffers` set while `stream_layers: false` → a footgun, refused
-- an architecture outside `llama` / `qwen2` / `qwen3` → named explicitly, with the allowlist
+- an architecture outside the supported list (llama / qwen2 / qwen3 / mistral / gemma / gemma2 / gemma3_text / phi / phi3) → named explicitly
 **Config example:**
@@ -309,12 +364,12 @@ data:
 training:
   epochs: 3
   lr: 2e-5
-  batch_size: 1           # required; larger batches in v0.72.3
-  gradient_accumulation_steps: 1   # required; accumulation in v0.72.3
+  batch_size: 1           # explicit sizes allowed; "auto" rejected
+  gradient_accumulation_steps: 1   # values > 1 now allowed (v0.72.3)
   quantization: 4bit      # NF4 — ~4x smaller RAM store than bf16 (or `none`)
   gradient_checkpointing: true     # handled per-layer by the streamer
   stream_layers: true     # Enable layer streaming
-  stream_source: auto     # RAM-based (disk in v0.72.3)
+  stream_source: auto     # RAM with auto-fallback to NVMe disk (v0.72.3)
   stream_buffers: 2       # double-buffering
   lora:
     r: 64

{soup_cli-0.72.2 → soup_cli-0.72.3}/docs/training.md RENAMED Viewed

@@ -5,9 +5,10 @@
 > SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, and the loop-hardening detectors.
 > **Training a model bigger than your GPU?** `training.stream_layers: true` streams the
-> frozen base from CPU RAM one decoder layer at a time, so peak VRAM is bounded by one
-> layer instead of the whole model —
-> see [Layer Streaming](performance-and-quantization.md#layer-streaming-beta-v0720).
+> frozen base from CPU RAM (with NVMe disk overflow in v0.72.3) one decoder layer at a time,
+> so peak VRAM is bounded by one layer instead of the whole model. Add `quantization: 4bit`
+> and an 8B base fits a 4 GB card — see
+> [Layer Streaming](performance-and-quantization.md#layer-streaming-beta-v0720-nf4-v0722-disk--wider-archs-v0723).
 **Contents:**

{soup_cli-0.72.2 → soup_cli-0.72.3}/pyproject.toml RENAMED Viewed

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 [project]
 name = "soup-cli"
-version = "0.72.2"
+version = "0.72.3"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"
@@ -105,7 +105,14 @@ modal = ["modal>=0.60.0"]
 # v0.71.28 - `soup mcp serve` MCP server. The official `mcp` python SDK is
 # lazy-imported (only src/soup_cli/mcp_server/server.py touches it), so the CLI
 # stays light without it. Floor pinned to guard against SDK API churn.
-mcp = ["mcp>=1.2.0"]
+#
+# v0.72.3: capped below 2.0. The floor alone was not enough — mcp 2.0.0 removed
+# `mcp.shared.memory.create_connected_server_and_client_session` and dropped
+# `Server.list_tools`, which broke every `soup mcp serve` round-trip test on all
+# nine CI jobs the day it was published. 1.29.0 (the newest 1.x) still exposes
+# both, verified directly. Migrating to the 2.x API is its own piece of work,
+# and shipping against a major nobody has validated is not a substitute for it.
+mcp = ["mcp>=1.2.0,<2"]
 [project.scripts]
 soup = "soup_cli.cli:run"

{soup_cli-0.72.2 → soup_cli-0.72.3}/src/soup_cli/__init__.py RENAMED Viewed

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
-__version__ = "0.72.2"
+__version__ = "0.72.3"

{soup_cli-0.72.2 → soup_cli-0.72.3}/src/soup_cli/commands/doctor.py RENAMED Viewed

@@ -56,6 +56,15 @@ def doctor(
     nccl: bool = typer.Option(
         False, "--nccl", help="Measure NCCL bandwidth and check against reference table."
     ),
+    disk: bool = typer.Option(
+        False,
+        "--disk",
+        help=(
+            "Detect the disk media type (NVMe / SSD / HDD). Layer streaming's "
+            "disk overflow tier needs NVMe. Off by default: the probe costs ~9 s "
+            "on Windows."
+        ),
+    ),
 ):
     """Check system dependencies, GPU, and compatibility."""
     console.print("[bold]Soup Doctor[/] - checking your environment...\n")
@@ -75,7 +84,7 @@ def doctor(
     _check_gpu()
     # Resources check
-    _check_resources()
+    _check_resources(probe_disk=disk)
     # Dependencies table
     table = Table(title="Dependencies")
@@ -367,7 +376,7 @@ def _get_ram_gb() -> str:
     return "Unknown"
-def _check_resources():
+def _check_resources(probe_disk: bool = False):
     """Check RAM and Disk space and display info."""
     import shutil
@@ -384,6 +393,32 @@ def _check_resources():
         disk_str = "Unknown"
     table.add_row("Disk", disk_str)
+    # v0.72.3 — layer streaming's disk overflow tier needs NVMe: on a spinning
+    # disk each step costs 2 seeks per layer (plan P11), which does not merely
+    # run slower, it thrashes. Reported here so the refusal is not the first
+    # time an operator hears about it.
+    #
+    # Opt-in behind --disk, matching this command's own --nccl convention for
+    # expensive probes. Measured on the dev box: the Windows PowerShell query
+    # costs ~9 s cold and ~2.4 s warm (17.6 s -> 20.0 s on `soup doctor`), paid
+    # by every user including the majority who never touch layer streaming. A
+    # streaming run does its own lazy probe when the tier decision actually
+    # depends on the answer, so nothing is lost by defaulting this off.
+    if probe_disk:
+        try:
+            from soup_cli.utils.layer_stream import detect_disk_kind
+            kind = detect_disk_kind(".")
+        except Exception:  # noqa: BLE001 — a diagnostic must never crash the report
+            kind = "unknown"
+        verdict = {
+            "nvme": "[green]NVMe[/] — layer streaming can use the disk overflow tier",
+            "ssd": "[yellow]SATA SSD[/] — layer streaming refuses the disk tier (RAM only)",
+            "hdd": "[red]HDD[/] — layer streaming refuses the disk tier (RAM only)",
+        }.get(kind, "[yellow]Unknown[/] — layer streaming will refuse the disk tier")
+        table.add_row("Disk type", verdict)
     console.print(table)
     console.print()

{soup_cli-0.72.2 → soup_cli-0.72.3}/src/soup_cli/commands/train.py RENAMED Viewed

@@ -530,23 +530,13 @@ def train(
         console.print(f"[red]{markup_escape(str(exc))}[/]")
         raise typer.Exit(code=2) from exc
-    # --- v0.72.0 BETA — layer streaming does not support resume yet ---
-    # Silently ignoring a resume would restart from scratch and look like it
-    # worked, so refuse before any heavy work happens.
-    #
-    # v0.72.1: --hf-resume must be covered too. It sets resume_from through a
-    # different branch (see prepare_hf_resume below), so a guard on --resume
-    # alone let it through. That mattered more after the adapter-key fix: the
-    # pushed checkpoint now carries canonical keys while a live streamed model
-    # is still `.inner.`-shaped, so PEFT's strict=False load would match
-    # NOTHING and silently continue with a freshly initialised adapter.
-    if (resume or hf_resume) and cfg.training.stream_layers:
-        console.print(
-            "[red]--resume / --hf-resume are not supported with "
-            "training.stream_layers[/] — checkpoint/resume for layer streaming "
-            "lands in v0.72.3. Drop the flag, or set stream_layers: false."
-        )
-        raise typer.Exit(code=2)
+    # v0.72.3 — --resume / --hf-resume now work with layer streaming. v0.72.0-.2
+    # refused them because a streamed model's `named_parameters()` carry an
+    # `.inner.` segment that `load_state_dict` narrows away, so PEFT matched
+    # NOTHING and silently continued with a freshly initialised adapter (measured:
+    # 0 of 12 tensors, and a resumed loss curve byte-identical to a from-scratch
+    # one). `StreamedDecoderLayer` now redirects canonical keys at load time,
+    # mirroring the v0.72.1 save-side delegation.
     # --- RA-DIT generator-stage auto-link (v0.71.10 #200) ---
     # When a generator stage has no retriever model set, splice in the latest

{soup_cli-0.72.2 → soup_cli-0.72.3}/src/soup_cli/config/schema.py RENAMED Viewed

@@ -4682,23 +4682,30 @@ class SoupConfig(BaseModel):
                 f"store weights in formats that cannot be streamed into a "
                 f"pooled buffer."
             )
-        if tcfg.stream_source == "disk":
-            raise ValueError(
-                "training.stream_source='disk' is not implemented yet — "
-                "the disk overflow tier lands in v0.72.3. Use 'ram' or 'auto'."
-            )
-        if tcfg.batch_size != 1:
-            raise ValueError(
-                f"training.stream_layers requires batch_size=1 (got "
-                f"{tcfg.batch_size!r}); larger batches land in v0.72.3."
-            )
-        if tcfg.gradient_accumulation_steps != 1:
-            raise ValueError(
-                f"training.stream_layers requires gradient_accumulation_steps=1 "
-                f"(got {tcfg.gradient_accumulation_steps}); every micro-batch "
-                f"re-reads the entire base, so accumulation multiplies streaming "
-                f"IO linearly. Accumulation lands in v0.72.3."
-            )
+        # v0.72.3: the disk overflow tier is live. It is still NVMe-only —
+        # `choose_tier` refuses spinning disks, where each step costs two seeks
+        # per layer (plan P11) and the run thrashes rather than merely running
+        # slower. `soup doctor` reports the detected media type.
+        # v0.72.3: any concrete batch size is supported — bigger batches are
+        # where streaming PAYS OFF, since one weight read is amortised over more
+        # tokens. "auto" is still refused: it resolves by OOM-probing a resident
+        # model, and under streaming that model never loads, so the probe would
+        # measure something that does not exist.
+        if tcfg.batch_size == "auto":
+            raise ValueError(
+                "training.stream_layers cannot use batch_size='auto': the probe "
+                "sizes a RESIDENT model, which a streaming run never loads. Set "
+                "a concrete batch_size — the pre-flight predicts peak VRAM for "
+                "it and refuses the run if it will not fit."
+            )
+        # v0.72.3: accumulation is supported. Measured on this box, it is
+        # per-TOKEN I/O-neutral — layer reads per 1k tokens held constant across
+        # accum 1/2/4 — because N micro-batches read the base N times but also
+        # process N times the tokens. Its cost is opportunity cost: reaching the
+        # same effective batch by raising batch_size was 2.52x faster. The
+        # pre-flight says so rather than the schema refusing it, because
+        # accumulation is the ONLY way to raise effective batch once the VRAM
+        # budget is exhausted (peak moved 0.842 -> 0.846 GB across accum 1->4).
         if tcfg.lora.r < 1:
             raise ValueError(
                 "training.stream_layers requires LoRA (training.lora.r >= 1) — "

soup-cli 0.72.2__tar.gz → 0.72.3__tar.gz

soup-cli 0.72.2tar.gz → 0.72.3tar.gz