streamft 0.2.0__tar.gz → 0.2.2__tar.gz

streamft-0.2.2/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

{streamft-0.2.0/streamft.egg-info → streamft-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: streamft
-Version: 0.2.0
+Version: 0.2.2
 Summary: Fine-tune models that do not fit: stream NF4 weights from disk, one transformer block at a time.
 Author-email: Gokhan Cava <byjameson@users.noreply.github.com>
 License: Apache-2.0
@@ -30,6 +30,8 @@ Dynamic: license-file
 
 # stream-finetune — fine-tune a 32B model on a 6 GB laptop GPU
 
+[![PyPI](https://img.shields.io/pypi/v/streamft)](https://pypi.org/project/streamft/) [![try it](https://img.shields.io/badge/try-will%20it%20fit%3F-6ee7b7)](https://huggingface.co/spaces/byjameson/will-it-fit)
+
 A small training engine that fine-tunes models that **do not fit in VRAM**. The NF4
 weights are mmap'd from disk, one transformer block is materialised on the GPU at a
 time, and the backward recomputes through it. The base stays frozen; a LoRA adapter is
@@ -53,7 +55,8 @@ The 32B weights are **19.2 GB**. They never fully live in VRAM. At 1024 tokens/s
 throughput is **3.1×** the 128-token rate, because the per-step I/O is amortised over
 more tokens. 50 steps of Dolly-15k SFT moved eval loss **1.8755 → 0.9661**.
 
-**Start with [`GUIDE_6GB_32B.md`](GUIDE_6GB_32B.md)** — the practical write-up: presets,
+**Installed from PyPI? Start with [`QUICKSTART.md`](QUICKSTART.md)** — it uses only the
+published CLI (no repo scripts). Deeper dive: [`GUIDE_6GB_32B.md`](GUIDE_6GB_32B.md) — the practical write-up: presets,
 the six traps, and which numbers are real. The measurement protocol, the discarded rows
 and the correction of an earlier wrong claim ("8.6× / 100 tok/s") are in
 [`LONGCTX_FINDINGS.md`](LONGCTX_FINDINGS.md).
@@ -61,7 +64,7 @@ and the correction of an earlier wrong claim ("8.6× / 100 tok/s") are in
 ## Quick start
 
 ```powershell
-pip install -e .            # or: pip install streamft (once it is on PyPI)
+pip install streamft
 
 # 0) will it fit? memory arithmetic + the measured reference, no weights downloaded
 streamft fit --model Qwen/Qwen2.5-14B-Instruct --gpu 6 --seq 512
@@ -142,30 +145,33 @@ Raw output: `results/gradcheck.json` (git-ignored; the numbers above are copied
 
 ## Compared with PEFT (matched runs)
 
-Same data, same LoRA (r=8, α=16, q/k/v/o), same AdamW lr 1e-4, batch 1, same prompt
+Same data, same LoRA (r=8, alpha=16, q/k/v/o), same AdamW lr 1e-4, batch 1, same prompt
 masking, same token-weighted masked eval on 8 held-out rows. PEFT uses
 `prepare_model_for_kbit_training` with gradient checkpointing (the standard QLoRA setup).
+Step times move by up to ~30% between runs on this laptop; quality is the stable number.
 
 **3B (fits entirely):**
 
 | | streamft (36 blocks resident) | PEFT / transformers |
 |---|---:|---:|
-| step time, median of 30 @ seq 512 | 0.938 s | **0.579 s** |
+| step time, median of 30 @ seq 512 | 0.94–1.24 s | **0.58–0.88 s** |
 | peak VRAM | **3.85 GB** | 4.90 GB |
 | eval before | 1.42087 | 1.42087 |
-| eval after | **1.30154** (−0.1193) | 1.32722 (−0.0936) |
+| eval after | **1.30720** (−0.1137) | 1.33002 (−0.0909) |
 
-**7B and 14B (do not fit comfortably):**
+**7B / 14B / 32B (do not fit comfortably), after the masked-gradient fix:**
 
-| model | step time, median @ seq 512 | peak VRAM | eval (start → after) |
+| model | step time, median @ seq 512 | peak VRAM | eval (start → after, 30 steps) |
 |---|---:|---:|---:|
-| 7B, streamft (28 blocks) | **1.70 s** | **5.51 GB** | 2.1928 → 1.1094 |
+| 7B, streamft (28 blocks) | **1.55 s** | **5.51 GB** | 2.1928 → **1.1055** |
 | 7B, PEFT | 8.96 s | 10.05 GB | 2.1914 → 1.1491 |
-| 14B, streamft (6 blocks) | **6.73 s** | **3.86 GB** | 2.3748 → 1.5312 (10 steps) |
-| 14B, PEFT | 18.56 s | 16.48 GB | 2.3748 (3 steps only) |
+| 14B, streamft (6 blocks) | **5.86 s** | **3.86 GB** | 2.3748 → **0.9742** |
+| 14B, PEFT | 18.56 s | 16.48 GB | 3 steps only |
+| 32B, streamft (2 blocks, seq 1024) | **12.48 s** | **3.90 GB** | 1.8755 → **1.0008** |
+| 32B, PEFT | fails before step 1 | — | — |
 
 At 7B the crossover has already happened: PEFT's 4.4 GB of weights plus activations no
-longer fit, so it spills (10.05 GB reserved) and pays 5.3× the step time. Note that
+longer fit, so it spills (10.05 GB reserved) and pays ~5.8× the step time. Note that
 streamft's 7B peak (5.51 GB) is *higher* than its 14B peak (3.86 GB) because the 7B run
 kept all 28 blocks resident while the 14B run kept 6 — the window is a choice, and the
 14B number is the one to copy for a 6 GB card.
@@ -174,16 +180,58 @@ kept all 28 blocks resident while the 14B run kept 6 — the window is a choice,
 into shared memory (19.45 GB reserved on a 6 GiB card), and `prepare_model_for_kbit_training`
 then dies allocating the fp32 norm copy: *CUDA out of memory. Tried to allocate 2.90 GiB;
 18.00 GiB is already allocated.* The streaming engine runs the same checkpoint at
-**12.5 s/step and 3.90 GB** (64/64 layers, held-out 1.8755 to 0.9959).
+**12.5 s/step and 3.90 GB** (64/64 layers).
 
 The 3B row is the honest answer to "is this faster?": **no** — on a model that fits, PEFT
-is 1.6× faster and uses 1.05 GB more. The 14B row is the design point: PEFT does not
-OOM on Windows, it **spills into shared system memory** (16.48 GB reserved on a 6.4 GB
-card) and pays 2.8× the step time; streamft stays at 3.86 GB. Over 30 steps and one seed
-the 3B quality difference is not meaningful.
+is faster and uses 1.05 GB more. The 14B/32B rows are the design point: PEFT does not
+OOM on Windows, it **spills into shared system memory** and pays 2.8–5.8× the step time.
+
+### Correction: masked gradient (25 Oct, after an external review)
+
+`chunked_lm_loss_and_backward_masked` divided every chunk by **that chunk's** valid-token
+count. The reported loss was still the correct mean, so the curves looked right, but the
+accumulated gradient over-weighted chunks with few unmasked tokens — i.e. the prompt-heavy
+start of an SFT row. Measured on a toy model with an uneven mask: **112–427% relative
+gradient error** at seq 512/1024 with 60% of the row masked (chunk 128), while balanced
+masks were unaffected.
+
+The fix divides every chunk by the **global** valid count (computed once, no-grad), so the
+gradient is exactly d(mean CE over valid tokens)/dh. Two tests pin it:
+`test_masked_chunked_gradient_matches_full_with_uneven_masks` and
+`test_masked_chunked_handles_fully_masked_chunks`.
+
+All SFT runs above were re-run with the fix. Quality held: 3B −0.114 (was −0.119),
+7B −1.087 (was −1.083), 32B −0.875 (was −0.880); 14B is now 30 steps (−1.40) instead of
+10. The numbers in the tables are the post-fix ones.
 
 Reproduce: `python scripts/compare_peft.py --model checkpoints/qwen3b_bnb4bit --steps 30`
-and `... --model checkpoints/qwen14b_nf4 --steps 3`.
+and `... --model checkpoints/qwen14b_nf4 --steps 30`.
+
+### Reproducibility probe
+
+Run this and compare the row; it is deterministic (seed 1234) apart from GPU-kernel
+non-determinism:
+
+```bash
+streamft train --model-dir checkpoints/qwen3b_bnb4bit --steps 3 --seq 128 \
+  --eval_chunks 1 --tag repro
+```
+
+Reference on the only machine tested so far (RTX 3050 Laptop 6 GB, torch 2.13.0+cu126,
+bitsandbytes 0.50.2, Windows 11):
+
+```text
+losses        3.347981  3.647039  3.049560
+ce_before     2.44565
+ce_after      2.44568
+peak_gb_max   3.068
+adapter_sha_after  422424c59ef004c5
+```
+
+Different hardware will differ in the last digits (kernels, bf16 rounding). What must
+match is the shape: the same three losses to ~1e-3, `grad_layers == n_layers`, and a
+changed adapter hash. If it does not, that is a bug report worth posting with the
+`streamft doctor` output.
 
 ## What this does not do
 
@@ -233,6 +281,7 @@ URLs. The engine lived at `experiments/exp54_disk_qlora.py` + `lcsa/` until the
 - [Running a 4B model in 6 GB and training 3B parameters on a laptop](https://dev.to/mycute_97b39c4e34027570f9/running-a-4b-model-in-6-gb-and-training-3b-parameters-on-a-laptop-4al)
 - [The same 32B post on Medium](https://medium.com/@myredditacc9393/fine-tuning-a-32b-model-on-a-6-gb-laptop-gpu-e749df447229)
 - [Will it fit? A free planner for fine-tuning models that do not fit](https://dev.to/mycute_97b39c4e34027570f9/will-it-fit-a-free-planner-for-fine-tuning-models-that-do-not-fit-on-your-gpu-o50)
+- [Will it fit? on Medium](https://medium.com/@myredditacc9393/will-it-fit-a-free-check-before-you-download-19-gb-of-weights-9097e1fcf93f)
 
 ## License
 

{streamft-0.2.0 → streamft-0.2.2}/README.md

@@ -1,5 +1,7 @@
 # stream-finetune — fine-tune a 32B model on a 6 GB laptop GPU
 
+[![PyPI](https://img.shields.io/pypi/v/streamft)](https://pypi.org/project/streamft/) [![try it](https://img.shields.io/badge/try-will%20it%20fit%3F-6ee7b7)](https://huggingface.co/spaces/byjameson/will-it-fit)
+
 A small training engine that fine-tunes models that **do not fit in VRAM**. The NF4
 weights are mmap'd from disk, one transformer block is materialised on the GPU at a
 time, and the backward recomputes through it. The base stays frozen; a LoRA adapter is
@@ -23,7 +25,8 @@ The 32B weights are **19.2 GB**. They never fully live in VRAM. At 1024 tokens/s
 throughput is **3.1×** the 128-token rate, because the per-step I/O is amortised over
 more tokens. 50 steps of Dolly-15k SFT moved eval loss **1.8755 → 0.9661**.
 
-**Start with [`GUIDE_6GB_32B.md`](GUIDE_6GB_32B.md)** — the practical write-up: presets,
+**Installed from PyPI? Start with [`QUICKSTART.md`](QUICKSTART.md)** — it uses only the
+published CLI (no repo scripts). Deeper dive: [`GUIDE_6GB_32B.md`](GUIDE_6GB_32B.md) — the practical write-up: presets,
 the six traps, and which numbers are real. The measurement protocol, the discarded rows
 and the correction of an earlier wrong claim ("8.6× / 100 tok/s") are in
 [`LONGCTX_FINDINGS.md`](LONGCTX_FINDINGS.md).
@@ -31,7 +34,7 @@ and the correction of an earlier wrong claim ("8.6× / 100 tok/s") are in
 ## Quick start
 
 ```powershell
-pip install -e .            # or: pip install streamft (once it is on PyPI)
+pip install streamft
 
 # 0) will it fit? memory arithmetic + the measured reference, no weights downloaded
 streamft fit --model Qwen/Qwen2.5-14B-Instruct --gpu 6 --seq 512
@@ -112,30 +115,33 @@ Raw output: `results/gradcheck.json` (git-ignored; the numbers above are copied
 
 ## Compared with PEFT (matched runs)
 
-Same data, same LoRA (r=8, α=16, q/k/v/o), same AdamW lr 1e-4, batch 1, same prompt
+Same data, same LoRA (r=8, alpha=16, q/k/v/o), same AdamW lr 1e-4, batch 1, same prompt
 masking, same token-weighted masked eval on 8 held-out rows. PEFT uses
 `prepare_model_for_kbit_training` with gradient checkpointing (the standard QLoRA setup).
+Step times move by up to ~30% between runs on this laptop; quality is the stable number.
 
 **3B (fits entirely):**
 
 | | streamft (36 blocks resident) | PEFT / transformers |
 |---|---:|---:|
-| step time, median of 30 @ seq 512 | 0.938 s | **0.579 s** |
+| step time, median of 30 @ seq 512 | 0.94–1.24 s | **0.58–0.88 s** |
 | peak VRAM | **3.85 GB** | 4.90 GB |
 | eval before | 1.42087 | 1.42087 |
-| eval after | **1.30154** (−0.1193) | 1.32722 (−0.0936) |
+| eval after | **1.30720** (−0.1137) | 1.33002 (−0.0909) |
 
-**7B and 14B (do not fit comfortably):**
+**7B / 14B / 32B (do not fit comfortably), after the masked-gradient fix:**
 
-| model | step time, median @ seq 512 | peak VRAM | eval (start → after) |
+| model | step time, median @ seq 512 | peak VRAM | eval (start → after, 30 steps) |
 |---|---:|---:|---:|
-| 7B, streamft (28 blocks) | **1.70 s** | **5.51 GB** | 2.1928 → 1.1094 |
+| 7B, streamft (28 blocks) | **1.55 s** | **5.51 GB** | 2.1928 → **1.1055** |
 | 7B, PEFT | 8.96 s | 10.05 GB | 2.1914 → 1.1491 |
-| 14B, streamft (6 blocks) | **6.73 s** | **3.86 GB** | 2.3748 → 1.5312 (10 steps) |
-| 14B, PEFT | 18.56 s | 16.48 GB | 2.3748 (3 steps only) |
+| 14B, streamft (6 blocks) | **5.86 s** | **3.86 GB** | 2.3748 → **0.9742** |
+| 14B, PEFT | 18.56 s | 16.48 GB | 3 steps only |
+| 32B, streamft (2 blocks, seq 1024) | **12.48 s** | **3.90 GB** | 1.8755 → **1.0008** |
+| 32B, PEFT | fails before step 1 | — | — |
 
 At 7B the crossover has already happened: PEFT's 4.4 GB of weights plus activations no
-longer fit, so it spills (10.05 GB reserved) and pays 5.3× the step time. Note that
+longer fit, so it spills (10.05 GB reserved) and pays ~5.8× the step time. Note that
 streamft's 7B peak (5.51 GB) is *higher* than its 14B peak (3.86 GB) because the 7B run
 kept all 28 blocks resident while the 14B run kept 6 — the window is a choice, and the
 14B number is the one to copy for a 6 GB card.
@@ -144,16 +150,58 @@ kept all 28 blocks resident while the 14B run kept 6 — the window is a choice,
 into shared memory (19.45 GB reserved on a 6 GiB card), and `prepare_model_for_kbit_training`
 then dies allocating the fp32 norm copy: *CUDA out of memory. Tried to allocate 2.90 GiB;
 18.00 GiB is already allocated.* The streaming engine runs the same checkpoint at
-**12.5 s/step and 3.90 GB** (64/64 layers, held-out 1.8755 to 0.9959).
+**12.5 s/step and 3.90 GB** (64/64 layers).
 
 The 3B row is the honest answer to "is this faster?": **no** — on a model that fits, PEFT
-is 1.6× faster and uses 1.05 GB more. The 14B row is the design point: PEFT does not
-OOM on Windows, it **spills into shared system memory** (16.48 GB reserved on a 6.4 GB
-card) and pays 2.8× the step time; streamft stays at 3.86 GB. Over 30 steps and one seed
-the 3B quality difference is not meaningful.
+is faster and uses 1.05 GB more. The 14B/32B rows are the design point: PEFT does not
+OOM on Windows, it **spills into shared system memory** and pays 2.8–5.8× the step time.
+
+### Correction: masked gradient (25 Oct, after an external review)
+
+`chunked_lm_loss_and_backward_masked` divided every chunk by **that chunk's** valid-token
+count. The reported loss was still the correct mean, so the curves looked right, but the
+accumulated gradient over-weighted chunks with few unmasked tokens — i.e. the prompt-heavy
+start of an SFT row. Measured on a toy model with an uneven mask: **112–427% relative
+gradient error** at seq 512/1024 with 60% of the row masked (chunk 128), while balanced
+masks were unaffected.
+
+The fix divides every chunk by the **global** valid count (computed once, no-grad), so the
+gradient is exactly d(mean CE over valid tokens)/dh. Two tests pin it:
+`test_masked_chunked_gradient_matches_full_with_uneven_masks` and
+`test_masked_chunked_handles_fully_masked_chunks`.
+
+All SFT runs above were re-run with the fix. Quality held: 3B −0.114 (was −0.119),
+7B −1.087 (was −1.083), 32B −0.875 (was −0.880); 14B is now 30 steps (−1.40) instead of
+10. The numbers in the tables are the post-fix ones.
 
 Reproduce: `python scripts/compare_peft.py --model checkpoints/qwen3b_bnb4bit --steps 30`
-and `... --model checkpoints/qwen14b_nf4 --steps 3`.
+and `... --model checkpoints/qwen14b_nf4 --steps 30`.
+
+### Reproducibility probe
+
+Run this and compare the row; it is deterministic (seed 1234) apart from GPU-kernel
+non-determinism:
+
+```bash
+streamft train --model-dir checkpoints/qwen3b_bnb4bit --steps 3 --seq 128 \
+  --eval_chunks 1 --tag repro
+```
+
+Reference on the only machine tested so far (RTX 3050 Laptop 6 GB, torch 2.13.0+cu126,
+bitsandbytes 0.50.2, Windows 11):
+
+```text
+losses        3.347981  3.647039  3.049560
+ce_before     2.44565
+ce_after      2.44568
+peak_gb_max   3.068
+adapter_sha_after  422424c59ef004c5
+```
+
+Different hardware will differ in the last digits (kernels, bf16 rounding). What must
+match is the shape: the same three losses to ~1e-3, `grad_layers == n_layers`, and a
+changed adapter hash. If it does not, that is a bug report worth posting with the
+`streamft doctor` output.
 
 ## What this does not do
 
@@ -203,6 +251,7 @@ URLs. The engine lived at `experiments/exp54_disk_qlora.py` + `lcsa/` until the
 - [Running a 4B model in 6 GB and training 3B parameters on a laptop](https://dev.to/mycute_97b39c4e34027570f9/running-a-4b-model-in-6-gb-and-training-3b-parameters-on-a-laptop-4al)
 - [The same 32B post on Medium](https://medium.com/@myredditacc9393/fine-tuning-a-32b-model-on-a-6-gb-laptop-gpu-e749df447229)
 - [Will it fit? A free planner for fine-tuning models that do not fit](https://dev.to/mycute_97b39c4e34027570f9/will-it-fit-a-free-planner-for-fine-tuning-models-that-do-not-fit-on-your-gpu-o50)
+- [Will it fit? on Medium](https://medium.com/@myredditacc9393/will-it-fit-a-free-check-before-you-download-19-gb-of-weights-9097e1fcf93f)
 
 ## License
 

{streamft-0.2.0 → streamft-0.2.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "streamft"
-version = "0.2.0"
+version = "0.2.2"
 description = "Fine-tune models that do not fit: stream NF4 weights from disk, one transformer block at a time."
 readme = "README.md"
 requires-python = ">=3.10"

{streamft-0.2.0 → streamft-0.2.2}/streamft/__main__.py

@@ -3,6 +3,7 @@
   streamft fit      will this model fit? (memory arithmetic + measured speed when known)
   streamft prepare  turn a bf16 model into a pre-quantized NF4 checkpoint
   streamft train    the streaming trainer (all exp54 flags)
+  streamft data     build the token shards (LM path only; SFT needs none)
   streamft doctor   environment report (torch, CUDA, bitsandbytes, VRAM)
 """
 from __future__ import annotations
@@ -14,6 +15,7 @@ USAGE = """streamft - fine-tune models that do not fit
   streamft fit --model <dir|hf-id> [--gpu 6] [--seq 512]
   streamft prepare --model <dir> --save-quant checkpoints/<name>
   streamft train --model-dir checkpoints/<name> --sft --steps 30 [--seq 1024]
+  streamft data --model <dir|hf-id> --out data
   streamft doctor
 
 Docs: https://github.com/byjameson/stream-finetune
@@ -33,6 +35,11 @@ def main() -> int:
 
         return fit_main(rest)
 
+    if cmd == "data":
+        from streamft.data import main as data_main
+
+        return data_main(rest)
+
     if cmd == "doctor":
         from streamft.doctor import main as doctor_main
 

{streamft-0.2.0 → streamft-0.2.2}/streamft/autotune.py

@@ -89,8 +89,20 @@ def checkpoint_stats(model_dir) -> dict:
     from safetensors import safe_open
 
     root = Path(model_dir)
-    index = json.loads((root / "model.safetensors.index.json").read_text())
-    weight_map: dict[str, str] = dict(index["weight_map"])
+    idx_path = root / "model.safetensors.index.json"
+    if idx_path.exists():
+        index = json.loads(idx_path.read_text())
+        weight_map: dict[str, str] = dict(index["weight_map"])
+    else:
+        # single-file checkpoint (small NF4 models): every key lives in one file
+        files = sorted(root.glob("*.safetensors"))
+        if len(files) != 1:
+            raise FileNotFoundError(
+                f"{root}: expected model.safetensors.index.json or exactly one "
+                f"*.safetensors file, found {len(files)}"
+            )
+        with safe_open(str(files[0]), framework="pt", device="cpu") as f:
+            weight_map = {k: files[0].name for k in f.keys()}
 
     handles: dict[str, object] = {}
     per_layer: dict[int, int] = {}

{streamft-0.2.0 → streamft-0.2.2}/streamft/cli.py

@@ -76,12 +76,22 @@ def adapter_sha(model) -> str:
     return h.hexdigest()[:16]
 
 
+def user_path(value) -> Path:
+    """Resolve a CLI argument against the current directory (absolute paths unchanged).
+
+    The package root (`ROOT`) is where *this file* lives; for a pip-installed wheel that
+    is site-packages, which must never be used to resolve user paths.
+    """
+    q = Path(value).expanduser()
+    return q if q.is_absolute() else Path.cwd() / q
+
+
 def prepare(model: str, save_quant: str) -> int:
     """Quantize a local bf16 model into a bnb-4bit checkpoint (uint8 + qstate)."""
     from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
 
-    src = str(ROOT / model)
-    dst = ROOT / save_quant
+    src = str(user_path(model))
+    dst = user_path(save_quant)
     print(f"[prepare] {model} -> {save_quant}", flush=True)
     bnb = BitsAndBytesConfig(
         load_in_4bit=True,
@@ -212,7 +222,7 @@ def main() -> int:
     if args.prepare:
         return prepare(args.model, args.save_quant)
 
-    model_dir = str(ROOT / args.model_dir)
+    model_dir = str(user_path(args.model_dir))
     print(f"[exp54] model_dir={Path(model_dir).name} steps={args.steps} seq={args.seq} "
           f"lora_r={args.lora_r} targets={args.targets} head={args.head}", flush=True)
 
@@ -361,12 +371,12 @@ def main() -> int:
         print(f"[eval] after loss={ce1:.4f} (delta {ce1-ce0:+.4f})", flush=True)
         wall = time.time() - t_all
     else:
-        tokens = load_tokens(str(ROOT / args.data))
+        tokens = load_tokens(str(user_path(args.data)))
         sampler = TrainSampler(tokens, args.seq, generator=torch.Generator().manual_seed(1234))
         batches = [sampler.batch(1) for _ in range(args.steps)]
 
         if args.eval_chunks > 0:
-            ev = chunks_from_tokens(load_tokens(str(ROOT / args.eval_data)), args.eval_seq,
+            ev = chunks_from_tokens(load_tokens(str(user_path(args.eval_data))), args.eval_seq,
                                     args.eval_chunks)
             ce0 = sum(stream_eval_ce(tr, ev[i].unsqueeze(0), args.chunk)
                       for i in range(len(ev))) / len(ev)
@@ -392,7 +402,7 @@ def main() -> int:
 
     saved = None
     if args.save:
-        save_path = ROOT / args.save
+        save_path = user_path(args.save)
         save_path.mkdir(parents=True, exist_ok=True)
         torch.save(lora_state_dict(model), str(save_path / "adapter.pt"))
         (save_path / "adapter_config.json").write_text(json.dumps({
@@ -437,11 +447,16 @@ def main() -> int:
         "ce_after": round(ce1, 5) if ce1 is not None else None,
         "ce_delta": round(ce1 - ce0, 6) if (ce0 is not None and ce1 is not None) else None,
         "saved": saved,
+        "license": "Apache-2.0",
+        "gpu": torch.cuda.get_device_name(0),
+        "torch": torch.__version__,
+        "cuda": torch.version.cuda,
         "ts": now(),
     }
     print(json.dumps(row, ensure_ascii=True), flush=True)
-    append_jsonl(str(ROOT / "results" / "results.jsonl"), row)
-    with open(ROOT / "results" / "exp54_disk_qlora.log", "a", encoding="utf-8") as f:
+    results_dir = user_path("results")
+    append_jsonl(str(results_dir / "results.jsonl"), row)
+    with open(results_dir / "exp54_disk_qlora.log", "a", encoding="utf-8") as f:
         f.write(json.dumps(row, ensure_ascii=False) + "\n")
     return 0
 

{streamft-0.2.0 → streamft-0.2.2}/streamft/data.py

@@ -79,3 +79,22 @@ class TrainSampler:
         xs = torch.stack([self.tokens[s: s + self.seq_len].long() for s in starts])
         ys = torch.stack([self.tokens[s + 1: s + self.seq_len + 1].long() for s in starts])
         return xs, ys
+
+
+def main(argv=None) -> int:
+    """`streamft data` -- build the token shards that --data/--eval_data expect."""
+    import argparse
+
+    from transformers import AutoTokenizer
+
+    ap = argparse.ArgumentParser(prog="streamft data")
+    ap.add_argument("--model", default="models/Qwen2.5-0.5B",
+                    help="tokenizer source (local dir or HF id)")
+    ap.add_argument("--out", default="data")
+    ap.add_argument("--force", action="store_true", help="rebuild existing shards")
+    args = ap.parse_args(argv)
+    tok = AutoTokenizer.from_pretrained(args.model)
+    out = build_all(tok, args.out, force=args.force)
+    for name, path in out.items():
+        print(f"{name:20s} {path}")
+    return 0

{streamft-0.2.0 → streamft-0.2.2}/streamft/disk_quant_store.py

@@ -43,8 +43,24 @@ class DiskQuantStore:
 
     def __init__(self, model_dir, device: str = "cuda"):
         self.dir = Path(model_dir)
-        index = json.loads((self.dir / "model.safetensors.index.json").read_text())
-        self.weight_map: dict[str, str] = dict(index["weight_map"])
+        idx_path = self.dir / "model.safetensors.index.json"
+        if idx_path.exists():
+            index = json.loads(idx_path.read_text())
+            self.weight_map: dict[str, str] = dict(index["weight_map"])
+        else:
+            # Single-file checkpoint (what `save_pretrained` writes for models whose
+            # quantized weights fit in one shard, e.g. 0.5B/1.5B NF4). The index is
+            # optional; every key lives in that one file.
+            files = sorted(self.dir.glob("*.safetensors"))
+            if len(files) != 1:
+                raise FileNotFoundError(
+                    f"{self.dir}: expected model.safetensors.index.json or exactly one "
+                    f"*.safetensors file, found {len(files)}"
+                )
+            from safetensors import safe_open
+
+            with safe_open(str(files[0]), framework="pt", device="cpu") as f:
+                self.weight_map = {k: files[0].name for k in f.keys()}
         self.device = device
         self._handles: dict[str, object] = {}
 

{streamft-0.2.0 → streamft-0.2.2}/streamft/fit.py

@@ -109,7 +109,9 @@ def main(argv: list[str] | None = None) -> int:
           f"+ reserve {RESERVE_GB} GB (embedding stays on CPU)")
     print(f"  -> whole model needs ~{out['needed_whole_gb']} GB on {args.gpu:g} GB")
     if fits_whole:
-        print("  verdict: fits whole at this seq (no streaming needed; >1k can still spill)")
+        print("  verdict: budget says the whole model fits at this seq")
+        print("  caution: activations and allocator headroom are not fully modelled;")
+        print("           treat this as a budget check, not a guarantee. >1k can still spill.")
     else:
         print("  verdict: needs streaming (one or more blocks on the GPU at a time)")
     if measured:

{streamft-0.2.0 → streamft-0.2.2}/streamft/stream_quant_lora.py

@@ -22,27 +22,39 @@ from .stream_train import _layer_out, chunked_lm_loss_and_backward
 
 
 def chunked_lm_loss_and_backward_masked(head, h, ys, chunk=128, ignore_index=-100):
-    """Chunked CE with an ignore_index (for SFT prompt masking)."""
+    """Chunked CE with an ignore_index (for SFT prompt masking).
+
+    Every chunk contributes `loss_sum / valid_total` where valid_total is the number of
+    unmasked targets in the **whole sequence**, so the accumulated gradient is exactly
+    d(mean CE over valid tokens)/dh.
+
+    FIX (25.10.2026, reported by an external reviewer): the first version divided each
+    chunk by that chunk's own valid count. Losses were unaffected (the returned value
+    summed to the right mean) but the gradient over-weighted chunks with few valid
+    tokens, i.e. the prompt-heavy start of an SFT row. tests/test_streamft_smoke.py
+    pins the corrected behaviour with an uneven mask.
+    """
     B, T, _ = h.shape
+    tgt_all = ys.to(h.device)
+    valid_total = int((tgt_all != ignore_index).sum())
+    if valid_total == 0:
+        zero = torch.zeros((), device=h.device)
+        return zero, zero
     total = torch.zeros((), device=h.device, dtype=torch.float32)
-    valid_total = 0
     for a in range(0, T, chunk):
         b = min(a + chunk, T)
-        logits = head(h[:, a:b])
-        tgt = ys[:, a:b].reshape(-1).to(h.device)
-        valid = int((tgt != ignore_index).sum())
-        if valid == 0:
+        tgt = tgt_all[:, a:b].reshape(-1)
+        if int((tgt != ignore_index).sum()) == 0:
             continue
+        logits = head(h[:, a:b])
         loss_sum = torch.nn.functional.cross_entropy(
             logits.reshape(-1, logits.shape[-1]).float(), tgt,
             reduction="sum", ignore_index=ignore_index,
         )
-        (loss_sum / valid).backward()
+        (loss_sum / valid_total).backward()
         with torch.no_grad():
             total = total + loss_sum.detach()
-        valid_total += valid
-    denom = max(valid_total, 1)
-    return total / denom, total / denom
+    return total / valid_total, total / valid_total
 
 
 def base_model_of(model):

{streamft-0.2.0 → streamft-0.2.2/streamft.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: streamft
-Version: 0.2.0
+Version: 0.2.2
 Summary: Fine-tune models that do not fit: stream NF4 weights from disk, one transformer block at a time.
 Author-email: Gokhan Cava <byjameson@users.noreply.github.com>
 License: Apache-2.0
@@ -30,6 +30,8 @@ Dynamic: license-file
 
 # stream-finetune — fine-tune a 32B model on a 6 GB laptop GPU
 
+[![PyPI](https://img.shields.io/pypi/v/streamft)](https://pypi.org/project/streamft/) [![try it](https://img.shields.io/badge/try-will%20it%20fit%3F-6ee7b7)](https://huggingface.co/spaces/byjameson/will-it-fit)
+
 A small training engine that fine-tunes models that **do not fit in VRAM**. The NF4
 weights are mmap'd from disk, one transformer block is materialised on the GPU at a
 time, and the backward recomputes through it. The base stays frozen; a LoRA adapter is
@@ -53,7 +55,8 @@ The 32B weights are **19.2 GB**. They never fully live in VRAM. At 1024 tokens/s
 throughput is **3.1×** the 128-token rate, because the per-step I/O is amortised over
 more tokens. 50 steps of Dolly-15k SFT moved eval loss **1.8755 → 0.9661**.
 
-**Start with [`GUIDE_6GB_32B.md`](GUIDE_6GB_32B.md)** — the practical write-up: presets,
+**Installed from PyPI? Start with [`QUICKSTART.md`](QUICKSTART.md)** — it uses only the
+published CLI (no repo scripts). Deeper dive: [`GUIDE_6GB_32B.md`](GUIDE_6GB_32B.md) — the practical write-up: presets,
 the six traps, and which numbers are real. The measurement protocol, the discarded rows
 and the correction of an earlier wrong claim ("8.6× / 100 tok/s") are in
 [`LONGCTX_FINDINGS.md`](LONGCTX_FINDINGS.md).
@@ -61,7 +64,7 @@ and the correction of an earlier wrong claim ("8.6× / 100 tok/s") are in
 ## Quick start
 
 ```powershell
-pip install -e .            # or: pip install streamft (once it is on PyPI)
+pip install streamft
 
 # 0) will it fit? memory arithmetic + the measured reference, no weights downloaded
 streamft fit --model Qwen/Qwen2.5-14B-Instruct --gpu 6 --seq 512
@@ -142,30 +145,33 @@ Raw output: `results/gradcheck.json` (git-ignored; the numbers above are copied
 
 ## Compared with PEFT (matched runs)
 
-Same data, same LoRA (r=8, α=16, q/k/v/o), same AdamW lr 1e-4, batch 1, same prompt
+Same data, same LoRA (r=8, alpha=16, q/k/v/o), same AdamW lr 1e-4, batch 1, same prompt
 masking, same token-weighted masked eval on 8 held-out rows. PEFT uses
 `prepare_model_for_kbit_training` with gradient checkpointing (the standard QLoRA setup).
+Step times move by up to ~30% between runs on this laptop; quality is the stable number.
 
 **3B (fits entirely):**
 
 | | streamft (36 blocks resident) | PEFT / transformers |
 |---|---:|---:|
-| step time, median of 30 @ seq 512 | 0.938 s | **0.579 s** |
+| step time, median of 30 @ seq 512 | 0.94–1.24 s | **0.58–0.88 s** |
 | peak VRAM | **3.85 GB** | 4.90 GB |
 | eval before | 1.42087 | 1.42087 |
-| eval after | **1.30154** (−0.1193) | 1.32722 (−0.0936) |
+| eval after | **1.30720** (−0.1137) | 1.33002 (−0.0909) |
 
-**7B and 14B (do not fit comfortably):**
+**7B / 14B / 32B (do not fit comfortably), after the masked-gradient fix:**
 
-| model | step time, median @ seq 512 | peak VRAM | eval (start → after) |
+| model | step time, median @ seq 512 | peak VRAM | eval (start → after, 30 steps) |
 |---|---:|---:|---:|
-| 7B, streamft (28 blocks) | **1.70 s** | **5.51 GB** | 2.1928 → 1.1094 |
+| 7B, streamft (28 blocks) | **1.55 s** | **5.51 GB** | 2.1928 → **1.1055** |
 | 7B, PEFT | 8.96 s | 10.05 GB | 2.1914 → 1.1491 |
-| 14B, streamft (6 blocks) | **6.73 s** | **3.86 GB** | 2.3748 → 1.5312 (10 steps) |
-| 14B, PEFT | 18.56 s | 16.48 GB | 2.3748 (3 steps only) |
+| 14B, streamft (6 blocks) | **5.86 s** | **3.86 GB** | 2.3748 → **0.9742** |
+| 14B, PEFT | 18.56 s | 16.48 GB | 3 steps only |
+| 32B, streamft (2 blocks, seq 1024) | **12.48 s** | **3.90 GB** | 1.8755 → **1.0008** |
+| 32B, PEFT | fails before step 1 | — | — |
 
 At 7B the crossover has already happened: PEFT's 4.4 GB of weights plus activations no
-longer fit, so it spills (10.05 GB reserved) and pays 5.3× the step time. Note that
+longer fit, so it spills (10.05 GB reserved) and pays ~5.8× the step time. Note that
 streamft's 7B peak (5.51 GB) is *higher* than its 14B peak (3.86 GB) because the 7B run
 kept all 28 blocks resident while the 14B run kept 6 — the window is a choice, and the
 14B number is the one to copy for a 6 GB card.
@@ -174,16 +180,58 @@ kept all 28 blocks resident while the 14B run kept 6 — the window is a choice,
 into shared memory (19.45 GB reserved on a 6 GiB card), and `prepare_model_for_kbit_training`
 then dies allocating the fp32 norm copy: *CUDA out of memory. Tried to allocate 2.90 GiB;
 18.00 GiB is already allocated.* The streaming engine runs the same checkpoint at
-**12.5 s/step and 3.90 GB** (64/64 layers, held-out 1.8755 to 0.9959).
+**12.5 s/step and 3.90 GB** (64/64 layers).
 
 The 3B row is the honest answer to "is this faster?": **no** — on a model that fits, PEFT
-is 1.6× faster and uses 1.05 GB more. The 14B row is the design point: PEFT does not
-OOM on Windows, it **spills into shared system memory** (16.48 GB reserved on a 6.4 GB
-card) and pays 2.8× the step time; streamft stays at 3.86 GB. Over 30 steps and one seed
-the 3B quality difference is not meaningful.
+is faster and uses 1.05 GB more. The 14B/32B rows are the design point: PEFT does not
+OOM on Windows, it **spills into shared system memory** and pays 2.8–5.8× the step time.
+
+### Correction: masked gradient (25 Oct, after an external review)
+
+`chunked_lm_loss_and_backward_masked` divided every chunk by **that chunk's** valid-token
+count. The reported loss was still the correct mean, so the curves looked right, but the
+accumulated gradient over-weighted chunks with few unmasked tokens — i.e. the prompt-heavy
+start of an SFT row. Measured on a toy model with an uneven mask: **112–427% relative
+gradient error** at seq 512/1024 with 60% of the row masked (chunk 128), while balanced
+masks were unaffected.
+
+The fix divides every chunk by the **global** valid count (computed once, no-grad), so the
+gradient is exactly d(mean CE over valid tokens)/dh. Two tests pin it:
+`test_masked_chunked_gradient_matches_full_with_uneven_masks` and
+`test_masked_chunked_handles_fully_masked_chunks`.
+
+All SFT runs above were re-run with the fix. Quality held: 3B −0.114 (was −0.119),
+7B −1.087 (was −1.083), 32B −0.875 (was −0.880); 14B is now 30 steps (−1.40) instead of
+10. The numbers in the tables are the post-fix ones.
 
 Reproduce: `python scripts/compare_peft.py --model checkpoints/qwen3b_bnb4bit --steps 30`
-and `... --model checkpoints/qwen14b_nf4 --steps 3`.
+and `... --model checkpoints/qwen14b_nf4 --steps 30`.
+
+### Reproducibility probe
+
+Run this and compare the row; it is deterministic (seed 1234) apart from GPU-kernel
+non-determinism:
+
+```bash
+streamft train --model-dir checkpoints/qwen3b_bnb4bit --steps 3 --seq 128 \
+  --eval_chunks 1 --tag repro
+```
+
+Reference on the only machine tested so far (RTX 3050 Laptop 6 GB, torch 2.13.0+cu126,
+bitsandbytes 0.50.2, Windows 11):
+
+```text
+losses        3.347981  3.647039  3.049560
+ce_before     2.44565
+ce_after      2.44568
+peak_gb_max   3.068
+adapter_sha_after  422424c59ef004c5
+```
+
+Different hardware will differ in the last digits (kernels, bf16 rounding). What must
+match is the shape: the same three losses to ~1e-3, `grad_layers == n_layers`, and a
+changed adapter hash. If it does not, that is a bug report worth posting with the
+`streamft doctor` output.
 
 ## What this does not do
 
@@ -233,6 +281,7 @@ URLs. The engine lived at `experiments/exp54_disk_qlora.py` + `lcsa/` until the
 - [Running a 4B model in 6 GB and training 3B parameters on a laptop](https://dev.to/mycute_97b39c4e34027570f9/running-a-4b-model-in-6-gb-and-training-3b-parameters-on-a-laptop-4al)
 - [The same 32B post on Medium](https://medium.com/@myredditacc9393/fine-tuning-a-32b-model-on-a-6-gb-laptop-gpu-e749df447229)
 - [Will it fit? A free planner for fine-tuning models that do not fit](https://dev.to/mycute_97b39c4e34027570f9/will-it-fit-a-free-planner-for-fine-tuning-models-that-do-not-fit-on-your-gpu-o50)
+- [Will it fit? on Medium](https://medium.com/@myredditacc9393/will-it-fit-a-free-check-before-you-download-19-gb-of-weights-9097e1fcf93f)
 
 ## License
 

{streamft-0.2.0 → streamft-0.2.2}/tests/test_streamft_smoke.py

@@ -1,114 +1,179 @@
-"""Smoke tests for the streamft package (CPU only, no model download).
-
-They pin the things that broke silently during development:
-  * a LoRA layer must be a *no-op* at init (B = 0), otherwise "before/after" is a lie;
-  * chunked cross-entropy must produce exactly d(mean CE)/dh, not a scaled version;
-  * only the adapter gets gradients (the NF4 base stays frozen).
-"""
-
-import math
-
-import torch
-import torch.nn as nn
-import torch.nn.functional as F
-
-from streamft import cli, stream_train
-from streamft.lora import LoRALinear, apply_lora, count_lora_params, lora_parameters
-
-
-# ---------------------------------------------------------------- lora
-
-def test_lora_is_noop_at_init():
-    torch.manual_seed(0)
-    base = nn.Linear(16, 8)
-    lora = LoRALinear(base, r=4, alpha=16)
-    x = torch.randn(3, 16)
-    assert torch.allclose(lora(x), base(x), atol=1e-6)
-    assert lora(x).shape == (3, 8)
-    assert sum(p.numel() for p in lora.parameters() if p.requires_grad) == 4 * (16 + 8)
-
-
-def test_gradients_reach_only_the_adapter():
-    torch.manual_seed(0)
-    base = nn.Linear(16, 8)
-    lora = LoRALinear(base, r=4)
-    lora(torch.randn(3, 16)).sum().backward()
-    assert base.weight.grad is None
-    assert lora.A.grad is not None
-    assert lora.B.grad is not None
-
-
-class _Attn(nn.Module):
-    def __init__(self, d):
-        super().__init__()
-        for name in ("q_proj", "k_proj", "v_proj", "o_proj"):
-            setattr(self, name, nn.Linear(d, d, bias=False))
-
-
-class _Mlp(nn.Module):
-    def __init__(self, d):
-        super().__init__()
-        self.gate_proj = nn.Linear(d, 2 * d, bias=False)
-        self.up_proj = nn.Linear(d, 2 * d, bias=False)
-        self.down_proj = nn.Linear(2 * d, d, bias=False)
-
-
-class _Layer(nn.Module):
-    def __init__(self, d):
-        super().__init__()
-        self.self_attn = _Attn(d)
-        self.mlp = _Mlp(d)
-
-
-class _FakeQwen(nn.Module):
-    """Just enough structure for apply_lora: model.layers[i].{self_attn,mlp}."""
-
-    def __init__(self, d=16, n=2):
-        super().__init__()
-        self.model = nn.Module()
-        self.model.layers = nn.ModuleList([_Layer(d) for _ in range(n)])
-
-
-def test_apply_lora_counts_and_freezes():
-    model = _FakeQwen()
-    apply_lora(model, r_attn=4, r_mlp=4)
-
-    r, d = 4, 16
-    per_proj_16 = r * (d + d)          # q/k/v/o and down_proj
-    per_proj_32 = r * (d + 2 * d)      # gate/up_proj
-    expected = 2 * (4 * per_proj_16 + 3 * per_proj_32)
-    assert count_lora_params(model) == expected
-
-    q = model.model.layers[0].self_attn.q_proj
-    assert isinstance(q, LoRALinear)
-    assert all(p.requires_grad is False for p in q.base.parameters())
-
-    trainable = [p for p in model.parameters() if p.requires_grad]
-    assert trainable and len(list(lora_parameters(model))) == len(trainable)
-
-
-# ---------------------------------------------------- chunked loss
-
-def test_chunked_ce_gradient_matches_full_mean():
-    torch.manual_seed(0)
-    d, T, V = 12, 9, 7
-    head = nn.Linear(d, V, bias=False)
-    ys = torch.randint(0, V, (1, T))
-
-    h1 = torch.randn(1, T, d, requires_grad=True)
-    full = F.cross_entropy(head(h1).reshape(-1, V), ys.reshape(-1), reduction="mean")
-    full.backward()
-
-    h2 = torch.randn(1, T, d, requires_grad=True)
-    with torch.no_grad():
-        h2.copy_(h1.detach())
-    chunked_last, chunked_mean = stream_train.chunked_lm_loss_and_backward(head, h2, ys, chunk=4)
-
-    assert math.isclose(float(chunked_mean.detach()), float(full.detach()), rel_tol=1e-5)
-    assert torch.allclose(h1.grad, h2.grad, atol=1e-6)
-
-
-# ---------------------------------------------------------------- cli
-
-def test_cli_is_importable():
-    assert callable(cli.main)
+"""Smoke tests for the streamft package (CPU only, no model download).
+
+They pin the things that broke silently during development:
+  * a LoRA layer must be a *no-op* at init (B = 0), otherwise "before/after" is a lie;
+  * chunked cross-entropy must produce exactly d(mean CE)/dh, not a scaled version;
+  * only the adapter gets gradients (the NF4 base stays frozen).
+"""
+
+import math
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from streamft import cli, stream_train
+from streamft.lora import LoRALinear, apply_lora, count_lora_params, lora_parameters
+
+
+# ---------------------------------------------------------------- lora
+
+def test_lora_is_noop_at_init():
+    torch.manual_seed(0)
+    base = nn.Linear(16, 8)
+    lora = LoRALinear(base, r=4, alpha=16)
+    x = torch.randn(3, 16)
+    assert torch.allclose(lora(x), base(x), atol=1e-6)
+    assert lora(x).shape == (3, 8)
+    assert sum(p.numel() for p in lora.parameters() if p.requires_grad) == 4 * (16 + 8)
+
+
+def test_gradients_reach_only_the_adapter():
+    torch.manual_seed(0)
+    base = nn.Linear(16, 8)
+    lora = LoRALinear(base, r=4)
+    lora(torch.randn(3, 16)).sum().backward()
+    assert base.weight.grad is None
+    assert lora.A.grad is not None
+    assert lora.B.grad is not None
+
+
+class _Attn(nn.Module):
+    def __init__(self, d):
+        super().__init__()
+        for name in ("q_proj", "k_proj", "v_proj", "o_proj"):
+            setattr(self, name, nn.Linear(d, d, bias=False))
+
+
+class _Mlp(nn.Module):
+    def __init__(self, d):
+        super().__init__()
+        self.gate_proj = nn.Linear(d, 2 * d, bias=False)
+        self.up_proj = nn.Linear(d, 2 * d, bias=False)
+        self.down_proj = nn.Linear(2 * d, d, bias=False)
+
+
+class _Layer(nn.Module):
+    def __init__(self, d):
+        super().__init__()
+        self.self_attn = _Attn(d)
+        self.mlp = _Mlp(d)
+
+
+class _FakeQwen(nn.Module):
+    """Just enough structure for apply_lora: model.layers[i].{self_attn,mlp}."""
+
+    def __init__(self, d=16, n=2):
+        super().__init__()
+        self.model = nn.Module()
+        self.model.layers = nn.ModuleList([_Layer(d) for _ in range(n)])
+
+
+def test_apply_lora_counts_and_freezes():
+    model = _FakeQwen()
+    apply_lora(model, r_attn=4, r_mlp=4)
+
+    r, d = 4, 16
+    per_proj_16 = r * (d + d)          # q/k/v/o and down_proj
+    per_proj_32 = r * (d + 2 * d)      # gate/up_proj
+    expected = 2 * (4 * per_proj_16 + 3 * per_proj_32)
+    assert count_lora_params(model) == expected
+
+    q = model.model.layers[0].self_attn.q_proj
+    assert isinstance(q, LoRALinear)
+    assert all(p.requires_grad is False for p in q.base.parameters())
+
+    trainable = [p for p in model.parameters() if p.requires_grad]
+    assert trainable and len(list(lora_parameters(model))) == len(trainable)
+
+
+# ---------------------------------------------------- chunked loss
+
+def test_chunked_ce_gradient_matches_full_mean():
+    torch.manual_seed(0)
+    d, T, V = 12, 9, 7
+    head = nn.Linear(d, V, bias=False)
+    ys = torch.randint(0, V, (1, T))
+
+    h1 = torch.randn(1, T, d, requires_grad=True)
+    full = F.cross_entropy(head(h1).reshape(-1, V), ys.reshape(-1), reduction="mean")
+    full.backward()
+
+    h2 = torch.randn(1, T, d, requires_grad=True)
+    with torch.no_grad():
+        h2.copy_(h1.detach())
+    chunked_last, chunked_mean = stream_train.chunked_lm_loss_and_backward(head, h2, ys, chunk=4)
+
+    assert math.isclose(float(chunked_mean.detach()), float(full.detach()), rel_tol=1e-5)
+    assert torch.allclose(h1.grad, h2.grad, atol=1e-6)
+
+
+# ---------------------------------------------------------------- cli
+
+def test_cli_is_importable():
+    assert callable(cli.main)
+
+
+# ------------------------------------------- masked (SFT) chunked loss
+
+def test_masked_chunked_gradient_matches_full_with_uneven_masks():
+    """The 25.10 fix: chunks with different valid counts must not be re-weighted.
+
+    With chunk=3 and a prompt mask over the first 4 positions, chunk 0 has zero valid
+    targets, chunk 1 has two, later chunks have three. The old implementation divided
+    each chunk by its own count, which inflated the sparse chunks' gradient.
+    """
+    from streamft.stream_quant_lora import chunked_lm_loss_and_backward_masked
+
+    torch.manual_seed(0)
+    d, T, V = 12, 10, 7
+    head = nn.Linear(d, V, bias=False)
+    ys = torch.randint(0, V, (1, T))
+    ys[0, :4] = -100     # prompt
+    ys[0, 7] = -100      # a masked middle position
+
+    h1 = torch.randn(1, T, d, requires_grad=True)
+    full = F.cross_entropy(head(h1).reshape(-1, V), ys.reshape(-1),
+                           ignore_index=-100, reduction="mean")
+    full.backward()
+
+    h2 = torch.randn(1, T, d, requires_grad=True)
+    with torch.no_grad():
+        h2.copy_(h1.detach())
+    _, mean = chunked_lm_loss_and_backward_masked(head, h2, ys, chunk=3)
+
+    assert math.isclose(float(mean.detach()), float(full.detach()), rel_tol=1e-6)
+    assert torch.allclose(h1.grad, h2.grad, atol=1e-6)
+
+
+def test_masked_chunked_handles_fully_masked_chunks():
+    """A chunk with no valid targets must contribute nothing at all."""
+    from streamft.stream_quant_lora import chunked_lm_loss_and_backward_masked
+
+    torch.manual_seed(0)
+    d, T, V = 8, 6, 5
+    head = nn.Linear(d, V, bias=False)
+    ys = torch.randint(0, V, (1, T))
+    ys[0, :3] = -100
+    h = torch.randn(1, T, d, requires_grad=True)
+    _, mean = chunked_lm_loss_and_backward_masked(head, h, ys, chunk=2)
+    logits = head(h).reshape(-1, V)
+    tgt = ys.reshape(-1)
+    mask = tgt != -100
+    ref = F.cross_entropy(logits[mask], tgt[mask], reduction="mean")
+    assert math.isclose(float(mean.detach()), float(ref.detach()), rel_tol=1e-6)
+
+
+# ------------------------------------------------ checkpoint file layouts
+
+def test_disk_store_supports_single_safetensors_file(tmp_path):
+    """save_pretrained writes one file for small NF4 models; the index is optional."""
+    from safetensors.torch import save_file
+
+    from streamft.disk_quant_store import DiskQuantStore
+
+    save_file({"a": torch.zeros(4, dtype=torch.uint8),
+               "b.c": torch.ones(3)}, str(tmp_path / "model.safetensors"))
+    store = DiskQuantStore(str(tmp_path))
+    assert store.has("a") and store.has("b.c")
+    assert store.get("b.c").tolist() == [1.0, 1.0, 1.0]

streamft-0.2.0/LICENSE

@@ -1,21 +0,0 @@
-MIT License
-
-Copyright (c) 2026 LCGA contributors
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.