streamft 0.2.3__tar.gz → 0.2.4__tar.gz

{streamft-0.2.3 → streamft-0.2.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: streamft
-Version: 0.2.3
+Version: 0.2.4
 Summary: Fine-tune models that do not fit: stream NF4 weights from disk, one transformer block at a time.
 Author-email: Gokhan Cava <byjameson@users.noreply.github.com>
 License: Apache-2.0
@@ -61,34 +61,34 @@ the six traps, and which numbers are real. The measurement protocol, the discard
 and the correction of an earlier wrong claim ("8.6× / 100 tok/s") are in
 [`LONGCTX_FINDINGS.md`](LONGCTX_FINDINGS.md).
 
-## Quick start
+## Quick start (installed package, nothing else needed)
 
-```powershell
+```bash
 pip install streamft
+streamft doctor                    # torch / CUDA / bitsandbytes / free VRAM
 
-# 0) will it fit? memory arithmetic + the measured reference, no weights downloaded
+# will it fit? memory arithmetic from config.json + the measured reference
 streamft fit --model Qwen/Qwen2.5-14B-Instruct --gpu 6 --seq 512
-streamft doctor            # torch / CUDA / bitsandbytes / free VRAM
 
-# 1) token shards for the LM path (SFT on Dolly needs none of this)
-streamft prepare --model models/Qwen2.5-0.5B --save-quant checkpoints/qwen0.5b_nf4
-python scripts/prep_data.py --model models/Qwen2.5-0.5B
+# one NF4 checkpoint per model (HF id or local directory)
+streamft prepare --model Qwen/Qwen2.5-0.5B --save-quant checkpoints/qwen0.5b_nf4
 
-# 1) turn a local bf16 model into a pre-quantized NF4 checkpoint (once)
-python -m streamft.cli --prepare --model models/Qwen2.5-32B-Instruct `
-    --save-quant checkpoints/qwen32b_nf4
+# SFT on Dolly-15k — no data prep
+streamft train --model-dir checkpoints/qwen0.5b_nf4 --sft --steps 20 --seq 512 --tag first
 
-# 2) fine-tune it, streaming from disk
-python tools/finetune.py --model checkpoints/qwen32b_nf4 --mode lm --seq 1024 --steps 50
-python tools/finetune.py --model checkpoints/qwen14b_nf4 --data dolly --seq 512 --steps 240 `
-    --save checkpoints/my_adapter
+# LM path (optional): build token shards, then train without --sft
+streamft data --model Qwen/Qwen2.5-0.5B --out data
+streamft train --model-dir checkpoints/qwen0.5b_nf4 --steps 20 --seq 256 --tag lm
+```
 
-# show the exact engine command without running it
-python tools/finetune.py ... --print-only
+Every run appends one JSON row to `results/results.jsonl` next to where you launched it.
+The same commands are in [`QUICKSTART.md`](QUICKSTART.md), which is what a pip user should
+read — this README also documents the experiments behind the numbers.
 
-# CPU smoke tests (5 cases, no GPU, no downloads)
-pip install -r requirements-dev.txt; python -m pytest tests -q
-```
+**From a source checkout only** (not part of the wheel): `tools/finetune.py` is a wrapper
+that encodes the VRAM presets, `scripts/compare_peft.py` reproduces the engine comparison,
+`scripts/verify_gradients.py` the gradient check, `scripts/prep_data.py` the token shards
+(the installed equivalent is `streamft data`), and `pytest tests -q` the CPU suite.
 
 `tools/finetune.py` encodes the VRAM presets we measured, because the *combination* of
 flags decides whether a run fits — not any single flag:

{streamft-0.2.3 → streamft-0.2.4}/README.md

@@ -31,34 +31,34 @@ the six traps, and which numbers are real. The measurement protocol, the discard
 and the correction of an earlier wrong claim ("8.6× / 100 tok/s") are in
 [`LONGCTX_FINDINGS.md`](LONGCTX_FINDINGS.md).
 
-## Quick start
+## Quick start (installed package, nothing else needed)
 
-```powershell
+```bash
 pip install streamft
+streamft doctor                    # torch / CUDA / bitsandbytes / free VRAM
 
-# 0) will it fit? memory arithmetic + the measured reference, no weights downloaded
+# will it fit? memory arithmetic from config.json + the measured reference
 streamft fit --model Qwen/Qwen2.5-14B-Instruct --gpu 6 --seq 512
-streamft doctor            # torch / CUDA / bitsandbytes / free VRAM
 
-# 1) token shards for the LM path (SFT on Dolly needs none of this)
-streamft prepare --model models/Qwen2.5-0.5B --save-quant checkpoints/qwen0.5b_nf4
-python scripts/prep_data.py --model models/Qwen2.5-0.5B
+# one NF4 checkpoint per model (HF id or local directory)
+streamft prepare --model Qwen/Qwen2.5-0.5B --save-quant checkpoints/qwen0.5b_nf4
 
-# 1) turn a local bf16 model into a pre-quantized NF4 checkpoint (once)
-python -m streamft.cli --prepare --model models/Qwen2.5-32B-Instruct `
-    --save-quant checkpoints/qwen32b_nf4
+# SFT on Dolly-15k — no data prep
+streamft train --model-dir checkpoints/qwen0.5b_nf4 --sft --steps 20 --seq 512 --tag first
 
-# 2) fine-tune it, streaming from disk
-python tools/finetune.py --model checkpoints/qwen32b_nf4 --mode lm --seq 1024 --steps 50
-python tools/finetune.py --model checkpoints/qwen14b_nf4 --data dolly --seq 512 --steps 240 `
-    --save checkpoints/my_adapter
+# LM path (optional): build token shards, then train without --sft
+streamft data --model Qwen/Qwen2.5-0.5B --out data
+streamft train --model-dir checkpoints/qwen0.5b_nf4 --steps 20 --seq 256 --tag lm
+```
 
-# show the exact engine command without running it
-python tools/finetune.py ... --print-only
+Every run appends one JSON row to `results/results.jsonl` next to where you launched it.
+The same commands are in [`QUICKSTART.md`](QUICKSTART.md), which is what a pip user should
+read — this README also documents the experiments behind the numbers.
 
-# CPU smoke tests (5 cases, no GPU, no downloads)
-pip install -r requirements-dev.txt; python -m pytest tests -q
-```
+**From a source checkout only** (not part of the wheel): `tools/finetune.py` is a wrapper
+that encodes the VRAM presets, `scripts/compare_peft.py` reproduces the engine comparison,
+`scripts/verify_gradients.py` the gradient check, `scripts/prep_data.py` the token shards
+(the installed equivalent is `streamft data`), and `pytest tests -q` the CPU suite.
 
 `tools/finetune.py` encodes the VRAM presets we measured, because the *combination* of
 flags decides whether a run fits — not any single flag:

{streamft-0.2.3 → streamft-0.2.4}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "streamft"
-version = "0.2.3"
+version = "0.2.4"
 description = "Fine-tune models that do not fit: stream NF4 weights from disk, one transformer block at a time."
 readme = "README.md"
 requires-python = ">=3.10"

{streamft-0.2.3 → streamft-0.2.4}/streamft/cli.py

@@ -184,7 +184,12 @@ def _masked_eval(trainer, rows, tokenizer, seq: int) -> float:
     if skipped:
         print(f"[eval] skipped {skipped}/{len(rows)} rows: prompt is not an exact prefix "
               f"of the templated row (would train on the prompt)", flush=True)
-    return total / max(n, 1)
+    if n == 0:
+        raise RuntimeError(
+            "_masked_eval: every row was skipped (no valid prompt mask), so there is no "
+            "loss to report. Returning 0.0 here would look like a perfect model."
+        )
+    return total / n
 
 
 def main() -> int:
@@ -369,12 +374,17 @@ def main() -> int:
               f"eval={len(eval_rows)}", flush=True)
         ce0 = _masked_eval(tr, eval_rows, tok, args.seq)
         print(f"[eval] before loss={ce0:.4f}", flush=True)
-        skipped_train = 0
+        # Filter once so `--steps N` means N optimizer updates, not "N attempts minus
+        # whatever the mask rejected". A silently short run is a lie about the budget.
+        valid_rows = [r for r in train_rows if _encode_chat(tok, r, args.seq) is not None]
+        skipped_train = len(train_rows) - len(valid_rows)
+        if not valid_rows:
+            raise SystemExit("[sft] no training row has a valid prompt mask; refusing to run")
+        if skipped_train:
+            print(f"[sft] {skipped_train}/{len(train_rows)} rows dropped: the prompt is not "
+                  f"an exact prefix of the templated row", flush=True)
         for i in range(args.steps):
-            enc = _encode_chat(tok, train_rows[i % len(train_rows)], args.seq)
-            if enc is None:
-                skipped_train += 1
-                continue
+            enc = _encode_chat(tok, valid_rows[i % len(valid_rows)], args.seq)
             ids, labels = enc
             ys = torch.full_like(ids, -100)
             ys[:, :-1] = labels[:, 1:]

{streamft-0.2.3 → streamft-0.2.4}/streamft/fit.py

@@ -25,16 +25,16 @@ from pathlib import Path
 MEASURED = [
     # Matched on (num_hidden_layers, hidden_size), not on a rounded parameter count: a
     # different 7B with a different architecture must not inherit this measurement.
-    {"arch": (36, 2048), "label": "Qwen2.5-3B", "seq": 512, "blocks": 36,
+    {"arch": ("qwen2", 36, 2048, 11008, 16, 2, 151936), "label": "Qwen2.5-3B", "seq": 512, "blocks": 36,
      "peak_gb": 3.85, "step_s": 0.94, "mode": "all resident",
      "tokens": "SFT: one Dolly row per step, median ~134 tokens"},
-    {"arch": (28, 3584), "label": "Qwen2.5-7B", "seq": 512, "blocks": 28,
+    {"arch": ("qwen2", 28, 3584, 18944, 28, 4, 152064), "label": "Qwen2.5-7B", "seq": 512, "blocks": 28,
      "peak_gb": 5.51, "step_s": 1.55, "mode": "all resident",
      "tokens": "SFT: one Dolly row per step, median ~134 tokens"},
-    {"arch": (48, 5120), "label": "Qwen2.5-14B", "seq": 512, "blocks": 6,
+    {"arch": ("qwen2", 48, 5120, 13824, 40, 8, 152064), "label": "Qwen2.5-14B", "seq": 512, "blocks": 6,
      "peak_gb": 3.86, "step_s": 5.86, "mode": "streaming",
      "tokens": "SFT: one Dolly row per step, median ~134 tokens"},
-    {"arch": (64, 5120), "label": "Qwen2.5-32B", "seq": 1024, "blocks": 2,
+    {"arch": ("qwen2", 64, 5120, 27648, 40, 8, 152064), "label": "Qwen2.5-32B", "seq": 1024, "blocks": 2,
      "peak_gb": 3.90, "step_s": 12.48, "mode": "streaming",
      "tokens": "SFT: one Dolly row per step, median ~134 tokens"},
 ]
@@ -51,6 +51,19 @@ def load_config(model: str) -> dict:
     return json.loads(Path(hf_hub_download(model, "config.json")).read_text(encoding="utf-8"))
 
 
+def model_fingerprint(cfg: dict) -> tuple:
+    """Everything that changes the measurement: two 7B models are not interchangeable."""
+    return (
+        str(cfg.get("model_type", "")),
+        int(cfg.get("num_hidden_layers") or cfg.get("n_layer") or 0),
+        int(cfg.get("hidden_size") or cfg.get("n_embd") or 0),
+        int(cfg.get("intermediate_size") or 0),
+        int(cfg.get("num_attention_heads") or 0),
+        int(cfg.get("num_key_value_heads") or cfg.get("num_attention_heads") or 0),
+        int(cfg.get("vocab_size") or 0),
+    )
+
+
 def model_math(cfg: dict) -> dict:
     h = cfg.get("hidden_size") or cfg.get("n_embd")
     layers = cfg.get("num_hidden_layers") or cfg.get("n_layer")
@@ -83,12 +96,13 @@ def main(argv: list[str] | None = None) -> int:
     ap.add_argument("--json", action="store_true")
     args = ap.parse_args(argv)
 
-    m = model_math(load_config(args.model))
+    cfg = load_config(args.model)
+    m = model_math(cfg)
     need = m["nf4_gb"] + m["head_gb"] + RESERVE_GB
     fits_whole = need <= args.gpu
     size_b = round(m["params"] / 1e9)
 
-    arch = (m["layers"], m["hidden"])
+    arch = model_fingerprint(cfg)
     measured = None
     for row in MEASURED:
         if row["arch"] == arch and row["seq"] == args.seq:
@@ -126,7 +140,8 @@ def main(argv: list[str] | None = None) -> int:
     else:
         print("  verdict: needs streaming (one or more blocks on the GPU at a time)")
     if measured:
-        print(f"  measured reference: {measured['label']} (layers {arch[0]}, hidden {arch[1]}) "
+        print(f"  measured reference: {measured['label']} "
+              f"(model_type {arch[0]}, layers {arch[1]}, hidden {arch[2]}) "
               f"on RTX 3050 6 GB -> {measured['blocks']} block(s), "
               f"{measured['peak_gb']} GB peak, {measured['step_s']} s/step at seq {measured['seq']}")
         print(f"  note: {measured['tokens']} (seq is a cap on this path)")

{streamft-0.2.3 → streamft-0.2.4}/streamft.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: streamft
-Version: 0.2.3
+Version: 0.2.4
 Summary: Fine-tune models that do not fit: stream NF4 weights from disk, one transformer block at a time.
 Author-email: Gokhan Cava <byjameson@users.noreply.github.com>
 License: Apache-2.0
@@ -61,34 +61,34 @@ the six traps, and which numbers are real. The measurement protocol, the discard
 and the correction of an earlier wrong claim ("8.6× / 100 tok/s") are in
 [`LONGCTX_FINDINGS.md`](LONGCTX_FINDINGS.md).
 
-## Quick start
+## Quick start (installed package, nothing else needed)
 
-```powershell
+```bash
 pip install streamft
+streamft doctor                    # torch / CUDA / bitsandbytes / free VRAM
 
-# 0) will it fit? memory arithmetic + the measured reference, no weights downloaded
+# will it fit? memory arithmetic from config.json + the measured reference
 streamft fit --model Qwen/Qwen2.5-14B-Instruct --gpu 6 --seq 512
-streamft doctor            # torch / CUDA / bitsandbytes / free VRAM
 
-# 1) token shards for the LM path (SFT on Dolly needs none of this)
-streamft prepare --model models/Qwen2.5-0.5B --save-quant checkpoints/qwen0.5b_nf4
-python scripts/prep_data.py --model models/Qwen2.5-0.5B
+# one NF4 checkpoint per model (HF id or local directory)
+streamft prepare --model Qwen/Qwen2.5-0.5B --save-quant checkpoints/qwen0.5b_nf4
 
-# 1) turn a local bf16 model into a pre-quantized NF4 checkpoint (once)
-python -m streamft.cli --prepare --model models/Qwen2.5-32B-Instruct `
-    --save-quant checkpoints/qwen32b_nf4
+# SFT on Dolly-15k — no data prep
+streamft train --model-dir checkpoints/qwen0.5b_nf4 --sft --steps 20 --seq 512 --tag first
 
-# 2) fine-tune it, streaming from disk
-python tools/finetune.py --model checkpoints/qwen32b_nf4 --mode lm --seq 1024 --steps 50
-python tools/finetune.py --model checkpoints/qwen14b_nf4 --data dolly --seq 512 --steps 240 `
-    --save checkpoints/my_adapter
+# LM path (optional): build token shards, then train without --sft
+streamft data --model Qwen/Qwen2.5-0.5B --out data
+streamft train --model-dir checkpoints/qwen0.5b_nf4 --steps 20 --seq 256 --tag lm
+```
 
-# show the exact engine command without running it
-python tools/finetune.py ... --print-only
+Every run appends one JSON row to `results/results.jsonl` next to where you launched it.
+The same commands are in [`QUICKSTART.md`](QUICKSTART.md), which is what a pip user should
+read — this README also documents the experiments behind the numbers.
 
-# CPU smoke tests (5 cases, no GPU, no downloads)
-pip install -r requirements-dev.txt; python -m pytest tests -q
-```
+**From a source checkout only** (not part of the wheel): `tools/finetune.py` is a wrapper
+that encodes the VRAM presets, `scripts/compare_peft.py` reproduces the engine comparison,
+`scripts/verify_gradients.py` the gradient check, `scripts/prep_data.py` the token shards
+(the installed equivalent is `streamft data`), and `pytest tests -q` the CPU suite.
 
 `tools/finetune.py` encodes the VRAM presets we measured, because the *combination* of
 flags decides whether a run fits — not any single flag: