streamft 0.2.2__tar.gz → 0.2.4__tar.gz

{streamft-0.2.2 → streamft-0.2.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: streamft
-Version: 0.2.2
+Version: 0.2.4
 Summary: Fine-tune models that do not fit: stream NF4 weights from disk, one transformer block at a time.
 Author-email: Gokhan Cava <byjameson@users.noreply.github.com>
 License: Apache-2.0
@@ -61,34 +61,34 @@ the six traps, and which numbers are real. The measurement protocol, the discard
 and the correction of an earlier wrong claim ("8.6× / 100 tok/s") are in
 [`LONGCTX_FINDINGS.md`](LONGCTX_FINDINGS.md).
 
-## Quick start
+## Quick start (installed package, nothing else needed)
 
-```powershell
+```bash
 pip install streamft
+streamft doctor                    # torch / CUDA / bitsandbytes / free VRAM
 
-# 0) will it fit? memory arithmetic + the measured reference, no weights downloaded
+# will it fit? memory arithmetic from config.json + the measured reference
 streamft fit --model Qwen/Qwen2.5-14B-Instruct --gpu 6 --seq 512
-streamft doctor            # torch / CUDA / bitsandbytes / free VRAM
 
-# 1) token shards for the LM path (SFT on Dolly needs none of this)
-streamft prepare --model models/Qwen2.5-0.5B --save-quant checkpoints/qwen0.5b_nf4
-python scripts/prep_data.py --model models/Qwen2.5-0.5B
+# one NF4 checkpoint per model (HF id or local directory)
+streamft prepare --model Qwen/Qwen2.5-0.5B --save-quant checkpoints/qwen0.5b_nf4
 
-# 1) turn a local bf16 model into a pre-quantized NF4 checkpoint (once)
-python -m streamft.cli --prepare --model models/Qwen2.5-32B-Instruct `
-    --save-quant checkpoints/qwen32b_nf4
+# SFT on Dolly-15k — no data prep
+streamft train --model-dir checkpoints/qwen0.5b_nf4 --sft --steps 20 --seq 512 --tag first
 
-# 2) fine-tune it, streaming from disk
-python tools/finetune.py --model checkpoints/qwen32b_nf4 --mode lm --seq 1024 --steps 50
-python tools/finetune.py --model checkpoints/qwen14b_nf4 --data dolly --seq 512 --steps 240 `
-    --save checkpoints/my_adapter
+# LM path (optional): build token shards, then train without --sft
+streamft data --model Qwen/Qwen2.5-0.5B --out data
+streamft train --model-dir checkpoints/qwen0.5b_nf4 --steps 20 --seq 256 --tag lm
+```
 
-# show the exact engine command without running it
-python tools/finetune.py ... --print-only
+Every run appends one JSON row to `results/results.jsonl` next to where you launched it.
+The same commands are in [`QUICKSTART.md`](QUICKSTART.md), which is what a pip user should
+read — this README also documents the experiments behind the numbers.
 
-# CPU smoke tests (5 cases, no GPU, no downloads)
-pip install -r requirements-dev.txt; python -m pytest tests -q
-```
+**From a source checkout only** (not part of the wheel): `tools/finetune.py` is a wrapper
+that encodes the VRAM presets, `scripts/compare_peft.py` reproduces the engine comparison,
+`scripts/verify_gradients.py` the gradient check, `scripts/prep_data.py` the token shards
+(the installed equivalent is `streamft data`), and `pytest tests -q` the CPU suite.
 
 `tools/finetune.py` encodes the VRAM presets we measured, because the *combination* of
 flags decides whether a run fits — not any single flag:

{streamft-0.2.2 → streamft-0.2.4}/README.md

@@ -31,34 +31,34 @@ the six traps, and which numbers are real. The measurement protocol, the discard
 and the correction of an earlier wrong claim ("8.6× / 100 tok/s") are in
 [`LONGCTX_FINDINGS.md`](LONGCTX_FINDINGS.md).
 
-## Quick start
+## Quick start (installed package, nothing else needed)
 
-```powershell
+```bash
 pip install streamft
+streamft doctor                    # torch / CUDA / bitsandbytes / free VRAM
 
-# 0) will it fit? memory arithmetic + the measured reference, no weights downloaded
+# will it fit? memory arithmetic from config.json + the measured reference
 streamft fit --model Qwen/Qwen2.5-14B-Instruct --gpu 6 --seq 512
-streamft doctor            # torch / CUDA / bitsandbytes / free VRAM
 
-# 1) token shards for the LM path (SFT on Dolly needs none of this)
-streamft prepare --model models/Qwen2.5-0.5B --save-quant checkpoints/qwen0.5b_nf4
-python scripts/prep_data.py --model models/Qwen2.5-0.5B
+# one NF4 checkpoint per model (HF id or local directory)
+streamft prepare --model Qwen/Qwen2.5-0.5B --save-quant checkpoints/qwen0.5b_nf4
 
-# 1) turn a local bf16 model into a pre-quantized NF4 checkpoint (once)
-python -m streamft.cli --prepare --model models/Qwen2.5-32B-Instruct `
-    --save-quant checkpoints/qwen32b_nf4
+# SFT on Dolly-15k — no data prep
+streamft train --model-dir checkpoints/qwen0.5b_nf4 --sft --steps 20 --seq 512 --tag first
 
-# 2) fine-tune it, streaming from disk
-python tools/finetune.py --model checkpoints/qwen32b_nf4 --mode lm --seq 1024 --steps 50
-python tools/finetune.py --model checkpoints/qwen14b_nf4 --data dolly --seq 512 --steps 240 `
-    --save checkpoints/my_adapter
+# LM path (optional): build token shards, then train without --sft
+streamft data --model Qwen/Qwen2.5-0.5B --out data
+streamft train --model-dir checkpoints/qwen0.5b_nf4 --steps 20 --seq 256 --tag lm
+```
 
-# show the exact engine command without running it
-python tools/finetune.py ... --print-only
+Every run appends one JSON row to `results/results.jsonl` next to where you launched it.
+The same commands are in [`QUICKSTART.md`](QUICKSTART.md), which is what a pip user should
+read — this README also documents the experiments behind the numbers.
 
-# CPU smoke tests (5 cases, no GPU, no downloads)
-pip install -r requirements-dev.txt; python -m pytest tests -q
-```
+**From a source checkout only** (not part of the wheel): `tools/finetune.py` is a wrapper
+that encodes the VRAM presets, `scripts/compare_peft.py` reproduces the engine comparison,
+`scripts/verify_gradients.py` the gradient check, `scripts/prep_data.py` the token shards
+(the installed equivalent is `streamft data`), and `pytest tests -q` the CPU suite.
 
 `tools/finetune.py` encodes the VRAM presets we measured, because the *combination* of
 flags decides whether a run fits — not any single flag:

{streamft-0.2.2 → streamft-0.2.4}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "streamft"
-version = "0.2.2"
+version = "0.2.4"
 description = "Fine-tune models that do not fit: stream NF4 weights from disk, one transformer block at a time."
 readme = "README.md"
 requires-python = ">=3.10"

{streamft-0.2.2 → streamft-0.2.4}/streamft/cli.py

@@ -86,11 +86,23 @@ def user_path(value) -> Path:
     return q if q.is_absolute() else Path.cwd() / q
 
 
+def resolve_model(value: str) -> str:
+    """A local directory if one exists at that path, otherwise an HF model id.
+
+    `user_path` must not be used for models: `Qwen/Qwen2.5-0.5B` is an id, not
+    `./Qwen/Qwen2.5-0.5B`. Absolute paths and existing directories win; everything else
+    is passed through to transformers/huggingface_hub unchanged.
+    """
+    p = Path(value).expanduser()
+    local = p if p.is_absolute() else Path.cwd() / p
+    return str(local) if local.exists() else value
+
+
 def prepare(model: str, save_quant: str) -> int:
     """Quantize a local bf16 model into a bnb-4bit checkpoint (uint8 + qstate)."""
     from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
 
-    src = str(user_path(model))
+    src = resolve_model(model)
     dst = user_path(save_quant)
     print(f"[prepare] {model} -> {save_quant}", flush=True)
     bnb = BitsAndBytesConfig(
@@ -131,11 +143,13 @@ def _encode_chat(tokenizer, row: dict, seq: int):
                                                 add_generation_prompt=True)
     prompt = tokenizer(prompt_text, add_special_tokens=False).input_ids
     full = full[:seq]
-    labels = list(full)
-    if 0 < len(prompt) < len(full) and full[:len(prompt)] == prompt[:len(full)]:
-        k = min(len(prompt), len(full))
-        labels = [-100] * k + labels[k:]
-    if len(labels) < 2:
+    # The prompt must be an exact prefix of the templated row. If it is not, the mask
+    # would silently cover nothing and the model would be trained on its own prompt, so
+    # the example is dropped instead (and counted by the caller).
+    if len(prompt) >= len(full) or full[: len(prompt)] != prompt:
+        return None
+    labels = [-100] * len(prompt) + list(full[len(prompt):])
+    if len(labels) < 2 or all(v == -100 for v in labels):
         return None
     return (torch.tensor([full], dtype=torch.long),
             torch.tensor([labels], dtype=torch.long))
@@ -145,10 +159,11 @@ def _encode_chat(tokenizer, row: dict, seq: int):
 def _masked_eval(trainer, rows, tokenizer, seq: int) -> float:
     """Masked SFT loss through the streaming (disk-backed) forward."""
     trainer.model.eval()
-    total, n = 0.0, 0
+    total, n, skipped = 0.0, 0, 0
     for row in rows:
         enc = _encode_chat(tokenizer, row, seq)
         if enc is None:
+            skipped += 1
             continue
         ids, labels = enc
         ys = torch.full_like(ids, -100)
@@ -166,7 +181,15 @@ def _masked_eval(trainer, rows, tokenizer, seq: int) -> float:
             tgt[valid].to(logits.device), reduction="sum"))
         n += v
     trainer.model.train()
-    return total / max(n, 1)
+    if skipped:
+        print(f"[eval] skipped {skipped}/{len(rows)} rows: prompt is not an exact prefix "
+              f"of the templated row (would train on the prompt)", flush=True)
+    if n == 0:
+        raise RuntimeError(
+            "_masked_eval: every row was skipped (no valid prompt mask), so there is no "
+            "loss to report. Returning 0.0 here would look like a perfect model."
+        )
+    return total / n
 
 
 def main() -> int:
@@ -222,7 +245,7 @@ def main() -> int:
     if args.prepare:
         return prepare(args.model, args.save_quant)
 
-    model_dir = str(user_path(args.model_dir))
+    model_dir = resolve_model(args.model_dir)
     print(f"[exp54] model_dir={Path(model_dir).name} steps={args.steps} seq={args.seq} "
           f"lora_r={args.lora_r} targets={args.targets} head={args.head}", flush=True)
 
@@ -351,10 +374,17 @@ def main() -> int:
               f"eval={len(eval_rows)}", flush=True)
         ce0 = _masked_eval(tr, eval_rows, tok, args.seq)
         print(f"[eval] before loss={ce0:.4f}", flush=True)
+        # Filter once so `--steps N` means N optimizer updates, not "N attempts minus
+        # whatever the mask rejected". A silently short run is a lie about the budget.
+        valid_rows = [r for r in train_rows if _encode_chat(tok, r, args.seq) is not None]
+        skipped_train = len(train_rows) - len(valid_rows)
+        if not valid_rows:
+            raise SystemExit("[sft] no training row has a valid prompt mask; refusing to run")
+        if skipped_train:
+            print(f"[sft] {skipped_train}/{len(train_rows)} rows dropped: the prompt is not "
+                  f"an exact prefix of the templated row", flush=True)
         for i in range(args.steps):
-            enc = _encode_chat(tok, train_rows[i % len(train_rows)], args.seq)
-            if enc is None:
-                continue
+            enc = _encode_chat(tok, valid_rows[i % len(valid_rows)], args.seq)
             ids, labels = enc
             ys = torch.full_like(ids, -100)
             ys[:, :-1] = labels[:, 1:]
@@ -435,6 +465,7 @@ def main() -> int:
         "adapter_sha_before": sha_before,
         "adapter_sha_after": adapter_sha(model),
         "steps": len(losses),
+        "sft_skipped": skipped_train if args.sft else 0,
         "seq": args.seq,
         "lr": args.lr,
         "losses": losses,

{streamft-0.2.2 → streamft-0.2.4}/streamft/fit.py

@@ -22,12 +22,22 @@ from pathlib import Path
 
 # Measured end-to-end on an RTX 3050 Laptop, 6 GB, Windows 11, AC power, NF4 base,
 # LoRA r=8 on q/k/v/o, batch 1, Dolly-15k. step_s = median warm step.
-MEASURED = {
-    (3, 512): {"blocks": 36, "peak_gb": 3.85, "step_s": 0.938, "mode": "all resident"},
-    (7, 512): {"blocks": 28, "peak_gb": 5.51, "step_s": 1.696, "mode": "all resident"},
-    (14, 512): {"blocks": 6, "peak_gb": 3.86, "step_s": 6.726, "mode": "streaming"},
-    (32, 1024): {"blocks": 2, "peak_gb": 3.90, "step_s": 12.541, "mode": "streaming"},
-}
+MEASURED = [
+    # Matched on (num_hidden_layers, hidden_size), not on a rounded parameter count: a
+    # different 7B with a different architecture must not inherit this measurement.
+    {"arch": ("qwen2", 36, 2048, 11008, 16, 2, 151936), "label": "Qwen2.5-3B", "seq": 512, "blocks": 36,
+     "peak_gb": 3.85, "step_s": 0.94, "mode": "all resident",
+     "tokens": "SFT: one Dolly row per step, median ~134 tokens"},
+    {"arch": ("qwen2", 28, 3584, 18944, 28, 4, 152064), "label": "Qwen2.5-7B", "seq": 512, "blocks": 28,
+     "peak_gb": 5.51, "step_s": 1.55, "mode": "all resident",
+     "tokens": "SFT: one Dolly row per step, median ~134 tokens"},
+    {"arch": ("qwen2", 48, 5120, 13824, 40, 8, 152064), "label": "Qwen2.5-14B", "seq": 512, "blocks": 6,
+     "peak_gb": 3.86, "step_s": 5.86, "mode": "streaming",
+     "tokens": "SFT: one Dolly row per step, median ~134 tokens"},
+    {"arch": ("qwen2", 64, 5120, 27648, 40, 8, 152064), "label": "Qwen2.5-32B", "seq": 1024, "blocks": 2,
+     "peak_gb": 3.90, "step_s": 12.48, "mode": "streaming",
+     "tokens": "SFT: one Dolly row per step, median ~134 tokens"},
+]
 
 RESERVE_GB = 0.9  # CUDA context + allocator fragmentation + the bf16 recompute buffers
 
@@ -41,6 +51,19 @@ def load_config(model: str) -> dict:
     return json.loads(Path(hf_hub_download(model, "config.json")).read_text(encoding="utf-8"))
 
 
+def model_fingerprint(cfg: dict) -> tuple:
+    """Everything that changes the measurement: two 7B models are not interchangeable."""
+    return (
+        str(cfg.get("model_type", "")),
+        int(cfg.get("num_hidden_layers") or cfg.get("n_layer") or 0),
+        int(cfg.get("hidden_size") or cfg.get("n_embd") or 0),
+        int(cfg.get("intermediate_size") or 0),
+        int(cfg.get("num_attention_heads") or 0),
+        int(cfg.get("num_key_value_heads") or cfg.get("num_attention_heads") or 0),
+        int(cfg.get("vocab_size") or 0),
+    )
+
+
 def model_math(cfg: dict) -> dict:
     h = cfg.get("hidden_size") or cfg.get("n_embd")
     layers = cfg.get("num_hidden_layers") or cfg.get("n_layer")
@@ -62,7 +85,7 @@ def model_math(cfg: dict) -> dict:
     # presets use --shared-dtype as-stored. A tied head has no separate tensor.
     head_gb = 0.0 if tied else vocab * h * 2 / 1e9
     return {"params": params, "layers": int(layers), "nf4_gb": nf4_gb,
-            "head_gb": head_gb, "tied": tied}
+            "head_gb": head_gb, "tied": tied, "hidden": int(h)}
 
 
 def main(argv: list[str] | None = None) -> int:
@@ -73,15 +96,17 @@ def main(argv: list[str] | None = None) -> int:
     ap.add_argument("--json", action="store_true")
     args = ap.parse_args(argv)
 
-    m = model_math(load_config(args.model))
+    cfg = load_config(args.model)
+    m = model_math(cfg)
     need = m["nf4_gb"] + m["head_gb"] + RESERVE_GB
     fits_whole = need <= args.gpu
     size_b = round(m["params"] / 1e9)
 
+    arch = model_fingerprint(cfg)
     measured = None
-    for (b, seq), row in MEASURED.items():
-        if abs(size_b - b) <= max(1, round(b * 0.1)) and seq == args.seq:
-            measured = dict(row, ref_size_b=b, ref_seq=seq)
+    for row in MEASURED:
+        if row["arch"] == arch and row["seq"] == args.seq:
+            measured = dict(row)
             break
 
     out = {
@@ -115,8 +140,11 @@ def main(argv: list[str] | None = None) -> int:
     else:
         print("  verdict: needs streaming (one or more blocks on the GPU at a time)")
     if measured:
-        print(f"  measured on RTX 3050 6 GB: {measured['blocks']} block(s), "
-              f"{measured['peak_gb']} GB peak, {measured['step_s']} s/step at seq {measured['ref_seq']}")
+        print(f"  measured reference: {measured['label']} "
+              f"(model_type {arch[0]}, layers {arch[1]}, hidden {arch[2]}) "
+              f"on RTX 3050 6 GB -> {measured['blocks']} block(s), "
+              f"{measured['peak_gb']} GB peak, {measured['step_s']} s/step at seq {measured['seq']}")
+        print(f"  note: {measured['tokens']} (seq is a cap on this path)")
     else:
         print("  speed/blocks: not measured for this size at this seq — start with")
         print("                 --blocks-on-gpu 1 and let autotune grow it (it measures")

{streamft-0.2.2 → streamft-0.2.4}/streamft.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: streamft
-Version: 0.2.2
+Version: 0.2.4
 Summary: Fine-tune models that do not fit: stream NF4 weights from disk, one transformer block at a time.
 Author-email: Gokhan Cava <byjameson@users.noreply.github.com>
 License: Apache-2.0
@@ -61,34 +61,34 @@ the six traps, and which numbers are real. The measurement protocol, the discard
 and the correction of an earlier wrong claim ("8.6× / 100 tok/s") are in
 [`LONGCTX_FINDINGS.md`](LONGCTX_FINDINGS.md).
 
-## Quick start
+## Quick start (installed package, nothing else needed)
 
-```powershell
+```bash
 pip install streamft
+streamft doctor                    # torch / CUDA / bitsandbytes / free VRAM
 
-# 0) will it fit? memory arithmetic + the measured reference, no weights downloaded
+# will it fit? memory arithmetic from config.json + the measured reference
 streamft fit --model Qwen/Qwen2.5-14B-Instruct --gpu 6 --seq 512
-streamft doctor            # torch / CUDA / bitsandbytes / free VRAM
 
-# 1) token shards for the LM path (SFT on Dolly needs none of this)
-streamft prepare --model models/Qwen2.5-0.5B --save-quant checkpoints/qwen0.5b_nf4
-python scripts/prep_data.py --model models/Qwen2.5-0.5B
+# one NF4 checkpoint per model (HF id or local directory)
+streamft prepare --model Qwen/Qwen2.5-0.5B --save-quant checkpoints/qwen0.5b_nf4
 
-# 1) turn a local bf16 model into a pre-quantized NF4 checkpoint (once)
-python -m streamft.cli --prepare --model models/Qwen2.5-32B-Instruct `
-    --save-quant checkpoints/qwen32b_nf4
+# SFT on Dolly-15k — no data prep
+streamft train --model-dir checkpoints/qwen0.5b_nf4 --sft --steps 20 --seq 512 --tag first
 
-# 2) fine-tune it, streaming from disk
-python tools/finetune.py --model checkpoints/qwen32b_nf4 --mode lm --seq 1024 --steps 50
-python tools/finetune.py --model checkpoints/qwen14b_nf4 --data dolly --seq 512 --steps 240 `
-    --save checkpoints/my_adapter
+# LM path (optional): build token shards, then train without --sft
+streamft data --model Qwen/Qwen2.5-0.5B --out data
+streamft train --model-dir checkpoints/qwen0.5b_nf4 --steps 20 --seq 256 --tag lm
+```
 
-# show the exact engine command without running it
-python tools/finetune.py ... --print-only
+Every run appends one JSON row to `results/results.jsonl` next to where you launched it.
+The same commands are in [`QUICKSTART.md`](QUICKSTART.md), which is what a pip user should
+read — this README also documents the experiments behind the numbers.
 
-# CPU smoke tests (5 cases, no GPU, no downloads)
-pip install -r requirements-dev.txt; python -m pytest tests -q
-```
+**From a source checkout only** (not part of the wheel): `tools/finetune.py` is a wrapper
+that encodes the VRAM presets, `scripts/compare_peft.py` reproduces the engine comparison,
+`scripts/verify_gradients.py` the gradient check, `scripts/prep_data.py` the token shards
+(the installed equivalent is `streamft data`), and `pytest tests -q` the CPU suite.
 
 `tools/finetune.py` encodes the VRAM presets we measured, because the *combination* of
 flags decides whether a run fits — not any single flag:

{streamft-0.2.2 → streamft-0.2.4}/tests/test_streamft_smoke.py

@@ -177,3 +177,35 @@ def test_disk_store_supports_single_safetensors_file(tmp_path):
     store = DiskQuantStore(str(tmp_path))
     assert store.has("a") and store.has("b.c")
     assert store.get("b.c").tolist() == [1.0, 1.0, 1.0]
+
+
+# ------------------------------------------------ SFT masking failures are loud
+
+class _StubTokenizer:
+    """apply_chat_template returns FULL for both messages, PROMPT for the prefix call."""
+
+    def __init__(self, full, prompt):
+        self.full, self.prompt = full, prompt
+
+    def apply_chat_template(self, messages, tokenize=False, add_generation_prompt=False):
+        return "FULL" if len(messages) == 2 else "PROMPT"
+
+    def __call__(self, text, add_special_tokens=False):
+        from types import SimpleNamespace
+
+        ids = self.full if text == "FULL" else self.prompt
+        return SimpleNamespace(input_ids=list(ids))
+
+
+def test_encode_chat_drops_rows_whose_prompt_is_not_a_prefix():
+    from streamft.cli import _encode_chat
+
+    row = {"instruction": "q", "response": "a"}
+    # prompt is not a prefix of the templated row -> the example must be dropped
+    assert _encode_chat(_StubTokenizer([1, 2, 3, 4], [9, 9]), row, 32) is None
+    # prompt longer than the row -> dropped
+    assert _encode_chat(_StubTokenizer([1, 2], [1, 2, 3]), row, 32) is None
+    # valid prefix -> every prompt token masked, the rest kept
+    ids, labels = _encode_chat(_StubTokenizer([1, 2, 3, 4], [1, 2]), row, 32)
+    assert ids.tolist() == [[1, 2, 3, 4]]
+    assert labels.tolist() == [[-100, -100, 3, 4]]