streamft 0.5.0__tar.gz → 0.6.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (30) hide show
  1. {streamft-0.5.0 → streamft-0.6.1}/PKG-INFO +49 -3
  2. {streamft-0.5.0 → streamft-0.6.1}/README.md +48 -2
  3. {streamft-0.5.0 → streamft-0.6.1}/pyproject.toml +1 -1
  4. {streamft-0.5.0 → streamft-0.6.1}/streamft/__main__.py +6 -0
  5. streamft-0.6.1/streamft/adapter.py +150 -0
  6. {streamft-0.5.0 → streamft-0.6.1}/streamft/lora.py +28 -9
  7. {streamft-0.5.0 → streamft-0.6.1}/streamft/serve.py +11 -3
  8. {streamft-0.5.0 → streamft-0.6.1}/streamft.egg-info/PKG-INFO +49 -3
  9. {streamft-0.5.0 → streamft-0.6.1}/streamft.egg-info/SOURCES.txt +2 -0
  10. streamft-0.6.1/tests/test_adapter_quant.py +84 -0
  11. {streamft-0.5.0 → streamft-0.6.1}/LICENSE +0 -0
  12. {streamft-0.5.0 → streamft-0.6.1}/setup.cfg +0 -0
  13. {streamft-0.5.0 → streamft-0.6.1}/streamft/__init__.py +0 -0
  14. {streamft-0.5.0 → streamft-0.6.1}/streamft/autotune.py +0 -0
  15. {streamft-0.5.0 → streamft-0.6.1}/streamft/cli.py +0 -0
  16. {streamft-0.5.0 → streamft-0.6.1}/streamft/data.py +0 -0
  17. {streamft-0.5.0 → streamft-0.6.1}/streamft/disk_quant_store.py +0 -0
  18. {streamft-0.5.0 → streamft-0.6.1}/streamft/doctor.py +0 -0
  19. {streamft-0.5.0 → streamft-0.6.1}/streamft/easy.py +0 -0
  20. {streamft-0.5.0 → streamft-0.6.1}/streamft/evals.py +0 -0
  21. {streamft-0.5.0 → streamft-0.6.1}/streamft/fit.py +0 -0
  22. {streamft-0.5.0 → streamft-0.6.1}/streamft/merge.py +0 -0
  23. {streamft-0.5.0 → streamft-0.6.1}/streamft/metrics.py +0 -0
  24. {streamft-0.5.0 → streamft-0.6.1}/streamft/stream_quant_lora.py +0 -0
  25. {streamft-0.5.0 → streamft-0.6.1}/streamft/stream_train.py +0 -0
  26. {streamft-0.5.0 → streamft-0.6.1}/streamft.egg-info/dependency_links.txt +0 -0
  27. {streamft-0.5.0 → streamft-0.6.1}/streamft.egg-info/entry_points.txt +0 -0
  28. {streamft-0.5.0 → streamft-0.6.1}/streamft.egg-info/requires.txt +0 -0
  29. {streamft-0.5.0 → streamft-0.6.1}/streamft.egg-info/top_level.txt +0 -0
  30. {streamft-0.5.0 → streamft-0.6.1}/tests/test_streamft_smoke.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: streamft
3
- Version: 0.5.0
3
+ Version: 0.6.1
4
4
  Summary: Fine-tune models that do not fit: stream NF4 weights from disk, one transformer block at a time.
5
5
  Author-email: Gokhan Cava <byjameson@users.noreply.github.com>
6
6
  License: Apache-2.0
@@ -150,8 +150,9 @@ streamft train --model-dir checkpoints/qwen7b_nf4 --sft --steps 1000 \
150
150
  - On the reference Ampere card fp16 is slightly faster than bf16 at 7B (1.79 vs 1.91 s/step,
151
151
  same 5.36 GB peak); on Pascal it is the only option.
152
152
  - On a **4 GB** card use `--blocks-on-gpu 1`, `--seq 512` and `--vram-hard-gb 3.4`.
153
- Those rows are not measured yet — if you run one, post the JSON row and it can become a
154
- measured entry in `streamft fit`.
153
+ Those rows are not measured yet — if you run one, open a
154
+ [hardware report](.github/ISSUE_TEMPLATE/hardware-report.md) and it can become a measured
155
+ entry in `streamft fit`.
155
156
  - `streamft fit` prints this hint automatically when it sees a pre-Ampere GPU.
156
157
 
157
158
  ### Tokenizer cost and the diffusion question
@@ -191,6 +192,41 @@ tools can point at it. This is **one machine computing, the rest as clients** ?
191
192
  peer-to-peer sharding and not volunteer compute. Tested end to end: `/health` and a
192
193
  16-token completion (1.9 s on the reference GPU).
193
194
 
195
+ ### Share one base, ship 5 MB adapters
196
+
197
+ A LoRA adapter is the per-behaviour artifact. Pack it to 4 bits and serve it on top of a
198
+ shared base:
199
+
200
+ ```bash
201
+ streamft adapter --adapter checkpoints/my_adapter.pt --bits 4 --out checkpoints/my_adapter_q4.pt
202
+ streamft serve --model checkpoints/qwen3b_bnb4bit --adapter checkpoints/my_adapter_q4.pt
203
+ ```
204
+
205
+ Measured on the reference machine (Qwen2.5-3B, 60-step Dolly LoRA):
206
+
207
+ | storage | size | bits / base-param |
208
+ |---|---:|---:|
209
+ | fp32 | 29.6 MB | 0.0766 |
210
+ | fp16 | 14.8 MB | 0.0382 |
211
+ | 8-bit | 8.56 MB | 0.0222 |
212
+ | **4-bit** | **4.88 MB** | **0.0126** |
213
+
214
+ 870 held-out Dolly rows, masked SFT loss: base 2.3752, original 2.3403, 8-bit 2.3401
215
+ (p = 0.81, no measurable loss), 4-bit 2.3420 (+0.0018, p = 0.001, 5% of the gain). A
216
+ second task, SST-2 (2,000 train / 400 test), goes from **71.25%** base to **93.75%**, and
217
+ both packed adapters stay at **93.75%** (masked loss 0.7940/0.7942 vs 0.7930; p = 0.50 /
218
+ 0.41). A third task, AG News (4-way), goes from 2.50% with the bare prompt (not followed)
219
+ to **62.25%**; 8-bit and 4-bit stay at **62.00%**. The adapter loads in 0.115 s. The LoRA
220
+ path is ~26% slower than the base on this card (8.75 vs 11.79 tok/s; a bf16 LoRA path
221
+ reaches 9.13); merge it into a bf16 base for zero per-token overhead, or accept the LoRA
222
+ path when storage/switching matters more than throughput.
223
+ Rank reduction is not viable: SVD to rank 8 loses 35.8% of the update.
224
+
225
+ **A prompt can be the better rung.** On the same base, a 441-byte 4-shot SST-2 prompt
226
+ scores 94.0% (adapter 93.75%) and a 242-byte 4-shot AG News prompt scores 61.75% (adapter
227
+ 62.25%). If a few examples express the task, use the prompt; the adapter is for behaviour
228
+ a prompt cannot express, or when prompt tokens/context cost more than the storage.
229
+
194
230
  ### Measured on
195
231
 
196
232
  Every speed number here comes from **one machine**: RTX 3050 Laptop 6 GB, Windows 11,
@@ -207,6 +243,16 @@ derives the architecture fingerprint and prints the snippets for `fit.py`, `fitc
207
243
  and the Space. Anything not in that table prints **"not measured"** in `streamft fit` ?
208
244
  that is the point.
209
245
 
246
+ ### Why 0.1-bit compression is not a universal target
247
+
248
+ A measured note on quantization vs distillation: post-hoc quantization dies below 3 bits;
249
+ a 340 KB student matches a 1.5B teacher on a narrow keyword task at 0.0019
250
+ bit/teacher-param; the same 4.4M student is within 1.3 points on DBPedia-14 but loses
251
+ 22.5 on SST-2 and 47.9 on MNLI; open-ended generation dies at 3 bits. A cheap 45 KB TF-IDF
252
+ student is a **lower bound** on the 4.4M student's normalized skill (6 tasks, R² = 0.89),
253
+ not a predictor: QQP (C 0.21, S 0.49) breaks the near-identity. Details, caveats and
254
+ reproduction: [`docs/BEHAVIORAL_BITS.md`](docs/BEHAVIORAL_BITS.md).
255
+
210
256
  ## Verification
211
257
 
212
258
  `python scripts/verify_gradients.py --model-dir checkpoints/qwen3b_bnb4bit --seq 128`
@@ -120,8 +120,9 @@ streamft train --model-dir checkpoints/qwen7b_nf4 --sft --steps 1000 \
120
120
  - On the reference Ampere card fp16 is slightly faster than bf16 at 7B (1.79 vs 1.91 s/step,
121
121
  same 5.36 GB peak); on Pascal it is the only option.
122
122
  - On a **4 GB** card use `--blocks-on-gpu 1`, `--seq 512` and `--vram-hard-gb 3.4`.
123
- Those rows are not measured yet — if you run one, post the JSON row and it can become a
124
- measured entry in `streamft fit`.
123
+ Those rows are not measured yet — if you run one, open a
124
+ [hardware report](.github/ISSUE_TEMPLATE/hardware-report.md) and it can become a measured
125
+ entry in `streamft fit`.
125
126
  - `streamft fit` prints this hint automatically when it sees a pre-Ampere GPU.
126
127
 
127
128
  ### Tokenizer cost and the diffusion question
@@ -161,6 +162,41 @@ tools can point at it. This is **one machine computing, the rest as clients** ?
161
162
  peer-to-peer sharding and not volunteer compute. Tested end to end: `/health` and a
162
163
  16-token completion (1.9 s on the reference GPU).
163
164
 
165
+ ### Share one base, ship 5 MB adapters
166
+
167
+ A LoRA adapter is the per-behaviour artifact. Pack it to 4 bits and serve it on top of a
168
+ shared base:
169
+
170
+ ```bash
171
+ streamft adapter --adapter checkpoints/my_adapter.pt --bits 4 --out checkpoints/my_adapter_q4.pt
172
+ streamft serve --model checkpoints/qwen3b_bnb4bit --adapter checkpoints/my_adapter_q4.pt
173
+ ```
174
+
175
+ Measured on the reference machine (Qwen2.5-3B, 60-step Dolly LoRA):
176
+
177
+ | storage | size | bits / base-param |
178
+ |---|---:|---:|
179
+ | fp32 | 29.6 MB | 0.0766 |
180
+ | fp16 | 14.8 MB | 0.0382 |
181
+ | 8-bit | 8.56 MB | 0.0222 |
182
+ | **4-bit** | **4.88 MB** | **0.0126** |
183
+
184
+ 870 held-out Dolly rows, masked SFT loss: base 2.3752, original 2.3403, 8-bit 2.3401
185
+ (p = 0.81, no measurable loss), 4-bit 2.3420 (+0.0018, p = 0.001, 5% of the gain). A
186
+ second task, SST-2 (2,000 train / 400 test), goes from **71.25%** base to **93.75%**, and
187
+ both packed adapters stay at **93.75%** (masked loss 0.7940/0.7942 vs 0.7930; p = 0.50 /
188
+ 0.41). A third task, AG News (4-way), goes from 2.50% with the bare prompt (not followed)
189
+ to **62.25%**; 8-bit and 4-bit stay at **62.00%**. The adapter loads in 0.115 s. The LoRA
190
+ path is ~26% slower than the base on this card (8.75 vs 11.79 tok/s; a bf16 LoRA path
191
+ reaches 9.13); merge it into a bf16 base for zero per-token overhead, or accept the LoRA
192
+ path when storage/switching matters more than throughput.
193
+ Rank reduction is not viable: SVD to rank 8 loses 35.8% of the update.
194
+
195
+ **A prompt can be the better rung.** On the same base, a 441-byte 4-shot SST-2 prompt
196
+ scores 94.0% (adapter 93.75%) and a 242-byte 4-shot AG News prompt scores 61.75% (adapter
197
+ 62.25%). If a few examples express the task, use the prompt; the adapter is for behaviour
198
+ a prompt cannot express, or when prompt tokens/context cost more than the storage.
199
+
164
200
  ### Measured on
165
201
 
166
202
  Every speed number here comes from **one machine**: RTX 3050 Laptop 6 GB, Windows 11,
@@ -177,6 +213,16 @@ derives the architecture fingerprint and prints the snippets for `fit.py`, `fitc
177
213
  and the Space. Anything not in that table prints **"not measured"** in `streamft fit` ?
178
214
  that is the point.
179
215
 
216
+ ### Why 0.1-bit compression is not a universal target
217
+
218
+ A measured note on quantization vs distillation: post-hoc quantization dies below 3 bits;
219
+ a 340 KB student matches a 1.5B teacher on a narrow keyword task at 0.0019
220
+ bit/teacher-param; the same 4.4M student is within 1.3 points on DBPedia-14 but loses
221
+ 22.5 on SST-2 and 47.9 on MNLI; open-ended generation dies at 3 bits. A cheap 45 KB TF-IDF
222
+ student is a **lower bound** on the 4.4M student's normalized skill (6 tasks, R² = 0.89),
223
+ not a predictor: QQP (C 0.21, S 0.49) breaks the near-identity. Details, caveats and
224
+ reproduction: [`docs/BEHAVIORAL_BITS.md`](docs/BEHAVIORAL_BITS.md).
225
+
180
226
  ## Verification
181
227
 
182
228
  `python scripts/verify_gradients.py --model-dir checkpoints/qwen3b_bnb4bit --seq 128`
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "streamft"
7
- version = "0.5.0"
7
+ version = "0.6.1"
8
8
  description = "Fine-tune models that do not fit: stream NF4 weights from disk, one transformer block at a time."
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.10"
@@ -8,6 +8,7 @@
8
8
  streamft synth generate training pairs locally with a small model
9
9
  streamft eval run a small local prompt set (repetition/refusal/keywords)
10
10
  streamft merge fold a saved adapter into the base model (for serving / GGUF)
11
+ streamft adapter pack a saved adapter to 8/4-bit (shared-base deployment)
11
12
  streamft serve host a merged model on the LAN (chat page + /v1/chat/completions)
12
13
  streamft doctor environment report (torch, CUDA, bitsandbytes, VRAM)
13
14
  """
@@ -25,6 +26,7 @@ USAGE = """streamft - fine-tune models that do not fit
25
26
  streamft synth --model <small-model> --n 50 --out data/synth.jsonl
26
27
  streamft eval --model <model> [--prompts mine.jsonl]
27
28
  streamft merge --model <base> --adapter <dir> --out merged/<name>
29
+ streamft adapter --adapter <pt> --bits 4 --out <pt>
28
30
  streamft serve --model merged/<name> --port 8000
29
31
  streamft doctor
30
32
 
@@ -57,6 +59,10 @@ def main() -> int:
57
59
  from streamft.merge import main as merge_main
58
60
  return merge_main(rest)
59
61
 
62
+ if cmd == "adapter":
63
+ from streamft.adapter import main as adapter_main
64
+ return adapter_main(rest)
65
+
60
66
  if cmd == "eval":
61
67
  from streamft.evals import main as eval_main
62
68
  return eval_main(rest)
@@ -0,0 +1,150 @@
1
+ """Quantized LoRA adapter storage (8-bit / 4-bit, group-wise).
2
+
3
+ The adapter is the per-behaviour artifact in `docs/BEHAVIORAL_BITS.md`: a shared base
4
+ plus a few MB of adapter. This module stores the A/B tensors as packed integers with
5
+ per-group scales, and dequantizes them at load time. It is deliberately separate from the
6
+ trainer: the adapter format is a deployment format, not a training format.
7
+
8
+ streamft adapter --adapter checkpoints/my_adapter.pt --bits 4 --out q4.pt
9
+
10
+ The 4-bit file is smaller than fp32 but carries per-group scales; the real file size is
11
+ what counts, not the packed-code size alone.
12
+ """
13
+ from __future__ import annotations
14
+
15
+ import argparse
16
+ import json
17
+ from pathlib import Path
18
+
19
+ import torch
20
+
21
+ FORMAT = "streamft-adapter-q1"
22
+
23
+
24
+ def _quantize(w: torch.Tensor, bits: int, group: int):
25
+ flat = w.reshape(-1).float()
26
+ n = flat.numel()
27
+ pad = (-n) % group
28
+ if pad:
29
+ flat = torch.cat([flat, torch.zeros(pad)])
30
+ g = flat.view(-1, group)
31
+ lo = g.amin(1, keepdim=True)
32
+ hi = g.amax(1, keepdim=True)
33
+ levels = 2 ** bits - 1
34
+ scale = (hi - lo).clamp_min(1e-12) / levels
35
+ code = ((g - lo) / scale).round().clamp(0, levels).to(torch.uint8)
36
+ if bits == 8:
37
+ packed = code.reshape(-1)
38
+ elif bits == 4:
39
+ c = code.reshape(-1, 2)
40
+ packed = (c[:, 0] | (c[:, 1] << 4)).to(torch.uint8)
41
+ elif bits == 2:
42
+ c = code.reshape(-1, 4)
43
+ packed = (c[:, 0] | (c[:, 1] << 2) | (c[:, 2] << 4) | (c[:, 3] << 6)).to(torch.uint8)
44
+ else:
45
+ raise ValueError("bits must be 2, 4 or 8")
46
+ return {"packed": packed, "lo": lo.squeeze(1), "scale": scale.squeeze(1),
47
+ "n": n, "shape": list(w.shape)}
48
+
49
+
50
+ def _unpack(q: dict, bits: int) -> torch.Tensor:
51
+ p = q["packed"]
52
+ if bits == 8:
53
+ code = p
54
+ elif bits == 4:
55
+ code = torch.stack([p & 0xF, (p >> 4) & 0xF], dim=1).reshape(-1)
56
+ elif bits == 2:
57
+ code = torch.stack([p & 0x3, (p >> 2) & 0x3, (p >> 4) & 0x3, (p >> 6) & 0x3],
58
+ dim=1).reshape(-1)
59
+ else:
60
+ raise ValueError("bits must be 2, 4 or 8")
61
+ return code
62
+
63
+
64
+ def _dequantize(q: dict, bits: int) -> torch.Tensor:
65
+ # _unpack returns the padded code vector (groups * group); slice after dequantizing.
66
+ code = _unpack(q, bits).float()
67
+ group = int(q.get("group", q["scale"].numel()))
68
+ scale = q["scale"].float()
69
+ lo = q["lo"].float()
70
+ flat = code.view(-1, group)
71
+ out = flat * scale[:, None] + lo[:, None]
72
+ return out.reshape(-1)[: q["n"]].reshape(q["shape"])
73
+
74
+
75
+ def quantize_adapter(state: dict, bits: int, group: int = 64, meta: dict | None = None) -> dict:
76
+ tensors = {}
77
+ for name, w in state.items():
78
+ if name.endswith(".A") or name.endswith(".B"):
79
+ q = _quantize(w, bits, group)
80
+ q["group"] = group
81
+ tensors[name] = q
82
+ return {"format": FORMAT, "bits": bits, "group": group,
83
+ "meta": meta or {}, "tensors": tensors}
84
+
85
+
86
+ def dequantize_adapter(obj: dict) -> dict:
87
+ return {name: _dequantize(q, obj["bits"]) for name, q in obj["tensors"].items()}
88
+
89
+
90
+ def save_quantized(state: dict, path: str | Path, bits: int, group: int = 64,
91
+ meta: dict | None = None) -> dict:
92
+ obj = quantize_adapter(state, bits, group, meta)
93
+ torch.save(obj, str(path))
94
+ return obj
95
+
96
+
97
+ def load_adapter(path: str | Path):
98
+ """Load a plain .pt adapter ({"lora": ...}) or a quantized one; return (state, meta)."""
99
+ obj = torch.load(str(path), map_location="cpu", weights_only=True)
100
+ if isinstance(obj, dict) and obj.get("format") == FORMAT:
101
+ return dequantize_adapter(obj), obj.get("meta", {})
102
+ if isinstance(obj, dict) and "lora" in obj:
103
+ return obj["lora"], obj.get("meta", {})
104
+ if isinstance(obj, dict):
105
+ return obj, {}
106
+ raise ValueError(f"not an adapter: {path}")
107
+
108
+
109
+ def tensor_error(a: torch.Tensor, b: torch.Tensor) -> float:
110
+ return float((a.float() - b.float()).norm() / a.float().norm().clamp_min(1e-12))
111
+
112
+
113
+ def main(argv=None) -> int:
114
+ ap = argparse.ArgumentParser(prog="streamft adapter")
115
+ ap.add_argument("--adapter", required=True,
116
+ help="adapter .pt, or a save dir containing adapter.pt")
117
+ ap.add_argument("--bits", type=int, choices=[8, 4, 2], default=4,
118
+ help="8=safe, 4=default, 2=not behaviourally validated")
119
+ ap.add_argument("--group", type=int, default=64)
120
+ ap.add_argument("--out", required=True)
121
+ args = ap.parse_args(argv)
122
+ src = Path(args.adapter)
123
+ if src.is_dir():
124
+ src = src / "adapter.pt"
125
+ state, meta = load_adapter(src)
126
+ if not state:
127
+ raise SystemExit(f"no tensors in adapter: {src}")
128
+ lora_keys = {k for k in state if k.endswith(".A") or k.endswith(".B")}
129
+ if not lora_keys:
130
+ raise SystemExit(f"no LoRA A/B tensors in {src}; nothing to pack")
131
+ skipped = sorted(set(state) - lora_keys)
132
+ out = Path(args.out)
133
+ obj = save_quantized(state, out, args.bits, args.group, meta)
134
+ back = dequantize_adapter(obj)
135
+ errs = [tensor_error(state[k], back[k]) for k in back]
136
+ params = sum(v.numel() for v in state.values())
137
+ print(json.dumps({
138
+ "adapter": str(src), "out": str(out), "bits": args.bits, "group": args.group,
139
+ "params": params,
140
+ "src_bytes": src.stat().st_size, "out_bytes": out.stat().st_size,
141
+ "compression": round(src.stat().st_size / max(out.stat().st_size, 1), 2),
142
+ "max_relative_error": round(max(errs) if errs else 0.0, 4),
143
+ "tensors": len(back),
144
+ "skipped_tensors": skipped,
145
+ }, indent=1))
146
+ return 0
147
+
148
+
149
+ if __name__ == "__main__":
150
+ raise SystemExit(main())
@@ -26,21 +26,40 @@ class LoRALinear(nn.Module):
26
26
  self.r = r
27
27
  self.scale = alpha / r
28
28
  w = base.weight
29
- self.A = nn.Parameter(torch.randn(r, w.shape[1], device=w.device,
30
- dtype=torch.float32) / math.sqrt(w.shape[1]))
31
- self.B = nn.Parameter(torch.zeros(w.shape[0], r, device=w.device, dtype=torch.float32))
29
+ # Use in_features/out_features, not weight.shape: bitsandbytes 4-bit layers
30
+ # store a packed weight whose shape is not (out, in).
31
+ self.A = nn.Parameter(torch.randn(r, base.in_features, device=w.device,
32
+ dtype=torch.float32) / math.sqrt(base.in_features))
33
+ self.B = nn.Parameter(torch.zeros(base.out_features, r, device=w.device, dtype=torch.float32))
32
34
  self.dropout = nn.Dropout(dropout) if dropout > 0 else nn.Identity()
35
+ self.fast_dtype = None # inference-only bf16/fp16 path; training stays fp32
33
36
 
34
37
  def forward(self, x):
35
38
  y = self.base(x)
36
- h = self.dropout(x).float()
37
- # A/B live on CPU (grads are accumulated there and the optimizer steps
38
- # there); hop them to the activation's device for the matmul. The low-rank
39
- # path is tiny, so this costs a negligible amount of PCIe traffic.
40
- A = self.A if self.A.device == h.device else self.A.to(h.device)
41
- B = self.B if self.B.device == h.device else self.B.to(h.device)
39
+ if self.fast_dtype is not None:
40
+ h = x.to(self.fast_dtype)
41
+ A = self.A.to(device=h.device, dtype=self.fast_dtype)
42
+ B = self.B.to(device=h.device, dtype=self.fast_dtype)
43
+ else:
44
+ h = self.dropout(x).float()
45
+ # A/B live on CPU (grads are accumulated there and the optimizer steps
46
+ # there); hop them to the activation's device for the matmul. The low-rank
47
+ # path is tiny, so this costs a negligible amount of PCIe traffic.
48
+ A = self.A if self.A.device == h.device else self.A.to(h.device)
49
+ B = self.B if self.B.device == h.device else self.B.to(h.device)
42
50
  return y + (self.scale * (h @ A.t() @ B.t())).to(y.dtype)
43
51
 
52
+ @torch.no_grad()
53
+ def set_fast(self, dtype=torch.bfloat16):
54
+ """Inference-only: run the low-rank path in `dtype` instead of fp32.
55
+
56
+ The adapter is trained in fp32 (small values would vanish in fp16); this is a
57
+ deployment switch. The weights are rounded once, at load time.
58
+ """
59
+ self.A.data = self.A.data.to(dtype)
60
+ self.B.data = self.B.data.to(dtype)
61
+ self.fast_dtype = dtype
62
+
44
63
  @torch.no_grad()
45
64
  def merge(self):
46
65
  self.base.weight.data += (self.scale * (self.B @ self.A)).to(self.base.weight.dtype)
@@ -108,7 +108,8 @@ def main(argv=None) -> int:
108
108
 
109
109
  ap = argparse.ArgumentParser(prog="streamft serve")
110
110
  ap.add_argument("--model", required=True, help="merged model dir or HF id")
111
- ap.add_argument("--adapter", default="", help="optional adapter dir with adapter.pt")
111
+ ap.add_argument("--adapter", default="",
112
+ help="optional adapter .pt, or a save dir containing adapter.pt")
112
113
  ap.add_argument("--host", default="0.0.0.0")
113
114
  ap.add_argument("--port", type=int, default=8000)
114
115
  ap.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
@@ -121,14 +122,21 @@ def main(argv=None) -> int:
121
122
  if args.adapter:
122
123
  from pathlib import Path
123
124
 
124
- from .lora import apply_lora, set_lora_state_dict
125
+ from .adapter import load_adapter
126
+ from .lora import LoRALinear, apply_lora, set_lora_state_dict
125
127
 
126
128
  p = Path(args.adapter)
127
129
  p = p / "adapter.pt" if p.is_dir() else p
128
- state = torch.load(str(p), map_location="cpu", weights_only=True)
130
+ state, _meta = load_adapter(p)
129
131
  r = int(next(v for k, v in state.items() if k.endswith(".A")).shape[0])
130
132
  apply_lora(model, r_attn=r, r_mlp=0)
131
133
  set_lora_state_dict(model, state)
134
+ if dtype != torch.float32:
135
+ # Inference-only bf16 low-rank path: ~4% faster than fp32 on the reference
136
+ # card and it matches the base dtype; training still uses fp32.
137
+ for m in model.modules():
138
+ if isinstance(m, LoRALinear):
139
+ m.set_fast(dtype)
132
140
  print(f"[serve] adapter {p.name} (r={r})", flush=True)
133
141
 
134
142
  _state.update(model=model, tok=tok, device=args.device, model_id=args.model)
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: streamft
3
- Version: 0.5.0
3
+ Version: 0.6.1
4
4
  Summary: Fine-tune models that do not fit: stream NF4 weights from disk, one transformer block at a time.
5
5
  Author-email: Gokhan Cava <byjameson@users.noreply.github.com>
6
6
  License: Apache-2.0
@@ -150,8 +150,9 @@ streamft train --model-dir checkpoints/qwen7b_nf4 --sft --steps 1000 \
150
150
  - On the reference Ampere card fp16 is slightly faster than bf16 at 7B (1.79 vs 1.91 s/step,
151
151
  same 5.36 GB peak); on Pascal it is the only option.
152
152
  - On a **4 GB** card use `--blocks-on-gpu 1`, `--seq 512` and `--vram-hard-gb 3.4`.
153
- Those rows are not measured yet — if you run one, post the JSON row and it can become a
154
- measured entry in `streamft fit`.
153
+ Those rows are not measured yet — if you run one, open a
154
+ [hardware report](.github/ISSUE_TEMPLATE/hardware-report.md) and it can become a measured
155
+ entry in `streamft fit`.
155
156
  - `streamft fit` prints this hint automatically when it sees a pre-Ampere GPU.
156
157
 
157
158
  ### Tokenizer cost and the diffusion question
@@ -191,6 +192,41 @@ tools can point at it. This is **one machine computing, the rest as clients** ?
191
192
  peer-to-peer sharding and not volunteer compute. Tested end to end: `/health` and a
192
193
  16-token completion (1.9 s on the reference GPU).
193
194
 
195
+ ### Share one base, ship 5 MB adapters
196
+
197
+ A LoRA adapter is the per-behaviour artifact. Pack it to 4 bits and serve it on top of a
198
+ shared base:
199
+
200
+ ```bash
201
+ streamft adapter --adapter checkpoints/my_adapter.pt --bits 4 --out checkpoints/my_adapter_q4.pt
202
+ streamft serve --model checkpoints/qwen3b_bnb4bit --adapter checkpoints/my_adapter_q4.pt
203
+ ```
204
+
205
+ Measured on the reference machine (Qwen2.5-3B, 60-step Dolly LoRA):
206
+
207
+ | storage | size | bits / base-param |
208
+ |---|---:|---:|
209
+ | fp32 | 29.6 MB | 0.0766 |
210
+ | fp16 | 14.8 MB | 0.0382 |
211
+ | 8-bit | 8.56 MB | 0.0222 |
212
+ | **4-bit** | **4.88 MB** | **0.0126** |
213
+
214
+ 870 held-out Dolly rows, masked SFT loss: base 2.3752, original 2.3403, 8-bit 2.3401
215
+ (p = 0.81, no measurable loss), 4-bit 2.3420 (+0.0018, p = 0.001, 5% of the gain). A
216
+ second task, SST-2 (2,000 train / 400 test), goes from **71.25%** base to **93.75%**, and
217
+ both packed adapters stay at **93.75%** (masked loss 0.7940/0.7942 vs 0.7930; p = 0.50 /
218
+ 0.41). A third task, AG News (4-way), goes from 2.50% with the bare prompt (not followed)
219
+ to **62.25%**; 8-bit and 4-bit stay at **62.00%**. The adapter loads in 0.115 s. The LoRA
220
+ path is ~26% slower than the base on this card (8.75 vs 11.79 tok/s; a bf16 LoRA path
221
+ reaches 9.13); merge it into a bf16 base for zero per-token overhead, or accept the LoRA
222
+ path when storage/switching matters more than throughput.
223
+ Rank reduction is not viable: SVD to rank 8 loses 35.8% of the update.
224
+
225
+ **A prompt can be the better rung.** On the same base, a 441-byte 4-shot SST-2 prompt
226
+ scores 94.0% (adapter 93.75%) and a 242-byte 4-shot AG News prompt scores 61.75% (adapter
227
+ 62.25%). If a few examples express the task, use the prompt; the adapter is for behaviour
228
+ a prompt cannot express, or when prompt tokens/context cost more than the storage.
229
+
194
230
  ### Measured on
195
231
 
196
232
  Every speed number here comes from **one machine**: RTX 3050 Laptop 6 GB, Windows 11,
@@ -207,6 +243,16 @@ derives the architecture fingerprint and prints the snippets for `fit.py`, `fitc
207
243
  and the Space. Anything not in that table prints **"not measured"** in `streamft fit` ?
208
244
  that is the point.
209
245
 
246
+ ### Why 0.1-bit compression is not a universal target
247
+
248
+ A measured note on quantization vs distillation: post-hoc quantization dies below 3 bits;
249
+ a 340 KB student matches a 1.5B teacher on a narrow keyword task at 0.0019
250
+ bit/teacher-param; the same 4.4M student is within 1.3 points on DBPedia-14 but loses
251
+ 22.5 on SST-2 and 47.9 on MNLI; open-ended generation dies at 3 bits. A cheap 45 KB TF-IDF
252
+ student is a **lower bound** on the 4.4M student's normalized skill (6 tasks, R² = 0.89),
253
+ not a predictor: QQP (C 0.21, S 0.49) breaks the near-identity. Details, caveats and
254
+ reproduction: [`docs/BEHAVIORAL_BITS.md`](docs/BEHAVIORAL_BITS.md).
255
+
210
256
  ## Verification
211
257
 
212
258
  `python scripts/verify_gradients.py --model-dir checkpoints/qwen3b_bnb4bit --seq 128`
@@ -3,6 +3,7 @@ README.md
3
3
  pyproject.toml
4
4
  streamft/__init__.py
5
5
  streamft/__main__.py
6
+ streamft/adapter.py
6
7
  streamft/autotune.py
7
8
  streamft/cli.py
8
9
  streamft/data.py
@@ -23,4 +24,5 @@ streamft.egg-info/dependency_links.txt
23
24
  streamft.egg-info/entry_points.txt
24
25
  streamft.egg-info/requires.txt
25
26
  streamft.egg-info/top_level.txt
27
+ tests/test_adapter_quant.py
26
28
  tests/test_streamft_smoke.py
@@ -0,0 +1,84 @@
1
+ import torch
2
+ from torch import nn
3
+
4
+ from streamft.adapter import load_adapter, save_quantized, tensor_error
5
+ from streamft.lora import LoRALinear
6
+
7
+
8
+ def _state():
9
+ torch.manual_seed(0)
10
+ return {
11
+ "model.layers.0.self_attn.q_proj.A": torch.randn(8, 2048) * 0.02,
12
+ "model.layers.0.self_attn.q_proj.B": torch.randn(2048, 8) * 0.02,
13
+ }
14
+
15
+
16
+ def test_roundtrip_8bit(tmp_path):
17
+ state = _state()
18
+ path = tmp_path / "a8.pt"
19
+ save_quantized(state, path, 8, group=64)
20
+ back, meta = load_adapter(path)
21
+ assert set(back) == set(state)
22
+ for k in state:
23
+ assert back[k].shape == state[k].shape
24
+ assert tensor_error(state[k], back[k]) < 0.02
25
+ assert path.stat().st_size < sum(v.numel() for v in state.values()) * 4
26
+
27
+
28
+ def test_roundtrip_4bit(tmp_path):
29
+ state = _state()
30
+ path = tmp_path / "a4.pt"
31
+ save_quantized(state, path, 4, group=64)
32
+ back, _ = load_adapter(path)
33
+ for k in state:
34
+ assert back[k].shape == state[k].shape
35
+ assert tensor_error(state[k], back[k]) < 0.15
36
+ assert path.stat().st_size < sum(v.numel() for v in state.values()) * 2
37
+
38
+
39
+ def test_plain_adapter_load(tmp_path):
40
+ state = _state()
41
+ path = tmp_path / "plain.pt"
42
+ torch.save({"lora": state, "meta": {"tag": "x"}}, path)
43
+ back, meta = load_adapter(path)
44
+ assert set(back) == set(state)
45
+ assert meta["tag"] == "x"
46
+
47
+
48
+ def test_lora_linear_shapes():
49
+ m = LoRALinear(nn.Linear(8, 4), r=2)
50
+ assert m.A.shape == (2, 8)
51
+ assert m.B.shape == (4, 2)
52
+ assert m(torch.randn(3, 8)).shape == (3, 4)
53
+
54
+
55
+ def test_cli_accepts_save_dir(tmp_path):
56
+ from streamft.adapter import main
57
+
58
+ save_dir = tmp_path / "run"
59
+ save_dir.mkdir()
60
+ torch.save(_state(), save_dir / "adapter.pt")
61
+ out = tmp_path / "adapter_q8.pt"
62
+ assert main(["--adapter", str(save_dir), "--bits", "8", "--out", str(out)]) == 0
63
+ assert out.exists()
64
+ back, _ = load_adapter(out)
65
+ assert set(back) == set(_state())
66
+
67
+
68
+ def test_cli_rejects_non_lora_state(tmp_path):
69
+ import pytest
70
+
71
+ from streamft.adapter import main
72
+
73
+ path = tmp_path / "plain.pt"
74
+ torch.save({"weight": torch.zeros(4, 4)}, path)
75
+ with pytest.raises(SystemExit, match="no LoRA"):
76
+ main(["--adapter", str(path), "--bits", "8", "--out", str(tmp_path / "q.pt")])
77
+
78
+
79
+ def test_lora_fast_path():
80
+ m = LoRALinear(nn.Linear(8, 4).to(torch.bfloat16), r=2)
81
+ m.set_fast(torch.bfloat16)
82
+ out = m(torch.randn(3, 8, dtype=torch.bfloat16))
83
+ assert out.shape == (3, 4)
84
+ assert out.dtype == torch.bfloat16
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes