streamft 0.4.0__tar.gz → 0.5.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (28) hide show
  1. {streamft-0.4.0 → streamft-0.5.0}/PKG-INFO +54 -1
  2. {streamft-0.4.0 → streamft-0.5.0}/README.md +53 -0
  3. {streamft-0.4.0 → streamft-0.5.0}/pyproject.toml +1 -1
  4. {streamft-0.4.0 → streamft-0.5.0}/streamft/__main__.py +12 -0
  5. streamft-0.5.0/streamft/merge.py +87 -0
  6. streamft-0.5.0/streamft/serve.py +146 -0
  7. {streamft-0.4.0 → streamft-0.5.0}/streamft.egg-info/PKG-INFO +54 -1
  8. {streamft-0.4.0 → streamft-0.5.0}/streamft.egg-info/SOURCES.txt +2 -0
  9. {streamft-0.4.0 → streamft-0.5.0}/tests/test_streamft_smoke.py +19 -0
  10. {streamft-0.4.0 → streamft-0.5.0}/LICENSE +0 -0
  11. {streamft-0.4.0 → streamft-0.5.0}/setup.cfg +0 -0
  12. {streamft-0.4.0 → streamft-0.5.0}/streamft/__init__.py +0 -0
  13. {streamft-0.4.0 → streamft-0.5.0}/streamft/autotune.py +0 -0
  14. {streamft-0.4.0 → streamft-0.5.0}/streamft/cli.py +0 -0
  15. {streamft-0.4.0 → streamft-0.5.0}/streamft/data.py +0 -0
  16. {streamft-0.4.0 → streamft-0.5.0}/streamft/disk_quant_store.py +0 -0
  17. {streamft-0.4.0 → streamft-0.5.0}/streamft/doctor.py +0 -0
  18. {streamft-0.4.0 → streamft-0.5.0}/streamft/easy.py +0 -0
  19. {streamft-0.4.0 → streamft-0.5.0}/streamft/evals.py +0 -0
  20. {streamft-0.4.0 → streamft-0.5.0}/streamft/fit.py +0 -0
  21. {streamft-0.4.0 → streamft-0.5.0}/streamft/lora.py +0 -0
  22. {streamft-0.4.0 → streamft-0.5.0}/streamft/metrics.py +0 -0
  23. {streamft-0.4.0 → streamft-0.5.0}/streamft/stream_quant_lora.py +0 -0
  24. {streamft-0.4.0 → streamft-0.5.0}/streamft/stream_train.py +0 -0
  25. {streamft-0.4.0 → streamft-0.5.0}/streamft.egg-info/dependency_links.txt +0 -0
  26. {streamft-0.4.0 → streamft-0.5.0}/streamft.egg-info/entry_points.txt +0 -0
  27. {streamft-0.4.0 → streamft-0.5.0}/streamft.egg-info/requires.txt +0 -0
  28. {streamft-0.4.0 → streamft-0.5.0}/streamft.egg-info/top_level.txt +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: streamft
3
- Version: 0.4.0
3
+ Version: 0.5.0
4
4
  Summary: Fine-tune models that do not fit: stream NF4 weights from disk, one transformer block at a time.
5
5
  Author-email: Gokhan Cava <byjameson@users.noreply.github.com>
6
6
  License: Apache-2.0
@@ -154,6 +154,59 @@ streamft train --model-dir checkpoints/qwen7b_nf4 --sft --steps 1000 \
154
154
  measured entry in `streamft fit`.
155
155
  - `streamft fit` prints this hint automatically when it sees a pre-Ampere GPU.
156
156
 
157
+ ### Tokenizer cost and the diffusion question
158
+
159
+ `seq` is in tokens, and tokenizers are not language-neutral. With the Qwen2.5 tokenizer,
160
+ the same passage costs **1.13 tokens/word in English, 1.74 in German and 2.39 in Turkish**
161
+ ? a Turkish run at `seq 512` sees about half the text an English run does. Measured with
162
+ `scripts/tokenizer_cost.py`; the numbers, the byte-level alternative and why diffusion LMs
163
+ (LLaDA/Mercury: 800?1,500 tok/s at batch 1, but the same memory footprint) do not change
164
+ this project's bottleneck are in [`docs/TOKENIZERS.md`](docs/TOKENIZERS.md).
165
+
166
+ ### Run it in a browser (no server)
167
+
168
+ ```bash
169
+ streamft merge --model Qwen/Qwen2.5-0.5B --adapter checkpoints/cpu_cookbook --out merged/q05
170
+ powershell -ExecutionPolicy Bypass -File scripts/export_gguf.ps1 # -> q05-q4_k_m.gguf
171
+ python scripts/serve_static.py --port 8765
172
+ # open http://127.0.0.1:8765/wasm/index.html?model=../q05-q4_k_m.gguf
173
+ ```
174
+
175
+ Measured on the reference laptop, 0.5B Q4_K_M (379 MB): **1.3 tok/s** with the plain
176
+ `http.server`, **6.0 tok/s** behind `scripts/serve_static.py` ? the COOP/COEP headers are
177
+ what let wllama use 12 threads instead of one. Details and limits in
178
+ [`wasm/README.md`](wasm/README.md).
179
+
180
+ ### Share one machine on the LAN
181
+
182
+ One PC with the model can serve everyone on the same network:
183
+
184
+ ```bash
185
+ streamft serve --model merged/q05_cpu --port 8000
186
+ # others open http://<that-machine-ip>:8000
187
+ ```
188
+
189
+ It ships a small chat page and an OpenAI-compatible `POST /v1/chat/completions`, so existing
190
+ tools can point at it. This is **one machine computing, the rest as clients** ? not
191
+ peer-to-peer sharding and not volunteer compute. Tested end to end: `/health` and a
192
+ 16-token completion (1.9 s on the reference GPU).
193
+
194
+ ### Measured on
195
+
196
+ Every speed number here comes from **one machine**: RTX 3050 Laptop 6 GB, Windows 11,
197
+ AC power, NF4 base, LoRA r=8, batch 1, Dolly-15k SFT. The four rows (3B/7B/14B/32B) are
198
+ listed in [`docs/MEASURED.md`](docs/MEASURED.md), together with the rules for adding one
199
+ from a different card:
200
+
201
+ ```bash
202
+ python scripts/measured_row.py --row "<json line>" --config checkpoints/<model>
203
+ ```
204
+
205
+ It validates the run (finished steps, `grad_layers == n_layers`, peak VRAM recorded),
206
+ derives the architecture fingerprint and prints the snippets for `fit.py`, `fitcore.py`
207
+ and the Space. Anything not in that table prints **"not measured"** in `streamft fit` ?
208
+ that is the point.
209
+
157
210
  ## Verification
158
211
 
159
212
  `python scripts/verify_gradients.py --model-dir checkpoints/qwen3b_bnb4bit --seq 128`
@@ -124,6 +124,59 @@ streamft train --model-dir checkpoints/qwen7b_nf4 --sft --steps 1000 \
124
124
  measured entry in `streamft fit`.
125
125
  - `streamft fit` prints this hint automatically when it sees a pre-Ampere GPU.
126
126
 
127
+ ### Tokenizer cost and the diffusion question
128
+
129
+ `seq` is in tokens, and tokenizers are not language-neutral. With the Qwen2.5 tokenizer,
130
+ the same passage costs **1.13 tokens/word in English, 1.74 in German and 2.39 in Turkish**
131
+ ? a Turkish run at `seq 512` sees about half the text an English run does. Measured with
132
+ `scripts/tokenizer_cost.py`; the numbers, the byte-level alternative and why diffusion LMs
133
+ (LLaDA/Mercury: 800?1,500 tok/s at batch 1, but the same memory footprint) do not change
134
+ this project's bottleneck are in [`docs/TOKENIZERS.md`](docs/TOKENIZERS.md).
135
+
136
+ ### Run it in a browser (no server)
137
+
138
+ ```bash
139
+ streamft merge --model Qwen/Qwen2.5-0.5B --adapter checkpoints/cpu_cookbook --out merged/q05
140
+ powershell -ExecutionPolicy Bypass -File scripts/export_gguf.ps1 # -> q05-q4_k_m.gguf
141
+ python scripts/serve_static.py --port 8765
142
+ # open http://127.0.0.1:8765/wasm/index.html?model=../q05-q4_k_m.gguf
143
+ ```
144
+
145
+ Measured on the reference laptop, 0.5B Q4_K_M (379 MB): **1.3 tok/s** with the plain
146
+ `http.server`, **6.0 tok/s** behind `scripts/serve_static.py` ? the COOP/COEP headers are
147
+ what let wllama use 12 threads instead of one. Details and limits in
148
+ [`wasm/README.md`](wasm/README.md).
149
+
150
+ ### Share one machine on the LAN
151
+
152
+ One PC with the model can serve everyone on the same network:
153
+
154
+ ```bash
155
+ streamft serve --model merged/q05_cpu --port 8000
156
+ # others open http://<that-machine-ip>:8000
157
+ ```
158
+
159
+ It ships a small chat page and an OpenAI-compatible `POST /v1/chat/completions`, so existing
160
+ tools can point at it. This is **one machine computing, the rest as clients** ? not
161
+ peer-to-peer sharding and not volunteer compute. Tested end to end: `/health` and a
162
+ 16-token completion (1.9 s on the reference GPU).
163
+
164
+ ### Measured on
165
+
166
+ Every speed number here comes from **one machine**: RTX 3050 Laptop 6 GB, Windows 11,
167
+ AC power, NF4 base, LoRA r=8, batch 1, Dolly-15k SFT. The four rows (3B/7B/14B/32B) are
168
+ listed in [`docs/MEASURED.md`](docs/MEASURED.md), together with the rules for adding one
169
+ from a different card:
170
+
171
+ ```bash
172
+ python scripts/measured_row.py --row "<json line>" --config checkpoints/<model>
173
+ ```
174
+
175
+ It validates the run (finished steps, `grad_layers == n_layers`, peak VRAM recorded),
176
+ derives the architecture fingerprint and prints the snippets for `fit.py`, `fitcore.py`
177
+ and the Space. Anything not in that table prints **"not measured"** in `streamft fit` ?
178
+ that is the point.
179
+
127
180
  ## Verification
128
181
 
129
182
  `python scripts/verify_gradients.py --model-dir checkpoints/qwen3b_bnb4bit --seq 128`
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "streamft"
7
- version = "0.4.0"
7
+ version = "0.5.0"
8
8
  description = "Fine-tune models that do not fit: stream NF4 weights from disk, one transformer block at a time."
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.10"
@@ -7,6 +7,8 @@
7
7
  streamft cpu SFT a small model on the CPU (no GPU; model must fit RAM)
8
8
  streamft synth generate training pairs locally with a small model
9
9
  streamft eval run a small local prompt set (repetition/refusal/keywords)
10
+ streamft merge fold a saved adapter into the base model (for serving / GGUF)
11
+ streamft serve host a merged model on the LAN (chat page + /v1/chat/completions)
10
12
  streamft doctor environment report (torch, CUDA, bitsandbytes, VRAM)
11
13
  """
12
14
  from __future__ import annotations
@@ -22,6 +24,8 @@ USAGE = """streamft - fine-tune models that do not fit
22
24
  streamft cpu --model <small-model> --steps 20
23
25
  streamft synth --model <small-model> --n 50 --out data/synth.jsonl
24
26
  streamft eval --model <model> [--prompts mine.jsonl]
27
+ streamft merge --model <base> --adapter <dir> --out merged/<name>
28
+ streamft serve --model merged/<name> --port 8000
25
29
  streamft doctor
26
30
 
27
31
  Docs: https://github.com/byjameson/stream-finetune
@@ -45,6 +49,14 @@ def main() -> int:
45
49
  from streamft.easy import cpu_main
46
50
  return cpu_main(rest)
47
51
 
52
+ if cmd == "serve":
53
+ from streamft.serve import main as serve_main
54
+ return serve_main(rest)
55
+
56
+ if cmd == "merge":
57
+ from streamft.merge import main as merge_main
58
+ return merge_main(rest)
59
+
48
60
  if cmd == "eval":
49
61
  from streamft.evals import main as eval_main
50
62
  return eval_main(rest)
@@ -0,0 +1,87 @@
1
+ """`streamft merge` — fold a saved LoRA adapter into the base model (CPU is fine).
2
+
3
+ The merged directory is a plain HF checkpoint, which is what GGUF conversion expects.
4
+
5
+ streamft merge --model Qwen/Qwen2.5-0.5B --adapter checkpoints/cpu_cookbook --out merged/q05
6
+
7
+ The adapter was saved by `streamft train --save` / `streamft cpu --save` as
8
+ `adapter.pt` (`lora_state_dict`). The rank is read from the tensor shapes, so no extra
9
+ config file is needed.
10
+ """
11
+ from __future__ import annotations
12
+
13
+ import argparse
14
+ import json
15
+ from pathlib import Path
16
+
17
+ import torch
18
+
19
+ from .lora import LoRALinear, apply_lora, merge_lora, set_lora_state_dict
20
+
21
+
22
+ def main(argv=None) -> int:
23
+ from transformers import AutoModelForCausalLM, AutoTokenizer
24
+
25
+ ap = argparse.ArgumentParser(prog="streamft merge")
26
+ ap.add_argument("--model", required=True, help="base model (HF id or local dir)")
27
+ ap.add_argument("--adapter", required=True, help="dir with adapter.pt")
28
+ ap.add_argument("--out", required=True, help="output directory for the merged model")
29
+ ap.add_argument("--dtype", choices=["fp32", "fp16", "bf16"], default="fp16",
30
+ help="storage dtype of the merged weights (fp16 halves the file)")
31
+ args = ap.parse_args(argv)
32
+
33
+ adapter_path = Path(args.adapter)
34
+ if adapter_path.is_dir():
35
+ adapter_path = adapter_path / "adapter.pt"
36
+ if not adapter_path.exists():
37
+ raise SystemExit(f"adapter not found: {adapter_path}")
38
+
39
+ state = torch.load(str(adapter_path), map_location="cpu", weights_only=True)
40
+ if not state:
41
+ raise SystemExit("adapter.pt is empty")
42
+ a_key = next((k for k in state if k.endswith(".A")), None)
43
+ if a_key is None:
44
+ raise SystemExit("adapter.pt has no LoRA A tensors; was it saved with lora_state_dict?")
45
+ r = int(state[a_key].shape[0])
46
+
47
+ dtype = {"fp32": torch.float32, "fp16": torch.float16, "bf16": torch.bfloat16}[args.dtype]
48
+ print(f"[merge] {args.model} + {adapter_path.name} (r={r}) -> {args.out} [{args.dtype}]",
49
+ flush=True)
50
+ model = AutoModelForCausalLM.from_pretrained(args.model, torch_dtype=dtype)
51
+ apply_lora(model, r_attn=r, r_mlp=0)
52
+ set_lora_state_dict(model, state)
53
+ merge_lora(model)
54
+
55
+ # After a real merge every LoRA factor is exactly zero; the base absorbed the delta.
56
+ not_merged = [n for n, p in model.named_parameters()
57
+ if (n.endswith(".A") or n.endswith(".B")) and float(p.detach().abs().sum()) > 0]
58
+ if not_merged:
59
+ raise SystemExit(f"adapter tensors left unmerged: {not_merged[:3]}")
60
+ for p in model.parameters():
61
+ p.requires_grad_(False)
62
+
63
+ # Unwrap: after merge the LoRA factors are zero, but `save_pretrained` would still
64
+ # write them, and GGUF converters refuse unknown tensors (`...k_proj.A`).
65
+ stripped = 0
66
+ for name, mod in list(model.named_modules()):
67
+ if isinstance(mod, LoRALinear):
68
+ parent_name, _, attr = name.rpartition(".")
69
+ parent = model.get_submodule(parent_name) if parent_name else model
70
+ setattr(parent, attr, mod.base)
71
+ stripped += 1
72
+ print(f"[merge] removed {stripped} LoRA wrappers", flush=True)
73
+
74
+ out = Path(args.out)
75
+ out.mkdir(parents=True, exist_ok=True)
76
+ model.save_pretrained(str(out), safe_serialization=True)
77
+ tok = AutoTokenizer.from_pretrained(args.model)
78
+ tok.save_pretrained(str(out))
79
+
80
+ sizes = {f.name: round(f.stat().st_size / 1e9, 2) for f in out.glob("*.safetensors")}
81
+ print(json.dumps({"merged": str(out), "dtype": args.dtype, "rank": r,
82
+ "weights_gb": sizes, "license": "same as the base model"}, indent=1))
83
+ print("\nnext (needs llama.cpp, several minutes of CPU):")
84
+ print(f" python llama.cpp/convert_hf_to_gguf.py {out} --outfile model-f16.gguf")
85
+ print(" llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M")
86
+ print(" open wasm/index.html and pick model-q4_k_m.gguf")
87
+ return 0
@@ -0,0 +1,146 @@
1
+ """`streamft serve` — one machine hosts, others on the LAN use it.
2
+
3
+ streamft serve --model merged/q05_cpu --port 8000
4
+
5
+ Open http://<this-machine-ip>:8000 from any phone/laptop on the same network. There is a
6
+ small chat page and an OpenAI-compatible `POST /v1/chat/completions` endpoint, so existing
7
+ tools can point at it too.
8
+
9
+ This is the practical version of "share the setup": one PC with the model, everyone else
10
+ uses a browser. It is *not* a peer-to-peer network — no sharding, no volunteer compute.
11
+
12
+ Adapters: pass `--adapter checkpoints/my_adapter` to serve an unmerged adapter on top of
13
+ the base.
14
+ """
15
+ from __future__ import annotations
16
+
17
+ import argparse
18
+ import json
19
+ import threading
20
+ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
21
+
22
+ import torch
23
+
24
+ _lock = threading.Lock()
25
+ _state: dict = {}
26
+
27
+ PAGE = """<!doctype html><meta charset="utf-8"><title>streamft serve</title>
28
+ <style>body{font:16px/1.5 system-ui;max-width:760px;margin:40px auto;padding:0 20px}
29
+ textarea{width:100%;min-height:90px}#o{white-space:pre-wrap;background:#f6f6f6;padding:12px;border-radius:8px}
30
+ </style>
31
+ <h2>streamft serve</h2><p id="m" style="color:#666"></p>
32
+ <textarea id="p">Write one sentence about small GPUs.</textarea>
33
+ <button id="g">Send</button><pre id="o"></pre>
34
+ <script>
35
+ fetch('/health').then(r=>r.json()).then(j=>document.getElementById('m').textContent='model: '+(j.model||'?'));
36
+ document.getElementById('g').onclick=async()=>{
37
+ const o=document.getElementById('o'); o.textContent='…';
38
+ const r=await fetch('/v1/chat/completions',{method:'POST',headers:{'Content-Type':'application/json'},
39
+ body:JSON.stringify({messages:[{role:'user',content:document.getElementById('p').value}],max_tokens:128})});
40
+ const j=await r.json();
41
+ o.textContent = j.error ? ('error: '+j.error) : j.choices[0].message.content;
42
+ };
43
+ </script>"""
44
+
45
+
46
+ def build_prompt(tok, messages) -> str:
47
+ if getattr(tok, "chat_template", None):
48
+ return tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
49
+ return "\n".join(m.get("content", "") for m in messages) + "\n"
50
+
51
+
52
+ class Handler(BaseHTTPRequestHandler):
53
+ protocol_version = "HTTP/1.1"
54
+
55
+ def _send(self, code: int, body: bytes, ctype: str = "application/json"):
56
+ self.send_response(code)
57
+ self.send_header("Content-Type", ctype)
58
+ self.send_header("Content-Length", str(len(body)))
59
+ self.send_header("Access-Control-Allow-Origin", "*")
60
+ self.end_headers()
61
+ self.wfile.write(body)
62
+
63
+ def do_GET(self):
64
+ if self.path == "/health":
65
+ self._send(200, json.dumps({"ok": True, "model": _state["model_id"]}).encode())
66
+ elif self.path in ("/", "/index.html"):
67
+ self._send(200, PAGE.encode(), "text/html; charset=utf-8")
68
+ else:
69
+ self._send(404, b'{"error":"not found"}')
70
+
71
+ def do_POST(self):
72
+ if self.path != "/v1/chat/completions":
73
+ return self._send(404, b'{"error":"not found"}')
74
+ try:
75
+ n = int(self.headers.get("Content-Length", "0"))
76
+ req = json.loads(self.rfile.read(n) or b"{}")
77
+ messages = req.get("messages") or [{"role": "user", "content": req.get("prompt", "")}]
78
+ max_new = int(req.get("max_tokens", 128))
79
+ prompt = build_prompt(_state["tok"], messages)
80
+ ids = _state["tok"](prompt, return_tensors="pt").to(_state["device"])
81
+ with _lock, torch.inference_mode():
82
+ out = _state["model"].generate(
83
+ **ids, max_new_tokens=max_new, do_sample=bool(req.get("temperature", 0) > 0),
84
+ temperature=float(req.get("temperature", 0.7) or 0.7),
85
+ top_p=float(req.get("top_p", 0.9)),
86
+ pad_token_id=_state["tok"].eos_token_id,
87
+ )
88
+ text = _state["tok"].decode(out[0][ids["input_ids"].shape[1]:],
89
+ skip_special_tokens=True).strip()
90
+ body = {
91
+ "object": "chat.completion",
92
+ "model": _state["model_id"],
93
+ "choices": [{"index": 0, "message": {"role": "assistant", "content": text},
94
+ "finish_reason": "stop"}],
95
+ "usage": {"prompt_tokens": int(ids["input_ids"].shape[1]),
96
+ "completion_tokens": int(out.shape[1] - ids["input_ids"].shape[1])},
97
+ }
98
+ self._send(200, json.dumps(body, ensure_ascii=False).encode())
99
+ except Exception as e: # keep the server alive, report the error
100
+ self._send(500, json.dumps({"error": f"{type(e).__name__}: {e}"}).encode())
101
+
102
+ def log_message(self, fmt, *args):
103
+ pass
104
+
105
+
106
+ def main(argv=None) -> int:
107
+ from transformers import AutoModelForCausalLM, AutoTokenizer
108
+
109
+ ap = argparse.ArgumentParser(prog="streamft serve")
110
+ ap.add_argument("--model", required=True, help="merged model dir or HF id")
111
+ ap.add_argument("--adapter", default="", help="optional adapter dir with adapter.pt")
112
+ ap.add_argument("--host", default="0.0.0.0")
113
+ ap.add_argument("--port", type=int, default=8000)
114
+ ap.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
115
+ args = ap.parse_args(argv)
116
+
117
+ dtype = torch.bfloat16 if args.device == "cuda" else torch.float32
118
+ tok = AutoTokenizer.from_pretrained(args.model)
119
+ model = AutoModelForCausalLM.from_pretrained(args.model, torch_dtype=dtype).to(args.device).eval()
120
+
121
+ if args.adapter:
122
+ from pathlib import Path
123
+
124
+ from .lora import apply_lora, set_lora_state_dict
125
+
126
+ p = Path(args.adapter)
127
+ p = p / "adapter.pt" if p.is_dir() else p
128
+ state = torch.load(str(p), map_location="cpu", weights_only=True)
129
+ r = int(next(v for k, v in state.items() if k.endswith(".A")).shape[0])
130
+ apply_lora(model, r_attn=r, r_mlp=0)
131
+ set_lora_state_dict(model, state)
132
+ print(f"[serve] adapter {p.name} (r={r})", flush=True)
133
+
134
+ _state.update(model=model, tok=tok, device=args.device, model_id=args.model)
135
+ srv = ThreadingHTTPServer((args.host, args.port), Handler)
136
+ print(f"[serve] http://{args.host}:{args.port} (device {args.device})", flush=True)
137
+ print("[serve] others on the same network: http://<this-machine-ip>:%d" % args.port, flush=True)
138
+ try:
139
+ srv.serve_forever()
140
+ except KeyboardInterrupt:
141
+ pass
142
+ return 0
143
+
144
+
145
+ if __name__ == "__main__":
146
+ raise SystemExit(main())
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: streamft
3
- Version: 0.4.0
3
+ Version: 0.5.0
4
4
  Summary: Fine-tune models that do not fit: stream NF4 weights from disk, one transformer block at a time.
5
5
  Author-email: Gokhan Cava <byjameson@users.noreply.github.com>
6
6
  License: Apache-2.0
@@ -154,6 +154,59 @@ streamft train --model-dir checkpoints/qwen7b_nf4 --sft --steps 1000 \
154
154
  measured entry in `streamft fit`.
155
155
  - `streamft fit` prints this hint automatically when it sees a pre-Ampere GPU.
156
156
 
157
+ ### Tokenizer cost and the diffusion question
158
+
159
+ `seq` is in tokens, and tokenizers are not language-neutral. With the Qwen2.5 tokenizer,
160
+ the same passage costs **1.13 tokens/word in English, 1.74 in German and 2.39 in Turkish**
161
+ ? a Turkish run at `seq 512` sees about half the text an English run does. Measured with
162
+ `scripts/tokenizer_cost.py`; the numbers, the byte-level alternative and why diffusion LMs
163
+ (LLaDA/Mercury: 800?1,500 tok/s at batch 1, but the same memory footprint) do not change
164
+ this project's bottleneck are in [`docs/TOKENIZERS.md`](docs/TOKENIZERS.md).
165
+
166
+ ### Run it in a browser (no server)
167
+
168
+ ```bash
169
+ streamft merge --model Qwen/Qwen2.5-0.5B --adapter checkpoints/cpu_cookbook --out merged/q05
170
+ powershell -ExecutionPolicy Bypass -File scripts/export_gguf.ps1 # -> q05-q4_k_m.gguf
171
+ python scripts/serve_static.py --port 8765
172
+ # open http://127.0.0.1:8765/wasm/index.html?model=../q05-q4_k_m.gguf
173
+ ```
174
+
175
+ Measured on the reference laptop, 0.5B Q4_K_M (379 MB): **1.3 tok/s** with the plain
176
+ `http.server`, **6.0 tok/s** behind `scripts/serve_static.py` ? the COOP/COEP headers are
177
+ what let wllama use 12 threads instead of one. Details and limits in
178
+ [`wasm/README.md`](wasm/README.md).
179
+
180
+ ### Share one machine on the LAN
181
+
182
+ One PC with the model can serve everyone on the same network:
183
+
184
+ ```bash
185
+ streamft serve --model merged/q05_cpu --port 8000
186
+ # others open http://<that-machine-ip>:8000
187
+ ```
188
+
189
+ It ships a small chat page and an OpenAI-compatible `POST /v1/chat/completions`, so existing
190
+ tools can point at it. This is **one machine computing, the rest as clients** ? not
191
+ peer-to-peer sharding and not volunteer compute. Tested end to end: `/health` and a
192
+ 16-token completion (1.9 s on the reference GPU).
193
+
194
+ ### Measured on
195
+
196
+ Every speed number here comes from **one machine**: RTX 3050 Laptop 6 GB, Windows 11,
197
+ AC power, NF4 base, LoRA r=8, batch 1, Dolly-15k SFT. The four rows (3B/7B/14B/32B) are
198
+ listed in [`docs/MEASURED.md`](docs/MEASURED.md), together with the rules for adding one
199
+ from a different card:
200
+
201
+ ```bash
202
+ python scripts/measured_row.py --row "<json line>" --config checkpoints/<model>
203
+ ```
204
+
205
+ It validates the run (finished steps, `grad_layers == n_layers`, peak VRAM recorded),
206
+ derives the architecture fingerprint and prints the snippets for `fit.py`, `fitcore.py`
207
+ and the Space. Anything not in that table prints **"not measured"** in `streamft fit` ?
208
+ that is the point.
209
+
157
210
  ## Verification
158
211
 
159
212
  `python scripts/verify_gradients.py --model-dir checkpoints/qwen3b_bnb4bit --seq 128`
@@ -12,7 +12,9 @@ streamft/easy.py
12
12
  streamft/evals.py
13
13
  streamft/fit.py
14
14
  streamft/lora.py
15
+ streamft/merge.py
15
16
  streamft/metrics.py
17
+ streamft/serve.py
16
18
  streamft/stream_quant_lora.py
17
19
  streamft/stream_train.py
18
20
  streamft.egg-info/PKG-INFO
@@ -279,3 +279,22 @@ def test_repetition_rate_counts_duplicate_ngrams():
279
279
  assert repetition_rate("a b c d e f g h") == 0.0
280
280
  assert repetition_rate("a b c d a b c d") == 0.2 # 1 duplicate of 5 grams
281
281
  assert repetition_rate("") == 0.0
282
+
283
+
284
+ def test_measured_row_validation_rejects_incomplete_runs():
285
+ import importlib.util
286
+ import sys
287
+ from pathlib import Path
288
+
289
+ path = Path(__file__).resolve().parents[1] / "scripts" / "measured_row.py"
290
+ spec = importlib.util.spec_from_file_location("measured_row", path)
291
+ mod = importlib.util.module_from_spec(spec)
292
+ sys.modules["measured_row"] = mod
293
+ spec.loader.exec_module(mod)
294
+
295
+ good = {"steps": 30, "grad_layers": 28, "n_layers": 28,
296
+ "peak_gb_max": 5.5, "step_s_med": 1.5}
297
+ assert mod.validate(good) == []
298
+ assert mod.validate({**good, "steps": 0})
299
+ assert mod.validate({**good, "grad_layers": 27})
300
+ assert mod.validate({**good, "peak_gb_max": None})
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes