make-localllm-easier 0.1.0__tar.gz

make_localllm_easier-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 homellm contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

make_localllm_easier-0.1.0/PKG-INFO

@@ -0,0 +1,130 @@
+Metadata-Version: 2.4
+Name: make-localllm-easier
+Version: 0.1.0
+Summary: One command runs the best local AI your PC can handle: measured model picks + a tuned llama.cpp (AMD, NVIDIA, Intel, Apple)
+License: MIT
+Project-URL: Homepage, https://github.com/phonology024/make-localllm-easier
+Keywords: llm,local-llm,llama.cpp,vulkan,amd,gguf,thai,offline-ai
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# make-localllm-easier — run the best local LLM your GPU can handle, in one command
+
+**`localllm` picks, downloads and runs the most accurate local AI model for your PC and your language, chosen from real
+benchmark measurements, with llama.cpp tuned for AMD, NVIDIA, Intel and Apple GPUs.**
+
+```
+pip install make-localllm-easier
+localllm
+```
+
+That's it. `localllm` checks your GPU and RAM, picks the most accurate model we have *measured* for your language that
+fits your card, downloads llama.cpp and the model, starts it with settings profiled op by op, and opens the chat page.
+You also get an OpenAI-compatible API at `http://127.0.0.1:8080/v1` for any app that speaks it. Offline, private, free.
+
+```
+localllm doctor     # what this GPU is good for: model sizes, speed, how much text it can hold
+localllm list       # every model we have measured, with scores per language
+localllm serve      # API only, no browser
+localllm eval       # score any running server in English + your language
+```
+
+## FAQ
+
+**Which local LLM should I run on my GPU?** Run `localllm doctor`. It lists which model sizes fit your card (4B up to
+120B MoE), at which quantization, how fast they should run, and the most accurate measured model for your language.
+
+**Can a 16 GB GPU run a 27B model?** Yes. Qwen3.8-27B at ~3.5 bits (12.2 GB) runs at ~50 tok/s on an RX 9070 XT and
+keeps 81.8% on English Global-MMLU-Lite. gemma-4-26B-A4B (13.3 GB) runs at ~69 tok/s with similar accuracy.
+
+**Is a 2-bit quantized model good enough?** Usually not for non-English use: 2-bit costs 8-13 accuracy points, and
+Hindi, Arabic and Thai lose the most (13 points).
+
+**Why is llama.cpp slow on my AMD (or Intel) GPU on Windows?** If Resizable BAR is off, llama.cpp's Vulkan backend puts
+buffers in a 256 MB host-visible heap backed by system RAM and decode drops up to 1.7x. `localllm` sets
+`GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1` for you ([llama.cpp#27097](https://github.com/ggml-org/llama.cpp/issues/27097)).
+
+**Does it work offline?** After the first download, yes. Nothing leaves your PC.
+
+**Which languages are measured?** 23 languages on Global-MMLU-Lite, 44 countries' own exams on INCLUDE, plus Thai
+(ThaiExam). `localllm eval --langs ...` measures any of them on your hardware.
+
+## What `localllm doctor` tells you
+
+```
+GPU AMD Radeon RX 9070 XT  15.9 GB  (640 GB/s)    RAM 32 GB    language: th
+
+Model sizes for this PC (whole model on the GPU = fast):
+  [OK  ] 4B                       Q8 4.2 GB  ~90 tok/s (est.)
+  [OK  ] 8B                       Q8 8.5 GB  ~45 tok/s (est.)
+  [OK  ] 14B                      Q6 11.5 GB  ~33 tok/s (est.)
+  [OK  ] 24-32B                   Q3 13.2 GB  ~29 tok/s (est.)
+  [SLOW] 30B MoE (3B active)      Q4 18.0 GB with experts in RAM - works, ~10-25 tok/s
+  [NO  ] 70B                      needs ~42.0 GB - too big for this PC
+
+Best measured model for you: gemma4-26b-a4b-qat  (MoE with ~4B active params: fastest)
+What it can do here:
+  TH  real local school/licence exams   65.7% correct  <- your language
+  holds ~78k tokens at once (~130 pages of text) next to the model
+  answers at ~69 tok/s
+```
+
+Speeds marked *est.* come from your card's memory bandwidth, calibrated on measured runs. Everything else is measured.
+
+## Measured results (RX 9070 XT 16 GB, Windows 11, llama.cpp Vulkan)
+
+Accuracy (%) on multiple-choice exams, zero-shot. **global** = Global-MMLU-Lite: the same 400 questions translated, so
+languages compare like for like. **regional** = INCLUDE: real exams written in each country (ThaiExam for Thai).
+
+| | Qwen3.8-27B Q3 (12.2 GB) | gemma-4-26B-A4B QAT Q4 (13.3 GB) | Qwen3.8-27B 2-bit (7.8 GB) |
+|---|---|---|---|
+| English | 81.8 | 82.2 | 74.2 |
+| Chinese | 76.2 / 74.7 | 73.5 / 66.5 | 67.8 / 67.8 |
+| Spanish | 80.2 / 76.8 | 74.5 / 75.2 | 70.8 / 69.2 |
+| Japanese | 73.5 / 87.6 | 74.5 / 81.9 | 65.8 / 77.9 |
+| Arabic | 70.8 / 71.2 | 71.5 / 73.6 | 60.8 / 57.2 |
+| Hindi | 69.0 / 74.3 | 69.5 / 71.0 | 56.2 / 55.5 |
+| Thai | – / 67.1 | – / 65.7 | – / 54.2 |
+| **decode speed** | **50 tok/s** (MTP) | **69 tok/s** | 40 tok/s |
+
+Cells are global / regional. Margins are about ±4 (global) and ±5 (regional) points at 95%, so `localllm` treats gaps
+under 2 points as a tie and picks the faster model.
+
+## Findings worth knowing
+
+1. **2-bit costs 8-13 points, and lower-resource languages pay the most.** Hindi, Arabic and Thai lose 13; English,
+   Chinese and Spanish about 8-9. A 177B MoE squeezed to 1.6 bits scored *below* a 27B at 3 bits.
+2. **Calibrating the quantization on your language doesn't help at ~3.5 bits.** A Thai-text importance matrix scored the
+   same as the stock one in Thai, English and Chinese (64.8 vs 64.6 Thai). At this level the number of bits matters,
+   the calibration text doesn't.
+3. **AMD/Intel cards without Resizable BAR lose up to 1.7x decode speed** in llama.cpp's Vulkan backend. Hybrid DeltaNet
+   models (Qwen3.5/3.8) suffer most: they rewrite a 3 MB state per layer per token.
+4. **Qwen3.8 GGUFs ship a multi-token-prediction head.** Drafting 2 tokens with it adds ~40% decode speed for free;
+   drafting 3 is slower.
+5. **The first run of a new llama.cpp build is slow** while the GPU driver compiles its shaders once (~15 s).
+
+## How the benchmark works
+
+`localllm eval` asks each question with thinking off and reads the log-probability of every answer letter from the first
+generated token, then takes the most likely one. It's prompt processing only, so a language takes a few minutes, and the
+result is deterministic. Data is downloaded at eval time from the original Apache-2.0 datasets
+([Global-MMLU-Lite](https://huggingface.co/datasets/CohereLabs/Global-MMLU-Lite),
+[INCLUDE](https://huggingface.co/datasets/CohereLabs/include-lite-44),
+[ThaiExam](https://huggingface.co/datasets/typhoon-ai/thai_exam)) and never redistributed. It measures knowledge and
+reasoning in multiple choice, not writing quality.
+
+## Contributing
+
+The catalog only grows with measurements. Run `localllm eval --langs en,<yours>` on your GPU and open a PR with
+`~/.localllm/results.json` and your GPU name. Other languages' local exams are very welcome. See [ROADMAP.md](ROADMAP.md)
+for what's next: using less system RAM (0.2), working alongside cloud provider APIs (0.3), and a speed-only release (0.4).
+
+## License
+
+MIT. Models keep their own licenses; benchmark data keeps its own (Apache-2.0).

make_localllm_easier-0.1.0/README.md

@@ -0,0 +1,114 @@
+# make-localllm-easier — run the best local LLM your GPU can handle, in one command
+
+**`localllm` picks, downloads and runs the most accurate local AI model for your PC and your language, chosen from real
+benchmark measurements, with llama.cpp tuned for AMD, NVIDIA, Intel and Apple GPUs.**
+
+```
+pip install make-localllm-easier
+localllm
+```
+
+That's it. `localllm` checks your GPU and RAM, picks the most accurate model we have *measured* for your language that
+fits your card, downloads llama.cpp and the model, starts it with settings profiled op by op, and opens the chat page.
+You also get an OpenAI-compatible API at `http://127.0.0.1:8080/v1` for any app that speaks it. Offline, private, free.
+
+```
+localllm doctor     # what this GPU is good for: model sizes, speed, how much text it can hold
+localllm list       # every model we have measured, with scores per language
+localllm serve      # API only, no browser
+localllm eval       # score any running server in English + your language
+```
+
+## FAQ
+
+**Which local LLM should I run on my GPU?** Run `localllm doctor`. It lists which model sizes fit your card (4B up to
+120B MoE), at which quantization, how fast they should run, and the most accurate measured model for your language.
+
+**Can a 16 GB GPU run a 27B model?** Yes. Qwen3.8-27B at ~3.5 bits (12.2 GB) runs at ~50 tok/s on an RX 9070 XT and
+keeps 81.8% on English Global-MMLU-Lite. gemma-4-26B-A4B (13.3 GB) runs at ~69 tok/s with similar accuracy.
+
+**Is a 2-bit quantized model good enough?** Usually not for non-English use: 2-bit costs 8-13 accuracy points, and
+Hindi, Arabic and Thai lose the most (13 points).
+
+**Why is llama.cpp slow on my AMD (or Intel) GPU on Windows?** If Resizable BAR is off, llama.cpp's Vulkan backend puts
+buffers in a 256 MB host-visible heap backed by system RAM and decode drops up to 1.7x. `localllm` sets
+`GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1` for you ([llama.cpp#27097](https://github.com/ggml-org/llama.cpp/issues/27097)).
+
+**Does it work offline?** After the first download, yes. Nothing leaves your PC.
+
+**Which languages are measured?** 23 languages on Global-MMLU-Lite, 44 countries' own exams on INCLUDE, plus Thai
+(ThaiExam). `localllm eval --langs ...` measures any of them on your hardware.
+
+## What `localllm doctor` tells you
+
+```
+GPU AMD Radeon RX 9070 XT  15.9 GB  (640 GB/s)    RAM 32 GB    language: th
+
+Model sizes for this PC (whole model on the GPU = fast):
+  [OK  ] 4B                       Q8 4.2 GB  ~90 tok/s (est.)
+  [OK  ] 8B                       Q8 8.5 GB  ~45 tok/s (est.)
+  [OK  ] 14B                      Q6 11.5 GB  ~33 tok/s (est.)
+  [OK  ] 24-32B                   Q3 13.2 GB  ~29 tok/s (est.)
+  [SLOW] 30B MoE (3B active)      Q4 18.0 GB with experts in RAM - works, ~10-25 tok/s
+  [NO  ] 70B                      needs ~42.0 GB - too big for this PC
+
+Best measured model for you: gemma4-26b-a4b-qat  (MoE with ~4B active params: fastest)
+What it can do here:
+  TH  real local school/licence exams   65.7% correct  <- your language
+  holds ~78k tokens at once (~130 pages of text) next to the model
+  answers at ~69 tok/s
+```
+
+Speeds marked *est.* come from your card's memory bandwidth, calibrated on measured runs. Everything else is measured.
+
+## Measured results (RX 9070 XT 16 GB, Windows 11, llama.cpp Vulkan)
+
+Accuracy (%) on multiple-choice exams, zero-shot. **global** = Global-MMLU-Lite: the same 400 questions translated, so
+languages compare like for like. **regional** = INCLUDE: real exams written in each country (ThaiExam for Thai).
+
+| | Qwen3.8-27B Q3 (12.2 GB) | gemma-4-26B-A4B QAT Q4 (13.3 GB) | Qwen3.8-27B 2-bit (7.8 GB) |
+|---|---|---|---|
+| English | 81.8 | 82.2 | 74.2 |
+| Chinese | 76.2 / 74.7 | 73.5 / 66.5 | 67.8 / 67.8 |
+| Spanish | 80.2 / 76.8 | 74.5 / 75.2 | 70.8 / 69.2 |
+| Japanese | 73.5 / 87.6 | 74.5 / 81.9 | 65.8 / 77.9 |
+| Arabic | 70.8 / 71.2 | 71.5 / 73.6 | 60.8 / 57.2 |
+| Hindi | 69.0 / 74.3 | 69.5 / 71.0 | 56.2 / 55.5 |
+| Thai | – / 67.1 | – / 65.7 | – / 54.2 |
+| **decode speed** | **50 tok/s** (MTP) | **69 tok/s** | 40 tok/s |
+
+Cells are global / regional. Margins are about ±4 (global) and ±5 (regional) points at 95%, so `localllm` treats gaps
+under 2 points as a tie and picks the faster model.
+
+## Findings worth knowing
+
+1. **2-bit costs 8-13 points, and lower-resource languages pay the most.** Hindi, Arabic and Thai lose 13; English,
+   Chinese and Spanish about 8-9. A 177B MoE squeezed to 1.6 bits scored *below* a 27B at 3 bits.
+2. **Calibrating the quantization on your language doesn't help at ~3.5 bits.** A Thai-text importance matrix scored the
+   same as the stock one in Thai, English and Chinese (64.8 vs 64.6 Thai). At this level the number of bits matters,
+   the calibration text doesn't.
+3. **AMD/Intel cards without Resizable BAR lose up to 1.7x decode speed** in llama.cpp's Vulkan backend. Hybrid DeltaNet
+   models (Qwen3.5/3.8) suffer most: they rewrite a 3 MB state per layer per token.
+4. **Qwen3.8 GGUFs ship a multi-token-prediction head.** Drafting 2 tokens with it adds ~40% decode speed for free;
+   drafting 3 is slower.
+5. **The first run of a new llama.cpp build is slow** while the GPU driver compiles its shaders once (~15 s).
+
+## How the benchmark works
+
+`localllm eval` asks each question with thinking off and reads the log-probability of every answer letter from the first
+generated token, then takes the most likely one. It's prompt processing only, so a language takes a few minutes, and the
+result is deterministic. Data is downloaded at eval time from the original Apache-2.0 datasets
+([Global-MMLU-Lite](https://huggingface.co/datasets/CohereLabs/Global-MMLU-Lite),
+[INCLUDE](https://huggingface.co/datasets/CohereLabs/include-lite-44),
+[ThaiExam](https://huggingface.co/datasets/typhoon-ai/thai_exam)) and never redistributed. It measures knowledge and
+reasoning in multiple choice, not writing quality.
+
+## Contributing
+
+The catalog only grows with measurements. Run `localllm eval --langs en,<yours>` on your GPU and open a PR with
+`~/.localllm/results.json` and your GPU name. Other languages' local exams are very welcome. See [ROADMAP.md](ROADMAP.md)
+for what's next: using less system RAM (0.2), working alongside cloud provider APIs (0.3), and a speed-only release (0.4).
+
+## License
+
+MIT. Models keep their own licenses; benchmark data keeps its own (Apache-2.0).

make_localllm_easier-0.1.0/localllm/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.1.0"

make_localllm_easier-0.1.0/localllm/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+main()

make_localllm_easier-0.1.0/localllm/bench.py

@@ -0,0 +1,135 @@
+"""Multilingual multiple-choice benchmark for any OpenAI-compatible server (llama-server, Ollama, LM Studio, vLLM ...).
+
+Two kinds of test per language, so a score means something wherever you live:
+  global    Global-MMLU-Lite (CohereLabs, Apache-2.0): the same 400 questions translated into 23 languages,
+            so languages and models compare like for like
+  regional  INCLUDE-lite-44 (CohereLabs, Apache-2.0): real exams written in each country (~250 per language,
+            44 languages); ThaiExam (typhoon-ai, Apache-2.0) fills in Thai
+Zero-shot, thinking off, one token: the answer is the option letter with the highest log-probability. Fast
+(prompt processing only) and deterministic. Data is downloaded at eval time and cached, never redistributed.
+"""
+from __future__ import annotations
+
+import ast
+import json
+import locale
+import time
+import urllib.parse
+import urllib.request
+
+from .runtime import HOME
+
+ROWS = "https://datasets-server.huggingface.co/rows?dataset={ds}&config={cfg}&split={split}&offset={off}&length=100"
+GLOBAL_LANGS = ["ar", "bn", "cs", "cy", "de", "en", "es", "fr", "hi", "hu", "id", "it", "ja", "ko", "my", "or", "pt",
+                "sk", "sq", "sw", "tg", "yo", "zh"]
+INCLUDE = {"sq": "Albanian", "ar": "Arabic", "hy": "Armenian", "az": "Azerbaijani", "eu": "Basque", "be": "Belarusian",
+           "bn": "Bengali", "bg": "Bulgarian", "zh": "Chinese", "hr": "Croatian", "nl": "Dutch", "et": "Estonian",
+           "fi": "Finnish", "fr": "French", "ka": "Georgian", "de": "German", "el": "Greek", "he": "Hebrew",
+           "hi": "Hindi", "hu": "Hungarian", "id": "Indonesian", "it": "Italian", "ja": "Japanese", "kk": "Kazakh",
+           "ko": "Korean", "lt": "Lithuanian", "ms": "Malay", "ml": "Malayalam", "ne": "Nepali",
+           "mk": "North Macedonian", "fa": "Persian", "pl": "Polish", "pt": "Portuguese", "ru": "Russian",
+           "sr": "Serbian", "es": "Spanish", "tl": "Tagalog", "ta": "Tamil", "te": "Telugu", "tr": "Turkish",
+           "uk": "Ukrainian", "ur": "Urdu", "uz": "Uzbek", "vi": "Vietnamese"}
+THAIEXAM = "https://huggingface.co/datasets/typhoon-ai/thai_exam/resolve/main/data/{s}/{s}_test.jsonl"
+SYSTEM = "Answer the multiple-choice question. Reply with only the letter of the correct option."
+
+
+def system_language() -> str:
+    try:
+        loc = locale.getlocale()[0] or ""
+    except ValueError:
+        loc = ""
+    if "_" in loc or len(loc) == 2:
+        return loc.split("_")[0].lower()[:2]
+    names = {n.lower(): c for c, n in INCLUDE.items()} | {"thai": "th", "english": "en"}
+    return names.get(loc.split("_")[0].lower(), "en")
+
+
+def available(lang: str) -> list[str]:
+    return [s for s, ok in (("global", lang in GLOBAL_LANGS), ("regional", lang in INCLUDE or lang == "th")) if ok]
+
+
+def _rows(ds: str, cfg: str, split: str = "test") -> list[dict]:
+    out, off = [], 0
+    while True:
+        u = ROWS.format(ds=urllib.parse.quote(ds), cfg=urllib.parse.quote(cfg), split=split, off=off)
+        d = json.load(urllib.request.urlopen(u, timeout=120))
+        out += [r["row"] for r in d["rows"]]
+        off += 100
+        if off >= d.get("num_rows_total", 0) or not d["rows"]:
+            return out
+
+
+def load(suite: str, lang: str) -> list[dict]:
+    """[{'q': question, 'opts': [..], 'ans': index}] cached under ~/.localllm/bench/."""
+    cache = HOME / "bench" / f"{suite}-{lang}.json"
+    if cache.exists():
+        return json.loads(cache.read_text(encoding="utf-8"))
+    items = []
+    if suite == "global":
+        for r in _rows("CohereLabs/Global-MMLU-Lite", lang):
+            opts = [r[f"option_{c}"] for c in "abcd"]
+            items.append({"q": r["question"], "opts": opts, "ans": "ABCD".index(r["answer"].strip().upper())})
+    elif suite == "regional" and lang == "th":
+        for s in ["onet", "ic", "tgat", "tpat1", "a_level"]:
+            for line in urllib.request.urlopen(THAIEXAM.format(s=s)).read().decode("utf-8").splitlines():
+                if line.strip():
+                    r = json.loads(line)
+                    keys = [c for c in "abcde" if r.get(c)]
+                    items.append({"q": r["question"], "opts": [r[c] for c in keys],
+                                  "ans": keys.index(r["answer"].strip().lower())})
+    elif suite == "regional":
+        for r in _rows("CohereLabs/include-lite-44", INCLUDE[lang]):
+            opts = r["choices"] if isinstance(r["choices"], list) else ast.literal_eval(r["choices"])
+            items.append({"q": r["question"], "opts": opts, "ans": int(r["answer"])})
+    else:
+        raise ValueError(f"no {suite} test for '{lang}'")
+    cache.parent.mkdir(parents=True, exist_ok=True)
+    cache.write_text(json.dumps(items, ensure_ascii=False), encoding="utf-8")
+    return items
+
+
+def ask(url: str, item: dict) -> int | None:
+    letters = "ABCDEFGHIJ"[: len(item["opts"])]
+    opts = "\n".join(f"{l}. {o}" for l, o in zip(letters, item["opts"]))
+    body = {"messages": [{"role": "system", "content": SYSTEM},
+                         {"role": "user", "content": f"{item['q']}\n\n{opts}\n\nAnswer ({'/'.join(letters)}):"}],
+            "max_tokens": 1, "temperature": 0, "logprobs": True, "top_logprobs": 20,
+            "chat_template_kwargs": {"enable_thinking": False}}
+    req = urllib.request.Request(url.rstrip("/") + "/v1/chat/completions", data=json.dumps(body).encode(),
+                                 headers={"Content-Type": "application/json"})
+    d = json.load(urllib.request.urlopen(req, timeout=600))
+    best, best_lp = None, -1e9
+    for t in d["choices"][0]["logprobs"]["content"][0]["top_logprobs"]:
+        tok = t["token"].strip().upper().strip(".()")
+        if len(tok) == 1 and tok in letters and t["logprob"] > best_lp:
+            best, best_lp = letters.index(tok), t["logprob"]
+    return best
+
+
+def _save(name: str, res: dict) -> None:
+    out = HOME / "results.json"
+    allres = json.loads(out.read_text(encoding="utf-8")) if out.exists() else {}
+    allres[name] = {**allres.get(name, {}), **res}
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text(json.dumps(allres, ensure_ascii=False, indent=1), encoding="utf-8")
+
+
+def run(url: str, name: str, langs: list[str], limit: int = 0) -> dict:
+    """Scores are saved after every test, so an interrupted run keeps what it finished."""
+    res, t0 = {}, time.time()
+    for lang in langs:
+        suites = available(lang)
+        if not suites:
+            print(f"  {lang}: no benchmark yet (contributions welcome)")
+        for suite in suites:
+            items = load(suite, lang)
+            items = items[:limit] if limit else items
+            ok = sum(ask(url, it) == it["ans"] for it in items)
+            acc = round(100 * ok / len(items), 1)
+            res[f"{lang}/{suite}"] = {"acc": acc, "correct": ok, "n": len(items)}
+            margin = round(196 * (acc / 100 * (1 - acc / 100) / len(items)) ** 0.5, 1)
+            print(f"  {lang:3} {suite:9} {acc:5.1f}%  ±{margin}  ({ok}/{len(items)})", flush=True)
+            _save(name, {**res, "_meta": {"model": name, "seconds": round(time.time() - t0), "limit": limit}})
+    print(f"saved to {HOME / 'results.json'}  (share it: open a PR adding your GPU's numbers)")
+    return res

make_localllm_easier-0.1.0/localllm/catalog.py

@@ -0,0 +1,53 @@
+"""Models we have measured end to end. `scores` = accuracy (%) from `localllm eval` keyed "lang/suite" (see bench.py).
+gb = weights in GiB; kv_kb_per_token = KV cache per token at q8 (from the GGUF attention layout).
+Speeds: llama-server decode tok/s on an RX 9070 XT 16 GB (Windows, Vulkan) with the tuned launch in runtime.py.
+Add a model only after measuring it with `localllm eval`."""
+
+MODELS = {
+    "qwen3.8-27b-q3": {
+        "repo": "unsloth/Qwen3.8-27B-GGUF", "file": "Qwen3.8-27B-UD-Q3_K_XL.gguf", "gb": 12.2,
+        "kv_kb_per_token": 34.8, "fixed_cache_gb": 0.15, "max_ctx": 262144, "tok_s_9070xt": 50, "mtp": True,
+        "scores": {"en/global": 81.5, "zh/global": 76.2, "zh/regional": 74.7, "es/global": 80.2, "es/regional": 76.8, "hi/global": 69.0, "hi/regional": 74.3, "ar/global": 70.8, "ar/regional": 71.2, "ja/global": 73.5, "ja/regional": 87.6, "th/regional": 67.1},
+        "note": "dense 27B; built-in MTP head drafts 2 tokens",
+    },
+    "gemma4-26b-a4b-qat": {
+        "repo": "unsloth/gemma-4-26B-A4B-it-qat-GGUF", "file": "gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf", "gb": 13.3,
+        "kv_kb_per_token": 10.9, "fixed_cache_gb": 0.11, "max_ctx": 262144, "tok_s_9070xt": 69, "mtp": False,
+        "scores": {"en/global": 82.2, "zh/global": 73.5, "zh/regional": 66.5, "es/global": 74.5, "es/regional": 75.2, "hi/global": 69.5, "hi/regional": 71.0, "ar/global": 71.5, "ar/regional": 73.6, "ja/global": 74.5, "ja/regional": 81.9, "th/regional": 65.7},
+        "note": "MoE with ~4B active params: fastest",
+    },
+    "qwen3.8-27b-iq2": {
+        "repo": "unsloth/Qwen3.8-27B-GGUF", "file": "Qwen3.8-27B-UD-IQ2_S.gguf", "gb": 7.8,
+        "kv_kb_per_token": 34.8, "fixed_cache_gb": 0.15, "max_ctx": 262144, "tok_s_9070xt": 40, "mtp": False,
+        "scores": {"en/global": 74.2, "zh/global": 67.8, "zh/regional": 67.8, "es/global": 70.8, "es/regional": 69.2, "hi/global": 56.2, "hi/regional": 55.5, "ar/global": 60.8, "ar/regional": 57.2, "ja/global": 65.8, "ja/regional": 77.9, "th/regional": 54.2},
+        "note": "for 10-12 GB cards only: 2-bit costs 8-13 points, most in Hindi, Arabic, Thai",
+    },
+}
+
+
+def score(key: str, lang: str | None = None) -> float:
+    """Mean accuracy over the user's language tests if we have them, else over everything measured."""
+    s = MODELS[key]["scores"]
+    mine = [v for k, v in s.items() if lang and k.startswith(lang + "/")]
+    pool = mine or list(s.values())
+    return sum(pool) / len(pool) if pool else 0.0
+
+
+MIN_CTX = 8192   # a model only "fits" if it also leaves room for an 8k-token conversation
+
+
+def fits(key: str, vram_gb: float) -> bool:
+    from .sizing import context_tokens
+    return context_tokens(vram_gb, MODELS[key]) >= MIN_CTX
+
+
+TIE_POINTS = 2.0   # accuracy gaps this small are inside the benchmark's margin: prefer the faster model
+
+
+def pick(vram_gb: float, lang: str | None = None) -> str | None:
+    """Most accurate model (for `lang` when measured) that fits the card whole with an 8k context; near-ties go to speed."""
+    ok = [k for k in MODELS if fits(k, vram_gb)]
+    if not ok:
+        return None
+    best = max(score(k, lang) for k in ok)
+    return max((k for k in ok if score(k, lang) >= best - TIE_POINTS), key=lambda k: MODELS[k]["tok_s_9070xt"])

make_localllm_easier-0.1.0/localllm/cli.py

@@ -0,0 +1,193 @@
+"""make-localllm-easier: one command, the best local AI your computer can run.
+
+  localllm                  check this PC, pick the best model, download, start, open the chat page
+  localllm doctor           what GPU/RAM you have and which model fits
+  localllm list             every model we have measured
+  localllm serve [MODEL]    start an OpenAI-compatible server only (http://127.0.0.1:8080/v1)
+  localllm eval             score a running server in English + your language (global and local exams)
+"""
+from __future__ import annotations
+
+import argparse
+import os
+import subprocess
+import sys
+import time
+import urllib.request
+import webbrowser
+from pathlib import Path
+
+from . import __version__, catalog, runtime
+
+
+def _say(msg: str) -> None:
+    print(f"[localllm] {msg}", flush=True)
+
+
+def _download(url: str, dest: Path, label: str) -> None:
+    """Resumable download with a one-line progress bar."""
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    tmp = dest.with_suffix(dest.suffix + ".part")
+    done = tmp.stat().st_size if tmp.exists() else 0
+    req = urllib.request.Request(url, headers={"Range": f"bytes={done}-"} if done else {})
+    with urllib.request.urlopen(req) as r, open(tmp, "ab") as f:
+        total = done + int(r.headers.get("Content-Length") or 0)
+        t0, got = time.time(), 0
+        while chunk := r.read(1 << 22):
+            f.write(chunk); got += len(chunk)
+            if total:
+                pct = 100 * (done + got) / total
+                speed = got / max(time.time() - t0, 1e-3) / 2**20
+                print(f"\r  {label}: {pct:5.1f}% of {total / 2**30:.1f} GB  ({speed:.0f} MB/s)  ", end="", flush=True)
+    print()
+    tmp.replace(dest)
+
+
+def _model_path(key: str) -> Path:
+    m = catalog.MODELS[key]
+    for d in filter(None, os.environ.get("LOCALLLM_MODELS", "").split(os.pathsep)):
+        if (Path(d) / m["file"]).exists():
+            return Path(d) / m["file"]
+    dest = runtime.HOME / "models" / m["file"]
+    if not dest.exists():
+        _say(f"downloading {key} ({m['gb']} GB, one time) ...")
+        _download(f"https://huggingface.co/{m['repo']}/resolve/main/{m['file']}", dest, m["file"])
+    return dest
+
+
+def _machine():
+    server = runtime.find_server()
+    devs = runtime.devices(server)
+    return server, devs, runtime.best_device(devs), runtime.ram_gb()
+
+
+def _start(key: str | None, port: int, ctx: int) -> tuple[subprocess.Popen, str]:
+    server, _devs, dev, _ram = _machine()
+    from .bench import system_language
+    key = key or (catalog.pick(dev["total_gb"], system_language()) if dev else None)
+    if not key:
+        sys.exit("[localllm] no measured model fits this GPU yet (need >= 10 GB VRAM). See `localllm list`.")
+    model = _model_path(key)
+    args = runtime.server_args(model, dev["id"] if dev else None, port, ctx, catalog.MODELS[key]["mtp"])
+    runtime.HOME.mkdir(parents=True, exist_ok=True)
+    log = open(runtime.HOME / "llama-server.log", "ab")
+    proc = subprocess.Popen([str(server), *args], env=runtime.server_env(), stdout=log, stderr=subprocess.STDOUT)
+    _say(f"loading {key} on {dev['name'] if dev else 'CPU'} ...")
+    url = f"http://127.0.0.1:{port}"
+    for _ in range(1000):
+        if proc.poll() is not None:
+            sys.exit(f"[localllm] llama-server stopped (exit {proc.returncode}); log: {runtime.HOME / 'llama-server.log'}")
+        try:
+            if b'"ok"' in urllib.request.urlopen(url + "/health", timeout=2).read():
+                return proc, url
+        except OSError:
+            pass
+        time.sleep(0.3)
+    proc.kill()
+    sys.exit("[localllm] model did not load within 5 minutes")
+
+
+def cmd_run(a) -> None:
+    proc, url = _start(a.model, a.port, a.ctx)
+    _say(f"ready. chat: {url}   API (OpenAI-compatible): {url}/v1   Ctrl+C to stop")
+    if not a.no_browser:
+        webbrowser.open(url)
+    try:
+        proc.wait()
+    except KeyboardInterrupt:
+        proc.terminate()
+
+
+def cmd_serve(a) -> None:
+    a.no_browser = True
+    cmd_run(a)
+
+
+ICON = {"fits": "OK  ", "low-bits": "WARN", "offload-moe": "SLOW", "offload-dense": "SLOW", "too-big": "NO  "}
+
+
+def cmd_doctor(_a) -> None:
+    from . import sizing
+    from .bench import system_language
+    server, devs, dev, ram = _machine()
+    lang = system_language()
+    gpu = dev["name"] if dev else "no GPU found"
+    vram = dev["total_gb"] if dev else 0.0
+    bw = sizing.bandwidth(gpu)
+    print(f"GPU {gpu}  {vram:.1f} GB" + (f"  ({bw} GB/s)" if bw else "") + f"    RAM {ram:.0f} GB    language: {lang}")
+    print("\nModel sizes for this PC (whole model on the GPU = fast):")
+    for r in sizing.tiers(vram, ram, gpu):
+        speed = f"~{r['tok_s']} tok/s (est.)" if r["tok_s"] else ""
+        what = {"fits": f"{r['quant']} {r['gb']} GB  {speed}",
+                "low-bits": f"only at {r['quant']} ({r['gb']} GB) - fits, but quality drops sharply below 3 bits",
+                "offload-moe": f"{r['quant']} {r['gb']} GB with experts in RAM - works, ~10-25 tok/s",
+                "offload-dense": f"{r['quant']} {r['gb']} GB with layers in RAM - very slow (< 5 tok/s)",
+                "too-big": f"needs ~{r['gb']} GB - too big for this PC"}[r["status"]]
+        print(f"  [{ICON[r['status']]}] {r['shape']:24} {what}")
+    key = catalog.pick(vram, lang) if dev else None
+    if not key:
+        print("\nNo measured model fits this GPU yet. Run `localllm list`, or help by measuring one (`localllm eval`).")
+        return
+    m = catalog.MODELS[key]
+    ctx = sizing.context_tokens(vram, m)
+    print(f"\nBest measured model for you: {key}  ({m['note']})")
+    print("What it can do here:")
+    shown = [t for t in sorted(m["scores"]) if t.split("/")[0] in (lang, "en")]
+    others = sorted({t.split("/")[0] for t in m["scores"]} - {lang, "en"})
+    for test in shown:
+        acc = m["scores"][test]
+        tl, suite = test.split("/")
+        kind = "translated world-knowledge exam" if suite == "global" else "real local school/licence exams"
+        mark = "  <- your language" if tl == lang else ""
+        print(f"  {tl.upper():3} {kind:32} {acc:5.1f}% correct{mark}")
+    if others:
+        print(f"  also measured in {', '.join(others)} (`localllm list`)")
+    print(f"  holds ~{ctx // 1000}k tokens at once (~{ctx // sizing.TOKENS_PER_PAGE} pages of text) next to the model")
+    same = bw == sizing.BANDWIDTH["rx 9070 xt"]
+    est = m["tok_s_9070xt"] if same else (int(m["tok_s_9070xt"] * bw / 640) if bw else None)
+    if est:
+        print(f"  answers at ~{est} tok/s" + ("" if same else " (estimated from memory bandwidth)"))
+    print(f"\nRun it: localllm        (llama.cpp: {server})")
+
+
+def cmd_list(_a) -> None:
+    print(f"{'model':22} {'weights':>8} {'tok/s*':>7}  scores")
+    for k, m in sorted(catalog.MODELS.items(), key=lambda kv: -catalog.score(kv[0])):
+        sc = "  ".join(f"{t} {v:.1f}" for t, v in m["scores"].items())
+        print(f"{k:22} {m['gb']:6.1f}GB {m['tok_s_9070xt']:7}  {sc}")
+    print("* decode speed on an RX 9070 XT 16 GB.  scores: accuracy % from `localllm eval` (lang/suite)")
+
+
+def cmd_eval(a) -> None:
+    from . import bench
+    langs = a.langs.split(",") if a.langs else sorted({"en", bench.system_language()})
+    _say(f"benchmarking {a.url} in: {', '.join(langs)}  (pick others with --langs ja,de,...)")
+    bench.run(a.url, a.name, langs, a.limit)
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(prog="localllm", description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--version", action="version", version=__version__)
+    ap.add_argument("--model", choices=list(catalog.MODELS), help="override the automatic pick")
+    ap.add_argument("--port", type=int, default=8080)
+    ap.add_argument("--ctx", type=int, default=8192, help="context length in tokens")
+    ap.add_argument("--no-browser", action="store_true")
+    ap.set_defaults(fn=cmd_run)
+    sub = ap.add_subparsers(dest="cmd")
+    sub.add_parser("doctor").set_defaults(fn=cmd_doctor)
+    sub.add_parser("list").set_defaults(fn=cmd_list)
+    s = sub.add_parser("serve"); s.add_argument("model", nargs="?", choices=list(catalog.MODELS))
+    s.add_argument("--port", type=int, default=8080); s.add_argument("--ctx", type=int, default=8192)
+    s.set_defaults(fn=cmd_serve)
+    e = sub.add_parser("eval"); e.add_argument("--url", default="http://127.0.0.1:8080")
+    e.add_argument("--name", default="model"); e.add_argument("--limit", type=int, default=0)
+    e.add_argument("--langs", help="comma-separated ISO codes, default: en + this PC's language")
+    e.set_defaults(fn=cmd_eval)
+    a = ap.parse_args()
+    if a.fn is cmd_serve:
+        a.model = a.model or None
+    a.fn(a)
+
+
+if __name__ == "__main__":
+    main()

make_localllm_easier-0.1.0/localllm/runtime.py

@@ -0,0 +1,102 @@
+"""Fetch an upstream llama.cpp build, list GPUs, and launch llama-server with the settings that measured fastest.
+
+Tuning (each measured on an RX 9070 XT, see README):
+  GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1  1.7x decode when Resizable BAR is off (llama.cpp#27097); harmless when on
+  -np 1 -kvu                             single user: one slot, unified KV
+  --spec-type draft-mtp --spec-draft-n-max 2   models that ship an MTP head (Qwen3.8): +40% decode
+  -fit off, --load-mode none             skip the fit dry-run (~0.6 s) and read weights straight into VRAM
+"""
+from __future__ import annotations
+
+import io
+import json
+import os
+import platform
+import re
+import subprocess
+import tarfile
+import urllib.request
+import zipfile
+from pathlib import Path
+
+HOME = Path(os.environ.get("LOCALLLM_HOME", Path.home() / ".localllm"))
+EXE = "llama-server.exe" if os.name == "nt" else "llama-server"
+
+
+def _asset() -> str:
+    system = {"Windows": "win", "Linux": "ubuntu", "Darwin": "macos"}[platform.system()]
+    arch = "arm64" if platform.machine().lower() in ("arm64", "aarch64") else "x64"
+    if system == "macos":
+        return f"bin-macos-{arch}"
+    return f"bin-{system}-vulkan-{arch}"
+
+
+def find_server() -> Path:
+    if os.environ.get("LOCALLLM_LLAMA_SERVER"):
+        return Path(os.environ["LOCALLLM_LLAMA_SERVER"])
+    found = sorted((HOME / "llama.cpp").rglob(EXE)) if (HOME / "llama.cpp").exists() else []
+    return found[-1] if found else fetch_server()
+
+
+def fetch_server() -> Path:
+    # "latest" can be a source-only tag (v0.6.0 had no binaries); the binaries ride on the per-build bNNNNN releases
+    rels = json.load(urllib.request.urlopen("https://api.github.com/repos/ggml-org/llama.cpp/releases?per_page=10"))
+    want = _asset()
+    rel, asset = next((r, a) for r in rels for a in r["assets"] if want in a["name"] and a["name"].startswith("llama-"))
+    dest = HOME / "llama.cpp" / rel["tag_name"]
+    print(f"downloading {asset['name']} ({asset['size'] / 2**20:.0f} MB) ...")
+    data = urllib.request.urlopen(asset["browser_download_url"]).read()
+    if asset["name"].endswith(".zip"):
+        zipfile.ZipFile(io.BytesIO(data)).extractall(dest)
+    else:
+        tarfile.open(fileobj=io.BytesIO(data)).extractall(dest, filter="data")
+    exe = next(dest.rglob(EXE))
+    if os.name != "nt":
+        exe.chmod(0o755)
+    return exe
+
+
+def devices(server: Path) -> list[dict]:
+    """[{'id': 'Vulkan0', 'name': ..., 'total_gb': .., 'free_gb': ..}] from `llama-server --list-devices`."""
+    out = subprocess.run([str(server), "--list-devices"], capture_output=True, text=True, errors="replace")
+    devs = []
+    for m in re.finditer(r"^\s*(\w+\d+): (.+?) \((\d+) MiB, (\d+) MiB free\)", out.stdout + out.stderr, re.M):
+        devs.append({"id": m[1], "name": m[2], "total_gb": int(m[3]) / 1024, "free_gb": int(m[4]) / 1024})
+    return devs
+
+
+def best_device(devs: list[dict]) -> dict | None:
+    """Largest-memory device; integrated GPUs report shared RAM, so prefer names that aren't iGPUs."""
+    igpu = re.compile(r"UHD|Iris|Radeon\(TM\) Graphics|Radeon Graphics|890M|780M|Apple", re.I)
+    pool = [d for d in devs if not igpu.search(d["name"])] or devs
+    return max(pool, key=lambda d: d["total_gb"]) if pool else None
+
+
+def ram_gb() -> float:
+    if os.name == "nt":
+        import ctypes
+
+        class MS(ctypes.Structure):
+            _fields_ = [("len", ctypes.c_ulong), ("load", ctypes.c_ulong), ("total", ctypes.c_ulonglong),
+                        ("avail", ctypes.c_ulonglong), ("a", ctypes.c_ulonglong), ("b", ctypes.c_ulonglong),
+                        ("c", ctypes.c_ulonglong), ("d", ctypes.c_ulonglong), ("e", ctypes.c_ulonglong)]
+        s = MS(); s.len = ctypes.sizeof(MS)
+        ctypes.windll.kernel32.GlobalMemoryStatusEx(ctypes.byref(s))
+        return s.total / 2**30
+    if Path("/proc/meminfo").exists():
+        return int(re.search(r"MemTotal:\s+(\d+)", Path("/proc/meminfo").read_text())[1]) / 2**20
+    return int(subprocess.run(["sysctl", "-n", "hw.memsize"], capture_output=True, text=True).stdout) / 2**30
+
+
+def server_args(model: Path, device: str | None, port: int, ctx: int, mtp: bool) -> list[str]:
+    args = ["-m", str(model), "--host", "127.0.0.1", "--port", str(port), "-c", str(ctx), "-ngl", "999",
+            "-fa", "on", "-ctk", "q8_0", "-ctv", "q8_0", "-np", "1", "-kvu", "-fit", "off", "--load-mode", "none"]
+    if device:
+        args += ["-dev", device]
+    if mtp:
+        args += ["--spec-type", "draft-mtp", "--spec-draft-n-max", "2"]
+    return args
+
+
+def server_env() -> dict:
+    return {**os.environ, "GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM": os.environ.get("GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM", "1")}

make_localllm_easier-0.1.0/localllm/sizing.py

@@ -0,0 +1,76 @@
+"""What size of model a GPU suits, how fast it should run, and how long a document it can hold.
+
+Everything here is arithmetic from the card's memory size and bandwidth, calibrated on measured runs:
+  dense, whole model in VRAM   decode ≈ 0.6 x bandwidth / bytes read per token   (Qwen3.8-27B Q3 on 640 GB/s: 35 tok/s)
+  MoE, whole model in VRAM     decode ≈ 0.3 x bandwidth / active bytes per token (gemma-4-26B-A4B Q4: 69 tok/s)
+Estimates are labelled as such; measured numbers live in catalog.py.
+"""
+from __future__ import annotations
+
+import re
+
+# memory bandwidth in GB/s of common consumer GPUs (vendor specs)
+BANDWIDTH = {
+    "rtx 5090": 1792, "rtx 5080": 960, "rtx 5070 ti": 896, "rtx 5070": 672, "rtx 5060 ti": 448, "rtx 5060": 448,
+    "rtx 4090": 1008, "rtx 4080": 717, "rtx 4070 ti": 504, "rtx 4070": 504, "rtx 4060 ti": 288, "rtx 4060": 272,
+    "rtx 3090": 936, "rtx 3080": 760, "rtx 3070": 448, "rtx 3060 ti": 448, "rtx 3060": 360,
+    "rx 9070 xt": 640, "rx 9070": 640, "rx 9060 xt": 320, "rx 7900 xtx": 960, "rx 7900 xt": 800, "rx 7800 xt": 624,
+    "rx 7700 xt": 432, "rx 7600": 288, "rx 6900 xt": 512, "rx 6800 xt": 512, "rx 6700 xt": 384,
+    "arc b580": 456, "arc a770": 560, "arc a750": 512,
+}
+# bits per weight including quantization overhead
+QUANTS = [("Q8", 8.5), ("Q6", 6.6), ("Q4", 4.8), ("Q3", 3.9), ("Q2", 2.9)]
+LOW_BITS = 3.0          # below this, measured accuracy drops sharply (Qwen3.8-27B: -17 ThaiExam points at 2.9 bpw)
+DESKTOP_GB = 1.0        # what the OS/desktop usually keeps on the card
+OVERHEAD_GB = 0.7       # compute buffers + small fixed caches
+TOKENS_PER_PAGE = 600   # ~450 English words
+
+# reference shapes people actually download: (label, total params B, active params B)
+SHAPES = [("4B", 4, 4), ("8B", 8, 8), ("14B", 14, 14), ("24-32B", 27, 27), ("30B MoE (3B active)", 30, 3),
+          ("70B", 70, 70), ("120B MoE (10B active)", 120, 10)]
+
+
+def bandwidth(gpu_name: str) -> int | None:
+    n = gpu_name.lower()
+    for key in sorted(BANDWIDTH, key=len, reverse=True):
+        if re.search(r"\b" + re.escape(key) + r"\b", n):
+            return BANDWIDTH[key]
+    return None
+
+
+def size_gb(params_b: float, bpw: float) -> float:
+    return params_b * bpw / 8
+
+
+def tok_s(total_b: float, active_b: float, bpw: float, bw: int | None) -> int | None:
+    if not bw:
+        return None
+    eff = 0.6 if active_b == total_b else 0.3
+    return int(eff * bw / size_gb(active_b, bpw))
+
+
+def tiers(vram_gb: float, ram_gb: float, gpu_name: str) -> list[dict]:
+    """One row per reference shape: best quant that fits the card whole, else whether RAM offload works."""
+    usable, bw, rows = vram_gb - DESKTOP_GB - OVERHEAD_GB, bandwidth(gpu_name), []
+    for label, total, active in SHAPES:
+        fit = next(((q, b) for q, b in QUANTS if size_gb(total, b) <= usable), None)
+        moe = active < total
+        if fit and fit[1] < LOW_BITS and moe and size_gb(total, 4.8) <= usable + 0.7 * ram_gb:
+            fit = None   # a MoE keeps its quality better at Q4 with some experts in RAM than at 2 bits on the GPU
+        if fit:
+            q, b = fit
+            rows.append({"shape": label, "status": "low-bits" if b < LOW_BITS else "fits", "quant": q,
+                         "gb": round(size_gb(total, b), 1), "tok_s": tok_s(total, active, b, bw)})
+        elif size_gb(total, 4.8) <= usable + 0.7 * ram_gb:
+            rows.append({"shape": label, "status": "offload-moe" if active < total else "offload-dense", "quant": "Q4",
+                         "gb": round(size_gb(total, 4.8), 1), "tok_s": None})
+        else:
+            rows.append({"shape": label, "status": "too-big", "quant": None, "gb": round(size_gb(total, 4.8), 1),
+                         "tok_s": None})
+    return rows
+
+
+def context_tokens(vram_gb: float, model: dict) -> int:
+    """How many tokens of conversation/document fit next to the weights (KV cache at q8)."""
+    free = vram_gb - DESKTOP_GB - OVERHEAD_GB - model["gb"] - model.get("fixed_cache_gb", 0)
+    return max(0, min(model.get("max_ctx", 131072), int(free * 2**20 / model["kv_kb_per_token"])))

make_localllm_easier-0.1.0/make_localllm_easier.egg-info/PKG-INFO

@@ -0,0 +1,130 @@
+Metadata-Version: 2.4
+Name: make-localllm-easier
+Version: 0.1.0
+Summary: One command runs the best local AI your PC can handle: measured model picks + a tuned llama.cpp (AMD, NVIDIA, Intel, Apple)
+License: MIT
+Project-URL: Homepage, https://github.com/phonology024/make-localllm-easier
+Keywords: llm,local-llm,llama.cpp,vulkan,amd,gguf,thai,offline-ai
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# make-localllm-easier — run the best local LLM your GPU can handle, in one command
+
+**`localllm` picks, downloads and runs the most accurate local AI model for your PC and your language, chosen from real
+benchmark measurements, with llama.cpp tuned for AMD, NVIDIA, Intel and Apple GPUs.**
+
+```
+pip install make-localllm-easier
+localllm
+```
+
+That's it. `localllm` checks your GPU and RAM, picks the most accurate model we have *measured* for your language that
+fits your card, downloads llama.cpp and the model, starts it with settings profiled op by op, and opens the chat page.
+You also get an OpenAI-compatible API at `http://127.0.0.1:8080/v1` for any app that speaks it. Offline, private, free.
+
+```
+localllm doctor     # what this GPU is good for: model sizes, speed, how much text it can hold
+localllm list       # every model we have measured, with scores per language
+localllm serve      # API only, no browser
+localllm eval       # score any running server in English + your language
+```
+
+## FAQ
+
+**Which local LLM should I run on my GPU?** Run `localllm doctor`. It lists which model sizes fit your card (4B up to
+120B MoE), at which quantization, how fast they should run, and the most accurate measured model for your language.
+
+**Can a 16 GB GPU run a 27B model?** Yes. Qwen3.8-27B at ~3.5 bits (12.2 GB) runs at ~50 tok/s on an RX 9070 XT and
+keeps 81.8% on English Global-MMLU-Lite. gemma-4-26B-A4B (13.3 GB) runs at ~69 tok/s with similar accuracy.
+
+**Is a 2-bit quantized model good enough?** Usually not for non-English use: 2-bit costs 8-13 accuracy points, and
+Hindi, Arabic and Thai lose the most (13 points).
+
+**Why is llama.cpp slow on my AMD (or Intel) GPU on Windows?** If Resizable BAR is off, llama.cpp's Vulkan backend puts
+buffers in a 256 MB host-visible heap backed by system RAM and decode drops up to 1.7x. `localllm` sets
+`GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1` for you ([llama.cpp#27097](https://github.com/ggml-org/llama.cpp/issues/27097)).
+
+**Does it work offline?** After the first download, yes. Nothing leaves your PC.
+
+**Which languages are measured?** 23 languages on Global-MMLU-Lite, 44 countries' own exams on INCLUDE, plus Thai
+(ThaiExam). `localllm eval --langs ...` measures any of them on your hardware.
+
+## What `localllm doctor` tells you
+
+```
+GPU AMD Radeon RX 9070 XT  15.9 GB  (640 GB/s)    RAM 32 GB    language: th
+
+Model sizes for this PC (whole model on the GPU = fast):
+  [OK  ] 4B                       Q8 4.2 GB  ~90 tok/s (est.)
+  [OK  ] 8B                       Q8 8.5 GB  ~45 tok/s (est.)
+  [OK  ] 14B                      Q6 11.5 GB  ~33 tok/s (est.)
+  [OK  ] 24-32B                   Q3 13.2 GB  ~29 tok/s (est.)
+  [SLOW] 30B MoE (3B active)      Q4 18.0 GB with experts in RAM - works, ~10-25 tok/s
+  [NO  ] 70B                      needs ~42.0 GB - too big for this PC
+
+Best measured model for you: gemma4-26b-a4b-qat  (MoE with ~4B active params: fastest)
+What it can do here:
+  TH  real local school/licence exams   65.7% correct  <- your language
+  holds ~78k tokens at once (~130 pages of text) next to the model
+  answers at ~69 tok/s
+```
+
+Speeds marked *est.* come from your card's memory bandwidth, calibrated on measured runs. Everything else is measured.
+
+## Measured results (RX 9070 XT 16 GB, Windows 11, llama.cpp Vulkan)
+
+Accuracy (%) on multiple-choice exams, zero-shot. **global** = Global-MMLU-Lite: the same 400 questions translated, so
+languages compare like for like. **regional** = INCLUDE: real exams written in each country (ThaiExam for Thai).
+
+| | Qwen3.8-27B Q3 (12.2 GB) | gemma-4-26B-A4B QAT Q4 (13.3 GB) | Qwen3.8-27B 2-bit (7.8 GB) |
+|---|---|---|---|
+| English | 81.8 | 82.2 | 74.2 |
+| Chinese | 76.2 / 74.7 | 73.5 / 66.5 | 67.8 / 67.8 |
+| Spanish | 80.2 / 76.8 | 74.5 / 75.2 | 70.8 / 69.2 |
+| Japanese | 73.5 / 87.6 | 74.5 / 81.9 | 65.8 / 77.9 |
+| Arabic | 70.8 / 71.2 | 71.5 / 73.6 | 60.8 / 57.2 |
+| Hindi | 69.0 / 74.3 | 69.5 / 71.0 | 56.2 / 55.5 |
+| Thai | – / 67.1 | – / 65.7 | – / 54.2 |
+| **decode speed** | **50 tok/s** (MTP) | **69 tok/s** | 40 tok/s |
+
+Cells are global / regional. Margins are about ±4 (global) and ±5 (regional) points at 95%, so `localllm` treats gaps
+under 2 points as a tie and picks the faster model.
+
+## Findings worth knowing
+
+1. **2-bit costs 8-13 points, and lower-resource languages pay the most.** Hindi, Arabic and Thai lose 13; English,
+   Chinese and Spanish about 8-9. A 177B MoE squeezed to 1.6 bits scored *below* a 27B at 3 bits.
+2. **Calibrating the quantization on your language doesn't help at ~3.5 bits.** A Thai-text importance matrix scored the
+   same as the stock one in Thai, English and Chinese (64.8 vs 64.6 Thai). At this level the number of bits matters,
+   the calibration text doesn't.
+3. **AMD/Intel cards without Resizable BAR lose up to 1.7x decode speed** in llama.cpp's Vulkan backend. Hybrid DeltaNet
+   models (Qwen3.5/3.8) suffer most: they rewrite a 3 MB state per layer per token.
+4. **Qwen3.8 GGUFs ship a multi-token-prediction head.** Drafting 2 tokens with it adds ~40% decode speed for free;
+   drafting 3 is slower.
+5. **The first run of a new llama.cpp build is slow** while the GPU driver compiles its shaders once (~15 s).
+
+## How the benchmark works
+
+`localllm eval` asks each question with thinking off and reads the log-probability of every answer letter from the first
+generated token, then takes the most likely one. It's prompt processing only, so a language takes a few minutes, and the
+result is deterministic. Data is downloaded at eval time from the original Apache-2.0 datasets
+([Global-MMLU-Lite](https://huggingface.co/datasets/CohereLabs/Global-MMLU-Lite),
+[INCLUDE](https://huggingface.co/datasets/CohereLabs/include-lite-44),
+[ThaiExam](https://huggingface.co/datasets/typhoon-ai/thai_exam)) and never redistributed. It measures knowledge and
+reasoning in multiple choice, not writing quality.
+
+## Contributing
+
+The catalog only grows with measurements. Run `localllm eval --langs en,<yours>` on your GPU and open a PR with
+`~/.localllm/results.json` and your GPU name. Other languages' local exams are very welcome. See [ROADMAP.md](ROADMAP.md)
+for what's next: using less system RAM (0.2), working alongside cloud provider APIs (0.3), and a speed-only release (0.4).
+
+## License
+
+MIT. Models keep their own licenses; benchmark data keeps its own (Apache-2.0).

make_localllm_easier-0.1.0/make_localllm_easier.egg-info/SOURCES.txt

@@ -0,0 +1,16 @@
+LICENSE
+README.md
+pyproject.toml
+localllm/__init__.py
+localllm/__main__.py
+localllm/bench.py
+localllm/catalog.py
+localllm/cli.py
+localllm/runtime.py
+localllm/sizing.py
+make_localllm_easier.egg-info/PKG-INFO
+make_localllm_easier.egg-info/SOURCES.txt
+make_localllm_easier.egg-info/dependency_links.txt
+make_localllm_easier.egg-info/entry_points.txt
+make_localllm_easier.egg-info/top_level.txt
+tests/test_core.py

make_localllm_easier-0.1.0/make_localllm_easier.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

make_localllm_easier-0.1.0/make_localllm_easier.egg-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+localllm = localllm.cli:main
+make-localllm-easier = localllm.cli:main

make_localllm_easier-0.1.0/make_localllm_easier.egg-info/top_level.txt

@@ -0,0 +1 @@
+localllm

make_localllm_easier-0.1.0/pyproject.toml

@@ -0,0 +1,24 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "make-localllm-easier"
+version = "0.1.0"
+description = "One command runs the best local AI your PC can handle: measured model picks + a tuned llama.cpp (AMD, NVIDIA, Intel, Apple)"
+readme = "README.md"
+license = { text = "MIT" }
+requires-python = ">=3.9"
+dependencies = []
+keywords = ["llm", "local-llm", "llama.cpp", "vulkan", "amd", "gguf", "thai", "offline-ai"]
+classifiers = ["Programming Language :: Python :: 3", "License :: OSI Approved :: MIT License", "Operating System :: OS Independent", "Topic :: Scientific/Engineering :: Artificial Intelligence"]
+
+[project.urls]
+Homepage = "https://github.com/phonology024/make-localllm-easier"
+
+[project.scripts]
+localllm = "localllm.cli:main"
+make-localllm-easier = "localllm.cli:main"
+
+[tool.setuptools]
+packages = ["localllm"]

make_localllm_easier-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

make_localllm_easier-0.1.0/tests/test_core.py

@@ -0,0 +1,48 @@
+from localllm import catalog, runtime
+
+
+def test_pick_by_vram():
+    assert catalog.pick(8) is None
+    assert catalog.pick(12) == "qwen3.8-27b-iq2"
+    assert catalog.pick(16, "th") == "gemma4-26b-a4b-qat"
+
+
+def test_server_args_mtp_and_device():
+    a = runtime.server_args(runtime.Path("m.gguf"), "Vulkan0", 8080, 4096, mtp=True)
+    assert a[a.index("-dev") + 1] == "Vulkan0" and "draft-mtp" in a and a[a.index("--spec-draft-n-max") + 1] == "2"
+    assert "draft-mtp" not in runtime.server_args(runtime.Path("m.gguf"), None, 8080, 4096, mtp=False)
+
+
+def test_env_disables_host_visible_vidmem_unless_set(monkeypatch):
+    monkeypatch.delenv("GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM", raising=False)
+    assert runtime.server_env()["GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM"] == "1"
+    monkeypatch.setenv("GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM", "0")
+    assert runtime.server_env()["GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM"] == "0"
+
+
+def test_best_device_skips_igpu():
+    devs = [{"id": "Vulkan0", "name": "AMD Radeon RX 9070 XT", "total_gb": 15.9},
+            {"id": "Vulkan1", "name": "Intel(R) UHD Graphics 770", "total_gb": 15.9}]
+    assert runtime.best_device(devs)["id"] == "Vulkan0"
+
+
+def test_bench_coverage():
+    from localllm import bench
+    assert bench.available("en") == ["global"]
+    assert bench.available("th") == ["regional"]
+    assert bench.available("ja") == ["global", "regional"]
+    assert bench.available("xx") == []
+
+
+def test_sizing_tiers_16gb():
+    from localllm import sizing
+    rows = {r["shape"]: r for r in sizing.tiers(15.9, 32, "AMD Radeon RX 9070 XT")}
+    assert rows["8B"]["status"] == "fits" and rows["24-32B"]["quant"] == "Q3"
+    assert rows["30B MoE (3B active)"]["status"] == "offload-moe"
+    assert rows["70B"]["status"] == "too-big"
+    assert sizing.bandwidth("NVIDIA GeForce RTX 4060 Ti") == 288 and sizing.bandwidth("Mystery GPU") is None
+
+
+def test_pick_prefers_accuracy_then_speed_on_ties():
+    assert catalog.pick(15.9, "zh") == "qwen3.8-27b-q3"      # 5.5 points better in Chinese
+    assert catalog.pick(15.9, "en") == "gemma4-26b-a4b-qat"  # tie on accuracy, faster