ggufit 0.1.0__tar.gz

ggufit-0.1.0/PKG-INFO

@@ -0,0 +1,102 @@
+Metadata-Version: 2.4
+Name: ggufit
+Version: 0.1.0
+Summary: Check whether a local LLM can run on your machine (CPU-only inference).
+Author: Mouad
+License: MIT
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Utilities
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: psutil>=5.9
+Requires-Dist: numpy>=1.24
+
+# ggufit
+
+Check whether a local LLM can run on your machine, CPU-only — from PowerShell or any terminal.
+
+Uses real formulas (model memory footprint, KV cache size, and a measured memory-bandwidth
+micro-benchmark) rather than guesses, to estimate whether a model fits in RAM and roughly
+how fast it'll generate tokens. Correctly handles MoE models (speed driven by active
+experts, not total params) and SSM/MLA architectures (Mamba, DeepSeek's MLA) that don't
+use standard multi-head attention.
+
+## Install
+
+```bash
+# Linux/macOS - recommended (avoids PEP 668 "externally managed environment" errors)
+sudo apt install pipx      # if not already installed
+pipx ensurepath            # reopen terminal after this
+cd ggufit/                 # the folder containing pyproject.toml
+pipx install .
+
+# Windows PowerShell
+pip install .
+
+# Active development (editable install - code changes apply without reinstalling)
+pipx install -e .
+```
+
+`ggufit` is now available as a global command.
+
+## Usage
+
+```bash
+# Full hardware scan: shows your CPU/RAM/bandwidth and which models fit
+ggufit scan
+
+# Check one specific model
+ggufit llama3.1
+ggufit mistral
+ggufit qwen2.5-14b
+
+# Force a specific quantization level
+ggufit llama3.1 --quant q4
+
+# Evaluate at a longer context length
+ggufit qwen2.5-32b --context 16384
+```
+
+## How it works
+
+For each model:
+1. **Model size (RAM)** = total params × bytes-per-parameter (varies by quant: FP16, Q8, Q6, Q5, Q4, Q3, Q2)
+2. **KV cache** = `2 × layers × hidden_size × context_len × bytes_per_param × kv_cache_multiplier`
+   - `kv_cache_multiplier` defaults to 1.0 (standard MHA), and is set lower for architectures
+     that don't use full attention at every layer: `0.0` for pure SSM/Mamba (no attention at all),
+     `~0.15` for MLA (DeepSeek-V2/V3/R1, MiniCPM3), `~0.125` for hybrid Mamba+attention (Jamba).
+3. **Total RAM needed** = `(model size + KV cache) × 1.2` overhead factor
+4. **Speed estimate** = `measured_memory_bandwidth / active_size`, where `active_size` is the
+   total model size for dense models, or just the active experts' size for MoE models
+   (`active_params_billion`) — since only those weights are streamed from RAM per token.
+
+`ggufit` runs a quick real memory-bandwidth benchmark (a large array copy) instead of guessing
+from RAM specs, since achievable bandwidth depends heavily on channel configuration.
+
+## Adding models
+
+Edit `ggufit/models_db.py` and add an entry to the `MODELS` dict with `params_billion`,
+`layers`, and `hidden_size` (found in the model's Hugging Face `config.json`). Optional:
+- `moe: True` + `active_params_billion: X` for Mixture-of-Experts models
+- `kv_cache_multiplier: X` for non-standard attention architectures (see above)
+
+## Known limitations
+
+- Speed estimates assume batch size 1, single-user chat.
+- The bandwidth benchmark is single-threaded; real inference engines use multiple threads.
+- Quant byte-per-param values are approximations of real GGUF file sizes.
+- KV cache still assumes full hidden_size for standard (non-flagged) models even though most
+  modern ones use GQA with fewer KV heads than query heads — this is intentionally
+  conservative (over-estimates KV, doesn't affect the dominant "does it load" answer).
+- `kv_cache_multiplier` values for MLA/hybrid architectures are approximate, not
+  per-model-measured.
+
+## Notes
+
+This is a side-project CLI. The full hardware-scan desktop app (Rust/Tauri) is a separate,
+more thorough tool still in development.

ggufit-0.1.0/README.md

@@ -0,0 +1,84 @@
+# ggufit
+
+Check whether a local LLM can run on your machine, CPU-only — from PowerShell or any terminal.
+
+Uses real formulas (model memory footprint, KV cache size, and a measured memory-bandwidth
+micro-benchmark) rather than guesses, to estimate whether a model fits in RAM and roughly
+how fast it'll generate tokens. Correctly handles MoE models (speed driven by active
+experts, not total params) and SSM/MLA architectures (Mamba, DeepSeek's MLA) that don't
+use standard multi-head attention.
+
+## Install
+
+```bash
+# Linux/macOS - recommended (avoids PEP 668 "externally managed environment" errors)
+sudo apt install pipx      # if not already installed
+pipx ensurepath            # reopen terminal after this
+cd ggufit/                 # the folder containing pyproject.toml
+pipx install .
+
+# Windows PowerShell
+pip install .
+
+# Active development (editable install - code changes apply without reinstalling)
+pipx install -e .
+```
+
+`ggufit` is now available as a global command.
+
+## Usage
+
+```bash
+# Full hardware scan: shows your CPU/RAM/bandwidth and which models fit
+ggufit scan
+
+# Check one specific model
+ggufit llama3.1
+ggufit mistral
+ggufit qwen2.5-14b
+
+# Force a specific quantization level
+ggufit llama3.1 --quant q4
+
+# Evaluate at a longer context length
+ggufit qwen2.5-32b --context 16384
+```
+
+## How it works
+
+For each model:
+1. **Model size (RAM)** = total params × bytes-per-parameter (varies by quant: FP16, Q8, Q6, Q5, Q4, Q3, Q2)
+2. **KV cache** = `2 × layers × hidden_size × context_len × bytes_per_param × kv_cache_multiplier`
+   - `kv_cache_multiplier` defaults to 1.0 (standard MHA), and is set lower for architectures
+     that don't use full attention at every layer: `0.0` for pure SSM/Mamba (no attention at all),
+     `~0.15` for MLA (DeepSeek-V2/V3/R1, MiniCPM3), `~0.125` for hybrid Mamba+attention (Jamba).
+3. **Total RAM needed** = `(model size + KV cache) × 1.2` overhead factor
+4. **Speed estimate** = `measured_memory_bandwidth / active_size`, where `active_size` is the
+   total model size for dense models, or just the active experts' size for MoE models
+   (`active_params_billion`) — since only those weights are streamed from RAM per token.
+
+`ggufit` runs a quick real memory-bandwidth benchmark (a large array copy) instead of guessing
+from RAM specs, since achievable bandwidth depends heavily on channel configuration.
+
+## Adding models
+
+Edit `ggufit/models_db.py` and add an entry to the `MODELS` dict with `params_billion`,
+`layers`, and `hidden_size` (found in the model's Hugging Face `config.json`). Optional:
+- `moe: True` + `active_params_billion: X` for Mixture-of-Experts models
+- `kv_cache_multiplier: X` for non-standard attention architectures (see above)
+
+## Known limitations
+
+- Speed estimates assume batch size 1, single-user chat.
+- The bandwidth benchmark is single-threaded; real inference engines use multiple threads.
+- Quant byte-per-param values are approximations of real GGUF file sizes.
+- KV cache still assumes full hidden_size for standard (non-flagged) models even though most
+  modern ones use GQA with fewer KV heads than query heads — this is intentionally
+  conservative (over-estimates KV, doesn't affect the dominant "does it load" answer).
+- `kv_cache_multiplier` values for MLA/hybrid architectures are approximate, not
+  per-model-measured.
+
+## Notes
+
+This is a side-project CLI. The full hardware-scan desktop app (Rust/Tauri) is a separate,
+more thorough tool still in development.

ggufit-0.1.0/ggufit/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.1.0"

ggufit-0.1.0/ggufit/calculator.py

@@ -0,0 +1,105 @@
+"""
+Core math: given a model's architecture specs, a quant level, a context length,
+and this machine's hardware profile, estimate whether it fits and how fast it'll run.
+
+Fixed in this version:
+  - MoE models now use active_params_billion (not total params_billion) for the
+    tokens/sec estimate. Total params still governs the RAM/fit check, since all
+    expert weights must be resident in memory even if only some are touched per
+    token. Previously this made every MoE entry (Mixtral, DeepSeek-V3, Qwen3-MoE,
+    etc.) look far slower than it actually is - up to ~40x for the biggest ones.
+  - KV cache now respects an optional per-model kv_cache_multiplier, so SSM/Mamba
+    (0.0, no attention at all) and MLA architectures (~0.15, heavily compressed KV)
+    aren't computed as if they used standard multi-head attention.
+"""
+from dataclasses import dataclass
+
+from .models_db import QUANT_BYTES_PER_PARAM, QUANT_FALLBACK_ORDER
+
+OVERHEAD_FACTOR = 1.2  # framework / buffer overhead on top of raw weights + KV cache
+
+
+@dataclass
+class FitResult:
+    quant: str
+    model_size_gb: float          # total weight size (what must fit in RAM)
+    active_size_gb: float         # size of weights touched per token (drives speed)
+    kv_cache_gb: float
+    total_ram_needed_gb: float
+    fits: bool
+    est_tokens_per_sec: float
+
+
+def model_size_gb(params_billion: float, quant: str) -> float:
+    bytes_per_param = QUANT_BYTES_PER_PARAM[quant]
+    total_bytes = params_billion * 1e9 * bytes_per_param
+    return total_bytes / (1024**3)
+
+
+def kv_cache_gb(layers: int, hidden_size: int, seq_len: int, batch: int = 1,
+                bytes_per_param: float = 2.0, kv_cache_multiplier: float = 1.0) -> float:
+    # 2x for K and V, one value per layer/hidden-dim/token/batch.
+    # kv_cache_multiplier corrects for architectures that don't use full MHA
+    # at every layer (SSM/Mamba = 0.0, MLA-compressed = ~0.15, hybrid = ~0.125).
+    total_bytes = 2 * layers * hidden_size * seq_len * batch * bytes_per_param * kv_cache_multiplier
+    return total_bytes / (1024**3)
+
+
+def estimate_tokens_per_sec(active_size_gb: float, bandwidth_gbps: float) -> float:
+    if active_size_gb <= 0:
+        return 0.0
+    return round(bandwidth_gbps / active_size_gb, 2)
+
+
+def evaluate_fit(params_billion: float, layers: int, hidden_size: int,
+                  quant: str, context_len: int, available_ram_gb: float,
+                  bandwidth_gbps: float, moe: bool = False,
+                  active_params_billion: float = None,
+                  kv_cache_multiplier: float = 1.0) -> FitResult:
+    m_gb = model_size_gb(params_billion, quant)
+
+    # Speed is driven by whatever weights are actually touched per token.
+    # For MoE models that's active_params_billion; for dense models it's the same
+    # as the total (m_gb).
+    if moe and active_params_billion is not None:
+        active_gb = model_size_gb(active_params_billion, quant)
+    else:
+        active_gb = m_gb
+
+    kv_gb = kv_cache_gb(layers, hidden_size, context_len, kv_cache_multiplier=kv_cache_multiplier)
+
+    # RAM/fit check always uses the TOTAL model size - every expert must be
+    # resident in memory even for MoE, regardless of how many are active per token.
+    total_needed = (m_gb + kv_gb) * OVERHEAD_FACTOR
+    fits = total_needed <= available_ram_gb
+
+    tok_s = estimate_tokens_per_sec(active_gb, bandwidth_gbps)
+
+    return FitResult(
+        quant=quant,
+        model_size_gb=round(m_gb, 2),
+        active_size_gb=round(active_gb, 2),
+        kv_cache_gb=round(kv_gb, 2),
+        total_ram_needed_gb=round(total_needed, 2),
+        fits=fits,
+        est_tokens_per_sec=tok_s,
+    )
+
+
+def best_fitting_quant(params_billion: float, layers: int, hidden_size: int,
+                        context_len: int, available_ram_gb: float,
+                        bandwidth_gbps: float, moe: bool = False,
+                        active_params_billion: float = None,
+                        kv_cache_multiplier: float = 1.0):
+    """Try quant levels from highest quality to lowest, return the best one that fits, or None."""
+    results = {}
+    best = None
+    for quant in QUANT_FALLBACK_ORDER:
+        result = evaluate_fit(params_billion, layers, hidden_size, quant,
+                               context_len, available_ram_gb, bandwidth_gbps,
+                               moe=moe, active_params_billion=active_params_billion,
+                               kv_cache_multiplier=kv_cache_multiplier)
+        results[quant] = result
+        if result.fits and best is None:
+            best = result
+    return best, results

ggufit-0.1.0/ggufit/cli.py

@@ -0,0 +1,104 @@
+import argparse
+import sys
+
+from . import hardware, calculator, formatting
+from .models_db import MODELS, find_model, suggest_models
+
+
+def _model_kwargs(info):
+    """Extract the optional architecture-aware fields, with safe defaults."""
+    return {
+        "moe": info.get("moe", False),
+        "active_params_billion": info.get("active_params_billion"),
+        "kv_cache_multiplier": info.get("kv_cache_multiplier", 1.0),
+    }
+
+
+def cmd_scan(args):
+    print("Scanning hardware (running a quick memory-bandwidth benchmark)...")
+    profile = hardware.get_full_profile()
+    formatting.print_hardware_profile(profile)
+
+    available_ram = profile["ram"]["available_gb"]
+    bandwidth = profile["measured_bandwidth_gbps"]
+    context_len = args.context
+
+    recommended, marginal, not_recommended = [], [], []
+
+    for key, info in MODELS.items():
+        best, _all = calculator.best_fitting_quant(
+            info["params_billion"], info["layers"], info["hidden_size"],
+            context_len, available_ram, bandwidth, **_model_kwargs(info)
+        )
+        if best is None:
+            not_recommended.append(info["display_name"])
+        elif best.quant in ("q4", "q3", "q2"):
+            marginal.append((info["display_name"], best.quant, best.est_tokens_per_sec))
+        else:
+            recommended.append((info["display_name"], best.quant, best.est_tokens_per_sec))
+
+    formatting.print_scan_summary(recommended, marginal, not_recommended)
+    print()
+    print(formatting.info("Run 'ggufit <model-name>' for a detailed breakdown of a specific model."))
+
+
+def cmd_model(args):
+    key, info = find_model(args.model)
+    if info is None:
+        print(formatting.bad(f"Model '{args.model}' not found in the database."))
+        suggestions = suggest_models(args.model)
+        if suggestions:
+            print("Did you mean:")
+            for s in suggestions:
+                print(f"  - {s}  ({MODELS[s]['display_name']})")
+        else:
+            print("Try 'ggufit scan' to see all known models.")
+        sys.exit(1)
+
+    print("Checking hardware (quick benchmark)...")
+    profile = hardware.get_full_profile()
+    available_ram = profile["ram"]["available_gb"]
+    bandwidth = profile["measured_bandwidth_gbps"]
+
+    formatting.print_hardware_profile(profile)
+
+    kwargs = _model_kwargs(info)
+
+    if args.quant:
+        result = calculator.evaluate_fit(
+            info["params_billion"], info["layers"], info["hidden_size"],
+            args.quant, args.context, available_ram, bandwidth, **kwargs
+        )
+        best = result if result.fits else None
+        all_results = {args.quant: result}
+    else:
+        best, all_results = calculator.best_fitting_quant(
+            info["params_billion"], info["layers"], info["hidden_size"],
+            args.context, available_ram, bandwidth, **kwargs
+        )
+
+    formatting.print_model_result(info["display_name"], args.context, best, all_results,
+                                   available_ram, is_moe=info.get("moe", False))
+
+
+def main():
+    parser = argparse.ArgumentParser(prog="ggufit", description="Check whether local LLMs can run on this machine (CPU-only inference).")
+    parser.add_argument("target", nargs="?", help="'scan' for a full hardware scan, or a model name e.g. llama3.1, mistral, qwen2.5-14b")
+    parser.add_argument("--quant", choices=["fp32", "fp16", "q8", "q6", "q5", "q4", "q3", "q2"], help="Force a specific quantization level (model mode only)")
+    parser.add_argument("--context", type=int, default=4096, help="Context length to evaluate (default 4096)")
+
+    args = parser.parse_args()
+
+    if args.target is None:
+        parser.print_help()
+        return
+
+    if args.target.lower() == "scan":
+        cmd_scan(args)
+    else:
+        args.model = args.target
+        cmd_model(args)
+
+
+if __name__ == "__main__":
+    main()

ggufit-0.1.0/ggufit/formatting.py

@@ -0,0 +1,88 @@
+"""
+Simple ANSI-colored terminal output. Works in Windows Terminal / PowerShell 7+.
+On very old cmd.exe without ANSI support, colors just render as raw escape
+codes doing nothing harmful - text still reads fine.
+"""
+
+GREEN = "\033[92m"
+RED = "\033[91m"
+YELLOW = "\033[93m"
+CYAN = "\033[96m"
+BOLD = "\033[1m"
+RESET = "\033[0m"
+
+
+def ok(text): return f"{GREEN}{text}{RESET}"
+def bad(text): return f"{RED}{text}{RESET}"
+def warn(text): return f"{YELLOW}{text}{RESET}"
+def info(text): return f"{CYAN}{text}{RESET}"
+def bold(text): return f"{BOLD}{text}{RESET}"
+
+
+def print_header(title):
+    print()
+    print(bold(f"=== {title} ==="))
+
+
+def print_hardware_profile(profile):
+    cpu = profile["cpu"]
+    ram = profile["ram"]
+    print_header("Hardware Profile")
+    print(f"  OS:              {cpu['os']}")
+    print(f"  CPU:             {cpu['processor']}")
+    print(f"  Cores:           {cpu['physical_cores']} physical / {cpu['logical_cores']} logical")
+    if cpu["max_freq_ghz"]:
+        print(f"  Max clock:       {cpu['max_freq_ghz']} GHz")
+    print(f"  RAM total:       {ram['total_gb']} GB")
+    print(f"  RAM available:   {ram['available_gb']} GB  ({ram['used_percent']}% in use)")
+    print(f"  Measured bandwidth: {profile['measured_bandwidth_gbps']} GB/s (single-thread, approximate)")
+
+
+def print_model_result(display_name, context_len, best_result, all_results, available_ram_gb, is_moe=False):
+    print_header(f"{display_name} @ {context_len} ctx")
+    if best_result is None:
+        if len(all_results) == 1:
+            # a specific quant was forced via --quant and it doesn't fit
+            forced = list(all_results.values())[0]
+            print(bad(f"  ✗ Does not fit at {forced.quant.upper()} on this machine."))
+            print(f"    Needs ~{forced.total_ram_needed_gb} GB, you have {available_ram_gb} GB available.")
+            print(warn("  Suggestion: try a lower quant level (e.g. --quant q4) or drop --quant to auto-pick the best fit."))
+        else:
+            print(bad("  ✗ Does not fit at any quant level on this machine."))
+            smallest = all_results[list(all_results.keys())[-1]]
+            print(f"    Even the smallest quant ({smallest.quant.upper()}) needs ~{smallest.total_ram_needed_gb} GB, "
+                  f"you have {available_ram_gb} GB available.")
+            print(warn("  Suggestion: this model likely needs more RAM, or run a smaller model."))
+        return
+
+    print(ok(f"  ✓ Runnable at {best_result.quant.upper()} quantization"))
+    print(f"    Model size (RAM): {best_result.model_size_gb} GB")
+    if is_moe:
+        print(f"    Active per token:  {best_result.active_size_gb} GB  (MoE - only active experts drive speed)")
+    print(f"    KV cache:          {best_result.kv_cache_gb} GB")
+    print(f"    Total RAM needed:  {best_result.total_ram_needed_gb} GB  (you have {available_ram_gb} GB)")
+    print(f"    Estimated speed:   ~{best_result.est_tokens_per_sec} tokens/sec  (memory-bandwidth bound)")
+
+    # Show whether a higher-quality quant is close to fitting too
+    quant_order = list(all_results.keys())
+    idx = quant_order.index(best_result.quant)
+    if idx > 0:
+        next_up = all_results[quant_order[idx - 1]]
+        print(info(f"    Note: {next_up.quant.upper()} needs {next_up.total_ram_needed_gb} GB "
+                    f"(short by {round(next_up.total_ram_needed_gb - available_ram_gb, 2)} GB) - close but no fit."))
+
+
+def print_scan_summary(recommended, marginal, not_recommended):
+    print_header("Model Recommendations For This Machine")
+    if recommended:
+        print(ok("  Comfortably runnable (Q5/Q6/Q8+):"))
+        for name, quant, speed in recommended:
+            print(f"    - {name}  [{quant.upper()}]  ~{speed} tok/s")
+    if marginal:
+        print(warn("\n  Runnable but tight (Q4 or lower):"))
+        for name, quant, speed in marginal:
+            print(f"    - {name}  [{quant.upper()}]  ~{speed} tok/s")
+    if not_recommended:
+        print(bad("\n  Not recommended (won't fit even at Q2):"))
+        for name in not_recommended:
+            print(f"    - {name}")

ggufit-0.1.0/ggufit/hardware.py

@@ -0,0 +1,74 @@
+"""
+Hardware detection: CPU info, RAM, and a quick real memory-bandwidth benchmark.
+We benchmark bandwidth instead of guessing from RAM generation, since actual
+achievable bandwidth depends heavily on channel config, which is hard to detect
+reliably cross-platform.
+"""
+import platform
+import time
+
+import psutil
+
+try:
+    import numpy as np
+    _HAS_NUMPY = True
+except ImportError:
+    _HAS_NUMPY = False
+
+
+def get_cpu_info():
+    freq = psutil.cpu_freq()
+    return {
+        "physical_cores": psutil.cpu_count(logical=False) or psutil.cpu_count(logical=True),
+        "logical_cores": psutil.cpu_count(logical=True),
+        "max_freq_ghz": round(freq.max / 1000, 2) if freq and freq.max else None,
+        "current_freq_ghz": round(freq.current / 1000, 2) if freq and freq.current else None,
+        "architecture": platform.machine(),
+        "processor": platform.processor() or platform.uname().processor or "Unknown",
+        "os": f"{platform.system()} {platform.release()}",
+    }
+
+
+def get_ram_info():
+    vm = psutil.virtual_memory()
+    return {
+        "total_gb": round(vm.total / (1024**3), 2),
+        "available_gb": round(vm.available / (1024**3), 2),
+        "used_percent": vm.percent,
+    }
+
+
+def benchmark_memory_bandwidth(size_mb: int = 256, repeats: int = 5) -> float:
+    """
+    Rough single-threaded memory bandwidth estimate in GB/s, via a large array copy.
+    Not a substitute for a proper tool like Intel MLC, but good enough to
+    ballpark CPU-inference speed since that's also largely single/few-thread
+    memory-bound work in most llama.cpp style runtimes.
+    Returns GB/s. Falls back to a conservative default if numpy isn't available.
+    """
+    if not _HAS_NUMPY:
+        return 15.0  # conservative fallback estimate (typical dual-channel DDR4)
+
+    n_elements = (size_mb * 1024 * 1024) // 8  # float64 = 8 bytes
+    src = np.random.rand(n_elements)
+    best = 0.0
+    for _ in range(repeats):
+        start = time.perf_counter()
+        dst = src.copy()
+        elapsed = time.perf_counter() - start
+        bytes_moved = n_elements * 8 * 2  # read + write
+        gbps = (bytes_moved / elapsed) / (1024**3)
+        best = max(best, gbps)
+        del dst
+    return round(best, 2)
+
+
+def get_full_profile():
+    cpu = get_cpu_info()
+    ram = get_ram_info()
+    bandwidth = benchmark_memory_bandwidth()
+    return {
+        "cpu": cpu,
+        "ram": ram,
+        "measured_bandwidth_gbps": bandwidth,
+    }