modelinfo-cli 1.2.0__tar.gz → 1.4.0__tar.gz

modelinfo_cli-1.4.0/PKG-INFO

@@ -0,0 +1,184 @@
+Metadata-Version: 2.4
+Name: modelinfo-cli
+Version: 1.4.0
+Summary: A sub-100ms, zero-dependency CLI to inspect ML models (.safetensors, .gguf) locally or via Hugging Face, calculate exact VRAM footprints, and determine hardware fit.
+Author: ModelInfo Contributors
+License: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: rich>=13.0.0
+Provides-Extra: dev
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Dynamic: license-file
+
+# ModelInfo CLI
+
+![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)
+![Dependencies](https://img.shields.io/badge/dependencies-rich-green.svg)
+![License](https://img.shields.io/badge/license-MIT-blue.svg)
+
+![ModelInfo Demo](modelinfo.gif)
+
+ModelInfo is a CLI tool that inspects machine learning model checkpoints (`.safetensors`, `.gguf`, `.pt`) and calculates hardware requirements completely offline.
+
+It reads binary headers directly using the Python standard library. By bypassing full tensor payload loading and strictly excluding heavy ecosystems like PyTorch or HuggingFace, the tool executes in under 100 milliseconds.
+
+## Features
+
+- **Zero-Dependency Parsing**: Reads `.safetensors` 8-byte JSON prefixes and `.gguf` binary key-value metadata directly via `struct` and `json` (falling back to `config.json` if needed).
+- **Remote Hugging Face Hub Inspection**: Pass a repo ID (e.g., `meta-llama/Llama-2-7b-hf`) and it uses concurrent byte-range requests to read the headers off the CDN in under 2 seconds. No need to download the checkpoint.
+- Parses `model.safetensors.index.json` to support sharded models without crashing on partial downloads.
+- **Dynamic VRAM & Subtractive vLLM Math**: Calculates exact VRAM limits based on the model's architecture and your target context length. If you use the `--vllm` flag, it switches to a subtractive "Serving Capacity" engine that calculates exactly how many tokens fit in the PagedAttention pool based on your `--gpu-util` ratio.
+- **Hardware Fit Diagnostics**: Check if a model fits your cluster with `--gpu` (e.g. `--gpu RTX4090` or `--gpu auto`). It enforces Apple Silicon's 75% unified memory wire limit, and you can explicitly model multi-GPU NCCL communication penalties with `--topology` and `--strategy`.
+- **Side-by-Side Comparison**: Pass multiple models to trigger a comparison table (parameters, data types, context lengths, VRAM footprints).
+- Uses exact `ggml_type` mappings for GGUF formats to calculate byte-scaling coefficients, preventing VRAM under-reporting.
+- **Secure Pickling**: Inspects legacy `.pt` files safely using a restricted `pickle.Unpickler`.
+- The UI (built with `rich`) groups repetitive layers and color-codes VRAM heatmaps.
+
+> [!NOTE]
+> **A Note on Performance & Remote Fetching**
+> Local `.gguf` and `.safetensors` files are parsed in under 100ms. However, querying remote Hugging Face repositories takes **1 to 10 seconds**. To remain zero-dependency, `modelinfo` opens connections via Python `urllib` instead of loading PyTorch. For massive sharded models (e.g., 100+ shards), it must fetch every header individually, capped at an 8-worker thread pool to prevent Cloudflare IP bans. Waiting ~8 seconds to map a model is faster than downloading 400GB just to see if it fits your hardware.
+
+## Installation
+
+Install directly from PyPI:
+
+```bash
+pip install modelinfo-cli
+```
+
+### Development
+
+To install from source and run the test suite:
+
+```bash
+git clone https://github.com/pipe1os/modelinfo-cli.git
+cd modelinfo-cli
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+```
+
+## Testing
+
+The testing suite enforces cross-platform structural integrity and guards the zero-dependency latency constraint. Tests are isolated against custom binary mocks in `tests/fixtures/`.
+
+Run the test suite using pytest:
+
+```bash
+pytest tests/ -v
+```
+
+## Usage
+
+Inspect a local model checkpoint:
+
+```bash
+modelinfo mistral-7b.safetensors
+```
+
+Inspect a remote model directly from the Hugging Face Hub:
+
+```bash
+modelinfo meta-llama/Llama-2-7b-hf
+```
+
+For gated models (e.g., Llama 2), you must provide authentication by setting the `HF_TOKEN` environment variable. You can create a token in your [Hugging Face settings](https://huggingface.co/settings/tokens).
+
+```bash
+export HF_TOKEN="hf_your_token_here"
+modelinfo meta-llama/Llama-2-7b-hf
+```
+
+Alternatively, the tool will automatically read tokens stored by the `hf auth login` command (located in `~/.cache/huggingface/token`).
+
+Calculate the memory footprint with a specific KV cache context window:
+
+```bash
+modelinfo mistral-7b.safetensors --context 8192
+```
+
+Adjust the VRAM heat-mapping thresholds for your specific hardware (e.g., an 80GB card):
+
+```bash
+modelinfo meta-llama/Llama-2-7b-hf --max-vram 80
+```
+
+Determine if a model fits your specific hardware:
+
+```bash
+modelinfo mistralai/Mistral-7B-v0.1 --gpu "RTX 4090"
+modelinfo mistralai/Mistral-7B-v0.1 --gpu auto
+```
+
+Compare multiple models side-by-side against a hardware target:
+
+```bash
+modelinfo mistralai/Mistral-7B-v0.1 Qwen/Qwen2.5-0.5B --gpu 12
+```
+
+Simulate exactly how many tokens you can serve using vLLM on a specific multi-GPU topology:
+
+```bash
+modelinfo mistralai/Mistral-7B-v0.1 --vllm --gpu 4xRTX4090 --topology pcie4 --strategy tp
+```
+
+### Example Output (Single Model)
+
+```text
+Format:          SafeTensors
+Architecture:    MistralForCausalLM (32 layers)
+Tensors:         291
+Parameters:      7.2B
+Dtype:           BF16
+Disk size:       13.49 GB
+VRAM (est):      ~15.07 GB Total Minimum Required
+                   ├─ Weights:    13.49 GB
+                   ├─ KV Cache:   1.0 GB (Default 8192 tokens. Native limit: 32,768)
+                   └─ Overhead:   600.0 MB (CUDA Context + Activations)
+Hardware Fit:    ✗ No (Requires 15.07 GB, Hardware has 12.0 GB)
+
+Top Tensors by Size:
+  model.embed_tokens.weight                     [32000 x 4096]   bf16   131.1M params
+  32x model.layers.[N].self_attn.q_proj.weight  [4096 x 4096]    bf16    16.8M params
+```
+
+### Example Output (Comparison)
+
+```text
+Model              Params    Dtype    Context    VRAM        Fits
+Mistral-7B-v0.1    7.2B      BF16     8K         15.07 GB    ✗
+Qwen2.5-0.5B       494.0M    BF16     8K         1.6 GB      ✓
+```
+
+## Command Reference
+
+| Argument | Example | Description |
+| :--- | :--- | :--- |
+| `[files...]` | `modelinfo model.safetensors` | Inspect a single model (local path or Hugging Face repo ID). |
+| `[files...]` | `modelinfo modelA modelB` | Pass multiple files/repos to automatically render a side-by-side comparison table instead of a deep-dive summary. |
+| `--gpu` | `--gpu rtx4090` | Check if the model fits. Accepts GPU names (`rtx4090`, `b200`, `rx7900xtx`), explicit VRAM limits in GB (`--gpu 24`), or local hardware auto-discovery (`--gpu auto`). |
+| `--context` | `--context 32768` | Adjust the target KV cache length. Essential for calculating the dynamic memory footprint of long-context models. Defaults to `8192`. |
+| `--max-vram` | `--max-vram 80` | Adjusts the color-coded heat mapping thresholds (Green/Yellow/Red) in the terminal output to match a specific hardware ceiling. |
+| `--vllm` | `--vllm --gpu auto` | Switches from additive memory checking to a subtractive serving capacity estimation. Shows exactly how many tokens fit in the PagedAttention pool. |
+| `--gpu-util` | `--gpu-util 0.9` | Sets the vLLM `gpu_memory_utilization` ratio. Defaults to `0.9` (reserves 10% for PyTorch context). |
+| `--topology` | `--topology nvlink` | Set interconnect topology to calculate exact communication overhead penalties (`nvlink`, `pcie4`, `pcie3`). Defaults to `pcie4`. |
+| `--strategy` | `--strategy tp` | Selects the parallelization strategy for multi-GPU setups (`tp` for Tensor Parallelism, `pp` for Pipeline Parallelism). Defaults to `tp`. |
+| `--tensors` | `--tensors` | Bypasses the algorithmic speed estimation and forces the tool to fetch all remote shards, displaying an exact size breakdown of every tensor. |
+
+## Architecture
+
+Three modules:
+
+1. **Presentation (`cli.py`, `ui.py`)**: Parses arguments and formats tables via `rich`.
+2. **Parsing Engine (`parsers/`)**: Specialized binary readers (`safetensors.py`, `gguf.py`, `pytorch.py`) that use only the standard library.
+3. **Math Engine (`calculator.py`)**: Determines total parameter counts, maps data types to byte coefficients, and calculates dynamic memory allocations based on tensor shape heuristics.
+
+## License
+
+This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.

modelinfo_cli-1.4.0/README.md

@@ -0,0 +1,166 @@
+# ModelInfo CLI
+
+![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)
+![Dependencies](https://img.shields.io/badge/dependencies-rich-green.svg)
+![License](https://img.shields.io/badge/license-MIT-blue.svg)
+
+![ModelInfo Demo](modelinfo.gif)
+
+ModelInfo is a CLI tool that inspects machine learning model checkpoints (`.safetensors`, `.gguf`, `.pt`) and calculates hardware requirements completely offline.
+
+It reads binary headers directly using the Python standard library. By bypassing full tensor payload loading and strictly excluding heavy ecosystems like PyTorch or HuggingFace, the tool executes in under 100 milliseconds.
+
+## Features
+
+- **Zero-Dependency Parsing**: Reads `.safetensors` 8-byte JSON prefixes and `.gguf` binary key-value metadata directly via `struct` and `json` (falling back to `config.json` if needed).
+- **Remote Hugging Face Hub Inspection**: Pass a repo ID (e.g., `meta-llama/Llama-2-7b-hf`) and it uses concurrent byte-range requests to read the headers off the CDN in under 2 seconds. No need to download the checkpoint.
+- Parses `model.safetensors.index.json` to support sharded models without crashing on partial downloads.
+- **Dynamic VRAM & Subtractive vLLM Math**: Calculates exact VRAM limits based on the model's architecture and your target context length. If you use the `--vllm` flag, it switches to a subtractive "Serving Capacity" engine that calculates exactly how many tokens fit in the PagedAttention pool based on your `--gpu-util` ratio.
+- **Hardware Fit Diagnostics**: Check if a model fits your cluster with `--gpu` (e.g. `--gpu RTX4090` or `--gpu auto`). It enforces Apple Silicon's 75% unified memory wire limit, and you can explicitly model multi-GPU NCCL communication penalties with `--topology` and `--strategy`.
+- **Side-by-Side Comparison**: Pass multiple models to trigger a comparison table (parameters, data types, context lengths, VRAM footprints).
+- Uses exact `ggml_type` mappings for GGUF formats to calculate byte-scaling coefficients, preventing VRAM under-reporting.
+- **Secure Pickling**: Inspects legacy `.pt` files safely using a restricted `pickle.Unpickler`.
+- The UI (built with `rich`) groups repetitive layers and color-codes VRAM heatmaps.
+
+> [!NOTE]
+> **A Note on Performance & Remote Fetching**
+> Local `.gguf` and `.safetensors` files are parsed in under 100ms. However, querying remote Hugging Face repositories takes **1 to 10 seconds**. To remain zero-dependency, `modelinfo` opens connections via Python `urllib` instead of loading PyTorch. For massive sharded models (e.g., 100+ shards), it must fetch every header individually, capped at an 8-worker thread pool to prevent Cloudflare IP bans. Waiting ~8 seconds to map a model is faster than downloading 400GB just to see if it fits your hardware.
+
+## Installation
+
+Install directly from PyPI:
+
+```bash
+pip install modelinfo-cli
+```
+
+### Development
+
+To install from source and run the test suite:
+
+```bash
+git clone https://github.com/pipe1os/modelinfo-cli.git
+cd modelinfo-cli
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+```
+
+## Testing
+
+The testing suite enforces cross-platform structural integrity and guards the zero-dependency latency constraint. Tests are isolated against custom binary mocks in `tests/fixtures/`.
+
+Run the test suite using pytest:
+
+```bash
+pytest tests/ -v
+```
+
+## Usage
+
+Inspect a local model checkpoint:
+
+```bash
+modelinfo mistral-7b.safetensors
+```
+
+Inspect a remote model directly from the Hugging Face Hub:
+
+```bash
+modelinfo meta-llama/Llama-2-7b-hf
+```
+
+For gated models (e.g., Llama 2), you must provide authentication by setting the `HF_TOKEN` environment variable. You can create a token in your [Hugging Face settings](https://huggingface.co/settings/tokens).
+
+```bash
+export HF_TOKEN="hf_your_token_here"
+modelinfo meta-llama/Llama-2-7b-hf
+```
+
+Alternatively, the tool will automatically read tokens stored by the `hf auth login` command (located in `~/.cache/huggingface/token`).
+
+Calculate the memory footprint with a specific KV cache context window:
+
+```bash
+modelinfo mistral-7b.safetensors --context 8192
+```
+
+Adjust the VRAM heat-mapping thresholds for your specific hardware (e.g., an 80GB card):
+
+```bash
+modelinfo meta-llama/Llama-2-7b-hf --max-vram 80
+```
+
+Determine if a model fits your specific hardware:
+
+```bash
+modelinfo mistralai/Mistral-7B-v0.1 --gpu "RTX 4090"
+modelinfo mistralai/Mistral-7B-v0.1 --gpu auto
+```
+
+Compare multiple models side-by-side against a hardware target:
+
+```bash
+modelinfo mistralai/Mistral-7B-v0.1 Qwen/Qwen2.5-0.5B --gpu 12
+```
+
+Simulate exactly how many tokens you can serve using vLLM on a specific multi-GPU topology:
+
+```bash
+modelinfo mistralai/Mistral-7B-v0.1 --vllm --gpu 4xRTX4090 --topology pcie4 --strategy tp
+```
+
+### Example Output (Single Model)
+
+```text
+Format:          SafeTensors
+Architecture:    MistralForCausalLM (32 layers)
+Tensors:         291
+Parameters:      7.2B
+Dtype:           BF16
+Disk size:       13.49 GB
+VRAM (est):      ~15.07 GB Total Minimum Required
+                   ├─ Weights:    13.49 GB
+                   ├─ KV Cache:   1.0 GB (Default 8192 tokens. Native limit: 32,768)
+                   └─ Overhead:   600.0 MB (CUDA Context + Activations)
+Hardware Fit:    ✗ No (Requires 15.07 GB, Hardware has 12.0 GB)
+
+Top Tensors by Size:
+  model.embed_tokens.weight                     [32000 x 4096]   bf16   131.1M params
+  32x model.layers.[N].self_attn.q_proj.weight  [4096 x 4096]    bf16    16.8M params
+```
+
+### Example Output (Comparison)
+
+```text
+Model              Params    Dtype    Context    VRAM        Fits
+Mistral-7B-v0.1    7.2B      BF16     8K         15.07 GB    ✗
+Qwen2.5-0.5B       494.0M    BF16     8K         1.6 GB      ✓
+```
+
+## Command Reference
+
+| Argument | Example | Description |
+| :--- | :--- | :--- |
+| `[files...]` | `modelinfo model.safetensors` | Inspect a single model (local path or Hugging Face repo ID). |
+| `[files...]` | `modelinfo modelA modelB` | Pass multiple files/repos to automatically render a side-by-side comparison table instead of a deep-dive summary. |
+| `--gpu` | `--gpu rtx4090` | Check if the model fits. Accepts GPU names (`rtx4090`, `b200`, `rx7900xtx`), explicit VRAM limits in GB (`--gpu 24`), or local hardware auto-discovery (`--gpu auto`). |
+| `--context` | `--context 32768` | Adjust the target KV cache length. Essential for calculating the dynamic memory footprint of long-context models. Defaults to `8192`. |
+| `--max-vram` | `--max-vram 80` | Adjusts the color-coded heat mapping thresholds (Green/Yellow/Red) in the terminal output to match a specific hardware ceiling. |
+| `--vllm` | `--vllm --gpu auto` | Switches from additive memory checking to a subtractive serving capacity estimation. Shows exactly how many tokens fit in the PagedAttention pool. |
+| `--gpu-util` | `--gpu-util 0.9` | Sets the vLLM `gpu_memory_utilization` ratio. Defaults to `0.9` (reserves 10% for PyTorch context). |
+| `--topology` | `--topology nvlink` | Set interconnect topology to calculate exact communication overhead penalties (`nvlink`, `pcie4`, `pcie3`). Defaults to `pcie4`. |
+| `--strategy` | `--strategy tp` | Selects the parallelization strategy for multi-GPU setups (`tp` for Tensor Parallelism, `pp` for Pipeline Parallelism). Defaults to `tp`. |
+| `--tensors` | `--tensors` | Bypasses the algorithmic speed estimation and forces the tool to fetch all remote shards, displaying an exact size breakdown of every tensor. |
+
+## Architecture
+
+Three modules:
+
+1. **Presentation (`cli.py`, `ui.py`)**: Parses arguments and formats tables via `rich`.
+2. **Parsing Engine (`parsers/`)**: Specialized binary readers (`safetensors.py`, `gguf.py`, `pytorch.py`) that use only the standard library.
+3. **Math Engine (`calculator.py`)**: Determines total parameter counts, maps data types to byte coefficients, and calculates dynamic memory allocations based on tensor shape heuristics.
+
+## License
+
+This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.

{modelinfo_cli-1.2.0 → modelinfo_cli-1.4.0}/pyproject.toml

@@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "modelinfo-cli"
-version = "1.2.0"
-description = "A sub-100ms, zero-dependency CLI tool to inspect ML model checkpoints and dynamically calculate VRAM requirements."
+version = "1.4.0"
+description = "A sub-100ms, zero-dependency CLI to inspect ML models (.safetensors, .gguf) locally or via Hugging Face, calculate exact VRAM footprints, and determine hardware fit."
 readme = "README.md"
 requires-python = ">=3.10"
 license = { text = "MIT" }
@@ -30,4 +30,4 @@ modelinfo = "modelinfo.cli:main"
 
 [tool.ruff]
 line-length = 88
-target-version = "py310"
+target-version = "py310"

{modelinfo_cli-1.2.0 → modelinfo_cli-1.4.0}/src/modelinfo/__init__.py

@@ -2,4 +2,4 @@
 modelinfo - A high-performance CLI utility for inspecting ML model checkpoints.
 """
 
-__version__ = "1.2.0"
+__version__ = "1.4.0"

modelinfo_cli-1.4.0/src/modelinfo/calculator.py

@@ -0,0 +1,178 @@
+import math
+from typing import Any, Dict
+
+from modelinfo.architecture import extract_architecture
+
+DTYPE_BYTES = {
+    "F64": 8.0,
+    "F32": 4.0,
+    "F16": 2.0,
+    "BF16": 2.0,
+    "F8": 1.0,
+    "F8_E5M2": 1.0,
+    "F8_E4M3": 1.0,
+    "I64": 8.0,
+    "I32": 4.0,
+    "I16": 2.0,
+    "I8": 1.0,
+    "U64": 8.0,
+    "U32": 4.0,
+    "Q8_0": 1.0625,
+    "Q8_1": 1.0625,
+    "Q8_K": 1.0625,
+    "Q6_K": 0.828125,
+    "Q5_0": 0.6875,
+    "Q5_1": 0.75,
+    "Q5_K": 0.6875,
+    "Q4_0": 0.5625,
+    "Q4_1": 0.625,
+    "Q4_K": 0.59375,
+    "Q3_K": 0.4375,
+    "Q2_K": 0.34375,
+    "IQ4_NL": 0.53125,
+    "IQ4_XS": 0.53125,
+    "IQ3_S": 0.4375,
+    "IQ3_XXS": 0.385,
+    "IQ2_S": 0.3125,
+    "IQ2_XS": 0.296875,
+    "IQ2_XXS": 0.28125,
+    "IQ1_M": 0.21875,
+    "IQ1_S": 0.1953125,
+    "Q8": 1.06,
+    "Q6": 0.82,
+    "Q5": 0.68,
+    "Q4": 0.58,
+    "Q3": 0.43,
+    "Q2": 0.28,
+}
+
+def _get_bytes_per_param(dtype: str) -> float:
+    """Return the size in bytes for a given data type."""
+    return DTYPE_BYTES.get(dtype.upper(), 2.0)
+
+def calculate_footprint(
+    tensors: Dict[str, Any], 
+    context_length: int = 0, 
+    batch_size: int = 1, 
+    config: Dict[str, Any] = None, 
+    gpu_count: int = 1,
+    topology: str = "pcie4",
+    strategy: str = "tp",
+    is_vllm: bool = False,
+    gpu_vram_bytes: float = 0.0,
+    gpu_util: float = 0.9
+) -> Dict[str, Any]:
+    """
+    Calculate the memory footprint of a model based on its tensors and context length.
+    """
+    total_params = 0
+    base_memory_bytes = 0.0
+    dtype_counts: Dict[str, int] = {}
+    
+    is_lazy = tensors.get("__metadata__", {}).get("lazy_fetch", False)
+    
+    if is_lazy:
+        base_memory_bytes = tensors.get("__metadata__", {}).get("total_size", 0.0)
+        # Assume predominantly FP16/BF16 for modern Hub architectures
+        primary_dtype = "BF16"
+        dtype_counts[primary_dtype] = 1
+        total_params = int(base_memory_bytes / 2.0)
+    else:
+        for name, metadata in tensors.items():
+            if name == "__metadata__":
+                continue
+                
+            shape = metadata.get("shape", [])
+            if not shape:
+                continue
+                
+            param_count = math.prod(shape)
+            total_params += param_count
+            
+            dtype = metadata.get("dtype", "F16").upper()
+            dtype_counts[dtype] = dtype_counts.get(dtype, 0) + 1
+            
+            bytes_per_param = _get_bytes_per_param(dtype)
+            base_memory_bytes += param_count * bytes_per_param
+        
+    num_layers, kv_dim, is_estimate = extract_architecture(tensors, config)
+    
+    # Formula: 2 * Layers * (KV_Heads * Head_Dim) * Context_Length * Batch_Size * Bytes_per_param
+    # Assume FP16 (2 bytes) for KV cache
+    kv_cache_bytes = 2 * num_layers * kv_dim * context_length * batch_size * 2
+    
+    primary_dtype = max(dtype_counts.items(), key=lambda x: x[1])[0] if dtype_counts else "Unknown"
+    # Topology & Strategy Penalties
+    penalty_percentage = 0.0
+    if gpu_count > 1:
+        if strategy == "pp":
+            penalty_percentage = 0.0
+        else: # strategy == "tp"
+            if topology == "nvlink":
+                penalty_percentage = 0.04
+            elif topology == "pcie3":
+                penalty_percentage = 0.20
+            else: # pcie4
+                penalty_percentage = 0.12
+                
+    distributed_overhead = base_memory_bytes * penalty_percentage if gpu_count > 1 else 0.0
+    
+    vllm_metrics = {}
+    if is_vllm and gpu_vram_bytes > 0:
+        usable_vram = gpu_vram_bytes * gpu_util
+        remaining_vram = usable_vram - (base_memory_bytes + distributed_overhead)
+        
+        bytes_per_token = 2 * num_layers * kv_dim * 2
+        
+        max_serving_capacity = 0
+        if remaining_vram > 0 and bytes_per_token > 0:
+            max_serving_capacity = math.floor(remaining_vram / bytes_per_token)
+            
+        overhead_bytes = distributed_overhead
+        total_memory_bytes = base_memory_bytes + overhead_bytes
+        
+        vllm_metrics = {
+            "usable_vram": usable_vram,
+            "static_weights": base_memory_bytes,
+            "distributed_penalty": distributed_overhead,
+            "paged_kv_pool": max(0.0, remaining_vram),
+            "max_serving_capacity": max_serving_capacity
+        }
+    else:
+        CUDA_CONTEXT_MB = 600 * gpu_count
+        overhead_bytes = (CUDA_CONTEXT_MB * 1024 * 1024) + distributed_overhead
+        total_memory_bytes = base_memory_bytes + kv_cache_bytes + overhead_bytes
+    
+    return {
+        "total_params": total_params,
+        "base_memory_bytes": base_memory_bytes,
+        "kv_cache_bytes": kv_cache_bytes,
+        "overhead_bytes": overhead_bytes,
+        "total_memory_bytes": total_memory_bytes,
+        "num_layers": num_layers,
+        "kv_dim": kv_dim,
+        "primary_dtype": primary_dtype,
+        "kv_is_estimate": is_estimate,
+        "penalty_percentage": penalty_percentage,
+        "vllm_metrics": vllm_metrics
+    }
+
+def format_bytes(size_bytes: float) -> str:
+    """Format bytes into a human-readable string (e.g. GB)."""
+    if size_bytes == 0:
+        return "0 B"
+    units = ["B", "KB", "MB", "GB", "TB", "PB"]
+    i = max(0, min(len(units) - 1, math.floor(math.log(size_bytes, 1024))))
+    p = math.pow(1024, i)
+    s = round(size_bytes / p, 2)
+    return f"{s} {units[i]}"
+
+def format_params(count: int) -> str:
+    """Format parameter count into a human-readable string (e.g. 7.2B)."""
+    if count >= 1_000_000_000:
+        return f"{count:,} ({count / 1_000_000_000:.1f}B)"
+    elif count >= 1_000_000:
+        return f"{count:,} ({count / 1_000_000:.1f}M)"
+    elif count >= 1_000:
+        return f"{count:,} ({count / 1_000:.1f}K)"
+    return f"{count:,}"