modelinfo-cli 1.3.0__tar.gz → 1.4.0__tar.gz

{modelinfo_cli-1.3.0 → modelinfo_cli-1.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: modelinfo-cli
-Version: 1.3.0
+Version: 1.4.0
 Summary: A sub-100ms, zero-dependency CLI to inspect ML models (.safetensors, .gguf) locally or via Hugging Face, calculate exact VRAM footprints, and determine hardware fit.
 Author: ModelInfo Contributors
 License: MIT
@@ -22,25 +22,27 @@ Dynamic: license-file
 ![Dependencies](https://img.shields.io/badge/dependencies-rich-green.svg)
 ![License](https://img.shields.io/badge/license-MIT-blue.svg)
 
-ModelInfo is a terminal-native utility that inspects machine learning model checkpoints (`.safetensors`, `.gguf`, `.pt`) and calculates hardware requirements completely offline.
+![ModelInfo Demo](modelinfo.gif)
+
+ModelInfo is a CLI tool that inspects machine learning model checkpoints (`.safetensors`, `.gguf`, `.pt`) and calculates hardware requirements completely offline.
 
 It reads binary headers directly using the Python standard library. By bypassing full tensor payload loading and strictly excluding heavy ecosystems like PyTorch or HuggingFace, the tool executes in under 100 milliseconds.
 
 ## Features
 
-- **Zero-Dependency Parsing**: Reads the 8-byte JSON prefix of `.safetensors` files and the binary key-value metadata of `.gguf` directly via `struct` and `json`. Reads adjacent `config.json` for architecture fallback.
-- **Remote Hugging Face Hub Inspection**: Inspect any public or gated model directly via its repo ID (e.g., `modelinfo meta-llama/Llama-2-7b-hf`) without downloading the checkpoint. Uses concurrent byte-range requests to read the binary headers directly off the CDN in under 2 seconds.
-- **Sharded Model Support**: Transparently parses `model.safetensors.index.json` to detect multi-file checkpoint distributions, gracefully guarding against partial downloads without crashing.
-- **Dynamic VRAM Estimation**: Extracts underlying model architecture to calculate exact VRAM limits, including dynamic KV cache footprints based on user-specified context lengths.
-- **Hardware Fit Diagnostics**: Pass the `--gpu` flag (e.g. `--gpu RTX4090` or `--gpu auto`) to calculate if the model fits in your specific cluster. Defends against fragmentation OOMs using a 3-tier heuristic (Safe, Warning, Fail), calculates overhead across multi-GPU setups, and enforces Apple Silicon's 75% unified memory wire limit.
-- **Side-by-Side Comparison**: Pass multiple models to automatically trigger a comparison table. Compares parameters, data types, context lengths, and VRAM footprints side-by-side to evaluate trade-offs.
-- **Precise Block Quantization**: Factors in exact byte-scaling coefficients for GGUF formats (e.g., Q8, Q6, Q4) rather than naive averages, eliminating VRAM under-reporting.
-- **Secure Pickling**: Inspects legacy `.pt` files without executing arbitrary code by using a highly restricted `pickle.Unpickler`.
-- **Terminal UI**: Groups repetitive structural layers and color-codes VRAM heatmaps using `rich`. Breaks down memory footprints into Weights, KV Cache, and Overhead.
+- **Zero-Dependency Parsing**: Reads `.safetensors` 8-byte JSON prefixes and `.gguf` binary key-value metadata directly via `struct` and `json` (falling back to `config.json` if needed).
+- **Remote Hugging Face Hub Inspection**: Pass a repo ID (e.g., `meta-llama/Llama-2-7b-hf`) and it uses concurrent byte-range requests to read the headers off the CDN in under 2 seconds. No need to download the checkpoint.
+- Parses `model.safetensors.index.json` to support sharded models without crashing on partial downloads.
+- **Dynamic VRAM & Subtractive vLLM Math**: Calculates exact VRAM limits based on the model's architecture and your target context length. If you use the `--vllm` flag, it switches to a subtractive "Serving Capacity" engine that calculates exactly how many tokens fit in the PagedAttention pool based on your `--gpu-util` ratio.
+- **Hardware Fit Diagnostics**: Check if a model fits your cluster with `--gpu` (e.g. `--gpu RTX4090` or `--gpu auto`). It enforces Apple Silicon's 75% unified memory wire limit, and you can explicitly model multi-GPU NCCL communication penalties with `--topology` and `--strategy`.
+- **Side-by-Side Comparison**: Pass multiple models to trigger a comparison table (parameters, data types, context lengths, VRAM footprints).
+- Uses exact `ggml_type` mappings for GGUF formats to calculate byte-scaling coefficients, preventing VRAM under-reporting.
+- **Secure Pickling**: Inspects legacy `.pt` files safely using a restricted `pickle.Unpickler`.
+- The UI (built with `rich`) groups repetitive layers and color-codes VRAM heatmaps.
 
 > [!NOTE]
 > **A Note on Performance & Remote Fetching**
-> Local `.gguf` and `.safetensors` files are parsed in under 100ms. However, querying remote Hugging Face repositories takes **1 to 10 seconds**. This is an intentional trade-off. To remain zero-dependency, `modelinfo` negotiates raw TCP/TLS via Python `urllib` instead of loading PyTorch. For massive sharded models (e.g., 100+ shards), it must fetch every header individually, capped at an 8-worker thread pool to prevent Cloudflare IP bans. Waiting ~8 seconds to map a model is faster than downloading 400GB just to see if it fits your hardware.
+> Local `.gguf` and `.safetensors` files are parsed in under 100ms. However, querying remote Hugging Face repositories takes **1 to 10 seconds**. To remain zero-dependency, `modelinfo` opens connections via Python `urllib` instead of loading PyTorch. For massive sharded models (e.g., 100+ shards), it must fetch every header individually, capped at an 8-worker thread pool to prevent Cloudflare IP bans. Waiting ~8 seconds to map a model is faster than downloading 400GB just to see if it fits your hardware.
 
 ## Installation
 
@@ -120,6 +122,12 @@ Compare multiple models side-by-side against a hardware target:
 modelinfo mistralai/Mistral-7B-v0.1 Qwen/Qwen2.5-0.5B --gpu 12
 ```
 
+Simulate exactly how many tokens you can serve using vLLM on a specific multi-GPU topology:
+
+```bash
+modelinfo mistralai/Mistral-7B-v0.1 --vllm --gpu 4xRTX4090 --topology pcie4 --strategy tp
+```
+
 ### Example Output (Single Model)
 
 ```text
@@ -157,13 +165,18 @@ Qwen2.5-0.5B       494.0M    BF16     8K         1.6 GB      ✓
 | `--gpu` | `--gpu rtx4090` | Check if the model fits. Accepts GPU names (`rtx4090`, `b200`, `rx7900xtx`), explicit VRAM limits in GB (`--gpu 24`), or local hardware auto-discovery (`--gpu auto`). |
 | `--context` | `--context 32768` | Adjust the target KV cache length. Essential for calculating the dynamic memory footprint of long-context models. Defaults to `8192`. |
 | `--max-vram` | `--max-vram 80` | Adjusts the color-coded heat mapping thresholds (Green/Yellow/Red) in the terminal output to match a specific hardware ceiling. |
+| `--vllm` | `--vllm --gpu auto` | Switches from additive memory checking to a subtractive serving capacity estimation. Shows exactly how many tokens fit in the PagedAttention pool. |
+| `--gpu-util` | `--gpu-util 0.9` | Sets the vLLM `gpu_memory_utilization` ratio. Defaults to `0.9` (reserves 10% for PyTorch context). |
+| `--topology` | `--topology nvlink` | Set interconnect topology to calculate exact communication overhead penalties (`nvlink`, `pcie4`, `pcie3`). Defaults to `pcie4`. |
+| `--strategy` | `--strategy tp` | Selects the parallelization strategy for multi-GPU setups (`tp` for Tensor Parallelism, `pp` for Pipeline Parallelism). Defaults to `tp`. |
+| `--tensors` | `--tensors` | Bypasses the algorithmic speed estimation and forces the tool to fetch all remote shards, displaying an exact size breakdown of every tensor. |
 
 ## Architecture
 
-The system operates across three modules:
+Three modules:
 
 1. **Presentation (`cli.py`, `ui.py`)**: Parses arguments and formats tables via `rich`.
-2. **Parsing Engine (`parsers/`)**: Specialized binary readers (`safetensors.py`, `gguf.py`, `pytorch.py`) strictly confined to standard library operations.
+2. **Parsing Engine (`parsers/`)**: Specialized binary readers (`safetensors.py`, `gguf.py`, `pytorch.py`) that use only the standard library.
 3. **Math Engine (`calculator.py`)**: Determines total parameter counts, maps data types to byte coefficients, and calculates dynamic memory allocations based on tensor shape heuristics.
 
 ## License

{modelinfo_cli-1.3.0 → modelinfo_cli-1.4.0}/README.md

@@ -4,25 +4,27 @@
 ![Dependencies](https://img.shields.io/badge/dependencies-rich-green.svg)
 ![License](https://img.shields.io/badge/license-MIT-blue.svg)
 
-ModelInfo is a terminal-native utility that inspects machine learning model checkpoints (`.safetensors`, `.gguf`, `.pt`) and calculates hardware requirements completely offline.
+![ModelInfo Demo](modelinfo.gif)
+
+ModelInfo is a CLI tool that inspects machine learning model checkpoints (`.safetensors`, `.gguf`, `.pt`) and calculates hardware requirements completely offline.
 
 It reads binary headers directly using the Python standard library. By bypassing full tensor payload loading and strictly excluding heavy ecosystems like PyTorch or HuggingFace, the tool executes in under 100 milliseconds.
 
 ## Features
 
-- **Zero-Dependency Parsing**: Reads the 8-byte JSON prefix of `.safetensors` files and the binary key-value metadata of `.gguf` directly via `struct` and `json`. Reads adjacent `config.json` for architecture fallback.
-- **Remote Hugging Face Hub Inspection**: Inspect any public or gated model directly via its repo ID (e.g., `modelinfo meta-llama/Llama-2-7b-hf`) without downloading the checkpoint. Uses concurrent byte-range requests to read the binary headers directly off the CDN in under 2 seconds.
-- **Sharded Model Support**: Transparently parses `model.safetensors.index.json` to detect multi-file checkpoint distributions, gracefully guarding against partial downloads without crashing.
-- **Dynamic VRAM Estimation**: Extracts underlying model architecture to calculate exact VRAM limits, including dynamic KV cache footprints based on user-specified context lengths.
-- **Hardware Fit Diagnostics**: Pass the `--gpu` flag (e.g. `--gpu RTX4090` or `--gpu auto`) to calculate if the model fits in your specific cluster. Defends against fragmentation OOMs using a 3-tier heuristic (Safe, Warning, Fail), calculates overhead across multi-GPU setups, and enforces Apple Silicon's 75% unified memory wire limit.
-- **Side-by-Side Comparison**: Pass multiple models to automatically trigger a comparison table. Compares parameters, data types, context lengths, and VRAM footprints side-by-side to evaluate trade-offs.
-- **Precise Block Quantization**: Factors in exact byte-scaling coefficients for GGUF formats (e.g., Q8, Q6, Q4) rather than naive averages, eliminating VRAM under-reporting.
-- **Secure Pickling**: Inspects legacy `.pt` files without executing arbitrary code by using a highly restricted `pickle.Unpickler`.
-- **Terminal UI**: Groups repetitive structural layers and color-codes VRAM heatmaps using `rich`. Breaks down memory footprints into Weights, KV Cache, and Overhead.
+- **Zero-Dependency Parsing**: Reads `.safetensors` 8-byte JSON prefixes and `.gguf` binary key-value metadata directly via `struct` and `json` (falling back to `config.json` if needed).
+- **Remote Hugging Face Hub Inspection**: Pass a repo ID (e.g., `meta-llama/Llama-2-7b-hf`) and it uses concurrent byte-range requests to read the headers off the CDN in under 2 seconds. No need to download the checkpoint.
+- Parses `model.safetensors.index.json` to support sharded models without crashing on partial downloads.
+- **Dynamic VRAM & Subtractive vLLM Math**: Calculates exact VRAM limits based on the model's architecture and your target context length. If you use the `--vllm` flag, it switches to a subtractive "Serving Capacity" engine that calculates exactly how many tokens fit in the PagedAttention pool based on your `--gpu-util` ratio.
+- **Hardware Fit Diagnostics**: Check if a model fits your cluster with `--gpu` (e.g. `--gpu RTX4090` or `--gpu auto`). It enforces Apple Silicon's 75% unified memory wire limit, and you can explicitly model multi-GPU NCCL communication penalties with `--topology` and `--strategy`.
+- **Side-by-Side Comparison**: Pass multiple models to trigger a comparison table (parameters, data types, context lengths, VRAM footprints).
+- Uses exact `ggml_type` mappings for GGUF formats to calculate byte-scaling coefficients, preventing VRAM under-reporting.
+- **Secure Pickling**: Inspects legacy `.pt` files safely using a restricted `pickle.Unpickler`.
+- The UI (built with `rich`) groups repetitive layers and color-codes VRAM heatmaps.
 
 > [!NOTE]
 > **A Note on Performance & Remote Fetching**
-> Local `.gguf` and `.safetensors` files are parsed in under 100ms. However, querying remote Hugging Face repositories takes **1 to 10 seconds**. This is an intentional trade-off. To remain zero-dependency, `modelinfo` negotiates raw TCP/TLS via Python `urllib` instead of loading PyTorch. For massive sharded models (e.g., 100+ shards), it must fetch every header individually, capped at an 8-worker thread pool to prevent Cloudflare IP bans. Waiting ~8 seconds to map a model is faster than downloading 400GB just to see if it fits your hardware.
+> Local `.gguf` and `.safetensors` files are parsed in under 100ms. However, querying remote Hugging Face repositories takes **1 to 10 seconds**. To remain zero-dependency, `modelinfo` opens connections via Python `urllib` instead of loading PyTorch. For massive sharded models (e.g., 100+ shards), it must fetch every header individually, capped at an 8-worker thread pool to prevent Cloudflare IP bans. Waiting ~8 seconds to map a model is faster than downloading 400GB just to see if it fits your hardware.
 
 ## Installation
 
@@ -102,6 +104,12 @@ Compare multiple models side-by-side against a hardware target:
 modelinfo mistralai/Mistral-7B-v0.1 Qwen/Qwen2.5-0.5B --gpu 12
 ```
 
+Simulate exactly how many tokens you can serve using vLLM on a specific multi-GPU topology:
+
+```bash
+modelinfo mistralai/Mistral-7B-v0.1 --vllm --gpu 4xRTX4090 --topology pcie4 --strategy tp
+```
+
 ### Example Output (Single Model)
 
 ```text
@@ -139,13 +147,18 @@ Qwen2.5-0.5B       494.0M    BF16     8K         1.6 GB      ✓
 | `--gpu` | `--gpu rtx4090` | Check if the model fits. Accepts GPU names (`rtx4090`, `b200`, `rx7900xtx`), explicit VRAM limits in GB (`--gpu 24`), or local hardware auto-discovery (`--gpu auto`). |
 | `--context` | `--context 32768` | Adjust the target KV cache length. Essential for calculating the dynamic memory footprint of long-context models. Defaults to `8192`. |
 | `--max-vram` | `--max-vram 80` | Adjusts the color-coded heat mapping thresholds (Green/Yellow/Red) in the terminal output to match a specific hardware ceiling. |
+| `--vllm` | `--vllm --gpu auto` | Switches from additive memory checking to a subtractive serving capacity estimation. Shows exactly how many tokens fit in the PagedAttention pool. |
+| `--gpu-util` | `--gpu-util 0.9` | Sets the vLLM `gpu_memory_utilization` ratio. Defaults to `0.9` (reserves 10% for PyTorch context). |
+| `--topology` | `--topology nvlink` | Set interconnect topology to calculate exact communication overhead penalties (`nvlink`, `pcie4`, `pcie3`). Defaults to `pcie4`. |
+| `--strategy` | `--strategy tp` | Selects the parallelization strategy for multi-GPU setups (`tp` for Tensor Parallelism, `pp` for Pipeline Parallelism). Defaults to `tp`. |
+| `--tensors` | `--tensors` | Bypasses the algorithmic speed estimation and forces the tool to fetch all remote shards, displaying an exact size breakdown of every tensor. |
 
 ## Architecture
 
-The system operates across three modules:
+Three modules:
 
 1. **Presentation (`cli.py`, `ui.py`)**: Parses arguments and formats tables via `rich`.
-2. **Parsing Engine (`parsers/`)**: Specialized binary readers (`safetensors.py`, `gguf.py`, `pytorch.py`) strictly confined to standard library operations.
+2. **Parsing Engine (`parsers/`)**: Specialized binary readers (`safetensors.py`, `gguf.py`, `pytorch.py`) that use only the standard library.
 3. **Math Engine (`calculator.py`)**: Determines total parameter counts, maps data types to byte coefficients, and calculates dynamic memory allocations based on tensor shape heuristics.
 
 ## License

{modelinfo_cli-1.3.0 → modelinfo_cli-1.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "modelinfo-cli"
-version = "1.3.0"
+version = "1.4.0"
 description = "A sub-100ms, zero-dependency CLI to inspect ML models (.safetensors, .gguf) locally or via Hugging Face, calculate exact VRAM footprints, and determine hardware fit."
 readme = "README.md"
 requires-python = ">=3.10"

{modelinfo_cli-1.3.0 → modelinfo_cli-1.4.0}/src/modelinfo/__init__.py

@@ -2,4 +2,4 @@
 modelinfo - A high-performance CLI utility for inspecting ML model checkpoints.
 """
 
-__version__ = "1.3.0"
+__version__ = "1.4.0"

modelinfo_cli-1.4.0/src/modelinfo/calculator.py

@@ -0,0 +1,178 @@
+import math
+from typing import Any, Dict
+
+from modelinfo.architecture import extract_architecture
+
+DTYPE_BYTES = {
+    "F64": 8.0,
+    "F32": 4.0,
+    "F16": 2.0,
+    "BF16": 2.0,
+    "F8": 1.0,
+    "F8_E5M2": 1.0,
+    "F8_E4M3": 1.0,
+    "I64": 8.0,
+    "I32": 4.0,
+    "I16": 2.0,
+    "I8": 1.0,
+    "U64": 8.0,
+    "U32": 4.0,
+    "Q8_0": 1.0625,
+    "Q8_1": 1.0625,
+    "Q8_K": 1.0625,
+    "Q6_K": 0.828125,
+    "Q5_0": 0.6875,
+    "Q5_1": 0.75,
+    "Q5_K": 0.6875,
+    "Q4_0": 0.5625,
+    "Q4_1": 0.625,
+    "Q4_K": 0.59375,
+    "Q3_K": 0.4375,
+    "Q2_K": 0.34375,
+    "IQ4_NL": 0.53125,
+    "IQ4_XS": 0.53125,
+    "IQ3_S": 0.4375,
+    "IQ3_XXS": 0.385,
+    "IQ2_S": 0.3125,
+    "IQ2_XS": 0.296875,
+    "IQ2_XXS": 0.28125,
+    "IQ1_M": 0.21875,
+    "IQ1_S": 0.1953125,
+    "Q8": 1.06,
+    "Q6": 0.82,
+    "Q5": 0.68,
+    "Q4": 0.58,
+    "Q3": 0.43,
+    "Q2": 0.28,
+}
+
+def _get_bytes_per_param(dtype: str) -> float:
+    """Return the size in bytes for a given data type."""
+    return DTYPE_BYTES.get(dtype.upper(), 2.0)
+
+def calculate_footprint(
+    tensors: Dict[str, Any], 
+    context_length: int = 0, 
+    batch_size: int = 1, 
+    config: Dict[str, Any] = None, 
+    gpu_count: int = 1,
+    topology: str = "pcie4",
+    strategy: str = "tp",
+    is_vllm: bool = False,
+    gpu_vram_bytes: float = 0.0,
+    gpu_util: float = 0.9
+) -> Dict[str, Any]:
+    """
+    Calculate the memory footprint of a model based on its tensors and context length.
+    """
+    total_params = 0
+    base_memory_bytes = 0.0
+    dtype_counts: Dict[str, int] = {}
+    
+    is_lazy = tensors.get("__metadata__", {}).get("lazy_fetch", False)
+    
+    if is_lazy:
+        base_memory_bytes = tensors.get("__metadata__", {}).get("total_size", 0.0)
+        # Assume predominantly FP16/BF16 for modern Hub architectures
+        primary_dtype = "BF16"
+        dtype_counts[primary_dtype] = 1
+        total_params = int(base_memory_bytes / 2.0)
+    else:
+        for name, metadata in tensors.items():
+            if name == "__metadata__":
+                continue
+                
+            shape = metadata.get("shape", [])
+            if not shape:
+                continue
+                
+            param_count = math.prod(shape)
+            total_params += param_count
+            
+            dtype = metadata.get("dtype", "F16").upper()
+            dtype_counts[dtype] = dtype_counts.get(dtype, 0) + 1
+            
+            bytes_per_param = _get_bytes_per_param(dtype)
+            base_memory_bytes += param_count * bytes_per_param
+        
+    num_layers, kv_dim, is_estimate = extract_architecture(tensors, config)
+    
+    # Formula: 2 * Layers * (KV_Heads * Head_Dim) * Context_Length * Batch_Size * Bytes_per_param
+    # Assume FP16 (2 bytes) for KV cache
+    kv_cache_bytes = 2 * num_layers * kv_dim * context_length * batch_size * 2
+    
+    primary_dtype = max(dtype_counts.items(), key=lambda x: x[1])[0] if dtype_counts else "Unknown"
+    # Topology & Strategy Penalties
+    penalty_percentage = 0.0
+    if gpu_count > 1:
+        if strategy == "pp":
+            penalty_percentage = 0.0
+        else: # strategy == "tp"
+            if topology == "nvlink":
+                penalty_percentage = 0.04
+            elif topology == "pcie3":
+                penalty_percentage = 0.20
+            else: # pcie4
+                penalty_percentage = 0.12
+                
+    distributed_overhead = base_memory_bytes * penalty_percentage if gpu_count > 1 else 0.0
+    
+    vllm_metrics = {}
+    if is_vllm and gpu_vram_bytes > 0:
+        usable_vram = gpu_vram_bytes * gpu_util
+        remaining_vram = usable_vram - (base_memory_bytes + distributed_overhead)
+        
+        bytes_per_token = 2 * num_layers * kv_dim * 2
+        
+        max_serving_capacity = 0
+        if remaining_vram > 0 and bytes_per_token > 0:
+            max_serving_capacity = math.floor(remaining_vram / bytes_per_token)
+            
+        overhead_bytes = distributed_overhead
+        total_memory_bytes = base_memory_bytes + overhead_bytes
+        
+        vllm_metrics = {
+            "usable_vram": usable_vram,
+            "static_weights": base_memory_bytes,
+            "distributed_penalty": distributed_overhead,
+            "paged_kv_pool": max(0.0, remaining_vram),
+            "max_serving_capacity": max_serving_capacity
+        }
+    else:
+        CUDA_CONTEXT_MB = 600 * gpu_count
+        overhead_bytes = (CUDA_CONTEXT_MB * 1024 * 1024) + distributed_overhead
+        total_memory_bytes = base_memory_bytes + kv_cache_bytes + overhead_bytes
+    
+    return {
+        "total_params": total_params,
+        "base_memory_bytes": base_memory_bytes,
+        "kv_cache_bytes": kv_cache_bytes,
+        "overhead_bytes": overhead_bytes,
+        "total_memory_bytes": total_memory_bytes,
+        "num_layers": num_layers,
+        "kv_dim": kv_dim,
+        "primary_dtype": primary_dtype,
+        "kv_is_estimate": is_estimate,
+        "penalty_percentage": penalty_percentage,
+        "vllm_metrics": vllm_metrics
+    }
+
+def format_bytes(size_bytes: float) -> str:
+    """Format bytes into a human-readable string (e.g. GB)."""
+    if size_bytes == 0:
+        return "0 B"
+    units = ["B", "KB", "MB", "GB", "TB", "PB"]
+    i = max(0, min(len(units) - 1, math.floor(math.log(size_bytes, 1024))))
+    p = math.pow(1024, i)
+    s = round(size_bytes / p, 2)
+    return f"{s} {units[i]}"
+
+def format_params(count: int) -> str:
+    """Format parameter count into a human-readable string (e.g. 7.2B)."""
+    if count >= 1_000_000_000:
+        return f"{count:,} ({count / 1_000_000_000:.1f}B)"
+    elif count >= 1_000_000:
+        return f"{count:,} ({count / 1_000_000:.1f}M)"
+    elif count >= 1_000:
+        return f"{count:,} ({count / 1_000:.1f}K)"
+    return f"{count:,}"

{modelinfo_cli-1.3.0 → modelinfo_cli-1.4.0}/src/modelinfo/cli.py

@@ -42,11 +42,51 @@ def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace:
         default=None,
         help="Target GPU hardware (e.g. 'RTX4090' or 'auto') to check if the model fits.",
     )
+    parser.add_argument(
+        "--tensors",
+        action="store_true",
+        help="Deep dive: Fetch all remote tensor shards to display the exact tensor size breakdown.",
+    )
+    parser.add_argument(
+        "--topology",
+        type=str,
+        choices=["nvlink", "pcie4", "pcie3"],
+        default="pcie4",
+        help="Interconnect topology to calculate distributed communication overhead.",
+    )
+    parser.add_argument(
+        "--strategy",
+        type=str,
+        choices=["tp", "pp"],
+        default="tp",
+        help="Distributed parallelism strategy (Tensor vs Pipeline).",
+    )
+    parser.add_argument(
+        "--vllm",
+        action="store_true",
+        help="Enable Subtractive Math Engine: Calculate max context tokens using vLLM PagedAttention allocation.",
+    )
+    parser.add_argument(
+        "--gpu-util",
+        type=float,
+        default=0.9,
+        help="vLLM gpu_memory_utilization ratio (default 0.9). Reserves 10 percent for PyTorch context.",
+    )
 
     return parser.parse_args(argv)
 
 
-def analyze_model(file_path: str, context_override: int | None, gpu_count: int = 1) -> dict:
+def analyze_model(
+    file_path: str, 
+    context_override: int | None, 
+    gpu_count: int = 1, 
+    fetch_tensors: bool = False,
+    topology: str = "pcie4",
+    strategy: str = "tp",
+    is_vllm: bool = False,
+    gpu_vram_gb: float = 0.0,
+    gpu_util: float = 0.9
+) -> dict:
     tensors = {}
     config = None
     disk_size = 0.0
@@ -55,7 +95,7 @@ def analyze_model(file_path: str, context_override: int | None, gpu_count: int =
     
     if not os.path.exists(file_path) and not file_path_lower.endswith((".safetensors", ".gguf", ".pt", ".bin", ".index.json")):
         from modelinfo.parsers.huggingface import fetch_huggingface_repo
-        tensors, config, format_name, disk_size = fetch_huggingface_repo(file_path)
+        tensors, config, format_name, disk_size = fetch_huggingface_repo(file_path, fetch_tensors=fetch_tensors)
     elif file_path_lower.endswith(".safetensors") or file_path_lower.endswith(".index.json"):
         tensors = parse_safetensors_header(file_path)
         format_name = "SafeTensors"
@@ -92,7 +132,17 @@ def analyze_model(file_path: str, context_override: int | None, gpu_count: int =
         context_length = min(8192, max_context) if max_context else 8192
         is_default_context = True
 
-    footprint = calculate_footprint(tensors, context_length=context_length, config=config, gpu_count=gpu_count)
+    footprint = calculate_footprint(
+        tensors, 
+        context_length=context_length,
+        config=config,
+        gpu_count=gpu_count,
+        topology=topology,
+        strategy=strategy,
+        is_vllm=is_vllm,
+        gpu_vram_bytes=gpu_vram_gb * 1024**3 if gpu_vram_gb else 0.0,
+        gpu_util=gpu_util
+    )
     num_layers = footprint["num_layers"]
     arch_name = identify_architecture_name(tensors, num_layers, config)
 
@@ -110,7 +160,14 @@ def analyze_model(file_path: str, context_override: int | None, gpu_count: int =
         "context_length": context_length,
         "is_default_context": is_default_context,
         "tensors": tensors,
-        "max_context": max_context
+        "max_context": max_context,
+        "is_lazy": tensors.get("__metadata__", {}).get("lazy_fetch", False),
+        "gpu_count": gpu_count,
+        "topology": topology,
+        "strategy": strategy,
+        "is_vllm": is_vllm,
+        "gpu_vram_gb": gpu_vram_gb,
+        "gpu_util": gpu_util
     }
 
 
@@ -118,20 +175,33 @@ def main(argv: Sequence[str] | None = None) -> int:
     args = parse_args(argv)
 
     gpu_name_display = None
+    gpu_vram_gb = None
     gpu_count = 1
-    if args.gpu:
+    
+    if args.gpu or args.vllm:
+        target = args.gpu if args.gpu else "auto"
         from modelinfo.hardware import resolve_gpu
-        try:
-            gpu_name_display, args.max_vram, gpu_count = resolve_gpu(args.gpu)
-        except Exception as e:
-            console.print(f"[red]{e}[/red]")
-            return 1
+        gpu_name_display, gpu_vram_gb, gpu_count = resolve_gpu(target)
 
     if len(args.file) > 1:
+        if args.vllm:
+            console.print("[red]Error: Side-by-side comparison does not currently support the subtractive --vllm engine. Compare models sequentially or remove --vllm.[/red]")
+            return 1
+            
         models = []
         for model_path in args.file:
             try:
-                info = analyze_model(model_path, args.context, gpu_count)
+                info = analyze_model(
+                    model_path, 
+                    args.context, 
+                    gpu_count, 
+                    fetch_tensors=args.tensors,
+                    topology=args.topology,
+                    strategy=args.strategy,
+                    is_vllm=args.vllm,
+                    gpu_vram_gb=gpu_vram_gb if gpu_vram_gb else 0.0,
+                    gpu_util=args.gpu_util
+                )
                 models.append((model_path.split("/")[-1], info))
             except Exception as e:
                 console.print(f"[red]Error analyzing model '{model_path}': {e}[/red]")
@@ -143,12 +213,22 @@ def main(argv: Sequence[str] | None = None) -> int:
     file_path = args.file[0]
     
     try:
-        info = analyze_model(file_path, args.context, gpu_count)
+        info = analyze_model(
+            file_path, 
+            args.context, 
+            gpu_count, 
+            fetch_tensors=args.tensors,
+            topology=args.topology,
+            strategy=args.strategy,
+            is_vllm=args.vllm,
+            gpu_vram_gb=gpu_vram_gb if gpu_vram_gb else 0.0,
+            gpu_util=args.gpu_util
+        )
     except Exception as e:
         console.print(f"[red]Error: {e}[/red]")
         return 1
 
-    print_model_info(**info, max_vram_gb=args.max_vram, gpu_name=gpu_name_display)
+    print_model_info(**info, max_vram_gb=gpu_vram_gb if gpu_vram_gb else 8.0, gpu_name=gpu_name_display)
     return 0
 
 

{modelinfo_cli-1.3.0 → modelinfo_cli-1.4.0}/src/modelinfo/parsers/gguf.py

@@ -1,6 +1,14 @@
 import struct
 from typing import Any, Dict
 
+GGML_TYPE_MAP = {
+    0: "F32", 1: "F16", 2: "Q4_0", 3: "Q4_1", 4: "Q4_1_O", 5: "Q4_0_O", 
+    6: "Q5_0", 7: "Q5_1", 8: "Q8_0", 9: "Q8_1", 10: "Q2_K", 11: "Q3_K", 
+    12: "Q4_K", 13: "Q5_K", 14: "Q6_K", 15: "Q8_K", 16: "IQ2_XXS", 17: "IQ2_XS", 
+    18: "IQ3_XXS", 19: "IQ1_S", 20: "IQ4_NL", 21: "IQ3_S", 22: "IQ2_S", 
+    23: "IQ4_XS", 24: "I8", 25: "I16", 26: "I32", 27: "I64", 28: "F64", 
+    29: "IQ1_M", 30: "BF16", 31: "Q4_0_4_4", 32: "Q4_0_4_8", 33: "Q4_0_8_8",
+}
 
 def _read_gguf_value(f: Any, val_type: int) -> Any:
     if val_type == 0:
@@ -73,12 +81,8 @@ def parse_gguf_header(path: str) -> Dict[str, Any]:
             t_type = struct.unpack("<I", f.read(4))[0]
             f.read(8)  # skip offset bytes
             
-            # Simplified GGUF tensor type mapping
-            dtype = "F32"
-            if t_type == 1:
-                dtype = "F16"
-            elif t_type > 1:
-                dtype = "Q4" # Generic placeholder for quantized types
+            # Strict GGUF tensor type mapping
+            dtype = GGML_TYPE_MAP.get(t_type, "Unknown")
                 
             tensors[name] = {"shape": shape, "dtype": dtype}
             

{modelinfo_cli-1.3.0 → modelinfo_cli-1.4.0}/src/modelinfo/parsers/huggingface.py

@@ -76,7 +76,7 @@ def _fetch_safetensors_header(repo_id: str, filename: str) -> Dict[str, Any]:
         
     return json.loads(json_bytes)
 
-def fetch_huggingface_repo(repo_id: str) -> Tuple[Dict[str, Any], Dict[str, Any] | None, str, float]:
+def fetch_huggingface_repo(repo_id: str, fetch_tensors: bool = False) -> Tuple[Dict[str, Any], Dict[str, Any] | None, str, float]:
     """
     Fetches the metadata directly from the Hugging Face Hub over the network.
     Returns: (tensors, config, format_name, disk_size)
@@ -110,31 +110,41 @@ def fetch_huggingface_repo(repo_id: str) -> Tuple[Dict[str, Any], Dict[str, Any]
         
         total_size = index_data.get("metadata", {}).get("total_size", 0.0)
         
-        def fetch_shard(shard: str):
-            return shard, _fetch_safetensors_header(repo_id, shard)
-            
-        with concurrent.futures.ThreadPoolExecutor(max_workers=min(8, len(unique_shards))) as executor:
-            future_to_shard = {executor.submit(fetch_shard, shard): shard for shard in unique_shards}
-            for future in concurrent.futures.as_completed(future_to_shard):
-                shard, shard_header = future.result()
-                for k, v in shard_header.items():
-                    if k != "__metadata__":
-                        tensors[k] = v
-                        
-        tensors["__metadata__"] = {
-            "missing_shards": 0,
-            "total_shards": len(unique_shards),
-            "is_sharded": True
-        }
+        if config and not fetch_tensors and total_size > 0:
+            # Lazy Fetch Paradigm
+            for tensor_name in weight_map.keys():
+                tensors[tensor_name] = {"shape": [], "dtype": "BF16"}
+                
+            tensors["__metadata__"] = {
+                "missing_shards": 0,
+                "total_shards": len(unique_shards),
+                "is_sharded": True,
+                "lazy_fetch": True,
+                "total_size": total_size
+            }
+        else:
+            def fetch_shard(shard: str):
+                return shard, _fetch_safetensors_header(repo_id, shard)
+                
+            with concurrent.futures.ThreadPoolExecutor(max_workers=min(8, len(unique_shards))) as executor:
+                future_to_shard = {executor.submit(fetch_shard, shard): shard for shard in unique_shards}
+                for future in concurrent.futures.as_completed(future_to_shard):
+                    shard, shard_header = future.result()
+                    for k, v in shard_header.items():
+                        if k != "__metadata__":
+                            tensors[k] = v
+                            
+            tensors["__metadata__"] = {
+                "missing_shards": 0,
+                "total_shards": len(unique_shards),
+                "is_sharded": True
+            }
         format_name = "SafeTensors"
         
     elif "model.safetensors" in filenames:
         # Single SafeTensors
-        header = _fetch_safetensors_header(repo_id, "model.safetensors")
-        tensors = header
-        format_name = "SafeTensors"
         
-        # We don't have total_size from index, so we could get it from Content-Length or just leave it 0
+        # Determine total size first
         req = urllib.request.Request(f"https://huggingface.co/{repo_id}/resolve/main/model.safetensors", method="HEAD")
         token = _get_hf_token()
         if token:
@@ -144,7 +154,12 @@ def fetch_huggingface_repo(repo_id: str) -> Tuple[Dict[str, Any], Dict[str, Any]
                 total_size = int(response.headers.get("Content-Length", 0))
         except Exception:
             pass
+
+        header = _fetch_safetensors_header(repo_id, "model.safetensors")
+        tensors = header
             
+        format_name = "SafeTensors"
+        
     else:
         raise ValueError(f"Repository {repo_id} does not contain SafeTensors weights.")
         

{modelinfo_cli-1.3.0 → modelinfo_cli-1.4.0}/src/modelinfo/ui.py

@@ -47,7 +47,14 @@ def print_model_info(
     tensors: Dict[str, Any],
     max_context: int | None = None,
     max_vram_gb: float = 8.0,
-    gpu_name: str | None = None
+    gpu_name: str | None = None,
+    is_lazy: bool = False,
+    gpu_count: int = 1,
+    topology: str = "pcie4",
+    strategy: str = "tp",
+    is_vllm: bool = False,
+    gpu_vram_gb: float = 0.0,
+    gpu_util: float = 0.9
 ) -> None:
     summary = Table(box=None, show_header=False, pad_edge=False, padding=(0, 2))
     summary.add_column("Property", style="bold")
@@ -92,7 +99,11 @@ def print_model_info(
         vram_display += f"  ├─ KV Cache:   {format_bytes(kv_cache_bytes)}{kv_note}\n"
         
         overhead_bytes = footprint.get("overhead_bytes", 600 * 1024 * 1024)
-        vram_display += f"  └─ Overhead:   {format_bytes(overhead_bytes)} (CUDA Context + Activations)"
+        if gpu_count > 1:
+            penalty_str = f"TP/{topology}" if strategy == "tp" else "PP"
+            vram_display += f"  └─ Overhead:   {format_bytes(overhead_bytes)} (CUDA Contexts + {penalty_str} Penalty)"
+        else:
+            vram_display += f"  └─ Overhead:   {format_bytes(overhead_bytes)} (CUDA Context + Activations)"
 
     summary.add_row("Format:", format_name)
     summary.add_row("Architecture:", arch_name)
@@ -100,17 +111,45 @@ def print_model_info(
     summary.add_row("Parameters:", param_text)
     summary.add_row("Dtype:", footprint["primary_dtype"])
     summary.add_row("Disk size:", disk_text)
-    summary.add_row("VRAM (est):", vram_display)
     
-    if gpu_name:
-        utilization = vram_bytes / (max_vram_gb * 1024**3) if max_vram_gb > 0 else 2.0
-        if utilization <= 0.90:
-            fit_text = f"[green]✓ Fits comfortably in {gpu_name} ({max_vram_gb:.1f} GB)[/green]"
-        elif utilization <= 0.99:
-            fit_text = f"[yellow]⚠ Warning: Extreme hardware limit on {gpu_name}. High risk of fragmentation OOM.[/yellow]"
+    if is_vllm:
+        vllm = footprint.get("vllm_metrics", {})
+        usable_vram = vllm.get("usable_vram", 0.0)
+        static_weights = vllm.get("static_weights", 0.0)
+        distributed_penalty = vllm.get("distributed_penalty", 0.0)
+        paged_kv_pool = vllm.get("paged_kv_pool", 0.0)
+        max_capacity = vllm.get("max_serving_capacity", 0)
+        
+        summary.add_row("VRAM Ceiling:", f"{max_vram_gb:.1f} GB ({gpu_name if gpu_name else 'Target'})")
+        
+        alloc_display = f"  ├─ Usable VRAM:      {format_bytes(usable_vram)} ({int(gpu_util*100)}% gpu_memory_utilization)\n"
+        alloc_display += f"  ├─ Static Weights:  -{format_bytes(static_weights)} ({footprint.get('primary_dtype', 'BF16')})\n"
+        if gpu_count > 1:
+            penalty_str = f"TP/{topology}" if strategy == "tp" else "PP"
+            alloc_display += f"  ├─ {penalty_str} Penalty: -{format_bytes(distributed_penalty)}\n"
+        alloc_display += f"  └─ Paged KV Pool:   = {format_bytes(paged_kv_pool)} Available for Context"
+        
+        summary.add_row("vLLM Allocation:", alloc_display)
+        summary.add_row("Max Capacity:", f"~{max_capacity:,} Tokens (Across all concurrent batches)")
+        
+        if paged_kv_pool <= 0:
+            summary.add_row("Hardware Fit:", "[red]✗ No (OOM before serving any tokens)[/red]")
         else:
-            fit_text = f"[red]✗ No (Requires {format_bytes(vram_bytes)}, Hardware has {max_vram_gb:.1f} GB)[/red]"
-        summary.add_row("Hardware Fit:", fit_text)
+            summary.add_row("Hardware Fit:", "[green]✓ Yes[/green]")
+            
+        if gpu_count > 1:
+            summary.add_row("", "[dim]*Note: Max capacity assumes perfect load balancing. Real capacity is bottlenecked by the most memory-constrained GPU in the array.[/dim]")
+    else:
+        summary.add_row("VRAM (est):", vram_display)
+        if gpu_name:
+            utilization = vram_bytes / (max_vram_gb * 1024**3) if max_vram_gb > 0 else 2.0
+            if utilization <= 0.90:
+                fit_text = f"[green]✓ Fits comfortably in {gpu_name} ({max_vram_gb:.1f} GB)[/green]"
+            elif utilization <= 0.99:
+                fit_text = f"[yellow]⚠ Warning: Extreme hardware limit on {gpu_name}. High risk of fragmentation OOM.[/yellow]"
+            else:
+                fit_text = f"[red]✗ No (Requires {format_bytes(vram_bytes)}, Hardware has {max_vram_gb:.1f} GB)[/red]"
+            summary.add_row("Hardware Fit:", fit_text)
     
     console.print(summary)
 
@@ -122,6 +161,10 @@ def print_model_info(
 
     console.print()
     
+    if is_lazy:
+        console.print("[yellow]Top Tensors omitted for speed. Run with --tensors to fetch remote shards.[/yellow]")
+        return
+        
     console.print("Top Tensors by Size:", style="bold")
     
     grouped_tensors = group_tensors_by_size(tensors)

{modelinfo_cli-1.3.0 → modelinfo_cli-1.4.0}/src/modelinfo_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: modelinfo-cli
-Version: 1.3.0
+Version: 1.4.0
 Summary: A sub-100ms, zero-dependency CLI to inspect ML models (.safetensors, .gguf) locally or via Hugging Face, calculate exact VRAM footprints, and determine hardware fit.
 Author: ModelInfo Contributors
 License: MIT
@@ -22,25 +22,27 @@ Dynamic: license-file
 ![Dependencies](https://img.shields.io/badge/dependencies-rich-green.svg)
 ![License](https://img.shields.io/badge/license-MIT-blue.svg)
 
-ModelInfo is a terminal-native utility that inspects machine learning model checkpoints (`.safetensors`, `.gguf`, `.pt`) and calculates hardware requirements completely offline.
+![ModelInfo Demo](modelinfo.gif)
+
+ModelInfo is a CLI tool that inspects machine learning model checkpoints (`.safetensors`, `.gguf`, `.pt`) and calculates hardware requirements completely offline.
 
 It reads binary headers directly using the Python standard library. By bypassing full tensor payload loading and strictly excluding heavy ecosystems like PyTorch or HuggingFace, the tool executes in under 100 milliseconds.
 
 ## Features
 
-- **Zero-Dependency Parsing**: Reads the 8-byte JSON prefix of `.safetensors` files and the binary key-value metadata of `.gguf` directly via `struct` and `json`. Reads adjacent `config.json` for architecture fallback.
-- **Remote Hugging Face Hub Inspection**: Inspect any public or gated model directly via its repo ID (e.g., `modelinfo meta-llama/Llama-2-7b-hf`) without downloading the checkpoint. Uses concurrent byte-range requests to read the binary headers directly off the CDN in under 2 seconds.
-- **Sharded Model Support**: Transparently parses `model.safetensors.index.json` to detect multi-file checkpoint distributions, gracefully guarding against partial downloads without crashing.
-- **Dynamic VRAM Estimation**: Extracts underlying model architecture to calculate exact VRAM limits, including dynamic KV cache footprints based on user-specified context lengths.
-- **Hardware Fit Diagnostics**: Pass the `--gpu` flag (e.g. `--gpu RTX4090` or `--gpu auto`) to calculate if the model fits in your specific cluster. Defends against fragmentation OOMs using a 3-tier heuristic (Safe, Warning, Fail), calculates overhead across multi-GPU setups, and enforces Apple Silicon's 75% unified memory wire limit.
-- **Side-by-Side Comparison**: Pass multiple models to automatically trigger a comparison table. Compares parameters, data types, context lengths, and VRAM footprints side-by-side to evaluate trade-offs.
-- **Precise Block Quantization**: Factors in exact byte-scaling coefficients for GGUF formats (e.g., Q8, Q6, Q4) rather than naive averages, eliminating VRAM under-reporting.
-- **Secure Pickling**: Inspects legacy `.pt` files without executing arbitrary code by using a highly restricted `pickle.Unpickler`.
-- **Terminal UI**: Groups repetitive structural layers and color-codes VRAM heatmaps using `rich`. Breaks down memory footprints into Weights, KV Cache, and Overhead.
+- **Zero-Dependency Parsing**: Reads `.safetensors` 8-byte JSON prefixes and `.gguf` binary key-value metadata directly via `struct` and `json` (falling back to `config.json` if needed).
+- **Remote Hugging Face Hub Inspection**: Pass a repo ID (e.g., `meta-llama/Llama-2-7b-hf`) and it uses concurrent byte-range requests to read the headers off the CDN in under 2 seconds. No need to download the checkpoint.
+- Parses `model.safetensors.index.json` to support sharded models without crashing on partial downloads.
+- **Dynamic VRAM & Subtractive vLLM Math**: Calculates exact VRAM limits based on the model's architecture and your target context length. If you use the `--vllm` flag, it switches to a subtractive "Serving Capacity" engine that calculates exactly how many tokens fit in the PagedAttention pool based on your `--gpu-util` ratio.
+- **Hardware Fit Diagnostics**: Check if a model fits your cluster with `--gpu` (e.g. `--gpu RTX4090` or `--gpu auto`). It enforces Apple Silicon's 75% unified memory wire limit, and you can explicitly model multi-GPU NCCL communication penalties with `--topology` and `--strategy`.
+- **Side-by-Side Comparison**: Pass multiple models to trigger a comparison table (parameters, data types, context lengths, VRAM footprints).
+- Uses exact `ggml_type` mappings for GGUF formats to calculate byte-scaling coefficients, preventing VRAM under-reporting.
+- **Secure Pickling**: Inspects legacy `.pt` files safely using a restricted `pickle.Unpickler`.
+- The UI (built with `rich`) groups repetitive layers and color-codes VRAM heatmaps.
 
 > [!NOTE]
 > **A Note on Performance & Remote Fetching**
-> Local `.gguf` and `.safetensors` files are parsed in under 100ms. However, querying remote Hugging Face repositories takes **1 to 10 seconds**. This is an intentional trade-off. To remain zero-dependency, `modelinfo` negotiates raw TCP/TLS via Python `urllib` instead of loading PyTorch. For massive sharded models (e.g., 100+ shards), it must fetch every header individually, capped at an 8-worker thread pool to prevent Cloudflare IP bans. Waiting ~8 seconds to map a model is faster than downloading 400GB just to see if it fits your hardware.
+> Local `.gguf` and `.safetensors` files are parsed in under 100ms. However, querying remote Hugging Face repositories takes **1 to 10 seconds**. To remain zero-dependency, `modelinfo` opens connections via Python `urllib` instead of loading PyTorch. For massive sharded models (e.g., 100+ shards), it must fetch every header individually, capped at an 8-worker thread pool to prevent Cloudflare IP bans. Waiting ~8 seconds to map a model is faster than downloading 400GB just to see if it fits your hardware.
 
 ## Installation
 
@@ -120,6 +122,12 @@ Compare multiple models side-by-side against a hardware target:
 modelinfo mistralai/Mistral-7B-v0.1 Qwen/Qwen2.5-0.5B --gpu 12
 ```
 
+Simulate exactly how many tokens you can serve using vLLM on a specific multi-GPU topology:
+
+```bash
+modelinfo mistralai/Mistral-7B-v0.1 --vllm --gpu 4xRTX4090 --topology pcie4 --strategy tp
+```
+
 ### Example Output (Single Model)
 
 ```text
@@ -157,13 +165,18 @@ Qwen2.5-0.5B       494.0M    BF16     8K         1.6 GB      ✓
 | `--gpu` | `--gpu rtx4090` | Check if the model fits. Accepts GPU names (`rtx4090`, `b200`, `rx7900xtx`), explicit VRAM limits in GB (`--gpu 24`), or local hardware auto-discovery (`--gpu auto`). |
 | `--context` | `--context 32768` | Adjust the target KV cache length. Essential for calculating the dynamic memory footprint of long-context models. Defaults to `8192`. |
 | `--max-vram` | `--max-vram 80` | Adjusts the color-coded heat mapping thresholds (Green/Yellow/Red) in the terminal output to match a specific hardware ceiling. |
+| `--vllm` | `--vllm --gpu auto` | Switches from additive memory checking to a subtractive serving capacity estimation. Shows exactly how many tokens fit in the PagedAttention pool. |
+| `--gpu-util` | `--gpu-util 0.9` | Sets the vLLM `gpu_memory_utilization` ratio. Defaults to `0.9` (reserves 10% for PyTorch context). |
+| `--topology` | `--topology nvlink` | Set interconnect topology to calculate exact communication overhead penalties (`nvlink`, `pcie4`, `pcie3`). Defaults to `pcie4`. |
+| `--strategy` | `--strategy tp` | Selects the parallelization strategy for multi-GPU setups (`tp` for Tensor Parallelism, `pp` for Pipeline Parallelism). Defaults to `tp`. |
+| `--tensors` | `--tensors` | Bypasses the algorithmic speed estimation and forces the tool to fetch all remote shards, displaying an exact size breakdown of every tensor. |
 
 ## Architecture
 
-The system operates across three modules:
+Three modules:
 
 1. **Presentation (`cli.py`, `ui.py`)**: Parses arguments and formats tables via `rich`.
-2. **Parsing Engine (`parsers/`)**: Specialized binary readers (`safetensors.py`, `gguf.py`, `pytorch.py`) strictly confined to standard library operations.
+2. **Parsing Engine (`parsers/`)**: Specialized binary readers (`safetensors.py`, `gguf.py`, `pytorch.py`) that use only the standard library.
 3. **Math Engine (`calculator.py`)**: Determines total parameter counts, maps data types to byte coefficients, and calculates dynamic memory allocations based on tensor shape heuristics.
 
 ## License

{modelinfo_cli-1.3.0 → modelinfo_cli-1.4.0}/tests/test_calculator.py

@@ -1,3 +1,4 @@
+import math
 from modelinfo.calculator import calculate_footprint, _get_bytes_per_param
 
 def test_quantization_byte_multipliers():
@@ -110,3 +111,61 @@ def test_framework_overhead_included():
     assert "overhead_bytes" in footprint
     assert footprint["overhead_bytes"] == 600 * 1024 * 1024
     assert footprint["total_memory_bytes"] == footprint["base_memory_bytes"] + footprint["kv_cache_bytes"] + footprint["overhead_bytes"]
+
+def test_explicit_gguf_quantization_byte_multipliers():
+    """Verify that explicit ggml_type enums are exactly mapped."""
+    assert _get_bytes_per_param("Q8_0") == 1.0625
+    assert _get_bytes_per_param("Q4_K") == 0.59375
+    assert _get_bytes_per_param("IQ2_XXS") == 0.28125
+    assert _get_bytes_per_param("F8_E5M2") == 1.0
+
+def test_topology_penalties():
+    """Verify multi-GPU distributed overhead logic."""
+    tensors = {
+        "model.layers.0.attn.weight": {"shape": [1024, 1024], "dtype": "F16"} # Base: 2,097,152 bytes
+    }
+    # NVLink (4%)
+    fp_nvlink = calculate_footprint(tensors, gpu_count=2, topology="nvlink", strategy="tp")
+    assert fp_nvlink["penalty_percentage"] == 0.04
+    assert fp_nvlink["overhead_bytes"] == (2 * 600 * 1024 * 1024) + (2097152 * 0.04)
+
+    # PCIe3 (20%)
+    fp_pcie3 = calculate_footprint(tensors, gpu_count=4, topology="pcie3", strategy="tp")
+    assert fp_pcie3["penalty_percentage"] == 0.20
+    assert fp_pcie3["overhead_bytes"] == (4 * 600 * 1024 * 1024) + (2097152 * 0.20)
+
+def test_strategy_pp():
+    """Verify Pipeline Parallelism incurs 0 distributed overhead."""
+    tensors = {
+        "model.layers.0.attn.weight": {"shape": [1024, 1024], "dtype": "F16"}
+    }
+    fp_pp = calculate_footprint(tensors, gpu_count=4, topology="pcie3", strategy="pp")
+    assert fp_pp["penalty_percentage"] == 0.0
+    assert fp_pp["overhead_bytes"] == (4 * 600 * 1024 * 1024)
+
+def test_vllm_subtractive_math():
+    """Verify the subtractive vLLM serving capacity engine calculates exact tokens."""
+    tensors = {
+        "model.layers.0.attn.weight": {"shape": [1024, 1024], "dtype": "F16"} # Base: 2MB
+    }
+    config = {
+        "num_hidden_layers": 10,
+        "num_attention_heads": 8,
+        "num_key_value_heads": 8,
+        "hidden_size": 1024
+    }
+    # 24GB VRAM. 90% util = 21.6GB. Base weights = 2MB. Remaining = ~21.59GB.
+    # Bytes per token: 2 (FP16) * 10 (layers) * 1024 (kv_dim) * 2 = 40960 bytes
+    gpu_vram = 24.0 * 1024**3
+    
+    fp_vllm = calculate_footprint(tensors, config=config, is_vllm=True, gpu_vram_bytes=gpu_vram, gpu_util=0.9, gpu_count=1)
+    
+    metrics = fp_vllm["vllm_metrics"]
+    assert "usable_vram" in metrics
+    assert metrics["usable_vram"] == gpu_vram * 0.9
+    assert metrics["static_weights"] == 2097152
+    assert metrics["paged_kv_pool"] == metrics["usable_vram"] - metrics["static_weights"]
+    
+    bytes_per_token = 40960
+    expected_capacity = math.floor(metrics["paged_kv_pool"] / bytes_per_token)
+    assert metrics["max_serving_capacity"] == expected_capacity

modelinfo_cli-1.3.0/src/modelinfo/calculator.py

@@ -1,98 +0,0 @@
-import math
-from typing import Any, Dict
-
-from modelinfo.architecture import extract_architecture
-
-DTYPE_BYTES = {
-    "F64": 8,
-    "F32": 4,
-    "F16": 2,
-    "BF16": 2,
-    "F8": 1,
-    "F8_E5M2": 1,
-    "F8_E4M3": 1,
-    "I64": 8,
-    "I32": 4,
-    "I16": 2,
-    "I8": 1,
-    "U64": 8,
-    "U32": 4,
-    "Q8": 1.06,
-    "Q6": 0.82,
-    "Q5": 0.68,
-    "Q4": 0.58,
-    "Q3": 0.43,
-    "Q2": 0.28,
-}
-
-def _get_bytes_per_param(dtype: str) -> float:
-    """Return the size in bytes for a given data type."""
-    return DTYPE_BYTES.get(dtype.upper(), 2.0)
-
-def calculate_footprint(tensors: Dict[str, Any], context_length: int = 0, batch_size: int = 1, config: Dict[str, Any] = None, gpu_count: int = 1) -> Dict[str, Any]:
-    """
-    Calculate the memory footprint of a model based on its tensors and context length.
-    """
-    total_params = 0
-    base_memory_bytes = 0.0
-    dtype_counts: Dict[str, int] = {}
-    
-    for name, metadata in tensors.items():
-        if name == "__metadata__":
-            continue
-            
-        shape = metadata.get("shape", [])
-        if not shape:
-            continue
-            
-        param_count = math.prod(shape)
-        total_params += param_count
-        
-        dtype = metadata.get("dtype", "F16").upper()
-        dtype_counts[dtype] = dtype_counts.get(dtype, 0) + 1
-        
-        bytes_per_param = _get_bytes_per_param(dtype)
-        base_memory_bytes += param_count * bytes_per_param
-        
-    num_layers, kv_dim, is_estimate = extract_architecture(tensors, config)
-    
-    # Formula: 2 * Layers * (KV_Heads * Head_Dim) * Context_Length * Batch_Size * Bytes_per_param
-    # Assume FP16 (2 bytes) for KV cache
-    kv_cache_bytes = 2 * num_layers * kv_dim * context_length * batch_size * 2
-    
-    primary_dtype = max(dtype_counts.items(), key=lambda x: x[1])[0] if dtype_counts else "Unknown"
-    
-    CUDA_CONTEXT_MB = 600 * gpu_count
-    overhead_bytes = CUDA_CONTEXT_MB * 1024 * 1024
-    
-    return {
-        "total_params": total_params,
-        "base_memory_bytes": base_memory_bytes,
-        "kv_cache_bytes": kv_cache_bytes,
-        "overhead_bytes": overhead_bytes,
-        "total_memory_bytes": base_memory_bytes + kv_cache_bytes + overhead_bytes,
-        "num_layers": num_layers,
-        "kv_dim": kv_dim,
-        "primary_dtype": primary_dtype,
-        "kv_is_estimate": is_estimate
-    }
-
-def format_bytes(size_bytes: float) -> str:
-    """Format bytes into a human-readable string (e.g. GB)."""
-    if size_bytes == 0:
-        return "0 B"
-    units = ["B", "KB", "MB", "GB", "TB", "PB"]
-    i = max(0, min(len(units) - 1, math.floor(math.log(size_bytes, 1024))))
-    p = math.pow(1024, i)
-    s = round(size_bytes / p, 2)
-    return f"{s} {units[i]}"
-
-def format_params(count: int) -> str:
-    """Format parameter count into a human-readable string (e.g. 7.2B)."""
-    if count >= 1_000_000_000:
-        return f"{count:,} ({count / 1_000_000_000:.1f}B)"
-    elif count >= 1_000_000:
-        return f"{count:,} ({count / 1_000_000:.1f}M)"
-    elif count >= 1_000:
-        return f"{count:,} ({count / 1_000:.1f}K)"
-    return f"{count:,}"