digvijay-llm 0.1.0__tar.gz

digvijay_llm-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Digvijay
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

digvijay_llm-0.1.0/PKG-INFO

@@ -0,0 +1,163 @@
+Metadata-Version: 2.4
+Name: digvijay_llm
+Version: 0.1.0
+Summary: Run 70B LLMs on low-RAM machines by streaming weights from disk
+Author: Digvijay
+License: MIT License
+        
+        Copyright (c) 2026 Digvijay
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: psutil>=5.9.0
+Provides-Extra: gguf
+Requires-Dist: llama-cpp-python>=0.2.0; extra == "gguf"
+Provides-Extra: hf
+Requires-Dist: torch>=2.0.0; extra == "hf"
+Requires-Dist: transformers>=4.38.0; extra == "hf"
+Requires-Dist: accelerate>=0.27.0; extra == "hf"
+Requires-Dist: safetensors>=0.4.0; extra == "hf"
+Provides-Extra: all
+Requires-Dist: llama-cpp-python>=0.2.0; extra == "all"
+Requires-Dist: torch>=2.0.0; extra == "all"
+Requires-Dist: transformers>=4.38.0; extra == "all"
+Requires-Dist: accelerate>=0.27.0; extra == "all"
+Requires-Dist: safetensors>=0.4.0; extra == "all"
+Dynamic: author
+Dynamic: license-file
+Dynamic: requires-python
+
+# digvijay_llm
+
+Run big open-source LLMs — including 70B-class models — on machines with as little as **16 GB of RAM**, by streaming weights from disk ("ROM") instead of loading the full model into RAM.
+
+## ⚠️ Read this first (important honesty note)
+
+A CPU/GPU can only *compute* on data sitting in RAM/VRAM — it is physically impossible to run matrix multiplications directly on a disk. What this library actually does, and what makes low-RAM 70B inference possible, is **disk-backed weight streaming**: weights live on disk and are pulled into RAM only for the layer currently being computed, then released. Peak RAM usage becomes roughly "one layer's worth of weights + activations" instead of "the entire model's weights." The tradeoff is speed — disk I/O is much slower than RAM, so expect lower tokens/sec than a fully-in-RAM setup, especially on HDDs. An NVMe SSD is strongly recommended.
+
+## How it works
+
+`digvijay_llm` ships two backends:
+
+| Backend | File format | Mechanism | Best for |
+|---|---|---|---|
+| `GGUFStreamingEngine` | `.gguf` (quantized) | Wraps `llama.cpp` via `llama-cpp-python`, using `mmap()` so the OS pages weights in/out of RAM on demand | Production use — fastest, most memory-efficient, supports 4-bit/5-bit quantization |
+| `SafetensorsLayerStreamer` | raw HuggingFace `.safetensors` checkpoints | Builds the model with empty weights, then uses `accelerate`'s disk-offload hooks to load **one transformer layer at a time** from disk right before it's needed | When you only have an unconverted HF checkpoint and don't want to quantize/convert to GGUF first |
+
+Both are wrapped behind one simple class, `LowRAMLLM`, so your code doesn't need to care which backend is active.
+
+## Install
+
+```bash
+pip install -r requirements.txt
+# or, pick only what you need:
+pip install digvijay_llm[gguf]   # GGUF backend only (lighter install)
+pip install digvijay_llm[hf]     # safetensors backend only
+pip install digvijay_llm[all]    # both
+```
+
+## Quick start
+
+### Option A — GGUF model (recommended, fastest)
+
+```python
+from digvijay_llm import LowRAMLLM
+
+llm = LowRAMLLM.from_gguf("models/llama-3-70b.Q4_K_M.gguf", n_ram_gb=16)
+print(llm.generate("Explain quantum entanglement simply."))
+
+# streaming token-by-token
+for token in llm.stream("Write a short poem about the sea."):
+    print(token, end="", flush=True)
+
+# chat interface
+reply = llm.chat([{"role": "user", "content": "Hello!"}])
+print(reply)
+```
+
+Get a `.gguf` file either by quantizing a model yourself with `llama.cpp`'s `convert_hf_to_gguf.py` + `llama-quantize`, or by downloading an already-quantized `.gguf` from a model hub.
+
+### Option B — raw HuggingFace safetensors checkpoint (no conversion needed)
+
+```python
+from digvijay_llm import LowRAMLLM
+
+llm = LowRAMLLM.from_safetensors("models/llama-3-70b-hf/", n_ram_gb=16)
+print(llm.generate("Write a haiku about the ocean."))
+```
+
+### Option C — auto-detect
+
+```python
+from digvijay_llm import LowRAMLLM
+
+llm = LowRAMLLM.auto("models/llama-3-70b.Q4_K_M.gguf", n_ram_gb=16)
+# or
+llm = LowRAMLLM.auto("models/llama-3-70b-hf/", n_ram_gb=16)
+```
+
+## Planning your RAM budget
+
+```python
+from digvijay_llm import plan_for_budget
+
+plan = plan_for_budget(
+    model_path="models/llama-3-70b-hf/",
+    total_params_billion=70,
+    n_layers=80,
+    bytes_per_param=2.0,  # fp16
+    n_ram_gb=16,
+)
+print(plan)
+for w in plan.warnings:
+    print("WARNING:", w)
+```
+
+This estimates how many transformer layers can safely stay resident in RAM at once given your budget, and warns you if your disk doesn't have enough free space for the model or if your RAM is so tight that throughput will suffer badly.
+
+## Practical tips for 70B-on-16GB
+
+1. **Use a quantized GGUF (Q4_K_M or smaller)** — this is by far the biggest lever. A 70B model at Q4 is roughly ~38–40 GB on disk vs ~140 GB at fp16; mmap streaming then only needs to page small chunks at a time.
+2. **Use an NVMe SSD**, not a spinning HDD or network drive — random-access read speed directly determines your tokens/sec.
+3. **Keep context length modest** (the KV cache also eats RAM) — `digvijay_llm` auto-shrinks context size on tight budgets, but you can override it.
+4. **Close other RAM-heavy apps** while running — the OS needs free RAM to use as page cache for the mmap'd weights.
+5. **`n_gpu_layers`** — if you have even a small GPU, offloading a few layers to VRAM (`n_gpu_layers=N`) reduces disk pressure further.
+
+## Project layout
+
+```
+digvijay_llm/
+├── __init__.py            # public API surface
+├── api.py                 # LowRAMLLM unified class
+├── engine_gguf.py          # llama.cpp mmap-based backend
+├── engine_safetensors.py   # accelerate disk-offload layer-streaming backend
+└── ram_planner.py          # RAM budget heuristics
+examples/
+├── run_gguf_example.py
+└── run_safetensors_example.py
+tests/
+└── test_ram_planner.py
+```
+
+## License
+
+MIT

digvijay_llm-0.1.0/README.md

@@ -0,0 +1,115 @@
+# digvijay_llm
+
+Run big open-source LLMs — including 70B-class models — on machines with as little as **16 GB of RAM**, by streaming weights from disk ("ROM") instead of loading the full model into RAM.
+
+## ⚠️ Read this first (important honesty note)
+
+A CPU/GPU can only *compute* on data sitting in RAM/VRAM — it is physically impossible to run matrix multiplications directly on a disk. What this library actually does, and what makes low-RAM 70B inference possible, is **disk-backed weight streaming**: weights live on disk and are pulled into RAM only for the layer currently being computed, then released. Peak RAM usage becomes roughly "one layer's worth of weights + activations" instead of "the entire model's weights." The tradeoff is speed — disk I/O is much slower than RAM, so expect lower tokens/sec than a fully-in-RAM setup, especially on HDDs. An NVMe SSD is strongly recommended.
+
+## How it works
+
+`digvijay_llm` ships two backends:
+
+| Backend | File format | Mechanism | Best for |
+|---|---|---|---|
+| `GGUFStreamingEngine` | `.gguf` (quantized) | Wraps `llama.cpp` via `llama-cpp-python`, using `mmap()` so the OS pages weights in/out of RAM on demand | Production use — fastest, most memory-efficient, supports 4-bit/5-bit quantization |
+| `SafetensorsLayerStreamer` | raw HuggingFace `.safetensors` checkpoints | Builds the model with empty weights, then uses `accelerate`'s disk-offload hooks to load **one transformer layer at a time** from disk right before it's needed | When you only have an unconverted HF checkpoint and don't want to quantize/convert to GGUF first |
+
+Both are wrapped behind one simple class, `LowRAMLLM`, so your code doesn't need to care which backend is active.
+
+## Install
+
+```bash
+pip install -r requirements.txt
+# or, pick only what you need:
+pip install digvijay_llm[gguf]   # GGUF backend only (lighter install)
+pip install digvijay_llm[hf]     # safetensors backend only
+pip install digvijay_llm[all]    # both
+```
+
+## Quick start
+
+### Option A — GGUF model (recommended, fastest)
+
+```python
+from digvijay_llm import LowRAMLLM
+
+llm = LowRAMLLM.from_gguf("models/llama-3-70b.Q4_K_M.gguf", n_ram_gb=16)
+print(llm.generate("Explain quantum entanglement simply."))
+
+# streaming token-by-token
+for token in llm.stream("Write a short poem about the sea."):
+    print(token, end="", flush=True)
+
+# chat interface
+reply = llm.chat([{"role": "user", "content": "Hello!"}])
+print(reply)
+```
+
+Get a `.gguf` file either by quantizing a model yourself with `llama.cpp`'s `convert_hf_to_gguf.py` + `llama-quantize`, or by downloading an already-quantized `.gguf` from a model hub.
+
+### Option B — raw HuggingFace safetensors checkpoint (no conversion needed)
+
+```python
+from digvijay_llm import LowRAMLLM
+
+llm = LowRAMLLM.from_safetensors("models/llama-3-70b-hf/", n_ram_gb=16)
+print(llm.generate("Write a haiku about the ocean."))
+```
+
+### Option C — auto-detect
+
+```python
+from digvijay_llm import LowRAMLLM
+
+llm = LowRAMLLM.auto("models/llama-3-70b.Q4_K_M.gguf", n_ram_gb=16)
+# or
+llm = LowRAMLLM.auto("models/llama-3-70b-hf/", n_ram_gb=16)
+```
+
+## Planning your RAM budget
+
+```python
+from digvijay_llm import plan_for_budget
+
+plan = plan_for_budget(
+    model_path="models/llama-3-70b-hf/",
+    total_params_billion=70,
+    n_layers=80,
+    bytes_per_param=2.0,  # fp16
+    n_ram_gb=16,
+)
+print(plan)
+for w in plan.warnings:
+    print("WARNING:", w)
+```
+
+This estimates how many transformer layers can safely stay resident in RAM at once given your budget, and warns you if your disk doesn't have enough free space for the model or if your RAM is so tight that throughput will suffer badly.
+
+## Practical tips for 70B-on-16GB
+
+1. **Use a quantized GGUF (Q4_K_M or smaller)** — this is by far the biggest lever. A 70B model at Q4 is roughly ~38–40 GB on disk vs ~140 GB at fp16; mmap streaming then only needs to page small chunks at a time.
+2. **Use an NVMe SSD**, not a spinning HDD or network drive — random-access read speed directly determines your tokens/sec.
+3. **Keep context length modest** (the KV cache also eats RAM) — `digvijay_llm` auto-shrinks context size on tight budgets, but you can override it.
+4. **Close other RAM-heavy apps** while running — the OS needs free RAM to use as page cache for the mmap'd weights.
+5. **`n_gpu_layers`** — if you have even a small GPU, offloading a few layers to VRAM (`n_gpu_layers=N`) reduces disk pressure further.
+
+## Project layout
+
+```
+digvijay_llm/
+├── __init__.py            # public API surface
+├── api.py                 # LowRAMLLM unified class
+├── engine_gguf.py          # llama.cpp mmap-based backend
+├── engine_safetensors.py   # accelerate disk-offload layer-streaming backend
+└── ram_planner.py          # RAM budget heuristics
+examples/
+├── run_gguf_example.py
+└── run_safetensors_example.py
+tests/
+└── test_ram_planner.py
+```
+
+## License
+
+MIT

digvijay_llm-0.1.0/digvijay_llm/__init__.py

@@ -0,0 +1,62 @@
+"""
+digvijay_llm
+============
+
+Run large open-source LLMs (including 70B-class models) on machines with
+very limited RAM (e.g. 16 GB) by NEVER loading the full weight set into
+memory at once. Instead, weights stay on disk (your "ROM" — SSD/HDD) and
+are streamed in just-in-time, layer by layer, then released.
+
+IMPORTANT HONEST NOTE (please read):
+-------------------------------------
+It is physically impossible to *compute* directly on disk/ROM — the CPU/GPU
+can only execute operations on data that is in RAM/VRAM. What IS possible,
+and what this library does, is keep the bulk of the model's weights on disk
+and only pull small pieces (one layer / one tensor at a time) into RAM right
+before they're needed, then drop them again. This keeps peak RAM usage close
+to "one layer's worth of weights" instead of "the whole model's worth",
+which is what makes running a 70B model on a 16GB machine feasible (at the
+cost of disk I/O speed -> slower tokens/sec, especially on HDD; an NVMe SSD
+is strongly recommended).
+
+Two backends are provided:
+
+1. GGUFStreamingEngine
+   - Wraps llama.cpp (via llama-cpp-python) which already implements
+     production-grade memory-mapped (mmap) weight streaming + quantization.
+   - Recommended for real, fast usage. Requires a .gguf quantized model file.
+
+2. SafetensorsLayerStreamer
+   - A pure PyTorch engine written from scratch in this library that loads
+     a HuggingFace-format (.safetensors) checkpoint ONE TRANSFORMER LAYER
+     AT A TIME directly from disk, runs the forward pass for that layer,
+     then frees it before loading the next layer.
+   - Useful when you only have raw HF weights and don't want to convert to
+     GGUF, or want full Python-level control over the streaming behavior.
+
+Quick start
+-----------
+    from digvijay_llm import LowRAMLLM
+
+    llm = LowRAMLLM.from_gguf("models/llama-70b.Q4_K_M.gguf", n_ram_gb=16)
+    print(llm.generate("Explain quantum entanglement simply."))
+
+    # or, for raw HF safetensors checkpoints:
+    llm = LowRAMLLM.from_safetensors("models/llama-70b-hf/")
+    print(llm.generate("Write a haiku about the ocean."))
+"""
+
+from .engine_gguf import GGUFStreamingEngine
+from .engine_safetensors import SafetensorsLayerStreamer
+from .api import LowRAMLLM
+from .ram_planner import RAMPlanner, plan_for_budget
+
+__all__ = [
+    "LowRAMLLM",
+    "GGUFStreamingEngine",
+    "SafetensorsLayerStreamer",
+    "RAMPlanner",
+    "plan_for_budget",
+]
+
+__version__ = "0.1.0"

digvijay_llm-0.1.0/digvijay_llm/api.py

@@ -0,0 +1,101 @@
+"""
+api.py
+======
+The single class most users will actually import: `LowRAMLLM`.
+Picks the right backend (GGUF mmap streaming vs safetensors layer
+streaming) based on what's passed in, and exposes one consistent
+.generate() / .stream() / .chat() interface regardless of backend.
+"""
+
+import os
+from typing import Optional, Iterator, List, Dict
+
+
+class LowRAMLLM:
+    """
+    Unified entrypoint for digvijay_llm.
+
+    Use the classmethods to construct it:
+        LowRAMLLM.from_gguf(path, n_ram_gb=16)
+        LowRAMLLM.from_safetensors(dir_path, n_ram_gb=16)
+        LowRAMLLM.auto(path_or_dir, n_ram_gb=16)   # detects which backend to use
+    """
+
+    def __init__(self, engine, backend: str):
+        self._engine = engine
+        self.backend = backend
+
+    # ---------- constructors ----------
+
+    @classmethod
+    def from_gguf(
+        cls,
+        model_path: str,
+        n_ram_gb: float = 16.0,
+        n_gpu_layers: int = 0,
+        n_threads: Optional[int] = None,
+        verbose: bool = False,
+    ) -> "LowRAMLLM":
+        from .engine_gguf import GGUFStreamingEngine
+
+        engine = GGUFStreamingEngine(
+            model_path=model_path,
+            n_ram_gb=n_ram_gb,
+            n_gpu_layers=n_gpu_layers,
+            n_threads=n_threads,
+            verbose=verbose,
+        )
+        return cls(engine, backend="gguf-mmap")
+
+    @classmethod
+    def from_safetensors(
+        cls,
+        model_dir: str,
+        offload_dir: Optional[str] = None,
+        n_ram_gb: float = 16.0,
+        dtype: str = "float16",
+    ) -> "LowRAMLLM":
+        from .engine_safetensors import SafetensorsLayerStreamer
+
+        engine = SafetensorsLayerStreamer(
+            model_dir=model_dir,
+            offload_dir=offload_dir,
+            n_ram_gb=n_ram_gb,
+            dtype=dtype,
+        )
+        return cls(engine, backend="safetensors-layer-stream")
+
+    @classmethod
+    def auto(cls, path: str, n_ram_gb: float = 16.0, **kwargs) -> "LowRAMLLM":
+        """Detects whether `path` is a .gguf file or an HF checkpoint dir."""
+        if os.path.isfile(path) and path.endswith(".gguf"):
+            return cls.from_gguf(path, n_ram_gb=n_ram_gb, **kwargs)
+        if os.path.isdir(path):
+            has_st = any(f.endswith(".safetensors") for f in os.listdir(path))
+            if has_st:
+                return cls.from_safetensors(path, n_ram_gb=n_ram_gb, **kwargs)
+        raise ValueError(
+            f"Could not detect model type at '{path}'. Expected a .gguf file "
+            "or a directory containing .safetensors shards."
+        )
+
+    # ---------- inference ----------
+
+    def generate(self, prompt: str, max_tokens: int = 256, temperature: float = 0.7, **kwargs) -> str:
+        return self._engine.generate(prompt, max_tokens=max_tokens, temperature=temperature, **kwargs)
+
+    def stream(self, prompt: str, max_tokens: int = 256, temperature: float = 0.7) -> Iterator[str]:
+        if self.backend == "gguf-mmap":
+            yield from self._engine.generate(prompt, max_tokens=max_tokens, temperature=temperature, stream=True)
+        else:
+            yield from self._engine.stream_generate(prompt, max_tokens=max_tokens, temperature=temperature)
+
+    def chat(self, messages: List[Dict[str, str]], max_tokens: int = 256, temperature: float = 0.7) -> str:
+        if self.backend == "gguf-mmap":
+            return self._engine.chat(messages, max_tokens=max_tokens, temperature=temperature)
+        # Fallback for safetensors backend: flatten chat messages into a prompt
+        prompt = "\n".join(f"{m['role']}: {m['content']}" for m in messages) + "\nassistant:"
+        return self._engine.generate(prompt, max_tokens=max_tokens, temperature=temperature)
+
+    def __repr__(self):
+        return f"<LowRAMLLM backend={self.backend!r}>"

digvijay_llm-0.1.0/digvijay_llm/engine_gguf.py

@@ -0,0 +1,125 @@
+"""
+engine_gguf.py
+==============
+Production-grade low-RAM backend. Wraps llama-cpp-python, which loads
+GGUF files via mmap() — the OS keeps weights on disk and pages them into
+RAM lazily as the model touches them, evicting cold pages under memory
+pressure. Combined with quantization (Q4_K_M, Q5_K_M, etc.) this is what
+lets a 70B model run on a 16GB machine.
+
+This module just gives that capability a clean, documented entrypoint
+inside digvijay_llm, with sane low-RAM defaults (no mlock, limited
+context, mmap forced on, conservative thread count).
+"""
+
+import os
+from typing import Optional, Iterator
+
+
+class GGUFStreamingEngine:
+    """
+    Streams a quantized .gguf model off disk instead of loading it fully
+    into RAM, via memory-mapping (mmap).
+
+    Parameters
+    ----------
+    model_path : str
+        Path to a .gguf quantized model file (e.g. produced by
+        llama.cpp's convert + quantize scripts, or downloaded pre-quantized
+        from a hub like Hugging Face / TheBloke-style repos).
+    n_ram_gb : float
+        Approximate RAM budget; used only to pick a sane default context
+        size / batch size so you don't accidentally request a KV cache
+        bigger than your machine can hold.
+    n_gpu_layers : int
+        Number of layers to offload to GPU if you have one (0 = pure CPU+
+        disk streaming, which is the "low RAM" use case this library is
+        built for).
+    n_threads : Optional[int]
+        CPU threads to use. Defaults to os.cpu_count().
+    """
+
+    def __init__(
+        self,
+        model_path: str,
+        n_ram_gb: float = 16.0,
+        n_gpu_layers: int = 0,
+        n_threads: Optional[int] = None,
+        verbose: bool = False,
+    ):
+        if not os.path.exists(model_path):
+            raise FileNotFoundError(f"GGUF model not found at: {model_path}")
+
+        try:
+            from llama_cpp import Llama
+        except ImportError as e:
+            raise ImportError(
+                "engine_gguf requires the 'llama-cpp-python' package. "
+                "Install it with: pip install llama-cpp-python"
+            ) from e
+
+        self.model_path = model_path
+        self.n_ram_gb = n_ram_gb
+
+        # Conservative context size: bigger context = bigger KV cache = more
+        # RAM. We scale it down automatically on tight budgets.
+        if n_ram_gb <= 8:
+            ctx = 1024
+        elif n_ram_gb <= 16:
+            ctx = 2048
+        elif n_ram_gb <= 32:
+            ctx = 4096
+        else:
+            ctx = 8192
+
+        self._llm = Llama(
+            model_path=model_path,
+            n_ctx=ctx,
+            n_threads=n_threads or os.cpu_count(),
+            n_gpu_layers=n_gpu_layers,
+            use_mmap=True,   # critical: keeps weights disk-backed ("ROM"), not fully copied into RAM
+            use_mlock=False, # critical: do NOT force-pin pages into RAM, let the OS evict freely
+            verbose=verbose,
+        )
+        self._ctx_size = ctx
+
+    def generate(
+        self,
+        prompt: str,
+        max_tokens: int = 256,
+        temperature: float = 0.7,
+        top_p: float = 0.9,
+        stream: bool = False,
+    ):
+        if stream:
+            return self._stream(prompt, max_tokens, temperature, top_p)
+
+        out = self._llm(
+            prompt,
+            max_tokens=max_tokens,
+            temperature=temperature,
+            top_p=top_p,
+        )
+        return out["choices"][0]["text"]
+
+    def _stream(self, prompt, max_tokens, temperature, top_p) -> Iterator[str]:
+        for chunk in self._llm(
+            prompt,
+            max_tokens=max_tokens,
+            temperature=temperature,
+            top_p=top_p,
+            stream=True,
+        ):
+            yield chunk["choices"][0]["text"]
+
+    def chat(self, messages, max_tokens: int = 256, temperature: float = 0.7):
+        """OpenAI-style chat interface, e.g.
+        messages=[{"role": "user", "content": "hi"}]"""
+        out = self._llm.create_chat_completion(
+            messages=messages, max_tokens=max_tokens, temperature=temperature
+        )
+        return out["choices"][0]["message"]["content"]
+
+    @property
+    def context_size(self) -> int:
+        return self._ctx_size

digvijay_llm-0.1.0/digvijay_llm/engine_safetensors.py

@@ -0,0 +1,175 @@
+"""
+engine_safetensors.py
+======================
+For raw HuggingFace-format checkpoints (.safetensors, no GGUF conversion
+done). This engine loads the model with EMPTY weights first, then attaches
+disk-offload hooks (built on top of Hugging Face `accelerate`) so that:
+
+  - Only ONE transformer layer's weights are materialized in RAM at a time.
+  - Right before a layer's forward() runs, its weights are read from the
+    .safetensors shard on disk straight into RAM.
+  - Immediately after that layer's forward() finishes, its weights are
+    freed from RAM again.
+
+This is the "layer-by-layer disk streaming" approach — RAM peak ~= size of
+the largest single layer + activations, instead of the whole model. It is
+slower than keeping everything in RAM (disk I/O per layer per token) but it
+is what makes a 70B model loadable on a 16 GB machine without GGUF
+conversion.
+
+Under the hood this uses `accelerate`'s `disk_offload` / `dispatch_model`
+machinery (the same machinery HF itself recommends for low-RAM inference),
+wrapped here with sane defaults + the digvijay_llm.generate() interface.
+"""
+
+import os
+import gc
+from typing import Optional, Iterator
+
+
+class SafetensorsLayerStreamer:
+    """
+    Parameters
+    ----------
+    model_dir : str
+        Path to a local directory containing a HuggingFace checkpoint:
+        config.json + one or more *.safetensors weight shards + tokenizer
+        files. (Use `huggingface-cli download <repo> --local-dir ...` to
+        fetch one without loading it into RAM first.)
+    offload_dir : str
+        Scratch directory on disk used as the staging area accelerate
+        streams weights through. Needs free space roughly equal to the
+        model size. Defaults to "<model_dir>/.digvijay_offload".
+    n_ram_gb : float
+        Approximate RAM budget, used to decide max_memory passed to
+        accelerate so it knows how aggressively to offload to disk.
+    dtype : str
+        "float16" (default, halves RAM vs float32) or "float32".
+    """
+
+    def __init__(
+        self,
+        model_dir: str,
+        offload_dir: Optional[str] = None,
+        n_ram_gb: float = 16.0,
+        dtype: str = "float16",
+        device: str = "cpu",
+    ):
+        if not os.path.isdir(model_dir):
+            raise FileNotFoundError(f"model_dir not found: {model_dir}")
+
+        try:
+            import torch
+            from transformers import AutoModelForCausalLM, AutoTokenizer
+            from accelerate import infer_auto_device_map, dispatch_model, init_empty_weights
+            from accelerate.utils import get_balanced_memory
+        except ImportError as e:
+            raise ImportError(
+                "engine_safetensors requires 'torch', 'transformers', and "
+                "'accelerate'. Install with: "
+                "pip install torch transformers accelerate safetensors"
+            ) from e
+
+        self._torch = torch
+        self.model_dir = model_dir
+        self.offload_dir = offload_dir or os.path.join(model_dir, ".digvijay_offload")
+        os.makedirs(self.offload_dir, exist_ok=True)
+
+        torch_dtype = torch.float16 if dtype == "float16" else torch.float32
+
+        # 1. Build the model architecture with NO weights materialized yet
+        #    (this costs ~0 RAM regardless of model size).
+        from transformers import AutoConfig
+        config = AutoConfig.from_pretrained(model_dir)
+        with init_empty_weights():
+            empty_model = AutoModelForCausalLM.from_config(config, torch_dtype=torch_dtype)
+
+        # 2. Reserve almost all RAM for ONE layer at a time by capping the
+        #    "max_memory" accelerate is allowed to use on this device; the
+        #    rest is automatically offloaded to `offload_dir` on disk and
+        #    streamed back in layer-by-layer during forward().
+        reserved_for_layer_gb = max(1.0, n_ram_gb * 0.5)  # heuristic safety margin
+        max_memory = {device: f"{reserved_for_layer_gb}GiB", "cpu": f"{reserved_for_layer_gb}GiB"}
+
+        device_map = infer_auto_device_map(
+            empty_model,
+            max_memory=max_memory,
+            no_split_module_classes=self._guess_no_split_classes(empty_model),
+            dtype=torch_dtype,
+        )
+
+        # 3. Load real weights directly from the safetensors shards on disk
+        #    INTO the device_map plan above — anything that doesn't fit the
+        #    RAM budget gets a disk-offload hook instead of being loaded.
+        from accelerate import load_checkpoint_and_dispatch
+
+        self._model = load_checkpoint_and_dispatch(
+            empty_model,
+            checkpoint=model_dir,
+            device_map=device_map,
+            offload_folder=self.offload_dir,
+            offload_state_dict=True,   # streams shards in piece by piece while loading too
+            dtype=torch_dtype,
+        )
+        self._model.eval()
+
+        self._tokenizer = AutoTokenizer.from_pretrained(model_dir)
+        self.device_map = device_map
+
+    @staticmethod
+    def _guess_no_split_classes(model):
+        """Find the repeated transformer-block class so accelerate never
+        splits a single decoder layer's weights across two offload
+        locations (which would break the "one layer in RAM at a time"
+        guarantee)."""
+        names = set()
+        for module in model.modules():
+            cls_name = module.__class__.__name__
+            if "Layer" in cls_name or "Block" in cls_name:
+                names.add(cls_name)
+        return list(names) if names else None
+
+    def generate(
+        self,
+        prompt: str,
+        max_tokens: int = 256,
+        temperature: float = 0.7,
+        top_p: float = 0.9,
+    ) -> str:
+        torch = self._torch
+        inputs = self._tokenizer(prompt, return_tensors="pt")
+        with torch.no_grad():
+            output_ids = self._model.generate(
+                **inputs,
+                max_new_tokens=max_tokens,
+                do_sample=temperature > 0,
+                temperature=max(temperature, 1e-5),
+                top_p=top_p,
+                pad_token_id=self._tokenizer.eos_token_id,
+            )
+        text = self._tokenizer.decode(output_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
+        gc.collect()
+        return text
+
+    def stream_generate(self, prompt: str, max_tokens: int = 256, temperature: float = 0.7) -> Iterator[str]:
+        """Token-by-token streaming using transformers' TextIteratorStreamer."""
+        from transformers import TextIteratorStreamer
+        import threading
+
+        torch = self._torch
+        inputs = self._tokenizer(prompt, return_tensors="pt")
+        streamer = TextIteratorStreamer(self._tokenizer, skip_prompt=True, skip_special_tokens=True)
+
+        gen_kwargs = dict(
+            **inputs,
+            max_new_tokens=max_tokens,
+            do_sample=temperature > 0,
+            temperature=max(temperature, 1e-5),
+            streamer=streamer,
+            pad_token_id=self._tokenizer.eos_token_id,
+        )
+        thread = threading.Thread(target=self._model.generate, kwargs=gen_kwargs)
+        thread.start()
+        for new_text in streamer:
+            yield new_text
+        thread.join()

digvijay_llm-0.1.0/digvijay_llm/ram_planner.py

@@ -0,0 +1,105 @@
+"""
+ram_planner.py
+==============
+Helps decide HOW to stream a model given a target RAM budget: how many
+layers (if any) can be safely kept "hot" in RAM at once, how big the
+on-disk page cache window should be, and whether the user's requested
+model will even fit on disk + meet a minimum tokens/sec expectation.
+
+This is intentionally simple/heuristic — its job is to stop people from
+footgunning themselves (e.g. trying to keep 40 layers "hot" on a 16GB
+machine) rather than to be a perfect simulator.
+"""
+
+from dataclasses import dataclass
+import os
+import shutil
+
+
+@dataclass
+class RAMPlan:
+    total_ram_gb: float
+    reserved_for_os_gb: float
+    usable_ram_gb: float
+    safe_hot_layers: int
+    layer_size_gb: float
+    disk_free_gb: float
+    warnings: list
+
+
+def _disk_free_gb(path: str) -> float:
+    try:
+        usage = shutil.disk_usage(path if os.path.isdir(path) else os.path.dirname(path) or ".")
+        return usage.free / (1024 ** 3)
+    except Exception:
+        return float("inf")
+
+
+def plan_for_budget(
+    model_path: str,
+    total_params_billion: float,
+    n_layers: int,
+    bytes_per_param: float = 2.0,  # fp16=2, 8bit=1, 4bit~=0.5-0.6
+    n_ram_gb: float = 16.0,
+    os_reserve_gb: float = 2.5,
+) -> RAMPlan:
+    """
+    Compute a conservative streaming plan: how many transformer layers can
+    be kept resident in RAM simultaneously without exceeding the budget.
+    """
+    warnings = []
+    usable = max(n_ram_gb - os_reserve_gb, 1.0)
+
+    total_weight_gb = (total_params_billion * 1e9 * bytes_per_param) / (1024 ** 3)
+    layer_size_gb = total_weight_gb / max(n_layers, 1)
+
+    # Keep a safety margin for activations / KV cache / runtime overhead
+    activation_overhead_gb = max(0.5, usable * 0.15)
+    budget_for_weights = max(usable - activation_overhead_gb, 0.5)
+
+    safe_hot_layers = max(1, int(budget_for_weights // layer_size_gb))
+    if safe_hot_layers < 2:
+        warnings.append(
+            "Budget is extremely tight: only 1 layer fits in RAM at a time. "
+            "Expect slow throughput; an NVMe SSD is strongly recommended."
+        )
+
+    disk_free = _disk_free_gb(model_path)
+    if disk_free < total_weight_gb:
+        warnings.append(
+            f"Model needs ~{total_weight_gb:.1f} GB on disk but only "
+            f"{disk_free:.1f} GB free was detected at '{model_path}'."
+        )
+
+    return RAMPlan(
+        total_ram_gb=n_ram_gb,
+        reserved_for_os_gb=os_reserve_gb,
+        usable_ram_gb=usable,
+        safe_hot_layers=safe_hot_layers,
+        layer_size_gb=layer_size_gb,
+        disk_free_gb=disk_free,
+        warnings=warnings,
+    )
+
+
+class RAMPlanner:
+    """Thin OO wrapper kept for ergonomic imports: RAMPlanner(...).plan()"""
+
+    def __init__(self, model_path: str, total_params_billion: float, n_layers: int,
+                 bytes_per_param: float = 2.0, n_ram_gb: float = 16.0, os_reserve_gb: float = 2.5):
+        self.model_path = model_path
+        self.total_params_billion = total_params_billion
+        self.n_layers = n_layers
+        self.bytes_per_param = bytes_per_param
+        self.n_ram_gb = n_ram_gb
+        self.os_reserve_gb = os_reserve_gb
+
+    def plan(self) -> RAMPlan:
+        return plan_for_budget(
+            self.model_path,
+            self.total_params_billion,
+            self.n_layers,
+            self.bytes_per_param,
+            self.n_ram_gb,
+            self.os_reserve_gb,
+        )

digvijay_llm-0.1.0/digvijay_llm.egg-info/PKG-INFO

@@ -0,0 +1,163 @@
+Metadata-Version: 2.4
+Name: digvijay_llm
+Version: 0.1.0
+Summary: Run 70B LLMs on low-RAM machines by streaming weights from disk
+Author: Digvijay
+License: MIT License
+        
+        Copyright (c) 2026 Digvijay
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: psutil>=5.9.0
+Provides-Extra: gguf
+Requires-Dist: llama-cpp-python>=0.2.0; extra == "gguf"
+Provides-Extra: hf
+Requires-Dist: torch>=2.0.0; extra == "hf"
+Requires-Dist: transformers>=4.38.0; extra == "hf"
+Requires-Dist: accelerate>=0.27.0; extra == "hf"
+Requires-Dist: safetensors>=0.4.0; extra == "hf"
+Provides-Extra: all
+Requires-Dist: llama-cpp-python>=0.2.0; extra == "all"
+Requires-Dist: torch>=2.0.0; extra == "all"
+Requires-Dist: transformers>=4.38.0; extra == "all"
+Requires-Dist: accelerate>=0.27.0; extra == "all"
+Requires-Dist: safetensors>=0.4.0; extra == "all"
+Dynamic: author
+Dynamic: license-file
+Dynamic: requires-python
+
+# digvijay_llm
+
+Run big open-source LLMs — including 70B-class models — on machines with as little as **16 GB of RAM**, by streaming weights from disk ("ROM") instead of loading the full model into RAM.
+
+## ⚠️ Read this first (important honesty note)
+
+A CPU/GPU can only *compute* on data sitting in RAM/VRAM — it is physically impossible to run matrix multiplications directly on a disk. What this library actually does, and what makes low-RAM 70B inference possible, is **disk-backed weight streaming**: weights live on disk and are pulled into RAM only for the layer currently being computed, then released. Peak RAM usage becomes roughly "one layer's worth of weights + activations" instead of "the entire model's weights." The tradeoff is speed — disk I/O is much slower than RAM, so expect lower tokens/sec than a fully-in-RAM setup, especially on HDDs. An NVMe SSD is strongly recommended.
+
+## How it works
+
+`digvijay_llm` ships two backends:
+
+| Backend | File format | Mechanism | Best for |
+|---|---|---|---|
+| `GGUFStreamingEngine` | `.gguf` (quantized) | Wraps `llama.cpp` via `llama-cpp-python`, using `mmap()` so the OS pages weights in/out of RAM on demand | Production use — fastest, most memory-efficient, supports 4-bit/5-bit quantization |
+| `SafetensorsLayerStreamer` | raw HuggingFace `.safetensors` checkpoints | Builds the model with empty weights, then uses `accelerate`'s disk-offload hooks to load **one transformer layer at a time** from disk right before it's needed | When you only have an unconverted HF checkpoint and don't want to quantize/convert to GGUF first |
+
+Both are wrapped behind one simple class, `LowRAMLLM`, so your code doesn't need to care which backend is active.
+
+## Install
+
+```bash
+pip install -r requirements.txt
+# or, pick only what you need:
+pip install digvijay_llm[gguf]   # GGUF backend only (lighter install)
+pip install digvijay_llm[hf]     # safetensors backend only
+pip install digvijay_llm[all]    # both
+```
+
+## Quick start
+
+### Option A — GGUF model (recommended, fastest)
+
+```python
+from digvijay_llm import LowRAMLLM
+
+llm = LowRAMLLM.from_gguf("models/llama-3-70b.Q4_K_M.gguf", n_ram_gb=16)
+print(llm.generate("Explain quantum entanglement simply."))
+
+# streaming token-by-token
+for token in llm.stream("Write a short poem about the sea."):
+    print(token, end="", flush=True)
+
+# chat interface
+reply = llm.chat([{"role": "user", "content": "Hello!"}])
+print(reply)
+```
+
+Get a `.gguf` file either by quantizing a model yourself with `llama.cpp`'s `convert_hf_to_gguf.py` + `llama-quantize`, or by downloading an already-quantized `.gguf` from a model hub.
+
+### Option B — raw HuggingFace safetensors checkpoint (no conversion needed)
+
+```python
+from digvijay_llm import LowRAMLLM
+
+llm = LowRAMLLM.from_safetensors("models/llama-3-70b-hf/", n_ram_gb=16)
+print(llm.generate("Write a haiku about the ocean."))
+```
+
+### Option C — auto-detect
+
+```python
+from digvijay_llm import LowRAMLLM
+
+llm = LowRAMLLM.auto("models/llama-3-70b.Q4_K_M.gguf", n_ram_gb=16)
+# or
+llm = LowRAMLLM.auto("models/llama-3-70b-hf/", n_ram_gb=16)
+```
+
+## Planning your RAM budget
+
+```python
+from digvijay_llm import plan_for_budget
+
+plan = plan_for_budget(
+    model_path="models/llama-3-70b-hf/",
+    total_params_billion=70,
+    n_layers=80,
+    bytes_per_param=2.0,  # fp16
+    n_ram_gb=16,
+)
+print(plan)
+for w in plan.warnings:
+    print("WARNING:", w)
+```
+
+This estimates how many transformer layers can safely stay resident in RAM at once given your budget, and warns you if your disk doesn't have enough free space for the model or if your RAM is so tight that throughput will suffer badly.
+
+## Practical tips for 70B-on-16GB
+
+1. **Use a quantized GGUF (Q4_K_M or smaller)** — this is by far the biggest lever. A 70B model at Q4 is roughly ~38–40 GB on disk vs ~140 GB at fp16; mmap streaming then only needs to page small chunks at a time.
+2. **Use an NVMe SSD**, not a spinning HDD or network drive — random-access read speed directly determines your tokens/sec.
+3. **Keep context length modest** (the KV cache also eats RAM) — `digvijay_llm` auto-shrinks context size on tight budgets, but you can override it.
+4. **Close other RAM-heavy apps** while running — the OS needs free RAM to use as page cache for the mmap'd weights.
+5. **`n_gpu_layers`** — if you have even a small GPU, offloading a few layers to VRAM (`n_gpu_layers=N`) reduces disk pressure further.
+
+## Project layout
+
+```
+digvijay_llm/
+├── __init__.py            # public API surface
+├── api.py                 # LowRAMLLM unified class
+├── engine_gguf.py          # llama.cpp mmap-based backend
+├── engine_safetensors.py   # accelerate disk-offload layer-streaming backend
+└── ram_planner.py          # RAM budget heuristics
+examples/
+├── run_gguf_example.py
+└── run_safetensors_example.py
+tests/
+└── test_ram_planner.py
+```
+
+## License
+
+MIT

digvijay_llm-0.1.0/digvijay_llm.egg-info/SOURCES.txt

@@ -0,0 +1,15 @@
+LICENSE
+README.md
+pyproject.toml
+setup.py
+digvijay_llm/__init__.py
+digvijay_llm/api.py
+digvijay_llm/engine_gguf.py
+digvijay_llm/engine_safetensors.py
+digvijay_llm/ram_planner.py
+digvijay_llm.egg-info/PKG-INFO
+digvijay_llm.egg-info/SOURCES.txt
+digvijay_llm.egg-info/dependency_links.txt
+digvijay_llm.egg-info/requires.txt
+digvijay_llm.egg-info/top_level.txt
+tests/test_ram_planner.py

digvijay_llm-0.1.0/digvijay_llm.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

digvijay_llm-0.1.0/digvijay_llm.egg-info/requires.txt

@@ -0,0 +1,17 @@
+psutil>=5.9.0
+
+[all]
+llama-cpp-python>=0.2.0
+torch>=2.0.0
+transformers>=4.38.0
+accelerate>=0.27.0
+safetensors>=0.4.0
+
+[gguf]
+llama-cpp-python>=0.2.0
+
+[hf]
+torch>=2.0.0
+transformers>=4.38.0
+accelerate>=0.27.0
+safetensors>=0.4.0

digvijay_llm-0.1.0/digvijay_llm.egg-info/top_level.txt

@@ -0,0 +1 @@
+digvijay_llm

digvijay_llm-0.1.0/pyproject.toml

@@ -0,0 +1,35 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "digvijay_llm"
+version = "0.1.0"
+description = "Run 70B LLMs on low-RAM machines by streaming weights from disk"
+readme = "README.md"
+license = { file = "LICENSE" }
+requires-python = ">=3.9"
+
+dependencies = [
+    "psutil>=5.9.0",
+]
+
+[project.optional-dependencies]
+gguf = [
+    "llama-cpp-python>=0.2.0",
+]
+
+hf = [
+    "torch>=2.0.0",
+    "transformers>=4.38.0",
+    "accelerate>=0.27.0",
+    "safetensors>=0.4.0",
+]
+
+all = [
+    "llama-cpp-python>=0.2.0",
+    "torch>=2.0.0",
+    "transformers>=4.38.0",
+    "accelerate>=0.27.0",
+    "safetensors>=0.4.0",
+]

digvijay_llm-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

digvijay_llm-0.1.0/setup.py

@@ -0,0 +1,33 @@
+from setuptools import setup, find_packages
+
+setup(
+    name="digvijay_llm",
+    version="0.1.0",
+    description="Run large LLMs (incl. 70B-class) on low-RAM machines by streaming weights from disk instead of holding the full model in RAM.",
+    long_description=open("README.md", encoding="utf-8").read() if __import__("os").path.exists("README.md") else "",
+    long_description_content_type="text/markdown",
+    author="Digvijay",
+    license="MIT",
+    packages=find_packages(),
+    python_requires=">=3.9",
+    install_requires=[
+        "psutil>=5.9.0",
+    ],
+    extras_require={
+        "gguf": ["llama-cpp-python>=0.2.0"],
+        "hf": ["torch>=2.0.0", "transformers>=4.38.0", "accelerate>=0.27.0", "safetensors>=0.4.0"],
+        "all": [
+            "llama-cpp-python>=0.2.0",
+            "torch>=2.0.0",
+            "transformers>=4.38.0",
+            "accelerate>=0.27.0",
+            "safetensors>=0.4.0",
+        ],
+    },
+    classifiers=[
+        "Programming Language :: Python :: 3",
+        "License :: OSI Approved :: MIT License",
+        "Operating System :: OS Independent",
+        "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    ],
+)

digvijay_llm-0.1.0/tests/test_ram_planner.py

@@ -0,0 +1,50 @@
+import os
+import sys
+import unittest
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+
+from digvijay_llm.ram_planner import plan_for_budget, RAMPlanner
+
+
+class TestRAMPlanner(unittest.TestCase):
+    def test_70b_on_16gb_warns_or_streams_safely(self):
+        plan = plan_for_budget(
+            model_path=".",
+            total_params_billion=70,
+            n_layers=80,
+            bytes_per_param=0.6,  # Q4 quantized
+            n_ram_gb=16,
+        )
+        self.assertGreaterEqual(plan.safe_hot_layers, 1)
+        self.assertLess(plan.safe_hot_layers, 80)  # must NOT fit all layers in RAM
+        self.assertGreater(plan.layer_size_gb, 0)
+
+    def test_small_model_fits_more_layers(self):
+        plan = plan_for_budget(
+            model_path=".",
+            total_params_billion=1.1,
+            n_layers=22,
+            bytes_per_param=2.0,
+            n_ram_gb=16,
+        )
+        self.assertGreaterEqual(plan.safe_hot_layers, 1)
+
+    def test_class_wrapper_matches_function(self):
+        planner = RAMPlanner(model_path=".", total_params_billion=7, n_layers=32, n_ram_gb=16)
+        plan = planner.plan()
+        self.assertEqual(plan.total_ram_gb, 16)
+
+    def test_extreme_budget_emits_warning(self):
+        plan = plan_for_budget(
+            model_path=".",
+            total_params_billion=70,
+            n_layers=80,
+            bytes_per_param=2.0,  # fp16, unrealistic for 16GB -> should warn
+            n_ram_gb=4,
+        )
+        self.assertTrue(len(plan.warnings) >= 1)
+
+
+if __name__ == "__main__":
+    unittest.main()