inline-core 1.1.1__tar.gz → 1.1.2__tar.gz

{inline_core-1.1.1 → inline_core-1.1.2}/CLAUDE.md

@@ -120,9 +120,26 @@ between nodes and are never takes.
 - **Server bind** — `INLINE_HOST` (default `127.0.0.1`), `INLINE_PORT` (default `8848`).
 - **Model overrides** — e.g. `INLINE_ZIMAGE_MODEL` (a single `.safetensors` file path, a local
   diffusers dir, or a HF repo id for Z-Image). Auto-resolved from `diffusion_models/` when unset.
-- **Memory** — always prefer the GPU: even under the `lowvram` profile, weights stay resident on the
-  GPU (VAE tiling/slicing + attention slicing + int8 do the saving); we do **not** auto-offload to CPU.
-  `INLINE_ALLOW_CPU_OFFLOAD=1` opts back into `enable_model_cpu_offload()` for extreme cases.
+- **Memory** — by default prefer the GPU, and **auto-fit the model to it**. When a runner hands the
+  policy the model's on-disk sizes (`DevicePolicy.set_footprint`, a `ModelFootprint`), the policy sizes
+  the weights against total VRAM (minus an activation headroom) and picks the lightest plan that fits:
+  full-precision **resident** → else int8 **resident** (torchao halves the transformer + text encoder,
+  no CPU offload — int8 + accelerate's `enable_model_cpu_offload` deadlock together) → else unquantized
+  `SEQUENTIAL` submodule streaming. So a card that can't hold Z-Image full-precision (a T4)
+  **auto-int8s with no flag**; `set_footprint(None)` / an unsizable whole-pipeline folder falls back to
+  the coarse total-VRAM buckets. Capacity is TOTAL VRAM (a fixed device property), not live-free, so the
+  plan — and the pipeline cache key it feeds — is stable across runs. The runner does a **pre-flight
+  check** (`DevicePolicy.fit_estimate`): a model too big for VRAM+RAM fails with a clean node error
+  before any load, instead of a host-RAM OOM-kill that would take the shared server down. Weights stream
+  straight to the GPU on load (`device=`/`device_map` in `models/loaders.py`) so peak host RAM ≈ one
+  tensor, and switching checkpoints **evicts** the previous model (`loaders.unload_components`,
+  `_evict_stale`) rather than stacking. `webui.sh` always sets `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`
+  to cut fragmentation OOMs. **Smart memory** (`INLINE_SMART_MEMORY=1`, `--smart-memory`) and
+  `INLINE_ALLOW_CPU_OFFLOAD=1` remain as explicit overrides when no footprint is set.
+- **Compute dtype** — bf16 on the GPU by default, but **fp16 on cards without bf16 acceleration**
+  (Turing/Volta — compute capability < 8.0, e.g. a T4): same footprint, but it uses the fp16 tensor
+  cores instead of bf16's slow path. The **VAE stays upcast** (bf16, or fp32 when the denoiser is
+  fp16) because fp16 VAE decode can overflow to black/NaN images (`_compute_dtype` in `device/`).
 - **Multi-GPU** — `INLINE_PARALLEL` (e.g. `pipefusion=2`, `pipefusion=2,ulysses=2`); degrees multiply
   to the world size, which must equal the GPU count.
 

{inline_core-1.1.1 → inline_core-1.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: inline-core
-Version: 1.1.1
+Version: 1.1.2
 Summary: The generation engine behind Inline.
 Requires-Python: >=3.11
 Requires-Dist: numpy>=1.26
@@ -19,7 +19,9 @@ Requires-Dist: diffusers>=0.36; extra == 'parallel'
 Requires-Dist: huggingface-hub>=0.23; extra == 'parallel'
 Requires-Dist: nvidia-ml-py>=12; extra == 'parallel'
 Requires-Dist: safetensors>=0.4; extra == 'parallel'
+Requires-Dist: scipy>=1.11; extra == 'parallel'
 Requires-Dist: torch>=2.2; extra == 'parallel'
+Requires-Dist: torchao>=0.14; extra == 'parallel'
 Requires-Dist: transformers>=4.44; extra == 'parallel'
 Requires-Dist: xfuser>=0.4; extra == 'parallel'
 Provides-Extra: server
@@ -31,5 +33,7 @@ Requires-Dist: accelerate>=0.30; extra == 'zimage'
 Requires-Dist: diffusers>=0.36; extra == 'zimage'
 Requires-Dist: huggingface-hub>=0.23; extra == 'zimage'
 Requires-Dist: safetensors>=0.4; extra == 'zimage'
+Requires-Dist: scipy>=1.11; extra == 'zimage'
 Requires-Dist: torch>=2.2; extra == 'zimage'
+Requires-Dist: torchao>=0.14; extra == 'zimage'
 Requires-Dist: transformers>=4.44; extra == 'zimage'

{inline_core-1.1.1 → inline_core-1.1.2}/README.md

@@ -136,6 +136,37 @@ Enabling it:
 The device policy and the worker-group IPC are in place and tested with a stub worker; the real
 xfuser denoise lands with the GPU-side Z-Image runner.
 
+## Memory: fit a big model onto a small GPU
+
+When a model is too large to hold full-precision on your GPU — or larger than your system RAM — Inline
+Core fits it to the machine automatically, **with no flags**. This is the low-VRAM path, and it aims to
+"just run" instead of making you tune profiles.
+
+- **Weights stream straight to the GPU.** Each component loads directly onto the GPU from its
+  memory-mapped `.safetensors` — the engine never materializes a full copy in system RAM first. So you
+  do **not** need roughly 2× the model size in RAM to load it, and a load can no longer exhaust host RAM
+  and take the server down.
+- **Auto-fit picks the lightest plan that fits.** The policy sizes the model against your VRAM and
+  chooses, in order: full-precision **resident** → **int8 resident** (weight-only quantization halves the
+  big weights so they fit on the GPU) → **CPU-offload streaming** (submodules move on/off the GPU). A card
+  that can't hold the model full-precision quantizes to int8 on its own — no need to know a flag.
+- **Load-time fit check.** A model too large for VRAM + RAM fails fast with a clear message on the node,
+  instead of crashing mid-load. The model popup also shows the estimate up front (fits / will run int8 /
+  won't fit).
+- **Switching models frees the old one.** Loading a different checkpoint evicts the previous weights
+  rather than stacking them, so you can move between models without accumulating VRAM.
+- **Generation fits too, not just loading.** Prompts encode under `no_grad` so the text-encoder forward
+  never retains its activation graph, and the encoder is parked on the CPU during denoise to free its
+  VRAM; large images (1024² and up) decode the VAE in tiles instead of one full-frame pass. So the
+  memory-heavy steps — encode and decode — stay bounded rather than spiking host RAM or VRAM and taking
+  the server down at high resolution.
+- **Fragmentation-resistant allocator.** `webui.sh` runs with PyTorch expandable CUDA segments, which
+  avoids the "allocation failed with VRAM still free" fragmentation OOMs common on low-VRAM cards.
+
+You can still steer it by hand: `--lowvram` for the tight-VRAM profile, `--smart-memory` to force the
+offload machinery, `--vram-budget GB` / `INLINE_VRAM_BUDGET_GB` to cap the assumed VRAM, or
+`INLINE_PROFILE` to pin `gpu-max | lowvram | cpu`. `GET /v1/health` reports the live VRAM/RAM budget.
+
 ## Run
 
 The easy path is `webui.sh`, which maps friendly flags onto the engine's `INLINE_*` env knobs:

{inline_core-1.1.1 → inline_core-1.1.2}/pyproject.toml

@@ -2,7 +2,7 @@
 # PyPI distribution name. The import package is `inline_core` (src/inline_core); PyPI treats the
 # hyphen and underscore as equivalent, so `pip install inline-core` installs the `inline_core` package.
 name = "inline-core"
-version = "1.1.1"
+version = "1.1.2"
 description = "The generation engine behind Inline."
 requires-python = ">=3.11"
 dependencies = [
@@ -21,6 +21,15 @@ zimage = [
   "transformers>=4.44",
   "accelerate>=0.30",
   "safetensors>=0.4",
+  # Smart memory (`webui.sh --smart-memory`) quantizes the big weights to int8 with torchao's
+  # weight-only quant — torch-native and movable, so it coexists with CPU offload. >=0.14 for the
+  # config-object API (Int8WeightOnlyConfig; the "int8_weight_only" string form was dropped).
+  # Best-effort at load time: an absent/incompatible torchao logs and falls back, never crashes.
+  "torchao>=0.14",
+  # scipy backs the Beta sigma schedule (FlowMatchEulerDiscreteScheduler.use_beta_sigmas). diffusers
+  # gates that option on scipy at scheduler construction; the sampling registry falls back to the
+  # model default spacing if it is ever absent, but declare it so Beta works out of the box.
+  "scipy>=1.11",
   # Explicit model downloads (the node's model popup) go through huggingface_hub. It ships
   # transitively with diffusers/transformers, but pin it here since we call snapshot_download
   # directly and rely on local_dir= to land files in models/ (never the hidden HF cache).

{inline_core-1.1.1 → inline_core-1.1.2}/src/inline_core/device/detect.py

@@ -23,6 +23,22 @@ def available_devices() -> tuple[Device, ...]:
     return (Device(DeviceKind.CPU),)
 
 
+def cuda_supports_bf16(device: Device) -> bool:
+    """True when the CUDA GPU has native bf16 acceleration — Ampere or newer (compute capability
+    >= 8.0). Turing/Volta (T4, V100) can run bf16 but only through a slow unaccelerated path, so the
+    policy prefers fp16 there (same memory footprint, but it uses the fp16 tensor cores). Unknown /
+    non-CUDA falls back to True so bf16 stays the default on anything we can't measure."""
+    if device.kind is not DeviceKind.CUDA:
+        return True
+    try:
+        import torch
+
+        major, _ = torch.cuda.get_device_capability(device.index)
+        return major >= 8
+    except Exception:
+        return True
+
+
 def has_nvlink() -> bool:
     """Best-effort: True only when NVLink is confirmed between two GPUs. Conservative (False) on any
     uncertainty, so the policy defaults to the PCIe-friendly PipeFusion split."""

inline_core-1.1.2/src/inline_core/device/memory.py

@@ -0,0 +1,361 @@
+"""The memory-aware device policy: measure RAM/VRAM, pick a profile, own dtype/offload/quant/tiling.
+
+Profiles: gpu-max (ample VRAM), lowvram (tight VRAM), cpu (fp32, tiling, int8 to fit RAM). By
+default we always prefer the GPU: even under lowvram, weights stay resident on the GPU (tiling +
+attention
+slicing + int8 do the memory saving) and we do NOT auto-offload to CPU — offloading is slow and
+defeats "use the GPU we have".
+
+Smart memory (INLINE_SMART_MEMORY=1, `webui.sh --smart-memory`) is the opt-in escape hatch for a
+model that simply does not fit resident: it spreads the model across VRAM + RAM + CPU by streaming
+components on/off the GPU (a graduated OffloadMode — MODEL, or SEQUENTIAL on a very small GPU) and
+quantizes the big weights to int8 so the offloaded half also fits in RAM. Slower per image, but it
+runs where full-resident OOMs. Set INLINE_ALLOW_CPU_OFFLOAD=1 for the older bare model-offload knob
+without quantization. Override the profile/budget with INLINE_PROFILE and INLINE_VRAM_BUDGET_GB.
+Detection is lazy so the core imports without torch or psutil; an unavailable measurement keeps the
+policy conservative.
+"""
+
+from __future__ import annotations
+
+import os
+
+from .detect import available_devices, cuda_supports_bf16, has_nvlink
+from .policy import (
+    AttentionBackend,
+    DevicePolicy,
+    FitEstimate,
+    ModelFootprint,
+    OffloadMode,
+    Parallel,
+    Placement,
+    Profile,
+    Quantization,
+)
+from .types import Device, DeviceKind, DType
+
+_GPU_MAX_MIN_VRAM_GB = 16.0  # at or above -> gpu-max, else lowvram
+_QUANT_VRAM_GB = 10.0  # lowvram below this -> int8
+_QUANT_RAM_GB = 48.0  # cpu below this -> int8
+# Smart memory: at/above this VRAM the int8 model fits RESIDENT on the GPU (no offload); below it,
+# even int8 won't fit, so fall to SEQUENTIAL submodule streaming (unquantized — torchao int8 and CPU
+# offload deadlock together) so a very small GPU can still run.
+_SMART_RESIDENT_MIN_VRAM_GB = 6.0
+
+# Fit estimate (size-aware placement): reserve this much VRAM beyond the weights for denoise
+# activations + the CUDA context + allocator fragmentation, and treat int8 weight-only quant as
+# ~half the fp16 weight bytes. Deliberately generous so the estimate errs toward a lighter plan.
+_ACTIVATION_HEADROOM_GB = 2.5
+_INT8_FACTOR = 0.5
+
+
+def _system_ram_gb() -> float | None:
+    try:
+        return os.sysconf("SC_PHYS_PAGES") * os.sysconf("SC_PAGE_SIZE") / 1e9
+    except (ValueError, OSError, AttributeError):
+        pass
+    try:
+        import psutil
+
+        return psutil.virtual_memory().total / 1e9
+    except ModuleNotFoundError:
+        return None
+
+
+def _vram_gb(device: Device) -> float | None:
+    if device.kind is not DeviceKind.CUDA:
+        return None
+    try:
+        import torch
+
+        return torch.cuda.mem_get_info(device.index)[1] / 1e9
+    except Exception:
+        return None
+
+
+def _free_vram_gb(device: Device) -> float | None:
+    """Live *free* VRAM (mem_get_info[0]) — for diagnostics/UI only. NOT used to choose the plan:
+    residency-dependent free readings would make the quant/offload decision (and thus the pipeline
+    cache key) oscillate. Capacity decisions use total VRAM (``_vram_gb``)."""
+    if device.kind is not DeviceKind.CUDA:
+        return None
+    try:
+        import torch
+
+        return torch.cuda.mem_get_info(device.index)[0] / 1e9
+    except Exception:
+        return None
+
+
+def _free_ram_gb() -> float | None:
+    """Live *available* system RAM (psutil), for the UI + a load guard."""
+    try:
+        import psutil
+
+        return psutil.virtual_memory().available / 1e9
+    except Exception:
+        return None
+
+
+def _env_profile() -> Profile | None:
+    value = os.environ.get("INLINE_PROFILE", "").strip().lower()
+    return next((p for p in Profile if p.value == value), None)
+
+
+def _env_budget() -> float | None:
+    value = os.environ.get("INLINE_VRAM_BUDGET_GB", "").strip()
+    try:
+        return float(value) if value else None
+    except ValueError:
+        return None
+
+
+def _env_flag(name: str) -> bool:
+    return os.environ.get(name, "").strip().lower() in ("1", "true", "yes", "on")
+
+
+def _env_allow_offload() -> bool:
+    """Opt back into bare CPU offload under lowvram. Off by default: we keep weights on the GPU."""
+    return _env_flag("INLINE_ALLOW_CPU_OFFLOAD")
+
+
+def _env_smart_memory() -> bool:
+    """Opt into smart memory: graduated CPU offload + int8 quant so a too-big model fits across
+    VRAM + RAM. Off by default. Set by `webui.sh --smart-memory`."""
+    return _env_flag("INLINE_SMART_MEMORY")
+
+
+def _env_parallel() -> Parallel | None:
+    """Parse INLINE_PARALLEL like `pipefusion=2,ulysses=2` into a degree spec."""
+    value = os.environ.get("INLINE_PARALLEL", "").strip()
+    if not value:
+        return None
+    valid = {"pipefusion", "ulysses", "ring", "cfg", "tensor"}
+    degrees: dict[str, int] = {}
+    for part in value.split(","):
+        key, _, raw = part.partition("=")
+        if key.strip() in valid:
+            try:
+                degrees[key.strip()] = int(raw)
+            except ValueError:
+                continue
+    return Parallel(**degrees) if degrees else None
+
+
+class MemoryPolicy(DevicePolicy):
+    def __init__(
+        self,
+        device: Device | None = None,
+        *,
+        ram_gb: float | None = None,
+        vram_gb: float | None = None,
+        profile: Profile | None = None,
+        devices: tuple[Device, ...] | None = None,
+        nvlink: bool | None = None,
+        parallel: Parallel | None = None,
+        allow_offload: bool | None = None,
+        smart_memory: bool | None = None,
+        supports_bf16: bool | None = None,
+    ) -> None:
+        self._devices = devices if devices is not None else available_devices()
+        self._device = device or self._devices[0]
+        self._ram_gb = ram_gb if ram_gb is not None else _system_ram_gb()
+        self._vram_gb = vram_gb if vram_gb is not None else _vram_gb(self._device)
+        self._profile = profile or _env_profile() or self._choose_profile()
+        self._nvlink = nvlink if nvlink is not None else has_nvlink()
+        self._parallel = parallel if parallel is not None else _env_parallel()
+        self._smart = smart_memory if smart_memory is not None else _env_smart_memory()
+        self._allow_offload = allow_offload if allow_offload is not None else _env_allow_offload()
+        self._bf16 = (
+            supports_bf16 if supports_bf16 is not None else cuda_supports_bf16(self._device)
+        )
+        self._footprint: ModelFootprint | None = None
+        self._fit: FitEstimate | None = None
+
+    # --- size-aware fit (auto-fit the model to the device without a --smart-memory flag) ---------
+
+    def set_footprint(self, footprint: ModelFootprint | None) -> None:
+        """Record the model's on-disk sizes so ``profile``/``quantization``/offload fit the device.
+        Called by the runner per run (files can change). Falls back to the coarse buckets when the
+        footprint is None or unmeasurable."""
+        self._footprint = footprint
+        self._fit = self._compute_fit(footprint) if footprint is not None else None
+
+    def estimate_fit(self, footprint: ModelFootprint) -> FitEstimate | None:
+        """A pure fit verdict — does NOT mutate the policy, so the UI thread can call it while a run
+        holds the policy on the worker thread."""
+        return self._compute_fit(footprint)
+
+    def fit_estimate(self) -> FitEstimate | None:
+        return self._fit
+
+    def _compute_fit(self, fp: ModelFootprint) -> FitEstimate | None:
+        """Pick the lightest plan whose weights fit the GPU: full-precision resident, else int8
+        resident, else CPU-offload streaming (unquantized — int8 + offload deadlock). Capacity is
+        TOTAL VRAM (a fixed device property) minus activation headroom, so the decision is stable
+        across runs and doesn't bust the pipeline cache. Returns None (→ coarse buckets) off-CUDA or
+        when sizes are unavailable (e.g. a whole-pipeline folder)."""
+        if self._device.kind is not DeviceKind.CUDA or fp.total_bytes <= 0:
+            return None
+        budget = _env_budget() or self._vram_gb
+        if budget is None:
+            return None
+        cap = max(0.0, budget - _ACTIVATION_HEADROOM_GB)
+        big = (fp.diffusion_bytes + fp.text_encoder_bytes) / 1e9
+        vae = fp.vae_bytes / 1e9
+        full = big + vae
+        int8 = big * _INT8_FACTOR + vae
+        forced = _env_profile() is not None  # explicit --profile pins the profile; fit picks quant
+
+        def prof(auto: Profile) -> Profile:
+            return self._profile if forced else auto
+
+        if full <= cap:
+            return FitEstimate(
+                "resident", Quantization.NONE, OffloadMode.NONE, prof(Profile.GPU_MAX),
+                full, budget, True, "Full-precision weights fit in VRAM.",
+            )
+        if int8 <= cap:
+            return FitEstimate(
+                "int8", Quantization.INT8, OffloadMode.NONE, prof(Profile.LOWVRAM),
+                int8, budget, True,
+                "Weights are int8-quantized to fit this GPU's VRAM.",
+            )
+        # int8 still won't fit resident -> CPU-offload streaming. Only viable if the (unquantized)
+        # model fits in system RAM, since sequential offload holds the off-GPU weights there.
+        ram = self._ram_gb
+        if ram is not None and full > ram:
+            return FitEstimate(
+                "wont-fit", Quantization.NONE, OffloadMode.SEQUENTIAL, prof(Profile.LOWVRAM),
+                full, budget, False,
+                "Model is too large for this GPU's VRAM and this machine's RAM.",
+            )
+        return FitEstimate(
+            "offload", Quantization.NONE, OffloadMode.SEQUENTIAL, prof(Profile.LOWVRAM),
+            int8, budget, True, "Weights stream between GPU and RAM (slower).",
+        )
+
+    def vram_budget_mb(self) -> int | None:
+        budget = _env_budget() or self._vram_gb
+        return int(budget * 1024) if budget is not None else None
+
+    def free_vram_mb(self) -> int | None:
+        gb = _free_vram_gb(self._device)
+        return int(gb * 1024) if gb is not None else None
+
+    def free_ram_mb(self) -> int | None:
+        gb = _free_ram_gb()
+        return int(gb * 1024) if gb is not None else None
+
+    def _choose_profile(self) -> Profile:
+        if self._device.kind is DeviceKind.CPU:
+            return Profile.CPU
+        if self._device.kind is DeviceKind.MPS:
+            return Profile.GPU_MAX  # unified memory; offload semantics differ
+        budget = _env_budget() or self._vram_gb
+        if budget is not None and budget < _GPU_MAX_MIN_VRAM_GB:
+            return Profile.LOWVRAM
+        return Profile.GPU_MAX
+
+    @property
+    def profile(self) -> Profile:
+        """The effective profile: the size-aware fit's profile when a footprint is set, else the
+        profile chosen at init from the coarse VRAM buckets / env override."""
+        return self._fit.profile if self._fit is not None else self._profile
+
+    def placement(self, role: str) -> Placement:
+        if self.profile is Profile.CPU:
+            return Placement(self._device, DType.FP32)
+        offload_mode = self._offload_mode()
+        dtype = self._compute_dtype(role)
+        if role == "denoiser":
+            parallel = self._denoiser_parallel()
+            if parallel is not None:
+                cuda = tuple(d for d in self._devices if d.kind is DeviceKind.CUDA)
+                return Placement(
+                    self._device,
+                    dtype,
+                    offload_mode=offload_mode,
+                    devices=cuda[: parallel.world_size],
+                    parallel=parallel,
+                )
+        return Placement(self._device, dtype, offload_mode=offload_mode)
+
+    def _compute_dtype(self, role: str) -> DType:
+        """The weight/compute dtype for a GPU role. bf16 by default, but fp16 on GPUs without bf16
+        acceleration (Turing/Volta) — same footprint, yet it uses their fp16 tensor cores instead of
+        bf16's slow path. The VAE is the exception: fp16 decode can overflow to black/NaN images, so
+        it stays upcast (bf16, or fp32 on a card that also lacks bf16) — it is tiny, so the cost is
+        nil.
+
+        **int8 overrides the fp16 preference to bf16.** torchao weight-only int8 only supports a
+        bf16 compute dtype — with fp16 the quantization silently no-ops, so the "int8" weights load
+        at *full* fp16 size and blow the VRAM budget (a T4 then OOMs mid-load). The int8 matmul still
+        runs on the card's int8 tensor cores; only the residual bf16 activations pay the slow path.
+        bf16 also has fp32's exponent range, so the VAE no longer needs the fp32 anti-overflow
+        upcast — it rides along at bf16."""
+        if self.quantization() is Quantization.INT8:
+            return DType.BF16
+        if self._bf16:
+            return DType.BF16
+        return DType.FP32 if role == "vae" else DType.FP16
+
+    def _offload_mode(self) -> OffloadMode:
+        """Whether (and how) to stream weights to CPU. Default: never — prefer the GPU we have, and
+        let tiling/slicing/int8 do the saving with weights resident.
+
+        Smart memory keeps weights RESIDENT and quantizes to int8 instead of offloading: int8 halves
+        the model so a lowvram GPU holds it, and that avoids the slow — and, with torchao int8,
+        hang-prone — accelerate CPU-offload path. Only a GPU too small for even int8-resident
+        streams submodules (SEQUENTIAL, unquantized). The older INLINE_ALLOW_CPU_OFFLOAD flag still
+        does bare (unquantized) MODEL offload for anyone who wants it."""
+        if self._fit is not None:
+            return self._fit.offload_mode
+        if self._profile is not Profile.LOWVRAM:
+            return OffloadMode.NONE
+        if self._smart:
+            if _below(self._vram_gb, _SMART_RESIDENT_MIN_VRAM_GB):
+                return OffloadMode.SEQUENTIAL
+            return OffloadMode.NONE
+        return OffloadMode.MODEL if self._allow_offload else OffloadMode.NONE
+
+    def _denoiser_parallel(self) -> Parallel | None:
+        """Split the denoiser across GPUs when there are 2+. An explicit override wins; else auto:
+        PipeFusion on PCIe (no NVLink), sequence-parallel (Ulysses) on NVLink."""
+        if self._parallel is not None:
+            return self._parallel if self._parallel.world_size > 1 else None
+        cuda = [d for d in self._devices if d.kind is DeviceKind.CUDA]
+        if len(cuda) < 2:
+            return None
+        return Parallel(ulysses=len(cuda)) if self._nvlink else Parallel(pipefusion=len(cuda))
+
+    def attention_backend(self) -> AttentionBackend:
+        return AttentionBackend.SDPA
+
+    def vae_tiling(self) -> bool:
+        return self.profile in (Profile.LOWVRAM, Profile.CPU)
+
+    def attention_slicing(self) -> bool:
+        return self.profile in (Profile.LOWVRAM, Profile.CPU)
+
+    def quantization(self) -> Quantization:
+        # A size-aware fit (when set) owns the quant choice — int8 auto-engages when full precision
+        # won't fit, no --smart-memory flag needed.
+        if self._fit is not None:
+            return self._fit.quant
+        # Smart memory quantizes to int8 so the halved model fits RESIDENT on the GPU (see
+        # _offload_mode). The one exception: a GPU so small it must SEQUENTIAL-offload instead runs
+        # unquantized — torchao int8 + CPU offload deadlock together.
+        if self._smart and self._profile is Profile.LOWVRAM:
+            if self._offload_mode() is OffloadMode.SEQUENTIAL:
+                return Quantization.NONE
+            return Quantization.INT8
+        if self._profile is Profile.LOWVRAM and _below(self._vram_gb, _QUANT_VRAM_GB):
+            return Quantization.INT8
+        if self._profile is Profile.CPU and _below(self._ram_gb, _QUANT_RAM_GB):
+            return Quantization.INT8
+        return Quantization.NONE
+
+
+def _below(measured: float | None, threshold: float) -> bool:
+    return measured is not None and measured < threshold

inline_core-1.1.2/src/inline_core/device/policy.py

@@ -0,0 +1,167 @@
+"""The device and memory policy interface. Components never self-assign a device; they ask here."""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from enum import Enum
+
+from .types import Device, DType
+
+
+class Profile(str, Enum):
+    GPU_MAX = "gpu-max"
+    LOWVRAM = "lowvram"
+    CPU = "cpu"
+
+
+class AttentionBackend(str, Enum):
+    FLASH = "flash"
+    XFORMERS = "xformers"
+    SDPA = "sdpa"
+
+
+class OffloadMode(str, Enum):
+    """How a component's weights are split between GPU VRAM and CPU RAM.
+
+    NONE       weights stay resident on the GPU (the default — prefer the GPU we have).
+    MODEL      diffusers ``enable_model_cpu_offload``: only the *active* component (text encoder,
+               then transformer, then VAE) sits on the GPU; the rest waits in CPU RAM. Peak VRAM ≈
+               the largest single component. Fast — the standard low-VRAM fit.
+    SEQUENTIAL diffusers ``enable_sequential_cpu_offload``: submodules stream on/off the GPU
+               layer-by-layer. Lowest peak VRAM, slowest — for GPUs too small for MODEL offload.
+    """
+
+    NONE = "none"
+    MODEL = "model"
+    SEQUENTIAL = "sequential"
+
+
+class Quantization(str, Enum):
+    NONE = "none"
+    INT8 = "int8"  # torch-native weight-only, portable
+    NF4 = "nf4"  # bitsandbytes, cuda-only
+
+
+@dataclass(frozen=True)
+class Parallel:
+    """How the denoiser is split across GPUs (xDiT degrees). Product of degrees = world size (GPUs).
+    PipeFusion is PCIe-friendly; Ulysses/Ring want NVLink; CFG applies only to guided models."""
+
+    pipefusion: int = 1
+    ulysses: int = 1
+    ring: int = 1
+    cfg: int = 1
+    tensor: int = 1
+
+    @property
+    def world_size(self) -> int:
+        return self.pipefusion * self.ulysses * self.ring * self.cfg * self.tensor
+
+
+@dataclass(frozen=True)
+class Placement:
+    """Where and how a component runs. Chosen by the policy, never by the component."""
+
+    device: Device
+    dtype: DType
+    offload_mode: OffloadMode = OffloadMode.NONE
+    # Multi-GPU (denoiser only): the device group + how it is split. Empty/None = single device.
+    devices: tuple[Device, ...] = ()
+    parallel: Parallel | None = None
+
+    @property
+    def offload(self) -> bool:
+        """Whether any CPU offload is in effect (either MODEL or SEQUENTIAL). Kept as a bool so the
+        many call sites that only care "is this streaming to CPU?" stay simple."""
+        return self.offload_mode is not OffloadMode.NONE
+
+
+@dataclass(frozen=True)
+class ModelFootprint:
+    """On-disk byte sizes of a model's big components — the input to a memory fit estimate. The
+    files ship already 16-bit, so a size is a good proxy for the fp16-resident weight footprint;
+    int8 halves the two big weights. The policy uses this to pick dtype/quant/offload to the GPU
+    instead of a coarse VRAM bucket."""
+
+    diffusion_bytes: int = 0
+    text_encoder_bytes: int = 0
+    vae_bytes: int = 0
+
+    @property
+    def total_bytes(self) -> int:
+        return self.diffusion_bytes + self.text_encoder_bytes + self.vae_bytes
+
+
+@dataclass(frozen=True)
+class FitEstimate:
+    """The policy's verdict for a given ModelFootprint on this device: what plan fits, how much VRAM
+    it needs, and whether it fits. Drives the runner's pre-flight check and the UI warning."""
+
+    plan: str  # "resident" | "int8" | "offload" | "wont-fit"
+    quant: Quantization
+    offload_mode: OffloadMode
+    profile: Profile
+    required_vram_gb: float
+    total_vram_gb: float | None
+    fits: bool
+    note: str
+
+
+class DevicePolicy(ABC):
+    """Owns dtype, device, offload, attention backend, and tiling for a worker."""
+
+    @property
+    @abstractmethod
+    def profile(self) -> Profile: ...
+
+    @abstractmethod
+    def placement(self, role: str) -> Placement:
+        """Placement for a component role: text_encoder, denoiser, vae, and so on."""
+
+    @abstractmethod
+    def attention_backend(self) -> AttentionBackend: ...
+
+    @abstractmethod
+    def vae_tiling(self) -> bool:
+        """Whether to tile VAE decode to cap peak memory."""
+
+    def attention_slicing(self) -> bool:
+        """Whether to slice attention to cap peak memory. Default off."""
+        return False
+
+    def quantization(self) -> Quantization:
+        """Weight quantization to fit low memory. Default none."""
+        return Quantization.NONE
+
+    # --- optional memory-fit surface (a size-aware policy overrides these) -----------------------
+    # Kept off the abstract contract with no-op/None defaults so simple policies (AutoDevicePolicy,
+    # test stubs) need not implement them; a memory-aware policy uses the model's on-disk sizes to
+    # fit dtype/quant/offload to the actual device.
+
+    def set_footprint(self, footprint: ModelFootprint | None) -> None:
+        """Receive the model's on-disk component sizes so later placement/quantization calls fit the
+        device. Default: ignore (placement decided without a size estimate)."""
+        return None
+
+    def estimate_fit(self, footprint: ModelFootprint) -> FitEstimate | None:
+        """A pure fit verdict for a footprint, WITHOUT mutating the policy (safe to call from the UI
+        thread while a run holds the policy). Default None (no estimate)."""
+        return None
+
+    def fit_estimate(self) -> FitEstimate | None:
+        """The verdict for the footprint last given to ``set_footprint`` (for the pre-flight check).
+        Default None."""
+        return None
+
+    def vram_budget_mb(self) -> int | None:
+        """Usable VRAM budget in MB (total or the env override), or None off-GPU. Default None."""
+        return None
+
+    def free_vram_mb(self) -> int | None:
+        """Live free VRAM in MB, or None off-GPU/unmeasurable. Default None."""
+        return None
+
+    def free_ram_mb(self) -> int | None:
+        """Live free system RAM in MB, or None when unmeasurable. Default None."""
+        return None