overflowml 0.1.0__tar.gz

overflowml-0.1.0/.gitignore

@@ -0,0 +1,6 @@
+__pycache__/
+*.pyc
+*.egg-info/
+dist/
+build/
+.pytest_cache/

overflowml-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 HybridLab AI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

overflowml-0.1.0/PKG-INFO

@@ -0,0 +1,224 @@
+Metadata-Version: 2.4
+Name: overflowml
+Version: 0.1.0
+Summary: Run AI models larger than your GPU. Auto-detects hardware and applies optimal memory strategy.
+Project-URL: Homepage, https://github.com/hybridlab-ai/overflowml
+Author: HybridLab AI
+License-Expression: MIT
+License-File: LICENSE
+Keywords: gpu,inference,machine-learning,offload,optimization,vram
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: psutil>=5.9
+Requires-Dist: torch>=2.0
+Provides-Extra: all
+Requires-Dist: accelerate; extra == 'all'
+Requires-Dist: diffusers>=0.30; extra == 'all'
+Requires-Dist: torchao>=0.5; extra == 'all'
+Requires-Dist: transformers; extra == 'all'
+Provides-Extra: dev
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Provides-Extra: diffusers
+Requires-Dist: accelerate; extra == 'diffusers'
+Requires-Dist: diffusers>=0.30; extra == 'diffusers'
+Requires-Dist: transformers; extra == 'diffusers'
+Provides-Extra: mlx
+Requires-Dist: mlx>=0.20; extra == 'mlx'
+Provides-Extra: quantize
+Requires-Dist: torchao>=0.5; extra == 'quantize'
+Description-Content-Type: text/markdown
+
+# OverflowML
+
+**Run AI models larger than your GPU.** One line of code.
+
+OverflowML auto-detects your hardware (NVIDIA, Apple Silicon, AMD, CPU) and applies the optimal memory strategy to load and run models that don't fit in VRAM. No manual configuration needed.
+
+```python
+import overflowml
+
+pipe = load_your_model()  # 40GB model, 24GB GPU? No problem.
+overflowml.optimize_pipeline(pipe, model_size_gb=40)
+result = pipe(prompt)     # Just works.
+```
+
+## The Problem
+
+AI models are getting bigger. A single image generation model can be 40GB+. LLMs regularly hit 70GB-400GB. But most GPUs have 8-24GB of VRAM.
+
+The current solutions are painful:
+- **Manual offloading** — you need to know which PyTorch function to call, which flags work together, and which combinations crash
+- **Quantization footguns** — FP8 is incompatible with CPU offload on Windows. Attention slicing crashes with sequential offload. INT4 needs specific libraries.
+- **Trial and error** — every hardware/model/framework combo has different gotchas
+
+OverflowML handles all of this automatically.
+
+## How It Works
+
+```
+Model: 40GB (BF16)          Your GPU: 24GB VRAM
+         │                           │
+    OverflowML detects mismatch      │
+         │                           │
+    ┌────▼────────────────────────────▼────┐
+    │  Strategy: Sequential CPU Offload    │
+    │  Move 1 layer (~1GB) to GPU at a    │
+    │  time, compute, move back.          │
+    │  Peak VRAM: ~3GB                     │
+    │  System RAM used: ~40GB              │
+    │  Speed: 33s/image (RTX 5090)        │
+    └──────────────────────────────────────┘
+```
+
+### Strategy Decision Tree
+
+| Model vs VRAM | Strategy | Peak VRAM | Speed |
+|---------------|----------|-----------|-------|
+| Model fits with 15% headroom | Direct GPU load | Full | Fastest |
+| FP8 model fits | FP8 quantization | ~55% of model | Fast |
+| Components fit individually | Model CPU offload | ~70% of model | Medium |
+| Nothing fits | Sequential CPU offload | ~3GB | Slower but works |
+| Not enough RAM either | INT4 quantization + sequential | ~3GB | Slowest |
+
+### Apple Silicon (Unified Memory)
+
+On Macs, CPU and GPU share the same memory pool — there's nothing to "offload." OverflowML detects this and skips offloading entirely. If the model fits in ~75% of your RAM, it loads directly. If not, quantization is recommended.
+
+| Mac | Unified Memory | Largest Model (4-bit) |
+|-----|---------------|----------------------|
+| M4 Max | 128GB | ~80B params |
+| M3/M4 Ultra | 192GB | ~120B params |
+| M3 Ultra | 512GB | 670B params |
+
+## Installation
+
+```bash
+pip install overflowml
+
+# With diffusers support:
+pip install overflowml[diffusers]
+
+# With quantization:
+pip install overflowml[all]
+```
+
+## Usage
+
+### Diffusers Pipeline (Recommended)
+
+```python
+import torch
+import overflowml
+from diffusers import FluxPipeline
+
+pipe = FluxPipeline.from_pretrained(
+    "black-forest-labs/FLUX.1-dev",
+    torch_dtype=torch.bfloat16,
+)
+
+# One line — auto-detects hardware, picks optimal strategy
+strategy = overflowml.optimize_pipeline(pipe, model_size_gb=24)
+print(strategy.summary())
+
+result = pipe("a sunset over mountains", num_inference_steps=20)
+```
+
+### Batch Generation with Memory Guard
+
+```python
+from overflowml import MemoryGuard
+
+guard = MemoryGuard(threshold=0.7)  # cleanup at 70% VRAM usage
+
+for prompt in prompts:
+    with guard:  # auto-cleans VRAM between iterations
+        result = pipe(prompt)
+        result.images[0].save(f"output.png")
+```
+
+### CLI — Hardware Detection
+
+```bash
+$ overflowml detect
+
+=== OverflowML Hardware Detection ===
+Accelerator: cuda
+GPU: NVIDIA GeForce RTX 5090 (32GB VRAM)
+System RAM: 194GB
+Overflow capacity: 178GB (total effective: 210GB)
+BF16: yes | FP8: yes
+
+$ overflowml plan 40
+
+=== Strategy for 40GB model ===
+Offload: sequential_cpu
+Dtype: bfloat16
+GC cleanup: enabled (threshold 70%)
+Estimated peak VRAM: 3.0GB
+  → Sequential offload: 1 layer at a time (~3GB VRAM), model lives in 194GB RAM
+WARNING: FP8 incompatible with CPU offload on Windows
+WARNING: Do NOT enable attention_slicing with sequential offload
+```
+
+### Standalone Model
+
+```python
+import overflowml
+
+model = load_my_transformer()
+strategy = overflowml.optimize_model(model, model_size_gb=14)
+```
+
+## Proven Results
+
+Built and battle-tested on a real production pipeline:
+
+| Metric | Before OverflowML | After |
+|--------|-------------------|-------|
+| Time per step | 530s (VRAM thrashing) | 6.7s |
+| Images generated | 0/30 (crashes) | 30/30 |
+| Total time | Impossible | 16.4 minutes |
+| Peak VRAM | 32GB (thrashing) | 3GB |
+| Reliability | Crashes after 3 images | Zero failures |
+
+*40GB model on RTX 5090 (32GB VRAM) + 194GB RAM, sequential offload, Lightning LoRA 4-step*
+
+## Known Incompatibilities
+
+These are automatically handled by OverflowML's strategy engine:
+
+| Combination | Issue | OverflowML Fix |
+|-------------|-------|----------------|
+| FP8 + CPU offload (Windows) | `Float8Tensor` can't move between devices | Skips FP8, uses BF16 |
+| `attention_slicing` + sequential offload | CUDA illegal memory access | Never enables both |
+| `enable_model_cpu_offload` + 40GB transformer | Transformer exceeds VRAM | Uses sequential offload instead |
+| `expandable_segments` on Windows WDDM | Not supported | Gracefully ignored |
+
+## Architecture
+
+```
+overflowml/
+├── detect.py      — Hardware detection (CUDA, MPS, MLX, ROCm, CPU)
+├── strategy.py    — Strategy engine (picks optimal offload + quantization)
+├── optimize.py    — Applies strategy to pipelines and models
+└── cli.py         — Command-line interface
+```
+
+## Cross-Platform Support
+
+| Platform | Accelerator | Status |
+|----------|-------------|--------|
+| Windows + NVIDIA | CUDA | Production-ready |
+| Linux + NVIDIA | CUDA | Production-ready |
+| macOS + Apple Silicon | MPS / MLX | Detection ready, optimization in progress |
+| Linux + AMD | ROCm | Planned |
+| CPU-only | CPU | Fallback always works |
+
+## License
+
+MIT

overflowml-0.1.0/README.md

@@ -0,0 +1,189 @@
+# OverflowML
+
+**Run AI models larger than your GPU.** One line of code.
+
+OverflowML auto-detects your hardware (NVIDIA, Apple Silicon, AMD, CPU) and applies the optimal memory strategy to load and run models that don't fit in VRAM. No manual configuration needed.
+
+```python
+import overflowml
+
+pipe = load_your_model()  # 40GB model, 24GB GPU? No problem.
+overflowml.optimize_pipeline(pipe, model_size_gb=40)
+result = pipe(prompt)     # Just works.
+```
+
+## The Problem
+
+AI models are getting bigger. A single image generation model can be 40GB+. LLMs regularly hit 70GB-400GB. But most GPUs have 8-24GB of VRAM.
+
+The current solutions are painful:
+- **Manual offloading** — you need to know which PyTorch function to call, which flags work together, and which combinations crash
+- **Quantization footguns** — FP8 is incompatible with CPU offload on Windows. Attention slicing crashes with sequential offload. INT4 needs specific libraries.
+- **Trial and error** — every hardware/model/framework combo has different gotchas
+
+OverflowML handles all of this automatically.
+
+## How It Works
+
+```
+Model: 40GB (BF16)          Your GPU: 24GB VRAM
+         │                           │
+    OverflowML detects mismatch      │
+         │                           │
+    ┌────▼────────────────────────────▼────┐
+    │  Strategy: Sequential CPU Offload    │
+    │  Move 1 layer (~1GB) to GPU at a    │
+    │  time, compute, move back.          │
+    │  Peak VRAM: ~3GB                     │
+    │  System RAM used: ~40GB              │
+    │  Speed: 33s/image (RTX 5090)        │
+    └──────────────────────────────────────┘
+```
+
+### Strategy Decision Tree
+
+| Model vs VRAM | Strategy | Peak VRAM | Speed |
+|---------------|----------|-----------|-------|
+| Model fits with 15% headroom | Direct GPU load | Full | Fastest |
+| FP8 model fits | FP8 quantization | ~55% of model | Fast |
+| Components fit individually | Model CPU offload | ~70% of model | Medium |
+| Nothing fits | Sequential CPU offload | ~3GB | Slower but works |
+| Not enough RAM either | INT4 quantization + sequential | ~3GB | Slowest |
+
+### Apple Silicon (Unified Memory)
+
+On Macs, CPU and GPU share the same memory pool — there's nothing to "offload." OverflowML detects this and skips offloading entirely. If the model fits in ~75% of your RAM, it loads directly. If not, quantization is recommended.
+
+| Mac | Unified Memory | Largest Model (4-bit) |
+|-----|---------------|----------------------|
+| M4 Max | 128GB | ~80B params |
+| M3/M4 Ultra | 192GB | ~120B params |
+| M3 Ultra | 512GB | 670B params |
+
+## Installation
+
+```bash
+pip install overflowml
+
+# With diffusers support:
+pip install overflowml[diffusers]
+
+# With quantization:
+pip install overflowml[all]
+```
+
+## Usage
+
+### Diffusers Pipeline (Recommended)
+
+```python
+import torch
+import overflowml
+from diffusers import FluxPipeline
+
+pipe = FluxPipeline.from_pretrained(
+    "black-forest-labs/FLUX.1-dev",
+    torch_dtype=torch.bfloat16,
+)
+
+# One line — auto-detects hardware, picks optimal strategy
+strategy = overflowml.optimize_pipeline(pipe, model_size_gb=24)
+print(strategy.summary())
+
+result = pipe("a sunset over mountains", num_inference_steps=20)
+```
+
+### Batch Generation with Memory Guard
+
+```python
+from overflowml import MemoryGuard
+
+guard = MemoryGuard(threshold=0.7)  # cleanup at 70% VRAM usage
+
+for prompt in prompts:
+    with guard:  # auto-cleans VRAM between iterations
+        result = pipe(prompt)
+        result.images[0].save(f"output.png")
+```
+
+### CLI — Hardware Detection
+
+```bash
+$ overflowml detect
+
+=== OverflowML Hardware Detection ===
+Accelerator: cuda
+GPU: NVIDIA GeForce RTX 5090 (32GB VRAM)
+System RAM: 194GB
+Overflow capacity: 178GB (total effective: 210GB)
+BF16: yes | FP8: yes
+
+$ overflowml plan 40
+
+=== Strategy for 40GB model ===
+Offload: sequential_cpu
+Dtype: bfloat16
+GC cleanup: enabled (threshold 70%)
+Estimated peak VRAM: 3.0GB
+  → Sequential offload: 1 layer at a time (~3GB VRAM), model lives in 194GB RAM
+WARNING: FP8 incompatible with CPU offload on Windows
+WARNING: Do NOT enable attention_slicing with sequential offload
+```
+
+### Standalone Model
+
+```python
+import overflowml
+
+model = load_my_transformer()
+strategy = overflowml.optimize_model(model, model_size_gb=14)
+```
+
+## Proven Results
+
+Built and battle-tested on a real production pipeline:
+
+| Metric | Before OverflowML | After |
+|--------|-------------------|-------|
+| Time per step | 530s (VRAM thrashing) | 6.7s |
+| Images generated | 0/30 (crashes) | 30/30 |
+| Total time | Impossible | 16.4 minutes |
+| Peak VRAM | 32GB (thrashing) | 3GB |
+| Reliability | Crashes after 3 images | Zero failures |
+
+*40GB model on RTX 5090 (32GB VRAM) + 194GB RAM, sequential offload, Lightning LoRA 4-step*
+
+## Known Incompatibilities
+
+These are automatically handled by OverflowML's strategy engine:
+
+| Combination | Issue | OverflowML Fix |
+|-------------|-------|----------------|
+| FP8 + CPU offload (Windows) | `Float8Tensor` can't move between devices | Skips FP8, uses BF16 |
+| `attention_slicing` + sequential offload | CUDA illegal memory access | Never enables both |
+| `enable_model_cpu_offload` + 40GB transformer | Transformer exceeds VRAM | Uses sequential offload instead |
+| `expandable_segments` on Windows WDDM | Not supported | Gracefully ignored |
+
+## Architecture
+
+```
+overflowml/
+├── detect.py      — Hardware detection (CUDA, MPS, MLX, ROCm, CPU)
+├── strategy.py    — Strategy engine (picks optimal offload + quantization)
+├── optimize.py    — Applies strategy to pipelines and models
+└── cli.py         — Command-line interface
+```
+
+## Cross-Platform Support
+
+| Platform | Accelerator | Status |
+|----------|-------------|--------|
+| Windows + NVIDIA | CUDA | Production-ready |
+| Linux + NVIDIA | CUDA | Production-ready |
+| macOS + Apple Silicon | MPS / MLX | Detection ready, optimization in progress |
+| Linux + AMD | ROCm | Planned |
+| CPU-only | CPU | Fallback always works |
+
+## License
+
+MIT

overflowml-0.1.0/examples/any_diffusers_pipeline.py

@@ -0,0 +1,49 @@
+"""Example: Use OverflowML with any diffusers pipeline.
+
+Works with Stable Diffusion, SDXL, Flux, Qwen-Image-Edit, etc.
+Just load your pipeline normally, then call optimize_pipeline().
+"""
+
+import torch
+import overflowml
+from overflowml import MemoryGuard
+
+# --- Example 1: SDXL (7GB model, fits most GPUs) ---
+from diffusers import StableDiffusionXLPipeline
+
+pipe = StableDiffusionXLPipeline.from_pretrained(
+    "stabilityai/stable-diffusion-xl-base-1.0",
+    torch_dtype=torch.float16,
+)
+strategy = overflowml.optimize_pipeline(pipe)
+# On RTX 4090: no offload needed, torch.compile enabled
+# On RTX 3060 (12GB): might use FP8 or model offload
+
+result = pipe("a cat in space", num_inference_steps=20)
+result.images[0].save("sdxl_output.png")
+
+
+# --- Example 2: Flux (24GB model, needs offload on most GPUs) ---
+from diffusers import FluxPipeline
+
+pipe = FluxPipeline.from_pretrained(
+    "black-forest-labs/FLUX.1-dev",
+    torch_dtype=torch.bfloat16,
+)
+strategy = overflowml.optimize_pipeline(pipe, model_size_gb=24)
+# On RTX 5090 (32GB): FP8 quantization, fits in VRAM
+# On RTX 4090 (24GB): sequential offload, ~3GB VRAM
+# On Mac M4 Max (128GB): direct load
+
+result = pipe("a sunset over mountains", num_inference_steps=20)
+result.images[0].save("flux_output.png")
+
+
+# --- Example 3: Batch generation with MemoryGuard ---
+guard = MemoryGuard(threshold=0.7)
+
+prompts = ["a cat", "a dog", "a bird", "a fish"]
+for i, prompt in enumerate(prompts):
+    with guard:  # auto-cleans VRAM between images
+        result = pipe(prompt, num_inference_steps=20)
+        result.images[0].save(f"batch_{i}.png")

overflowml-0.1.0/examples/qwen_image_edit.py

@@ -0,0 +1,59 @@
+"""Example: Run Qwen-Image-Edit-2511 (40GB model) on any GPU.
+
+This model is 40GB in BF16 — too large for most GPUs.
+OverflowML auto-detects your hardware and picks the best strategy:
+  - RTX 5090 (32GB): sequential offload, ~3GB VRAM, 33s/image
+  - RTX 4090 (24GB): sequential offload, ~3GB VRAM, ~50s/image
+  - RTX 3090 (24GB): sequential offload, ~3GB VRAM, ~80s/image
+  - Mac M2 Ultra (192GB): direct load, full speed
+  - Mac M4 Max (128GB): direct load, full speed
+"""
+
+import torch
+from pathlib import Path
+from PIL import Image
+from diffusers import QwenImageEditPlusPipeline
+
+import overflowml
+from overflowml import MemoryGuard
+
+# 1. Detect hardware
+hw = overflowml.detect_hardware()
+print(hw.summary())
+
+# 2. Load model
+pipe = QwenImageEditPlusPipeline.from_pretrained(
+    "Qwen/Qwen-Image-Edit-2511",
+    torch_dtype=torch.bfloat16,
+    low_cpu_mem_usage=True,
+)
+
+# 3. Optimize — ONE LINE does everything
+strategy = overflowml.optimize_pipeline(pipe, model_size_gb=40)
+print(strategy.summary())
+
+# 4. Generate with memory guard (prevents VRAM fragmentation)
+face = Image.open("avatar.png").convert("RGB")
+guard = MemoryGuard(threshold=0.7, verbose=True)
+
+prompts = [
+    "Photograph of this woman, clean white studio, confident gaze, editorial headshot.",
+    "Photograph of this woman, outdoor golden hour, casual white linen shirt, natural smile.",
+    "Photograph of this woman, cozy cafe, holding ceramic coffee cup, relaxed expression.",
+]
+
+for i, prompt in enumerate(prompts):
+    with guard:
+        result = pipe(
+            image=[face],
+            prompt=f"[image 1] is the reference face. {prompt}",
+            negative_prompt="low quality, blurry, distorted",
+            true_cfg_scale=4.0,
+            guidance_scale=1.0,
+            num_inference_steps=25,
+            width=832,
+            height=1040,
+            generator=torch.manual_seed(42),
+        )
+        result.images[0].save(f"output_{i+1}.png")
+        print(f"Image {i+1} saved")

overflowml-0.1.0/overflowml/__init__.py

@@ -0,0 +1,16 @@
+"""OverflowML — Run AI models larger than your GPU."""
+
+__version__ = "0.1.0"
+
+from .detect import detect_hardware, HardwareProfile
+from .strategy import pick_strategy, Strategy
+from .optimize import optimize_pipeline, optimize_model
+
+__all__ = [
+    "detect_hardware",
+    "HardwareProfile",
+    "pick_strategy",
+    "Strategy",
+    "optimize_pipeline",
+    "optimize_model",
+]

overflowml-0.1.0/overflowml/cli.py

@@ -0,0 +1,75 @@
+"""CLI tool for hardware detection and strategy recommendation."""
+
+import argparse
+import logging
+import sys
+
+if sys.platform == "win32":
+    sys.stdout.reconfigure(encoding="utf-8", errors="replace")
+    sys.stderr.reconfigure(encoding="utf-8", errors="replace")
+
+from .detect import detect_hardware
+from .strategy import pick_strategy
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        prog="overflowml",
+        description="OverflowML — Run AI models larger than your GPU",
+    )
+    sub = parser.add_subparsers(dest="command")
+
+    # --- detect
+    sub.add_parser("detect", help="Detect hardware and show capabilities")
+
+    # --- plan
+    plan = sub.add_parser("plan", help="Plan optimal loading strategy for a model")
+    plan.add_argument("model_size", type=float, help="Model size in GB (BF16 weights)")
+    plan.add_argument("--fast", action="store_true", help="Prefer speed over VRAM savings")
+    plan.add_argument("--no-quantize", action="store_true", help="Disable quantization")
+
+    args = parser.parse_args()
+
+    logging.basicConfig(level=logging.INFO, format="%(message)s")
+
+    if args.command == "detect":
+        hw = detect_hardware()
+        print("\n=== OverflowML Hardware Detection ===")
+        print(hw.summary())
+        print(f"\nFor a model that needs loading, run:")
+        print(f"  overflowml plan <size_in_gb>")
+        print()
+
+    elif args.command == "plan":
+        hw = detect_hardware()
+        print("\n=== Hardware ===")
+        print(hw.summary())
+        print()
+
+        strategy = pick_strategy(
+            hw, args.model_size,
+            prefer_speed=args.fast,
+            allow_quantization=not args.no_quantize,
+        )
+        print(f"=== Strategy for {args.model_size:.0f}GB model ===")
+        print(strategy.summary())
+        print()
+
+        # Show code example
+        print("=== Usage ===")
+        print("```python")
+        print("import overflowml")
+        print("from diffusers import SomePipeline")
+        print()
+        print('pipe = SomePipeline.from_pretrained("model", torch_dtype=torch.bfloat16)')
+        print(f"strategy = overflowml.optimize_pipeline(pipe, model_size_gb={args.model_size})")
+        print("result = pipe(prompt)")
+        print("```")
+        print()
+
+    else:
+        parser.print_help()
+
+
+if __name__ == "__main__":
+    main()