kandiga 0.1.0__tar.gz

kandiga-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kandiga Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

kandiga-0.1.0/PKG-INFO

@@ -0,0 +1,17 @@
+Metadata-Version: 2.4
+Name: kandiga
+Version: 0.1.0
+Summary: Run 35B AI models in 1.5GB of RAM. Any Mac.
+License: MIT
+Requires-Python: >=3.9
+License-File: LICENSE
+Requires-Dist: mlx>=0.30.0
+Requires-Dist: mlx-lm>=0.24.0
+Requires-Dist: numpy
+Requires-Dist: rich
+Provides-Extra: serve
+Requires-Dist: fastapi; extra == "serve"
+Requires-Dist: uvicorn[standard]; extra == "serve"
+Provides-Extra: tools
+Requires-Dist: duckduckgo-search; extra == "tools"
+Dynamic: license-file

kandiga-0.1.0/README.md

@@ -0,0 +1,153 @@
+# Kandiga
+
+Run 35B AI models in 1.5GB of RAM. Any Mac.
+
+Kandiga is an open-source MoE inference engine that uses **Selective Expert Materialization** to run models that would normally require 20GB+ of memory in under 2GB on any Apple Silicon Mac.
+
+## How it works
+
+Large MoE (Mixture of Experts) models like Qwen3.5-35B-A3B have 256 experts per layer, but only activate 8 per token. Kandiga exploits this sparsity:
+
+1. **Shared layers** (attention, norms, embeddings) load to GPU memory (~1.5GB)
+2. **Expert MLP weights** stay on disk in packed binary files (~17GB SSD)
+3. **Per token**: the router selects 8 experts, which are read from SSD via `pread`
+4. **CPU computes** expert MLP with NEON-vectorized 4-bit dequant + GCD parallelism
+5. **GPU computes** attention simultaneously via MLX (unified memory, zero copy)
+
+This is the [KTransformers](https://github.com/kvcache-ai/ktransformers) architecture adapted for Apple Silicon's unified memory.
+
+## Install
+
+```bash
+pip install kandiga
+```
+
+Requirements: macOS with Apple Silicon (M1/M2/M3/M4), Python 3.10+
+
+## Quick start
+
+```bash
+# One-time setup: download model + prepare expert files (~20 min)
+kandiga setup
+
+# Interactive chat
+kandiga chat
+
+# Fast mode (K=4 experts instead of 8, ~2x speed, slightly less quality)
+kandiga chat --fast
+
+# One-shot prompt
+kandiga "What is the capital of France?"
+
+# Start an OpenAI-compatible API server
+kandiga serve
+
+# Run benchmarks
+kandiga bench
+```
+
+## Benchmarks
+
+Measured on M4 Mac Mini (16GB), Qwen3.5-35B-A3B-4bit:
+
+| Mode | Experts | Speed | RAM | Quality |
+|------|---------|-------|-----|---------|
+| Quality (K=8) | 8/256 per layer | ~3.5 tok/s | 1.5GB | Full |
+| Fast (K=4) | 4/256 per layer | ~6.5 tok/s | 1.5GB | Near-equal |
+
+For comparison, loading the full model requires 20.4GB of RAM and MLX alone achieves ~25 tok/s when it fits in memory. Kandiga trades speed for accessibility: if your Mac has 8-16GB of RAM, you can now run a 35B model that previously required 24GB+.
+
+## Architecture
+
+```
+User prompt
+    |
+    v
+[Tokenizer + Chat Template]
+    |
+    v
+[MLX Forward Pass]
+    |
+    +---> GPU: Attention + Norms + Router + Shared Expert + Blending
+    |
+    +---> CPU: Routed Expert MLP (NEON 4-bit dequant + GCD parallel)
+    |         |
+    |         +-- pread expert weights from SSD (OS page cache)
+    |         +-- gate_proj matvec (512x2048)
+    |         +-- up_proj matvec (512x2048)
+    |         +-- SwiGLU activation
+    |         +-- down_proj matvec (2048x512)
+    |
+    v
+[Token Output]
+```
+
+Both CPU and GPU operate on the same physical DRAM (Apple Silicon unified memory), so there is zero data transfer overhead between them.
+
+## API Server
+
+Kandiga includes an OpenAI-compatible HTTP API:
+
+```bash
+kandiga serve --port 8340
+```
+
+```python
+import openai
+
+client = openai.OpenAI(base_url="http://localhost:8340/v1", api_key="unused")
+response = client.chat.completions.create(
+    model="mlx-community/Qwen3.5-35B-A3B-4bit",
+    messages=[{"role": "user", "content": "Hello!"}],
+    stream=True,
+)
+for chunk in response:
+    print(chunk.choices[0].delta.content or "", end="")
+```
+
+## Project structure
+
+```
+kandiga/
+  __init__.py          # Package version
+  cli.py               # CLI entry point (argparse)
+  engine.py            # Core inference engine (SEM)
+  chat.py              # Interactive chat (Rich terminal UI)
+  serve.py             # OpenAI-compatible HTTP API (FastAPI)
+  bench.py             # Benchmarking suite
+  setup.py             # Model download + expert splitting + packing
+  _split_experts.py    # Split stacked weights into per-expert files
+  _pack_experts.py     # Pack per-expert files into binary format
+  _build.py            # Compile CPU expert dylib from source
+  metal/
+    kandiga_cpu_expert.h   # C API header
+    kandiga_cpu_expert.m   # NEON + GCD implementation
+    Makefile               # Build the dylib
+  tools/
+    __init__.py            # Future: web search, file access
+scripts/
+  install.sh           # Quick install script
+tests/
+  ...
+```
+
+## Development
+
+```bash
+# Clone
+git clone https://github.com/yourusername/kandiga.git
+cd kandiga
+
+# Install in development mode
+pip install -e ".[serve]"
+
+# Build the CPU expert library
+cd kandiga/metal && make && cd ../..
+
+# Run tests
+pytest tests/ -v
+```
+
+## License
+
+MIT

kandiga-0.1.0/kandiga/__init__.py

@@ -0,0 +1,2 @@
+"""Kandiga: Run 35B AI models in 1.5GB of RAM."""
+__version__ = "0.1.0"

kandiga-0.1.0/kandiga/__main__.py

@@ -0,0 +1,4 @@
+"""Allow running kandiga as a module: python -m kandiga"""
+from kandiga.cli import main
+
+main()

kandiga-0.1.0/kandiga/_build.py

@@ -0,0 +1,48 @@
+"""Build the CPU expert dylib from source."""
+
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+
+
+def build_cpu_expert_dylib() -> str:
+    """Compile libkandiga_cpu_expert.dylib from the Objective-C source.
+
+    Returns the path to the built dylib.
+    """
+    metal_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "metal")
+    source = os.path.join(metal_dir, "kandiga_cpu_expert.m")
+    header = os.path.join(metal_dir, "kandiga_cpu_expert.h")
+    dylib = os.path.join(metal_dir, "libkandiga_cpu_expert.dylib")
+
+    if not os.path.exists(source):
+        raise FileNotFoundError(f"Source file not found: {source}")
+
+    # Build command
+    cmd = [
+        "clang",
+        "-shared",
+        "-o", dylib,
+        source,
+        "-fobjc-arc",
+        "-framework", "Foundation",
+        "-O2",
+        "-march=native",
+    ]
+
+    print(f"  Building: {' '.join(cmd)}")
+    result = subprocess.run(cmd, capture_output=True, text=True)
+
+    if result.returncode != 0:
+        raise RuntimeError(
+            f"Compilation failed (exit {result.returncode}):\n"
+            f"stdout: {result.stdout}\n"
+            f"stderr: {result.stderr}"
+        )
+
+    if not os.path.exists(dylib):
+        raise RuntimeError(f"Build succeeded but dylib not found at {dylib}")
+
+    return dylib

kandiga-0.1.0/kandiga/_pack_experts.py

@@ -0,0 +1,167 @@
+"""Pack per-expert safetensors into single raw binary files per layer.
+
+Reads the split per-expert safetensors files and packs them into a compact
+binary format that can be read with zero parsing overhead using pread().
+
+Binary format per file:
+  Header (4096 bytes):
+    magic:       4 bytes  "BKEX"
+    version:     uint32   1
+    num_experts: uint32   256
+    expert_size: uint64   1769472 (bytes per expert block)
+    num_tensors: uint32   9
+    tensor descriptors...
+    padding to 4096 bytes
+
+  Expert data (256 x 1769472 bytes):
+    expert_000: [gate.weight][gate.scales][gate.biases]
+                [up.weight][up.scales][up.biases]
+                [down.weight][down.scales][down.biases]
+    ...
+    expert_255: same layout
+"""
+
+from __future__ import annotations
+
+import os
+import struct
+import time
+
+import mlx.core as mx
+import numpy as np
+
+NUM_EXPERTS = 256
+NUM_MOE_LAYERS = 40
+HEADER_SIZE = 4096
+EXPERT_SIZE = 1_769_472  # 1728KB exactly
+
+# Tensor order must match the C library's byte offsets
+TENSOR_ORDER = [
+    ("gate_proj.weight", (512, 256), "uint32"),
+    ("gate_proj.scales", (512, 32), "bfloat16"),
+    ("gate_proj.biases", (512, 32), "bfloat16"),
+    ("up_proj.weight", (512, 256), "uint32"),
+    ("up_proj.scales", (512, 32), "bfloat16"),
+    ("up_proj.biases", (512, 32), "bfloat16"),
+    ("down_proj.weight", (2048, 64), "uint32"),
+    ("down_proj.scales", (2048, 8), "bfloat16"),
+    ("down_proj.biases", (2048, 8), "bfloat16"),
+]
+
+
+def _tensor_nbytes(shape: tuple, dtype_str: str) -> int:
+    """Calculate raw byte size for a tensor."""
+    itemsize = 4 if dtype_str == "uint32" else 2
+    return shape[0] * shape[1] * itemsize
+
+
+def _build_header() -> bytes:
+    """Build the 4096-byte binary header."""
+    buf = bytearray(HEADER_SIZE)
+
+    buf[0:4] = b"BKEX"
+    struct.pack_into("<I", buf, 4, 1)  # version
+    struct.pack_into("<I", buf, 8, NUM_EXPERTS)
+    struct.pack_into("<Q", buf, 12, EXPERT_SIZE)
+    struct.pack_into("<I", buf, 20, len(TENSOR_ORDER))
+
+    offset_in_expert = 0
+    pos = 24
+    for name, shape, dtype_str in TENSOR_ORDER:
+        nbytes = _tensor_nbytes(shape, dtype_str)
+        dtype_code = 0 if dtype_str == "uint32" else 1
+
+        name_bytes = name.encode("ascii")
+        struct.pack_into("<B", buf, pos, len(name_bytes))
+        pos += 1
+        buf[pos: pos + len(name_bytes)] = name_bytes
+        pos += 24
+        struct.pack_into("<I", buf, pos, offset_in_expert)
+        pos += 4
+        struct.pack_into("<I", buf, pos, nbytes)
+        pos += 4
+        struct.pack_into("<I", buf, pos, shape[0])
+        pos += 4
+        struct.pack_into("<I", buf, pos, shape[1])
+        pos += 4
+        struct.pack_into("<B", buf, pos, dtype_code)
+        pos += 1
+
+        offset_in_expert += nbytes
+
+    assert offset_in_expert == EXPERT_SIZE, (
+        f"Tensor sizes don't sum to EXPERT_SIZE: {offset_in_expert} != {EXPERT_SIZE}"
+    )
+    return bytes(buf)
+
+
+def _expert_to_bytes(tensors: dict[str, mx.array]) -> bytes:
+    """Convert an expert's tensor dict to raw bytes in canonical order."""
+    parts = []
+    for name, shape, dtype_str in TENSOR_ORDER:
+        tensor = tensors[name]
+        mx.eval(tensor)
+
+        if dtype_str == "uint32":
+            np_arr = np.array(tensor, copy=False)
+            raw = np_arr.tobytes()
+        else:
+            u16 = tensor.view(mx.uint16)
+            mx.eval(u16)
+            np_arr = np.array(u16, copy=False)
+            raw = np_arr.tobytes()
+
+        expected = _tensor_nbytes(shape, dtype_str)
+        assert len(raw) == expected, f"{name}: got {len(raw)} bytes, expected {expected}"
+        parts.append(raw)
+
+    data = b"".join(parts)
+    assert len(data) == EXPERT_SIZE, f"Expert data {len(data)} != {EXPERT_SIZE}"
+    return data
+
+
+def _pack_layer(layer_idx: int, input_dir: str, output_dir: str) -> None:
+    """Pack all 256 experts for one layer into a single binary file."""
+    layer_dir = os.path.join(input_dir, f"layer_{layer_idx:02d}")
+    out_path = os.path.join(output_dir, f"layer_{layer_idx:02d}.bin")
+
+    header = _build_header()
+
+    with open(out_path, "wb") as f:
+        f.write(header)
+        for expert_idx in range(NUM_EXPERTS):
+            st_path = os.path.join(layer_dir, f"expert_{expert_idx:03d}.safetensors")
+            tensors = mx.load(st_path)
+            mx.eval(*tensors.values())
+            raw = _expert_to_bytes(tensors)
+            f.write(raw)
+            del tensors
+
+    # Verify file size
+    expected_size = HEADER_SIZE + NUM_EXPERTS * EXPERT_SIZE
+    actual_size = os.path.getsize(out_path)
+    assert actual_size == expected_size, (
+        f"File size mismatch: {actual_size} != {expected_size}"
+    )
+
+
+def pack_experts(
+    input_dir: str,
+    output_dir: str,
+    num_layers: int = NUM_MOE_LAYERS,
+) -> None:
+    """Pack all layers from split expert files into binary format."""
+    os.makedirs(output_dir, exist_ok=True)
+
+    total_start = time.time()
+    for layer_idx in range(num_layers):
+        layer_start = time.time()
+        print(f"  Packing layer {layer_idx:2d}/{num_layers - 1}...", end=" ", flush=True)
+        _pack_layer(layer_idx, input_dir, output_dir)
+        elapsed = time.time() - layer_start
+        total_elapsed = time.time() - total_start
+        eta = (total_elapsed / (layer_idx + 1)) * (num_layers - layer_idx - 1)
+        print(f"done ({elapsed:.1f}s, ETA {eta:.0f}s)")
+
+    total_elapsed = time.time() - total_start
+    print(f"  {num_layers} layer files packed in {total_elapsed:.1f}s")

kandiga-0.1.0/kandiga/_split_experts.py

@@ -0,0 +1,120 @@
+"""Split stacked MoE expert weights into per-expert safetensors files.
+
+Reads the downloaded Qwen3.5-35B-A3B-4bit model's stacked expert weights
+(256 experts per layer, 40 MoE layers) and splits them into individual
+per-expert files for selective loading.
+
+Output structure:
+  ~/.kandiga/experts/Qwen3.5-35B-A3B-4bit/
+    layer_00/
+      expert_000.safetensors
+      ...
+      expert_255.safetensors
+    layer_01/
+      ...
+    layer_39/
+      ...
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import time
+
+import mlx.core as mx
+
+WEIGHT_PREFIX = "language_model.model.layers"
+PROJECTIONS = ("gate_proj", "up_proj", "down_proj")
+COMPONENTS = ("weight", "scales", "biases")
+NUM_EXPERTS = 256
+NUM_MOE_LAYERS = 40
+
+
+def _build_weight_map(model_dir: str) -> dict[str, str]:
+    """Map tensor names to absolute shard file paths."""
+    index_file = os.path.join(model_dir, "model.safetensors.index.json")
+    if not os.path.exists(index_file):
+        raise FileNotFoundError(f"No index file found at {index_file}")
+    with open(index_file) as f:
+        data = json.load(f)
+    return {k: os.path.join(model_dir, v) for k, v in data["weight_map"].items()}
+
+
+def _split_layer(
+    layer_idx: int,
+    weight_map: dict[str, str],
+    output_dir: str,
+) -> None:
+    """Split one layer's stacked expert weights into per-expert files."""
+    layer_dir = os.path.join(output_dir, f"layer_{layer_idx:02d}")
+    os.makedirs(layer_dir, exist_ok=True)
+
+    prefix = f"{WEIGHT_PREFIX}.{layer_idx}.mlp.switch_mlp."
+
+    # Collect all 9 tensor keys for this layer's experts
+    tensor_keys = {}
+    for proj in PROJECTIONS:
+        for comp in COMPONENTS:
+            key = f"{prefix}{proj}.{comp}"
+            if key not in weight_map:
+                raise KeyError(f"Missing weight key: {key}")
+            tensor_keys[(proj, comp)] = key
+
+    # Group by shard file to minimize file opens
+    shards: dict[str, list[tuple[str, str, str]]] = {}
+    for (proj, comp), key in tensor_keys.items():
+        shard_file = weight_map[key]
+        shards.setdefault(shard_file, []).append((proj, comp, key))
+
+    # Load all stacked tensors for this layer
+    stacked: dict[tuple[str, str], mx.array] = {}
+    for shard_file, entries in shards.items():
+        shard_data = mx.load(shard_file)
+        for proj, comp, key in entries:
+            tensor = shard_data[key]
+            mx.eval(tensor)
+            stacked[(proj, comp)] = tensor
+        del shard_data
+
+    # Split and save per-expert files
+    for expert_idx in range(NUM_EXPERTS):
+        expert_tensors = {}
+        for proj in PROJECTIONS:
+            for comp in COMPONENTS:
+                full_tensor = stacked[(proj, comp)]
+                sliced = full_tensor[expert_idx]
+                mx.eval(sliced)
+                expert_tensors[f"{proj}.{comp}"] = sliced
+
+        out_path = os.path.join(layer_dir, f"expert_{expert_idx:03d}.safetensors")
+        mx.save_safetensors(out_path, expert_tensors)
+        del expert_tensors
+
+    del stacked
+
+
+def split_experts(
+    model_dir: str,
+    output_dir: str,
+    num_layers: int = NUM_MOE_LAYERS,
+) -> None:
+    """Split all layers' expert weights into per-expert files."""
+    os.makedirs(output_dir, exist_ok=True)
+
+    print(f"  Building weight map...")
+    weight_map = _build_weight_map(model_dir)
+
+    total_start = time.time()
+    for layer_idx in range(num_layers):
+        layer_start = time.time()
+        print(f"  Splitting layer {layer_idx:2d}/{num_layers - 1}...", end=" ", flush=True)
+        _split_layer(layer_idx, weight_map, output_dir)
+        elapsed = time.time() - layer_start
+        total_elapsed = time.time() - total_start
+        eta = (total_elapsed / (layer_idx + 1)) * (num_layers - layer_idx - 1)
+        print(f"done ({elapsed:.1f}s, ETA {eta:.0f}s)")
+
+    total_elapsed = time.time() - total_start
+    total_files = num_layers * NUM_EXPERTS
+    print(f"  {total_files:,} expert files created in {total_elapsed:.1f}s")

kandiga-0.1.0/kandiga/bench.py

@@ -0,0 +1,104 @@
+"""Benchmarking suite for Kandiga inference engine."""
+
+from __future__ import annotations
+
+import time
+
+from rich.console import Console
+from rich.table import Table
+
+console = Console()
+
+PROMPTS = [
+    ("Short", "What is 2+2?"),
+    ("Medium", "Explain how a transformer neural network works in 3 sentences."),
+    ("Long", "Write a detailed comparison of Python and Rust for systems programming. "
+     "Cover performance, safety, ecosystem, and learning curve."),
+]
+
+
+def _bench_one(engine, prompt: str, max_tokens: int = 256) -> dict:
+    """Benchmark a single prompt. Returns timing stats."""
+    # Warmup: ensure model is loaded
+    if not engine.is_ready:
+        engine.load()
+
+    # Time to first token
+    t_start = time.time()
+    tokens = []
+    t_first = None
+
+    for token in engine.generate(prompt, max_tokens=max_tokens, stream=True):
+        if t_first is None:
+            t_first = time.time()
+        tokens.append(token)
+
+    t_end = time.time()
+
+    total_time = t_end - t_start
+    ttft = (t_first - t_start) if t_first else total_time
+    gen_time = (t_end - t_first) if t_first else 0
+    num_tokens = len(tokens)
+    tps = num_tokens / gen_time if gen_time > 0 else 0
+
+    return {
+        "num_tokens": num_tokens,
+        "total_time": total_time,
+        "ttft": ttft,
+        "gen_time": gen_time,
+        "tps": tps,
+    }
+
+
+def run_bench():
+    """Run inference benchmarks and display results."""
+    console.print()
+    console.print("[bold cyan]Kandiga Benchmark[/]")
+    console.print()
+
+    from kandiga.engine import KandigaEngine
+
+    # Benchmark both modes
+    for mode_name, fast in [("Quality (K=8)", False), ("Fast (K=4)", True)]:
+        console.print(f"[bold]{mode_name}[/]")
+        console.print("[dim]Loading model...[/]")
+
+        engine = KandigaEngine(fast_mode=fast, log_memory=False)
+        engine.load()
+
+        stats = engine.stats
+        console.print(
+            f"[dim]RSS: {stats['rss_mb']:.0f}MB | "
+            f"GPU: {stats['gpu_active_mb']:.0f}MB[/]"
+        )
+        console.print()
+
+        table = Table(show_header=True, header_style="bold")
+        table.add_column("Prompt", width=12)
+        table.add_column("Tokens", justify="right")
+        table.add_column("TTFT", justify="right")
+        table.add_column("Gen Time", justify="right")
+        table.add_column("tok/s", justify="right", style="cyan")
+
+        for label, prompt in PROMPTS:
+            console.print(f"  Running: {label}...", end=" ")
+            result = _bench_one(engine, prompt, max_tokens=256)
+            console.print(f"[green]done[/]")
+
+            table.add_row(
+                label,
+                str(result["num_tokens"]),
+                f"{result['ttft']:.2f}s",
+                f"{result['gen_time']:.1f}s",
+                f"{result['tps']:.1f}",
+            )
+
+        console.print()
+        console.print(table)
+        console.print()
+
+        # Clean up
+        del engine
+
+    console.print("[bold green]Benchmark complete.[/]")
+    console.print()