moe-l2 0.2.0__tar.gz

moe_l2-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 yalund
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

moe_l2-0.2.0/PKG-INFO

@@ -0,0 +1,192 @@
+Metadata-Version: 2.2
+Name: moe-l2
+Version: 0.2.0
+Summary: MoE inference L2 hot-cache scheduler — run large MoE models on consumer GPUs
+License: MIT
+Project-URL: Homepage, https://github.com/yalund/moe-l2
+Project-URL: Source, https://github.com/yalund/moe-l2
+Keywords: moe,mixture-of-experts,llm,inference,caching,ollama,llama.cpp
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: System :: Hardware :: Hardware Drivers
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: httpx>=0.27
+Requires-Dist: gguf>=0.10
+Provides-Extra: predictor
+Requires-Dist: sentence-transformers>=3.0; extra == "predictor"
+
+# moe-l2
+
+**Run large MoE models on consumer GPUs.** A transparent proxy that predicts which experts your prompt needs, preloads them into a shared-memory LRU cache, so you can run 16 GB+ models on 8 GB GPUs.
+
+## How it works
+
+MoE models have many "experts" but only activate a few per token. moe-l2 predicts your prompt's domain (codegen, math, chinese_tech, etc.) and preloads the relevant experts into an mmap'd LRU cache before they're needed.
+
+```
+user → moe-l2 proxy (localhost:11435)
+    ├── predict domain
+    ├── preload domain experts → /dev/shm/moe_l2/
+    └── forward to ollama (localhost:11434)
+```
+
+## Quick start
+
+```bash
+pip install moe-l2                   # keyword-only predictor (zero extra deps)
+pip install moe-l2[predictor]        # hybrid: keyword + semantic embedding
+moe-l2 start --model model.gguf --l2-size 4GB
+```
+
+## Usage
+
+### 1. L2 proxy (recommended)
+
+Start the transparent proxy — sits between your client and ollama:
+
+```bash
+moe-l2 start --model /models/DeepSeek-V2-Lite.Q4_K_M.gguf --l2-size 4GB
+```
+
+All default ollama tools work through it (curl, open-webui, langchain):
+
+```bash
+# streaming
+curl http://localhost:11435/api/chat -d '{
+  "model":"qwen3:4b",
+  "messages":[{"role":"user","content":"write a Python script"}],
+  "stream":true
+}'
+
+# blocking
+curl http://localhost:11435/api/chat -d '{
+  "model":"qwen3:4b",
+  "messages":[{"role":"user","content":"hello"}],
+  "stream":false
+}'
+```
+
+### 2. Monitor cache stats
+
+```bash
+moe-l2 stats --port 11435
+```
+
+Example output:
+```
+moe-l2 cache stats
+  requests:     47
+  hits:         42     (89.4%)
+  misses:        5
+  slots_used:  32/48  (66.7%)
+  memory:     456 MB  (68.3% of 668 MB)
+```
+
+### 3. Use as a library
+
+```python
+from moe_l2 import predict, predict_hybrid, domain_to_expert_ids
+from moe_l2.cache import L2Cache
+
+# Predict domain (zero-dependency mode)
+domain = predict("print hello world")  # → "codegen"
+
+# Hybrid mode (falls back to semantic embedding)
+from moe_l2 import enable_semantic
+enable_semantic()
+domain = predict_hybrid("deploy nginx on ubuntu")  # → "chinese_tech"
+
+# Setup L2 cache
+cache = L2Cache(
+    model_path="/models/model.gguf",
+    slots_per_layer=48  # auto-calculated from --l2-size
+)
+expert_map = load_mapping()
+cache.preload_domain("codegen", expert_map)
+
+# Stats
+cache.stats()  # → {"hits": ..., "misses": ..., ...}
+```
+
+## Supported domains
+
+| Domain | Examples |
+|--------|---------|
+| `codegen` | Python, JS, bash, API design |
+| `debug` | error logs, stack traces, crash analysis |
+| `math` | algebra, calculus, equations |
+| `logic` | reasoning, puzzles, proofs |
+| `general_qa` | general knowledge, facts, explanations |
+| `chinese_tech` | 中文技术内容, NAS, 部署, 教程 |
+| `creative_write` | storytelling, poetry, marketing |
+| `translate` | translation between languages |
+
+## Architecture
+
+```
+┌────────────────────────────────────────────────┐
+│         Ollama Client (user-facing)             │
+│    curl / open-webui / langchain / any tool     │
+└──────────┬─────────────────────────────────────┘
+           │ POST to :11435
+┌──────────▼─────────────────────────────────────┐
+│            moe-l2 proxy                         │
+│  ┌─────────────────────────────────────────┐    │
+│  │ Domain Predictor                        │    │
+│  │  ┌──────────┐   ┌───────────────────┐   │    │
+│  │  │ Keywords │ → │ Semantic (opt)    │   │    │
+│  │  │ ~210词    │   │ all-MiniLM-L6-v2 │   │    │
+│  │  └──────────┘   └───────────────────┘   │    │
+│  │         ↓ domain                          │    │
+│  │  ┌──────────────────────────────────┐    │    │
+│  │  │ L2 Cache (LRU + mmap /dev/shm/) │    │    │
+│  │  │  preload domain experts async   │    │    │
+│  │  └──────────────────────────────────┘    │    │
+│  └─────────────────────────────────────────┘    │
+│           │ POST to :11434 (transparent)        │
+└──────────┬─────────────────────────────────────┘
+           │
+┌──────────▼─────────────────────────────────────┐
+│           Ollama / llama.cpp                    │
+│    MoE inference with hot-cached experts        │
+└────────────────────────────────────────────────┘
+```
+
+## CLI reference
+
+| Command | Description |
+|---------|-------------|
+| `moe-l2 start --model <path> --l2-size <size>` | Start proxy + cache |
+| `moe-l2 stats --port <port>` | Show live cache stats |
+| `moe-l2 stop --port <port>` | Stop proxy |
+
+Options:
+- `--model auto`: scan `/opt/data/models/*.gguf`
+- `--l2-size 4GB` / `--l2-size 512MB`: target cache size
+- `--port 11435` (default)
+
+## Project status
+
+**Phase 2** — core components complete (2026-07-30):
+- ✅ Domain predictor (keyword + optional semantic)
+- ✅ L2 cache (mmap LRU, thread-safe, async preload)
+- ✅ GGUF weight reader (direct memmap from .gguf)
+- ✅ Transparent proxy (HTTP/SSE forwarding, predict+preload)
+- ✅ CLI (start/stats with auto model detection)
+- ✅ GPU end-to-end pipeline verified (DS-V2-Lite on RTX 4090 24GB, Dec 2026)
+- 🔲 llama.cpp C++ integration (direct mmap from L2 cache) — Phase 3
+- 🔲 PyPI v0.2.0 release
+
+## License
+
+MIT

moe_l2-0.2.0/README.md

@@ -0,0 +1,165 @@
+# moe-l2
+
+**Run large MoE models on consumer GPUs.** A transparent proxy that predicts which experts your prompt needs, preloads them into a shared-memory LRU cache, so you can run 16 GB+ models on 8 GB GPUs.
+
+## How it works
+
+MoE models have many "experts" but only activate a few per token. moe-l2 predicts your prompt's domain (codegen, math, chinese_tech, etc.) and preloads the relevant experts into an mmap'd LRU cache before they're needed.
+
+```
+user → moe-l2 proxy (localhost:11435)
+    ├── predict domain
+    ├── preload domain experts → /dev/shm/moe_l2/
+    └── forward to ollama (localhost:11434)
+```
+
+## Quick start
+
+```bash
+pip install moe-l2                   # keyword-only predictor (zero extra deps)
+pip install moe-l2[predictor]        # hybrid: keyword + semantic embedding
+moe-l2 start --model model.gguf --l2-size 4GB
+```
+
+## Usage
+
+### 1. L2 proxy (recommended)
+
+Start the transparent proxy — sits between your client and ollama:
+
+```bash
+moe-l2 start --model /models/DeepSeek-V2-Lite.Q4_K_M.gguf --l2-size 4GB
+```
+
+All default ollama tools work through it (curl, open-webui, langchain):
+
+```bash
+# streaming
+curl http://localhost:11435/api/chat -d '{
+  "model":"qwen3:4b",
+  "messages":[{"role":"user","content":"write a Python script"}],
+  "stream":true
+}'
+
+# blocking
+curl http://localhost:11435/api/chat -d '{
+  "model":"qwen3:4b",
+  "messages":[{"role":"user","content":"hello"}],
+  "stream":false
+}'
+```
+
+### 2. Monitor cache stats
+
+```bash
+moe-l2 stats --port 11435
+```
+
+Example output:
+```
+moe-l2 cache stats
+  requests:     47
+  hits:         42     (89.4%)
+  misses:        5
+  slots_used:  32/48  (66.7%)
+  memory:     456 MB  (68.3% of 668 MB)
+```
+
+### 3. Use as a library
+
+```python
+from moe_l2 import predict, predict_hybrid, domain_to_expert_ids
+from moe_l2.cache import L2Cache
+
+# Predict domain (zero-dependency mode)
+domain = predict("print hello world")  # → "codegen"
+
+# Hybrid mode (falls back to semantic embedding)
+from moe_l2 import enable_semantic
+enable_semantic()
+domain = predict_hybrid("deploy nginx on ubuntu")  # → "chinese_tech"
+
+# Setup L2 cache
+cache = L2Cache(
+    model_path="/models/model.gguf",
+    slots_per_layer=48  # auto-calculated from --l2-size
+)
+expert_map = load_mapping()
+cache.preload_domain("codegen", expert_map)
+
+# Stats
+cache.stats()  # → {"hits": ..., "misses": ..., ...}
+```
+
+## Supported domains
+
+| Domain | Examples |
+|--------|---------|
+| `codegen` | Python, JS, bash, API design |
+| `debug` | error logs, stack traces, crash analysis |
+| `math` | algebra, calculus, equations |
+| `logic` | reasoning, puzzles, proofs |
+| `general_qa` | general knowledge, facts, explanations |
+| `chinese_tech` | 中文技术内容, NAS, 部署, 教程 |
+| `creative_write` | storytelling, poetry, marketing |
+| `translate` | translation between languages |
+
+## Architecture
+
+```
+┌────────────────────────────────────────────────┐
+│         Ollama Client (user-facing)             │
+│    curl / open-webui / langchain / any tool     │
+└──────────┬─────────────────────────────────────┘
+           │ POST to :11435
+┌──────────▼─────────────────────────────────────┐
+│            moe-l2 proxy                         │
+│  ┌─────────────────────────────────────────┐    │
+│  │ Domain Predictor                        │    │
+│  │  ┌──────────┐   ┌───────────────────┐   │    │
+│  │  │ Keywords │ → │ Semantic (opt)    │   │    │
+│  │  │ ~210词    │   │ all-MiniLM-L6-v2 │   │    │
+│  │  └──────────┘   └───────────────────┘   │    │
+│  │         ↓ domain                          │    │
+│  │  ┌──────────────────────────────────┐    │    │
+│  │  │ L2 Cache (LRU + mmap /dev/shm/) │    │    │
+│  │  │  preload domain experts async   │    │    │
+│  │  └──────────────────────────────────┘    │    │
+│  └─────────────────────────────────────────┘    │
+│           │ POST to :11434 (transparent)        │
+└──────────┬─────────────────────────────────────┘
+           │
+┌──────────▼─────────────────────────────────────┐
+│           Ollama / llama.cpp                    │
+│    MoE inference with hot-cached experts        │
+└────────────────────────────────────────────────┘
+```
+
+## CLI reference
+
+| Command | Description |
+|---------|-------------|
+| `moe-l2 start --model <path> --l2-size <size>` | Start proxy + cache |
+| `moe-l2 stats --port <port>` | Show live cache stats |
+| `moe-l2 stop --port <port>` | Stop proxy |
+
+Options:
+- `--model auto`: scan `/opt/data/models/*.gguf`
+- `--l2-size 4GB` / `--l2-size 512MB`: target cache size
+- `--port 11435` (default)
+
+## Project status
+
+**Phase 2** — core components complete (2026-07-30):
+- ✅ Domain predictor (keyword + optional semantic)
+- ✅ L2 cache (mmap LRU, thread-safe, async preload)
+- ✅ GGUF weight reader (direct memmap from .gguf)
+- ✅ Transparent proxy (HTTP/SSE forwarding, predict+preload)
+- ✅ CLI (start/stats with auto model detection)
+- ✅ GPU end-to-end pipeline verified (DS-V2-Lite on RTX 4090 24GB, Dec 2026)
+- 🔲 llama.cpp C++ integration (direct mmap from L2 cache) — Phase 3
+- 🔲 PyPI v0.2.0 release
+
+## License
+
+MIT

moe_l2-0.2.0/moe_l2/__init__.py

@@ -0,0 +1,28 @@
+"""moe-l2: MoE inference L2 hot-cache scheduler."""
+
+from .predictor import (
+    DOMAINS,
+    domain_to_expert_ids,
+    enable_semantic,
+    get_backbone_experts,
+    get_layer_specificity,
+    get_preload_set,
+    is_semantic_available,
+    load_mapping,
+    predict,
+    predict_hybrid,
+)
+
+__version__ = "0.2.0"
+__all__ = [
+    "DOMAINS",
+    "load_mapping",
+    "predict",
+    "predict_hybrid",
+    "domain_to_expert_ids",
+    "get_preload_set",
+    "get_backbone_experts",
+    "get_layer_specificity",
+    "enable_semantic",
+    "is_semantic_available",
+]