coderouter-cli 2.7.5__tar.gz → 2.7.6__tar.gz

{coderouter_cli-2.7.5 → coderouter_cli-2.7.6}/CHANGELOG.md

@@ -9,6 +9,75 @@ are kept verbatim where the Japanese text itself is the subject).
 
 ---
 
+## [v2.7.6] — 2026-07-09 (Launcher MTP / speculative-decoding support)
+
+**PR #59 / #60**. Adds Multi-Token Prediction (MTP) / speculative-decoding
+support to the llama.cpp launcher (both the Web UI and the desktop GUI), closing the
+gap where operators had to hand-assemble `--spec-type` / `--model-draft`
+flags. Fully backward compatible: the new fields default to auto-detection
+with a silent no-op fallback, no config-schema changes, no new dependencies.
+
+### Added
+
+- **`resolve_speculative()` / `find_draft_companion()`**
+  (`coderouter/launcher_speculative.py`) — shared decision logic for both
+  launchers. `mtp_mode="auto"` (default) first checks the selected gguf's
+  own metadata for embedded nextn layers (`{arch}.nextn_predict_layers > 0`)
+  and, if present, emits `--spec-type draft-mtp` with no separate draft
+  model; otherwise it scans the *same folder* as the main gguf for a
+  companion draft/MTP file (name hint or shared prefix, size under 50% of
+  the main model, architecture-matched when readable) and wires up
+  `--spec-type draft-mtp|draft-simple --model-draft <path>`; otherwise it
+  starts normally and logs that no companion was found. An explicit
+  `draft_model_path` skips detection entirely (400 if the path doesn't
+  exist); `mtp_mode="off"` never emits speculative flags. When the operator
+  supplies `--spec-type` in extra args, auto-detection defers to it and
+  adds nothing.
+  Both knobs are llama.cpp-only — vllm/mlx reject them with a 400.
+- **`POST /api/launcher/start`**: new optional `draft_model_path` and
+  `mtp_mode` (`"auto"` / `"off"`) fields; the response now carries the
+  resolved flags as `"speculative"`. `-md` / `--model-draft` /
+  `--spec-draft-model` join the existing `-m`/`--model` denylist for
+  `options`/`extra_args` (llama.cpp) — the draft model can only be set via
+  `draft_model_path`. The remaining spec knobs (`--spec-type`,
+  `--spec-draft-n-max/-n-min/-p-min`, `-ngld`, `-devd`) stay free-form.
+- **GGUF introspection** (`coderouter/gguf_introspect.py`): `GGUFInfo` now
+  reports `n_nextn` and the derived `supports_mtp` property, read from
+  `{arch}.nextn_predict_layers`.
+- **`--split-mode tensor` crash warning**: known llama.cpp issue #24309 —
+  a nextn-embedded model combined with tensor split can crash. The launcher
+  detects the combination and logs a warning recommending `--split-mode
+  layer` without blocking the launch.
+- **UI**: Web LAUNCH form gets "MTP/draft gguf (空欄で自動検出)" text field
+  + "MTP" `auto`/`off` select. Desktop GUI (`launcher_gui.py`) gets the
+  matching Entry + a "MTP自動検出" checkbox (default on).
+- **Automatic MTP startup-crash fallback**: when speculative flags were added
+  by AUTO detection (`mtp_mode="auto"`, no explicit `draft_model_path`) and the
+  backend dies during startup (non-zero exit within ~3 min), the launcher
+  relaunches it ONCE without the speculative flags and logs
+  `[launcher] MTP startup failure detected (exit code ...); retrying without
+  speculative decoding` plus the flag-free command. This covers architectures
+  whose `draft-mtp` support in llama.cpp is still immature (detection is right
+  but the MTP context fails to initialize, e.g. `failed to measure MTP context
+  memory`). Explicit `draft_model_path` / operator-supplied `--spec-type` are
+  never auto-retried, and the retry can never loop (single-shot, guarded).
+  Implemented in both the Web launcher (`launcher_routes.py`) and the desktop
+  GUI (`launcher_gui.py`).
+
+### Docs
+
+- `docs/backends/launcher.md`/`.en.md`: new "MTP / speculative decoding"
+  section (auto-detection order, companion-file convention, `mtp_mode=off`,
+  extra-args deferral, `-md` denylist, issue #24309 caveat, API fields), plus
+  the LAUNCH form field table and denylist note updated.
+- `docs/backends/llamacpp-direct.md`/`.en.md`: manual `--spec-type
+  draft-mtp` / `--model-draft` / `--spec-draft-n-max` usage example, with a
+  pointer to the launcher's auto-detection.
+- `docs/backends/launcher-quickstart.md`/`.en.md`: one-line pointer to the
+  new MTP section.
+
+---
+
 ## [v2.7.5] — 2026-07-06 (External-bind startup warning + remote-access guide + docs i18n)
 
 Driven by the first field report against the v2.7.0 Host-validation

{coderouter_cli-2.7.5 → coderouter_cli-2.7.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coderouter-cli
-Version: 2.7.5
+Version: 2.7.6
 Summary: Local-first, free-first, fallback-built-in LLM router. Claude Code / OpenAI compatible.
 Project-URL: Homepage, https://github.com/zephel01/CodeRouter
 Project-URL: Repository, https://github.com/zephel01/CodeRouter

{coderouter_cli-2.7.5 → coderouter_cli-2.7.6}/coderouter/gguf_introspect.py

@@ -136,6 +136,17 @@ class GGUFInfo:
     n_kv_heads: int | None
     file_type: int | None
     file_size_bytes: int
+    n_nextn: int | None = None
+
+    @property
+    def supports_mtp(self) -> bool:
+        """True when the GGUF embeds Multi-Token-Prediction (nextn) layers.
+
+        Derived from ``{arch}.nextn_predict_layers`` — a positive count means
+        the main model carries MTP/nextn tensors and can drive llama.cpp's
+        ``--spec-type draft-mtp`` without a separate draft gguf.
+        """
+        return bool(self.n_nextn and self.n_nextn > 0)
 
     @property
     def quant_name(self) -> str | None:
@@ -224,6 +235,10 @@ _KEY_BLOCK_COUNT = ".block_count"
 _KEY_EMBED_LEN = ".embedding_length"
 _KEY_HEAD_COUNT = ".attention.head_count"
 _KEY_HEAD_COUNT_KV = ".attention.head_count_kv"
+# Number of Multi-Token-Prediction (nextn) layers embedded in the main model
+# (e.g. ``glm4moe.nextn_predict_layers``). A positive value means the GGUF can
+# drive llama.cpp speculative decoding via ``--spec-type draft-mtp`` alone.
+_KEY_NEXTN = ".nextn_predict_layers"
 
 
 def read_gguf_metadata(path: str | Path) -> GGUFInfo:
@@ -245,6 +260,7 @@ def read_gguf_metadata(path: str | Path) -> GGUFInfo:
     n_heads: int | None = None
     n_kv_heads: int | None = None
     file_type: int | None = None
+    n_nextn: int | None = None
 
     with p.open("rb") as fh:
         magic = fh.read(4)
@@ -275,6 +291,8 @@ def read_gguf_metadata(path: str | Path) -> GGUFInfo:
                 n_kv_heads = value
             elif key.endswith(_KEY_HEAD_COUNT) and isinstance(value, int):
                 n_heads = value
+            elif key.endswith(_KEY_NEXTN) and isinstance(value, int):
+                n_nextn = value
 
     return GGUFInfo(
         architecture=arch,
@@ -284,6 +302,7 @@ def read_gguf_metadata(path: str | Path) -> GGUFInfo:
         n_kv_heads=n_kv_heads,
         file_type=file_type,
         file_size_bytes=file_size,
+        n_nextn=n_nextn,
     )
 
 

{coderouter_cli-2.7.5 → coderouter_cli-2.7.6}/coderouter/ingress/launcher_routes.py

@@ -30,6 +30,7 @@ import secrets
 import shlex
 import shutil
 import subprocess
+import time
 import uuid
 from collections import deque
 from dataclasses import dataclass, field
@@ -40,6 +41,7 @@ from fastapi import APIRouter, HTTPException, Request
 from fastapi.responses import HTMLResponse
 from pydantic import BaseModel
 
+from coderouter.launcher_speculative import resolve_speculative
 from coderouter.logging import get_logger
 
 logger = get_logger(__name__)
@@ -115,9 +117,23 @@ class ManagedProcess:
     options: dict[str, Any]
     extra_args: str
     status: str = "starting"   # "starting" | "running" | "stopped" | "error"
+    # MTP / speculative-decoding controls (defaults keep existing call sites
+    # working). Recorded for introspection; the resolved flags live in the cmd.
+    draft_model_path: str | None = None
+    mtp_mode: str = "auto"
     pid: int | None = None
     returncode: int | None = None
     log_tail: deque = field(default_factory=lambda: deque(maxlen=200))
+    # MTP auto-fallback state. When the speculative flags were added by AUTO
+    # detection and the child dies during startup, the process is relaunched
+    # ONCE without them (some archs' draft-mtp support in llama.cpp is
+    # immature and crashes the context init). These fields make that possible
+    # and impossible to loop.
+    spec_tokens: list = field(default_factory=list)
+    spec_auto: bool = False
+    mtp_fallback_done: bool = False
+    fallback_cmd: list | None = None
+    started_at: float = 0.0
     # asyncio subprocess handle — not serialised
     _proc: Any = field(default=None, repr=False, compare=False)
 
@@ -208,8 +224,15 @@ _BACKEND_DEFAULTS: dict[str, str] = {
 #   - llama.cpp: ``-m`` / ``--model`` select the GGUF file.
 #   - vllm / mlx: ``--model`` selects the model; ``-m`` selects the python
 #     module that ``_build_cmd`` pins, so it must not be re-specified either.
+#   - llama.cpp: the draft/MTP model is likewise set only via the dedicated
+#     ``draft_model_path`` field, so its aliases (``-md`` / ``--model-draft``
+#     / ``--spec-draft-model``) are rejected in options / extra_args too. The
+#     remaining spec knobs (``--spec-type``, ``--spec-draft-n-max`` …) stay
+#     free-form.
 _MODEL_FLAGS: dict[str, frozenset[str]] = {
-    "llama.cpp": frozenset({"-m", "--model"}),
+    "llama.cpp": frozenset(
+        {"-m", "--model", "-md", "--model-draft", "--spec-draft-model"}
+    ),
     "vllm": frozenset({"-m", "--model"}),
     "mlx": frozenset({"-m", "--model"}),
 }
@@ -407,6 +430,24 @@ def _resolve_within_model_dirs(model_path: str, model_dirs: list[str]) -> Path:
     raise ValueError("model_path is not under any configured model_dirs.")
 
 
+def _option_tokens(options: dict[str, Any]) -> list[str]:
+    """Flatten an ``options`` dict into CLI tokens (``{flag: val}`` semantics).
+
+    Boolean values become a bare flag when true and vanish when false;
+    everything else becomes ``[flag, str(val)]``. Shared by ``_build_cmd`` and
+    ``api_start`` so the tokens fed to ``resolve_speculative`` match exactly
+    what lands on the command line (no double-parsing drift).
+    """
+    tokens: list[str] = []
+    for flag, val in options.items():
+        if isinstance(val, bool):
+            if val:
+                tokens.append(flag)
+        else:
+            tokens.extend([flag, str(val)])
+    return tokens
+
+
 def _build_cmd(
     backend: str,
     model_path: str,
@@ -414,17 +455,27 @@ def _build_cmd(
     options: dict[str, Any],
     extra_args: str,
     binary: str | None = None,
+    spec_tokens: list[str] | None = None,
 ) -> list[str]:
     """Build the CLI command list for the given backend and options.
 
     ``binary`` overrides the default executable (``llama-server`` /
     ``python``).  When None, the default is used and PATH resolution
     is left to the OS.
+
+    ``spec_tokens`` are pre-resolved speculative-decoding / MTP flags
+    (from :func:`coderouter.launcher_speculative.resolve_speculative`).
+    For llama.cpp they are appended right after the port args and BEFORE
+    the profile / extra args. They are trusted (the launcher, not the
+    caller, produced them) so they bypass the model-override guard even
+    though they may contain ``--model-draft``.
     """
     exe = _resolve_binary(backend, binary)
 
     if backend == "llama.cpp":
         cmd: list[str] = [exe, "-m", model_path, "--port", str(port)]
+        if spec_tokens:
+            cmd.extend(spec_tokens)
     elif backend == "vllm":
         cmd = [
             exe, "-m", "vllm.entrypoints.openai.api_server",
@@ -446,12 +497,7 @@ def _build_cmd(
     # H8: reject model-path re-specification via options keys.
     _assert_no_model_override(backend, list(options.keys()))
 
-    for flag, val in options.items():
-        if isinstance(val, bool):
-            if val:
-                cmd.append(flag)
-        else:
-            cmd.extend([flag, str(val)])
+    cmd.extend(_option_tokens(options))
 
     if extra_args.strip():
         extra_tokens = shlex.split(extra_args)
@@ -474,12 +520,37 @@ def _build_cmd(
 # recover by reading the buffered chunk and continuing so log tailing survives.
 _LOG_STREAM_LIMIT = 256 * 1024  # 256 KB
 
+# Window (seconds) after spawn in which a non-zero exit is treated as a
+# startup crash eligible for the MTP auto-fallback. A backend that ran fine
+# for longer and then died is a normal crash, not a load-time failure, so it
+# is never retried.
+_MTP_FALLBACK_WINDOW_SECS = 180.0
 
-async def _tail_logs(proc: ManagedProcess) -> None:
-    """Read stdout+stderr into proc.log_tail until the process exits."""
+
+def _should_mtp_fallback(proc: ManagedProcess) -> bool:
+    """True when a just-crashed process qualifies for the one-shot MTP retry.
+
+    Only auto-detected speculative launches that die within the startup
+    window are eligible, and only once (guarded by ``mtp_fallback_done``).
+    """
     p = proc._proc
-    if p is None:
-        return
+    rc = p.returncode if p is not None else proc.returncode
+    return (
+        proc.spec_auto
+        and not proc.mtp_fallback_done
+        and bool(proc.fallback_cmd)
+        and rc not in (0, None)
+        and (time.monotonic() - proc.started_at) <= _MTP_FALLBACK_WINDOW_SECS
+    )
+
+
+async def _tail_logs(proc: ManagedProcess) -> None:
+    """Read stdout+stderr into proc.log_tail until the process exits.
+
+    Loops so that a one-shot MTP fallback (relaunch without the auto-detected
+    speculative flags) can be tailed by the same task. The loop exits as soon
+    as no fallback applies.
+    """
 
     async def _drain(stream: asyncio.StreamReader | None) -> None:
         if stream is None:
@@ -505,14 +576,53 @@ async def _tail_logs(proc: ManagedProcess) -> None:
                 break
             proc.log_tail.append(line.decode(errors="replace").rstrip())
 
-    await asyncio.gather(_drain(p.stdout), _drain(p.stderr))
-    await p.wait()
-    proc.returncode = p.returncode
-    proc.pid = None
-    proc.status = "stopped" if (p.returncode or 0) == 0 else "error"
-    proc.log_tail.append(
-        f"[launcher] process exited with code {p.returncode}"
-    )
+    while True:
+        p = proc._proc
+        if p is None:
+            return
+
+        await asyncio.gather(_drain(p.stdout), _drain(p.stderr))
+        await p.wait()
+        proc.returncode = p.returncode
+        proc.pid = None
+        proc.status = "stopped" if (p.returncode or 0) == 0 else "error"
+        proc.log_tail.append(
+            f"[launcher] process exited with code {p.returncode}"
+        )
+
+        if not _should_mtp_fallback(proc):
+            return
+
+        # Auto-detected MTP crashed during startup — retry ONCE without the
+        # speculative flags. The port is unchanged, so the existing provider
+        # registration still stands (do not re-register).
+        rc = proc.returncode
+        proc.mtp_fallback_done = True
+        proc.log_tail.append(
+            f"[launcher] MTP startup failure detected (exit code {rc}); "
+            "retrying without speculative decoding"
+        )
+        proc.log_tail.append(
+            f"[launcher] cmd: {' '.join(proc.fallback_cmd)}"
+        )
+        try:
+            p = await asyncio.create_subprocess_exec(
+                *proc.fallback_cmd,
+                stdout=asyncio.subprocess.PIPE,
+                stderr=asyncio.subprocess.PIPE,
+                limit=_LOG_STREAM_LIMIT,
+            )
+        except (FileNotFoundError, OSError) as exc:
+            proc.log_tail.append(f"[launcher] fallback relaunch failed: {exc}")
+            proc.status = "error"
+            return
+        proc._proc = p
+        proc.pid = p.pid
+        proc.status = "running"
+        proc.returncode = None
+        proc.started_at = time.monotonic()
+        proc.log_tail.append(f"[launcher] started PID {p.pid}")
+        # Loop to tail the relaunched process.
 
 
 async def shutdown_launcher(app: Any) -> None:
@@ -555,6 +665,11 @@ class StartRequest(BaseModel):
     port: int
     options: dict[str, Any] = {}
     extra_args: str = ""
+    # MTP / speculative-decoding controls (llama.cpp only). ``draft_model_path``
+    # is an explicit companion draft/MTP gguf; ``mtp_mode`` is "auto" (detect)
+    # or "off" (never emit speculative flags).
+    draft_model_path: str | None = None
+    mtp_mode: str = "auto"
 
 
 # ---------------------------------------------------------------------------
@@ -690,15 +805,69 @@ async def api_start(req: StartRequest, request: Request) -> dict[str, Any]:
         if bc and bc.binary:
             configured_binary = bc.binary
 
+    # Validate the MTP mode up front (only "auto" / "off" are accepted).
+    if req.mtp_mode not in ("auto", "off"):
+        raise HTTPException(
+            status_code=400,
+            detail=f"mtp_mode must be 'auto' or 'off', got {req.mtp_mode!r}.",
+        )
+
+    # Build the user token list once (options flags + extra_args) so
+    # resolve_speculative sees exactly what _build_cmd will emit. shlex.split
+    # can raise on unbalanced quotes → surface as 400 like other bad input.
+    try:
+        user_tokens = _option_tokens(req.options)
+        if req.extra_args.strip():
+            user_tokens += shlex.split(req.extra_args)
+    except ValueError as exc:
+        raise HTTPException(status_code=400, detail=str(exc)) from exc
+
+    # Resolve MTP / speculative-decoding flags (llama.cpp only; no-op elsewhere).
+    try:
+        spec_tokens, spec_notes = resolve_speculative(
+            req.backend,
+            req.model_path,
+            req.draft_model_path,
+            req.mtp_mode,
+            user_tokens,
+        )
+    except ValueError as exc:
+        raise HTTPException(status_code=400, detail=str(exc)) from exc
+
     try:
         cmd = _build_cmd(
             req.backend, req.model_path, req.port,
             req.options, req.extra_args,
             binary=configured_binary,
+            spec_tokens=spec_tokens,
         )
     except ValueError as exc:
         raise HTTPException(status_code=400, detail=str(exc)) from exc
 
+    # Speculative flags qualify for the one-shot startup-crash fallback only
+    # when they came from AUTO detection (mtp_mode="auto", no explicit draft
+    # model, and detection actually emitted flags). Explicit draft paths and
+    # operator-supplied --spec-type are never auto-retried. The fallback cmd
+    # is rebuilt from scratch with spec_tokens=None (exact — never spliced).
+    spec_auto = (
+        req.mtp_mode == "auto"
+        and req.draft_model_path is None
+        and bool(spec_tokens)
+    )
+    fallback_cmd: list[str] | None = None
+    if spec_auto:
+        try:
+            fallback_cmd = _build_cmd(
+                req.backend, req.model_path, req.port,
+                req.options, req.extra_args,
+                binary=configured_binary,
+                spec_tokens=None,
+            )
+        except ValueError:
+            # A failure to rebuild the non-spec command simply disables the
+            # fallback; it must never block the primary start.
+            fallback_cmd = None
+
     proc_id = uuid.uuid4().hex[:8]
     proc = ManagedProcess(
         id=proc_id,
@@ -708,9 +877,16 @@ async def api_start(req: StartRequest, request: Request) -> dict[str, Any]:
         port=req.port,
         options=req.options,
         extra_args=req.extra_args,
+        draft_model_path=req.draft_model_path,
+        mtp_mode=req.mtp_mode,
         status="starting",
+        spec_tokens=spec_tokens,
+        spec_auto=spec_auto and fallback_cmd is not None,
+        fallback_cmd=fallback_cmd,
     )
     proc.log_tail.append(f"[launcher] cmd: {' '.join(cmd)}")
+    for note in spec_notes:
+        proc.log_tail.append(f"[launcher] {note}")
 
     try:
         p = await asyncio.create_subprocess_exec(
@@ -732,6 +908,7 @@ async def api_start(req: StartRequest, request: Request) -> dict[str, Any]:
     proc._proc = p
     proc.pid = p.pid
     proc.status = "running"
+    proc.started_at = time.monotonic()
     proc.log_tail.append(f"[launcher] started PID {p.pid}")
 
     _registry(request).add(proc)
@@ -767,6 +944,7 @@ async def api_start(req: StartRequest, request: Request) -> dict[str, Any]:
         "pid": p.pid,
         "command": cmd,
         "provider_sync": provider_sync,
+        "speculative": spec_tokens,
     }
 
 
@@ -971,6 +1149,20 @@ _LAUNCHER_HTML = r"""<!doctype html>
         <div id="profile-args" class="mt-1 text-xs font-mono text-slate-400 bg-slate-800/50 rounded p-2 hidden"></div>
       </div>
 
+      <div class="grid grid-cols-2 gap-2">
+        <div>
+          <label class="block text-xs text-slate-400 mb-1">MTP/draft gguf (空欄で自動検出)</label>
+          <input id="f-draft" type="text" placeholder="companion .gguf (任意)" />
+        </div>
+        <div>
+          <label class="block text-xs text-slate-400 mb-1">MTP</label>
+          <select id="f-mtp">
+            <option value="auto">auto</option>
+            <option value="off">off</option>
+          </select>
+        </div>
+      </div>
+
       <div>
         <div class="flex items-center justify-between mb-1">
           <label class="block text-xs text-slate-400">追加オプション(自由入力)</label>
@@ -1292,6 +1484,8 @@ _LAUNCHER_HTML = r"""<!doctype html>
     const backend = document.getElementById("f-backend").value;
     const model = document.getElementById("f-model").value.trim();
     const extra = document.getElementById("f-extra").value.trim();
+    const draft = document.getElementById("f-draft").value.trim();
+    const mtp = document.getElementById("f-mtp").value;
 
     if (!name) { showLaunchErr("名前を入力してください"); return; }
     if (!model) { showLaunchErr("モデルパスを入力してください"); return; }
@@ -1306,7 +1500,8 @@ _LAUNCHER_HTML = r"""<!doctype html>
         method: "POST",
         headers: authHeaders({"Content-Type": "application/json"}),
         body: JSON.stringify({name, backend, model_path: model, port,
-                              options: selectedProfileArgs(), extra_args: extra}),
+                              options: selectedProfileArgs(), extra_args: extra,
+                              draft_model_path: draft || null, mtp_mode: mtp}),
       });
       const d = await res.json();
       if (!res.ok) { showLaunchErr(d.detail || "起動失敗"); return; }