@khanglvm/llm-router 2.4.1 → 2.5.2

package/CHANGELOG.md

@@ -7,6 +7,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.5.2] - 2026-04-23
+
+### Fixed
+- `yarn dev` now force-reclaims stale dev web-console listeners on startup and restarts matching stale dev routers so the next dev session takes over the sandbox cleanly instead of inheriting the old process.
+
+## [2.5.1] - 2026-04-23
+
+### Fixed
+- Relaxed the live Claude Code publish smoke check so short affirmative routed replies such as `OK` or `好的` no longer fail `npm publish` when the end-to-end router path is otherwise healthy.
+
+## [2.5.0] - 2026-04-23
+
+### Added
+- Local Models can now use a native macOS file/folder picker to attach GGUF files in place, scan a selected folder recursively for GGUF artifacts, and browse directly to a local `llama-server` runtime binary.
+
+### Changed
+- Hugging Face GGUF search results for Local Models now rank quantizations more intelligently, show tighter Mac memory-fit guidance, and call out better long-context download choices for 64 GB Macs.
+- `llama.cpp` runtime detection now searches common local source-build locations in addition to `PATH` and Homebrew installs, and server validation now recognizes more `llama-server` help output variants including TurboQuant builds.
+
+### Fixed
+- OpenAI-to-Claude response translation now preserves Anthropic-compatible usage metadata such as `speed`, `service_tier`, cache counters, and tool-usage fields so Claude Code no longer trips over missing `usage.speed` on routed responses.
+
 ## [2.4.1] - 2026-04-19
 
 ### Fixed

package/README.md

@@ -29,13 +29,25 @@ llr ai-help  # agent-oriented setup brief
 - **Model aliases with routing** — group models into stable alias names with weighted round-robin, quota-aware balancing, and automatic fallback
 - **Rate limiting** — set request caps per model or across all models over configurable time windows
 - **Coding tool routing** — one-click routing config for Codex CLI, Claude Code, Factory Droid, and AMP
-- **Dev sandbox** — `yarn dev` runs the console against a dedicated dev config/router port, highlights dev mode in terminal + UI, and can clone the production config into the sandbox for quick iteration
+- **Dev sandbox** — `yarn dev` runs the console against a dedicated dev config/router port, highlights dev mode in terminal + UI, can clone the production config into the sandbox for quick iteration, and automatically reclaims stale dev listeners before the next session starts
 - **Claude native web tools** — local handling for Claude web search and page fetch requests, with selectable Claude Code web-search providers from the shared Web Search config
 - **Seamless local updates** — `llr update` keeps the fixed local router endpoint online, drains in-flight requests, and automatically retries through backend restart windows
 - **Web search** — built-in web search for AMP and other router-managed tools
 - **Deployable** — run locally or deploy to Cloudflare Workers
 - **AI-agent friendly** — full CLI parity with `llr config --operation=...` so agents can configure everything programmatically
 
+## Local Models
+
+Open `llr` and use the **Local Models** tab to manage local inference sources alongside hosted providers.
+
+- **`llama.cpp` runtime** — detect or point at a local `llama-server`, attach GGUF files in place, or download public GGUF artifacts into the router-managed library under `~/.llm-router/local-models`
+- **Native macOS browsing** — use the built-in file picker to choose a single GGUF file, scan a folder recursively for GGUF models, or browse directly to a local `llama-server` binary
+- **Managed + attached model library** — stale or moved files stay visible instead of crashing the app, and can be repaired by locating the file again or removed cleanly
+- **Router-visible local variants** — create friendly model variants with bounded presets, context-window metadata, preload toggles, and Mac unified-memory fit guidance with clearer safe/tight recommendations
+- **Alias-ready local routing** — once saved, local variants behave like normal router models and can be used in aliases, capability flags, and fallback chains
+
+For v1, the managed download flow only searches public Hugging Face GGUF files and the fit guidance is tuned for Macs with unified memory.
+
 ## Local Runtime Reliability
 
 `llr start` keeps a small supervisor bound to the fixed local router port and runs the real router backend behind it on an internal loopback port.
@@ -48,7 +60,7 @@ That means `llr update` can install a new package version and gracefully swap th
 yarn dev
 ```
 
-Development mode uses the dedicated `~/.llm-router-dev.json` config and its own local router port so it can run alongside a startup-managed or manually started production router. The terminal and Web UI both show a dev-mode indicator, and the dev Web UI includes a one-click sync action to copy the current production config into the sandbox without changing the dev router binding.
+Development mode uses the dedicated `~/.llm-router-dev.json` config and its own local router port so it can run alongside a startup-managed or manually started production router. The terminal and Web UI both show a dev-mode indicator, the dev Web UI includes a one-click sync action to copy the current production config into the sandbox without changing the dev router binding, and each new `yarn dev` run automatically takes over any stale dev web-console/router listeners from a prior session.
 
 ## Web UI
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@khanglvm/llm-router",
-  "version": "2.4.1",
+  "version": "2.5.2",
   "description": "LLM Router: single gateway endpoint for multi-provider LLMs with unified OpenAI+Anthropic format and seamless fallback",
   "keywords": [
     "llm-router",

package/src/node/dev-command.js

@@ -0,0 +1,114 @@
+import { getActiveRuntimeState } from "./instance-state.js";
+import { reclaimPort } from "./port-reclaim.js";
+import { startWebConsoleServer } from "./web-console-server.js";
+
+const DEV_ROUTER_STOP_REASON = "Stopping the dev router because the dev web console exited.";
+
+function normalizeHost(value) {
+  return String(value || "127.0.0.1").trim() || "127.0.0.1";
+}
+
+function shouldRestartStaleDevRouter(runtimeBeforeStart, runtimeAfterStart, snapshot) {
+  if (!runtimeBeforeStart || !runtimeAfterStart || !snapshot?.router?.running) return false;
+  if (Number(runtimeBeforeStart.pid) !== Number(runtimeAfterStart.pid)) return false;
+  if (snapshot?.config?.parseError) return false;
+  if (!Number(snapshot?.config?.providerCount)) return false;
+
+  const localServer = snapshot?.config?.localServer || {};
+  return Number(runtimeAfterStart.port) === Number(localServer.port)
+    && normalizeHost(runtimeAfterStart.host) === normalizeHost(localServer.host);
+}
+
+async function stopDevRouterAfterExit(server, onError) {
+  if (!server || typeof server.stopRouter !== "function") return;
+
+  try {
+    await server.stopRouter({
+      reason: DEV_ROUTER_STOP_REASON,
+      reclaimPortIfStopped: true
+    });
+  } catch (error) {
+    onError(`Failed stopping the dev router during shutdown: ${error instanceof Error ? error.message : String(error)}`);
+  }
+}
+
+export async function startManagedDevWebConsole(options = {}, deps = {}) {
+  const line = typeof deps.line === "function" ? deps.line : console.log;
+  const error = typeof deps.error === "function" ? deps.error : console.error;
+  const startWebConsoleServerFn = typeof deps.startWebConsoleServer === "function"
+    ? deps.startWebConsoleServer
+    : startWebConsoleServer;
+  const getActiveRuntimeStateFn = typeof deps.getActiveRuntimeState === "function"
+    ? deps.getActiveRuntimeState
+    : getActiveRuntimeState;
+  const reclaimPortFn = typeof deps.reclaimPort === "function"
+    ? deps.reclaimPort
+    : (args) => reclaimPort(args, deps);
+  const serverOptions = {
+    ...options,
+    devMode: true
+  };
+  const runtimeBeforeStart = await getActiveRuntimeStateFn().catch(() => null);
+
+  let server;
+  try {
+    server = await startWebConsoleServerFn(serverOptions);
+  } catch (startError) {
+    if (startError?.code !== "EADDRINUSE") throw startError;
+
+    const reclaimed = await reclaimPortFn({
+      port: serverOptions.port,
+      line,
+      error
+    });
+    if (!reclaimed?.ok) {
+      throw new Error(reclaimed?.errorMessage || `Failed to reclaim port ${serverOptions.port}.`);
+    }
+
+    line(`Port ${serverOptions.port} reclaimed successfully.`);
+    server = await startWebConsoleServerFn(serverOptions);
+  }
+
+  const startupSnapshot = typeof server.getSnapshot === "function"
+    ? await server.getSnapshot().catch(() => null)
+    : null;
+  const runtimeAfterStart = await getActiveRuntimeStateFn().catch(() => null);
+  if (shouldRestartStaleDevRouter(runtimeBeforeStart, runtimeAfterStart, startupSnapshot)
+    && typeof server.restartRouter === "function") {
+    await server.restartRouter(startupSnapshot.config.localServer);
+  }
+
+  let stopRouterPromise = null;
+  const ensureDevRouterStopped = () => {
+    if (stopRouterPromise) return stopRouterPromise;
+    stopRouterPromise = stopDevRouterAfterExit(server, error);
+    return stopRouterPromise;
+  };
+
+  const done = (async () => {
+    let result;
+    try {
+      result = await server.done;
+    } finally {
+      await ensureDevRouterStopped();
+    }
+    return result;
+  })();
+
+  let shutdownPromise = null;
+  const shutdown = async (reason = "dev-console-closed") => {
+    if (shutdownPromise) return shutdownPromise;
+    shutdownPromise = (async () => {
+      await ensureDevRouterStopped();
+      await server.close(reason);
+      return done;
+    })();
+    return shutdownPromise;
+  };
+
+  return {
+    ...server,
+    done,
+    shutdown
+  };
+}

package/src/node/huggingface-gguf.js

@@ -0,0 +1,273 @@
+import path from "node:path";
+import { promises as fs } from "node:fs";
+
+const HUGGING_FACE_API_URL = "https://huggingface.co/api/models";
+const HUGGING_FACE_BASE_URL = "https://huggingface.co";
+const POTENTIAL_MODEL_ARTIFACT_PATTERN = /\.(gguf|safetensors|bin|pth|pt)$/i;
+const DEFAULT_EXPECTED_CONTEXT_WINDOW = 200000;
+
+function normalizeString(value) {
+  return typeof value === "string" ? value.trim() : "";
+}
+
+function normalizePositiveNumber(value) {
+  const parsed = Number(value);
+  if (!Number.isFinite(parsed) || parsed <= 0) return undefined;
+  return parsed;
+}
+
+function parseQuantizationFromFileName(fileName) {
+  const match = String(fileName || "").match(/(UD-[A-Z0-9_]+|IQ\d+_[A-Z]+|Q\d+_[A-Z0-9]+|Q\d+_0|MXFP4_MOE|BF16|F16|F32)/i);
+  return match ? match[1].toUpperCase() : "";
+}
+
+function scoreQuantization(fileName) {
+  const quantization = parseQuantizationFromFileName(fileName);
+  if (!quantization) return 0;
+  if (quantization.startsWith("Q5")) return 6;
+  if (quantization.startsWith("IQ")) return 5;
+  if (quantization === "Q4_K_M" || quantization === "Q4_K_S" || quantization.startsWith("Q4")) return 4;
+  if (quantization.startsWith("Q6")) return 3;
+  if (quantization.startsWith("Q8")) return 2;
+  if (quantization === "BF16" || quantization === "F16" || quantization === "F32") return 1;
+  return 1;
+}
+
+function buildCompatibilityBadges(fileName, fit, recommendation = "") {
+  const badges = [];
+  if (/\.gguf$/i.test(fileName)) badges.push("GGUF");
+  badges.push("llama.cpp");
+  if (fit === "safe") badges.push("Mac OK");
+  else if (fit === "tight") badges.push("Mac Tight");
+  else badges.push("Mac review");
+  if (/best fit/i.test(recommendation)) badges.push("Best fit");
+  return badges;
+}
+
+function isPotentialModelArtifact(fileName) {
+  return POTENTIAL_MODEL_ARTIFACT_PATTERN.test(String(fileName || ""));
+}
+
+function encodePathSegments(rawPath) {
+  return String(rawPath || "")
+    .split("/")
+    .filter(Boolean)
+    .map((segment) => encodeURIComponent(segment))
+    .join("/");
+}
+
+function extractHuggingFaceFiles(models = []) {
+  const files = [];
+  for (const model of Array.isArray(models) ? models : []) {
+    const repo = normalizeString(model?.id || model?.modelId);
+    if (!repo) continue;
+    for (const sibling of Array.isArray(model?.siblings) ? model.siblings : []) {
+      const file = normalizeString(sibling?.rfilename);
+      if (!file || !isPotentialModelArtifact(file)) continue;
+      files.push({
+        repo,
+        file,
+        size: normalizePositiveNumber(sibling?.size) ?? normalizePositiveNumber(sibling?.lfs?.size),
+        downloads: normalizePositiveNumber(model?.downloads) || 0,
+        likes: normalizePositiveNumber(model?.likes) || 0,
+        gguf: model?.gguf || undefined,
+        private: model?.private === true,
+        gated: model?.gated === true
+      });
+    }
+  }
+  return files;
+}
+
+export function classifyGgufCandidateForMac(candidate, { totalMemoryBytes } = {}) {
+  const fileName = normalizeString(candidate?.file || candidate?.rfilename);
+  const sizeBytes = normalizePositiveNumber(candidate?.sizeBytes ?? candidate?.size);
+  const expectedContextWindow = normalizePositiveNumber(candidate?.expectedContextWindow) || DEFAULT_EXPECTED_CONTEXT_WINDOW;
+
+  if (!/\.gguf$/i.test(fileName)) {
+    return {
+      fit: "unsupported",
+      disabled: true,
+      reason: "Not a GGUF file",
+      recommendation: "Unsupported for llama.cpp in v1."
+    };
+  }
+
+  if (sizeBytes && totalMemoryBytes && sizeBytes > Number(totalMemoryBytes) * 0.85) {
+    return {
+      fit: "over-budget",
+      disabled: true,
+      reason: "Too large for this Mac",
+      recommendation: "Skip this one on a 64 GB Mac."
+    };
+  }
+
+  if (!sizeBytes || !totalMemoryBytes) {
+    return {
+      fit: "unknown",
+      disabled: false,
+      reason: "",
+      recommendation: "Review memory fit manually before download."
+    };
+  }
+
+  const memoryRatio = sizeBytes / Number(totalMemoryBytes);
+  const quantScore = scoreQuantization(fileName);
+
+  if (expectedContextWindow >= 200000 && memoryRatio >= 0.5) {
+    return {
+      fit: "tight",
+      disabled: false,
+      reason: "200K context will be tight on this Mac",
+      recommendation: quantScore >= 2
+        ? "200K context needs review on a 64 GB Mac."
+        : "Large context and heavy quantization choice need review."
+    };
+  }
+
+  if (memoryRatio >= 0.4) {
+    return {
+      fit: "tight",
+      disabled: false,
+      reason: "Fits, but leaves limited unified memory headroom",
+      recommendation: "Reasonable fit, but memory headroom will be tight."
+    };
+  }
+
+  return {
+    fit: "safe",
+    disabled: false,
+    reason: "",
+    recommendation: quantScore >= 4
+      ? "Best fit for a 64 GB Mac and long-context testing."
+      : "Fits this Mac comfortably."
+  };
+}
+
+export function shapeHuggingFaceGgufResults(files, systemInfo = {}) {
+  const results = (Array.isArray(files) ? files : []).map((entry) => {
+    const file = normalizeString(entry?.file || entry?.rfilename);
+    const sizeBytes = normalizePositiveNumber(entry?.sizeBytes ?? entry?.size);
+    const status = classifyGgufCandidateForMac({
+      file,
+      sizeBytes,
+      expectedContextWindow: systemInfo?.expectedContextWindow
+    }, systemInfo);
+    const quantization = parseQuantizationFromFileName(file);
+    const fitScore = status.fit === "safe" ? 30 : status.fit === "tight" ? 15 : status.fit === "unknown" ? 8 : -20;
+    const rankingScore = fitScore
+      + (status.disabled ? -100 : 0)
+      + (scoreQuantization(file) * 10)
+      + Math.min(15, Math.log10(Number(entry?.downloads || 0) + 1) * 4)
+      + Math.min(8, Math.log10(Number(entry?.likes || 0) + 1) * 3)
+      - Math.min(12, (sizeBytes || 0) / (1024 ** 3));
+    return {
+      repo: normalizeString(entry?.repo || entry?.id || entry?.modelId),
+      file,
+      quantization,
+      sizeBytes,
+      disabled: status.disabled,
+      disabledReason: status.reason,
+      fit: status.fit,
+      recommendation: status.recommendation,
+      badges: buildCompatibilityBadges(file, status.fit, status.recommendation),
+      rankingScore
+    };
+  });
+
+  return results.sort((left, right) => {
+    if (right.rankingScore !== left.rankingScore) return right.rankingScore - left.rankingScore;
+    return String(left.file || "").localeCompare(String(right.file || ""));
+  });
+}
+
+export async function searchHuggingFaceGgufCandidates(query, {
+  limit = 20,
+  totalMemoryBytes,
+  expectedContextWindow = DEFAULT_EXPECTED_CONTEXT_WINDOW,
+  fetchImpl = fetch
+} = {}) {
+  const search = normalizeString(query);
+  const url = new URL(HUGGING_FACE_API_URL);
+  if (search) url.searchParams.set("search", search);
+  url.searchParams.set("limit", String(Math.max(1, Math.min(50, Number(limit) || 20))));
+  for (const field of ["siblings", "gguf", "downloads", "likes", "gated", "private"]) {
+    url.searchParams.append("expand[]", field);
+  }
+
+  const response = await fetchImpl(url, {
+    headers: {
+      accept: "application/json"
+    }
+  });
+  if (!response.ok) {
+    throw new Error(`Hugging Face search failed (${response.status}).`);
+  }
+
+  const payload = await response.json();
+  return shapeHuggingFaceGgufResults(
+    extractHuggingFaceFiles(payload),
+    { totalMemoryBytes, expectedContextWindow }
+  );
+}
+
+export function buildHuggingFaceFileDownloadUrl(repo, file) {
+  const normalizedRepo = encodePathSegments(repo);
+  const normalizedFile = encodePathSegments(file);
+  return `${HUGGING_FACE_BASE_URL}/${normalizedRepo}/resolve/main/${normalizedFile}?download=true`;
+}
+
+export async function downloadManagedHuggingFaceGguf({
+  repo,
+  file,
+  destinationPath
+} = {}, {
+  fetchImpl = fetch,
+  onProgress = () => {}
+} = {}) {
+  const targetRepo = normalizeString(repo);
+  const targetFile = normalizeString(file);
+  const outputPath = normalizeString(destinationPath);
+  if (!targetRepo || !targetFile || !outputPath) {
+    throw new Error("repo, file, and destinationPath are required.");
+  }
+
+  const url = buildHuggingFaceFileDownloadUrl(targetRepo, targetFile);
+  const response = await fetchImpl(url, {
+    headers: {
+      accept: "application/octet-stream"
+    }
+  });
+  if (!response.ok || !response.body) {
+    throw new Error(`Hugging Face download failed (${response.status}).`);
+  }
+
+  await fs.mkdir(path.dirname(outputPath), { recursive: true });
+  const tempPath = `${outputPath}.part`;
+  const fileHandle = await fs.open(tempPath, "w");
+  const totalBytes = normalizePositiveNumber(response.headers.get("content-length"));
+  let receivedBytes = 0;
+
+  try {
+    const reader = response.body.getReader();
+    while (true) {
+      const { value, done } = await reader.read();
+      if (done) break;
+      const chunk = value || new Uint8Array();
+      if (chunk.byteLength > 0) {
+        await fileHandle.write(chunk);
+        receivedBytes += chunk.byteLength;
+        onProgress({ receivedBytes, totalBytes });
+      }
+    }
+  } finally {
+    await fileHandle.close();
+  }
+
+  await fs.rename(tempPath, outputPath);
+  return {
+    filePath: outputPath,
+    sizeBytes: receivedBytes || totalBytes || undefined,
+    downloadUrl: url
+  };
+}