local-lemonade 1.0.0 → 1.1.0

package/README.md

@@ -58,9 +58,11 @@ how pi itself works:
   in `lemonade.json`. Moving the server, changing ports, or tracking upstream
   API changes is config, not code.
 - **Live state where it matters.** Pulls stream real progress from
-  server-owned download jobs with Esc-to-cancel; status shows host resources
-  and last-request performance; logs stream over websocket. Destructive
-  actions gate behind confirm dialogs and a typed phrase.
+  server-owned download jobs with Esc-to-cancel; the status panel streams the
+  full state of the box — every metric from five endpoints, grouped by source,
+  re-fetched live while open, cyclable across instances with `n`/`p`; logs
+  stream over websocket. Destructive actions gate behind confirm dialogs and
+  a typed phrase.
 - **Failures speak.** Missing models produce errors naming what to pull — the
   agent relays and recovers. Never a silent failure.
 - **One entry point.** A single command — `/lemonade-setup` — every capability
@@ -220,7 +222,7 @@ A navigable TUI (↑↓ / enter / esc):
 
 | Menu | What you can do |
 |---|---|
-| **Server status** | Live health: version, loaded models with type/device/pinned/context, per-type slot limits, websocket port — plus host resources (CPU/RAM/GPU/VRAM) and last-request performance (tok/s, time-to-first-token) |
+| **Server status** | The complete state of the box, grouped by the endpoint each metric came from — every server metric flat on one **live** panel that re-fetches every `statusPollMs` (default 2 s) while open, with each snapshot timestamped in the header; a transient fetch failure keeps the last good snapshot. Scrollable (wrap-aware `↑↓/j/k/pgup/pgdn/home/end` scrolling + mouse wheel, `q`/`esc`/`enter` return; position indicator in the footer). With multiple instances configured, `n`/`p` cycles between boxes without leaving the view — the title names the instance shown, and the menu's active instance follows it on exit. Sections: `/v1/health` (loaded models with busy/streaming/backend-health/pid/slot-pool, per-type slot limits, pinned counts, telemetry state), `/v1/system-stats` (CPU/RAM used-total-%, GPU, VRAM, NPU), `/v1/stats` (last-request tok/s, TTFT, in/out tokens, prefix-cache hit rate, lifetime token/request counters, routing decisions), `/v1/system-info` (CPU cores/threads, OS, GPU/NPU inventory incl. NPU power mode and TOPS, model-storage drive usage), and `/metrics` (llama.cpp backend: request queue depth, busy slots per decode, peak sequence length). Each section degrades to an "(unreachable)" line independently — one dead endpoint never blanks the panel |
 | **Server settings** | Discover servers (UDP beacon + HTTP fallback), edit base URL, edit API key, test connection |
 | **Model management** | List catalog (including not-yet-downloaded registry entries), load / unload, pull (download) with live progress + esc-to-cancel, install from Hugging Face (search → pick variant → install as `user.*`), delete, change context size (unload → reload with `ctx_size`, saved), toggle chat-only filter, refresh provider |
 | **Transcription settings** | Default Whisper model, endpoint path, smoke-test a file |
@@ -327,7 +329,8 @@ The full shape (values shown are the built-in defaults):
   "loadTimeoutMs": 300000,
   "pullTimeoutMs": 1800000,
   "transcriptionTimeoutMs": 300000,
-  "generationTimeoutMs": 600000
+  "generationTimeoutMs": 600000,
+  "statusPollMs": 2000
 }
 ```
 
@@ -340,6 +343,7 @@ The full shape (values shown are the built-in defaults):
 | `defaultTranscriptionModel` / `defaultImageModel` / `defaultUpscaleModel` / `defaultClassifierModel` | Fixed model ids for the tools. Empty = auto-pick from the live catalog (by label, or by `onnxruntime` recipe for classifiers). |
 | `outputDir` | Where generated files (images, audio, meshes) are saved. Empty = agent's working directory. |
 | `*TimeoutMs` | Per-operation timeouts. `pullTimeoutMs` only caps the *blocking* pull fallback — normal pulls use download jobs with live progress. |
+| `statusPollMs` | How often the setup-menu status panel re-fetches while open (live refresh). `0` = static snapshot. |
 
 ## Offline & unreachable networks
 
@@ -433,10 +437,11 @@ rename — the cost of a perfectly uniform namespace.
   instances by name and URL every session, so virgin chats know the fleet
   without hunting.
 - **Setup menu:** a *Switch instance* item (appears when extras exist) picks
-  which box Status, Model management, and Live logs act on; Server settings →
-  *Manage instances* lists (with live reachability), adds, and removes
-  instances; discovered servers can be registered as default or as a named
-  instance.
+  which box Status, Model management, and Live logs act on — and the status
+  panel itself cycles instances in-place with `n`/`p`, syncing the menu's
+  active instance on exit; Server settings → *Manage instances* lists (with
+  live reachability), adds, and removes instances; discovered servers can be
+  registered as default or as a named instance.
 
 **The one-place rule:** instance semantics are defined exactly once —
 `instanceView()` returns a *config view* (same settings, swapped
@@ -484,6 +489,18 @@ Empirically confirmed against lemonade 11.7.0 (cross-checked with the
   chat model.
 - **change-ctx.** Implemented as unload → reload with `ctx_size` +
   `save_options: true`, which persists the choice for future loads.
+- **Status panel sources.** All five metric endpoints are live-confirmed:
+  `/v1/health` (incl. per-model `is_busy`/`is_streaming`/`backend_health`/
+  `slot_pool`/`residency_class`/`last_use`/`pid` and `pinned_models` counters),
+  `/v1/system-stats`, `/v1/stats` (cumulative `*_total` counters + per-request
+  prefix-cache hits via `cache_tokens`), `/v1/system-info` (hardware inventory
+  + `model_storage` drive usage), and root-level `/metrics` (Prometheus text;
+  `lemonade_llamacpp_*` backend gauges incl. `requests_processing`/
+  `requests_deferred` queue depth and `n_tokens_max` peak sequence length).
+  Caveats seen in the field: the Linux `amd_gpu` device `name` can come back as
+  a raw device id (the panel falls back to `family`), and `/metrics` requires
+  the bearer key when `LEMONADE_API_KEY` is set. Power draw in watts is *not*
+  exposed by any lemonade endpoint — utilization only (`*_percent`).
 
 ## Troubleshooting
 
@@ -496,6 +513,8 @@ Empirically confirmed against lemonade 11.7.0 (cross-checked with the
 | Discovery finds nothing | Expected inside WSL2 (UDP broadcasts don't cross the NAT); set the URL manually |
 | Server unreachable at startup | The instance registers nothing — no lemonade models in `/model` until it's reachable; fix the URL and Refresh |
 | Pull seems to hang | It's downloading — the blocking pull path sends no progress. Large models take minutes; the `pullTimeoutMs` cap guarantees control returns |
+| Status panel shows "(unreachable)" for some sections | That one endpoint failed for this box — `/metrics` in particular requires the API key when the server sets `LEMONADE_API_KEY`. The rest of the panel stays live |
+| Status panel never refreshes | `statusPollMs` is `0` in the config — that disables live refresh. Set it to a poll interval in ms (default 2000) |
 
 ## MCP Gateway (lemonade as an MCP server)
 

package/index.ts

@@ -41,7 +41,7 @@ import dgram from "node:dgram";
 import { spawn } from "node:child_process";
 import { Type } from "typebox";
 import { DynamicBorder, getAgentDir } from "@earendil-works/pi-coding-agent";
-import { Container, Key, matchesKey, type SelectItem, SelectList, Text } from "@earendil-works/pi-tui";
+import { Container, Key, matchesKey, type SelectItem, SelectList, Text, type TuiMouseEvent, type TuiMouseEventResult } from "@earendil-works/pi-tui";
 import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
 
 // ─── Configuration ──────────────────────────────────────────────────────────
@@ -79,6 +79,7 @@ interface LemonadeConfig {
   defaultUpscaleModel: string;
   defaultClassifierModel: string; // "" = auto-pick first classification model
   outputDir: string; // "" = process cwd
+  statusPollMs: number; // status-panel live-refresh interval while viewing
   discoveryTimeoutMs: number;
   beaconTimeoutMs: number;
   loadTimeoutMs: number;
@@ -132,6 +133,7 @@ const DEFAULT_CONFIG: LemonadeConfig = {
   defaultUpscaleModel: "RealESRGAN-x4plus",
   defaultClassifierModel: "",
   outputDir: "",
+  statusPollMs: 2000,
   discoveryTimeoutMs: 5000,
   beaconTimeoutMs: 3000,
   loadTimeoutMs: 300_000,
@@ -290,6 +292,17 @@ interface LoadedModelInfo {
   device?: string;
   pinned?: boolean;
   max_context_window?: number;
+  checkpoint?: string;
+  recipe?: string;
+  is_busy?: boolean;
+  is_streaming?: boolean;
+  backend_health?: string;
+  backend_alive?: boolean;
+  pid?: number;
+  last_use?: number;
+  slot_pool?: string;
+  residency_class?: string;
+  watchdog_reset?: boolean;
 }
 
 interface LemonadeHealth {
@@ -299,6 +312,9 @@ interface LemonadeHealth {
   all_models_loaded?: LoadedModelInfo[];
   websocket_port?: number;
   max_models?: Record<string, number>;
+  pinned_models?: Record<string, number>;
+  pinned_helper_models?: Record<string, number>;
+  telemetry?: { enabled?: boolean; captures?: string[] };
 }
 
 interface SystemStats {
@@ -314,7 +330,40 @@ interface PerfStats {
   tokens_per_second?: number;
   input_tokens?: number;
   output_tokens?: number;
+  prompt_tokens?: number | null;
+  cache_tokens?: number | null;
   request_count_total?: number;
+  input_tokens_total?: number;
+  output_tokens_total?: number;
+  prompt_tokens_total?: number;
+  cache_tokens_total?: number;
+  routing_decisions_total?: number;
+  routing_switches_total?: number;
+}
+
+/** GPU device entry — shared shape (amd_gpu carries virtual_mem_gb, nvidia_gpu carries vram_gb). */
+interface GpuInfo {
+  name?: string;
+  family?: string;
+  integrated?: boolean;
+  virtual_mem_gb?: number;
+  vram_gb?: number;
+  available?: boolean;
+}
+
+/** Hardware inventory from GET /v1/system-info (fields are platform-dependent). */
+interface SystemInfo {
+  "OS Version"?: string;
+  Processor?: string;
+  "Physical Memory"?: string;
+  "Windows Power Setting"?: string;
+  model_storage?: { path?: string; used_bytes?: number; total_bytes?: number; free_bytes?: number };
+  devices?: {
+    cpu?: { name?: string; cores?: number; threads?: number; available?: boolean };
+    amd_gpu?: GpuInfo[];
+    nvidia_gpu?: GpuInfo[];
+    amd_npu?: { name?: string; family?: string; power_mode?: string; tops_max_int?: number; utilization?: number; available?: boolean };
+  };
 }
 
 interface DiscoveredServer {
@@ -401,6 +450,31 @@ async function fetchJsonOrNull<T>(config: LemonadeConfig, p: string): Promise<T
   }
 }
 
+/** Fetch a plain-text endpoint (e.g. Prometheus /metrics); null when unreachable. */
+async function fetchTextOrNull(config: LemonadeConfig, p: string): Promise<string | null> {
+  try {
+    const res = await fetch(url(config, p), {
+      headers: authHeaders(config),
+      signal: AbortSignal.timeout(config.discoveryTimeoutMs),
+    });
+    return res.ok ? await res.text() : null;
+  } catch {
+    return null;
+  }
+}
+
+/** Parse Prometheus text exposition into a name → value map (last sample wins). */
+function parsePrometheus(body: string): Map<string, number> {
+  const values = new Map<string, number>();
+  for (const raw of body.split("\n")) {
+    const line = raw.trim();
+    if (!line || line.startsWith("#")) continue;
+    const m = line.match(/^([a-zA-Z_:][a-zA-Z0-9_:]*)(?:\{[^}]*\})?\s+(-?[\d.]+(?:[eE][+-]?\d+)?)/);
+    if (m) values.set(m[1], parseFloat(m[2]));
+  }
+  return values;
+}
+
 async function fetchHealth(config: LemonadeConfig): Promise<LemonadeHealth | null> {
   return fetchJsonOrNull<LemonadeHealth>(config, config.healthPath);
 }
@@ -1299,19 +1373,226 @@ function menu<T extends string>(ctx: UiCtx, title: string, items: SelectItem<T>[
   });
 }
 
-/** Full-screen framed text view (status output); any key returns. */
-async function textView(ctx: UiCtx, title: string, lines: string[]): Promise<void> {
+/** Live-refresh wiring for the scrollable textView (status panel). */
+interface TextViewLive {
+  /** Poll interval in ms; failed/empty refetches keep the last good snapshot. */
+  pollMs?: number;
+  refetch?: () => Promise<string[]>;
+  /** Extra footer key hint (e.g. "n/p instance"), shown when multiple targets exist. */
+  extraHint?: string;
+  /** Hook for keys beyond scroll/quit; return true when the key was consumed. */
+  onKey?: (data: string, api: { setTitle: (t: string) => void }) => boolean;
+}
+
+/** Full-screen framed text view; any key returns. `scrollable` adds keyboard
+ *  paging; `live` adds polling + a key hook (scrollable mode only). */
+async function textView(
+  ctx: UiCtx,
+  title: string,
+  lines: string[],
+  scrollable = false,
+  live?: TextViewLive
+): Promise<void> {
   await ctx.ui.custom<null>((tui, theme, _kb, done) => {
     const container = new Container();
     container.addChild(new DynamicBorder((s: string) => theme.fg("accent", s)));
-    container.addChild(new Text(theme.fg("accent", theme.bold(title)), 1, 0));
-    for (const line of lines) container.addChild(new Text(line, 0, 0));
-    container.addChild(new Text(theme.fg("dim", "press any key to return"), 1, 0));
+    const titleText = new Text(theme.fg("accent", theme.bold(title)), 1, 0);
+    container.addChild(titleText);
+    if (!scrollable) {
+      for (const line of lines) container.addChild(new Text(line, 0, 0));
+      const footer = new Text(theme.fg("dim", "press any key to return"), 1, 0);
+      container.addChild(footer);
+      container.addChild(new DynamicBorder((s: string) => theme.fg("accent", s)));
+      return {
+        render: (w) => container.render(w),
+        invalidate: () => container.invalidate(),
+        handleInput: () => { done(null); tui.requestRender(); },
+      };
+    }
+
+    // Scrollable: render only a terminal-height window of the lines, with a
+    // live position indicator. The main screen has no layout engine for this
+    // custom view, so ScrollView is inert here — the window is sliced by hand.
+    // The slice is wrap-aware: lines longer than the terminal width occupy
+    // more than one rendered row, so the window is filled by *rows*, not by
+    // line count (otherwise long lines push the footer off-screen on narrow
+    // terminals and scrolling can stop short of the real end).
+    const body = new Text("", 0, 0);
+    container.addChild(body);
+    const footer = new Text("", 1, 0);
+    container.addChild(footer);
     container.addChild(new DynamicBorder((s: string) => theme.fg("accent", s)));
+
+    let cur = lines; // swapped in place by live refetches
+
+    // top border + title + footer + bottom border = 4 lines; keep one spare
+    // row so word-wrap loss (wrap estimates are a lower bound) never pushes
+    // the bottom border off-screen.
+    const overhead = 5;
+    const viewport = () => Math.max(3, tui.terminal.rows - overhead);
+    const rowsOf = (l: string) => Math.max(1, Math.ceil(l.length / Math.max(10, tui.terminal.columns)));
+
+    // Row offset of each line's first rendered row; offsets[cur.length] = total rows.
+    const rowOffsets = (): number[] => {
+      const off: number[] = [];
+      let acc = 0;
+      for (const l of cur) {
+        off.push(acc);
+        acc += rowsOf(l);
+      }
+      off.push(acc);
+      return off;
+    };
+
+    let top = 0;
+    /** Largest line index that still leaves a full viewport of rows below. */
+    const maxTop = () => {
+      const off = rowOffsets();
+      const vh = viewport();
+      let i = cur.length - 1;
+      while (i > 0 && off[cur.length] - off[i] < vh) i--;
+      return i;
+    };
+    /** Line index `byRows` rendered rows down/up from `top`. */
+    const advance = (byRows: number): number => {
+      const off = rowOffsets();
+      const target = off[top] + byRows;
+      let i = top;
+      while (i < cur.length - 1 && off[i] < target) i++;
+      return i;
+    };
+    const retreat = (byRows: number): number => {
+      const off = rowOffsets();
+      const target = Math.max(0, off[top] - byRows);
+      let i = top;
+      while (i > 0 && off[i] > target) i--;
+      return i;
+    };
+
+    const refresh = () => {
+      const off = rowOffsets();
+      const vh = viewport();
+      top = Math.min(top, maxTop());
+      const slice: string[] = [];
+      let used = 0;
+      let j = top;
+      while (j < cur.length) {
+        const r = rowsOf(cur[j]);
+        if (used + r > vh && slice.length) break; // always show at least the top line
+        used += r;
+        slice.push(cur[j]);
+        j++;
+      }
+      body.setText(slice.join("\n"));
+      // Hint ladder: the full hint (scroll keys + instance keys + quit keys +
+      // position) is ~80 chars and wraps on 80-col terminals, so progressively
+      // shorter variants are tried until one fits the usable width. Keys are
+      // unaffected — the hint is best-effort documentation.
+      const base = [...(live?.extraHint ? [live.extraHint] : []), "q/esc/enter return"].join(" · ");
+      const pos = `lines ${top + 1}–${top + slice.length} of ${cur.length}`;
+      const scrollableNow = off[cur.length] > vh;
+      const candidates = scrollableNow
+        ? [
+            `↑↓/jk/pgup·pgdn/home/end · ${base} · ${pos}`,
+            `↑↓/jk/pgup·pgdn · ${base} · ${pos}`,
+            `↑↓/jk · ${base}`,
+            base,
+          ]
+        : [base, "q/esc return"];
+      const budget = Math.max(10, tui.terminal.columns) - 2; // Text paddingX 1 each side
+      const hint = candidates.find((c) => c.length <= budget) ?? candidates[candidates.length - 1];
+      footer.setText(theme.fg("dim", hint));
+    };
+    refresh();
+
+    // Live polling: swap in fresh lines on each tick. Transient fetch failures
+    // keep the last good snapshot on screen; the timer dies with the view.
+    // Ticks coalesce with a trailing edge: a tick requested while one is in
+    // flight (fast n/p cycling) runs once after it, with the latest target.
+    let inFlight = false;
+    let pending = false;
+    const tick = async (): Promise<void> => {
+      if (!live?.refetch) return;
+      if (inFlight) {
+        pending = true;
+        return;
+      }
+      inFlight = true;
+      try {
+        const next = await live.refetch();
+        if (next && next.length) {
+          cur = next;
+          refresh();
+          tui.requestRender();
+        }
+      } catch {
+        /* keep showing the last good snapshot */
+      } finally {
+        inFlight = false;
+        if (pending) {
+          pending = false;
+          void tick();
+        }
+      }
+    };
+    let timer: ReturnType<typeof setInterval> | undefined;
+    if (live?.refetch) timer = setInterval(() => void tick(), Math.max(250, live.pollMs ?? 2000));
+    const cleanup = () => {
+      if (timer !== undefined) {
+        clearInterval(timer);
+        timer = undefined;
+      }
+    };
+
     return {
       render: (w) => container.render(w),
       invalidate: () => container.invalidate(),
-      handleInput: () => { done(null); tui.requestRender(); },
+      handleInput: (data) => {
+        // Letters have no Key.* members (Key only covers special/symbol keys);
+        // "j"/"k"/"q" are valid bare KeyIds. Never let an input error become an
+        // uncaught exception — it would take down the whole pi process.
+        try {
+          const page = () => Math.max(1, viewport() - 1);
+          if (matchesKey(data, Key.up) || matchesKey(data, "k")) top = retreat(1);
+          else if (matchesKey(data, Key.down) || matchesKey(data, "j")) top = Math.min(maxTop(), advance(1));
+          else if (matchesKey(data, Key.pageUp)) top = retreat(page());
+          else if (matchesKey(data, Key.pageDown)) top = Math.min(maxTop(), advance(page()));
+          else if (matchesKey(data, Key.home)) top = 0;
+          else if (matchesKey(data, Key.end)) top = maxTop();
+          else if (
+            matchesKey(data, Key.escape) ||
+            matchesKey(data, Key.enter) ||
+            matchesKey(data, "q")
+          ) {
+            cleanup();
+            done(null);
+            tui.requestRender();
+            return;
+          } else if (
+            live?.onKey &&
+            live.onKey(data, {
+              setTitle: (t: string) => titleText.setText(theme.fg("accent", theme.bold(t))),
+            })
+          ) {
+            void tick(); // target changed — refresh immediately, poll continues on it
+          }
+          refresh();
+          tui.requestRender();
+        } catch {
+          // A failed key match must not kill the process — swallow and continue.
+        }
+      },
+      dispose: cleanup,
+      handleMouse: (event: TuiMouseEvent): TuiMouseEventResult | undefined => {
+        if (event.type !== "wheel" || !event.wheelDelta) return undefined;
+        top =
+          event.wheelDelta > 0
+            ? Math.min(maxTop(), advance(event.wheelDelta))
+            : retreat(-event.wheelDelta);
+        refresh();
+        tui.requestRender();
+        return { handled: true };
+      },
     };
   });
 }
@@ -1481,7 +1762,49 @@ async function liveLogsView(ctx: UiCtx, config: LemonadeConfig): Promise<void> {
   });
 }
 
-function formatHealth(config: LemonadeConfig, health: LemonadeHealth | null): string[] {
+/** Format a metric family section title, e.g. "── /v1/stats ────────────". */
+function sectionTitle(name: string): string {
+  const label = ` ${name} `;
+  return `──${label}${"─".repeat(Math.max(2, 44 - label.length))}`;
+}
+
+/** Compact token counts: 15,231,250 → "15.2M", 243,182 stays locale-grouped. */
+function fmtTokens(n: number): string {
+  if (n >= 1e9) return `${(n / 1e9).toFixed(2)}B`;
+  if (n >= 1e6) return `${(n / 1e6).toFixed(1)}M`;
+  return Math.round(n).toLocaleString("en-US");
+}
+
+/** Byte counts → human-readable storage sizes. */
+function fmtBytes(n: number): string {
+  if (n >= 1e12) return `${(n / 1e12).toFixed(1)} TB`;
+  if (n >= 1e9) return `${(n / 1e9).toFixed(1)} GB`;
+  if (n >= 1e6) return `${(n / 1e6).toFixed(1)} MB`;
+  return `${Math.round(n / 1e3)} kB`;
+}
+
+/** Parse "125.07 GB"-style totals from system-info into a GiB number. */
+function parseGbTotal(s?: string): number | null {
+  if (!s) return null;
+  const m = s.match(/([\d.]+)\s*GB/i);
+  return m ? parseFloat(m[1]) : null;
+}
+
+/**
+ * Full status panel: every metric family the server exposes, grouped under
+ * a title naming the endpoint it came from. Each section degrades to an
+ * "(unreachable)" line when its endpoint fails — one dead endpoint never
+ * blanks the rest of the panel.
+ */
+function formatStatus(
+  config: LemonadeConfig,
+  health: LemonadeHealth | null,
+  sys: SystemStats | null,
+  perf: PerfStats | null,
+  sysinfo: SystemInfo | null,
+  metrics: Map<string, number> | null,
+  updated?: Date
+): string[] {
   if (!health) {
     return [
       `Unreachable: ${url(config, config.healthPath)}`,
@@ -1490,55 +1813,199 @@ function formatHealth(config: LemonadeConfig, health: LemonadeHealth | null): st
       "under Server settings (or run Discover servers).",
     ];
   }
-  const lines = [
-    `Status:   ${health.status ?? "?"}   Version: ${health.version ?? "?"}`,
+
+  const lines: string[] = [];
+  const telemetry = health.telemetry?.enabled
+    ? `on${health.telemetry.captures?.length ? ` (${health.telemetry.captures.join(", ")})` : ""}`
+    : "off";
+
+  // Header
+  lines.push(
+    `Status: ${health.status ?? "?"}   Lemonade v${health.version ?? "?"}   Telemetry: ${telemetry}` +
+      (updated ? `   · updated ${updated.toTimeString().slice(0, 8)}` : ""),
     `Endpoint: ${config.baseUrl}   WebSocket port: ${health.websocket_port ?? "?"}`,
-    "",
-    `Loaded models (${health.all_models_loaded?.length ?? 0}):`,
-  ];
+    ""
+  );
+
+  // ── /v1/health ───────────────────────────────────────────────────────
+  lines.push(sectionTitle(config.healthPath));
+  lines.push(`Loaded models (${health.all_models_loaded?.length ?? 0}):`);
   for (const m of health.all_models_loaded ?? []) {
     lines.push(
       `  • ${m.model_name} — ${m.type}, ${m.status}, ${m.device ?? "?"}${m.pinned ? ", pinned" : ""}` +
         `, ctx ${m.max_context_window ?? "?"}`
     );
+    const live: string[] = [];
+    live.push(m.is_busy ? "busy" : "idle");
+    if (m.is_streaming) live.push("streaming");
+    live.push(
+      `backend ${m.backend_health ?? "?"}${m.backend_alive === false ? " (backend not responding!)" : ""}`
+    );
+    if (m.watchdog_reset) live.push("watchdog reset");
+    lines.push(`      ${live.join("  ")}`);
+    const meta: string[] = [];
+    if (m.recipe) meta.push(m.recipe);
+    if (m.checkpoint) meta.push(m.checkpoint);
+    if (m.slot_pool) meta.push(`pool ${m.slot_pool}`);
+    if (m.residency_class) meta.push(m.residency_class);
+    if (typeof m.pid === "number") meta.push(`pid ${m.pid}`);
+    if (typeof m.last_use === "number") meta.push(`last use ${m.last_use}`);
+    if (meta.length) lines.push(`      ${meta.join("  ")}`);
   }
   if (!health.all_models_loaded?.length) lines.push("  (none)");
   const slots = Object.entries(health.max_models ?? {})
     .map(([k, v]) => `${k}:${v}`)
     .join("  ");
-  if (slots) lines.push("", `Slots: ${slots}`);
-  return lines;
-}
+  if (slots) lines.push(`Slot limits: ${slots}`);
+  const pinned = Object.entries(health.pinned_models ?? {})
+    .filter(([, v]) => v > 0)
+    .map(([k, v]) => `${k}:${v}`)
+    .join("  ");
+  const helpers = Object.values(health.pinned_helper_models ?? {}).reduce((a, b) => a + b, 0);
+  if (pinned) lines.push(`Pinned: ${pinned}${helpers ? `  (+${helpers} pinned helper${helpers > 1 ? "s" : ""})` : ""}`);
+  lines.push("");
 
-function formatStatus(
-  config: LemonadeConfig,
-  health: LemonadeHealth | null,
-  sys: SystemStats | null,
-  perf: PerfStats | null
-): string[] {
-  const lines = formatHealth(config, health);
+  // ── /v1/system-stats ─────────────────────────────────────────────────
+  lines.push(sectionTitle("/v1/system-stats"));
   if (sys) {
     const pct = (v?: number | null) => (typeof v === "number" ? `${v.toFixed(0)}%` : "?");
     const gb = (v?: number | null) => (typeof v === "number" ? `${v.toFixed(1)} GB` : "?");
+    const totalRam = parseGbTotal(sysinfo?.["Physical Memory"]);
+    const ramPart =
+      typeof sys.memory_gb === "number"
+        ? totalRam
+          ? `RAM ${sys.memory_gb.toFixed(1)}/${totalRam.toFixed(1)} GB (${((sys.memory_gb / totalRam) * 100).toFixed(0)}%)`
+          : `RAM ${sys.memory_gb.toFixed(1)} GB`
+        : "RAM ?";
     lines.push(
-      "",
-      `Host: CPU ${pct(sys.cpu_percent)}  RAM ${gb(sys.memory_gb)}  GPU ${pct(sys.gpu_percent)}  VRAM ${gb(sys.vram_gb)}` +
-        (sys.npu_percent !== undefined && sys.npu_percent !== null ? `  NPU ${pct(sys.npu_percent)}` : "")
+      `CPU ${pct(sys.cpu_percent)}   ${ramPart}   GPU ${pct(sys.gpu_percent)}   VRAM ${gb(sys.vram_gb)}` +
+        (sys.npu_percent !== undefined && sys.npu_percent !== null ? `   NPU ${pct(sys.npu_percent)}` : "")
     );
+  } else {
+    lines.push("  (unreachable)");
   }
-  if (perf && typeof perf.tokens_per_second === "number") {
-    lines.push(
-      "",
-      `Last request: ${perf.tokens_per_second.toFixed(1)} tok/s, TTFT ${
-        typeof perf.time_to_first_token === "number" ? perf.time_to_first_token.toFixed(2) : "?"
-      }s` +
-        ` (in ${perf.input_tokens ?? "?"} / out ${perf.output_tokens ?? "?"} tokens)` +
-        (typeof perf.request_count_total === "number" ? `  lifetime requests: ${perf.request_count_total}` : "")
-    );
+  lines.push("");
+
+  // ── /v1/stats ─────────────────────────────────────────────────────────
+  lines.push(sectionTitle("/v1/stats"));
+  if (perf) {
+    if (typeof perf.tokens_per_second === "number") {
+      const num = (v: number | null | undefined, digits = 0) =>
+        typeof v === "number" ? v.toLocaleString("en-US", { maximumFractionDigits: digits }) : "?";
+      lines.push(
+        `Last request: ${perf.tokens_per_second.toFixed(1)} tok/s, TTFT ${
+          typeof perf.time_to_first_token === "number" ? perf.time_to_first_token.toFixed(2) : "?"
+        }s (in ${num(perf.input_tokens)} / out ${num(perf.output_tokens)} tokens)`
+      );
+      if (typeof perf.prompt_tokens === "number") {
+        const cachePart =
+          typeof perf.cache_tokens === "number"
+            ? perf.prompt_tokens > 0
+              ? `, ${num(perf.cache_tokens)} from prefix cache (${((perf.cache_tokens / perf.prompt_tokens) * 100).toFixed(1)}%)`
+              : `, ${num(perf.cache_tokens)} from prefix cache`
+            : "";
+        lines.push(`  prompt ${num(perf.prompt_tokens)} tokens${cachePart}`);
+      }
+      const lifetime: string[] = [];
+      if (typeof perf.request_count_total === "number") lifetime.push(`${num(perf.request_count_total)} requests`);
+      if (typeof perf.input_tokens_total === "number") lifetime.push(`in ${fmtTokens(perf.input_tokens_total)}`);
+      if (typeof perf.output_tokens_total === "number") lifetime.push(`out ${fmtTokens(perf.output_tokens_total)}`);
+      if (typeof perf.prompt_tokens_total === "number") lifetime.push(`prompt ${fmtTokens(perf.prompt_tokens_total)}`);
+      if (typeof perf.cache_tokens_total === "number") lifetime.push(`cache ${fmtTokens(perf.cache_tokens_total)}`);
+      if (lifetime.length) lines.push(`Lifetime: ${lifetime.join(" · ")}`);
+      if (typeof perf.routing_decisions_total === "number" || typeof perf.routing_switches_total === "number") {
+        lines.push(
+          `Routing: ${num(perf.routing_decisions_total)} decisions, ${num(perf.routing_switches_total)} switches`
+        );
+      }
+    } else {
+      lines.push("No requests served since server start.");
+    }
+  } else {
+    lines.push("  (unreachable)");
   }
+  lines.push("");
+
+  // ── /v1/system-info ──────────────────────────────────────────────────
+  lines.push(sectionTitle("/v1/system-info"));
+  if (sysinfo) {
+    const cpu = sysinfo.devices?.cpu;
+    if (sysinfo.Processor || cpu?.cores) {
+      const coresPart = cpu?.cores ? ` (${cpu.cores}c/${cpu.threads ?? "?"}t)` : "";
+      lines.push(`CPU: ${sysinfo.Processor ?? cpu?.name ?? "?"}${coresPart}`);
+    }
+    if (sysinfo["OS Version"]) lines.push(`OS: ${sysinfo["OS Version"]}`);
+    const gpu = sysinfo.devices?.amd_gpu?.find((g) => g.available !== false) ?? sysinfo.devices?.nvidia_gpu?.find((g) => g.available !== false);
+    if (gpu) {
+      // On Linux the amd_gpu "name" can be a raw device id (e.g. "110501");
+      // fall back to the family in that case.
+      const gpuName = gpu.name && !/^\d+$/.test(gpu.name) ? gpu.name : gpu.family || "?";
+      const gpuParts: string[] = [];
+      if (typeof gpu.integrated === "boolean") gpuParts.push(gpu.integrated ? "integrated" : "discrete");
+      if (gpu.family && gpuName !== gpu.family) gpuParts.push(gpu.family);
+      if (typeof gpu.virtual_mem_gb === "number") gpuParts.push(`virtual mem ${gpu.virtual_mem_gb.toFixed(1)} GB`);
+      else if (typeof gpu.vram_gb === "number" && gpu.vram_gb > 0) gpuParts.push(`VRAM ${gpu.vram_gb.toFixed(1)} GB`);
+      lines.push(`GPU: ${gpuName}${gpuParts.length ? ` — ${gpuParts.join(", ")}` : ""}`);
+    }
+    const npu = sysinfo.devices?.amd_npu;
+    if (npu && npu.available !== false) {
+      const npuParts: string[] = [];
+      if (npu.power_mode) npuParts.push(`power mode ${npu.power_mode}`);
+      if (typeof npu.tops_max_int === "number") npuParts.push(`${npu.tops_max_int} TOPS`);
+      if (npuParts.length || npu.name || npu.family) {
+        lines.push(`NPU: ${npu.name || "?"}${npu.family ? ` (${npu.family})` : ""}${npuParts.length ? ` — ${npuParts.join(", ")}` : ""}`);
+      }
+    }
+    if (sysinfo["Windows Power Setting"]) lines.push(`Power plan: ${sysinfo["Windows Power Setting"]}`);
+    const store = sysinfo.model_storage;
+    if (store && typeof store.total_bytes === "number") {
+      const free = typeof store.free_bytes === "number" ? ` (${fmtBytes(store.free_bytes)} free)` : "";
+      lines.push(
+        `Model storage: ${fmtBytes(store.used_bytes ?? 0)} used / ${fmtBytes(store.total_bytes)} total${free}`
+      );
+      if (store.path) lines.push(`  ${store.path}`);
+    }
+  } else {
+    lines.push("  (unreachable)");
+  }
+  lines.push("");
+
+  // ── /metrics ──────────────────────────────────────────────────────────
+  lines.push(sectionTitle("/metrics (llama.cpp backend)"));
+  if (metrics && metrics.size) {
+    const proc = metrics.get("lemonade_llamacpp_requests_processing");
+    const deferred = metrics.get("lemonade_llamacpp_requests_deferred");
+    const busy = metrics.get("lemonade_llamacpp_n_busy_slots_per_decode");
+    const peak = metrics.get("lemonade_llamacpp_n_tokens_max");
+    if (proc === undefined && deferred === undefined && peak === undefined) {
+      lines.push("  (no llamacpp backend metrics — no llama.cpp model loaded)");
+    } else {
+      lines.push(
+        `Queue: ${proc ?? 0} processing, ${deferred ?? 0} deferred` +
+          (busy !== undefined ? `   Busy slots/decode: ${busy.toFixed(2)}` : "")
+      );
+      if (peak !== undefined) lines.push(`Peak sequence: ${fmtTokens(peak)} tokens (largest prompt+generation observed)`);
+    }
+  } else {
+    lines.push("  (unreachable)");
+  }
+
   return lines;
 }
 
+/** Fetch all five status endpoints and format the panel, timestamped "now". */
+async function fetchStatusLines(config: LemonadeConfig): Promise<string[]> {
+  const [h, sys, perf, info, metricsBody] = await Promise.all([
+    fetchHealth(config),
+    fetchJsonOrNull<SystemStats>(config, "/v1/system-stats"),
+    fetchJsonOrNull<PerfStats>(config, "/v1/stats"),
+    fetchJsonOrNull<SystemInfo>(config, "/v1/system-info"),
+    fetchTextOrNull(config, "/metrics"),
+  ]);
+  const metrics = metricsBody ? parsePrometheus(metricsBody) : null;
+  return formatStatus(config, h, sys, perf, info, metrics, new Date());
+}
+
 /** Run an async op with a status indicator; returns [ok, result]. */
 async function withStatus<T>(
   ctx: UiCtx,
@@ -1776,19 +2243,43 @@ function registerSetupCommand(pi: ExtensionAPI, configRef: { current: LemonadeCo
         }
 
         // ── Status ────────────────────────────────────────────────────────
+        // Live panel: polls every statusPollMs while open, and n/p cycles
+        // through every configured instance without leaving the view.
         if (choice === "status") {
+          const instances = config.servers;
+          let idx = Math.max(0, instances.findIndex((s) => s.name === activeInstance));
+          const cfgFor = (): LemonadeConfig => {
+            const view = instanceView(configRef.current, instances[idx]?.name ?? activeInstance);
+            return typeof view === "string" ? iconfig : view; // stale name → keep the menu's view
+          };
           ctx.ui.setStatus("lemonade-setup", "Contacting server…");
-          const [h, sys, perf] = await Promise.all([
-            fetchHealth(iconfig),
-            fetchJsonOrNull<SystemStats>(iconfig, "/v1/system-stats"),
-            fetchJsonOrNull<PerfStats>(iconfig, "/v1/stats"),
-          ]);
+          const first = await fetchStatusLines(cfgFor());
           ctx.ui.setStatus("lemonade-setup", undefined);
           await textView(
             ctx,
-            `Server Status — ${activeInstance}`,
-            formatStatus(iconfig, h, sys, perf)
+            `Server Status — ${instances[idx]?.name ?? activeInstance}`,
+            first,
+            true,
+            {
+              pollMs: config.statusPollMs,
+              refetch: config.statusPollMs > 0 ? () => fetchStatusLines(cfgFor()) : undefined,
+              extraHint: instances.length > 1 ? "n/p instance" : undefined,
+              onKey:
+                instances.length > 1
+                  ? (data, api) => {
+                      if (matchesKey(data, "n")) idx = (idx + 1) % instances.length;
+                      else if (matchesKey(data, "p")) idx = (idx - 1 + instances.length) % instances.length;
+                      else return false;
+                      const name = instances[idx]?.name ?? activeInstance;
+                      api.setTitle(`Server Status — ${name}`);
+                      return true;
+                    }
+                  : undefined,
+            }
           );
+          // The panel was the last thing the user looked at — the menu follows it.
+          const lastName = instances[idx]?.name;
+          if (lastName) activeInstance = lastName;
           continue;
         }
 

package/lemonade.example.json

@@ -67,5 +67,9 @@
   "loadTimeoutMs": 300000,      // model load (5 min)
   "pullTimeoutMs": 1800000,     // blocking model pull (30 min)
   "transcriptionTimeoutMs": 300000,
-  "generationTimeoutMs": 600000
+  "generationTimeoutMs": 600000,
+
+  // How often (ms) the /lemonade-setup status panel re-fetches while it is
+  // open. 0 = static snapshot (no live refresh).
+  "statusPollMs": 2000
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "local-lemonade",
-  "version": "1.0.0",
+  "version": "1.1.0",
   "description": "Unified pi extension for a self-hosted Lemonade server: dynamic chat-model discovery, multimodal agent tools (transcription, image gen/edit/upscale, TTS, audio, 3D mesh), and a live /lemonade-setup management TUI.",
   "keywords": [
     "pi-package",
@@ -21,4 +21,4 @@
   "pi": {
     "extensions": ["./index.ts"]
   }
-}
+}