pi-llamacpp-infra 1.2.0 → 1.2.2

package/README.md

@@ -1,5 +1,7 @@
 # llamacpp-infra — Discovery, Metrics & Control for llama.cpp-family Servers
 
+![llamacpp-infra banner](https://raw.githubusercontent.com/noguerol/llamacpp-infra/main/docs/banner.jpeg)
+
 **llamacpp-infra** turns pi into a first-class citizen of local llama.cpp infrastructure. It probes any number of machines — localhost, LAN or Tailscale — discovers every model served by llama.cpp and its variants (including LM Studio), registers them into pi's native `/model` list, and gives you live Prometheus metrics, per-model thinking budgets, vision detection and a full configuration UI — all without leaving the pi prompt.
 
 ---
@@ -24,7 +26,7 @@ Anything else (vLLM, Ollama, cloud APIs…) is out of scope — use pi's built-i
 - **Compact model ids** — models appear as `Name (host:port)` in pi's `/model` picker, like a native provider; the raw GGUF path/alias is sent to the server automatically on every request
 - **Single-model & router modes** — llama.cpp single-model mode (one GGUF per instance) and router mode (multiple models per server, with per-model status and args)
 - **Per-model metadata badges** — 👁️ vision (mmproj / modalities), 🚀 drafter (speculative decoding), 🗜️ quant tag from GGUF filename, 🧠 KV cache quantization (from server args or `/proc`)
-- **Live Prometheus metrics** — polls `/metrics` (or `/stats`) and renders a compact widget with instantaneous prompt/gen throughput; auto-activates for llamacpp-infra models only
+- **Live speed & metrics** — a constantly updating footer reading of the active model's prefill (⚡) and generation (🔥) token speed, measured straight from the stream (per token, ~10 updates/s); when pi is idle it also mirrors other clients the server's `/metrics` endpoint reports. Lives in the footer's status line, so no extra terminal row is taken. Works even without `--metrics`
 - **Thinking budgets** — llama.cpp accepts `thinking_budget_tokens` per request; configure budgets per thinking level (minimal/low/medium/high/xhigh/max) per model; models with budgets are registered with reasoning enabled
 - **Header warmup** — pre-caches the system prompt KV on llama.cpp-family servers so the first real request is faster
 - **LM Studio support** — uses LM Studio's OpenAI-compatible `/v1` API, enriches names/context/quant/vision from `/api/v1/models` (or legacy `/api/v0/models`), and avoids llama.cpp-only request fields
@@ -35,7 +37,7 @@ Anything else (vLLM, Ollama, cloud APIs…) is out of scope — use pi's built-i
 
 ## Install
 
-llamacpp-infra is a [pi package](https://pi.dev/packages): one extension (`src/index.ts`) plus an inlined warmup module, declared in `package.json`.
+llamacpp-infra is a [pi package](https://pi.dev/packages): a small entrypoint (`src/index.ts`) backed by several focused modules. Heavier pieces (UI, scan engine, metrics, header warmup) are loaded lazily on first use so the extension stays light at startup. The package is declared in `package.json`.
 
 ```bash
 # From GitHub
@@ -97,7 +99,7 @@ No special payload workaround is required: LM Studio accepts standard OpenAI cha
 | `/llamacpp-infra scan` | Rescan all servers now |
 | `/llamacpp-infra status` | Detailed per-endpoint report |
 | `/llamacpp-infra list` | List discovered models with metadata badges |
-| `/llamacpp-infra metrics` | Toggle the live metrics widget |
+| `/llamacpp-infra metrics` | Toggle live speed & metrics in the footer |
 | `/llamacpp-infra help` | Command help |
 
 ### `/llamacpp-infra config`
@@ -109,7 +111,7 @@ The main config menu branches into submenus:
 - **📋 Models** — per-model options (thinking budgets, replace/remove)
 - **🧪 Test** — connectivity test of all configured servers
 - **🧠 Thinking budgets** — configure per-model thinking_budget_tokens per level
-- **📈 Metrics** — enable/disable widget, poll interval
+- **📈 Metrics** — enable/disable footer metrics, server poll interval
 - **⚙️ Settings** — discovery timeout, poll interval/budget, startup grace, fail limit, vision detection, prefix model IDs, name badges, unloaded router models, header warmup
 - **ℹ️ About** — extension info
 
@@ -222,7 +224,7 @@ Everything is configurable through the UI, but the persisted file is `~/.pi/agen
 | `showBadgesInNames` | `true` | Append 👁️🚀💤 badges to model display names |
 | `includeUnloadedRouterModels` | `false` | Router mode: list models that are not currently loaded |
 | `warmup` | `true` | Pre-cache system prompt KV on llama.cpp servers |
-| `metricsEnabled` | `true` | Auto-show live metrics widget for llamacpp-infra models |
+| `metricsEnabled` | `true` | Show live speed & metrics in the footer for llamacpp-infra models |
 | `metricsPollMs` | `5000` | How often `/metrics` is fetched |
 
 ### Thinking budgets
@@ -243,33 +245,59 @@ With `prefixModelIds: false` the machine tag is omitted (`ModelName`); it is re-
 
 llama.cpp-family models are registered as reasoning models, exactly like a native pi provider: the footer shows `ModelName (host:port) • <level>`, the thinking selector offers levels with token estimates, and pi sends the configured `thinking_budget_tokens` budget on each request. Per-model budgets configured in the extension override pi's global per-level budgets.
 
-## Live Metrics Widget
+## Live Speed & Metrics (footer)
 
-When enabled, the metrics widget appears automatically when the active model is from llamacpp-infra:
+When enabled, the speed reading appears in the footer's status line (no extra terminal row) whenever the active model is from llamacpp-infra, updating constantly while tokens flow. Both entries are kept ultra-compact so they coexist with other extensions on pi's single status line (which truncates from the end):
 
 ```
-📊 local:8080  ⚡ 42.3 t/s prompt · 38.1 t/s gen · 1.2k tokens
+🦙(12) ⚡…            (before the first token)
+🦙(12) ⚡ 420 t/s 🔥 38.1 t/s  (while streaming)
+🦙(12) ⚡ 420 t/s 🔥 38.1 t/s  (just after the answer ends)
+🦙(12) ⏸             (between turns)
+🦙(12) ▶2 ⚡ 150 t/s 🔥 18.0 t/s  (pi idle, server busy for other clients)
 ```
 
-It polls the server's Prometheus `/metrics` endpoint (or JSON `/stats`) and shows instantaneous throughput. The poll interval is configurable (default 5s).
+(`🦙(n)` is the extension's model-count status; both live on the same footer line, so no extra row is consumed.)
+
+- **Client measurement (always, no `--metrics` needed)** — prefill speed = `prompt tokens ÷ (request → first token)` (pi's `usage.input`, OpenAI-style `prompt_tokens` as fallback); generation speed = a moving 1.5 s window over per-token arrival samples. Updated ~every 100 ms while a stream is live (throttled, and unchanged text is skipped, so the footer never churns).
+- **Server supplement (only when pi is idle)** — the poller fetches the server's Prometheus `/metrics` endpoint (or JSON `/stats`) every `metricsPollMs` (default 5 s). If the server reports other clients processing, their ⚡/🔥 rates are shown (`▶n`); when the server is idle, the plain `⏸` reading returns.
 
 ## Architecture
 
 ```
 llamacpp-infra/
-├── package.json        # pi package manifest (pi-package)
-├── LICENSE             # MIT
+├── package.json         # pi package manifest (pi-package)
+├── LICENSE              # MIT
 ├── README.md
 └── src/
-    ├── index.ts        # Extension entry point (~2600 lines)
-    └── prompt-warmup.ts # Header warmup module (inlined, ~600 lines)
+    ├── index.ts         # Entrypoint: hooks, command, lifecycle. Statically imports core + types.
+    ├── core.ts          # Config persistence, shared state, id helpers, compat profile (loaded at startup).
+    ├── types.ts         # Shared interfaces (type-only; erased at runtime).
+    ├── scan.ts          # Discovery engine (lazy: HTTP probing, /props, LM Studio catalog, /proc, kind detection).
+    ├── registration.ts  # Scan → pi-model mapping + provider registration (lazy).
+    ├── metrics.ts       # Server /metrics poller → ServerMetricsState (lazy; only if `metricsEnabled`).
+    ├── speed.ts         # Client-side speed tracker + footer status line (lazy; only if `metricsEnabled`).
+    ├── ui.ts            # /llamacpp-infra subcommands, menus, status, help (lazy).
+    └── prompt-warmup.ts # Header warmup: capture + cache system prompt KV (lazy; only if `warmup`).
 ```
 
-Two-file extension with zero external dependencies (only pi's bundled `@earendil-works/pi-coding-agent` + Node built-ins):
+Module load profile:
+
+| Module | Loaded when | Approx. size |
+|---|---|---|
+| `index.ts` + `core.ts` (+ `types.ts`) | Startup (static) | ~25 KB |
+| `scan.ts` + `registration.ts` | First discovery (dynamic) | ~27 KB |
+| `prompt-warmup.ts` | Primed at load if `warmup` enabled; not loaded when disabled | ~15 KB; skipped entirely when `warmup` is OFF |
+| `metrics.ts` + `speed.ts` | Primed at load if `metricsEnabled`; not loaded when disabled | ~18 KB; skipped entirely when `metricsEnabled` is OFF |
+| `ui.ts` | First `/llamacpp-infra …` command (dynamic) | ~32 KB |
+
+Zero external npm dependencies (only pi's bundled `@earendil-works/pi-coding-agent` + Node built-ins).
+
+Subsystems:
 
 - **Discovery engine** — multi-server probing with timeouts, retry budgets, and per-server kind detection (llama.cpp, ZINC, DwarfStar, lucebox, LM Studio)
 - **Router support** — single-model and multi-model llama.cpp modes with per-model status, args parsing and metadata extraction
-- **Metrics subsystem** — Prometheus endpoint discovery, polling, and compact widget rendering
+- **Speed & metrics subsystem** — client-side per-token speed measurement (prefill + moving-window generation), throttled footer status updates, and server `/metrics` polling that supplements the footer while the client is idle
 - **Thinking budgets** — per-model per-level configuration with automatic `reasoning` registration
 - **Config persistence** — `~/.pi/agent/llamacpp-infra.json` with one-time migration from `local-models.json`
 - **/proc scanner** — local llama-server process detection for vision, KV cache quant, and drafter flags

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "pi-llamacpp-infra",
-  "version": "1.2.0",
-  "description": "Discovery, metrics and control of llama.cpp-family servers for pi: probes any number of machines (localhost, LAN, Tailscale), registers every model into pi's native /model list, and provides live Prometheus metrics, per-model thinking budgets, vision detection and a native config UI. Supports llama.cpp, ZINC, DwarfStar/ds4, lucebox and LM Studio.",
+  "version": "1.2.2",
+  "description": "Discovery, metrics and control of llama.cpp-family servers for pi (llama.cpp, ZINC, DwarfStar/ds4, lucebox, LM Studio): scan machines (localhost, LAN, Tailscale), register models, live metrics, per-model thinking budgets, vision detection, native config UI.",
   "keywords": [
     "pi-package",
     "llamacpp",
@@ -28,7 +28,8 @@
   "pi": {
     "extensions": [
       "./src/index.ts"
-    ]
+    ],
+    "image": "https://raw.githubusercontent.com/noguerol/llamacpp-infra/main/docs/preview.jpeg"
   },
   "files": [
     "src"

package/src/core.ts

@@ -0,0 +1,200 @@
+// Core: config persistence, constants, id helpers, shared mutable state.
+// Statically imported by the entrypoint → must stay small.
+
+import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
+import { join } from "node:path";
+import { getAgentDir } from "@earendil-works/pi-coding-agent";
+import type {
+	CompatProfile,
+	InfraConfig,
+	ModelOptions,
+	ServerConfig,
+	ServerKind,
+	SettingsConfig,
+} from "./types.ts";
+
+// ── Provider identity ───────────────────────────────────────────────────────
+export const PROVIDER_NAME = "llamacpp-infra";
+export const STATUS_KEY = "llamacpp-infra";
+export const CONFIG_FILE = "llamacpp-infra.json";
+export const LEGACY_CONFIG_FILE = "local-models.json";
+export const METRICS_STATUS_KEY = "llamacpp-infra-speed";
+export const DEFAULT_API_KEY = "no-auth";
+export const THINKING_BUDGET_FIELD = "thinking_budget_tokens";
+
+// ── Settings defaults ───────────────────────────────────────────────────────
+export const DEFAULT_SETTINGS: SettingsConfig = {
+	discoveryTimeoutMs: 2000,
+	pollIntervalMs: 4000,
+	pollMaxMs: 90_000,
+	startupGraceMs: 40_000,
+	knownGoodFailLimit: 3,
+	detectVision: true,
+	prefixModelIds: true,
+	warmup: true,
+	metricsEnabled: true,
+	metricsPollMs: 5000,
+	includeUnloadedRouterModels: false,
+	showBadgesInNames: true,
+};
+
+export const DEFAULT_SERVERS: ServerConfig[] = [
+	{
+		id: "local",
+		host: "127.0.0.1",
+		label: "Local",
+		ports: [8000, 8001, 8002, 8080, 8081, 8082, 1234],
+		enabled: true,
+		probeDs4: false,
+	},
+];
+
+// ── Debug logging ──────────────────────────────────────────────────────────
+export const DEBUG = process.env.PI_LLAMACPP_INFRA_DEBUG === "1" || process.env.PI_LLAMACPP_INFRA_DEBUG === "true";
+export function debugLog(...args: unknown[]): void {
+	if (DEBUG) console.debug(`[llamacpp-infra]`, ...args);
+}
+
+// ── Config persistence ─────────────────────────────────────────────────────
+export function getConfigPath(): string {
+	return join(getAgentDir(), CONFIG_FILE);
+}
+
+export function loadConfig(): InfraConfig {
+	const defaults: InfraConfig = {
+		servers: DEFAULT_SERVERS,
+		settings: { ...DEFAULT_SETTINGS },
+		modelOptions: {},
+	};
+	const path = getConfigPath();
+	if (existsSync(path)) {
+		try {
+			const raw = JSON.parse(readFileSync(path, "utf-8")) as Partial<InfraConfig>;
+			return {
+				servers: Array.isArray(raw.servers) ? raw.servers : defaults.servers,
+				settings: { ...DEFAULT_SETTINGS, ...(raw.settings ?? {}) },
+				modelOptions: raw.modelOptions ?? {},
+			};
+		} catch (err) {
+			console.error(`[llamacpp-infra] Config load error: ${err}`);
+		}
+		return defaults;
+	}
+	const legacyPath = join(getAgentDir(), LEGACY_CONFIG_FILE);
+	if (existsSync(legacyPath)) {
+		try {
+			const raw = JSON.parse(readFileSync(legacyPath, "utf-8")) as Partial<InfraConfig>;
+			const migrated: InfraConfig = {
+				servers: Array.isArray(raw.servers) ? raw.servers : defaults.servers,
+				settings: { ...DEFAULT_SETTINGS, ...(raw.settings ?? {}) },
+				modelOptions: raw.modelOptions ?? {},
+			};
+			debugLog(`migrated legacy config from ${legacyPath}`);
+			return migrated;
+		} catch {
+			// fall through to defaults
+		}
+	}
+	return defaults;
+}
+
+export function saveConfig(config: InfraConfig): void {
+	try {
+		const dir = getAgentDir();
+		if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
+		writeFileSync(getConfigPath(), JSON.stringify(config, null, 2), "utf-8");
+		debugLog(`config saved to ${getConfigPath()}`);
+	} catch (err) {
+		console.error(`[llamacpp-infra] Config save error: ${err}`);
+	}
+}
+
+// ── Compat profile ─────────────────────────────────────────────────────────
+export function supportsThinkingBudget(kind: ServerKind | "unknown" | "auto" | undefined): boolean {
+	return kind === "llamacpp" || kind === "lucebox";
+}
+
+export function makeCompat(kind: ServerKind | "unknown" | "auto"): CompatProfile {
+	const usageInStreaming = kind !== "zinc";
+	return {
+		supportsDeveloperRole: false,
+		supportsReasoningEffort: false,
+		maxTokensField: "max_tokens" as const,
+		supportsUsageInStreaming: usageInStreaming,
+		supportsStrictMode: false,
+		...(supportsThinkingBudget(kind) ? { thinkingTokenBudgetField: THINKING_BUDGET_FIELD } : {}),
+	};
+}
+
+// ── Shared mutable state ───────────────────────────────────────────────────
+// Maps used by hooks + lazy modules. Kept in one place so lazy imports share
+// them via static import of core.ts.
+export const shared = {
+	activeConfig: undefined as InfraConfig | undefined,
+	zincModelIds: new Set<string>(),
+	modelBaseUrls: new Map<string, string>(),
+	endpointKinds: new Map<string, string>(),
+	serverModelIds: new Map<string, string>(),
+	compactModelIds: new Map<string, string>(),
+	lastScan: undefined as import("./types.ts").ScanResult | undefined,
+	lastModels: [] as import("./types.ts").PiModel[],
+	registeredCount: 0,
+	lastError: undefined as string | undefined,
+	rIncludeUnloaded: false,
+};
+
+export function setActiveConfig(c: InfraConfig | undefined): void {
+	shared.activeConfig = c;
+}
+
+export function modelOptions(): Record<string, ModelOptions> {
+	return shared.activeConfig?.modelOptions ?? {};
+}
+
+/** Compact id registered in pi for a raw server model id (or the input). */
+export function compactIdFor(modelId: string | undefined): string | undefined {
+	if (!modelId) return undefined;
+	return shared.compactModelIds.get(modelId) ?? modelId;
+}
+
+/** Raw server-side id to send in requests for a registered model id (or the input). */
+export function rawIdFor(modelId: string | undefined): string | undefined {
+	if (!modelId) return undefined;
+	return shared.serverModelIds.get(modelId) ?? modelId;
+}
+
+// ── Thinking-level normalization (used by the per-model budget hook) ──────
+export function normalizeLevel(level: string | undefined): "minimal" | "low" | "medium" | "high" | undefined {
+	switch (level) {
+		case "minimal":
+		case "low":
+		case "medium":
+		case "high":
+			return level;
+		case "xhigh":
+		case "max":
+			return "high";
+		default:
+			return undefined;
+	}
+}
+
+// ── Server-label / id helpers ──────────────────────────────────────────────
+export function serverLabel(srv: ServerConfig): string {
+	return srv.label?.trim() || srv.host;
+}
+
+export function idSafeHost(host: string): string {
+	return host.trim().toLowerCase() || "host";
+}
+
+export function isLocalHost(host: string): boolean {
+	const h = host.trim().toLowerCase();
+	return h === "127.0.0.1" || h === "localhost" || h === "::1";
+}
+
+export function baseName(p: string): string {
+	const cleaned = p.replace(/\\/g, "/");
+	const last = cleaned.split("/").pop() || cleaned;
+	return last;
+}