pi-llama-cpp 0.7.1 → 0.8.0

package/PLAN.md

@@ -0,0 +1,349 @@
+# Model Download Feature — Implementation Plan
+
+## Overview
+
+Add a `/models-download` command that lets users:
+
+- **Download** a model by typing a HuggingFace repo string (e.g. `ggml-org/gemma-3-4b-it-GGUF:Q4_K_M`)
+- **Cancel** an in-progress download
+- **Delete** a cached model via `/models-download delete`
+
+Download progress is shown in real time via the existing SSE infrastructure. After completion, the user is notified and invited to run `/models` to refresh.
+
+Extra documentation of the server's internals can be found in `~/Proyectos/llama-server/README.md`.
+
+---
+
+## Key Insight
+
+When downloading, the model **does not yet exist** in the local model list. The HF repo string itself acts as the model identifier in SSE events. This means:
+
+- Download/cancel/delete are **`Server` operations**, not `BaseModel` methods.
+- SSE subscriptions use the raw HF repo string as the `modelId`.
+- No changes to `BaseModel` are needed.
+
+## Command Structure
+
+The `/models-download` command has three subcommands:
+
+| Subcommand                   | Description                    |
+| ---------------------------- | ------------------------------ |
+| `/models-download` (no args) | Interactive download flow      |
+| `/models-download delete`    | Interactive delete flow        |
+| `/models-download cancel`    | Cancel an in-progress download |
+
+Each subcommand is routed by `DownloadManager.handleCommand(args, ctx, pi)`.
+
+---
+
+## Files to Create
+
+### `src/managers/download.ts` — DownloadManager
+
+Orchestrates the download, cancel, and delete workflows.
+
+```
+class DownloadManager
+├── constructor(serverManager, pi)
+├── handleCommand(args, ctx, pi)   → routes to download / delete / cancel flow
+├── handleDownloadFlow(ctx)        → interactive: input HF repo → confirm → download with progress
+├── handleDeleteFlow(ctx)          → interactive: select deletable model → confirm → delete
+├── handleCancelFlow(ctx)          → cancel in-progress download
+└── helpers: selectServer, selectModel
+```
+
+**Dependencies:** `ServerManager` (needs `getServers()` accessor), `pi: ExtensionAPI` (for cleanup notifications).
+
+---
+
+## Files to Modify
+
+### 1. `src/server.ts`
+
+Add four methods:
+
+| Method                   | HTTP                                 | Purpose                                   |
+| ------------------------ | ------------------------------------ | ----------------------------------------- |
+| `downloadModel(hfRepo)`  | `POST /models`                       | Trigger download with `{ model: hfRepo }` |
+| `deleteModel(modelId)`   | `DELETE /models?model={id}`          | Remove cached model                       |
+| `cancelDownload(hfRepo)` | `POST /models/unload`                | Cancel in-progress download               |
+| `probeSSESupport()`      | delegates to `SSEManager.probeSSE()` | Check `/models/sse` availability          |
+
+**Note:** All four endpoints are confirmed by the llama-server [API documentation](https://github.com/ggml-org/llama.cpp/blob/main/server/README.md). `POST /models/unload` works for both unloading loaded models AND canceling downloads.
+
+**Also:** Add `ApiClient.do_delete()` for DELETE requests. Unlike `get`/`post`, `delete` does **not** need caching since it's a mutation. No public `delete()` wrapper is needed (unlike `get`/`post`).
+
+```typescript
+// New signature
+async do_delete<T>(endpoint: string): Promise<T>
+```
+
+`do_delete` makes a raw DELETE request with `Authorization: Bearer ${this.apiKey}` (same auth pattern as `do_get`/`do_post`).
+
+**Note:** No changes to `BaseModel`. Download and delete are server-level operations that work with model IDs/strings directly, not model instances.
+
+### 2. `src/managers/server.ts`
+
+Add a public accessor (currently `servers` is private):
+
+```typescript
+getServers(): Server[] {
+  return this.servers;
+}
+```
+
+### 3. `src/interfaces/endpoints/models.ts`
+
+Add `can_remove` and `need_download` to `DataProperty` (they're already there — confirmed). No changes needed.
+
+### 4. `src/index.ts`
+
+- Import and instantiate `DownloadManager`
+- Register the `/models-download` command
+- Pass `serverManager` and `pi` to it
+
+### 5. `src/sse/manager.ts`
+
+Add `subscribeToDownloadStatus(modelId: string): Promise<'success' | 'failed'>` — see implementation details in the SSE Event Mapping section above.
+
+### 6. `src/sse/types.ts`
+
+Add `download_finished` and `download_failed` to `SSEEventType` (they're already there — confirmed). No changes needed.
+
+---
+
+## Download Flow (detailed)
+
+```
+User: /models-download
+  │
+  ├─ SSE supported? (probeSSESupport)
+  │   └─ No → notify error, exit
+  │
+  ├─ Always select server (even if only one) → ctx.ui.select("Select server:", [...])
+  │   └─ Format: "Llama.cpp (http://127.0.0.1:8080)"
+  │
+  ├─ ctx.ui.input("Enter HuggingFace model", "user/repo:quant")
+  │   └─ Cancelled → exit
+  │
+  ├─ Basic validation (must contain "/")
+  │   └─ Invalid → notify error, exit
+  │
+  ├─ ctx.ui.confirm("Confirm Download", "Download <repo>?")
+  │   └─ No → exit
+  │
+  ├─ server.downloadModel(hfRepo)
+  │   └─ Response: { success: true } or { error: { code, message, type } }
+  │       └─ If error → notify error, exit
+  │
+  ├─ Subscribe to SSE progress (hfRepo as modelId)
+  │   └─ On each event: ctx.ui.notify("Downloading ... X%", "info")
+  │
+  ├─ Wait for terminal SSE event (download_finished / download_failed)
+  │   ├─ Success → notify "Download complete. Run /models to refresh."
+  │   └─ Failure → notify "Download failed: <reason>"
+  │
+  ├─ Call GET /models to update the model list
+  │
+  └─ Cleanup SSE subscription
+```
+
+### Download Timeout
+
+Downloads can take several minutes. **No timeout** is applied to the download status subscription — the SSE connection will wait indefinitely for `download_finished` or `download_failed`. The `subscribeToProgress` call already handles progress updates in real time.
+
+## Delete Flow (detailed)
+
+```
+User: /models-download delete
+  │
+  ├─ Always select server (even if only one) → ctx.ui.select("Select server:", [...])
+  │   └─ Format: "Llama.cpp (http://127.0.0.1:8080)"
+  │
+  ├─ Fetch fresh model list from server (GET /models)
+  │
+  ├─ Filter models where can_remove === true
+  │   └─ None → notify "No deletable models", exit
+  │   └─ Note: models still downloading are NOT in the list yet, so they cannot be deleted.
+  │
+  ├─ ctx.ui.select("Select model to delete:", [...])
+  │   └─ Cancelled → exit
+  │
+  ├─ ctx.ui.confirm("Confirm Delete", "Delete <name>? This cannot be undone.")
+  │   └─ No → exit
+  │
+  ├─ server.deleteModel(modelId)
+  │   └─ Response: { success: true }
+  │   ├─ Success → notify "Deleted <name>. Run /models to refresh."
+  │   └─ Failure → notify error
+  │
+  └─ Done
+```
+
+## Cancel Download
+
+The `cancelDownload` method on `Server` sends `POST /models/unload` with the HF repo string. Per the llama-server docs, `POST /models/unload` cancels in-progress downloads (it's the same endpoint used to unload loaded models).
+
+Exposed as `/models-download cancel` subcommand. The flow:
+
+```
+User: /models-download cancel
+  │
+  ├─ Always select server (even if only one) → ctx.ui.select("Select server:", [...])
+  │   └─ Format: "Llama.cpp (http://127.0.0.1:8080)"
+  │
+  ├─ ctx.ui.input("Enter HF repo to cancel", "user/repo:quant")
+  │   └─ Cancelled → exit
+  │
+  ├─ server.cancelDownload(hfRepo)
+  │   └─ Sends POST /models/unload with { model: hfRepo }
+  │
+  ├─ Notify: "Download cancelled for <repo>."
+  │
+  └─ Done
+```
+
+**Note:** `cancelDownload` is semantically distinct from `downloadModel` even though they use the same endpoint. It should be a separate method for clarity.
+
+---
+
+## SSE Event Mapping
+
+The existing `SSEManager` already handles these event types:
+
+| SSE Event                       | Current handling                                             | Used by       |
+| ------------------------------- | ------------------------------------------------------------ | ------------- |
+| `download_progress`             | `subscribeToProgress()` parses `done/total` → percentage     | Download flow |
+| `status_change` with `progress` | `subscribeToProgress()` parses `value` → percentage + stage  | Download flow |
+| `download_finished`             | Not explicitly handled yet — need to add terminal resolution | Download flow |
+| `download_failed`               | Not explicitly handled yet — need to add terminal rejection  | Download flow |
+
+**Gap:** `SSEManager.subscribeToStatus()` currently only resolves on `status_change` events with `loaded`/`failed`. For downloads, we need it to also resolve on `download_finished`/`download_failed` events. Two options:
+
+- **(A)** Add a new `subscribeToDownloadStatus(modelId)` method that resolves on download terminal events.
+- **(B)** Extend `subscribeToStatus()` to also handle download terminal events.
+
+**Recommendation:** Option (A) — keep concerns separated. The download manager calls `subscribeToDownloadStatus()`.
+
+**`subscribeToDownloadStatus(modelId)` implementation:**
+
+```typescript
+subscribeToDownloadStatus(modelId: string): Promise<'success' | 'failed'> {
+  return new Promise((resolve, reject) => {
+    // No timeout — downloads can take several minutes.
+    // The SSE connection waits indefinitely for a terminal event.
+
+    this.subscribeToSSE(modelId, (event: SSEEvent) => {
+      if (event.event === SSEEventType.download_finished) {
+        resolve('success');
+      } else if (event.event === SSEEventType.download_failed) {
+        reject(new Error(`Download failed for model: ${modelId}`));
+      }
+    });
+  });
+}
+```
+
+- Subscribes to SSE for the HF repo string as `modelId`.
+- Resolves with `"success"` on `download_finished`.
+- Rejects with an error on `download_failed`.
+- The event `data` field contains the model identifier string (e.g., `"ggml-org/gemma-3-4b-it-GGUF:Q4_K_M"`), but we don't need to parse it — the `modelId` parameter already identifies the model.
+
+**SSE Event Payloads** (from llama-server docs):
+
+```json
+// download_finished / download_failed
+{
+  "model": "ggml-org/gemma-3-4b-it-GGUF:Q4_K_M"
+}
+```
+
+Both events carry the model identifier in `data`. No additional error details — errors are surfaced via the event type itself (`download_failed` vs `download_finished`).
+
+```json
+// model_remove — sent after successful DELETE /models
+{
+  "model": "ggml-org/gemma-3-4b-it-GGUF:Q4_K_M"
+}
+```
+
+The `model_remove` event is sent by the server after a successful delete. The extension doesn't need to handle it explicitly — the user runs `/models` to refresh the list anyway. But it's good to be aware of it in case we want to auto-refresh later.
+
+```json
+// models_reload — sent after model list changes (download completes, delete completes)
+{
+  "model": "*"
+}
+```
+
+The `models_reload` event uses `"model": "*"` (global event). This is sent after downloads complete and after deletes. We don't need to handle it in v1 — the download flow explicitly calls `GET /models` to refresh. But it's useful context for future auto-refresh features.
+
+**Note:** `SSEManager.probeSSE()` already exists and is cached. `SSEEventType` already includes `download_finished` and `download_failed`. The `subscribeToProgress()` method already handles `download_progress` events. We just need to add a terminal resolver.
+
+---
+
+## Notification Types
+
+Pi's `ctx.ui.notify()` accepts `"info" | "warning" | "error"`. There is no `"success"` type. Use `"info"` for success messages.
+
+---
+
+## API Response Formats
+
+### `POST /models` (download)
+
+**Success:**
+
+```json
+{ "success": true }
+```
+
+**Error:**
+
+```json
+{
+  "error": {
+    "code": 400,
+    "message": "model validation failed, unable to download",
+    "type": "invalid_request_error"
+  }
+}
+```
+
+### `DELETE /models?model={id}` (delete)
+
+**Success:**
+
+```json
+{ "success": true }
+```
+
+### `POST /models/unload` (cancel download)
+
+Same endpoint as unload — response format not explicitly documented by llama-server. Likely returns `{ success: true }` or an empty object. We'll treat any non-throwing response as success.
+
+---
+
+## Server Selection UX
+
+In all three flows (download, delete, cancel), the user always selects a server first — even if there's only one server configured. This keeps the UX consistent and future-proofs the feature.
+
+```typescript
+// Example: server list labels
+const choices = servers.map((s) => `${PROVIDER_NAME} (${s.baseUrl})`);
+// → ["Llama.cpp (http://127.0.0.1:8080)", "Llama.cpp (http://127.0.0.1:8081)"]
+const choice = await ctx.ui.select("Select server:", choices);
+```
+
+---
+
+## TODO Comments
+
+Add `// TODO: Verify if download/delete needs to be gated to router mode` in:
+
+- `DownloadManager.handleCommand()`
+- `Server.downloadModel()`
+- `Server.deleteModel()`
+- `Server.cancelDownload()`
+
+---

package/README.md

@@ -13,6 +13,7 @@ A [Pi Coding Agent](https://pi.dev/) extension that integrates with running [lla
 - **Auth support** — allows to login into a llama.cpp server that was secured with an API key
 - **Multiple server support** — connect to multiple llama.cpp servers simultaneously by separating URLs with semicolons
 - **Thinking budget support** — configurable token budgets for model reasoning/thinking, mapped to Pi's thinking levels
+- **Real-time progress tracking** — live loading progress via SSE (falls back to polling)
 
 ### Status Indicators
 
@@ -127,10 +128,10 @@ llama-server --model path/to/model.gguf ...
 The extension determines the context size as follows:
 
 - **Router mode**
-  - When loaded, reads `meta.n_ctx` from the `/models` endpoint
+  - When loaded, reads `meta.n_ctx` from the `/v1/models` endpoint
   - When not loaded, reads `--ctx-size` and/or `--fit-ctx` from the server arguments (which can also originate from the **presets.ini** file the llama.cpp server uses to load its models).
-- **Single mode** — reads `meta.n_ctx` from the `/models` endpoint
-- **Legacy mode** — reads `max_model_len` from `/models`, falling back to `n_ctx` from `/props`
+- **Single mode** — reads `meta.n_ctx` from the `/v1/models` endpoint
+- **Legacy mode** — reads `max_model_len` from `/v1/models`, falling back to `n_ctx` from `/props`
 - Falls back to `128000` if not available
 
 ### Commands
@@ -200,16 +201,18 @@ This keeps the server in sync with the active model in Pi, regardless of how the
 
 ### Loading Models
 
-When you trigger a load, switch, or retry action, the extension polls the server to track progress. If a model takes longer than **60 seconds** to load, the polling times out with an error.
+When you trigger a load, switch, or retry action, the extension uses SSE (Server-Sent Events) to receive real-time progress updates from the server. If SSE is not available, it falls back to polling.
 
-> **Note:** The timeout is only for the polling. The model might still be loading.
+If loading takes longer than **60 seconds**, the operation times out with an error.
+
+> **Note:** The timeout only applies to the progress detection. The model might still be loading in the background.
 
 ### Model Configuration
 
 Each model exposed to Pi includes the following defaults:
 
 - **`maxTokens`** — dynamically set to the model's context window (detected from llama-server)
-- **`reasoning`** — `true` (assumed, as llama.cpp's `/models` endpoint does not expose it)
+- **`reasoning`** — `true` (assumed, as llama.cpp's `/v1/models` endpoint does not expose it)
 - **`cost`** — all zero (local models)
 
 ## Dependencies

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-llama-cpp",
-  "version": "0.7.1",
+  "version": "0.8.0",
   "description": "Pi extension for llama.cpp integration. Supports router, single and legacy models. Supports multiple servers.",
   "keywords": [
     "pi",
@@ -36,8 +36,8 @@
     "@earendil-works/pi-tui": "*"
   },
   "devDependencies": {
-    "@types/node": "^25.9.3",
+    "@types/node": "^26.0.1",
     "prettier-plugin-organize-imports": "^4.3.0",
-    "vitest": "^4.1.8"
+    "vitest": "^4.1.9"
   }
 }

package/src/api/client.ts

@@ -0,0 +1,121 @@
+import { POLLING_INTERVAL } from "../constants";
+import { Cache } from "../utils/cache";
+import { Mutex } from "../utils/mutex";
+
+/**
+ * HTTP client for llama-server with caching and deduplication.
+ */
+export class ApiClient {
+  private cache = new Cache(POLLING_INTERVAL / 2);
+  private mutex = new Mutex();
+
+  /**
+   * Creates a new ApiClient.
+   *
+   * @param baseUrl The base URL of the llama-server
+   * @param apiKey The API key for authentication
+   */
+  constructor(
+    private readonly baseUrl: string,
+    private readonly apiKey: string,
+  ) {}
+
+  /**
+   * Makes a cached, deduplicated GET request to the llama-server.
+   * Results are cached for half the polling interval and in-flight requests are deduplicated.
+   *
+   * @param endpoint The endpoint path to fetch (e.g. "/health")
+   * @returns The parsed JSON response from the server
+   */
+  async get<T>(endpoint: string): Promise<T> {
+    const cached = this.cache.get<T>(endpoint);
+    if (cached !== undefined) return cached;
+
+    return this.mutex.getOrCreate(endpoint, async () => {
+      const data = (await this.do_get<T>(endpoint)) as T;
+      this.cache.set(endpoint, data);
+      return data;
+    });
+  }
+
+  /**
+   * Makes a cached, deduplicated POST request to the llama-server.
+   * Results are cached for half the polling interval and in-flight requests are deduplicated.
+   *
+   * @param endpoint The endpoint path to post to
+   * @param body The optional request body
+   * @returns The parsed JSON response from the server
+   */
+  async post<T>(endpoint: string, body?: Record<string, unknown>): Promise<T> {
+    const key = this.cacheKey(endpoint, body);
+    const cached = this.cache.get<T>(key);
+    if (cached !== undefined) return cached;
+
+    return this.mutex.getOrCreate(key, async () => {
+      const data = (await this.do_post<T>(endpoint, body)) as T;
+      this.cache.set(key, data);
+      return data;
+    });
+  }
+
+  /**
+   * Clears the entire cache.
+   */
+  clearCache(): void {
+    this.cache.clear();
+  }
+
+  /**
+   * Makes a raw GET request to the llama-server.
+   * This bypasses caching and deduplication.
+   *
+   * @param endpoint The endpoint path to fetch (e.g. "/health")
+   * @returns The parsed JSON response from the server
+   */
+  private async do_get<T>(endpoint: string): Promise<T> {
+    const url = `${this.baseUrl}${endpoint}`;
+
+    const res = await fetch(url, {
+      headers: { Authorization: `Bearer ${this.apiKey}` },
+    });
+
+    return res.json();
+  }
+
+  /**
+   * Makes a raw POST request to the llama-server.
+   * This bypasses caching and deduplication.
+   *
+   * @param endpoint The endpoint path to post to
+   * @param body The optional request body
+   * @returns The parsed JSON response from the server
+   */
+  private async do_post<T>(
+    endpoint: string,
+    body?: Record<string, unknown>,
+  ): Promise<T> {
+    const url = `${this.baseUrl}${endpoint}`;
+
+    const res = await fetch(url, {
+      method: "POST",
+      headers: {
+        "Content-Type": "application/json",
+        Authorization: `Bearer ${this.apiKey}`,
+      },
+      body: body ? JSON.stringify(body) : undefined,
+    });
+
+    return res.json();
+  }
+
+  /**
+   * Sets a cache key
+   *
+   * @param endpoint The endpoint path to post to
+   * @param body The optional request body
+   * @returns The cache key
+   */
+  private cacheKey(endpoint: string, body?: Record<string, unknown>): string {
+    return body ? `${endpoint}:${JSON.stringify(body)}` : endpoint;
+  }
+}

package/src/constants.ts

@@ -44,7 +44,7 @@ export const POLLING_TIMEOUT = 60000;
 export const READABLE_TIMEOUT = 15000;
 
 /**
- * Timeout (ms) for server verification before assuming failure
+ * Timeout (ms) for server verification and SSE support probe
  */
 export const SERVER_TIMEOUT = 1000;
 

package/src/interfaces/endpoints/models.ts

@@ -40,6 +40,9 @@ export interface DataProperty {
   created: number;
   status?: StatusProperty;
   architecture?: ArchitectureProperty;
+  source?: string;
+  can_remove?: boolean;
+  need_download?: boolean;
   meta?: MetaProperty;
 }
 

package/src/interfaces/endpoints/props.ts

@@ -2,28 +2,56 @@
  * The structure of llama-server's /props endpoint
  */
 export interface PropsEndpoint {
-  role?: "router";
-  error?: PropsError;
+  role: "router";
+  max_instances: number;
+  models_autoload: boolean;
+  model_alias: string;
+  model_path: string;
   default_generation_settings: Record<string, any>;
+  ui_settings: Record<string, any>;
+  build_info: string;
+  cors_proxy_enabled: boolean;
+}
+
+/**
+ * The structure of llama-server's /props?model=<id> endpoint
+ */
+export interface PropsModelEndpoint {
+  error?: PropsError;
+  default_generation_settings: {
+    params: Record<string, any>;
+    n_ctx: number;
+  };
   total_slots: number;
   model_alias: string;
   model_path: string;
   modalities: {
     vision: boolean;
+    video: boolean;
     audio: boolean;
   };
   media_marker: string;
   endpoint_slots: boolean;
   endpoint_props: boolean;
   endpoint_metrics: boolean;
-  webui: boolean;
-  webui_settings: Record<string, any>;
+  ui: boolean;
+  ui_settings: Record<string, any>;
   chat_template: string;
-  chat_template_caps: Record<string, boolean>;
+  chat_template_caps: {
+    supports_object_arguments: boolean;
+    supports_parallel_tool_calls: boolean;
+    supports_preserve_reasoning: boolean;
+    supports_string_content: boolean;
+    supports_system_role: boolean;
+    supports_tool_calls: boolean;
+    supports_tools: boolean;
+    supports_typed_content: boolean;
+  };
   bos_token: string;
   eos_token: string;
   build_info: string;
   is_sleeping: boolean;
+  cors_proxy_enabled: boolean;
 }
 
 export interface PropsError {

package/src/managers/command.ts

@@ -134,6 +134,17 @@ export class CommandManager {
       ctx.ui.notify(`Loading ${model.name}...`, "info");
       EventManager.inflightModel = model;
 
+      // Subscribe to progress events
+      const cleanupProgress = this.serverManager
+        .getServer(model)
+        .sseManager.subscribeToProgress(model.id, (percentage, stage) => {
+          const stageText = stage ? ` (${stage})` : "";
+          ctx.ui.notify(
+            `Loading ${model.name}... [${percentage}%${stageText}]`,
+            "info",
+          );
+        });
+
       const onSuccess = async () => {
         const { serverId } = model;
         const piModel = ctx.modelRegistry.find(serverId, model.id);
@@ -171,7 +182,10 @@ export class CommandManager {
         .load()
         .then(onSuccess)
         .catch(onFailure)
-        .finally(EventManager.resetInflightModel);
+        .finally(() => {
+          cleanupProgress();
+          EventManager.resetInflightModel();
+        });
     }
   }
 

package/src/managers/server.ts

@@ -117,6 +117,16 @@ export class ServerManager {
     return warnings;
   }
 
+  /**
+   * Returns the server for a given model.
+   *
+   * @param model - The model to find the server for
+   * @returns The server containing the model
+   */
+  getServer(model: BaseModel): Server {
+    return this.servers.find((s) => s.baseUrl === model.serverUrl)!;
+  }
+
   /**
    * Returns all models from all servers.
    *

package/src/models/baseModel.ts

@@ -87,9 +87,11 @@ export abstract class BaseModel {
       const model = data.find((d) => d.id === this.id);
       if (!model) return ["text"];
 
-      const { input_modalities } = model.architecture!;
+      const input_modalities: ("text" | "image" | "audio")[] = model
+        .architecture?.input_modalities ?? ["text"];
+
       const response = input_modalities.filter(
-        (mod) => mod === "text" || mod === "image",
+        (mod): mod is "text" | "image" => mod === "text" || mod === "image",
       );
 
       return response;
@@ -189,14 +191,25 @@ export abstract class BaseModel {
   }
 
   /**
-   * Loads the model in llama-server
+   * Loads the model in llama-server.
+   * Uses SSE status events when available, falling back to polling.
    */
   async load(): Promise<void> {
     const status = await this.getStatus();
     if (status === Status.LOADED || status === Status.SLEEPING) return;
 
     await this.server.postRequest("load", this.id);
-    await this.pollStatus();
+
+    if (await this.server.sseManager.probeSSE()) {
+      const { status, exit_code } =
+        await this.server.sseManager.subscribeToStatus(this.id);
+
+      if (status === "failed" || (status === "unloaded" && exit_code !== 0)) {
+        throw new Error(`Model loading failed: ${this.id}`);
+      }
+    } else {
+      await this.pollStatus();
+    }
   }
 
   /**

package/src/models/routerModel.ts

@@ -14,8 +14,7 @@ export class RouterModel extends BaseModel {
   }
 
   /**
-   * Workaround for the currently-bugged /models status detection
-   * (I suspect it was introduced in PR #22683 of llama.cpp)
+   * Workaround for /models status detection
    *
    * When a model is loaded for the very first time,
    * this workaround will try to poll to /props instead of /models

package/src/server.ts

@@ -1,21 +1,35 @@
+import { ApiClient } from "./api/client";
 import { PROVIDER_NAME, PROVIDER_PREFIX } from "./constants";
 import { Mode } from "./enums/mode";
 import { ServerStatus } from "./enums/serverStatus";
 import { HealthEndpoint } from "./interfaces/endpoints/health";
 import { ModelsEndpoint } from "./interfaces/endpoints/models";
-import { PropsEndpoint } from "./interfaces/endpoints/props";
+import {
+  PropsEndpoint,
+  PropsModelEndpoint,
+} from "./interfaces/endpoints/props";
 import { BaseModel } from "./models/baseModel";
 import { LegacyModel } from "./models/legacyModel";
 import { RouterModel } from "./models/routerModel";
 import { SingleModel } from "./models/singleModel";
 import { ConfigResolver } from "./resolver";
+import { SSEManager } from "./sse/manager";
 
 export class Server {
   public readonly models: BaseModel[] = [];
   private configResolver = new ConfigResolver();
+  private apiClient!: ApiClient;
+  private sse!: SSEManager;
 
   constructor(readonly baseUrl: string) {}
 
+  /**
+   * Provides access to the SSE manager for direct subscriptions.
+   */
+  get sseManager(): SSEManager {
+    return this.sse;
+  }
+
   /**
    * Generates a unique provider ID from a server URL.
    */
@@ -39,9 +53,13 @@ export class Server {
   }
 
   /**
-   * Fetches models from the server and populates {@link models}
+   * Fetches models from the server and populates {@link models}.
+   * Clears the cache first so we always fetch fresh data.
    */
   async initialize() {
+    const apiKey = await this.getApiKey();
+    this.apiClient = new ApiClient(this.baseUrl, apiKey);
+    this.sse = new SSEManager(this.baseUrl, apiKey);
     const { data } = await this.fetchModels();
     const mode = await this.detectServerMode();
 
@@ -81,6 +99,8 @@ export class Server {
    * @returns The server status
    */
   async isReady(timeout: number): Promise<ServerStatus> {
+    this.apiClient ??= new ApiClient(this.baseUrl, await this.getApiKey());
+
     try {
       const timeoutPromise = new Promise<never>((_, reject) =>
         setTimeout(() => reject(new Error("timeout")), timeout),
@@ -107,7 +127,7 @@ export class Server {
    * @returns The health status
    */
   async fetchServerHealth(): Promise<HealthEndpoint> {
-    return await this.rpc<HealthEndpoint>("/health");
+    return await this.apiClient.get<HealthEndpoint>("/health");
   }
 
   /**
@@ -116,7 +136,7 @@ export class Server {
    * @return The models from the server
    */
   async fetchModels(): Promise<ModelsEndpoint> {
-    return await this.rpc<ModelsEndpoint>("/v1/models");
+    return await this.apiClient.get<ModelsEndpoint>("/v1/models");
   }
 
   /**
@@ -125,7 +145,7 @@ export class Server {
    * @return The properties of the server
    */
   async fetchServerProps(): Promise<PropsEndpoint> {
-    return await this.rpc<PropsEndpoint>("/props?autoload=false");
+    return await this.apiClient.get<PropsEndpoint>("/props?autoload=false");
   }
 
   /**
@@ -134,8 +154,8 @@ export class Server {
    * @param modelId The ID of the model
    * @return The properties of the specified model
    */
-  async fetchModelProps(modelId: string): Promise<PropsEndpoint> {
-    return await this.rpc<PropsEndpoint>(
+  async fetchModelProps(modelId: string): Promise<PropsModelEndpoint> {
+    return await this.apiClient.get<PropsModelEndpoint>(
       `/props?model=${modelId}&autoload=false`,
     );
   }
@@ -150,38 +170,9 @@ export class Server {
     resource: "load" | "unload",
     model: string,
   ): Promise<ModelsEndpoint> {
-    return await this.rpc<ModelsEndpoint>(`/models/${resource}`, { model });
-  }
-
-  /**
-   * Makes an HTTP request to the llama-server and returns the parsed JSON response
-   *
-   * @param endpoint The endpoint path to fetch (e.g. "/health")
-   * @param body The optional request body for POST requests
-   * @returns The parsed JSON response from the server
-   */
-  private async rpc<T>(
-    endpoint: string,
-    body?: Record<string, unknown>,
-  ): Promise<T> {
-    const url = `${this.baseUrl}${endpoint}`;
-    const apiKey = await this.getApiKey();
-
-    const data = {
-      method: body ? "POST" : "GET",
-      headers: body ? { "Content-Type": "application/json" } : undefined,
-      body: body ? JSON.stringify(body) : undefined,
-    };
-
-    const res = await fetch(url, {
-      ...data,
-      headers: {
-        ...data.headers,
-        ...(apiKey ? { Authorization: `Bearer ${apiKey}` } : {}),
-      },
+    this.apiClient.clearCache();
+    return await this.apiClient.post<ModelsEndpoint>(`/models/${resource}`, {
+      model,
     });
-
-    const response: T = await res.json();
-    return response;
   }
 }

package/src/sse/client.ts

@@ -0,0 +1,140 @@
+import { POLLING_INTERVAL } from "../constants";
+import type { SSECallback, SSECleanup, SSEEvent } from "./types";
+
+/**
+ * SSE client for llama-server's /models/sse endpoint.
+ *
+ * Uses a single shared EventSource per server instance.
+ * Supports multiple model subscriptions with automatic event routing.
+ * Handles reconnection by re-subscribing all callbacks.
+ */
+export class SSEClient {
+  private eventSource: EventSource | null = null;
+  private subscribers: Map<string, SSECallback> = new Map();
+  private connected: boolean = false;
+  private reconnecting: boolean = false; // tracks if EventSource auto-reconnect is in progress
+
+  /**
+   * @param sseEndpoint - The full SSE endpoint URL (e.g., "http://127.0.0.1:8080/models/sse")
+   * @param apiKey - Optional API key for authenticated servers
+   */
+  constructor(
+    private readonly sseEndpoint: string,
+    private readonly apiKey?: string,
+  ) {}
+
+  /**
+   * Connects to the SSE endpoint.
+   *
+   * @returns true if the connection was established successfully
+   */
+  async connect(): Promise<boolean> {
+    if (this.connected) return true;
+
+    const url = this.buildUrl();
+
+    try {
+      this.eventSource = new EventSource(url);
+    } catch {
+      this.connected = false;
+      return false;
+    }
+
+    this.eventSource.onopen = () => {
+      this.connected = true;
+      this.reconnecting = false;
+    };
+
+    this.eventSource.onerror = () => {
+      // EventSource will auto-reconnect; we just track state
+      this.connected = false;
+      this.reconnecting = true;
+    };
+
+    this.eventSource.onmessage = (event: MessageEvent) => {
+      try {
+        const data = JSON.parse(event.data);
+        const sseEvent: SSEEvent = {
+          event: data.event ?? "unknown",
+          model: data.model ?? "*",
+          data: data.data,
+        };
+        this.dispatch(sseEvent);
+      } catch {
+        // Invalid JSON, ignore
+      }
+    };
+
+    // Wait a bit for the connection to establish
+    await new Promise<void>((resolve) => {
+      const timeout = setTimeout(() => resolve(), POLLING_INTERVAL);
+      this.eventSource!.onopen = () => {
+        clearTimeout(timeout);
+        this.connected = true;
+        this.reconnecting = false;
+        resolve();
+      };
+    });
+
+    return this.connected;
+  }
+
+  /**
+   * Subscribes to SSE events for a specific model.
+   * Auto-connects if not already connected.
+   *
+   * @param modelId - The model ID to subscribe to
+   * @param callback - Callback to receive SSE events
+   * @returns A cleanup function to unsubscribe
+   */
+  subscribe(modelId: string, callback: SSECallback): SSECleanup {
+    this.subscribers.set(modelId, callback);
+
+    if (!this.connected && !this.reconnecting) {
+      this.connect();
+    }
+
+    return () => {
+      this.subscribers.delete(modelId);
+    };
+  }
+
+  /**
+   * Disconnects from the SSE endpoint and clears all subscriptions.
+   */
+  disconnect(): void {
+    if (this.eventSource) {
+      this.eventSource.close();
+      this.eventSource = null;
+    }
+    this.connected = false;
+    this.subscribers.clear();
+  }
+
+  /**
+   * Builds the full URL with optional API key query param.
+   */
+  private buildUrl(): string {
+    if (this.apiKey) {
+      return `${this.sseEndpoint}?api_key=${encodeURIComponent(this.apiKey)}`;
+    }
+    return this.sseEndpoint;
+  }
+
+  /**
+   * Dispatches an SSE event to all matching subscribers.
+   */
+  private dispatch(event: SSEEvent): void {
+    // Dispatch to model-specific subscriber
+    const modelCallback = this.subscribers.get(event.model);
+    if (modelCallback) {
+      modelCallback(event);
+    }
+
+    // Also dispatch to wildcard subscriber if present
+    const wildcardCallback = this.subscribers.get("*");
+    if (wildcardCallback && event.model !== "*") {
+      wildcardCallback(event);
+    }
+  }
+}

package/src/sse/manager.ts

@@ -0,0 +1,202 @@
+import { POLLING_TIMEOUT, SERVER_TIMEOUT } from "../constants";
+import { SSEClient } from "./client";
+import {
+  DownloadProgressData,
+  ProgressData,
+  SSECallback,
+  SSECleanup,
+  SSEEvent,
+  SSEEventType,
+  StatusChangeData,
+} from "./types";
+
+/**
+ * Manages SSE connections and event routing for a single llama-server instance.
+ *
+ * Handles:
+ * - Shared EventSource connection
+ * - Model-based event subscription with callback aggregation
+ * - Progress parsing and callback dispatch
+ */
+export class SSEManager {
+  private sseClient: SSEClient | null = null;
+  private sseSubscribers: Map<string, SSECleanup> = new Map();
+  private modelCallbacks: Map<string, SSECallback[]> = new Map();
+  private sseSupported: boolean | null = null;
+
+  constructor(
+    private readonly baseUrl: string,
+    private readonly apiKey: string,
+  ) {}
+
+  /**
+   * The SSE endpoint URL.
+   */
+  private get sseEndpoint(): string {
+    return `${this.baseUrl}/models/sse`;
+  }
+
+  /**
+   * Probes the SSE endpoint to check if it's supported.
+   * Result is cached for the lifetime of the manager.
+   *
+   * @returns true if SSE is supported
+   */
+  async probeSSE(): Promise<boolean> {
+    if (this.sseSupported !== null) return this.sseSupported;
+
+    try {
+      const headers: Record<string, string> = {};
+      if (this.apiKey) {
+        headers["Authorization"] = `Bearer ${this.apiKey}`;
+      }
+      const response = await fetch(this.sseEndpoint, {
+        method: "GET",
+        headers,
+        signal: AbortSignal.timeout(SERVER_TIMEOUT),
+      });
+      this.sseSupported =
+        response.ok &&
+        !!response.headers.get("content-type")?.includes("text/event-stream");
+    } catch {
+      this.sseSupported = false;
+    }
+
+    return this.sseSupported;
+  }
+
+  /**
+   * Subscribes to SSE events for a specific model.
+   * Uses a shared SSE connection per server.
+   * Aggregates multiple callbacks into one SSEClient subscription.
+   *
+   * @param modelId - The model ID to subscribe to
+   * @param callback - Callback to receive SSE events
+   * @returns A cleanup function to unsubscribe
+   */
+  private subscribeToSSE(
+    modelId: string,
+    callback: (event: SSEEvent) => void,
+  ): SSECleanup {
+    // Aggregate callbacks for this model
+    const callbacks = this.modelCallbacks.get(modelId) ?? [];
+    callbacks.push(callback);
+    this.modelCallbacks.set(modelId, callbacks);
+
+    // Create SSE client if not already created
+    this.sseClient ??= new SSEClient(this.sseEndpoint, this.apiKey);
+
+    // Subscribe a single dispatching callback to the SSE client
+    if (!this.sseSubscribers.has(modelId)) {
+      const dispatch = (event: SSEEvent) => {
+        for (const cb of callbacks) cb(event);
+      };
+      const cleanup = this.sseClient!.subscribe(modelId, dispatch);
+      this.sseSubscribers.set(modelId, cleanup);
+    }
+
+    return () => {
+      const list = this.modelCallbacks.get(modelId);
+      if (list) {
+        const idx = list.indexOf(callback);
+        if (idx !== -1) list.splice(idx, 1);
+        if (list.length === 0) {
+          this.modelCallbacks.delete(modelId);
+          const cleanup = this.sseSubscribers.get(modelId);
+          if (cleanup) cleanup();
+          this.sseSubscribers.delete(modelId);
+        }
+      }
+    };
+  }
+
+  /**
+   * Subscribes to SSE progress events for a specific model.
+   * Parses SSE events and calls the progress callback with percentage and stage.
+   *
+   * @param modelId - The model ID to subscribe to
+   * @param onProgress - Callback to receive progress updates (percentage 0-100, stage name)
+   * @returns A cleanup function to unsubscribe
+   */
+  subscribeToProgress(
+    modelId: string,
+    onProgress: (percentage: number, stage?: string) => void,
+  ): SSECleanup {
+    // Track download progress across multiple URLs
+    let totalDownloaded = 0;
+    let totalToDownload = 0;
+
+    return this.subscribeToSSE(modelId, (event: SSEEvent) => {
+      if (event.event === SSEEventType.status_change && event.data) {
+        const data = event.data as unknown as StatusChangeData;
+
+        if (data.status === "loading" && data.progress) {
+          const progress = data.progress as ProgressData;
+          const percentage = Math.round(progress.value * 100);
+          onProgress(percentage, progress.current);
+        } else if (data.status === "loaded" || data.status === "failed") {
+          // Reset download tracking on final state
+          totalDownloaded = 0;
+          totalToDownload = 0;
+        }
+      } else if (event.event === SSEEventType.download_progress && event.data) {
+        const downloadData = event.data as DownloadProgressData;
+        totalDownloaded = 0;
+        totalToDownload = 0;
+
+        for (const urlData of Object.values(downloadData)) {
+          totalDownloaded += urlData.done;
+          totalToDownload += urlData.total;
+        }
+
+        if (totalToDownload > 0) {
+          const percentage = Math.round(
+            (totalDownloaded / totalToDownload) * 100,
+          );
+          onProgress(percentage, "downloading");
+        }
+      }
+    });
+  }
+
+  /**
+   * Subscribes to SSE status change events for a specific model.
+   * Resolves with the final status string once the model reaches a terminal state.
+   *
+   * @param modelId - The model ID to subscribe to
+   * @returns Promise that resolves with the final status string
+   */
+  subscribeToStatus(modelId: string): Promise<StatusChangeData> {
+    return new Promise((resolve, reject) => {
+      const timeout = setTimeout(
+        () => reject(new Error(`SSE status timeout for model: ${modelId}`)),
+        POLLING_TIMEOUT,
+      );
+
+      this.subscribeToSSE(modelId, (event: SSEEvent) => {
+        if (event.event === SSEEventType.status_change && event.data) {
+          const data = event.data as unknown as StatusChangeData;
+          if (["loaded", "unloaded", "failed"].includes(data.status)) {
+            clearTimeout(timeout);
+            resolve(data);
+          }
+        }
+      });
+    });
+  }
+
+  /**
+   * Disconnects the SSE client and cleans up all subscriptions.
+   */
+  disconnect(): void {
+    for (const cleanup of this.sseSubscribers.values()) {
+      cleanup();
+    }
+    this.sseSubscribers.clear();
+    this.modelCallbacks.clear();
+    if (this.sseClient) {
+      this.sseClient.disconnect();
+      this.sseClient = null;
+    }
+  }
+}

package/src/sse/types.ts

@@ -0,0 +1,65 @@
+/**
+ * SSE event types from llama-server's /models/sse endpoint
+ */
+
+/**
+ * Possible event types from the SSE stream
+ */
+export const SSEEventType = {
+  status_change: "status_change",
+  download_progress: "download_progress",
+  download_finished: "download_finished",
+  download_failed: "download_failed",
+  models_reload: "models_reload",
+  model_remove: "model_remove",
+} as const;
+
+export type SSEEventType = (typeof SSEEventType)[keyof typeof SSEEventType];
+
+/**
+ * A parsed SSE event from the /models/sse endpoint
+ */
+export interface SSEEvent {
+  event: SSEEventType;
+  model: string; // model ID or "*" for global events
+  data?: Record<string, unknown>;
+}
+
+/**
+ * Progress data sent during model loading
+ */
+export interface ProgressData {
+  stages: string[];
+  current: string;
+  value: number; // 0.0 to 1.0
+}
+
+/**
+ * Data payload for status_change events
+ */
+export interface StatusChangeData {
+  status: string;
+  exit_code?: number;
+  info?: Record<string, unknown>;
+  progress?: ProgressData;
+}
+
+/**
+ * Data payload for download_progress events
+ */
+export interface DownloadProgressData {
+  [url: string]: {
+    done: number;
+    total: number;
+  };
+}
+
+/**
+ * Subscriber callback type for SSE events
+ */
+export type SSECallback = (event: SSEEvent) => void;
+
+/**
+ * Cleanup function to unsubscribe from SSE events
+ */
+export type SSECleanup = () => void;

package/src/utils/cache.ts

@@ -0,0 +1,39 @@
+/**
+ * Generic TTL cache.
+ * Entries expire after `ttl` milliseconds from the time they were set.
+ */
+export class Cache {
+  private entries = new Map<string, { data: unknown; timestamp: number }>();
+
+  /**
+   * @param ttl Time-to-live in milliseconds
+   */
+  constructor(private readonly ttl: number) {}
+
+  /**
+   * Gets a cached value by key. Returns `undefined` if missing or expired.
+   */
+  get<T>(key: string): T | undefined {
+    const entry = this.entries.get(key);
+    if (!entry) return undefined;
+    if (Date.now() - entry.timestamp > this.ttl) {
+      this.entries.delete(key);
+      return undefined;
+    }
+    return entry.data as T;
+  }
+
+  /**
+   * Stores a value in the cache with the current timestamp.
+   */
+  set(key: string, data: unknown): void {
+    this.entries.set(key, { data, timestamp: Date.now() });
+  }
+
+  /**
+   * Clears all cached entries.
+   */
+  clear(): void {
+    this.entries.clear();
+  }
+}

package/src/utils/mutex.ts

@@ -0,0 +1,24 @@
+/**
+ * Ensures only one in-flight operation exists per key.
+ * Concurrent callers for the same key share the same promise.
+ */
+export class Mutex {
+  private promises = new Map<string, Promise<unknown>>();
+
+  /**
+   * Runs `fn` for the given key, or returns an existing in-flight promise.
+   * Concurrent callers for the same key share the same promise.
+   */
+  getOrCreate<T>(key: string, fn: () => Promise<T>): Promise<T> {
+    const existing = this.promises.get(key);
+    if (existing) {
+      return existing as Promise<T>;
+    }
+
+    const promise = fn().finally(() => {
+      this.promises.delete(key);
+    });
+    this.promises.set(key, promise);
+    return promise;
+  }
+}