ollama-agent-router 0.1.7 → 0.1.9

package/README.md

@@ -94,6 +94,221 @@ Start with:
 ollama-agent-router serve --config examples/gex44.yaml
 ```
 
+`examples/gex44-secured.yaml` is the same hardware profile with the standalone plane locked down: API key required, anonymous access rejected, per-key rate limits, and the admin plane enabled on localhost. Use it as a starting point when the router is exposed beyond a single user or process.
+
+## Routing Algorithm
+
+### Candidate selection
+
+For every request the router builds a candidate list from three sources, merged in order:
+
+1. `router.preferredModels` from the request — added first, regardless of `routes`.
+2. `routes[taskType]` — the ordered list for the classified task type.
+3. Any model whose `purpose` or `tags` array contains the task type — acts as a catch-all fallback.
+
+Models listed in `router.forbiddenModels` are dropped from the candidate list entirely.
+
+### Blocking checks
+
+Before scoring, each candidate is checked for hard blocks:
+
+- **`gpu_only`** — `requireGpuOnly` is set (globally or per-request) and the model is not fully on GPU, has a CPU/GPU split in `ollama ps`, or there is not enough free VRAM to load it.
+- **`busy`** — the model has `exclusive: true` and is already running, or `allowWhenBusy: false` and has reached `maxConcurrent`.
+
+Blocked models are excluded from sync selection but can still be picked for async jobs.
+
+### Scoring
+
+Every non-blocked candidate receives a numeric score. Higher score wins. Starting value: **100**.
+
+| Component | Delta | Notes |
+|---|---|---|
+| Route position | `+50` for index 0, `−8` per step | First entry in `routes[taskType]` gets the full bonus |
+| `model.priority` | `+priority` | Set per model, 1–100 |
+| `purpose` match | `+25` | Model's `purpose` array contains the task type |
+| `preferredModels` | `+80` for index 0, `−10` per step | Request-level override |
+| Already loaded in Ollama | **`+20`** | Model appears in `ollama ps` output |
+| Heavy complexity + `costClass: high` | `+20` | Classifier returned `heavy`; rewards large models |
+| Light complexity + `costClass: low` | `+15` | Classifier returned `light`; rewards small models |
+| Free VRAM headroom | `+0..+25` | Scales with `(freeMb − requiredMb) / 512`, capped at 25 |
+| Insufficient VRAM | **`−60`** | `model.sizeGb × 1024 + vramSafetyReserveMb > freeMb` |
+| Queue depth | `−18 × queueDepth` | Per-model queue length |
+| Running count | `−25 × running` | Per-model active executions |
+| Exclusive + running | `−80 × running` additional | `exclusive: true` models penalised heavily while in use |
+
+The candidate with the highest score is selected. The others appear in `fallbackModels` in the response.
+
+### Model config fields that affect routing
+
+```yaml
+models:
+  - name: gpt-oss:20b
+    sizeGb: 14.0          # used for VRAM headroom calculation
+    purpose: [agentic_reasoning, large_context, planning, tool_use, complex_debugging]
+                          # +25 score when task type matches; also adds model to the candidate list
+    priority: 95          # added directly to score; use to rank models of similar capability
+    maxConcurrent: 1      # hard cap on parallel executions
+    costClass: high       # low | medium | high — matched against request complexity for bonus/penalty
+    exclusive: true       # if running, gets −80 extra penalty per execution; only one at a time
+    allowWhenBusy: false  # if false and maxConcurrent reached → blocked entirely
+```
+
+**`purpose`** — declares what the model can do. Each entry that matches the request's task type adds `+25` to the score and also makes the model a candidate even when it is not listed in `routes[taskType]`. Use it for every task type the model handles well, including secondary ones (e.g. add `agentic_reasoning` to a coder model that works as a capable fallback).
+
+**`costClass`** — signals the relative weight of the model:
+- `high`: gets `+20` when the classifier decides the request is complex (`heavy`). Intended for large reasoning models.
+- `low`: gets `+15` when the request is simple (`light`). Intended for small triage/chat models.
+- `medium`: no complexity bonus in either direction.
+
+**`exclusive`** — intended for large models that cannot safely share GPU memory with another concurrent execution. While one request is running, the model accumulates `−80` per running job on top of the standard `−25`, making it effectively unselectable for sync requests until free.
+
+### `routes` config and its relation to scoring
+
+```yaml
+routes:
+  agentic_reasoning: [gpt-oss:20b, qwen2.5-coder:7b]
+```
+
+Order matters: `gpt-oss:20b` at index 0 gets `+50`, `qwen2.5-coder:7b` at index 1 gets `+42`. Each additional position costs `−8`.
+
+A model does not need to be in `routes` to be selected — if it declares the task type in `purpose` or `tags` it will still enter the candidate list (with a route-position score of 0).
+
+### Sync vs async decision
+
+After scoring, the router checks whether to run synchronously or push to the async queue:
+
+1. If `router.mode: async` — always async.
+2. If heavy load is detected (total queue depth ≥ `router.heavyLoadQueueDepth` **or** free VRAM < `router.heavyLoadGpuFreeMbThreshold`) and `allowAsync: true` — async.
+3. If the top-scored model is busy and `allowAsync: true` — async on that model.
+4. Otherwise — sync on the top-scored model.
+
+`allowAsync` defaults to `true`. Set `"router": {"mode": "sync"}` in the request to force synchronous execution regardless of load.
+
+### Forcing a specific model
+
+`preferredModels` adds `+80` to the first entry, making it win unless blocked by VRAM or busy constraints. `forbiddenModels` removes models from the candidate list entirely — useful when testing a specific model in isolation.
+
+### Request examples
+
+**Explicit task type — let the router pick the best model for the task:**
+
+```bash
+curl -s http://127.0.0.1:11435/v1/chat/completions \
+  -H 'content-type: application/json' \
+  -H 'authorization: Bearer <api-key>' \
+  -d '{
+    "model": "auto",
+    "messages": [{"role": "user", "content": "Plan a multi-service refactor"}],
+    "router": {
+      "taskType": "agentic_reasoning"
+    }
+  }'
+```
+
+**Explicit task type with async fallback on heavy load:**
+
+```bash
+curl -s http://127.0.0.1:11435/v1/chat/completions \
+  -H 'content-type: application/json' \
+  -H 'authorization: Bearer <api-key>' \
+  -d '{
+    "model": "auto",
+    "messages": [{"role": "user", "content": "Plan a multi-service refactor"}],
+    "router": {
+      "taskType": "agentic_reasoning",
+      "allowAsync": true
+    }
+  }'
+```
+
+Returns `202` with a job id when load is high; `200` with the result when run synchronously.
+
+**Force a specific model, block all others:**
+
+```bash
+curl -s http://127.0.0.1:11435/v1/chat/completions \
+  -H 'content-type: application/json' \
+  -H 'authorization: Bearer <api-key>' \
+  -d '{
+    "model": "auto",
+    "messages": [{"role": "user", "content": "Review this PR diff"}],
+    "router": {
+      "taskType": "code_review",
+      "preferredModels": ["gpt-oss:20b"],
+      "forbiddenModels": ["qwen2.5-coder:7b", "deepseek-coder:6.7b"]
+    }
+  }'
+```
+
+**Force sync, no async fallback even under load:**
+
+```bash
+curl -s http://127.0.0.1:11435/v1/chat/completions \
+  -H 'content-type: application/json' \
+  -H 'authorization: Bearer <api-key>' \
+  -d '{
+    "model": "auto",
+    "messages": [{"role": "user", "content": "Fix the off-by-one error"}],
+    "router": {
+      "taskType": "code_fix",
+      "mode": "sync",
+      "allowAsync": false
+    }
+  }'
+```
+
+**High priority request — jumps ahead in the queue:**
+
+```bash
+curl -s http://127.0.0.1:11435/v1/chat/completions \
+  -H 'content-type: application/json' \
+  -H 'authorization: Bearer <api-key>' \
+  -d '{
+    "model": "auto",
+    "messages": [{"role": "user", "content": "Summarize this log"}],
+    "router": {
+      "taskType": "summarize",
+      "priority": "high"
+    }
+  }'
+```
+
+**GPU-only — reject if model would run on CPU or with a CPU/GPU split:**
+
+```bash
+curl -s http://127.0.0.1:11435/v1/chat/completions \
+  -H 'content-type: application/json' \
+  -H 'authorization: Bearer <api-key>' \
+  -d '{
+    "model": "auto",
+    "messages": [{"role": "user", "content": "Generate a REST API scaffold"}],
+    "router": {
+      "taskType": "code_generate",
+      "requireGpuOnly": true
+    }
+  }'
+```
+
+Returns `503` if no GPU-only candidate is available.
+
+**Check what the router decided** — every `200` response includes a `router` object:
+
+```json
+{
+  "router": {
+    "mode": "sync",
+    "taskType": "agentic_reasoning",
+    "selectedModel": "gpt-oss:20b",
+    "fallbackModels": ["gpt-oss:20b", "qwen2.5-coder:7b"],
+    "queueTimeMs": 3,
+    "executionTimeMs": 8420,
+    "decisionReason": "Selected gpt-oss:20b for agentic_reasoning with score 290.0"
+  }
+}
+```
+
+`decisionReason` includes the winning score, which helps diagnose unexpected model selection — compare it against the scoring table above to see which component tipped the balance.
+
 ## Config Reference
 
 Lookup order:
@@ -252,17 +467,22 @@ This prevents the admin API from changing the rules that protect itself. When `a
 
 Admin API:
 
+**Read the current managed access config**
+
 ```bash
 curl http://127.0.0.1:11435/v1/admin/access/config \
   -H 'authorization: Bearer admin-secret'
+```
+
+**Replace the entire managed access config** (planes + all keys at once)
 
+```bash
 curl -X PUT http://127.0.0.1:11435/v1/admin/access/config \
   -H 'authorization: Bearer admin-secret' \
   -H 'content-type: application/json' \
   -d '{
     "expectedVersion": 1,
     "config": {
-      "version": 1,
       "planes": {
         "standalone": {
           "enabled": true,
@@ -280,7 +500,51 @@ curl -X PUT http://127.0.0.1:11435/v1/admin/access/config \
   }'
 ```
 
-The admin `PUT` supports optimistic concurrency. If `expectedVersion` is present and does not match the active managed access config, the router returns `409`.
+`expectedVersion` enables optimistic concurrency. If present and the value does not match the active managed config version, the router returns `409`.
+
+**Add an API key**
+
+Generate a key and its SHA-256 hash first:
+
+```bash
+node -e "
+const c = require('crypto'), k = 'onr-' + c.randomBytes(20).toString('hex');
+console.log('key: ', k);
+console.log('hash: sha256:' + c.createHash('sha256').update(k).digest('hex'));
+"
+```
+
+Then add the key:
+
+```bash
+curl -X POST http://127.0.0.1:11435/v1/admin/access/keys \
+  -H 'authorization: Bearer admin-secret' \
+  -H 'content-type: application/json' \
+  -d '{
+    "id": "user-alice",
+    "name": "Alice",
+    "keyHash": "sha256:<hash>",
+    "scopes": ["standalone"],
+    "limits": {
+      "standalone": {"requests": 100, "windowSeconds": 60}
+    }
+  }'
+```
+
+Returns `201` with the created key entry. Returns `409` if the `id` is already in use.
+
+`scopes` controls which planes accept the key. Valid values are `standalone`, `runtimeAgent`, or both. `limits` is optional; when omitted, the plane's `defaultLimit` applies.
+
+**Revoke an API key**
+
+```bash
+curl -X DELETE http://127.0.0.1:11435/v1/admin/access/keys/user-alice \
+  -H 'authorization: Bearer admin-secret'
+```
+
+Returns `200 { "revoked": { ... } }` with the removed key entry. Returns `404` if the id is not found. The change takes effect immediately without a restart.
+
+All admin operations are written atomically to `access.managedConfigPath` and appended to the audit log when `auditLog: true`.
 
 For admin client certificate checks, enable HTTPS, configure `server.https.caPath`, and set:
 

package/dist/cli.js

@@ -44,6 +44,17 @@ var protectedPlaneConfigSchema = z.object({
   }).default({ requireApiKey: false, anonymous: "allow" }),
   defaultLimit: trafficLimitSchema.optional()
 });
+var apiKeySchema = z.object({
+  id: z.string().min(1).regex(/^[a-zA-Z0-9._:-]+$/, "api key id may contain only letters, numbers, dots, underscores, colons, and dashes"),
+  name: z.string().min(1).optional(),
+  keyHash: z.string().regex(/^sha256:[a-fA-F0-9]{64}$/, "keyHash must use sha256:<64 hex chars>"),
+  enabled: z.boolean().default(true),
+  scopes: z.array(z.enum(["standalone", "runtimeAgent"])).min(1),
+  limits: z.object({
+    standalone: trafficLimitSchema.optional(),
+    runtimeAgent: trafficLimitSchema.optional()
+  }).optional()
+});
 var managedAccessConfigSchema = z.object({
   version: z.number().int().nonnegative().default(1),
   updatedAt: z.string().datetime().optional(),
@@ -60,19 +71,7 @@ var managedAccessConfigSchema = z.object({
     standalone: { enabled: true, auth: { requireApiKey: false, anonymous: "allow" } },
     runtimeAgent: { enabled: true, auth: { requireApiKey: false, anonymous: "allow" } }
   }),
-  apiKeys: z.array(
-    z.object({
-      id: z.string().min(1).regex(/^[a-zA-Z0-9._:-]+$/, "api key id may contain only letters, numbers, dots, underscores, colons, and dashes"),
-      name: z.string().min(1).optional(),
-      keyHash: z.string().regex(/^sha256:[a-fA-F0-9]{64}$/, "keyHash must use sha256:<64 hex chars>"),
-      enabled: z.boolean().default(true),
-      scopes: z.array(z.enum(["standalone", "runtimeAgent"])).min(1),
-      limits: z.object({
-        standalone: trafficLimitSchema.optional(),
-        runtimeAgent: trafficLimitSchema.optional()
-      }).optional()
-    })
-  ).default([])
+  apiKeys: z.array(apiKeySchema).default([])
 });
 var defaultManagedAccessConfig = managedAccessConfigSchema.parse({});
 var adminPlaneConfigSchema = z.object({
@@ -159,8 +158,7 @@ var modelSpecSchema = z2.object({
   timeoutMs: z2.number().int().positive(),
   costClass: z2.enum(["low", "medium", "high"]).default("medium"),
   exclusive: z2.boolean().default(false),
-  allowWhenBusy: z2.boolean().default(false),
-  tags: z2.array(z2.string()).default([])
+  allowWhenBusy: z2.boolean().default(false)
 });
 var appConfigSchema = z2.object({
   server: z2.object({
@@ -364,7 +362,6 @@ models:
     costClass: low
     exclusive: false
     allowWhenBusy: true
-    tags: [general]
 routes:
   triage: [llama3.2:3b]
   simple_chat: [llama3.2:3b]
@@ -962,8 +959,7 @@ function buildModelSpec(name, role, sizeGb, cpuOnly) {
     timeoutMs: heavy ? 3e5 : code ? 18e4 : 9e4,
     costClass: heavy ? "high" : code ? "medium" : "low",
     exclusive: heavy,
-    allowWhenBusy: !heavy,
-    tags: tagsForRole(role)
+    allowWhenBusy: !heavy
   };
 }
 function purposesForRole(role) {
@@ -981,21 +977,6 @@ function purposesForRole(role) {
       return ["triage", "simple_chat", "summarize"];
   }
 }
-function tagsForRole(role) {
-  switch (role) {
-    case "code":
-      return ["code", "fallback"];
-    case "review":
-      return ["code", "review"];
-    case "heavy":
-      return ["reasoning", "large_context"];
-    case "tool":
-      return ["tool_use"];
-    case "fast":
-    default:
-      return ["fast", "chat"];
-  }
-}
 function generateRoutes(models) {
   const fast = models.filter((model) => model.costClass === "low").map((model) => model.name);
   const code = models.filter((model) => model.purpose.includes("code_generate")).map((model) => model.name);
@@ -1551,7 +1532,6 @@ var RoutingEngine = class {
     score += Math.max(0, 50 - routeIndex * 8);
     score += model.priority;
     if (model.purpose.includes(context.classification.taskType)) score += 25;
-    if (model.tags.includes(context.classification.taskType)) score += 15;
     if (preferredIndex >= 0) score += 80 - preferredIndex * 10;
     if (loaded) score += 20;
     if (context.classification.complexity === "heavy" && model.costClass === "high") score += 20;
@@ -1573,7 +1553,7 @@ var RoutingEngine = class {
     for (const name of context.router.preferredModels) names.add(name);
     for (const name of routeNames) names.add(name);
     for (const model of this.config.models) {
-      if (model.purpose.includes(context.classification.taskType) || model.tags.includes(context.classification.taskType)) {
+      if (model.purpose.includes(context.classification.taskType)) {
         names.add(model.name);
       }
     }
@@ -1652,6 +1632,51 @@ var AccessControlStore = class {
     });
     return structuredClone(updated);
   }
+  async addApiKey(input2) {
+    const key = apiKeySchema.parse(input2);
+    await this.enqueueWrite(async () => {
+      if (this.managed.apiKeys.some((k) => k.id === key.id)) {
+        throw new AccessHttpError(409, `API key with id '${key.id}' already exists`);
+      }
+      if (!this.access.managedConfigPath) {
+        throw new AccessHttpError(500, "access.managedConfigPath is not configured");
+      }
+      const next = {
+        ...this.managed,
+        version: this.managed.version + 1,
+        updatedAt: (/* @__PURE__ */ new Date()).toISOString(),
+        apiKeys: [...this.managed.apiKeys, key]
+      };
+      await writeManagedAccessConfig(this.access.managedConfigPath, next);
+      this.managed = next;
+      this.access.managed = next;
+    });
+    return structuredClone(key);
+  }
+  async revokeApiKey(id) {
+    let removed;
+    await this.enqueueWrite(async () => {
+      const idx = this.managed.apiKeys.findIndex((k) => k.id === id);
+      if (idx === -1) {
+        throw new AccessHttpError(404, `API key '${id}' not found`);
+      }
+      if (!this.access.managedConfigPath) {
+        throw new AccessHttpError(500, "access.managedConfigPath is not configured");
+      }
+      removed = this.managed.apiKeys[idx];
+      const next = {
+        ...this.managed,
+        version: this.managed.version + 1,
+        updatedAt: (/* @__PURE__ */ new Date()).toISOString(),
+        apiKeys: this.managed.apiKeys.filter((_, i) => i !== idx)
+      };
+      await writeManagedAccessConfig(this.access.managedConfigPath, next);
+      this.managed = next;
+      this.access.managed = next;
+      this.limiter.clear();
+    });
+    return structuredClone(removed);
+  }
   publicMiddleware(planeOrPlanes) {
     return (req, res, next) => {
       const planes = Array.isArray(planeOrPlanes) ? planeOrPlanes : [planeOrPlanes];
@@ -1962,6 +1987,26 @@ function createApp(config, deps) {
       next(error);
     }
   });
+  api.post("/v1/admin/access/keys", adminAccess, async (req, res, next) => {
+    try {
+      const key = await access3.addApiKey(req.body);
+      auditAdmin(config.access.admin, req, "success", "key_added", res.locals.admin?.remoteIp, key.id);
+      res.status(201).json(key);
+    } catch (error) {
+      auditAdmin(config.access.admin, req, "failure", error instanceof Error ? error.message : String(error), res.locals.admin?.remoteIp);
+      next(error);
+    }
+  });
+  api.delete("/v1/admin/access/keys/:id", adminAccess, async (req, res, next) => {
+    try {
+      const revoked = await access3.revokeApiKey(req.params.id);
+      auditAdmin(config.access.admin, req, "success", "key_revoked", res.locals.admin?.remoteIp, revoked.id);
+      res.json({ revoked });
+    } catch (error) {
+      auditAdmin(config.access.admin, req, "failure", error instanceof Error ? error.message : String(error), res.locals.admin?.remoteIp);
+      next(error);
+    }
+  });
   api.get("/v1/router/capabilities", runtimeAgentAccess, (_req, res) => {
     res.json(buildCapabilities(config));
   });