@nemesis-oss/ollama-sdk 1.11.0 → 1.12.0

package/CHANGELOG.md

@@ -1,5 +1,33 @@
 # Changelog
 
+## [1.12.0] — Daemon liveness & telemetry standardization: formatTelemetry, dynamic host model-affinity routing, root ping
+
+The October 10 documentation digest ("Daemon Liveness & Telemetry Standardization") reviewed the official operational surfaces — the root liveness probe (`HEAD /` → 200, body "Ollama is running"), `GET /api/version`, `GET /api/ps` runner introspection, and the nanosecond duration counters on every generation response. Verified against the codebase first (see [ADR 0028](./docs/adr/0028-telemetry-normalization-host-affinity-root-ping.md)): two of the digest's gap assessments were stale — multi-host pooling with least-connections routing, capacity caps, and 503/`OLLAMA_MAX_QUEUE` saturation failover already exist (`endpoints` + `endpointHealth` + `DEFAULT_FAILOVER_CODES`), so the proposed `LoadBalancedOllamaClient` was **not** added (it would fork the existing routing engine); the real gaps — telemetry math, dynamic multi-host affinity, and the root ping — are implemented, all additive, no breaking changes. 865 pre-existing tests stay green, 33 new tests added.
+
+### Standardized telemetry normalizer (`formatTelemetry`, `src/telemetry/metrics.ts`)
+
+- **`formatTelemetry(raw)`** derives the human-scale summary from Ollama's raw nanosecond/token counters: `totalLatencyMs`, `modelLoadMs`, `promptEvalMs`, `generationMs` (ns→ms, 2 decimals), `tokensPerSecond` (`eval_count / (eval_duration / 1e9)` — the official Usage-doc formula), `promptTokensPerSecond`, and `cacheHitRatio` — pure, never throws, missing counters read as `0` (never `NaN`/`Infinity`, so partial payloads like stream final events and older servers format cleanly).
+- The input is **structural**: any response object carrying the optional counters — `ChatResponse`, `GenerateResponse`, the final `GenerateStreamEvent`, `EmbedResponse` — passes as-is, no plucking. Purity and structural acceptance are pinned by tests with real response objects (no casts).
+- **`cacheHitRatio` deliberately deviates from the digest's sketch**: `cached / (cached + evaluated)`, not `cached / prompt_eval_count` — Ollama reports `prompt_eval_count: 0` on a _full_ cache hit, so the digest's formula divides by zero and reads a perfect cache as "no caching". The sum denominator is the SDK's established `ConversationSession.cacheStats.hitRate` semantics; the two surfaces now share documented, tested math.
+
+### Dynamic host model-affinity routing (`endpointHealth.modelAffinity` + `ModelAffinityRouter`)
+
+- The digest's "tomorrow's focus" item, shipped: with several interchangeable endpoints, **opt-in** `endpointHealth: { modelAffinity: { ttlMs?, failureRetryMs? } }` reorders each request's candidate endpoints so hosts whose `GET /api/ps` shows the requested model already resident in VRAM are tried first — avoiding the unload/cold-load swap a miss forces on `OLLAMA_MAX_LOADED_MODELS`-bounded daemons. The multi-host counterpart of `ModelAffinityScheduler` (v1.11.0); the two compose.
+- **Zero added request latency**: reordering is synchronous and cache-only — a request never waits for a probe. On a cache miss the request proceeds with the unchanged order while a background refresh (TTL 30s default, deduped, 5s failure backoff) warms the cache. `client.warmModelAffinity()` pre-warms at startup; `client.modelAffinityStatus()` exposes per-endpoint snapshots (loaded models, fetch time, freshness) for observability.
+- **Partial knowledge suppresses the reorder** — if any candidate's snapshot is stale or unknown, the original order stands (routing on partial residency data risks the exact cold-load swap the feature prevents). Probe failures are never cached as data; probes bypass `executeWithFailover` entirely (no reentry, no circuit-breaker pollution, no cross-host failover — the answer is per-host by definition).
+- **Race-free with `'least-connections'`**: no `await` between candidate selection, the reorder, and the endpoint's `acquire()` — the strategy's deterministic one-per-account guarantee is untouched; priority tiers, capacity caps, `models`-scoped credentials, and single-endpoint pinning all keep working.
+- **Matching is shared, not duplicated**: "resident" uses the same matcher as context-window discovery (`isModelResident`, refactored out of `findRunningModelContextLength` — exact `name`/`model` match or tag-less prefix `llama3` ↔ `llama3:latest`, never `llama` → `llama3.1`). Off unless configured: without `modelAffinity` the router isn't constructed and no `/api/ps` probes are ever issued.
+
+### Root liveness probe (`client.ping()`)
+
+- The official ultra-lightweight liveness check: `HEAD /` against the current best candidate endpoint, resolving `{ healthy, latencyMs, baseUrl, error? }` — the docs define this exact probe for load-balancer health checks (`GET /` answers the plain-text body "Ollama is running"; HEAD doesn't transfer it).
+- **Single-shot by design** — no retry, no failover: a liveness probe's value is that it answers about _one concrete host_, not "some endpoint eventually answered". Fleet audits remain `healthCheck()`'s job (every endpoint, plus the version via the heavier `GET /api/version`). Resolves rather than throws on connection failure/timeout/error status (monitoring loops treat it as a datapoint); only caller-initiated cancellation (`signal`, `destroy()`) rejects, like any other aborted request. Uses the client's `fetch`/middleware/lifecycle pipeline like every other request.
+
+### Documentation
+
+- README: three Key Features bullets; a "Standardized Telemetry" section (before OpenTelemetry, which it complements); a "Model-affinity routing across hosts" subsection in the Multi-Endpoint guide (with the `warmModelAffinity`/`modelAffinityStatus` flow); `ping()` in the health-check intro. ADR 0028 records the audit verdicts, the declined `LoadBalancedOllamaClient` fork, and the `cacheHitRatio` deviation rationale.
+- New examples: `examples/telemetry.ts` and `examples/multi-host-affinity.ts`.
+
 ## [1.11.0] — Documentation digest + production-readiness audit: context discovery, compaction, affinity scheduling, agent cycle detection, URL-join hardening, vision-history hygiene
 
 Three real gaps from the follow-up documentation digest on concurrency & memory-scaling mechanics (see [ADR 0026](./docs/adr/0026-context-discovery-history-compaction-model-affinity.md)) and three verified findings from the October 9 production-readiness audit on agent-loop ergonomics, transport hardening, and multi-turn memory retention (see [ADR 0027](./docs/adr/0027-agent-cycle-detection-url-join-history-hygiene.md)) — all additive, no breaking changes; 778 pre-existing tests stay green, 87 new tests added (40 digest + 47 audit). Stale claims in both reports were verified against the codebase and answered with documentation and tests instead of new code (one digest claim, two audit claims).

package/README.md

@@ -29,6 +29,9 @@
 - 🧠 **Dynamic Context-Window Discovery**: `client.models.getContextLength({ model })` resolves the real window from the server — the allocated window of the running instance (`/api/ps`), the Modelfile `num_ctx` default, and the native GGUF maximum (`model_info`), in precedence order — so applications stop hardcoding 2048/4096 guesses.
 - 🔁 **Model-Affinity Scheduling**: `ModelAffinityScheduler` runs multi-model workloads through per-model serial queues with a distinct-model cap (default 1), deepening the already-hot model before loading the next — the anti-thrashing order for `OLLAMA_NUM_PARALLEL`/`OLLAMA_MAX_LOADED_MODELS`-bounded daemons — and routes candidate lists to whichever model is already loaded.
 - 🛡️ **Agent Loop Guardrails**: `Agent` is a boundary-checked runner out of the box — `maxIterations` (default 10), an opt-in `maxToolCalls` budget, and opt-in **cycle detection** (`maxRepeatedToolCalls`) that fails fast with `OllamaAgentToolLoopError` the moment the model re-emits one identical tool call instead of burning the whole iteration budget on it; tool rejections (including unregistered names) are always encapsulated as `role: 'tool'` results, never loop crashes.
+- 📊 **Standardized Telemetry (`formatTelemetry`)**: One pure call turns Ollama's raw nanosecond counters into the summary humans want — ms latencies, `tokensPerSecond`/`promptTokensPerSecond` per the official Usage-doc formulas, and the KV `cacheHitRatio` — accepting any response object structurally (`chat`, `generate`, stream final events, `embed`), with missing counters reading as `0`, never `NaN`.
+- 🧲 **Dynamic Host Model-Affinity Routing**: With several interchangeable endpoints, opt-in `endpointHealth.modelAffinity` reorders each request's candidates so the host already holding the requested model in VRAM (`GET /api/ps`, TTL-cached, zero added request latency) is tried first — no more forced unload/cold-load swaps under `OLLAMA_MAX_LOADED_MODELS`.
+- 🩺 **Root Liveness Probe (`ping`)**: `client.ping()` is the official ultra-lightweight `HEAD /` health check — single-shot (no retry, no failover: the answer is about one concrete host), resolves instead of throwing, reports latency; `healthCheck()` remains the fleet-wide audit.
 - 🧼 **Vision History Hygiene**: `sanitizeHistoryForNextTurn()` evicts stale base64 image payloads from consumer-managed multi-turn vision histories — every `/api/chat` request re-sends the whole history, so old images silently bloat payload, context, and heap; the helper keeps images only on the trailing window you choose, conversation text fully preserved.
 - 🌊 **Web Stream Adapters**: Drop-in adapters (`toTextStream`, `toDataStream`, `toResponse`) for Next.js Route Handlers and Vercel AI SDK.
 - 📈 **OpenTelemetry Instrumentation**: Automatic spans for HTTP requests, endpoint failover, chat/generate calls, and agent runs — zero-cost when OpenTelemetry isn't installed.
@@ -936,6 +939,11 @@ const client = new OllamaClient({
 // Active health check probe
 const health = await client.healthCheck();
 console.log(health);
+
+// Ultra-lightweight single-host liveness probe — the official `HEAD /` root
+// ping (no retry, no failover: an answer about one concrete host).
+const { healthy, latencyMs } = await client.ping();
+console.log(healthy, `${latencyMs}ms`);
 ```
 
 Failover applies to inference calls (`chat`, `generate`, `embed`, `embeddings`,
@@ -1164,6 +1172,64 @@ initial HTTP round trip:
   between turns, outside any `chat()` call, so a slow tool never ties up one of your
   scarce concurrent-request accounts.
 
+#### Model-affinity routing across hosts (prefer the resident host)
+
+With several *interchangeable* endpoints serving the same models — the fleet-of-workstations
+case, not the per-credential Cloud case — sending a request to a host that doesn't currently
+hold the model forces that host to evict something and cold-load it: the multi-host version
+of the thrashing `ModelAffinityScheduler` prevents inside one client. Opt in with
+`endpointHealth.modelAffinity` and each request's candidates are reordered so hosts whose
+`GET /api/ps` shows the requested model already resident in VRAM are tried first:
+
+```typescript
+const client = new OllamaClient({
+  endpoints: [
+    { name: 'gpu-a', baseUrl: 'http://gpu-a.internal:11434' },
+    { name: 'gpu-b', baseUrl: 'http://gpu-b.internal:11434' },
+    { name: 'gpu-c', baseUrl: 'http://gpu-c.internal:11434' },
+  ],
+  endpointHealth: {
+    strategy: 'least-connections',
+    modelAffinity: { ttlMs: 30_000 }, // /api/ps snapshot freshness (default 30s)
+  },
+});
+
+// Optional but recommended at startup: pre-warm the residency cache so even the
+// first request routes to the host already holding the model.
+await client.warmModelAffinity();
+
+await client.chat({ model: 'llama3.2', messages }); // → the host with llama3.2 in VRAM
+
+// Observability: what each host held, when it was last probed, whether it's fresh.
+console.log(client.modelAffinityStatus());
+```
+
+The design never trades latency for affinity (see
+[ADR 0028](./docs/adr/0028-telemetry-normalization-host-affinity-root-ping.md)):
+
+- **Zero added request latency.** Reordering is synchronous and consults only cached
+  `/api/ps` snapshots. A request never waits for a probe: on a cache miss it proceeds
+  with the unchanged order while a background refresh warms the cache for the next one.
+- **Partial knowledge suppresses the reorder.** If any candidate's snapshot is stale or
+  unknown (a host was just rebooted, a probe failed), the original order stands —
+  routing on partial residency data risks exactly the cold-load swap this exists to
+  avoid. Probe failures are never cached as data and a 5s backoff keeps a dead host
+  from being re-probed on every request.
+- **No probes at all unless configured.** Without `modelAffinity`, the router isn't even
+  constructed — zero overhead, and no `/api/ps` traffic.
+- **Composes with everything above.** Affinity reorders *within* the runnable set:
+  priority tiers, `least-connections` ordering, capacity caps, and `models`-scoped
+  credentials keep working exactly as before, and single-endpoint-pinned operations
+  (model/blob management, `capabilities()`) are never moved to a different host.
+  Residency matching follows Ollama's own name resolution (`llama3` matches
+  `llama3:latest` but never `llama3.1`), shared with context-window discovery so the
+  two features can't disagree about what "loaded" means.
+
+For the *single-host* counterpart — per-model serial queues that deepen the already-hot
+model before loading the next one — see
+[Model-Affinity Scheduling](#model-affinity-scheduling-multi-model-anti-thrashing); the
+two compose: a scheduler per host, affinity routing between hosts.
+
 ### Client Teardown: `destroy()`
 
 Inside `node:worker_threads` tasks, CLI runners, serverless handlers, and other
@@ -1197,6 +1263,60 @@ returns `0`.
 
 ---
 
+### Standardized Telemetry (`formatTelemetry`)
+
+Ollama reports every duration on its generation responses as an **integer in
+nanoseconds** (`total_duration`, `load_duration`, `prompt_eval_duration`, `eval_duration`)
+alongside token counters (`prompt_eval_count`, `prompt_eval_cached_count`, `eval_count`).
+Turning those into the numbers humans actually want — milliseconds, tokens/second, cache
+hit rate — is the same division-and-rounding math every consumer would otherwise
+hand-roll (and the easy mistakes are real: `NaN` on missing counters, integer division,
+a cache ratio that divides by the wrong denominator). `formatTelemetry` is that math,
+once, with the official Usage-doc formulas:
+
+```typescript
+import { OllamaClient, formatTelemetry } from '@nemesis-oss/ollama-sdk';
+
+const client = new OllamaClient();
+
+const res = await client.chat({
+  model: 'llama3.2',
+  messages: [{ role: 'user', content: 'Why is the sky blue?' }],
+});
+
+// Any response object works as-is — chat, generate, the final stream event,
+// embed. No plucking required: the counters are read structurally.
+const t = formatTelemetry(res);
+console.log(t);
+// {
+//   totalLatencyMs: 3712.5,       // total_duration / 1e6, 2 decimals
+//   modelLoadMs: 250,             // cold-load into VRAM (0 when resident)
+//   promptEvalMs: 1000,           // prompt pre-fill
+//   generationMs: 2400,           // token generation
+//   tokensPerSecond: 50,          // eval_count / (eval_duration / 1e9) — official formula
+//   promptTokensPerSecond: 200,   // prompt_eval_count / (prompt_eval_duration / 1e9)
+//   cacheHitRatio: 0.429          // cached / (cached + evaluated), 3 decimals
+// }
+```
+
+Two behaviors worth knowing:
+
+- **Missing counters read as `0`** — partial payloads (stream final events, older
+  servers without `prompt_eval_cached_count`, embeddings) format cleanly instead of
+  producing `NaN`/`Infinity`, and zero durations never divide into rates.
+- **`cacheHitRatio` uses `cached / (cached + evaluated)`**, not
+  `cached / prompt_eval_count`: Ollama reports `prompt_eval_count: 0` on a *full* cache
+  hit, so dividing by it alone would read a perfect cache as "no caching at all". This
+  is the same semantics as [`ConversationSession`'s
+  `cacheStats.hitRate`](#kv-cache-aware-conversation-sessions), so the two surfaces can
+  never disagree.
+
+`formatTelemetry` is pure — no fetching, no throwing, no mutation — so it drops
+straight into log lines, dashboards, and streaming pipelines alike. For span-based
+instrumentation of requests themselves, see
+[Observability with OpenTelemetry](#observability-with-opentelemetry) below; the two
+compose (spans for *where/when*, `formatTelemetry` for *how fast*).
+
 ### Observability with OpenTelemetry
 
 The client automatically emits [OpenTelemetry](https://opentelemetry.io/) spans for HTTP

package/dist/{anthropic-CfricAhK.d.ts → anthropic-CKp5TmAP.d.ts}

@@ -1,4 +1,4 @@
-import { O as OllamaRuntime, H as HttpClient, A as AbortableAsyncIterable, af as SseEvent, y as OllamaClientError } from './runtime-CCm4CtU_.js';
+import { O as OllamaRuntime, H as HttpClient, A as AbortableAsyncIterable, af as SseEvent, y as OllamaClientError } from './runtime-CIQYCRL3.js';
 
 /**
  * AUTO-GENERATED by `npm run contract:generate`. DO NOT EDIT.

package/dist/{anthropic-DinrABof.d.cts → anthropic-Did0lQxP.d.cts}

@@ -1,4 +1,4 @@
-import { O as OllamaRuntime, H as HttpClient, A as AbortableAsyncIterable, af as SseEvent, y as OllamaClientError } from './runtime-DTtWCxDs.cjs';
+import { O as OllamaRuntime, H as HttpClient, A as AbortableAsyncIterable, af as SseEvent, y as OllamaClientError } from './runtime-DmI1cOEq.cjs';
 
 /**
  * AUTO-GENERATED by `npm run contract:generate`. DO NOT EDIT.

package/dist/{compose-BQYXdpar.d.ts → compose-6UOPEknN.d.ts}

@@ -1,4 +1,4 @@
-import { l as ToolDefinition, i as ToolCall, n as Message } from './types-BtQexFpT.js';
+import { l as ToolDefinition, i as ToolCall, n as Message } from './types-B57ZZfp3.js';
 import { z } from 'zod';
 
 /**

package/dist/{compose-lt-Qz3lI.d.cts → compose-DhvoUrWa.d.cts}

@@ -1,4 +1,4 @@
-import { l as ToolDefinition, i as ToolCall, n as Message } from './types-BtQexFpT.cjs';
+import { l as ToolDefinition, i as ToolCall, n as Message } from './types-B57ZZfp3.cjs';
 import { z } from 'zod';
 
 /**

package/dist/generated-api.d.cts

@@ -1,8 +1,8 @@
-import { O as OpenAIChatCompletionRequest, a as OpenAIChatCompletionResponse, b as OpenAIChatCompletionChunk, c as OpenAICompletionRequest, d as OpenAICompletionResponse, e as OpenAICompletionChunk, f as OpenAIEmbeddingRequest, g as OpenAIEmbeddingResponse, h as OpenAIListModelsResponse, i as OpenAIModelItem, j as OpenAIResponsesRequest, k as OpenAIResponsesResponse, l as OpenAIResponsesStreamEvent, A as AnthropicMessagesRequest, m as AnthropicMessagesResponse, n as AnthropicMessageStreamEvent } from './anthropic-DinrABof.cjs';
-export { N as NativeApi } from './anthropic-DinrABof.cjs';
-import { O as OllamaRuntime } from './runtime-DTtWCxDs.cjs';
-export { a as OperationDefinition } from './runtime-DTtWCxDs.cjs';
-import './types-BtQexFpT.cjs';
+import { O as OpenAIChatCompletionRequest, a as OpenAIChatCompletionResponse, b as OpenAIChatCompletionChunk, c as OpenAICompletionRequest, d as OpenAICompletionResponse, e as OpenAICompletionChunk, f as OpenAIEmbeddingRequest, g as OpenAIEmbeddingResponse, h as OpenAIListModelsResponse, i as OpenAIModelItem, j as OpenAIResponsesRequest, k as OpenAIResponsesResponse, l as OpenAIResponsesStreamEvent, A as AnthropicMessagesRequest, m as AnthropicMessagesResponse, n as AnthropicMessageStreamEvent } from './anthropic-Did0lQxP.cjs';
+export { N as NativeApi } from './anthropic-Did0lQxP.cjs';
+import { O as OllamaRuntime } from './runtime-DmI1cOEq.cjs';
+export { a as OperationDefinition } from './runtime-DmI1cOEq.cjs';
+import './types-B57ZZfp3.cjs';
 import 'zod';
 
 /**

package/dist/generated-api.d.ts

@@ -1,8 +1,8 @@
-import { O as OpenAIChatCompletionRequest, a as OpenAIChatCompletionResponse, b as OpenAIChatCompletionChunk, c as OpenAICompletionRequest, d as OpenAICompletionResponse, e as OpenAICompletionChunk, f as OpenAIEmbeddingRequest, g as OpenAIEmbeddingResponse, h as OpenAIListModelsResponse, i as OpenAIModelItem, j as OpenAIResponsesRequest, k as OpenAIResponsesResponse, l as OpenAIResponsesStreamEvent, A as AnthropicMessagesRequest, m as AnthropicMessagesResponse, n as AnthropicMessageStreamEvent } from './anthropic-CfricAhK.js';
-export { N as NativeApi } from './anthropic-CfricAhK.js';
-import { O as OllamaRuntime } from './runtime-CCm4CtU_.js';
-export { a as OperationDefinition } from './runtime-CCm4CtU_.js';
-import './types-BtQexFpT.js';
+import { O as OpenAIChatCompletionRequest, a as OpenAIChatCompletionResponse, b as OpenAIChatCompletionChunk, c as OpenAICompletionRequest, d as OpenAICompletionResponse, e as OpenAICompletionChunk, f as OpenAIEmbeddingRequest, g as OpenAIEmbeddingResponse, h as OpenAIListModelsResponse, i as OpenAIModelItem, j as OpenAIResponsesRequest, k as OpenAIResponsesResponse, l as OpenAIResponsesStreamEvent, A as AnthropicMessagesRequest, m as AnthropicMessagesResponse, n as AnthropicMessageStreamEvent } from './anthropic-CKp5TmAP.js';
+export { N as NativeApi } from './anthropic-CKp5TmAP.js';
+import { O as OllamaRuntime } from './runtime-CIQYCRL3.js';
+export { a as OperationDefinition } from './runtime-CIQYCRL3.js';
+import './types-B57ZZfp3.js';
 import 'zod';
 
 /**

package/dist/generated-runtime.d.cts

@@ -1,6 +1,6 @@
-export { I as InvokeRequest, O as OllamaRuntime, b as OllamaRuntimeOptions, a as OperationDefinition, S as SchemaRef } from './runtime-DTtWCxDs.cjs';
+export { I as InvokeRequest, O as OllamaRuntime, b as OllamaRuntimeOptions, a as OperationDefinition, S as SchemaRef } from './runtime-DmI1cOEq.cjs';
 import { z } from 'zod';
-import './types-BtQexFpT.cjs';
+import './types-B57ZZfp3.cjs';
 
 /**
  * AUTO-GENERATED by `npm run contract:generate`. DO NOT EDIT.

package/dist/generated-runtime.d.ts

@@ -1,6 +1,6 @@
-export { I as InvokeRequest, O as OllamaRuntime, b as OllamaRuntimeOptions, a as OperationDefinition, S as SchemaRef } from './runtime-CCm4CtU_.js';
+export { I as InvokeRequest, O as OllamaRuntime, b as OllamaRuntimeOptions, a as OperationDefinition, S as SchemaRef } from './runtime-CIQYCRL3.js';
 import { z } from 'zod';
-import './types-BtQexFpT.js';
+import './types-B57ZZfp3.js';
 
 /**
  * AUTO-GENERATED by `npm run contract:generate`. DO NOT EDIT.