@nemesis-oss/ollama-sdk 1.14.0 → 1.15.0

package/CHANGELOG.md

@@ -1,5 +1,24 @@
 # Changelog
 
+## [1.15.0] — Draft benchmark utility & request-aware mock routes: `compareDraftPerformance`, mock `responder`
+
+The October 13 digest ("Speculative Performance & Telemetry Observations") reviewed how upstream reports speculative efficiency — no dedicated draft-token acceptance counter, throughput-only observability, a noisy signal (rejected drafts _add_ compute on creative prompts; accepted drafts slash `eval_duration` on structured ones) — and asked for a benchmark helper plus mock scenario hooks. Audited against the codebase first (see [ADR 0030](./docs/adr/0030-draft-benchmark-request-aware-mock-routes.md)): the "manual nanosecond ratios" claim was partially stale — `formatTelemetry` derives tokens/second since v1.12.0 and `lab/15` implements the full protocol, but neither reaches npm consumers (`files` publishes only `dist/`), so the packaged-harness gap was real. The mock scenario claims were stale: parallel tool calls in one frame, malformed-argument tool calls (string `arguments`), in-band mid-stream error injection, and 503 bodies all shipped in v1.14.0. The genuine engine gap the audit surfaced: routes couldn't vary responses **per request**. All additive, no breaking changes. 960 pre-existing tests stay green, 17 new tests added.
+
+### Speculative draft benchmark (`compareDraftPerformance`)
+
+- **`compareDraftPerformance(client, options)`** — the deterministic A/B for "does `draft_num_predict` pay off for _this_ prompt": identical prompt on both sides, `temperature: 0` forced + a fixed `seed` (default 42), one warm-up generation per setting (pays the model load), `samples` measured runs per side (default 5) with the **median** compared — single samples mislead. Throughput flows through `formatTelemetry` (the official Usage-doc math; never hand-rolled nanosecond division).
+- Result: `baselineTokensPerSec` / `speculativeTokensPerSec` (1dp), `speedupMultiplier` (2dp, computed from the two reported medians so the numbers are self-consistent), `effectiveGain` (`> gainThreshold`, default 1.05), plus full per-side `DraftRunStats` (every sample, medians).
+- **Structural dependency** — accepts any `DraftBenchmarkClient` (`{ generate }` with the non-streaming signature); `OllamaClient` satisfies it without being named, and tests can pass a plain object. A `client.benchmark` namespace was deliberately not added, and the digest's `SpeculativeBenchmarkRunner` class was folded into the function — rationale in ADR 0030.
+- Guarded: `RangeError` on invalid numeric options; non-positive/unusable samples are skipped, and a side with **no** valid samples throws `OllamaClientError` (`draft_benchmark_no_valid_samples`) instead of fabricating a zero/Infinity speedup; the abort signal is checked between runs and forwarded to every call.
+- The no-draft-network reading is documented on the utility and in the README: a ≈1.0 speedup means the model has no draft network (and no embedded MTP tensors) — the server silently ignores the option.
+
+### Request-aware mock routes (`responder`)
+
+- `MockEndpointOptions` now accepts `responder(record) => MockRouteResponse | undefined` — per-request responses computed from the recorded method/path/body. Returning `undefined` falls back to the route's static fields, so fail-twice-then-succeed retry drills, body-dependent payloads (e.g. draft-setting-aware frames), and call-count scenarios are closure-local one-liners on the same engine — no subclassing, no one-shot global scenario queue. The digest's `ScenarioMockOllamaServer` sketch was declined for exactly that reason (it forks the request path ADR 0029 protects; see ADR 0030).
+- The emission pipeline is shared: `chunkDelayMs`, `injectErrorFrame`, `dropConnectionMidStream`, and `rawBody` all keep their semantics inside responder routes; request recording and `onRequest` still fire.
+- Type-level: the response fields moved to a new `MockRouteResponse` interface that `MockEndpointOptions` extends — pure refactor, existing routes unaffected. `MockRouteResponse` is exported from the `/testing` subpath.
+- The SDK's own benchmark tests run daemonless through responder routes (draft-aware `/api/generate` frames) — the exact consumer CI pattern the digest's downstream stories describe, exercising the public engine rather than test-only fixtures.
+
 ## [1.14.0] — Speculative-decoding model assembly & the public testing subpath: `draftGguf`, `draft_num_predict` docs, `@nemesis-oss/ollama-sdk/testing`
 
 The October 12 digest ("Speculative Decoding & Draft Model Creation") reviewed the official model-creation surfaces — `draft_files` + `draft_quantize` on `POST /api/create`, the `draft_num_predict` Modelfile parameter and runtime option — and asked for an exportable zero-network mock server. Audited against the codebase first (see [ADR 0029](./docs/adr/0029-public-testing-subpath-speculative-draft-conveniences.md)): the digest's headline typing claims were stale — `draft_files`/`draft_quantize` already shipped on both the generated `CreateRequest` and the hand-written `CreateRequestOptions` (official OpenAPI, 2026-10-07 pin), and `draft_num_predict` was already typed + documented on the runtime `ModelOptions`; the "DRAFT `<model>`" Modelfile instruction the digest describes is not in the current official docs (only the parameter is). The real gaps — a path-based draft upload convenience, wire tests, discoverable documentation, and the not-exported mock server — are closed here. All additive, no breaking changes. 932 pre-existing tests stay green, 28 new tests added.

package/README.md

@@ -34,7 +34,8 @@ Nexum’s model harness (2026-10-09) exercised this SDK via `gemma4:cloud` and `
 - 💬 **KV-Cache-Aware Conversation Sessions**: `client.session(model, systemPrompt)` keeps an append-only, prefix-stable history that maximizes Ollama's KV-cache reuse across turns, surfaces per-turn + cumulative cache hit rates (`prompt_eval_cached_count`) so cache degradation is visible instead of silent, and offers a one-call `compact()` sliding-window escape hatch when a session outgrows its window.
 - 🛑 **Context-Window Safety**: Heuristic client-side token estimation warns (or throws, `onContextOverflow: 'throw'`) before a request is sent when the prompt approaches the effective `num_ctx` window — Ollama's default behavior is to _silently truncate_ oversized prompts — and `defaultContextLength` makes the window explicit on every request.
 - 📦 **Blob & Custom Model Publishing**: Content-addressed blob management per Ollama's documented protocol — `computeBlobDigest()` (SHA-256 → `sha256:<hex>`), `createBlobFromData()`/`createBlobFromFile()` (HEAD-check + upload, skipping existing blobs), and one-shot `createModelFromGguf()` (upload GGUF shard blobs, then `/api/create` with `files: {name: digest}`) — with **speculative-decoding assembly**: `draftGguf` uploads the draft network's shards into `draft_files`, `draft_quantize` compresses them independently, and `parameters.draft_num_predict` tunes the per-step draft length.
-- 🧪 **In-Memory Mock Server (`@nemesis-oss/ollama-sdk/testing`)**: The SDK's own deterministic test double, promoted public — `createMockOllamaServer()` with scriptable NDJSON routes, built-in liveness/version defaults (so `ping()` works out of the box), request recording for wire-level assertions, mid-stream **in-band error injection** (`{"error": "..."}` frames at HTTP 200, exactly how a real daemon reports GPU OOM), connection drops, and correctly-shaped chunk builders for streaming text and **parallel tool calls** — zero network, zero daemon, CI-deterministic.
+- 🏁 **Speculative Draft Benchmark (`compareDraftPerformance`)**: One call answers "does drafting pay off for _this_ prompt?" — identical prompt on both sides, deterministic sampling (`temperature: 0` + fixed seed), warm-up discard, several samples per side with **medians** compared, throughput via `formatTelemetry` — reported as baseline vs. speculative tok/s, `speedupMultiplier`, and `effectiveGain` past a configurable threshold. A ≈1.0 speedup is itself diagnostic: the model has no draft network, so the server silently ignores the option.
+- 🧪 **In-Memory Mock Server (`@nemesis-oss/ollama-sdk/testing`)**: The SDK's own deterministic test double, promoted public — `createMockOllamaServer()` with scriptable NDJSON routes, built-in liveness/version defaults (so `ping()` works out of the box), request recording for wire-level assertions, **request-aware routes** (`responder` — per-request responses for fail-twice-then-succeed retry drills and body-dependent payloads), mid-stream **in-band error injection** (`{"error": "..."}` frames at HTTP 200, exactly how a real daemon reports GPU OOM), connection drops, and correctly-shaped chunk builders for streaming text and **parallel tool calls** — zero network, zero daemon, CI-deterministic.
 - 🧵 **Batched Embeddings with Backpressure**: `client.embedBatch()` splits corpora of any size into bounded `/api/embed` batches (default 32 inputs × 3 in flight) — order-preserving, fail-fast, progress-reporting — so RAG/vector ingestion can't OOM a local daemon or saturate `OLLAMA_MAX_QUEUE` the way `Promise.all` floods do, with per-string context-window pre-flight against silent truncation.
 - 🧹 **Client Teardown (`destroy()`)**: One call aborts every in-flight request, active stream, and queued capacity waiter with `OllamaAbortError` — the clean-exit path for `worker_threads`, CLI runners, and short-lived scripts where dangling fetch bodies keep the event loop alive.
 - 🧠 **Dynamic Context-Window Discovery**: `client.models.getContextLength({ model })` resolves the real window from the server — the allocated window of the running instance (`/api/ps`), the Modelfile `num_ctx` default, and the native GGUF maximum (`model_info`), in precedence order — so applications stop hardcoding 2048/4096 guesses.
@@ -533,9 +534,40 @@ console.log(`${t.tokensPerSecond} tok/s (load ${t.modelLoadMs}ms)`);
 
 Already have the draft blobs uploaded? Pass the digest map directly —
 `createModel({ draft_files: { 'draft.gguf': digest } })` — or merge both
-(path uploads win on basename collision). A live measurement harness
-comparing `draft_num_predict` settings lives at
-`lab/15-speculative-draft/measure-draft-speedup.ts`.
+(path uploads win on basename collision).
+
+**Does drafting actually pay off?** Upstream emits no draft-token
+acceptance counter — speculative efficiency is only observable as effective
+token throughput, and the signal is noisy: rejected draft predictions _add_
+compute (high-temperature, creative prompts) while accepted drafts slash
+`eval_duration` on predictable workloads (structured code, JSON extraction).
+`compareDraftPerformance()` runs the deterministic A/B for a specific
+prompt — identical prompt both sides, `temperature: 0` + a fixed seed, one
+warm-up generation per setting (pays the model load), several samples per
+side with the **median** compared, throughput derived by `formatTelemetry`:
+
+```typescript
+import { OllamaClient, compareDraftPerformance } from '@nemesis-oss/ollama-sdk';
+
+const comparison = await compareDraftPerformance(new OllamaClient(), {
+  model: 'my-model',
+  prompt: 'Extract the sender, subject, and date from this email as JSON: ...',
+  draftNumPredict: 4, // default; the baseline side always runs with 0
+  samples: 5, // median per side — single runs mislead
+});
+
+console.log(
+  `${comparison.baselineTokensPerSec} → ${comparison.speculativeTokensPerSec} tok/s ` +
+    `(${comparison.speedupMultiplier}x, gain: ${comparison.effectiveGain})`,
+);
+```
+
+`effectiveGain` flips to `true` when the reported multiplier exceeds
+`gainThreshold` (default `1.05` — five percent, past run-to-run noise). A
+speedup of ≈1.0 is itself the diagnosis: the model has no draft network
+(and no embedded MTP tensors), so the server silently ignores
+`draft_num_predict` — the same reading `lab/15-speculative-draft/measure-draft-speedup.ts`
+makes obvious across 0/4/8/16.
 
 ### Token Log Probabilities (`logprobs`)
 
@@ -1552,12 +1584,40 @@ server.clearRequests(); // and server.clear() between tests
 await server.stop(); // idempotent — safe in afterEach
 ```
 
+**Request-aware routes** — `responder` computes the response _per request_
+from the recorded method/path/body, covering the scenario drills a static
+route can't: fail-twice-then-succeed retries, body-dependent payloads,
+call-count behavior:
+
+```typescript
+// Fail twice, then serve — retry/backoff drills:
+let hits = 0;
+server.register('/api/chat', {
+  chunks: mockTextChunks('recovered'),
+  responder: () =>
+    hits++ < 2 ? { status: 503, chunks: ['{"error":"server queue is full"}'] } : undefined,
+});
+
+// Body-dependent — answer differently by what the client sent:
+server.register('/api/generate', {
+  responder: (record) => ({
+    chunks: [frameFor(readDraftSetting(record.body))],
+  }),
+});
+```
+
+Returning `undefined` falls back to the route's static fields; a returned
+response feeds the full emission pipeline, so `chunkDelayMs`,
+`injectErrorFrame`, `dropConnectionMidStream`, and `rawBody` all keep their
+semantics inside responder routes.
+
 Per-route controls: `status` (503/502/429 bodies), `chunkDelayMs`,
 `dropConnectionMidStream` (connection reset mid-response), `onRequest`
 (per-route body capture), and `injectErrorFrame.after` (error position).
 Chunk builders: `mockChatChunk` (one frame with counter defaults),
 `mockTextChunks` (chunked text + done frame), `mockToolCallChunks` (parallel
-tool calls + done frame). Node.js only — the subpath builds on `node:http`
+tool calls + done frame — malformed-argument cases via string
+`arguments`). Node.js only — the subpath builds on `node:http`
 and is never imported from the Edge-safe main entry.
 
 ### Edge Runtime Compatibility

package/dist/index.cjs

@@ -7750,6 +7750,111 @@ var Agent = class {
   }
 };
 
+// src/draft-benchmark.ts
+var DEFAULT_DRAFT_TOKENS = 4;
+var DEFAULT_SAMPLES = 5;
+var DEFAULT_NUM_PREDICT = 256;
+var DEFAULT_SEED = 42;
+var DEFAULT_GAIN_THRESHOLD = 1.05;
+function median(values) {
+  const sorted = [...values].sort((a, b) => a - b);
+  const mid = Math.floor(sorted.length / 2);
+  return sorted.length % 2 === 1 ? sorted[mid] ?? 0 : ((sorted[mid - 1] ?? 0) + (sorted[mid] ?? 0)) / 2;
+}
+function roundTo(value, decimals) {
+  const factor = 10 ** decimals;
+  return Math.round(value * factor) / factor;
+}
+function requirePositiveInteger(name, value) {
+  if (!Number.isInteger(value) || value < 1) {
+    throw new RangeError(
+      `compareDraftPerformance: ${name} must be an integer >= 1 (got ${value}).`
+    );
+  }
+}
+async function runSide(client, options, draftNumPredict, samples, numPredict, seed) {
+  const samplingOptions = {
+    num_predict: numPredict,
+    temperature: 0,
+    seed,
+    draft_num_predict: draftNumPredict
+  };
+  if (options.warmup ?? true) {
+    await client.generate({
+      model: options.model,
+      prompt: options.prompt,
+      options: { ...samplingOptions, num_predict: 1 },
+      stream: false,
+      signal: options.signal
+    });
+  }
+  const tokensPerSecond = [];
+  const generationMs = [];
+  for (let i = 0; i < samples; i++) {
+    options.signal?.throwIfAborted();
+    const response = await client.generate({
+      model: options.model,
+      prompt: options.prompt,
+      options: samplingOptions,
+      stream: false,
+      signal: options.signal
+    });
+    const telemetry = formatTelemetry(response);
+    if (Number.isFinite(telemetry.tokensPerSecond) && telemetry.tokensPerSecond > 0) {
+      tokensPerSecond.push(telemetry.tokensPerSecond);
+      generationMs.push(telemetry.generationMs);
+    }
+  }
+  if (tokensPerSecond.length === 0) {
+    throw new OllamaClientError(
+      `compareDraftPerformance: every draft_num_predict=${draftNumPredict} sample returned a non-positive tokens/second. The model name is likely wrong, or the server returned empty generations.`,
+      { code: "draft_benchmark_no_valid_samples" }
+    );
+  }
+  return {
+    draftNumPredict,
+    medianTokensPerSecond: median(tokensPerSecond),
+    sampleTokensPerSecond: tokensPerSecond,
+    medianEvalDurationMs: median(generationMs)
+  };
+}
+async function compareDraftPerformance(client, options) {
+  const draftNumPredict = options.draftNumPredict ?? DEFAULT_DRAFT_TOKENS;
+  requirePositiveInteger("draftNumPredict", draftNumPredict);
+  const samples = options.samples ?? DEFAULT_SAMPLES;
+  requirePositiveInteger("samples", samples);
+  const numPredict = options.numPredict ?? DEFAULT_NUM_PREDICT;
+  requirePositiveInteger("numPredict", numPredict);
+  const seed = options.seed ?? DEFAULT_SEED;
+  if (!Number.isInteger(seed)) {
+    throw new RangeError(`compareDraftPerformance: seed must be an integer (got ${seed}).`);
+  }
+  const gainThreshold = options.gainThreshold ?? DEFAULT_GAIN_THRESHOLD;
+  if (!Number.isFinite(gainThreshold) || gainThreshold <= 0) {
+    throw new RangeError(
+      `compareDraftPerformance: gainThreshold must be a positive finite number (got ${gainThreshold}).`
+    );
+  }
+  options.signal?.throwIfAborted();
+  const baseline = await runSide(client, options, 0, samples, numPredict, seed);
+  const speculative = await runSide(client, options, draftNumPredict, samples, numPredict, seed);
+  const baselineTokensPerSec = roundTo(baseline.medianTokensPerSecond, 1);
+  const speculativeTokensPerSec = roundTo(speculative.medianTokensPerSecond, 1);
+  const speedupMultiplier = baselineTokensPerSec > 0 ? roundTo(speculativeTokensPerSec / baselineTokensPerSec, 2) : 0;
+  return {
+    model: options.model,
+    prompt: options.prompt,
+    draftNumPredict,
+    baselineTokensPerSec,
+    speculativeTokensPerSec,
+    speedupMultiplier,
+    effectiveGain: speedupMultiplier > gainThreshold,
+    gainThreshold,
+    baseline,
+    speculative
+  };
+}
+
 // src/affinity.ts
 var DEFAULT_CONCURRENT_MODELS = 1;
 var DEFAULT_PER_MODEL_CONCURRENCY = 1;
@@ -8011,6 +8116,7 @@ exports.checkChatContext = checkChatContext;
 exports.checkEndpointHealth = checkEndpointHealth;
 exports.checkGenerateContext = checkGenerateContext;
 exports.compactConversationHistory = compactConversationHistory;
+exports.compareDraftPerformance = compareDraftPerformance;
 exports.composeMiddleware = composeMiddleware;
 exports.contextWarningMessage = contextWarningMessage;
 exports.createConsoleLogger = createConsoleLogger;