@lenne.tech/nest-server 11.41.0 → 11.41.1

package/migration-guides/11.41.0-to-11.41.1.md

@@ -0,0 +1,114 @@
+# Migration Guide: 11.41.0 → 11.41.1
+
+## Overview
+
+| Category | Effort | Applies to |
+|----------|--------|-----------|
+| **Bugfix** | none | Projects whose AI connection points at a reasoning model — prompts that silently returned nothing now return an answer |
+| Behaviour change | none, but worth knowing | Projects using `ai.budget` — one prompt can now cost two upstream calls, and both are metered |
+| New (read-only) | none | `LlmResponse.finishReason`, `LlmUsage.reasoningTokens` — both optional |
+
+Most projects update with `pnpm update @lenne.tech/nest-server` and read no further. **Projects that
+do not use the AI module are unaffected by all of it.**
+
+## Quick Migration
+
+```bash
+pnpm update @lenne.tech/nest-server
+```
+
+```bash
+# Do you use the AI module at all? If this is empty, you are done.
+grep -rn "CoreAiModule\|aiConnections\|AiTool" src/ 2>/dev/null
+```
+
+## Bugfix: a reasoning model that thinks past its budget now gets a second chance
+
+### What was broken
+
+A reasoning model spends output tokens on its thinking phase **before** it writes a single character
+of the answer. When that phase consumes the whole `max_tokens` allowance, the endpoint answers a
+perfectly ordinary `HTTP 200` carrying `finish_reason: 'length'`, empty content, and
+`reasoning_tokens == completion_tokens`.
+
+`chat()` handed that on as an empty string. Nothing distinguished it from "the model had nothing to
+say", so a consumer with a modest `maxTokens` received nothing — silently, on every prompt, for as
+long as the connection pointed at such a model. Call sites that degrade to a null-fallback reported
+no error at all.
+
+Measured against an OpenAI-compatible hosting endpoint with a 900-token budget:
+
+| Model | Default | With the thinking phase off |
+|-------|---------|-----------------------------|
+| Mistral-Medium-3.5-128B | empty, 900/900 spent thinking | 348 characters, 0.8 s |
+| Qwen3.6-35B-A3B-FP8 | empty, 900/900 spent thinking | 399 characters, 0.8 s |
+| gpt-oss-120b | 537 characters | HTTP 400 — rejects the parameter |
+| Ministral-3-14B-Instruct | 1213 characters | 901 characters |
+
+**Raising the token budget does not help.** The model spends whatever it is given: 900 of 900, 1500
+of 1500, 2048 of 2048, 4096 of 4096 — always with empty content.
+
+### What happens now
+
+When, and only when, a completion comes back with `finish_reason: 'length'`, no content and no tool
+call, the provider retries the identical request once with `reasoning_effort: 'none'`. Everything
+else is unchanged: a truncated answer that HAS content is kept (retrying would discard usable text),
+an empty answer with `finish_reason: 'stop'` is kept (the model chose to say nothing, so no budget
+ran out), and a tool call is kept (the model did answer, in the tool channel).
+
+The parameter cannot be sent pre-emptively — `gpt-oss-120b` rejects it with a `400` while working
+perfectly well without it. When the retry fails for any reason, the original response is returned
+with its `finishReason` and usage intact, and the log names the actual failure.
+
+### Do I need to do anything?
+
+**No.** There is no new configuration, and deliberately so: the behaviour only triggers on a
+response that previously reached you as an empty string, which is not an outcome anyone can want.
+
+## Behaviour change: one prompt can now cost two upstream calls
+
+Worth knowing if you run with `ai.budget` limits or watch provider spend.
+
+On the retry path both calls are genuinely billed by the provider, so **both are reported in
+`usage`** — `promptTokens`, `completionTokens` and `reasoningTokens` are the sum of the two. That is
+what keeps `ai.budget` honest: the accounting sums `totalTokens` from the audit records, so
+reporting only the retry would hide the starved call, which by definition burned the entire
+`max_tokens` allowance, from the very limit meant to bound it.
+
+Two consequences:
+
+- On a model that starves on every prompt, real token spend against a configured limit is roughly
+  double what the same workload cost before — because it always was, and is now measured. If you
+  set `ai.budget.user.maxTokens` against observed usage from 11.41.0, re-check the number.
+- `contextWindow.used` counts the prompt twice on that path, since the same prompt really was sent
+  twice. It is clamped to the window size, so it can saturate at 100 % but never report nonsense.
+
+The retry shares the original call's timeout budget rather than starting a fresh one, so the
+documented ceiling of `maxIterations` × the per-call timeout still holds. A connection whose entire
+`timeoutMs` is below one second never retries.
+
+## New: two optional fields on the provider contract
+
+```ts
+interface LlmResponse {
+  /** 'stop', 'length', 'tool_calls', … — undefined when the backend omits it. */
+  finishReason?: string;
+}
+
+interface LlmUsage {
+  /** Output tokens spent thinking. Part of completionTokens, not additional to it. */
+  reasoningTokens?: number;
+}
+```
+
+Both are optional additions, so every existing `ILlmProvider` implementation and every `LlmResponse`
+literal still compiles unchanged. They exist because without them a caller cannot tell an answer
+apart from a fragment: `length` means the budget ran out mid-flight, so short or empty text is a
+truncation rather than the model's verdict.
+
+## Under the hood
+
+`OpenAiCompatibleProvider` gained two protected seams a subclass can override: `postCompletion()`
+(one request, mapped to `LlmResponse`) and `isReasoningStarved()` (the detection predicate). The SSRF
+egress allowlist (`ai.allowedBaseUrlHosts`) is applied before both calls, as before — the retry
+reuses the already-validated URL and changes only the request body.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lenne.tech/nest-server",
-  "version": "11.41.0",
+  "version": "11.41.1",
   "description": "Modern, fast, powerful Node.js web framework in TypeScript based on Nest with a GraphQL API and a connection to MongoDB (or other databases).",
   "keywords": [
     "node",

package/src/core/modules/ai/interfaces/llm-provider.interface.ts

@@ -100,6 +100,17 @@ export interface LlmToolCall {
 export interface LlmUsage {
   completionTokens?: number;
   promptTokens?: number;
+  /**
+   * Output tokens the model spent THINKING before answering, where the backend
+   * reports them (`completion_tokens_details.reasoning_tokens`). Part of
+   * {@link completionTokens}, not additional to it.
+   *
+   * Worth surfacing because the thinking phase competes with the answer for the
+   * SAME budget: when it equals `completionTokens` the model never got to the
+   * answer, which is a very different failure from a model that had nothing to
+   * say. See {@link LlmResponse.finishReason}.
+   */
+  reasoningTokens?: number;
   totalTokens?: number;
 }
 
@@ -140,6 +151,19 @@ export interface LlmCompletionOptions {
  * Normalized response of a single LLM completion.
  */
 export interface LlmResponse {
+  /**
+   * Why the model stopped, as reported by the backend (`stop`, `length`,
+   * `tool_calls`, …). Undefined when the backend omits it.
+   *
+   * Without this a caller cannot tell an answer apart from a fragment: `length`
+   * means the output budget ran out mid-flight, so short or empty text is a
+   * truncation and not the model's verdict. Retrying an identical request is
+   * pointless in that case — and actively misleading against a backend that
+   * caches identical prompts, which answers the retry from cache in
+   * milliseconds.
+   */
+  finishReason?: string;
+
   /** Raw provider payload (for debugging/audit, never sent to clients). */
   raw?: unknown;
 

package/src/core/modules/ai/providers/openai-compatible.provider.ts

@@ -9,6 +9,7 @@ import {
   LlmMessage,
   LlmResponse,
   LlmToolSchema,
+  LlmUsage,
 } from '../interfaces/llm-provider.interface';
 import { ResolvedAiConnection } from '../interfaces/resolved-ai-connection.interface';
 
@@ -29,6 +30,22 @@ export class OpenAiCompatibleProvider implements ILlmProvider {
   readonly capabilities: LlmCapabilities;
   readonly name = 'openai-compatible';
 
+  /**
+   * Floor below which the reasoning retry is not attempted at all.
+   *
+   * The retry shares the ORIGINAL call's timeout budget (see {@link chat}), so a
+   * first call that nearly exhausted it leaves too little for a second. Starting one
+   * anyway would spend the remainder waiting for a request that cannot finish, and
+   * then report the timeout as "the model rejects reasoning_effort". Measured retries
+   * answer in well under a second once the thinking phase is off.
+   *
+   * Consequence worth stating: a connection whose whole `timeoutMs` is below this floor
+   * never retries at all. That is the right trade for a value this far below any usable
+   * LLM timeout (the default is 120 s), but it is a behaviour the number decides, so it
+   * belongs here rather than in a reader's head.
+   */
+  protected static readonly MIN_REASONING_RETRY_MS = 1_000;
+
   private readonly logger = new Logger(OpenAiCompatibleProvider.name);
   private readonly defaultTimeoutMs: number;
 
@@ -93,6 +110,93 @@ export class OpenAiCompatibleProvider implements ILlmProvider {
     }
 
     const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
+    const startedAt = Date.now();
+    const answer = await this.postCompletion(url, body, timeoutMs);
+    if (!this.isReasoningStarved(answer)) {
+      return answer;
+    }
+
+    // The thinking phase ate the whole budget before a single character of the
+    // answer. Say so with the numbers — the failure is otherwise indistinguishable
+    // in a log from "the model had nothing to say", which is what made it so
+    // expensive to diagnose.
+    this.logger.warn(
+      `AI completion for model "${body.model}" returned NO content: finish_reason=${answer.finishReason}, ` +
+        `${answer.usage?.reasoningTokens ?? 0} of ${answer.usage?.completionTokens ?? 0} output tokens were spent ` +
+        `thinking against a budget of ${body.max_tokens}. Retrying once without the thinking phase.`,
+    );
+    // The retry shares the ORIGINAL call's timeout budget instead of starting a fresh
+    // one. `CoreAiService` checks `ai.maxRunMs` BETWEEN agent-loop iterations and never
+    // mid-call, so a second full timeout here would silently double the ceiling the
+    // framework documents as `maxIterations` x the per-call timeout — 20 minutes instead
+    // of 10 at the defaults. On the SSE path that is time the client spends in total
+    // silence: `promptStream()` yields nothing but tool actions until the run settles,
+    // and a starved completion produces no tool call to report.
+    const remainingMs = timeoutMs - (Date.now() - startedAt);
+    if (remainingMs < OpenAiCompatibleProvider.MIN_REASONING_RETRY_MS) {
+      this.logger.warn(
+        `Not retrying model "${body.model}" without the thinking phase: only ${Math.max(0, remainingMs)}ms of the ` +
+          `${timeoutMs}ms budget remain. Keeping the original, empty completion.`,
+      );
+      return answer;
+    }
+
+    try {
+      const retried = await this.postCompletion(url, { ...body, reasoning_effort: 'none' }, remainingMs);
+      // BOTH calls were billed upstream, so both must reach the caller. `CoreAiService`
+      // accumulates only what `chat()` returns, that total lands in the audit record's
+      // `totalTokens`, and `CoreAiBudgetService` enforces `ai.budget` from exactly that
+      // field. Returning the retry's usage alone would hide the STARVED call — the one
+      // that by definition burned the entire `max_tokens` allowance — from the limit
+      // that exists to bound it.
+      return { ...retried, usage: this.mergeUsage(answer.usage, retried.usage) };
+    } catch (err) {
+      // Usually a 400: the backend does not accept `reasoning_effort` for this model.
+      // But the same catch also sees timeouts and transport failures, and naming the
+      // 400 for one of those would send the reader after a cause that is not there —
+      // the precise kind of misdirection this whole change exists to remove. Report
+      // what actually happened and let the original answer stand: it keeps its
+      // `finishReason` and the usage the caller already paid for.
+      this.logger.warn(
+        `Retry without the thinking phase failed for model "${body.model}" ` +
+          `(${(err as Error)?.message ?? 'unknown error'}) — keeping the original, empty completion. ` +
+          'Raise the token budget or configure a model that answers within it.',
+      );
+      return answer;
+    }
+  }
+
+  /**
+   * Add up the usage of the two calls a retried completion actually made.
+   *
+   * Kept separate from {@link chat} because "what did this run cost" is a question the
+   * budget, the audit record and the client's usage summary all read from one number,
+   * and a provider that reports only half of it under-enforces every limit built on it.
+   *
+   * A field absent from BOTH sides stays absent — a backend that reports no breakdown
+   * must not be made to look like it reported zero.
+   */
+  protected mergeUsage(first?: LlmUsage, second?: LlmUsage): LlmUsage | undefined {
+    if (!first) {
+      return second;
+    }
+    if (!second) {
+      return first;
+    }
+    const sum = (a?: number, b?: number) => (a === undefined && b === undefined ? undefined : (a ?? 0) + (b ?? 0));
+    return {
+      completionTokens: sum(first.completionTokens, second.completionTokens),
+      promptTokens: sum(first.promptTokens, second.promptTokens),
+      reasoningTokens: sum(first.reasoningTokens, second.reasoningTokens),
+      totalTokens: sum(first.totalTokens, second.totalTokens),
+    };
+  }
+
+  /**
+   * POST one completion and map it to {@link LlmResponse}. Transport failures and
+   * non-2xx responses throw, exactly as a single-shot `chat()` always did.
+   */
+  protected async postCompletion(url: string, body: Record<string, any>, timeoutMs: number): Promise<LlmResponse> {
     let response: Response;
     try {
       response = await fetch(url, {
@@ -117,26 +221,64 @@ export class OpenAiCompatibleProvider implements ILlmProvider {
     }
 
     const result = (await response.json()) as {
-      choices?: { message?: { content?: string; tool_calls?: any[] } }[];
-      usage?: { completion_tokens?: number; prompt_tokens?: number; total_tokens?: number };
+      choices?: { finish_reason?: string; message?: { content?: string; tool_calls?: any[] } }[];
+      usage?: {
+        completion_tokens?: number;
+        completion_tokens_details?: { reasoning_tokens?: number };
+        prompt_tokens?: number;
+        total_tokens?: number;
+      };
     };
 
-    const choice = result.choices?.[0]?.message;
-    const text = choice?.content ?? '';
-    const nativeToolCalls = this.capabilities.nativeTools ? this.mapNativeToolCalls(choice?.tool_calls) : undefined;
+    const choice = result.choices?.[0];
+    const text = choice?.message?.content ?? '';
+    const nativeToolCalls = this.capabilities.nativeTools
+      ? this.mapNativeToolCalls(choice?.message?.tool_calls)
+      : undefined;
 
     return {
+      finishReason: choice?.finish_reason,
       raw: result,
       text,
       toolCalls: nativeToolCalls,
       usage: {
         completionTokens: result.usage?.completion_tokens,
         promptTokens: result.usage?.prompt_tokens,
+        reasoningTokens: result.usage?.completion_tokens_details?.reasoning_tokens,
         totalTokens: result.usage?.total_tokens,
       },
     };
   }
 
+  /**
+   * True when the output budget was exhausted before the model produced ANY
+   * answer — a reasoning model that spent every token on its thinking phase.
+   *
+   * The backend answers `200` with `finish_reason: 'length'`, empty content and
+   * (where it reports the breakdown) `reasoning_tokens == completion_tokens`.
+   * Measured against an OpenAI-compatible hosting endpoint on 2026-09-07 with a 900-token budget:
+   * `Mistral-Medium-3.5-128B` and `Qwen3.6-35B-A3B-FP8` both return nothing,
+   * while `gpt-oss-120b` and `Ministral-3-14B-Instruct` answer normally.
+   *
+   * The narrowness is deliberate, in both directions:
+   *
+   * - **Content present** → the answer merely got cut short. That is a budget
+   *   question the caller can now see via `finishReason`, and discarding the
+   *   partial text to re-ask would lose something usable.
+   * - **`finish_reason: 'stop'` with empty content** → the model chose to say
+   *   nothing. No budget ran out, so removing the thinking phase addresses
+   *   nothing and would cost a second upstream call on every such answer.
+   * - **Tool calls present** → the model DID answer, in the tool channel.
+   *
+   * `reasoning_tokens` is treated as corroborating, not required: backends that
+   * omit the breakdown produce exactly the same symptom, and the retry is
+   * harmless where the diagnosis is wrong (one extra call on a request that
+   * returned nothing either way).
+   */
+  protected isReasoningStarved(response: LlmResponse): boolean {
+    return response.finishReason === 'length' && !response.text && !response.toolCalls?.length;
+  }
+
   /**
    * Optional SSRF hardening: when `ai.allowedBaseUrlHosts` is configured (non-empty),
    * only allow requests to those hosts (matched by `host` incl. port, or bare