flexinference 1.4.1 → 1.5.1

package/CHANGELOG.md

@@ -1,5 +1,33 @@
 # Changelog
 
+## 1.5.1
+
+Small robustness fixes for the transport, no API changes. A non-streaming request that hits
+its total `timeout` now throws a clear timeout error instead of a bare abort that looked like
+a caller cancellation. A streamed response flushes its decoder and delivers the final frame
+even when the stream ends without a trailing blank line, so the last event is never dropped,
+and the streaming buffer is capped so a stream that never sends a separator cannot grow
+without bound. The `start_within` type now matches the runtime rule, so a value the client
+would reject at run time no longer type-checks.
+
+## 1.5.0
+
+Reworks request timeouts so a healthy stream is never cut off yet a hung request never hangs
+forever. Streaming requests now have **no total cap** -- they run as long as tokens keep
+arriving -- and are bounded instead by two new options: `firstByteTimeout` (how long to wait
+for the first response, default `60000` ms) and `idleTimeout` (max silence between chunks
+before the stream is treated as hung, default `60000` ms). Non-streaming requests keep the
+total `timeout` (default `600000` ms). The first-byte wait is **auto-raised for a flex
+`start_within`**, fixing flex requests whose deadline exceeded the old hardcoded 10s connect
+cap and aborted before the router replied. All three timeouts are configurable on the client
+and per request. Behavior change, no API breaks: `timeout` no longer caps a stream, and the
+old 10s first-response cap is now 60s (auto-sized for flex).
+
+Also teaches `FlexInferenceError` to parse the router's per-surface error shapes: the router now
+returns errors in the dialect of the endpoint you called (OpenAI on responses/chat, Anthropic on
+`messages`, Google on `interactions`), and the error class reads all three (`message` always,
+`type` from the OpenAI/Anthropic `type` or Gemini `status`).
+
 ## 1.4.1
 
 Rewrites the README in plain language from the copy audit. It explains what `start_within` does, how the cheaper flex tier runs first and falls back up to your standard tier when it cannot start in time, and how you only pay a share of what a flex request saves you. No code or API changes.

package/README.md

@@ -102,19 +102,29 @@ console.log(messageText(res));
 
 ## Timeouts and cancellation
 
-Every client has a built-in timeout: **10 seconds to connect** and **600 seconds (10 min) total**, matching the router's budget, so a request can never hang forever. Override the total with the `timeout` option (raise it for very long streams):
+Streaming and non-streaming requests are timed differently, so a long generation is never cut off yet a hung request never hangs forever:
+
+- **Non-streaming** requests have a total wall-clock budget: `timeout`, default **600000 ms (10 min)**, from send through the full body.
+- **Streaming** requests have **no total cap** (a healthy stream runs as long as tokens keep arriving). They are bounded by two clocks: `firstByteTimeout` (wait for the first response, default **60000 ms**) and `idleTimeout` (max silence between chunks before the stream is treated as hung, default **60000 ms**).
+
+The first-byte wait is **auto-raised for a flex `start_within`** (the router withholds response headers until the flex race resolves), so a long deadline just works. Raise `firstByteTimeout` yourself only for very large non-flex contexts whose first token is slow.
 
 ```ts
-const client = new FlexInference({ apiKey: "flex_live_...", timeout: 120_000 });
+const client = new FlexInference({
+  apiKey: "flex_live_...",
+  timeout: 120_000, // non-streaming total budget
+  firstByteTimeout: 90_000, // wait for the first response
+  idleTimeout: 30_000, // max silence between streamed chunks
+});
 ```
 
-Every call also accepts an `AbortSignal` as a second argument, so you can cancel a request yourself. Cancelling a stream stops it mid-flight:
+Any of these can be overridden per request, alongside an `AbortSignal` to cancel yourself. Cancelling a stream stops it mid-flight:
 
 ```ts
 const controller = new AbortController();
 const res = await client.responses.create(
   { model: "gpt-5.5", input: "Write a haiku about cheap GPUs.", start_within: "priority" },
-  { signal: controller.signal },
+  { signal: controller.signal, idleTimeout: 15_000 },
 );
 // controller.abort() from elsewhere to cancel
 console.log(outputText(res));
@@ -122,7 +132,7 @@ console.log(outputText(res));
 
 ## Errors
 
-Non-2xx responses throw `FlexInferenceError`, carrying the OpenAI-shaped `status`, `type`, `code`, and `param`:
+Non-2xx responses throw `FlexInferenceError`, carrying `status`, `type`, `code`, and `param`. The router shapes error bodies to match the endpoint you called (OpenAI on `responses`/`chat`, Anthropic on `messages`, Google on `interactions`) so the SDK you would use for that surface parses them; `FlexInferenceError` reads all three. `message` and `status` are always set; `code`, `param`, and `docUrl` are populated on the OpenAI surface.
 
 ```ts
 import { FlexInferenceError } from "flexinference";
@@ -138,7 +148,7 @@ try {
 }
 ```
 
-Every FlexInference error tells you four things. It says what went wrong, why it went wrong, how to fix it, and it shows an example of a request that works. The `message` reads like a note from a person, so an agent can act on it instead of guessing. Errors that come from the provider pass straight through with their status and body intact, so you always see the real cause. For instance, a duration on a `claude-*` model returns `400 flex_unsupported_for_anthropic` with a message that tells you to drop the duration or switch to `default`, `priority`, or `auto`.
+Every FlexInference error tells you four things. It says what went wrong, why it went wrong, how to fix it, and it shows an example of a request that works. The `message` reads like a note from a person, so an agent can act on it instead of guessing. Provider errors are reshaped into the same surface envelope (and normalized into `FlexInferenceError`), so you get one consistent error type no matter which model ran. For instance, a duration on a `claude-*` model returns `400 flex_unsupported_for_anthropic` with a message that tells you to drop the duration or switch to `default`, `priority`, or `auto`.
 
 ## Billing / 402
 
@@ -169,12 +179,16 @@ Because `PaymentRequiredError extends FlexInferenceError`, existing
 
 ## Configuration
 
-| Option    | Default                            | Description                                      |
-| --------- | ---------------------------------- | ------------------------------------------------ |
-| `apiKey`  | -                                  | Your `flex_live_` key (required).                |
-| `baseURL` | `https://api.flexinference.com/v1` | Override the router endpoint.                    |
-| `fetch`   | global `fetch`                     | Provide a custom fetch implementation.           |
-| `timeout` | `600000`                           | Total request budget in ms (connect capped 10s). |
+| Option             | Default                            | Description                                                                     |
+| ------------------ | ---------------------------------- | ------------------------------------------------------------------------------- |
+| `apiKey`           | -                                  | Your `flex_live_` key (required).                                               |
+| `baseURL`          | `https://api.flexinference.com/v1` | Override the router endpoint.                                                   |
+| `fetch`            | global `fetch`                     | Provide a custom fetch implementation.                                          |
+| `timeout`          | `600000`                           | Non-streaming total budget in ms (streaming has no total).                      |
+| `firstByteTimeout` | `60000`                            | Wait for the first response in ms; auto-raised for a flex `start_within`.       |
+| `idleTimeout`      | `60000`                            | Max silence between streamed chunks in ms before the stream is treated as hung. |
+
+All three timeouts can also be passed per request (with `signal`) in the second argument.
 
 ## License
 

package/dist/index.d.ts

@@ -70,7 +70,7 @@ export declare class FlexInferenceError extends Error {
     readonly param: string | null | undefined;
     /** Deep link into the docs for this error's code, when the router supplied one. */
     readonly docUrl: string | null | undefined;
-    constructor(status: number, body: Partial<FlexErrorBody> | undefined, fallback: string);
+    constructor(status: number, body: unknown, fallback: string);
 }
 /**
  * Thrown on HTTP 402: billing is past due, so billable flex is paused. Free routing
@@ -86,19 +86,41 @@ export interface ClientOptions {
     /** Override the global fetch (e.g. for tests or non-standard runtimes). */
     fetch?: typeof fetch;
     /**
-     * Total request budget in ms (connect through the full response/stream). Defaults
-     * to 600000 (10 min). Connect alone is capped at 10s. Raise it for very long streams.
+     * Non-streaming wall-clock budget in ms (from send through the full body). Defaults to
+     * 600000 (10 min). Streaming requests ignore this: they are governed by `firstByteTimeout`
+     * and `idleTimeout` and have no total cap, so a healthy long stream is never cut off.
      */
     timeout?: number;
+    /**
+     * How long to wait in ms for the first response (headers) before the router has sent
+     * anything. Defaults to 60000. For a flex `start_within` it is auto-raised to cover the
+     * deadline (the router withholds headers during the race), so you rarely set this; raise
+     * it for very large non-flex contexts whose first token is slow.
+     */
+    firstByteTimeout?: number;
+    /**
+     * Max silence in ms between streamed chunks before the stream is treated as hung and
+     * aborted. Defaults to 60000. Streaming requests only.
+     */
+    idleTimeout?: number;
 }
 export interface RequestOptions {
     signal?: AbortSignal;
+    /** Override the client `firstByteTimeout` for this request (ms). */
+    firstByteTimeout?: number;
+    /** Override the client `idleTimeout` for this streaming request (ms). */
+    idleTimeout?: number;
+    /** Override the client `timeout` (non-streaming total budget) for this request (ms). */
+    timeout?: number;
 }
-interface PostResult {
+interface OpenResult {
     res: Response;
+    controller: AbortController;
     cleanup: () => void;
+    idleMs: number;
+    totalMs: number;
 }
-type Post = (path: string, body: unknown, signal: AbortSignal | undefined) => Promise<PostResult>;
+type Send = (path: string, body: unknown, options: RequestOptions | undefined) => Promise<OpenResult>;
 export declare class FlexInference {
     readonly responses: Responses;
     readonly chat: Chat;
@@ -107,13 +129,17 @@ export declare class FlexInference {
     private readonly apiKey;
     private readonly baseURL;
     private readonly fetchImpl;
-    private readonly timeoutMs;
+    private readonly firstByteMs;
+    private readonly idleMs;
+    private readonly totalMs;
     constructor(options: ClientOptions);
-    private post;
+    private timeoutsFor;
+    private send;
+    private open;
 }
 declare class Responses {
-    private readonly post;
-    constructor(post: Post);
+    private readonly send;
+    constructor(send: Send);
     create(body: ResponseCreateParams & {
         stream?: false | null;
     }, options?: RequestOptions): Promise<ResponseObject>;
@@ -123,11 +149,11 @@ declare class Responses {
 }
 declare class Chat {
     readonly completions: ChatCompletions;
-    constructor(post: Post);
+    constructor(send: Send);
 }
 declare class ChatCompletions {
-    private readonly post;
-    constructor(post: Post);
+    private readonly send;
+    constructor(send: Send);
     create(body: ChatCompletionCreateParams & {
         stream?: false | null;
     }, options?: RequestOptions): Promise<ChatCompletion>;
@@ -136,8 +162,8 @@ declare class ChatCompletions {
     }, options?: RequestOptions): Promise<AsyncIterable<ChatCompletionChunk>>;
 }
 declare class Interactions {
-    private readonly post;
-    constructor(post: Post);
+    private readonly send;
+    constructor(send: Send);
     create(body: InteractionCreateParams & {
         stream?: false | null;
     }, options?: RequestOptions): Promise<InteractionObject>;
@@ -146,8 +172,8 @@ declare class Interactions {
     }, options?: RequestOptions): Promise<AsyncIterable<InteractionStreamEvent>>;
 }
 declare class Messages {
-    private readonly post;
-    constructor(post: Post);
+    private readonly send;
+    constructor(send: Send);
     create(body: MessageCreateParams & {
         stream?: false | null;
     }, options?: RequestOptions): Promise<MessageObject>;

package/dist/index.js

@@ -92,15 +92,26 @@ export class FlexInferenceError extends Error {
     param;
     /** Deep link into the docs for this error's code, when the router supplied one. */
     docUrl;
+    // The router shapes errors by the endpoint you called, so `body` may be the OpenAI
+    // envelope (responses/chat), the Anthropic one (messages: `{type:"error", error:{...}}`),
+    // or the Google one (interactions: `{error:{code,message,status}}`). All three nest the
+    // detail under `error`, so this parses whichever arrived. `message` is always populated;
+    // `type` reads OpenAI/Anthropic `error.type` or falls back to Gemini's `error.status`;
+    // `code`/`param`/`doc_url` are OpenAI-only and undefined on the other shapes.
     constructor(status, body, fallback) {
-        const err = body?.error;
-        super(err?.message ?? fallback);
+        const err = isRecord(body) && isRecord(body.error) ? body.error : undefined;
+        super(typeof err?.message === "string" ? err.message : fallback);
         this.name = "FlexInferenceError";
         this.status = status;
-        this.type = err?.type;
-        this.code = err?.code;
-        this.param = err?.param;
-        this.docUrl = err?.doc_url;
+        this.type =
+            typeof err?.type === "string"
+                ? err.type
+                : typeof err?.status === "string"
+                    ? err.status
+                    : undefined;
+        this.code = typeof err?.code === "string" ? err.code : undefined;
+        this.param = typeof err?.param === "string" ? err.param : undefined;
+        this.docUrl = typeof err?.doc_url === "string" ? err.doc_url : undefined;
     }
 }
 /**
@@ -111,13 +122,28 @@ export class FlexInferenceError extends Error {
 export class PaymentRequiredError extends FlexInferenceError {
 }
 const DEFAULT_BASE_URL = "https://api.flexinference.com/v1";
-const DEFAULT_TIMEOUT_MS = 600_000; // 10 min total budget, matching the router's generosity
-const CONNECT_TIMEOUT_MS = 10_000; // fail fast if the router is unreachable
+// A streaming request is governed by two clocks and no total: how long we wait for the
+// first response (headers), and the maximum silence between chunks once it is flowing. A
+// healthy long generation is allowed to run as long as tokens keep arriving. A
+// non-streaming request has no stream to idle-watch, so it keeps a total wall-clock budget
+// instead.
+const DEFAULT_FIRST_BYTE_MS = 60_000; // wait for the first response; auto-raised for a flex start_within
+const FLEX_FIRST_BYTE_MARGIN_MS = 30_000; // headroom over the flex deadline for the fallback-to-standard path
+const DEFAULT_IDLE_MS = 60_000; // max silence between streamed chunks before the stream is treated as hung
+const DEFAULT_TOTAL_MS = 600_000; // non-streaming wall-clock budget (streaming has none)
+const ERROR_BODY_READ_MS = 30_000; // bound the read of a non-2xx error body
+// Cap the streaming buffer above the largest legitimate single frame (a terminal
+// response object is well under this). A stream that never sends a blank-line separator
+// would otherwise grow the buffer without bound; error clearly instead. See T2.
+const MAX_SSE_BUFFER_CHARS = 8 * 1024 * 1024;
 const MIN_DEADLINE_SECONDS = 5;
 const MAX_DEADLINE_SECONDS = 600;
 const SECONDS_PER_HOUR = 3600;
 const SECONDS_PER_MINUTE = 60;
-const DURATION_RE = /^(\d{2})h-(\d{2})m-(\d{2})s$/;
+// One or two digits per field so the runtime accepts the same shapes the StartWithin
+// template-literal type does (e.g. "0h-5m-0s"): `${number}` in the type imposes no digit
+// count, so a two-digit-only regex would reject values that type-check. See T4.
+const DURATION_RE = /^(\d{1,2})h-(\d{1,2})m-(\d{1,2})s$/;
 const START_WITHIN_VALUES = '"default", "priority", "auto", or a duration "HHh-MMm-SSs"';
 // Runtime guard so plain-JS callers (who get no compile-time check) still fail
 // locally with a clear message instead of a provider round-trip. TypeScript callers
@@ -138,13 +164,61 @@ function assertStartWithin(body) {
     const match = DURATION_RE.exec(value);
     if (match === null) {
         throw new Error(`FlexInference: \`start_within\` "${value}" is not ${START_WITHIN_VALUES}. ` +
-            `A duration is two digits each for hours, minutes, and seconds, for example "00h-05m-00s" for a 5-minute deadline.`);
+            `A duration is one or two digits each for hours, minutes, and seconds, for example "00h-05m-00s" for a 5-minute deadline.`);
     }
     const seconds = Number(match[1]) * SECONDS_PER_HOUR + Number(match[2]) * SECONDS_PER_MINUTE + Number(match[3]);
     if (seconds < MIN_DEADLINE_SECONDS || seconds > MAX_DEADLINE_SECONDS) {
         throw new Error("FlexInference: `start_within` duration must be between 5s and 10m.");
     }
 }
+// Distinct abort reasons so an aborted fetch/read can be told apart: a timeout we fired
+// (turned into a clear message) versus the caller's own AbortSignal (propagated as-is).
+const FIRST_BYTE_TIMEOUT = Symbol("first-byte-timeout");
+const IDLE_TIMEOUT = Symbol("idle-timeout");
+const TOTAL_TIMEOUT = Symbol("total-timeout");
+/**
+ * Seconds for a flex-duration `start_within`, or null for a tier value ("default" etc.),
+ * a missing field, or a malformed one. Used to auto-size the first-byte wait: the router
+ * withholds response headers until the flex race commits or falls back, so the first-byte
+ * clock must cover the whole deadline plus the fallback path.
+ */
+function flexDeadlineSeconds(body) {
+    const value = typeof body === "object" && body !== null
+        ? body["start_within"]
+        : undefined;
+    if (typeof value !== "string")
+        return null;
+    const match = DURATION_RE.exec(value);
+    if (match === null)
+        return null;
+    return (Number(match[1]) * SECONDS_PER_HOUR + Number(match[2]) * SECONDS_PER_MINUTE + Number(match[3]));
+}
+// Read a non-streamed JSON body under the total wall-clock budget, then release the timer
+// and the abort listener. Non-streaming only: a stream is drained by streamSSE instead.
+async function readJson(open) {
+    const { res, controller, cleanup, totalMs } = open;
+    // Abort with a dedicated reason so a fired total timeout can be told apart from the
+    // caller cancelling on purpose, and translated into a clear message (parity with the
+    // first-byte and idle timeouts, and with the Python SDK). See T1.
+    const totalTimer = setTimeout(() => {
+        controller.abort(TOTAL_TIMEOUT);
+    }, totalMs);
+    try {
+        return (await res.json());
+    }
+    catch (err) {
+        if (controller.signal.reason === TOTAL_TIMEOUT) {
+            throw new Error(`FlexInference: no full response from the router within ${String(totalMs)}ms ` +
+                "(total timeout). Raise `timeout` for very large non-streaming responses, or " +
+                "stream the request so a long generation is governed by the idle timeout instead.");
+        }
+        throw err;
+    }
+    finally {
+        clearTimeout(totalTimer);
+        cleanup();
+    }
+}
 export class FlexInference {
     responses;
     chat;
@@ -153,28 +227,54 @@ export class FlexInference {
     apiKey;
     baseURL;
     fetchImpl;
-    timeoutMs;
+    firstByteMs;
+    idleMs;
+    totalMs;
     constructor(options) {
         if (!options.apiKey)
             throw new Error("FlexInference: `apiKey` is required.");
         this.apiKey = options.apiKey;
         this.baseURL = (options.baseURL ?? DEFAULT_BASE_URL).replace(/\/+$/, "");
-        this.timeoutMs = options.timeout ?? DEFAULT_TIMEOUT_MS;
+        this.firstByteMs = options.firstByteTimeout ?? DEFAULT_FIRST_BYTE_MS;
+        this.idleMs = options.idleTimeout ?? DEFAULT_IDLE_MS;
+        this.totalMs = options.timeout ?? DEFAULT_TOTAL_MS;
         const f = options.fetch ?? globalThis.fetch;
         if (typeof f !== "function") {
             throw new Error("FlexInference: no global fetch found; pass `fetch` in options.");
         }
         this.fetchImpl = f;
-        const post = (path, body, signal) => this.post(path, body, signal);
-        this.responses = new Responses(post);
-        this.chat = new Chat(post);
-        this.interactions = new Interactions(post);
-        this.messages = new Messages(post);
+        const send = (path, body, options) => this.send(path, body, options);
+        this.responses = new Responses(send);
+        this.chat = new Chat(send);
+        this.interactions = new Interactions(send);
+        this.messages = new Messages(send);
+    }
+    // Resolve the three per-request clocks. A per-request override always wins. Otherwise the
+    // first-byte wait is auto-sized for a flex `start_within`: it can never be shorter than
+    // the deadline plus the fallback margin (headers are withheld for the whole race), but a
+    // larger client default still wins over a short deadline.
+    timeoutsFor(body, options) {
+        const deadline = flexDeadlineSeconds(body);
+        const autoFirstByte = deadline === null
+            ? this.firstByteMs
+            : Math.max(this.firstByteMs, deadline * 1000 + FLEX_FIRST_BYTE_MARGIN_MS);
+        return {
+            firstByteMs: options?.firstByteTimeout ?? autoFirstByte,
+            idleMs: options?.idleTimeout ?? this.idleMs,
+            totalMs: options?.timeout ?? this.totalMs,
+        };
+    }
+    async send(path, body, options) {
+        const { firstByteMs, idleMs, totalMs } = this.timeoutsFor(body, options);
+        const { res, controller, cleanup } = await this.open(path, body, options?.signal, firstByteMs);
+        return { res, controller, cleanup, idleMs, totalMs };
     }
-    async post(path, body, signal) {
-        // One AbortController fed by three sources: the caller's signal, a 10s connect
-        // timer (cleared once headers arrive), and the total budget (cleared by the
-        // consumer once the body/stream is done). No dangling timers after completion.
+    // Open the request and return once response headers arrive (or throw on an error status).
+    // One AbortController is fed by two sources here: the caller's signal, and a first-byte
+    // timer (cleared once headers arrive). The idle timer (streaming) or total timer
+    // (non-streaming) is armed by the caller against the returned controller. Distinct abort
+    // reasons let a fired timeout be told apart from a caller abort.
+    async open(path, body, signal, firstByteMs) {
         const controller = new AbortController();
         const onCallerAbort = () => {
             controller.abort(signal?.reason);
@@ -185,17 +285,12 @@ export class FlexInference {
             else
                 signal.addEventListener("abort", onCallerAbort, { once: true });
         }
-        const connectTimer = setTimeout(() => {
-            controller.abort();
-        }, CONNECT_TIMEOUT_MS);
-        const totalTimer = setTimeout(() => {
-            controller.abort();
-        }, this.timeoutMs);
         const cleanup = () => {
-            clearTimeout(connectTimer);
-            clearTimeout(totalTimer);
             signal?.removeEventListener("abort", onCallerAbort);
         };
+        const firstByteTimer = setTimeout(() => {
+            controller.abort(FIRST_BYTE_TIMEOUT);
+        }, firstByteMs);
         let res;
         try {
             res = await this.fetchImpl(`${this.baseURL}${path}`, {
@@ -210,24 +305,30 @@ export class FlexInference {
             });
         }
         catch (err) {
+            clearTimeout(firstByteTimer);
             cleanup();
-            if (controller.signal.aborted && signal?.aborted !== true) {
-                throw new Error(`FlexInference: request to the router timed out (connect ${String(CONNECT_TIMEOUT_MS)}ms / total ${String(this.timeoutMs)}ms).`);
+            if (controller.signal.reason === FIRST_BYTE_TIMEOUT) {
+                throw new Error(`FlexInference: no response from the router within ${String(firstByteMs)}ms ` +
+                    "(time to first byte). Raise `firstByteTimeout` for very large contexts or long " +
+                    "`start_within` deadlines.");
             }
             throw err;
         }
-        clearTimeout(connectTimer); // headers arrived; only the total budget governs the body now
+        clearTimeout(firstByteTimer); // headers arrived
         if (!res.ok) {
-            // Read the error body under the same timeout, then clear the timers. Clearing
-            // before the read would leave a stalled error body with no budget to abort it.
+            // Read the error body under a bounded budget, then release the abort listener.
+            const errTimer = setTimeout(() => {
+                controller.abort();
+            }, ERROR_BODY_READ_MS);
             let parsed;
             try {
-                parsed = (await res.json());
+                parsed = await res.json();
             }
             catch {
                 parsed = undefined;
             }
             finally {
+                clearTimeout(errTimer);
                 cleanup();
             }
             if (res.status === 402) {
@@ -235,92 +336,82 @@ export class FlexInference {
             }
             throw new FlexInferenceError(res.status, parsed, `HTTP ${String(res.status)} ${res.statusText}`);
         }
-        return { res, cleanup };
+        return { res, controller, cleanup };
     }
 }
 class Responses {
-    post;
-    constructor(post) {
-        this.post = post;
+    send;
+    constructor(send) {
+        this.send = send;
     }
     async create(body, options) {
         assertStartWithin(body);
-        const { res, cleanup } = await this.post("/responses", body, options?.signal);
-        if (body.stream) {
-            return streamSSE(res, cleanup);
-        }
-        try {
-            return (await res.json());
-        }
-        finally {
-            cleanup();
-        }
+        const open = await this.send("/responses", body, options);
+        if (body.stream)
+            return streamSSE(open);
+        return readJson(open);
     }
 }
 class Chat {
     completions;
-    constructor(post) {
-        this.completions = new ChatCompletions(post);
+    constructor(send) {
+        this.completions = new ChatCompletions(send);
     }
 }
 class ChatCompletions {
-    post;
-    constructor(post) {
-        this.post = post;
+    send;
+    constructor(send) {
+        this.send = send;
     }
     async create(body, options) {
         assertStartWithin(body);
-        const { res, cleanup } = await this.post("/chat/completions", body, options?.signal);
-        if (body.stream) {
-            return streamSSE(res, cleanup);
-        }
-        try {
-            return (await res.json());
-        }
-        finally {
-            cleanup();
-        }
+        const open = await this.send("/chat/completions", body, options);
+        if (body.stream)
+            return streamSSE(open);
+        return readJson(open);
     }
 }
 class Interactions {
-    post;
-    constructor(post) {
-        this.post = post;
+    send;
+    constructor(send) {
+        this.send = send;
     }
     async create(body, options) {
         assertStartWithin(body);
-        const { res, cleanup } = await this.post("/interactions", body, options?.signal);
-        if (body.stream) {
-            return streamSSE(res, cleanup);
-        }
-        try {
-            return (await res.json());
-        }
-        finally {
-            cleanup();
-        }
+        const open = await this.send("/interactions", body, options);
+        if (body.stream)
+            return streamSSE(open);
+        return readJson(open);
     }
 }
 class Messages {
-    post;
-    constructor(post) {
-        this.post = post;
+    send;
+    constructor(send) {
+        this.send = send;
     }
     async create(body, options) {
         assertStartWithin(body);
-        const { res, cleanup } = await this.post("/messages", body, options?.signal);
-        if (body.stream) {
-            return streamSSE(res, cleanup);
-        }
-        try {
-            return (await res.json());
-        }
-        finally {
-            cleanup();
-        }
+        const open = await this.send("/messages", body, options);
+        if (body.stream)
+            return streamSSE(open);
+        return readJson(open);
+    }
+}
+function parseSSEFrame(frame) {
+    const dataLines = [];
+    for (const line of frame.split("\n")) {
+        if (line.startsWith("data:"))
+            dataLines.push(line.slice(5).replace(/^ /, ""));
     }
+    if (dataLines.length === 0)
+        return null;
+    const data = dataLines.join("\n");
+    if (data === "[DONE]")
+        return { kind: "done" };
+    return { kind: "value", value: JSON.parse(data) };
 }
-async function* streamSSE(res, cleanup) {
+async function* streamSSE(open) {
+    const { res, controller, idleMs, cleanup } = open;
     if (!res.body) {
         cleanup();
         throw new Error("FlexInference: streaming response has no body.");
@@ -331,9 +422,39 @@ async function* streamSSE(res, cleanup) {
     let exhausted = false;
     try {
         for (;;) {
-            const { done, value } = await reader.read();
+            // Idle timeout: reset on every chunk. Fires only when the stream goes silent for
+            // idleMs, aborting the underlying fetch so a hung upstream can't pin the stream. A
+            // healthy stream (chunks flowing) keeps resetting it, so a long generation is fine.
+            const idleTimer = setTimeout(() => {
+                controller.abort(IDLE_TIMEOUT);
+            }, idleMs);
+            let chunk;
+            try {
+                chunk = await reader.read();
+            }
+            catch (err) {
+                if (controller.signal.reason === IDLE_TIMEOUT) {
+                    throw new Error(`FlexInference: stream stalled - no data for ${String(idleMs)}ms (idle timeout).`);
+                }
+                throw err;
+            }
+            finally {
+                clearTimeout(idleTimer);
+            }
+            const { done, value } = chunk;
             if (done) {
+                // The stream ended. Flush any bytes the decoder held back for an incomplete
+                // multibyte sequence, then emit a final frame that arrived without its trailing
+                // blank line so the last piece of the answer is not dropped. See T3.
                 exhausted = true;
+                buffer += decoder.decode();
+                const frame = buffer;
+                buffer = "";
+                if (frame.length > 0) {
+                    const parsed = parseSSEFrame(frame);
+                    if (parsed !== null && parsed.kind === "value")
+                        yield parsed.value;
+                }
                 break;
             }
             buffer += decoder.decode(value, { stream: true }).replace(/\r/g, "");
@@ -341,19 +462,21 @@ async function* streamSSE(res, cleanup) {
             while ((sep = buffer.indexOf("\n\n")) !== -1) {
                 const frame = buffer.slice(0, sep);
                 buffer = buffer.slice(sep + 2);
-                const dataLines = [];
-                for (const line of frame.split("\n")) {
-                    if (line.startsWith("data:"))
-                        dataLines.push(line.slice(5).replace(/^ /, ""));
-                }
-                if (dataLines.length === 0)
+                const parsed = parseSSEFrame(frame);
+                if (parsed === null)
                     continue;
-                const data = dataLines.join("\n");
-                if (data === "[DONE]") {
+                if (parsed.kind === "done") {
                     exhausted = true;
                     return;
                 }
-                yield JSON.parse(data);
+                yield parsed.value;
+            }
+            // With every complete frame drained, whatever remains is a single unterminated
+            // frame. If it grows past the cap the stream is never sending a separator; error
+            // clearly instead of buffering without bound. See T2.
+            if (buffer.length > MAX_SSE_BUFFER_CHARS) {
+                throw new Error(`FlexInference: stream buffer exceeded ${String(MAX_SSE_BUFFER_CHARS)} characters ` +
+                    "without a frame boundary; aborting to avoid unbounded memory.");
             }
         }
     }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "flexinference",
-  "version": "1.4.1",
+  "version": "1.5.1",
   "description": "Official TypeScript SDK for FlexInference - a deadline-aware, OpenAI-compatible inference router.",
   "license": "MIT",
   "author": "Aditya Perswal <adityaperswal@gmail.com>",
@@ -34,7 +34,7 @@
     "build": "tsc -p tsconfig.json",
     "typecheck": "tsc -p tsconfig.json --noEmit",
     "test:types": "tsc -p tsconfig.test.json --noEmit",
-    "test": "node test/smoke.mjs && npm run test:types",
+    "test": "node test/smoke.mjs && node test/fixes.test.mjs && npm run test:types",
     "lint": "eslint .",
     "lint:fix": "eslint . --fix",
     "format": "prettier --write .",