flexinference 1.4.0 → 1.5.0

package/CHANGELOG.md

@@ -1,5 +1,27 @@
 # Changelog
 
+## 1.5.0
+
+Reworks request timeouts so a healthy stream is never cut off yet a hung request never hangs
+forever. Streaming requests now have **no total cap** -- they run as long as tokens keep
+arriving -- and are bounded instead by two new options: `firstByteTimeout` (how long to wait
+for the first response, default `60000` ms) and `idleTimeout` (max silence between chunks
+before the stream is treated as hung, default `60000` ms). Non-streaming requests keep the
+total `timeout` (default `600000` ms). The first-byte wait is **auto-raised for a flex
+`start_within`**, fixing flex requests whose deadline exceeded the old hardcoded 10s connect
+cap and aborted before the router replied. All three timeouts are configurable on the client
+and per request. Behavior change, no API breaks: `timeout` no longer caps a stream, and the
+old 10s first-response cap is now 60s (auto-sized for flex).
+
+Also teaches `FlexInferenceError` to parse the router's per-surface error shapes: the router now
+returns errors in the dialect of the endpoint you called (OpenAI on responses/chat, Anthropic on
+`messages`, Google on `interactions`), and the error class reads all three (`message` always,
+`type` from the OpenAI/Anthropic `type` or Gemini `status`).
+
+## 1.4.1
+
+Rewrites the README in plain language from the copy audit. It explains what `start_within` does, how the cheaper flex tier runs first and falls back up to your standard tier when it cannot start in time, and how you only pay a share of what a flex request saves you. No code or API changes.
+
 ## 1.4.0
 
 Surfaces the router's **`doc_url`** on `FlexInferenceError` as a new `docUrl` field: a deep

package/README.md

@@ -1,6 +1,6 @@
 # FlexInference (TypeScript)
 
-The official TypeScript SDK for [FlexInference](https://flexinference.com) - a deadline-aware inference router across **OpenAI, Google Gemini, and Anthropic**. Send the OpenAI requests you already send, bring your own provider key, and set one required field - `start_within` - to trade latency for cost. Four caller formats are supported: `responses`, `chat.completions`, `interactions` (Gemini shape), and `messages` (Anthropic shape) - any of them reaches any provider.
+The official TypeScript SDK for [FlexInference](https://flexinference.com), an inference router that works with OpenAI, Google Gemini, and Anthropic. Send the OpenAI requests you already send and bring your own provider key. Set one required field, `start_within`, to tell us how long you will wait, and we trade a little latency for a smaller bill. You can call it four ways. Use `responses`, `chat.completions`, `interactions` (the Gemini shape), or `messages` (the Anthropic shape). Any of them reaches any provider.
 
 ```bash
 npm install flexinference
@@ -22,21 +22,23 @@ const res = await client.responses.create({
 console.log(outputText(res));
 ```
 
-Responses come back as the **raw OpenAI JSON** (we never reshape the body), so there is no `output_text` field on the wire - that is computed by OpenAI's own SDKs. `outputText(res)` pulls the assistant's text out of either a response or a chat completion for you.
+Responses come back as the **raw OpenAI JSON** (we never reshape the body), so there is no `output_text` field on the wire. OpenAI's own SDKs compute that. `outputText(res)` pulls the assistant's text out of either a response or a chat completion for you.
 
-`start_within` is **required** on every request. It takes `"default"`, `"priority"`, `"auto"`, or a duration `"HHh-MMm-SSs"` (5s-10m). The duration races OpenAI's flex tier on a flex-capable model and falls back to standard if it can't start in time; `"default"`, `"priority"`, and `"auto"` map to those OpenAI service tiers and proxy any model. It is typed, so a missing value or a value outside the allowed shapes is a **compile-time** error; a duration that fits the shape but is malformed or out of range is caught at **runtime**, the same check plain-JavaScript callers get. See the [docs](https://flexinference.mintlify.app/deadline-routing).
+`start_within` is required on every request. It takes `"default"`, `"priority"`, `"auto"`, or a duration in the form `"HHh-MMm-SSs"` from 5 seconds to 10 minutes. A duration tells us how long you will wait for the request to start. We try the cheaper flex tier first and give it that long to begin. If flex cannot start in time, we move your request up to your standard tier so it still finishes. The words `"default"`, `"priority"`, and `"auto"` map to those OpenAI service tiers and proxy any model. Types catch most mistakes for you. A missing value, or a value that does not fit the allowed shapes, fails at compile time. A duration that fits the shape but is malformed or out of range fails at runtime, the same check plain-JavaScript callers get. See the [docs](https://flexinference.mintlify.app/deadline-routing) for the full rules.
+
+This runs the opposite way from the usual fallback. Most setups start on the strong model and drop to a cheaper one when they need to. We start on the cheaper flex tier and escalate up to your standard model if flex cannot start in time, so a slow flex tier never costs you the answer.
 
 ## Providers (OpenAI, Gemini, and Anthropic)
 
-FlexInference routes to **OpenAI**, **Google Gemini**, and **Anthropic**. Send the same OpenAI-shaped request and pass whichever model id you want - `gpt-5.5`, `o4-mini`, `gemini-3.5-flash`, `claude-opus-4-8`, and so on. We translate Gemini and Anthropic to and from the OpenAI shape, so your code is identical for all three.
+FlexInference routes to **OpenAI**, **Google Gemini**, and **Anthropic**. Send the same OpenAI-shaped request and pass whichever model id you want. Use `gpt-5.5`, `o4-mini`, `gemini-3.5-flash`, `claude-opus-4-8`, and so on. We translate Gemini and Anthropic to and from the OpenAI shape, so your code is identical for all three.
 
-- **OpenAI:** `default` (standard tier), `priority`, `auto`, and the flex race (a duration) on flex-capable models.
-- **Gemini:** `default` maps to Gemini's **standard** tier, plus `priority` and the flex race on the Gemini flex models (`gemini-3.5-flash`, `gemini-3.1-flash-lite`, `gemini-3.1-pro-preview`, `gemini-3-flash-preview`, `gemini-2.5-pro`, `gemini-2.5-flash`, `gemini-2.5-flash-lite`). Gemini has no `auto` tier, so `start_within: "auto"` on a Gemini model returns `400`.
-- **Anthropic (Claude):** proxy-only. `default`, `priority`, and `auto` work; there is **no flex race**, so a duration `start_within` on a `claude-*` model returns `400 flex_unsupported_for_anthropic`. Anthropic requires a token cap, so set `max_output_tokens` (`max_completion_tokens` on Chat, `max_tokens` on Messages) or you get `400 missing_max_tokens`. You keep the unified API and tier control, and draw down your own Anthropic credits.
+- **OpenAI:** `default` (the standard tier), `priority`, `auto`, and flex on flex-capable models. You reach flex by passing a duration.
+- **Gemini:** `default` maps to Gemini's standard tier. You also get `priority` and flex on the Gemini flex models (`gemini-3.5-flash`, `gemini-3.1-flash-lite`, `gemini-3.1-pro-preview`, `gemini-3-flash-preview`, `gemini-2.5-pro`, `gemini-2.5-flash`, `gemini-2.5-flash-lite`). Gemini has no `auto` tier, so `start_within: "auto"` on a Gemini model returns `400`.
+- **Anthropic (Claude):** proxy-only. `default`, `priority`, and `auto` work; there is **no flex tier**, so a duration `start_within` on a `claude-*` model returns `400 flex_unsupported_for_anthropic`. Anthropic requires a token cap, so set `max_output_tokens` (`max_completion_tokens` on Chat, `max_tokens` on Messages) or you get `400 missing_max_tokens`. You keep the unified API and tier control, and draw down your own Anthropic credits.
 
 Add the provider key you'll use (OpenAI, Gemini, and/or Anthropic) in the [dashboard](https://www.flexinference.com/dashboard). Text, streaming, structured outputs, function calling, image input, and web search work across providers (send a Responses `web_search` tool; we map it to Gemini's `google_search`).
 
-Don't send `service_tier` - the router controls the tier from `start_within` and rejects a caller-supplied `service_tier` with `400 service_tier_not_allowed`.
+Do not send `service_tier`. The router sets the tier from `start_within`, and it rejects a caller-supplied `service_tier` with `400 service_tier_not_allowed`.
 
 ## Streaming
 
@@ -100,19 +102,29 @@ console.log(messageText(res));
 
 ## Timeouts and cancellation
 
-Every client has a built-in timeout: **10 seconds to connect** and **600 seconds (10 min) total**, matching the router's budget, so a request can never hang forever. Override the total with the `timeout` option (raise it for very long streams):
+Streaming and non-streaming requests are timed differently, so a long generation is never cut off yet a hung request never hangs forever:
+
+- **Non-streaming** requests have a total wall-clock budget: `timeout`, default **600000 ms (10 min)**, from send through the full body.
+- **Streaming** requests have **no total cap** (a healthy stream runs as long as tokens keep arriving). They are bounded by two clocks: `firstByteTimeout` (wait for the first response, default **60000 ms**) and `idleTimeout` (max silence between chunks before the stream is treated as hung, default **60000 ms**).
+
+The first-byte wait is **auto-raised for a flex `start_within`** (the router withholds response headers until the flex race resolves), so a long deadline just works. Raise `firstByteTimeout` yourself only for very large non-flex contexts whose first token is slow.
 
 ```ts
-const client = new FlexInference({ apiKey: "flex_live_...", timeout: 120_000 });
+const client = new FlexInference({
+  apiKey: "flex_live_...",
+  timeout: 120_000, // non-streaming total budget
+  firstByteTimeout: 90_000, // wait for the first response
+  idleTimeout: 30_000, // max silence between streamed chunks
+});
 ```
 
-Every call also accepts an `AbortSignal` as a second argument, so you can cancel a request yourself. Cancelling a stream stops it mid-flight:
+Any of these can be overridden per request, alongside an `AbortSignal` to cancel yourself. Cancelling a stream stops it mid-flight:
 
 ```ts
 const controller = new AbortController();
 const res = await client.responses.create(
   { model: "gpt-5.5", input: "Write a haiku about cheap GPUs.", start_within: "priority" },
-  { signal: controller.signal },
+  { signal: controller.signal, idleTimeout: 15_000 },
 );
 // controller.abort() from elsewhere to cancel
 console.log(outputText(res));
@@ -120,7 +132,7 @@ console.log(outputText(res));
 
 ## Errors
 
-Non-2xx responses throw `FlexInferenceError`, carrying the OpenAI-shaped `status`, `type`, `code`, and `param`:
+Non-2xx responses throw `FlexInferenceError`, carrying `status`, `type`, `code`, and `param`. The router shapes error bodies to match the endpoint you called (OpenAI on `responses`/`chat`, Anthropic on `messages`, Google on `interactions`) so the SDK you would use for that surface parses them; `FlexInferenceError` reads all three. `message` and `status` are always set; `code`, `param`, and `docUrl` are populated on the OpenAI surface.
 
 ```ts
 import { FlexInferenceError } from "flexinference";
@@ -136,10 +148,14 @@ try {
 }
 ```
 
+Every FlexInference error tells you four things. It says what went wrong, why it went wrong, how to fix it, and it shows an example of a request that works. The `message` reads like a note from a person, so an agent can act on it instead of guessing. Provider errors are reshaped into the same surface envelope (and normalized into `FlexInferenceError`), so you get one consistent error type no matter which model ran. For instance, a duration on a `claude-*` model returns `400 flex_unsupported_for_anthropic` with a message that tells you to drop the duration or switch to `default`, `priority`, or `auto`.
+
 ## Billing / 402
 
-If your account's billing is past due, the router pauses **billable flex** and returns
-`402 Payment Required` on those requests; free routing keeps working. The SDK throws a
+If your account's billing is past due, the router pauses paid flex requests and returns
+`402 Payment Required` on them. Free routing keeps working, so your standard-tier calls
+still go through. Flex is the only part you ever pay for, and you only pay it when it
+saves you money. The SDK throws a
 typed `PaymentRequiredError` (a subclass of `FlexInferenceError`) for HTTP 402, so you
 can catch it on its own and prompt the user to update payment while letting other errors
 propagate:
@@ -163,12 +179,16 @@ Because `PaymentRequiredError extends FlexInferenceError`, existing
 
 ## Configuration
 
-| Option    | Default                            | Description                                      |
-| --------- | ---------------------------------- | ------------------------------------------------ |
-| `apiKey`  | -                                  | Your `flex_live_` key (required).                |
-| `baseURL` | `https://api.flexinference.com/v1` | Override the router endpoint.                    |
-| `fetch`   | global `fetch`                     | Provide a custom fetch implementation.           |
-| `timeout` | `600000`                           | Total request budget in ms (connect capped 10s). |
+| Option             | Default                            | Description                                                                     |
+| ------------------ | ---------------------------------- | ------------------------------------------------------------------------------- |
+| `apiKey`           | -                                  | Your `flex_live_` key (required).                                               |
+| `baseURL`          | `https://api.flexinference.com/v1` | Override the router endpoint.                                                   |
+| `fetch`            | global `fetch`                     | Provide a custom fetch implementation.                                          |
+| `timeout`          | `600000`                           | Non-streaming total budget in ms (streaming has no total).                      |
+| `firstByteTimeout` | `60000`                            | Wait for the first response in ms; auto-raised for a flex `start_within`.       |
+| `idleTimeout`      | `60000`                            | Max silence between streamed chunks in ms before the stream is treated as hung. |
+
+All three timeouts can also be passed per request (with `signal`) in the second argument.
 
 ## License
 

package/dist/index.d.ts

@@ -70,7 +70,7 @@ export declare class FlexInferenceError extends Error {
     readonly param: string | null | undefined;
     /** Deep link into the docs for this error's code, when the router supplied one. */
     readonly docUrl: string | null | undefined;
-    constructor(status: number, body: Partial<FlexErrorBody> | undefined, fallback: string);
+    constructor(status: number, body: unknown, fallback: string);
 }
 /**
  * Thrown on HTTP 402: billing is past due, so billable flex is paused. Free routing
@@ -86,19 +86,41 @@ export interface ClientOptions {
     /** Override the global fetch (e.g. for tests or non-standard runtimes). */
     fetch?: typeof fetch;
     /**
-     * Total request budget in ms (connect through the full response/stream). Defaults
-     * to 600000 (10 min). Connect alone is capped at 10s. Raise it for very long streams.
+     * Non-streaming wall-clock budget in ms (from send through the full body). Defaults to
+     * 600000 (10 min). Streaming requests ignore this: they are governed by `firstByteTimeout`
+     * and `idleTimeout` and have no total cap, so a healthy long stream is never cut off.
      */
     timeout?: number;
+    /**
+     * How long to wait in ms for the first response (headers) before the router has sent
+     * anything. Defaults to 60000. For a flex `start_within` it is auto-raised to cover the
+     * deadline (the router withholds headers during the race), so you rarely set this; raise
+     * it for very large non-flex contexts whose first token is slow.
+     */
+    firstByteTimeout?: number;
+    /**
+     * Max silence in ms between streamed chunks before the stream is treated as hung and
+     * aborted. Defaults to 60000. Streaming requests only.
+     */
+    idleTimeout?: number;
 }
 export interface RequestOptions {
     signal?: AbortSignal;
+    /** Override the client `firstByteTimeout` for this request (ms). */
+    firstByteTimeout?: number;
+    /** Override the client `idleTimeout` for this streaming request (ms). */
+    idleTimeout?: number;
+    /** Override the client `timeout` (non-streaming total budget) for this request (ms). */
+    timeout?: number;
 }
-interface PostResult {
+interface OpenResult {
     res: Response;
+    controller: AbortController;
     cleanup: () => void;
+    idleMs: number;
+    totalMs: number;
 }
-type Post = (path: string, body: unknown, signal: AbortSignal | undefined) => Promise<PostResult>;
+type Send = (path: string, body: unknown, options: RequestOptions | undefined) => Promise<OpenResult>;
 export declare class FlexInference {
     readonly responses: Responses;
     readonly chat: Chat;
@@ -107,13 +129,17 @@ export declare class FlexInference {
     private readonly apiKey;
     private readonly baseURL;
     private readonly fetchImpl;
-    private readonly timeoutMs;
+    private readonly firstByteMs;
+    private readonly idleMs;
+    private readonly totalMs;
     constructor(options: ClientOptions);
-    private post;
+    private timeoutsFor;
+    private send;
+    private open;
 }
 declare class Responses {
-    private readonly post;
-    constructor(post: Post);
+    private readonly send;
+    constructor(send: Send);
     create(body: ResponseCreateParams & {
         stream?: false | null;
     }, options?: RequestOptions): Promise<ResponseObject>;
@@ -123,11 +149,11 @@ declare class Responses {
 }
 declare class Chat {
     readonly completions: ChatCompletions;
-    constructor(post: Post);
+    constructor(send: Send);
 }
 declare class ChatCompletions {
-    private readonly post;
-    constructor(post: Post);
+    private readonly send;
+    constructor(send: Send);
     create(body: ChatCompletionCreateParams & {
         stream?: false | null;
     }, options?: RequestOptions): Promise<ChatCompletion>;
@@ -136,8 +162,8 @@ declare class ChatCompletions {
     }, options?: RequestOptions): Promise<AsyncIterable<ChatCompletionChunk>>;
 }
 declare class Interactions {
-    private readonly post;
-    constructor(post: Post);
+    private readonly send;
+    constructor(send: Send);
     create(body: InteractionCreateParams & {
         stream?: false | null;
     }, options?: RequestOptions): Promise<InteractionObject>;
@@ -146,8 +172,8 @@ declare class Interactions {
     }, options?: RequestOptions): Promise<AsyncIterable<InteractionStreamEvent>>;
 }
 declare class Messages {
-    private readonly post;
-    constructor(post: Post);
+    private readonly send;
+    constructor(send: Send);
     create(body: MessageCreateParams & {
         stream?: false | null;
     }, options?: RequestOptions): Promise<MessageObject>;

package/dist/index.js

@@ -92,15 +92,26 @@ export class FlexInferenceError extends Error {
     param;
     /** Deep link into the docs for this error's code, when the router supplied one. */
     docUrl;
+    // The router shapes errors by the endpoint you called, so `body` may be the OpenAI
+    // envelope (responses/chat), the Anthropic one (messages: `{type:"error", error:{...}}`),
+    // or the Google one (interactions: `{error:{code,message,status}}`). All three nest the
+    // detail under `error`, so this parses whichever arrived. `message` is always populated;
+    // `type` reads OpenAI/Anthropic `error.type` or falls back to Gemini's `error.status`;
+    // `code`/`param`/`doc_url` are OpenAI-only and undefined on the other shapes.
     constructor(status, body, fallback) {
-        const err = body?.error;
-        super(err?.message ?? fallback);
+        const err = isRecord(body) && isRecord(body.error) ? body.error : undefined;
+        super(typeof err?.message === "string" ? err.message : fallback);
         this.name = "FlexInferenceError";
         this.status = status;
-        this.type = err?.type;
-        this.code = err?.code;
-        this.param = err?.param;
-        this.docUrl = err?.doc_url;
+        this.type =
+            typeof err?.type === "string"
+                ? err.type
+                : typeof err?.status === "string"
+                    ? err.status
+                    : undefined;
+        this.code = typeof err?.code === "string" ? err.code : undefined;
+        this.param = typeof err?.param === "string" ? err.param : undefined;
+        this.docUrl = typeof err?.doc_url === "string" ? err.doc_url : undefined;
     }
 }
 /**
@@ -111,8 +122,16 @@ export class FlexInferenceError extends Error {
 export class PaymentRequiredError extends FlexInferenceError {
 }
 const DEFAULT_BASE_URL = "https://api.flexinference.com/v1";
-const DEFAULT_TIMEOUT_MS = 600_000; // 10 min total budget, matching the router's generosity
-const CONNECT_TIMEOUT_MS = 10_000; // fail fast if the router is unreachable
+// A streaming request is governed by two clocks and no total: how long we wait for the
+// first response (headers), and the maximum silence between chunks once it is flowing. A
+// healthy long generation is allowed to run as long as tokens keep arriving. A
+// non-streaming request has no stream to idle-watch, so it keeps a total wall-clock budget
+// instead.
+const DEFAULT_FIRST_BYTE_MS = 60_000; // wait for the first response; auto-raised for a flex start_within
+const FLEX_FIRST_BYTE_MARGIN_MS = 30_000; // headroom over the flex deadline for the fallback-to-standard path
+const DEFAULT_IDLE_MS = 60_000; // max silence between streamed chunks before the stream is treated as hung
+const DEFAULT_TOTAL_MS = 600_000; // non-streaming wall-clock budget (streaming has none)
+const ERROR_BODY_READ_MS = 30_000; // bound the read of a non-2xx error body
 const MIN_DEADLINE_SECONDS = 5;
 const MAX_DEADLINE_SECONDS = 600;
 const SECONDS_PER_HOUR = 3600;
@@ -145,6 +164,42 @@ function assertStartWithin(body) {
         throw new Error("FlexInference: `start_within` duration must be between 5s and 10m.");
     }
 }
+// Distinct abort reasons so an aborted fetch/read can be told apart: a timeout we fired
+// (turned into a clear message) versus the caller's own AbortSignal (propagated as-is).
+const FIRST_BYTE_TIMEOUT = Symbol("first-byte-timeout");
+const IDLE_TIMEOUT = Symbol("idle-timeout");
+/**
+ * Seconds for a flex-duration `start_within`, or null for a tier value ("default" etc.),
+ * a missing field, or a malformed one. Used to auto-size the first-byte wait: the router
+ * withholds response headers until the flex race commits or falls back, so the first-byte
+ * clock must cover the whole deadline plus the fallback path.
+ */
+function flexDeadlineSeconds(body) {
+    const value = typeof body === "object" && body !== null
+        ? body["start_within"]
+        : undefined;
+    if (typeof value !== "string")
+        return null;
+    const match = DURATION_RE.exec(value);
+    if (match === null)
+        return null;
+    return (Number(match[1]) * SECONDS_PER_HOUR + Number(match[2]) * SECONDS_PER_MINUTE + Number(match[3]));
+}
+// Read a non-streamed JSON body under the total wall-clock budget, then release the timer
+// and the abort listener. Non-streaming only: a stream is drained by streamSSE instead.
+async function readJson(open) {
+    const { res, controller, cleanup, totalMs } = open;
+    const totalTimer = setTimeout(() => {
+        controller.abort();
+    }, totalMs);
+    try {
+        return (await res.json());
+    }
+    finally {
+        clearTimeout(totalTimer);
+        cleanup();
+    }
+}
 export class FlexInference {
     responses;
     chat;
@@ -153,28 +208,54 @@ export class FlexInference {
     apiKey;
     baseURL;
     fetchImpl;
-    timeoutMs;
+    firstByteMs;
+    idleMs;
+    totalMs;
     constructor(options) {
         if (!options.apiKey)
             throw new Error("FlexInference: `apiKey` is required.");
         this.apiKey = options.apiKey;
         this.baseURL = (options.baseURL ?? DEFAULT_BASE_URL).replace(/\/+$/, "");
-        this.timeoutMs = options.timeout ?? DEFAULT_TIMEOUT_MS;
+        this.firstByteMs = options.firstByteTimeout ?? DEFAULT_FIRST_BYTE_MS;
+        this.idleMs = options.idleTimeout ?? DEFAULT_IDLE_MS;
+        this.totalMs = options.timeout ?? DEFAULT_TOTAL_MS;
         const f = options.fetch ?? globalThis.fetch;
         if (typeof f !== "function") {
             throw new Error("FlexInference: no global fetch found; pass `fetch` in options.");
         }
         this.fetchImpl = f;
-        const post = (path, body, signal) => this.post(path, body, signal);
-        this.responses = new Responses(post);
-        this.chat = new Chat(post);
-        this.interactions = new Interactions(post);
-        this.messages = new Messages(post);
+        const send = (path, body, options) => this.send(path, body, options);
+        this.responses = new Responses(send);
+        this.chat = new Chat(send);
+        this.interactions = new Interactions(send);
+        this.messages = new Messages(send);
+    }
+    // Resolve the three per-request clocks. A per-request override always wins. Otherwise the
+    // first-byte wait is auto-sized for a flex `start_within`: it can never be shorter than
+    // the deadline plus the fallback margin (headers are withheld for the whole race), but a
+    // larger client default still wins over a short deadline.
+    timeoutsFor(body, options) {
+        const deadline = flexDeadlineSeconds(body);
+        const autoFirstByte = deadline === null
+            ? this.firstByteMs
+            : Math.max(this.firstByteMs, deadline * 1000 + FLEX_FIRST_BYTE_MARGIN_MS);
+        return {
+            firstByteMs: options?.firstByteTimeout ?? autoFirstByte,
+            idleMs: options?.idleTimeout ?? this.idleMs,
+            totalMs: options?.timeout ?? this.totalMs,
+        };
+    }
+    async send(path, body, options) {
+        const { firstByteMs, idleMs, totalMs } = this.timeoutsFor(body, options);
+        const { res, controller, cleanup } = await this.open(path, body, options?.signal, firstByteMs);
+        return { res, controller, cleanup, idleMs, totalMs };
     }
-    async post(path, body, signal) {
-        // One AbortController fed by three sources: the caller's signal, a 10s connect
-        // timer (cleared once headers arrive), and the total budget (cleared by the
-        // consumer once the body/stream is done). No dangling timers after completion.
+    // Open the request and return once response headers arrive (or throw on an error status).
+    // One AbortController is fed by two sources here: the caller's signal, and a first-byte
+    // timer (cleared once headers arrive). The idle timer (streaming) or total timer
+    // (non-streaming) is armed by the caller against the returned controller. Distinct abort
+    // reasons let a fired timeout be told apart from a caller abort.
+    async open(path, body, signal, firstByteMs) {
         const controller = new AbortController();
         const onCallerAbort = () => {
             controller.abort(signal?.reason);
@@ -185,17 +266,12 @@ export class FlexInference {
             else
                 signal.addEventListener("abort", onCallerAbort, { once: true });
         }
-        const connectTimer = setTimeout(() => {
-            controller.abort();
-        }, CONNECT_TIMEOUT_MS);
-        const totalTimer = setTimeout(() => {
-            controller.abort();
-        }, this.timeoutMs);
         const cleanup = () => {
-            clearTimeout(connectTimer);
-            clearTimeout(totalTimer);
             signal?.removeEventListener("abort", onCallerAbort);
         };
+        const firstByteTimer = setTimeout(() => {
+            controller.abort(FIRST_BYTE_TIMEOUT);
+        }, firstByteMs);
         let res;
         try {
             res = await this.fetchImpl(`${this.baseURL}${path}`, {
@@ -210,24 +286,30 @@ export class FlexInference {
             });
         }
         catch (err) {
+            clearTimeout(firstByteTimer);
             cleanup();
-            if (controller.signal.aborted && signal?.aborted !== true) {
-                throw new Error(`FlexInference: request to the router timed out (connect ${String(CONNECT_TIMEOUT_MS)}ms / total ${String(this.timeoutMs)}ms).`);
+            if (controller.signal.reason === FIRST_BYTE_TIMEOUT) {
+                throw new Error(`FlexInference: no response from the router within ${String(firstByteMs)}ms ` +
+                    "(time to first byte). Raise `firstByteTimeout` for very large contexts or long " +
+                    "`start_within` deadlines.");
             }
             throw err;
         }
-        clearTimeout(connectTimer); // headers arrived; only the total budget governs the body now
+        clearTimeout(firstByteTimer); // headers arrived
         if (!res.ok) {
-            // Read the error body under the same timeout, then clear the timers. Clearing
-            // before the read would leave a stalled error body with no budget to abort it.
+            // Read the error body under a bounded budget, then release the abort listener.
+            const errTimer = setTimeout(() => {
+                controller.abort();
+            }, ERROR_BODY_READ_MS);
             let parsed;
             try {
-                parsed = (await res.json());
+                parsed = await res.json();
             }
             catch {
                 parsed = undefined;
             }
             finally {
+                clearTimeout(errTimer);
                 cleanup();
             }
             if (res.status === 402) {
@@ -235,92 +317,69 @@ export class FlexInference {
             }
             throw new FlexInferenceError(res.status, parsed, `HTTP ${String(res.status)} ${res.statusText}`);
         }
-        return { res, cleanup };
+        return { res, controller, cleanup };
     }
 }
 class Responses {
-    post;
-    constructor(post) {
-        this.post = post;
+    send;
+    constructor(send) {
+        this.send = send;
     }
     async create(body, options) {
         assertStartWithin(body);
-        const { res, cleanup } = await this.post("/responses", body, options?.signal);
-        if (body.stream) {
-            return streamSSE(res, cleanup);
-        }
-        try {
-            return (await res.json());
-        }
-        finally {
-            cleanup();
-        }
+        const open = await this.send("/responses", body, options);
+        if (body.stream)
+            return streamSSE(open);
+        return readJson(open);
     }
 }
 class Chat {
     completions;
-    constructor(post) {
-        this.completions = new ChatCompletions(post);
+    constructor(send) {
+        this.completions = new ChatCompletions(send);
     }
 }
 class ChatCompletions {
-    post;
-    constructor(post) {
-        this.post = post;
+    send;
+    constructor(send) {
+        this.send = send;
     }
     async create(body, options) {
         assertStartWithin(body);
-        const { res, cleanup } = await this.post("/chat/completions", body, options?.signal);
-        if (body.stream) {
-            return streamSSE(res, cleanup);
-        }
-        try {
-            return (await res.json());
-        }
-        finally {
-            cleanup();
-        }
+        const open = await this.send("/chat/completions", body, options);
+        if (body.stream)
+            return streamSSE(open);
+        return readJson(open);
     }
 }
 class Interactions {
-    post;
-    constructor(post) {
-        this.post = post;
+    send;
+    constructor(send) {
+        this.send = send;
     }
     async create(body, options) {
         assertStartWithin(body);
-        const { res, cleanup } = await this.post("/interactions", body, options?.signal);
-        if (body.stream) {
-            return streamSSE(res, cleanup);
-        }
-        try {
-            return (await res.json());
-        }
-        finally {
-            cleanup();
-        }
+        const open = await this.send("/interactions", body, options);
+        if (body.stream)
+            return streamSSE(open);
+        return readJson(open);
     }
 }
 class Messages {
-    post;
-    constructor(post) {
-        this.post = post;
+    send;
+    constructor(send) {
+        this.send = send;
     }
     async create(body, options) {
         assertStartWithin(body);
-        const { res, cleanup } = await this.post("/messages", body, options?.signal);
-        if (body.stream) {
-            return streamSSE(res, cleanup);
-        }
-        try {
-            return (await res.json());
-        }
-        finally {
-            cleanup();
-        }
+        const open = await this.send("/messages", body, options);
+        if (body.stream)
+            return streamSSE(open);
+        return readJson(open);
     }
 }
-async function* streamSSE(res, cleanup) {
+async function* streamSSE(open) {
+    const { res, controller, idleMs, cleanup } = open;
     if (!res.body) {
         cleanup();
         throw new Error("FlexInference: streaming response has no body.");
@@ -331,7 +390,26 @@ async function* streamSSE(res, cleanup) {
     let exhausted = false;
     try {
         for (;;) {
-            const { done, value } = await reader.read();
+            // Idle timeout: reset on every chunk. Fires only when the stream goes silent for
+            // idleMs, aborting the underlying fetch so a hung upstream can't pin the stream. A
+            // healthy stream (chunks flowing) keeps resetting it, so a long generation is fine.
+            const idleTimer = setTimeout(() => {
+                controller.abort(IDLE_TIMEOUT);
+            }, idleMs);
+            let chunk;
+            try {
+                chunk = await reader.read();
+            }
+            catch (err) {
+                if (controller.signal.reason === IDLE_TIMEOUT) {
+                    throw new Error(`FlexInference: stream stalled - no data for ${String(idleMs)}ms (idle timeout).`);
+                }
+                throw err;
+            }
+            finally {
+                clearTimeout(idleTimer);
+            }
+            const { done, value } = chunk;
             if (done) {
                 exhausted = true;
                 break;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "flexinference",
-  "version": "1.4.0",
+  "version": "1.5.0",
   "description": "Official TypeScript SDK for FlexInference - a deadline-aware, OpenAI-compatible inference router.",
   "license": "MIT",
   "author": "Aditya Perswal <adityaperswal@gmail.com>",