flexinference 0.1.0 → 1.0.1

package/CHANGELOG.md

@@ -0,0 +1,59 @@
+# Changelog
+
+## 1.0.1
+
+Gemini now supports **image input** and **web search** (send a Responses `web_search`
+tool; the router maps it to Gemini's `google_search`), in addition to text, streaming,
+structured outputs, and function calling. No API or type changes - translation is
+handled server-side, so existing code keeps working.
+
+## 1.0.0
+
+The stable v1 of the FlexInference TypeScript SDK. This release makes `start_within`
+required and aligns its values with OpenAI's `service_tier` vocabulary.
+
+### Breaking changes
+
+- **`start_within` is now required on every request.** It is typed as
+  `StartWithin = "default" | "priority" | "auto" | \`${number}h-${number}m-${number}s\``,
+so omitting it (or passing a value outside those shapes) is a **compile-time** error.
+A duration that fits the shape but is malformed or out of range is caught at **runtime**;
+plain-JS callers get every check at **runtime** (a thrown `Error` before any network call).
+- **Accepted values changed.** `start_within` now takes `"default"`, `"priority"`,
+  `"auto"`, or a duration `"HHh-MMm-SSs"` (5s-10m). The old `"standard"` value is
+  removed; use `"default"` (it maps to the same OpenAI service tier).
+- **Any model proxies for the non-flex values.** `"default"`, `"priority"`, and
+  `"auto"` proxy any model to OpenAI. Only the duration (flex race) requires a
+  flex-capable model; a duration on a non-flex-capable model returns the router's new
+  `model_not_flex_capable` error (the old `unsupported_model` is retired).
+
+### Added
+
+- **Default request timeout.** Clients now bound every request: 10s to connect and
+  600s (10 min) total, matching the router. Override the total with the new `timeout`
+  client option, still composable with a per-call `AbortSignal`.
+- **`outputText(res)` helper.** Pulls the assistant's text out of a raw response
+  (Responses or chat completion) without hand-walking `output[].content[]`. The raw
+  wire body has no `output_text` field (that is computed by OpenAI's SDKs, not sent),
+  so this fills the gap. Read-only; nothing is reshaped.
+
+### Notes
+
+- An arbitrary `string` is not assignable to `StartWithin`; use `value as StartWithin`
+  for a dynamically-built value.
+- Validation is request-only: unknown fields pass straight through to the provider,
+  and responses are never validated or reshaped.
+
+### Migration
+
+```diff
+- await client.responses.create({ model: "gpt-5.5", input: "hi" });
++ await client.responses.create({ model: "gpt-5.5", input: "hi", start_within: "default" });
+
+- start_within: "standard"
++ start_within: "default"
+```
+
+## 0.1.0
+
+- Initial release.

package/README.md

@@ -1,6 +1,6 @@
 # FlexInference (TypeScript)
 
-The official TypeScript SDK for [FlexInference](https://flexinference.com) - a deadline-aware, OpenAI-compatible inference router. Send the OpenAI requests you already send, bring your own OpenAI key, and add one field - `start_within` - to trade latency for cost.
+The official TypeScript SDK for [FlexInference](https://flexinference.com) - a deadline-aware, OpenAI-compatible inference router. Send the OpenAI requests you already send, bring your own OpenAI key, and set one required field - `start_within` - to trade latency for cost.
 
 ```bash
 npm install flexinference
@@ -9,7 +9,7 @@ npm install flexinference
 ## Quickstart
 
 ```ts
-import { FlexInference } from "flexinference";
+import { FlexInference, outputText } from "flexinference";
 
 const client = new FlexInference({ apiKey: "flex_live_..." });
 
@@ -19,10 +19,23 @@ const res = await client.responses.create({
   start_within: "00h-00m-30s", // typed - no cast needed
 });
 
-console.log(res.output_text);
+console.log(outputText(res));
 ```
 
-`start_within` takes `"priority"`, `"standard"`, or a duration `"HHh-MMm-SSs"` (5s-10m) that races OpenAI's flex tier and falls back to standard if it can't start in time. See the [docs](https://flexinference.com/docs/deadline-routing).
+Responses come back as the **raw OpenAI JSON** (we never reshape the body), so there is no `output_text` field on the wire - that is computed by OpenAI's own SDKs. `outputText(res)` pulls the assistant's text out of either a response or a chat completion for you.
+
+`start_within` is **required** on every request. It takes `"default"`, `"priority"`, `"auto"`, or a duration `"HHh-MMm-SSs"` (5s-10m). The duration races OpenAI's flex tier on a flex-capable model and falls back to standard if it can't start in time; `"default"`, `"priority"`, and `"auto"` map to those OpenAI service tiers and proxy any model. It is typed, so a missing value or a value outside the allowed shapes is a **compile-time** error; a duration that fits the shape but is malformed or out of range is caught at **runtime**, the same check plain-JavaScript callers get. See the [docs](https://flexinference.mintlify.app/deadline-routing).
+
+## Providers (OpenAI and Gemini)
+
+FlexInference routes to **OpenAI** and **Google Gemini**. Send the same OpenAI-shaped request and pass whichever model id you want - `gpt-5.5`, `o4-mini`, `gemini-3.5-flash`, and so on. We run Gemini through its Interactions API and translate it back to the OpenAI shape, so your code is identical for both.
+
+- **OpenAI:** `default` (standard tier), `priority`, `auto`, and the flex race (a duration) on flex-capable models.
+- **Gemini:** `default` maps to Gemini's **standard** tier, plus `priority` and the flex race on the Gemini flex models (`gemini-3.5-flash`, `gemini-3.1-flash-lite`, `gemini-3.1-pro-preview`, `gemini-3-flash-preview`, `gemini-2.5-pro`, `gemini-2.5-flash`, `gemini-2.5-flash-lite`). Gemini has no `auto` tier, so `start_within: "auto"` on a Gemini model returns `400`.
+
+Add the provider key you'll use (OpenAI and/or Gemini) in the [dashboard](https://www.flexinference.com/dashboard). Text, streaming, structured outputs, function calling, image input, and web search work on both providers (send a Responses `web_search` tool; we map it to Gemini's `google_search`).
+
+Don't send `service_tier` - the router controls the tier from `start_within` and rejects a caller-supplied `service_tier` with `400 service_tier_not_allowed`.
 
 ## Streaming
 
@@ -47,27 +60,28 @@ The Chat Completions endpoint works identically:
 const res = await client.chat.completions.create({
   model: "gpt-5.5",
   messages: [{ role: "user", content: "Hello!" }],
-  start_within: "standard",
+  start_within: "default",
 });
 ```
 
-## Cancellation
+## Timeouts and cancellation
 
-Every call accepts an `AbortSignal` as a second argument, so you can cancel a request (or wire up your own timeout). Cancelling a stream stops it mid-flight:
+Every client has a built-in timeout: **10 seconds to connect** and **600 seconds (10 min) total**, matching the router's budget, so a request can never hang forever. Override the total with the `timeout` option (raise it for very long streams):
 
 ```ts
-const controller = new AbortController();
-const timeout = setTimeout(() => controller.abort(), 5_000);
+const client = new FlexInference({ apiKey: "flex_live_...", timeout: 120_000 });
+```
 
-try {
-  const res = await client.responses.create(
-    { model: "gpt-5.5", input: "Write a haiku about cheap GPUs.", start_within: "priority" },
-    { signal: controller.signal },
-  );
-  console.log(res.output_text);
-} finally {
-  clearTimeout(timeout);
-}
+Every call also accepts an `AbortSignal` as a second argument, so you can cancel a request yourself. Cancelling a stream stops it mid-flight:
+
+```ts
+const controller = new AbortController();
+const res = await client.responses.create(
+  { model: "gpt-5.5", input: "Write a haiku about cheap GPUs.", start_within: "priority" },
+  { signal: controller.signal },
+);
+// controller.abort() from elsewhere to cancel
+console.log(outputText(res));
 ```
 
 ## Errors
@@ -90,11 +104,12 @@ try {
 
 ## Configuration
 
-| Option    | Default                            | Description                            |
-| --------- | ---------------------------------- | -------------------------------------- |
-| `apiKey`  | -                                  | Your `flex_live_` key (required).      |
-| `baseURL` | `https://api.flexinference.com/v1` | Override the router endpoint.          |
-| `fetch`   | global `fetch`                     | Provide a custom fetch implementation. |
+| Option    | Default                            | Description                                      |
+| --------- | ---------------------------------- | ------------------------------------------------ |
+| `apiKey`  | -                                  | Your `flex_live_` key (required).                |
+| `baseURL` | `https://api.flexinference.com/v1` | Override the router endpoint.                    |
+| `fetch`   | global `fetch`                     | Provide a custom fetch implementation.           |
+| `timeout` | `600000`                           | Total request budget in ms (connect capped 10s). |
 
 ## License
 

package/dist/index.d.ts

@@ -1,11 +1,32 @@
 import type { components } from "./types.js";
 type Schemas = components["schemas"];
-export type ResponseCreateParams = Schemas["CreateResponse"];
+/**
+ * The required FlexInference routing field. "default", "priority", and "auto" map to
+ * those OpenAI service tiers and proxy any model; a duration "HHh-MMm-SSs" (5s-10m)
+ * runs the flex race on a flex-capable model. An arbitrary string needs `as StartWithin`.
+ */
+export type StartWithin = "default" | "priority" | "auto" | `${number}h-${number}m-${number}s`;
+/** Make `start_within` a required, typed field on a generated request body. */
+type WithStartWithin<T> = Omit<T, "start_within"> & {
+    start_within: StartWithin;
+};
+export type ResponseCreateParams = WithStartWithin<Schemas["CreateResponse"]>;
 export type ResponseObject = Schemas["Response"];
 export type ResponseStreamEvent = Schemas["ResponseStreamEvent"];
-export type ChatCompletionCreateParams = Schemas["CreateChatCompletionRequest"];
+type RawChatRequest = Schemas["CreateChatCompletionRequest"];
+export type ChatCompletionCreateParams = WithStartWithin<Partial<RawChatRequest> & Required<Pick<RawChatRequest, "model" | "messages">>>;
 export type ChatCompletion = Schemas["CreateChatCompletionResponse"];
 export type ChatCompletionChunk = Schemas["CreateChatCompletionStreamResponse"];
+/**
+ * Concatenate the assistant's text from a non-streamed response. Works on a Responses
+ * object (walks `output[].content[]` for `output_text` parts) and on a chat completion
+ * (`choices[0].message.content`). Read-only: the raw body is never reshaped. Returns ""
+ * when there is no text (e.g. a tool-call-only or refusal response). The raw wire body
+ * has no `output_text` field - that is computed by OpenAI's own SDKs, not sent on the
+ * wire - so use this to pull the text out. Streamed responses are not objects; collect
+ * their `output_text.delta` events instead.
+ */
+export declare function outputText(response: ResponseObject | ChatCompletion): string;
 export interface FlexErrorBody {
     error: {
         message: string;
@@ -27,17 +48,27 @@ export interface ClientOptions {
     baseURL?: string;
     /** Override the global fetch (e.g. for tests or non-standard runtimes). */
     fetch?: typeof fetch;
+    /**
+     * Total request budget in ms (connect through the full response/stream). Defaults
+     * to 600000 (10 min). Connect alone is capped at 10s. Raise it for very long streams.
+     */
+    timeout?: number;
 }
 export interface RequestOptions {
     signal?: AbortSignal;
 }
-type Post = (path: string, body: unknown, signal: AbortSignal | undefined) => Promise<Response>;
+interface PostResult {
+    res: Response;
+    cleanup: () => void;
+}
+type Post = (path: string, body: unknown, signal: AbortSignal | undefined) => Promise<PostResult>;
 export declare class FlexInference {
     readonly responses: Responses;
     readonly chat: Chat;
     private readonly apiKey;
     private readonly baseURL;
     private readonly fetchImpl;
+    private readonly timeoutMs;
     constructor(options: ClientOptions);
     private post;
 }

package/dist/index.js

@@ -1,3 +1,41 @@
+function isRecord(value) {
+    return typeof value === "object" && value !== null;
+}
+/**
+ * Concatenate the assistant's text from a non-streamed response. Works on a Responses
+ * object (walks `output[].content[]` for `output_text` parts) and on a chat completion
+ * (`choices[0].message.content`). Read-only: the raw body is never reshaped. Returns ""
+ * when there is no text (e.g. a tool-call-only or refusal response). The raw wire body
+ * has no `output_text` field - that is computed by OpenAI's own SDKs, not sent on the
+ * wire - so use this to pull the text out. Streamed responses are not objects; collect
+ * their `output_text.delta` events instead.
+ */
+export function outputText(response) {
+    const obj = response;
+    if (!isRecord(obj))
+        return "";
+    const output = obj.output;
+    if (Array.isArray(output)) {
+        let text = "";
+        for (const item of output) {
+            if (!isRecord(item) || item.type !== "message" || !Array.isArray(item.content))
+                continue;
+            for (const part of item.content) {
+                if (isRecord(part) && part.type === "output_text" && typeof part.text === "string") {
+                    text += part.text;
+                }
+            }
+        }
+        return text;
+    }
+    const choices = obj.choices;
+    if (Array.isArray(choices) && isRecord(choices[0])) {
+        const message = choices[0].message;
+        if (isRecord(message) && typeof message.content === "string")
+            return message.content;
+    }
+    return "";
+}
 export class FlexInferenceError extends Error {
     status;
     type;
@@ -14,17 +52,51 @@ export class FlexInferenceError extends Error {
     }
 }
 const DEFAULT_BASE_URL = "https://api.flexinference.com/v1";
+const DEFAULT_TIMEOUT_MS = 600_000; // 10 min total budget, matching the router's generosity
+const CONNECT_TIMEOUT_MS = 10_000; // fail fast if the router is unreachable
+const MIN_DEADLINE_SECONDS = 5;
+const MAX_DEADLINE_SECONDS = 600;
+const SECONDS_PER_HOUR = 3600;
+const SECONDS_PER_MINUTE = 60;
+const DURATION_RE = /^(\d{2})h-(\d{2})m-(\d{2})s$/;
+const START_WITHIN_VALUES = '"default", "priority", "auto", or a duration "HHh-MMm-SSs"';
+// Runtime guard so plain-JS callers (who get no compile-time check) still fail
+// locally with a clear message instead of a provider round-trip. TypeScript callers
+// are already protected by the required StartWithin field above.
+function assertStartWithin(body) {
+    const value = typeof body === "object" && body !== null
+        ? body["start_within"]
+        : undefined;
+    if (value === undefined) {
+        throw new Error(`FlexInference: \`start_within\` is required. Set it to ${START_WITHIN_VALUES}.`);
+    }
+    if (typeof value !== "string") {
+        throw new Error(`FlexInference: \`start_within\` must be a string (${START_WITHIN_VALUES}).`);
+    }
+    if (value === "default" || value === "priority" || value === "auto")
+        return;
+    const match = DURATION_RE.exec(value);
+    if (match === null) {
+        throw new Error(`FlexInference: \`start_within\` "${value}" is not ${START_WITHIN_VALUES}.`);
+    }
+    const seconds = Number(match[1]) * SECONDS_PER_HOUR + Number(match[2]) * SECONDS_PER_MINUTE + Number(match[3]);
+    if (seconds < MIN_DEADLINE_SECONDS || seconds > MAX_DEADLINE_SECONDS) {
+        throw new Error("FlexInference: `start_within` duration must be between 5s and 10m.");
+    }
+}
 export class FlexInference {
     responses;
     chat;
     apiKey;
     baseURL;
     fetchImpl;
+    timeoutMs;
     constructor(options) {
         if (!options.apiKey)
             throw new Error("FlexInference: `apiKey` is required.");
         this.apiKey = options.apiKey;
         this.baseURL = (options.baseURL ?? DEFAULT_BASE_URL).replace(/\/+$/, "");
+        this.timeoutMs = options.timeout ?? DEFAULT_TIMEOUT_MS;
         const f = options.fetch ?? globalThis.fetch;
         if (typeof f !== "function") {
             throw new Error("FlexInference: no global fetch found; pass `fetch` in options.");
@@ -35,17 +107,54 @@ export class FlexInference {
         this.chat = new Chat(post);
     }
     async post(path, body, signal) {
-        const res = await this.fetchImpl(`${this.baseURL}${path}`, {
-            method: "POST",
-            headers: {
-                Authorization: `Bearer ${this.apiKey}`,
-                "Content-Type": "application/json",
-                Accept: "text/event-stream, application/json",
-            },
-            body: JSON.stringify(body),
-            signal,
-        });
+        // One AbortController fed by three sources: the caller's signal, a 10s connect
+        // timer (cleared once headers arrive), and the total budget (cleared by the
+        // consumer once the body/stream is done). No dangling timers after completion.
+        const controller = new AbortController();
+        const onCallerAbort = () => {
+            controller.abort(signal?.reason);
+        };
+        if (signal) {
+            if (signal.aborted)
+                controller.abort(signal.reason);
+            else
+                signal.addEventListener("abort", onCallerAbort, { once: true });
+        }
+        const connectTimer = setTimeout(() => {
+            controller.abort();
+        }, CONNECT_TIMEOUT_MS);
+        const totalTimer = setTimeout(() => {
+            controller.abort();
+        }, this.timeoutMs);
+        const cleanup = () => {
+            clearTimeout(connectTimer);
+            clearTimeout(totalTimer);
+            signal?.removeEventListener("abort", onCallerAbort);
+        };
+        let res;
+        try {
+            res = await this.fetchImpl(`${this.baseURL}${path}`, {
+                method: "POST",
+                headers: {
+                    Authorization: `Bearer ${this.apiKey}`,
+                    "Content-Type": "application/json",
+                    Accept: "text/event-stream, application/json",
+                },
+                body: JSON.stringify(body),
+                signal: controller.signal,
+            });
+        }
+        catch (err) {
+            cleanup();
+            if (controller.signal.aborted && signal?.aborted !== true) {
+                throw new Error(`FlexInference: request to the router timed out (connect ${String(CONNECT_TIMEOUT_MS)}ms / total ${String(this.timeoutMs)}ms).`);
+            }
+            throw err;
+        }
+        clearTimeout(connectTimer); // headers arrived; only the total budget governs the body now
         if (!res.ok) {
+            // Read the error body under the same timeout, then clear the timers. Clearing
+            // before the read would leave a stalled error body with no budget to abort it.
             let parsed;
             try {
                 parsed = (await res.json());
@@ -53,9 +162,12 @@ export class FlexInference {
             catch {
                 parsed = undefined;
             }
+            finally {
+                cleanup();
+            }
             throw new FlexInferenceError(res.status, parsed, `HTTP ${String(res.status)} ${res.statusText}`);
         }
-        return res;
+        return { res, cleanup };
     }
 }
 class Responses {
@@ -64,11 +176,17 @@ class Responses {
         this.post = post;
     }
     async create(body, options) {
-        const res = await this.post("/responses", body, options?.signal);
+        assertStartWithin(body);
+        const { res, cleanup } = await this.post("/responses", body, options?.signal);
         if (body.stream) {
-            return streamSSE(res);
+            return streamSSE(res, cleanup);
+        }
+        try {
+            return (await res.json());
+        }
+        finally {
+            cleanup();
         }
-        return (await res.json());
     }
 }
 class Chat {
@@ -83,24 +201,35 @@ class ChatCompletions {
         this.post = post;
     }
     async create(body, options) {
-        const res = await this.post("/chat/completions", body, options?.signal);
+        assertStartWithin(body);
+        const { res, cleanup } = await this.post("/chat/completions", body, options?.signal);
         if (body.stream) {
-            return streamSSE(res);
+            return streamSSE(res, cleanup);
+        }
+        try {
+            return (await res.json());
+        }
+        finally {
+            cleanup();
         }
-        return (await res.json());
     }
 }
-async function* streamSSE(res) {
-    if (!res.body)
+async function* streamSSE(res, cleanup) {
+    if (!res.body) {
+        cleanup();
         throw new Error("FlexInference: streaming response has no body.");
+    }
     const reader = res.body.getReader();
     const decoder = new TextDecoder();
     let buffer = "";
+    let exhausted = false;
     try {
         for (;;) {
             const { done, value } = await reader.read();
-            if (done)
+            if (done) {
+                exhausted = true;
                 break;
+            }
             buffer += decoder.decode(value, { stream: true }).replace(/\r/g, "");
             let sep;
             while ((sep = buffer.indexOf("\n\n")) !== -1) {
@@ -114,13 +243,26 @@ async function* streamSSE(res) {
                 if (dataLines.length === 0)
                     continue;
                 const data = dataLines.join("\n");
-                if (data === "[DONE]")
+                if (data === "[DONE]") {
+                    exhausted = true;
                     return;
+                }
                 yield JSON.parse(data);
             }
         }
     }
     finally {
-        reader.releaseLock();
+        // A consumer that breaks early (or a parse that throws) leaves the body undrained.
+        // Cancel it so the socket is released now instead of lingering until GC; a fully
+        // drained stream just releases the lock.
+        if (exhausted) {
+            reader.releaseLock();
+        }
+        else {
+            await reader.cancel().catch(() => {
+                // already cancelled or the body errored; nothing to release
+            });
+        }
+        cleanup();
     }
 }

package/dist/types.d.ts

@@ -1105,7 +1105,7 @@ export interface components {
              */
             top_logprobs?: number;
             /**
-             * @description FlexInference deadline control. "priority" or "standard" route the request to those OpenAI service tiers. A duration "XXh-YYm-ZZs" between 5 seconds and 10 minutes runs the flex tier and automatically falls back to standard if it has not started within the window.
+             * @description FlexInference deadline control. Required on every request. "default", "priority", and "auto" map to those OpenAI service tiers ("auto" lets OpenAI pick) and proxy any model. A duration "HHh-MMm-SSs" between 5 seconds and 10 minutes runs the flex tier on a flex-capable model and automatically falls back to standard if it has not started within the window. The retired value "standard" is no longer accepted (use "default").
              * @example 00h-00m-30s
              */
             start_within?: string;
@@ -1773,7 +1773,7 @@ export interface components {
         /**
          * @description Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`).
          *     For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend
-         *     phase on all assistant messages - dropping it can degrade performance. Not used for user messages.
+         *     phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
          * @enum {string}
          */
         MessagePhase: "commentary" | "final_answer";
@@ -5398,7 +5398,7 @@ export interface operations {
                     "text/event-stream": components["schemas"]["CreateChatCompletionStreamResponse"];
                 };
             };
-            /** @description Bad request. A FlexInference-layer rejection (type flexinference_error, e.g. invalid_start_within, unsupported_model, no_byok_key) or an OpenAI rejection passed through unchanged. */
+            /** @description Bad request. A FlexInference-layer rejection (type flexinference_error, e.g. missing_start_within, invalid_start_within, model_not_flex_capable, no_byok_key) or an OpenAI rejection passed through unchanged. */
             400: {
                 headers: {
                     [name: string]: unknown;
@@ -5441,7 +5441,7 @@ export interface operations {
                     "text/event-stream": components["schemas"]["ResponseStreamEvent"];
                 };
             };
-            /** @description Bad request. A FlexInference-layer rejection (type flexinference_error, e.g. invalid_start_within, unsupported_model, no_byok_key) or an OpenAI rejection passed through unchanged. */
+            /** @description Bad request. A FlexInference-layer rejection (type flexinference_error, e.g. missing_start_within, invalid_start_within, model_not_flex_capable, no_byok_key) or an OpenAI rejection passed through unchanged. */
             400: {
                 headers: {
                     [name: string]: unknown;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "flexinference",
-  "version": "0.1.0",
+  "version": "1.0.1",
   "description": "Official TypeScript SDK for FlexInference - a deadline-aware, OpenAI-compatible inference router.",
   "license": "MIT",
   "author": "Aditya Perswal <adityaperswal@gmail.com>",
@@ -27,11 +27,14 @@
   "files": [
     "dist",
     "README.md",
+    "CHANGELOG.md",
     "LICENSE"
   ],
   "scripts": {
     "build": "tsc -p tsconfig.json",
     "typecheck": "tsc -p tsconfig.json --noEmit",
+    "test:types": "tsc -p tsconfig.test.json --noEmit",
+    "test": "node test/smoke.mjs && npm run test:types",
     "lint": "eslint .",
     "lint:fix": "eslint . --fix",
     "format": "prettier --write .",