flexinference 1.4.0 → 1.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +4 -0
- package/README.md +16 -10
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,9 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 1.4.1
|
|
4
|
+
|
|
5
|
+
Rewrites the README in plain language from the copy audit. It explains what `start_within` does, how the cheaper flex tier runs first and falls back up to your standard tier when it cannot start in time, and how you only pay a share of what a flex request saves you. No code or API changes.
|
|
6
|
+
|
|
3
7
|
## 1.4.0
|
|
4
8
|
|
|
5
9
|
Surfaces the router's **`doc_url`** on `FlexInferenceError` as a new `docUrl` field: a deep
|
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# FlexInference (TypeScript)
|
|
2
2
|
|
|
3
|
-
The official TypeScript SDK for [FlexInference](https://flexinference.com)
|
|
3
|
+
The official TypeScript SDK for [FlexInference](https://flexinference.com), an inference router that works with OpenAI, Google Gemini, and Anthropic. Send the OpenAI requests you already send and bring your own provider key. Set one required field, `start_within`, to tell us how long you will wait, and we trade a little latency for a smaller bill. You can call it four ways. Use `responses`, `chat.completions`, `interactions` (the Gemini shape), or `messages` (the Anthropic shape). Any of them reaches any provider.
|
|
4
4
|
|
|
5
5
|
```bash
|
|
6
6
|
npm install flexinference
|
|
@@ -22,21 +22,23 @@ const res = await client.responses.create({
|
|
|
22
22
|
console.log(outputText(res));
|
|
23
23
|
```
|
|
24
24
|
|
|
25
|
-
Responses come back as the **raw OpenAI JSON** (we never reshape the body), so there is no `output_text` field on the wire
|
|
25
|
+
Responses come back as the **raw OpenAI JSON** (we never reshape the body), so there is no `output_text` field on the wire. OpenAI's own SDKs compute that. `outputText(res)` pulls the assistant's text out of either a response or a chat completion for you.
|
|
26
26
|
|
|
27
|
-
`start_within` is
|
|
27
|
+
`start_within` is required on every request. It takes `"default"`, `"priority"`, `"auto"`, or a duration in the form `"HHh-MMm-SSs"` from 5 seconds to 10 minutes. A duration tells us how long you will wait for the request to start. We try the cheaper flex tier first and give it that long to begin. If flex cannot start in time, we move your request up to your standard tier so it still finishes. The words `"default"`, `"priority"`, and `"auto"` map to those OpenAI service tiers and proxy any model. Types catch most mistakes for you. A missing value, or a value that does not fit the allowed shapes, fails at compile time. A duration that fits the shape but is malformed or out of range fails at runtime, the same check plain-JavaScript callers get. See the [docs](https://flexinference.mintlify.app/deadline-routing) for the full rules.
|
|
28
|
+
|
|
29
|
+
This runs the opposite way from the usual fallback. Most setups start on the strong model and drop to a cheaper one when they need to. We start on the cheaper flex tier and escalate up to your standard model if flex cannot start in time, so a slow flex tier never costs you the answer.
|
|
28
30
|
|
|
29
31
|
## Providers (OpenAI, Gemini, and Anthropic)
|
|
30
32
|
|
|
31
|
-
FlexInference routes to **OpenAI**, **Google Gemini**, and **Anthropic**. Send the same OpenAI-shaped request and pass whichever model id you want
|
|
33
|
+
FlexInference routes to **OpenAI**, **Google Gemini**, and **Anthropic**. Send the same OpenAI-shaped request and pass whichever model id you want. Use `gpt-5.5`, `o4-mini`, `gemini-3.5-flash`, `claude-opus-4-8`, and so on. We translate Gemini and Anthropic to and from the OpenAI shape, so your code is identical for all three.
|
|
32
34
|
|
|
33
|
-
- **OpenAI:** `default` (standard tier), `priority`, `auto`, and
|
|
34
|
-
- **Gemini:** `default` maps to Gemini's
|
|
35
|
-
- **Anthropic (Claude):** proxy-only. `default`, `priority`, and `auto` work; there is **no flex
|
|
35
|
+
- **OpenAI:** `default` (the standard tier), `priority`, `auto`, and flex on flex-capable models. You reach flex by passing a duration.
|
|
36
|
+
- **Gemini:** `default` maps to Gemini's standard tier. You also get `priority` and flex on the Gemini flex models (`gemini-3.5-flash`, `gemini-3.1-flash-lite`, `gemini-3.1-pro-preview`, `gemini-3-flash-preview`, `gemini-2.5-pro`, `gemini-2.5-flash`, `gemini-2.5-flash-lite`). Gemini has no `auto` tier, so `start_within: "auto"` on a Gemini model returns `400`.
|
|
37
|
+
- **Anthropic (Claude):** proxy-only. `default`, `priority`, and `auto` work; there is **no flex tier**, so a duration `start_within` on a `claude-*` model returns `400 flex_unsupported_for_anthropic`. Anthropic requires a token cap, so set `max_output_tokens` (`max_completion_tokens` on Chat, `max_tokens` on Messages) or you get `400 missing_max_tokens`. You keep the unified API and tier control, and draw down your own Anthropic credits.
|
|
36
38
|
|
|
37
39
|
Add the provider key you'll use (OpenAI, Gemini, and/or Anthropic) in the [dashboard](https://www.flexinference.com/dashboard). Text, streaming, structured outputs, function calling, image input, and web search work across providers (send a Responses `web_search` tool; we map it to Gemini's `google_search`).
|
|
38
40
|
|
|
39
|
-
|
|
41
|
+
Do not send `service_tier`. The router sets the tier from `start_within`, and it rejects a caller-supplied `service_tier` with `400 service_tier_not_allowed`.
|
|
40
42
|
|
|
41
43
|
## Streaming
|
|
42
44
|
|
|
@@ -136,10 +138,14 @@ try {
|
|
|
136
138
|
}
|
|
137
139
|
```
|
|
138
140
|
|
|
141
|
+
Every FlexInference error tells you four things. It says what went wrong, why it went wrong, how to fix it, and it shows an example of a request that works. The `message` reads like a note from a person, so an agent can act on it instead of guessing. Errors that come from the provider pass straight through with their status and body intact, so you always see the real cause. For instance, a duration on a `claude-*` model returns `400 flex_unsupported_for_anthropic` with a message that tells you to drop the duration or switch to `default`, `priority`, or `auto`.
|
|
142
|
+
|
|
139
143
|
## Billing / 402
|
|
140
144
|
|
|
141
|
-
If your account's billing is past due, the router pauses
|
|
142
|
-
`402 Payment Required` on
|
|
145
|
+
If your account's billing is past due, the router pauses paid flex requests and returns
|
|
146
|
+
`402 Payment Required` on them. Free routing keeps working, so your standard-tier calls
|
|
147
|
+
still go through. Flex is the only part you ever pay for, and you only pay it when it
|
|
148
|
+
saves you money. The SDK throws a
|
|
143
149
|
typed `PaymentRequiredError` (a subclass of `FlexInferenceError`) for HTTP 402, so you
|
|
144
150
|
can catch it on its own and prompt the user to update payment while letting other errors
|
|
145
151
|
propagate:
|
package/package.json
CHANGED