@bitbaum/ai-kit 0.6.2 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +100 -14
- package/dist/complete.d.ts +160 -0
- package/dist/complete.js +237 -0
- package/dist/index.d.ts +16 -7
- package/dist/index.js +16 -7
- package/dist/liveness.d.ts +112 -0
- package/dist/liveness.js +198 -0
- package/package.json +9 -7
- package/src/complete.ts +340 -0
- package/src/index.ts +33 -7
- package/src/liveness.ts +268 -0
package/README.md
CHANGED
|
@@ -6,7 +6,7 @@ what to do when you're going too fast, how to share a free tier fairly between
|
|
|
6
6
|
users, and how to fill a form from plain language.
|
|
7
7
|
|
|
8
8
|
```bash
|
|
9
|
-
|
|
9
|
+
pnpm add @bitbaum/ai-kit
|
|
10
10
|
```
|
|
11
11
|
|
|
12
12
|
---
|
|
@@ -58,6 +58,95 @@ the same org-wide daily budget, so when the day runs dry every link in that
|
|
|
58
58
|
via a text tool protocol, not native `tool_calls`. A native-only client would
|
|
59
59
|
have silently lost most of the chain.
|
|
60
60
|
|
|
61
|
+
### Make the call — `complete()` owns the fetch
|
|
62
|
+
|
|
63
|
+
```ts
|
|
64
|
+
import { complete, freeChain, usableChain, createHealthTracker } from '@bitbaum/ai-kit';
|
|
65
|
+
|
|
66
|
+
export const llmHealth = createHealthTracker();
|
|
67
|
+
const chain = usableChain(freeChain('MYAPP'), process.env);
|
|
68
|
+
|
|
69
|
+
const { text, id } = await complete({
|
|
70
|
+
chain,
|
|
71
|
+
health: llmHealth,
|
|
72
|
+
maxTokens: 500,
|
|
73
|
+
messages: [{ role: 'user', content: 'Summarise this in one line.' }],
|
|
74
|
+
});
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
For four releases this package shipped the *decisions* and told you to keep the
|
|
78
|
+
fetch. The rule read well and it was wrong: measured 2026-09-05, this fleet ran
|
|
79
|
+
**eight** hand-rolled clients, **two** of which told the three kinds of 429
|
|
80
|
+
apart — while `ai-forms`, which ships a working route factory, had more than
|
|
81
|
+
twice this package's adoption. A package that hands you a working call gets
|
|
82
|
+
installed; one that hands you advice about calls does not.
|
|
83
|
+
|
|
84
|
+
`complete()` is the chain walk plus the request, and it carries the parts that
|
|
85
|
+
kept getting left out of the hand-rolled ones:
|
|
86
|
+
|
|
87
|
+
- a **200 with empty content is a failure**, not an answer — reasoning models
|
|
88
|
+
and some vendors return exactly that, and every client that read
|
|
89
|
+
`choices[0].message.content || ''` shipped the empty string to a user;
|
|
90
|
+
- a **daily** 429 marks the whole vendor dead for the walk, instead of trying
|
|
91
|
+
its other models against the same exhausted org-wide budget;
|
|
92
|
+
- a **size** 429 ends the walk rather than demoting to a *smaller* ceiling,
|
|
93
|
+
which is strictly worse;
|
|
94
|
+
- the vendor's response body survives into the error, so an exhausted day is
|
|
95
|
+
distinguishable from a momentary burst in a log.
|
|
96
|
+
|
|
97
|
+
**`maxTokens` has a floor, and it is higher than you think.** The chain leads
|
|
98
|
+
with reasoning models, which spend the budget on hidden thinking before emitting
|
|
99
|
+
a visible token: `groq/openai/gpt-oss-20b` answered *empty* at 16 and correctly
|
|
100
|
+
at 256 for the same one-word question. A mean budget makes a healthy model look
|
|
101
|
+
dead.
|
|
102
|
+
|
|
103
|
+
`tryChain` stays for a caller with a genuinely unusual request to make.
|
|
104
|
+
|
|
105
|
+
### Does it work RIGHT NOW? — a probe, not a guess
|
|
106
|
+
|
|
107
|
+
```ts
|
|
108
|
+
// app/api/health/ai/route.ts — Next App Router, Hono, Deno and Bun all take
|
|
109
|
+
// this shape directly.
|
|
110
|
+
import { createAiHealthHandler, freeChain, usableChain } from '@bitbaum/ai-kit';
|
|
111
|
+
import { llmHealth } from '@/lib/llm-health';
|
|
112
|
+
|
|
113
|
+
const handler = createAiHealthHandler({
|
|
114
|
+
chain: usableChain(freeChain('MYAPP'), process.env),
|
|
115
|
+
health: llmHealth,
|
|
116
|
+
secret: process.env.AI_PROBE_SECRET,
|
|
117
|
+
});
|
|
118
|
+
|
|
119
|
+
export const GET = handler;
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
```
|
|
123
|
+
GET /api/health/ai free. What happened last time.
|
|
124
|
+
GET /api/health/ai?probe=1 + the secret makes a real call. 200 or 503.
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
**Why a probe and not a passive read.** Absence of failure is not evidence of
|
|
128
|
+
success. A tracker that has recorded nothing looks identical whether the chain
|
|
129
|
+
is perfect or every key is missing — and straight after a deploy that is exactly
|
|
130
|
+
the state it is in. Observed converting the first app: the deploy was green, the
|
|
131
|
+
bundle provably held the new code, both keys were present, `/api/health`
|
|
132
|
+
returned 200, and `llm.status` was `"unknown"`. Every available signal said
|
|
133
|
+
"probably fine" and none said "works". The only paths that would have answered
|
|
134
|
+
were an admin-authenticated form and two cron jobs that **email real users** —
|
|
135
|
+
verifying a deploy must never require spamming somebody.
|
|
136
|
+
|
|
137
|
+
**Why it is gated and cached.** A probe spends real tokens from a daily budget
|
|
138
|
+
shared with the app's actual features, so an ungated one on a health route is a
|
|
139
|
+
self-inflicted outage: a monitor polling every 30s would drain the allowance and
|
|
140
|
+
take the AI features down with it. So a probe runs only on `?probe=1` **and**
|
|
141
|
+
with the secret, a *success* is cached for 10 minutes (returned with `cached`
|
|
142
|
+
and its age, because a nine-minute-old success is a different claim from a fresh
|
|
143
|
+
one), and a **failure is never cached** — the whole point is the truth about
|
|
144
|
+
right now.
|
|
145
|
+
|
|
146
|
+
With no secret configured the route answers **501**, not an open probe: an app
|
|
147
|
+
that forgets to set one gets a route that cannot spend money, rather than one
|
|
148
|
+
that can.
|
|
149
|
+
|
|
61
150
|
### Is it up? — walk the chain, and know when none of it worked
|
|
62
151
|
|
|
63
152
|
A chain nobody walks is a list, not a fallback. This was found sitting unused
|
|
@@ -79,7 +168,7 @@ const { text } = await tryChain(chain, {
|
|
|
79
168
|
});
|
|
80
169
|
```
|
|
81
170
|
|
|
82
|
-
|
|
171
|
+
`attempt` makes the real request; `tryChain` only
|
|
83
172
|
decides which link goes next and throws `ChainExhaustedError` (naming every
|
|
84
173
|
link's failure, not just the last) when none of them work.
|
|
85
174
|
|
|
@@ -165,7 +254,7 @@ stays its own package — it works, four apps run it, and it is useful well outs
|
|
|
165
254
|
this fleet. Swallowing it would have broken those four for the sake of a filing
|
|
166
255
|
system.
|
|
167
256
|
|
|
168
|
-
**Note the subpath.** Form filling is at
|
|
257
|
+
**Note the subpath.** Form filling is at `@bitbaum/ai-kit/forms`, not at the root. For one
|
|
169
258
|
release it was both, and the first app to adopt the merged package paid for it:
|
|
170
259
|
`ai-forms` is ESM-only, so importing the *chain* from the root dragged the forms
|
|
171
260
|
package in behind it and the app's Jest run — which executes CJS — died inside a
|
|
@@ -173,21 +262,18 @@ module it never asked for. One install is still the whole promise; the exports
|
|
|
173
262
|
map is what keeps it, while letting a server that only wants a provider chain
|
|
174
263
|
stop paying for a form library.
|
|
175
264
|
|
|
176
|
-
React lives on its own subpath and is an **optional** peer, so importing
|
|
177
|
-
on a server never pulls in a UI library.
|
|
265
|
+
React lives on its own subpath and is an **optional** peer, so importing
|
|
266
|
+
`@bitbaum/ai-kit` on a server never pulls in a UI library.
|
|
178
267
|
|
|
179
268
|
---
|
|
180
269
|
|
|
181
270
|
## What it deliberately does not ship
|
|
182
271
|
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
in this fleet and it is the one that broke the rule, by shipping a route factory
|
|
189
|
-
and a React hook. A package that hands you a working route gets installed; one
|
|
190
|
-
that hands you advice about routes does not.
|
|
272
|
+
**~~An HTTP client.~~** It ships one now — see [`complete()`](#make-the-call--complete-owns-the-fetch).
|
|
273
|
+
The old rule ("every app has its own calling conventions, and replacing those is
|
|
274
|
+
a rewrite rather than an adoption") described this fleet's duplication
|
|
275
|
+
accurately and then protected it: the conventions differed because nothing had
|
|
276
|
+
ever offered to own them.
|
|
191
277
|
|
|
192
278
|
**Model values.** Which ids are free, which are billed, and which your account
|
|
193
279
|
may use are properties of *your* deployment. Centralise the rule, assert it
|
|
@@ -210,7 +296,7 @@ model catalogue to do it.
|
|
|
210
296
|
## Development
|
|
211
297
|
|
|
212
298
|
```bash
|
|
213
|
-
|
|
299
|
+
pnpm run verify # lint + typecheck + build + test
|
|
214
300
|
```
|
|
215
301
|
|
|
216
302
|
MIT.
|
|
@@ -0,0 +1,160 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The call itself — the one thing this package refused to ship, and the reason
|
|
3
|
+
* the rest of it went unused.
|
|
4
|
+
*
|
|
5
|
+
* ── WHY THIS REVERSES A STATED RULE ──────────────────────────────────────────
|
|
6
|
+
* Every module here was written against a real outage, and every one of them is
|
|
7
|
+
* correct. None of that reached the apps that were actually failing. Measured
|
|
8
|
+
* across the fleet on 2026-09-05:
|
|
9
|
+
*
|
|
10
|
+
* this package's decisions adopted by 2 repos
|
|
11
|
+
* `ai-forms`, which ships a working handler adopted by 5 repos
|
|
12
|
+
* hand-rolled LLM clients still in service 8, ~1400-1700 lines each
|
|
13
|
+
*
|
|
14
|
+
* The pattern is not about quality, it is about shape. `ai-forms` was adopted
|
|
15
|
+
* because `createFormAssistHandler` does the job; this package was not, because
|
|
16
|
+
* it hands back advice the caller must then wire up. The old rule — "every app
|
|
17
|
+
* has its own calling conventions, replacing them is a rewrite rather than an
|
|
18
|
+
* adoption" — describes the duplication accurately and then protects it. The
|
|
19
|
+
* conventions differ because nothing ever offered to own them.
|
|
20
|
+
*
|
|
21
|
+
* The cost of that is not theoretical. Of the 8 hand-rolled clients, 2 tell the
|
|
22
|
+
* three kinds of 429 apart; the other 6 treat a spent daily budget as a busy
|
|
23
|
+
* minute — `limits.ts` has explained why that is harmful since 2026-08-14, in a
|
|
24
|
+
* module those 6 apps do not import. `tryChain` says it plainly: a chain nobody
|
|
25
|
+
* walks is a list, not a fallback. A decision nobody calls is a comment.
|
|
26
|
+
*
|
|
27
|
+
* So: this owns the fetch. `tryChain` stays for callers with a genuinely
|
|
28
|
+
* unusual request to make; this is the answer for everyone else.
|
|
29
|
+
*
|
|
30
|
+
* ── WHAT IT KNOWS THAT A HAND-ROLLED LOOP DOES NOT ───────────────────────────
|
|
31
|
+
* Walking the chain is the easy half. Three behaviours below are the ones every
|
|
32
|
+
* hand-rolled client in this fleet got wrong, each traced to an incident:
|
|
33
|
+
*
|
|
34
|
+
* HTTP 200 IS NOT SUCCESS. `nvidia/nemotron-nano-12b-v2-vl` returns 200 with
|
|
35
|
+
* empty content, and `gemini-2.5-flash` does the same after a tool call — it
|
|
36
|
+
* spends its whole budget on internal thinking and emits no text part. A
|
|
37
|
+
* client that checks `res.ok` returns "" to the user and reports success, so
|
|
38
|
+
* the chain never advances and health stays green through a total outage.
|
|
39
|
+
* Empty content is a FAILURE here, and it demotes to the next link.
|
|
40
|
+
*
|
|
41
|
+
* A DAILY 429 CONDEMNS THE WHOLE VENDOR, not one model. The budget is
|
|
42
|
+
* org-wide and shared across models, so every remaining link at that provider
|
|
43
|
+
* is already dead. Walking them costs a dead round trip each and reaches the
|
|
44
|
+
* same failure. They are skipped.
|
|
45
|
+
*
|
|
46
|
+
* A SIZE 429 ENDS THE WALK. One request exceeded the entire per-minute
|
|
47
|
+
* allowance; the next model down has a SMALLER ceiling (measured: 12000 TPM
|
|
48
|
+
* vs 6000), so demoting makes it strictly worse. The only cure is a shorter
|
|
49
|
+
* prompt, and the caller is told exactly that instead of watching the chain
|
|
50
|
+
* burn itself down to reach a worse version of the same error.
|
|
51
|
+
*
|
|
52
|
+
* ── AND ONE IT INHERITS ──────────────────────────────────────────────────────
|
|
53
|
+
* The response BODY is kept in every error. A status-only message ("groq 429")
|
|
54
|
+
* makes an exhausted day indistinguishable from a momentary burst, and the
|
|
55
|
+
* obvious remedy for the latter — wait and retry — can never work for the
|
|
56
|
+
* former. That misdiagnosis cost an hour once; it is not free to repeat.
|
|
57
|
+
*/
|
|
58
|
+
import { type Env, type Link, type Provider } from "./chain.js";
|
|
59
|
+
import type { HealthTracker } from "./health.js";
|
|
60
|
+
import { type RateLimitKind } from "./limits.js";
|
|
61
|
+
/** One message in the OpenAI chat-completions shape every provider here speaks. */
|
|
62
|
+
export interface ChatMessage {
|
|
63
|
+
role: "system" | "user" | "assistant" | "tool";
|
|
64
|
+
content: string;
|
|
65
|
+
/** Present on `role: "tool"` replies; passed through untouched. */
|
|
66
|
+
tool_call_id?: string;
|
|
67
|
+
name?: string;
|
|
68
|
+
}
|
|
69
|
+
/**
|
|
70
|
+
* A tool call the model asked for, normalised across the two protocols models
|
|
71
|
+
* actually answer on.
|
|
72
|
+
*
|
|
73
|
+
* Both exist in the default chain: of nine free models probed live, four
|
|
74
|
+
* answered with native `tool_calls` and five only in text. Callers get the
|
|
75
|
+
* native shape here; parsing the text protocol is the caller's business,
|
|
76
|
+
* because its convention differs per app.
|
|
77
|
+
*/
|
|
78
|
+
export interface ToolCall {
|
|
79
|
+
id: string;
|
|
80
|
+
name: string;
|
|
81
|
+
/** Raw JSON string as the model emitted it — NOT parsed, because a model can emit invalid JSON and the caller decides what to do about that. */
|
|
82
|
+
args: string;
|
|
83
|
+
}
|
|
84
|
+
export interface CompleteOptions {
|
|
85
|
+
messages: ChatMessage[];
|
|
86
|
+
/**
|
|
87
|
+
* Links to try, in order. Defaults to `usableChain(freeChain())` — every free
|
|
88
|
+
* provider that has a key in `env`.
|
|
89
|
+
*/
|
|
90
|
+
chain?: Link[];
|
|
91
|
+
/** Providers to derive the chain from when `chain` is not given. */
|
|
92
|
+
providers?: Provider[];
|
|
93
|
+
/**
|
|
94
|
+
* Start the chain at this model rather than the front, falling through to the
|
|
95
|
+
* rest. The usual home for an app's "use this model" env var.
|
|
96
|
+
*/
|
|
97
|
+
model?: string;
|
|
98
|
+
env?: Env;
|
|
99
|
+
health?: HealthTracker;
|
|
100
|
+
signal?: AbortSignal;
|
|
101
|
+
/**
|
|
102
|
+
* Set this GENEROUSLY, or a healthy model looks dead.
|
|
103
|
+
*
|
|
104
|
+
* The default chain leads with reasoning models, which spend this budget
|
|
105
|
+
* thinking before emitting a visible token. Set it too low and the vendor
|
|
106
|
+
* returns 200 with empty content — which this module correctly treats as a
|
|
107
|
+
* failure and demotes, so a small `maxTokens` silently walks the whole chain
|
|
108
|
+
* and reports every link broken. Measured 2026-09-05: groq/openai/gpt-oss-20b
|
|
109
|
+
* answered EMPTY at 16 and answered correctly at 256, for the same one-word
|
|
110
|
+
* question.
|
|
111
|
+
*/
|
|
112
|
+
maxTokens?: number;
|
|
113
|
+
temperature?: number;
|
|
114
|
+
/** Tool definitions in the OpenAI shape; passed through untouched. */
|
|
115
|
+
tools?: unknown[];
|
|
116
|
+
/** Extra body fields for a vendor-specific parameter. Merged last, so it can override. */
|
|
117
|
+
extraBody?: Record<string, unknown>;
|
|
118
|
+
/** Called on each link's failure before moving on — e.g. to log which id rotted. */
|
|
119
|
+
onLinkFailure?: (link: Link, error: Error) => void;
|
|
120
|
+
/** Injected for tests. Defaults to global `fetch`. */
|
|
121
|
+
fetchImpl?: typeof fetch;
|
|
122
|
+
}
|
|
123
|
+
export interface CompleteResult {
|
|
124
|
+
/** The assistant's text. Never empty — an empty completion is treated as a failure. */
|
|
125
|
+
text: string;
|
|
126
|
+
/** `provider/model`, the id worth logging: it says which link actually served the turn. */
|
|
127
|
+
id: string;
|
|
128
|
+
link: Link;
|
|
129
|
+
toolCalls: ToolCall[];
|
|
130
|
+
/** The parsed response body, for a caller that needs a field this does not surface. */
|
|
131
|
+
raw: unknown;
|
|
132
|
+
}
|
|
133
|
+
/**
|
|
134
|
+
* A link failed in a way that says something about the WALK, not just this link.
|
|
135
|
+
*
|
|
136
|
+
* `kind` is what the walker acts on; it is carried on the error so a caller
|
|
137
|
+
* reading `ChainExhaustedError.failures` can see why the walk stopped where it
|
|
138
|
+
* did rather than inferring it from prose.
|
|
139
|
+
*/
|
|
140
|
+
export declare class LinkFailure extends Error {
|
|
141
|
+
readonly link: Link;
|
|
142
|
+
readonly status?: number;
|
|
143
|
+
readonly kind?: RateLimitKind;
|
|
144
|
+
readonly retryAfter?: number | null;
|
|
145
|
+
constructor(link: Link, message: string, init?: {
|
|
146
|
+
status?: number;
|
|
147
|
+
kind?: RateLimitKind;
|
|
148
|
+
retryAfter?: number | null;
|
|
149
|
+
});
|
|
150
|
+
}
|
|
151
|
+
/** `provider/model` — the id worth logging, because the model alone does not say whose meter it drew on. */
|
|
152
|
+
export declare function linkId(link: Link): string;
|
|
153
|
+
/**
|
|
154
|
+
* Call the first link that works, and return what it said.
|
|
155
|
+
*
|
|
156
|
+
* Throws `ChainExhaustedError` carrying every link's failure, so a log shows
|
|
157
|
+
* what was actually tried — the failure that explains an outage is usually not
|
|
158
|
+
* the last one.
|
|
159
|
+
*/
|
|
160
|
+
export declare function complete(options: CompleteOptions): Promise<CompleteResult>;
|
package/dist/complete.js
ADDED
|
@@ -0,0 +1,237 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The call itself — the one thing this package refused to ship, and the reason
|
|
3
|
+
* the rest of it went unused.
|
|
4
|
+
*
|
|
5
|
+
* ── WHY THIS REVERSES A STATED RULE ──────────────────────────────────────────
|
|
6
|
+
* Every module here was written against a real outage, and every one of them is
|
|
7
|
+
* correct. None of that reached the apps that were actually failing. Measured
|
|
8
|
+
* across the fleet on 2026-09-05:
|
|
9
|
+
*
|
|
10
|
+
* this package's decisions adopted by 2 repos
|
|
11
|
+
* `ai-forms`, which ships a working handler adopted by 5 repos
|
|
12
|
+
* hand-rolled LLM clients still in service 8, ~1400-1700 lines each
|
|
13
|
+
*
|
|
14
|
+
* The pattern is not about quality, it is about shape. `ai-forms` was adopted
|
|
15
|
+
* because `createFormAssistHandler` does the job; this package was not, because
|
|
16
|
+
* it hands back advice the caller must then wire up. The old rule — "every app
|
|
17
|
+
* has its own calling conventions, replacing them is a rewrite rather than an
|
|
18
|
+
* adoption" — describes the duplication accurately and then protects it. The
|
|
19
|
+
* conventions differ because nothing ever offered to own them.
|
|
20
|
+
*
|
|
21
|
+
* The cost of that is not theoretical. Of the 8 hand-rolled clients, 2 tell the
|
|
22
|
+
* three kinds of 429 apart; the other 6 treat a spent daily budget as a busy
|
|
23
|
+
* minute — `limits.ts` has explained why that is harmful since 2026-08-14, in a
|
|
24
|
+
* module those 6 apps do not import. `tryChain` says it plainly: a chain nobody
|
|
25
|
+
* walks is a list, not a fallback. A decision nobody calls is a comment.
|
|
26
|
+
*
|
|
27
|
+
* So: this owns the fetch. `tryChain` stays for callers with a genuinely
|
|
28
|
+
* unusual request to make; this is the answer for everyone else.
|
|
29
|
+
*
|
|
30
|
+
* ── WHAT IT KNOWS THAT A HAND-ROLLED LOOP DOES NOT ───────────────────────────
|
|
31
|
+
* Walking the chain is the easy half. Three behaviours below are the ones every
|
|
32
|
+
* hand-rolled client in this fleet got wrong, each traced to an incident:
|
|
33
|
+
*
|
|
34
|
+
* HTTP 200 IS NOT SUCCESS. `nvidia/nemotron-nano-12b-v2-vl` returns 200 with
|
|
35
|
+
* empty content, and `gemini-2.5-flash` does the same after a tool call — it
|
|
36
|
+
* spends its whole budget on internal thinking and emits no text part. A
|
|
37
|
+
* client that checks `res.ok` returns "" to the user and reports success, so
|
|
38
|
+
* the chain never advances and health stays green through a total outage.
|
|
39
|
+
* Empty content is a FAILURE here, and it demotes to the next link.
|
|
40
|
+
*
|
|
41
|
+
* A DAILY 429 CONDEMNS THE WHOLE VENDOR, not one model. The budget is
|
|
42
|
+
* org-wide and shared across models, so every remaining link at that provider
|
|
43
|
+
* is already dead. Walking them costs a dead round trip each and reaches the
|
|
44
|
+
* same failure. They are skipped.
|
|
45
|
+
*
|
|
46
|
+
* A SIZE 429 ENDS THE WALK. One request exceeded the entire per-minute
|
|
47
|
+
* allowance; the next model down has a SMALLER ceiling (measured: 12000 TPM
|
|
48
|
+
* vs 6000), so demoting makes it strictly worse. The only cure is a shorter
|
|
49
|
+
* prompt, and the caller is told exactly that instead of watching the chain
|
|
50
|
+
* burn itself down to reach a worse version of the same error.
|
|
51
|
+
*
|
|
52
|
+
* ── AND ONE IT INHERITS ──────────────────────────────────────────────────────
|
|
53
|
+
* The response BODY is kept in every error. A status-only message ("groq 429")
|
|
54
|
+
* makes an exhausted day indistinguishable from a momentary burst, and the
|
|
55
|
+
* obvious remedy for the latter — wait and retry — can never work for the
|
|
56
|
+
* former. That misdiagnosis cost an hour once; it is not free to repeat.
|
|
57
|
+
*/
|
|
58
|
+
import { chainFrom, freeChain, usableChain } from "./chain.js";
|
|
59
|
+
import { ChainExhaustedError } from "./attempt.js";
|
|
60
|
+
import { classifyRateLimit, retryAfterSeconds } from "./limits.js";
|
|
61
|
+
/**
|
|
62
|
+
* A link failed in a way that says something about the WALK, not just this link.
|
|
63
|
+
*
|
|
64
|
+
* `kind` is what the walker acts on; it is carried on the error so a caller
|
|
65
|
+
* reading `ChainExhaustedError.failures` can see why the walk stopped where it
|
|
66
|
+
* did rather than inferring it from prose.
|
|
67
|
+
*/
|
|
68
|
+
export class LinkFailure extends Error {
|
|
69
|
+
link;
|
|
70
|
+
status;
|
|
71
|
+
kind;
|
|
72
|
+
retryAfter;
|
|
73
|
+
constructor(link, message, init = {}) {
|
|
74
|
+
super(message);
|
|
75
|
+
this.name = "LinkFailure";
|
|
76
|
+
this.link = link;
|
|
77
|
+
this.status = init.status;
|
|
78
|
+
this.kind = init.kind;
|
|
79
|
+
this.retryAfter = init.retryAfter;
|
|
80
|
+
}
|
|
81
|
+
}
|
|
82
|
+
/** `provider/model` — the id worth logging, because the model alone does not say whose meter it drew on. */
|
|
83
|
+
export function linkId(link) {
|
|
84
|
+
return `${link.provider.id}/${link.model}`;
|
|
85
|
+
}
|
|
86
|
+
function firstText(message) {
|
|
87
|
+
if (!message)
|
|
88
|
+
return "";
|
|
89
|
+
const content = message.content;
|
|
90
|
+
if (typeof content === "string")
|
|
91
|
+
return content;
|
|
92
|
+
// Some vendors return content as an array of parts. Concatenate the text ones
|
|
93
|
+
// rather than stringifying the array, which would hand the caller JSON.
|
|
94
|
+
if (Array.isArray(content)) {
|
|
95
|
+
return content
|
|
96
|
+
.map((part) => part && typeof part === "object" && typeof part.text === "string"
|
|
97
|
+
? part.text
|
|
98
|
+
: "")
|
|
99
|
+
.join("");
|
|
100
|
+
}
|
|
101
|
+
return "";
|
|
102
|
+
}
|
|
103
|
+
function toolCallsFrom(message) {
|
|
104
|
+
const raw = message?.tool_calls;
|
|
105
|
+
if (!Array.isArray(raw))
|
|
106
|
+
return [];
|
|
107
|
+
const out = [];
|
|
108
|
+
for (const entry of raw) {
|
|
109
|
+
if (!entry || typeof entry !== "object")
|
|
110
|
+
continue;
|
|
111
|
+
const fn = entry.function;
|
|
112
|
+
if (!fn || typeof fn.name !== "string")
|
|
113
|
+
continue;
|
|
114
|
+
out.push({
|
|
115
|
+
id: String(entry.id ?? ""),
|
|
116
|
+
name: fn.name,
|
|
117
|
+
args: typeof fn.arguments === "string" ? fn.arguments : JSON.stringify(fn.arguments ?? {}),
|
|
118
|
+
});
|
|
119
|
+
}
|
|
120
|
+
return out;
|
|
121
|
+
}
|
|
122
|
+
/**
|
|
123
|
+
* Truncated so a failure message stays readable in a log line, but long enough
|
|
124
|
+
* to carry the sentence that matters: Groq states the real reset ~90 characters
|
|
125
|
+
* into a daily-cap body, and cutting before it throws away the one number the
|
|
126
|
+
* user can act on.
|
|
127
|
+
*/
|
|
128
|
+
function excerpt(body, limit = 300) {
|
|
129
|
+
const flat = body.replace(/\s+/g, " ").trim();
|
|
130
|
+
return flat.length > limit ? `${flat.slice(0, limit)}…` : flat;
|
|
131
|
+
}
|
|
132
|
+
async function callLink(link, options, key) {
|
|
133
|
+
const doFetch = options.fetchImpl ?? globalThis.fetch;
|
|
134
|
+
const body = {
|
|
135
|
+
model: link.model,
|
|
136
|
+
messages: options.messages,
|
|
137
|
+
...(options.maxTokens === undefined ? {} : { max_tokens: options.maxTokens }),
|
|
138
|
+
...(options.temperature === undefined ? {} : { temperature: options.temperature }),
|
|
139
|
+
...(options.tools === undefined ? {} : { tools: options.tools }),
|
|
140
|
+
...options.extraBody,
|
|
141
|
+
};
|
|
142
|
+
let res;
|
|
143
|
+
try {
|
|
144
|
+
res = await doFetch(`${link.provider.baseUrl}/chat/completions`, {
|
|
145
|
+
method: "POST",
|
|
146
|
+
headers: { "content-type": "application/json", authorization: `Bearer ${key}` },
|
|
147
|
+
body: JSON.stringify(body),
|
|
148
|
+
signal: options.signal,
|
|
149
|
+
});
|
|
150
|
+
}
|
|
151
|
+
catch (error) {
|
|
152
|
+
// A transport failure (DNS, TLS, abort) is not the vendor's answer, so it
|
|
153
|
+
// carries no rate-limit kind — it demotes like any other link failure.
|
|
154
|
+
throw new LinkFailure(link, `${linkId(link)}: ${error.message}`);
|
|
155
|
+
}
|
|
156
|
+
const text = await res.text();
|
|
157
|
+
if (!res.ok) {
|
|
158
|
+
if (res.status === 429) {
|
|
159
|
+
const kind = classifyRateLimit(text);
|
|
160
|
+
const retryAfter = retryAfterSeconds(text);
|
|
161
|
+
throw new LinkFailure(link, `${linkId(link)}: 429 ${kind} — ${excerpt(text)}`, {
|
|
162
|
+
status: 429,
|
|
163
|
+
kind,
|
|
164
|
+
retryAfter,
|
|
165
|
+
});
|
|
166
|
+
}
|
|
167
|
+
throw new LinkFailure(link, `${linkId(link)}: ${res.status} — ${excerpt(text)}`, {
|
|
168
|
+
status: res.status,
|
|
169
|
+
});
|
|
170
|
+
}
|
|
171
|
+
let parsed;
|
|
172
|
+
try {
|
|
173
|
+
parsed = JSON.parse(text);
|
|
174
|
+
}
|
|
175
|
+
catch {
|
|
176
|
+
throw new LinkFailure(link, `${linkId(link)}: 200 with unparseable body — ${excerpt(text)}`, {
|
|
177
|
+
status: res.status,
|
|
178
|
+
});
|
|
179
|
+
}
|
|
180
|
+
const choice = parsed
|
|
181
|
+
?.choices?.[0];
|
|
182
|
+
const content = firstText(choice?.message);
|
|
183
|
+
const toolCalls = toolCallsFrom(choice?.message);
|
|
184
|
+
// A 200 that carries neither text nor a tool call is an outage wearing a
|
|
185
|
+
// success code — see the header. Demote, so the chain gets its chance.
|
|
186
|
+
if (content.trim() === "" && toolCalls.length === 0) {
|
|
187
|
+
throw new LinkFailure(link, `${linkId(link)}: 200 with empty content — model produced no output`, {
|
|
188
|
+
status: res.status,
|
|
189
|
+
});
|
|
190
|
+
}
|
|
191
|
+
return { text: content, id: linkId(link), link, toolCalls, raw: parsed };
|
|
192
|
+
}
|
|
193
|
+
/**
|
|
194
|
+
* Call the first link that works, and return what it said.
|
|
195
|
+
*
|
|
196
|
+
* Throws `ChainExhaustedError` carrying every link's failure, so a log shows
|
|
197
|
+
* what was actually tried — the failure that explains an outage is usually not
|
|
198
|
+
* the last one.
|
|
199
|
+
*/
|
|
200
|
+
export async function complete(options) {
|
|
201
|
+
const env = options.env ?? process.env;
|
|
202
|
+
const base = options.chain ?? usableChain(options.providers ?? freeChain(), env);
|
|
203
|
+
const chain = chainFrom(options.model, base);
|
|
204
|
+
const failures = [];
|
|
205
|
+
const deadProviders = new Set();
|
|
206
|
+
for (const link of chain) {
|
|
207
|
+
// A daily cap already condemned this vendor earlier in the walk. Its other
|
|
208
|
+
// models draw on the same exhausted budget, so trying them buys a dead
|
|
209
|
+
// round trip and the identical error.
|
|
210
|
+
if (deadProviders.has(link.provider.id))
|
|
211
|
+
continue;
|
|
212
|
+
const key = env[link.provider.keyEnv]?.trim();
|
|
213
|
+
if (!key) {
|
|
214
|
+
failures.push({ link, message: `${linkId(link)}: no ${link.provider.keyEnv} in env` });
|
|
215
|
+
continue;
|
|
216
|
+
}
|
|
217
|
+
try {
|
|
218
|
+
const result = await callLink(link, options, key);
|
|
219
|
+
options.health?.recordSuccess();
|
|
220
|
+
return result;
|
|
221
|
+
}
|
|
222
|
+
catch (error) {
|
|
223
|
+
const failure = error;
|
|
224
|
+
failures.push({ link, message: failure.message });
|
|
225
|
+
options.onLinkFailure?.(link, failure);
|
|
226
|
+
if (failure.kind === "daily")
|
|
227
|
+
deadProviders.add(link.provider.id);
|
|
228
|
+
// Stepping down after a size 429 reaches a model with a smaller ceiling —
|
|
229
|
+
// strictly worse. Stop, and let the caller shorten the prompt.
|
|
230
|
+
if (failure.kind === "size")
|
|
231
|
+
break;
|
|
232
|
+
}
|
|
233
|
+
}
|
|
234
|
+
const exhausted = new ChainExhaustedError(failures);
|
|
235
|
+
options.health?.recordFailure(exhausted);
|
|
236
|
+
throw exhausted;
|
|
237
|
+
}
|
package/dist/index.d.ts
CHANGED
|
@@ -34,17 +34,26 @@
|
|
|
34
34
|
* retired model id — the exact failure the `chain` and `catalog` modules exist
|
|
35
35
|
* to prevent. "Ration" described one of five modules and buried the other four.
|
|
36
36
|
*
|
|
37
|
-
*
|
|
38
|
-
*
|
|
39
|
-
*
|
|
40
|
-
*
|
|
41
|
-
*
|
|
42
|
-
*
|
|
43
|
-
*
|
|
37
|
+
* NOW INCLUDED: an HTTP client. `complete()` makes the call.
|
|
38
|
+
*
|
|
39
|
+
* The old rule was "every app has its own calling conventions, and replacing
|
|
40
|
+
* those is a rewrite rather than an adoption" — accurate about the fleet, and
|
|
41
|
+
* it protected the very duplication it described. The conventions differed
|
|
42
|
+
* because nothing ever offered to own them. Measured 2026-09-05: 8 hand-rolled
|
|
43
|
+
* clients in service, 2 of which tell the three kinds of 429 apart, while this
|
|
44
|
+
* package explained the distinction to the 2 repos that imported it.
|
|
45
|
+
* `ai-forms` — adopted by 5 — is not better code, it is code that does the job
|
|
46
|
+
* rather than advising on it. See `complete.ts` for the full argument.
|
|
47
|
+
*
|
|
48
|
+
* `tryChain` remains for a caller with a genuinely unusual request to make: it
|
|
49
|
+
* walks the chain and lets the caller keep the fetch. `complete` is the answer
|
|
50
|
+
* for everyone else.
|
|
44
51
|
*/
|
|
45
52
|
export { type Provider, type Env, type Link, type CostVerdict, providerModels, withEnvPrefix, freeChain, modelCost, modelCostAt, paidModelsIn, dayCapacityTokens, usableChain, chainFrom, } from "./chain.js";
|
|
46
53
|
export { type CatalogVerdict, type CheckCatalogOptions, checkCatalog, hasRot, deadProviders, catalogReport, } from "./catalog.js";
|
|
47
54
|
export { type ChainAttemptFailure, type TryChainOptions, ChainExhaustedError, tryChain, } from "./attempt.js";
|
|
55
|
+
export { type ChatMessage, type ToolCall, type CompleteOptions, type CompleteResult, LinkFailure, complete, linkId, } from "./complete.js";
|
|
48
56
|
export { type HealthStatus, type Health, type HealthTrackerOptions, type HealthTracker, createHealthTracker, } from "./health.js";
|
|
57
|
+
export { type LivenessResult, type LivenessOptions, type LivenessProbe, type AiHealthHandlerOptions, createLivenessProbe, createAiHealthHandler, } from "./liveness.js";
|
|
49
58
|
export { type RateLimitKind, classifyRateLimit, retryAfterSeconds, humanizeWait, rateLimitMessage, } from "./limits.js";
|
|
50
59
|
export { DAY_SECONDS, DEFAULT_BURST, type ShareInput, type ShareReason, type ShareDecision, fairShare, utcDayElapsed, utcDayKey, } from "./fair-share.js";
|
package/dist/index.js
CHANGED
|
@@ -34,18 +34,27 @@
|
|
|
34
34
|
* retired model id — the exact failure the `chain` and `catalog` modules exist
|
|
35
35
|
* to prevent. "Ration" described one of five modules and buried the other four.
|
|
36
36
|
*
|
|
37
|
-
*
|
|
38
|
-
*
|
|
39
|
-
*
|
|
40
|
-
*
|
|
41
|
-
*
|
|
42
|
-
*
|
|
43
|
-
*
|
|
37
|
+
* NOW INCLUDED: an HTTP client. `complete()` makes the call.
|
|
38
|
+
*
|
|
39
|
+
* The old rule was "every app has its own calling conventions, and replacing
|
|
40
|
+
* those is a rewrite rather than an adoption" — accurate about the fleet, and
|
|
41
|
+
* it protected the very duplication it described. The conventions differed
|
|
42
|
+
* because nothing ever offered to own them. Measured 2026-09-05: 8 hand-rolled
|
|
43
|
+
* clients in service, 2 of which tell the three kinds of 429 apart, while this
|
|
44
|
+
* package explained the distinction to the 2 repos that imported it.
|
|
45
|
+
* `ai-forms` — adopted by 5 — is not better code, it is code that does the job
|
|
46
|
+
* rather than advising on it. See `complete.ts` for the full argument.
|
|
47
|
+
*
|
|
48
|
+
* `tryChain` remains for a caller with a genuinely unusual request to make: it
|
|
49
|
+
* walks the chain and lets the caller keep the fetch. `complete` is the answer
|
|
50
|
+
* for everyone else.
|
|
44
51
|
*/
|
|
45
52
|
export { providerModels, withEnvPrefix, freeChain, modelCost, modelCostAt, paidModelsIn, dayCapacityTokens, usableChain, chainFrom, } from "./chain.js";
|
|
46
53
|
export { checkCatalog, hasRot, deadProviders, catalogReport, } from "./catalog.js";
|
|
47
54
|
export { ChainExhaustedError, tryChain, } from "./attempt.js";
|
|
55
|
+
export { LinkFailure, complete, linkId, } from "./complete.js";
|
|
48
56
|
export { createHealthTracker, } from "./health.js";
|
|
57
|
+
export { createLivenessProbe, createAiHealthHandler, } from "./liveness.js";
|
|
49
58
|
export { classifyRateLimit, retryAfterSeconds, humanizeWait, rateLimitMessage, } from "./limits.js";
|
|
50
59
|
export { DAY_SECONDS, DEFAULT_BURST, fairShare, utcDayElapsed, utcDayKey, } from "./fair-share.js";
|
|
51
60
|
// Form filling lives at `ai-kit/forms`, NOT here.
|