@12-apps/llm 0.0.0-stage → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +428 -2
- package/dist/index.d.ts +814 -0
- package/dist/index.js +1457 -0
- package/dist/index.js.map +1 -0
- package/package.json +71 -4
package/README.md
CHANGED
|
@@ -1,3 +1,429 @@
|
|
|
1
|
-
#
|
|
1
|
+
# @12-apps/llm
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
A provider-agnostic model connector. A host writes its prompt, its tools and
|
|
4
|
+
its metering once, against one interface. An adapter per provider maps that
|
|
5
|
+
interface onto the provider's official SDK and maps the answer back.
|
|
6
|
+
|
|
7
|
+
- **One interface** for a chat turn with tools, and one normalized result.
|
|
8
|
+
- **An abstract factory**, `createModelClient`, that picks the adapter by
|
|
9
|
+
provider id from a registry the host can extend.
|
|
10
|
+
- **Built-in adapters** for Anthropic (`@anthropic-ai/sdk`), OpenAI (`openai`),
|
|
11
|
+
Gemini (`@google/genai`) and xAI's Grok (over the `openai` SDK, since xAI
|
|
12
|
+
speaks OpenAI's Chat Completions). Swapping providers is a change of
|
|
13
|
+
`provider` and credentials, nothing else. The SDKs are optional peers:
|
|
14
|
+
install the ones you use. Each SDK is imported only when its adapter makes its first call.
|
|
15
|
+
- **Credential strategies**: an API key, or an OAuth access token with a
|
|
16
|
+
refresh hook, both behind one interface.
|
|
17
|
+
- **A tool-use loop** that reports every turn's usage.
|
|
18
|
+
- **Typed errors** and one retry hook the host supplies.
|
|
19
|
+
- **A support ladder**: automation, then model tiers that escalate, then a
|
|
20
|
+
person, with least-privilege tools per tier bound to the host's scope.
|
|
21
|
+
|
|
22
|
+
Only chat ships today. The `Capability` union leaves room for others (a
|
|
23
|
+
transcription capability is planned), and an adapter declares what it can do.
|
|
24
|
+
|
|
25
|
+
## Install
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
pnpm add @12-apps/llm @anthropic-ai/sdk # plus `openai` and/or `@google/genai` if you use them
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
## A chat turn
|
|
32
|
+
|
|
33
|
+
```ts
|
|
34
|
+
import { createModelClient } from "@12-apps/llm";
|
|
35
|
+
|
|
36
|
+
const client = createModelClient({
|
|
37
|
+
provider: "anthropic", // default model: claude-haiku-5-5
|
|
38
|
+
credentials: { kind: "apiKey", apiKey: () => secrets.read("ANTHROPIC_API_KEY") },
|
|
39
|
+
});
|
|
40
|
+
|
|
41
|
+
const result = await client.chat({
|
|
42
|
+
system: [
|
|
43
|
+
{ text: GENERAL_INSTRUCTIONS }, // the same for every tenant
|
|
44
|
+
{ text: renderCatalog(catalog), cache: true }, // a breakpoint: everything up to here is cached
|
|
45
|
+
],
|
|
46
|
+
messages: [{ role: "user", text: "Do you deliver on Sundays?" }],
|
|
47
|
+
tools: [searchCatalog],
|
|
48
|
+
toolChoice: "auto",
|
|
49
|
+
maxOutputTokens: 1024,
|
|
50
|
+
effort: "low",
|
|
51
|
+
});
|
|
52
|
+
|
|
53
|
+
result.text; // the reply text
|
|
54
|
+
result.toolCalls; // [{ id, name, input }], input already parsed
|
|
55
|
+
result.stopReason; // "end" | "tool_use" | "max_tokens" | "refusal" | "other"
|
|
56
|
+
result.usage; // { inputTokens, cacheCreationTokens, cacheReadTokens, outputTokens, raw }
|
|
57
|
+
result.message; // the turn as a message, ready to append to the conversation
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
### The request
|
|
61
|
+
|
|
62
|
+
| field | meaning |
|
|
63
|
+
|---|---|
|
|
64
|
+
| `system` | Ordered blocks, **most stable first**. `cache: true` (or `{ ttl: "1h" }`) puts a cache breakpoint at the end of the block. |
|
|
65
|
+
| `messages` | `user` (text), `assistant` (text, tool calls and the provider's `raw` turn) and `tool` (every result of the previous turn's calls, together). |
|
|
66
|
+
| `tools` | `{ name, description, inputSchema }`, where the schema is **strict** (below). |
|
|
67
|
+
| `toolChoice` | `auto`, `none` or `required` (at least one call). |
|
|
68
|
+
| `maxOutputTokens` | Required. |
|
|
69
|
+
| `effort` | `low`, `medium` or `high`. Each adapter maps it to its own setting. |
|
|
70
|
+
| `temperature` | A hint. Adapters whose API refuses sampling parameters drop it (Anthropic). |
|
|
71
|
+
| `cacheConversation` | Also cache the conversation so far (Anthropic). |
|
|
72
|
+
| `cacheKey` | A stable key that routes requests sharing a prefix together (OpenAI's `prompt_cache_key`). |
|
|
73
|
+
| `model` | Overrides the client's model for one call. |
|
|
74
|
+
| `signal` | An `AbortSignal`. |
|
|
75
|
+
|
|
76
|
+
### Strict tool schemas
|
|
77
|
+
|
|
78
|
+
Every provider's strict mode guarantees the model's arguments validate against
|
|
79
|
+
the schema, but each provider accepts a slightly different subset of JSON
|
|
80
|
+
Schema. The client accepts only schemas inside the subset all four share, and
|
|
81
|
+
refuses any other schema with an `InvalidRequestError` before any call:
|
|
82
|
+
|
|
83
|
+
- the root is `type: "object"`;
|
|
84
|
+
- every object sets `additionalProperties: false`;
|
|
85
|
+
- every object lists **all** its properties in `required`. An optional field is
|
|
86
|
+
a nullable type: `{ "type": ["integer", "null"] }`.
|
|
87
|
+
|
|
88
|
+
`toolSchemaProblems(tools)` returns the same findings with a path to each.
|
|
89
|
+
|
|
90
|
+
### Usage
|
|
91
|
+
|
|
92
|
+
`inputTokens` counts only the input that was neither written to nor read from
|
|
93
|
+
the cache, so the four counters add up to everything billed and each can be
|
|
94
|
+
priced at its own rate. `raw` is the provider's own usage object.
|
|
95
|
+
|
|
96
|
+
| | Anthropic | OpenAI | Gemini | xAI |
|
|
97
|
+
|---|---|---|---|---|
|
|
98
|
+
| `inputTokens` | `input_tokens` | `prompt_tokens` − cached − cache writes | `promptTokenCount` (+ tool-use prompt) − cached | same as OpenAI |
|
|
99
|
+
| `cacheCreationTokens` | `cache_creation_input_tokens` | `cache_write_tokens` | 0 | same as OpenAI |
|
|
100
|
+
| `cacheReadTokens` | `cache_read_input_tokens` | `cached_tokens` | `cachedContentTokenCount` | same as OpenAI |
|
|
101
|
+
| `outputTokens` | `output_tokens` | `completion_tokens` | `candidatesTokenCount` + `thoughtsTokenCount` | same as OpenAI |
|
|
102
|
+
|
|
103
|
+
### Prompt caching per provider
|
|
104
|
+
|
|
105
|
+
- **Anthropic**: a `cache` block gets a `cache_control` breakpoint. Tools are
|
|
106
|
+
rendered before the system prompt, so the first breakpoint caches the tool
|
|
107
|
+
list too. A request may carry at most four breakpoints, `cacheConversation`
|
|
108
|
+
included. A prefix shorter than the model's minimum cacheable length is
|
|
109
|
+
silently not cached: check `usage.cacheReadTokens` on the second call.
|
|
110
|
+
- **OpenAI** and **Gemini** cache repeated prefixes by themselves. Breakpoints
|
|
111
|
+
have nothing to map to, but the order still matters: keep the stable blocks
|
|
112
|
+
first.
|
|
113
|
+
|
|
114
|
+
### Refusals
|
|
115
|
+
|
|
116
|
+
A refusal is a **result**: `stopReason: "refusal"` with `refusal: { category,
|
|
117
|
+
explanation }` when the provider says why (Anthropic's `stop_details`, OpenAI's
|
|
118
|
+
`message.refusal` or `content_filter`, Gemini's safety finish reasons and
|
|
119
|
+
blocked prompts). `ensureNotRefused(result)` throws a `RefusalError` for code
|
|
120
|
+
that prefers an exception.
|
|
121
|
+
|
|
122
|
+
## The factory and the registry
|
|
123
|
+
|
|
124
|
+
`createModelClient({ provider, model?, credentials, retry?, registry? })`
|
|
125
|
+
looks the provider id up in `registry`. When no registry is given, it uses a
|
|
126
|
+
registry of the four built-in adapters. It refuses an unknown provider or a
|
|
127
|
+
credential kind the adapter does not declare (`UnsupportedError`). It also
|
|
128
|
+
refuses a missing model when the adapter has none (`InvalidRequestError`):
|
|
129
|
+
only the Anthropic adapter ships a default model.
|
|
130
|
+
|
|
131
|
+
### Adding an adapter
|
|
132
|
+
|
|
133
|
+
An adapter implements `ProviderAdapter`:
|
|
134
|
+
|
|
135
|
+
```ts
|
|
136
|
+
import { createAdapterRegistry, builtInAdapters, mapProviderError, type ProviderAdapter } from "@12-apps/llm";
|
|
137
|
+
|
|
138
|
+
const localModel: ProviderAdapter = {
|
|
139
|
+
id: "local",
|
|
140
|
+
defaultModel: "my-model",
|
|
141
|
+
capabilities: ["chat"],
|
|
142
|
+
credentialKinds: ["apiKey"],
|
|
143
|
+
async chat({ model, request, credential }) {
|
|
144
|
+
try {
|
|
145
|
+
const response = await myHttpClient.post("/chat", toWire(model, request), credential.secret);
|
|
146
|
+
return fromWire(response); // a ChatResult
|
|
147
|
+
} catch (error) {
|
|
148
|
+
throw mapProviderError("local", error, [credential.secret]);
|
|
149
|
+
}
|
|
150
|
+
},
|
|
151
|
+
};
|
|
152
|
+
|
|
153
|
+
const registry = createAdapterRegistry(builtInAdapters()).register(localModel);
|
|
154
|
+
const client = createModelClient({ provider: "local", credentials, registry });
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
`register(adapter, { replace: true })` replaces a built-in adapter. Use it, for
|
|
158
|
+
example, to point Anthropic at a gateway with `anthropicAdapter({ baseURL })`.
|
|
159
|
+
Each built-in adapter also takes a `client` factory (`(credential) => sdkClient`).
|
|
160
|
+
Tests use it to inject a fake SDK client.
|
|
161
|
+
|
|
162
|
+
An adapter maps both directions and nothing else. The client does the work
|
|
163
|
+
every provider shares: the strict-schema check, the credential strategy and
|
|
164
|
+
its OAuth refresh, and the retry policy. A returned assistant turn may carry
|
|
165
|
+
`raw: { provider, content }`. The same adapter replays that content verbatim
|
|
166
|
+
on the next turn, so thinking blocks and thought signatures survive a tool
|
|
167
|
+
loop. Any other adapter rebuilds the turn from `text` and `toolCalls`.
|
|
168
|
+
|
|
169
|
+
## Credentials
|
|
170
|
+
|
|
171
|
+
```ts
|
|
172
|
+
// A static secret the host supplies, read through the resolver before every call.
|
|
173
|
+
{ kind: "apiKey", apiKey: () => vault.read("provider-key") }
|
|
174
|
+
|
|
175
|
+
// An access token with an expiry, and the host's refresh.
|
|
176
|
+
{
|
|
177
|
+
kind: "oauth",
|
|
178
|
+
token: () => store.currentToken(), // { accessToken, expiresAt? (epoch ms) }
|
|
179
|
+
refresh: async (reason) => { // "expiring" | "unauthorized"
|
|
180
|
+
const next = await myOAuth.refresh();
|
|
181
|
+
await store.save(next); // the host keeps it, the package does not
|
|
182
|
+
return next;
|
|
183
|
+
},
|
|
184
|
+
refreshSkewMs: 60_000, // refresh this close to expiry (default 60 s)
|
|
185
|
+
isRevoked: () => accounts.isDisconnected(id), // optional: the host's "account disconnected" check
|
|
186
|
+
}
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
- The package **never stores, logs or returns a secret**. The credential an
|
|
190
|
+
adapter receives prints as `[credential apiKey]` and keeps its value
|
|
191
|
+
non-enumerable. Error messages are scrubbed of the secret the call used.
|
|
192
|
+
- **OAuth refresh**: a token within `refreshSkewMs` of `expiresAt` is
|
|
193
|
+
refreshed before the call. After a 401, the client refreshes **once** and
|
|
194
|
+
retries **once**. A second 401 is thrown as an `AuthError`.
|
|
195
|
+
- **A refresh is single-flight per client.** Concurrent calls that find the
|
|
196
|
+
token expiring, or that get a 401, share one `refresh()`: providers rotate
|
|
197
|
+
the refresh token on use, and two refreshes racing can revoke the login. A
|
|
198
|
+
401 on a token older than the last refresh takes that refresh's token
|
|
199
|
+
instead of rotating again.
|
|
200
|
+
- **Revoking an OAuth credential**: pass `isRevoked` and return true once the
|
|
201
|
+
account is disconnected. From then on `current()` and `afterUnauthorized()`
|
|
202
|
+
throw `CredentialRevokedError` (an `AuthError`, no status), and a refresh
|
|
203
|
+
that was in flight when the host revoked is discarded: its waiters get the
|
|
204
|
+
same error and its token is neither returned nor cached.
|
|
205
|
+
- **Which strategies a provider accepts is the adapter's declaration**:
|
|
206
|
+
Anthropic `apiKey`, `oauth`; OpenAI `apiKey`; Gemini `apiKey`, `oauth`;
|
|
207
|
+
xAI `apiKey`. The
|
|
208
|
+
package ships **no login UI and no login flow**, and it hard-codes no
|
|
209
|
+
consumer-subscription sign-in. Obtaining a token is the host's job.
|
|
210
|
+
- **A store's credential reaches its provider alone.** Left unset, the SDKs
|
|
211
|
+
read the process environment: `OPENAI_ORG_ID`, `OPENAI_PROJECT_ID`,
|
|
212
|
+
`OPENAI_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_BASE_URL`,
|
|
213
|
+
`GOOGLE_GENAI_USE_VERTEXAI`, `GOOGLE_CLOUD_PROJECT`, the `*_CUSTOM_HEADERS`
|
|
214
|
+
variables and more. On a server holding the platform's own account, a store's
|
|
215
|
+
key would then be billed to the platform's project, routed through its proxy
|
|
216
|
+
or sent beside its token. So the built-in adapters pass every such option
|
|
217
|
+
explicitly: the base URL, a `null` organisation and project, a `null` second
|
|
218
|
+
credential, and `vertexai: false`. And every request goes through a sealed
|
|
219
|
+
`fetch`, the shape of a credential gateway. It refuses any URL outside the
|
|
220
|
+
adapter's base URL, **and any path or method the adapter does not use**
|
|
221
|
+
(Anthropic `POST /v1/messages`, OpenAI and xAI `POST /chat/completions`,
|
|
222
|
+
Gemini `POST /v1[beta]/models/<model>:generateContent`; `InvalidRequestError`
|
|
223
|
+
otherwise), drops every auth and account header the SDK set, and sets only
|
|
224
|
+
the call's own credential. It never follows a redirect: a 3xx from the
|
|
225
|
+
provider is thrown as an `InvalidRequestError` carrying its status, so a key
|
|
226
|
+
header cannot ride to another origin. A `baseURL` the host passes is honoured.
|
|
227
|
+
- Gemini's SDK prefers `GOOGLE_API_KEY` or `GEMINI_API_KEY` over an OAuth
|
|
228
|
+
token, so the Gemini adapter refuses an OAuth call while either is set.
|
|
229
|
+
- **The tool loop's transcript is always a valid next request.** A turn the
|
|
230
|
+
loop ends on (`max_turns`, `max_tokens`, a refusal) keeps its text but never
|
|
231
|
+
a tool call nobody answered, and a turn with no text and no call is not
|
|
232
|
+
appended at all.
|
|
233
|
+
|
|
234
|
+
### Terms: enable a provider's OAuth only where its terms allow it
|
|
235
|
+
|
|
236
|
+
The `oauth` strategy is generic, but a provider's terms decide which tokens a
|
|
237
|
+
third-party product may use:
|
|
238
|
+
|
|
239
|
+
- **Anthropic**: the consumer terms bar using Claude Free, Pro or Max
|
|
240
|
+
subscription OAuth in a third-party product. That includes doing it through
|
|
241
|
+
the Agent SDK. Use API keys, or OAuth tokens Anthropic issues for API access
|
|
242
|
+
(for example, workload identity federation).
|
|
243
|
+
- **OpenAI**: a ChatGPT or Codex sign-in is in the same position. The OpenAI
|
|
244
|
+
adapter accepts API keys only.
|
|
245
|
+
- **Google**: Google's own OAuth for the Gemini API is a documented way to
|
|
246
|
+
authenticate, so the Gemini adapter accepts it.
|
|
247
|
+
|
|
248
|
+
A host enables a provider's OAuth strategy only where that provider's terms
|
|
249
|
+
allow it, and a tenant that brings its own credentials answers for its own
|
|
250
|
+
provider's terms.
|
|
251
|
+
|
|
252
|
+
## The tool-use loop
|
|
253
|
+
|
|
254
|
+
```ts
|
|
255
|
+
import { runToolLoop } from "@12-apps/llm";
|
|
256
|
+
|
|
257
|
+
const outcome = await runToolLoop({
|
|
258
|
+
client,
|
|
259
|
+
request, // the first turn's request
|
|
260
|
+
executors: { search_catalog: (input) => searchCatalog(input) },
|
|
261
|
+
maxTurns: 6, // default 8
|
|
262
|
+
onUsage: (usage, turn) => meter.record(usage), // once per turn, refused turns included
|
|
263
|
+
});
|
|
264
|
+
|
|
265
|
+
outcome.status; // "completed" | "refused" | "max_turns" | "max_tokens" | "stopped" | "halted"
|
|
266
|
+
outcome.messages; // the conversation with every turn and tool result appended
|
|
267
|
+
outcome.usage; // the summed usage
|
|
268
|
+
outcome.refusal; // set when status is "refused"
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
The loop runs every tool call of a turn and sends all the results back
|
|
272
|
+
together. An unknown tool, or an executor that throws, becomes an error result
|
|
273
|
+
the model can read and recover from. When the turn limit is reached, the loop
|
|
274
|
+
stops without running the last turn's calls. An executor that returns
|
|
275
|
+
`{ output, halt: true }` ends the loop with `status: "halted"` once every call
|
|
276
|
+
of that turn has its result, and no further model call is made.
|
|
277
|
+
|
|
278
|
+
## The support ladder
|
|
279
|
+
|
|
280
|
+
`createSupportLadder` routes each customer message up a ladder. Automation
|
|
281
|
+
answers what the host recognises with no model call. Anything else goes to
|
|
282
|
+
the first model tier, which answers or escalates. A tier escalates to the next
|
|
283
|
+
tier, which runs on the same message in the same call, or to a person. The
|
|
284
|
+
ladder carries no domain: tiers, tools, copy and the person are all the host's.
|
|
285
|
+
|
|
286
|
+
```ts
|
|
287
|
+
import { createSupportLadder, mcpTools } from "@12-apps/llm";
|
|
288
|
+
|
|
289
|
+
const ladder = createSupportLadder<{ accountId: string }, { locale: string }>({
|
|
290
|
+
automation: {
|
|
291
|
+
greeting: () => "Hi! Reply 1 for opening hours, 2 to reset your password.",
|
|
292
|
+
match: (message) => (message.trim() === "1" ? { reply: "We answer 9 to 5, Monday to Friday." } : null),
|
|
293
|
+
},
|
|
294
|
+
tiers: [
|
|
295
|
+
{
|
|
296
|
+
id: "triage",
|
|
297
|
+
client: { provider: "openai", model: "gpt-5-mini", credentials }, // or a ModelClient
|
|
298
|
+
instructions: "You answer simple helpdesk questions from the knowledge base.",
|
|
299
|
+
tools: [searchArticles], // read-only
|
|
300
|
+
},
|
|
301
|
+
{
|
|
302
|
+
id: "specialist",
|
|
303
|
+
client: { provider: "anthropic", model: "claude-opus-5-5", credentials },
|
|
304
|
+
instructions: "You resolve account problems, or hand them to a person.",
|
|
305
|
+
tools: [
|
|
306
|
+
searchArticles,
|
|
307
|
+
mcpTools({
|
|
308
|
+
connect: (scope) => accountServer.connectAs(scope.accountId), // the scope's own token
|
|
309
|
+
allow: ["reset_password"], // nothing else of the server's
|
|
310
|
+
}),
|
|
311
|
+
],
|
|
312
|
+
},
|
|
313
|
+
],
|
|
314
|
+
person: { handOff: ({ conversationId, reason, fromTier }) => queue.push({ conversationId, reason, fromTier }) },
|
|
315
|
+
state: {
|
|
316
|
+
get: (id) => levels.get(id),
|
|
317
|
+
// ATOMIC compare-and-set: store `to` only while the level is still `from`.
|
|
318
|
+
raise: (id, from, to) => levels.compareAndSet(id, from, to), // true when it moved
|
|
319
|
+
},
|
|
320
|
+
scopeFields: ["accountId"], // the model may never set these
|
|
321
|
+
onUsage: (tier, usage) => meter.record(tier, usage),
|
|
322
|
+
});
|
|
323
|
+
|
|
324
|
+
const result = await ladder.handle({
|
|
325
|
+
conversationId: "conv-42",
|
|
326
|
+
message: "I can't sign in",
|
|
327
|
+
transcript, // the conversation before this message
|
|
328
|
+
context: { locale: "en" },
|
|
329
|
+
scope: { accountId: "acct-7" }, // from the host's session, never from the model
|
|
330
|
+
});
|
|
331
|
+
|
|
332
|
+
switch (result.kind) {
|
|
333
|
+
case "automation": send(result.reply); break; // source: "greeting" | "match"
|
|
334
|
+
case "model": send(result.reply); break; // result.tier answered
|
|
335
|
+
case "person": notifyAgent(result); break; // fromTier, reason, handedOff
|
|
336
|
+
}
|
|
337
|
+
result.usage; // [{ tier, turns, usage }] per tier that ran, to meter each model separately
|
|
338
|
+
result.escalations; // [{ from, to, reason }]
|
|
339
|
+
```
|
|
340
|
+
|
|
341
|
+
- **Each tier answers or escalates.** Every tier gets the ladder's `escalate`
|
|
342
|
+
tool, `{ to: "next" | "person", reason }`; `"next"` is not offered on the last
|
|
343
|
+
tier. A refusal escalates with reason `"refused"`, and a turn with no words
|
|
344
|
+
with `"no_answer"`: to the next tier, or to a person from the last. A turn
|
|
345
|
+
that calls `escalate` leaves the tier: the other tools it called in that
|
|
346
|
+
turn are answered "not run" and never execute, and when it calls `escalate`
|
|
347
|
+
more than once, the FIRST call (in the model's order) decides and the rest
|
|
348
|
+
are ignored.
|
|
349
|
+
- **The next tier hears why.** The escalating tier's reason reaches the next
|
|
350
|
+
tier as a delimited USER-role note just before the customer's message
|
|
351
|
+
(`<handover from="…">…</handover>`, at most 500 characters, angle brackets
|
|
352
|
+
removed), never as a system block: it is the prior model's words, derived
|
|
353
|
+
from the customer's. The note is left out of the `messages` a result returns.
|
|
354
|
+
- **A conversation stays where it reached.** The `state` port stores the tier
|
|
355
|
+
that answered, or `"person"` (`WITH_PERSON`). The next message goes straight
|
|
356
|
+
to that tier; automation is consulted only before a conversation reached a
|
|
357
|
+
tier. Once a person has it, `handle` returns `{ kind: "person", handedOff:
|
|
358
|
+
false }` and never calls a model or `handOff` again.
|
|
359
|
+
- **Two messages in flight.** A customer can send two lines before the first
|
|
360
|
+
is answered, so the ladder never writes a level blindly. The port's one
|
|
361
|
+
write is `raise(conversationId, from, to) → boolean`, which the host makes
|
|
362
|
+
ATOMIC (a conditional `UPDATE … WHERE level = $from`, a Redis `WATCH`/Lua
|
|
363
|
+
compare-and-set, a Map on one event loop). The ladder only raises: an answer
|
|
364
|
+
from a lower tier leaves a higher stored level alone, and an answer that
|
|
365
|
+
finishes after the conversation reached a person returns `{ kind: "person",
|
|
366
|
+
handedOff: false, fromTier: null }` instead of the model's reply. Only the
|
|
367
|
+
call whose raise to `"person"` wins calls `handOff`, once, after the raise.
|
|
368
|
+
A raise that loses re-reads and retries; a port whose level keeps moving
|
|
369
|
+
is refused (`state_conflict`).
|
|
370
|
+
- **Resetting a conversation.** The host resets one by deleting its stored
|
|
371
|
+
level, so `get` returns nothing: the next message starts at automation and
|
|
372
|
+
the first tier again. The ladder has no reset of its own and never lowers a
|
|
373
|
+
level. A conversation stored on a tier id that no longer exists (a tier
|
|
374
|
+
renamed or removed) starts again at the first tier, skipping automation.
|
|
375
|
+
- **Least privilege per tier.** A tier's model sees exactly that tier's tools
|
|
376
|
+
plus `escalate`. A call to another tier's tool is answered as an unknown
|
|
377
|
+
tool and never runs.
|
|
378
|
+
- **Tools from MCP.** `mcpTools({ connect, allow, overrides? })` connects for
|
|
379
|
+
each call's scope and offers ONLY the allow-listed tools; the rest of the
|
|
380
|
+
server's list is never shown, and a call to one never reaches the server.
|
|
381
|
+
An allowed tool the server lacks is refused (`tool_missing`; the
|
|
382
|
+
conversation then goes to a person, see "A tool source that fails"). The client is
|
|
383
|
+
structural (`listTools`, `callTool`, as the official MCP SDK's `Client`), so
|
|
384
|
+
the package depends on no MCP SDK. Tool schemas must be strict; `overrides`
|
|
385
|
+
replaces a description or schema the server ships loose.
|
|
386
|
+
- **What `mcpTools` costs.** Every `handle` call, for every tier that runs,
|
|
387
|
+
calls `connect(scope)` and lists the server's tools (every page). Nothing is
|
|
388
|
+
cached inside the package, on purpose, so one scope's client never serves
|
|
389
|
+
another's. Cache on the host's side, keyed by the scope: have `connect`
|
|
390
|
+
return a client kept per scope (an LRU of connections, with the scope's own
|
|
391
|
+
credentials), and close it when the scope's session ends.
|
|
392
|
+
- **A tool source that fails.** When a source throws (`connect` fails, an
|
|
393
|
+
allowed tool is missing, a server tool clashes with `escalate` or exposes a
|
|
394
|
+
scope field), the conversation goes to a person with reason
|
|
395
|
+
`"tool_error"`, and `usage` still lists the tiers that already ran.
|
|
396
|
+
- **Scope binding.** `scope` is opaque to the ladder. Every tool's `run` gets
|
|
397
|
+
it in its context, and an MCP server gets it through the host's `connect`,
|
|
398
|
+
never through the model. For a server whose tools take the tenant as an
|
|
399
|
+
ARGUMENT, `mcpTools({ …, bindArgs: (scope, toolName) => ({ tenantId:
|
|
400
|
+
scope.tenantId }) })` merges those arguments AFTER the model's input (the
|
|
401
|
+
host's value wins), removes their names from the schema the model sees, and
|
|
402
|
+
refuses, without reaching the server, a model call that sets one. A tool
|
|
403
|
+
whose schema declares a `scopeFields` name is refused (`scope_exposed`), and a model call whose arguments carry one is
|
|
404
|
+
refused without running, so a conversation for one account cannot reach
|
|
405
|
+
another's data even when both use the same tools.
|
|
406
|
+
- **Refused configurations** throw `SupportLadderError` (an
|
|
407
|
+
`InvalidRequestError`) with a `code`: `empty_tiers`, `duplicate_tier` (a tier
|
|
408
|
+
id listed twice), `reserved_tier` (a tier named `"person"`), `tool_conflict`,
|
|
409
|
+
`tool_missing`, `scope_exposed`; and `state_conflict` when the state port's
|
|
410
|
+
level keeps changing while the ladder raises it.
|
|
411
|
+
|
|
412
|
+
## Errors and retries
|
|
413
|
+
|
|
414
|
+
| class | when |
|
|
415
|
+
|---|---|
|
|
416
|
+
| `AuthError` | 401 or 403 |
|
|
417
|
+
| `RateLimitError` | 429; `retryAfterMs` comes from `retry-after-ms` or `retry-after` |
|
|
418
|
+
| `InvalidRequestError` | 400, 404, 413 or 422, a non-strict tool schema, too many breakpoints |
|
|
419
|
+
| `ProviderUnavailableError` | 5xx, 529, 408, network failures |
|
|
420
|
+
| `InvalidResponseError` | an answer the adapter cannot read (tool arguments that are not JSON) |
|
|
421
|
+
| `RefusalError` | thrown only by `ensureNotRefused` |
|
|
422
|
+
| `UnsupportedError` | an unknown provider, an undeclared credential kind, a missing optional SDK |
|
|
423
|
+
| `SupportLadderError` | a support ladder's configuration or state refused; see `code` |
|
|
424
|
+
|
|
425
|
+
All of them extend `LlmError`, which has `kind`, `provider` and `status`. The
|
|
426
|
+
SDKs' own retries are switched off. The host's `retry` hook is the whole
|
|
427
|
+
policy: `(error, attempt) => delayMs | null`. With no hook, nothing is retried.
|
|
428
|
+
`retryTransient({ maxAttempts, baseDelayMs })` retries rate limits (using the
|
|
429
|
+
provider's hint) and outages with exponential backoff.
|