@12-apps/llm 0.0.0-stage → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,3 +1,429 @@
1
- # Temporary Holding Version
1
+ # @12-apps/llm
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ A provider-agnostic model connector. A host writes its prompt, its tools and
4
+ its metering once, against one interface. An adapter per provider maps that
5
+ interface onto the provider's official SDK and maps the answer back.
6
+
7
+ - **One interface** for a chat turn with tools, and one normalized result.
8
+ - **An abstract factory**, `createModelClient`, that picks the adapter by
9
+ provider id from a registry the host can extend.
10
+ - **Built-in adapters** for Anthropic (`@anthropic-ai/sdk`), OpenAI (`openai`),
11
+ Gemini (`@google/genai`) and xAI's Grok (over the `openai` SDK, since xAI
12
+ speaks OpenAI's Chat Completions). Swapping providers is a change of
13
+ `provider` and credentials, nothing else. The SDKs are optional peers:
14
+ install the ones you use. Each SDK is imported only when its adapter makes its first call.
15
+ - **Credential strategies**: an API key, or an OAuth access token with a
16
+ refresh hook, both behind one interface.
17
+ - **A tool-use loop** that reports every turn's usage.
18
+ - **Typed errors** and one retry hook the host supplies.
19
+ - **A support ladder**: automation, then model tiers that escalate, then a
20
+ person, with least-privilege tools per tier bound to the host's scope.
21
+
22
+ Only chat ships today. The `Capability` union leaves room for others (a
23
+ transcription capability is planned), and an adapter declares what it can do.
24
+
25
+ ## Install
26
+
27
+ ```bash
28
+ pnpm add @12-apps/llm @anthropic-ai/sdk # plus `openai` and/or `@google/genai` if you use them
29
+ ```
30
+
31
+ ## A chat turn
32
+
33
+ ```ts
34
+ import { createModelClient } from "@12-apps/llm";
35
+
36
+ const client = createModelClient({
37
+ provider: "anthropic", // default model: claude-haiku-5-5
38
+ credentials: { kind: "apiKey", apiKey: () => secrets.read("ANTHROPIC_API_KEY") },
39
+ });
40
+
41
+ const result = await client.chat({
42
+ system: [
43
+ { text: GENERAL_INSTRUCTIONS }, // the same for every tenant
44
+ { text: renderCatalog(catalog), cache: true }, // a breakpoint: everything up to here is cached
45
+ ],
46
+ messages: [{ role: "user", text: "Do you deliver on Sundays?" }],
47
+ tools: [searchCatalog],
48
+ toolChoice: "auto",
49
+ maxOutputTokens: 1024,
50
+ effort: "low",
51
+ });
52
+
53
+ result.text; // the reply text
54
+ result.toolCalls; // [{ id, name, input }], input already parsed
55
+ result.stopReason; // "end" | "tool_use" | "max_tokens" | "refusal" | "other"
56
+ result.usage; // { inputTokens, cacheCreationTokens, cacheReadTokens, outputTokens, raw }
57
+ result.message; // the turn as a message, ready to append to the conversation
58
+ ```
59
+
60
+ ### The request
61
+
62
+ | field | meaning |
63
+ |---|---|
64
+ | `system` | Ordered blocks, **most stable first**. `cache: true` (or `{ ttl: "1h" }`) puts a cache breakpoint at the end of the block. |
65
+ | `messages` | `user` (text), `assistant` (text, tool calls and the provider's `raw` turn) and `tool` (every result of the previous turn's calls, together). |
66
+ | `tools` | `{ name, description, inputSchema }`, where the schema is **strict** (below). |
67
+ | `toolChoice` | `auto`, `none` or `required` (at least one call). |
68
+ | `maxOutputTokens` | Required. |
69
+ | `effort` | `low`, `medium` or `high`. Each adapter maps it to its own setting. |
70
+ | `temperature` | A hint. Adapters whose API refuses sampling parameters drop it (Anthropic). |
71
+ | `cacheConversation` | Also cache the conversation so far (Anthropic). |
72
+ | `cacheKey` | A stable key that routes requests sharing a prefix together (OpenAI's `prompt_cache_key`). |
73
+ | `model` | Overrides the client's model for one call. |
74
+ | `signal` | An `AbortSignal`. |
75
+
76
+ ### Strict tool schemas
77
+
78
+ Every provider's strict mode guarantees the model's arguments validate against
79
+ the schema, but each provider accepts a slightly different subset of JSON
80
+ Schema. The client accepts only schemas inside the subset all four share, and
81
+ refuses any other schema with an `InvalidRequestError` before any call:
82
+
83
+ - the root is `type: "object"`;
84
+ - every object sets `additionalProperties: false`;
85
+ - every object lists **all** its properties in `required`. An optional field is
86
+ a nullable type: `{ "type": ["integer", "null"] }`.
87
+
88
+ `toolSchemaProblems(tools)` returns the same findings with a path to each.
89
+
90
+ ### Usage
91
+
92
+ `inputTokens` counts only the input that was neither written to nor read from
93
+ the cache, so the four counters add up to everything billed and each can be
94
+ priced at its own rate. `raw` is the provider's own usage object.
95
+
96
+ | | Anthropic | OpenAI | Gemini | xAI |
97
+ |---|---|---|---|---|
98
+ | `inputTokens` | `input_tokens` | `prompt_tokens` − cached − cache writes | `promptTokenCount` (+ tool-use prompt) − cached | same as OpenAI |
99
+ | `cacheCreationTokens` | `cache_creation_input_tokens` | `cache_write_tokens` | 0 | same as OpenAI |
100
+ | `cacheReadTokens` | `cache_read_input_tokens` | `cached_tokens` | `cachedContentTokenCount` | same as OpenAI |
101
+ | `outputTokens` | `output_tokens` | `completion_tokens` | `candidatesTokenCount` + `thoughtsTokenCount` | same as OpenAI |
102
+
103
+ ### Prompt caching per provider
104
+
105
+ - **Anthropic**: a `cache` block gets a `cache_control` breakpoint. Tools are
106
+ rendered before the system prompt, so the first breakpoint caches the tool
107
+ list too. A request may carry at most four breakpoints, `cacheConversation`
108
+ included. A prefix shorter than the model's minimum cacheable length is
109
+ silently not cached: check `usage.cacheReadTokens` on the second call.
110
+ - **OpenAI** and **Gemini** cache repeated prefixes by themselves. Breakpoints
111
+ have nothing to map to, but the order still matters: keep the stable blocks
112
+ first.
113
+
114
+ ### Refusals
115
+
116
+ A refusal is a **result**: `stopReason: "refusal"` with `refusal: { category,
117
+ explanation }` when the provider says why (Anthropic's `stop_details`, OpenAI's
118
+ `message.refusal` or `content_filter`, Gemini's safety finish reasons and
119
+ blocked prompts). `ensureNotRefused(result)` throws a `RefusalError` for code
120
+ that prefers an exception.
121
+
122
+ ## The factory and the registry
123
+
124
+ `createModelClient({ provider, model?, credentials, retry?, registry? })`
125
+ looks the provider id up in `registry`. When no registry is given, it uses a
126
+ registry of the four built-in adapters. It refuses an unknown provider or a
127
+ credential kind the adapter does not declare (`UnsupportedError`). It also
128
+ refuses a missing model when the adapter has none (`InvalidRequestError`):
129
+ only the Anthropic adapter ships a default model.
130
+
131
+ ### Adding an adapter
132
+
133
+ An adapter implements `ProviderAdapter`:
134
+
135
+ ```ts
136
+ import { createAdapterRegistry, builtInAdapters, mapProviderError, type ProviderAdapter } from "@12-apps/llm";
137
+
138
+ const localModel: ProviderAdapter = {
139
+ id: "local",
140
+ defaultModel: "my-model",
141
+ capabilities: ["chat"],
142
+ credentialKinds: ["apiKey"],
143
+ async chat({ model, request, credential }) {
144
+ try {
145
+ const response = await myHttpClient.post("/chat", toWire(model, request), credential.secret);
146
+ return fromWire(response); // a ChatResult
147
+ } catch (error) {
148
+ throw mapProviderError("local", error, [credential.secret]);
149
+ }
150
+ },
151
+ };
152
+
153
+ const registry = createAdapterRegistry(builtInAdapters()).register(localModel);
154
+ const client = createModelClient({ provider: "local", credentials, registry });
155
+ ```
156
+
157
+ `register(adapter, { replace: true })` replaces a built-in adapter. Use it, for
158
+ example, to point Anthropic at a gateway with `anthropicAdapter({ baseURL })`.
159
+ Each built-in adapter also takes a `client` factory (`(credential) => sdkClient`).
160
+ Tests use it to inject a fake SDK client.
161
+
162
+ An adapter maps both directions and nothing else. The client does the work
163
+ every provider shares: the strict-schema check, the credential strategy and
164
+ its OAuth refresh, and the retry policy. A returned assistant turn may carry
165
+ `raw: { provider, content }`. The same adapter replays that content verbatim
166
+ on the next turn, so thinking blocks and thought signatures survive a tool
167
+ loop. Any other adapter rebuilds the turn from `text` and `toolCalls`.
168
+
169
+ ## Credentials
170
+
171
+ ```ts
172
+ // A static secret the host supplies, read through the resolver before every call.
173
+ { kind: "apiKey", apiKey: () => vault.read("provider-key") }
174
+
175
+ // An access token with an expiry, and the host's refresh.
176
+ {
177
+ kind: "oauth",
178
+ token: () => store.currentToken(), // { accessToken, expiresAt? (epoch ms) }
179
+ refresh: async (reason) => { // "expiring" | "unauthorized"
180
+ const next = await myOAuth.refresh();
181
+ await store.save(next); // the host keeps it, the package does not
182
+ return next;
183
+ },
184
+ refreshSkewMs: 60_000, // refresh this close to expiry (default 60 s)
185
+ isRevoked: () => accounts.isDisconnected(id), // optional: the host's "account disconnected" check
186
+ }
187
+ ```
188
+
189
+ - The package **never stores, logs or returns a secret**. The credential an
190
+ adapter receives prints as `[credential apiKey]` and keeps its value
191
+ non-enumerable. Error messages are scrubbed of the secret the call used.
192
+ - **OAuth refresh**: a token within `refreshSkewMs` of `expiresAt` is
193
+ refreshed before the call. After a 401, the client refreshes **once** and
194
+ retries **once**. A second 401 is thrown as an `AuthError`.
195
+ - **A refresh is single-flight per client.** Concurrent calls that find the
196
+ token expiring, or that get a 401, share one `refresh()`: providers rotate
197
+ the refresh token on use, and two refreshes racing can revoke the login. A
198
+ 401 on a token older than the last refresh takes that refresh's token
199
+ instead of rotating again.
200
+ - **Revoking an OAuth credential**: pass `isRevoked` and return true once the
201
+ account is disconnected. From then on `current()` and `afterUnauthorized()`
202
+ throw `CredentialRevokedError` (an `AuthError`, no status), and a refresh
203
+ that was in flight when the host revoked is discarded: its waiters get the
204
+ same error and its token is neither returned nor cached.
205
+ - **Which strategies a provider accepts is the adapter's declaration**:
206
+ Anthropic `apiKey`, `oauth`; OpenAI `apiKey`; Gemini `apiKey`, `oauth`;
207
+ xAI `apiKey`. The
208
+ package ships **no login UI and no login flow**, and it hard-codes no
209
+ consumer-subscription sign-in. Obtaining a token is the host's job.
210
+ - **A store's credential reaches its provider alone.** Left unset, the SDKs
211
+ read the process environment: `OPENAI_ORG_ID`, `OPENAI_PROJECT_ID`,
212
+ `OPENAI_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_BASE_URL`,
213
+ `GOOGLE_GENAI_USE_VERTEXAI`, `GOOGLE_CLOUD_PROJECT`, the `*_CUSTOM_HEADERS`
214
+ variables and more. On a server holding the platform's own account, a store's
215
+ key would then be billed to the platform's project, routed through its proxy
216
+ or sent beside its token. So the built-in adapters pass every such option
217
+ explicitly: the base URL, a `null` organisation and project, a `null` second
218
+ credential, and `vertexai: false`. And every request goes through a sealed
219
+ `fetch`, the shape of a credential gateway. It refuses any URL outside the
220
+ adapter's base URL, **and any path or method the adapter does not use**
221
+ (Anthropic `POST /v1/messages`, OpenAI and xAI `POST /chat/completions`,
222
+ Gemini `POST /v1[beta]/models/<model>:generateContent`; `InvalidRequestError`
223
+ otherwise), drops every auth and account header the SDK set, and sets only
224
+ the call's own credential. It never follows a redirect: a 3xx from the
225
+ provider is thrown as an `InvalidRequestError` carrying its status, so a key
226
+ header cannot ride to another origin. A `baseURL` the host passes is honoured.
227
+ - Gemini's SDK prefers `GOOGLE_API_KEY` or `GEMINI_API_KEY` over an OAuth
228
+ token, so the Gemini adapter refuses an OAuth call while either is set.
229
+ - **The tool loop's transcript is always a valid next request.** A turn the
230
+ loop ends on (`max_turns`, `max_tokens`, a refusal) keeps its text but never
231
+ a tool call nobody answered, and a turn with no text and no call is not
232
+ appended at all.
233
+
234
+ ### Terms: enable a provider's OAuth only where its terms allow it
235
+
236
+ The `oauth` strategy is generic, but a provider's terms decide which tokens a
237
+ third-party product may use:
238
+
239
+ - **Anthropic**: the consumer terms bar using Claude Free, Pro or Max
240
+ subscription OAuth in a third-party product. That includes doing it through
241
+ the Agent SDK. Use API keys, or OAuth tokens Anthropic issues for API access
242
+ (for example, workload identity federation).
243
+ - **OpenAI**: a ChatGPT or Codex sign-in is in the same position. The OpenAI
244
+ adapter accepts API keys only.
245
+ - **Google**: Google's own OAuth for the Gemini API is a documented way to
246
+ authenticate, so the Gemini adapter accepts it.
247
+
248
+ A host enables a provider's OAuth strategy only where that provider's terms
249
+ allow it, and a tenant that brings its own credentials answers for its own
250
+ provider's terms.
251
+
252
+ ## The tool-use loop
253
+
254
+ ```ts
255
+ import { runToolLoop } from "@12-apps/llm";
256
+
257
+ const outcome = await runToolLoop({
258
+ client,
259
+ request, // the first turn's request
260
+ executors: { search_catalog: (input) => searchCatalog(input) },
261
+ maxTurns: 6, // default 8
262
+ onUsage: (usage, turn) => meter.record(usage), // once per turn, refused turns included
263
+ });
264
+
265
+ outcome.status; // "completed" | "refused" | "max_turns" | "max_tokens" | "stopped" | "halted"
266
+ outcome.messages; // the conversation with every turn and tool result appended
267
+ outcome.usage; // the summed usage
268
+ outcome.refusal; // set when status is "refused"
269
+ ```
270
+
271
+ The loop runs every tool call of a turn and sends all the results back
272
+ together. An unknown tool, or an executor that throws, becomes an error result
273
+ the model can read and recover from. When the turn limit is reached, the loop
274
+ stops without running the last turn's calls. An executor that returns
275
+ `{ output, halt: true }` ends the loop with `status: "halted"` once every call
276
+ of that turn has its result, and no further model call is made.
277
+
278
+ ## The support ladder
279
+
280
+ `createSupportLadder` routes each customer message up a ladder. Automation
281
+ answers what the host recognises with no model call. Anything else goes to
282
+ the first model tier, which answers or escalates. A tier escalates to the next
283
+ tier, which runs on the same message in the same call, or to a person. The
284
+ ladder carries no domain: tiers, tools, copy and the person are all the host's.
285
+
286
+ ```ts
287
+ import { createSupportLadder, mcpTools } from "@12-apps/llm";
288
+
289
+ const ladder = createSupportLadder<{ accountId: string }, { locale: string }>({
290
+ automation: {
291
+ greeting: () => "Hi! Reply 1 for opening hours, 2 to reset your password.",
292
+ match: (message) => (message.trim() === "1" ? { reply: "We answer 9 to 5, Monday to Friday." } : null),
293
+ },
294
+ tiers: [
295
+ {
296
+ id: "triage",
297
+ client: { provider: "openai", model: "gpt-5-mini", credentials }, // or a ModelClient
298
+ instructions: "You answer simple helpdesk questions from the knowledge base.",
299
+ tools: [searchArticles], // read-only
300
+ },
301
+ {
302
+ id: "specialist",
303
+ client: { provider: "anthropic", model: "claude-opus-5-5", credentials },
304
+ instructions: "You resolve account problems, or hand them to a person.",
305
+ tools: [
306
+ searchArticles,
307
+ mcpTools({
308
+ connect: (scope) => accountServer.connectAs(scope.accountId), // the scope's own token
309
+ allow: ["reset_password"], // nothing else of the server's
310
+ }),
311
+ ],
312
+ },
313
+ ],
314
+ person: { handOff: ({ conversationId, reason, fromTier }) => queue.push({ conversationId, reason, fromTier }) },
315
+ state: {
316
+ get: (id) => levels.get(id),
317
+ // ATOMIC compare-and-set: store `to` only while the level is still `from`.
318
+ raise: (id, from, to) => levels.compareAndSet(id, from, to), // true when it moved
319
+ },
320
+ scopeFields: ["accountId"], // the model may never set these
321
+ onUsage: (tier, usage) => meter.record(tier, usage),
322
+ });
323
+
324
+ const result = await ladder.handle({
325
+ conversationId: "conv-42",
326
+ message: "I can't sign in",
327
+ transcript, // the conversation before this message
328
+ context: { locale: "en" },
329
+ scope: { accountId: "acct-7" }, // from the host's session, never from the model
330
+ });
331
+
332
+ switch (result.kind) {
333
+ case "automation": send(result.reply); break; // source: "greeting" | "match"
334
+ case "model": send(result.reply); break; // result.tier answered
335
+ case "person": notifyAgent(result); break; // fromTier, reason, handedOff
336
+ }
337
+ result.usage; // [{ tier, turns, usage }] per tier that ran, to meter each model separately
338
+ result.escalations; // [{ from, to, reason }]
339
+ ```
340
+
341
+ - **Each tier answers or escalates.** Every tier gets the ladder's `escalate`
342
+ tool, `{ to: "next" | "person", reason }`; `"next"` is not offered on the last
343
+ tier. A refusal escalates with reason `"refused"`, and a turn with no words
344
+ with `"no_answer"`: to the next tier, or to a person from the last. A turn
345
+ that calls `escalate` leaves the tier: the other tools it called in that
346
+ turn are answered "not run" and never execute, and when it calls `escalate`
347
+ more than once, the FIRST call (in the model's order) decides and the rest
348
+ are ignored.
349
+ - **The next tier hears why.** The escalating tier's reason reaches the next
350
+ tier as a delimited USER-role note just before the customer's message
351
+ (`<handover from="…">…</handover>`, at most 500 characters, angle brackets
352
+ removed), never as a system block: it is the prior model's words, derived
353
+ from the customer's. The note is left out of the `messages` a result returns.
354
+ - **A conversation stays where it reached.** The `state` port stores the tier
355
+ that answered, or `"person"` (`WITH_PERSON`). The next message goes straight
356
+ to that tier; automation is consulted only before a conversation reached a
357
+ tier. Once a person has it, `handle` returns `{ kind: "person", handedOff:
358
+ false }` and never calls a model or `handOff` again.
359
+ - **Two messages in flight.** A customer can send two lines before the first
360
+ is answered, so the ladder never writes a level blindly. The port's one
361
+ write is `raise(conversationId, from, to) → boolean`, which the host makes
362
+ ATOMIC (a conditional `UPDATE … WHERE level = $from`, a Redis `WATCH`/Lua
363
+ compare-and-set, a Map on one event loop). The ladder only raises: an answer
364
+ from a lower tier leaves a higher stored level alone, and an answer that
365
+ finishes after the conversation reached a person returns `{ kind: "person",
366
+ handedOff: false, fromTier: null }` instead of the model's reply. Only the
367
+ call whose raise to `"person"` wins calls `handOff`, once, after the raise.
368
+ A raise that loses re-reads and retries; a port whose level keeps moving
369
+ is refused (`state_conflict`).
370
+ - **Resetting a conversation.** The host resets one by deleting its stored
371
+ level, so `get` returns nothing: the next message starts at automation and
372
+ the first tier again. The ladder has no reset of its own and never lowers a
373
+ level. A conversation stored on a tier id that no longer exists (a tier
374
+ renamed or removed) starts again at the first tier, skipping automation.
375
+ - **Least privilege per tier.** A tier's model sees exactly that tier's tools
376
+ plus `escalate`. A call to another tier's tool is answered as an unknown
377
+ tool and never runs.
378
+ - **Tools from MCP.** `mcpTools({ connect, allow, overrides? })` connects for
379
+ each call's scope and offers ONLY the allow-listed tools; the rest of the
380
+ server's list is never shown, and a call to one never reaches the server.
381
+ An allowed tool the server lacks is refused (`tool_missing`; the
382
+ conversation then goes to a person, see "A tool source that fails"). The client is
383
+ structural (`listTools`, `callTool`, as the official MCP SDK's `Client`), so
384
+ the package depends on no MCP SDK. Tool schemas must be strict; `overrides`
385
+ replaces a description or schema the server ships loose.
386
+ - **What `mcpTools` costs.** Every `handle` call, for every tier that runs,
387
+ calls `connect(scope)` and lists the server's tools (every page). Nothing is
388
+ cached inside the package, on purpose, so one scope's client never serves
389
+ another's. Cache on the host's side, keyed by the scope: have `connect`
390
+ return a client kept per scope (an LRU of connections, with the scope's own
391
+ credentials), and close it when the scope's session ends.
392
+ - **A tool source that fails.** When a source throws (`connect` fails, an
393
+ allowed tool is missing, a server tool clashes with `escalate` or exposes a
394
+ scope field), the conversation goes to a person with reason
395
+ `"tool_error"`, and `usage` still lists the tiers that already ran.
396
+ - **Scope binding.** `scope` is opaque to the ladder. Every tool's `run` gets
397
+ it in its context, and an MCP server gets it through the host's `connect`,
398
+ never through the model. For a server whose tools take the tenant as an
399
+ ARGUMENT, `mcpTools({ …, bindArgs: (scope, toolName) => ({ tenantId:
400
+ scope.tenantId }) })` merges those arguments AFTER the model's input (the
401
+ host's value wins), removes their names from the schema the model sees, and
402
+ refuses, without reaching the server, a model call that sets one. A tool
403
+ whose schema declares a `scopeFields` name is refused (`scope_exposed`), and a model call whose arguments carry one is
404
+ refused without running, so a conversation for one account cannot reach
405
+ another's data even when both use the same tools.
406
+ - **Refused configurations** throw `SupportLadderError` (an
407
+ `InvalidRequestError`) with a `code`: `empty_tiers`, `duplicate_tier` (a tier
408
+ id listed twice), `reserved_tier` (a tier named `"person"`), `tool_conflict`,
409
+ `tool_missing`, `scope_exposed`; and `state_conflict` when the state port's
410
+ level keeps changing while the ladder raises it.
411
+
412
+ ## Errors and retries
413
+
414
+ | class | when |
415
+ |---|---|
416
+ | `AuthError` | 401 or 403 |
417
+ | `RateLimitError` | 429; `retryAfterMs` comes from `retry-after-ms` or `retry-after` |
418
+ | `InvalidRequestError` | 400, 404, 413 or 422, a non-strict tool schema, too many breakpoints |
419
+ | `ProviderUnavailableError` | 5xx, 529, 408, network failures |
420
+ | `InvalidResponseError` | an answer the adapter cannot read (tool arguments that are not JSON) |
421
+ | `RefusalError` | thrown only by `ensureNotRefused` |
422
+ | `UnsupportedError` | an unknown provider, an undeclared credential kind, a missing optional SDK |
423
+ | `SupportLadderError` | a support ladder's configuration or state refused; see `code` |
424
+
425
+ All of them extend `LlmError`, which has `kind`, `provider` and `status`. The
426
+ SDKs' own retries are switched off. The host's `retry` hook is the whole
427
+ policy: `(error, attempt) => delayMs | null`. With no hook, nothing is retried.
428
+ `retryTransient({ maxAttempts, baseDelayMs })` retries rate limits (using the
429
+ provider's hint) and outages with exponential backoff.