@12-apps/llm 0.0.0-stage → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,3 +1,290 @@
1
- # Temporary Holding Version
1
+ # @12-apps/llm
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ A provider-agnostic model connector. A host writes its prompt, its tools and
4
+ its metering once, against one interface. An adapter per provider maps that
5
+ interface onto the provider's official SDK and maps the answer back.
6
+
7
+ - **One interface** for a chat turn with tools, and one normalized result.
8
+ - **An abstract factory**, `createModelClient`, that picks the adapter by
9
+ provider id from a registry the host can extend.
10
+ - **Built-in adapters** for Anthropic (`@anthropic-ai/sdk`), OpenAI (`openai`),
11
+ Gemini (`@google/genai`) and xAI's Grok (over the `openai` SDK, since xAI
12
+ speaks OpenAI's Chat Completions). Swapping providers is a change of
13
+ `provider` and credentials, nothing else. The SDKs are optional peers:
14
+ install the ones you use. Each SDK is imported only when its adapter makes its first call.
15
+ - **Credential strategies**: an API key, or an OAuth access token with a
16
+ refresh hook, both behind one interface.
17
+ - **A tool-use loop** that reports every turn's usage.
18
+ - **Typed errors** and one retry hook the host supplies.
19
+
20
+ Only chat ships today. The `Capability` union leaves room for others (a
21
+ transcription capability is planned), and an adapter declares what it can do.
22
+
23
+ ## Install
24
+
25
+ ```bash
26
+ pnpm add @12-apps/llm @anthropic-ai/sdk # plus `openai` and/or `@google/genai` if you use them
27
+ ```
28
+
29
+ ## A chat turn
30
+
31
+ ```ts
32
+ import { createModelClient } from "@12-apps/llm";
33
+
34
+ const client = createModelClient({
35
+ provider: "anthropic", // default model: claude-haiku-5-5
36
+ credentials: { kind: "apiKey", apiKey: () => secrets.read("ANTHROPIC_API_KEY") },
37
+ });
38
+
39
+ const result = await client.chat({
40
+ system: [
41
+ { text: GENERAL_INSTRUCTIONS }, // the same for every tenant
42
+ { text: renderCatalog(catalog), cache: true }, // a breakpoint: everything up to here is cached
43
+ ],
44
+ messages: [{ role: "user", text: "Do you deliver on Sundays?" }],
45
+ tools: [searchCatalog],
46
+ toolChoice: "auto",
47
+ maxOutputTokens: 1024,
48
+ effort: "low",
49
+ });
50
+
51
+ result.text; // the reply text
52
+ result.toolCalls; // [{ id, name, input }], input already parsed
53
+ result.stopReason; // "end" | "tool_use" | "max_tokens" | "refusal" | "other"
54
+ result.usage; // { inputTokens, cacheCreationTokens, cacheReadTokens, outputTokens, raw }
55
+ result.message; // the turn as a message, ready to append to the conversation
56
+ ```
57
+
58
+ ### The request
59
+
60
+ | field | meaning |
61
+ |---|---|
62
+ | `system` | Ordered blocks, **most stable first**. `cache: true` (or `{ ttl: "1h" }`) puts a cache breakpoint at the end of the block. |
63
+ | `messages` | `user` (text), `assistant` (text, tool calls and the provider's `raw` turn) and `tool` (every result of the previous turn's calls, together). |
64
+ | `tools` | `{ name, description, inputSchema }`, where the schema is **strict** (below). |
65
+ | `toolChoice` | `auto`, `none` or `required` (at least one call). |
66
+ | `maxOutputTokens` | Required. |
67
+ | `effort` | `low`, `medium` or `high`. Each adapter maps it to its own setting. |
68
+ | `temperature` | A hint. Adapters whose API refuses sampling parameters drop it (Anthropic). |
69
+ | `cacheConversation` | Also cache the conversation so far (Anthropic). |
70
+ | `cacheKey` | A stable key that routes requests sharing a prefix together (OpenAI's `prompt_cache_key`). |
71
+ | `model` | Overrides the client's model for one call. |
72
+ | `signal` | An `AbortSignal`. |
73
+
74
+ ### Strict tool schemas
75
+
76
+ Every provider's strict mode guarantees the model's arguments validate against
77
+ the schema, but each provider accepts a slightly different subset of JSON
78
+ Schema. The client accepts only schemas inside the subset all four share, and
79
+ refuses any other schema with an `InvalidRequestError` before any call:
80
+
81
+ - the root is `type: "object"`;
82
+ - every object sets `additionalProperties: false`;
83
+ - every object lists **all** its properties in `required`. An optional field is
84
+ a nullable type: `{ "type": ["integer", "null"] }`.
85
+
86
+ `toolSchemaProblems(tools)` returns the same findings with a path to each.
87
+
88
+ ### Usage
89
+
90
+ `inputTokens` counts only the input that was neither written to nor read from
91
+ the cache, so the four counters add up to everything billed and each can be
92
+ priced at its own rate. `raw` is the provider's own usage object.
93
+
94
+ | | Anthropic | OpenAI | Gemini | xAI |
95
+ |---|---|---|---|---|
96
+ | `inputTokens` | `input_tokens` | `prompt_tokens` − cached − cache writes | `promptTokenCount` (+ tool-use prompt) − cached | same as OpenAI |
97
+ | `cacheCreationTokens` | `cache_creation_input_tokens` | `cache_write_tokens` | 0 | same as OpenAI |
98
+ | `cacheReadTokens` | `cache_read_input_tokens` | `cached_tokens` | `cachedContentTokenCount` | same as OpenAI |
99
+ | `outputTokens` | `output_tokens` | `completion_tokens` | `candidatesTokenCount` + `thoughtsTokenCount` | same as OpenAI |
100
+
101
+ ### Prompt caching per provider
102
+
103
+ - **Anthropic**: a `cache` block gets a `cache_control` breakpoint. Tools are
104
+ rendered before the system prompt, so the first breakpoint caches the tool
105
+ list too. A request may carry at most four breakpoints, `cacheConversation`
106
+ included. A prefix shorter than the model's minimum cacheable length is
107
+ silently not cached: check `usage.cacheReadTokens` on the second call.
108
+ - **OpenAI** and **Gemini** cache repeated prefixes by themselves. Breakpoints
109
+ have nothing to map to, but the order still matters: keep the stable blocks
110
+ first.
111
+
112
+ ### Refusals
113
+
114
+ A refusal is a **result**: `stopReason: "refusal"` with `refusal: { category,
115
+ explanation }` when the provider says why (Anthropic's `stop_details`, OpenAI's
116
+ `message.refusal` or `content_filter`, Gemini's safety finish reasons and
117
+ blocked prompts). `ensureNotRefused(result)` throws a `RefusalError` for code
118
+ that prefers an exception.
119
+
120
+ ## The factory and the registry
121
+
122
+ `createModelClient({ provider, model?, credentials, retry?, registry? })`
123
+ looks the provider id up in `registry`. When no registry is given, it uses a
124
+ registry of the four built-in adapters. It refuses an unknown provider or a
125
+ credential kind the adapter does not declare (`UnsupportedError`). It also
126
+ refuses a missing model when the adapter has none (`InvalidRequestError`):
127
+ only the Anthropic adapter ships a default model.
128
+
129
+ ### Adding an adapter
130
+
131
+ An adapter implements `ProviderAdapter`:
132
+
133
+ ```ts
134
+ import { createAdapterRegistry, builtInAdapters, mapProviderError, type ProviderAdapter } from "@12-apps/llm";
135
+
136
+ const localModel: ProviderAdapter = {
137
+ id: "local",
138
+ defaultModel: "my-model",
139
+ capabilities: ["chat"],
140
+ credentialKinds: ["apiKey"],
141
+ async chat({ model, request, credential }) {
142
+ try {
143
+ const response = await myHttpClient.post("/chat", toWire(model, request), credential.secret);
144
+ return fromWire(response); // a ChatResult
145
+ } catch (error) {
146
+ throw mapProviderError("local", error, [credential.secret]);
147
+ }
148
+ },
149
+ };
150
+
151
+ const registry = createAdapterRegistry(builtInAdapters()).register(localModel);
152
+ const client = createModelClient({ provider: "local", credentials, registry });
153
+ ```
154
+
155
+ `register(adapter, { replace: true })` replaces a built-in adapter. Use it, for
156
+ example, to point Anthropic at a gateway with `anthropicAdapter({ baseURL })`.
157
+ Each built-in adapter also takes a `client` factory (`(credential) => sdkClient`).
158
+ Tests use it to inject a fake SDK client.
159
+
160
+ An adapter maps both directions and nothing else. The client does the work
161
+ every provider shares: the strict-schema check, the credential strategy and
162
+ its OAuth refresh, and the retry policy. A returned assistant turn may carry
163
+ `raw: { provider, content }`. The same adapter replays that content verbatim
164
+ on the next turn, so thinking blocks and thought signatures survive a tool
165
+ loop. Any other adapter rebuilds the turn from `text` and `toolCalls`.
166
+
167
+ ## Credentials
168
+
169
+ ```ts
170
+ // A static secret the host supplies, read through the resolver before every call.
171
+ { kind: "apiKey", apiKey: () => vault.read("provider-key") }
172
+
173
+ // An access token with an expiry, and the host's refresh.
174
+ {
175
+ kind: "oauth",
176
+ token: () => store.currentToken(), // { accessToken, expiresAt? (epoch ms) }
177
+ refresh: async (reason) => { // "expiring" | "unauthorized"
178
+ const next = await myOAuth.refresh();
179
+ await store.save(next); // the host keeps it, the package does not
180
+ return next;
181
+ },
182
+ refreshSkewMs: 60_000, // refresh this close to expiry (default 60 s)
183
+ isRevoked: () => accounts.isDisconnected(id), // optional: the host's "account disconnected" check
184
+ }
185
+ ```
186
+
187
+ - The package **never stores, logs or returns a secret**. The credential an
188
+ adapter receives prints as `[credential apiKey]` and keeps its value
189
+ non-enumerable. Error messages are scrubbed of the secret the call used.
190
+ - **OAuth refresh**: a token within `refreshSkewMs` of `expiresAt` is
191
+ refreshed before the call. After a 401, the client refreshes **once** and
192
+ retries **once**. A second 401 is thrown as an `AuthError`.
193
+ - **A refresh is single-flight per client.** Concurrent calls that find the
194
+ token expiring, or that get a 401, share one `refresh()`: providers rotate
195
+ the refresh token on use, and two refreshes racing can revoke the login. A
196
+ 401 on a token older than the last refresh takes that refresh's token
197
+ instead of rotating again.
198
+ - **Revoking an OAuth credential**: pass `isRevoked` and return true once the
199
+ account is disconnected. From then on `current()` and `afterUnauthorized()`
200
+ throw `CredentialRevokedError` (an `AuthError`, no status), and a refresh
201
+ that was in flight when the host revoked is discarded: its waiters get the
202
+ same error and its token is neither returned nor cached.
203
+ - **Which strategies a provider accepts is the adapter's declaration**:
204
+ Anthropic `apiKey`, `oauth`; OpenAI `apiKey`; Gemini `apiKey`, `oauth`;
205
+ xAI `apiKey`. The
206
+ package ships **no login UI and no login flow**, and it hard-codes no
207
+ consumer-subscription sign-in. Obtaining a token is the host's job.
208
+ - **A store's credential reaches its provider alone.** Left unset, the SDKs
209
+ read the process environment: `OPENAI_ORG_ID`, `OPENAI_PROJECT_ID`,
210
+ `OPENAI_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_BASE_URL`,
211
+ `GOOGLE_GENAI_USE_VERTEXAI`, `GOOGLE_CLOUD_PROJECT`, the `*_CUSTOM_HEADERS`
212
+ variables and more. On a server holding the platform's own account, a store's
213
+ key would then be billed to the platform's project, routed through its proxy
214
+ or sent beside its token. So the built-in adapters pass every such option
215
+ explicitly: the base URL, a `null` organisation and project, a `null` second
216
+ credential, and `vertexai: false`. And every request goes through a sealed
217
+ `fetch`, the shape of a credential gateway. It refuses any URL outside the
218
+ adapter's base URL, **and any path or method the adapter does not use**
219
+ (Anthropic `POST /v1/messages`, OpenAI and xAI `POST /chat/completions`,
220
+ Gemini `POST /v1[beta]/models/<model>:generateContent`; `InvalidRequestError`
221
+ otherwise), drops every auth and account header the SDK set, and sets only
222
+ the call's own credential. It never follows a redirect: a 3xx from the
223
+ provider is thrown as an `InvalidRequestError` carrying its status, so a key
224
+ header cannot ride to another origin. A `baseURL` the host passes is honoured.
225
+ - Gemini's SDK prefers `GOOGLE_API_KEY` or `GEMINI_API_KEY` over an OAuth
226
+ token, so the Gemini adapter refuses an OAuth call while either is set.
227
+ - **The tool loop's transcript is always a valid next request.** A turn the
228
+ loop ends on (`max_turns`, `max_tokens`, a refusal) keeps its text but never
229
+ a tool call nobody answered, and a turn with no text and no call is not
230
+ appended at all.
231
+
232
+ ### Terms: enable a provider's OAuth only where its terms allow it
233
+
234
+ The `oauth` strategy is generic, but a provider's terms decide which tokens a
235
+ third-party product may use:
236
+
237
+ - **Anthropic**: the consumer terms bar using Claude Free, Pro or Max
238
+ subscription OAuth in a third-party product. That includes doing it through
239
+ the Agent SDK. Use API keys, or OAuth tokens Anthropic issues for API access
240
+ (for example, workload identity federation).
241
+ - **OpenAI**: a ChatGPT or Codex sign-in is in the same position. The OpenAI
242
+ adapter accepts API keys only.
243
+ - **Google**: Google's own OAuth for the Gemini API is a documented way to
244
+ authenticate, so the Gemini adapter accepts it.
245
+
246
+ A host enables a provider's OAuth strategy only where that provider's terms
247
+ allow it, and a tenant that brings its own credentials answers for its own
248
+ provider's terms.
249
+
250
+ ## The tool-use loop
251
+
252
+ ```ts
253
+ import { runToolLoop } from "@12-apps/llm";
254
+
255
+ const outcome = await runToolLoop({
256
+ client,
257
+ request, // the first turn's request
258
+ executors: { search_catalog: (input) => searchCatalog(input) },
259
+ maxTurns: 6, // default 8
260
+ onUsage: (usage, turn) => meter.record(usage), // once per turn, refused turns included
261
+ });
262
+
263
+ outcome.status; // "completed" | "refused" | "max_turns" | "max_tokens" | "stopped"
264
+ outcome.messages; // the conversation with every turn and tool result appended
265
+ outcome.usage; // the summed usage
266
+ outcome.refusal; // set when status is "refused"
267
+ ```
268
+
269
+ The loop runs every tool call of a turn and sends all the results back
270
+ together. An unknown tool, or an executor that throws, becomes an error result
271
+ the model can read and recover from. When the turn limit is reached, the loop
272
+ stops without running the last turn's calls.
273
+
274
+ ## Errors and retries
275
+
276
+ | class | when |
277
+ |---|---|
278
+ | `AuthError` | 401 or 403 |
279
+ | `RateLimitError` | 429; `retryAfterMs` comes from `retry-after-ms` or `retry-after` |
280
+ | `InvalidRequestError` | 400, 404, 413 or 422, a non-strict tool schema, too many breakpoints |
281
+ | `ProviderUnavailableError` | 5xx, 529, 408, network failures |
282
+ | `InvalidResponseError` | an answer the adapter cannot read (tool arguments that are not JSON) |
283
+ | `RefusalError` | thrown only by `ensureNotRefused` |
284
+ | `UnsupportedError` | an unknown provider, an undeclared credential kind, a missing optional SDK |
285
+
286
+ All of them extend `LlmError`, which has `kind`, `provider` and `status`. The
287
+ SDKs' own retries are switched off. The host's `retry` hook is the whole
288
+ policy: `(error, attempt) => delayMs | null`. With no hook, nothing is retried.
289
+ `retryTransient({ maxAttempts, baseDelayMs })` retries rate limits (using the
290
+ provider's hint) and outages with exponential backoff.