@12-apps/llm 0.0.0-stage → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +289 -2
- package/dist/index.d.ts +586 -0
- package/dist/index.js +1115 -0
- package/dist/index.js.map +1 -0
- package/package.json +71 -4
package/README.md
CHANGED
|
@@ -1,3 +1,290 @@
|
|
|
1
|
-
#
|
|
1
|
+
# @12-apps/llm
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
A provider-agnostic model connector. A host writes its prompt, its tools and
|
|
4
|
+
its metering once, against one interface. An adapter per provider maps that
|
|
5
|
+
interface onto the provider's official SDK and maps the answer back.
|
|
6
|
+
|
|
7
|
+
- **One interface** for a chat turn with tools, and one normalized result.
|
|
8
|
+
- **An abstract factory**, `createModelClient`, that picks the adapter by
|
|
9
|
+
provider id from a registry the host can extend.
|
|
10
|
+
- **Built-in adapters** for Anthropic (`@anthropic-ai/sdk`), OpenAI (`openai`),
|
|
11
|
+
Gemini (`@google/genai`) and xAI's Grok (over the `openai` SDK, since xAI
|
|
12
|
+
speaks OpenAI's Chat Completions). Swapping providers is a change of
|
|
13
|
+
`provider` and credentials, nothing else. The SDKs are optional peers:
|
|
14
|
+
install the ones you use. Each SDK is imported only when its adapter makes its first call.
|
|
15
|
+
- **Credential strategies**: an API key, or an OAuth access token with a
|
|
16
|
+
refresh hook, both behind one interface.
|
|
17
|
+
- **A tool-use loop** that reports every turn's usage.
|
|
18
|
+
- **Typed errors** and one retry hook the host supplies.
|
|
19
|
+
|
|
20
|
+
Only chat ships today. The `Capability` union leaves room for others (a
|
|
21
|
+
transcription capability is planned), and an adapter declares what it can do.
|
|
22
|
+
|
|
23
|
+
## Install
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
pnpm add @12-apps/llm @anthropic-ai/sdk # plus `openai` and/or `@google/genai` if you use them
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
## A chat turn
|
|
30
|
+
|
|
31
|
+
```ts
|
|
32
|
+
import { createModelClient } from "@12-apps/llm";
|
|
33
|
+
|
|
34
|
+
const client = createModelClient({
|
|
35
|
+
provider: "anthropic", // default model: claude-haiku-5-5
|
|
36
|
+
credentials: { kind: "apiKey", apiKey: () => secrets.read("ANTHROPIC_API_KEY") },
|
|
37
|
+
});
|
|
38
|
+
|
|
39
|
+
const result = await client.chat({
|
|
40
|
+
system: [
|
|
41
|
+
{ text: GENERAL_INSTRUCTIONS }, // the same for every tenant
|
|
42
|
+
{ text: renderCatalog(catalog), cache: true }, // a breakpoint: everything up to here is cached
|
|
43
|
+
],
|
|
44
|
+
messages: [{ role: "user", text: "Do you deliver on Sundays?" }],
|
|
45
|
+
tools: [searchCatalog],
|
|
46
|
+
toolChoice: "auto",
|
|
47
|
+
maxOutputTokens: 1024,
|
|
48
|
+
effort: "low",
|
|
49
|
+
});
|
|
50
|
+
|
|
51
|
+
result.text; // the reply text
|
|
52
|
+
result.toolCalls; // [{ id, name, input }], input already parsed
|
|
53
|
+
result.stopReason; // "end" | "tool_use" | "max_tokens" | "refusal" | "other"
|
|
54
|
+
result.usage; // { inputTokens, cacheCreationTokens, cacheReadTokens, outputTokens, raw }
|
|
55
|
+
result.message; // the turn as a message, ready to append to the conversation
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
### The request
|
|
59
|
+
|
|
60
|
+
| field | meaning |
|
|
61
|
+
|---|---|
|
|
62
|
+
| `system` | Ordered blocks, **most stable first**. `cache: true` (or `{ ttl: "1h" }`) puts a cache breakpoint at the end of the block. |
|
|
63
|
+
| `messages` | `user` (text), `assistant` (text, tool calls and the provider's `raw` turn) and `tool` (every result of the previous turn's calls, together). |
|
|
64
|
+
| `tools` | `{ name, description, inputSchema }`, where the schema is **strict** (below). |
|
|
65
|
+
| `toolChoice` | `auto`, `none` or `required` (at least one call). |
|
|
66
|
+
| `maxOutputTokens` | Required. |
|
|
67
|
+
| `effort` | `low`, `medium` or `high`. Each adapter maps it to its own setting. |
|
|
68
|
+
| `temperature` | A hint. Adapters whose API refuses sampling parameters drop it (Anthropic). |
|
|
69
|
+
| `cacheConversation` | Also cache the conversation so far (Anthropic). |
|
|
70
|
+
| `cacheKey` | A stable key that routes requests sharing a prefix together (OpenAI's `prompt_cache_key`). |
|
|
71
|
+
| `model` | Overrides the client's model for one call. |
|
|
72
|
+
| `signal` | An `AbortSignal`. |
|
|
73
|
+
|
|
74
|
+
### Strict tool schemas
|
|
75
|
+
|
|
76
|
+
Every provider's strict mode guarantees the model's arguments validate against
|
|
77
|
+
the schema, but each provider accepts a slightly different subset of JSON
|
|
78
|
+
Schema. The client accepts only schemas inside the subset all four share, and
|
|
79
|
+
refuses any other schema with an `InvalidRequestError` before any call:
|
|
80
|
+
|
|
81
|
+
- the root is `type: "object"`;
|
|
82
|
+
- every object sets `additionalProperties: false`;
|
|
83
|
+
- every object lists **all** its properties in `required`. An optional field is
|
|
84
|
+
a nullable type: `{ "type": ["integer", "null"] }`.
|
|
85
|
+
|
|
86
|
+
`toolSchemaProblems(tools)` returns the same findings with a path to each.
|
|
87
|
+
|
|
88
|
+
### Usage
|
|
89
|
+
|
|
90
|
+
`inputTokens` counts only the input that was neither written to nor read from
|
|
91
|
+
the cache, so the four counters add up to everything billed and each can be
|
|
92
|
+
priced at its own rate. `raw` is the provider's own usage object.
|
|
93
|
+
|
|
94
|
+
| | Anthropic | OpenAI | Gemini | xAI |
|
|
95
|
+
|---|---|---|---|---|
|
|
96
|
+
| `inputTokens` | `input_tokens` | `prompt_tokens` − cached − cache writes | `promptTokenCount` (+ tool-use prompt) − cached | same as OpenAI |
|
|
97
|
+
| `cacheCreationTokens` | `cache_creation_input_tokens` | `cache_write_tokens` | 0 | same as OpenAI |
|
|
98
|
+
| `cacheReadTokens` | `cache_read_input_tokens` | `cached_tokens` | `cachedContentTokenCount` | same as OpenAI |
|
|
99
|
+
| `outputTokens` | `output_tokens` | `completion_tokens` | `candidatesTokenCount` + `thoughtsTokenCount` | same as OpenAI |
|
|
100
|
+
|
|
101
|
+
### Prompt caching per provider
|
|
102
|
+
|
|
103
|
+
- **Anthropic**: a `cache` block gets a `cache_control` breakpoint. Tools are
|
|
104
|
+
rendered before the system prompt, so the first breakpoint caches the tool
|
|
105
|
+
list too. A request may carry at most four breakpoints, `cacheConversation`
|
|
106
|
+
included. A prefix shorter than the model's minimum cacheable length is
|
|
107
|
+
silently not cached: check `usage.cacheReadTokens` on the second call.
|
|
108
|
+
- **OpenAI** and **Gemini** cache repeated prefixes by themselves. Breakpoints
|
|
109
|
+
have nothing to map to, but the order still matters: keep the stable blocks
|
|
110
|
+
first.
|
|
111
|
+
|
|
112
|
+
### Refusals
|
|
113
|
+
|
|
114
|
+
A refusal is a **result**: `stopReason: "refusal"` with `refusal: { category,
|
|
115
|
+
explanation }` when the provider says why (Anthropic's `stop_details`, OpenAI's
|
|
116
|
+
`message.refusal` or `content_filter`, Gemini's safety finish reasons and
|
|
117
|
+
blocked prompts). `ensureNotRefused(result)` throws a `RefusalError` for code
|
|
118
|
+
that prefers an exception.
|
|
119
|
+
|
|
120
|
+
## The factory and the registry
|
|
121
|
+
|
|
122
|
+
`createModelClient({ provider, model?, credentials, retry?, registry? })`
|
|
123
|
+
looks the provider id up in `registry`. When no registry is given, it uses a
|
|
124
|
+
registry of the four built-in adapters. It refuses an unknown provider or a
|
|
125
|
+
credential kind the adapter does not declare (`UnsupportedError`). It also
|
|
126
|
+
refuses a missing model when the adapter has none (`InvalidRequestError`):
|
|
127
|
+
only the Anthropic adapter ships a default model.
|
|
128
|
+
|
|
129
|
+
### Adding an adapter
|
|
130
|
+
|
|
131
|
+
An adapter implements `ProviderAdapter`:
|
|
132
|
+
|
|
133
|
+
```ts
|
|
134
|
+
import { createAdapterRegistry, builtInAdapters, mapProviderError, type ProviderAdapter } from "@12-apps/llm";
|
|
135
|
+
|
|
136
|
+
const localModel: ProviderAdapter = {
|
|
137
|
+
id: "local",
|
|
138
|
+
defaultModel: "my-model",
|
|
139
|
+
capabilities: ["chat"],
|
|
140
|
+
credentialKinds: ["apiKey"],
|
|
141
|
+
async chat({ model, request, credential }) {
|
|
142
|
+
try {
|
|
143
|
+
const response = await myHttpClient.post("/chat", toWire(model, request), credential.secret);
|
|
144
|
+
return fromWire(response); // a ChatResult
|
|
145
|
+
} catch (error) {
|
|
146
|
+
throw mapProviderError("local", error, [credential.secret]);
|
|
147
|
+
}
|
|
148
|
+
},
|
|
149
|
+
};
|
|
150
|
+
|
|
151
|
+
const registry = createAdapterRegistry(builtInAdapters()).register(localModel);
|
|
152
|
+
const client = createModelClient({ provider: "local", credentials, registry });
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
`register(adapter, { replace: true })` replaces a built-in adapter. Use it, for
|
|
156
|
+
example, to point Anthropic at a gateway with `anthropicAdapter({ baseURL })`.
|
|
157
|
+
Each built-in adapter also takes a `client` factory (`(credential) => sdkClient`).
|
|
158
|
+
Tests use it to inject a fake SDK client.
|
|
159
|
+
|
|
160
|
+
An adapter maps both directions and nothing else. The client does the work
|
|
161
|
+
every provider shares: the strict-schema check, the credential strategy and
|
|
162
|
+
its OAuth refresh, and the retry policy. A returned assistant turn may carry
|
|
163
|
+
`raw: { provider, content }`. The same adapter replays that content verbatim
|
|
164
|
+
on the next turn, so thinking blocks and thought signatures survive a tool
|
|
165
|
+
loop. Any other adapter rebuilds the turn from `text` and `toolCalls`.
|
|
166
|
+
|
|
167
|
+
## Credentials
|
|
168
|
+
|
|
169
|
+
```ts
|
|
170
|
+
// A static secret the host supplies, read through the resolver before every call.
|
|
171
|
+
{ kind: "apiKey", apiKey: () => vault.read("provider-key") }
|
|
172
|
+
|
|
173
|
+
// An access token with an expiry, and the host's refresh.
|
|
174
|
+
{
|
|
175
|
+
kind: "oauth",
|
|
176
|
+
token: () => store.currentToken(), // { accessToken, expiresAt? (epoch ms) }
|
|
177
|
+
refresh: async (reason) => { // "expiring" | "unauthorized"
|
|
178
|
+
const next = await myOAuth.refresh();
|
|
179
|
+
await store.save(next); // the host keeps it, the package does not
|
|
180
|
+
return next;
|
|
181
|
+
},
|
|
182
|
+
refreshSkewMs: 60_000, // refresh this close to expiry (default 60 s)
|
|
183
|
+
isRevoked: () => accounts.isDisconnected(id), // optional: the host's "account disconnected" check
|
|
184
|
+
}
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
- The package **never stores, logs or returns a secret**. The credential an
|
|
188
|
+
adapter receives prints as `[credential apiKey]` and keeps its value
|
|
189
|
+
non-enumerable. Error messages are scrubbed of the secret the call used.
|
|
190
|
+
- **OAuth refresh**: a token within `refreshSkewMs` of `expiresAt` is
|
|
191
|
+
refreshed before the call. After a 401, the client refreshes **once** and
|
|
192
|
+
retries **once**. A second 401 is thrown as an `AuthError`.
|
|
193
|
+
- **A refresh is single-flight per client.** Concurrent calls that find the
|
|
194
|
+
token expiring, or that get a 401, share one `refresh()`: providers rotate
|
|
195
|
+
the refresh token on use, and two refreshes racing can revoke the login. A
|
|
196
|
+
401 on a token older than the last refresh takes that refresh's token
|
|
197
|
+
instead of rotating again.
|
|
198
|
+
- **Revoking an OAuth credential**: pass `isRevoked` and return true once the
|
|
199
|
+
account is disconnected. From then on `current()` and `afterUnauthorized()`
|
|
200
|
+
throw `CredentialRevokedError` (an `AuthError`, no status), and a refresh
|
|
201
|
+
that was in flight when the host revoked is discarded: its waiters get the
|
|
202
|
+
same error and its token is neither returned nor cached.
|
|
203
|
+
- **Which strategies a provider accepts is the adapter's declaration**:
|
|
204
|
+
Anthropic `apiKey`, `oauth`; OpenAI `apiKey`; Gemini `apiKey`, `oauth`;
|
|
205
|
+
xAI `apiKey`. The
|
|
206
|
+
package ships **no login UI and no login flow**, and it hard-codes no
|
|
207
|
+
consumer-subscription sign-in. Obtaining a token is the host's job.
|
|
208
|
+
- **A store's credential reaches its provider alone.** Left unset, the SDKs
|
|
209
|
+
read the process environment: `OPENAI_ORG_ID`, `OPENAI_PROJECT_ID`,
|
|
210
|
+
`OPENAI_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_BASE_URL`,
|
|
211
|
+
`GOOGLE_GENAI_USE_VERTEXAI`, `GOOGLE_CLOUD_PROJECT`, the `*_CUSTOM_HEADERS`
|
|
212
|
+
variables and more. On a server holding the platform's own account, a store's
|
|
213
|
+
key would then be billed to the platform's project, routed through its proxy
|
|
214
|
+
or sent beside its token. So the built-in adapters pass every such option
|
|
215
|
+
explicitly: the base URL, a `null` organisation and project, a `null` second
|
|
216
|
+
credential, and `vertexai: false`. And every request goes through a sealed
|
|
217
|
+
`fetch`, the shape of a credential gateway. It refuses any URL outside the
|
|
218
|
+
adapter's base URL, **and any path or method the adapter does not use**
|
|
219
|
+
(Anthropic `POST /v1/messages`, OpenAI and xAI `POST /chat/completions`,
|
|
220
|
+
Gemini `POST /v1[beta]/models/<model>:generateContent`; `InvalidRequestError`
|
|
221
|
+
otherwise), drops every auth and account header the SDK set, and sets only
|
|
222
|
+
the call's own credential. It never follows a redirect: a 3xx from the
|
|
223
|
+
provider is thrown as an `InvalidRequestError` carrying its status, so a key
|
|
224
|
+
header cannot ride to another origin. A `baseURL` the host passes is honoured.
|
|
225
|
+
- Gemini's SDK prefers `GOOGLE_API_KEY` or `GEMINI_API_KEY` over an OAuth
|
|
226
|
+
token, so the Gemini adapter refuses an OAuth call while either is set.
|
|
227
|
+
- **The tool loop's transcript is always a valid next request.** A turn the
|
|
228
|
+
loop ends on (`max_turns`, `max_tokens`, a refusal) keeps its text but never
|
|
229
|
+
a tool call nobody answered, and a turn with no text and no call is not
|
|
230
|
+
appended at all.
|
|
231
|
+
|
|
232
|
+
### Terms: enable a provider's OAuth only where its terms allow it
|
|
233
|
+
|
|
234
|
+
The `oauth` strategy is generic, but a provider's terms decide which tokens a
|
|
235
|
+
third-party product may use:
|
|
236
|
+
|
|
237
|
+
- **Anthropic**: the consumer terms bar using Claude Free, Pro or Max
|
|
238
|
+
subscription OAuth in a third-party product. That includes doing it through
|
|
239
|
+
the Agent SDK. Use API keys, or OAuth tokens Anthropic issues for API access
|
|
240
|
+
(for example, workload identity federation).
|
|
241
|
+
- **OpenAI**: a ChatGPT or Codex sign-in is in the same position. The OpenAI
|
|
242
|
+
adapter accepts API keys only.
|
|
243
|
+
- **Google**: Google's own OAuth for the Gemini API is a documented way to
|
|
244
|
+
authenticate, so the Gemini adapter accepts it.
|
|
245
|
+
|
|
246
|
+
A host enables a provider's OAuth strategy only where that provider's terms
|
|
247
|
+
allow it, and a tenant that brings its own credentials answers for its own
|
|
248
|
+
provider's terms.
|
|
249
|
+
|
|
250
|
+
## The tool-use loop
|
|
251
|
+
|
|
252
|
+
```ts
|
|
253
|
+
import { runToolLoop } from "@12-apps/llm";
|
|
254
|
+
|
|
255
|
+
const outcome = await runToolLoop({
|
|
256
|
+
client,
|
|
257
|
+
request, // the first turn's request
|
|
258
|
+
executors: { search_catalog: (input) => searchCatalog(input) },
|
|
259
|
+
maxTurns: 6, // default 8
|
|
260
|
+
onUsage: (usage, turn) => meter.record(usage), // once per turn, refused turns included
|
|
261
|
+
});
|
|
262
|
+
|
|
263
|
+
outcome.status; // "completed" | "refused" | "max_turns" | "max_tokens" | "stopped"
|
|
264
|
+
outcome.messages; // the conversation with every turn and tool result appended
|
|
265
|
+
outcome.usage; // the summed usage
|
|
266
|
+
outcome.refusal; // set when status is "refused"
|
|
267
|
+
```
|
|
268
|
+
|
|
269
|
+
The loop runs every tool call of a turn and sends all the results back
|
|
270
|
+
together. An unknown tool, or an executor that throws, becomes an error result
|
|
271
|
+
the model can read and recover from. When the turn limit is reached, the loop
|
|
272
|
+
stops without running the last turn's calls.
|
|
273
|
+
|
|
274
|
+
## Errors and retries
|
|
275
|
+
|
|
276
|
+
| class | when |
|
|
277
|
+
|---|---|
|
|
278
|
+
| `AuthError` | 401 or 403 |
|
|
279
|
+
| `RateLimitError` | 429; `retryAfterMs` comes from `retry-after-ms` or `retry-after` |
|
|
280
|
+
| `InvalidRequestError` | 400, 404, 413 or 422, a non-strict tool schema, too many breakpoints |
|
|
281
|
+
| `ProviderUnavailableError` | 5xx, 529, 408, network failures |
|
|
282
|
+
| `InvalidResponseError` | an answer the adapter cannot read (tool arguments that are not JSON) |
|
|
283
|
+
| `RefusalError` | thrown only by `ensureNotRefused` |
|
|
284
|
+
| `UnsupportedError` | an unknown provider, an undeclared credential kind, a missing optional SDK |
|
|
285
|
+
|
|
286
|
+
All of them extend `LlmError`, which has `kind`, `provider` and `status`. The
|
|
287
|
+
SDKs' own retries are switched off. The host's `retry` hook is the whole
|
|
288
|
+
policy: `(error, attempt) => delayMs | null`. With no hook, nothing is retried.
|
|
289
|
+
`retryTransient({ maxAttempts, baseDelayMs })` retries rate limits (using the
|
|
290
|
+
provider's hint) and outages with exponential backoff.
|