@ultimat3/ai 21.0.0 → 22.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CLAUDE.md CHANGED
@@ -6,20 +6,15 @@ query jobs realtime`. **Never** `mcp manifest render pwa ui admin testing cli`.
6
6
  Declared today: `action` (the primitive `llm()` returns), `cache` (semantic cache), `core`,
7
7
  `db` (pgvector), `jobs` (`agentJob()`), `money`, `policy`, `schema`, `time`.
8
8
 
9
- **`jobs` was declared 2026-08, for `agentJob()`.** `packages/action/src/job-handle.ts`'s header
10
- already named this package as the home: `isJobHandle` needs `kind === 'job'` plus membership of a
11
- WeakMap only `job()` writes, and `action` and `jobs` are both tier 3, so the bridge has to live at
12
- tier 4+. Downward edge, nothing new in the tier table.
9
+ `jobs` is declared for `agentJob()`: the action→job bridge has to live at tier 4+.
13
10
 
14
11
  `mcp` is the same tier, so the LLM-tool projection is restated structurally in `tools.ts`
15
12
  rather than imported. Same contract, two wire formats — and the same *decision*: `toLlmTools` and
16
13
  `runLlmToolCall` ask `isMcpExposed` from `@ultimat3/core` (tier 0, reachable by both), never a
17
14
  local `=== true`. An in-app agent and an external one must be offered exactly the same tools.
18
15
 
19
- **And the same NAME**: `toLlmTool` passes `action.name` through untouched, which is the export name
20
- `@ultimat3/mcp` serves. Never derive one here. `llm()` and `agent()` return actions, so their
21
- `.tool()` name is the verbatim export name too — it read `summarize_post` / `projecting_agent`
22
- until 2026-08, naming a tool no catalog contained (`llm.test.ts`, `agent.test.ts`).
16
+ **And the same NAME**: `toLlmTool` passes `action.name` through untouched (the export name
17
+ `@ultimat3/mcp` serves); `llm()` and `agent()` `.tool()` names are verbatim too. Never derive one.
23
18
 
24
19
  ## Owns
25
20
 
@@ -68,599 +63,164 @@ until 2026-08, naming a tool no catalog contained (`llm.test.ts`, `agent.test.ts
68
63
  | `runtime.ts` | the ambient gateway / embedder / semantic caches an `llm()` reaches |
69
64
  | `fix-line.ts` / `fix-line.evals.ts` / `fix-line.v1.baseline.json` / `fix-line.eval.test.ts` | the package's own dogfood eval — the first framework-level `*.eval.test.ts`, proving the `defineEval`/baseline convention actually fails a build |
70
65
 
71
- ## Invariants
66
+ ## Invariants — bounds and budget
72
67
 
73
- - **Every numeric option in this package is screened, and `??` is not the screen — `As of
74
- 2026-08-26`.** `NaN` is not nullish, so a default never fires for it, and `Math.max`, `Math.min`
75
- and `Math.floor` all PROPAGATE it. `finiteOption`/`finiteCount` from `@ultimat3/core` are the one
76
- form (never a local copy — `scripts/flight-copies.ts` exists because four grew once), and
77
- `bun run finite-bounds` is the ratchet — it saw 21 of these sites, and the ones it CANNOT see (a
78
- required option with no `??`, a default parameter, a cross-field product like
79
- `Math.max(input.k * 4, 20)`) were the worse half. What they DID:
80
- `chunk({ size: NaN })` was a synchronous infinite loop past every `AbortSignal`;
81
- `embedBatched(…, 0)` re-issued the same empty batch to a paid endpoint forever;
82
- `hive({ concurrency: NaN })` ran `Array.from({ length: NaN })` workers, i.e. none, and returned
83
- `0 ok / 0 failed / 0 skipped` as a clean run; `RemoteEmbedder({ batchSize: NaN })` sent ONE
84
- request of zero inputs and answered zero vectors; `k1`, `k`, `rrfK` each turned a search into an
85
- empty list or an unranked one. **The worst is `maxTokens`, and it is not about the request**: it
86
- IS the pre-flight estimate, `BudgetLedger.assertScope` asks `want > remaining`, every comparison
87
- against a `NaN` is false, and `debit` then writes that `NaN` onto the ambient ledger AND the
88
- per-process `BudgetStore` — measured, a 5,000,000-token call passed a 1,000-token ceiling on the
89
- next reserve. One unscreened declaration turns every actor and org ceiling in the process off for
90
- the life of the process. Hence `Gateway.generate`/`stream` screen it at the one seam every model
91
- call passes, `registerModel` screens `maxOutput` (which reaches the same estimate through
92
- `Math.min`), and `llm()`/`agent()` screen theirs at DECLARATION, beside `respondToolFor` and the
93
- `X_AGENT_TOOL_UNEXPOSED` check, so a module-scope declaration fails the boot rather than the
94
- ninetieth second of a run.
95
- - **A bound is screened under the key the DECLARATION uses, even when that costs a second check.**
96
- `budget.tokensPerRun` reaches `BudgetLimits` as `request`, so `limitsOf` in `llm.ts`, `agent.ts`
97
- and `hive.ts` screens it before the ledger's own constructor does — a `fix:` naming a key the app
98
- never wrote is not a fix. Same reason `retrieve()` screens its own `k` rather than letting the
99
- store refuse the `k * 3` it becomes, and `RemoteEmbedder` screens `maxResponseBytes` itself
100
- rather than letting core's reader answer `readWithinLimit was given a limit of NaN` — which is
101
- correct, unactionable, and only arrives after the request has been paid for.
102
- - **`llm()` returns an `action`. It is not a ninth primitive** (root `CLAUDE.md`, 2026-08). It
103
- never re-implements parse, authz or invoke — `action()` owns those and `invoke` runs them.
104
- - `src/index.ts` re-exports `t` from `@ultimat3/schema` **verbatim**, so an `llm` file imports one
105
- package. Never wrap, spread or re-declare it: `t` delegates to `schemaProvider()` on every
106
- access, and a copy would freeze the provider at import time. `index.test.ts` asserts identity.
107
- - The model half is the only thing `llm.ts` adds: render the prompt, project `output` into the
108
- one tool the model may answer through, reserve the budget, consult the semantic cache.
109
- - One repair turn, then `X_LLM_OUTPUT_INVALID`. Two schema failures is a prompt/schema
110
- disagreement; a third attempt only spends money.
111
- - Semantic scopes are separate cache INSTANCES, never a filter over a shared one — cosine
112
- similarity has no notion of a tenant. The instance key carries the prompt hash too, which is
113
- what makes a version bump invalidate the cache.
114
- - **The instance key carries `ctx.locale` too, `As of 2026-08-24`, and it sits with the prompt hash
115
- rather than with the scope.** A prompt taking `locale` as a var — the reference app's `summarize`
116
- does, and the model obeys it — differs by ONE token between languages while carrying a whole
117
- document, so the two rendered prompts are neighbours: measured with this package's own
118
- `HashEmbedder` over that template, **0.9986**, against the declaration's `threshold: 0.97`. The
119
- Spanish summary was a hit for an English reader, and the model was never wrong — the cache was.
120
- No threshold repairs it, because the same number has to keep an honest repeat above it. It is in
121
- the UNCONDITIONAL half of the key because a `scope` answers "who may share this answer" and a
122
- locale is part of what the answer IS: an app writing `scope: () => 'global'` is saying its
123
- callers may read one another's summaries, never that a Spanish one will do. Same rule
124
- `@ultimat3/render` applies to ISR.
125
- - **The default scope is the calling ACTOR, and `scope` receives `{ input, ctx }`** (`As of
126
- 2026-08`). It defaulted to the literal string `'global'` and took the bare `input`, so the rule
127
- in the bullet above was contradicted by the very default that shipped: `cache: { semantic: { ttl:
128
- '1h' } }` put every tenant in one store, and `lookup` is a nearest neighbour with no tenant
129
- predicate — proven by execution, tenant B asking a prompt within 0.92 cosine of tenant A's
130
- received A's completion verbatim. And with only `input` to decide from, the one thing a partition
131
- may never be chosen by (a value the caller sends) was the only thing it could be chosen by, while
132
- `vars()` on the same declaration already took the pair. The default is
133
- `JSON.stringify([actor.kind, actor.id, actor.orgId ?? null])` — `@ultimat3/query`'s
134
- `readAuthority` rule, verbatim: a declaration that says nothing gets the NARROWEST key, and
135
- widening is a written statement about what the answers are (`scope: () => 'global'`). **Breaking**
136
- in both halves. `semanticCacheFor`'s instance map is bounded as a consequence
137
- (`MAX_SEMANTIC_CACHE_SCOPES`, which IS core's `MAX_CACHED_FORMATTERS` — one bounded FIFO map in
138
- the framework, whose name is about its first caller and not its contract): one entry per actor in
139
- a process that never restarts is a leak where one entry per process was not.
140
- - A per-call budget `derive`s from the ambient ledger, so it can only TIGHTEN the actor and org
141
- ceilings it runs inside. Widening them from a declaration would be a budget that is not one.
142
- **A derived ledger reports back up the chain**: every debit and every recorded cost lands on it
143
- AND on every ledger it was derived from, and `reserve` checks the `request` scope of each one.
144
- Without that link a child was a fresh counter with no parent — `llm()` derives one per call, so
145
- `gateway.spent()` answered zero after a hundred calls and a `request` ceiling of 5,000 was
146
- re-granted in full to each of them. The STORE is written once, by the ledger the call was made
147
- on: a child shares its parent's store and keys, so writing through both bills the actor twice.
148
- Reservations queue on the ROOT's turnstile for the same reason — a per-ledger queue serialises
149
- nothing once every call has its own ledger.
150
- - `cache.invalidates` from `docs/idea/05-caching.md` is **not** on `llm()` yet: `invalidateTags`
151
- fans out to `CacheTier`s, and a `SemanticCache` is not one. Storing tags nothing visits would
152
- read as wired and silently not be. Version bump + `ttl` is the invalidation today.
153
- - The gateway is ambient (`configureAi`) because a declaration is evaluated at module scope,
154
- long before a provider exists. Absent at call time is `X_AI_GATEWAY_MISSING`, never a default
155
- provider — a silent fallback would spend real money on a boot mistake.
68
+ - **Every numeric option is screened; `??` is not the screen.** `finiteOption`/`finiteCount` from
69
+ `@ultimat3/core` are the one form; `bun run finite-bounds` is the ratchet, and it cannot see a
70
+ required option with no `??`, a default parameter or a cross-field product. **`maxTokens` is the
71
+ worst**: it IS the budget's pre-flight estimate, and a `NaN` switches every actor and org ceiling
72
+ off. `Gateway.generate`/`stream` screen it at the one seam, `registerModel` screens `maxOutput`, and
73
+ `llm()`/`agent()` screen theirs at DECLARATION.
74
+ - **A bound is screened under the key the DECLARATION uses** (`budget.tokensPerRun` in `limitsOf`
75
+ before the ledger's `request`; `retrieve()`'s own `k`; `RemoteEmbedder`'s `maxResponseBytes`).
76
+ - A per-call budget `derive`s from the ambient ledger and only TIGHTENS. **A derived ledger reports
77
+ back up the chain** (every debit on every ancestor; `reserve` checks each `request` scope); the
78
+ STORE is written once, by the ledger the call was made on; reservations queue on the ROOT's turnstile.
79
+ - **`BudgetStore` is where `actor` and `org` live; the default is per PROCESS**
80
+ (`createGateway({ budgetStore })`). `add` takes a negative `tokens` for a release — a store that
81
+ clamps at zero leaks the ceiling.
82
+ - Cost is `Money` (integer minor units), rounded **up**. A budget throws `X_AI_BUDGET_EXCEEDED`
83
+ **before** the provider call. Never truncate. List price over-reserves (no introductory prices).
84
+ - **`Gateway.stream` routes BEFORE it reserves**, and `settled = true` comes after
85
+ `await ledger.record(...)`.
156
86
 
157
- - **`BudgetStore` is where `actor` and `org` actually live, and the default is per PROCESS.**
158
- `createGateway({ budgetStore })` is the one install point; omitted, it is `MemoryBudgetStore`,
159
- so an `org` ceiling is multiplied by the replica count exactly as `jobs`' `LimitConfig` is.
160
- `request` is unaffected — it is one call chain and never crosses a process. `add` takes a
161
- **negative** `tokens` (releasing an unspent reservation is a credit), so a store that clamps at
162
- zero leaks the ceiling upward on every release.
163
- - Cost is `Money` (integer minor units), rounded **up**. Never a float, never a division
164
- that loses a fraction.
165
- - **`attempt` collects TRANSPORT failures only, `As of 2026-08-23`.** A caught value that is an
166
- `UltimateError` whose code is not `X_AI_PROVIDER_UNAVAILABLE` is rethrown verbatim, on the spot:
167
- those are raised locally, before the socket opens — a missing credential, a control the model does
168
- not accept — so the same rejection is waiting on every provider and every attempt, which is
169
- exactly the reason a 400 is not retried. Collecting one flattened `X_AI_KEY_MISSING` and its
170
- runnable `export ANTHROPIC_API_KEY=…` into "no provider could serve model X", and made
171
- `generate()` and `stream()` answer the SAME misconfiguration two different ways — `stream()`
172
- never routes through `attempt`. `AiTransportError` carries `X_AI_PROVIDER_UNAVAILABLE`, so
173
- "provider one 503'd, provider two timed out" still collects across candidates unchanged.
174
- **Breaking**: a caller catching `X_AI_PROVIDER_UNAVAILABLE` today starts seeing the real code.
175
- The read is `stringField(error, 'code')`, never `error.code` — the value came from an app's
176
- `Provider` and a property read on it is a getter call or a `Proxy` trap.
177
- - **`Gateway.stream` routes BEFORE it reserves.** `providerFor` throws for a registered model no
178
- configured provider serves, and it used to run between `reserve()` and the `try` that releases —
179
- so the estimate was debited, landed on the `BudgetStore` under `actorKey`/`orgKey`, and nothing
180
- credited it back. `MemoryBudgetStore` is per process and never expires, so five refused streams
181
- spent an org's whole ceiling with nothing ever sent, and every later call in that process was
182
- `X_AI_BUDGET_EXCEEDED` forever. `settled = true` moved AFTER `await ledger.record(...)` for the
183
- same reason: a store that throws at `done` left the reservation both unreleased and half-recorded.
184
- - **The gateway's two reads of a provider's throw are total.** A `Provider` is the APP's object, so
185
- the value it rejects with is one the framework did not build: `isRetryable` indexes it (a getter,
186
- or a `Proxy` trap) and fails closed if the read raises, and the failure line goes through core's
187
- `renderThrowable` rather than `error.message` / `String(error)` — a renderer that throws replaces
188
- `X_AI_PROVIDER_UNAVAILABLE` with a bare `TypeError` nothing catches by code, and it bounds a
189
- provider's 1MB body out of the `cause`.
190
- - **`X_AI_PROVIDER_UNAVAILABLE` is classified `retryable`, and it is the only one, `As of
191
- 2026-08-23`.** `AI_ERROR_RETRY`. It rendered `retry: "terminal"` in every problem document while
192
- the gateway three lines away was backing off and trying again. `retryable`, not `retry-after`:
193
- `statedDelayMs` and `@ultimat3/http`'s `retryAfterOf` read one field, `meta.retryAfterSeconds`, and
194
- neither wire format here parses `Retry-After` off a 429 — so there is no time to name. The code's
195
- other half, "no configured provider serves this model", is an `app.config.ts` edit and carries a
196
- per-instance `retry: 'terminal'` from `AiProviderUnavailableError({ unserved: true })`; registering
197
- the code is what makes that override honoured, because core reads an instance `terminal` on an
198
- unregistered code as unclassified. Every other code is left UNREGISTERED rather than registered as
199
- terminal: `@ultimat3/jobs` dead-letters a registered `terminal` on attempt 1, which is very likely
200
- right for `X_LLM_REFUSED` and `X_AI_BUDGET_EXCEEDED` — a re-run is a second identical bill — and is
201
- a change to how every app's jobs fail, so it belongs to whoever makes it on purpose.
202
- - **The retry SCHEDULE is core's, the retry LOOP is this package's, `As of 2026-08-23`.**
203
- `backoffMs` is `@ultimat3/core`'s `backoffDelay` with `baseDelayMs`/`maxDelayMs` mapped onto
204
- `base`/`max`, and `isRetryable`'s status half is core's `isRetryableStatus`. Two behaviour changes
205
- came with it and both are pinned in `gateway-backoff.test.ts`: a delay is **rounded** where it was
206
- floored (<=1ms), and **408, 409 and 425 are now retried** where only `429 || >= 500` was. What did
207
- NOT move is `retry()`, core's executor, and the reason is the direction it fails: it stops on a
208
- `terminal` classification and retries everything else, so an UNCLASSIFIED throw is retried. Every
209
- value in this loop is an app `Provider`'s — a plain object with a `status`, never an
210
- `UltimateError` — so adopting it would have retried a 400 three times per provider, which is the
211
- one thing the loop's own comment says it must not do. `RetryPolicy` and `DEFAULT_RETRY` stay
212
- declared here for a second reason: their field names are what an app writes.
213
- - Every non-2xx and every in-band `error` frame becomes `AiTransportError`, which carries a real
214
- `status` field — that field IS the gateway's retry rule. A body parsed as a message would read
215
- as an empty, successful answer, which is the one outcome nothing downstream can detect.
216
- - **The two wire formats answer the same question the same way, and `provider-parity.test.ts` is
217
- what makes that a build error — added 2026-08.** Four rules were held by one format and not the
218
- other, all of them in the RESPONSE half, and each one is a failure that reads as a success:
219
- - an in-band `error` object in a **200 body** is `AiTransportError` on both. `parseMessage` read
220
- one as an empty `end_turn` answer while `MessageStream` — the *same provider's* streamed half —
221
- had always refused it. `throwInBandError` is `wire.ts`'s, exported, so one status table decides
222
- the gateway's retry on both transports.
223
- - a `stopDetails` of type `refusal` **forces** `stopReason: 'refusal'`, because `llm()` and
224
- `agent()` branch on the reason and nothing reads the detail. `parseStopReason` answers
225
- `end_turn` for a spelling this build has never seen, so a refusal in a new vocabulary arrived
226
- as a complete answer that happened to be empty. The OpenAI-format read always forced it.
227
- - a tool call's `input` is **parsed, never cast**: `asToolInput`, one copy. `(b['input'] ?? {}) as
228
- Record<string, unknown>` put a string under that type, and `runLlmToolCall` indexes it.
229
- - the **credential is scrubbed** out of `AiTransportError.detail` on both. `withoutKey` was the
230
- OpenAI provider's alone, so a proxy echoing `x-api-key` into its 400 body put an Anthropic key
231
- in an error — and an error reaches a log index, a span and a problem document. Both providers
232
- now call `error-body.ts`'s pair; `detailOf` moved there from `provider.ts` with it (internal,
233
- never in `src/index.ts`).
234
- What is NOT parity: the two status tables (529 vs 503 for "overloaded") and the OpenAI-format
235
- `estimatedUsage` fallback. Those are the formats differing, and the parity suite asserts them
236
- *as* differences rather than flattening them.
237
- - A stream that ends without `message_stop` throws. A truncated answer that returns `end_turn`
238
- is a confidently wrong answer with no signal, which the budget rule already forbids.
239
- - **`readSse` caps the unterminated buffer** (`MAX_FRAME_CHARS`, `As of 2026-08`) and refuses with
240
- `AiTransportError`. A peer that never sends a frame boundary — an HTML error page, a proxy on the
241
- model's port — grew it without limit and no read deadline interrupted it, because every read
242
- SUCCEEDED. Same call `@ultimat3/mail`'s `createReplyParser` makes: coded failure > OOM. `provider`
243
- is a required argument for that reason — a transport error names the endpoint it is about.
244
- - **`llm()` forwards `ctx.signal` onto `GenerateRequest`**, `As of 2026-08`, the way `agent()`
245
- always did. Without it a model call had no cancellation and no deadline: a caller that hung up
246
- left the provider call in flight, billed and unread, and the repair turn bought a second one.
247
- `.stream()` inherits it from the same `base`, which is where it matters most.
248
- - A tool call is emitted whole. `input_json_delta` fragments are not arguments until the block
249
- closes, so nothing partial reaches a caller.
250
- - Thinking chunks are never appended to `text`. A consumer concatenating every chunk must not
251
- end up shipping the reasoning to the user.
252
- - **`embedBatched` enforces the `Embedder` arity, per batch** (`As of 2026-09`). "One vector per
253
- input text, in the order the texts arrived" is the interface's own invariant and nothing
254
- downstream can restore it: `indexDocument` writes `vectors[index]` per chunk, so an app embedder
255
- that answered short stored `undefined` and surfaced a layer later as a `TypeError` inside the
256
- vector store, naming nothing an author wrote. `X_AI_EMBEDDER_INVALID` — the shipped code whose
257
- registered title already read "an Embedder returned fewer vectors than texts it was given" — now
258
- carries the two counts in its cause and `meta`. `RemoteEmbedder.decode` refuses its own provider
259
- on the same rule; an app's own `Embedder` was the unchecked half.
260
- - One `RemoteEmbedder` for every vendor: `baseUrl` selects the provider, the wire shape is the
261
- same. Vectors are L2-normalised on arrival so `cosine` stays a dot product, and a width other
262
- than the declared one is `X_VECTOR_DIM_MISMATCH` before anything reaches a store.
263
- - A budget throws `X_AI_BUDGET_EXCEEDED` **before** the provider call. Never truncate.
87
+ ## Invariants — `llm()`
88
+
89
+ - **`llm()` returns an `action`**, never a ninth primitive; it adds only the model half: render the
90
+ prompt, project `output` into the one `respond` tool, reserve the budget, consult the semantic cache.
91
+ - `src/index.ts` re-exports `t` from `@ultimat3/schema` **verbatim** (`index.test.ts`).
92
+ - One repair turn, then `X_LLM_OUTPUT_INVALID`. A refusal (a 200 with no answer) is `X_LLM_REFUSED`
93
+ before the parse, carrying `stopDetails.category`; a truncated invalid answer is `X_LLM_TRUNCATED`.
94
+ The gateway never caches a refusal.
95
+ - **Semantic scopes are separate cache INSTANCES**; the instance key carries the prompt hash AND
96
+ `ctx.locale` (in the unconditional half). **The default scope is the calling ACTOR**
97
+ (`JSON.stringify([actor.kind, actor.id, actor.orgId ?? null])`); `scope` receives
98
+ `{ input, ctx }`; widening is `scope: () => 'global'`. The instance map is bounded
99
+ (`MAX_SEMANTIC_CACHE_SCOPES`).
100
+ - `cache.invalidates` is not on `llm()`: a `SemanticCache` is not a `CacheTier`. Version bump + `ttl`
101
+ is the invalidation.
102
+ - The gateway is ambient (`configureAi`); absent at call time is `X_AI_GATEWAY_MISSING`, never a
103
+ default provider.
104
+ - **`llm()` forwards `ctx.signal`** onto `GenerateRequest`; `.stream()` inherits it.
105
+ - **`.stream()` is the SAME action**, marked with an ambient sink: it yields UNVALIDATED text and one
106
+ final `done` with the value that satisfied `output`; no repair turn (`X_LLM_STREAM_INVALID`); the
107
+ budget is reserved before the provider, debited on the first pull, reconciled at `done`; no
108
+ `respond` tool; a cache hit yields `done` alone. LAZY. `named()` is re-narrowed so the twin still streams.
109
+ - **`configureAi({ redact })` is the one seam between `vars()` and the provider**; a `Secret` in
110
+ `vars()` is `X_AI_PROMPT_SECRET` regardless. The `llm.redacted` span attribute records it.
111
+ - **Fallback is across PROVIDERS serving one model, never across models**; `GenerateResult.provider`
112
+ is stamped by the gateway and put on the span as `llm.provider`.
113
+ - Server-side `fallbacks` (beta) are never sent — the stable `2023-06-01` surface only.
114
+ - `definePrompt` refuses a re-registered version whose hash moved; the hash is over core's
115
+ `canonicalJson` (`prompt.test.ts` pins one hash literally); an absent schema hashes as `''`.
116
+
117
+ ## Invariants — providers and the gateway
118
+
119
+ - **`attempt` collects TRANSPORT failures only**: an `UltimateError` other than
120
+ `X_AI_PROVIDER_UNAVAILABLE` is rethrown on the spot (the same misconfiguration everywhere). Read
121
+ with `stringField(error, 'code')`.
122
+ - **The gateway's reads of a provider's throw are total**: `isRetryable` fails closed; the line goes
123
+ through `renderThrowable`.
124
+ - **`X_AI_PROVIDER_UNAVAILABLE` is the one code classified `retryable`** (`AI_ERROR_RETRY`);
125
+ `AiProviderUnavailableError({ unserved: true })` carries a per-instance `terminal`. Every other code
126
+ stays UNREGISTERED on purpose.
127
+ - **The retry SCHEDULE is core's (`backoffDelay`, `isRetryableStatus`); the LOOP is this package's** —
128
+ core's `retry()` retries unclassified throws, and every value here is an app `Provider`'s plain
129
+ object, so it would retry a 400. `gateway-backoff.test.ts`. `RetryPolicy`/`DEFAULT_RETRY` keep the
130
+ field names an app writes.
131
+ - Every non-2xx and every in-band `error` frame is `AiTransportError` with a real `status`.
132
+ - **Both wire formats answer the same question the same way** (`provider-parity.test.ts`): an in-band
133
+ `error` in a 200 body throws (`throwInBandError`, `wire.ts`); a `refusal` stop detail forces
134
+ `stopReason: 'refusal'`; a tool call's `input` is parsed (`asToolInput`); the credential is scrubbed
135
+ from `detail` (`error-body.ts`'s `withoutKey`/`detailOf`). The status tables and the OpenAI
136
+ `estimatedUsage` fallback are asserted as differences.
137
+ - A stream ending without `message_stop` throws. **`readSse` caps the unterminated buffer**
138
+ (`MAX_FRAME_CHARS`); `provider` is required so a transport error names its endpoint.
139
+ - A tool call is emitted whole. Thinking chunks are never appended to `text`.
264
140
  - Anthropic body: no `temperature`/`top_p`/`top_k`, no `budget_tokens`, `effort` inside
265
- `output_config`. All 400s otherwise.
266
- - **`ModelId` is `string`, and the catalogue is an open registry — decided 2026-08.** The routing
267
- seam (`Provider`, `createGateway`) was always open; the VOCABULARY was not, so a company's own
268
- gateway serving `llama-internal-70b` could not be typed and the only way past `tsc` was to claim
269
- a Claude id — after which `costOf` charged list price for a model nobody ran, `BudgetLedger`
270
- reserved against the wrong number and the manifest recorded a model the company does not use.
271
- What replaces the union as the guard is `modelSpec(id)`: an unregistered id is
272
- `X_AI_MODEL_UNKNOWN` at the first read, naming the registered set. **The three built-ins register
273
- through the same `registerModel` an app calls**, at the bottom of `models.ts`, so the default
274
- path is the app's path and there is one way to put a model in the catalogue.
275
- - **Re-registering an id REPLACES its spec and keeps its rung.** That is the negotiated-rate
276
- mechanism, and the reason there is no `overrideModel`: an app whose contract prices
277
- `claude-opus-5` below list registers it again with its own prices, and every `costOf`, every
278
- reservation and every recorded cost is that number from then on. Boot runs after this module is
279
- imported, so the app always wins.
280
- - **Registration order IS the ladder within a `family`, most capable first, and `moreCapableThan`
281
- is its only reader.** `ModelSpec.family` is what makes that true once more than one vendor's
282
- list is registered: the built-ins are `anthropic` then `openai`, so without it the rung above
283
- `gpt-5.6-sol` was `claude-haiku-4-5` — the cheapest model in the catalogue, from a vendor the
284
- app's gateway may not serve, offered as an UPGRADE in `X_LLM_REFUSED`'s fix line. Absent is its
285
- own family, so an app that registers its whole catalogue in one order still compares across all
286
- of it. A refusal is worth retrying upward and nowhere else: `MODEL_IDS.find((id) => id !==
287
- refused)` answered a refusal on the default model with the next entry DOWN, so `X_LLM_REFUSED`'s
288
- fix line told an operator to buy the same refusal from a weaker model. When there is no rung
289
- above — including for a model nobody registered — `alternative` is `undefined` and the fix line
290
- drops the suggestion rather than inventing a downgrade. A model appended after the built-ins is
291
- the least capable rung, because that is what appending to a most-capable-first list means; an app
292
- wanting its own ladder registers its whole catalogue in order.
293
- - `AnthropicProvider.models` is `ANTHROPIC_MODEL_IDS`, its OWN list, never the registry's — an
294
- app's internal model must not be routed to Anthropic. `EchoProvider.models` is a getter over the
295
- registry, because a test double has to serve whatever the test registered.
296
- - **The reasoning half of the body is PER MODEL, and `models.ts` owns which model takes what.**
297
- `output_config.effort` and adaptive thinking arrived with 4.6, so one body sent to the whole
298
- catalogue is a guaranteed 400 on the oldest entry — which is how `claude-haiku-4-5` shipped
299
- blessed and uncallable. A control the caller never asked for is omitted; a control they DID
300
- ask for is refused locally with `X_AI_REQUEST_INVALID`, never dropped, because a declaration
301
- reading `effort: 'max'` that quietly runs at the default is the failure nobody can see. Adding
302
- a model is a row in `MODELS`, never an `if` in the request builder. Omission is literal: an
303
- absent `thinking` sends no block at all, where `(thinking ?? 'adaptive')` sent an adaptive one
304
- for every adaptive-capable model — harmless on the wire, since adaptive is the server default,
305
- but it made a defaulted control indistinguishable from a declared one, which is the whole
306
- distinction this rule draws.
307
- - **`openAiProvider()` is a FORMAT, not a vendor — decided 2026-08.** Azure OpenAI, vLLM, Ollama,
308
- LiteLLM, OpenRouter, Together and most company gateways speak the OpenAI chat-completions wire
309
- format, so one provider plus `baseUrl` is what makes "point Ultimate at our internal gateway"
310
- real. Never add a second class per vendor: `baseUrl`, `auth` and `headers` are the differences.
311
- - **Structured output is the `respond` tool, never `response_format`.** `llm()` already projects
312
- `output` into one tool and reads the answer out of `toolCalls`; `json_schema` + `strict` would
313
- be a SECOND structured-output path (axiom 1), would need the provider to synthesise a
314
- `respond` call out of a content string, and is the one feature most OpenAI-*compatible* servers
315
- do not implement. Forcing the function is what buys the reliability instead: `tool_choice`
316
- names the tool when the request offers **exactly one**, which is precisely `llm()`'s shape and
317
- never `agent()`'s — forcing a name inside a tool loop decides the model's next step for it.
318
- - **`strict: true` is derived from the schema, never forwarded.** `LlmTool.strict` is `true` on
319
- every projection; on this wire the server CHECKS it, and one optional field (a key in
320
- `properties` absent from `required`) is a 400. `satisfiesStrictMode` is the gate, recursive
321
- because the server's check is.
322
- - `max_completion_tokens`, never `max_tokens`: the old field is rejected outright by every
323
- current reasoning model. No `temperature`/`top_p`, same rule as the Anthropic body.
324
- - **`stream_options: { include_usage: true }` on every streamed request, and an ESTIMATE when
325
- usage never arrives.** Usage comes once, in a trailing chunk with an empty `choices` array, and
326
- only when that field was sent — a compatible server that ignores it would otherwise leave the
327
- budget reconciling a real call against zero, refunding the whole reservation. Zero is wrong by
328
- all of it; an estimate is wrong by a few percent in the safe direction.
329
- - `prompt_tokens` INCLUDES the cached prefix here, where Anthropic's `input_tokens` excludes it.
330
- Subtract `prompt_tokens_details.cached_tokens` out of the input count or the cached half is
331
- billed twice — once at the input rate, once at the cache rate.
332
- - Tool-call deltas are merged by `tool_calls[].index`. The id and the name arrive on the FIRST
333
- fragment and on no other, so merging by array position builds one call per chunk and keeps only
334
- the last slice of arguments. The call is emitted whole at the finish reason — there is no
335
- per-block stop event in this format.
336
- - `isComplete()` accepts `[DONE]` **or** a finish reason: plenty of servers in the family close
337
- the socket straight after the finish chunk, and a finish reason is the model saying why it
338
- stopped, which a cut connection cannot produce. **With one exception, `As of 2026-08`:
339
- `[DONE]` while tool-call fragments are still pending is NOT complete.** `onFinish` is the only
340
- drain of `pending` and the finish reason is the only close this format has, so the sentinel
341
- alone cannot tell "the model finished asking" from "the connection died mid-arguments" —
342
- reporting complete discarded a whole tool call and answered an empty, successful `end_turn`.
343
- Refused rather than flushed, exactly as the Anthropic half refuses a missing `message_stop`:
344
- emitting the fragments would run a tool's side effects from half a JSON object.
345
- - `role: 'system'`, not `developer` — the newer role is OpenAI's alone and every other server in
346
- the family knows only `system`.
347
- - **Only three models are priced** (`gpt-5.6-sol` / `-terra` / `-luna`, list price read
348
- 2026-08-16). `gpt-4o` and the `o1` family cache at 0.5x input where `costOf` assumes 0.1x, and
349
- the `pro` tiers publish no cached rate — a wrong price is worse than a missing one, because
350
- `costOf` answers confidently either way and `X_AI_MODEL_UNKNOWN` at least says so.
351
- - The specs register at module scope, like the Anthropic three — so a suite that calls
352
- `resetModels()` drops them. `registerOpenAiModels()` is exported for exactly that, and every
353
- `openai-*.test.ts` calls it in `beforeEach`.
354
- - **No new `X_*` code.** A non-2xx and an in-band `error` object are `AiTransportError`, a missing
355
- key is `X_AI_KEY_MISSING`, a control the endpoint has not got is `X_AI_REQUEST_INVALID` — the
356
- failures are the same failures, and a second code per provider would be a vocabulary that grows
357
- with the driver list. What DID change: `AiTransportError` now takes the provider's `envVar`,
358
- because the 401 fix line was a hardcoded `ANTHROPIC_API_KEY` for every provider in the package.
359
- - The key is revealed as late as possible, never stored on the instance, and scrubbed out of the
360
- error `detail` — a proxy echoing request headers into its 4xx body is the one path by which a
361
- key reaches an error, and an error reaches a log index, a span and a problem document.
362
- - Model IDs are exact aliases. Never append a date suffix.
363
- - **No fix line may name `x ai`.** That command is PLANNED and throws (`packages/cli/src/cmd-planned.ts`),
364
- so a fix citing `x ai reindex` sends an operator to a wall — an axiom-4 violation. Two shipped;
365
- both now name the app-code fix instead.
366
- - **An eval is selected with `x test eval --filter <name>`, never `x test <name>`.** `x test`'s
367
- positional is a `TestType` (`unit contract live job e2e eval`), so the eval's own name there is
368
- `X_CLI_BAD_FLAG`. `X_EVAL_THRESHOLD` shipped that fix line until 2026-08; `eval-errors.test.ts`
369
- now asserts every `x test <word>` these five classes emit is one of the six types. The
370
- `errors` step's `fix-command.ts` resolves the *command*, not its positional, so nothing else
371
- would have caught it.
372
- - The introductory price on a model is deliberately not modelled. A price that lapses on a date
373
- makes a recorded cost depend on when it was read, and under-reporting spend after the lapse is
374
- a budget that is not one. List price over-reserves, which is the safe direction.
375
- - `generate()` above `STREAM_ONLY_MAX_TOKENS` runs the STREAMING transport and assembles the
376
- result, rather than refusing. The ceiling is the transport's, not the model's.
377
- - **`llm()` streams through `.stream()`, and it is the SAME action — decided 2026-08.** The
378
- invocation is an ordinary one, marked with an ambient sink; policy, input parse, budget scope,
379
- semantic cache, span, audit and `.tool()` all still apply, because there is no second execution
380
- path. Before it existed, the first feature needing tokens on a screen called `aiGateway()`
381
- directly and lost every one of them.
382
- - **Output schema:** a schema cannot be checked until the last token lands, so a stream yields
383
- UNVALIDATED text and one final `done` carrying the value that DID satisfy `output`. **No repair
384
- turn** — the consumer has already read the tokens, and a second answer over the top is two
385
- answers to one question. One attempt, then `X_LLM_STREAM_INVALID`, whose fix is the
386
- non-streaming call.
387
- - **Budget:** unchanged and still reserved before the provider is touched. `Gateway.stream`
388
- debits the worst-case estimate on the first pull and reconciles at `done`, releasing in a
389
- `finally`. The whole stream is driven inside the handler, so reservation and reconciliation sit
390
- on one async chain; abandoning the iterator stops delivery, never the accounting.
391
- - A streamed call offers **no `respond` tool**: a tool call is emitted whole, so forcing one
392
- leaves nothing to stream. The answer is prose, and its JSON parse is what a non-string `output`
393
- validates. A semantic-cache hit yields `done` alone, with no text increments.
394
- - `.stream()` is LAZY. Nothing is authorised, budgeted or sent until the first pull. `named()` is
395
- re-narrowed for the same reason `stream` is assigned in place: `action()`'s `named` builds a
396
- fresh twin that would silently not stream.
397
- - **`agent()` is a job for the tool loop, and one of the factory rule's instances** — the list is
398
- `PRIMITIVE_FACTORIES` in `@ultimat3/core`, never an ordinal in a header — it returns an `action`,
399
- never a ninth primitive. It exists because
400
- the alternative is a hand-rolled loop, and a hand-rolled loop is where the dangerous mistake
401
- lives: **taking the actor from the model's output.** `ctx.actor` is read once and is the only
402
- identity any tool runs as; nothing the model emits can reach it. Bounded by `maxTurns`
403
- (`X_AGENT_MAX_TURNS`, never a partial answer), by `budget.tokensPerRun` (the ledger's `request`
404
- scope, which accumulates across turns) and by `maxToolResultChars` — the transcript IS the
405
- request, so an untruncated tool result is re-billed once per remaining turn.
406
- - **`tools` takes the real `action()`, adapted at this package's edge — decided 2026-08 (issue
407
- #124).** It took `ProjectableAction` alone, which no `action()` structurally satisfies (an
408
- action is `as`/`tool`/`openapi`/`job`/`contract` and the callable, never `run`), so the shape
409
- the README documents was a `TS2741` and every test here hand-built a stand-in — which is why
410
- the suite was green over an API that did not compile. `AgentTool = AnyAction |
411
- ProjectableAction` and `asProjectableAction` are the fix, the same union `@ultimat3/mcp`'s
412
- `ListedPrimitive` already accepted. **Not** a `run` member on the action facade, which was the
413
- obvious alternative and is wrong three times over: it duplicates `.as()` (axiom 1), it would
414
- offer a tool named `''` for an action `registerActions` has not named yet, and it would NOT
415
- collapse `mcp`'s adapter, whose `toWireSchema` narrows to the subset that server's arg
416
- validator can enforce while this one publishes the Messages API's (`packages/mcp/src/
417
- from-action.ts` header). Two wire formats, two projections, one `invoke`.
418
- - Projection happens on the FIRST RUN, memoised — never at declaration. `agent()` is evaluated at
419
- module scope and `registerAction` stamps a name at boot, so `actionName()` at declaration would
420
- make the ordinary `export const publishPost = action(...)` beside it `X_ACTION_UNREGISTERED`.
421
- - **An `agent()` is a tool of another `agent()`** — it returns an action, and an action is what a
422
- tool is. That is the supervisor/sub-agent shape, with no `hive()` and no ninth primitive; the
423
- sub-agent runs under the same actor, through its own policy.
424
- - A tool listed in `agent({ tools })` that is not `mcp: { expose: true }` is
425
- `X_AGENT_TOOL_UNEXPOSED` **at declaration**, not filtered at the call: a silently dropped tool
426
- reads as offered and is not. `isMcpExposed` is the one predicate, so an in-app agent and an
427
- external MCP client see the same catalogue. Asked of the DECLARATION, not of the projection,
428
- because exposure needs no name and the name does not exist yet.
429
- - **`ctx.signal` is read, at three points.** `throwIfAborted` at the top of every turn and before
430
- every tool batch, and `GenerateRequest.signal` forwarded into `fetch` by both providers. The
431
- transcript IS the request, so a loop that keeps going after the caller disconnects re-sends it
432
- once per remaining turn, runs every remaining tool's side effect and discards the answer —
433
- eight provider calls for an answer nothing reads. `X_ABORTED` is core's, already shipped; no
434
- new code. `signal` is deliberately absent from `cacheKeyFor` and from every estimate: it says
435
- whether a call was ABANDONED, never what it asked for.
436
- - **The tools of ONE turn run concurrently**, unbounded within the turn, results paired by index
437
- so a fast tool cannot be matched to a slow tool's `tool_use` id. Serial cost 5x wall clock for a
438
- turn that asked for five tools and nothing in the types or the docs said so. No second ceiling
439
- here: the batch is what one model turn asked for, each entry is an action with its own `policy`
440
- and `rateLimit`, and a tool that calls a model still queues on the ledger's root turnstile.
441
- Guarantee is "no tool STARTS after the abort" — a tool already in flight unwinds through its
442
- own handler's reading of `ctx.signal`, which is the action's to make.
443
- - **`onTurn` is the per-turn observation, and `.stream()` on `agent()` is deliberately not shipped
444
- yet.** A 90-second multi-turn run emitted nothing until it returned. `onTurn` reports facts
445
- only — turn, model, tool names, stop reason, usage, that turn's cost — never the transcript and
446
- never the actor, and the same facts land on the span as an `agent.turn` event so a run
447
- declaring no hook is still readable in a trace. A throw from it FAILS the run: an observer that
448
- quietly stopped working reads exactly like one that is fine. Tokens on a screen is a different
449
- contract from turns in a loop, and half-shipping it would be the second path axiom 1 refuses.
450
- - **No semantic cache on `agent()`.** Similar prompts do not have similar answers once the answer
451
- depends on what `lookupOrder` returned this second.
452
- - **Every `tool_use` block the transcript replays is answered by a `tool_result` in the very next
453
- message** — the Messages API's own rule, and `agent-transcript.ts` owns both halves of it for
454
- that reason. Two paths broke it, both through `respond`, which is filtered out of the calls
455
- that RUN and replayed like any other block: a turn emitting a tool call AND `respond` together
456
- (ordinary parallel tool use), and a `respond` whose input failed the output schema, followed by
457
- a plain user message. Both were a 400 (`tool_use ids were found without tool_result blocks`),
458
- i.e. `X_AI_PROVIDER_UNAVAILABLE` in place of a completed run, and `agent.test.ts` never mixed
459
- the two so neither shipped visible. An unaccepted `respond` now comes back as its own
460
- `tool_result`, `is_error`, saying why — the answer is SUPERSEDED, not wrong: it was written
461
- before the results of the tools the same turn asked for existed, so the loop continues and the
462
- model answers again with them in hand. Discarding the block instead would have been the other
463
- legal fix and loses the record; USING the speculative answer would skip the tool results the
464
- model itself asked for, after those tools already ran.
465
- - **A tool result is rendered totally.** `runLlmToolCall` returns `content: string` and the loop
466
- TRUNCATES it, so `JSON.stringify`'s other two answers both have to be handled: `undefined` for
467
- an action that returns nothing (`'null'`), and a throw on a bigint, a cycle or a `toJSON` of
468
- the value's own — reported as "the tool ran and its result is not JSON", never as a failure,
469
- because a model told the tool failed calls it again and buys its side effects twice. The throw
470
- it catches is read with `stringField`, never `typeof error.code === 'string'`: the value is an
471
- app's, so the probe is a getter call or a `Proxy` trap inside the catch block.
472
- - `AiMessage.content` widened to `string | readonly AiContentBlock[]` for this: a `tool_result`
473
- has to name the `tool_use` it answers and a string has nowhere to put the id. The block field
474
- names are the Messages API's, so `body()` passes them through untouched.
475
- - **`hive()` is a fan-out, and another of the factory rule's instances** (`PRIMITIVE_FACTORIES`
476
- again) — it returns an `action`, never a ninth primitive. It exists because
477
- the alternative is a hand-rolled `Promise.all` over `agent()` calls, and that loop gets four
478
- things wrong every time: the actor, the order, the difference between ran-and-failed and
479
- never-ran, and the ceiling.
480
- - **`HiveResult` is a SCHEMA, not an interface**, built from the member action's own `output` and
481
- embedded in the `ok` arm. That is what makes a hive project to OpenAPI, the typed client, the
482
- MCP `outputSchema` and the manifest like any other action; a hand-written interface would have
483
- given the type and none of the six projections, which is the whole reason it is a factory.
484
- - **Three arms — `ok` / `failed` / `skipped` — never two.** A member that ran and threw and a
485
- member that never ran are different facts, and an aborted sibling is the second. Two arms make
486
- "the hive stopped early" indistinguishable from "every remaining item is bad data", which is
487
- the difference between retrying the tail and fixing the source. `skipped` gets its own counter
488
- beside `ok` and `failed` for the same reason: three arms and two counters means every caller
489
- writes `members.length - ok - failed` once, and writes it wrong once.
490
- - **`members` is in SPLIT order, always**, filled by index rather than pushed on settle, with
491
- `index` on every arm so a caller can join a result back to its row without depending on array
492
- position surviving a filter.
493
- - **The hive never names an actor.** `split` derives member inputs from `input` and `ctx` and
494
- from nothing a model emitted; each member runs through its own callable, so `invoke` applies
495
- the member's own `policy` with `ctx.actor` untouched. There is no `as(actor)` in the factory —
496
- the same boundary `agent()` holds, and the reason both belong in the framework.
497
- - **No hive-specific budget code, deliberately.** One derived ledger for the run, `withBudget`
498
- around the pool, each member's `agent()` deriving again — and the ceiling holds under
499
- parallelism because `reserve` DEBITS on the root's turnstile before the call. Three members
500
- against a ceiling only one fits leave exactly one `ok`; that is asserted through the hive
501
- rather than asserted about the ledger, because the ledger already promised it.
502
- - **A member's throw is RECORDED, whatever it is.** `failureOf` reads it with `isThrownError` and
503
- `stringField` from core, never `error instanceof Error` and `.message`: a member is an app's
504
- action, so a `Proxy` makes `instanceof` run a `getPrototypeOf` trap, and a throw there takes
505
- down the whole hive — the one outcome the three arms exist to prevent. `skipped` has two
506
- reasons, because they are two facts: `SKIPPED_ABORTED` (a sibling failed under `'abort'`) and
507
- `SKIPPED_NO_INPUT` (the split produced nothing at that index). One string for both sent a
508
- caller to retry a tail that was never cut.
509
- - **`onMemberError` is required.** `'abort'` stops and leaves the rest `skipped`; `'collect'`
510
- harvests. Both are right for somebody, so neither may be inherited silently.
511
- - `concurrency` defaults to 4 and `minMembers` to 2, and neither number is measured off any run —
512
- the framework cannot know a provider's concurrency allowance. A below-floor split still runs
513
- every input it produced, serially: dropping one would be silent data loss.
514
- - An empty split is `X_HIVE_EMPTY`, never a successful run of zero members — "0 ok, 0 failed"
515
- cannot be told apart from a query that returned no rows and nobody noticed.
516
- - An aborted `ctx` unwinds the whole hive with `X_ABORTED`, which is a DIFFERENT event from
517
- `onMemberError: 'abort'`: the latter is a completed run with a partial harvest worth returning,
518
- the former has nobody left to hand it to.
519
- - **`describeAgents()` is OFFERED, not published, `As of 2026-08-23`** — no reader exists anywhere
520
- in the tree, and neither does one for `registeredModels()`. Both are tier-4 exports whose only
521
- legal consumer is `@ultimat3/cli` (tier 5); `@ultimat3/manifest` is tier 4, so the obvious import
522
- is a sideways edge `bun run boundaries` refuses. Their doc comments say so now, because a comment
523
- claiming a consumer that does not exist is the documentation axiom 3 calls no rule at all.
524
- - **`describeAgents()` describes what an `ActionDescriptor` cannot.** An agent projects to the same
525
- descriptor as any other action, and that descriptor knows nothing about turns, tools, models or
526
- prompt hashes — so "how far can this loop and what may it call" had no answer outside the source.
527
- Same shape as `describePrompts()` / `describeEvals()`, and deliberately NOT a new
528
- `ActionDescriptor` field: `@ultimat3/action` is tier 3 and knows nothing about models.
529
- The facts are a THUNK, resolved when asked: `agent()` runs at module scope beside the actions it
530
- lists, and every name in a row is stamped by `registerAction` at boot. An agent still carrying no
531
- name has no row — not a silent drop, but the absence of a capability: an action with no name
532
- reaches no route, no tool catalogue and no queue. `named()` builds a TWIN where registration
533
- names in place, so an agent renamed that way is absent for the same reason; register it instead.
534
- - **`agentJob()` closes #125 for the agent case, by COMPOSING `job()`** — the returned value is one
535
- `job()` seated in that package's own registry, so `.enqueue()`, the outbox, the worker's
536
- cancellation, the dead-letter path, `x jobs show` and its manifest row arrive without a line here.
537
- Never an imitation handle: `isJobHandle` needs `kind === 'job'` plus membership of a WeakMap only
538
- `job()` writes, which is exactly what stops a second execution path existing.
539
- - `name`, `tenant` and `retry` are REQUIRED, no defaults. `name` because a job name is the durable
540
- queue key that queued, retrying and dead-lettered rows already carry — deriving it from the
541
- export name would move delivery when somebody renames a variable. `tenant` and `retry` because
542
- `jobs` states that every candidate default for `tenant` is a cross-tenant read waiting for the
543
- first job that takes an org id in its input. Both are `TS2741` when omitted AND have runtime
544
- backstops (`X_JOB_TENANT_REQUIRED`, the `retry.attempts` assert), for generated and JS callers.
545
- - **Both reads of `target.job()` are LAZY**, and that is load-bearing: `actionName()` throws
546
- `X_ACTION_UNREGISTERED` until boot stamps the export name, and `agentJob()` is evaluated at
547
- module scope right beside the `agent()` it wraps. Same rule as `agent()`'s tool projection and
548
- `describeAgents()`' thunk — third instance in this package.
549
- - **The at-least-once trap is DOCUMENTED, not enforced, and that is a decision with evidence.**
550
- `idempotencyKey` dedupes the ENQUEUE, never the ATTEMPT: a lost lease re-runs the agent from the
551
- top, as does every page `backfill()` replays. So every tool an `agentJob()`'d agent may call has
552
- to be idempotent. The framework cannot check it: `mutates` is not a fact an `action()` declares
553
- — it exists only in `@ultimat3/mcp`, whose `projectable.ts` sets it to `true` for EVERY action —
554
- so a read-only `lookupOrder` and a destructive `issueRefund` are indistinguishable, and
555
- `ActionFacade`'s `Pick<>` carries no `idempotent` either (only `describe()` does, and that needs
556
- a name). `isMutator` IS legible without a name, but `mutator()` is the local-first write
557
- primitive: refusing only those would catch almost none of the risk while reading as if it caught
558
- all of it. A wrong refusal is worse than a stated obligation, so the obligation is stated — in
559
- `AgentJobOptions.idempotencyKey`'s doc comment and in the README, both naming the second refund.
141
+ `output_config`. Model IDs are exact aliases, never date-suffixed.
142
+ - **`ModelId` is `string`; the catalogue is an open registry** — `modelSpec(id)` refuses an
143
+ unregistered id (`X_AI_MODEL_UNKNOWN`). The built-ins register through the same `registerModel`.
144
+ **Re-registering REPLACES the spec and keeps its rung** (negotiated rates). **Registration order is
145
+ the ladder within a `family`**, read only by `moreCapableThan`; `X_LLM_REFUSED`'s fix names a rung
146
+ ABOVE or nothing.
147
+ - `AnthropicProvider.models` is its own list; `EchoProvider.models` reads the registry.
148
+ - **The reasoning controls are PER MODEL (`models.ts`)**: an unasked control is omitted; an asked
149
+ control the model lacks is `X_AI_REQUEST_INVALID`, never dropped. Adding a model is a row in `MODELS`.
150
+ - `generate()` above `STREAM_ONLY_MAX_TOKENS` runs the streaming transport.
151
+ - **`openAiProvider()` is a FORMAT, not a vendor** — `baseUrl`, `auth`, `headers` are the
152
+ differences; never a second class per vendor:
153
+ - structured output is the `respond` tool, never `response_format`; `tool_choice` names the tool
154
+ only when the request offers exactly one;
155
+ - `strict: true` only when `satisfiesStrictMode` says the schema qualifies;
156
+ - `max_completion_tokens`; `role: 'system'`; `stream_options: { include_usage: true }` and an
157
+ ESTIMATE when usage never arrives; subtract `cached_tokens` from `prompt_tokens`;
158
+ - tool-call deltas merge by `tool_calls[].index`; `isComplete()` accepts `[DONE]` or a finish reason,
159
+ except `[DONE]` with pending tool fragments, which is refused;
160
+ - only `gpt-5.6-sol` / `-terra` / `-luna` are priced; `registerOpenAiModels()` re-registers after
161
+ `resetModels()` (every `openai-*.test.ts` calls it in `beforeEach`);
162
+ - no new `X_*` code; `AiTransportError` takes the provider's `envVar`; the key is revealed late,
163
+ never stored, scrubbed from `detail`.
164
+ - **`embedBatched` enforces the `Embedder` arity per batch** (`X_AI_EMBEDDER_INVALID` with both
165
+ counts). One `RemoteEmbedder` for every vendor; vectors L2-normalised; a wrong width is
166
+ `X_VECTOR_DIM_MISMATCH`.
167
+ - **A caught value is read with `renderThrowable`** (`remote-embedder.ts`, `wire.ts`,
168
+ `openai-wire.ts`) — `scripts/error-render.ts` cannot see `catch` bindings.
169
+ - **A caller's string is never an object KEY**: the wire tables are `Map`s; `prompt.render` and
170
+ `EchoProvider`'s `replies` use `Object.hasOwn`.
171
+
172
+ ## Invariants — `agent()`, `hive()`, `agentJob()`
560
173
 
561
- - **`configureAi({ redact })` is the one seam between `vars()` and the provider.** `vars()` is the
562
- one declared place a model call loads data, so it is the one place a redactor can see the row
563
- before it leaves the process; the redactor sees the whole RENDERED prompt and the system prompt,
564
- template as well as values. WHAT to remove is the app's (a PII classifier is a model choice —
565
- axiom 8). The framework ships the seam, the `llm.redacted` span attribute, and the one rule it
566
- can enforce structurally: **a `Secret` in `vars()` is `X_AI_PROMPT_SECRET`**, whether or not a
567
- redactor is installed. Not a leak — `Secret` renders `[redacted]` by value — but a prompt that
568
- reads fine, means something else, and costs full price.
569
- - **Fallback is across PROVIDERS serving one model, never across models — decided 2026-08.** The
570
- wiki's LLM-gateway table claimed an ordered model list; there never was one, and building one was
571
- rejected: a silent model swap changes what answered, what it cost, and which eval baseline the
572
- answer belongs to, and `X_LLM_REFUSED` already names a more capable model for the DECLARATION to
573
- adopt. What was missing is the other half of the claim — "never silent" — so the gateway now
574
- stamps `GenerateResult.provider` with the provider that actually answered and `llm()` puts it on
575
- the span as `llm.provider`. Stamped by the gateway, not the provider: routing is a gateway
576
- concept, and an app's own `Provider` cannot report on a decision it did not make.
577
- - A **refusal is a 200 with no answer in it**, so it becomes `X_LLM_REFUSED` at the `llm()` seam,
578
- before the output is parsed. Parsing it first reports a schema disagreement — wrong cause,
579
- inapplicable fix — and spends a repair turn buying the same refusal again. A truncated answer
580
- that also fails its schema is `X_LLM_TRUNCATED` for the same reason: the ceiling does not move
581
- between attempts. `stopDetails.category` is carried, not dropped: it is the only thing that
582
- says whether another model would answer.
583
- - The gateway does not cache a refusal. Caching one keeps serving a classifier decision long
584
- after the prompt that provoked it was fixed.
585
- - Server-side `fallbacks` (beta) are deliberately NOT sent. The provider speaks the stable
586
- `2023-06-01` surface, and a 1.0 package that promises semver cannot pin a beta wire contract;
587
- the typed refusal plus the gateway's own model routing is the framework's answer instead.
588
- - `definePrompt` refuses a re-registered version whose hash moved, and the hash is taken over
589
- `@ultimat3/core`'s `canonicalJson` (`As of 2026-08-22`). It was a local sorted-key
590
- `JSON.stringify` — the framework's third copy of one canonical form — which spells `-0` as `0`
591
- and every non-finite number as `null`: a schema `default` the model is told about could move
592
- while the ref did not, and every score already filed goes on claiming to describe the new prompt.
593
- Ordinary JSON hashes byte-identically between the two, so **no committed baseline is
594
- invalidated**; `prompt.test.ts` pins one hash literally to keep it that way. An ABSENT schema
595
- stays the empty string rather than `canonicalJson(undefined)`'s `null`, for the same reason.
596
- - **A caught value is read with `renderThrowable`, never `error instanceof Error ? error.message :
597
- String(error)`.** `instanceof` RUNS a `Proxy`'s `getPrototypeOf` trap, and a throw there escapes
598
- the very `catch` that exists to produce a coded refusal — proven against `RemoteEmbedder`, whose
599
- injected `fetch` made it reachable from app config. Five sites in this package
600
- (`remote-embedder.ts`, `wire.ts` x2, `openai-wire.ts` x2); `scripts/error-render.ts` cannot see
601
- any of them, because it reads PARAMETERS typed `unknown` and these are `catch` bindings.
602
- - Every eval result carries the prompt hash. A score without one is not a measurement.
603
- - An eval gates on the DROP from its recorded baseline, never on an absolute score. An absolute
604
- floor fails every eval at once the day a provider ships a slightly different model, which
605
- teaches everyone to lower thresholds until they measure nothing.
606
- - The run mean AND every case are compared. A mean that holds while one case collapses is the
607
- regression an eval exists to catch.
608
- - A baseline that has never been recorded is `X_EVAL_BASELINE_MISSING`, and a corrupt one is
609
- `X_EVAL_BASELINE_INVALID` — never "absent, so pass". A step that cannot fail is not running.
610
- - `baseline` is `import.meta.resolve('./…')`. A cwd-relative path resolves to a different file
611
- depending on where the suite was started, which is how a gate silently stops gating.
612
- - Every registered prompt must be named by an eval (`promptsWithoutEvals`, `X_EVAL_MISSING`).
613
- Coverage is by prompt ID, not ref: old versions are retained, and an eval on the current one
614
- evaluates that lineage.
615
- - `ULTIMATE_EVAL_RECORD=1` writes baselines instead of gating on them. A test that deliberately
616
- scores a worse model calls `run`, never `assert` — `assert` would re-record during that pass.
617
- - **Recording and the gate are mutually exclusive.** `x verify` with that variable set is
618
- `X_EVAL_RECORDING` and runs no eval suite at all. Recording passes by definition, so a gate run
619
- that inherited the flag is green over numbers it wrote itself — and rewrites every committed
620
- baseline on its way through, which is the half a red step would not undo. Hence refuse *before*
621
- the suite, never after it.
622
- - The gate asks whether an eval has a baseline, not only whether one is declared. `defineEval`
623
- proves a prompt is named; it proves nothing measured it, and an eval whose numbers were never
624
- recorded — one no test asserts, one whose `baseline:` is a cwd-relative string — would otherwise
625
- satisfy `X_EVAL_MISSING` while gating on nothing.
626
- - Retrieval is hybrid by default. Do not add a vector-only convenience path.
627
- - **`chunk()` performs all three splits its header names — paragraph, sentence, HARD WRAP** (`As of
628
- 2026-08`). The wrap is what bounds a unit, and an oversized unit is one the size check can never
629
- flush (it only fires when something is already in the buffer), so it re-seeded every following
630
- chunk: a ~1,000-token document indexed as nine chunks totalling ~9,000, each carrying the same
631
- sentence. The overlap carry stops at `buffer.length - 1` for the same reason — a flushed unit may
632
- never become the whole of the next buffer.
633
- - **A caller's string is never used as an object KEY.** A `Record` lookup on one answers
634
- `constructor`, `toString` and `valueOf` off the prototype chain, and every read here is of a
635
- string a provider, a template author or a test fixture chose: the two wire tables are `Map`s,
636
- `prompt.render` uses `Object.hasOwn` (a `{{constructor}}` slot rendered JS source into a billed
637
- prompt and hashed it into the cache key), and so does `EchoProvider`'s `replies`.
638
- - `PgVectorStore` is the ONLY production vector path — pgvector and Postgres FTS in the app's own
639
- Postgres, never a second datastore. `MemoryVectorStore` is the dev twin and enforces the same
640
- envelope; a leak that only reproduces against real Postgres is a leak nobody finds.
641
- - **It is proved against a real pgvector, not only against a recording client.**
642
- `pg-vector.live.test.ts` runs the whole chain — `ddl()` -> a live server -> `upsert` -> cosine,
643
- FTS and the RRF fusion -> decoded hit — and REFUSES to skip when `TEST_DATABASE_URL` names a
644
- Postgres without the extension, because a suite that stands down reports green for the one
645
- store that runs in front of real traffic. CI's service container is `pgvector/pgvector:pg17`
646
- for that reason. Asserting statement *text* cannot catch a statement Postgres rejects, nor a
647
- filter that compiles cleanly and excludes nothing: that is exactly how metadata shipped bound
648
- `::jsonb`. A new operator, read path or scope rule is not done until it round-trips there.
649
- - The distance ordering lives in a subquery, ascending and raw, because that is the only shape
650
- hnsw answers — `order by 1 - (…) desc` is a sequential scan. Both halves are pinned by a plan
651
- assertion in the live suite, since only a planner can say which one shipped.
652
- - hnsw applies the scope AFTER the index scan, so an approximate index can return fewer rows
653
- than asked for once a tenant filter is selective. The planner takes the exact path instead
654
- when it has stats — which is why a bulk backfill that skips `analyze` is how a search that
655
- used the index yesterday scans today. Assert the rows a scoped read returns, never the node.
656
- - Tenant and policy filters go **in SQL**, on every statement, through `conditionsSql` — and on
657
- BOTH halves of the fusion. Filtering after the rows are loaded is not filtering.
658
- - `(tenant, id)` is the primary key. A cross-tenant overwrite is impossible at the storage layer
659
- rather than conditional on every upsert remembering to check.
660
- - `scoped()` only ever TIGHTENS: tenants are set once, allow-lists intersect. Widening is
661
- `X_VECTOR_SCOPE_WIDENED`. Same rule as `budget.derive`, for the same reason.
662
- - Metadata is bound `::text::jsonb`. A bound string cast straight to `::jsonb` is JSON-encoded
663
- twice, reads back correctly, and makes every `metadata ->> key` filter match nothing.
174
+ - **`agent()` returns an action** (listed in `PRIMITIVE_FACTORIES`). `ctx.actor` is read once and is
175
+ the only identity any tool runs as — never taken from the model's output. Bounded by `maxTurns`
176
+ (`X_AGENT_MAX_TURNS`), `budget.tokensPerRun` and `maxToolResultChars`.
177
+ - `tools` takes a real `action()` (`AgentTool = AnyAction | ProjectableAction`,
178
+ `asProjectableAction`); projection happens on the FIRST RUN, memoised. An agent can be another
179
+ agent's tool.
180
+ - A tool not `mcp: { expose: true }` is `X_AGENT_TOOL_UNEXPOSED` at DECLARATION.
181
+ - `ctx.signal` is read at the top of every turn, before every tool batch, and forwarded to
182
+ `fetch`; `X_ABORTED` is core's.
183
+ - One turn's tools run concurrently, results paired by index; no tool STARTS after an abort.
184
+ - `onTurn` reports facts only (also an `agent.turn` span event); a throw from it fails the run.
185
+ `.stream()` on `agent()` is deliberately not shipped. No semantic cache.
186
+ - Every replayed `tool_use` is answered by a `tool_result` in the very next message
187
+ (`agent-transcript.ts`); an unaccepted `respond` comes back as an `is_error` `tool_result`
188
+ ("superseded").
189
+ - A tool result is rendered totally (`'null'` for `undefined`; a non-JSON result reported as such,
190
+ never as a failure); the throw is read with `stringField`.
191
+ - **`hive()` is a fan-out action** (`PRIMITIVE_FACTORIES`): `HiveResult` is a SCHEMA from the
192
+ member's `output`; three arms (`ok`/`failed`/`skipped`) and three counters; `members` in SPLIT
193
+ order with `index`; the hive never names an actor; no hive budget code (the root turnstile holds
194
+ it); a member's throw is recorded via `isThrownError`/`stringField`; `skipped` reasons
195
+ `SKIPPED_ABORTED` / `SKIPPED_NO_INPUT`; `onMemberError` is required; `concurrency` 4, `minMembers`
196
+ 2 (a below-floor split still runs serially); an empty split is `X_HIVE_EMPTY`; an aborted `ctx`
197
+ unwinds with `X_ABORTED`.
198
+ - **`describeAgents()` / `registeredModels()` are offered, not published** — their only legal consumer
199
+ is `@ultimat3/cli`. The facts are a THUNK; an unnamed agent has no row.
200
+ - **`agentJob()` composes `job()`** (never an imitation handle). `name`, `tenant`, `retry` are
201
+ REQUIRED; both reads of `target.job()` are LAZY. **Every tool an `agentJob()`'d agent may call must
202
+ be idempotent** — stated in `AgentJobOptions.idempotencyKey`'s doc and the README, not enforceable.
203
+
204
+ ## Invariants — evals, retrieval, fix lines
205
+
206
+ - **No fix line may name `x ai`** (planned). An eval is selected with `x test eval --filter <name>`;
207
+ `eval-errors.test.ts` asserts every `x test <word>` is one of the six types.
208
+ - Every eval result carries the prompt hash. An eval gates on the DROP from its recorded baseline,
209
+ mean AND per case. Never-recorded is `X_EVAL_BASELINE_MISSING`, corrupt is `X_EVAL_BASELINE_INVALID`.
210
+ `baseline` is `import.meta.resolve('./…')`. Every prompt is named by an eval (`X_EVAL_MISSING`, by
211
+ prompt ID). `ULTIMATE_EVAL_RECORD=1` records; a deliberately-worse test calls `run`, never
212
+ `assert`; `x verify` with the flag is `X_EVAL_RECORDING` and runs no eval. The gate asks for a
213
+ recorded baseline, not only a declaration.
214
+ - Retrieval is hybrid by default; no vector-only path. **`chunk()` hard-wraps**, and the overlap
215
+ carry stops at `buffer.length - 1`.
216
+ - `PgVectorStore` is the only production vector path (pgvector + FTS in the app's Postgres);
217
+ `MemoryVectorStore` enforces the same envelope. **`pg-vector.live.test.ts` refuses to skip** on a
218
+ Postgres without the extension (CI uses `pgvector/pgvector:pg17`).
219
+ - The distance ordering lives in an ascending raw subquery (the only shape hnsw answers), pinned by a
220
+ plan assertion. hnsw scopes after the scan — assert rows, never the node; `analyze` after a backfill.
221
+ - Tenant and policy filters are in SQL on every statement (`conditionsSql`), on BOTH fusion halves.
222
+ `(tenant, id)` is the primary key. `scoped()` only TIGHTENS (`X_VECTOR_SCOPE_WIDENED`). Metadata is
223
+ bound `::text::jsonb`.
664
224
 
665
225
  ## Commands
666
226
 
@@ -673,3 +233,5 @@ docker run -d -e POSTGRES_PASSWORD=ultimate -p 5432:5432 pgvector/pgvector:pg17
673
233
  TEST_DATABASE_URL=postgres://postgres:ultimate@localhost:5432/postgres \
674
234
  bun test packages/ai/src/pg-vector.live.test.ts
675
235
  ```
236
+
237
+ Why each rule above is shaped the way it is: [`docs/history/ai.md`](../../docs/history/ai.md).