@ultimat3/ai 21.0.0 → 22.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +159 -597
- package/README.md +37 -3
- package/package.json +10 -10
- package/src/agent-transcript.ts +7 -1
- package/src/agent.ts +11 -3
- package/src/budget.ts +5 -1
- package/src/gateway.ts +35 -3
- package/src/index.ts +2 -9
- package/src/llm.ts +11 -5
- package/src/pg-vector-sql.ts +13 -0
- package/src/pg-vector.ts +10 -1
- package/src/provider.ts +7 -1
- package/src/rag.ts +11 -1
- package/src/remote-embedder.ts +34 -1
- package/src/vector.ts +16 -0
- package/src/wire.ts +7 -1
package/CLAUDE.md
CHANGED
|
@@ -6,20 +6,15 @@ query jobs realtime`. **Never** `mcp manifest render pwa ui admin testing cli`.
|
|
|
6
6
|
Declared today: `action` (the primitive `llm()` returns), `cache` (semantic cache), `core`,
|
|
7
7
|
`db` (pgvector), `jobs` (`agentJob()`), `money`, `policy`, `schema`, `time`.
|
|
8
8
|
|
|
9
|
-
|
|
10
|
-
already named this package as the home: `isJobHandle` needs `kind === 'job'` plus membership of a
|
|
11
|
-
WeakMap only `job()` writes, and `action` and `jobs` are both tier 3, so the bridge has to live at
|
|
12
|
-
tier 4+. Downward edge, nothing new in the tier table.
|
|
9
|
+
`jobs` is declared for `agentJob()`: the action→job bridge has to live at tier 4+.
|
|
13
10
|
|
|
14
11
|
`mcp` is the same tier, so the LLM-tool projection is restated structurally in `tools.ts`
|
|
15
12
|
rather than imported. Same contract, two wire formats — and the same *decision*: `toLlmTools` and
|
|
16
13
|
`runLlmToolCall` ask `isMcpExposed` from `@ultimat3/core` (tier 0, reachable by both), never a
|
|
17
14
|
local `=== true`. An in-app agent and an external one must be offered exactly the same tools.
|
|
18
15
|
|
|
19
|
-
**And the same NAME**: `toLlmTool` passes `action.name` through untouched
|
|
20
|
-
`@ultimat3/mcp` serves
|
|
21
|
-
`.tool()` name is the verbatim export name too — it read `summarize_post` / `projecting_agent`
|
|
22
|
-
until 2026-08, naming a tool no catalog contained (`llm.test.ts`, `agent.test.ts`).
|
|
16
|
+
**And the same NAME**: `toLlmTool` passes `action.name` through untouched (the export name
|
|
17
|
+
`@ultimat3/mcp` serves); `llm()` and `agent()` `.tool()` names are verbatim too. Never derive one.
|
|
23
18
|
|
|
24
19
|
## Owns
|
|
25
20
|
|
|
@@ -68,599 +63,164 @@ until 2026-08, naming a tool no catalog contained (`llm.test.ts`, `agent.test.ts
|
|
|
68
63
|
| `runtime.ts` | the ambient gateway / embedder / semantic caches an `llm()` reaches |
|
|
69
64
|
| `fix-line.ts` / `fix-line.evals.ts` / `fix-line.v1.baseline.json` / `fix-line.eval.test.ts` | the package's own dogfood eval — the first framework-level `*.eval.test.ts`, proving the `defineEval`/baseline convention actually fails a build |
|
|
70
65
|
|
|
71
|
-
## Invariants
|
|
66
|
+
## Invariants — bounds and budget
|
|
72
67
|
|
|
73
|
-
- **Every numeric option
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
call passes, `registerModel` screens `maxOutput` (which reaches the same estimate through
|
|
92
|
-
`Math.min`), and `llm()`/`agent()` screen theirs at DECLARATION, beside `respondToolFor` and the
|
|
93
|
-
`X_AGENT_TOOL_UNEXPOSED` check, so a module-scope declaration fails the boot rather than the
|
|
94
|
-
ninetieth second of a run.
|
|
95
|
-
- **A bound is screened under the key the DECLARATION uses, even when that costs a second check.**
|
|
96
|
-
`budget.tokensPerRun` reaches `BudgetLimits` as `request`, so `limitsOf` in `llm.ts`, `agent.ts`
|
|
97
|
-
and `hive.ts` screens it before the ledger's own constructor does — a `fix:` naming a key the app
|
|
98
|
-
never wrote is not a fix. Same reason `retrieve()` screens its own `k` rather than letting the
|
|
99
|
-
store refuse the `k * 3` it becomes, and `RemoteEmbedder` screens `maxResponseBytes` itself
|
|
100
|
-
rather than letting core's reader answer `readWithinLimit was given a limit of NaN` — which is
|
|
101
|
-
correct, unactionable, and only arrives after the request has been paid for.
|
|
102
|
-
- **`llm()` returns an `action`. It is not a ninth primitive** (root `CLAUDE.md`, 2026-08). It
|
|
103
|
-
never re-implements parse, authz or invoke — `action()` owns those and `invoke` runs them.
|
|
104
|
-
- `src/index.ts` re-exports `t` from `@ultimat3/schema` **verbatim**, so an `llm` file imports one
|
|
105
|
-
package. Never wrap, spread or re-declare it: `t` delegates to `schemaProvider()` on every
|
|
106
|
-
access, and a copy would freeze the provider at import time. `index.test.ts` asserts identity.
|
|
107
|
-
- The model half is the only thing `llm.ts` adds: render the prompt, project `output` into the
|
|
108
|
-
one tool the model may answer through, reserve the budget, consult the semantic cache.
|
|
109
|
-
- One repair turn, then `X_LLM_OUTPUT_INVALID`. Two schema failures is a prompt/schema
|
|
110
|
-
disagreement; a third attempt only spends money.
|
|
111
|
-
- Semantic scopes are separate cache INSTANCES, never a filter over a shared one — cosine
|
|
112
|
-
similarity has no notion of a tenant. The instance key carries the prompt hash too, which is
|
|
113
|
-
what makes a version bump invalidate the cache.
|
|
114
|
-
- **The instance key carries `ctx.locale` too, `As of 2026-08-24`, and it sits with the prompt hash
|
|
115
|
-
rather than with the scope.** A prompt taking `locale` as a var — the reference app's `summarize`
|
|
116
|
-
does, and the model obeys it — differs by ONE token between languages while carrying a whole
|
|
117
|
-
document, so the two rendered prompts are neighbours: measured with this package's own
|
|
118
|
-
`HashEmbedder` over that template, **0.9986**, against the declaration's `threshold: 0.97`. The
|
|
119
|
-
Spanish summary was a hit for an English reader, and the model was never wrong — the cache was.
|
|
120
|
-
No threshold repairs it, because the same number has to keep an honest repeat above it. It is in
|
|
121
|
-
the UNCONDITIONAL half of the key because a `scope` answers "who may share this answer" and a
|
|
122
|
-
locale is part of what the answer IS: an app writing `scope: () => 'global'` is saying its
|
|
123
|
-
callers may read one another's summaries, never that a Spanish one will do. Same rule
|
|
124
|
-
`@ultimat3/render` applies to ISR.
|
|
125
|
-
- **The default scope is the calling ACTOR, and `scope` receives `{ input, ctx }`** (`As of
|
|
126
|
-
2026-08`). It defaulted to the literal string `'global'` and took the bare `input`, so the rule
|
|
127
|
-
in the bullet above was contradicted by the very default that shipped: `cache: { semantic: { ttl:
|
|
128
|
-
'1h' } }` put every tenant in one store, and `lookup` is a nearest neighbour with no tenant
|
|
129
|
-
predicate — proven by execution, tenant B asking a prompt within 0.92 cosine of tenant A's
|
|
130
|
-
received A's completion verbatim. And with only `input` to decide from, the one thing a partition
|
|
131
|
-
may never be chosen by (a value the caller sends) was the only thing it could be chosen by, while
|
|
132
|
-
`vars()` on the same declaration already took the pair. The default is
|
|
133
|
-
`JSON.stringify([actor.kind, actor.id, actor.orgId ?? null])` — `@ultimat3/query`'s
|
|
134
|
-
`readAuthority` rule, verbatim: a declaration that says nothing gets the NARROWEST key, and
|
|
135
|
-
widening is a written statement about what the answers are (`scope: () => 'global'`). **Breaking**
|
|
136
|
-
in both halves. `semanticCacheFor`'s instance map is bounded as a consequence
|
|
137
|
-
(`MAX_SEMANTIC_CACHE_SCOPES`, which IS core's `MAX_CACHED_FORMATTERS` — one bounded FIFO map in
|
|
138
|
-
the framework, whose name is about its first caller and not its contract): one entry per actor in
|
|
139
|
-
a process that never restarts is a leak where one entry per process was not.
|
|
140
|
-
- A per-call budget `derive`s from the ambient ledger, so it can only TIGHTEN the actor and org
|
|
141
|
-
ceilings it runs inside. Widening them from a declaration would be a budget that is not one.
|
|
142
|
-
**A derived ledger reports back up the chain**: every debit and every recorded cost lands on it
|
|
143
|
-
AND on every ledger it was derived from, and `reserve` checks the `request` scope of each one.
|
|
144
|
-
Without that link a child was a fresh counter with no parent — `llm()` derives one per call, so
|
|
145
|
-
`gateway.spent()` answered zero after a hundred calls and a `request` ceiling of 5,000 was
|
|
146
|
-
re-granted in full to each of them. The STORE is written once, by the ledger the call was made
|
|
147
|
-
on: a child shares its parent's store and keys, so writing through both bills the actor twice.
|
|
148
|
-
Reservations queue on the ROOT's turnstile for the same reason — a per-ledger queue serialises
|
|
149
|
-
nothing once every call has its own ledger.
|
|
150
|
-
- `cache.invalidates` from `docs/idea/05-caching.md` is **not** on `llm()` yet: `invalidateTags`
|
|
151
|
-
fans out to `CacheTier`s, and a `SemanticCache` is not one. Storing tags nothing visits would
|
|
152
|
-
read as wired and silently not be. Version bump + `ttl` is the invalidation today.
|
|
153
|
-
- The gateway is ambient (`configureAi`) because a declaration is evaluated at module scope,
|
|
154
|
-
long before a provider exists. Absent at call time is `X_AI_GATEWAY_MISSING`, never a default
|
|
155
|
-
provider — a silent fallback would spend real money on a boot mistake.
|
|
68
|
+
- **Every numeric option is screened; `??` is not the screen.** `finiteOption`/`finiteCount` from
|
|
69
|
+
`@ultimat3/core` are the one form; `bun run finite-bounds` is the ratchet, and it cannot see a
|
|
70
|
+
required option with no `??`, a default parameter or a cross-field product. **`maxTokens` is the
|
|
71
|
+
worst**: it IS the budget's pre-flight estimate, and a `NaN` switches every actor and org ceiling
|
|
72
|
+
off. `Gateway.generate`/`stream` screen it at the one seam, `registerModel` screens `maxOutput`, and
|
|
73
|
+
`llm()`/`agent()` screen theirs at DECLARATION.
|
|
74
|
+
- **A bound is screened under the key the DECLARATION uses** (`budget.tokensPerRun` in `limitsOf`
|
|
75
|
+
before the ledger's `request`; `retrieve()`'s own `k`; `RemoteEmbedder`'s `maxResponseBytes`).
|
|
76
|
+
- A per-call budget `derive`s from the ambient ledger and only TIGHTENS. **A derived ledger reports
|
|
77
|
+
back up the chain** (every debit on every ancestor; `reserve` checks each `request` scope); the
|
|
78
|
+
STORE is written once, by the ledger the call was made on; reservations queue on the ROOT's turnstile.
|
|
79
|
+
- **`BudgetStore` is where `actor` and `org` live; the default is per PROCESS**
|
|
80
|
+
(`createGateway({ budgetStore })`). `add` takes a negative `tokens` for a release — a store that
|
|
81
|
+
clamps at zero leaks the ceiling.
|
|
82
|
+
- Cost is `Money` (integer minor units), rounded **up**. A budget throws `X_AI_BUDGET_EXCEEDED`
|
|
83
|
+
**before** the provider call. Never truncate. List price over-reserves (no introductory prices).
|
|
84
|
+
- **`Gateway.stream` routes BEFORE it reserves**, and `settled = true` comes after
|
|
85
|
+
`await ledger.record(...)`.
|
|
156
86
|
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
`
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
-
|
|
166
|
-
`
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
`
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
- **
|
|
203
|
-
`
|
|
204
|
-
`
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
`
|
|
209
|
-
|
|
210
|
-
`UltimateError` — so adopting it would have retried a 400 three times per provider, which is the
|
|
211
|
-
one thing the loop's own comment says it must not do. `RetryPolicy` and `DEFAULT_RETRY` stay
|
|
212
|
-
declared here for a second reason: their field names are what an app writes.
|
|
213
|
-
- Every non-2xx and every in-band `error` frame becomes `AiTransportError`, which carries a real
|
|
214
|
-
`status` field — that field IS the gateway's retry rule. A body parsed as a message would read
|
|
215
|
-
as an empty, successful answer, which is the one outcome nothing downstream can detect.
|
|
216
|
-
- **The two wire formats answer the same question the same way, and `provider-parity.test.ts` is
|
|
217
|
-
what makes that a build error — added 2026-08.** Four rules were held by one format and not the
|
|
218
|
-
other, all of them in the RESPONSE half, and each one is a failure that reads as a success:
|
|
219
|
-
- an in-band `error` object in a **200 body** is `AiTransportError` on both. `parseMessage` read
|
|
220
|
-
one as an empty `end_turn` answer while `MessageStream` — the *same provider's* streamed half —
|
|
221
|
-
had always refused it. `throwInBandError` is `wire.ts`'s, exported, so one status table decides
|
|
222
|
-
the gateway's retry on both transports.
|
|
223
|
-
- a `stopDetails` of type `refusal` **forces** `stopReason: 'refusal'`, because `llm()` and
|
|
224
|
-
`agent()` branch on the reason and nothing reads the detail. `parseStopReason` answers
|
|
225
|
-
`end_turn` for a spelling this build has never seen, so a refusal in a new vocabulary arrived
|
|
226
|
-
as a complete answer that happened to be empty. The OpenAI-format read always forced it.
|
|
227
|
-
- a tool call's `input` is **parsed, never cast**: `asToolInput`, one copy. `(b['input'] ?? {}) as
|
|
228
|
-
Record<string, unknown>` put a string under that type, and `runLlmToolCall` indexes it.
|
|
229
|
-
- the **credential is scrubbed** out of `AiTransportError.detail` on both. `withoutKey` was the
|
|
230
|
-
OpenAI provider's alone, so a proxy echoing `x-api-key` into its 400 body put an Anthropic key
|
|
231
|
-
in an error — and an error reaches a log index, a span and a problem document. Both providers
|
|
232
|
-
now call `error-body.ts`'s pair; `detailOf` moved there from `provider.ts` with it (internal,
|
|
233
|
-
never in `src/index.ts`).
|
|
234
|
-
What is NOT parity: the two status tables (529 vs 503 for "overloaded") and the OpenAI-format
|
|
235
|
-
`estimatedUsage` fallback. Those are the formats differing, and the parity suite asserts them
|
|
236
|
-
*as* differences rather than flattening them.
|
|
237
|
-
- A stream that ends without `message_stop` throws. A truncated answer that returns `end_turn`
|
|
238
|
-
is a confidently wrong answer with no signal, which the budget rule already forbids.
|
|
239
|
-
- **`readSse` caps the unterminated buffer** (`MAX_FRAME_CHARS`, `As of 2026-08`) and refuses with
|
|
240
|
-
`AiTransportError`. A peer that never sends a frame boundary — an HTML error page, a proxy on the
|
|
241
|
-
model's port — grew it without limit and no read deadline interrupted it, because every read
|
|
242
|
-
SUCCEEDED. Same call `@ultimat3/mail`'s `createReplyParser` makes: coded failure > OOM. `provider`
|
|
243
|
-
is a required argument for that reason — a transport error names the endpoint it is about.
|
|
244
|
-
- **`llm()` forwards `ctx.signal` onto `GenerateRequest`**, `As of 2026-08`, the way `agent()`
|
|
245
|
-
always did. Without it a model call had no cancellation and no deadline: a caller that hung up
|
|
246
|
-
left the provider call in flight, billed and unread, and the repair turn bought a second one.
|
|
247
|
-
`.stream()` inherits it from the same `base`, which is where it matters most.
|
|
248
|
-
- A tool call is emitted whole. `input_json_delta` fragments are not arguments until the block
|
|
249
|
-
closes, so nothing partial reaches a caller.
|
|
250
|
-
- Thinking chunks are never appended to `text`. A consumer concatenating every chunk must not
|
|
251
|
-
end up shipping the reasoning to the user.
|
|
252
|
-
- **`embedBatched` enforces the `Embedder` arity, per batch** (`As of 2026-09`). "One vector per
|
|
253
|
-
input text, in the order the texts arrived" is the interface's own invariant and nothing
|
|
254
|
-
downstream can restore it: `indexDocument` writes `vectors[index]` per chunk, so an app embedder
|
|
255
|
-
that answered short stored `undefined` and surfaced a layer later as a `TypeError` inside the
|
|
256
|
-
vector store, naming nothing an author wrote. `X_AI_EMBEDDER_INVALID` — the shipped code whose
|
|
257
|
-
registered title already read "an Embedder returned fewer vectors than texts it was given" — now
|
|
258
|
-
carries the two counts in its cause and `meta`. `RemoteEmbedder.decode` refuses its own provider
|
|
259
|
-
on the same rule; an app's own `Embedder` was the unchecked half.
|
|
260
|
-
- One `RemoteEmbedder` for every vendor: `baseUrl` selects the provider, the wire shape is the
|
|
261
|
-
same. Vectors are L2-normalised on arrival so `cosine` stays a dot product, and a width other
|
|
262
|
-
than the declared one is `X_VECTOR_DIM_MISMATCH` before anything reaches a store.
|
|
263
|
-
- A budget throws `X_AI_BUDGET_EXCEEDED` **before** the provider call. Never truncate.
|
|
87
|
+
## Invariants — `llm()`
|
|
88
|
+
|
|
89
|
+
- **`llm()` returns an `action`**, never a ninth primitive; it adds only the model half: render the
|
|
90
|
+
prompt, project `output` into the one `respond` tool, reserve the budget, consult the semantic cache.
|
|
91
|
+
- `src/index.ts` re-exports `t` from `@ultimat3/schema` **verbatim** (`index.test.ts`).
|
|
92
|
+
- One repair turn, then `X_LLM_OUTPUT_INVALID`. A refusal (a 200 with no answer) is `X_LLM_REFUSED`
|
|
93
|
+
before the parse, carrying `stopDetails.category`; a truncated invalid answer is `X_LLM_TRUNCATED`.
|
|
94
|
+
The gateway never caches a refusal.
|
|
95
|
+
- **Semantic scopes are separate cache INSTANCES**; the instance key carries the prompt hash AND
|
|
96
|
+
`ctx.locale` (in the unconditional half). **The default scope is the calling ACTOR**
|
|
97
|
+
(`JSON.stringify([actor.kind, actor.id, actor.orgId ?? null])`); `scope` receives
|
|
98
|
+
`{ input, ctx }`; widening is `scope: () => 'global'`. The instance map is bounded
|
|
99
|
+
(`MAX_SEMANTIC_CACHE_SCOPES`).
|
|
100
|
+
- `cache.invalidates` is not on `llm()`: a `SemanticCache` is not a `CacheTier`. Version bump + `ttl`
|
|
101
|
+
is the invalidation.
|
|
102
|
+
- The gateway is ambient (`configureAi`); absent at call time is `X_AI_GATEWAY_MISSING`, never a
|
|
103
|
+
default provider.
|
|
104
|
+
- **`llm()` forwards `ctx.signal`** onto `GenerateRequest`; `.stream()` inherits it.
|
|
105
|
+
- **`.stream()` is the SAME action**, marked with an ambient sink: it yields UNVALIDATED text and one
|
|
106
|
+
final `done` with the value that satisfied `output`; no repair turn (`X_LLM_STREAM_INVALID`); the
|
|
107
|
+
budget is reserved before the provider, debited on the first pull, reconciled at `done`; no
|
|
108
|
+
`respond` tool; a cache hit yields `done` alone. LAZY. `named()` is re-narrowed so the twin still streams.
|
|
109
|
+
- **`configureAi({ redact })` is the one seam between `vars()` and the provider**; a `Secret` in
|
|
110
|
+
`vars()` is `X_AI_PROMPT_SECRET` regardless. The `llm.redacted` span attribute records it.
|
|
111
|
+
- **Fallback is across PROVIDERS serving one model, never across models**; `GenerateResult.provider`
|
|
112
|
+
is stamped by the gateway and put on the span as `llm.provider`.
|
|
113
|
+
- Server-side `fallbacks` (beta) are never sent — the stable `2023-06-01` surface only.
|
|
114
|
+
- `definePrompt` refuses a re-registered version whose hash moved; the hash is over core's
|
|
115
|
+
`canonicalJson` (`prompt.test.ts` pins one hash literally); an absent schema hashes as `''`.
|
|
116
|
+
|
|
117
|
+
## Invariants — providers and the gateway
|
|
118
|
+
|
|
119
|
+
- **`attempt` collects TRANSPORT failures only**: an `UltimateError` other than
|
|
120
|
+
`X_AI_PROVIDER_UNAVAILABLE` is rethrown on the spot (the same misconfiguration everywhere). Read
|
|
121
|
+
with `stringField(error, 'code')`.
|
|
122
|
+
- **The gateway's reads of a provider's throw are total**: `isRetryable` fails closed; the line goes
|
|
123
|
+
through `renderThrowable`.
|
|
124
|
+
- **`X_AI_PROVIDER_UNAVAILABLE` is the one code classified `retryable`** (`AI_ERROR_RETRY`);
|
|
125
|
+
`AiProviderUnavailableError({ unserved: true })` carries a per-instance `terminal`. Every other code
|
|
126
|
+
stays UNREGISTERED on purpose.
|
|
127
|
+
- **The retry SCHEDULE is core's (`backoffDelay`, `isRetryableStatus`); the LOOP is this package's** —
|
|
128
|
+
core's `retry()` retries unclassified throws, and every value here is an app `Provider`'s plain
|
|
129
|
+
object, so it would retry a 400. `gateway-backoff.test.ts`. `RetryPolicy`/`DEFAULT_RETRY` keep the
|
|
130
|
+
field names an app writes.
|
|
131
|
+
- Every non-2xx and every in-band `error` frame is `AiTransportError` with a real `status`.
|
|
132
|
+
- **Both wire formats answer the same question the same way** (`provider-parity.test.ts`): an in-band
|
|
133
|
+
`error` in a 200 body throws (`throwInBandError`, `wire.ts`); a `refusal` stop detail forces
|
|
134
|
+
`stopReason: 'refusal'`; a tool call's `input` is parsed (`asToolInput`); the credential is scrubbed
|
|
135
|
+
from `detail` (`error-body.ts`'s `withoutKey`/`detailOf`). The status tables and the OpenAI
|
|
136
|
+
`estimatedUsage` fallback are asserted as differences.
|
|
137
|
+
- A stream ending without `message_stop` throws. **`readSse` caps the unterminated buffer**
|
|
138
|
+
(`MAX_FRAME_CHARS`); `provider` is required so a transport error names its endpoint.
|
|
139
|
+
- A tool call is emitted whole. Thinking chunks are never appended to `text`.
|
|
264
140
|
- Anthropic body: no `temperature`/`top_p`/`top_k`, no `budget_tokens`, `effort` inside
|
|
265
|
-
`output_config`.
|
|
266
|
-
- **`ModelId` is `string
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
a
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
`output_config.effort` and adaptive thinking arrived with 4.6, so one body sent to the whole
|
|
298
|
-
catalogue is a guaranteed 400 on the oldest entry — which is how `claude-haiku-4-5` shipped
|
|
299
|
-
blessed and uncallable. A control the caller never asked for is omitted; a control they DID
|
|
300
|
-
ask for is refused locally with `X_AI_REQUEST_INVALID`, never dropped, because a declaration
|
|
301
|
-
reading `effort: 'max'` that quietly runs at the default is the failure nobody can see. Adding
|
|
302
|
-
a model is a row in `MODELS`, never an `if` in the request builder. Omission is literal: an
|
|
303
|
-
absent `thinking` sends no block at all, where `(thinking ?? 'adaptive')` sent an adaptive one
|
|
304
|
-
for every adaptive-capable model — harmless on the wire, since adaptive is the server default,
|
|
305
|
-
but it made a defaulted control indistinguishable from a declared one, which is the whole
|
|
306
|
-
distinction this rule draws.
|
|
307
|
-
- **`openAiProvider()` is a FORMAT, not a vendor — decided 2026-08.** Azure OpenAI, vLLM, Ollama,
|
|
308
|
-
LiteLLM, OpenRouter, Together and most company gateways speak the OpenAI chat-completions wire
|
|
309
|
-
format, so one provider plus `baseUrl` is what makes "point Ultimate at our internal gateway"
|
|
310
|
-
real. Never add a second class per vendor: `baseUrl`, `auth` and `headers` are the differences.
|
|
311
|
-
- **Structured output is the `respond` tool, never `response_format`.** `llm()` already projects
|
|
312
|
-
`output` into one tool and reads the answer out of `toolCalls`; `json_schema` + `strict` would
|
|
313
|
-
be a SECOND structured-output path (axiom 1), would need the provider to synthesise a
|
|
314
|
-
`respond` call out of a content string, and is the one feature most OpenAI-*compatible* servers
|
|
315
|
-
do not implement. Forcing the function is what buys the reliability instead: `tool_choice`
|
|
316
|
-
names the tool when the request offers **exactly one**, which is precisely `llm()`'s shape and
|
|
317
|
-
never `agent()`'s — forcing a name inside a tool loop decides the model's next step for it.
|
|
318
|
-
- **`strict: true` is derived from the schema, never forwarded.** `LlmTool.strict` is `true` on
|
|
319
|
-
every projection; on this wire the server CHECKS it, and one optional field (a key in
|
|
320
|
-
`properties` absent from `required`) is a 400. `satisfiesStrictMode` is the gate, recursive
|
|
321
|
-
because the server's check is.
|
|
322
|
-
- `max_completion_tokens`, never `max_tokens`: the old field is rejected outright by every
|
|
323
|
-
current reasoning model. No `temperature`/`top_p`, same rule as the Anthropic body.
|
|
324
|
-
- **`stream_options: { include_usage: true }` on every streamed request, and an ESTIMATE when
|
|
325
|
-
usage never arrives.** Usage comes once, in a trailing chunk with an empty `choices` array, and
|
|
326
|
-
only when that field was sent — a compatible server that ignores it would otherwise leave the
|
|
327
|
-
budget reconciling a real call against zero, refunding the whole reservation. Zero is wrong by
|
|
328
|
-
all of it; an estimate is wrong by a few percent in the safe direction.
|
|
329
|
-
- `prompt_tokens` INCLUDES the cached prefix here, where Anthropic's `input_tokens` excludes it.
|
|
330
|
-
Subtract `prompt_tokens_details.cached_tokens` out of the input count or the cached half is
|
|
331
|
-
billed twice — once at the input rate, once at the cache rate.
|
|
332
|
-
- Tool-call deltas are merged by `tool_calls[].index`. The id and the name arrive on the FIRST
|
|
333
|
-
fragment and on no other, so merging by array position builds one call per chunk and keeps only
|
|
334
|
-
the last slice of arguments. The call is emitted whole at the finish reason — there is no
|
|
335
|
-
per-block stop event in this format.
|
|
336
|
-
- `isComplete()` accepts `[DONE]` **or** a finish reason: plenty of servers in the family close
|
|
337
|
-
the socket straight after the finish chunk, and a finish reason is the model saying why it
|
|
338
|
-
stopped, which a cut connection cannot produce. **With one exception, `As of 2026-08`:
|
|
339
|
-
`[DONE]` while tool-call fragments are still pending is NOT complete.** `onFinish` is the only
|
|
340
|
-
drain of `pending` and the finish reason is the only close this format has, so the sentinel
|
|
341
|
-
alone cannot tell "the model finished asking" from "the connection died mid-arguments" —
|
|
342
|
-
reporting complete discarded a whole tool call and answered an empty, successful `end_turn`.
|
|
343
|
-
Refused rather than flushed, exactly as the Anthropic half refuses a missing `message_stop`:
|
|
344
|
-
emitting the fragments would run a tool's side effects from half a JSON object.
|
|
345
|
-
- `role: 'system'`, not `developer` — the newer role is OpenAI's alone and every other server in
|
|
346
|
-
the family knows only `system`.
|
|
347
|
-
- **Only three models are priced** (`gpt-5.6-sol` / `-terra` / `-luna`, list price read
|
|
348
|
-
2026-08-16). `gpt-4o` and the `o1` family cache at 0.5x input where `costOf` assumes 0.1x, and
|
|
349
|
-
the `pro` tiers publish no cached rate — a wrong price is worse than a missing one, because
|
|
350
|
-
`costOf` answers confidently either way and `X_AI_MODEL_UNKNOWN` at least says so.
|
|
351
|
-
- The specs register at module scope, like the Anthropic three — so a suite that calls
|
|
352
|
-
`resetModels()` drops them. `registerOpenAiModels()` is exported for exactly that, and every
|
|
353
|
-
`openai-*.test.ts` calls it in `beforeEach`.
|
|
354
|
-
- **No new `X_*` code.** A non-2xx and an in-band `error` object are `AiTransportError`, a missing
|
|
355
|
-
key is `X_AI_KEY_MISSING`, a control the endpoint has not got is `X_AI_REQUEST_INVALID` — the
|
|
356
|
-
failures are the same failures, and a second code per provider would be a vocabulary that grows
|
|
357
|
-
with the driver list. What DID change: `AiTransportError` now takes the provider's `envVar`,
|
|
358
|
-
because the 401 fix line was a hardcoded `ANTHROPIC_API_KEY` for every provider in the package.
|
|
359
|
-
- The key is revealed as late as possible, never stored on the instance, and scrubbed out of the
|
|
360
|
-
error `detail` — a proxy echoing request headers into its 4xx body is the one path by which a
|
|
361
|
-
key reaches an error, and an error reaches a log index, a span and a problem document.
|
|
362
|
-
- Model IDs are exact aliases. Never append a date suffix.
|
|
363
|
-
- **No fix line may name `x ai`.** That command is PLANNED and throws (`packages/cli/src/cmd-planned.ts`),
|
|
364
|
-
so a fix citing `x ai reindex` sends an operator to a wall — an axiom-4 violation. Two shipped;
|
|
365
|
-
both now name the app-code fix instead.
|
|
366
|
-
- **An eval is selected with `x test eval --filter <name>`, never `x test <name>`.** `x test`'s
|
|
367
|
-
positional is a `TestType` (`unit contract live job e2e eval`), so the eval's own name there is
|
|
368
|
-
`X_CLI_BAD_FLAG`. `X_EVAL_THRESHOLD` shipped that fix line until 2026-08; `eval-errors.test.ts`
|
|
369
|
-
now asserts every `x test <word>` these five classes emit is one of the six types. The
|
|
370
|
-
`errors` step's `fix-command.ts` resolves the *command*, not its positional, so nothing else
|
|
371
|
-
would have caught it.
|
|
372
|
-
- The introductory price on a model is deliberately not modelled. A price that lapses on a date
|
|
373
|
-
makes a recorded cost depend on when it was read, and under-reporting spend after the lapse is
|
|
374
|
-
a budget that is not one. List price over-reserves, which is the safe direction.
|
|
375
|
-
- `generate()` above `STREAM_ONLY_MAX_TOKENS` runs the STREAMING transport and assembles the
|
|
376
|
-
result, rather than refusing. The ceiling is the transport's, not the model's.
|
|
377
|
-
- **`llm()` streams through `.stream()`, and it is the SAME action — decided 2026-08.** The
|
|
378
|
-
invocation is an ordinary one, marked with an ambient sink; policy, input parse, budget scope,
|
|
379
|
-
semantic cache, span, audit and `.tool()` all still apply, because there is no second execution
|
|
380
|
-
path. Before it existed, the first feature needing tokens on a screen called `aiGateway()`
|
|
381
|
-
directly and lost every one of them.
|
|
382
|
-
- **Output schema:** a schema cannot be checked until the last token lands, so a stream yields
|
|
383
|
-
UNVALIDATED text and one final `done` carrying the value that DID satisfy `output`. **No repair
|
|
384
|
-
turn** — the consumer has already read the tokens, and a second answer over the top is two
|
|
385
|
-
answers to one question. One attempt, then `X_LLM_STREAM_INVALID`, whose fix is the
|
|
386
|
-
non-streaming call.
|
|
387
|
-
- **Budget:** unchanged and still reserved before the provider is touched. `Gateway.stream`
|
|
388
|
-
debits the worst-case estimate on the first pull and reconciles at `done`, releasing in a
|
|
389
|
-
`finally`. The whole stream is driven inside the handler, so reservation and reconciliation sit
|
|
390
|
-
on one async chain; abandoning the iterator stops delivery, never the accounting.
|
|
391
|
-
- A streamed call offers **no `respond` tool**: a tool call is emitted whole, so forcing one
|
|
392
|
-
leaves nothing to stream. The answer is prose, and its JSON parse is what a non-string `output`
|
|
393
|
-
validates. A semantic-cache hit yields `done` alone, with no text increments.
|
|
394
|
-
- `.stream()` is LAZY. Nothing is authorised, budgeted or sent until the first pull. `named()` is
|
|
395
|
-
re-narrowed for the same reason `stream` is assigned in place: `action()`'s `named` builds a
|
|
396
|
-
fresh twin that would silently not stream.
|
|
397
|
-
- **`agent()` is a job for the tool loop, and one of the factory rule's instances** — the list is
|
|
398
|
-
`PRIMITIVE_FACTORIES` in `@ultimat3/core`, never an ordinal in a header — it returns an `action`,
|
|
399
|
-
never a ninth primitive. It exists because
|
|
400
|
-
the alternative is a hand-rolled loop, and a hand-rolled loop is where the dangerous mistake
|
|
401
|
-
lives: **taking the actor from the model's output.** `ctx.actor` is read once and is the only
|
|
402
|
-
identity any tool runs as; nothing the model emits can reach it. Bounded by `maxTurns`
|
|
403
|
-
(`X_AGENT_MAX_TURNS`, never a partial answer), by `budget.tokensPerRun` (the ledger's `request`
|
|
404
|
-
scope, which accumulates across turns) and by `maxToolResultChars` — the transcript IS the
|
|
405
|
-
request, so an untruncated tool result is re-billed once per remaining turn.
|
|
406
|
-
- **`tools` takes the real `action()`, adapted at this package's edge — decided 2026-08 (issue
|
|
407
|
-
#124).** It took `ProjectableAction` alone, which no `action()` structurally satisfies (an
|
|
408
|
-
action is `as`/`tool`/`openapi`/`job`/`contract` and the callable, never `run`), so the shape
|
|
409
|
-
the README documents was a `TS2741` and every test here hand-built a stand-in — which is why
|
|
410
|
-
the suite was green over an API that did not compile. `AgentTool = AnyAction |
|
|
411
|
-
ProjectableAction` and `asProjectableAction` are the fix, the same union `@ultimat3/mcp`'s
|
|
412
|
-
`ListedPrimitive` already accepted. **Not** a `run` member on the action facade, which was the
|
|
413
|
-
obvious alternative and is wrong three times over: it duplicates `.as()` (axiom 1), it would
|
|
414
|
-
offer a tool named `''` for an action `registerActions` has not named yet, and it would NOT
|
|
415
|
-
collapse `mcp`'s adapter, whose `toWireSchema` narrows to the subset that server's arg
|
|
416
|
-
validator can enforce while this one publishes the Messages API's (`packages/mcp/src/
|
|
417
|
-
from-action.ts` header). Two wire formats, two projections, one `invoke`.
|
|
418
|
-
- Projection happens on the FIRST RUN, memoised — never at declaration. `agent()` is evaluated at
|
|
419
|
-
module scope and `registerAction` stamps a name at boot, so `actionName()` at declaration would
|
|
420
|
-
make the ordinary `export const publishPost = action(...)` beside it `X_ACTION_UNREGISTERED`.
|
|
421
|
-
- **An `agent()` is a tool of another `agent()`** — it returns an action, and an action is what a
|
|
422
|
-
tool is. That is the supervisor/sub-agent shape, with no `hive()` and no ninth primitive; the
|
|
423
|
-
sub-agent runs under the same actor, through its own policy.
|
|
424
|
-
- A tool listed in `agent({ tools })` that is not `mcp: { expose: true }` is
|
|
425
|
-
`X_AGENT_TOOL_UNEXPOSED` **at declaration**, not filtered at the call: a silently dropped tool
|
|
426
|
-
reads as offered and is not. `isMcpExposed` is the one predicate, so an in-app agent and an
|
|
427
|
-
external MCP client see the same catalogue. Asked of the DECLARATION, not of the projection,
|
|
428
|
-
because exposure needs no name and the name does not exist yet.
|
|
429
|
-
- **`ctx.signal` is read, at three points.** `throwIfAborted` at the top of every turn and before
|
|
430
|
-
every tool batch, and `GenerateRequest.signal` forwarded into `fetch` by both providers. The
|
|
431
|
-
transcript IS the request, so a loop that keeps going after the caller disconnects re-sends it
|
|
432
|
-
once per remaining turn, runs every remaining tool's side effect and discards the answer —
|
|
433
|
-
eight provider calls for an answer nothing reads. `X_ABORTED` is core's, already shipped; no
|
|
434
|
-
new code. `signal` is deliberately absent from `cacheKeyFor` and from every estimate: it says
|
|
435
|
-
whether a call was ABANDONED, never what it asked for.
|
|
436
|
-
- **The tools of ONE turn run concurrently**, unbounded within the turn, results paired by index
|
|
437
|
-
so a fast tool cannot be matched to a slow tool's `tool_use` id. Serial cost 5x wall clock for a
|
|
438
|
-
turn that asked for five tools and nothing in the types or the docs said so. No second ceiling
|
|
439
|
-
here: the batch is what one model turn asked for, each entry is an action with its own `policy`
|
|
440
|
-
and `rateLimit`, and a tool that calls a model still queues on the ledger's root turnstile.
|
|
441
|
-
Guarantee is "no tool STARTS after the abort" — a tool already in flight unwinds through its
|
|
442
|
-
own handler's reading of `ctx.signal`, which is the action's to make.
|
|
443
|
-
- **`onTurn` is the per-turn observation, and `.stream()` on `agent()` is deliberately not shipped
|
|
444
|
-
yet.** A 90-second multi-turn run emitted nothing until it returned. `onTurn` reports facts
|
|
445
|
-
only — turn, model, tool names, stop reason, usage, that turn's cost — never the transcript and
|
|
446
|
-
never the actor, and the same facts land on the span as an `agent.turn` event so a run
|
|
447
|
-
declaring no hook is still readable in a trace. A throw from it FAILS the run: an observer that
|
|
448
|
-
quietly stopped working reads exactly like one that is fine. Tokens on a screen is a different
|
|
449
|
-
contract from turns in a loop, and half-shipping it would be the second path axiom 1 refuses.
|
|
450
|
-
- **No semantic cache on `agent()`.** Similar prompts do not have similar answers once the answer
|
|
451
|
-
depends on what `lookupOrder` returned this second.
|
|
452
|
-
- **Every `tool_use` block the transcript replays is answered by a `tool_result` in the very next
|
|
453
|
-
message** — the Messages API's own rule, and `agent-transcript.ts` owns both halves of it for
|
|
454
|
-
that reason. Two paths broke it, both through `respond`, which is filtered out of the calls
|
|
455
|
-
that RUN and replayed like any other block: a turn emitting a tool call AND `respond` together
|
|
456
|
-
(ordinary parallel tool use), and a `respond` whose input failed the output schema, followed by
|
|
457
|
-
a plain user message. Both were a 400 (`tool_use ids were found without tool_result blocks`),
|
|
458
|
-
i.e. `X_AI_PROVIDER_UNAVAILABLE` in place of a completed run, and `agent.test.ts` never mixed
|
|
459
|
-
the two so neither shipped visible. An unaccepted `respond` now comes back as its own
|
|
460
|
-
`tool_result`, `is_error`, saying why — the answer is SUPERSEDED, not wrong: it was written
|
|
461
|
-
before the results of the tools the same turn asked for existed, so the loop continues and the
|
|
462
|
-
model answers again with them in hand. Discarding the block instead would have been the other
|
|
463
|
-
legal fix and loses the record; USING the speculative answer would skip the tool results the
|
|
464
|
-
model itself asked for, after those tools already ran.
|
|
465
|
-
- **A tool result is rendered totally.** `runLlmToolCall` returns `content: string` and the loop
|
|
466
|
-
TRUNCATES it, so `JSON.stringify`'s other two answers both have to be handled: `undefined` for
|
|
467
|
-
an action that returns nothing (`'null'`), and a throw on a bigint, a cycle or a `toJSON` of
|
|
468
|
-
the value's own — reported as "the tool ran and its result is not JSON", never as a failure,
|
|
469
|
-
because a model told the tool failed calls it again and buys its side effects twice. The throw
|
|
470
|
-
it catches is read with `stringField`, never `typeof error.code === 'string'`: the value is an
|
|
471
|
-
app's, so the probe is a getter call or a `Proxy` trap inside the catch block.
|
|
472
|
-
- `AiMessage.content` widened to `string | readonly AiContentBlock[]` for this: a `tool_result`
|
|
473
|
-
has to name the `tool_use` it answers and a string has nowhere to put the id. The block field
|
|
474
|
-
names are the Messages API's, so `body()` passes them through untouched.
|
|
475
|
-
- **`hive()` is a fan-out, and another of the factory rule's instances** (`PRIMITIVE_FACTORIES`
|
|
476
|
-
again) — it returns an `action`, never a ninth primitive. It exists because
|
|
477
|
-
the alternative is a hand-rolled `Promise.all` over `agent()` calls, and that loop gets four
|
|
478
|
-
things wrong every time: the actor, the order, the difference between ran-and-failed and
|
|
479
|
-
never-ran, and the ceiling.
|
|
480
|
-
- **`HiveResult` is a SCHEMA, not an interface**, built from the member action's own `output` and
|
|
481
|
-
embedded in the `ok` arm. That is what makes a hive project to OpenAPI, the typed client, the
|
|
482
|
-
MCP `outputSchema` and the manifest like any other action; a hand-written interface would have
|
|
483
|
-
given the type and none of the six projections, which is the whole reason it is a factory.
|
|
484
|
-
- **Three arms — `ok` / `failed` / `skipped` — never two.** A member that ran and threw and a
|
|
485
|
-
member that never ran are different facts, and an aborted sibling is the second. Two arms make
|
|
486
|
-
"the hive stopped early" indistinguishable from "every remaining item is bad data", which is
|
|
487
|
-
the difference between retrying the tail and fixing the source. `skipped` gets its own counter
|
|
488
|
-
beside `ok` and `failed` for the same reason: three arms and two counters means every caller
|
|
489
|
-
writes `members.length - ok - failed` once, and writes it wrong once.
|
|
490
|
-
- **`members` is in SPLIT order, always**, filled by index rather than pushed on settle, with
|
|
491
|
-
`index` on every arm so a caller can join a result back to its row without depending on array
|
|
492
|
-
position surviving a filter.
|
|
493
|
-
- **The hive never names an actor.** `split` derives member inputs from `input` and `ctx` and
|
|
494
|
-
from nothing a model emitted; each member runs through its own callable, so `invoke` applies
|
|
495
|
-
the member's own `policy` with `ctx.actor` untouched. There is no `as(actor)` in the factory —
|
|
496
|
-
the same boundary `agent()` holds, and the reason both belong in the framework.
|
|
497
|
-
- **No hive-specific budget code, deliberately.** One derived ledger for the run, `withBudget`
|
|
498
|
-
around the pool, each member's `agent()` deriving again — and the ceiling holds under
|
|
499
|
-
parallelism because `reserve` DEBITS on the root's turnstile before the call. Three members
|
|
500
|
-
against a ceiling only one fits leave exactly one `ok`; that is asserted through the hive
|
|
501
|
-
rather than asserted about the ledger, because the ledger already promised it.
|
|
502
|
-
- **A member's throw is RECORDED, whatever it is.** `failureOf` reads it with `isThrownError` and
|
|
503
|
-
`stringField` from core, never `error instanceof Error` and `.message`: a member is an app's
|
|
504
|
-
action, so a `Proxy` makes `instanceof` run a `getPrototypeOf` trap, and a throw there takes
|
|
505
|
-
down the whole hive — the one outcome the three arms exist to prevent. `skipped` has two
|
|
506
|
-
reasons, because they are two facts: `SKIPPED_ABORTED` (a sibling failed under `'abort'`) and
|
|
507
|
-
`SKIPPED_NO_INPUT` (the split produced nothing at that index). One string for both sent a
|
|
508
|
-
caller to retry a tail that was never cut.
|
|
509
|
-
- **`onMemberError` is required.** `'abort'` stops and leaves the rest `skipped`; `'collect'`
|
|
510
|
-
harvests. Both are right for somebody, so neither may be inherited silently.
|
|
511
|
-
- `concurrency` defaults to 4 and `minMembers` to 2, and neither number is measured off any run —
|
|
512
|
-
the framework cannot know a provider's concurrency allowance. A below-floor split still runs
|
|
513
|
-
every input it produced, serially: dropping one would be silent data loss.
|
|
514
|
-
- An empty split is `X_HIVE_EMPTY`, never a successful run of zero members — "0 ok, 0 failed"
|
|
515
|
-
cannot be told apart from a query that returned no rows and nobody noticed.
|
|
516
|
-
- An aborted `ctx` unwinds the whole hive with `X_ABORTED`, which is a DIFFERENT event from
|
|
517
|
-
`onMemberError: 'abort'`: the latter is a completed run with a partial harvest worth returning,
|
|
518
|
-
the former has nobody left to hand it to.
|
|
519
|
-
- **`describeAgents()` is OFFERED, not published, `As of 2026-08-23`** — no reader exists anywhere
|
|
520
|
-
in the tree, and neither does one for `registeredModels()`. Both are tier-4 exports whose only
|
|
521
|
-
legal consumer is `@ultimat3/cli` (tier 5); `@ultimat3/manifest` is tier 4, so the obvious import
|
|
522
|
-
is a sideways edge `bun run boundaries` refuses. Their doc comments say so now, because a comment
|
|
523
|
-
claiming a consumer that does not exist is the documentation axiom 3 calls no rule at all.
|
|
524
|
-
- **`describeAgents()` describes what an `ActionDescriptor` cannot.** An agent projects to the same
|
|
525
|
-
descriptor as any other action, and that descriptor knows nothing about turns, tools, models or
|
|
526
|
-
prompt hashes — so "how far can this loop and what may it call" had no answer outside the source.
|
|
527
|
-
Same shape as `describePrompts()` / `describeEvals()`, and deliberately NOT a new
|
|
528
|
-
`ActionDescriptor` field: `@ultimat3/action` is tier 3 and knows nothing about models.
|
|
529
|
-
The facts are a THUNK, resolved when asked: `agent()` runs at module scope beside the actions it
|
|
530
|
-
lists, and every name in a row is stamped by `registerAction` at boot. An agent still carrying no
|
|
531
|
-
name has no row — not a silent drop, but the absence of a capability: an action with no name
|
|
532
|
-
reaches no route, no tool catalogue and no queue. `named()` builds a TWIN where registration
|
|
533
|
-
names in place, so an agent renamed that way is absent for the same reason; register it instead.
|
|
534
|
-
- **`agentJob()` closes #125 for the agent case, by COMPOSING `job()`** — the returned value is one
|
|
535
|
-
`job()` seated in that package's own registry, so `.enqueue()`, the outbox, the worker's
|
|
536
|
-
cancellation, the dead-letter path, `x jobs show` and its manifest row arrive without a line here.
|
|
537
|
-
Never an imitation handle: `isJobHandle` needs `kind === 'job'` plus membership of a WeakMap only
|
|
538
|
-
`job()` writes, which is exactly what stops a second execution path existing.
|
|
539
|
-
- `name`, `tenant` and `retry` are REQUIRED, no defaults. `name` because a job name is the durable
|
|
540
|
-
queue key that queued, retrying and dead-lettered rows already carry — deriving it from the
|
|
541
|
-
export name would move delivery when somebody renames a variable. `tenant` and `retry` because
|
|
542
|
-
`jobs` states that every candidate default for `tenant` is a cross-tenant read waiting for the
|
|
543
|
-
first job that takes an org id in its input. Both are `TS2741` when omitted AND have runtime
|
|
544
|
-
backstops (`X_JOB_TENANT_REQUIRED`, the `retry.attempts` assert), for generated and JS callers.
|
|
545
|
-
- **Both reads of `target.job()` are LAZY**, and that is load-bearing: `actionName()` throws
|
|
546
|
-
`X_ACTION_UNREGISTERED` until boot stamps the export name, and `agentJob()` is evaluated at
|
|
547
|
-
module scope right beside the `agent()` it wraps. Same rule as `agent()`'s tool projection and
|
|
548
|
-
`describeAgents()`' thunk — third instance in this package.
|
|
549
|
-
- **The at-least-once trap is DOCUMENTED, not enforced, and that is a decision with evidence.**
|
|
550
|
-
`idempotencyKey` dedupes the ENQUEUE, never the ATTEMPT: a lost lease re-runs the agent from the
|
|
551
|
-
top, as does every page `backfill()` replays. So every tool an `agentJob()`'d agent may call has
|
|
552
|
-
to be idempotent. The framework cannot check it: `mutates` is not a fact an `action()` declares
|
|
553
|
-
— it exists only in `@ultimat3/mcp`, whose `projectable.ts` sets it to `true` for EVERY action —
|
|
554
|
-
so a read-only `lookupOrder` and a destructive `issueRefund` are indistinguishable, and
|
|
555
|
-
`ActionFacade`'s `Pick<>` carries no `idempotent` either (only `describe()` does, and that needs
|
|
556
|
-
a name). `isMutator` IS legible without a name, but `mutator()` is the local-first write
|
|
557
|
-
primitive: refusing only those would catch almost none of the risk while reading as if it caught
|
|
558
|
-
all of it. A wrong refusal is worse than a stated obligation, so the obligation is stated — in
|
|
559
|
-
`AgentJobOptions.idempotencyKey`'s doc comment and in the README, both naming the second refund.
|
|
141
|
+
`output_config`. Model IDs are exact aliases, never date-suffixed.
|
|
142
|
+
- **`ModelId` is `string`; the catalogue is an open registry** — `modelSpec(id)` refuses an
|
|
143
|
+
unregistered id (`X_AI_MODEL_UNKNOWN`). The built-ins register through the same `registerModel`.
|
|
144
|
+
**Re-registering REPLACES the spec and keeps its rung** (negotiated rates). **Registration order is
|
|
145
|
+
the ladder within a `family`**, read only by `moreCapableThan`; `X_LLM_REFUSED`'s fix names a rung
|
|
146
|
+
ABOVE or nothing.
|
|
147
|
+
- `AnthropicProvider.models` is its own list; `EchoProvider.models` reads the registry.
|
|
148
|
+
- **The reasoning controls are PER MODEL (`models.ts`)**: an unasked control is omitted; an asked
|
|
149
|
+
control the model lacks is `X_AI_REQUEST_INVALID`, never dropped. Adding a model is a row in `MODELS`.
|
|
150
|
+
- `generate()` above `STREAM_ONLY_MAX_TOKENS` runs the streaming transport.
|
|
151
|
+
- **`openAiProvider()` is a FORMAT, not a vendor** — `baseUrl`, `auth`, `headers` are the
|
|
152
|
+
differences; never a second class per vendor:
|
|
153
|
+
- structured output is the `respond` tool, never `response_format`; `tool_choice` names the tool
|
|
154
|
+
only when the request offers exactly one;
|
|
155
|
+
- `strict: true` only when `satisfiesStrictMode` says the schema qualifies;
|
|
156
|
+
- `max_completion_tokens`; `role: 'system'`; `stream_options: { include_usage: true }` and an
|
|
157
|
+
ESTIMATE when usage never arrives; subtract `cached_tokens` from `prompt_tokens`;
|
|
158
|
+
- tool-call deltas merge by `tool_calls[].index`; `isComplete()` accepts `[DONE]` or a finish reason,
|
|
159
|
+
except `[DONE]` with pending tool fragments, which is refused;
|
|
160
|
+
- only `gpt-5.6-sol` / `-terra` / `-luna` are priced; `registerOpenAiModels()` re-registers after
|
|
161
|
+
`resetModels()` (every `openai-*.test.ts` calls it in `beforeEach`);
|
|
162
|
+
- no new `X_*` code; `AiTransportError` takes the provider's `envVar`; the key is revealed late,
|
|
163
|
+
never stored, scrubbed from `detail`.
|
|
164
|
+
- **`embedBatched` enforces the `Embedder` arity per batch** (`X_AI_EMBEDDER_INVALID` with both
|
|
165
|
+
counts). One `RemoteEmbedder` for every vendor; vectors L2-normalised; a wrong width is
|
|
166
|
+
`X_VECTOR_DIM_MISMATCH`.
|
|
167
|
+
- **A caught value is read with `renderThrowable`** (`remote-embedder.ts`, `wire.ts`,
|
|
168
|
+
`openai-wire.ts`) — `scripts/error-render.ts` cannot see `catch` bindings.
|
|
169
|
+
- **A caller's string is never an object KEY**: the wire tables are `Map`s; `prompt.render` and
|
|
170
|
+
`EchoProvider`'s `replies` use `Object.hasOwn`.
|
|
171
|
+
|
|
172
|
+
## Invariants — `agent()`, `hive()`, `agentJob()`
|
|
560
173
|
|
|
561
|
-
- **`
|
|
562
|
-
|
|
563
|
-
|
|
564
|
-
|
|
565
|
-
|
|
566
|
-
|
|
567
|
-
|
|
568
|
-
|
|
569
|
-
|
|
570
|
-
|
|
571
|
-
|
|
572
|
-
|
|
573
|
-
|
|
574
|
-
|
|
575
|
-
|
|
576
|
-
|
|
577
|
-
|
|
578
|
-
|
|
579
|
-
|
|
580
|
-
|
|
581
|
-
|
|
582
|
-
|
|
583
|
-
|
|
584
|
-
|
|
585
|
-
-
|
|
586
|
-
|
|
587
|
-
|
|
588
|
-
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
597
|
-
|
|
598
|
-
|
|
599
|
-
|
|
600
|
-
|
|
601
|
-
|
|
602
|
-
|
|
603
|
-
-
|
|
604
|
-
|
|
605
|
-
|
|
606
|
-
- The
|
|
607
|
-
|
|
608
|
-
-
|
|
609
|
-
`
|
|
610
|
-
|
|
611
|
-
depending on where the suite was started, which is how a gate silently stops gating.
|
|
612
|
-
- Every registered prompt must be named by an eval (`promptsWithoutEvals`, `X_EVAL_MISSING`).
|
|
613
|
-
Coverage is by prompt ID, not ref: old versions are retained, and an eval on the current one
|
|
614
|
-
evaluates that lineage.
|
|
615
|
-
- `ULTIMATE_EVAL_RECORD=1` writes baselines instead of gating on them. A test that deliberately
|
|
616
|
-
scores a worse model calls `run`, never `assert` — `assert` would re-record during that pass.
|
|
617
|
-
- **Recording and the gate are mutually exclusive.** `x verify` with that variable set is
|
|
618
|
-
`X_EVAL_RECORDING` and runs no eval suite at all. Recording passes by definition, so a gate run
|
|
619
|
-
that inherited the flag is green over numbers it wrote itself — and rewrites every committed
|
|
620
|
-
baseline on its way through, which is the half a red step would not undo. Hence refuse *before*
|
|
621
|
-
the suite, never after it.
|
|
622
|
-
- The gate asks whether an eval has a baseline, not only whether one is declared. `defineEval`
|
|
623
|
-
proves a prompt is named; it proves nothing measured it, and an eval whose numbers were never
|
|
624
|
-
recorded — one no test asserts, one whose `baseline:` is a cwd-relative string — would otherwise
|
|
625
|
-
satisfy `X_EVAL_MISSING` while gating on nothing.
|
|
626
|
-
- Retrieval is hybrid by default. Do not add a vector-only convenience path.
|
|
627
|
-
- **`chunk()` performs all three splits its header names — paragraph, sentence, HARD WRAP** (`As of
|
|
628
|
-
2026-08`). The wrap is what bounds a unit, and an oversized unit is one the size check can never
|
|
629
|
-
flush (it only fires when something is already in the buffer), so it re-seeded every following
|
|
630
|
-
chunk: a ~1,000-token document indexed as nine chunks totalling ~9,000, each carrying the same
|
|
631
|
-
sentence. The overlap carry stops at `buffer.length - 1` for the same reason — a flushed unit may
|
|
632
|
-
never become the whole of the next buffer.
|
|
633
|
-
- **A caller's string is never used as an object KEY.** A `Record` lookup on one answers
|
|
634
|
-
`constructor`, `toString` and `valueOf` off the prototype chain, and every read here is of a
|
|
635
|
-
string a provider, a template author or a test fixture chose: the two wire tables are `Map`s,
|
|
636
|
-
`prompt.render` uses `Object.hasOwn` (a `{{constructor}}` slot rendered JS source into a billed
|
|
637
|
-
prompt and hashed it into the cache key), and so does `EchoProvider`'s `replies`.
|
|
638
|
-
- `PgVectorStore` is the ONLY production vector path — pgvector and Postgres FTS in the app's own
|
|
639
|
-
Postgres, never a second datastore. `MemoryVectorStore` is the dev twin and enforces the same
|
|
640
|
-
envelope; a leak that only reproduces against real Postgres is a leak nobody finds.
|
|
641
|
-
- **It is proved against a real pgvector, not only against a recording client.**
|
|
642
|
-
`pg-vector.live.test.ts` runs the whole chain — `ddl()` -> a live server -> `upsert` -> cosine,
|
|
643
|
-
FTS and the RRF fusion -> decoded hit — and REFUSES to skip when `TEST_DATABASE_URL` names a
|
|
644
|
-
Postgres without the extension, because a suite that stands down reports green for the one
|
|
645
|
-
store that runs in front of real traffic. CI's service container is `pgvector/pgvector:pg17`
|
|
646
|
-
for that reason. Asserting statement *text* cannot catch a statement Postgres rejects, nor a
|
|
647
|
-
filter that compiles cleanly and excludes nothing: that is exactly how metadata shipped bound
|
|
648
|
-
`::jsonb`. A new operator, read path or scope rule is not done until it round-trips there.
|
|
649
|
-
- The distance ordering lives in a subquery, ascending and raw, because that is the only shape
|
|
650
|
-
hnsw answers — `order by 1 - (…) desc` is a sequential scan. Both halves are pinned by a plan
|
|
651
|
-
assertion in the live suite, since only a planner can say which one shipped.
|
|
652
|
-
- hnsw applies the scope AFTER the index scan, so an approximate index can return fewer rows
|
|
653
|
-
than asked for once a tenant filter is selective. The planner takes the exact path instead
|
|
654
|
-
when it has stats — which is why a bulk backfill that skips `analyze` is how a search that
|
|
655
|
-
used the index yesterday scans today. Assert the rows a scoped read returns, never the node.
|
|
656
|
-
- Tenant and policy filters go **in SQL**, on every statement, through `conditionsSql` — and on
|
|
657
|
-
BOTH halves of the fusion. Filtering after the rows are loaded is not filtering.
|
|
658
|
-
- `(tenant, id)` is the primary key. A cross-tenant overwrite is impossible at the storage layer
|
|
659
|
-
rather than conditional on every upsert remembering to check.
|
|
660
|
-
- `scoped()` only ever TIGHTENS: tenants are set once, allow-lists intersect. Widening is
|
|
661
|
-
`X_VECTOR_SCOPE_WIDENED`. Same rule as `budget.derive`, for the same reason.
|
|
662
|
-
- Metadata is bound `::text::jsonb`. A bound string cast straight to `::jsonb` is JSON-encoded
|
|
663
|
-
twice, reads back correctly, and makes every `metadata ->> key` filter match nothing.
|
|
174
|
+
- **`agent()` returns an action** (listed in `PRIMITIVE_FACTORIES`). `ctx.actor` is read once and is
|
|
175
|
+
the only identity any tool runs as — never taken from the model's output. Bounded by `maxTurns`
|
|
176
|
+
(`X_AGENT_MAX_TURNS`), `budget.tokensPerRun` and `maxToolResultChars`.
|
|
177
|
+
- `tools` takes a real `action()` (`AgentTool = AnyAction | ProjectableAction`,
|
|
178
|
+
`asProjectableAction`); projection happens on the FIRST RUN, memoised. An agent can be another
|
|
179
|
+
agent's tool.
|
|
180
|
+
- A tool not `mcp: { expose: true }` is `X_AGENT_TOOL_UNEXPOSED` at DECLARATION.
|
|
181
|
+
- `ctx.signal` is read at the top of every turn, before every tool batch, and forwarded to
|
|
182
|
+
`fetch`; `X_ABORTED` is core's.
|
|
183
|
+
- One turn's tools run concurrently, results paired by index; no tool STARTS after an abort.
|
|
184
|
+
- `onTurn` reports facts only (also an `agent.turn` span event); a throw from it fails the run.
|
|
185
|
+
`.stream()` on `agent()` is deliberately not shipped. No semantic cache.
|
|
186
|
+
- Every replayed `tool_use` is answered by a `tool_result` in the very next message
|
|
187
|
+
(`agent-transcript.ts`); an unaccepted `respond` comes back as an `is_error` `tool_result`
|
|
188
|
+
("superseded").
|
|
189
|
+
- A tool result is rendered totally (`'null'` for `undefined`; a non-JSON result reported as such,
|
|
190
|
+
never as a failure); the throw is read with `stringField`.
|
|
191
|
+
- **`hive()` is a fan-out action** (`PRIMITIVE_FACTORIES`): `HiveResult` is a SCHEMA from the
|
|
192
|
+
member's `output`; three arms (`ok`/`failed`/`skipped`) and three counters; `members` in SPLIT
|
|
193
|
+
order with `index`; the hive never names an actor; no hive budget code (the root turnstile holds
|
|
194
|
+
it); a member's throw is recorded via `isThrownError`/`stringField`; `skipped` reasons
|
|
195
|
+
`SKIPPED_ABORTED` / `SKIPPED_NO_INPUT`; `onMemberError` is required; `concurrency` 4, `minMembers`
|
|
196
|
+
2 (a below-floor split still runs serially); an empty split is `X_HIVE_EMPTY`; an aborted `ctx`
|
|
197
|
+
unwinds with `X_ABORTED`.
|
|
198
|
+
- **`describeAgents()` / `registeredModels()` are offered, not published** — their only legal consumer
|
|
199
|
+
is `@ultimat3/cli`. The facts are a THUNK; an unnamed agent has no row.
|
|
200
|
+
- **`agentJob()` composes `job()`** (never an imitation handle). `name`, `tenant`, `retry` are
|
|
201
|
+
REQUIRED; both reads of `target.job()` are LAZY. **Every tool an `agentJob()`'d agent may call must
|
|
202
|
+
be idempotent** — stated in `AgentJobOptions.idempotencyKey`'s doc and the README, not enforceable.
|
|
203
|
+
|
|
204
|
+
## Invariants — evals, retrieval, fix lines
|
|
205
|
+
|
|
206
|
+
- **No fix line may name `x ai`** (planned). An eval is selected with `x test eval --filter <name>`;
|
|
207
|
+
`eval-errors.test.ts` asserts every `x test <word>` is one of the six types.
|
|
208
|
+
- Every eval result carries the prompt hash. An eval gates on the DROP from its recorded baseline,
|
|
209
|
+
mean AND per case. Never-recorded is `X_EVAL_BASELINE_MISSING`, corrupt is `X_EVAL_BASELINE_INVALID`.
|
|
210
|
+
`baseline` is `import.meta.resolve('./…')`. Every prompt is named by an eval (`X_EVAL_MISSING`, by
|
|
211
|
+
prompt ID). `ULTIMATE_EVAL_RECORD=1` records; a deliberately-worse test calls `run`, never
|
|
212
|
+
`assert`; `x verify` with the flag is `X_EVAL_RECORDING` and runs no eval. The gate asks for a
|
|
213
|
+
recorded baseline, not only a declaration.
|
|
214
|
+
- Retrieval is hybrid by default; no vector-only path. **`chunk()` hard-wraps**, and the overlap
|
|
215
|
+
carry stops at `buffer.length - 1`.
|
|
216
|
+
- `PgVectorStore` is the only production vector path (pgvector + FTS in the app's Postgres);
|
|
217
|
+
`MemoryVectorStore` enforces the same envelope. **`pg-vector.live.test.ts` refuses to skip** on a
|
|
218
|
+
Postgres without the extension (CI uses `pgvector/pgvector:pg17`).
|
|
219
|
+
- The distance ordering lives in an ascending raw subquery (the only shape hnsw answers), pinned by a
|
|
220
|
+
plan assertion. hnsw scopes after the scan — assert rows, never the node; `analyze` after a backfill.
|
|
221
|
+
- Tenant and policy filters are in SQL on every statement (`conditionsSql`), on BOTH fusion halves.
|
|
222
|
+
`(tenant, id)` is the primary key. `scoped()` only TIGHTENS (`X_VECTOR_SCOPE_WIDENED`). Metadata is
|
|
223
|
+
bound `::text::jsonb`.
|
|
664
224
|
|
|
665
225
|
## Commands
|
|
666
226
|
|
|
@@ -673,3 +233,5 @@ docker run -d -e POSTGRES_PASSWORD=ultimate -p 5432:5432 pgvector/pgvector:pg17
|
|
|
673
233
|
TEST_DATABASE_URL=postgres://postgres:ultimate@localhost:5432/postgres \
|
|
674
234
|
bun test packages/ai/src/pg-vector.live.test.ts
|
|
675
235
|
```
|
|
236
|
+
|
|
237
|
+
Why each rule above is shaped the way it is: [`docs/history/ai.md`](../../docs/history/ai.md).
|