@ultimat3/ai 9.0.0 → 10.0.0

package/CLAUDE.md

@@ -54,7 +54,7 @@ until 2026-08, naming a tool no catalog contained (`llm.test.ts`, `agent.test.ts
 | `llm-stream.ts` | `.stream()`'s plumbing: the sink, the ambient mark, the one-turn drive |
 | `agent.ts` | `agent()` — the tool loop, declared as an `action` |
 | `agent-transcript.ts` | what one turn leaves in the transcript: the assistant replay, the tool results, the correction |
-| `agent-facts.ts` | `describeAgents()` — the agent registry and the row a manifest publishes |
+| `agent-facts.ts` | `describeAgents()` — the agent registry, and the row a manifest WOULD publish: nothing reads it yet, because `manifest` is tier 4 like this package and the consumer has to be `cli` at tier 5 |
 | `agent-job.ts` | `agentJob()` — an agent as a real `JobHandle`, composed from `job()` |
 | `hive.ts` | `hive()` — one action fanned out over many inputs, declared as an `action` |
 | `hive-result.ts` | `HiveMember` / `HiveResult`, and the SCHEMA built from the member's own `output` |
@@ -120,6 +120,25 @@ until 2026-08, naming a tool no catalog contained (`llm.test.ts`, `agent.test.ts
   zero leaks the ceiling upward on every release.
 - Cost is `Money` (integer minor units), rounded **up**. Never a float, never a division
   that loses a fraction.
+- **`attempt` collects TRANSPORT failures only, `As of 2026-08-23`.** A caught value that is an
+  `UltimateError` whose code is not `X_AI_PROVIDER_UNAVAILABLE` is rethrown verbatim, on the spot:
+  those are raised locally, before the socket opens — a missing credential, a control the model does
+  not accept — so the same rejection is waiting on every provider and every attempt, which is
+  exactly the reason a 400 is not retried. Collecting one flattened `X_AI_KEY_MISSING` and its
+  runnable `export ANTHROPIC_API_KEY=…` into "no provider could serve model X", and made
+  `generate()` and `stream()` answer the SAME misconfiguration two different ways — `stream()`
+  never routes through `attempt`. `AiTransportError` carries `X_AI_PROVIDER_UNAVAILABLE`, so
+  "provider one 503'd, provider two timed out" still collects across candidates unchanged.
+  **Breaking**: a caller catching `X_AI_PROVIDER_UNAVAILABLE` today starts seeing the real code.
+  The read is `stringField(error, 'code')`, never `error.code` — the value came from an app's
+  `Provider` and a property read on it is a getter call or a `Proxy` trap.
+- **`Gateway.stream` routes BEFORE it reserves.** `providerFor` throws for a registered model no
+  configured provider serves, and it used to run between `reserve()` and the `try` that releases —
+  so the estimate was debited, landed on the `BudgetStore` under `actorKey`/`orgKey`, and nothing
+  credited it back. `MemoryBudgetStore` is per process and never expires, so five refused streams
+  spent an org's whole ceiling with nothing ever sent, and every later call in that process was
+  `X_AI_BUDGET_EXCEEDED` forever. `settled = true` moved AFTER `await ledger.record(...)` for the
+  same reason: a store that throws at `done` left the reservation both unreleased and half-recorded.
 - **The gateway's two reads of a provider's throw are total.** A `Provider` is the APP's object, so
   the value it rejects with is one the framework did not build: `isRetryable` indexes it (a getter,
   or a `Proxy` trap) and fails closed if the read raises, and the failure line goes through core's
@@ -424,7 +443,12 @@ until 2026-08, naming a tool no catalog contained (`llm.test.ts`, `agent.test.ts
   - An aborted `ctx` unwinds the whole hive with `X_ABORTED`, which is a DIFFERENT event from
     `onMemberError: 'abort'`: the latter is a completed run with a partial harvest worth returning,
     the former has nobody left to hand it to.
-- **`describeAgents()` publishes what an `ActionDescriptor` cannot.** An agent projects to the same
+- **`describeAgents()` is OFFERED, not published, `As of 2026-08-23`** — no reader exists anywhere
+  in the tree, and neither does one for `registeredModels()`. Both are tier-4 exports whose only
+  legal consumer is `@ultimat3/cli` (tier 5); `@ultimat3/manifest` is tier 4, so the obvious import
+  is a sideways edge `bun run boundaries` refuses. Their doc comments say so now, because a comment
+  claiming a consumer that does not exist is the documentation axiom 3 calls no rule at all.
+- **`describeAgents()` describes what an `ActionDescriptor` cannot.** An agent projects to the same
   descriptor as any other action, and that descriptor knows nothing about turns, tools, models or
   prompt hashes — so "how far can this loop and what may it call" had no answer outside the source.
   Same shape as `describePrompts()` / `describeEvals()`, and deliberately NOT a new

package/README.md

@@ -79,6 +79,7 @@ default store at `replicas: 6` is six ledgers of twenty million, which is a budg
 | A control nobody asked for is **omitted**, never defaulted | a default sent as a request is indistinguishable on the wire from one that was declared |
 | A refusal is `X_LLM_REFUSED`, not a schema failure | it is a 200 with no answer in it, and a repair turn buys the same refusal again |
 | The refusal's `alternative` is only ever a **more capable** model | registration order is most-capable-first and `moreCapableThan` walks it upward; retrying a refusal on a weaker model is the one retry that cannot help, so an unbeatable model gets no suggestion at all |
+| A local refusal is never collected into `X_AI_PROVIDER_UNAVAILABLE` | `X_AI_KEY_MISSING` and `X_AI_REQUEST_INVALID` are raised before the request leaves; retrying them across providers burns attempts on the same answer and discards the runnable `fix:`. `generate()` and `stream()` therefore answer the same misconfiguration the same way |
 | Fallback is across **providers serving one model**, never across models | a silent model swap changes what answered, what it cost and which eval baseline the answer belongs to; the gateway stamps `result.provider`, and `llm()` puts it on the span as `llm.provider`, so the fallback that does exist is never silent |
 | The repair turn replays the tool call's arguments, never an empty `text` | an answer through the `respond` tool leaves `text` empty, and an empty text block is a 400 — the repair came back as `X_AI_PROVIDER_UNAVAILABLE` |
 | `reserve()` **debits** the estimate and takes a turn | three concurrent calls otherwise read the same `spent()`, all pass, and all three record against a ceiling only one of them fitted; `record` reconciles and `release` gives it back |
@@ -451,6 +452,12 @@ describeAgents();
 //    mcp: true }]
 ```
 
+**Offered, not yet published, `As of 2026-08-23`.** Nothing in the framework reads it: `describeAgents()`
+lives at tier 4, `@ultimat3/manifest` is tier 4 too, and a sideways import is a build error — so the
+consumer has to be `@ultimat3/cli` at tier 5, and that wiring has not landed. Call it yourself and
+the rows are real; wait for `x manifest` to carry them and you will wait. Same for
+`registeredModels()`.
+
 An agent projects to an `ActionDescriptor` like any other action, and that descriptor knows nothing
 about turns or tools — so "how far can this loop, and what may it call" had no answer outside the
 source. Names are read when you ask, not when the agent was declared: `registerAction` stamps them
@@ -625,7 +632,7 @@ actor comes from the request context, never from the model.
 
 | Code | Meaning |
 |---|---|
-| `X_AI_PROVIDER_UNAVAILABLE` | every provider for the model failed; lists what each said |
+| `X_AI_PROVIDER_UNAVAILABLE` | every provider for the model was unreachable; lists what each said. A TRANSPORT failure only — a coded refusal raised before the socket opens (`X_AI_KEY_MISSING`, `X_AI_REQUEST_INVALID`) reaches the caller as itself, `As of 2026-08-23`, because the same rejection waits on every provider and every attempt and its `fix:` is the whole point of it |
 | `X_AI_BUDGET_EXCEEDED` | refused pre-flight, naming the scope and what remains |
 | `X_AI_GATEWAY_MISSING` | an `llm()` action ran before `configureAi` |
 | `X_AI_PROMPT_VERSION` | version drift, or a render missing a declared variable |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ultimat3/ai",
-  "version": "9.0.0",
+  "version": "10.0.0",
   "description": "LLM gateway, versioned prompts, evals as tests, embeddings, hybrid vector search, RAG",
   "license": "MIT",
   "type": "module",
@@ -32,14 +32,14 @@
     "test": "bun test"
   },
   "dependencies": {
-    "@ultimat3/action": "9.0.0",
-    "@ultimat3/cache": "9.0.0",
-    "@ultimat3/core": "9.0.0",
-    "@ultimat3/db": "9.0.0",
-    "@ultimat3/jobs": "9.0.0",
-    "@ultimat3/money": "9.0.0",
-    "@ultimat3/policy": "9.0.0",
-    "@ultimat3/schema": "9.0.0",
-    "@ultimat3/time": "9.0.0"
+    "@ultimat3/action": "10.0.0",
+    "@ultimat3/cache": "10.0.0",
+    "@ultimat3/core": "10.0.0",
+    "@ultimat3/db": "10.0.0",
+    "@ultimat3/jobs": "10.0.0",
+    "@ultimat3/money": "10.0.0",
+    "@ultimat3/policy": "10.0.0",
+    "@ultimat3/schema": "10.0.0",
+    "@ultimat3/time": "10.0.0"
   }
 }

package/src/agent-facts.ts

@@ -51,6 +51,11 @@ export function registerAgentFact(target: AnyAction, facts: () => Omit<AgentFact
 /**
  * Every registered agent, by name.
  *
+ * OFFERED, not yet published. Nothing outside this package reads it: `@ultimat3/manifest` is tier 4
+ * and so is this package, so a direct import is a sideways edge the boundary check refuses, and the
+ * consumer has to be `@ultimat3/cli` at tier 5. Until that wiring lands, "the row a manifest
+ * publishes" describes the SHAPE, never a row any manifest carries.
+ *
  * An agent still carrying no name is left out, and that is not a silent drop: a name is stamped by
  * `registerAction`, an action without one reaches no route, no tool catalogue and no queue, so
  * there is no capability for a row to describe. `named()` builds a TWIN rather than naming in

package/src/errors.ts

@@ -65,7 +65,12 @@ registerErrorCodes(
   Object.fromEntries(Object.entries(AI_ERROR_TITLES).map(([code, title]) => [code, { title }])),
 );
 
-const docsFor = (code: AiErrorCode): string => `https://ultimate.dev/errors/${code}`;
+// No `docs:` on the subclasses below. `UltimateError` fills it from `describeErrorCode(code).docs`,
+// which is `@ultimat3/core`'s `ERROR_DOCS_URL` — one page for every code, never one per code, because
+// `wiki/` is the framework's only public documentation surface and a code lives there in a TABLE ROW,
+// which has no anchor. The `https://ultimate.dev/errors/<code>` links this file built until 9.x
+// answered 404, host included, on every error it has ever thrown; restating the replacement here
+// would be the same constant in eight places waiting to drift again.
 
 /** Every configured provider refused or errored. Carries what each one said. */
 export class AiProviderUnavailableError extends UltimateError {
@@ -74,7 +79,6 @@ export class AiProviderUnavailableError extends UltimateError {
       code: 'X_AI_PROVIDER_UNAVAILABLE',
       cause: `no provider could serve model "${input.model}" (${input.attempts.join(' | ')})`,
       fix: 'check ai.providers in app.config.ts and the provider API key env var',
-      docs: docsFor('X_AI_PROVIDER_UNAVAILABLE'),
     });
   }
 }
@@ -99,7 +103,6 @@ export class AiBudgetExceededError extends UltimateError {
         `request needs ${input.requested} ${input.unit ?? 'tokens'} but scope ` +
         `"${input.scope}" has ${input.remaining} of ${input.limit} left`,
       fix: `raise ai.budget for "${input.scope}" in app.config.ts, or shorten the prompt`,
-      docs: docsFor('X_AI_BUDGET_EXCEEDED'),
     });
   }
 }
@@ -115,7 +118,6 @@ export class AiGatewayMissingError extends UltimateError {
       code: 'X_AI_GATEWAY_MISSING',
       cause: `an llm action on prompt "${input.prompt}" ran before any gateway was configured`,
       fix: 'configureAi({ gateway: createGateway({ providers: [new AnthropicProvider()] }) }) at boot',
-      docs: docsFor('X_AI_GATEWAY_MISSING'),
     });
   }
 }
@@ -137,7 +139,6 @@ export class AiModelUnknownError extends UltimateError {
       // The `errors` gate blanks every interpolation, so the literal half alone has to name the
       // call. Which ids ARE registered is a fact of the failure, and cause is where facts live.
       fix: 'registerModel({ id, contextWindow, maxOutput, inputPerMillion, outputPerMillion, cacheMinimumTokens, reasoning }) at boot, before configureAi',
-      docs: docsFor('X_AI_MODEL_UNKNOWN'),
       meta: { model: input.model },
     });
   }
@@ -155,7 +156,6 @@ export class AiPromptSecretError extends UltimateError {
       code: 'X_AI_PROMPT_SECRET',
       cause: `prompt "${input.ref}" was given a Secret in vars(): ${input.keys.join(', ')}`,
       fix: 'drop the key from vars() and from the template, or revealSecret(value) in vars() if the model genuinely has to read it',
-      docs: docsFor('X_AI_PROMPT_SECRET'),
     });
   }
 }
@@ -172,7 +172,6 @@ export class LlmStreamInvalidError extends UltimateError {
       code: 'X_LLM_STREAM_INVALID',
       cause: `streamed answer to prompt "${input.prompt}" failed its output schema: ${input.issues}`,
       fix: 'call the action instead of .stream() when the answer must satisfy a structured schema — a stream has already delivered its tokens and cannot take a repair turn',
-      docs: docsFor('X_LLM_STREAM_INVALID'),
     });
   }
 }
@@ -190,7 +189,6 @@ export class LlmOutputInvalidError extends UltimateError {
         `prompt "${input.prompt}" returned output failing its schema on all ` +
         `${input.attempts} attempts: ${input.issues}`,
       fix: 'describe the output shape in the prompt template and bump its version, or widen `output` in the llm() declaration',
-      docs: docsFor('X_LLM_OUTPUT_INVALID'),
     });
   }
 }
@@ -213,7 +211,6 @@ export class AgentMaxTurnsError extends UltimateError {
         `agent "${input.agent}" used all ${input.turns} turns and ${input.calls} tool calls ` +
         `without calling the respond tool`,
       fix: 'tell the template when to stop and answer through the respond tool, then bump its version — raise maxTurns only once the run demonstrably converges',
-      docs: docsFor('X_AGENT_MAX_TURNS'),
       meta: { agent: input.agent, turns: input.turns },
     });
   }
@@ -234,7 +231,6 @@ export class AgentToolUnexposedError extends UltimateError {
       code: 'X_AGENT_TOOL_UNEXPOSED',
       cause: `agent "${input.agent}" lists tools no MCP surface exposes: ${input.tools.join(', ')}`,
       fix: 'add mcp: { expose: true } to the action named in cause, or drop it from the agent tools list',
-      docs: docsFor('X_AGENT_TOOL_UNEXPOSED'),
     });
   }
 }
@@ -270,7 +266,6 @@ export class LlmRefusedError extends UltimateError {
         input.alternative === undefined
           ? `edit the template in definePrompt('${input.prompt}') and bump its version — no blessed model is more capable than '${input.model}'`
           : `set model: '${input.alternative}' on the llm() declaration, or edit the template in definePrompt('${input.prompt}') and bump its version`,
-      docs: docsFor('X_LLM_REFUSED'),
       meta: { model: input.model, category: input.category },
     });
   }
@@ -287,7 +282,6 @@ export class LlmTruncatedError extends UltimateError {
       code: 'X_LLM_TRUNCATED',
       cause: `prompt "${input.prompt}" was cut off at its ${input.maxTokens}-token ceiling`,
       fix: `set maxTokens: ${input.maxTokens * 2} on the llm() declaration, or drop fields from its output schema`,
-      docs: docsFor('X_LLM_TRUNCATED'),
     });
   }
 }
@@ -301,7 +295,6 @@ export class AiPromptVersionError extends UltimateError {
         input.available.length > 0 ? input.available.join(', ') : 'none'
       })`,
       fix: 'bump the version in definePrompt after editing the template, then x manifest',
-      docs: docsFor('X_AI_PROMPT_VERSION'),
     });
   }
 }
@@ -317,7 +310,6 @@ export class AiPromptRenderError extends UltimateError {
       code: 'X_AI_PROMPT_VERSION',
       cause: `prompt "${input.ref}" was rendered without: ${input.missing.join(', ')}`,
       fix: 'pass every {{variable}} the template declares, or remove it from the template',
-      docs: docsFor('X_AI_PROMPT_VERSION'),
     });
   }
 }
@@ -331,7 +323,6 @@ export class VectorDimMismatchError extends UltimateError {
       // Not `x ai reindex`: that command is PLANNED and throws, so a fix line naming it sends an
       // operator to a wall. A fix has to be performable today, which here means app code.
       fix: 'use the same embedder that created the store, or re-embed every record at the new width and upsert it',
-      docs: docsFor('X_VECTOR_DIM_MISMATCH'),
     });
   }
 }
@@ -349,7 +340,6 @@ export class VectorScopeWidenedError extends UltimateError {
         `store "${input.store}" is bound to tenant "${input.held}" and cannot be re-scoped ` +
         `to "${input.requested}"`,
       fix: `derive from the unscoped store instead: vectorStore.scoped({ tenant: '${input.requested}' })`,
-      docs: docsFor('X_VECTOR_SCOPE_WIDENED'),
     });
   }
 }
@@ -367,7 +357,6 @@ export class EmbedderDimMismatchError extends UltimateError {
         `embedder "${input.embedder}" is declared with ${input.expected} dimensions but the ` +
         `provider returned ${input.received}`,
       fix: `set dimension: ${input.received} on the embedder, then re-embed every record at that width and upsert it`,
-      docs: docsFor('X_VECTOR_DIM_MISMATCH'),
     });
   }
 }
@@ -386,7 +375,6 @@ export class AiEmbedderInvalidError extends UltimateError {
       // interpolation — so the literal half alone has to name the call. Which embedder broke the
       // invariant is a fact of the failure, and the cause and `meta` are where facts live.
       fix: 'return one vector per input text from embed(), in the order the texts arrived',
-      docs: docsFor('X_AI_EMBEDDER_INVALID'),
       meta: { embedder: input.embedder },
     });
   }
@@ -399,7 +387,6 @@ export class AiKeyMissingError extends UltimateError {
       code: 'X_AI_KEY_MISSING',
       cause: `provider "${input.provider}" has no API key: ${input.envVar} is unset and none was passed to its constructor`,
       fix: `export ${input.envVar}=<key>, or pass { apiKey } when constructing the provider`,
-      docs: docsFor('X_AI_KEY_MISSING'),
       meta: { provider: input.provider, envVar: input.envVar },
     });
   }
@@ -416,7 +403,6 @@ export class AiRequestInvalidError extends UltimateError {
       code: 'X_AI_REQUEST_INVALID',
       cause: input.detail,
       fix: input.fix,
-      docs: docsFor('X_AI_REQUEST_INVALID'),
     });
   }
 }
@@ -446,7 +432,6 @@ export class AiTransportError extends UltimateError {
         input.status === undefined ? 'failed' : `returned ${input.status}`
       }: ${input.detail}`,
       fix: fixForStatus(input.status, input.envVar),
-      docs: docsFor('X_AI_PROVIDER_UNAVAILABLE'),
       meta: { provider: input.provider, status: input.status },
     });
     this.status = input.status;

package/src/eval-errors.ts

@@ -3,9 +3,13 @@
 // stay in ./errors — one owner, one registration, one place a duplicate can surface.
 
 import { UltimateError } from '@ultimat3/core';
-import type { AiErrorCode } from './errors';
 
-const docsFor = (code: AiErrorCode): string => `https://ultimate.dev/errors/${code}`;
+// No `docs:` on the subclasses below. `UltimateError` fills it from `describeErrorCode(code).docs`,
+// which is `@ultimat3/core`'s `ERROR_DOCS_URL` — one page for every code, never one per code, because
+// `wiki/` is the framework's only public documentation surface and a code lives there in a TABLE ROW,
+// which has no anchor. The `https://ultimate.dev/errors/<code>` links this file built until 9.x
+// answered 404, host included, on every error it has ever thrown; restating the replacement here
+// would be the same constant in eight places waiting to drift again.
 
 /**
  * An eval scored further below its recorded baseline than its tolerance allows. The gate is the
@@ -33,7 +37,6 @@ export class EvalThresholdError extends UltimateError {
       // so the eval's own name there is `X_CLI_BAD_FLAG` ("not a test type") — a fix line that
       // cannot be run is axiom 4 broken at the one moment it is needed.
       fix: `x test eval --filter ${input.eval} to see per-case scores, then fix the prompt — or ULTIMATE_EVAL_RECORD=1 x test eval to accept the new numbers as a reviewed diff`,
-      docs: docsFor('X_EVAL_THRESHOLD'),
     });
   }
 }
@@ -48,7 +51,6 @@ export class EvalBaselineMissingError extends UltimateError {
       code: 'X_EVAL_BASELINE_MISSING',
       cause: `eval "${input.eval}" gates against ${input.path}, which ${input.reason}`,
       fix: input.fix ?? `ULTIMATE_EVAL_RECORD=1 x test eval, then commit ${input.path}`,
-      docs: docsFor('X_EVAL_BASELINE_MISSING'),
     });
   }
 }
@@ -60,7 +62,6 @@ export class EvalBaselineInvalidError extends UltimateError {
       code: 'X_EVAL_BASELINE_INVALID',
       cause: `the recorded baseline ${input.path} ${input.problem}`,
       fix: `ULTIMATE_EVAL_RECORD=1 x test eval to re-record ${input.path}`,
-      docs: docsFor('X_EVAL_BASELINE_INVALID'),
     });
   }
 }
@@ -75,7 +76,6 @@ export class EvalMissingError extends UltimateError {
       code: 'X_EVAL_MISSING',
       cause: `prompt "${input.prompt}" has no eval`,
       fix: `defineEval({ name: '${input.id}', prompt, cases, scorers, tolerance, baseline }) beside the prompt, then ULTIMATE_EVAL_RECORD=1 x test eval`,
-      docs: docsFor('X_EVAL_MISSING'),
     });
   }
 }
@@ -92,7 +92,6 @@ export class EvalRecordingError extends UltimateError {
       code: 'X_EVAL_RECORDING',
       cause: `${input.env} is set, so every eval would re-record its baseline instead of gating on it`,
       fix: `env -u ${input.env} x verify`,
-      docs: docsFor('X_EVAL_RECORDING'),
     });
   }
 }

package/src/evals.ts

@@ -183,6 +183,10 @@ async function runEval<V extends PromptVars>(
       ...(input.prompt.system !== undefined ? { system: input.prompt.system } : {}),
       ...(input.prompt.model !== undefined ? { model: input.prompt.model } : {}),
       ...(input.prompt.effort !== undefined ? { effort: input.prompt.effort } : {}),
+      // `thinking` too, and not because it is tidy: `contentHash` covers it, so the baseline is
+      // filed under a hash describing a configuration the measurement did not use. `llm()` and
+      // `agent()` have always sent both.
+      ...(input.prompt.thinking !== undefined ? { thinking: input.prompt.thinking } : {}),
     });
     const perScorer: Record<string, number> = {};
     for (const scorer of input.scorers) {

package/src/gateway.ts

@@ -4,7 +4,7 @@
 // budgeted, and cost-accounted in integer minor units. Everything an app does with a model
 // goes through here, so budgets and accounting cannot be bypassed by a stray fetch.
 
-import { renderThrowable } from '@ultimat3/core';
+import { isUltimateError, renderThrowable, stringField } from '@ultimat3/core';
 import type { Money } from '@ultimat3/money';
 import type { BudgetLimits, BudgetStore } from './budget';
 import { BudgetLedger, currentBudget, estimateSpend, withBudget } from './budget';
@@ -32,6 +32,9 @@ export interface RetryPolicy {
 
 export const DEFAULT_RETRY: RetryPolicy = { attempts: 3, baseDelayMs: 500, maxDelayMs: 8_000 };
 
+/** The one code `attempt` may collect into. Every other coded refusal is the caller's answer. */
+const PROVIDER_UNAVAILABLE = 'X_AI_PROVIDER_UNAVAILABLE';
+
 export interface CreateGatewayInput {
   /** Tried in order for a given model. First provider that lists the model wins. */
   readonly providers: readonly Provider[];
@@ -124,9 +127,13 @@ class GatewayImpl implements Gateway {
   async *stream(request: GenerateRequest): AsyncIterable<StreamChunk> {
     const model = request.model ?? this.config.defaultModel ?? DEFAULT_MODEL;
     const resolved: GenerateRequest = { ...request, model };
-    const ledger = currentBudget();
-    const reservation = await ledger?.reserve(estimateSpend(resolved));
 
+    // Routed BEFORE the reservation, not after it. `providerFor` throws for a registered model no
+    // configured provider serves — an ordinary boot misconfiguration — and a debit taken first has
+    // nothing to credit it back: the throw is outside the `finally` below, and the estimate has
+    // already landed on the `BudgetStore`, which is per process and never expires. Five refused
+    // streams and the org's ceiling is gone for the life of the process, with nothing ever sent.
+    //
     // The streaming path does not retry AT ALL — not mid-flight, and not on the handshake either.
     // Mid-flight is the obvious one: the consumer has already been handed tokens, and replaying
     // from the top would duplicate them. The handshake is not separable from it here, because
@@ -136,6 +143,9 @@ class GatewayImpl implements Gateway {
     // its fallback across providers belong to `generate` alone. A caller that wants either uses
     // `generate`, or reconnects itself and knows what it has already shown.
     const provider = this.providerFor(model);
+
+    const ledger = currentBudget();
+    const reservation = await ledger?.reserve(estimateSpend(resolved));
     let settled = false;
     try {
       for await (const chunk of provider.stream(resolved)) {
@@ -143,8 +153,10 @@ class GatewayImpl implements Gateway {
           yield chunk;
           continue;
         }
-        settled = true;
+        // Settled only once `record` has LANDED. Marking it first left a store that threw here
+        // holding the reservation and half the record — the `finally` saw a settled stream.
         await ledger?.record(chunk.result.usage, chunk.result.cost, reservation);
+        settled = true;
         yield { type: 'done', result: { ...chunk.result, provider: provider.name } };
       }
     } finally {
@@ -190,6 +202,19 @@ class GatewayImpl implements Gateway {
         try {
           return { ...(await call(provider)), provider: provider.name };
         } catch (error) {
+          // A coded refusal that is NOT `X_AI_PROVIDER_UNAVAILABLE` reaches the caller verbatim.
+          // Those are raised locally, before the socket opens — a missing credential, a control
+          // the model does not accept — so the same rejection is waiting on every provider and
+          // every attempt, which is the reason a 400 is not retried three lines below. Collecting
+          // one into `X_AI_PROVIDER_UNAVAILABLE` discards its runnable `fix:` and answers the same
+          // failure a different way from `stream`, which does not route through here at all.
+          // A transport failure keeps the old path: `AiTransportError` IS
+          // `X_AI_PROVIDER_UNAVAILABLE`, so "provider one 503'd, provider two timed out" still
+          // collects across the candidates. `stringField` rather than `error.code`, because the
+          // value came from an app's `Provider` and a property read on it can trap.
+          if (isUltimateError(error) && stringField(error, 'code') !== PROVIDER_UNAVAILABLE) {
+            throw error;
+          }
           // `renderThrowable`, never `error.message` or `String(error)`: this line becomes the
           // `cause` of `X_AI_PROVIDER_UNAVAILABLE`, and a renderer that throws replaces the coded
           // refusal with a `TypeError` nothing downstream can catch by code. It bounds the text

package/src/hive-errors.ts

@@ -4,9 +4,13 @@
 // duplicate can surface.
 
 import { UltimateError } from '@ultimat3/core';
-import type { AiErrorCode } from './errors';
 
-const docsFor = (code: AiErrorCode): string => `https://ultimate.dev/errors/${code}`;
+// No `docs:` on the subclasses below. `UltimateError` fills it from `describeErrorCode(code).docs`,
+// which is `@ultimat3/core`'s `ERROR_DOCS_URL` — one page for every code, never one per code, because
+// `wiki/` is the framework's only public documentation surface and a code lives there in a TABLE ROW,
+// which has no anchor. The `https://ultimate.dev/errors/<code>` links this file built until 9.x
+// answered 404, host included, on every error it has ever thrown; restating the replacement here
+// would be the same constant in eight places waiting to drift again.
 
 /**
  * `split` handed back an empty list, so the hive fanned out to nobody and would have reported a
@@ -23,7 +27,6 @@ export class HiveEmptyError extends UltimateError {
       code: 'X_HIVE_EMPTY',
       cause: `the hive over "${input.member}" split into 0 members, so no member ran`,
       fix: `return at least one member input from the hive's split() over "${input.member}", or skip the hive call when the source is empty — a hive reporting 0 ok and 0 failed cannot be told apart from one whose query returned no rows`,
-      docs: docsFor('X_HIVE_EMPTY'),
       meta: { member: input.member },
     });
   }

package/src/models.ts

@@ -128,7 +128,14 @@ export function modelIds(): readonly ModelId[] {
   return [...registry.keys()];
 }
 
-/** Every registered spec, in ladder order. Consumed by `x manifest` and by doctor-style checks. */
+/**
+ * Every registered spec, in ladder order.
+ *
+ * OFFERED, not yet published: nothing in the tree reads it. `@ultimat3/manifest` is tier 4 and so
+ * is this package, so the consumer has to be `@ultimat3/cli` (tier 5) — a direct import would be a
+ * sideways edge the boundary check refuses. The doc claimed `x manifest` consumed it, which is
+ * exactly the kind of statement axiom 3 says is not a rule.
+ */
 export function registeredModels(): readonly ModelSpec[] {
   return [...registry.values()];
 }

package/src/openai-provider.ts

@@ -113,8 +113,13 @@ class OpenAiProvider implements Provider {
    * open socket past the HTTP timeout and fails after the completion was generated and billed.
    */
   async generate(request: GenerateRequest): Promise<GenerateResult> {
-    if (requiresStreaming(request)) return this.assemble(request);
+    // Resolved BEFORE the transport question is asked. `requiresStreaming` clamps `maxTokens` to
+    // the model's `maxOutput`, and its own fallback for an absent model is the framework's
+    // `DEFAULT_MODEL` — a Claude id no OpenAI-format endpoint serves. Sized against that, a
+    // request naming no model was measured against somebody else's ceiling, seventy lines above
+    // `modelOf`'s comment saying this provider never does that.
     const model = this.modelOf(request);
+    if (requiresStreaming({ ...request, model })) return this.assemble(request);
     const response = await this.send(
       chatCompletionBody({ request, model, stream: false }),
       false,

package/src/prompt.ts

@@ -27,7 +27,17 @@ export interface DefinePromptInput<V extends PromptVars> {
   readonly system?: string;
   /** Schema of the variables, for the manifest. */
   readonly input?: JsonSchema;
-  /** Expected output shape, fed to `output_config.format` when the caller opts in. */
+  /**
+   * The output shape this prompt PROMISES. Declarative: it is hashed into `ref` and published by
+   * `describePrompts()`, and it is deliberately never sent on the wire — structured output is the
+   * `respond` tool `llm()` projects from the ACTION's `output`, and a second path through
+   * `output_config.format` / `response_format` is the ambiguity axiom 1 refuses (it is also the
+   * one feature most OpenAI-compatible servers do not implement).
+   *
+   * Editing it moves the hash and so needs a `version` bump, which is the point: a prompt whose
+   * promised shape changed is a different prompt, and every score already filed against the old
+   * hash describes the old one.
+   */
   readonly output?: JsonSchema;
   readonly model?: ModelId;
   readonly effort?: Effort;

package/src/scorers.ts

@@ -13,8 +13,22 @@ export interface Scorer {
   score(input: { output: string; expected?: string }): Promise<number> | number;
 }
 
-/** Scores outside 0..1 are a scorer bug; clamping keeps one from skewing a whole run's mean. */
-export const clampScore = (n: number): number => (n < 0 ? 0 : n > 1 ? 1 : n);
+/**
+ * Scores outside 0..1 are a scorer bug; clamping keeps one from skewing a whole run's mean.
+ *
+ * `NaN` is the case a pair of comparisons cannot express, and it is the dangerous one: both
+ * `n < 0` and `n > 1` are false for it, so it propagated through `mean()` into `EvalResult.score`,
+ * and `regressionsAgainst`'s `now < was - tolerance` is false for `NaN` too — the run reported a
+ * pass over a number that measured nothing. The trigger is ordinary arithmetic in an app's own
+ * scorer: `output.length / expected.length` on a case that declares no `expected`.
+ *
+ * Zero rather than a throw, and rather than a new code: zero is a REGRESSION against any recorded
+ * baseline, so the scorer bug surfaces through `X_EVAL_THRESHOLD`, which already names the case
+ * that produced it. It also makes recording and gating agree — `ULTIMATE_EVAL_RECORD=1` wrote
+ * `"score": null` for a `NaN`, which `parseBaseline` then refuses as `X_EVAL_BASELINE_INVALID`.
+ */
+export const clampScore = (n: number): number =>
+  Number.isFinite(n) ? (n < 0 ? 0 : n > 1 ? 1 : n) : n > 0 ? 1 : 0;
 
 /** Exact match after trimming. The strictest and cheapest scorer; prefer it when it fits. */
 export const exact: Scorer = {
@@ -98,6 +112,10 @@ export function llmJudge(input: {
         maxTokens: input.maxTokens ?? 256,
         ...(input.judge.system !== undefined ? { system: input.judge.system } : {}),
         ...(input.judge.model !== undefined ? { model: input.judge.model } : {}),
+        // The judge prompt's hash is this scorer's NAME, and `contentHash` covers `effort` and
+        // `thinking` — so dropping either measures with a judge the name does not describe.
+        ...(input.judge.effort !== undefined ? { effort: input.judge.effort } : {}),
+        ...(input.judge.thinking !== undefined ? { thinking: input.judge.thinking } : {}),
       });
       // The judge is asked for a bare 0..1; anything else scores 0 rather than guessing.
       const parsed = Number.parseFloat(generated.text.trim());

package/src/sse.ts

@@ -8,7 +8,8 @@
 import { AiTransportError } from './errors';
 
 /**
- * The most one unterminated frame may buffer. A peer that sends a body with no frame boundary in
+ * The most one unterminated frame may buffer — the REMAINDER after every complete frame has been
+ * split off, never the whole decode buffer. A peer that sends a body with no frame boundary in
  * it — an HTML error page, a proxy answering on the model's port, a hung gateway — grows `buffer`
  * without limit, and no read deadline interrupts it because every individual read SUCCEEDS. Coded
  * failure > OOM, the same call `@ultimat3/mail`'s `createReplyParser` makes for the same shape.
@@ -86,10 +87,14 @@ export async function* readSse(
       const { done, value } = await reader.read();
       if (done) break;
       buffer += decoder.decode(value, { stream: true });
-      guard(buffer, provider);
       const decoded = decodeSse(buffer);
       buffer = decoded.rest;
       for (const frame of decoded.frames) yield frame;
+      // The REMAINDER, after the split, and after the frames that did complete were delivered.
+      // Measured against the whole decode buffer it refused a busy provider that landed a
+      // megabyte of perfectly framed deltas in one read — every one of them terminated — for
+      // "sending more than 1048576 characters without completing one SSE frame".
+      guard(buffer, provider);
     }
     buffer += decoder.decode();
     const tail = frameOf(buffer);

package/src/vector-scope.ts

@@ -49,17 +49,28 @@ function narrowTenant(
   throw new VectorScopeWidenedError({ store, held: base, requested: next });
 }
 
+/**
+ * A `Map`, never a `Record`, because every key here is a CALLER's string — an app's metadata field
+ * name, chosen by whoever wrote the policy. An object gets it wrong in both directions:
+ *   - reading `merged['constructor']` answers the `Object` function off the prototype chain, so
+ *     `held === undefined` is false and `held.includes(value)` is a bare `TypeError` raised inside
+ *     a path whose only contract is `X_VECTOR_SCOPE_WIDENED`;
+ *   - writing `merged['__proto__'] = [...]` runs `Object.prototype`'s setter rather than creating
+ *     the key, so the rule the caller declared is absent from `Object.entries` and the derived
+ *     scope comes out WIDER than the one that was asked for.
+ * `Object.fromEntries` defines own properties, so the object handed back has neither hazard.
+ */
 function narrowAllow(
   base: Readonly<Record<string, readonly string[]>> | undefined,
   next: Readonly<Record<string, readonly string[]>> | undefined,
 ): Readonly<Record<string, readonly string[]>> | undefined {
   if (next === undefined) return base;
-  const merged: Record<string, readonly string[]> = { ...(base ?? {}) };
+  const merged = new Map<string, readonly string[]>(Object.entries(base ?? {}));
   for (const [key, values] of Object.entries(next)) {
-    const held = merged[key];
-    merged[key] = held === undefined ? values : values.filter((value) => held.includes(value));
+    const held = merged.get(key);
+    merged.set(key, held === undefined ? values : values.filter((value) => held.includes(value)));
   }
-  return merged;
+  return Object.fromEntries(merged);
 }
 
 /** Whether one stored row survives the scope. The in-memory twin of the SQL conditions. */
@@ -70,7 +81,11 @@ export function scopeAdmits(
 ): boolean {
   if (scope.tenant !== undefined && scope.tenant !== tenant) return false;
   return Object.entries(scope.allow ?? {}).every(([key, values]) => {
-    const value = metadata[key];
+    // Own properties only, for the reason `narrowAllow` uses a `Map`: `key` is a caller's string
+    // and a metadata bag is a caller's object. This half already failed CLOSED — an inherited
+    // member is never one of the allowed strings — but relying on that is relying on the value
+    // types, not on the rule, and the rule is that a caller's string is never an object key.
+    const value = Object.hasOwn(metadata, key) ? metadata[key] : undefined;
     return value !== undefined && values.includes(value);
   });
 }

package/src/vector.ts

@@ -218,4 +218,12 @@ export function fuse(rankings: readonly (readonly SearchHit[])[], rrfK = 60): re
     .sort(byScoreDesc);
 }
 
-const byScoreDesc = (a: SearchHit, b: SearchHit): number => b.score - a.score;
+/**
+ * Score descending, then id ascending — the second key is `pg-vector-sql.ts`'s
+ * `order by f.score desc, d."id" asc`, and the two have to agree or the developer machine and the
+ * deployed app return different pages of the same search. Ties are the COMMON case in RRF: two
+ * documents that swap rank between the dense and the lexical list score identically, and a stable
+ * sort then resolves them by dense-list insertion order, which no SQL engine reproduces.
+ */
+const byScoreDesc = (a: SearchHit, b: SearchHit): number =>
+  b.score - a.score || (a.id < b.id ? -1 : a.id > b.id ? 1 : 0);