@ultimat3/ai 27.9.0 → 27.10.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +104 -108
- package/README.md +21 -5
- package/package.json +11 -11
- package/src/agent-continue.ts +434 -0
- package/src/agent-declared.ts +157 -0
- package/src/agent-def.ts +28 -5
- package/src/agent-errors.ts +20 -1
- package/src/agent-turn.ts +54 -0
- package/src/agent-usage.ts +49 -13
- package/src/agent.ts +163 -193
- package/src/answer-errors.ts +88 -0
- package/src/billed.ts +87 -0
- package/src/budget.ts +37 -10
- package/src/cost.ts +146 -0
- package/src/echo-provider.ts +3 -1
- package/src/errors-retry.ts +8 -0
- package/src/errors.ts +22 -19
- package/src/failure-disclosure.ts +37 -1
- package/src/gateway.ts +53 -7
- package/src/in-process-tool.ts +77 -0
- package/src/index.ts +29 -3
- package/src/llm-def.ts +97 -0
- package/src/llm.ts +93 -75
- package/src/models.ts +45 -1
- package/src/openai-body.ts +6 -3
- package/src/openai-provider.ts +11 -2
- package/src/openai-wire.ts +168 -62
- package/src/provider.ts +32 -34
- package/src/run-meter.ts +147 -0
- package/src/sampling.ts +113 -0
- package/src/tool-arguments.ts +88 -0
- package/src/tools.ts +51 -6
- package/src/wire.ts +15 -14
package/CLAUDE.md
CHANGED
|
@@ -9,18 +9,13 @@ does), `jobs` (`agentJob()`), `money`, `policy`, `schema`, `time`.
|
|
|
9
9
|
|
|
10
10
|
`jobs` is declared for `agentJob()`: the action→job bridge has to live at tier 4+.
|
|
11
11
|
|
|
12
|
-
`mcp` is the same tier, so the LLM-tool projection is restated structurally in `tools.ts
|
|
13
|
-
|
|
14
|
-
`
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
**And the same TOOL**: `asProjectableAction` and mcp's `primitiveFromAction` are two copies the
|
|
21
|
-
tier table forces; `scripts/ai-mcp-tool-parity.test.ts` holds them equal — name, description,
|
|
22
|
-
exposure, the whole input schema (an idempotent action's reserved `idempotencyKey` argument,
|
|
23
|
-
`tool-idempotency.ts`), the policy verdict and the replay. A field one adds, the other adds.
|
|
12
|
+
`mcp` is the same tier, so the LLM-tool projection is restated structurally in `tools.ts`, never
|
|
13
|
+
imported: same contract, same *decision* (`isMcpExposed` from `@ultimat3/core`, never a local
|
|
14
|
+
`=== true`), same NAME (`action.name` verbatim, never derived) and the same TOOL —
|
|
15
|
+
`asProjectableAction` and mcp's `primitiveFromAction` are two copies the tier table forces, held
|
|
16
|
+
equal by `scripts/ai-mcp-tool-parity.test.ts` (name, description, exposure, the whole input schema
|
|
17
|
+
incl. `tool-idempotency.ts`'s reserved `idempotencyKey`, the policy verdict, the replay). A field
|
|
18
|
+
one adds, the other adds.
|
|
24
19
|
|
|
25
20
|
## Owns
|
|
26
21
|
|
|
@@ -29,7 +24,7 @@ exposure, the whole input schema (an idempotent action's reserved `idempotencyKe
|
|
|
29
24
|
| `models.ts` | the model REGISTRY: `registerModel`, limits, prices, the reasoning controls each one accepts — and NO row of its own |
|
|
30
25
|
| `model-resolve.ts` | `resolveModel` / `modelForCall` — declaration → prompt → gateway `defaultModel`, none is `X_AI_MODEL_UNRESOLVED` naming all three; `describedModel` for a published fact |
|
|
31
26
|
| `model-fixture.ts` | the rows this package's own suites register (`useFixtureModels()`). Not shipped |
|
|
32
|
-
| `provider.ts` | `Provider` interface, the request half,
|
|
27
|
+
| `provider.ts` | `Provider` interface, the request half, `anthropicProvider()` |
|
|
33
28
|
| `echo-provider.ts` | `echoProvider()`, the deterministic double — split from `provider.ts` at its line ceiling |
|
|
34
29
|
| `content-blocks.ts` | the `image` / `document` block shapes, the screen every request runs them through, their token estimate |
|
|
35
30
|
| `content-errors.ts` | the `X_AI_CONTENT_UNSUPPORTED` class; its code and title belong in `errors.ts` |
|
|
@@ -57,17 +52,21 @@ exposure, the whole input schema (an idempotent action's reserved `idempotencyKe
|
|
|
57
52
|
| `rag.ts` | chunker, retriever, reranker, budgeted context assembler |
|
|
58
53
|
| `tools.ts` | action → LLM tool definition; the `AgentTool` union and `asProjectableAction`; `runLlmToolCall` |
|
|
59
54
|
| `tool-idempotency.ts` | an idempotent action's reserved `idempotencyKey` tool argument — MCP's, restated (same tier), held equal by the parity test |
|
|
60
|
-
| `
|
|
55
|
+
| `tool-arguments.ts` / `answer-errors.ts` | one argument-string reader for both wires and what an unreadable one means (`assertToolArguments`); `X_LLM_TRUNCATED` + `X_LLM_TOOL_ARGUMENTS_INVALID` |
|
|
56
|
+
| `billed.ts` / `run-meter.ts` | `meta.billed` + `billedOf`; the meter every `llm()`/`agent()` provider call goes through |
|
|
57
|
+
| `failure-disclosure.ts` | `discloseFailure` — the ONE rule for what of a throw a remote reader sees (tool result, hive member); the withheld half is logged, never its message or cause |
|
|
61
58
|
| `llm.ts` | `llm()` — the model call, declared as an `action`; and what a streamed answer must satisfy |
|
|
62
59
|
| `respond.ts` | the `respond` tool for one `output` (a non-object one wrapped in `{ value }`), its reader, and `parseJsonish` |
|
|
63
60
|
| `llm-stream.ts` | `.stream()`'s plumbing: the sink, the ambient mark, the one-turn drive |
|
|
64
61
|
| `agent.ts` | `agent()` — the tool loop, declared as an `action` |
|
|
65
|
-
| `agent-def.ts` / `agent-system.ts` / `agent-usage.ts` | the declaration's types; the system one run sends; its running usage |
|
|
62
|
+
| `agent-def.ts` / `agent-declared.ts` / `agent-system.ts` / `agent-usage.ts` | the declaration's types; its defaults, bound screen, limits and fact row; the system one run sends; its running usage |
|
|
63
|
+
| `llm-def.ts` / `cost.ts` | `llm()`'s declaration types and `LlmCall`; the money arithmetic (`costOf`, `exactCostOf`) |
|
|
66
64
|
| `agent-guards.ts` / `agent-errors.ts` | the repeated-turn detector and the run deadline; their two classes |
|
|
67
65
|
| `prompt-cache.ts` | PROVIDER prompt caching: `SystemParts`, `PromptCacheHint`, `PROMPT_CACHE_RULES` (sources in its header) |
|
|
68
66
|
| `prompt-cache-blocks.ts` | the Anthropic wire's marks (chat-completions': `openai-messages.ts`) |
|
|
67
|
+
| `agent-continue.ts` / `agent-turn.ts` | `.run()` / `.continue()`, the transcript handle and its screen; one turn's report and refusals |
|
|
69
68
|
| `agent-transcript.ts` | what one turn leaves in the transcript: the assistant replay, the tool results, the correction |
|
|
70
|
-
| `agent-facts.ts` | `describeAgents()` — the agent registry, and the row a manifest WOULD publish
|
|
69
|
+
| `agent-facts.ts` | `describeAgents()` — the agent registry, and the row a manifest WOULD publish (its consumer has to be `cli`, tier 5) |
|
|
71
70
|
| `agent-job.ts` | `agentJob()` — an agent as a real `JobHandle`, composed from `job()` |
|
|
72
71
|
| `hive.ts` | `hive()` — one action fanned out over many inputs, declared as an `action` |
|
|
73
72
|
| `hive-result.ts` | `HiveMember` / `HiveResult`, and the SCHEMA built from the member's own `output` |
|
|
@@ -81,65 +80,54 @@ exposure, the whole input schema (an idempotent action's reserved `idempotencyKe
|
|
|
81
80
|
|
|
82
81
|
## Invariants — bounds and budget
|
|
83
82
|
|
|
83
|
+
Why each is shaped this way: [`docs/history/ai.md`](../../docs/history/ai.md#bounds-and-budget).
|
|
84
|
+
|
|
84
85
|
- **Every numeric option is screened; `??` is not the screen** (`finiteOption`/`finiteCount`;
|
|
85
|
-
`bun run finite-bounds` is the
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
- **`reserve` debits, then checks — never a read-then-write across an `await`.** The store's scopes
|
|
95
|
-
go through `BudgetStore.take(key, tokens, limit)`: REQUIRED, atomic (never `spent` + `add`). A
|
|
96
|
-
refusal at a later scope gives back what the earlier ones took.
|
|
97
|
-
- **The store is touched only for a scope whose ceiling is DECLARED** (`takeScope`, `meteredKeys`).
|
|
98
|
-
`memoryBudgetStore()` has no window and no eviction (a ceiling bypass); a zero counter is deleted.
|
|
86
|
+
`bun run finite-bounds`). **`maxTokens` is the worst**: it IS the pre-flight estimate, and a `NaN`
|
|
87
|
+
switches every ceiling off — screened by the gateway, `registerModel` and `llm()`/`agent()`.
|
|
88
|
+
- **A bound is screened under the key the DECLARATION uses** (`budget.tokensPerRun`, not `request`).
|
|
89
|
+
- A per-call budget `derive`s from the ambient ledger and only TIGHTENS; a derived ledger reports
|
|
90
|
+
back up the chain; the STORE is written once, by the ledger the call was made on.
|
|
91
|
+
- **`reserve` debits, then checks — never a read-then-write across an `await`**:
|
|
92
|
+
`BudgetStore.take(key, tokens, limit)` is REQUIRED and atomic; a later refusal gives back.
|
|
93
|
+
- **The store is touched only for a scope whose ceiling is DECLARED**; `memoryBudgetStore()` has no
|
|
94
|
+
window and no eviction; a zero counter is deleted.
|
|
99
95
|
- **`llm()`, `agent()`, `hive()` root as `currentBudget() ?? gateway.callLedger(budgetKeysFor(ctx.actor))`**
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
- Cost is `Money`
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
96
|
+
(`actor:<kind>:<id>` / `org:<orgId>`). `callLedger` is required on `Gateway`.
|
|
97
|
+
- **`BudgetStore` is where `actor` and `org` live; the default is per PROCESS**; `add` takes a
|
|
98
|
+
negative `tokens` for a release — a store that clamps at zero leaks the ceiling.
|
|
99
|
+
- A budget throws `X_AI_BUDGET_EXCEEDED` **before** the provider call. Never truncate. List price
|
|
100
|
+
over-reserves.
|
|
101
|
+
- **Cost is exact in bigint (`cost.ts`, `Money` at `costScale`), rounded UP once**: per call by
|
|
102
|
+
`costOf`, per ledger at `report()`, per run from the summed usage — never a sum of round-ups. A
|
|
103
|
+
price may state a `scale` (≤ `PRICE_SCALE_DIGITS` finer); it is always read through `minorAt`.
|
|
104
|
+
- **`Gateway.stream` routes BEFORE it reserves**; `settled = true` comes after `ledger.record`.
|
|
109
105
|
|
|
110
106
|
## Invariants — `llm()`
|
|
111
107
|
|
|
108
|
+
Why each is shaped this way: [`docs/history/ai.md`](../../docs/history/ai.md#llm-invariants).
|
|
109
|
+
|
|
112
110
|
- **`llm()` returns an `action`**, never a ninth primitive; it adds only the model half: render the
|
|
113
|
-
prompt, project `output` into the one `respond` tool, reserve the budget, consult the
|
|
114
|
-
- **A non-object `output` is wrapped in `{ value }` and unwrapped** (`respond.ts`)
|
|
115
|
-
|
|
116
|
-
`indexOf` scans, never a backtracking pattern.
|
|
111
|
+
prompt, project `output` into the one `respond` tool, reserve the budget, consult the cache.
|
|
112
|
+
- **A non-object `output` is wrapped in `{ value }` and unwrapped** (`respond.ts`), never refused;
|
|
113
|
+
`parseJsonish` is two `indexOf` scans, never a backtracking pattern.
|
|
117
114
|
- `src/index.ts` re-exports `t` from `@ultimat3/schema` **verbatim** (`index.test.ts`).
|
|
118
115
|
- One repair turn, then `X_LLM_OUTPUT_INVALID`. A refusal (a 200 with no answer) is `X_LLM_REFUSED`
|
|
119
|
-
before the parse
|
|
120
|
-
The gateway never caches a refusal.
|
|
121
|
-
- **Semantic scopes are separate cache INSTANCES
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
(core's `MAX_CACHED_FORMATTERS`).
|
|
126
|
-
- `cache.invalidates` is not on `llm()`: a `SemanticCache` is not a `CacheTier`. Version bump + `ttl`
|
|
127
|
-
is the invalidation.
|
|
128
|
-
- The gateway is ambient (`configureAi`); absent at call time is `X_AI_GATEWAY_MISSING`, never a
|
|
129
|
-
default provider.
|
|
116
|
+
before the parse; a truncated turn is `X_LLM_TRUNCATED` before the schema AND before any tool
|
|
117
|
+
runs (a cut `respond` reads `{}`). The gateway never caches a refusal.
|
|
118
|
+
- **Semantic scopes are separate cache INSTANCES** (key: prompt hash AND `ctx.locale`); **the
|
|
119
|
+
default scope is the calling ACTOR**; widening is `scope: () => 'global'`; the map is bounded.
|
|
120
|
+
`cache.invalidates` is not on `llm()`: version bump + `ttl` is the invalidation.
|
|
121
|
+
- The gateway is ambient (`configureAi`); absent at call time is `X_AI_GATEWAY_MISSING`.
|
|
130
122
|
- **`llm()` forwards `ctx.signal`** onto `GenerateRequest`; `.stream()` inherits it.
|
|
131
123
|
- **`.stream()` is the SAME action**, marked with an ambient sink: UNVALIDATED text, then one `done`
|
|
132
|
-
with the validated value; no repair turn (`X_LLM_STREAM_INVALID`), no `respond` tool
|
|
133
|
-
reserved before the provider, reconciled at `done`. LAZY.
|
|
124
|
+
with the validated value; no repair turn (`X_LLM_STREAM_INVALID`), no `respond` tool. LAZY.
|
|
134
125
|
- **`configureAi({ redact })` is the one seam between `vars()` and the provider**; a `Secret` in
|
|
135
|
-
`vars()` is `X_AI_PROMPT_SECRET` regardless.
|
|
126
|
+
`vars()` is `X_AI_PROMPT_SECRET` regardless.
|
|
136
127
|
- **Fallback is across PROVIDERS serving one model, never across models**; `GenerateResult.provider`
|
|
137
|
-
is stamped by the gateway
|
|
138
|
-
-
|
|
139
|
-
|
|
140
|
-
prompt, every fence closer a fenced value wrote any character of (never per value: two slots can
|
|
141
|
-
each hold half) — never deletes, never touches a template-only closer. The
|
|
142
|
-
fences are read once at `definePrompt`; no hash moves.
|
|
128
|
+
is stamped by the gateway. Server-side `fallbacks` (beta) are never sent.
|
|
129
|
+
- **A slot inside a tag pair is DATA** (`prompt-fence.ts`, #689): `render` breaks every fence closer
|
|
130
|
+
a fenced value wrote, in the ASSEMBLED prompt — never deletes, never touches a template's own.
|
|
143
131
|
- `definePrompt` refuses a re-registered version whose hash moved; the hash is over core's
|
|
144
132
|
`canonicalJson`; an absent schema hashes as `''`.
|
|
145
133
|
|
|
@@ -147,7 +135,8 @@ exposure, the whole input schema (an idempotent action's reserved `idempotencyKe
|
|
|
147
135
|
|
|
148
136
|
- **The response cache key is core's `fingerprint` over the request, each tool WHOLE** (every `llm()`
|
|
149
137
|
tool is `respond`); a throwing `cache.set` is logged (`ai.cache.write_failed`), never a failure.
|
|
150
|
-
- **An unknown stop
|
|
138
|
+
- **An unknown stop reason FAILS CLOSED as `max_tokens` on the Anthropic wire; on chat-completions
|
|
139
|
+
it is UNKNOWN** (`readFinishReason`): a whole turn is a stop, else `X_AI_PROVIDER_UNEXPECTED`.
|
|
151
140
|
- **One failure-body reader** (`error-body.ts`'s `detailOf`, bounded at `READ_LIMIT`) and one
|
|
152
141
|
`withoutKey` scrub — the embedder included.
|
|
153
142
|
- **`attempt` collects TRANSPORT failures only**: an `UltimateError` other than
|
|
@@ -156,33 +145,37 @@ exposure, the whole input schema (an idempotent action's reserved `idempotencyKe
|
|
|
156
145
|
- **The gateway's reads of a provider's throw are total**: `isRetryable` fails closed; the line goes
|
|
157
146
|
through `renderThrowable`.
|
|
158
147
|
- **`X_AI_PROVIDER_UNAVAILABLE` is the one code classified `retryable`** (`AI_ERROR_RETRY`);
|
|
159
|
-
`AiProviderUnavailableError({ unserved: true })` carries a per-instance `terminal`.
|
|
160
|
-
|
|
148
|
+
`AiProviderUnavailableError({ unserved: true })` carries a per-instance `terminal`. Six
|
|
149
|
+
deterministic failures are `terminal` (`errors-retry.ts`); the rest stay UNREGISTERED on purpose.
|
|
161
150
|
- **The retry SCHEDULE is core's (`backoffDelay`, `isRetryableStatus`); the LOOP is this package's** —
|
|
162
151
|
core's `retry()` retries unclassified throws, so it would retry a 400 (`gateway-backoff.test.ts`).
|
|
163
152
|
- Every non-2xx and every in-band `error` frame is `AiTransportError` with a real `status`.
|
|
164
153
|
- **Both wire formats answer the same question the same way** (`provider-parity.test.ts`): an in-band
|
|
165
154
|
`error` in a 200 throws (`throwInBandError`); a `refusal` detail forces `stopReason: 'refusal'`; a
|
|
166
|
-
tool call's `input` is parsed (`
|
|
155
|
+
tool call's `input` is parsed (`parseToolArguments`); the credential is scrubbed from `detail`.
|
|
156
|
+
- **Arguments that are not JSON are never thrown by a wire reader**: recorded as `malformed`, and
|
|
157
|
+
the PROVIDER decides once usage is known — dropped on a truncated turn, else
|
|
158
|
+
`X_LLM_TOOL_ARGUMENTS_INVALID` with `meta.billed`, which the gateway RECORDS (`settleFailure`).
|
|
159
|
+
- **Every throw of `llm()`/`agent()` after a provider call STARTED carries `meta.billed`**
|
|
160
|
+
(`meter.fail`): answered calls summed, `unsettled` counted and never priced.
|
|
167
161
|
- A stream ending without `message_stop` throws. **`readSse` caps the unterminated buffer**
|
|
168
162
|
(`MAX_FRAME_CHARS`); `provider` is required so a transport error names its endpoint.
|
|
169
163
|
- A tool call is emitted whole. Thinking chunks are never appended to `text`.
|
|
170
|
-
- Anthropic body: no `
|
|
171
|
-
|
|
172
|
-
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
164
|
+
- Anthropic body: no `top_p`/`top_k`, no `budget_tokens`, `effort` inside `output_config`. Model
|
|
165
|
+
IDs are exact aliases, never date-suffixed.
|
|
166
|
+
- **`temperature`/`seed` (`sampling.ts`) are sent ONLY to a row with `sampling: { … : true }`** —
|
|
167
|
+
else left out (never refused; `llm.sampling.ignored`); no `seed` on the Anthropic wire; in
|
|
168
|
+
`promptCacheKey` only when set.
|
|
169
|
+
- **Apps bring their own models and providers (M12, 25.0.0): no vendor choice, no vendor data.** The
|
|
170
|
+
registry starts EMPTY; a model resolves declaration → prompt → `Gateway.defaultModel`
|
|
171
|
+
(`resolveModel` / `modelForCall`), none is `X_AI_MODEL_UNRESOLVED` (no gateway at all:
|
|
172
|
+
`X_AI_GATEWAY_MISSING`). `vendor-neutral.test.ts` proves it; no fix line names a vendor id.
|
|
178
173
|
- **`ModelId` is `string`; the catalogue is an open registry** — `modelSpec(id)` refuses an
|
|
179
|
-
unregistered id (`X_AI_MODEL_UNKNOWN
|
|
180
|
-
the
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
require a non-empty one (`X_AI_REQUEST_INVALID`); `echoProvider().models` reads the registry. **No
|
|
185
|
-
provider has a model of its own**: a direct call naming none is `X_AI_MODEL_UNRESOLVED` from all three.
|
|
174
|
+
unregistered id (`X_AI_MODEL_UNKNOWN`) before a socket or a reservation. Re-registering REPLACES
|
|
175
|
+
the spec and keeps its rung; registration order is the ladder within a `family`
|
|
176
|
+
(`moreCapableThan`); `X_LLM_REFUSED`'s fix names a rung ABOVE or nothing.
|
|
177
|
+
- **Every provider serves the app's list** (`{ models }` required, non-empty); no provider has a
|
|
178
|
+
model of its own.
|
|
186
179
|
- **The reasoning controls are PER MODEL (`models.ts`)**: an unasked control is omitted; an asked
|
|
187
180
|
control the model lacks is `X_AI_REQUEST_INVALID`, never dropped. Adding a model is a row the app
|
|
188
181
|
registers. `disableThinkingUpTo: 'never'` refuses `thinking: 'disabled'` at every effort;
|
|
@@ -197,12 +190,13 @@ exposure, the whole input schema (an idempotent action's reserved `idempotencyKe
|
|
|
197
190
|
- **`openAiProvider()` is a FORMAT, not a vendor** — `baseUrl`, `auth`, `headers` are the
|
|
198
191
|
differences; never a second class per vendor:
|
|
199
192
|
- structured output is the `respond` tool, never `response_format`; `tool_choice` names the tool
|
|
200
|
-
only when the request offers exactly one
|
|
193
|
+
only when the request offers exactly one and the row is not `forcedToolChoice: false`;
|
|
201
194
|
- `strict: true` only when `satisfiesStrictMode` says the schema qualifies;
|
|
202
195
|
- `max_completion_tokens`; `role: 'system'`; `stream_options: { include_usage: true }` and an
|
|
203
196
|
ESTIMATE when usage never arrives; subtract `cached_tokens` from `prompt_tokens`;
|
|
204
|
-
- tool-call deltas merge by `tool_calls[].index
|
|
205
|
-
|
|
197
|
+
- tool-call deltas merge by `tool_calls[].index`, a differing `id` starting a new call; calls are
|
|
198
|
+
read at `end()` (`[DONE]` or socket close), never at the finish reason; `isComplete()` accepts
|
|
199
|
+
`[DONE]` or a finish reason, except `[DONE]` with pending tool fragments, which is refused;
|
|
206
200
|
- no code of its own (a media refusal is the shared `X_AI_CONTENT_UNSUPPORTED`);
|
|
207
201
|
`AiTransportError` takes the provider's `envVar`; the key is revealed late,
|
|
208
202
|
never stored, scrubbed from `detail`.
|
|
@@ -223,7 +217,9 @@ exposure, the whole input schema (an idempotent action's reserved `idempotencyKe
|
|
|
223
217
|
- `tools` takes a real `action()` (`AgentTool = AnyAction | ProjectableAction`,
|
|
224
218
|
`asProjectableAction`); projection happens on the FIRST RUN, memoised. An agent can be another
|
|
225
219
|
agent's tool.
|
|
226
|
-
- A tool
|
|
220
|
+
- A tool is an exposed action or an `inProcessTool()` (`isOfferedTool`); anything else is
|
|
221
|
+
`X_AGENT_TOOL_UNEXPOSED` at DECLARATION. An in-process tool is never an action, never projected
|
|
222
|
+
to MCP, has no policy (the agent's decides) and parses its own input.
|
|
227
223
|
- `ctx.signal` is read at the top of every turn, before every tool batch, and forwarded to
|
|
228
224
|
`fetch`; `X_ABORTED` is core's.
|
|
229
225
|
- One turn's tools run concurrently, results paired by index; no tool STARTS after an abort.
|
|
@@ -237,7 +233,14 @@ exposure, the whole input schema (an idempotent action's reserved `idempotencyKe
|
|
|
237
233
|
- **A 5xx code's cause reaches the model only when core's `hasPublicCause` allows it** (status from
|
|
238
234
|
`@ultimat3/http`'s `statusFor`) — else `CODE: HIDDEN_TOOL_CAUSE`, plus a branded `callerFix`. The
|
|
239
235
|
same verdict a problem document and `@ultimat3/mcp` give, from `failure-disclosure.ts` — the
|
|
240
|
-
rule a hive member's `reason` follows too. Never a second copy.
|
|
236
|
+
rule a hive member's `reason` follows too. Never a second copy. One widening, for an
|
|
237
|
+
`inProcess` tool only (`discloseOwnFailure`): a branded error under a code with NO declared
|
|
238
|
+
status is shown as `{ error: { code, cause, fix } }`; `discloseErrors: false` opts out.
|
|
239
|
+
- **`.run()`/`.continue()` are the SAME action**, marked by an ambient session only the declaring
|
|
240
|
+
agent may claim, once (a nested agent starts fresh). The handle is SIGNED (`sealTranscript`: the
|
|
241
|
+
cursor secret, own label) and bound to agent, model, tool names and `orgId`; screened before
|
|
242
|
+
anything is loaded. `maxTurns`/`tokensPerRun` are cumulative — the remainder is the ceiling,
|
|
243
|
+
NEVER a ledger debit. A failure seals an updated handle onto the error (`transcriptOf`).
|
|
241
244
|
- **`hive()` is a fan-out action** (`PRIMITIVE_FACTORIES`): `HiveResult` is a SCHEMA from the
|
|
242
245
|
member's `output`; three arms (`ok`/`failed`/`skipped`) and three counters; `members` in SPLIT
|
|
243
246
|
order with `index`; the hive never names an actor; no hive budget code; a member's throw goes
|
|
@@ -270,27 +273,20 @@ exposure, the whole input schema (an idempotent action's reserved `idempotencyKe
|
|
|
270
273
|
|
|
271
274
|
## Invariants — evals, retrieval, fix lines
|
|
272
275
|
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
`memoryVectorStore()` enforces the same envelope. **`pg-vector.live.test.ts` refuses to skip** on a
|
|
288
|
-
Postgres without the extension (CI uses `pgvector/pgvector:pg17`).
|
|
289
|
-
- The distance ordering lives in an ascending raw subquery (the only shape hnsw answers), pinned by a
|
|
290
|
-
plan assertion. hnsw scopes after the scan — assert rows, never the node; `analyze` after a backfill.
|
|
291
|
-
- Tenant and policy filters are in SQL on every statement (`conditionsSql`), on BOTH fusion halves.
|
|
292
|
-
`(tenant, id)` is the primary key. `scoped()` only TIGHTENS (`X_VECTOR_SCOPE_WIDENED`). Metadata is
|
|
293
|
-
bound `::text::jsonb`.
|
|
276
|
+
Full text of each rule: [`docs/history/ai.md`](../../docs/history/ai.md#evals-retrieval-fix-lines).
|
|
277
|
+
|
|
278
|
+
- **No fix line may name `x ai`** (planned); an eval is selected with `x test eval --filter <name>`.
|
|
279
|
+
- Every eval result carries the prompt hash; an eval gates on the DROP from its recorded baseline,
|
|
280
|
+
mean AND per case (`X_EVAL_BASELINE_MISSING` / `_INVALID`); every prompt is named by an eval
|
|
281
|
+
(`X_EVAL_MISSING`); `ULTIMATE_EVAL_RECORD=1` records, and `x verify` with it is `X_EVAL_RECORDING`.
|
|
282
|
+
- Retrieval is hybrid by default, no vector-only path. `chunk()` hard-wraps (no chunk over `size`);
|
|
283
|
+
`metadata.source` is written AFTER the caller's spread; `VectorStore.prune` is required.
|
|
284
|
+
- **An unbound store read inside a request whose actor has an `orgId` is `X_VECTOR_UNSCOPED`**
|
|
285
|
+
(both stores, every read); `scope: UNSCOPED` is the named opt-in, an omitted scope is `UNBOUND`.
|
|
286
|
+
- `postgresVectorStore()` is the only production vector path; `pg-vector.live.test.ts` refuses to
|
|
287
|
+
skip without the extension. Distance ordering is an ascending raw subquery (the shape hnsw
|
|
288
|
+
answers), pinned by a plan assertion. Tenant and policy filters are in SQL on every statement and
|
|
289
|
+
BOTH fusion halves; `(tenant, id)` is the key; `scoped()` only TIGHTENS (`X_VECTOR_SCOPE_WIDENED`).
|
|
294
290
|
|
|
295
291
|
## Commands
|
|
296
292
|
|
package/README.md
CHANGED
|
@@ -205,7 +205,7 @@ registerModel({
|
|
|
205
205
|
id: 'llama-internal-70b',
|
|
206
206
|
contextWindow: 128_000,
|
|
207
207
|
maxOutput: 8_192,
|
|
208
|
-
inputPerMillion: { minor: 20, currency: 'USD' }, // YOUR price, integer minor units
|
|
208
|
+
inputPerMillion: { minor: 20, currency: 'USD' }, // YOUR price, integer minor units — or finer: { minor: 75, currency: 'USD', scale: 3 } is $0.075
|
|
209
209
|
outputPerMillion: { minor: 40, currency: 'USD' },
|
|
210
210
|
cacheMinimumTokens: 0,
|
|
211
211
|
reasoning: { effort: false, adaptive: false, disableThinkingUpTo: undefined },
|
|
@@ -214,6 +214,8 @@ registerModel({
|
|
|
214
214
|
configureAi({ gateway: providerGateway({ providers: [new InternalGatewayProvider()] }) });
|
|
215
215
|
```
|
|
216
216
|
|
|
217
|
+
Two optional per-model rows: `sampling: { temperature: true, seed: true }` (the model takes the `temperature` / `seed` an `llm()` or `agent()` declares — absent, they are not sent) and `forcedToolChoice: false` (the model rejects a forced `tool_choice`, so `llm()`'s one `respond` tool is offered under the endpoint's default).
|
|
218
|
+
|
|
217
219
|
**The framework registers no model.** `registerModel` is the one way into the catalogue and only the
|
|
218
220
|
app calls it — at boot, by convention in its own `models.ts`, each row with the page and the date its
|
|
219
221
|
numbers were read from, so a stale price is a diff in the app's repo rather than a framework release.
|
|
@@ -433,7 +435,7 @@ sub-agent in its own `tools` and the sub-agent runs under the same actor, throug
|
|
|
433
435
|
| an aborted `ctx` unwinds the run — at the top of every turn, before every tool batch, and on the socket | the transcript IS the request, so a loop that keeps going after the caller disconnects re-sends it once per remaining turn, runs every remaining side effect and discards the answer. `ctx.signal` rides on `GenerateRequest` too, so a call already in flight is cut rather than paid for |
|
|
434
436
|
| the tools of **one turn** run concurrently, results paired by `tool_use` id | a turn asking for five tools cost 5x wall clock and nothing said so. Order is positional, never by completion; the batch is bounded by what one turn asked for, and each tool is an action with its own `policy` and `rateLimit`, so a second ceiling here would be a throttle competing with those |
|
|
435
437
|
| `onTurn` reports each completed turn as it happens (and an `agent.turn` span event, always) | a 90-second run emitted nothing until it returned. Observation only — it cannot steer the loop, see the transcript or reach the actor — and a throw from it fails the run rather than being swallowed |
|
|
436
|
-
| a tool
|
|
438
|
+
| a tool is an `mcp: { expose: true }` action or an `inProcessTool()`; anything else is `X_AGENT_TOOL_UNEXPOSED` **at declaration** | a silently dropped tool reads as offered and is not; `isMcpExposed` is the one predicate for ACTIONS, so an in-app agent and an external MCP client see the same catalogue of them — and a tool no MCP client may see is declared as one that is this process's alone |
|
|
437
439
|
| running out of turns is `X_AGENT_MAX_TURNS`, never a partial answer | a half-finished transcript returned as a result is working notes presented as a decision |
|
|
438
440
|
| a cycle of 1 to 3 tool turns — same calls, arguments, results and failures — gone round `maxRepeatedToolCalls` (default 3) times is `X_AGENT_TOOL_LOOP` | such a turn can only be followed by another like it; a changed result is progress; a repair turn neither counts nor resets |
|
|
439
441
|
| `deadlineMs` (no default) is the run's wall-clock ceiling: `X_AGENT_DEADLINE` | the in-flight model call is aborted on the socket and every tool's `ctx.signal` fires; a tool that ignores it still finishes late. A caller that leaves first is still `X_ABORTED` |
|
|
@@ -443,6 +445,14 @@ sub-agent in its own `tools` and the sub-agent runs under the same actor, throug
|
|
|
443
445
|
| a tool result is truncated, and says so | the transcript IS the request, so an untruncated result is re-billed once per remaining turn |
|
|
444
446
|
| **no semantic cache** | similar prompts do not have similar answers once the answer depends on what `lookupOrder` returned this second |
|
|
445
447
|
|
|
448
|
+
### What a call cost
|
|
449
|
+
|
|
450
|
+
`cost` is the currency's minor units, rounded up once per call; `costExact` beside it is the same figure exactly (`Money` at a finer `scale`, from `exactCostOf`). `agent()` reports both per turn and for the run in `onTurn`; `llm()` per provider call in `onCall`; a ledger sums exact costs and rounds once when read. `repairOf` on either event is what a repair turn was told to fix.
|
|
451
|
+
|
|
452
|
+
### Continuing a run
|
|
453
|
+
|
|
454
|
+
`agent()` returns an `AgentAction`: the action, plus `.run(input, opts)` → `{ output, transcript, turns, usage }` and `.continue(input, { transcript, user }, opts)`, which appends one user turn to a stored transcript and resumes the loop. The transcript is re-sent byte for byte (the provider's prompt cache reads it back; no tool re-runs); `maxTurns` and `budget.tokensPerRun` are cumulative; the handle is signed, bound to the agent, model, tools and tenant, and screened (`X_AGENT_TRANSCRIPT_INVALID`); a failed call leaves an updated one on the error (`transcriptOf(error)`), which is the one a job stores. Full contract: [Agents → Continuing a run](https://github.com/developerz-ai/ultimate/wiki/Agents#continuing-a-run-run--continue).
|
|
455
|
+
|
|
446
456
|
### Provider prompt caching
|
|
447
457
|
|
|
448
458
|
Not the gateway's response cache and not `llm()`'s semantic one: this asks the **provider** to keep
|
|
@@ -504,7 +514,7 @@ ever sent.
|
|
|
504
514
|
|
|
505
515
|
| Limit | Detail |
|
|
506
516
|
|---|---|
|
|
507
|
-
| `run.cost`
|
|
517
|
+
| `run.cost` and the ledger | one contract: cost is computed exactly and rounded up once. `AgentTurn.run.cost` is the run's summed usage priced once; `gateway.spent()` / `BudgetLedger.report().cost` is the exact cost of every recorded call summed, rounded once when read (`costExact` beside it). They agree for a run on built-in providers; a `Provider` that states no `costExact` is summed from its rounded `cost`. Budgets are enforced against the ledger |
|
|
508
518
|
| the Anthropic row matches any host | an `anthropicProvider({ baseUrl })` behind a proxy still sends marks |
|
|
509
519
|
| more than about 9 tool calls in one turn, on OpenRouter | only the tail is marked there and the provider looks back 20 blocks, so that turn's transcript is rewritten, not read |
|
|
510
520
|
| a mark never lands on an empty or whitespace-only text block, and never makes a fifth | it moves back a block or is dropped; a raw caller's own marks are counted first |
|
|
@@ -925,6 +935,8 @@ An in-app agent and an external MCP agent both end at the same `invoke` — `run
|
|
|
925
935
|
carries it, and an action facade has no `.run` of its own. So they authorize identically. The
|
|
926
936
|
actor comes from the request context, never from the model.
|
|
927
937
|
|
|
938
|
+
`inProcessTool({ name, description, input, execute })` is a tool that is this process's alone: no `mcp`, no policy of its own (the agent's decides), in no catalogue but the agent's. Its own typed refusal reads back as `{"error":{"code","cause","fix"}}`. [Agents → In-process tools](https://github.com/developerz-ai/ultimate/wiki/Agents#in-process-tools).
|
|
939
|
+
|
|
928
940
|
A failed tool reads back as `CODE: cause (fix: …)` — except a 5xx code whose cause core's
|
|
929
941
|
`hasPublicCause` does not declare public, which reads `CODE: ` plus a fixed sentence (and a
|
|
930
942
|
`callerFix`, when the `UltimateError` declared one): the rule a production problem document follows,
|
|
@@ -953,7 +965,7 @@ fail-closed `terminal` default.
|
|
|
953
965
|
| `X_LLM_OUTPUT_INVALID` | the model failed its `output` schema on the answer and on the repair turn |
|
|
954
966
|
| `X_LLM_STREAM_INVALID` | a streamed answer failed its schema, and a stream cannot take a repair turn |
|
|
955
967
|
| `X_AGENT_MAX_TURNS` | an `agent()` used every turn without answering |
|
|
956
|
-
| `X_AGENT_TOOL_UNEXPOSED` | an `agent()` lists
|
|
968
|
+
| `X_AGENT_TOOL_UNEXPOSED` | an `agent()` lists a tool that is neither in-process nor an MCP-exposed action |
|
|
957
969
|
| `X_AGENT_TOOL_LOOP` | an `agent()` made the same tool calls and read the same results `maxRepeatedToolCalls` turns in a row |
|
|
958
970
|
| `X_AGENT_DEADLINE` | an `agent()` outlived its declared `deadlineMs` |
|
|
959
971
|
| `X_EVAL_THRESHOLD` | an eval scored below its bar |
|
|
@@ -962,6 +974,8 @@ fail-closed `terminal` default.
|
|
|
962
974
|
| `X_VECTOR_UNSCOPED` | a store with no tenant bound was read inside a request acting for an org; `scope: UNSCOPED` is the named opt-out |
|
|
963
975
|
| `X_NOT_IMPLEMENTED` | a remote driver with no key or transport; the fix names the env var |
|
|
964
976
|
|
|
977
|
+
A failure thrown after a provider call was started carries what the run was billed on `meta.billed` (`billedOf(error)` → `{ usage, cost, calls, unsettled }`); `unsettled` counts the calls whose bill cannot be known. [Agents → Unattended runs](https://github.com/developerz-ai/ultimate/wiki/Agents#unattended-runs).
|
|
978
|
+
|
|
965
979
|
### Error classes
|
|
966
980
|
|
|
967
981
|
Every error class `src/index.ts` exports, for `instanceof` inside one process. Across a wire or
|
|
@@ -972,6 +986,7 @@ a job boundary the class is gone and the `code` is what survives — match on th
|
|
|
972
986
|
| `AgentDeadlineError` | `X_AGENT_DEADLINE` | `src/agent-errors.ts` |
|
|
973
987
|
| `AgentMaxTurnsError` | `X_AGENT_MAX_TURNS` | `src/errors.ts` |
|
|
974
988
|
| `AgentToolLoopError` | `X_AGENT_TOOL_LOOP` | `src/agent-errors.ts` |
|
|
989
|
+
| `AgentTranscriptInvalidError` | `X_AGENT_TRANSCRIPT_INVALID` | `src/agent-errors.ts` |
|
|
975
990
|
| `AgentToolUnexposedError` | `X_AGENT_TOOL_UNEXPOSED` | `src/errors.ts` |
|
|
976
991
|
| `AiBudgetExceededError` | `X_AI_BUDGET_EXCEEDED` | `src/errors.ts` |
|
|
977
992
|
| `AiContentUnsupportedError` | `X_AI_CONTENT_UNSUPPORTED` | `src/content-errors.ts` |
|
|
@@ -994,7 +1009,8 @@ a job boundary the class is gone and the `code` is what survives — match on th
|
|
|
994
1009
|
| `LlmOutputInvalidError` | `X_LLM_OUTPUT_INVALID` | `src/errors.ts` |
|
|
995
1010
|
| `LlmRefusedError` | `X_LLM_REFUSED` | `src/errors.ts` |
|
|
996
1011
|
| `LlmStreamInvalidError` | `X_LLM_STREAM_INVALID` | `src/errors.ts` |
|
|
997
|
-
| `
|
|
1012
|
+
| `LlmToolArgumentsError` | `X_LLM_TOOL_ARGUMENTS_INVALID` | `src/answer-errors.ts` |
|
|
1013
|
+
| `LlmTruncatedError` | `X_LLM_TRUNCATED` | `src/answer-errors.ts` |
|
|
998
1014
|
| `VectorDimMismatchError` | `X_VECTOR_DIM_MISMATCH` | `src/errors.ts` |
|
|
999
1015
|
| `VectorScopeWidenedError` | `X_VECTOR_SCOPE_WIDENED` | `src/errors.ts` |
|
|
1000
1016
|
| `VectorUnscopedError` | `X_VECTOR_UNSCOPED` | `src/vector-scope.ts` |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ultimat3/ai",
|
|
3
|
-
"version": "27.
|
|
3
|
+
"version": "27.10.1",
|
|
4
4
|
"description": "LLM gateway, versioned prompts, evals as tests, embeddings, hybrid vector search, RAG",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
@@ -32,15 +32,15 @@
|
|
|
32
32
|
"test": "bun test"
|
|
33
33
|
},
|
|
34
34
|
"dependencies": {
|
|
35
|
-
"@ultimat3/action": "27.
|
|
36
|
-
"@ultimat3/cache": "27.
|
|
37
|
-
"@ultimat3/core": "27.
|
|
38
|
-
"@ultimat3/db": "27.
|
|
39
|
-
"@ultimat3/http": "27.
|
|
40
|
-
"@ultimat3/jobs": "27.
|
|
41
|
-
"@ultimat3/money": "27.
|
|
42
|
-
"@ultimat3/policy": "27.
|
|
43
|
-
"@ultimat3/schema": "27.
|
|
44
|
-
"@ultimat3/time": "27.
|
|
35
|
+
"@ultimat3/action": "27.10.1",
|
|
36
|
+
"@ultimat3/cache": "27.10.1",
|
|
37
|
+
"@ultimat3/core": "27.10.1",
|
|
38
|
+
"@ultimat3/db": "27.10.1",
|
|
39
|
+
"@ultimat3/http": "27.10.1",
|
|
40
|
+
"@ultimat3/jobs": "27.10.1",
|
|
41
|
+
"@ultimat3/money": "27.10.1",
|
|
42
|
+
"@ultimat3/policy": "27.10.1",
|
|
43
|
+
"@ultimat3/schema": "27.10.1",
|
|
44
|
+
"@ultimat3/time": "27.10.1"
|
|
45
45
|
}
|
|
46
46
|
}
|