@ultimat3/ai 16.0.0 → 18.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +31 -0
- package/package.json +11 -11
- package/src/agent.ts +27 -3
- package/src/budget.ts +21 -2
- package/src/embeddings.ts +12 -2
- package/src/evals.ts +5 -1
- package/src/gateway.ts +33 -4
- package/src/hive.ts +22 -8
- package/src/llm.ts +16 -2
- package/src/models.ts +17 -0
- package/src/pg-vector.ts +21 -5
- package/src/rag.ts +18 -4
- package/src/remote-embedder.ts +30 -5
- package/src/scorers.ts +5 -1
- package/src/vector.ts +36 -9
package/CLAUDE.md
CHANGED
|
@@ -63,11 +63,42 @@ until 2026-08, naming a tool no catalog contained (`llm.test.ts`, `agent.test.ts
|
|
|
63
63
|
| `eval-errors.ts` | the five `X_EVAL_*` classes; their codes and titles stay in `errors.ts` |
|
|
64
64
|
| `llm-cache.ts` | the semantic cache half of `llm()`: what a declaration may partition on, and the store it reaches |
|
|
65
65
|
| `llm-fixture.ts` | the harness `llm.test.ts` and `llm-cache.test.ts` share. Not shipped (`!src/**/*-fixture.ts`) |
|
|
66
|
+
| `bounds-fixture.ts` | the one `refusal`/`asyncRefusal` every numeric-bound suite here asserts through. Not shipped, same rule |
|
|
67
|
+
| `agent-bounds.test.ts` | the agent loop's ceilings, split out because `agent.test.ts` is at the 500-line ceiling |
|
|
66
68
|
| `runtime.ts` | the ambient gateway / embedder / semantic caches an `llm()` reaches |
|
|
67
69
|
| `fix-line.ts` / `fix-line.evals.ts` / `fix-line.v1.baseline.json` / `fix-line.eval.test.ts` | the package's own dogfood eval — the first framework-level `*.eval.test.ts`, proving the `defineEval`/baseline convention actually fails a build |
|
|
68
70
|
|
|
69
71
|
## Invariants
|
|
70
72
|
|
|
73
|
+
- **Every numeric option in this package is screened, and `??` is not the screen — `As of
|
|
74
|
+
2026-08-26`.** `NaN` is not nullish, so a default never fires for it, and `Math.max`, `Math.min`
|
|
75
|
+
and `Math.floor` all PROPAGATE it. `finiteOption`/`finiteCount` from `@ultimat3/core` are the one
|
|
76
|
+
form (never a local copy — `scripts/flight-copies.ts` exists because four grew once), and
|
|
77
|
+
`bun run finite-bounds` is the ratchet — it saw 21 of these sites, and the ones it CANNOT see (a
|
|
78
|
+
required option with no `??`, a default parameter, a cross-field product like
|
|
79
|
+
`Math.max(input.k * 4, 20)`) were the worse half. What they DID:
|
|
80
|
+
`chunk({ size: NaN })` was a synchronous infinite loop past every `AbortSignal`;
|
|
81
|
+
`embedBatched(…, 0)` re-issued the same empty batch to a paid endpoint forever;
|
|
82
|
+
`hive({ concurrency: NaN })` ran `Array.from({ length: NaN })` workers, i.e. none, and returned
|
|
83
|
+
`0 ok / 0 failed / 0 skipped` as a clean run; `RemoteEmbedder({ batchSize: NaN })` sent ONE
|
|
84
|
+
request of zero inputs and answered zero vectors; `k1`, `k`, `rrfK` each turned a search into an
|
|
85
|
+
empty list or an unranked one. **The worst is `maxTokens`, and it is not about the request**: it
|
|
86
|
+
IS the pre-flight estimate, `BudgetLedger.assertScope` asks `want > remaining`, every comparison
|
|
87
|
+
against a `NaN` is false, and `debit` then writes that `NaN` onto the ambient ledger AND the
|
|
88
|
+
per-process `BudgetStore` — measured, a 5,000,000-token call passed a 1,000-token ceiling on the
|
|
89
|
+
next reserve. One unscreened declaration turns every actor and org ceiling in the process off for
|
|
90
|
+
the life of the process. Hence `Gateway.generate`/`stream` screen it at the one seam every model
|
|
91
|
+
call passes, `registerModel` screens `maxOutput` (which reaches the same estimate through
|
|
92
|
+
`Math.min`), and `llm()`/`agent()` screen theirs at DECLARATION, beside `respondToolFor` and the
|
|
93
|
+
`X_AGENT_TOOL_UNEXPOSED` check, so a module-scope declaration fails the boot rather than the
|
|
94
|
+
ninetieth second of a run.
|
|
95
|
+
- **A bound is screened under the key the DECLARATION uses, even when that costs a second check.**
|
|
96
|
+
`budget.tokensPerRun` reaches `BudgetLimits` as `request`, so `limitsOf` in `llm.ts`, `agent.ts`
|
|
97
|
+
and `hive.ts` screens it before the ledger's own constructor does — a `fix:` naming a key the app
|
|
98
|
+
never wrote is not a fix. Same reason `retrieve()` screens its own `k` rather than letting the
|
|
99
|
+
store refuse the `k * 3` it becomes, and `RemoteEmbedder` screens `maxResponseBytes` itself
|
|
100
|
+
rather than letting core's reader answer `readWithinLimit was given a limit of NaN` — which is
|
|
101
|
+
correct, unactionable, and only arrives after the request has been paid for.
|
|
71
102
|
- **`llm()` returns an `action`. It is not a ninth primitive** (root `CLAUDE.md`, 2026-08). It
|
|
72
103
|
never re-implements parse, authz or invoke — `action()` owns those and `invoke` runs them.
|
|
73
104
|
- `src/index.ts` re-exports `t` from `@ultimat3/schema` **verbatim**, so an `llm` file imports one
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ultimat3/ai",
|
|
3
|
-
"version": "
|
|
3
|
+
"version": "18.0.0",
|
|
4
4
|
"description": "LLM gateway, versioned prompts, evals as tests, embeddings, hybrid vector search, RAG",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
@@ -25,21 +25,21 @@
|
|
|
25
25
|
"LICENSE"
|
|
26
26
|
],
|
|
27
27
|
"engines": {
|
|
28
|
-
"bun": ">=1.
|
|
28
|
+
"bun": ">=1.4.0"
|
|
29
29
|
},
|
|
30
30
|
"scripts": {
|
|
31
31
|
"typecheck": "tsc --noEmit -p tsconfig.json",
|
|
32
32
|
"test": "bun test"
|
|
33
33
|
},
|
|
34
34
|
"dependencies": {
|
|
35
|
-
"@ultimat3/action": "
|
|
36
|
-
"@ultimat3/cache": "
|
|
37
|
-
"@ultimat3/core": "
|
|
38
|
-
"@ultimat3/db": "
|
|
39
|
-
"@ultimat3/jobs": "
|
|
40
|
-
"@ultimat3/money": "
|
|
41
|
-
"@ultimat3/policy": "
|
|
42
|
-
"@ultimat3/schema": "
|
|
43
|
-
"@ultimat3/time": "
|
|
35
|
+
"@ultimat3/action": "18.0.0",
|
|
36
|
+
"@ultimat3/cache": "18.0.0",
|
|
37
|
+
"@ultimat3/core": "18.0.0",
|
|
38
|
+
"@ultimat3/db": "18.0.0",
|
|
39
|
+
"@ultimat3/jobs": "18.0.0",
|
|
40
|
+
"@ultimat3/money": "18.0.0",
|
|
41
|
+
"@ultimat3/policy": "18.0.0",
|
|
42
|
+
"@ultimat3/schema": "18.0.0",
|
|
43
|
+
"@ultimat3/time": "18.0.0"
|
|
44
44
|
}
|
|
45
45
|
}
|
package/src/agent.ts
CHANGED
|
@@ -23,7 +23,7 @@
|
|
|
23
23
|
import type { Action, ActionMcp, ActionPolicy } from '@ultimat3/action';
|
|
24
24
|
import { action } from '@ultimat3/action';
|
|
25
25
|
import type { Ctx, Span } from '@ultimat3/core';
|
|
26
|
-
import { isMcpExposed, throwIfAborted, withSpan } from '@ultimat3/core';
|
|
26
|
+
import { finiteCount, isMcpExposed, throwIfAborted, withSpan } from '@ultimat3/core';
|
|
27
27
|
import type { Money } from '@ultimat3/money';
|
|
28
28
|
import type { InferOutput, StandardSchemaV1 } from '@ultimat3/schema';
|
|
29
29
|
import { formatIssues, validateAsync } from '@ultimat3/schema';
|
|
@@ -73,6 +73,9 @@ const DEFAULT_MAX_TOKENS = 4_096;
|
|
|
73
73
|
*/
|
|
74
74
|
const DEFAULT_TOOL_RESULT_CHARS = 4_000;
|
|
75
75
|
|
|
76
|
+
/** Named in every bound refusal, so the fix names the key the declaration carries. */
|
|
77
|
+
const SUBJECT = 'agent';
|
|
78
|
+
|
|
76
79
|
export interface AgentBudget extends LlmBudget {
|
|
77
80
|
/**
|
|
78
81
|
* Token ceiling for the WHOLE run, every turn counted. The one ceiling `llm()` does not need:
|
|
@@ -161,6 +164,21 @@ export function agent<
|
|
|
161
164
|
tools: unexposed.map(toolLabel),
|
|
162
165
|
});
|
|
163
166
|
}
|
|
167
|
+
// The three loop bounds, screened at DECLARATION for the same reason the tool check above is:
|
|
168
|
+
// the values are here, and an `agent()` runs at module scope, so a bound that is not one fails
|
|
169
|
+
// the boot instead of the ninetieth second of a run. None of them is checked by what it lands
|
|
170
|
+
// on — `turn <= NaN` is false on the FIRST comparison, so the loop takes zero turns and reports
|
|
171
|
+
// `X_AGENT_MAX_TURNS` about a model that was never called; `slice(0, NaN)` is `''`, so every
|
|
172
|
+
// tool result reaches the model empty while the run keeps paying for turns; and `maxTokens`
|
|
173
|
+
// becomes the pre-flight estimate, where a `NaN` disables the ledger for the whole process.
|
|
174
|
+
finiteCount(SUBJECT, 'maxTurns', def.maxTurns ?? DEFAULT_MAX_TURNS, 1);
|
|
175
|
+
finiteCount(
|
|
176
|
+
SUBJECT,
|
|
177
|
+
'maxToolResultChars',
|
|
178
|
+
def.maxToolResultChars ?? DEFAULT_TOOL_RESULT_CHARS,
|
|
179
|
+
1,
|
|
180
|
+
);
|
|
181
|
+
finiteCount(SUBJECT, 'maxTokens', def.maxTokens ?? DEFAULT_MAX_TOKENS, 1);
|
|
164
182
|
// Projected on FIRST RUN, memoised, for the reason above: `agent()` is evaluated at module
|
|
165
183
|
// scope and a tool's name is stamped by `registerAction` at boot, so naming it here would make
|
|
166
184
|
// the ordinary `export const publishPost = action(...)` beside it `X_ACTION_UNREGISTERED`.
|
|
@@ -389,10 +407,16 @@ function limitsOf<
|
|
|
389
407
|
>(def: AgentDef<TInput, TOutput, V>): BudgetLimits {
|
|
390
408
|
const budget = def.budget;
|
|
391
409
|
return {
|
|
392
|
-
|
|
410
|
+
// Screened under the names the DECLARATION uses, not the ledger's: `tokensPerRun` arrives at
|
|
411
|
+
// `BudgetLimits` as `request`, and a fix line naming a key the app never wrote is not a fix.
|
|
412
|
+
...(budget?.tokensIn === undefined
|
|
413
|
+
? {}
|
|
414
|
+
: { tokensIn: finiteCount(SUBJECT, 'budget.tokensIn', budget.tokensIn) }),
|
|
393
415
|
...(budget?.costPerCall === undefined ? {} : { costPerCall: budget.costPerCall }),
|
|
394
416
|
// The ledger's `request` scope accumulates across every call made under one ledger, which for
|
|
395
417
|
// a run under `withBudget` is exactly "the whole run".
|
|
396
|
-
...(budget?.tokensPerRun === undefined
|
|
418
|
+
...(budget?.tokensPerRun === undefined
|
|
419
|
+
? {}
|
|
420
|
+
: { request: finiteCount(SUBJECT, 'budget.tokensPerRun', budget.tokensPerRun) }),
|
|
397
421
|
};
|
|
398
422
|
}
|
package/src/budget.ts
CHANGED
|
@@ -10,7 +10,7 @@
|
|
|
10
10
|
// module-scope `new` threw at EVALUATION in a browser bundle, where the bundler stubs
|
|
11
11
|
// `node:async_hooks` to `{}`, and took every importer of `@ultimat3/ai` with it.
|
|
12
12
|
|
|
13
|
-
import { asyncContext } from '@ultimat3/core';
|
|
13
|
+
import { asyncContext, finiteCount } from '@ultimat3/core';
|
|
14
14
|
import type { Money } from '@ultimat3/money';
|
|
15
15
|
import { assertSameCurrency } from '@ultimat3/money';
|
|
16
16
|
import { AiBudgetExceededError } from './errors';
|
|
@@ -138,7 +138,13 @@ export class BudgetLedger {
|
|
|
138
138
|
private turnstile: Promise<unknown> = Promise.resolve();
|
|
139
139
|
|
|
140
140
|
constructor(input: BudgetLedgerInput) {
|
|
141
|
-
|
|
141
|
+
// The ceilings are screened where they LAND, because `assertScope` compares with `>`: a `NaN`
|
|
142
|
+
// limit makes `limit - spent` a `NaN`, `want > NaN` false, and the scope silently unlimited —
|
|
143
|
+
// the ceiling does not become wrong, it stops existing. `llm()`, `agent()` and `hive()` screen
|
|
144
|
+
// the same numbers first under the key names their declarations use (`tokensPerRun` is this
|
|
145
|
+
// `request`), so this is the backstop for a `createGateway({ budget })` or a hand-built ledger,
|
|
146
|
+
// where these ARE the names the caller wrote.
|
|
147
|
+
this.limits = assertFiniteLimits(input.limits);
|
|
142
148
|
this.actorKey = input.actorKey;
|
|
143
149
|
this.orgKey = input.orgKey;
|
|
144
150
|
this.store = input.store ?? new MemoryBudgetStore();
|
|
@@ -290,6 +296,19 @@ export class BudgetLedger {
|
|
|
290
296
|
}
|
|
291
297
|
}
|
|
292
298
|
|
|
299
|
+
/**
|
|
300
|
+
* Every declared token ceiling, proven to be a number. A limit is optional and an absent one is
|
|
301
|
+
* "unlimited" by design — which is exactly why a `NaN` one is the dangerous value: it reads as a
|
|
302
|
+
* declared ceiling everywhere (`report()`, a manifest row, a log line) and enforces nothing.
|
|
303
|
+
*/
|
|
304
|
+
function assertFiniteLimits(limits: BudgetLimits): BudgetLimits {
|
|
305
|
+
for (const scope of ['request', 'tokensIn', 'actor', 'org'] as const) {
|
|
306
|
+
const limit = limits[scope];
|
|
307
|
+
if (limit !== undefined) finiteCount('the AI budget', scope, limit);
|
|
308
|
+
}
|
|
309
|
+
return limits;
|
|
310
|
+
}
|
|
311
|
+
|
|
293
312
|
/** Spreadable single-key record, so an absent limit stays absent under exactOptionalPropertyTypes. */
|
|
294
313
|
function pick<K extends string, V>(key: K, value: V | undefined): Partial<Record<K, V>> {
|
|
295
314
|
return value === undefined ? {} : ({ [key]: value } as Record<K, V>);
|
package/src/embeddings.ts
CHANGED
|
@@ -5,6 +5,7 @@
|
|
|
5
5
|
// queried with another is a silent relevance collapse, and the only place to catch it is
|
|
6
6
|
// where the two meet. `VectorStore` compares the declared dimension and refuses.
|
|
7
7
|
|
|
8
|
+
import { finiteCount } from '@ultimat3/core';
|
|
8
9
|
import { AiEmbedderInvalidError } from './errors';
|
|
9
10
|
|
|
10
11
|
export interface Embedder {
|
|
@@ -29,8 +30,14 @@ export async function embedOne(embedder: Embedder, text: string): Promise<Float3
|
|
|
29
30
|
export async function embedBatched(
|
|
30
31
|
embedder: Embedder,
|
|
31
32
|
texts: readonly string[],
|
|
32
|
-
size = 96,
|
|
33
|
+
size: number = 96,
|
|
33
34
|
): Promise<readonly Float32Array[]> {
|
|
35
|
+
// A default PARAMETER is a bound like any option, and this one is the loop's stride: `size: 0`
|
|
36
|
+
// never advances `i` and issues the same empty batch to a paid endpoint forever — measured, a
|
|
37
|
+
// synchronous-looking `await` loop that never returns — while `size: NaN` sends ONE request of
|
|
38
|
+
// zero inputs and answers with zero vectors for however many texts it was given. Hence a floor
|
|
39
|
+
// of 1: there is no batch of nothing.
|
|
40
|
+
finiteCount('embedBatched', 'size', size, 1);
|
|
34
41
|
const out: Float32Array[] = [];
|
|
35
42
|
for (let i = 0; i < texts.length; i += size) {
|
|
36
43
|
out.push(...(await embedder.embed(texts.slice(i, i + size))));
|
|
@@ -53,7 +60,10 @@ export class HashEmbedder implements Embedder {
|
|
|
53
60
|
readonly dimension: number;
|
|
54
61
|
|
|
55
62
|
constructor(input: HashEmbedderInput = {}) {
|
|
56
|
-
|
|
63
|
+
// Floored at 1: `new Float32Array(NaN)` is a vector of LENGTH ZERO, so every embedding is
|
|
64
|
+
// empty, `cosine` answers 0 for every pair, and the ranking collapses with nothing thrown —
|
|
65
|
+
// the silent relevance collapse this file's own header says the dimension exists to catch.
|
|
66
|
+
this.dimension = finiteCount('HashEmbedder', 'dimension', input.dimension ?? 256, 1);
|
|
57
67
|
}
|
|
58
68
|
|
|
59
69
|
async embed(texts: readonly string[]): Promise<readonly Float32Array[]> {
|
package/src/evals.ts
CHANGED
|
@@ -11,6 +11,7 @@
|
|
|
11
11
|
// Every result is filed against a prompt's content hash, so a score is always attributable
|
|
12
12
|
// to an exact prompt rather than "whatever was in main that day".
|
|
13
13
|
|
|
14
|
+
import { finiteCount } from '@ultimat3/core';
|
|
14
15
|
import type { EvalBaseline, Regression } from './eval-baseline';
|
|
15
16
|
import {
|
|
16
17
|
baselinePath,
|
|
@@ -179,7 +180,10 @@ async function runEval<V extends PromptVars>(
|
|
|
179
180
|
for (const testCase of input.cases) {
|
|
180
181
|
const generated = await gateway.generate({
|
|
181
182
|
messages: [{ role: 'user' as const, content: input.prompt.render(testCase.vars) }],
|
|
182
|
-
|
|
183
|
+
// A ceiling of `NaN` is not a ceiling: it becomes the pre-flight estimate, passes every
|
|
184
|
+
// budget check and then poisons the ledger it was checked against. Refused here, per RUN,
|
|
185
|
+
// because `defineEval` takes it as a plain field and there is no earlier seam.
|
|
186
|
+
maxTokens: finiteCount('defineEval', 'maxTokens', input.maxTokens ?? 1_024, 1),
|
|
183
187
|
...(input.prompt.system !== undefined ? { system: input.prompt.system } : {}),
|
|
184
188
|
...(input.prompt.model !== undefined ? { model: input.prompt.model } : {}),
|
|
185
189
|
...(input.prompt.effort !== undefined ? { effort: input.prompt.effort } : {}),
|
package/src/gateway.ts
CHANGED
|
@@ -7,6 +7,7 @@
|
|
|
7
7
|
import type { Random } from '@ultimat3/core';
|
|
8
8
|
import {
|
|
9
9
|
backoffDelay,
|
|
10
|
+
finiteCount,
|
|
10
11
|
isRetryableStatus,
|
|
11
12
|
isUltimateError,
|
|
12
13
|
renderThrowable,
|
|
@@ -91,6 +92,15 @@ class GatewayImpl implements Gateway {
|
|
|
91
92
|
constructor(config: CreateGatewayInput) {
|
|
92
93
|
this.config = config;
|
|
93
94
|
this.retry = config.retry ?? DEFAULT_RETRY;
|
|
95
|
+
// `attempts` is the retry loop's only exit condition and nothing screened it: `attempt <= NaN`
|
|
96
|
+
// is false on the first comparison, so `attempt()` calls no provider at all and raises
|
|
97
|
+
// `X_AI_PROVIDER_UNAVAILABLE` with an EMPTY attempt list — measured, "no provider could serve
|
|
98
|
+
// model claude-opus-5 ()" for a provider that was never asked. A floor of 1 because the field
|
|
99
|
+
// is documented as total attempts INCLUDING the first, so zero of them is not a policy.
|
|
100
|
+
// `baseDelayMs` and `maxDelayMs` are deliberately not screened here: `backoffDelay` refuses a
|
|
101
|
+
// non-finite one already, and a NEGATIVE base is clamped to a zero wait on purpose
|
|
102
|
+
// (`gateway-backoff.test.ts` pins it), which a count check here would start refusing.
|
|
103
|
+
finiteCount('createGateway', 'retry.attempts', this.retry.attempts, 1);
|
|
94
104
|
this.sleep = config.sleep ?? ((ms) => new Promise((resolve) => setTimeout(resolve, ms)));
|
|
95
105
|
this.random = config.random;
|
|
96
106
|
}
|
|
@@ -113,7 +123,7 @@ class GatewayImpl implements Gateway {
|
|
|
113
123
|
|
|
114
124
|
async generate(request: GenerateRequest): Promise<GenerateResult> {
|
|
115
125
|
const model = request.model ?? this.config.defaultModel ?? DEFAULT_MODEL;
|
|
116
|
-
const resolved: GenerateRequest = { ...request, model };
|
|
126
|
+
const resolved: GenerateRequest = { ...request, model, maxTokens: ceilingOf(request) };
|
|
117
127
|
|
|
118
128
|
const cacheKey = cacheKeyFor(resolved);
|
|
119
129
|
const cached = await this.config.cache?.get(cacheKey);
|
|
@@ -150,7 +160,7 @@ class GatewayImpl implements Gateway {
|
|
|
150
160
|
|
|
151
161
|
async *stream(request: GenerateRequest): AsyncIterable<StreamChunk> {
|
|
152
162
|
const model = request.model ?? this.config.defaultModel ?? DEFAULT_MODEL;
|
|
153
|
-
const resolved: GenerateRequest = { ...request, model };
|
|
163
|
+
const resolved: GenerateRequest = { ...request, model, maxTokens: ceilingOf(request) };
|
|
154
164
|
|
|
155
165
|
// Routed BEFORE the reservation, not after it. `providerFor` throws for a registered model no
|
|
156
166
|
// configured provider serves — an ordinary boot misconfiguration — and a debit taken first has
|
|
@@ -258,6 +268,21 @@ class GatewayImpl implements Gateway {
|
|
|
258
268
|
}
|
|
259
269
|
}
|
|
260
270
|
|
|
271
|
+
/**
|
|
272
|
+
* The request's completion ceiling, screened at the one seam every model call in an app passes
|
|
273
|
+
* through — `llm()`, `agent()`, an eval, a judge and a hand-built `generate()` alike.
|
|
274
|
+
*
|
|
275
|
+
* It is not the request that a `NaN` here breaks, and that is why it is refused before anything
|
|
276
|
+
* else happens: `maxTokens` IS the pre-flight estimate, `want > remaining` is false for a `NaN`
|
|
277
|
+
* want, and `BudgetLedger.debit` then writes it onto the ambient ledger and the per-process
|
|
278
|
+
* `BudgetStore`. Both counters are `NaN` from then on, so every later comparison against them is
|
|
279
|
+
* false too — one unscreened declaration turns off every actor and org ceiling in the process,
|
|
280
|
+
* permanently, and reports nothing. Screened before the reservation, so the ledger never sees it.
|
|
281
|
+
*/
|
|
282
|
+
function ceilingOf(request: GenerateRequest): number {
|
|
283
|
+
return finiteCount('the AI gateway', 'maxTokens', request.maxTokens, 1);
|
|
284
|
+
}
|
|
285
|
+
|
|
261
286
|
/**
|
|
262
287
|
* Full jitter: a uniform pick from [0, exponential], capped BEFORE the roll.
|
|
263
288
|
*
|
|
@@ -265,8 +290,12 @@ class GatewayImpl implements Gateway {
|
|
|
265
290
|
* onto it, and nothing else. Two things came with the delegation and neither is cosmetic: the
|
|
266
291
|
* result is ROUNDED where this floored it (a shift of at most 1ms, and the same rounding
|
|
267
292
|
* `@ultimat3/jobs` and `@ultimat3/realtime` already use), and a policy carrying a `NaN` — which is
|
|
268
|
-
* what `Number(process.env.…)` answers for an unset variable —
|
|
269
|
-
*
|
|
293
|
+
* what `Number(process.env.…)` answers for an unset variable — is REFUSED rather than clamped.
|
|
294
|
+
* This paragraph said "waits 0" until 2026-08-26, which was the safe answer to the wrong question
|
|
295
|
+
* and had already stopped being true: a schedule of zeroes still spins, it just spins on purpose,
|
|
296
|
+
* so core's `backoffDelay` refuses a non-finite bound before it clamps and this gateway inherits
|
|
297
|
+
* the refusal with the curve. `gateway-backoff.test.ts` pins both halves — the refusal, and the
|
|
298
|
+
* negative base that IS still clamped to zero.
|
|
270
299
|
*/
|
|
271
300
|
export function backoffMs(policy: RetryPolicy, attempt: number, random?: Random): number {
|
|
272
301
|
return backoffDelay({
|
package/src/hive.ts
CHANGED
|
@@ -17,7 +17,7 @@
|
|
|
17
17
|
import type { Action, ActionMcp, ActionPolicy } from '@ultimat3/action';
|
|
18
18
|
import { action, actionName } from '@ultimat3/action';
|
|
19
19
|
import type { Ctx } from '@ultimat3/core';
|
|
20
|
-
import { throwIfAborted, withSpan } from '@ultimat3/core';
|
|
20
|
+
import { finiteCount, throwIfAborted, withSpan } from '@ultimat3/core';
|
|
21
21
|
import type { AnySchema, InferInput, InferOutput, StandardSchemaV1 } from '@ultimat3/schema';
|
|
22
22
|
import type { BudgetLimits } from './budget';
|
|
23
23
|
import { BudgetLedger, currentBudget, withBudget } from './budget';
|
|
@@ -42,6 +42,9 @@ const DEFAULT_CONCURRENCY = 4;
|
|
|
42
42
|
*/
|
|
43
43
|
const DEFAULT_MIN_MEMBERS = 2;
|
|
44
44
|
|
|
45
|
+
/** Named in every bound refusal, so the fix names the key the declaration carries. */
|
|
46
|
+
const SUBJECT = 'hive';
|
|
47
|
+
|
|
45
48
|
export interface HiveSplitArgs<TInput extends StandardSchemaV1> {
|
|
46
49
|
readonly input: InferOutput<TInput>;
|
|
47
50
|
readonly ctx: Ctx;
|
|
@@ -108,13 +111,18 @@ async function run<
|
|
|
108
111
|
const inputs = await def.split({ input: args.input, ctx: args.ctx });
|
|
109
112
|
if (inputs.length === 0) throw new HiveEmptyError({ member: name });
|
|
110
113
|
|
|
111
|
-
|
|
114
|
+
// Both bounds screened before either clamp reads them, because neither clamp is a screen:
|
|
115
|
+
// `Math.max` and `Math.min` PROPAGATE a `NaN`, so `concurrency: NaN` survived both and reached
|
|
116
|
+
// `Array.from({ length: NaN }, worker)` — an empty worker list, a `members` array of holes, and
|
|
117
|
+
// `0 ok / 0 failed / 0 skipped` returned as a clean run over inputs nothing ever touched.
|
|
118
|
+
// `minMembers: NaN` is quieter and the same shape: `inputs.length < NaN` is false, so the floor
|
|
119
|
+
// stops existing rather than being wrong. The floor stays 0 for both — `Math.max(1, …)` already
|
|
120
|
+
// reads a declared 0 as "one worker", and refusing it here would be a new rule, not a repair.
|
|
121
|
+
const floor = finiteCount(SUBJECT, 'minMembers', def.minMembers ?? DEFAULT_MIN_MEMBERS);
|
|
122
|
+
const declared = finiteCount(SUBJECT, 'concurrency', def.concurrency ?? DEFAULT_CONCURRENCY);
|
|
112
123
|
// A split below the floor still runs every input it produced — dropping one would be silent
|
|
113
124
|
// data loss — it just stops paying for a pool to do it.
|
|
114
|
-
const width =
|
|
115
|
-
inputs.length < floor
|
|
116
|
-
? 1
|
|
117
|
-
: Math.max(1, Math.min(def.concurrency ?? DEFAULT_CONCURRENCY, inputs.length));
|
|
125
|
+
const width = inputs.length < floor ? 1 : Math.max(1, Math.min(declared, inputs.length));
|
|
118
126
|
|
|
119
127
|
return withSpan('ai.hive', async (span) => {
|
|
120
128
|
span.setAttributes({
|
|
@@ -170,10 +178,16 @@ function limitsOf<
|
|
|
170
178
|
>(def: HiveDef<TInput, MIn, MOut>): BudgetLimits {
|
|
171
179
|
const budget = def.budget;
|
|
172
180
|
return {
|
|
173
|
-
|
|
181
|
+
// Under the declaration's own key names — `tokensPerRun` is the ledger's `request`, and a fix
|
|
182
|
+
// line has to name what the app wrote.
|
|
183
|
+
...(budget?.tokensIn === undefined
|
|
184
|
+
? {}
|
|
185
|
+
: { tokensIn: finiteCount(SUBJECT, 'budget.tokensIn', budget.tokensIn) }),
|
|
174
186
|
...(budget?.costPerCall === undefined ? {} : { costPerCall: budget.costPerCall }),
|
|
175
187
|
// The ledger's `request` scope accumulates across every call made under it, which for a hive
|
|
176
188
|
// under `withBudget` is every member's every turn.
|
|
177
|
-
...(budget?.tokensPerRun === undefined
|
|
189
|
+
...(budget?.tokensPerRun === undefined
|
|
190
|
+
? {}
|
|
191
|
+
: { request: finiteCount(SUBJECT, 'budget.tokensPerRun', budget.tokensPerRun) }),
|
|
178
192
|
};
|
|
179
193
|
}
|
package/src/llm.ts
CHANGED
|
@@ -24,7 +24,7 @@
|
|
|
24
24
|
import type { Action, ActionMcp, ActionPolicy, InvokeOptions } from '@ultimat3/action';
|
|
25
25
|
import { action } from '@ultimat3/action';
|
|
26
26
|
import type { Ctx, Span, SpanAttributes } from '@ultimat3/core';
|
|
27
|
-
import { withSpan } from '@ultimat3/core';
|
|
27
|
+
import { finiteCount, withSpan } from '@ultimat3/core';
|
|
28
28
|
import type { Money } from '@ultimat3/money';
|
|
29
29
|
import type { InferInput, InferOutput, StandardSchemaV1 } from '@ultimat3/schema';
|
|
30
30
|
import { formatIssues, toMcpInputSchema, validateAsync } from '@ultimat3/schema';
|
|
@@ -138,6 +138,15 @@ export function llm<
|
|
|
138
138
|
V extends PromptVars,
|
|
139
139
|
>(def: LlmDef<TInput, TOutput, V>): LlmAction<TInput, TOutput> {
|
|
140
140
|
const respond = respondToolFor(def.output);
|
|
141
|
+
// Screened at DECLARATION, beside `respondToolFor`'s own refusal and for the same reason: an
|
|
142
|
+
// `llm()` is evaluated at module scope, so a bound that is not one fails the boot rather than
|
|
143
|
+
// the first request. It is the read at `generate` below that makes it urgent — `maxTokens`
|
|
144
|
+
// becomes the pre-flight ESTIMATE, a `NaN` estimate passes every `want > remaining` check, and
|
|
145
|
+
// `debit` then writes that `NaN` onto the ambient ledger AND the per-process `BudgetStore`,
|
|
146
|
+
// where it never expires: one bad declaration turns every actor and org ceiling in the process
|
|
147
|
+
// off for the life of the process. A floor of 1 because a completion ceiling of zero tokens is
|
|
148
|
+
// a call that cannot answer.
|
|
149
|
+
finiteCount('llm', 'maxTokens', def.maxTokens ?? DEFAULT_MAX_TOKENS, 1);
|
|
141
150
|
const built = action<TInput, TOutput>({
|
|
142
151
|
input: def.input,
|
|
143
152
|
output: def.output,
|
|
@@ -378,7 +387,12 @@ function repair(issues: string): AiMessage {
|
|
|
378
387
|
|
|
379
388
|
function limitsOf(budget: LlmBudget | undefined): BudgetLimits {
|
|
380
389
|
return {
|
|
381
|
-
|
|
390
|
+
// Screened under the name the DECLARATION uses. `BudgetLedger` screens its own `limits` too,
|
|
391
|
+
// but its field is called `tokensIn` there by coincidence and `request` for `tokensPerRun` —
|
|
392
|
+
// a fix line has to name the key the app actually wrote.
|
|
393
|
+
...(budget?.tokensIn === undefined
|
|
394
|
+
? {}
|
|
395
|
+
: { tokensIn: finiteCount('llm', 'budget.tokensIn', budget.tokensIn) }),
|
|
382
396
|
...(budget?.costPerCall === undefined ? {} : { costPerCall: budget.costPerCall }),
|
|
383
397
|
};
|
|
384
398
|
}
|
package/src/models.ts
CHANGED
|
@@ -3,6 +3,7 @@
|
|
|
3
3
|
// has never heard of — a closed union made them untypeable, so the only way past `tsc` was to
|
|
4
4
|
// claim a Claude id and be billed Anthropic list prices for a model nobody ran. As of 2026-08.
|
|
5
5
|
|
|
6
|
+
import { finiteCount } from '@ultimat3/core';
|
|
6
7
|
import type { Money } from '@ultimat3/money';
|
|
7
8
|
import { AiModelUnknownError, AiRequestInvalidError } from './errors';
|
|
8
9
|
|
|
@@ -103,6 +104,9 @@ const usd = (minor: number): Money => ({ minor, currency: 'USD' });
|
|
|
103
104
|
*/
|
|
104
105
|
const registry = new Map<ModelId, ModelSpec>();
|
|
105
106
|
|
|
107
|
+
/** Named in every bound refusal, so the fix names the call an app makes at boot. */
|
|
108
|
+
const SUBJECT = 'registerModel';
|
|
109
|
+
|
|
106
110
|
/**
|
|
107
111
|
* Add a model to the catalogue, or restate one that is already in it. **The three built-ins
|
|
108
112
|
* register through this same call**, at the bottom of this file — so the default path is the
|
|
@@ -119,6 +123,19 @@ const registry = new Map<ModelId, ModelSpec>();
|
|
|
119
123
|
* catalogue in the order it wants, re-registering the built-in ids it keeps.
|
|
120
124
|
*/
|
|
121
125
|
export function registerModel(spec: ModelSpec): ModelSpec {
|
|
126
|
+
// Screened at the ONE seam every model in the catalogue passes through, and at boot, which is
|
|
127
|
+
// the earliest a wrong row can be caught. Not a formality: `maxOutput` reaches the pre-flight
|
|
128
|
+
// estimate through `Math.min(request.maxTokens, spec.maxOutput)` — which propagates a `NaN`
|
|
129
|
+
// rather than screening it — and a `NaN` estimate passes every `want > remaining` budget check
|
|
130
|
+
// and then writes itself onto the ledger and the per-process `BudgetStore`, where every later
|
|
131
|
+
// comparison against it is false too. A price is the same story for the money ceiling, and
|
|
132
|
+
// `costOf` answers confidently either way, so a row nobody can price is refused rather than
|
|
133
|
+
// billed. Minor units are whole by the framework's money rule, so `finiteCount` is the check.
|
|
134
|
+
finiteCount(SUBJECT, `${spec.id} contextWindow`, spec.contextWindow, 1);
|
|
135
|
+
finiteCount(SUBJECT, `${spec.id} maxOutput`, spec.maxOutput, 1);
|
|
136
|
+
finiteCount(SUBJECT, `${spec.id} cacheMinimumTokens`, spec.cacheMinimumTokens);
|
|
137
|
+
finiteCount(SUBJECT, `${spec.id} inputPerMillion.minor`, spec.inputPerMillion.minor);
|
|
138
|
+
finiteCount(SUBJECT, `${spec.id} outputPerMillion.minor`, spec.outputPerMillion.minor);
|
|
122
139
|
registry.set(spec.id, spec);
|
|
123
140
|
return spec;
|
|
124
141
|
}
|
package/src/pg-vector.ts
CHANGED
|
@@ -3,6 +3,7 @@
|
|
|
3
3
|
// the tenant and policy envelope has to be un-bypassable — every statement it emits is built on
|
|
4
4
|
// `conditionsSql`, and the fusion happens in SQL rather than after the rows are already loaded.
|
|
5
5
|
|
|
6
|
+
import { finiteCount, finiteOption } from '@ultimat3/core';
|
|
6
7
|
import { type DbClient, db, type SqlFragment } from '@ultimat3/db';
|
|
7
8
|
import { VectorDimMismatchError } from './errors';
|
|
8
9
|
import {
|
|
@@ -36,6 +37,9 @@ export interface PgVectorStoreInput {
|
|
|
36
37
|
readonly scope?: VectorScope | undefined;
|
|
37
38
|
}
|
|
38
39
|
|
|
40
|
+
/** Named in every bound refusal here, matching `vector.ts`: one store contract, one subject. */
|
|
41
|
+
const HYBRID = 'hybrid search';
|
|
42
|
+
|
|
39
43
|
/** The row every statement projects. `metadata` arrives as jsonb; drivers differ on parsing it. */
|
|
40
44
|
interface HitRow {
|
|
41
45
|
readonly id: string;
|
|
@@ -103,7 +107,12 @@ export class PgVectorStore implements VectorStore {
|
|
|
103
107
|
filter?: MetadataFilter,
|
|
104
108
|
): Promise<readonly SearchHit[]> {
|
|
105
109
|
this.assertDimension(vector.length);
|
|
106
|
-
|
|
110
|
+
// `k` is the statement's `limit` and reaches Postgres as a bound parameter, so a `NaN` is the
|
|
111
|
+
// database's error to report rather than this store's — and a fractional one is an error there
|
|
112
|
+
// too. Refused here instead, before a connection is taken, naming the argument the caller wrote.
|
|
113
|
+
return this.run(
|
|
114
|
+
searchSql(this.target, vector, { scope: this.scope, filter, k: finiteCount(HYBRID, 'k', k) }),
|
|
115
|
+
);
|
|
107
116
|
}
|
|
108
117
|
|
|
109
118
|
async searchText(
|
|
@@ -111,17 +120,24 @@ export class PgVectorStore implements VectorStore {
|
|
|
111
120
|
k: number,
|
|
112
121
|
filter?: MetadataFilter,
|
|
113
122
|
): Promise<readonly SearchHit[]> {
|
|
114
|
-
return this.run(
|
|
123
|
+
return this.run(
|
|
124
|
+
textSql(this.target, query, { scope: this.scope, filter, k: finiteCount(HYBRID, 'k', k) }),
|
|
125
|
+
);
|
|
115
126
|
}
|
|
116
127
|
|
|
117
128
|
async hybrid(input: HybridSearchInput): Promise<readonly SearchHit[]> {
|
|
118
129
|
this.assertDimension(input.vector.length);
|
|
130
|
+
// The same three bounds `MemoryVectorStore.hybrid` screens, and they have to be screened in
|
|
131
|
+
// BOTH stores: `rrfK` lands in `1.0 / (rrfK + rank)`, Postgres has a float8 `NaN`, and it
|
|
132
|
+
// sorts as the LARGEST value — so every fused score ties, the order collapses to the `id`
|
|
133
|
+
// tiebreak, and the fusion this method exists for is gone with nothing raised anywhere.
|
|
134
|
+
const k = finiteCount(HYBRID, 'k', input.k);
|
|
119
135
|
const args: PgHybridArgs = {
|
|
120
136
|
scope: this.scope,
|
|
121
137
|
filter: input.filter,
|
|
122
|
-
k
|
|
123
|
-
candidates: input.candidates ?? Math.max(
|
|
124
|
-
rrfK: input.rrfK ?? 60,
|
|
138
|
+
k,
|
|
139
|
+
candidates: finiteCount(HYBRID, 'candidates', input.candidates ?? Math.max(k * 4, 20)),
|
|
140
|
+
rrfK: finiteOption(HYBRID, 'rrfK', input.rrfK ?? 60),
|
|
125
141
|
};
|
|
126
142
|
return this.run(hybridSql(this.target, input.query, input.vector, args));
|
|
127
143
|
}
|
package/src/rag.ts
CHANGED
|
@@ -4,6 +4,7 @@
|
|
|
4
4
|
// discover at request time that they don't fit is a truncation bug waiting to happen; the
|
|
5
5
|
// assembler fills to a declared ceiling and reports what it dropped.
|
|
6
6
|
|
|
7
|
+
import { finiteCount, finiteOption } from '@ultimat3/core';
|
|
7
8
|
import type { Embedder } from './embeddings';
|
|
8
9
|
import { embedBatched, embedOne } from './embeddings';
|
|
9
10
|
import { estimateTextTokens as estimateChunkTokens } from './provider';
|
|
@@ -34,8 +35,14 @@ export function chunk(input: ChunkInput): readonly Chunk[] {
|
|
|
34
35
|
// Floored at one token: `size: 0` makes every comparison below meaningless and the wrap's cut
|
|
35
36
|
// point zero-width, which is a loop that never advances rather than a chunker that produces
|
|
36
37
|
// nothing. A budget under one token is not a budget.
|
|
37
|
-
|
|
38
|
-
|
|
38
|
+
//
|
|
39
|
+
// The floor is not a SCREEN, and that is a separate line for a separate reason: `Math.max` and
|
|
40
|
+
// `Math.floor` both propagate a `NaN`, so `size: NaN` walked past both, made `<= size` false at
|
|
41
|
+
// every cut point and put `cutToBudget` in a loop with no exit — measured, a SYNCHRONOUS spin
|
|
42
|
+
// past every `AbortSignal`, on a worker's only thread. `finiteOption` and not `finiteCount`,
|
|
43
|
+
// because the `Math.floor` above is a deliberate acceptance of a fractional budget.
|
|
44
|
+
const size = Math.max(1, Math.floor(finiteOption('chunk', 'size', input.size ?? 512)));
|
|
45
|
+
const overlap = Math.min(finiteOption('chunk', 'overlap', input.overlap ?? 64), size - 1);
|
|
39
46
|
const units = splitUnits(input.text, size);
|
|
40
47
|
const chunks: Chunk[] = [];
|
|
41
48
|
let buffer: string[] = [];
|
|
@@ -196,7 +203,10 @@ export interface RetrieveInput {
|
|
|
196
203
|
|
|
197
204
|
/** Hybrid retrieval then rerank. Hybrid by default because pure vector loses on exact terms. */
|
|
198
205
|
export async function retrieve(input: RetrieveInput): Promise<readonly SearchHit[]> {
|
|
199
|
-
|
|
206
|
+
// Screened here as well as in the store, so the refusal names the argument this caller wrote
|
|
207
|
+
// rather than the `k * 3` it becomes: `k: NaN` retrieved nothing and reported a successful
|
|
208
|
+
// retrieval of zero documents, which downstream is an answer given with no context at all.
|
|
209
|
+
const k = finiteCount('retrieve', 'k', input.k ?? 8);
|
|
200
210
|
const vector = await embedOne(input.embedder, input.query);
|
|
201
211
|
const hits = await input.store.hybrid({
|
|
202
212
|
query: input.query,
|
|
@@ -248,6 +258,10 @@ export function assembleContext(input: {
|
|
|
248
258
|
/** Between BLOCKS, never inside one — the block fence is what separates documents. */
|
|
249
259
|
readonly separator?: string;
|
|
250
260
|
}): AssembledContext {
|
|
261
|
+
// The one bound in this function, and it carries no default — so nothing screened it: `tokens +
|
|
262
|
+
// cost > NaN` is false for every hit, every document is accepted, and the ceiling that exists to
|
|
263
|
+
// keep an assembled context inside the model's window stops existing while `dropped` stays empty.
|
|
264
|
+
const maxTokens = finiteCount('assembleContext', 'maxTokens', input.maxTokens);
|
|
251
265
|
const separator = input.separator ?? '\n\n';
|
|
252
266
|
const separatorTokens = estimateChunkTokens(separator);
|
|
253
267
|
const parts: string[] = [];
|
|
@@ -258,7 +272,7 @@ export function assembleContext(input: {
|
|
|
258
272
|
for (const hit of input.hits) {
|
|
259
273
|
const block = documentBlock(hit.id, hit.text);
|
|
260
274
|
const cost = estimateChunkTokens(block) + (parts.length === 0 ? 0 : separatorTokens);
|
|
261
|
-
if (tokens + cost >
|
|
275
|
+
if (tokens + cost > maxTokens) {
|
|
262
276
|
dropped.push(hit.id);
|
|
263
277
|
continue;
|
|
264
278
|
}
|
package/src/remote-embedder.ts
CHANGED
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
// vendor: `baseUrl` selects the provider and nothing else changes. A second class per vendor
|
|
7
7
|
// would be a second thing to learn for a difference that does not exist on the wire.
|
|
8
8
|
|
|
9
|
-
import { readWithinLimit, renderThrowable } from '@ultimat3/core';
|
|
9
|
+
import { finiteCount, readWithinLimit, renderThrowable } from '@ultimat3/core';
|
|
10
10
|
import type { Embedder } from './embeddings';
|
|
11
11
|
import { normalize } from './embeddings';
|
|
12
12
|
import { AiKeyMissingError, AiTransportError, EmbedderDimMismatchError } from './errors';
|
|
@@ -25,6 +25,8 @@ const DETAIL_LIMIT = 300;
|
|
|
25
25
|
*/
|
|
26
26
|
const DEFAULT_TIMEOUT_MS = 30_000;
|
|
27
27
|
const DEFAULT_MAX_RESPONSE_BYTES = 32 * 1024 * 1024;
|
|
28
|
+
/** Named in every bound refusal, so the fix names the constructor argument the app wrote. */
|
|
29
|
+
const SUBJECT = 'RemoteEmbedder';
|
|
28
30
|
|
|
29
31
|
export interface RemoteEmbedderInput {
|
|
30
32
|
/** The provider's model id. Doubles as the embedder name, so a store records what wrote it. */
|
|
@@ -52,10 +54,33 @@ export class RemoteEmbedder implements Embedder {
|
|
|
52
54
|
readonly name: string;
|
|
53
55
|
readonly dimension: number;
|
|
54
56
|
private readonly config: RemoteEmbedderInput;
|
|
57
|
+
/**
|
|
58
|
+
* The three numeric bounds, resolved and SCREENED once when the embedder is built rather than
|
|
59
|
+
* read fresh per batch. Each of the three failed differently and none of them raised anything a
|
|
60
|
+
* caller could act on: `batchSize: 0` never advanced the loop and issued the same empty request
|
|
61
|
+
* to a paid endpoint forever, `batchSize: NaN` sent ONE request of zero inputs and answered zero
|
|
62
|
+
* vectors for however many texts it was handed, `timeoutMs: NaN` threw a `TypeError` out of
|
|
63
|
+
* `AbortSignal.timeout` that this file's own `catch` re-dressed as `X_AI_PROVIDER_UNAVAILABLE` —
|
|
64
|
+
* a config typo reported as a transport failure, which the gateway then RETRIES — and
|
|
65
|
+
* `maxResponseBytes: NaN` reached core's reader, which refuses it correctly but only after the
|
|
66
|
+
* request has been paid for, in a message about `readWithinLimit` rather than about this option.
|
|
67
|
+
*/
|
|
68
|
+
private readonly batchSize: number;
|
|
69
|
+
private readonly timeoutMs: number;
|
|
70
|
+
private readonly maxResponseBytes: number;
|
|
55
71
|
|
|
56
72
|
constructor(input: RemoteEmbedderInput) {
|
|
57
73
|
this.name = input.name;
|
|
58
|
-
this.dimension = input.dimension;
|
|
74
|
+
this.dimension = finiteCount(SUBJECT, 'dimension', input.dimension, 1);
|
|
75
|
+
this.batchSize = finiteCount(SUBJECT, 'batchSize', input.batchSize ?? DEFAULT_BATCH_SIZE, 1);
|
|
76
|
+
// A floor of 1 rather than 0: `AbortSignal.timeout(0)` aborts on the next tick, so a zero here
|
|
77
|
+
// is not "no deadline" — it is every request failing before the socket is even opened.
|
|
78
|
+
this.timeoutMs = finiteCount(SUBJECT, 'timeoutMs', input.timeoutMs ?? DEFAULT_TIMEOUT_MS, 1);
|
|
79
|
+
this.maxResponseBytes = finiteCount(
|
|
80
|
+
SUBJECT,
|
|
81
|
+
'maxResponseBytes',
|
|
82
|
+
input.maxResponseBytes ?? DEFAULT_MAX_RESPONSE_BYTES,
|
|
83
|
+
);
|
|
59
84
|
this.config = input;
|
|
60
85
|
}
|
|
61
86
|
|
|
@@ -65,7 +90,7 @@ export class RemoteEmbedder implements Embedder {
|
|
|
65
90
|
*/
|
|
66
91
|
async embed(texts: readonly string[]): Promise<readonly Float32Array[]> {
|
|
67
92
|
if (texts.length === 0) return [];
|
|
68
|
-
const size = this.
|
|
93
|
+
const size = this.batchSize;
|
|
69
94
|
const vectors: Float32Array[] = [];
|
|
70
95
|
for (let start = 0; start < texts.length; start += size) {
|
|
71
96
|
vectors.push(...(await this.batch(texts.slice(start, start + size))));
|
|
@@ -79,7 +104,7 @@ export class RemoteEmbedder implements Embedder {
|
|
|
79
104
|
throw new AiKeyMissingError({ provider: this.name, envVar: API_KEY_ENV });
|
|
80
105
|
}
|
|
81
106
|
const doFetch: AiFetch = this.config.fetch ?? fetch;
|
|
82
|
-
const timeoutMs = this.
|
|
107
|
+
const timeoutMs = this.timeoutMs;
|
|
83
108
|
const url = `${this.config.baseUrl ?? DEFAULT_BASE_URL}/embeddings`;
|
|
84
109
|
let response: Response;
|
|
85
110
|
try {
|
|
@@ -113,7 +138,7 @@ export class RemoteEmbedder implements Embedder {
|
|
|
113
138
|
}
|
|
114
139
|
// Read through core's counting reader rather than `response.json()`: a body is buffered whole
|
|
115
140
|
// before anything measures it otherwise, and a `content-length` a remote wrote is not a bound.
|
|
116
|
-
const maxBytes = this.
|
|
141
|
+
const maxBytes = this.maxResponseBytes;
|
|
117
142
|
const read = await readWithinLimit(response.body, maxBytes);
|
|
118
143
|
if ('over' in read) {
|
|
119
144
|
throw new AiTransportError({
|
package/src/scorers.ts
CHANGED
|
@@ -4,6 +4,7 @@
|
|
|
4
4
|
// is a model call, which means it is itself a measuring instrument that can drift — so its
|
|
5
5
|
// prompt is a versioned artifact and its hash is part of the scorer's name.
|
|
6
6
|
|
|
7
|
+
import { finiteCount } from '@ultimat3/core';
|
|
7
8
|
import type { Gateway } from './gateway';
|
|
8
9
|
import type { Prompt } from './prompt';
|
|
9
10
|
|
|
@@ -109,7 +110,10 @@ export function llmJudge(input: {
|
|
|
109
110
|
content: input.judge.render({ output, expected: expected ?? '' }),
|
|
110
111
|
},
|
|
111
112
|
],
|
|
112
|
-
|
|
113
|
+
// Same rule as every other completion ceiling here, and this one is read per SCORE: a
|
|
114
|
+
// `NaN` reaches the gateway as the estimate, and an estimate that is not a number turns
|
|
115
|
+
// the ambient budget off rather than exceeding it.
|
|
116
|
+
maxTokens: finiteCount('llmJudge', 'maxTokens', input.maxTokens ?? 256, 1),
|
|
113
117
|
...(input.judge.system !== undefined ? { system: input.judge.system } : {}),
|
|
114
118
|
...(input.judge.model !== undefined ? { model: input.judge.model } : {}),
|
|
115
119
|
// The judge prompt's hash is this scorer's NAME, and `contentHash` covers `effort` and
|
package/src/vector.ts
CHANGED
|
@@ -8,6 +8,7 @@
|
|
|
8
8
|
// Reciprocal-rank fusion combines the two rankings without needing the two score scales to
|
|
9
9
|
// be comparable — which they never are.
|
|
10
10
|
|
|
11
|
+
import { finiteCount, finiteOption } from '@ultimat3/core';
|
|
11
12
|
import { cosine, tokenize } from './embeddings';
|
|
12
13
|
import { VectorDimMismatchError } from './errors';
|
|
13
14
|
import { NO_TENANT, narrowScope, scopeAdmits, UNSCOPED, type VectorScope } from './vector-scope';
|
|
@@ -69,6 +70,12 @@ export interface MemoryVectorStoreInput {
|
|
|
69
70
|
readonly records?: Map<string, StoredRecord>;
|
|
70
71
|
}
|
|
71
72
|
|
|
73
|
+
/** Named in every bound refusal below, so a caller knows which call it is about to edit. */
|
|
74
|
+
const SUBJECT = 'MemoryVectorStore';
|
|
75
|
+
|
|
76
|
+
/** The same subject for the hybrid read, whose options are the caller's per-call arguments. */
|
|
77
|
+
const HYBRID = 'hybrid search';
|
|
78
|
+
|
|
72
79
|
/** A record plus the tenant it was written under — the in-memory twin of the `tenant` column. */
|
|
73
80
|
export interface StoredRecord extends VectorRecord {
|
|
74
81
|
readonly tenant: string;
|
|
@@ -89,8 +96,12 @@ export class MemoryVectorStore implements VectorStore {
|
|
|
89
96
|
this.dimension = input.dimension;
|
|
90
97
|
this.scope = input.scope ?? UNSCOPED;
|
|
91
98
|
this.records = input.records ?? new Map<string, StoredRecord>();
|
|
92
|
-
|
|
93
|
-
|
|
99
|
+
// Screened, not clamped: a `NaN` here makes every BM25 score `NaN`, `hit.score > 0` reads
|
|
100
|
+
// false for every document, and `searchText` answers an empty list — zero work reported as a
|
|
101
|
+
// successful search. `finiteOption` and not `finiteCount`, because both are tuning constants
|
|
102
|
+
// and fractional by nature (1.2 and 0.75 are the values the BM25 paper settled on).
|
|
103
|
+
this.k1 = finiteOption(SUBJECT, 'k1', input.k1 ?? 1.2);
|
|
104
|
+
this.b = finiteOption(SUBJECT, 'b', input.b ?? 0.75);
|
|
94
105
|
}
|
|
95
106
|
|
|
96
107
|
/**
|
|
@@ -119,10 +130,12 @@ export class MemoryVectorStore implements VectorStore {
|
|
|
119
130
|
filter?: MetadataFilter,
|
|
120
131
|
): Promise<readonly SearchHit[]> {
|
|
121
132
|
this.assertDimension(vector.length);
|
|
133
|
+
// `k` carries no default, so `??` never sees it and nothing else does either: `slice(0, NaN)`
|
|
134
|
+
// is `[]`, which is a search that answers "no matches" for every query and reports success.
|
|
122
135
|
return this.candidates(filter)
|
|
123
136
|
.map((record) => this.hit(record, cosine(vector, record.vector)))
|
|
124
137
|
.sort(byScoreDesc)
|
|
125
|
-
.slice(0, k);
|
|
138
|
+
.slice(0, finiteCount(SUBJECT, 'k', k));
|
|
126
139
|
}
|
|
127
140
|
|
|
128
141
|
/** BM25 over the stored text. Real lexical scoring, so a rare exact term actually wins. */
|
|
@@ -131,6 +144,7 @@ export class MemoryVectorStore implements VectorStore {
|
|
|
131
144
|
k: number,
|
|
132
145
|
filter?: MetadataFilter,
|
|
133
146
|
): Promise<readonly SearchHit[]> {
|
|
147
|
+
const width = finiteCount(SUBJECT, 'k', k);
|
|
134
148
|
const candidates = this.candidates(filter);
|
|
135
149
|
if (candidates.length === 0) return [];
|
|
136
150
|
const docs = candidates.map((record) => ({ record, tokens: tokenize(record.text) }));
|
|
@@ -152,7 +166,7 @@ export class MemoryVectorStore implements VectorStore {
|
|
|
152
166
|
})
|
|
153
167
|
.filter((hit) => hit.score > 0)
|
|
154
168
|
.sort(byScoreDesc)
|
|
155
|
-
.slice(0,
|
|
169
|
+
.slice(0, width);
|
|
156
170
|
}
|
|
157
171
|
|
|
158
172
|
/**
|
|
@@ -161,13 +175,17 @@ export class MemoryVectorStore implements VectorStore {
|
|
|
161
175
|
* by exact term match beats one that merely leads a flat vector ranking.
|
|
162
176
|
*/
|
|
163
177
|
async hybrid(input: HybridSearchInput): Promise<readonly SearchHit[]> {
|
|
164
|
-
|
|
165
|
-
|
|
178
|
+
// Three bounds, none of which `Math.max` screens — it PROPAGATES a `NaN`. A `NaN` `k` collapses
|
|
179
|
+
// both candidate widths and the final slice to `[]`; a `NaN` `rrfK` makes every fused score
|
|
180
|
+
// `NaN`, and the ranking this method exists to produce becomes whatever order the sort left.
|
|
181
|
+
const k = finiteCount(HYBRID, 'k', input.k);
|
|
182
|
+
const width = finiteCount(HYBRID, 'candidates', input.candidates ?? Math.max(k * 4, 20));
|
|
183
|
+
const rrfK = finiteOption(HYBRID, 'rrfK', input.rrfK ?? 60);
|
|
166
184
|
const [dense, lexical] = await Promise.all([
|
|
167
185
|
this.search(input.vector, width, input.filter),
|
|
168
186
|
this.searchText(input.query, width, input.filter),
|
|
169
187
|
]);
|
|
170
|
-
return fuse([dense, lexical], rrfK).slice(0,
|
|
188
|
+
return fuse([dense, lexical], rrfK).slice(0, k);
|
|
171
189
|
}
|
|
172
190
|
|
|
173
191
|
/** Scoped, exactly like the SQL `delete ... where id in (...) and <scope>`. */
|
|
@@ -203,8 +221,17 @@ export class MemoryVectorStore implements VectorStore {
|
|
|
203
221
|
}
|
|
204
222
|
}
|
|
205
223
|
|
|
206
|
-
/**
|
|
207
|
-
|
|
224
|
+
/**
|
|
225
|
+
* Fuse ranked lists by reciprocal rank. Exported so a reranker can reuse it — which is why the
|
|
226
|
+
* damping is screened HERE too and not only in `hybrid`: a default parameter is a bound like any
|
|
227
|
+
* other, and `1 / (NaN + rank)` scores every document `NaN`, so the fused order is no longer a
|
|
228
|
+
* ranking while every caller still reads a full result list.
|
|
229
|
+
*/
|
|
230
|
+
export function fuse(
|
|
231
|
+
rankings: readonly (readonly SearchHit[])[],
|
|
232
|
+
rrfK: number = 60,
|
|
233
|
+
): readonly SearchHit[] {
|
|
234
|
+
finiteOption('fuse', 'rrfK', rrfK);
|
|
208
235
|
const scores = new Map<string, number>();
|
|
209
236
|
const hits = new Map<string, SearchHit>();
|
|
210
237
|
for (const ranking of rankings) {
|