metergraph 0.4.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +37 -0
- package/dist/batch-first.d.ts +112 -0
- package/dist/batch-first.js +207 -0
- package/dist/index.d.ts +10 -0
- package/dist/index.js +9 -0
- package/dist/provider-batch.d.ts +160 -0
- package/dist/provider-batch.js +240 -0
- package/dist/version.d.ts +1 -1
- package/dist/version.js +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -125,6 +125,43 @@ inference while iterating a wrapped `client.messages.batches.results()` result.
|
|
|
125
125
|
Consume results inside `route()` so the asynchronous batch retains its product
|
|
126
126
|
route. Job-management polls are deliberately not counted as model calls.
|
|
127
127
|
|
|
128
|
+
## Batch-first execution (opt-in)
|
|
129
|
+
|
|
130
|
+
`batchFirst()` is a separate, explicitly opt-in code path from `wrap()`/capture: submit one request through a provider's Batch API, wait up to a caller-chosen deadline, and fall back to exactly one direct call if the batch hasn't finished in time.
|
|
131
|
+
|
|
132
|
+
```ts
|
|
133
|
+
import { batchFirst } from "metergraph";
|
|
134
|
+
import OpenAI from "openai";
|
|
135
|
+
|
|
136
|
+
const client = new OpenAI(); // unwrapped — batchFirst() drives it directly, not through wrap()
|
|
137
|
+
|
|
138
|
+
const outcome = await batchFirst(
|
|
139
|
+
client,
|
|
140
|
+
"openai",
|
|
141
|
+
{ model: "gpt-5-mini", input: "Summarize this document." },
|
|
142
|
+
{
|
|
143
|
+
deadlineMs: 60_000,
|
|
144
|
+
acceptDuplicateProviderExecution: true, // required: a missed deadline can execute the request twice
|
|
145
|
+
onLateBatchSettled: (info) => {
|
|
146
|
+
// Fires later, asynchronously, only if a losing batch eventually
|
|
147
|
+
// settles. Never blocks batchFirst()'s own return, and never exposes
|
|
148
|
+
// the late result's content — only whether it contained a
|
|
149
|
+
// tool-call plan.
|
|
150
|
+
},
|
|
151
|
+
},
|
|
152
|
+
);
|
|
153
|
+
|
|
154
|
+
outcome.source; // "batch" | "direct"
|
|
155
|
+
outcome.result; // the provider response
|
|
156
|
+
outcome.metadata.batch_outcome; // "completed" | "failed" | "expired" | "pending_at_deadline"
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
`provider` is `"openai" | "anthropic" | "google"`, matching `wrap()`'s own explicit-provider option — never inferred from the client instance. A request with `stream: true` is rejected before any provider call. A request carrying `tools` is rejected unless `allowDuplicateToolCallPlans: true` is also set, acknowledging that the batch result and the direct fallback are independent provider executions that may each choose a different tool-call plan. `acceptDuplicateProviderExecution` must be exactly `true` — there is no default and no environment-variable override, and a missed deadline can execute (and bill) the same request twice. Neither the batch nor the direct path executes a tool call automatically; the caller receives the tool-call plan in `outcome.result` and is responsible for executing it, exactly as with a normal (non-batch-first) provider response.
|
|
160
|
+
|
|
161
|
+
`batchFirst()` is not integrated with `wrap()`'s capture/telemetry pipeline; its result and metadata are returned directly to the caller, never enqueued for delivery. The background poll that watches a losing batch for late telemetry uses an ordinary `setTimeout`, not an unref'd one — after a deadline-triggered direct fallback it keeps the Node process alive until the batch reaches a terminal state, which can take up to 24 hours; this is currently unmitigated, so a short-lived script or CLI process should not assume it exits promptly after a fallback.
|
|
162
|
+
|
|
163
|
+
OpenAI, Anthropic, and Google Gemini all have adapters (`createOpenAIBatchAdapter`, `createAnthropicBatchAdapter`, `createGoogleBatchAdapter`), built against each provider's documented Batch API shape and covered by fake-client contract tests, but **none has been exercised against a live provider Batch API from this SDK** — treat this as a beta-quality, code-reviewed-but-not-live-verified surface.
|
|
164
|
+
|
|
128
165
|
## Set up with an AI coding agent
|
|
129
166
|
|
|
130
167
|
Paste this into Claude Code, Codex, Cursor, or any coding agent inside the
|
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Explicit, opt-in batch-first execution: submit one request through a
|
|
3
|
+
* provider's Batch API, wait a caller-selected deadline, and fall back to
|
|
4
|
+
* a single direct call if the batch hasn't finished in time. Never enabled
|
|
5
|
+
* by wrap()/capture defaults or by an environment variable — a caller
|
|
6
|
+
* reaches this only by importing and calling batchFirst() directly, and
|
|
7
|
+
* only after explicitly acknowledging the duplicate-execution semantic
|
|
8
|
+
* below.
|
|
9
|
+
*
|
|
10
|
+
* On a missed deadline, the request may execute twice against the
|
|
11
|
+
* provider (once via batch, once via direct) — an accepted, deliberate
|
|
12
|
+
* semantic, never silently avoided. What this module guarantees instead:
|
|
13
|
+
* exactly one direct fallback is ever issued, and a batch result that
|
|
14
|
+
* arrives after the fallback already won is never returned, never
|
|
15
|
+
* executed, and never mutates an already-resolved result.
|
|
16
|
+
*/
|
|
17
|
+
import { type BatchFirstRequest, type ProviderBatchAdapter } from "./provider-batch.js";
|
|
18
|
+
export type { BatchFirstRequest } from "./provider-batch.js";
|
|
19
|
+
/** An injectable time source. Real timers by default; tests can supply a
|
|
20
|
+
* fake to drive the deadline deterministically, with no real waiting. */
|
|
21
|
+
export interface BatchFirstClock {
|
|
22
|
+
setTimeout(handler: () => void, ms: number): unknown;
|
|
23
|
+
clearTimeout(handle: unknown): void;
|
|
24
|
+
}
|
|
25
|
+
/** Reported asynchronously, after batchFirst() has already returned via the
|
|
26
|
+
* direct fallback, if the losing batch eventually reaches a terminal
|
|
27
|
+
* state. Its actual result content is never included here and never
|
|
28
|
+
* returned from batchFirst() — only whether it happened to contain a
|
|
29
|
+
* tool-call plan, which may differ from the one the direct fallback
|
|
30
|
+
* produced (see allowDuplicateToolCallPlans on BatchFirstPolicy). */
|
|
31
|
+
export interface LateBatchInfo {
|
|
32
|
+
outcome: "completed" | "failed" | "expired";
|
|
33
|
+
containedToolCallPlan: boolean;
|
|
34
|
+
}
|
|
35
|
+
export interface BatchFirstPolicy {
|
|
36
|
+
/** How long to wait for the batch before issuing the one direct
|
|
37
|
+
* fallback. Required — there is no default deadline. */
|
|
38
|
+
deadlineMs: number;
|
|
39
|
+
/**
|
|
40
|
+
* Required, explicit acknowledgement that a missed deadline can result
|
|
41
|
+
* in the same request executing twice against the provider. Must be
|
|
42
|
+
* exactly `true`; never read from an environment variable, and never
|
|
43
|
+
* defaulted.
|
|
44
|
+
*/
|
|
45
|
+
acceptDuplicateProviderExecution: true;
|
|
46
|
+
/**
|
|
47
|
+
* Required, explicit acknowledgement for a request that includes
|
|
48
|
+
* `tools`: the batch result and the direct fallback are two independent
|
|
49
|
+
* provider executions of the same prompt, and may each choose a
|
|
50
|
+
* DIFFERENT tool-call plan (different tool, different arguments). A
|
|
51
|
+
* caller whose tools have side effects must not assume the two plans
|
|
52
|
+
* agree. Without this flag, a request carrying `tools` is rejected
|
|
53
|
+
* before any provider call is made.
|
|
54
|
+
*/
|
|
55
|
+
allowDuplicateToolCallPlans?: boolean;
|
|
56
|
+
/** Fired once, later, only when the losing batch eventually settles.
|
|
57
|
+
* Never blocks batchFirst()'s own return. */
|
|
58
|
+
onLateBatchSettled?: (info: LateBatchInfo) => void;
|
|
59
|
+
}
|
|
60
|
+
/** Non-public knobs for driving runBatchFirst()'s poll interval and time
|
|
61
|
+
* source from tests — deliberately kept off BatchFirstPolicy so a real
|
|
62
|
+
* caller of batchFirst() has no way to set them. */
|
|
63
|
+
interface RunBatchFirstTestControls {
|
|
64
|
+
/** How often to re-check batch status while waiting. Default 2000ms. */
|
|
65
|
+
pollIntervalMs?: number;
|
|
66
|
+
/** Injectable time source — see BatchFirstClock. */
|
|
67
|
+
clock?: BatchFirstClock;
|
|
68
|
+
}
|
|
69
|
+
export type BatchFirstSource = "batch" | "direct";
|
|
70
|
+
export interface BatchFirstMetadata {
|
|
71
|
+
execution_mode: "batch_first";
|
|
72
|
+
deadline_ms: number;
|
|
73
|
+
/** Wall-clock time from submission to the canonical result settling. */
|
|
74
|
+
batch_wait_ms: number;
|
|
75
|
+
/** The batch's own status as of when batchFirst() returned — not its
|
|
76
|
+
* eventual status if that differs (see onLateBatchSettled). */
|
|
77
|
+
batch_outcome: "completed" | "failed" | "expired" | "pending_at_deadline";
|
|
78
|
+
canonical_result: BatchFirstSource;
|
|
79
|
+
duplicate_provider_execution: boolean;
|
|
80
|
+
/** Always false when canonical_result is "batch" (no lateness is
|
|
81
|
+
* possible — the batch IS the canonical result). When
|
|
82
|
+
* canonical_result is "direct", this reflects what was known AT THE
|
|
83
|
+
* MOMENT batchFirst() returned, which is always false: a completion
|
|
84
|
+
* confirmed later only reaches the caller through
|
|
85
|
+
* onLateBatchSettled, never by mutating this object. */
|
|
86
|
+
late_batch_completed: boolean;
|
|
87
|
+
late_batch_contained_tool_call_plan: boolean;
|
|
88
|
+
}
|
|
89
|
+
export interface BatchFirstResult<TResult = unknown> {
|
|
90
|
+
source: BatchFirstSource;
|
|
91
|
+
result: TResult;
|
|
92
|
+
metadata: BatchFirstMetadata;
|
|
93
|
+
}
|
|
94
|
+
/** Thrown before any provider call when a request/policy combination is
|
|
95
|
+
* not eligible for batch-first execution — streaming, tools without
|
|
96
|
+
* acknowledgement, a missing/false acceptDuplicateProviderExecution, or
|
|
97
|
+
* an adapter-specific ineligibility. */
|
|
98
|
+
export declare class BatchFirstIneligibleError extends Error {
|
|
99
|
+
}
|
|
100
|
+
/**
|
|
101
|
+
* The adapter-injected core state machine — exported so fake-adapter,
|
|
102
|
+
* fake-clock tests can drive it directly, and used internally by the
|
|
103
|
+
* public, provider-explicit batchFirst() below.
|
|
104
|
+
*/
|
|
105
|
+
export declare function runBatchFirst<TResult>(adapter: ProviderBatchAdapter<TResult>, request: BatchFirstRequest, policy: BatchFirstPolicy, testControls?: RunBatchFirstTestControls): Promise<BatchFirstResult<TResult>>;
|
|
106
|
+
export type BatchFirstProvider = "openai" | "anthropic" | "google";
|
|
107
|
+
/**
|
|
108
|
+
* Explicit, provider-specific batch-first execution. Never inferred from
|
|
109
|
+
* the client instance — the caller states the provider, matching wrap()'s
|
|
110
|
+
* own explicit-provider option.
|
|
111
|
+
*/
|
|
112
|
+
export declare function batchFirst<TResult = unknown>(client: unknown, provider: BatchFirstProvider, request: BatchFirstRequest, policy: BatchFirstPolicy): Promise<BatchFirstResult<TResult>>;
|
|
@@ -0,0 +1,207 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Explicit, opt-in batch-first execution: submit one request through a
|
|
3
|
+
* provider's Batch API, wait a caller-selected deadline, and fall back to
|
|
4
|
+
* a single direct call if the batch hasn't finished in time. Never enabled
|
|
5
|
+
* by wrap()/capture defaults or by an environment variable — a caller
|
|
6
|
+
* reaches this only by importing and calling batchFirst() directly, and
|
|
7
|
+
* only after explicitly acknowledging the duplicate-execution semantic
|
|
8
|
+
* below.
|
|
9
|
+
*
|
|
10
|
+
* On a missed deadline, the request may execute twice against the
|
|
11
|
+
* provider (once via batch, once via direct) — an accepted, deliberate
|
|
12
|
+
* semantic, never silently avoided. What this module guarantees instead:
|
|
13
|
+
* exactly one direct fallback is ever issued, and a batch result that
|
|
14
|
+
* arrives after the fallback already won is never returned, never
|
|
15
|
+
* executed, and never mutates an already-resolved result.
|
|
16
|
+
*/
|
|
17
|
+
import { createAnthropicBatchAdapter, createGoogleBatchAdapter, createOpenAIBatchAdapter, } from "./provider-batch.js";
|
|
18
|
+
function realClock() {
|
|
19
|
+
return {
|
|
20
|
+
setTimeout: (handler, ms) => setTimeout(handler, ms),
|
|
21
|
+
clearTimeout: (handle) => clearTimeout(handle),
|
|
22
|
+
};
|
|
23
|
+
}
|
|
24
|
+
/** Thrown before any provider call when a request/policy combination is
|
|
25
|
+
* not eligible for batch-first execution — streaming, tools without
|
|
26
|
+
* acknowledgement, a missing/false acceptDuplicateProviderExecution, or
|
|
27
|
+
* an adapter-specific ineligibility. */
|
|
28
|
+
export class BatchFirstIneligibleError extends Error {
|
|
29
|
+
}
|
|
30
|
+
function hasTools(request) {
|
|
31
|
+
return Array.isArray(request.tools) && request.tools.length > 0;
|
|
32
|
+
}
|
|
33
|
+
function validate(request, policy) {
|
|
34
|
+
if (policy.acceptDuplicateProviderExecution !== true) {
|
|
35
|
+
throw new BatchFirstIneligibleError("batchFirst() requires policy.acceptDuplicateProviderExecution: true — "
|
|
36
|
+
+ "a missed deadline can execute the request twice against the provider");
|
|
37
|
+
}
|
|
38
|
+
if (!Number.isFinite(policy.deadlineMs) || policy.deadlineMs <= 0) {
|
|
39
|
+
throw new BatchFirstIneligibleError("batchFirst() requires a positive policy.deadlineMs");
|
|
40
|
+
}
|
|
41
|
+
if (request.stream === true) {
|
|
42
|
+
throw new BatchFirstIneligibleError("batchFirst() does not support streaming requests — streaming is direct-only");
|
|
43
|
+
}
|
|
44
|
+
if (hasTools(request) && policy.allowDuplicateToolCallPlans !== true) {
|
|
45
|
+
throw new BatchFirstIneligibleError("batchFirst() requires policy.allowDuplicateToolCallPlans: true for requests with "
|
|
46
|
+
+ "tools — the batch result and the direct fallback are independent provider "
|
|
47
|
+
+ "executions and may each choose a different tool call plan");
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
const DEFAULT_POLL_INTERVAL_MS = 2_000;
|
|
51
|
+
/**
|
|
52
|
+
* The adapter-injected core state machine — exported so fake-adapter,
|
|
53
|
+
* fake-clock tests can drive it directly, and used internally by the
|
|
54
|
+
* public, provider-explicit batchFirst() below.
|
|
55
|
+
*/
|
|
56
|
+
export async function runBatchFirst(adapter, request, policy, testControls) {
|
|
57
|
+
validate(request, policy);
|
|
58
|
+
const eligibility = adapter.eligibility(request);
|
|
59
|
+
if (!eligibility.eligible) {
|
|
60
|
+
throw new BatchFirstIneligibleError(`request is not eligible for batch-first execution: ${eligibility.reason ?? "unsupported by this adapter"}`);
|
|
61
|
+
}
|
|
62
|
+
const clock = testControls?.clock ?? realClock();
|
|
63
|
+
const pollIntervalMs = testControls?.pollIntervalMs ?? DEFAULT_POLL_INTERVAL_MS;
|
|
64
|
+
const startedAt = Date.now();
|
|
65
|
+
const handle = await adapter.submitOne(request);
|
|
66
|
+
let settleDeadline;
|
|
67
|
+
const deadlineSettled = new Promise((resolve) => {
|
|
68
|
+
settleDeadline = resolve;
|
|
69
|
+
});
|
|
70
|
+
const deadlineTimer = clock.setTimeout(() => settleDeadline?.(), policy.deadlineMs);
|
|
71
|
+
// Resolves at most once, and only from inside the poll loop below, the
|
|
72
|
+
// moment a real terminal status is observed — never resolves at all if
|
|
73
|
+
// the deadline wins first. Racing this directly (rather than racing the
|
|
74
|
+
// loop's own completion promise) means there is no "loop stopped without
|
|
75
|
+
// a terminal status" case to fake a result for: that case simply never
|
|
76
|
+
// resolves this promise, which is exactly the semantics Promise.race
|
|
77
|
+
// needs.
|
|
78
|
+
let resolveBatchTerminal;
|
|
79
|
+
const batchTerminalSettled = new Promise((resolve) => {
|
|
80
|
+
resolveBatchTerminal = resolve;
|
|
81
|
+
});
|
|
82
|
+
let stopPolling = false;
|
|
83
|
+
const pollUntilTerminal = (async () => {
|
|
84
|
+
while (!stopPolling) {
|
|
85
|
+
const outcome = await adapter.poll(handle);
|
|
86
|
+
if (outcome.status !== "pending") {
|
|
87
|
+
resolveBatchTerminal?.(outcome.status);
|
|
88
|
+
return outcome.status;
|
|
89
|
+
}
|
|
90
|
+
if (stopPolling)
|
|
91
|
+
return undefined;
|
|
92
|
+
await new Promise((resolve) => clock.setTimeout(resolve, pollIntervalMs));
|
|
93
|
+
}
|
|
94
|
+
return undefined;
|
|
95
|
+
})();
|
|
96
|
+
const winner = await Promise.race([
|
|
97
|
+
batchTerminalSettled.then((status) => ({ kind: "batch", status })),
|
|
98
|
+
deadlineSettled.then(() => ({ kind: "deadline" })),
|
|
99
|
+
]);
|
|
100
|
+
clock.clearTimeout(deadlineTimer);
|
|
101
|
+
// A batch reported "completed" but whose result cannot be read (a
|
|
102
|
+
// missing output file, an item-level provider error, a malformed or
|
|
103
|
+
// missing matching line, a transient read failure) is neither a valid
|
|
104
|
+
// canonical batch result nor grounds to throw out of batchFirst()
|
|
105
|
+
// instead of the promised fallback — it is treated exactly like a batch
|
|
106
|
+
// that reported "failed": exactly one direct fallback, never a second
|
|
107
|
+
// read attempt, never surfaced as a rejection.
|
|
108
|
+
let unreadableCompletedBatch = false;
|
|
109
|
+
if (winner.kind === "batch" && winner.status === "completed") {
|
|
110
|
+
stopPolling = true;
|
|
111
|
+
try {
|
|
112
|
+
const { result } = await adapter.readResult(handle);
|
|
113
|
+
return {
|
|
114
|
+
source: "batch",
|
|
115
|
+
result,
|
|
116
|
+
metadata: {
|
|
117
|
+
execution_mode: "batch_first",
|
|
118
|
+
deadline_ms: policy.deadlineMs,
|
|
119
|
+
batch_wait_ms: Date.now() - startedAt,
|
|
120
|
+
batch_outcome: "completed",
|
|
121
|
+
canonical_result: "batch",
|
|
122
|
+
duplicate_provider_execution: false,
|
|
123
|
+
late_batch_completed: false,
|
|
124
|
+
late_batch_contained_tool_call_plan: false,
|
|
125
|
+
},
|
|
126
|
+
};
|
|
127
|
+
}
|
|
128
|
+
catch {
|
|
129
|
+
unreadableCompletedBatch = true;
|
|
130
|
+
}
|
|
131
|
+
}
|
|
132
|
+
// Either the deadline fired first, the batch reached a non-completed
|
|
133
|
+
// terminal status before the deadline, or the batch completed but its
|
|
134
|
+
// result could not be read — either way, issue exactly one direct
|
|
135
|
+
// fallback now, and never wait further (or retry a read) on the batch
|
|
136
|
+
// for the canonical result. Only stop the poll loop when the batch
|
|
137
|
+
// itself already produced a terminal status (it has nothing left to do,
|
|
138
|
+
// so this is a no-op) — when the DEADLINE won, deliberately leave
|
|
139
|
+
// polling running in the background: "keep polling only to write
|
|
140
|
+
// terminal telemetry" requires the loop to keep going, not stop here.
|
|
141
|
+
const batchOutcomeAtFallback = unreadableCompletedBatch
|
|
142
|
+
? "failed"
|
|
143
|
+
: winner.kind === "batch"
|
|
144
|
+
? winner.status
|
|
145
|
+
: "pending_at_deadline";
|
|
146
|
+
if (winner.kind === "batch")
|
|
147
|
+
stopPolling = true;
|
|
148
|
+
const directResultPromise = adapter.direct(request);
|
|
149
|
+
if (winner.kind === "deadline") {
|
|
150
|
+
// Observe the batch purely for telemetry — never read for its content
|
|
151
|
+
// to be returned, executed, or used to mutate the already-in-flight
|
|
152
|
+
// direct result.
|
|
153
|
+
pollUntilTerminal
|
|
154
|
+
.then(async (status) => {
|
|
155
|
+
if (status !== "completed" && status !== "failed" && status !== "expired")
|
|
156
|
+
return;
|
|
157
|
+
let containedToolCallPlan = false;
|
|
158
|
+
if (status === "completed") {
|
|
159
|
+
try {
|
|
160
|
+
const late = await adapter.readResult(handle);
|
|
161
|
+
containedToolCallPlan = late.containedToolCallPlan;
|
|
162
|
+
}
|
|
163
|
+
catch {
|
|
164
|
+
/* telemetry only — never thrown, never surfaced */
|
|
165
|
+
}
|
|
166
|
+
}
|
|
167
|
+
policy.onLateBatchSettled?.({ outcome: status, containedToolCallPlan });
|
|
168
|
+
})
|
|
169
|
+
.catch(() => { });
|
|
170
|
+
}
|
|
171
|
+
const { result } = await directResultPromise;
|
|
172
|
+
return {
|
|
173
|
+
source: "direct",
|
|
174
|
+
result,
|
|
175
|
+
metadata: {
|
|
176
|
+
execution_mode: "batch_first",
|
|
177
|
+
deadline_ms: policy.deadlineMs,
|
|
178
|
+
batch_wait_ms: Date.now() - startedAt,
|
|
179
|
+
batch_outcome: batchOutcomeAtFallback,
|
|
180
|
+
canonical_result: "direct",
|
|
181
|
+
duplicate_provider_execution: true,
|
|
182
|
+
late_batch_completed: false,
|
|
183
|
+
late_batch_contained_tool_call_plan: false,
|
|
184
|
+
},
|
|
185
|
+
};
|
|
186
|
+
}
|
|
187
|
+
function resolveAdapter(client, provider) {
|
|
188
|
+
if (provider === "openai") {
|
|
189
|
+
return createOpenAIBatchAdapter(client);
|
|
190
|
+
}
|
|
191
|
+
if (provider === "anthropic") {
|
|
192
|
+
return createAnthropicBatchAdapter(client);
|
|
193
|
+
}
|
|
194
|
+
if (provider === "google") {
|
|
195
|
+
return createGoogleBatchAdapter(client);
|
|
196
|
+
}
|
|
197
|
+
throw new BatchFirstIneligibleError(`batchFirst() has no adapter for provider "${provider}" yet`);
|
|
198
|
+
}
|
|
199
|
+
/**
|
|
200
|
+
* Explicit, provider-specific batch-first execution. Never inferred from
|
|
201
|
+
* the client instance — the caller states the provider, matching wrap()'s
|
|
202
|
+
* own explicit-provider option.
|
|
203
|
+
*/
|
|
204
|
+
export async function batchFirst(client, provider, request, policy) {
|
|
205
|
+
const adapter = resolveAdapter(client, provider);
|
|
206
|
+
return runBatchFirst(adapter, request, policy);
|
|
207
|
+
}
|
package/dist/index.d.ts
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { route, setSession, setTags, trace, type RouteOptions, type TraceOptions } from "./context.js";
|
|
2
|
+
import { BatchFirstIneligibleError, batchFirst, type BatchFirstMetadata, type BatchFirstPolicy, type BatchFirstProvider, type BatchFirstRequest, type BatchFirstResult, type BatchFirstSource, type LateBatchInfo } from "./batch-first.js";
|
|
2
3
|
import { track } from "./track.js";
|
|
3
4
|
import { type TransportMode, type WaitUntil } from "./transport.js";
|
|
4
5
|
import { type VercelAISDKMiddleware, type VercelAISDKMiddlewareOptions, type VercelAISDKMiddlewareSpecificationOptions, type VercelAISDKMiddlewareVersionOptions, type VercelAISDKSpecificationVersion } from "./vercel-ai.js";
|
|
@@ -60,3 +61,12 @@ export declare function vercelAISDKMiddleware(options: VercelAISDKMiddlewareVers
|
|
|
60
61
|
export declare function vercelAISDKMiddleware<TVersion extends VercelAISDKSpecificationVersion = "v3">(options?: VercelAISDKMiddlewareSpecificationOptions<TVersion>): VercelAISDKMiddleware<TVersion>;
|
|
61
62
|
export { route, setSession, setTags, trace, track };
|
|
62
63
|
export type { RouteOptions, TraceOptions, TransportMode, VercelAISDKMiddleware, VercelAISDKMiddlewareOptions, VercelAISDKSpecificationVersion, };
|
|
64
|
+
/**
|
|
65
|
+
* Explicit, opt-in batch-first execution over a provider's Batch API —
|
|
66
|
+
* never enabled by wrap()/capture defaults or by an environment variable.
|
|
67
|
+
* See BatchFirstPolicy for the required, explicit acknowledgements
|
|
68
|
+
* (acceptDuplicateProviderExecution, and allowDuplicateToolCallPlans for
|
|
69
|
+
* requests that include tools).
|
|
70
|
+
*/
|
|
71
|
+
export { batchFirst, BatchFirstIneligibleError };
|
|
72
|
+
export type { BatchFirstMetadata, BatchFirstPolicy, BatchFirstProvider, BatchFirstRequest, BatchFirstResult, BatchFirstSource, LateBatchInfo, };
|
package/dist/index.js
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
import { CaptureRuntime } from "./capture.js";
|
|
2
2
|
import { ConfigPoller } from "./config.js";
|
|
3
3
|
import { contextSnapshot, route, setSession, setTags, trace, } from "./context.js";
|
|
4
|
+
import { BatchFirstIneligibleError, batchFirst, } from "./batch-first.js";
|
|
4
5
|
import { ensureRepoConfig } from "./repo-config.js";
|
|
5
6
|
import { SessionManager } from "./session.js";
|
|
6
7
|
import { track } from "./track.js";
|
|
@@ -158,3 +159,11 @@ export function vercelAISDKMiddleware(options = {}) {
|
|
|
158
159
|
return createVercelAISDKMiddleware(options);
|
|
159
160
|
}
|
|
160
161
|
export { route, setSession, setTags, trace, track };
|
|
162
|
+
/**
|
|
163
|
+
* Explicit, opt-in batch-first execution over a provider's Batch API —
|
|
164
|
+
* never enabled by wrap()/capture defaults or by an environment variable.
|
|
165
|
+
* See BatchFirstPolicy for the required, explicit acknowledgements
|
|
166
|
+
* (acceptDuplicateProviderExecution, and allowDuplicateToolCallPlans for
|
|
167
|
+
* requests that include tools).
|
|
168
|
+
*/
|
|
169
|
+
export { batchFirst, BatchFirstIneligibleError };
|
|
@@ -0,0 +1,160 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The narrow adapter boundary batchFirst() drives, plus a worker-only OpenAI
|
|
3
|
+
* Batch API implementation. Deliberately duck-typed against the openai
|
|
4
|
+
* package's shape (see OpenAIBatchCapableClient) rather than importing it —
|
|
5
|
+
* `openai` is an optional peer dependency of this package, and structural
|
|
6
|
+
* typing already lets a real `OpenAI` client instance satisfy this
|
|
7
|
+
* interface without any import, type-only or otherwise.
|
|
8
|
+
*
|
|
9
|
+
* Every method here returns only bounded, structural information — a
|
|
10
|
+
* status, a result the caller explicitly asked for, and a boolean noting
|
|
11
|
+
* whether that result happened to contain a tool-call plan. No adapter
|
|
12
|
+
* method inspects or logs a provider error body, a credential, or a raw
|
|
13
|
+
* prompt.
|
|
14
|
+
*/
|
|
15
|
+
/** A caller's own provider request body (e.g. an OpenAI Responses payload).
|
|
16
|
+
* Only `stream` and `tools` are inspected by this module; every other
|
|
17
|
+
* field passes through untouched. */
|
|
18
|
+
export type BatchFirstRequest = Record<string, unknown> & {
|
|
19
|
+
stream?: boolean;
|
|
20
|
+
tools?: unknown[];
|
|
21
|
+
};
|
|
22
|
+
export interface BatchHandle {
|
|
23
|
+
/** The provider's own batch identifier — never returned to a caller of
|
|
24
|
+
* batchFirst() or logged; adapters use it only to poll/read their own
|
|
25
|
+
* batch. */
|
|
26
|
+
providerBatchId: string;
|
|
27
|
+
}
|
|
28
|
+
export type BatchPollStatus = "pending" | "completed" | "failed" | "expired";
|
|
29
|
+
export interface BatchPollResult {
|
|
30
|
+
status: BatchPollStatus;
|
|
31
|
+
}
|
|
32
|
+
export interface ProviderBatchResult<TResult = unknown> {
|
|
33
|
+
result: TResult;
|
|
34
|
+
/** Whether this result's own output happens to include a tool/function
|
|
35
|
+
* call — reported as a boolean only, never by exposing the call's
|
|
36
|
+
* arguments or the surrounding response through this flag. */
|
|
37
|
+
containedToolCallPlan: boolean;
|
|
38
|
+
}
|
|
39
|
+
export interface ProviderBatchEligibility {
|
|
40
|
+
eligible: boolean;
|
|
41
|
+
reason?: string;
|
|
42
|
+
}
|
|
43
|
+
/**
|
|
44
|
+
* The seam batchFirst() is written against. An adapter submits exactly one
|
|
45
|
+
* request per batch and knows how to poll it, read its terminal result,
|
|
46
|
+
* and run the same request directly (bypassing batch entirely) as a
|
|
47
|
+
* fallback.
|
|
48
|
+
*/
|
|
49
|
+
export interface ProviderBatchAdapter<TResult = unknown> {
|
|
50
|
+
eligibility(request: BatchFirstRequest): ProviderBatchEligibility;
|
|
51
|
+
submitOne(request: BatchFirstRequest): Promise<BatchHandle>;
|
|
52
|
+
poll(handle: BatchHandle): Promise<BatchPollResult>;
|
|
53
|
+
readResult(handle: BatchHandle): Promise<ProviderBatchResult<TResult>>;
|
|
54
|
+
direct(request: BatchFirstRequest): Promise<ProviderBatchResult<TResult>>;
|
|
55
|
+
}
|
|
56
|
+
/** The minimal slice of the `openai` package's client shape this adapter
|
|
57
|
+
* calls. A real `OpenAI` client instance satisfies this structurally. */
|
|
58
|
+
export interface OpenAIBatchCapableClient {
|
|
59
|
+
files: {
|
|
60
|
+
create(params: {
|
|
61
|
+
file: Blob;
|
|
62
|
+
purpose: string;
|
|
63
|
+
}): Promise<{
|
|
64
|
+
id: string;
|
|
65
|
+
}>;
|
|
66
|
+
content(fileId: string): Promise<{
|
|
67
|
+
text(): Promise<string>;
|
|
68
|
+
}>;
|
|
69
|
+
};
|
|
70
|
+
batches: {
|
|
71
|
+
create(params: {
|
|
72
|
+
input_file_id: string;
|
|
73
|
+
endpoint: string;
|
|
74
|
+
completion_window: string;
|
|
75
|
+
}): Promise<{
|
|
76
|
+
id: string;
|
|
77
|
+
status?: string;
|
|
78
|
+
}>;
|
|
79
|
+
retrieve(batchId: string): Promise<{
|
|
80
|
+
id: string;
|
|
81
|
+
status?: string;
|
|
82
|
+
output_file_id?: string | null;
|
|
83
|
+
}>;
|
|
84
|
+
};
|
|
85
|
+
responses: {
|
|
86
|
+
create(request: Record<string, unknown>): Promise<Record<string, unknown>>;
|
|
87
|
+
};
|
|
88
|
+
}
|
|
89
|
+
export declare class ProviderBatchError extends Error {
|
|
90
|
+
}
|
|
91
|
+
export declare function createOpenAIBatchAdapter(client: OpenAIBatchCapableClient): ProviderBatchAdapter;
|
|
92
|
+
/** The minimal slice of the `@anthropic-ai/sdk` package's client shape this
|
|
93
|
+
* adapter calls. A real `Anthropic` client instance satisfies this
|
|
94
|
+
* structurally — see `patchAnthropicBatchResults` in wrap.ts, which
|
|
95
|
+
* confirms `messages.batches.results()` returns an async iterable of
|
|
96
|
+
* `{custom_id, result: {type, message}}` items against the real SDK. */
|
|
97
|
+
export interface AnthropicBatchCapableClient {
|
|
98
|
+
messages: {
|
|
99
|
+
create(request: Record<string, unknown>): Promise<Record<string, unknown>>;
|
|
100
|
+
batches: {
|
|
101
|
+
create(params: {
|
|
102
|
+
requests: Array<{
|
|
103
|
+
custom_id: string;
|
|
104
|
+
params: Record<string, unknown>;
|
|
105
|
+
}>;
|
|
106
|
+
}): Promise<{
|
|
107
|
+
id: string;
|
|
108
|
+
processing_status?: string;
|
|
109
|
+
}>;
|
|
110
|
+
retrieve(batchId: string): Promise<{
|
|
111
|
+
id: string;
|
|
112
|
+
processing_status?: string;
|
|
113
|
+
request_counts?: Record<string, number>;
|
|
114
|
+
}>;
|
|
115
|
+
results(batchId: string): AsyncIterable<unknown>;
|
|
116
|
+
};
|
|
117
|
+
};
|
|
118
|
+
}
|
|
119
|
+
export declare function createAnthropicBatchAdapter(client: AnthropicBatchCapableClient): ProviderBatchAdapter;
|
|
120
|
+
/** The minimal slice of the `@google/genai` package's client shape this
|
|
121
|
+
* adapter calls. A real Google GenAI client instance satisfies this
|
|
122
|
+
* structurally. The exact batch job/response wire shape below (`state` vs
|
|
123
|
+
* `metadata.state`, `dest.inlinedResponses`) is our best-effort reading of
|
|
124
|
+
* the current Gemini Batch API and needs live/gated verification before
|
|
125
|
+
* production use — see this module's caller-facing report for specifics. */
|
|
126
|
+
export interface GoogleBatchCapableClient {
|
|
127
|
+
models: {
|
|
128
|
+
generateContent(request: Record<string, unknown>): Promise<Record<string, unknown>>;
|
|
129
|
+
};
|
|
130
|
+
batches: {
|
|
131
|
+
create(params: {
|
|
132
|
+
model: string;
|
|
133
|
+
src: {
|
|
134
|
+
inlinedRequests: Array<Record<string, unknown>>;
|
|
135
|
+
};
|
|
136
|
+
}): Promise<{
|
|
137
|
+
name: string;
|
|
138
|
+
state?: string;
|
|
139
|
+
metadata?: {
|
|
140
|
+
state?: string;
|
|
141
|
+
};
|
|
142
|
+
}>;
|
|
143
|
+
get(params: {
|
|
144
|
+
name: string;
|
|
145
|
+
}): Promise<{
|
|
146
|
+
name: string;
|
|
147
|
+
state?: string;
|
|
148
|
+
metadata?: {
|
|
149
|
+
state?: string;
|
|
150
|
+
};
|
|
151
|
+
dest?: {
|
|
152
|
+
inlinedResponses?: Array<{
|
|
153
|
+
response?: Record<string, unknown>;
|
|
154
|
+
error?: Record<string, unknown>;
|
|
155
|
+
}>;
|
|
156
|
+
};
|
|
157
|
+
}>;
|
|
158
|
+
};
|
|
159
|
+
}
|
|
160
|
+
export declare function createGoogleBatchAdapter(client: GoogleBatchCapableClient): ProviderBatchAdapter;
|
|
@@ -0,0 +1,240 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The narrow adapter boundary batchFirst() drives, plus a worker-only OpenAI
|
|
3
|
+
* Batch API implementation. Deliberately duck-typed against the openai
|
|
4
|
+
* package's shape (see OpenAIBatchCapableClient) rather than importing it —
|
|
5
|
+
* `openai` is an optional peer dependency of this package, and structural
|
|
6
|
+
* typing already lets a real `OpenAI` client instance satisfy this
|
|
7
|
+
* interface without any import, type-only or otherwise.
|
|
8
|
+
*
|
|
9
|
+
* Every method here returns only bounded, structural information — a
|
|
10
|
+
* status, a result the caller explicitly asked for, and a boolean noting
|
|
11
|
+
* whether that result happened to contain a tool-call plan. No adapter
|
|
12
|
+
* method inspects or logs a provider error body, a credential, or a raw
|
|
13
|
+
* prompt.
|
|
14
|
+
*/
|
|
15
|
+
const OPENAI_BATCH_ENDPOINT = "/v1/responses";
|
|
16
|
+
const OPENAI_COMPLETION_WINDOW = "24h";
|
|
17
|
+
// "validating" | "in_progress" | "finalizing" | "cancelling" all fall
|
|
18
|
+
// through to "pending" below, alongside any future status OpenAI adds —
|
|
19
|
+
// only a recognized terminal status is ever treated as terminal.
|
|
20
|
+
const OPENAI_FAILED_STATUSES = new Set(["failed", "cancelled"]);
|
|
21
|
+
function get(value, key) {
|
|
22
|
+
return value && typeof value === "object" ? value[key] : undefined;
|
|
23
|
+
}
|
|
24
|
+
function hasFunctionCall(body) {
|
|
25
|
+
const output = get(body, "output");
|
|
26
|
+
if (!Array.isArray(output))
|
|
27
|
+
return false;
|
|
28
|
+
return output.some((item) => get(item, "type") === "function_call");
|
|
29
|
+
}
|
|
30
|
+
function randomCustomId() {
|
|
31
|
+
if (typeof globalThis.crypto?.randomUUID === "function")
|
|
32
|
+
return globalThis.crypto.randomUUID();
|
|
33
|
+
return `batch-first-${Date.now().toString(36)}-${Math.random().toString(36).slice(2)}`;
|
|
34
|
+
}
|
|
35
|
+
export class ProviderBatchError extends Error {
|
|
36
|
+
}
|
|
37
|
+
export function createOpenAIBatchAdapter(client) {
|
|
38
|
+
return {
|
|
39
|
+
eligibility(request) {
|
|
40
|
+
if (request.stream === true) {
|
|
41
|
+
return { eligible: false, reason: "streaming requests are direct-only" };
|
|
42
|
+
}
|
|
43
|
+
return { eligible: true };
|
|
44
|
+
},
|
|
45
|
+
async submitOne(request) {
|
|
46
|
+
const customId = randomCustomId();
|
|
47
|
+
const line = JSON.stringify({
|
|
48
|
+
custom_id: customId,
|
|
49
|
+
method: "POST",
|
|
50
|
+
url: OPENAI_BATCH_ENDPOINT,
|
|
51
|
+
body: request,
|
|
52
|
+
});
|
|
53
|
+
const file = new Blob([`${line}\n`], { type: "application/jsonl" });
|
|
54
|
+
const uploaded = await client.files.create({ file, purpose: "batch" });
|
|
55
|
+
const batch = await client.batches.create({
|
|
56
|
+
input_file_id: uploaded.id,
|
|
57
|
+
endpoint: OPENAI_BATCH_ENDPOINT,
|
|
58
|
+
completion_window: OPENAI_COMPLETION_WINDOW,
|
|
59
|
+
});
|
|
60
|
+
const handle = { providerBatchId: batch.id, customId };
|
|
61
|
+
return handle;
|
|
62
|
+
},
|
|
63
|
+
async poll(handle) {
|
|
64
|
+
const batch = await client.batches.retrieve(handle.providerBatchId);
|
|
65
|
+
const status = batch.status ?? "";
|
|
66
|
+
if (status === "completed")
|
|
67
|
+
return { status: "completed" };
|
|
68
|
+
if (status === "expired")
|
|
69
|
+
return { status: "expired" };
|
|
70
|
+
if (OPENAI_FAILED_STATUSES.has(status))
|
|
71
|
+
return { status: "failed" };
|
|
72
|
+
return { status: "pending" };
|
|
73
|
+
},
|
|
74
|
+
async readResult(handle) {
|
|
75
|
+
const openaiHandle = handle;
|
|
76
|
+
const batch = await client.batches.retrieve(handle.providerBatchId);
|
|
77
|
+
const outputFileId = batch.output_file_id;
|
|
78
|
+
if (!outputFileId) {
|
|
79
|
+
throw new ProviderBatchError("openai batch has no output file to read");
|
|
80
|
+
}
|
|
81
|
+
const file = await client.files.content(outputFileId);
|
|
82
|
+
const text = await file.text();
|
|
83
|
+
for (const rawLine of text.split(/\r?\n/)) {
|
|
84
|
+
const line = rawLine.trim();
|
|
85
|
+
if (!line)
|
|
86
|
+
continue;
|
|
87
|
+
let item;
|
|
88
|
+
try {
|
|
89
|
+
item = JSON.parse(line);
|
|
90
|
+
}
|
|
91
|
+
catch {
|
|
92
|
+
continue;
|
|
93
|
+
}
|
|
94
|
+
if (get(item, "custom_id") !== openaiHandle.customId)
|
|
95
|
+
continue;
|
|
96
|
+
const response = get(item, "response");
|
|
97
|
+
const statusCode = Number(get(response, "status_code"));
|
|
98
|
+
const body = get(response, "body");
|
|
99
|
+
const error = get(item, "error");
|
|
100
|
+
if (error != null || (Number.isFinite(statusCode) && statusCode >= 400)) {
|
|
101
|
+
// Never surface the provider's own error text — only that this
|
|
102
|
+
// one item failed.
|
|
103
|
+
throw new ProviderBatchError("openai batch item returned an error response");
|
|
104
|
+
}
|
|
105
|
+
return { result: body, containedToolCallPlan: hasFunctionCall(body) };
|
|
106
|
+
}
|
|
107
|
+
throw new ProviderBatchError("openai batch output did not contain our submitted item");
|
|
108
|
+
},
|
|
109
|
+
async direct(request) {
|
|
110
|
+
const response = await client.responses.create(request);
|
|
111
|
+
return { result: response, containedToolCallPlan: hasFunctionCall(response) };
|
|
112
|
+
},
|
|
113
|
+
};
|
|
114
|
+
}
|
|
115
|
+
function hasToolUse(message) {
|
|
116
|
+
const content = get(message, "content");
|
|
117
|
+
if (!Array.isArray(content))
|
|
118
|
+
return false;
|
|
119
|
+
return content.some((block) => get(block, "type") === "tool_use");
|
|
120
|
+
}
|
|
121
|
+
export function createAnthropicBatchAdapter(client) {
|
|
122
|
+
return {
|
|
123
|
+
eligibility(request) {
|
|
124
|
+
if (request.stream === true) {
|
|
125
|
+
return { eligible: false, reason: "streaming requests are direct-only" };
|
|
126
|
+
}
|
|
127
|
+
return { eligible: true };
|
|
128
|
+
},
|
|
129
|
+
async submitOne(request) {
|
|
130
|
+
const customId = randomCustomId();
|
|
131
|
+
const batch = await client.messages.batches.create({
|
|
132
|
+
requests: [{ custom_id: customId, params: request }],
|
|
133
|
+
});
|
|
134
|
+
const handle = { providerBatchId: batch.id, customId };
|
|
135
|
+
return handle;
|
|
136
|
+
},
|
|
137
|
+
async poll(handle) {
|
|
138
|
+
const batch = await client.messages.batches.retrieve(handle.providerBatchId);
|
|
139
|
+
if (batch.processing_status !== "ended")
|
|
140
|
+
return { status: "pending" };
|
|
141
|
+
// "ended" is itself a terminal signal — there is no un-ending it, so
|
|
142
|
+
// any bucket shape we don't recognize is bounded to "failed" rather
|
|
143
|
+
// than left pending forever.
|
|
144
|
+
const counts = batch.request_counts ?? {};
|
|
145
|
+
if ((counts.succeeded ?? 0) > 0)
|
|
146
|
+
return { status: "completed" };
|
|
147
|
+
if ((counts.expired ?? 0) > 0)
|
|
148
|
+
return { status: "expired" };
|
|
149
|
+
return { status: "failed" };
|
|
150
|
+
},
|
|
151
|
+
async readResult(handle) {
|
|
152
|
+
const anthropicHandle = handle;
|
|
153
|
+
for await (const item of client.messages.batches.results(handle.providerBatchId)) {
|
|
154
|
+
if (get(item, "custom_id") !== anthropicHandle.customId)
|
|
155
|
+
continue;
|
|
156
|
+
const result = get(item, "result");
|
|
157
|
+
if (get(result, "type") !== "succeeded") {
|
|
158
|
+
// Never surface the provider's own error text — only that this
|
|
159
|
+
// one item failed.
|
|
160
|
+
throw new ProviderBatchError("anthropic batch item did not succeed");
|
|
161
|
+
}
|
|
162
|
+
const message = get(result, "message");
|
|
163
|
+
return { result: message, containedToolCallPlan: hasToolUse(message) };
|
|
164
|
+
}
|
|
165
|
+
throw new ProviderBatchError("anthropic batch results did not contain our submitted item");
|
|
166
|
+
},
|
|
167
|
+
async direct(request) {
|
|
168
|
+
const response = await client.messages.create(request);
|
|
169
|
+
return { result: response, containedToolCallPlan: hasToolUse(response) };
|
|
170
|
+
},
|
|
171
|
+
};
|
|
172
|
+
}
|
|
173
|
+
const GOOGLE_TERMINAL_STATUS = {
|
|
174
|
+
JOB_STATE_SUCCEEDED: "completed",
|
|
175
|
+
JOB_STATE_FAILED: "failed",
|
|
176
|
+
JOB_STATE_CANCELLED: "failed",
|
|
177
|
+
JOB_STATE_EXPIRED: "expired",
|
|
178
|
+
};
|
|
179
|
+
function hasFunctionCallPart(response) {
|
|
180
|
+
const candidates = get(response, "candidates");
|
|
181
|
+
if (!Array.isArray(candidates))
|
|
182
|
+
return false;
|
|
183
|
+
return candidates.some((candidate) => {
|
|
184
|
+
const parts = get(get(candidate, "content"), "parts");
|
|
185
|
+
if (!Array.isArray(parts))
|
|
186
|
+
return false;
|
|
187
|
+
return parts.some((part) => get(part, "functionCall") != null || get(part, "function_call") != null);
|
|
188
|
+
});
|
|
189
|
+
}
|
|
190
|
+
export function createGoogleBatchAdapter(client) {
|
|
191
|
+
return {
|
|
192
|
+
eligibility(request) {
|
|
193
|
+
if (request.stream === true) {
|
|
194
|
+
return { eligible: false, reason: "streaming requests are direct-only" };
|
|
195
|
+
}
|
|
196
|
+
return { eligible: true };
|
|
197
|
+
},
|
|
198
|
+
async submitOne(request) {
|
|
199
|
+
const { model, ...inlineRequest } = request;
|
|
200
|
+
if (typeof model !== "string" || model.trim().length === 0) {
|
|
201
|
+
throw new ProviderBatchError("google batch requests require a non-blank request.model");
|
|
202
|
+
}
|
|
203
|
+
const job = await client.batches.create({
|
|
204
|
+
model,
|
|
205
|
+
src: { inlinedRequests: [inlineRequest] },
|
|
206
|
+
});
|
|
207
|
+
const handle = { providerBatchId: job.name };
|
|
208
|
+
return handle;
|
|
209
|
+
},
|
|
210
|
+
async poll(handle) {
|
|
211
|
+
const job = await client.batches.get({ name: handle.providerBatchId });
|
|
212
|
+
const state = job.state ?? job.metadata?.state ?? "";
|
|
213
|
+
return { status: GOOGLE_TERMINAL_STATUS[state] ?? "pending" };
|
|
214
|
+
},
|
|
215
|
+
async readResult(handle) {
|
|
216
|
+
const job = await client.batches.get({ name: handle.providerBatchId });
|
|
217
|
+
const entries = job.dest?.inlinedResponses;
|
|
218
|
+
// A single-item batch is unambiguous by construction: index 0 is
|
|
219
|
+
// "our" item, the same way a custom_id would be for the other
|
|
220
|
+
// providers — there is no cross-item confusion possible here.
|
|
221
|
+
const entry = entries?.[0];
|
|
222
|
+
if (!entry) {
|
|
223
|
+
throw new ProviderBatchError("google batch produced no inline response");
|
|
224
|
+
}
|
|
225
|
+
if (entry.error != null) {
|
|
226
|
+
// Never surface the provider's own error text — only that this
|
|
227
|
+
// one item failed.
|
|
228
|
+
throw new ProviderBatchError("google batch item returned an error response");
|
|
229
|
+
}
|
|
230
|
+
if (!entry.response) {
|
|
231
|
+
throw new ProviderBatchError("google batch item had no response body");
|
|
232
|
+
}
|
|
233
|
+
return { result: entry.response, containedToolCallPlan: hasFunctionCallPart(entry.response) };
|
|
234
|
+
},
|
|
235
|
+
async direct(request) {
|
|
236
|
+
const response = await client.models.generateContent(request);
|
|
237
|
+
return { result: response, containedToolCallPlan: hasFunctionCallPart(response) };
|
|
238
|
+
},
|
|
239
|
+
};
|
|
240
|
+
}
|
package/dist/version.d.ts
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
export declare const SDK_VERSION = "0.
|
|
1
|
+
export declare const SDK_VERSION = "0.5.0";
|
package/dist/version.js
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
export const SDK_VERSION = "0.
|
|
1
|
+
export const SDK_VERSION = "0.5.0";
|