@ai-sdk/google-vertex 4.0.23 → 4.0.25
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +18 -0
- package/dist/anthropic/edge/index.js +1 -1
- package/dist/anthropic/edge/index.mjs +1 -1
- package/dist/edge/index.js +1 -1
- package/dist/edge/index.mjs +1 -1
- package/dist/index.js +1 -1
- package/dist/index.mjs +1 -1
- package/docs/16-google-vertex.mdx +1407 -0
- package/package.json +10 -5
- package/src/__snapshots__/google-vertex-embedding-model.test.ts.snap +39 -0
- package/src/anthropic/edge/google-vertex-anthropic-provider-edge.test.ts +87 -0
- package/src/anthropic/edge/google-vertex-anthropic-provider-edge.ts +41 -0
- package/src/anthropic/edge/index.ts +8 -0
- package/src/anthropic/google-vertex-anthropic-messages-options.ts +15 -0
- package/src/anthropic/google-vertex-anthropic-provider-node.test.ts +73 -0
- package/src/anthropic/google-vertex-anthropic-provider-node.ts +40 -0
- package/src/anthropic/google-vertex-anthropic-provider.test.ts +208 -0
- package/src/anthropic/google-vertex-anthropic-provider.ts +210 -0
- package/src/anthropic/index.ts +8 -0
- package/src/edge/google-vertex-auth-edge.test.ts +308 -0
- package/src/edge/google-vertex-auth-edge.ts +161 -0
- package/src/edge/google-vertex-provider-edge.test.ts +105 -0
- package/src/edge/google-vertex-provider-edge.ts +50 -0
- package/src/edge/index.ts +5 -0
- package/src/google-vertex-auth-google-auth-library.test.ts +59 -0
- package/src/google-vertex-auth-google-auth-library.ts +27 -0
- package/src/google-vertex-config.ts +8 -0
- package/src/google-vertex-embedding-model.test.ts +315 -0
- package/src/google-vertex-embedding-model.ts +135 -0
- package/src/google-vertex-embedding-options.ts +63 -0
- package/src/google-vertex-error.ts +19 -0
- package/src/google-vertex-image-model.test.ts +926 -0
- package/src/google-vertex-image-model.ts +288 -0
- package/src/google-vertex-image-settings.ts +8 -0
- package/src/google-vertex-options.ts +32 -0
- package/src/google-vertex-provider-node.test.ts +88 -0
- package/src/google-vertex-provider-node.ts +49 -0
- package/src/google-vertex-provider.test.ts +318 -0
- package/src/google-vertex-provider.ts +217 -0
- package/src/google-vertex-tools.ts +11 -0
- package/src/index.ts +7 -0
- package/src/version.ts +6 -0
|
@@ -0,0 +1,1407 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Google Vertex AI
|
|
3
|
+
description: Learn how to use the Google Vertex AI provider.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Google Vertex Provider
|
|
7
|
+
|
|
8
|
+
The Google Vertex provider for the [AI SDK](/docs) contains language model support for the [Google Vertex AI](https://cloud.google.com/vertex-ai) APIs. This includes support for [Google's Gemini models](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models) and [Anthropic's Claude partner models](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-claude).
|
|
9
|
+
|
|
10
|
+
<Note>
|
|
11
|
+
The Google Vertex provider is compatible with both Node.js and Edge runtimes.
|
|
12
|
+
The Edge runtime is supported through the `@ai-sdk/google-vertex/edge`
|
|
13
|
+
sub-module. More details can be found in the [Google Vertex Edge
|
|
14
|
+
Runtime](#google-vertex-edge-runtime) and [Google Vertex Anthropic Edge
|
|
15
|
+
Runtime](#google-vertex-anthropic-edge-runtime) sections below.
|
|
16
|
+
</Note>
|
|
17
|
+
|
|
18
|
+
## Setup
|
|
19
|
+
|
|
20
|
+
The Google Vertex and Google Vertex Anthropic providers are both available in the `@ai-sdk/google-vertex` module. You can install it with
|
|
21
|
+
|
|
22
|
+
<Tabs items={['pnpm', 'npm', 'yarn', 'bun']}>
|
|
23
|
+
<Tab>
|
|
24
|
+
<Snippet text="pnpm add @ai-sdk/google-vertex" dark />
|
|
25
|
+
</Tab>
|
|
26
|
+
<Tab>
|
|
27
|
+
<Snippet text="npm install @ai-sdk/google-vertex" dark />
|
|
28
|
+
</Tab>
|
|
29
|
+
<Tab>
|
|
30
|
+
<Snippet
|
|
31
|
+
text="yarn add @ai-sdk/google-vertex @google-cloud/vertexai"
|
|
32
|
+
dark
|
|
33
|
+
/>
|
|
34
|
+
</Tab>
|
|
35
|
+
|
|
36
|
+
<Tab>
|
|
37
|
+
<Snippet text="bun add @ai-sdk/google-vertex" dark />
|
|
38
|
+
</Tab>
|
|
39
|
+
</Tabs>
|
|
40
|
+
|
|
41
|
+
## Google Vertex Provider Usage
|
|
42
|
+
|
|
43
|
+
The Google Vertex provider instance is used to create model instances that call the Vertex AI API. The models available with this provider include [Google's Gemini models](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models). If you're looking to use [Anthropic's Claude models](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-claude), see the [Google Vertex Anthropic Provider](#google-vertex-anthropic-provider-usage) section below.
|
|
44
|
+
|
|
45
|
+
### Provider Instance
|
|
46
|
+
|
|
47
|
+
You can import the default provider instance `vertex` from `@ai-sdk/google-vertex`:
|
|
48
|
+
|
|
49
|
+
```ts
|
|
50
|
+
import { vertex } from '@ai-sdk/google-vertex';
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
If you need a customized setup, you can import `createVertex` from `@ai-sdk/google-vertex` and create a provider instance with your settings:
|
|
54
|
+
|
|
55
|
+
```ts
|
|
56
|
+
import { createVertex } from '@ai-sdk/google-vertex';
|
|
57
|
+
|
|
58
|
+
const vertex = createVertex({
|
|
59
|
+
project: 'my-project', // optional
|
|
60
|
+
location: 'us-central1', // optional
|
|
61
|
+
});
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
Google Vertex supports multiple authentication methods depending on your runtime environment and requirements.
|
|
65
|
+
|
|
66
|
+
#### Node.js Runtime
|
|
67
|
+
|
|
68
|
+
The Node.js runtime is the default runtime supported by the AI SDK. It supports all standard Google Cloud authentication options through the [`google-auth-library`](https://github.com/googleapis/google-auth-library-nodejs?tab=readme-ov-file#ways-to-authenticate). Typical use involves setting a path to a json credentials file in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. The credentials file can be obtained from the [Google Cloud Console](https://console.cloud.google.com/apis/credentials).
|
|
69
|
+
|
|
70
|
+
If you want to customize the Google authentication options you can pass them as options to the `createVertex` function, for example:
|
|
71
|
+
|
|
72
|
+
```ts
|
|
73
|
+
import { createVertex } from '@ai-sdk/google-vertex';
|
|
74
|
+
|
|
75
|
+
const vertex = createVertex({
|
|
76
|
+
googleAuthOptions: {
|
|
77
|
+
credentials: {
|
|
78
|
+
client_email: 'my-email',
|
|
79
|
+
private_key: 'my-private-key',
|
|
80
|
+
},
|
|
81
|
+
},
|
|
82
|
+
});
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
##### Optional Provider Settings
|
|
86
|
+
|
|
87
|
+
You can use the following optional settings to customize the provider instance:
|
|
88
|
+
|
|
89
|
+
- **project** _string_
|
|
90
|
+
|
|
91
|
+
The Google Cloud project ID that you want to use for the API calls.
|
|
92
|
+
It uses the `GOOGLE_VERTEX_PROJECT` environment variable by default.
|
|
93
|
+
|
|
94
|
+
- **location** _string_
|
|
95
|
+
|
|
96
|
+
The Google Cloud location that you want to use for the API calls, e.g. `us-central1`.
|
|
97
|
+
It uses the `GOOGLE_VERTEX_LOCATION` environment variable by default.
|
|
98
|
+
|
|
99
|
+
- **googleAuthOptions** _object_
|
|
100
|
+
|
|
101
|
+
Optional. The Authentication options used by the [Google Auth Library](https://github.com/googleapis/google-auth-library-nodejs/). See also the [GoogleAuthOptions](https://github.com/googleapis/google-auth-library-nodejs/blob/08978822e1b7b5961f0e355df51d738e012be392/src/auth/googleauth.ts#L87C18-L87C35) interface.
|
|
102
|
+
|
|
103
|
+
- **authClient** _object_
|
|
104
|
+
An `AuthClient` to use.
|
|
105
|
+
|
|
106
|
+
- **keyFilename** _string_
|
|
107
|
+
Path to a .json, .pem, or .p12 key file.
|
|
108
|
+
|
|
109
|
+
- **keyFile** _string_
|
|
110
|
+
Path to a .json, .pem, or .p12 key file.
|
|
111
|
+
|
|
112
|
+
- **credentials** _object_
|
|
113
|
+
Object containing client_email and private_key properties, or the external account client options.
|
|
114
|
+
|
|
115
|
+
- **clientOptions** _object_
|
|
116
|
+
Options object passed to the constructor of the client.
|
|
117
|
+
|
|
118
|
+
- **scopes** _string | string[]_
|
|
119
|
+
Required scopes for the desired API request.
|
|
120
|
+
|
|
121
|
+
- **projectId** _string_
|
|
122
|
+
Your project ID.
|
|
123
|
+
|
|
124
|
+
- **universeDomain** _string_
|
|
125
|
+
The default service domain for a given Cloud universe.
|
|
126
|
+
|
|
127
|
+
- **headers** _Resolvable<Record<string, string | undefined>>_
|
|
128
|
+
|
|
129
|
+
Headers to include in the requests. Can be provided in multiple formats:
|
|
130
|
+
|
|
131
|
+
- A record of header key-value pairs: `Record<string, string | undefined>`
|
|
132
|
+
- A function that returns headers: `() => Record<string, string | undefined>`
|
|
133
|
+
- An async function that returns headers: `async () => Record<string, string | undefined>`
|
|
134
|
+
- A promise that resolves to headers: `Promise<Record<string, string | undefined>>`
|
|
135
|
+
|
|
136
|
+
- **fetch** _(input: RequestInfo, init?: RequestInit) => Promise<Response>_
|
|
137
|
+
|
|
138
|
+
Custom [fetch](https://developer.mozilla.org/en-US/docs/Web/API/fetch) implementation.
|
|
139
|
+
Defaults to the global `fetch` function.
|
|
140
|
+
You can use it as a middleware to intercept requests,
|
|
141
|
+
or to provide a custom fetch implementation for e.g. testing.
|
|
142
|
+
|
|
143
|
+
- **baseURL** _string_
|
|
144
|
+
|
|
145
|
+
Optional. Base URL for the Google Vertex API calls e.g. to use proxy servers. By default, it is constructed using the location and project:
|
|
146
|
+
`https://${location}-aiplatform.googleapis.com/v1/projects/${project}/locations/${location}/publishers/google`
|
|
147
|
+
|
|
148
|
+
<a id="google-vertex-edge-runtime"></a>
|
|
149
|
+
#### Edge Runtime
|
|
150
|
+
|
|
151
|
+
Edge runtimes (like Vercel Edge Functions and Cloudflare Workers) are lightweight JavaScript environments that run closer to users at the network edge.
|
|
152
|
+
They only provide a subset of the standard Node.js APIs.
|
|
153
|
+
For example, direct file system access is not available, and many Node.js-specific libraries
|
|
154
|
+
(including the standard Google Auth library) are not compatible.
|
|
155
|
+
|
|
156
|
+
The Edge runtime version of the Google Vertex provider supports Google's [Application Default Credentials](https://github.com/googleapis/google-auth-library-nodejs?tab=readme-ov-file#application-default-credentials) through environment variables. The values can be obtained from a json credentials file from the [Google Cloud Console](https://console.cloud.google.com/apis/credentials).
|
|
157
|
+
|
|
158
|
+
You can import the default provider instance `vertex` from `@ai-sdk/google-vertex/edge`:
|
|
159
|
+
|
|
160
|
+
```ts
|
|
161
|
+
import { vertex } from '@ai-sdk/google-vertex/edge';
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
<Note>
|
|
165
|
+
The `/edge` sub-module is included in the `@ai-sdk/google-vertex` package, so
|
|
166
|
+
you don't need to install it separately. You must import from
|
|
167
|
+
`@ai-sdk/google-vertex/edge` to differentiate it from the Node.js provider.
|
|
168
|
+
</Note>
|
|
169
|
+
|
|
170
|
+
If you need a customized setup, you can import `createVertex` from `@ai-sdk/google-vertex/edge` and create a provider instance with your settings:
|
|
171
|
+
|
|
172
|
+
```ts
|
|
173
|
+
import { createVertex } from '@ai-sdk/google-vertex/edge';
|
|
174
|
+
|
|
175
|
+
const vertex = createVertex({
|
|
176
|
+
project: 'my-project', // optional
|
|
177
|
+
location: 'us-central1', // optional
|
|
178
|
+
});
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
For Edge runtime authentication, you'll need to set these environment variables from your Google Default Application Credentials JSON file:
|
|
182
|
+
|
|
183
|
+
- `GOOGLE_CLIENT_EMAIL`
|
|
184
|
+
- `GOOGLE_PRIVATE_KEY`
|
|
185
|
+
- `GOOGLE_PRIVATE_KEY_ID` (optional)
|
|
186
|
+
|
|
187
|
+
These values can be obtained from a service account JSON file from the [Google Cloud Console](https://console.cloud.google.com/apis/credentials).
|
|
188
|
+
|
|
189
|
+
##### Optional Provider Settings
|
|
190
|
+
|
|
191
|
+
You can use the following optional settings to customize the provider instance:
|
|
192
|
+
|
|
193
|
+
- **project** _string_
|
|
194
|
+
|
|
195
|
+
The Google Cloud project ID that you want to use for the API calls.
|
|
196
|
+
It uses the `GOOGLE_VERTEX_PROJECT` environment variable by default.
|
|
197
|
+
|
|
198
|
+
- **location** _string_
|
|
199
|
+
|
|
200
|
+
The Google Cloud location that you want to use for the API calls, e.g. `us-central1`.
|
|
201
|
+
It uses the `GOOGLE_VERTEX_LOCATION` environment variable by default.
|
|
202
|
+
|
|
203
|
+
- **googleCredentials** _object_
|
|
204
|
+
|
|
205
|
+
Optional. The credentials used by the Edge provider for authentication. These credentials are typically set through environment variables and are derived from a service account JSON file.
|
|
206
|
+
|
|
207
|
+
- **clientEmail** _string_
|
|
208
|
+
The client email from the service account JSON file. Defaults to the contents of the `GOOGLE_CLIENT_EMAIL` environment variable.
|
|
209
|
+
|
|
210
|
+
- **privateKey** _string_
|
|
211
|
+
The private key from the service account JSON file. Defaults to the contents of the `GOOGLE_PRIVATE_KEY` environment variable.
|
|
212
|
+
|
|
213
|
+
- **privateKeyId** _string_
|
|
214
|
+
The private key ID from the service account JSON file (optional). Defaults to the contents of the `GOOGLE_PRIVATE_KEY_ID` environment variable.
|
|
215
|
+
|
|
216
|
+
- **headers** _Resolvable<Record<string, string | undefined>>_
|
|
217
|
+
|
|
218
|
+
Headers to include in the requests. Can be provided in multiple formats:
|
|
219
|
+
|
|
220
|
+
- A record of header key-value pairs: `Record<string, string | undefined>`
|
|
221
|
+
- A function that returns headers: `() => Record<string, string | undefined>`
|
|
222
|
+
- An async function that returns headers: `async () => Record<string, string | undefined>`
|
|
223
|
+
- A promise that resolves to headers: `Promise<Record<string, string | undefined>>`
|
|
224
|
+
|
|
225
|
+
- **fetch** _(input: RequestInfo, init?: RequestInit) => Promise<Response>_
|
|
226
|
+
|
|
227
|
+
Custom [fetch](https://developer.mozilla.org/en-US/docs/Web/API/fetch) implementation.
|
|
228
|
+
Defaults to the global `fetch` function.
|
|
229
|
+
You can use it as a middleware to intercept requests,
|
|
230
|
+
or to provide a custom fetch implementation for e.g. testing.
|
|
231
|
+
|
|
232
|
+
#### Express Mode
|
|
233
|
+
|
|
234
|
+
Express mode provides a simplified authentication method using an API key instead of OAuth or service account credentials. When using express mode, the `project` and `location` settings are not required.
|
|
235
|
+
|
|
236
|
+
```ts
|
|
237
|
+
import { createVertex } from '@ai-sdk/google-vertex';
|
|
238
|
+
|
|
239
|
+
const vertex = createVertex({
|
|
240
|
+
apiKey: process.env.GOOGLE_VERTEX_API_KEY,
|
|
241
|
+
});
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
##### Optional Provider Settings
|
|
245
|
+
|
|
246
|
+
- **apiKey** _string_
|
|
247
|
+
|
|
248
|
+
The API key for Google Vertex AI. When provided, the provider uses express mode with API key authentication instead of OAuth.
|
|
249
|
+
It uses the `GOOGLE_VERTEX_API_KEY` environment variable by default.
|
|
250
|
+
|
|
251
|
+
### Language Models
|
|
252
|
+
|
|
253
|
+
You can create models that call the Vertex API using the provider instance.
|
|
254
|
+
The first argument is the model id, e.g. `gemini-1.5-pro`.
|
|
255
|
+
|
|
256
|
+
```ts
|
|
257
|
+
const model = vertex('gemini-1.5-pro');
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
<Note>
|
|
261
|
+
If you are using [your own
|
|
262
|
+
models](https://cloud.google.com/vertex-ai/docs/training-overview), the name
|
|
263
|
+
of your model needs to start with `projects/`.
|
|
264
|
+
</Note>
|
|
265
|
+
|
|
266
|
+
Google Vertex models support also some model specific settings that are not part
|
|
267
|
+
of the [standard call settings](/docs/ai-sdk-core/settings). You can pass them as
|
|
268
|
+
an options argument:
|
|
269
|
+
|
|
270
|
+
```ts
|
|
271
|
+
const model = vertex('gemini-1.5-pro');
|
|
272
|
+
|
|
273
|
+
await generateText({
|
|
274
|
+
model,
|
|
275
|
+
providerOptions: {
|
|
276
|
+
google: {
|
|
277
|
+
safetySettings: [
|
|
278
|
+
{
|
|
279
|
+
category: 'HARM_CATEGORY_UNSPECIFIED',
|
|
280
|
+
threshold: 'BLOCK_LOW_AND_ABOVE',
|
|
281
|
+
},
|
|
282
|
+
],
|
|
283
|
+
},
|
|
284
|
+
},
|
|
285
|
+
});
|
|
286
|
+
```
|
|
287
|
+
|
|
288
|
+
The following optional provider options are available for Google Vertex models:
|
|
289
|
+
|
|
290
|
+
- **structuredOutputs** _boolean_
|
|
291
|
+
|
|
292
|
+
Optional. Enable structured output. Default is true.
|
|
293
|
+
|
|
294
|
+
This is useful when the JSON Schema contains elements that are
|
|
295
|
+
not supported by the OpenAPI schema version that
|
|
296
|
+
Google Vertex uses. You can use this to disable
|
|
297
|
+
structured outputs if you need to.
|
|
298
|
+
|
|
299
|
+
See [Troubleshooting: Schema Limitations](#schema-limitations) for more details.
|
|
300
|
+
|
|
301
|
+
- **safetySettings** _Array\<\{ category: string; threshold: string \}\>_
|
|
302
|
+
|
|
303
|
+
Optional. Safety settings for the model.
|
|
304
|
+
|
|
305
|
+
- **category** _string_
|
|
306
|
+
|
|
307
|
+
The category of the safety setting. Can be one of the following:
|
|
308
|
+
|
|
309
|
+
- `HARM_CATEGORY_UNSPECIFIED`
|
|
310
|
+
- `HARM_CATEGORY_HATE_SPEECH`
|
|
311
|
+
- `HARM_CATEGORY_DANGEROUS_CONTENT`
|
|
312
|
+
- `HARM_CATEGORY_HARASSMENT`
|
|
313
|
+
- `HARM_CATEGORY_SEXUALLY_EXPLICIT`
|
|
314
|
+
- `HARM_CATEGORY_CIVIC_INTEGRITY`
|
|
315
|
+
|
|
316
|
+
- **threshold** _string_
|
|
317
|
+
|
|
318
|
+
The threshold of the safety setting. Can be one of the following:
|
|
319
|
+
|
|
320
|
+
- `HARM_BLOCK_THRESHOLD_UNSPECIFIED`
|
|
321
|
+
- `BLOCK_LOW_AND_ABOVE`
|
|
322
|
+
- `BLOCK_MEDIUM_AND_ABOVE`
|
|
323
|
+
- `BLOCK_ONLY_HIGH`
|
|
324
|
+
- `BLOCK_NONE`
|
|
325
|
+
|
|
326
|
+
- **audioTimestamp** _boolean_
|
|
327
|
+
|
|
328
|
+
Optional. Enables timestamp understanding for audio files. Defaults to false.
|
|
329
|
+
|
|
330
|
+
This is useful for generating transcripts with accurate timestamps.
|
|
331
|
+
Consult [Google's Documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/audio-understanding) for usage details.
|
|
332
|
+
|
|
333
|
+
- **labels** _object_
|
|
334
|
+
|
|
335
|
+
Optional. Defines labels used in billing reports.
|
|
336
|
+
|
|
337
|
+
Consult [Google's Documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/add-labels-to-api-calls) for usage details.
|
|
338
|
+
|
|
339
|
+
You can use Google Vertex language models to generate text with the `generateText` function:
|
|
340
|
+
|
|
341
|
+
```ts highlight="1,4"
|
|
342
|
+
import { vertex } from '@ai-sdk/google-vertex';
|
|
343
|
+
import { generateText } from 'ai';
|
|
344
|
+
|
|
345
|
+
const { text } = await generateText({
|
|
346
|
+
model: vertex('gemini-1.5-pro'),
|
|
347
|
+
prompt: 'Write a vegetarian lasagna recipe for 4 people.',
|
|
348
|
+
});
|
|
349
|
+
```
|
|
350
|
+
|
|
351
|
+
Google Vertex language models can also be used in the `streamText` function
|
|
352
|
+
(see [AI SDK Core](/docs/ai-sdk-core)).
|
|
353
|
+
|
|
354
|
+
#### Code Execution
|
|
355
|
+
|
|
356
|
+
With [Code Execution](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/code-execution), certain Gemini models on Vertex AI can generate and execute Python code. This allows the model to perform calculations, data manipulation, and other programmatic tasks to enhance its responses.
|
|
357
|
+
|
|
358
|
+
You can enable code execution by adding the `code_execution` tool to your request.
|
|
359
|
+
|
|
360
|
+
```ts
|
|
361
|
+
import { vertex } from '@ai-sdk/google-vertex';
|
|
362
|
+
import { generateText } from 'ai';
|
|
363
|
+
|
|
364
|
+
const result = await generateText({
|
|
365
|
+
model: vertex('gemini-2.5-pro'),
|
|
366
|
+
tools: { code_execution: vertex.tools.codeExecution({}) },
|
|
367
|
+
prompt:
|
|
368
|
+
'Use python to calculate 20th fibonacci number. Then find the nearest palindrome to it.',
|
|
369
|
+
});
|
|
370
|
+
```
|
|
371
|
+
|
|
372
|
+
The response will contain `tool-call` and `tool-result` parts for the executed code.
|
|
373
|
+
|
|
374
|
+
#### URL Context
|
|
375
|
+
|
|
376
|
+
URL Context allows Gemini models to retrieve and analyze content from URLs. Supported models: Gemini 2.5 Flash-Lite, 2.5 Pro, 2.5 Flash, 2.0 Flash.
|
|
377
|
+
|
|
378
|
+
```ts
|
|
379
|
+
import { vertex } from '@ai-sdk/google-vertex';
|
|
380
|
+
import { generateText } from 'ai';
|
|
381
|
+
|
|
382
|
+
const result = await generateText({
|
|
383
|
+
model: vertex('gemini-2.5-pro'),
|
|
384
|
+
tools: { url_context: vertex.tools.urlContext({}) },
|
|
385
|
+
prompt: 'What are the key points from https://example.com/article?',
|
|
386
|
+
});
|
|
387
|
+
```
|
|
388
|
+
|
|
389
|
+
#### Google Search
|
|
390
|
+
|
|
391
|
+
Google Search enables Gemini models to access real-time web information. Supported models: Gemini 2.5 Flash-Lite, 2.5 Flash, 2.0 Flash, 2.5 Pro.
|
|
392
|
+
|
|
393
|
+
```ts
|
|
394
|
+
import { vertex } from '@ai-sdk/google-vertex';
|
|
395
|
+
import { generateText } from 'ai';
|
|
396
|
+
|
|
397
|
+
const result = await generateText({
|
|
398
|
+
model: vertex('gemini-2.5-pro'),
|
|
399
|
+
tools: { google_search: vertex.tools.googleSearch({}) },
|
|
400
|
+
prompt: 'What are the latest developments in AI?',
|
|
401
|
+
});
|
|
402
|
+
```
|
|
403
|
+
|
|
404
|
+
#### Enterprise Web Search
|
|
405
|
+
|
|
406
|
+
[Enterprise Web Search](https://cloud.google.com/vertex-ai/generative-ai/docs/grounding/web-grounding-enterprise) provides grounding using a compliance-focused web index designed for highly-regulated industries such as finance, healthcare, and the public sector. Unlike standard Google Search grounding, Enterprise Web Search does not log customer data and supports VPC service controls. Supported models: Gemini 2.0 and newer.
|
|
407
|
+
|
|
408
|
+
```ts
|
|
409
|
+
import { vertex } from '@ai-sdk/google-vertex';
|
|
410
|
+
import { generateText } from 'ai';
|
|
411
|
+
|
|
412
|
+
const result = await generateText({
|
|
413
|
+
model: vertex('gemini-2.5-flash'),
|
|
414
|
+
tools: {
|
|
415
|
+
enterprise_web_search: vertex.tools.enterpriseWebSearch({}),
|
|
416
|
+
},
|
|
417
|
+
prompt: 'What are the latest FDA regulations for clinical trials?',
|
|
418
|
+
});
|
|
419
|
+
```
|
|
420
|
+
|
|
421
|
+
#### Google Maps
|
|
422
|
+
|
|
423
|
+
Google Maps grounding enables Gemini models to access Google Maps data for location-aware responses. Supported models: Gemini 2.5 Flash-Lite, 2.5 Flash, 2.0 Flash, 2.5 Pro, 3.0 Pro.
|
|
424
|
+
|
|
425
|
+
```ts
|
|
426
|
+
import { vertex } from '@ai-sdk/google-vertex';
|
|
427
|
+
import { generateText } from 'ai';
|
|
428
|
+
|
|
429
|
+
const result = await generateText({
|
|
430
|
+
model: vertex('gemini-2.5-flash'),
|
|
431
|
+
tools: {
|
|
432
|
+
google_maps: vertex.tools.googleMaps({}),
|
|
433
|
+
},
|
|
434
|
+
providerOptions: {
|
|
435
|
+
google: {
|
|
436
|
+
retrievalConfig: {
|
|
437
|
+
latLng: { latitude: 34.090199, longitude: -117.881081 },
|
|
438
|
+
},
|
|
439
|
+
},
|
|
440
|
+
},
|
|
441
|
+
prompt: 'What are the best Italian restaurants nearby?',
|
|
442
|
+
});
|
|
443
|
+
```
|
|
444
|
+
|
|
445
|
+
The optional `retrievalConfig.latLng` provider option provides location context for queries about nearby places. This configuration applies to any grounding tools that support location context.
|
|
446
|
+
|
|
447
|
+
#### Reasoning (Thinking Tokens)
|
|
448
|
+
|
|
449
|
+
Google Vertex AI, through its support for Gemini models, can also emit "thinking" tokens, representing the model's reasoning process. The AI SDK exposes these as reasoning information.
|
|
450
|
+
|
|
451
|
+
To enable thinking tokens for compatible Gemini models via Vertex, set `includeThoughts: true` in the `thinkingConfig` provider option. Since the Vertex provider uses the Google provider's underlying language model, these options are passed through `providerOptions.google`:
|
|
452
|
+
|
|
453
|
+
```ts
|
|
454
|
+
import { vertex } from '@ai-sdk/google-vertex';
|
|
455
|
+
import { GoogleGenerativeAIProviderOptions } from '@ai-sdk/google'; // Note: importing from @ai-sdk/google
|
|
456
|
+
import { generateText, streamText } from 'ai';
|
|
457
|
+
|
|
458
|
+
// For generateText:
|
|
459
|
+
const { text, reasoningText, reasoning } = await generateText({
|
|
460
|
+
model: vertex('gemini-2.0-flash-001'), // Or other supported model via Vertex
|
|
461
|
+
providerOptions: {
|
|
462
|
+
google: {
|
|
463
|
+
// Options are nested under 'google' for Vertex provider
|
|
464
|
+
thinkingConfig: {
|
|
465
|
+
includeThoughts: true,
|
|
466
|
+
// thinkingBudget: 2048, // Optional
|
|
467
|
+
},
|
|
468
|
+
} satisfies GoogleGenerativeAIProviderOptions,
|
|
469
|
+
},
|
|
470
|
+
prompt: 'Explain quantum computing in simple terms.',
|
|
471
|
+
});
|
|
472
|
+
|
|
473
|
+
console.log('Reasoning:', reasoningText);
|
|
474
|
+
console.log('Reasoning Details:', reasoning);
|
|
475
|
+
console.log('Final Text:', text);
|
|
476
|
+
|
|
477
|
+
// For streamText:
|
|
478
|
+
const result = streamText({
|
|
479
|
+
model: vertex('gemini-2.0-flash-001'), // Or other supported model via Vertex
|
|
480
|
+
providerOptions: {
|
|
481
|
+
google: {
|
|
482
|
+
// Options are nested under 'google' for Vertex provider
|
|
483
|
+
thinkingConfig: {
|
|
484
|
+
includeThoughts: true,
|
|
485
|
+
// thinkingBudget: 2048, // Optional
|
|
486
|
+
},
|
|
487
|
+
} satisfies GoogleGenerativeAIProviderOptions,
|
|
488
|
+
},
|
|
489
|
+
prompt: 'Explain quantum computing in simple terms.',
|
|
490
|
+
});
|
|
491
|
+
|
|
492
|
+
for await (const part of result.fullStream) {
|
|
493
|
+
if (part.type === 'reasoning') {
|
|
494
|
+
process.stdout.write(`THOUGHT: ${part.textDelta}\n`);
|
|
495
|
+
} else if (part.type === 'text-delta') {
|
|
496
|
+
process.stdout.write(part.textDelta);
|
|
497
|
+
}
|
|
498
|
+
}
|
|
499
|
+
```
|
|
500
|
+
|
|
501
|
+
When `includeThoughts` is true, parts of the API response marked with `thought: true` will be processed as reasoning.
|
|
502
|
+
|
|
503
|
+
- In `generateText`, these contribute to the `reasoningText` (string) and `reasoning` (array) fields.
|
|
504
|
+
- In `streamText`, these are emitted as `reasoning` stream parts.
|
|
505
|
+
|
|
506
|
+
<Note>
|
|
507
|
+
Refer to the [Google Vertex AI documentation on
|
|
508
|
+
"thinking"](https://cloud.google.com/vertex-ai/generative-ai/docs/thinking)
|
|
509
|
+
for model compatibility and further details.
|
|
510
|
+
</Note>
|
|
511
|
+
|
|
512
|
+
#### File Inputs
|
|
513
|
+
|
|
514
|
+
The Google Vertex provider supports file inputs, e.g. PDF files.
|
|
515
|
+
|
|
516
|
+
```ts
|
|
517
|
+
import { vertex } from '@ai-sdk/google-vertex';
|
|
518
|
+
import { generateText } from 'ai';
|
|
519
|
+
|
|
520
|
+
const { text } = await generateText({
|
|
521
|
+
model: vertex('gemini-1.5-pro'),
|
|
522
|
+
messages: [
|
|
523
|
+
{
|
|
524
|
+
role: 'user',
|
|
525
|
+
content: [
|
|
526
|
+
{
|
|
527
|
+
type: 'text',
|
|
528
|
+
text: 'What is an embedding model according to this document?',
|
|
529
|
+
},
|
|
530
|
+
{
|
|
531
|
+
type: 'file',
|
|
532
|
+
data: fs.readFileSync('./data/ai.pdf'),
|
|
533
|
+
mediaType: 'application/pdf',
|
|
534
|
+
},
|
|
535
|
+
],
|
|
536
|
+
},
|
|
537
|
+
],
|
|
538
|
+
});
|
|
539
|
+
```
|
|
540
|
+
|
|
541
|
+
<Note>
|
|
542
|
+
The AI SDK will automatically download URLs if you pass them as data, except
|
|
543
|
+
for `gs://` URLs. You can use the Google Cloud Storage API to upload larger
|
|
544
|
+
files to that location.
|
|
545
|
+
</Note>
|
|
546
|
+
|
|
547
|
+
See [File Parts](/docs/foundations/prompts#file-parts) for details on how to use files in prompts.
|
|
548
|
+
|
|
549
|
+
### Safety Ratings
|
|
550
|
+
|
|
551
|
+
The safety ratings provide insight into the safety of the model's response.
|
|
552
|
+
See [Google Vertex AI documentation on configuring safety filters](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/configure-safety-filters).
|
|
553
|
+
|
|
554
|
+
Example response excerpt:
|
|
555
|
+
|
|
556
|
+
```json
|
|
557
|
+
{
|
|
558
|
+
"safetyRatings": [
|
|
559
|
+
{
|
|
560
|
+
"category": "HARM_CATEGORY_HATE_SPEECH",
|
|
561
|
+
"probability": "NEGLIGIBLE",
|
|
562
|
+
"probabilityScore": 0.11027937,
|
|
563
|
+
"severity": "HARM_SEVERITY_LOW",
|
|
564
|
+
"severityScore": 0.28487435
|
|
565
|
+
},
|
|
566
|
+
{
|
|
567
|
+
"category": "HARM_CATEGORY_DANGEROUS_CONTENT",
|
|
568
|
+
"probability": "HIGH",
|
|
569
|
+
"blocked": true,
|
|
570
|
+
"probabilityScore": 0.95422274,
|
|
571
|
+
"severity": "HARM_SEVERITY_MEDIUM",
|
|
572
|
+
"severityScore": 0.43398145
|
|
573
|
+
},
|
|
574
|
+
{
|
|
575
|
+
"category": "HARM_CATEGORY_HARASSMENT",
|
|
576
|
+
"probability": "NEGLIGIBLE",
|
|
577
|
+
"probabilityScore": 0.11085559,
|
|
578
|
+
"severity": "HARM_SEVERITY_NEGLIGIBLE",
|
|
579
|
+
"severityScore": 0.19027223
|
|
580
|
+
},
|
|
581
|
+
{
|
|
582
|
+
"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
|
|
583
|
+
"probability": "NEGLIGIBLE",
|
|
584
|
+
"probabilityScore": 0.22901751,
|
|
585
|
+
"severity": "HARM_SEVERITY_NEGLIGIBLE",
|
|
586
|
+
"severityScore": 0.09089675
|
|
587
|
+
}
|
|
588
|
+
]
|
|
589
|
+
}
|
|
590
|
+
```
|
|
591
|
+
|
|
592
|
+
For more details, see the [Google Vertex AI documentation on grounding with Google Search](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/ground-gemini#ground-to-search).
|
|
593
|
+
|
|
594
|
+
### Troubleshooting
|
|
595
|
+
|
|
596
|
+
#### Schema Limitations
|
|
597
|
+
|
|
598
|
+
The Google Vertex API uses a subset of the OpenAPI 3.0 schema,
|
|
599
|
+
which does not support features such as unions.
|
|
600
|
+
The errors that you get in this case look like this:
|
|
601
|
+
|
|
602
|
+
`GenerateContentRequest.generation_config.response_schema.properties[occupation].type: must be specified`
|
|
603
|
+
|
|
604
|
+
By default, structured outputs are enabled (and for tool calling they are required).
|
|
605
|
+
You can disable structured outputs for object generation as a workaround:
|
|
606
|
+
|
|
607
|
+
```ts highlight="3,8"
|
|
608
|
+
const result = await generateObject({
|
|
609
|
+
model: vertex('gemini-1.5-pro'),
|
|
610
|
+
providerOptions: {
|
|
611
|
+
google: {
|
|
612
|
+
structuredOutputs: false,
|
|
613
|
+
},
|
|
614
|
+
},
|
|
615
|
+
schema: z.object({
|
|
616
|
+
name: z.string(),
|
|
617
|
+
age: z.number(),
|
|
618
|
+
contact: z.union([
|
|
619
|
+
z.object({
|
|
620
|
+
type: z.literal('email'),
|
|
621
|
+
value: z.string(),
|
|
622
|
+
}),
|
|
623
|
+
z.object({
|
|
624
|
+
type: z.literal('phone'),
|
|
625
|
+
value: z.string(),
|
|
626
|
+
}),
|
|
627
|
+
]),
|
|
628
|
+
}),
|
|
629
|
+
prompt: 'Generate an example person for testing.',
|
|
630
|
+
});
|
|
631
|
+
```
|
|
632
|
+
|
|
633
|
+
The following Zod features are known to not work with Google Vertex:
|
|
634
|
+
|
|
635
|
+
- `z.union`
|
|
636
|
+
- `z.record`
|
|
637
|
+
|
|
638
|
+
### Model Capabilities
|
|
639
|
+
|
|
640
|
+
| Model | Image Input | Object Generation | Tool Usage | Tool Streaming |
|
|
641
|
+
| ---------------------- | ------------------- | ------------------- | ------------------- | ------------------- |
|
|
642
|
+
| `gemini-3-pro-preview` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
|
|
643
|
+
| `gemini-2.5-pro` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
|
|
644
|
+
| `gemini-2.5-flash` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
|
|
645
|
+
| `gemini-2.0-flash-001` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
|
|
646
|
+
| `gemini-1.5-flash` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
|
|
647
|
+
| `gemini-1.5-pro` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
|
|
648
|
+
|
|
649
|
+
<Note>
|
|
650
|
+
The table above lists popular models. Please see the [Google Vertex AI
|
|
651
|
+
docs](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models)
|
|
652
|
+
for a full list of available models. The table above lists popular models. You
|
|
653
|
+
can also pass any available provider model ID as a string if needed.
|
|
654
|
+
</Note>
|
|
655
|
+
|
|
656
|
+
### Embedding Models
|
|
657
|
+
|
|
658
|
+
You can create models that call the Google Vertex AI embeddings API using the `.embeddingModel()` factory method:
|
|
659
|
+
|
|
660
|
+
```ts
|
|
661
|
+
const model = vertex.embeddingModel('text-embedding-004');
|
|
662
|
+
```
|
|
663
|
+
|
|
664
|
+
Google Vertex AI embedding models support additional settings. You can pass them as an options argument:
|
|
665
|
+
|
|
666
|
+
```ts
|
|
667
|
+
import { vertex } from '@ai-sdk/google-vertex';
|
|
668
|
+
import { embed } from 'ai';
|
|
669
|
+
|
|
670
|
+
const model = vertex.embeddingModel('text-embedding-004');
|
|
671
|
+
|
|
672
|
+
const { embedding } = await embed({
|
|
673
|
+
model,
|
|
674
|
+
value: 'sunny day at the beach',
|
|
675
|
+
providerOptions: {
|
|
676
|
+
google: {
|
|
677
|
+
outputDimensionality: 512, // optional, number of dimensions for the embedding
|
|
678
|
+
taskType: 'SEMANTIC_SIMILARITY', // optional, specifies the task type for generating embeddings
|
|
679
|
+
autoTruncate: false, // optional
|
|
680
|
+
},
|
|
681
|
+
},
|
|
682
|
+
});
|
|
683
|
+
```
|
|
684
|
+
|
|
685
|
+
The following optional provider options are available for Google Vertex AI embedding models:
|
|
686
|
+
|
|
687
|
+
- **outputDimensionality**: _number_
|
|
688
|
+
|
|
689
|
+
Optional reduced dimension for the output embedding. If set, excessive values in the output embedding are truncated from the end.
|
|
690
|
+
|
|
691
|
+
- **taskType**: _string_
|
|
692
|
+
|
|
693
|
+
Optional. Specifies the task type for generating embeddings. Supported task types include:
|
|
694
|
+
|
|
695
|
+
- `SEMANTIC_SIMILARITY`: Optimized for text similarity.
|
|
696
|
+
- `CLASSIFICATION`: Optimized for text classification.
|
|
697
|
+
- `CLUSTERING`: Optimized for clustering texts based on similarity.
|
|
698
|
+
- `RETRIEVAL_DOCUMENT`: Optimized for document retrieval.
|
|
699
|
+
- `RETRIEVAL_QUERY`: Optimized for query-based retrieval.
|
|
700
|
+
- `QUESTION_ANSWERING`: Optimized for answering questions.
|
|
701
|
+
- `FACT_VERIFICATION`: Optimized for verifying factual information.
|
|
702
|
+
- `CODE_RETRIEVAL_QUERY`: Optimized for retrieving code blocks based on natural language queries.
|
|
703
|
+
|
|
704
|
+
- **title**: _string_
|
|
705
|
+
|
|
706
|
+
Optional. The title of the document being embedded. This helps the model produce better embeddings by providing additional context. Only valid when `taskType` is set to `'RETRIEVAL_DOCUMENT'`.
|
|
707
|
+
|
|
708
|
+
- **autoTruncate**: _boolean_
|
|
709
|
+
|
|
710
|
+
Optional. When set to `true`, input text will be truncated if it exceeds the maximum length. When set to `false`, an error is returned if the input text is too long. Defaults to `true`.
|
|
711
|
+
|
|
712
|
+
#### Model Capabilities
|
|
713
|
+
|
|
714
|
+
| Model | Max Values Per Call | Parallel Calls |
|
|
715
|
+
| -------------------- | ------------------- | ------------------- |
|
|
716
|
+
| `text-embedding-004` | 2048 | <Check size={18} /> |
|
|
717
|
+
|
|
718
|
+
<Note>
|
|
719
|
+
The table above lists popular models. You can also pass any available provider
|
|
720
|
+
model ID as a string if needed.
|
|
721
|
+
</Note>
|
|
722
|
+
|
|
723
|
+
### Image Models
|
|
724
|
+
|
|
725
|
+
You can create [Imagen](https://cloud.google.com/vertex-ai/generative-ai/docs/image/overview) models that call the [Imagen on Vertex AI API](https://cloud.google.com/vertex-ai/generative-ai/docs/image/generate-images)
|
|
726
|
+
using the `.image()` factory method. For more on image generation with the AI SDK see [generateImage()](/docs/reference/ai-sdk-core/generate-image).
|
|
727
|
+
|
|
728
|
+
```ts
|
|
729
|
+
import { vertex } from '@ai-sdk/google-vertex';
|
|
730
|
+
import { generateImage } from 'ai';
|
|
731
|
+
|
|
732
|
+
const { image } = await generateImage({
|
|
733
|
+
model: vertex.image('imagen-4.0-generate-001'),
|
|
734
|
+
prompt: 'A futuristic cityscape at sunset',
|
|
735
|
+
aspectRatio: '16:9',
|
|
736
|
+
});
|
|
737
|
+
```
|
|
738
|
+
|
|
739
|
+
Further configuration can be done using Google Vertex provider options. You can validate the provider options using the `GoogleVertexImageProviderOptions` type.
|
|
740
|
+
|
|
741
|
+
```ts
|
|
742
|
+
import { vertex } from '@ai-sdk/google-vertex';
|
|
743
|
+
import { GoogleVertexImageProviderOptions } from '@ai-sdk/google-vertex';
|
|
744
|
+
import { generateImage } from 'ai';
|
|
745
|
+
|
|
746
|
+
const { image } = await generateImage({
|
|
747
|
+
model: vertex.image('imagen-4.0-generate-001'),
|
|
748
|
+
providerOptions: {
|
|
749
|
+
vertex: {
|
|
750
|
+
negativePrompt: 'pixelated, blurry, low-quality',
|
|
751
|
+
} satisfies GoogleVertexImageProviderOptions,
|
|
752
|
+
},
|
|
753
|
+
// ...
|
|
754
|
+
});
|
|
755
|
+
```
|
|
756
|
+
|
|
757
|
+
The following provider options are available:
|
|
758
|
+
|
|
759
|
+
- **negativePrompt** _string_
|
|
760
|
+
A description of what to discourage in the generated images.
|
|
761
|
+
|
|
762
|
+
- **personGeneration** `allow_adult` | `allow_all` | `dont_allow`
|
|
763
|
+
Whether to allow person generation. Defaults to `allow_adult`.
|
|
764
|
+
|
|
765
|
+
- **safetySetting** `block_low_and_above` | `block_medium_and_above` | `block_only_high` | `block_none`
|
|
766
|
+
Whether to block unsafe content. Defaults to `block_medium_and_above`.
|
|
767
|
+
|
|
768
|
+
- **addWatermark** _boolean_
|
|
769
|
+
Whether to add an invisible watermark to the generated images. Defaults to `true`.
|
|
770
|
+
|
|
771
|
+
- **storageUri** _string_
|
|
772
|
+
Cloud Storage URI to store the generated images.
|
|
773
|
+
|
|
774
|
+
<Note>
|
|
775
|
+
Imagen models do not support the `size` parameter. Use the `aspectRatio`
|
|
776
|
+
parameter instead.
|
|
777
|
+
</Note>
|
|
778
|
+
|
|
779
|
+
Additional information about the images can be retrieved using Google Vertex meta data.
|
|
780
|
+
|
|
781
|
+
```ts
|
|
782
|
+
import { vertex } from '@ai-sdk/google-vertex';
|
|
783
|
+
import { GoogleVertexImageProviderOptions } from '@ai-sdk/google-vertex';
|
|
784
|
+
import { generateImage } from 'ai';
|
|
785
|
+
|
|
786
|
+
const { image, providerMetadata } = await generateImage({
|
|
787
|
+
model: vertex.image('imagen-4.0-generate-001'),
|
|
788
|
+
prompt: 'A futuristic cityscape at sunset',
|
|
789
|
+
aspectRatio: '16:9',
|
|
790
|
+
});
|
|
791
|
+
|
|
792
|
+
console.log(
|
|
793
|
+
`Revised prompt: ${providerMetadata.vertex.images[0].revisedPrompt}`,
|
|
794
|
+
);
|
|
795
|
+
```
|
|
796
|
+
|
|
797
|
+
#### Image Editing
|
|
798
|
+
|
|
799
|
+
Google Vertex Imagen models support image editing through inpainting, outpainting, and other edit modes. Pass input images via `prompt.images` and optionally a mask via `prompt.mask`.
|
|
800
|
+
|
|
801
|
+
<Note>
|
|
802
|
+
Image editing is supported by `imagen-3.0-capability-001`. The
|
|
803
|
+
`imagen-4.0-generate-001` model does not currently support editing operations.
|
|
804
|
+
</Note>
|
|
805
|
+
|
|
806
|
+
##### Inpainting (Insert Objects)
|
|
807
|
+
|
|
808
|
+
Insert or replace objects in specific areas using a mask:
|
|
809
|
+
|
|
810
|
+
```ts
|
|
811
|
+
import {
|
|
812
|
+
vertex,
|
|
813
|
+
GoogleVertexImageProviderOptions,
|
|
814
|
+
} from '@ai-sdk/google-vertex';
|
|
815
|
+
import { generateImage } from 'ai';
|
|
816
|
+
import fs from 'fs';
|
|
817
|
+
|
|
818
|
+
const image = fs.readFileSync('./input-image.png');
|
|
819
|
+
const mask = fs.readFileSync('./mask.png'); // White = edit area
|
|
820
|
+
|
|
821
|
+
const { images } = await generateImage({
|
|
822
|
+
model: vertex.image('imagen-3.0-capability-001'),
|
|
823
|
+
prompt: {
|
|
824
|
+
text: 'A sunlit indoor lounge area with a pool containing a flamingo',
|
|
825
|
+
images: [image],
|
|
826
|
+
mask,
|
|
827
|
+
},
|
|
828
|
+
providerOptions: {
|
|
829
|
+
vertex: {
|
|
830
|
+
edit: {
|
|
831
|
+
baseSteps: 50,
|
|
832
|
+
mode: 'EDIT_MODE_INPAINT_INSERTION',
|
|
833
|
+
maskMode: 'MASK_MODE_USER_PROVIDED',
|
|
834
|
+
maskDilation: 0.01,
|
|
835
|
+
},
|
|
836
|
+
} satisfies GoogleVertexImageProviderOptions,
|
|
837
|
+
},
|
|
838
|
+
});
|
|
839
|
+
```
|
|
840
|
+
|
|
841
|
+
##### Outpainting (Extend Image)
|
|
842
|
+
|
|
843
|
+
Extend an image beyond its original boundaries:
|
|
844
|
+
|
|
845
|
+
```ts
|
|
846
|
+
import {
|
|
847
|
+
vertex,
|
|
848
|
+
GoogleVertexImageProviderOptions,
|
|
849
|
+
} from '@ai-sdk/google-vertex';
|
|
850
|
+
import { generateImage } from 'ai';
|
|
851
|
+
import fs from 'fs';
|
|
852
|
+
|
|
853
|
+
const image = fs.readFileSync('./input-image.png');
|
|
854
|
+
const mask = fs.readFileSync('./outpaint-mask.png'); // White = extend area
|
|
855
|
+
|
|
856
|
+
const { images } = await generateImage({
|
|
857
|
+
model: vertex.image('imagen-3.0-capability-001'),
|
|
858
|
+
prompt: {
|
|
859
|
+
text: 'Extend the scene with more of the forest background',
|
|
860
|
+
images: [image],
|
|
861
|
+
mask,
|
|
862
|
+
},
|
|
863
|
+
providerOptions: {
|
|
864
|
+
vertex: {
|
|
865
|
+
edit: {
|
|
866
|
+
baseSteps: 50,
|
|
867
|
+
mode: 'EDIT_MODE_OUTPAINT',
|
|
868
|
+
maskMode: 'MASK_MODE_USER_PROVIDED',
|
|
869
|
+
},
|
|
870
|
+
} satisfies GoogleVertexImageProviderOptions,
|
|
871
|
+
},
|
|
872
|
+
});
|
|
873
|
+
```
|
|
874
|
+
|
|
875
|
+
##### Edit Provider Options
|
|
876
|
+
|
|
877
|
+
The following options are available under `providerOptions.vertex.edit`:
|
|
878
|
+
|
|
879
|
+
- **mode** - The edit mode to use:
|
|
880
|
+
|
|
881
|
+
- `EDIT_MODE_INPAINT_INSERTION` - Insert objects into masked areas
|
|
882
|
+
- `EDIT_MODE_INPAINT_REMOVAL` - Remove objects from masked areas
|
|
883
|
+
- `EDIT_MODE_OUTPAINT` - Extend image beyond boundaries
|
|
884
|
+
- `EDIT_MODE_CONTROLLED_EDITING` - Controlled editing
|
|
885
|
+
- `EDIT_MODE_PRODUCT_IMAGE` - Product image editing
|
|
886
|
+
- `EDIT_MODE_BGSWAP` - Background swap
|
|
887
|
+
|
|
888
|
+
- **baseSteps** _number_ - Number of sampling steps (35-75). Higher values = better quality but slower.
|
|
889
|
+
|
|
890
|
+
- **maskMode** - How to interpret the mask:
|
|
891
|
+
|
|
892
|
+
- `MASK_MODE_USER_PROVIDED` - Use the provided mask directly
|
|
893
|
+
- `MASK_MODE_DEFAULT` - Default mask mode
|
|
894
|
+
- `MASK_MODE_DETECTION_BOX` - Mask from detected bounding boxes
|
|
895
|
+
- `MASK_MODE_CLOTHING_AREA` - Mask from clothing segmentation
|
|
896
|
+
- `MASK_MODE_PARSED_PERSON` - Mask from person parsing
|
|
897
|
+
|
|
898
|
+
- **maskDilation** _number_ - Percentage (0-1) to grow the mask. Recommended: 0.01.
|
|
899
|
+
|
|
900
|
+
<Note>
|
|
901
|
+
Input images must be provided as `Buffer`, `ArrayBuffer`, `Uint8Array`, or
|
|
902
|
+
base64-encoded strings. URL-based images are not supported for Google Vertex
|
|
903
|
+
image editing.
|
|
904
|
+
</Note>
|
|
905
|
+
|
|
906
|
+
#### Model Capabilities
|
|
907
|
+
|
|
908
|
+
| Model | Aspect Ratios |
|
|
909
|
+
| ------------------------------- | ------------------------- |
|
|
910
|
+
| `imagen-3.0-generate-001` | 1:1, 3:4, 4:3, 9:16, 16:9 |
|
|
911
|
+
| `imagen-3.0-generate-002` | 1:1, 3:4, 4:3, 9:16, 16:9 |
|
|
912
|
+
| `imagen-3.0-fast-generate-001` | 1:1, 3:4, 4:3, 9:16, 16:9 |
|
|
913
|
+
| `imagen-4.0-generate-001` | 1:1, 3:4, 4:3, 9:16, 16:9 |
|
|
914
|
+
| `imagen-4.0-fast-generate-001` | 1:1, 3:4, 4:3, 9:16, 16:9 |
|
|
915
|
+
| `imagen-4.0-ultra-generate-001` | 1:1, 3:4, 4:3, 9:16, 16:9 |
|
|
916
|
+
|
|
917
|
+
## Google Vertex Anthropic Provider Usage
|
|
918
|
+
|
|
919
|
+
The Google Vertex Anthropic provider for the [AI SDK](/docs) offers support for Anthropic's Claude models through the Google Vertex AI APIs. This section provides details on how to set up and use the Google Vertex Anthropic provider.
|
|
920
|
+
|
|
921
|
+
### Provider Instance
|
|
922
|
+
|
|
923
|
+
You can import the default provider instance `vertexAnthropic` from `@ai-sdk/google-vertex/anthropic`:
|
|
924
|
+
|
|
925
|
+
```typescript
|
|
926
|
+
import { vertexAnthropic } from '@ai-sdk/google-vertex/anthropic';
|
|
927
|
+
```
|
|
928
|
+
|
|
929
|
+
If you need a customized setup, you can import `createVertexAnthropic` from `@ai-sdk/google-vertex/anthropic` and create a provider instance with your settings:
|
|
930
|
+
|
|
931
|
+
```typescript
|
|
932
|
+
import { createVertexAnthropic } from '@ai-sdk/google-vertex/anthropic';
|
|
933
|
+
|
|
934
|
+
const vertexAnthropic = createVertexAnthropic({
|
|
935
|
+
project: 'my-project', // optional
|
|
936
|
+
location: 'us-central1', // optional
|
|
937
|
+
});
|
|
938
|
+
```
|
|
939
|
+
|
|
940
|
+
#### Node.js Runtime
|
|
941
|
+
|
|
942
|
+
For Node.js environments, the Google Vertex Anthropic provider supports all standard Google Cloud authentication options through the `google-auth-library`. You can customize the authentication options by passing them to the `createVertexAnthropic` function:
|
|
943
|
+
|
|
944
|
+
```typescript
|
|
945
|
+
import { createVertexAnthropic } from '@ai-sdk/google-vertex/anthropic';
|
|
946
|
+
|
|
947
|
+
const vertexAnthropic = createVertexAnthropic({
|
|
948
|
+
googleAuthOptions: {
|
|
949
|
+
credentials: {
|
|
950
|
+
client_email: 'my-email',
|
|
951
|
+
private_key: 'my-private-key',
|
|
952
|
+
},
|
|
953
|
+
},
|
|
954
|
+
});
|
|
955
|
+
```
|
|
956
|
+
|
|
957
|
+
##### Optional Provider Settings
|
|
958
|
+
|
|
959
|
+
You can use the following optional settings to customize the Google Vertex Anthropic provider instance:
|
|
960
|
+
|
|
961
|
+
- **project** _string_
|
|
962
|
+
|
|
963
|
+
The Google Cloud project ID that you want to use for the API calls.
|
|
964
|
+
It uses the `GOOGLE_VERTEX_PROJECT` environment variable by default.
|
|
965
|
+
|
|
966
|
+
- **location** _string_
|
|
967
|
+
|
|
968
|
+
The Google Cloud location that you want to use for the API calls, e.g. `us-central1`.
|
|
969
|
+
It uses the `GOOGLE_VERTEX_LOCATION` environment variable by default.
|
|
970
|
+
|
|
971
|
+
- **googleAuthOptions** _object_
|
|
972
|
+
|
|
973
|
+
Optional. The Authentication options used by the [Google Auth Library](https://github.com/googleapis/google-auth-library-nodejs/). See also the [GoogleAuthOptions](https://github.com/googleapis/google-auth-library-nodejs/blob/08978822e1b7b5961f0e355df51d738e012be392/src/auth/googleauth.ts#L87C18-L87C35) interface.
|
|
974
|
+
|
|
975
|
+
- **authClient** _object_
|
|
976
|
+
An `AuthClient` to use.
|
|
977
|
+
|
|
978
|
+
- **keyFilename** _string_
|
|
979
|
+
Path to a .json, .pem, or .p12 key file.
|
|
980
|
+
|
|
981
|
+
- **keyFile** _string_
|
|
982
|
+
Path to a .json, .pem, or .p12 key file.
|
|
983
|
+
|
|
984
|
+
- **credentials** _object_
|
|
985
|
+
Object containing client_email and private_key properties, or the external account client options.
|
|
986
|
+
|
|
987
|
+
- **clientOptions** _object_
|
|
988
|
+
Options object passed to the constructor of the client.
|
|
989
|
+
|
|
990
|
+
- **scopes** _string | string[]_
|
|
991
|
+
Required scopes for the desired API request.
|
|
992
|
+
|
|
993
|
+
- **projectId** _string_
|
|
994
|
+
Your project ID.
|
|
995
|
+
|
|
996
|
+
- **universeDomain** _string_
|
|
997
|
+
The default service domain for a given Cloud universe.
|
|
998
|
+
|
|
999
|
+
- **headers** _Resolvable<Record<string, string | undefined>>_
|
|
1000
|
+
|
|
1001
|
+
Headers to include in the requests. Can be provided in multiple formats:
|
|
1002
|
+
|
|
1003
|
+
- A record of header key-value pairs: `Record<string, string | undefined>`
|
|
1004
|
+
- A function that returns headers: `() => Record<string, string | undefined>`
|
|
1005
|
+
- An async function that returns headers: `async () => Record<string, string | undefined>`
|
|
1006
|
+
- A promise that resolves to headers: `Promise<Record<string, string | undefined>>`
|
|
1007
|
+
|
|
1008
|
+
- **fetch** _(input: RequestInfo, init?: RequestInit) => Promise<Response>_
|
|
1009
|
+
|
|
1010
|
+
Custom [fetch](https://developer.mozilla.org/en-US/docs/Web/API/fetch) implementation.
|
|
1011
|
+
Defaults to the global `fetch` function.
|
|
1012
|
+
You can use it as a middleware to intercept requests,
|
|
1013
|
+
or to provide a custom fetch implementation for e.g. testing.
|
|
1014
|
+
|
|
1015
|
+
<a id="google-vertex-anthropic-edge-runtime"></a>
|
|
1016
|
+
#### Edge Runtime
|
|
1017
|
+
|
|
1018
|
+
Edge runtimes (like Vercel Edge Functions and Cloudflare Workers) are lightweight JavaScript environments that run closer to users at the network edge.
|
|
1019
|
+
They only provide a subset of the standard Node.js APIs.
|
|
1020
|
+
For example, direct file system access is not available, and many Node.js-specific libraries
|
|
1021
|
+
(including the standard Google Auth library) are not compatible.
|
|
1022
|
+
|
|
1023
|
+
The Edge runtime version of the Google Vertex Anthropic provider supports Google's [Application Default Credentials](https://github.com/googleapis/google-auth-library-nodejs?tab=readme-ov-file#application-default-credentials) through environment variables. The values can be obtained from a json credentials file from the [Google Cloud Console](https://console.cloud.google.com/apis/credentials).
|
|
1024
|
+
|
|
1025
|
+
For Edge runtimes, you can import the provider instance from `@ai-sdk/google-vertex/anthropic/edge`:
|
|
1026
|
+
|
|
1027
|
+
```typescript
|
|
1028
|
+
import { vertexAnthropic } from '@ai-sdk/google-vertex/anthropic/edge';
|
|
1029
|
+
```
|
|
1030
|
+
|
|
1031
|
+
To customize the setup, use `createVertexAnthropic` from the same module:
|
|
1032
|
+
|
|
1033
|
+
```typescript
|
|
1034
|
+
import { createVertexAnthropic } from '@ai-sdk/google-vertex/anthropic/edge';
|
|
1035
|
+
|
|
1036
|
+
const vertexAnthropic = createVertexAnthropic({
|
|
1037
|
+
project: 'my-project', // optional
|
|
1038
|
+
location: 'us-central1', // optional
|
|
1039
|
+
});
|
|
1040
|
+
```
|
|
1041
|
+
|
|
1042
|
+
For Edge runtime authentication, set these environment variables from your Google Default Application Credentials JSON file:
|
|
1043
|
+
|
|
1044
|
+
- `GOOGLE_CLIENT_EMAIL`
|
|
1045
|
+
- `GOOGLE_PRIVATE_KEY`
|
|
1046
|
+
- `GOOGLE_PRIVATE_KEY_ID` (optional)
|
|
1047
|
+
|
|
1048
|
+
##### Optional Provider Settings
|
|
1049
|
+
|
|
1050
|
+
You can use the following optional settings to customize the provider instance:
|
|
1051
|
+
|
|
1052
|
+
- **project** _string_
|
|
1053
|
+
|
|
1054
|
+
The Google Cloud project ID that you want to use for the API calls.
|
|
1055
|
+
It uses the `GOOGLE_VERTEX_PROJECT` environment variable by default.
|
|
1056
|
+
|
|
1057
|
+
- **location** _string_
|
|
1058
|
+
|
|
1059
|
+
The Google Cloud location that you want to use for the API calls, e.g. `us-central1`.
|
|
1060
|
+
It uses the `GOOGLE_VERTEX_LOCATION` environment variable by default.
|
|
1061
|
+
|
|
1062
|
+
- **googleCredentials** _object_
|
|
1063
|
+
|
|
1064
|
+
Optional. The credentials used by the Edge provider for authentication. These credentials are typically set through environment variables and are derived from a service account JSON file.
|
|
1065
|
+
|
|
1066
|
+
- **clientEmail** _string_
|
|
1067
|
+
The client email from the service account JSON file. Defaults to the contents of the `GOOGLE_CLIENT_EMAIL` environment variable.
|
|
1068
|
+
|
|
1069
|
+
- **privateKey** _string_
|
|
1070
|
+
The private key from the service account JSON file. Defaults to the contents of the `GOOGLE_PRIVATE_KEY` environment variable.
|
|
1071
|
+
|
|
1072
|
+
- **privateKeyId** _string_
|
|
1073
|
+
The private key ID from the service account JSON file (optional). Defaults to the contents of the `GOOGLE_PRIVATE_KEY_ID` environment variable.
|
|
1074
|
+
|
|
1075
|
+
- **headers** _Resolvable<Record<string, string | undefined>>_
|
|
1076
|
+
|
|
1077
|
+
Headers to include in the requests. Can be provided in multiple formats:
|
|
1078
|
+
|
|
1079
|
+
- A record of header key-value pairs: `Record<string, string | undefined>`
|
|
1080
|
+
- A function that returns headers: `() => Record<string, string | undefined>`
|
|
1081
|
+
- An async function that returns headers: `async () => Record<string, string | undefined>`
|
|
1082
|
+
- A promise that resolves to headers: `Promise<Record<string, string | undefined>>`
|
|
1083
|
+
|
|
1084
|
+
- **fetch** _(input: RequestInfo, init?: RequestInit) => Promise<Response>_
|
|
1085
|
+
|
|
1086
|
+
Custom [fetch](https://developer.mozilla.org/en-US/docs/Web/API/fetch) implementation.
|
|
1087
|
+
Defaults to the global `fetch` function.
|
|
1088
|
+
You can use it as a middleware to intercept requests,
|
|
1089
|
+
or to provide a custom fetch implementation for e.g. testing.
|
|
1090
|
+
|
|
1091
|
+
### Language Models
|
|
1092
|
+
|
|
1093
|
+
You can create models that call the [Anthropic Messages API](https://docs.anthropic.com/claude/reference/messages_post) using the provider instance.
|
|
1094
|
+
The first argument is the model id, e.g. `claude-3-haiku-20240307`.
|
|
1095
|
+
Some models have multi-modal capabilities.
|
|
1096
|
+
|
|
1097
|
+
```ts
|
|
1098
|
+
const model = anthropic('claude-3-haiku-20240307');
|
|
1099
|
+
```
|
|
1100
|
+
|
|
1101
|
+
You can use Anthropic language models to generate text with the `generateText` function:
|
|
1102
|
+
|
|
1103
|
+
```ts
|
|
1104
|
+
import { vertexAnthropic } from '@ai-sdk/google-vertex/anthropic';
|
|
1105
|
+
import { generateText } from 'ai';
|
|
1106
|
+
|
|
1107
|
+
const { text } = await generateText({
|
|
1108
|
+
model: vertexAnthropic('claude-3-haiku-20240307'),
|
|
1109
|
+
prompt: 'Write a vegetarian lasagna recipe for 4 people.',
|
|
1110
|
+
});
|
|
1111
|
+
```
|
|
1112
|
+
|
|
1113
|
+
Anthropic language models can also be used in the `streamText`, `generateObject`, and `streamObject` functions
|
|
1114
|
+
(see [AI SDK Core](/docs/ai-sdk-core)).
|
|
1115
|
+
|
|
1116
|
+
<Note>
|
|
1117
|
+
The Anthropic API returns streaming tool calls all at once after a delay. This
|
|
1118
|
+
causes the `streamObject` function to generate the object fully after a delay
|
|
1119
|
+
instead of streaming it incrementally.
|
|
1120
|
+
</Note>
|
|
1121
|
+
|
|
1122
|
+
The following optional provider options are available for Anthropic models:
|
|
1123
|
+
|
|
1124
|
+
- `sendReasoning` _boolean_
|
|
1125
|
+
|
|
1126
|
+
Optional. Include reasoning content in requests sent to the model. Defaults to `true`.
|
|
1127
|
+
|
|
1128
|
+
If you are experiencing issues with the model handling requests involving
|
|
1129
|
+
reasoning content, you can set this to `false` to omit them from the request.
|
|
1130
|
+
|
|
1131
|
+
- `thinking` _object_
|
|
1132
|
+
|
|
1133
|
+
Optional. See [Reasoning section](#reasoning) for more details.
|
|
1134
|
+
|
|
1135
|
+
### Reasoning
|
|
1136
|
+
|
|
1137
|
+
Anthropic has reasoning support for the `claude-3-7-sonnet@20250219` model.
|
|
1138
|
+
|
|
1139
|
+
You can enable it using the `thinking` provider option
|
|
1140
|
+
and specifying a thinking budget in tokens.
|
|
1141
|
+
|
|
1142
|
+
```ts
|
|
1143
|
+
import { vertexAnthropic } from '@ai-sdk/google-vertex/anthropic';
|
|
1144
|
+
import { generateText } from 'ai';
|
|
1145
|
+
|
|
1146
|
+
const { text, reasoningText, reasoning } = await generateText({
|
|
1147
|
+
model: vertexAnthropic('claude-3-7-sonnet@20250219'),
|
|
1148
|
+
prompt: 'How many people will live in the world in 2040?',
|
|
1149
|
+
providerOptions: {
|
|
1150
|
+
anthropic: {
|
|
1151
|
+
thinking: { type: 'enabled', budgetTokens: 12000 },
|
|
1152
|
+
},
|
|
1153
|
+
},
|
|
1154
|
+
});
|
|
1155
|
+
|
|
1156
|
+
console.log(reasoningText); // reasoning text
|
|
1157
|
+
console.log(reasoning); // reasoning details including redacted reasoning
|
|
1158
|
+
console.log(text); // text response
|
|
1159
|
+
```
|
|
1160
|
+
|
|
1161
|
+
See [AI SDK UI: Chatbot](/docs/ai-sdk-ui/chatbot#reasoning) for more details
|
|
1162
|
+
on how to integrate reasoning into your chatbot.
|
|
1163
|
+
|
|
1164
|
+
#### Cache Control
|
|
1165
|
+
|
|
1166
|
+
<Note>
|
|
1167
|
+
Anthropic cache control is in a Pre-Generally Available (GA) state on Google
|
|
1168
|
+
Vertex. For more see [Google Vertex Anthropic cache control
|
|
1169
|
+
documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude-prompt-caching).
|
|
1170
|
+
</Note>
|
|
1171
|
+
|
|
1172
|
+
In the messages and message parts, you can use the `providerOptions` property to set cache control breakpoints.
|
|
1173
|
+
You need to set the `anthropic` property in the `providerOptions` object to `{ cacheControl: { type: 'ephemeral' } }` to set a cache control breakpoint.
|
|
1174
|
+
|
|
1175
|
+
The cache creation input tokens are then returned in the `providerMetadata` object
|
|
1176
|
+
for `generateText` and `generateObject`, again under the `anthropic` property.
|
|
1177
|
+
When you use `streamText` or `streamObject`, the response contains a promise
|
|
1178
|
+
that resolves to the metadata. Alternatively you can receive it in the
|
|
1179
|
+
`onFinish` callback.
|
|
1180
|
+
|
|
1181
|
+
```ts highlight="8,18-20,29-30"
|
|
1182
|
+
import { vertexAnthropic } from '@ai-sdk/google-vertex/anthropic';
|
|
1183
|
+
import { generateText } from 'ai';
|
|
1184
|
+
|
|
1185
|
+
const errorMessage = '... long error message ...';
|
|
1186
|
+
|
|
1187
|
+
const result = await generateText({
|
|
1188
|
+
model: vertexAnthropic('claude-3-5-sonnet-20240620'),
|
|
1189
|
+
messages: [
|
|
1190
|
+
{
|
|
1191
|
+
role: 'user',
|
|
1192
|
+
content: [
|
|
1193
|
+
{ type: 'text', text: 'You are a JavaScript expert.' },
|
|
1194
|
+
{
|
|
1195
|
+
type: 'text',
|
|
1196
|
+
text: `Error message: ${errorMessage}`,
|
|
1197
|
+
providerOptions: {
|
|
1198
|
+
anthropic: { cacheControl: { type: 'ephemeral' } },
|
|
1199
|
+
},
|
|
1200
|
+
},
|
|
1201
|
+
{ type: 'text', text: 'Explain the error message.' },
|
|
1202
|
+
],
|
|
1203
|
+
},
|
|
1204
|
+
],
|
|
1205
|
+
});
|
|
1206
|
+
|
|
1207
|
+
console.log(result.text);
|
|
1208
|
+
console.log(result.providerMetadata?.anthropic);
|
|
1209
|
+
// e.g. { cacheCreationInputTokens: 2118, cacheReadInputTokens: 0 }
|
|
1210
|
+
```
|
|
1211
|
+
|
|
1212
|
+
You can also use cache control on system messages by providing multiple system messages at the head of your messages array:
|
|
1213
|
+
|
|
1214
|
+
```ts highlight="3,9-11"
|
|
1215
|
+
const result = await generateText({
|
|
1216
|
+
model: vertexAnthropic('claude-3-5-sonnet-20240620'),
|
|
1217
|
+
messages: [
|
|
1218
|
+
{
|
|
1219
|
+
role: 'system',
|
|
1220
|
+
content: 'Cached system message part',
|
|
1221
|
+
providerOptions: {
|
|
1222
|
+
anthropic: { cacheControl: { type: 'ephemeral' } },
|
|
1223
|
+
},
|
|
1224
|
+
},
|
|
1225
|
+
{
|
|
1226
|
+
role: 'system',
|
|
1227
|
+
content: 'Uncached system message part',
|
|
1228
|
+
},
|
|
1229
|
+
{
|
|
1230
|
+
role: 'user',
|
|
1231
|
+
content: 'User prompt',
|
|
1232
|
+
},
|
|
1233
|
+
],
|
|
1234
|
+
});
|
|
1235
|
+
```
|
|
1236
|
+
|
|
1237
|
+
For more on prompt caching with Anthropic, see [Google Vertex AI's Claude prompt caching documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude-prompt-caching) and [Anthropic's Cache Control documentation](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching).
|
|
1238
|
+
|
|
1239
|
+
### Tools
|
|
1240
|
+
|
|
1241
|
+
Google Vertex Anthropic supports a subset of Anthropic's built-in tools. The following tools are available via the `tools` property of the provider instance:
|
|
1242
|
+
|
|
1243
|
+
1. **Bash Tool**: Allows running bash commands.
|
|
1244
|
+
2. **Text Editor Tool**: Provides functionality for viewing and editing text files.
|
|
1245
|
+
3. **Computer Tool**: Enables control of keyboard and mouse actions on a computer.
|
|
1246
|
+
4. **Web Search Tool**: Provides access to real-time web content.
|
|
1247
|
+
|
|
1248
|
+
<Note>
|
|
1249
|
+
Only a subset of Anthropic tools are supported on Google Vertex. Tools like
|
|
1250
|
+
Code Execution, Memory, and Web Fetch are not available. Use the regular
|
|
1251
|
+
`@ai-sdk/anthropic` provider if you need access to all Anthropic tools.
|
|
1252
|
+
</Note>
|
|
1253
|
+
|
|
1254
|
+
For more background on Anthropic tools, see [Anthropic's documentation](https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview).
|
|
1255
|
+
|
|
1256
|
+
#### Bash Tool
|
|
1257
|
+
|
|
1258
|
+
The Bash Tool allows running bash commands. Here's how to create and use it:
|
|
1259
|
+
|
|
1260
|
+
```ts
|
|
1261
|
+
const bashTool = vertexAnthropic.tools.bash_20250124({
|
|
1262
|
+
execute: async ({ command, restart }) => {
|
|
1263
|
+
// Implement your bash command execution logic here
|
|
1264
|
+
// Return the result of the command execution
|
|
1265
|
+
},
|
|
1266
|
+
});
|
|
1267
|
+
```
|
|
1268
|
+
|
|
1269
|
+
Parameters:
|
|
1270
|
+
|
|
1271
|
+
- `command` (string): The bash command to run. Required unless the tool is being restarted.
|
|
1272
|
+
- `restart` (boolean, optional): Specifying true will restart this tool.
|
|
1273
|
+
|
|
1274
|
+
#### Text Editor Tool
|
|
1275
|
+
|
|
1276
|
+
The Text Editor Tool provides functionality for viewing and editing text files:
|
|
1277
|
+
|
|
1278
|
+
```ts
|
|
1279
|
+
const textEditorTool = vertexAnthropic.tools.textEditor_20250124({
|
|
1280
|
+
execute: async ({
|
|
1281
|
+
command,
|
|
1282
|
+
path,
|
|
1283
|
+
file_text,
|
|
1284
|
+
insert_line,
|
|
1285
|
+
new_str,
|
|
1286
|
+
old_str,
|
|
1287
|
+
view_range,
|
|
1288
|
+
}) => {
|
|
1289
|
+
// Implement your text editing logic here
|
|
1290
|
+
// Return the result of the text editing operation
|
|
1291
|
+
},
|
|
1292
|
+
});
|
|
1293
|
+
```
|
|
1294
|
+
|
|
1295
|
+
Parameters:
|
|
1296
|
+
|
|
1297
|
+
- `command` ('view' | 'create' | 'str_replace' | 'insert' | 'undo_edit'): The command to run. Note: `undo_edit` is not supported in `textEditor_20250429` and `textEditor_20250728`.
|
|
1298
|
+
- `path` (string): Absolute path to file or directory, e.g. `/repo/file.py` or `/repo`.
|
|
1299
|
+
- `file_text` (string, optional): Required for `create` command, with the content of the file to be created.
|
|
1300
|
+
- `insert_line` (number, optional): Required for `insert` command. The line number after which to insert the new string.
|
|
1301
|
+
- `new_str` (string, optional): New string for `str_replace` or `insert` commands.
|
|
1302
|
+
- `old_str` (string, optional): Required for `str_replace` command, containing the string to replace.
|
|
1303
|
+
- `view_range` (number[], optional): Optional for `view` command to specify line range to show.
|
|
1304
|
+
- `max_characters` (number, optional): Optional maximum number of characters to view in the file (only available in `textEditor_20250728`).
|
|
1305
|
+
|
|
1306
|
+
#### Computer Tool
|
|
1307
|
+
|
|
1308
|
+
The Computer Tool enables control of keyboard and mouse actions on a computer:
|
|
1309
|
+
|
|
1310
|
+
```ts
|
|
1311
|
+
const computerTool = vertexAnthropic.tools.computer_20241022({
|
|
1312
|
+
displayWidthPx: 1920,
|
|
1313
|
+
displayHeightPx: 1080,
|
|
1314
|
+
displayNumber: 0, // Optional, for X11 environments
|
|
1315
|
+
|
|
1316
|
+
execute: async ({ action, coordinate, text }) => {
|
|
1317
|
+
// Implement your computer control logic here
|
|
1318
|
+
// Return the result of the action
|
|
1319
|
+
|
|
1320
|
+
// Example code:
|
|
1321
|
+
switch (action) {
|
|
1322
|
+
case 'screenshot': {
|
|
1323
|
+
// multipart result:
|
|
1324
|
+
return {
|
|
1325
|
+
type: 'image',
|
|
1326
|
+
data: fs
|
|
1327
|
+
.readFileSync('./data/screenshot-editor.png')
|
|
1328
|
+
.toString('base64'),
|
|
1329
|
+
};
|
|
1330
|
+
}
|
|
1331
|
+
default: {
|
|
1332
|
+
console.log('Action:', action);
|
|
1333
|
+
console.log('Coordinate:', coordinate);
|
|
1334
|
+
console.log('Text:', text);
|
|
1335
|
+
return `executed ${action}`;
|
|
1336
|
+
}
|
|
1337
|
+
}
|
|
1338
|
+
},
|
|
1339
|
+
|
|
1340
|
+
// map to tool result content for LLM consumption:
|
|
1341
|
+
toModelOutput({ output }) {
|
|
1342
|
+
return typeof output === 'string'
|
|
1343
|
+
? [{ type: 'text', text: output }]
|
|
1344
|
+
: [{ type: 'image', data: output.data, mediaType: 'image/png' }];
|
|
1345
|
+
},
|
|
1346
|
+
});
|
|
1347
|
+
```
|
|
1348
|
+
|
|
1349
|
+
Parameters:
|
|
1350
|
+
|
|
1351
|
+
- `action` ('key' | 'type' | 'mouse_move' | 'left_click' | 'left_click_drag' | 'right_click' | 'middle_click' | 'double_click' | 'screenshot' | 'cursor_position'): The action to perform.
|
|
1352
|
+
- `coordinate` (number[], optional): Required for `mouse_move` and `left_click_drag` actions. Specifies the (x, y) coordinates.
|
|
1353
|
+
- `text` (string, optional): Required for `type` and `key` actions.
|
|
1354
|
+
|
|
1355
|
+
#### Web Search Tool
|
|
1356
|
+
|
|
1357
|
+
The Web Search Tool provides Claude with direct access to real-time web content:
|
|
1358
|
+
|
|
1359
|
+
```ts
|
|
1360
|
+
const webSearchTool = vertexAnthropic.tools.webSearch_20250305({
|
|
1361
|
+
maxUses: 5, // Optional: Maximum number of web searches Claude can perform
|
|
1362
|
+
allowedDomains: ['example.com'], // Optional: Only search these domains
|
|
1363
|
+
blockedDomains: ['spam.com'], // Optional: Never search these domains
|
|
1364
|
+
userLocation: {
|
|
1365
|
+
// Optional: Provide location for geographically relevant results
|
|
1366
|
+
type: 'approximate',
|
|
1367
|
+
city: 'San Francisco',
|
|
1368
|
+
region: 'CA',
|
|
1369
|
+
country: 'US',
|
|
1370
|
+
timezone: 'America/Los_Angeles',
|
|
1371
|
+
},
|
|
1372
|
+
});
|
|
1373
|
+
```
|
|
1374
|
+
|
|
1375
|
+
Parameters:
|
|
1376
|
+
|
|
1377
|
+
- `maxUses` (number, optional): Maximum number of web searches Claude can perform during the conversation.
|
|
1378
|
+
- `allowedDomains` (string[], optional): Optional list of domains that Claude is allowed to search.
|
|
1379
|
+
- `blockedDomains` (string[], optional): Optional list of domains that Claude should avoid when searching.
|
|
1380
|
+
- `userLocation` (object, optional): Optional user location information to provide geographically relevant search results.
|
|
1381
|
+
- `type` ('approximate'): The type of location (must be approximate).
|
|
1382
|
+
- `city` (string, optional): The city name.
|
|
1383
|
+
- `region` (string, optional): The region or state.
|
|
1384
|
+
- `country` (string, optional): The country.
|
|
1385
|
+
- `timezone` (string, optional): The IANA timezone ID.
|
|
1386
|
+
|
|
1387
|
+
These tools can be used in conjunction with supported Claude models to enable more complex interactions and tasks.
|
|
1388
|
+
|
|
1389
|
+
### Model Capabilities
|
|
1390
|
+
|
|
1391
|
+
The latest Anthropic model list on Vertex AI is available [here](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-claude#model-list).
|
|
1392
|
+
See also [Anthropic Model Comparison](https://docs.anthropic.com/en/docs/about-claude/models#model-comparison).
|
|
1393
|
+
|
|
1394
|
+
| Model | Image Input | Object Generation | Tool Usage | Tool Streaming | Computer Use |
|
|
1395
|
+
| ------------------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- |
|
|
1396
|
+
| `claude-3-7-sonnet@20250219` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
|
|
1397
|
+
| `claude-3-5-sonnet-v2@20241022` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
|
|
1398
|
+
| `claude-3-5-sonnet@20240620` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Cross size={18} /> |
|
|
1399
|
+
| `claude-3-5-haiku@20241022` | <Cross size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Cross size={18} /> |
|
|
1400
|
+
| `claude-3-sonnet@20240229` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Cross size={18} /> |
|
|
1401
|
+
| `claude-3-haiku@20240307` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Cross size={18} /> |
|
|
1402
|
+
| `claude-3-opus@20240229` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Cross size={18} /> |
|
|
1403
|
+
|
|
1404
|
+
<Note>
|
|
1405
|
+
The table above lists popular models. You can also pass any available provider
|
|
1406
|
+
model ID as a string if needed.
|
|
1407
|
+
</Note>
|