zuplo 7.1.1 → 7.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (36) hide show
  1. package/docs/ai-gateway/apps.mdx +28 -10
  2. package/docs/ai-gateway/cookbooks/custom-fallback.mdx +96 -0
  3. package/docs/ai-gateway/cookbooks/dynamic-model-routing.mdx +119 -0
  4. package/docs/ai-gateway/custom-policies.mdx +213 -0
  5. package/docs/ai-gateway/custom-providers.mdx +11 -2
  6. package/docs/ai-gateway/fallback.mdx +48 -75
  7. package/docs/ai-gateway/getting-started.mdx +202 -133
  8. package/docs/ai-gateway/integrations/ai-sdk.mdx +71 -28
  9. package/docs/ai-gateway/integrations/claude-code.mdx +32 -8
  10. package/docs/ai-gateway/integrations/codex.mdx +28 -10
  11. package/docs/ai-gateway/integrations/goose.mdx +26 -19
  12. package/docs/ai-gateway/integrations/langchain.mdx +31 -15
  13. package/docs/ai-gateway/integrations/openai.mdx +28 -16
  14. package/docs/ai-gateway/introduction.mdx +73 -32
  15. package/docs/ai-gateway/managing-apps.mdx +62 -17
  16. package/docs/ai-gateway/managing-providers.mdx +26 -8
  17. package/docs/ai-gateway/managing-teams.mdx +29 -12
  18. package/docs/ai-gateway/policies/akamai-ai-firewall.mdx +92 -96
  19. package/docs/ai-gateway/policies/authentication.mdx +44 -0
  20. package/docs/ai-gateway/policies/comet-opik-tracing.mdx +40 -2
  21. package/docs/ai-gateway/policies/fallback-model.mdx +40 -0
  22. package/docs/ai-gateway/policies/galileo-tracing.mdx +36 -22
  23. package/docs/ai-gateway/policies/metering.mdx +47 -0
  24. package/docs/ai-gateway/policies/model-filtering.mdx +49 -0
  25. package/docs/ai-gateway/policies/overview.mdx +28 -0
  26. package/docs/ai-gateway/policies/semantic-cache.mdx +48 -0
  27. package/docs/ai-gateway/policy-chains.mdx +178 -0
  28. package/docs/ai-gateway/policy-templates.mdx +68 -0
  29. package/docs/ai-gateway/providers.mdx +47 -9
  30. package/docs/ai-gateway/source-control.mdx +88 -0
  31. package/docs/ai-gateway/teams.mdx +16 -10
  32. package/docs/ai-gateway/universal-api.mdx +39 -12
  33. package/docs/ai-gateway/usage-limits.mdx +87 -55
  34. package/docs/policies/cdn-cache-control-outbound/doc.md +13 -13
  35. package/package.json +5 -5
  36. package/docs/ai-gateway/guardrails.mdx +0 -65
@@ -1,22 +1,40 @@
1
1
  ---
2
2
  title: AI Gateway Apps
3
3
  sidebar_label: Overview
4
+ description:
5
+ Apps represent the services and integrations that call your AI Gateway. Each
6
+ app has its own URL, API key, and policy chain.
4
7
  ---
5
8
 
6
- Apps in the Zuplo AI Gateway represent any app or integration that will call the
7
- AI Gateway. For example, you might have a custom support chatbot on your
8
- website - that chatbot would be an App in the AI Gateway. Each App has its own
9
- API Key so that usage is tracked independently. Apps are owned by a specific
10
- [team](./teams.mdx) and can access the AI Providers assigned to that team.
9
+ An app represents one caller of your AI Gateway—a service, an agent, or a
10
+ feature in a larger codebase. A support chatbot on your website is one app; the
11
+ batch job that summarizes tickets overnight is another. Each app belongs to a
12
+ [team](./teams.mdx) and can reach the AI providers assigned to that team.
13
+
14
+ Each app has three things of its own:
15
+
16
+ - **An API URL**—the endpoint the app's code calls, shown at the top of the app
17
+ page. The gateway attributes requests made to this URL to the app, tracking
18
+ usage independently per app. See the [Universal API](./universal-api.mdx).
19
+ - **An API key**—authenticates the app's requests when the gateway runs the
20
+ [authentication policy](./policy-chains.mdx#authentication).
21
+ - **A [policy chain](./policy-chains.mdx)**—the ordered policies that run on the
22
+ app's requests: model access, budgets, caching, guardrails, and custom
23
+ policies. The chain starts out empty unless the app's team has a
24
+ [policy template](./policy-templates.mdx).
11
25
 
12
26
  ## API Keys
13
27
 
14
- Each App in the AI Gateway has its own API Key. This allows you to track usage
15
- independently for each App. You can find the API Key for an App by opening the
28
+ Each app has its own API key, which the gateway validates when the
29
+ [authentication policy](./policy-chains.mdx#authentication) applies. The key is
30
+ optional: the gateway attributes usage by app ID, resolving it either from a
31
+ validated key or from the `{app_id}` segment of the request URL, so an app
32
+ without authentication still tracks usage independently.
33
+
34
+ To find an app's API key, open the
16
35
  [Apps](https://portal.zuplo.com/+/account/project/ai/apps) tab of your AI
17
- Gateway project in the Zuplo Portal. Select the App you want to view the API Key
18
- for. With the App open you will see the API Key section at the top of the app
19
- page.
36
+ Gateway project in the Zuplo Portal and select the app. The API key appears in
37
+ the API Key section at the top of the app page.
20
38
 
21
39
  **Additional Resources**
22
40
 
@@ -0,0 +1,96 @@
1
+ ---
2
+ title: "Cookbook: Custom fallback logic"
3
+ sidebar_label: Custom fallback logic
4
+ description:
5
+ Write a custom policy that chooses a backup model dynamically—for example,
6
+ picking any active Anthropic model as the fallback—instead of a fixed entry in
7
+ the Fallback Model policy.
8
+ ---
9
+
10
+ The [Fallback Model policy](../fallback.mdx) covers a fixed backup and quota
11
+ fallback. When the backup should be chosen dynamically—the cheapest available
12
+ model, a provider-specific preference, whatever is currently active—a custom
13
+ [chain policy](../custom-policies.mdx) can replace it using the same public
14
+ primitives.
15
+
16
+ A fallback policy enriches the selection an earlier policy created. Because
17
+ `AIGatewayModelRouting.set()` replaces the complete stored selection rather than
18
+ partially updating it, the policy must read, merge, and set. It should also
19
+ leave the request unchanged when no earlier policy created a selection—so a
20
+ misplaced chain entry doesn't bypass model filtering—and when the chosen backup
21
+ equals the current main model, which `set()` rejects.
22
+
23
+ ## The policy
24
+
25
+ This example adds the first active Anthropic completions model as the backup,
26
+ with a 30-second timeout:
27
+
28
+ ```ts title="modules/add-fallback.ts"
29
+ import {
30
+ AIGatewayModelRouting,
31
+ AIGatewayModels,
32
+ ZuploContext,
33
+ ZuploRequest,
34
+ } from "@zuplo/runtime";
35
+
36
+ export default async function addFallback(
37
+ request: ZuploRequest,
38
+ context: ZuploContext,
39
+ ): Promise<ZuploRequest> {
40
+ const routing = AIGatewayModelRouting.get(context);
41
+ const current = routing?.completions;
42
+ if (!routing || !current) {
43
+ return request;
44
+ }
45
+ const target = typeof current === "string" ? { main: current } : current;
46
+
47
+ const providers = await AIGatewayModels.load(context);
48
+ const anthropic = providers.find(
49
+ (provider) => provider.providerName === "anthropic",
50
+ );
51
+ const fallback = anthropic?.models.find(
52
+ (candidate) =>
53
+ candidate.capability === "completions" && candidate.status === "active",
54
+ );
55
+ const backup = fallback ? `anthropic/${fallback.model}` : undefined;
56
+ if (!backup || backup.toLowerCase() === target.main.toLowerCase()) {
57
+ return request;
58
+ }
59
+
60
+ await AIGatewayModelRouting.set(context, {
61
+ ...routing,
62
+ completions: {
63
+ ...target,
64
+ backup,
65
+ fallbackTimeoutSeconds: 30,
66
+ },
67
+ });
68
+ return request;
69
+ }
70
+ ```
71
+
72
+ ## Declare and use it
73
+
74
+ ```json title="config/policies.json (one entry in the policies array)"
75
+ {
76
+ "name": "add-fallback",
77
+ "policyType": "custom-code-inbound",
78
+ "handler": {
79
+ "export": "default",
80
+ "module": "$import(./modules/add-fallback)"
81
+ }
82
+ }
83
+ ```
84
+
85
+ Add `add-fallback` to an app's chain directly after Model Filtering, replacing
86
+ the Fallback Model policy. Keep the no-selection guard at the top of the
87
+ policy—it prevents a misplaced entry from creating a primary selection and
88
+ bypassing filtering.
89
+
90
+ ## Next steps
91
+
92
+ - [Fallback Models](../fallback.mdx): the built-in fallback and quota-fallback
93
+ behavior
94
+ - [Cookbook: Dynamic model routing](./dynamic-model-routing.mdx): create the
95
+ primary selection from code
96
+ - [Custom Policies](../custom-policies.mdx): the full custom-policy quickstart
@@ -0,0 +1,119 @@
1
+ ---
2
+ title: "Cookbook: Dynamic model routing"
3
+ sidebar_label: Dynamic model routing
4
+ description:
5
+ Write a custom policy that picks the model per request—for example, always
6
+ routing to the cheapest active model—using the AI Gateway's public routing
7
+ primitives.
8
+ ---
9
+
10
+ The Model Filtering policy covers static allow and block lists, but model
11
+ selection can also be code. A custom [chain policy](../custom-policies.mdx) can
12
+ pick the model per request based on price, capability, the calling app, or
13
+ anything else in the request. This recipe routes every request to the cheapest
14
+ active completions model.
15
+
16
+ Two public primitives from `@zuplo/runtime` do the work:
17
+
18
+ - `AIGatewayModels.load(context)` returns the cached provider catalog, including
19
+ each model's capability, status, and per-token pricing. Credentials are never
20
+ returned.
21
+ - `AIGatewayModelRouting.set(context, routing)` validates the routing, resolves
22
+ provider credentials internally, and stores the selection the AI Gateway
23
+ handler uses. `AIGatewayModelRouting.get(context)` reads the current
24
+ selection.
25
+
26
+ ## The policy
27
+
28
+ ```ts title="modules/cheapest-model.ts"
29
+ import {
30
+ AIGatewayModelRouting,
31
+ AIGatewayModels,
32
+ ZuploContext,
33
+ ZuploRequest,
34
+ } from "@zuplo/runtime";
35
+
36
+ interface CheapestModelOptions {
37
+ providers?: string[];
38
+ }
39
+
40
+ export default async function cheapestModel(
41
+ request: ZuploRequest,
42
+ context: ZuploContext,
43
+ options: CheapestModelOptions,
44
+ ): Promise<ZuploRequest | Response> {
45
+ const allowed = options.providers ?? ["openai", "anthropic"];
46
+ const candidates = (await AIGatewayModels.load(context))
47
+ .filter(({ providerName }) => allowed.includes(providerName.toLowerCase()))
48
+ .flatMap((provider) =>
49
+ provider.models.map((model) => ({
50
+ providerName: provider.providerName,
51
+ model,
52
+ })),
53
+ )
54
+ .filter(
55
+ ({ model }) =>
56
+ model.capability === "completions" && model.status === "active",
57
+ )
58
+ .sort(
59
+ (left, right) =>
60
+ left.model.inputCostPerToken +
61
+ left.model.outputCostPerToken -
62
+ (right.model.inputCostPerToken + right.model.outputCostPerToken),
63
+ );
64
+
65
+ const cheapest = candidates[0];
66
+ if (!cheapest) {
67
+ // Returning a Response short-circuits the chain and answers the request.
68
+ return new Response("No eligible model", { status: 503 });
69
+ }
70
+
71
+ await AIGatewayModelRouting.set(context, {
72
+ completions: `${cheapest.providerName}/${cheapest.model.model}`,
73
+ });
74
+ return request;
75
+ }
76
+ ```
77
+
78
+ ## Declare and use it
79
+
80
+ Declare the module in `config/policies.json` and push:
81
+
82
+ ```json title="config/policies.json (one entry in the policies array)"
83
+ {
84
+ "name": "cheapest-model",
85
+ "policyType": "custom-code-inbound",
86
+ "handler": {
87
+ "export": "default",
88
+ "module": "$import(./modules/cheapest-model)",
89
+ "options": {
90
+ "providers": ["openai", "anthropic"]
91
+ }
92
+ }
93
+ }
94
+ ```
95
+
96
+ Then add `cheapest-model` to an app's chain on its Policies tab, in place of
97
+ Model Filtering.
98
+
99
+ ## Routing precedence
100
+
101
+ Policy order determines which selection wins:
102
+
103
+ 1. Routing selected before Model Filtering stays authoritative—Model Filtering
104
+ leaves an existing selection unchanged.
105
+ 2. Model Filtering creates routing when no earlier policy selected it.
106
+ 3. A custom policy placed after Model Filtering may deliberately replace that
107
+ selection.
108
+ 4. If no policy selects routing, the handler derives it from the request's
109
+ `providerName/model`.
110
+
111
+ Prefer one policy as the primary selector so the route's intent is easy to
112
+ understand.
113
+
114
+ ## Next steps
115
+
116
+ - [Custom Policies](../custom-policies.mdx): the full custom-policy quickstart
117
+ - [Cookbook: Custom fallback logic](./custom-fallback.mdx): enrich a selection
118
+ instead of creating one
119
+ - [Policy Chains](../policy-chains.mdx): how chain order works
@@ -0,0 +1,213 @@
1
+ ---
2
+ title: "Quickstart: Write a Custom Policy"
3
+ sidebar_label: Custom Policies
4
+ description:
5
+ Write a custom AI Gateway policy in TypeScript—a content filter that blocks
6
+ prompts containing banned terms—declare it in policies.json, deploy it with a
7
+ push, and add it to an app's policy chain in the portal.
8
+ ---
9
+
10
+ The AI Gateway's built-in policies cover model access, budgets, caching,
11
+ guardrails, and tracing—but your gateway can run any policy you can write in
12
+ TypeScript. A custom policy lives in your gateway's
13
+ [repository](./source-control.mdx), is declared in `config/policies.json`, and
14
+ from then on appears in the portal's **Add Policy** dialog like any built-in
15
+ policy. Apps add it to their [policy chains](./policy-chains.mdx), and it runs
16
+ on every request for those apps.
17
+
18
+ This quickstart builds a content filter that blocks prompts containing banned
19
+ terms. By the end, one app on your gateway rejects a prompt containing
20
+ "acme-secret-project" with a `400` response, while other apps are unaffected.
21
+
22
+ ## Prerequisites
23
+
24
+ - An AI Gateway project connected to a Git repository, with a provider, a team,
25
+ and an app—the [Getting Started](./getting-started.mdx) guide covers this
26
+ - A local clone of the gateway's repository
27
+
28
+ <Stepper>
29
+
30
+ 1. **Write the policy module**
31
+
32
+ In your clone of the gateway repository, create a `modules/` directory next
33
+ to `config/` if it doesn't exist yet, and add `modules/content-filter.ts`. A
34
+ policy is a function that receives the request, a context, and the options
35
+ configured for it—and returns the request to continue the chain, or a
36
+ `Response` to answer immediately:
37
+
38
+ ```ts title="modules/content-filter.ts"
39
+ import type { ZuploContext, ZuploRequest } from "@zuplo/runtime";
40
+
41
+ interface ContentFilterOptions {
42
+ blockedTerms: string[];
43
+ }
44
+
45
+ export default async function contentFilter(
46
+ request: ZuploRequest,
47
+ context: ZuploContext,
48
+ options: ContentFilterOptions,
49
+ ): Promise<ZuploRequest | Response> {
50
+ const body = await request.clone().json();
51
+ const text = JSON.stringify(body.messages ?? body.input ?? "");
52
+
53
+ const match = options.blockedTerms.find((term) =>
54
+ text.toLowerCase().includes(term.toLowerCase()),
55
+ );
56
+
57
+ if (match) {
58
+ context.log.warn(
59
+ `Blocked request from app ${request.user?.sub}: matched "${match}"`,
60
+ );
61
+ return new Response(
62
+ JSON.stringify({
63
+ error: {
64
+ message: "This request was blocked by your content policy.",
65
+ type: "invalid_request_error",
66
+ },
67
+ }),
68
+ { status: 400, headers: { "content-type": "application/json" } },
69
+ );
70
+ }
71
+
72
+ return request;
73
+ }
74
+ ```
75
+
76
+ The calling app is available on `request.user` when the request's API key
77
+ resolves to one—`sub` is the app's name—so a policy can log, branch, or
78
+ report per app. It can be `undefined`, so use optional chaining. The example
79
+ inspects chat and response payloads; embeddings requests pass through
80
+ unfiltered.
81
+
82
+ 2. **Declare the policy in `config/policies.json`**
83
+
84
+ The repository already contains `config/policies.json` with the built-in
85
+ policy declarations. Add one more entry to its `policies` array—the entry's
86
+ `name` is how the policy appears in the portal. The declaration makes the
87
+ policy available for apps to select; it doesn't run for any app yet:
88
+
89
+ ```json title="config/policies.json (one entry in the policies array)"
90
+ {
91
+ "name": "content-filter",
92
+ "policyType": "custom-code-inbound",
93
+ "handler": {
94
+ "export": "default",
95
+ "module": "$import(./modules/content-filter)",
96
+ "options": {
97
+ "blockedTerms": ["acme-secret-project"]
98
+ }
99
+ }
100
+ }
101
+ ```
102
+
103
+ 3. **Deploy**
104
+
105
+ Commit both files and push to your default branch:
106
+
107
+ ```bash
108
+ git add modules/content-filter.ts config/policies.json
109
+ git commit -m "Add content-filter policy"
110
+ git push
111
+ ```
112
+
113
+ The next production deploy puts the new policy on the menu—with GitHub, the
114
+ push itself deploys; see [Source Control](./source-control.mdx) for the other
115
+ Git providers.
116
+
117
+ 4. **Add the policy to an app's chain**
118
+
119
+ In the Zuplo Portal, open
120
+ [**Apps**](https://portal.zuplo.com/+/account/project/ai/apps), select your
121
+ app, and open its **Policies** tab. Click **Add Policy**—`content-filter` now
122
+ appears alongside the built-in policies. Add it, drag it to where in the
123
+ chain it should run, and save.
124
+
125
+ :::tip{title="Where in the chain?"}
126
+
127
+ If the chain has the Budgets and Costs policy, placing the filter before it
128
+ means blocked requests aren't counted against the app's budget; placing it
129
+ after means they are. The same reasoning applies to any policy that can
130
+ answer a request itself.
131
+
132
+ :::
133
+
134
+ The change applies within about a minute—no deploy.
135
+
136
+ 5. **Test it**
137
+
138
+ Send a prompt containing a blocked term through the app, using the URL and
139
+ API key from the app page (the URL below is a stand-in for it):
140
+
141
+ ```bash
142
+ curl https://my-gateway-main-2e18f50.zuplo.app/config_fe0a04972d2848e0a94ae4b8bcd1497e/v1/chat/completions \
143
+ -H "Authorization: Bearer YOUR_APP_API_KEY" \
144
+ -H "Content-Type: application/json" \
145
+ -d '{
146
+ "model": "openai/gpt-5-mini",
147
+ "messages": [{"role": "user", "content": "Tell me about acme-secret-project"}]
148
+ }'
149
+ ```
150
+
151
+ The gateway answers with the policy's `400` response. Send a harmless prompt
152
+ and the request flows through to the provider as usual. Apps that don't
153
+ include `content-filter` in their chains are unaffected.
154
+
155
+ </Stepper>
156
+
157
+ ## Per-app settings
158
+
159
+ An app's chain entry can override the declared options completely. In the
160
+ portal, edit the entry's options to give one app its own list:
161
+
162
+ ```json
163
+ {
164
+ "blockedTerms": ["acme-secret-project", "codename-falcon"]
165
+ }
166
+ ```
167
+
168
+ An entry that overrides options replaces the declaration's entire options object
169
+ —fields aren't merged. An entry without options inherits the declaration's
170
+ options exactly.
171
+
172
+ ## Configure credentials
173
+
174
+ If a policy needs a credential—say it calls an external moderation API—set it in
175
+ the declaration's options, and let chain entries inherit it:
176
+
177
+ ```json title="config/policies.json"
178
+ {
179
+ "name": "my-moderation-policy",
180
+ "policyType": "custom-code-inbound",
181
+ "handler": {
182
+ "export": "default",
183
+ "module": "$import(./modules/my-moderation-policy)",
184
+ "options": {
185
+ "apiKey": "your-moderation-api-key"
186
+ }
187
+ }
188
+ }
189
+ ```
190
+
191
+ Leave the chain entry's options out so it inherits the declaration's values—an
192
+ entry that sets its own options replaces them completely.
193
+
194
+ ## Beyond filtering
195
+
196
+ A custom policy can do more than block requests—the cookbooks walk through
197
+ complete recipes:
198
+
199
+ - **Route models dynamically.** Read the live model catalog and select the model
200
+ per request—for example, always the cheapest active model. See
201
+ [Dynamic model routing](./cookbooks/dynamic-model-routing.mdx).
202
+ - **Choose fallbacks from code.** Enrich the model selection with a dynamically
203
+ chosen backup. See [Custom fallback logic](./cookbooks/custom-fallback.mdx).
204
+ - **Enrich or annotate.** Add headers, log structured events, or call out to
205
+ other services.
206
+
207
+ ## Next steps
208
+
209
+ - [Policy Chains](./policy-chains.mdx): execution order, options inheritance,
210
+ and the built-in policies
211
+ - [Policy Templates](./policy-templates.mdx): roll a custom policy out to every
212
+ new app in a team
213
+ - [Source Control](./source-control.mdx): how repository changes deploy
@@ -1,6 +1,10 @@
1
1
  ---
2
2
  title: Using Custom AI Providers
3
3
  sidebar_label: Custom Providers
4
+ description:
5
+ Route AI Gateway apps to self-hosted or third-party OpenAI-compatible services
6
+ by adding a custom provider with its own name, URL, models, and optional
7
+ per-token costs.
4
8
  ---
5
9
 
6
10
  Zuplo's AI Gateway supports the addition of custom AI providers. This allows
@@ -29,8 +33,13 @@ To add a custom AI provider to your Zuplo AI Gateway, follow these steps:
29
33
 
30
34
  1. Select the **Custom** option from the Custom Providers section of the list
31
35
 
32
- 1. Specify a label for the custom provider instance. You can change the label
33
- later if required.
36
+ 1. Replace the pre-filled **Provider Name** (`custom`) with a name for your
37
+ provider. The name is how apps reference the provider's models: a custom
38
+ provider named `acme-llm` serves models as `acme-llm/<model>`. Names are
39
+ lowercase (letters, numbers, dots, dashes, and underscores), and Zuplo
40
+ reserves the built-in provider names (`openai`, `anthropic`, `google`,
41
+ `mistral`, `xai`, `moonshot`, `zuplo`, `zuplodemo`, `zuplo-demo`). The name
42
+ is permanent after creation.
34
43
 
35
44
  1. Specify the API URL of the custom provider you are using.
36
45
 
@@ -2,64 +2,51 @@
2
2
  title: Fallback Models
3
3
  sidebar_label: Fallbacks
4
4
  description:
5
- Configure error, timeout, and quota fallback models for an AI Gateway app in
6
- the Zuplo Portal so it keeps serving requests when the primary model fails or
7
- goes over quota.
5
+ Configure error, timeout, and quota fallback models for an AI Gateway app so
6
+ it keeps serving requests when the primary model fails or goes over quota.
8
7
  ---
9
8
 
10
- Each AI Gateway app calls a primary model: the provider, completions model, and
11
- optional embeddings model you select under **AI Models** on the app's
12
- **Settings** tab. Fallbacks let an app keep serving requests when that primary
13
- model fails, times out, or runs over its usage limits, instead of returning an
14
- error to the caller.
9
+ Fallbacks let an app keep serving requests when its primary model fails, times
10
+ out, or runs over its usage limits, instead of returning an error to the caller.
11
+ They're configured with the **Fallback Model** policy in the app's
12
+ [policy chain](./policy-chains.mdx).
15
13
 
16
14
  The AI Gateway offers two independent fallback mechanisms, each triggered by a
17
15
  different condition:
18
16
 
19
- | Mechanism | Triggers when… | Without a fallback set… |
20
- | ---------------------- | -------------------------------------------------------- | ----------------------------------- |
21
- | **Fallback & Timeout** | The primary returns a `4xx`/`5xx` or exceeds the timeout | The error is returned to the caller |
22
- | **Quota Fallback** | One of the app's usage limits is exceeded | The request is blocked with a `429` |
17
+ | Mechanism | Triggers when… | Without a fallback set… |
18
+ | ---------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------------- |
19
+ | **Fallback & Timeout** | The primary fails with a retryable error (`5xx`, `408`, `425`, `429`), a network failure, or a timeout | The gateway returns the error to the caller |
20
+ | **Quota Fallback** | One of the app's usage limits is exceeded | The request is blocked with a `429` |
23
21
 
24
22
  Both are configured entirely in the Zuplo Portal, and either can route to _any_
25
- provider. The fallback doesn't have to share the primary's provider.
23
+ provider—the fallback doesn't have to share the primary's provider. Fallback
24
+ models are referenced like any model, as `providerName/model`.
26
25
 
27
- ## Error and timeout fallback
28
-
29
- The **Fallback & Timeout** section fails an app over to a second model when the
30
- primary model returns a `4xx` or `5xx` response, or when the request takes
31
- longer than the configured timeout. This protects against provider outages, rate
32
- limiting on the primary provider, and slow responses.
33
-
34
- ### Configure an error and timeout fallback
26
+ ## Configure fallbacks
35
27
 
36
28
  <Stepper>
37
29
 
38
30
  1. Open the [Apps](https://portal.zuplo.com/+/account/project/ai/apps) tab of
39
31
  your AI Gateway project and select the app to edit.
40
32
 
41
- 1. Select the **Settings** tab and find the **Fallback & Timeout** section.
42
-
43
- 1. Choose a **Fallback Provider**. This can be the same provider as the primary
44
- or a different one, including a [custom provider](./custom-providers.mdx).
33
+ 1. Select the **Policies** tab. If the chain doesn't have a **Fallback Model**
34
+ policy yet, click **Add Policy** and add it—placed directly after Model
35
+ Filtering.
45
36
 
46
- 1. Select the **Fallback Completions** model. If the app uses embeddings, also
47
- select a **Fallback Embeddings** model.
37
+ 1. Configure the policy:
38
+ - **Fallback**: the model attempted after a retryable error or timeout, for
39
+ example `anthropic/claude-haiku-4-5`.
40
+ - **Quota fallback**: the model used once a usage limit is exceeded, for
41
+ example `openai/gpt-4o-mini`.
42
+ - **Fallback timeout (seconds)**: how long the primary call can run before
43
+ the gateway fails over. The default is `60`; values from `1` to `300` are
44
+ accepted.
48
45
 
49
- 1. Set the **Request timeout (seconds)** value to bound how long the primary
50
- model call can run before the gateway fails over. The default is `60`, but
51
- you can set any value that suits your app.
52
-
53
- 1. Click **Save Changes**.
46
+ 1. Save. The change applies within about a minute.
54
47
 
55
48
  </Stepper>
56
49
 
57
- <Framed>
58
-
59
- ![The Fallback & Timeout section of the app Settings tab, showing the Fallback Provider, Fallback Completions, Fallback Embeddings, and Request timeout fields](./fallback-and-timeout.png)
60
-
61
- </Framed>
62
-
63
50
  :::note
64
51
 
65
52
  The request timeout applies _only_ when a fallback model is set. If no fallback
@@ -69,38 +56,16 @@ is configured, the primary model call runs unbounded.
69
56
 
70
57
  ## Quota fallback
71
58
 
72
- The **Quota Fallback** section routes requests to an alternate, usually cheaper,
73
- model when one of the app's [usage limits](./usage-limits.mdx) is exceeded,
74
- rather than blocking the request with a `429`. This keeps an app available after
75
- it crosses a budget, token, or request threshold, while shifting the overflow
76
- traffic to a lower-cost model.
77
-
78
- If you leave the quota fallback empty, the app blocks requests with a `429` once
79
- it goes over quota.
80
-
81
- ### Configure a quota fallback
82
-
83
- <Stepper>
84
-
85
- 1. Open the [Apps](https://portal.zuplo.com/+/account/project/ai/apps) tab and
86
- select the app to edit.
59
+ The quota fallback routes requests to an alternate, usually cheaper, model when
60
+ one of the app's [usage limits](./usage-limits.mdx) is exceeded, rather than
61
+ blocking the request with a `429`. This keeps an app available after it crosses
62
+ a budget, token, or request threshold, while shifting the overflow traffic to a
63
+ lower-cost model.
87
64
 
88
- 1. Select the **Settings** tab. Set the limits you want to enforce under **Usage
89
- Limits & Thresholds**, then find the **Quota Fallback** section below them.
90
-
91
- 1. Choose a **Quota Fallback Provider**, then select the **Quota Fallback
92
- Completions** model. If the app uses embeddings, also select a **Quota
93
- Fallback Embeddings** model.
94
-
95
- 1. Click **Save Changes**.
96
-
97
- </Stepper>
98
-
99
- <Framed>
100
-
101
- ![The Quota Fallback section of the app Settings tab, below Usage Limits & Thresholds, showing the Quota Fallback Provider, Quota Fallback Completions, and Quota Fallback Embeddings fields](./quota-fallback.png)
102
-
103
- </Framed>
65
+ For the quota fallback to activate, the chain needs the **Budgets and Costs**
66
+ policy after Fallback Model—it detects the exceeded limit and switches to the
67
+ fallback selection. If you leave the quota fallback empty, the app blocks
68
+ requests with a `429` once it goes over quota.
104
69
 
105
70
  :::tip
106
71
 
@@ -115,18 +80,26 @@ toward the app's limits.
115
80
  The mechanisms are evaluated independently and can both be active on the same
116
81
  app:
117
82
 
118
- - A request that is **over quota** routes to the quota fallback model.
119
- - A request that is **within quota** but hits an **error or timeout** on the
83
+ - A request that's **over quota** routes to the quota fallback model.
84
+ - A request that's **within quota** but hits an **error or timeout** on the
120
85
  primary routes to the error and timeout fallback model.
121
86
 
122
87
  Set whichever fallbacks match the failure modes you want to protect against.
123
88
  Neither is required.
124
89
 
90
+ :::note
91
+
92
+ Error and timeout fallback applies to Chat Completions and Embeddings requests.
93
+ Requests to the native `/v1/messages` and `/v1/responses` endpoints pass through
94
+ without retrying a backup model; the quota fallback still applies to them.
95
+
96
+ :::
97
+
125
98
  ## Related resources
126
99
 
127
- - [Managing Apps](./managing-apps.mdx) - Create, edit, and delete AI Gateway
128
- apps.
129
- - [Usage Limits & Thresholds](./usage-limits.mdx) - Configure the budget, token,
130
- and request limits that trigger a quota fallback.
100
+ - [Policy Chains](./policy-chains.mdx) - How the app's policy chain executes and
101
+ the recommended policy order.
102
+ - [Usage Limits](./usage-limits.mdx) - Configure the budget, token, and request
103
+ limits that trigger a quota fallback.
131
104
  - [Custom Providers](./custom-providers.mdx) - Add your own provider to use as a
132
105
  primary or fallback model.