zuplo 7.1.0 → 7.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/ai-gateway/apps.mdx +28 -10
- package/docs/ai-gateway/cookbooks/custom-fallback.mdx +96 -0
- package/docs/ai-gateway/cookbooks/dynamic-model-routing.mdx +119 -0
- package/docs/ai-gateway/custom-policies.mdx +213 -0
- package/docs/ai-gateway/custom-providers.mdx +10 -2
- package/docs/ai-gateway/fallback.mdx +48 -75
- package/docs/ai-gateway/getting-started.mdx +200 -132
- package/docs/ai-gateway/integrations/ai-sdk.mdx +71 -28
- package/docs/ai-gateway/integrations/claude-code.mdx +32 -8
- package/docs/ai-gateway/integrations/codex.mdx +28 -10
- package/docs/ai-gateway/integrations/goose.mdx +26 -19
- package/docs/ai-gateway/integrations/langchain.mdx +31 -15
- package/docs/ai-gateway/integrations/openai.mdx +28 -16
- package/docs/ai-gateway/introduction.mdx +73 -32
- package/docs/ai-gateway/managing-apps.mdx +62 -17
- package/docs/ai-gateway/managing-providers.mdx +23 -7
- package/docs/ai-gateway/managing-teams.mdx +29 -12
- package/docs/ai-gateway/policies/akamai-ai-firewall.mdx +92 -96
- package/docs/ai-gateway/policies/authentication.mdx +44 -0
- package/docs/ai-gateway/policies/comet-opik-tracing.mdx +40 -2
- package/docs/ai-gateway/policies/fallback-model.mdx +40 -0
- package/docs/ai-gateway/policies/galileo-tracing.mdx +36 -22
- package/docs/ai-gateway/policies/metering.mdx +47 -0
- package/docs/ai-gateway/policies/model-filtering.mdx +49 -0
- package/docs/ai-gateway/policies/overview.mdx +28 -0
- package/docs/ai-gateway/policies/semantic-cache.mdx +48 -0
- package/docs/ai-gateway/policy-chains.mdx +178 -0
- package/docs/ai-gateway/policy-templates.mdx +68 -0
- package/docs/ai-gateway/providers.mdx +23 -9
- package/docs/ai-gateway/source-control.mdx +88 -0
- package/docs/ai-gateway/teams.mdx +16 -10
- package/docs/ai-gateway/universal-api.mdx +39 -12
- package/docs/ai-gateway/usage-limits.mdx +87 -55
- package/docs/policies/cdn-cache-control-outbound/doc.md +13 -13
- package/package.json +5 -5
- package/docs/ai-gateway/guardrails.mdx +0 -65
package/docs/ai-gateway/apps.mdx
CHANGED
|
@@ -1,22 +1,40 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: AI Gateway Apps
|
|
3
3
|
sidebar_label: Overview
|
|
4
|
+
description:
|
|
5
|
+
Apps represent the services and integrations that call your AI Gateway. Each
|
|
6
|
+
app has its own URL, API key, and policy chain.
|
|
4
7
|
---
|
|
5
8
|
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
9
|
+
An app represents one caller of your AI Gateway—a service, an agent, or a
|
|
10
|
+
feature in a larger codebase. A support chatbot on your website is one app; the
|
|
11
|
+
batch job that summarizes tickets overnight is another. Each app belongs to a
|
|
12
|
+
[team](./teams.mdx) and can reach the AI providers assigned to that team.
|
|
13
|
+
|
|
14
|
+
Each app has three things of its own:
|
|
15
|
+
|
|
16
|
+
- **An API URL**—the endpoint the app's code calls, shown at the top of the app
|
|
17
|
+
page. The gateway attributes requests made to this URL to the app, tracking
|
|
18
|
+
usage independently per app. See the [Universal API](./universal-api.mdx).
|
|
19
|
+
- **An API key**—authenticates the app's requests when the gateway runs the
|
|
20
|
+
[authentication policy](./policy-chains.mdx#authentication).
|
|
21
|
+
- **A [policy chain](./policy-chains.mdx)**—the ordered policies that run on the
|
|
22
|
+
app's requests: model access, budgets, caching, guardrails, and custom
|
|
23
|
+
policies. The chain starts out empty unless the app's team has a
|
|
24
|
+
[policy template](./policy-templates.mdx).
|
|
11
25
|
|
|
12
26
|
## API Keys
|
|
13
27
|
|
|
14
|
-
Each
|
|
15
|
-
|
|
28
|
+
Each app has its own API key, which the gateway validates when the
|
|
29
|
+
[authentication policy](./policy-chains.mdx#authentication) applies. The key is
|
|
30
|
+
optional: the gateway attributes usage by app ID, resolving it either from a
|
|
31
|
+
validated key or from the `{app_id}` segment of the request URL, so an app
|
|
32
|
+
without authentication still tracks usage independently.
|
|
33
|
+
|
|
34
|
+
To find an app's API key, open the
|
|
16
35
|
[Apps](https://portal.zuplo.com/+/account/project/ai/apps) tab of your AI
|
|
17
|
-
Gateway project in the Zuplo Portal
|
|
18
|
-
|
|
19
|
-
page.
|
|
36
|
+
Gateway project in the Zuplo Portal and select the app. The API key appears in
|
|
37
|
+
the API Key section at the top of the app page.
|
|
20
38
|
|
|
21
39
|
**Additional Resources**
|
|
22
40
|
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Cookbook: Custom fallback logic"
|
|
3
|
+
sidebar_label: Custom fallback logic
|
|
4
|
+
description:
|
|
5
|
+
Write a custom policy that chooses a backup model dynamically—for example,
|
|
6
|
+
picking any active Anthropic model as the fallback—instead of a fixed entry in
|
|
7
|
+
the Fallback Model policy.
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
The [Fallback Model policy](../fallback.mdx) covers a fixed backup and quota
|
|
11
|
+
fallback. When the backup should be chosen dynamically—the cheapest available
|
|
12
|
+
model, a provider-specific preference, whatever is currently active—a custom
|
|
13
|
+
[chain policy](../custom-policies.mdx) can replace it using the same public
|
|
14
|
+
primitives.
|
|
15
|
+
|
|
16
|
+
A fallback policy enriches the selection an earlier policy created. Because
|
|
17
|
+
`AIGatewayModelRouting.set()` replaces the complete stored selection rather than
|
|
18
|
+
partially updating it, the policy must read, merge, and set. It should also
|
|
19
|
+
leave the request unchanged when no earlier policy created a selection—so a
|
|
20
|
+
misplaced chain entry doesn't bypass model filtering—and when the chosen backup
|
|
21
|
+
equals the current main model, which `set()` rejects.
|
|
22
|
+
|
|
23
|
+
## The policy
|
|
24
|
+
|
|
25
|
+
This example adds the first active Anthropic completions model as the backup,
|
|
26
|
+
with a 30-second timeout:
|
|
27
|
+
|
|
28
|
+
```ts title="modules/add-fallback.ts"
|
|
29
|
+
import {
|
|
30
|
+
AIGatewayModelRouting,
|
|
31
|
+
AIGatewayModels,
|
|
32
|
+
ZuploContext,
|
|
33
|
+
ZuploRequest,
|
|
34
|
+
} from "@zuplo/runtime";
|
|
35
|
+
|
|
36
|
+
export default async function addFallback(
|
|
37
|
+
request: ZuploRequest,
|
|
38
|
+
context: ZuploContext,
|
|
39
|
+
): Promise<ZuploRequest> {
|
|
40
|
+
const routing = AIGatewayModelRouting.get(context);
|
|
41
|
+
const current = routing?.completions;
|
|
42
|
+
if (!routing || !current) {
|
|
43
|
+
return request;
|
|
44
|
+
}
|
|
45
|
+
const target = typeof current === "string" ? { main: current } : current;
|
|
46
|
+
|
|
47
|
+
const providers = await AIGatewayModels.load(context);
|
|
48
|
+
const anthropic = providers.find(
|
|
49
|
+
(provider) => provider.providerName === "anthropic",
|
|
50
|
+
);
|
|
51
|
+
const fallback = anthropic?.models.find(
|
|
52
|
+
(candidate) =>
|
|
53
|
+
candidate.capability === "completions" && candidate.status === "active",
|
|
54
|
+
);
|
|
55
|
+
const backup = fallback ? `anthropic/${fallback.model}` : undefined;
|
|
56
|
+
if (!backup || backup.toLowerCase() === target.main.toLowerCase()) {
|
|
57
|
+
return request;
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
await AIGatewayModelRouting.set(context, {
|
|
61
|
+
...routing,
|
|
62
|
+
completions: {
|
|
63
|
+
...target,
|
|
64
|
+
backup,
|
|
65
|
+
fallbackTimeoutSeconds: 30,
|
|
66
|
+
},
|
|
67
|
+
});
|
|
68
|
+
return request;
|
|
69
|
+
}
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
## Declare and use it
|
|
73
|
+
|
|
74
|
+
```json title="config/policies.json (one entry in the policies array)"
|
|
75
|
+
{
|
|
76
|
+
"name": "add-fallback",
|
|
77
|
+
"policyType": "custom-code-inbound",
|
|
78
|
+
"handler": {
|
|
79
|
+
"export": "default",
|
|
80
|
+
"module": "$import(./modules/add-fallback)"
|
|
81
|
+
}
|
|
82
|
+
}
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Add `add-fallback` to an app's chain directly after Model Filtering, replacing
|
|
86
|
+
the Fallback Model policy. Keep the no-selection guard at the top of the
|
|
87
|
+
policy—it prevents a misplaced entry from creating a primary selection and
|
|
88
|
+
bypassing filtering.
|
|
89
|
+
|
|
90
|
+
## Next steps
|
|
91
|
+
|
|
92
|
+
- [Fallback Models](../fallback.mdx): the built-in fallback and quota-fallback
|
|
93
|
+
behavior
|
|
94
|
+
- [Cookbook: Dynamic model routing](./dynamic-model-routing.mdx): create the
|
|
95
|
+
primary selection from code
|
|
96
|
+
- [Custom Policies](../custom-policies.mdx): the full custom-policy quickstart
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Cookbook: Dynamic model routing"
|
|
3
|
+
sidebar_label: Dynamic model routing
|
|
4
|
+
description:
|
|
5
|
+
Write a custom policy that picks the model per request—for example, always
|
|
6
|
+
routing to the cheapest active model—using the AI Gateway's public routing
|
|
7
|
+
primitives.
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
The Model Filtering policy covers static allow and block lists, but model
|
|
11
|
+
selection can also be code. A custom [chain policy](../custom-policies.mdx) can
|
|
12
|
+
pick the model per request based on price, capability, the calling app, or
|
|
13
|
+
anything else in the request. This recipe routes every request to the cheapest
|
|
14
|
+
active completions model.
|
|
15
|
+
|
|
16
|
+
Two public primitives from `@zuplo/runtime` do the work:
|
|
17
|
+
|
|
18
|
+
- `AIGatewayModels.load(context)` returns the cached provider catalog, including
|
|
19
|
+
each model's capability, status, and per-token pricing. Credentials are never
|
|
20
|
+
returned.
|
|
21
|
+
- `AIGatewayModelRouting.set(context, routing)` validates the routing, resolves
|
|
22
|
+
provider credentials internally, and stores the selection the AI Gateway
|
|
23
|
+
handler uses. `AIGatewayModelRouting.get(context)` reads the current
|
|
24
|
+
selection.
|
|
25
|
+
|
|
26
|
+
## The policy
|
|
27
|
+
|
|
28
|
+
```ts title="modules/cheapest-model.ts"
|
|
29
|
+
import {
|
|
30
|
+
AIGatewayModelRouting,
|
|
31
|
+
AIGatewayModels,
|
|
32
|
+
ZuploContext,
|
|
33
|
+
ZuploRequest,
|
|
34
|
+
} from "@zuplo/runtime";
|
|
35
|
+
|
|
36
|
+
interface CheapestModelOptions {
|
|
37
|
+
providers?: string[];
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
export default async function cheapestModel(
|
|
41
|
+
request: ZuploRequest,
|
|
42
|
+
context: ZuploContext,
|
|
43
|
+
options: CheapestModelOptions,
|
|
44
|
+
): Promise<ZuploRequest | Response> {
|
|
45
|
+
const allowed = options.providers ?? ["openai", "anthropic"];
|
|
46
|
+
const candidates = (await AIGatewayModels.load(context))
|
|
47
|
+
.filter(({ providerName }) => allowed.includes(providerName.toLowerCase()))
|
|
48
|
+
.flatMap((provider) =>
|
|
49
|
+
provider.models.map((model) => ({
|
|
50
|
+
providerName: provider.providerName,
|
|
51
|
+
model,
|
|
52
|
+
})),
|
|
53
|
+
)
|
|
54
|
+
.filter(
|
|
55
|
+
({ model }) =>
|
|
56
|
+
model.capability === "completions" && model.status === "active",
|
|
57
|
+
)
|
|
58
|
+
.sort(
|
|
59
|
+
(left, right) =>
|
|
60
|
+
left.model.inputCostPerToken +
|
|
61
|
+
left.model.outputCostPerToken -
|
|
62
|
+
(right.model.inputCostPerToken + right.model.outputCostPerToken),
|
|
63
|
+
);
|
|
64
|
+
|
|
65
|
+
const cheapest = candidates[0];
|
|
66
|
+
if (!cheapest) {
|
|
67
|
+
// Returning a Response short-circuits the chain and answers the request.
|
|
68
|
+
return new Response("No eligible model", { status: 503 });
|
|
69
|
+
}
|
|
70
|
+
|
|
71
|
+
await AIGatewayModelRouting.set(context, {
|
|
72
|
+
completions: `${cheapest.providerName}/${cheapest.model.model}`,
|
|
73
|
+
});
|
|
74
|
+
return request;
|
|
75
|
+
}
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
## Declare and use it
|
|
79
|
+
|
|
80
|
+
Declare the module in `config/policies.json` and push:
|
|
81
|
+
|
|
82
|
+
```json title="config/policies.json (one entry in the policies array)"
|
|
83
|
+
{
|
|
84
|
+
"name": "cheapest-model",
|
|
85
|
+
"policyType": "custom-code-inbound",
|
|
86
|
+
"handler": {
|
|
87
|
+
"export": "default",
|
|
88
|
+
"module": "$import(./modules/cheapest-model)",
|
|
89
|
+
"options": {
|
|
90
|
+
"providers": ["openai", "anthropic"]
|
|
91
|
+
}
|
|
92
|
+
}
|
|
93
|
+
}
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Then add `cheapest-model` to an app's chain on its Policies tab, in place of
|
|
97
|
+
Model Filtering.
|
|
98
|
+
|
|
99
|
+
## Routing precedence
|
|
100
|
+
|
|
101
|
+
Policy order determines which selection wins:
|
|
102
|
+
|
|
103
|
+
1. Routing selected before Model Filtering stays authoritative—Model Filtering
|
|
104
|
+
leaves an existing selection unchanged.
|
|
105
|
+
2. Model Filtering creates routing when no earlier policy selected it.
|
|
106
|
+
3. A custom policy placed after Model Filtering may deliberately replace that
|
|
107
|
+
selection.
|
|
108
|
+
4. If no policy selects routing, the handler derives it from the request's
|
|
109
|
+
`providerName/model`.
|
|
110
|
+
|
|
111
|
+
Prefer one policy as the primary selector so the route's intent is easy to
|
|
112
|
+
understand.
|
|
113
|
+
|
|
114
|
+
## Next steps
|
|
115
|
+
|
|
116
|
+
- [Custom Policies](../custom-policies.mdx): the full custom-policy quickstart
|
|
117
|
+
- [Cookbook: Custom fallback logic](./custom-fallback.mdx): enrich a selection
|
|
118
|
+
instead of creating one
|
|
119
|
+
- [Policy Chains](../policy-chains.mdx): how chain order works
|
|
@@ -0,0 +1,213 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Quickstart: Write a Custom Policy"
|
|
3
|
+
sidebar_label: Custom Policies
|
|
4
|
+
description:
|
|
5
|
+
Write a custom AI Gateway policy in TypeScript—a content filter that blocks
|
|
6
|
+
prompts containing banned terms—declare it in policies.json, deploy it with a
|
|
7
|
+
push, and add it to an app's policy chain in the portal.
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
The AI Gateway's built-in policies cover model access, budgets, caching,
|
|
11
|
+
guardrails, and tracing—but your gateway can run any policy you can write in
|
|
12
|
+
TypeScript. A custom policy lives in your gateway's
|
|
13
|
+
[repository](./source-control.mdx), is declared in `config/policies.json`, and
|
|
14
|
+
from then on appears in the portal's **Add Policy** dialog like any built-in
|
|
15
|
+
policy. Apps add it to their [policy chains](./policy-chains.mdx), and it runs
|
|
16
|
+
on every request for those apps.
|
|
17
|
+
|
|
18
|
+
This quickstart builds a content filter that blocks prompts containing banned
|
|
19
|
+
terms. By the end, one app on your gateway rejects a prompt containing
|
|
20
|
+
"acme-secret-project" with a `400` response, while other apps are unaffected.
|
|
21
|
+
|
|
22
|
+
## Prerequisites
|
|
23
|
+
|
|
24
|
+
- An AI Gateway project connected to a Git repository, with a provider, a team,
|
|
25
|
+
and an app—the [Getting Started](./getting-started.mdx) guide covers this
|
|
26
|
+
- A local clone of the gateway's repository
|
|
27
|
+
|
|
28
|
+
<Stepper>
|
|
29
|
+
|
|
30
|
+
1. **Write the policy module**
|
|
31
|
+
|
|
32
|
+
In your clone of the gateway repository, create a `modules/` directory next
|
|
33
|
+
to `config/` if it doesn't exist yet, and add `modules/content-filter.ts`. A
|
|
34
|
+
policy is a function that receives the request, a context, and the options
|
|
35
|
+
configured for it—and returns the request to continue the chain, or a
|
|
36
|
+
`Response` to answer immediately:
|
|
37
|
+
|
|
38
|
+
```ts title="modules/content-filter.ts"
|
|
39
|
+
import type { ZuploContext, ZuploRequest } from "@zuplo/runtime";
|
|
40
|
+
|
|
41
|
+
interface ContentFilterOptions {
|
|
42
|
+
blockedTerms: string[];
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
export default async function contentFilter(
|
|
46
|
+
request: ZuploRequest,
|
|
47
|
+
context: ZuploContext,
|
|
48
|
+
options: ContentFilterOptions,
|
|
49
|
+
): Promise<ZuploRequest | Response> {
|
|
50
|
+
const body = await request.clone().json();
|
|
51
|
+
const text = JSON.stringify(body.messages ?? body.input ?? "");
|
|
52
|
+
|
|
53
|
+
const match = options.blockedTerms.find((term) =>
|
|
54
|
+
text.toLowerCase().includes(term.toLowerCase()),
|
|
55
|
+
);
|
|
56
|
+
|
|
57
|
+
if (match) {
|
|
58
|
+
context.log.warn(
|
|
59
|
+
`Blocked request from app ${request.user?.sub}: matched "${match}"`,
|
|
60
|
+
);
|
|
61
|
+
return new Response(
|
|
62
|
+
JSON.stringify({
|
|
63
|
+
error: {
|
|
64
|
+
message: "This request was blocked by your content policy.",
|
|
65
|
+
type: "invalid_request_error",
|
|
66
|
+
},
|
|
67
|
+
}),
|
|
68
|
+
{ status: 400, headers: { "content-type": "application/json" } },
|
|
69
|
+
);
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
return request;
|
|
73
|
+
}
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
The calling app is available on `request.user` when the request's API key
|
|
77
|
+
resolves to one—`sub` is the app's name—so a policy can log, branch, or
|
|
78
|
+
report per app. It can be `undefined`, so use optional chaining. The example
|
|
79
|
+
inspects chat and response payloads; embeddings requests pass through
|
|
80
|
+
unfiltered.
|
|
81
|
+
|
|
82
|
+
2. **Declare the policy in `config/policies.json`**
|
|
83
|
+
|
|
84
|
+
The repository already contains `config/policies.json` with the built-in
|
|
85
|
+
policy declarations. Add one more entry to its `policies` array—the entry's
|
|
86
|
+
`name` is how the policy appears in the portal. The declaration makes the
|
|
87
|
+
policy available for apps to select; it doesn't run for any app yet:
|
|
88
|
+
|
|
89
|
+
```json title="config/policies.json (one entry in the policies array)"
|
|
90
|
+
{
|
|
91
|
+
"name": "content-filter",
|
|
92
|
+
"policyType": "custom-code-inbound",
|
|
93
|
+
"handler": {
|
|
94
|
+
"export": "default",
|
|
95
|
+
"module": "$import(./modules/content-filter)",
|
|
96
|
+
"options": {
|
|
97
|
+
"blockedTerms": ["acme-secret-project"]
|
|
98
|
+
}
|
|
99
|
+
}
|
|
100
|
+
}
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
3. **Deploy**
|
|
104
|
+
|
|
105
|
+
Commit both files and push to your default branch:
|
|
106
|
+
|
|
107
|
+
```bash
|
|
108
|
+
git add modules/content-filter.ts config/policies.json
|
|
109
|
+
git commit -m "Add content-filter policy"
|
|
110
|
+
git push
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
The next production deploy puts the new policy on the menu—with GitHub, the
|
|
114
|
+
push itself deploys; see [Source Control](./source-control.mdx) for the other
|
|
115
|
+
Git providers.
|
|
116
|
+
|
|
117
|
+
4. **Add the policy to an app's chain**
|
|
118
|
+
|
|
119
|
+
In the Zuplo Portal, open
|
|
120
|
+
[**Apps**](https://portal.zuplo.com/+/account/project/ai/apps), select your
|
|
121
|
+
app, and open its **Policies** tab. Click **Add Policy**—`content-filter` now
|
|
122
|
+
appears alongside the built-in policies. Add it, drag it to where in the
|
|
123
|
+
chain it should run, and save.
|
|
124
|
+
|
|
125
|
+
:::tip{title="Where in the chain?"}
|
|
126
|
+
|
|
127
|
+
If the chain has the Budgets and Costs policy, placing the filter before it
|
|
128
|
+
means blocked requests aren't counted against the app's budget; placing it
|
|
129
|
+
after means they are. The same reasoning applies to any policy that can
|
|
130
|
+
answer a request itself.
|
|
131
|
+
|
|
132
|
+
:::
|
|
133
|
+
|
|
134
|
+
The change applies within about a minute—no deploy.
|
|
135
|
+
|
|
136
|
+
5. **Test it**
|
|
137
|
+
|
|
138
|
+
Send a prompt containing a blocked term through the app, using the URL and
|
|
139
|
+
API key from the app page (the URL below is a stand-in for it):
|
|
140
|
+
|
|
141
|
+
```bash
|
|
142
|
+
curl https://my-gateway-main-2e18f50.zuplo.app/config_fe0a04972d2848e0a94ae4b8bcd1497e/v1/chat/completions \
|
|
143
|
+
-H "Authorization: Bearer YOUR_APP_API_KEY" \
|
|
144
|
+
-H "Content-Type: application/json" \
|
|
145
|
+
-d '{
|
|
146
|
+
"model": "openai/gpt-5-mini",
|
|
147
|
+
"messages": [{"role": "user", "content": "Tell me about acme-secret-project"}]
|
|
148
|
+
}'
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
The gateway answers with the policy's `400` response. Send a harmless prompt
|
|
152
|
+
and the request flows through to the provider as usual. Apps that don't
|
|
153
|
+
include `content-filter` in their chains are unaffected.
|
|
154
|
+
|
|
155
|
+
</Stepper>
|
|
156
|
+
|
|
157
|
+
## Per-app settings
|
|
158
|
+
|
|
159
|
+
An app's chain entry can override the declared options completely. In the
|
|
160
|
+
portal, edit the entry's options to give one app its own list:
|
|
161
|
+
|
|
162
|
+
```json
|
|
163
|
+
{
|
|
164
|
+
"blockedTerms": ["acme-secret-project", "codename-falcon"]
|
|
165
|
+
}
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
An entry that overrides options replaces the declaration's entire options object
|
|
169
|
+
—fields aren't merged. An entry without options inherits the declaration's
|
|
170
|
+
options exactly.
|
|
171
|
+
|
|
172
|
+
## Configure credentials
|
|
173
|
+
|
|
174
|
+
If a policy needs a credential—say it calls an external moderation API—set it in
|
|
175
|
+
the declaration's options, and let chain entries inherit it:
|
|
176
|
+
|
|
177
|
+
```json title="config/policies.json"
|
|
178
|
+
{
|
|
179
|
+
"name": "my-moderation-policy",
|
|
180
|
+
"policyType": "custom-code-inbound",
|
|
181
|
+
"handler": {
|
|
182
|
+
"export": "default",
|
|
183
|
+
"module": "$import(./modules/my-moderation-policy)",
|
|
184
|
+
"options": {
|
|
185
|
+
"apiKey": "your-moderation-api-key"
|
|
186
|
+
}
|
|
187
|
+
}
|
|
188
|
+
}
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
Leave the chain entry's options out so it inherits the declaration's values—an
|
|
192
|
+
entry that sets its own options replaces them completely.
|
|
193
|
+
|
|
194
|
+
## Beyond filtering
|
|
195
|
+
|
|
196
|
+
A custom policy can do more than block requests—the cookbooks walk through
|
|
197
|
+
complete recipes:
|
|
198
|
+
|
|
199
|
+
- **Route models dynamically.** Read the live model catalog and select the model
|
|
200
|
+
per request—for example, always the cheapest active model. See
|
|
201
|
+
[Dynamic model routing](./cookbooks/dynamic-model-routing.mdx).
|
|
202
|
+
- **Choose fallbacks from code.** Enrich the model selection with a dynamically
|
|
203
|
+
chosen backup. See [Custom fallback logic](./cookbooks/custom-fallback.mdx).
|
|
204
|
+
- **Enrich or annotate.** Add headers, log structured events, or call out to
|
|
205
|
+
other services.
|
|
206
|
+
|
|
207
|
+
## Next steps
|
|
208
|
+
|
|
209
|
+
- [Policy Chains](./policy-chains.mdx): execution order, options inheritance,
|
|
210
|
+
and the built-in policies
|
|
211
|
+
- [Policy Templates](./policy-templates.mdx): roll a custom policy out to every
|
|
212
|
+
new app in a team
|
|
213
|
+
- [Source Control](./source-control.mdx): how repository changes deploy
|
|
@@ -1,6 +1,10 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Using Custom AI Providers
|
|
3
3
|
sidebar_label: Custom Providers
|
|
4
|
+
description:
|
|
5
|
+
Route AI Gateway apps to self-hosted or third-party OpenAI-compatible services
|
|
6
|
+
by adding a custom provider with its own name, URL, models, and optional
|
|
7
|
+
per-token costs.
|
|
4
8
|
---
|
|
5
9
|
|
|
6
10
|
Zuplo's AI Gateway supports the addition of custom AI providers. This allows
|
|
@@ -29,8 +33,12 @@ To add a custom AI provider to your Zuplo AI Gateway, follow these steps:
|
|
|
29
33
|
|
|
30
34
|
1. Select the **Custom** option from the Custom Providers section of the list
|
|
31
35
|
|
|
32
|
-
1.
|
|
33
|
-
|
|
36
|
+
1. Replace the pre-filled **Provider Name** (`custom`) with a name for your
|
|
37
|
+
provider. The name is how apps reference the provider's models: a custom
|
|
38
|
+
provider named `acme-llm` serves models as `acme-llm/<model>`. Names are
|
|
39
|
+
lowercase (letters, numbers, dots, dashes, and underscores), and the built-in
|
|
40
|
+
provider names (`openai`, `anthropic`, `google`, `mistral`, `xai`,
|
|
41
|
+
`moonshot`) are reserved. The name is permanent after creation.
|
|
34
42
|
|
|
35
43
|
1. Specify the API URL of the custom provider you are using.
|
|
36
44
|
|
|
@@ -2,64 +2,51 @@
|
|
|
2
2
|
title: Fallback Models
|
|
3
3
|
sidebar_label: Fallbacks
|
|
4
4
|
description:
|
|
5
|
-
Configure error, timeout, and quota fallback models for an AI Gateway app
|
|
6
|
-
|
|
7
|
-
goes over quota.
|
|
5
|
+
Configure error, timeout, and quota fallback models for an AI Gateway app so
|
|
6
|
+
it keeps serving requests when the primary model fails or goes over quota.
|
|
8
7
|
---
|
|
9
8
|
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
error to the caller.
|
|
9
|
+
Fallbacks let an app keep serving requests when its primary model fails, times
|
|
10
|
+
out, or runs over its usage limits, instead of returning an error to the caller.
|
|
11
|
+
They're configured with the **Fallback Model** policy in the app's
|
|
12
|
+
[policy chain](./policy-chains.mdx).
|
|
15
13
|
|
|
16
14
|
The AI Gateway offers two independent fallback mechanisms, each triggered by a
|
|
17
15
|
different condition:
|
|
18
16
|
|
|
19
|
-
| Mechanism | Triggers when…
|
|
20
|
-
| ---------------------- |
|
|
21
|
-
| **Fallback & Timeout** | The primary
|
|
22
|
-
| **Quota Fallback** | One of the app's usage limits is exceeded
|
|
17
|
+
| Mechanism | Triggers when… | Without a fallback set… |
|
|
18
|
+
| ---------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------------- |
|
|
19
|
+
| **Fallback & Timeout** | The primary fails with a retryable error (`5xx`, `408`, `425`, `429`), a network failure, or a timeout | The gateway returns the error to the caller |
|
|
20
|
+
| **Quota Fallback** | One of the app's usage limits is exceeded | The request is blocked with a `429` |
|
|
23
21
|
|
|
24
22
|
Both are configured entirely in the Zuplo Portal, and either can route to _any_
|
|
25
|
-
provider
|
|
23
|
+
provider—the fallback doesn't have to share the primary's provider. Fallback
|
|
24
|
+
models are referenced like any model, as `providerName/model`.
|
|
26
25
|
|
|
27
|
-
##
|
|
28
|
-
|
|
29
|
-
The **Fallback & Timeout** section fails an app over to a second model when the
|
|
30
|
-
primary model returns a `4xx` or `5xx` response, or when the request takes
|
|
31
|
-
longer than the configured timeout. This protects against provider outages, rate
|
|
32
|
-
limiting on the primary provider, and slow responses.
|
|
33
|
-
|
|
34
|
-
### Configure an error and timeout fallback
|
|
26
|
+
## Configure fallbacks
|
|
35
27
|
|
|
36
28
|
<Stepper>
|
|
37
29
|
|
|
38
30
|
1. Open the [Apps](https://portal.zuplo.com/+/account/project/ai/apps) tab of
|
|
39
31
|
your AI Gateway project and select the app to edit.
|
|
40
32
|
|
|
41
|
-
1. Select the **
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
or a different one, including a [custom provider](./custom-providers.mdx).
|
|
33
|
+
1. Select the **Policies** tab. If the chain doesn't have a **Fallback Model**
|
|
34
|
+
policy yet, click **Add Policy** and add it—placed directly after Model
|
|
35
|
+
Filtering.
|
|
45
36
|
|
|
46
|
-
1.
|
|
47
|
-
|
|
37
|
+
1. Configure the policy:
|
|
38
|
+
- **Fallback**: the model attempted after a retryable error or timeout, for
|
|
39
|
+
example `anthropic/claude-haiku-4-5`.
|
|
40
|
+
- **Quota fallback**: the model used once a usage limit is exceeded, for
|
|
41
|
+
example `openai/gpt-4o-mini`.
|
|
42
|
+
- **Fallback timeout (seconds)**: how long the primary call can run before
|
|
43
|
+
the gateway fails over. The default is `60`; values from `1` to `300` are
|
|
44
|
+
accepted.
|
|
48
45
|
|
|
49
|
-
1.
|
|
50
|
-
model call can run before the gateway fails over. The default is `60`, but
|
|
51
|
-
you can set any value that suits your app.
|
|
52
|
-
|
|
53
|
-
1. Click **Save Changes**.
|
|
46
|
+
1. Save. The change applies within about a minute.
|
|
54
47
|
|
|
55
48
|
</Stepper>
|
|
56
49
|
|
|
57
|
-
<Framed>
|
|
58
|
-
|
|
59
|
-

|
|
60
|
-
|
|
61
|
-
</Framed>
|
|
62
|
-
|
|
63
50
|
:::note
|
|
64
51
|
|
|
65
52
|
The request timeout applies _only_ when a fallback model is set. If no fallback
|
|
@@ -69,38 +56,16 @@ is configured, the primary model call runs unbounded.
|
|
|
69
56
|
|
|
70
57
|
## Quota fallback
|
|
71
58
|
|
|
72
|
-
The
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
If you leave the quota fallback empty, the app blocks requests with a `429` once
|
|
79
|
-
it goes over quota.
|
|
80
|
-
|
|
81
|
-
### Configure a quota fallback
|
|
82
|
-
|
|
83
|
-
<Stepper>
|
|
84
|
-
|
|
85
|
-
1. Open the [Apps](https://portal.zuplo.com/+/account/project/ai/apps) tab and
|
|
86
|
-
select the app to edit.
|
|
59
|
+
The quota fallback routes requests to an alternate, usually cheaper, model when
|
|
60
|
+
one of the app's [usage limits](./usage-limits.mdx) is exceeded, rather than
|
|
61
|
+
blocking the request with a `429`. This keeps an app available after it crosses
|
|
62
|
+
a budget, token, or request threshold, while shifting the overflow traffic to a
|
|
63
|
+
lower-cost model.
|
|
87
64
|
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
Completions** model. If the app uses embeddings, also select a **Quota
|
|
93
|
-
Fallback Embeddings** model.
|
|
94
|
-
|
|
95
|
-
1. Click **Save Changes**.
|
|
96
|
-
|
|
97
|
-
</Stepper>
|
|
98
|
-
|
|
99
|
-
<Framed>
|
|
100
|
-
|
|
101
|
-

|
|
102
|
-
|
|
103
|
-
</Framed>
|
|
65
|
+
For the quota fallback to activate, the chain needs the **Budgets and Costs**
|
|
66
|
+
policy after Fallback Model—it detects the exceeded limit and switches to the
|
|
67
|
+
fallback selection. If you leave the quota fallback empty, the app blocks
|
|
68
|
+
requests with a `429` once it goes over quota.
|
|
104
69
|
|
|
105
70
|
:::tip
|
|
106
71
|
|
|
@@ -115,18 +80,26 @@ toward the app's limits.
|
|
|
115
80
|
The mechanisms are evaluated independently and can both be active on the same
|
|
116
81
|
app:
|
|
117
82
|
|
|
118
|
-
- A request that
|
|
119
|
-
- A request that
|
|
83
|
+
- A request that's **over quota** routes to the quota fallback model.
|
|
84
|
+
- A request that's **within quota** but hits an **error or timeout** on the
|
|
120
85
|
primary routes to the error and timeout fallback model.
|
|
121
86
|
|
|
122
87
|
Set whichever fallbacks match the failure modes you want to protect against.
|
|
123
88
|
Neither is required.
|
|
124
89
|
|
|
90
|
+
:::note
|
|
91
|
+
|
|
92
|
+
Error and timeout fallback applies to Chat Completions and Embeddings requests.
|
|
93
|
+
Requests to the native `/v1/messages` and `/v1/responses` endpoints pass through
|
|
94
|
+
without retrying a backup model; the quota fallback still applies to them.
|
|
95
|
+
|
|
96
|
+
:::
|
|
97
|
+
|
|
125
98
|
## Related resources
|
|
126
99
|
|
|
127
|
-
- [
|
|
128
|
-
|
|
129
|
-
- [Usage Limits
|
|
130
|
-
|
|
100
|
+
- [Policy Chains](./policy-chains.mdx) - How the app's policy chain executes and
|
|
101
|
+
the recommended policy order.
|
|
102
|
+
- [Usage Limits](./usage-limits.mdx) - Configure the budget, token, and request
|
|
103
|
+
limits that trigger a quota fallback.
|
|
131
104
|
- [Custom Providers](./custom-providers.mdx) - Add your own provider to use as a
|
|
132
105
|
primary or fallback model.
|